Gitea keeping taking 100% CPU usage, and runs slowly in git pushing

[duchenpaul]

• Gitea version (or commit ref): 1.12.0+dev-440-gf5a20250a

• Git version: git version 2.20.1

• Operating system: Raspbian GNU/Linux 10 (buster) on Raspberry Pi zero w

• Database (use [x]):

  • PostgreSQL
  • MySQL
  • MSSQL
  • SQLite

• Can you reproduce the bug at https://try.gitea.io:

  • Yes (provide example URL)
  • No
  • Not relevant

• Log gist:

• Architecture: linux-arm-6

Description

After upgrading the gitea to this version, I found that gitea runs very slow.
By checking the CPU usage, the CPU usage keeps 100%.
And I checked the ps, found that seems the gitea hook processes (2395, 964) are taking up lots resource, I would like to know what are they and how can I track them, and how to stop them.
As far as I know, there is no git hook setting in my repo that runs on this server. So I think this could be the gitea's system hook.
And I see PID 2366 saying duchenpaul/homebridge-docker.git is executing hook, but there is no hook script set in that repo
As the PID 961, I dont know what is going on there.

I checked the log, only the sql that queried in the log, no errors

  928 ?        Ss     0:00  \_ sshd: git [priv]
  937 ?        S      0:00  |   \_ sshd: git@notty
  939 ?        Rsl   14:49  |       \_ /home/git/gitea/gitea serv key-3 --config=/home/git/gitea/custom/conf/app.ini
  987 ?        Ss     0:00  \_ sshd: git [priv]
  996 ?        S      0:00  |   \_ sshd: git@notty
  998 ?        Rsl   12:43  |       \_ /home/git/gitea/gitea serv key-3 --config=/home/git/gitea/custom/conf/app.ini
 1310 ?        Ss     0:00  \_ sshd: git [priv]
 1319 ?        S      0:00  |   \_ sshd: git@notty
 1321 ?        Rsl   10:04  |       \_ /home/git/gitea/gitea serv key-3 --config=/home/git/gitea/custom/conf/app.ini
 2062 ?        Ss     0:00  \_ sshd: pi [priv]
 2071 ?        S      0:02  |   \_ sshd: pi@pts/1
 2074 pts/1    Ss     0:00  |       \_ -bash
 2482 pts/1    S+     0:00  |           \_ vi gitea.log
 2349 ?        Ss     0:00  \_ sshd: git [priv]
 2358 ?        S      0:00      \_ sshd: git@notty
 2360 ?        Ssl    0:03          \_ /home/git/gitea/gitea serv key-3 --config=/home/git/gitea/custom/conf/app.ini
 2366 ?        Sl     0:00              \_ git-receive-pack duchenpaul/homebridge-docker.git
 2390 ?        S      0:00                  \_ bash hooks/post-receive
 2394 ?        S      0:00                      \_ bash ./hooks/post-receive.d/gitea
 2395 ?        Rl     4:55                          \_ /home/git/gitea/gitea hook --config=/home/git/gitea/custom/conf/app.ini post-receive
  454 tty1     Ss+    0:00 /sbin/agetty -o -p -- \u --noclear tty1 linux
  525 ?        Ss     0:00 php-fpm: master process (/etc/php5/fpm/php-fpm.conf)
  619 ?        S      0:00  \_ php-fpm: pool www
  620 ?        S      0:00  \_ php-fpm: pool www
  586 ?        Ss     0:00 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
  590 ?        S      0:00  \_ nginx: worker process
  654 ?        Ss     0:00 /lib/systemd/systemd --user
  680 ?        S      0:00  \_ (sd-pam)
  700 ?        Ss     0:00 /lib/systemd/systemd --user
  723 ?        S      0:00  \_ (sd-pam)
  753 ?        Ss     0:00 /usr/sbin/exim4 -bd -q30m
  961 ?        S      0:00 bash hooks/update refs/heads/master 2b1fb9ae3617e7fc0b9b496897b542e1e3e24375 9a610f1bb0fdd659a4fa5ef7576ff3dc27690f22
  963 ?        S      0:00  \_ bash ./hooks/update.d/gitea refs/heads/master 2b1fb9ae3617e7fc0b9b496897b542e1e3e24375 9a610f1bb0fdd659a4fa5ef7576ff3dc27690f22
  964 ?        Rl    14:45      \_ /home/git/gitea/gitea hook --config=/home/git/gitea/custom/conf/app.ini update refs/heads/master 2b1fb9ae3617e7fc0b9b496897b542e1e3e24375 9a610f1bb0fdd659a4fa5ef7576ff3dc27690f22
 2175 ?        Ssl    1:49 /home/git/gitea/gitea web

[Update] Summarize

• The hanging of gitea serv key-3 --config=/home/git/gitea/custom/conf/app.ini is introduced from git client's requests, for some reason, some git request did not finishes properly, instead, taking up all the CPU for hours and slow down the system.
• And these process does not related to the gitea main service, even i stops the gitea service, these process is still hanging there.
• It looks the bug started from v1.11.2 (0768823) in the release, I am using v1.11.1 (ff24f81), it is ok, but as long as I switch to v1.11.2, above thing start to happen in a short time.

Screenshots

[duchenpaul]

Also, I observed that there are 3 processes called gitea serv key-3 --config=/home/git/gitea/custom/conf/app.ini which taking up the most of CPU usage, what are they? How can it be stopped

[guillep2k]

Those three processes (939, 998 and 1321) sound like the server side of git clients. The fact that all of them have key-3 means that they all used the same credentials (public key). It's probably you trying to push, then cancelling, but for some reason the server side didn't realize about that yet.

But the culprit seems to be:

  964 ?        Rl    14:45      \_ /home/git/gitea/gitea hook --config=/home/git/gitea/custom/conf/app.ini update refs/heads/master 2b1fb9ae3617e7fc0b9b496897b542e1e3e24375 9a610f1bb0fdd659a4fa5ef7576ff3dc27690f22

which is Gitea's update hook on the repository. The 14:45 value is the accumulated CPU time used, which is not normal unless you're pushing on the Linux kernel source code in an old Pentium computer.

And yes, those hooks are Gitea's git plumbing that make things work behind the scenes.

Some ideas:

• Try rebooting. Seriously. If for some fortuitous reason some process hanged on you but it's still locking some files, this will kill it.
• Try doing a git fsck and a git gc on the repo via the admin menu. It's possible that there's something wrong with it.

Also, a Raspberry Pi Zero W doesn't have much RAM in it. Try checking whether your system is paging too much. Use the top program and hit the m key until the memory graph shows up:

Or maybe vmstat 3 999 (you cancel that with CTRL+C):

The si and so columns should be 0 most of the time. Paging in and out is something that really slows down the system.

(Note: don't take the values in my pictures as expected for your system: mine is a CentOS 7 VM with 12GB of RAM assigned to it).

[duchenpaul]

Thank you for your replying,
I did a reboot again, looks ok for now.
I didn't cancel any transactions in-fight, from what I see for client side, all transactions are finished normally.
I believe PID 964 I referring to is blocked due to the high CPU usage of ssh transactions

My questions are:

• Is there are timeout settings that cancel the in-fight ssh transactions from server side which taking too much time?
• And is there a way to track back what is going on with those ssh transactions.

anyway, I will keep monitoring for 3 days, if it is ok, I will close this issue.

This gitea server is only serving myself, I think it still is able to handle the job well. 😄

output for vmstat 3 999

procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  0   4036  68112  28512 141908    0    5   523    45  576 1210 17  9 72  2  0
 0  0   4036  68112  28512 141908    0    0     0     0  434  877  4  3 93  0  0
 0  0   4036  68112  28512 141908    0    0     0     0  405  856  6  1 93  0  0
 1  0   4036  68112  28512 141908    0    0     0     0  454  917  4  2 94  0  0
 3  0   4036  68112  28512 141908    0    0     0     0  532 1086  7  2 91  0  0

[duchenpaul]

[Update] I found these 2 gitea serv process pops up again, while I didn't even do anything.
Anyone can help me find out who caused this?

pi@Venus ~ $ vmstat 3 999
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 4  0  16172  84076  11680 106668    0    1    42     5  548  996 84  3 13  0  0
 3  0  16172  84076  11680 106668    0    0     0     0  554  958 99  1  0  0  0
 3  0  16172  84076  11680 106668    0    0     0     0  582 1027 96  4  0  0  0
 4  0  16172  84076  11680 106668    0    0     0     0  704 1296 98  2  0  0  0
 3  0  16172  84076  11680 106668    0    0     0     0  561 1015 97  3  0  0  0
 4  0  16172  84076  11680 106668    0    0     0     0  648 1170 97  3  0  0  0

[duchenpaul]

Close for now, I found it could be my github desktop causing this. I change to GitExtensions, it never hangs.

[lunny]

Some git tools may check repo status on background.

[duchenpaul]

Yes, you can tell from the log of github desktop, but how come the process stalls on the server side, and taking up all cpu resource, even the client is quit, the client end PC is powered off.

[zeripath]

Not sure - we would need to see some logs from gitea as to what it's doing.

For example the update hook should just be basically empty - I've suggested it's removal entirely as it only can slow things down.

[duchenpaul]

Yeah, I will reopen this issue, and set my gitea log level to debug to collect more info.
I didn't use update hook in any repo. only the post-receive one in a few repos just to deploy the code on remote server.
Now I found that even gitea dump can stall sometime, just hanging there and put the CPU in full throttle.

[duchenpaul]

This is my log conf for debugging:

[log]
MODE      = file
LEVEL     = Debug
ROOT_PATH = /home/git/gitea/log
REDIRECT_MACARON_LOG = true
STACKTRACE_LEVEL = Debug
ENABLE_ACCESS_LOG = true
ENABLE_XORM_LOG = true
XORM = ,

is there any thing to add to help locate the problem?

[duchenpaul]

It got stall again.
The PID 8480 takes 100% CPU usage. Even the gitea service is stoped by me.

 8469 ?        Ss     0:00  \_ sshd: git [priv]
 8478 ?        S      0:00  |   \_ sshd: git@notty
 8480 ?        Rsl    5:04  |       \_ /home/git/gitea/gitea --config=/home/git/gitea/custom/conf/app.ini serv key-6
 8486 ?        Ss     0:00  \_ sshd: pi [priv]
 8495 ?        S      0:00      \_ sshd: pi@pts/3
 8498 pts/3    Ss     0:00          \_ -bash
 8637 pts/3    R+     0:00              \_ ps xfa
  452 tty1     Ss+    0:00 /sbin/agetty -o -p -- \u --noclear tty1 linux
  568 ?        Ss     0:08 php-fpm: master process (/etc/php5/fpm/php-fpm.conf)
  653 ?        S      0:00  \_ php-fpm: pool www
  656 ?        S      0:00  \_ php-fpm: pool www
  633 ?        Ss     0:00 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
  637 ?        S      0:00  \_ nginx: worker process
  725 ?        Ss     0:00 /usr/sbin/exim4 -bd -q30m
 7762 ?        Ss     0:00 /lib/systemd/systemd --user
 7766 ?        S      0:00  \_ (sd-pam)
 8334 ?        Ss     0:00 /lib/systemd/systemd --user
 8337 ?        S      0:00  \_ (sd-pam)
[ 09:52 - 192.168.31.150  ]
pi@Venus ~ $ ps -o lstart -o etime -p 8480
                 STARTED     ELAPSED
Mon Mar  9 09:46:14 2020       05:59
[ 09:52 - 192.168.31.150  ]
pi@Venus ~ $ 

This process starts at 09:46:14, I checked each log around that time, what I find is only Started Session c187 of user git. which my git client is tring to update the repo in background

attached the logs I can find below.

syslog.log
access.log
auth.log
daemon.log
gitea.log

[guillep2k]

Have you tried running the fsck and gc? Those are important, given that you're using an SD-card to store the repos. If you are unable to do it via Gitea (e.g. system gets locked up before you get the chance), you can shutdown Gitea and run the commands yourself:

cd /home/git/gitea-repositories
( for user in * ; do cd "$user" ; echo "--- USER: $user ---"
    for repo in * ; do cd $repo ; echo "--- REPO: $repo ---"
        git fsck && git gc || exit
    cd .. ; done ; cd .. ; done )

(Change the initial path accordingly if required)