Defunct git processes when running Gitea as PID 1 on docker

[jswolf19]

Gitea Version

1.15.7

Git Version

2.34.1

Operating System

OpenSuse Leap 15.3 (Docker)

How are you running Gitea?

I used an rpm built using the repo at https://build.opensuse.org/package/show/devel:tools:scm/gitea and installed with zypper.
I'm running in a docker container based on OpenSuse Leap 15.3, base os also OpenSuse Leap 15.3

Database

MySQL

Can you reproduce the bug on the Gitea demo site?

No

Log Gist

No response

Description

I have zombie processes whose start time coincides with log entries such as below:
Upon updating my github login credentials, the zombie processes seem to no longer occur.
#13987 is documented as being fixed by #14006 in the 1.14.0 milestone, so I'm reporting this as a possible regression. I'm sorry I haven't been able to upgrade to the latest version to check if the issue still exists. If you need any additional information, please let me know.

2022/03/14 15:22:27 ...irror/mirror_pull.go:176:runSync() [E] Failed to update mirror repository &{21 2 Towa <nil> purchasevisualization PurchaseVisualization 購入品管理  2 https://:xxxxxx@github.com/Towa-Japan/PurchaseVisualization.git main 4 0 1 0 0 0 0 0 0 0 0 0 0 0 0 false false false true <nil> [] 0 map[] map[] [] <nil> false 0 <nil> false 0 <nil> 103641 <nil> <nil> false false [] default  1610340108 1647238881}:
        Stdout: Fetching origin

        Stderr: remote: Repository not found.
        fatal: Authentication failed for 'https://github.com/Towa-Japan/PurchaseVisualization.git/'
        error: Could not fetch origin

        Err: exit status 1

Screenshots

No response

[lunny]

Please upgrade to v1.16.3

[jswolf19]

Please upgrade to v1.16.3

I hope to have the time soon. Does that mean that this is not an issue in 1.16.3?

[lunny]

If I'm not wrong, it should be.

[kiriharu]

Same issue on 1.16.2 in Docker

[lunny]

Same issue on 1.16.2 in Docker

Please upgrade to v1.16.5

[jswolf19]

I'm assuming I'll just be told to upgrade, but for what it's worth, the issue persists in v1.16.3. I'm also using docker, if that makes a difference.

[silentcodeg]

If I'm not wrong, it should be.

Could you please share why you think upgrading to 1.16.5 fixes the problem?

[zeripath]

The below was a misunderstanding when the title was zombie processes rather than defunct. #13987 related to stuck open git processes - whereas this case relates to processes that are dead but have not been reaped.

This should be finally fixed by #19454 and its backport #19466 which is in 1.16.7

Fixing this was extremely difficult before #19207 was merged because we were never provided with a simple reproducing case, logs or pprof data to figure out what is really happening.

[jswolf19]

@zeripath thanks for the info. I think I'm on 1.16.8 now, so when I get the time, I'll verify.

[jswolf19]

@zeripath The zombie (defunct) processes still get created at a roughly 10-20 minute interval with 1.16.8.

we were never provided with a simple reproducing case, logs or pprof data to figure out what is really happening.

If there's something I can provide, then I'm more than willing to, but I'm not sure what I can give you that will help, or how to get it. It's also not a major issue for me, because it only happens in error cases and I've just found out that running docker with the --init flag seems to at least clean up the zombies. In fact, it's entirely possible that the zombie processes are due to how git works, in which case using the --init flag may be the only solution to stop them.

However, the settings page of the mirror reports the latest update date as being the date of the last failed sync job, so it is not obvious that an error is occurring without comparing the two repos or looking at the execution logs of gitea for an error.

As best I can tell, the following steps should reproduce the issue (assuming a gitea instance running in a docker container):

1. create (or have access to) a private repo on github.
2. create a personal access token
3. create a mirror in gitea using the token created above.
4. regenerate the personal access token created above.
5. wait for the mirror sync timeout to expire.

After the above steps, every time the mirror update job runs in gitea, a zombie git process results (sample ps output)

~ > ps -ax -O lstart | grep defunct
 3204 Wed Jun  1 17:15:11 2022 Z ?        00:00:00 git <defunct>
 3618 Wed Jun  1 16:25:11 2022 Z ?        00:00:00 git <defunct>
 9033 Wed Jun  1 16:35:11 2022 Z ?        00:00:00 git <defunct>
 9666 Wed Jun  1 09:56:35 2022 Z ?        00:00:00 git <defunct>
10932 Wed Jun  1 17:25:11 2022 Z ?        00:00:00 git <defunct>
15320 Wed Jun  1 17:35:11 2022 Z ?        00:00:00 git <defunct>
15877 Wed Jun  1 16:45:11 2022 Z ?        00:00:00 git <defunct>
20352 Wed Jun  1 17:45:11 2022 Z ?        00:00:00 git <defunct>
21406 Wed Jun  1 16:05:11 2022 Z ?        00:00:00 git <defunct>
22423 Wed Jun  1 17:55:11 2022 Z ?        00:00:00 git <defunct>
23985 Wed Jun  1 16:55:11 2022 Z ?        00:00:00 git <defunct>

and an error similar to below is generated in the log file

2022/06/01 16:55:13 ...irror/mirror_pull.go:256:runSync() [E] SyncMirrors [repo: 60:jon_s/TowaIoT]: failed to update mirror  repository:  <-- pid 23985
        Stdout: Fetching origin

        Stderr: remote: Invalid username or password.
        fatal: Authentication failed for 'https://github.com/Towa-Japan/TowaIoT.git/'
        error: could not fetch origin

        Err: exit status 1
2022/06/01 17:05:13 ...irror/mirror_pull.go:256:runSync() [E] SyncMirrors [repo: 60:jon_s/TowaIoT]: failed to update mirror  repository: <-- no zombie?
        Stdout: Fetching origin

        Stderr: remote: Invalid username or password.
        fatal: Authentication failed for 'https://github.com/Towa-Japan/TowaIoT.git/'
        error: could not fetch origin

        Err: exit status 1
2022/06/01 17:15:13 ...irror/mirror_pull.go:256:runSync() [E] SyncMirrors [repo: 60:jon_s/TowaIoT]: failed to update mirror repository: <-- pid 3204
        Stdout: Fetching origin

        Stderr: remote: Invalid username or password.
        fatal: Authentication failed for 'https://github.com/Towa-Japan/TowaIoT.git/'
        error: could not fetch origin

        Err: exit status 1

[zeripath]

Ah so you think that these defunct processes are related to SyncMirrors? I suspect that these defunct processes are actually because of git created child processes.

Does this happen on 1.17 too? What is your version of Git?

(I wonder if it's possibly related to a version of Git?)

I have one further possible solution - we could try to set a process attribute Setpgid: true

Could you apply the below patch?

diff --git a/modules/git/command.go b/modules/git/command.go
index 3dd12e421..b3266ce99 100644
--- a/modules/git/command.go
+++ b/modules/git/command.go
@@ -13,6 +13,7 @@ import (
 	"os"
 	"os/exec"
 	"strings"
+	"syscall"
 	"time"
 	"unsafe"
 
@@ -157,6 +158,7 @@ func (c *Command) Run(opts *RunOpts) error {
 		"GIT_NO_REPLACE_OBJECTS=1",
 	)
 
+	cmd.SysProcAttr = &syscall.SysProcAttr{Setpgid: true}
 	cmd.Dir = opts.Dir
 	cmd.Stdout = opts.Stdout
 	cmd.Stderr = opts.Stderr

and see if that prevents these defunct processes?

[singuliere]

It would help to know if the zombie processes are orphans or if their parent is the Gitea process. Running something like ps -o pid,ppid,comm,args within the container will show the PPID (parent process id) as well as the PID.

If the git process is indeed an orphan, setting the process group id as @zeripath suggests will help. The Gitea children then becomes a process group leader and killing it will also kill all its children instead of not.

The system call setpgid is used to set the process group ID of a process, thereby either joining the process to an existing process group, or creating a new process group within the session of the process with the process becoming the process group leader of the newly created group.[5] POSIX prohibits the re-use of a process ID where a process group with that identifier still exists (i.e. where the leader of a process group has exited, but other processes in the group still exist). It thereby guarantees that processes may not accidentally become process group leaders.

That being said I'd be surprised that it is the cause. A process is a zombie when the parent fails to wait(2). And it is more likely that said process is Gitea itself.

The reproducer you provide is a good one 👍 and I'll give it a try.