Out of disk space on dockerized shared hosts

[richardlau]

Reported via Slack:

From @Trott :

Example

10:51:23  > git config remote.origin.url git@github.com:nodejs/node.git # timeout=10
10:51:23 ERROR: Error fetching remote repo 'origin'
10:51:23 hudson.plugins.git.GitException: Failed to fetch from git@github.com:nodejs/node.git
10:51:23 	at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:996)
10:51:23 	at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1237)

Different example

11:01:46 collect2: fatal error: ld terminated with signal 9 [Killed]
11:01:46 compilation terminated.
11:01:46 cctest.target.mk:231: recipe for target '/home/iojs/build/workspace/node-test-commit-linux-> containered/out/Debug/cctest' failed
11:01:46 make[2]: *** [/home/iojs/build/workspace/node-test-commit-linux-containered/out/Debug/cctest] Error 1
11:01:46 make[2]: *** Waiting for unfinished jobs....
11:02:58 rm fa309b4689a758e9e8a16895fbbf2b4922a45c96.intermediate
11:02:58 Makefile:104: recipe for target 'node_g' failed
11:02:58 make[1]: *** [node_g] Error 2
11:02:58 Makefile:530: recipe for target 'build-ci' failed
11:02:58 make: *** [build-ci] Error 2

From @danielleadams :

I’m running into some issues with the pull-request jobs today (releasing 15.4.0). Different tests seem to be raising the same error with git, and they are inconsistently failing:
https://ci.nodejs.org/job/node-test-commit-linux-containered/nodes=ubuntu1804_sharedlibs_debug_x64/23997/console
https://ci.nodejs.org/job/node-test-commit-linux-containered/nodes=ubuntu1804_sharedlibs_withoutssl_x64/23997/console
https://ci.nodejs.org/job/node-test-commit-linux-containered/nodes=ubuntu1804_sharedlibs_openssl110_x64/23999/console
https://ci.nodejs.org/job/node-test-commit-linux-containered/nodes=ubuntu1804_sharedlibs_openssl111_x64/24008/
https://ci.nodejs.org/job/node-test-commit-linux-containered/nodes=ubuntu1804_sharedlibs_withoutssl_x64/24008/console
https://ci.nodejs.org/job/node-test-commit-linux/nodes=alpine-latest-x64/38744/console
Does anyone know how to address this?

[richardlau]

It looks like the disks are full. e.g. from https://ci.nodejs.org/job/node-test-commit-linux-containered/24011/nodes=ubuntu1804_sharedlibs_withoutssl_x64/console the relevant lines are:

18:51:23 Caused by: hudson.plugins.git.GitException: Command "git config remote.origin.url git@github.com:nodejs/node.git" returned status code 4:
18:51:23 stdout: 
18:51:23 stderr: error: failed to write new configuration file /home/iojs/build/workspace/node-test-commit-linux-containered/.git/config.lock
18:51:23 
...
18:51:23 FATAL: Unable to produce a script file
18:51:23 java.io.IOException: No space left on device
18:51:23 	at java.io.UnixFileSystem.createFileExclusively(Native Method)

root@test-softlayer-ubuntu1804-docker-x64-1:/home/iojs# df -h .
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvda2       99G   94G     0 100% /
root@test-softlayer-ubuntu1804-docker-x64-1:/home/iojs# du -hs test-softlayer-*
4.0G	test-softlayer-alpine310_container-x64-1
2.3G	test-softlayer-alpine311_container-x64-1
2.3G	test-softlayer-alpine312_container-x64-1
3.7G	test-softlayer-alpine39_container-x64-1
2.2G	test-softlayer-ubi81_container-x64-1
2.5G	test-softlayer-ubuntu1604_arm_cross_container-x64-1
6.6G	test-softlayer-ubuntu1804_arm_cross_container-x64-1
208M	test-softlayer-ubuntu1804_container-x64-1
2.1G	test-softlayer-ubuntu1804_sharedlibs_container-x64-1
16G	test-softlayer-ubuntu1804_sharedlibs_container-x64-2
14G	test-softlayer-ubuntu1804_sharedlibs_container-x64-3
2.3G	test-softlayer-ubuntu1804_sharedlibs_container-x64-4
2.5G	test-softlayer-ubuntu1804_sharedlibs_container-x64-5
root@test-softlayer-ubuntu1804-docker-x64-1:/home/iojs#

I'll try removing the 14G/16G workspaces.

[richardlau]

I've cleaned up the workspaces for test-softlayer-ubuntu1804_sharedlibs_container-x64-2 and test-softlayer-ubuntu1804_sharedlibs_container-x64-3 to free up 30G of space.

[rvagg]

feel free to clean out all of the workspaces, it's not a huge slowdown to recreate them here (like it is on the Pi hosts)

[richardlau]

Out of space again this morning:

root@test-softlayer-ubuntu1804-docker-x64-1:/home/iojs# df .
Filesystem     1K-blocks     Used Available Use% Mounted on
/dev/xvda2     102821812 97779460    582800 100% /
root@test-softlayer-ubuntu1804-docker-x64-1:/home/iojs# du -hs test-softlayer-*
4.0G    test-softlayer-alpine310_container-x64-1
1.6G    test-softlayer-alpine311_container-x64-1
2.3G    test-softlayer-alpine312_container-x64-1
3.7G    test-softlayer-alpine39_container-x64-1
2.2G    test-softlayer-ubi81_container-x64-1
2.5G    test-softlayer-ubuntu1604_arm_cross_container-x64-1
6.6G    test-softlayer-ubuntu1804_arm_cross_container-x64-1
208M    test-softlayer-ubuntu1804_container-x64-1
1.8G    test-softlayer-ubuntu1804_sharedlibs_container-x64-1
15G     test-softlayer-ubuntu1804_sharedlibs_container-x64-2
16G     test-softlayer-ubuntu1804_sharedlibs_container-x64-3
2.3G    test-softlayer-ubuntu1804_sharedlibs_container-x64-4
2.4G    test-softlayer-ubuntu1804_sharedlibs_container-x64-5
root@test-softlayer-ubuntu1804-docker-x64-1:/home/iojs#

So the 15G workpaces are back and look to be Debug builds. I don't know what the typical expected workspace size is -- whether the sizes have been creeping up slowly over time or if there's been a recent change to cause a jump.

FWIW

root@test-softlayer-ubuntu1804-docker-x64-1:/home/iojs/test-softlayer-ubuntu1804_sharedlibs_container-x64-3/build/workspace# du -hs *
16G     node-test-commit-linux-containered
4.0K    node-test-commit-linux-containered@tmp
root@test-softlayer-ubuntu1804-docker-x64-1:/home/iojs/test-softlayer-ubuntu1804_sharedlibs_container-x64-3/build/workspace# cd node-test-commit-linux-containered
root@test-softlayer-ubuntu1804-docker-x64-1:/home/iojs/test-softlayer-ubuntu1804_sharedlibs_container-x64-3/build/workspace/node-test-commit-linux-containered# du -hs *
120K    AUTHORS
4.0K    BSDmakefile
32K     BUILDING.md
56K     CHANGELOG.md
4.0K    CODE_OF_CONDUCT.md
4.0K    CONTRIBUTING.md
8.0K    GOVERNANCE.md
84K     LICENSE
48K     Makefile
32K     README.md
4.0K    SECURITY.md
4.0K    android-configure
1.5M    benchmark
4.0K    codecov.yml
20K     common.gypi
4.0K    config.gypi
4.0K    config.mk
4.0K    config.status
4.0K    configure
72K     configure.py
56K     configure.pyc
386M    deps
9.9M    doc
4.0K    env.properties
4.0K    glossary.md
72K     icu_config.gypi
2.9M    lib
0       node
52K     node.gyp
12K     node.gypi
0       node_g
16K     onboarding.md
15G     out
4.9M    src
46M     test
248K    test.tap
35M     tools
32K     vcbuild.bat
root@test-softlayer-ubuntu1804-docker-x64-1:/home/iojs/test-softlayer-ubuntu1804_sharedlibs_container-x64-3/build/workspace/node-test-commit-linux-containered# cd out/
root@test-softlayer-ubuntu1804-docker-x64-1:/home/iojs/test-softlayer-ubuntu1804_sharedlibs_container-x64-3/build/workspace/node-test-commit-linux-containered/out# du -hs *
14G     Debug
36K     Makefile
854M    Release
12K     cctest.target.mk
156K    deps
12K     embedtest.target.mk
8.0K    fuzz_env.target.mk
8.0K    fuzz_url.target.mk
240K    junit
40K     libnode.target.mk
12K     mkcodecache.target.mk
16K     node.target.mk
4.0K    node_dtrace_header.target.mk
4.0K    node_dtrace_provider.target.mk
4.0K    node_dtrace_ustack.target.mk
4.0K    node_etw.target.mk
12K     node_mksnapshot.target.mk
4.0K    node_text_start.target.mk
4.0K    overlapped-checker.target.mk
4.0K    specialize_node_d.target.mk
508K    tools
root@test-softlayer-ubuntu1804-docker-x64-1:/home/iojs/test-softlayer-ubuntu1804_sharedlibs_container-x64-3/build/workspace/node-test-commit-linux-containered/out# cd Debug/
root@test-softlayer-ubuntu1804-docker-x64-1:/home/iojs/test-softlayer-ubuntu1804_sharedlibs_container-x64-3/build/workspace/node-test-commit-linux-containered/out/Debug# du -hs *
1.7M    bytecode_builtins_list_generator
1.1G    cctest
1.1G    embedtest
34M     gen-regexp-special-case
15M     genccode
16M     icupkg
1.1G    mkcodecache
1.4G    mksnapshot
1.1G    node
1.1G    node_mksnapshot
154M    obj
62M     obj.host
6.5G    obj.target
9.9M    openssl-cli
16K     overlapped-checker
39M     torque
root@test-softlayer-ubuntu1804-docker-x64-1:/home/iojs/test-softlayer-ubuntu1804_sharedlibs_container-x64-3/build/workspace/node-test-commit-linux-containered/out/Debug# ls -alh
total 6.8G
drwxr-xr-x  6 iojs iojs 4.0K Dec  8 19:53 .
drwxr-xr-x  7 iojs iojs 4.0K Dec  8 20:03 ..
drwxr-xr-x  3 iojs iojs 4.0K Dec  8 19:50 .deps
-rwxr-xr-x  1 iojs iojs 1.7M Dec  8 19:38 bytecode_builtins_list_generator
-rwxr-xr-x  1 iojs iojs 1.1G Dec  8 19:53 cctest
-rwxr-xr-x  1 iojs iojs 1.1G Dec  8 19:53 embedtest
-rwxr-xr-x  1 iojs iojs  34M Dec  8 19:40 gen-regexp-special-case
-rwxr-xr-x  1 iojs iojs  15M Dec  8 19:38 genccode
-rwxr-xr-x  1 iojs iojs  16M Dec  8 19:38 icupkg
-rwxr-xr-x  1 iojs iojs 1.1G Dec  8 19:53 mkcodecache
-rwxr-xr-x  1 iojs iojs 1.4G Dec  8 19:50 mksnapshot
-rwxr-xr-x  1 iojs iojs 1.1G Dec  8 19:54 node
-rwxr-xr-x  1 iojs iojs 1.1G Dec  8 19:53 node_mksnapshot
drwxr-xr-x  3 iojs iojs 4.0K Dec  8 19:37 obj
drwxr-xr-x  6 iojs iojs 4.0K Dec  8 19:38 obj.host
drwxr-xr-x 39 iojs iojs 4.0K Dec  8 19:53 obj.target
-rwxr-xr-x  1 iojs iojs 9.9M Dec  8 19:38 openssl-cli
-rwxr-xr-x  1 iojs iojs  16K Dec  8 19:37 overlapped-checker
-rwxr-xr-x  1 iojs iojs  39M Dec  8 19:38 torque
root@test-softlayer-ubuntu1804-docker-x64-1:/home/iojs/test-softlayer-ubuntu1804_sharedlibs_container-x64-3/build/workspace/node-test-commit-linux-containered/out/Debug#

[richardlau]

I was waiting for the inflight https://ci.nodejs.org/job/node-test-commit-linux-containered/ jobs to complete before manually removing things but it looks like the two most recent builds passed. The current disk space usage with nothing running looks like this:

root@test-softlayer-ubuntu1804-docker-x64-1:/home/iojs# df -h .
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvda2       99G   82G   13G  87% /
root@test-softlayer-ubuntu1804-docker-x64-1:/home/iojs# du -hs *
4.0G    test-softlayer-alpine310_container-x64-1
1.6G    test-softlayer-alpine311_container-x64-1
2.3G    test-softlayer-alpine312_container-x64-1
3.7G    test-softlayer-alpine39_container-x64-1
2.2G    test-softlayer-ubi81_container-x64-1
2.5G    test-softlayer-ubuntu1604_arm_cross_container-x64-1
6.6G    test-softlayer-ubuntu1804_arm_cross_container-x64-1
208M    test-softlayer-ubuntu1804_container-x64-1
1.8G    test-softlayer-ubuntu1804_sharedlibs_container-x64-1
2.3G    test-softlayer-ubuntu1804_sharedlibs_container-x64-2
16G     test-softlayer-ubuntu1804_sharedlibs_container-x64-3
2.3G    test-softlayer-ubuntu1804_sharedlibs_container-x64-4
2.4G    test-softlayer-ubuntu1804_sharedlibs_container-x64-5
root@test-softlayer-ubuntu1804-docker-x64-1:/home/iojs#

i.e. we only have one 16G (i.e. Debug build) workspace. We're probably seeing flaky behaviour if more than one https://ci.nodejs.org/job/node-test-commit-linux-containered/ is running and we end up with two inflight Debug builds which fills up the disk.

[richardlau]

Disk full again, with 15G workspaces on two hosts (suggesting we had concurrent debug builds again):

root@test-softlayer-ubuntu1804-docker-x64-1:/home/iojs# df -h .
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvda2       99G   94G  298M 100% /
root@test-softlayer-ubuntu1804-docker-x64-1:/home/iojs# du -hs *
4.0G    test-softlayer-alpine310_container-x64-1
1.6G    test-softlayer-alpine311_container-x64-1
2.3G    test-softlayer-alpine312_container-x64-1
3.7G    test-softlayer-alpine39_container-x64-1
2.2G    test-softlayer-ubi81_container-x64-1
2.5G    test-softlayer-ubuntu1604_arm_cross_container-x64-1
6.6G    test-softlayer-ubuntu1804_arm_cross_container-x64-1
208M    test-softlayer-ubuntu1804_container-x64-1
1.8G    test-softlayer-ubuntu1804_sharedlibs_container-x64-1
15G     test-softlayer-ubuntu1804_sharedlibs_container-x64-2
16G     test-softlayer-ubuntu1804_sharedlibs_container-x64-3
2.3G    test-softlayer-ubuntu1804_sharedlibs_container-x64-4
2.4G    test-softlayer-ubuntu1804_sharedlibs_container-x64-5
root@test-softlayer-ubuntu1804-docker-x64-1:

feel free to clean out all of the workspaces, it's not a huge slowdown to recreate them here (like it is on the Pi hosts)

I've gone and wiped all the workspaces.

root@test-softlayer-ubuntu1804-docker-x64-1:/home/iojs# ls -1 | xargs -i bash -c "rm -rf {}/build/workspace/*"
root@test-softlayer-ubuntu1804-docker-x64-1:/home/iojs# du -hs *
162M    test-softlayer-alpine310_container-x64-1
199M    test-softlayer-alpine311_container-x64-1
191M    test-softlayer-alpine312_container-x64-1
146M    test-softlayer-alpine39_container-x64-1
218M    test-softlayer-ubi81_container-x64-1
173M    test-softlayer-ubuntu1604_arm_cross_container-x64-1
173M    test-softlayer-ubuntu1804_arm_cross_container-x64-1
208M    test-softlayer-ubuntu1804_container-x64-1
271M    test-softlayer-ubuntu1804_sharedlibs_container-x64-1
277M    test-softlayer-ubuntu1804_sharedlibs_container-x64-2
269M    test-softlayer-ubuntu1804_sharedlibs_container-x64-3
270M    test-softlayer-ubuntu1804_sharedlibs_container-x64-4
265M    test-softlayer-ubuntu1804_sharedlibs_container-x64-5
root@test-softlayer-ubuntu1804-docker-x64-1:/home/iojs# df -h .
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvda2       99G   38G   57G  40% /
root@test-softlayer-ubuntu1804-docker-x64-1:/home/iojs#

[richardlau]

Cleaned up test-softlayer-ubuntu1804-docker-x64-1 again:
before:

root@test-softlayer-ubuntu1804-docker-x64-1:~# df -h /home/iojs/
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvda2       99G   94G   97M 100% /
root@test-softlayer-ubuntu1804-docker-x64-1:~# du -hs /home/iojs/*
162M    /home/iojs/test-softlayer-alpine310_container-x64-1
1.8G    /home/iojs/test-softlayer-alpine311_container-x64-1
2.1G    /home/iojs/test-softlayer-alpine312_container-x64-1
146M    /home/iojs/test-softlayer-alpine39_container-x64-1
2.2G    /home/iojs/test-softlayer-ubi81_container-x64-1
2.0G    /home/iojs/test-softlayer-ubuntu1604_arm_cross_container-x64-1
11G     /home/iojs/test-softlayer-ubuntu1804_arm_cross_container-x64-1
246M    /home/iojs/test-softlayer-ubuntu1804_container-x64-1
2.4G    /home/iojs/test-softlayer-ubuntu1804_sharedlibs_container-x64-1
17G     /home/iojs/test-softlayer-ubuntu1804_sharedlibs_container-x64-2
16G     /home/iojs/test-softlayer-ubuntu1804_sharedlibs_container-x64-3
2.5G    /home/iojs/test-softlayer-ubuntu1804_sharedlibs_container-x64-4
2.3G    /home/iojs/test-softlayer-ubuntu1804_sharedlibs_container-x64-5
root@test-softlayer-ubuntu1804-docker-x64-1:~#

[Trott]

Any chance something similar is going on with #2588?

[richardlau]

Possibly? Maybe bring test-softlayer-ubuntu1804_sharedlibs_container-x64-3 back online and see if builds still fail on it?

[Trott]

Hmmm, looks like it's already back online.

[rvagg]

probably related to an ld failure, likely all goes back to #2573

[mhdawson]

@richardlau is the cleanup that originated this issues something that we might enable build helpers to be able to do?