cmd/compile: killed while compiling gogoprotobuf

[gpaul]

1. What version of Go are you using (go version)?
   go version devel +ba6765c Wed May 4 05:55:07 2016 +0000 linux/amd64
2. What operating system and processor architecture are you using (go env)?
   linux x86_64
3. What did you do?

mkdir /tmp/gopath
cd $!
export GOPATH=`pwd`
go get -d -v github.com/gogo/protobuf
go install -v ...

1. What did you expect to see?
   should succeed
2. What did you see instead?
   a couple of signal: killed errors:

github.com/gogo/protobuf/test/casttype/combos/unsafeunmarshaler
go build github.com/gogo/protobuf/test: /home/gustav/go/pkg/tool/linux_amd64/compile: signal: killed
github.com/gogo/protobuf/test/castvalue
go build github.com/gogo/protobuf/test/casttype/combos/neither: /home/gustav/go/pkg/tool/linux_amd64/compile: signal: killed

This is on a pretty decent laptop with 16GB of RAM and is doing little else:

$ free -g
              total        used        free      shared  buff/cache   available
Mem:             15           1          13           0           0          13
Swap:             0           0           0

[bradfitz]

What's your ulimit -a?

What does time go install -v .. say?

[gpaul]

$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 63873
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 63873
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Sometimes strace shows failure with ENOMEM, other times I just see SIGKILLs as the oom-killer steps in.

dmesg shows the following (oomkiller)

[Wed May  4 17:04:11 2016] [ pid ]   uid  tgid total_vm      rss nr_ptes nr_pmds swapents oom_score_adj name
...
[Wed May  4 17:04:11 2016] [ 9770]     0  9770    93361    26797      90       3        0             0 perf
[Wed May  4 17:04:11 2016] [13022]  1000 13022     1828       92       9       3        0             0 strace
[Wed May  4 17:04:11 2016] [13024]  1000 13024    75524     2265      36       6        0             0 go
[Wed May  4 17:04:11 2016] [13123]  1000 13123   661874   518769    1140       7        0             0 compile
[Wed May  4 17:04:11 2016] [13131]  1000 13131   662488   521653    1146       7        0             0 compile
[Wed May  4 17:04:11 2016] [13132]  1000 13132   597042   437581    1005       7        0             0 compile
[Wed May  4 17:04:11 2016] [13143]  1000 13143   685774   497712    1099       7        0             0 compile
[Wed May  4 17:04:11 2016] [13144]  1000 13144   588702   389245     897       7        0             0 compile
[Wed May  4 17:04:11 2016] [13154]  1000 13154   661903   522499    1170       7        0             0 compile
[Wed May  4 17:04:11 2016] [13157]  1000 13157   661655   518735    1141       8        0             0 compile
[Wed May  4 17:04:11 2016] [14667]  1000 14667    30977    27699      64       5        0             0 compile
[Wed May  4 17:04:11 2016] [14713]  1000 14713    91364     1211      30       5        0             0 go
[Wed May  4 17:04:11 2016] Out of memory: Kill process 13154 (compile) score 127 or sacrifice child
[Wed May  4 17:04:11 2016] Killed process 13154 (compile) total-vm:2647612kB, anon-rss:2089996kB, file-rss:0kB

[gpaul]

Specifically, strace sometimes shows the following kind of failure:

17263 mmap(0xc8a5dc0000, 256507904, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = -1 ENOMEM (Cannot allocate memory)
17263 mincore(0xc8a5dc0000, 4096, 0xb104a9) = -1 ENOMEM (Cannot allocate memory)
17263 mincore(0xc8a5dc1000, 4096, 0xb104a9) = -1 ENOMEM (Cannot allocate memory)
17263 mincore(0xc8a5dc2000, 4096, 0xb104a9) = -1 ENOMEM (Cannot allocate memory)
17263 mincore(0xc8a5dc3000, 4096, 0xb104a9) = -1 ENOMEM (Cannot allocate memory)
...

[gpaul]

The output of ps ax is given below. Watching these with htop shows them growing rapidly to the point where they use upwards of 1GB each and finally trigger the oom-killer.

    17225 pts/1    Rl+    0:10 /home/gustav/go/pkg/tool/linux_amd64/compile -o /dev/shm/go-build625944181/github.com/gogo/protobuf/test.a -trimpath /dev/shm/go-build625944181 -p github.com/gogo/protobuf/test -complete -buildid 1bb3edbfa185d998f7210687c649cc52b6422a76 -D _/home/gustav/gopath2/src/github.com/gogo/protobuf/test -I /dev/shm/go-build625944181 -I /home/gustav/gopath2/pkg/linux_amd64 -pack /home/gustav/gopath2/src/github.com/gogo/protobuf/test/thetest.pb.go /home/gustav/gopath2/src/github.com/gogo/protobuf/test/uuid.go
    17226 pts/1    Rl+    0:10 /home/gustav/go/pkg/tool/linux_amd64/compile -o /dev/shm/go-build625944181/github.com/gogo/protobuf/test/casttype/combos/both.a -trimpath /dev/shm/go-build625944181 -p github.com/gogo/protobuf/test/casttype/combos/both -complete -buildid 0dec27d7c21ec48f91578f15d0f446edc5ee76aa -D _/home/gustav/gopath2/src/github.com/gogo/protobuf/test/casttype/combos/both -I /dev/shm/go-build625944181 -I /home/gustav/gopath2/pkg/linux_amd64 -pack /home/gustav/gopath2/src/github.com/gogo/protobuf/test/casttype/combos/both/casttype.pb.go
    17236 pts/1    Sl+    0:10 /home/gustav/go/pkg/tool/linux_amd64/compile -o /dev/shm/go-build625944181/github.com/gogo/protobuf/test/casttype/combos/marshaler.a -trimpath /dev/shm/go-build625944181 -p github.com/gogo/protobuf/test/casttype/combos/marshaler -complete -buildid 0dec27d7c21ec48f91578f15d0f446edc5ee76aa -D _/home/gustav/gopath2/src/github.com/gogo/protobuf/test/casttype/combos/marshaler -I /dev/shm/go-build625944181 -I /home/gustav/gopath2/pkg/linux_amd64 -pack /home/gustav/gopath2/src/github.com/gogo/protobuf/test/casttype/combos/marshaler/casttype.pb.go
    17238 pts/1    Rl+    0:10 /home/gustav/go/pkg/tool/linux_amd64/compile -o /dev/shm/go-build625944181/github.com/gogo/protobuf/test/casttype/combos/neither.a -trimpath /dev/shm/go-build625944181 -p github.com/gogo/protobuf/test/casttype/combos/neither -complete -buildid 0dec27d7c21ec48f91578f15d0f446edc5ee76aa -D _/home/gustav/gopath2/src/github.com/gogo/protobuf/test/casttype/combos/neither -I /dev/shm/go-build625944181 -I /home/gustav/gopath2/pkg/linux_amd64 -pack /home/gustav/gopath2/src/github.com/gogo/protobuf/test/casttype/combos/neither/casttype.pb.go
    17242 pts/1    Rl+    0:10 /home/gustav/go/pkg/tool/linux_amd64/compile -o /dev/shm/go-build625944181/github.com/gogo/protobuf/test/casttype/combos/unmarshaler.a -trimpath /dev/shm/go-build625944181 -p github.com/gogo/protobuf/test/casttype/combos/unmarshaler -complete -buildid 0dec27d7c21ec48f91578f15d0f446edc5ee76aa -D _/home/gustav/gopath2/src/github.com/gogo/protobuf/test/casttype/combos/unmarshaler -I /dev/shm/go-build625944181 -I /home/gustav/gopath2/pkg/linux_amd64 -pack /home/gustav/gopath2/src/github.com/gogo/protobuf/test/casttype/combos/unmarshaler/casttype.pb.go
    17248 pts/1    Sl+    0:09 /home/gustav/go/pkg/tool/linux_amd64/compile -o /dev/shm/go-build625944181/github.com/gogo/protobuf/test/casttype/combos/unsafeboth.a -trimpath /dev/shm/go-build625944181 -p github.com/gogo/protobuf/test/casttype/combos/unsafeboth -complete -buildid 0dec27d7c21ec48f91578f15d0f446edc5ee76aa -D _/home/gustav/gopath2/src/github.com/gogo/protobuf/test/casttype/combos/unsafeboth -I /dev/shm/go-build625944181 -I /home/gustav/gopath2/pkg/linux_amd64 -pack /home/gustav/gopath2/src/github.com/gogo/protobuf/test/casttype/combos/unsafeboth/casttype.pb.go
    17253 pts/1    Sl+    0:10 /home/gustav/go/pkg/tool/linux_amd64/compile -o /dev/shm/go-build625944181/github.com/gogo/protobuf/test/casttype/combos/unsafemarshaler.a -trimpath /dev/shm/go-build625944181 -p github.com/gogo/protobuf/test/casttype/combos/unsafemarshaler -complete -buildid 0dec27d7c21ec48f91578f15d0f446edc5ee76aa -D _/home/gustav/gopath2/src/github.com/gogo/protobuf/test/casttype/combos/unsafemarshaler -I /dev/shm/go-build625944181 -I /home/gustav/gopath2/pkg/linux_amd64 -pack /home/gustav/gopath2/src/github.com/gogo/protobuf/test/casttype/combos/unsafemarshaler/casttype.pb.go
    17254 pts/1    Sl+    0:10 /home/gustav/go/pkg/tool/linux_amd64/compile -o /dev/shm/go-build625944181/github.com/gogo/protobuf/test/casttype/combos/unsafeunmarshaler.a -trimpath /dev/shm/go-build625944181 -p github.com/gogo/protobuf/test/casttype/combos/unsafeunmarshaler -complete -buildid 0dec27d7c21ec48f91578f15d0f446edc5ee76aa -D _/home/gustav/gopath2/src/github.com/gogo/protobuf/test/casttype/combos/unsafeunmarshaler -I /dev/shm/go-build625944181 -I /home/gustav/gopath2/pkg/linux_amd64 -pack /home/gustav/gopath2/src/github.com/gogo/protobuf/test/casttype/combos/unsafeunmarshaler/casttype.pb.go

[kostya-sh]

Possible reason: the generated casttype.pb.go files have lots (1000+) anonymous function invocations like this:

   {Name: func(v string) *string { return &v }("google/protobuf/descriptor.proto"),
    Package: func(v string) *string { return &v }("google.protobuf"),
    MessageType: []*descriptor.DescriptorProto{{Name: func(v string) *string { return &v }("FileDescriptorSet"),
        Field: []*descriptor.FieldDescriptorProto{{Name: func(v string) *string { return &v }("file"),
            Number:   func(v int32) *int32 { return &v }(1),
            Label:    func(v descriptor.FieldDescriptorProto_Label) *descriptor.FieldDescriptorProto_Label { return &v }(3),
            Type:     func(v descriptor.FieldDescriptorProto_Type) *descriptor.FieldDescriptorProto_Type { return &v }(11),
            TypeName: func(v string) *string { return &v }(".google.protobuf.FileDescriptorProto"),
            JsonName: func(v string) *string { return &v }("file"),
        }},
    }

Memprofile shows:

  299.64MB 36.26% 36.26%   300.37MB 36.35%  cmd/compile/internal/ssa.(*stackAllocState).stackalloc
  195.68MB 23.68% 59.94%   342.44MB 41.44%  cmd/compile/internal/ssa.(*stackAllocState).init
  146.77MB 17.76% 77.70%   146.77MB 17.76%  cmd/compile/internal/ssa.(*Func).newSparseSet
  146.76MB 17.76% 95.46%   146.76MB 17.76%  cmd/compile/internal/ssa.(*stackAllocState).buildInterferenceGraph

[josharian]

Thanks for profiling. If you have the binary and profile still around, will you re-run pprof with the -lines argument added as well? Thanks!

[davecheney]

Does your machine had any swap partition or file allocated? Even though you may have huge amounts of physical memory available, Linux behaves very conservatively if no swap is available.

On 4 May 2016, at 23:34, Gustav Paul notifications@github.com wrote:

The output of ps ax is given below. Watching these with htop shows them growing rapidly to the point where they use upwards of 1GB each and finally trigger the oom-killer.

17225 pts/1    Rl+    0:10 /home/gustav/go/pkg/tool/linux_amd64/compile -o /dev/shm/go-build625944181/github.com/gogo/protobuf/test.a -trimpath /dev/shm/go-build625944181 -p github.com/gogo/protobuf/test -complete -buildid 1bb3edbfa185d998f7210687c649cc52b6422a76 -D _/home/gustav/gopath2/src/github.com/gogo/protobuf/test -I /dev/shm/go-build625944181 -I /home/gustav/gopath2/pkg/linux_amd64 -pack /home/gustav/gopath2/src/github.com/gogo/protobuf/test/thetest.pb.go /home/gustav/gopath2/src/github.com/gogo/protobuf/test/uuid.go
17226 pts/1    Rl+    0:10 /home/gustav/go/pkg/tool/linux_amd64/compile -o /dev/shm/go-build625944181/github.com/gogo/protobuf/test/casttype/combos/both.a -trimpath /dev/shm/go-build625944181 -p github.com/gogo/protobuf/test/casttype/combos/both -complete -buildid 0dec27d7c21ec48f91578f15d0f446edc5ee76aa -D _/home/gustav/gopath2/src/github.com/gogo/protobuf/test/casttype/combos/both -I /dev/shm/go-build625944181 -I /home/gustav/gopath2/pkg/linux_amd64 -pack /home/gustav/gopath2/src/github.com/gogo/protobuf/test/casttype/combos/both/casttype.pb.go
17236 pts/1    Sl+    0:10 /home/gustav/go/pkg/tool/linux_amd64/compile -o /dev/shm/go-build625944181/github.com/gogo/protobuf/test/casttype/combos/marshaler.a -trimpath /dev/shm/go-build625944181 -p github.com/gogo/protobuf/test/casttype/combos/marshaler -complete -buildid 0dec27d7c21ec48f91578f15d0f446edc5ee76aa -D _/home/gustav/gopath2/src/github.com/gogo/protobuf/test/casttype/combos/marshaler -I /dev/shm/go-build625944181 -I /home/gustav/gopath2/pkg/linux_amd64 -pack /home/gustav/gopath2/src/github.com/gogo/protobuf/test/casttype/combos/marshaler/casttype.pb.go
17238 pts/1    Rl+    0:10 /home/gustav/go/pkg/tool/linux_amd64/compile -o /dev/shm/go-build625944181/github.com/gogo/protobuf/test/casttype/combos/neither.a -trimpath /dev/shm/go-build625944181 -p github.com/gogo/protobuf/test/casttype/combos/neither -complete -buildid 0dec27d7c21ec48f91578f15d0f446edc5ee76aa -D _/home/gustav/gopath2/src/github.com/gogo/protobuf/test/casttype/combos/neither -I /dev/shm/go-build625944181 -I /home/gustav/gopath2/pkg/linux_amd64 -pack /home/gustav/gopath2/src/github.com/gogo/protobuf/test/casttype/combos/neither/casttype.pb.go
17242 pts/1    Rl+    0:10 /home/gustav/go/pkg/tool/linux_amd64/compile -o /dev/shm/go-build625944181/github.com/gogo/protobuf/test/casttype/combos/unmarshaler.a -trimpath /dev/shm/go-build625944181 -p github.com/gogo/protobuf/test/casttype/combos/unmarshaler -complete -buildid 0dec27d7c21ec48f91578f15d0f446edc5ee76aa -D _/home/gustav/gopath2/src/github.com/gogo/protobuf/test/casttype/combos/unmarshaler -I /dev/shm/go-build625944181 -I /home/gustav/gopath2/pkg/linux_amd64 -pack /home/gustav/gopath2/src/github.com/gogo/protobuf/test/casttype/combos/unmarshaler/casttype.pb.go
17248 pts/1    Sl+    0:09 /home/gustav/go/pkg/tool/linux_amd64/compile -o /dev/shm/go-build625944181/github.com/gogo/protobuf/test/casttype/combos/unsafeboth.a -trimpath /dev/shm/go-build625944181 -p github.com/gogo/protobuf/test/casttype/combos/unsafeboth -complete -buildid 0dec27d7c21ec48f91578f15d0f446edc5ee76aa -D _/home/gustav/gopath2/src/github.com/gogo/protobuf/test/casttype/combos/unsafeboth -I /dev/shm/go-build625944181 -I /home/gustav/gopath2/pkg/linux_amd64 -pack /home/gustav/gopath2/src/github.com/gogo/protobuf/test/casttype/combos/unsafeboth/casttype.pb.go
17253 pts/1    Sl+    0:10 /home/gustav/go/pkg/tool/linux_amd64/compile -o /dev/shm/go-build625944181/github.com/gogo/protobuf/test/casttype/combos/unsafemarshaler.a -trimpath /dev/shm/go-build625944181 -p github.com/gogo/protobuf/test/casttype/combos/unsafemarshaler -complete -buildid 0dec27d7c21ec48f91578f15d0f446edc5ee76aa -D _/home/gustav/gopath2/src/github.com/gogo/protobuf/test/casttype/combos/unsafemarshaler -I /dev/shm/go-build625944181 -I /home/gustav/gopath2/pkg/linux_amd64 -pack /home/gustav/gopath2/src/github.com/gogo/protobuf/test/casttype/combos/unsafemarshaler/casttype.pb.go
17254 pts/1    Sl+    0:10 /home/gustav/go/pkg/tool/linux_amd64/compile -o /dev/shm/go-build625944181/github.com/gogo/protobuf/test/casttype/combos/unsafeunmarshaler.a -trimpath /dev/shm/go-build625944181 -p github.com/gogo/protobuf/test/casttype/combos/unsafeunmarshaler -complete -buildid 0dec27d7c21ec48f91578f15d0f446edc5ee76aa -D _/home/gustav/gopath2/src/github.com/gogo/protobuf/test/casttype/combos/unsafeunmarshaler -I /dev/shm/go-build625944181 -I /home/gustav/gopath2/pkg/linux_amd64 -pack /home/gustav/gopath2/src/github.com/gogo/protobuf/test/casttype/combos/unsafeunmarshaler/casttype.pb.go

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub

[josharian]

I just reproduced here. (Not the killing, but high memory usage.) I'm investigating, although no promises. An initial observation is that we're generating way too many OpCopies during SSA construction.

[randall77]

@dr2chase : if this is phi insertion your CL might help.

[josharian]

I think I have a simpler fix. Hang tight.

[dr2chase]

Sparse-phi code looks like a big help. On my laptop (8GB, OS X) it appears to max out at 4 simultaneous compiles each over 2G if sparse-phi is activated, but it runs to completion with only a brief instant of red in the system memory monitor.

With the sparse-phi code deactivated, at an earlier stage one of the compiles grows to 6.7G (at least -- I killed it, it did this twice in a row so it is repeatable enough for me) alongside three others, one of those over 2G and two over 1G, and the system memory monitor goes red and sticks there.

[gopherbot]

CL https://golang.org/cl/22790 mentions this issue.