cmd/go: module dependencies not updated with `go get -u` in 1.17

[niallnsec]

What version of Go are you using (go version)?

$ go version
go version go1.17 darwin/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/USER/Library/Caches/go-build"
GOENV="/Users/USER/Library/Application Support/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/USER/go/pkg/mod"
GONOPROXY="private1.repo,private2.repo"
GONOSUMDB="private1.repo,private2.repo"
GOOS="darwin"
GOPATH="/Users/USER/go"
GOPRIVATE="private1.repo,private2.repo"
GOPROXY="https://private2.repo:4443"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
GOVCS=""
GOVERSION="go1.17"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/dev/null"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -arch x86_64 -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/zx/4vdy2gqd5b1g26g7n8lt0zk80000gn/T/go-build4221827946=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

I ran go get -u

What did you expect to see?

I expected the module dependencies to be update as usual.

What did you see instead?

Only a small number of dependencies were updated. Even directly imported packages with new versions were not updated. The command appeared to download the new versions but did not update the go.mod file.

USER@MacBook-Pro ‹ master › : ~/project/auth
[0] % go get -u           
go: downloading private2.repo/project/mfa v0.2.5
go: downloading private2.repo/project/webauthn v1.0.3
go: downloading github.com/fxamacker/cbor v1.5.1
go get: upgraded github.com/aws/aws-sdk-go v1.40.23 => v1.40.24

After running the above command, go.mod still has a require line for private2.repo/project/mfa v0.2.4 and private2.repo/project/webauthn v1.0.2 instead of upgrading them to the new version it just downloaded. Also, many package updates are completely ignored.

If I use go 1.16.4 instead I get the following and go.mod is updated as expected with all the appropriate new versions.

USER@MacBook-Pro ‹ master › : ~/project/auth
[0] % go1.16.4 get -u
go: downloading github.com/cenkalti/backoff v1.1.0
go: downloading github.com/dgrijalva/jwt-go v1.0.2
go: downloading github.com/gofrs/uuid v1.2.0
go: downloading github.com/pierrec/lz4 v1.0.1
go get: upgraded golang.org/x/crypto v0.0.0-20210711020723-a769d52b0f97 => v0.0.0-20210817164053-32db794688a5
go get: upgraded golang.org/x/net v0.0.0-20210805182204-aaa1db679c0d => v0.0.0-20210813160813-60bc85c4be6d
go get: upgraded golang.org/x/sys v0.0.0-20210806184541-e5e7981a1069 => v0.0.0-20210817190340-bfb29a6856f2
go get: upgraded golang.org/x/text v0.3.6 => v0.3.7
go get: upgraded private2.repo/project/cryptoengine v0.1.3 => v0.1.4
go get: upgraded private2.repo/project/mfa v0.2.4 => v0.2.5
go get: upgraded private2.repo/project/webauthn v1.0.2 => v1.0.3

[mknyszek]

@niallnsec is this new in 1.17? Can you reproduce with 1.16?

CC @bcmills @jayconrod @matloob via https://dev.golang.org/owners

[niallnsec]

Yes this only seems to happen with go 1.17.

I don't think I made it very clear above (sorry about that), but the two code blocks in the report above show me using go get -u on the same repository sequentially. Running the first go get -u using Go 1.17 updated a single entry and downloaded updated code for two others but did not update go.mod. (Running the command again using 1.17 had no effect). Then I ran the same command using Go 1.16.4 and it proceeded to update all the dependencies as expected.

Go 1.16.4 worked as expected, 1.17 appeared to ignore most of the available updates, even though it had downloaded updated code for two of them.

[gopherbot]

Change https://golang.org/cl/343431 mentions this issue: modfile: in SetRequireSeparateIndirect, arrange requirements consistently

[jayconrod]

@niallnsec Thanks for reporting this. It sounds like there might be a bug here, but we need more information to understand what's going on.

• Ideally, do you have a public repo that reproduces this problem? Or can you put together a small example?
• Does go list -m -u all show the updates you expect?
• For modules that are not updated by 1.17, are the packages they contain imported directly from the package in the current directory? Indirectly? By tests?
• What version does the go directive have in go.mod?

[niallnsec]

I have not got a public repo example at the moment, but will keep looking or try to create a test case that has the same behaviour. In the mean time, to answer your other questions:

Does go list -m -u all show the updates you expect?

Yes, for both go1.16.7 and go1.17 the output of go list -m -u all are identical.

For modules that are not updated by 1.17, are the packages they contain imported directly from the package in the current directory? Indirectly? By tests?

The modules not updated are imported both directly and indirectly.

What version does the go directive have in go.mod?

When I upgraded to go 1.17 I update the go directive to 1.17, then did a go get -u followed by a go mod tidy. Prior to that the directive was go 1.16. I now have two require blocks in my repository as was expected.
So all tests show were done with a go directive of 1.17.

I am using a private company Go proxy in my environment, so I disabled that before running the tests to make sure it wasn't playing a part. I get the same results with it in use or not.

[jayconrod]

Thanks, that helps cross out a few possible causes, but we'll probably still need a reproducible example to figure out what's going on.

[niallnsec]

I have created a small demo project here: https://github.com/niallnsec/demo_auth

I took the go.mod and go.sum files from my private module exactly as they were following the initial update to Go 1.17. The only edits done manually on these two files (ever) is to remove references to three other internal projects. Otherwise, the module is still importing all of the same public module as in my previous tests.

Using that repository, when using Go 1.16 go get -u gives me the following:

[0] % go get -u
go get: upgraded github.com/aws/aws-sdk-go v1.40.23 => v1.40.27
go get: upgraded golang.org/x/crypto v0.0.0-20210711020723-a769d52b0f97 => v0.0.0-20210817164053-32db794688a5
go get: upgraded golang.org/x/net v0.0.0-20210805182204-aaa1db679c0d => v0.0.0-20210813160813-60bc85c4be6d
go get: upgraded golang.org/x/sys v0.0.0-20210806184541-e5e7981a1069 => v0.0.0-20210820121016-41cdb8703e55
go get: upgraded golang.org/x/text v0.3.6 => v0.3.7

However, Go 1.17 does not update all of the packages in the same way when I run the command:

[0] % go1.17 get -u
go get: upgraded github.com/aws/aws-sdk-go v1.40.23 => v1.40.27

In this specific case, all of the modules not updated are indirect imports. The three private modules I removed are all direct imports of the original module these files were taken from and they did not update either when using Go 1.17, although one of them was downloaded.

[jayconrod]

Thanks, that's really helpful. I was able to reproduce this and narrow it down a bit. Working to find the root cause.

It's likely we'll want to backport the fix to 1.17.1.