cmd/go: go get should not add a dependency to go.mod

[natefinch]

go get is not the right command for adding a dependency to the current module. We have a command for dealing with modules, and it's go mod, that's the command that should be used to add dependencies.

This is the first paragraph of help on go get:

Get downloads the packages named by the import paths, along with their dependencies. It then installs the named packages, like 'go install'.

None of that says "add the library you specified to the dependencies of the current project". And why would it? it's for getting code from somewhere else and bringing it to this machine.

Notably, go get will even add go commands as a dependency... so like if you run go get golang.org/x/tools/cmd/goimports and your current directory happens to be a go module, guess what? Your go.mod now has

golang.org/x/tools v0.0.0-20180911133044-677d2ff680c1 // indirect

This UX is extremely confusing for the ~2 million people who used go before modules. It's also way too implicit and magic regardless of the experience. Adding a dependency to your project should be an explicit action:

go mod add golang.org/x/tools/cmd/goimports  

What did you do?

from inside a module directory

$ ls
go.mod
$ go get github.com/natefinch/lumberjack
go: finding github.com/natefinch/lumberjack latest
go: downloading github.com/natefinch/lumberjack v0.0.0-20180817145747-7d6a1875575e
~/dev/test$ more go.mod
module app

require github.com/natefinch/lumberjack v0.0.0-20180817145747-7d6a1875575e // indirect

What did you expect to see?

No change to my go.mod

What did you see instead?

go get added a library to my go.mod file.

System details

go version go1.11 darwin/amd64
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/finchnat/Library/Caches/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/finchnat"
GOPROXY=""
GORACE=""
GOROOT="/Users/finchnat/sdk/go1.11"
GOTMPDIR=""
GOTOOLDIR="/Users/finchnat/sdk/go1.11/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD=""
GOROOT/bin/go version: go version go1.11 darwin/amd64
GOROOT/bin/go tool compile -V: compile version go1.11
uname -v: Darwin Kernel Version 17.7.0: Thu Jun 21 22:53:14 PDT 2018; root:xnu-4570.71.2~1/RELEASE_X86_64
ProductName:	Mac OS X
ProductVersion:	10.13.6
BuildVersion:	17G65
lldb --version: lldb-902.0.79.7
  Swift-4.1

[myitcv]

From the bottom of go help get:

...

This text describes the behavior of get when using GOPATH
to manage source code and dependencies.
If instead the go command is running in module-aware mode,
the details of get's flags and effects change, as does 'go help get'.
See 'go help modules' and 'go help module-get'.

From go help module-get:

The 'go get' command changes behavior depending on whether the
go command is running in module-aware mode or legacy GOPATH mode.
This help text, accessible as 'go help module-get' even in legacy GOPATH mode,
describes 'go get' as it operates in module-aware mode.

...

[myitcv]

@natefinch I think the documentation on this covers the points you raise, but please do say if you think it could be better signposted.

cc @bcmills

[natefinch]

It's not about the documentation. It's a complete change to what go get has meant for the last 6+ years. Which has meant "download and compile some code so I have it locally". That's still a useful command to have in the universe of go modules.

Changing what an established command does, in such an important way, is a really bad idea.

I think it's pretty clear, if I go get golang.org/x/tools/cmd/goimports ... what I mean is "download and compile goimports, and put it somewhere for general use". The fact that this command also stealthfully adds golang.org/x/tools/ to my current directory's dependencies is surprising and extremely unwelcome.

Like I said in the bug report... we have a command for dealing with modules. Why would we not just add this functionality there? go mod add github.com/... is perfectly clear, and wouldn't clash with current usage.

[komuw]

if I go get golang.org/x/tools/cmd/goimports ... what I mean is "download and compile goimports, and put it somewhere for general use".

I think this is; #25922 and #24250

[cespare]

I agree with this. I've already been surprised three separate times now by running go get and having tools unexpectedly added to my go.mod, or having a new go.mod file dropped into a directory where I didn't intend.

Before modules, go get was used for two pretty distinct use cases: (1) downloading tools and (2) downloading libraries. It seems much, much less surprising and user-friendly for post-modules go get to specialize on use case (1) and leave (2) for other tools. In my experience with modules so far, I've handled (2) by simply adding new imports and running go test/go build.

[myitcv]

I think it's pretty clear, if I go get golang.org/x/tools/cmd/goimports ... what I mean is "download and compile goimports, and put it somewhere for general use".

In a GOPATH world I would agree. But we're talking a modules world here. From my perspective I'm fully anticipating any go command to do something with my go.mod.

Put another way (and this is where #24250 comes in), I would expect, in a modules world, to have to do something "special" in order to performa a "global" install (by definition outside the context of my go.mod)

[vdemario]

I was also surprised by this behavior. The suggestion that go get will have a different meaning in a "modules world" doesn't solve the use case of go get golang.org/x/tools/cmd/goimports or installing any other tool. Do I need to change directories every time I need to install a new tool? What if a go.mod is created anywhere I am because in the modules world any place is a valid place for a Go project?

Regardless of these examples, what bothers me the most is that we're using the same command that has existed for several years to do a different thing. Are we all supposed to accept that every tutorial, every doc, everything that has ever been said about go get is not valid anymore?

The global place for installing any library or tool still exists, so why do we need to do something special to get back the behavior that already exists?

[kaedys]

Have to chime in that that I agree with @natefinch and @vdemario here. Using go get for installing tools has been the standard practice since go get was created. Changing that semantic is extremely problematic from a usability perspective, especially since the Go tooling currently lacks a command equivalent to "download this Go library, compile it, and install it somewhere I can use, but do not add it to my current project".

We're adding entirely new behavior via the go mod command. It seems to make the most sense to have explicitly adding a dependency to be a subset of that command, rather than hijacking an existing command and completely obliterating one of the two most common ways that command is currently used.

[vdemario]

Also, apparently trying to install a tool outside a module doesn't even work:

/tmp
$ set -gx GO111MODULE on # fish shell syntax
/tmp
$ go get golang.org/x/tools/cmd/goimports
go: cannot find main module; see 'go help modules'

[myitcv]

@vdemario please see, as linked above, #24250

[kaedys]

That issue clarifies this one even more. Prior to Go 1.11, the semantics for go get were identical to go install, except that it pulled the code from a remote repository before running go install on them. The module functionality in Go 1.11 has completely abandoned the go install piece (actually, it's still there, but it's mostly an anachronism at this point) because go get has been hijacked for adding a dependency to a module.

And that's a bit of a problem, since go get was validly used just as commonly for the go install functionality as it was for the abstraction of git clone. Abandoning the install functionality is extremely jarring and leads to some very unexpected side-effects, like the necessity of running go mod tidy after installing a tool that's not also a dependency, and as @vdemario just pointed out, the complete inability to install tools unless you're currently in a module (which you then need to tidy).

[myitcv]

@kaedys

The module functionality in Go 1.11 has completely abandoned the go install piece (actually, it's still there, but it's mostly an anachronism at this point) because go get has been hijacked for adding a dependency to a module.

And that's a bit of a problem, since go get was validly used just as commonly for the go install functionality as it was for the abstraction of git clone. Abandoning the install functionality is extremely jarring and leads to some very unexpected side-effects, like the necessity of running go mod tidy after installing a tool that's not also a dependency

go get does still perform an install. Given:

export GOPATH=$(mktemp -d)
cd $(mktemp -d)
mkdir hello
cd hello
go mod init example.com/hello
export GOBIN=$PWD/.bin
go get golang.org/x/tools/cmd/stringer

Then:

$ ls $GOBIN
stringer

like the necessity of running go mod tidy after installing a tool that's not also a dependency, and as @vdemario just pointed out, the complete inability to install tools unless you're currently in a module (which you then need to tidy).

This falls under the umbrella of #24250.

As @komuw linked above, for discussion on the current "best practice" for defining tool dependencies, please see #25922

[kaedys]

But it's not about tool dependencies. It's about tools you want to install on a global level, unrelated to the current module. That's the issue at the root of this, that under the Go module framework, go get is inherently and unavoidably linked to the module you are currently operating on (and completely fails if you're not on a module). There's no reason to abandon the ability to install tools unrelated to the current module (or outside the context of a module), and there's no reason to assume that every package retrieved is necessarily a dependency of the current module, especially given the pre-1.11 semantics and usage of go get.

[myitcv]

It's about tools you want to install on a global level

Indeed, that's why I linked to #24250.

Just to emphasise what @rsc said in #24250 (comment):

$ vgo get github.com/golang/dep/cmd/dep@v0.4.1
cannot determine module root; please create a go.mod file there

This clearly must work eventually.

(we can now substitute go get for vgo get)

So there is agreement that this "global" concept indeed very much has merit. It's just that this didn't make Go 1.11.

There's no reason to abandon the ability to install tools unrelated to the current module (or outside the context of a module), and there's no reason to assume that every package retrieved is necessarily a dependency of the current module, especially given the pre-1.11 semantics and usage of go get.

Hopefully the link above covers this point?