cmd/go: add modvendor sub-command

[myitcv]

Creating this issue as a follow up to #26366 (and others).

go mod vendor is documented as follows:

Vendor resets the main module's vendor directory to include all packages
needed to build and test all the main module's packages.
It does not include test code for vendored packages.

Much of the surprise in #26366 comes about because people are expecting "other" files to also be included in vendor.

An alternative to the Go 1.5 vendor is to instead "vendor" the module download cache. A proof of concept of this approach is presented here:

https://github.com/myitcv/go-modules-by-example/blob/master/012_modvendor/README.md

Hence I propose go mod modvendor, which would be documented as follows:

Modvendor resets the main module's modvendor directory to include a 
copy of the module download cache required for the main module and its 
transitive dependencies.

Name and the documentation clearly not final.

Benefits (WIP)

• Eliminates any potential confusion around what is in/not in vendor
• Easier to contribute patches/fixes to upstream module authors (via something like gohack), because the entire module is available
• The modules included in modvendor are an exact copy of the original modules. This makes it easier to check their fidelity at any point in time, with either the source or some other reference (e.g. Athens)
• Makes clear the source of modules, via the use of GOPROXY=/path/to/modvendor. No potential for confusion like "will the modvendor of my dependencies be used?"
• A single deliverable
• Fully reproducible and high fidelity builds (modules in general gives us this, so just re-emphasising the point)
• ...

Costs (WIP)

• The above steps are currently manual; tooling (the go tool?) can fix this
• Reviewing "vendored" dependencies is now more involved without further tooling. For example it's no longer possible to simply browse the source of a dependency via a GitHub PR when it is added. Again, tooling could help here. As could some central source of truth for trusted, reviewed modules (Athens? cc @bketelsen @arschles)
• ...

Related discussion

Somewhat related to discussion in #27227 (cc @rasky) where it is suggested the existence of vendor should imply the -mod=vendor flag. The same argument could be applied here, namely the existence of modvendor implying the setting of GOPROXY=/path/to/modvendor. This presupposes, however, that the idea of modvendor makes sense in the first place.

Background discussion:

https://twitter.com/_myitcv/status/1038885458950934528

cc @StabbyCutyou @fatih

cc @bcmills

[bcmills]

I don't think the proposed resets the main module's modvendor directory behavior is quite the right workflow.

One of the benefits of versioned modules over vendoring is that they can reduce redundancy globally: instead of N copies of the same code spread across N repos, we can have a single canonical copy shared by all builds of those repos. A per-module modvendor cache would revert that advantage.

[bcmills]

Instead, perhaps we should make it easier to maintain per-user or per-organization module proxies.

For example, we could add an optional argument to go mod download to tell it where to save the downloaded modules.

go mod download $path could copy all active modules to $path, and go mod verify $path could verify that the modules already stored in $path match the go.sum of the current module. Then, the modvendor operation would essentially be:

go mod download $GOPROXY
go mod verify $GOPROXY

Then the user could commit the contents of $GOPROXY to a separate (personal or org-wide) repository.

[flibustenet]

We should also do the opposite, to fill the cache from a downloaded directory.

$ go mod download -export $path 

somewhere else, maybe an other machine
$ go mod download -import $path
that will fill the cache

[bcmills]

@flibustenet GOPROXY already does the opposite: GOPROXY=$path go mod download populates the active modules into the user's module cache from an arbitrary directory.

We don't currently have a command that populates more than the active modules, but that seems like a job for rsync or git rather than go itself.

[myitcv]

@bcmills

I don't think the proposed resets the main module's modvendor directory behavior is quite the right workflow.

I think there are actually two use cases here:

1. "vendoring" all dependencies within the same repo as the module(s) that depend on them
2. a per-user/organisation module proxy repo, separate from the repo(s) that use it

I should update the description to make clear that this issue is trying to address point 1. Hence why I think the logic to "reset the main module's modvendor directory" is correct; because I don't want this directory to grow like a cache.

Point 2 is the approach I've taken with https://github.com/myitcv/cachex, which is the "organisation repo" for https://github.com/myitcv/x, my mono repo. In this case, https://github.com/myitcv/cachex is an append-only repo that is a cache, and hence grows over time. It's separate from (and a subset of) $GOPATH/pkg/mod/cache/download because that can (and does) include downloads of private repos that I don't want made public. As you say, this approach reduces redundancy. Your proposal of go mod download $path is effectively what I do via bash with a GOPROXY+GOPATH+rsync dance; in this situation, I agree, I don't want the reset semantics.

But I can see use cases (i.e. deploying code or similar) where there is real benefit in point 1, for everything to be "bundled (in the same repo).

Assuming we want to address/support both use cases (and it seems sensible to my mind to do so), they could be solved by the same sub-command; I'm certainly not precious about that 😄. But I think there are separate use cases to cover here.

[bcmills]

I can see use cases (i.e. deploying code or similar) where there is real benefit in point 1, for everything to be "bundled" (in the same repo).

I'm not certain about those cases one way or the other. Given versioning, it seems like you can address all of the same use-cases — and more! — using a separate repository. If folks are doing the cost/benefit analysis and coming to a different conclusion, I'd like to see more of the details of the costs and benefits involved (beyond just “that's the way we've done things without versioning”).

[myitcv]

If folks are doing the cost/benefit analysis and coming to a different conclusion, I'd like to see more of the details of the costs and benefits involved (beyond just “that's the way we've done things without versioning”).

I'd second this request because, unless it wasn't clear already, I'm a fan of point 2.

I'm only putting up point 1 as a "better" alternative to go mod vendor (better in the sense that it doesn't suffer from the pitfalls associated with #26366 amongst other things). But, and I totally grant you this, I haven't articulated all (any?) of the costs associated with keeping workflows oriented around a single repo, a la vendor.

[bcmills]

Hmm. With the go mod download $path approach, it's still possible to put $path in the same repository (cutting it off from the modules in that repo using an explicit go.mod file, or perhaps with a well-known subdirectory such as vendor/mod/ or vendormod/), and you can even unpack it easily with a single command (GOPROXY=$path go mod vendor).

[myitcv]

Yes absolutely; I think the only difference between these two use cases is the use of "reset" semantics or not.