cmd/go: issues with shared $GOPATH/pkg/mod cache and autogenerated modules

[sirkon]

Hi!

At my job we have a single repository with proto-files grouped by directories. Something like gitlab.example.com/common/schema/<service path>/<file.proto>

We were generating this into $GOPATH/src/gen/<service path>/<file.go> and import this as gen/<service path>

Now we did a server which acts like a good go modules proxy except the case when it is requested for gitlab.example.com/common/schema/<service path>/v34. It generates go code from gitlab.example.com/common/schema/<service path>/*.proto files and return them as regular go module. Sounds nice and it works great in most cases except one thing: most our projects use code generated with protoc-gen-go, but some are playing with code generated by protoc-gen-gofast.

Our decision was to have two URL serving goproxies: https://goproxy.example.com/go and https://goproxy.example.com/gofast where the first one requests for 3rd party goproxies and returns code generated with protoc-gen-go and the other one works exactly the same except it would return code generated with protoc-gen-gofast. We thought we would simply use different GOPROXY for different projects and everything will be great.

And this doesn't work that easy unfortunately: we have shared $GOPATH/pkg/mod – cache directory for downloaded modules. And we will have exactly the same autogenerated module of certain version for project A if we have already requested this version of the module for project B despite having different sources.

Sure, we can manually clean cache dir but this is … ugly. We would probably have different cache directories with different GOPATH, but we would prefer to keep it the same.

So I have a proposal: a new environment variable called GOCACHE or GOMODCACHE or whatever to control modules cache. This would ease the development if editor/IDE supports project profiles.

[thepudds]

I think I might have misunderstood the problem, or at least I didn't follow this paragraph:

And this doesn't work that easy unfortunately: we have shared $GOPATH/pkg/mod – cache directory for downloaded modules. And we will have exactly the same autogenerated module of certain version for project A if we have already requested this version of the module for project B despite having different sources.

Could you expand slightly on the actual problem you encountered?

We would probably have different cache directories with different GOPATH, but we would prefer to keep it the same.

Can you expand why different GOPATH wouldn't work for you?

[sirkon]

Could you expand slightly on the actual problem you encountered?

Our situation: we have service called marker whose gRPC declarations are defined in gitlab.example.com/common/schema/marker and we have web server with autogenerated handlers providing REST interface to some gRPC handlers which is using these declarations to access marker.

But the difference is marker works with code generated with protoc-gen-go, while the web server working with code generated by protoc-gen-gofast because of some additional functionality we needed (finer JSON marshaling control mainly).

Both are using helpers what import gitlab.example.com/common/schema/marker and thus work both for protoc-gen-go and protoc-gen-gofast generated code.

Can you expand why different GOPATH wouldn't work for you?

It would actually, we would just prefer to keep it the same for things like go install.

[bcmills]

We thought we would simply use different GOPROXY for different projects and everything will be great.

Sorry, but that's not how GOPROXY works. It is a proxy, not a source of alternative truths.

As you've observed, every proxy must serve the same contents for a given module path and version; that is fundamental to the design of modules (and to the reproducibility of module-enabled builds), and enforced by checksums in go.sum and the database enabled in #25530.

As an alternative, you might be able to serve the two variants at different module paths and let users apply their own replace directives to substitute one for the other.