cmd/go: can't run go builds concurrently if they download modules

[allenpetersen]

What version of Go are you using (go version)?

go1.11beta2

What operating system and processor architecture are you using (go env)?

amd64 docker container

Our current build workflow is to run glide install first and then start a number of go build jobs in parallel.

When I take a clean system and start the go build jobs in parallel they fail due to issues downloading modules, git shallow.lock files and race conditions creating and populating /go/src/mod directories.

If I first do something like go get ./foo ./bar ./baz to download the modules I can run the builds in parallel.

Is there an equivalent of glide install or dep ensure?
Is there a plan to make downloading of modules thread safe?

[ianlancetaylor]

CC @bcmills

[rsc]

For now it should suffice to do 'go list ./foo ./bar ./baz' before your builds.

There is a plan to make downloading of modules by parallel go commands
safe but we haven't done that yet.

There is also a plan to provide 'go mod download' to prefetch all the
downloaded modules. That could also stand in for 'glide install' but
go list is enough and exists today.

Why are you running multiple go build commands in parallel instead of
a single go build command?

[allenpetersen]

Why are you running multiple go build commands in parallel instead of
a single go build command?

We have many services in a monorepo written in a number of different languages. The top level CI builder doesn't know about what language they are written it. It just invokes a number of Makefiles most of which happen to be Go programs.

[apparentlymart]

We have a different "multiple builds at once" use-case at hashicorp/terraform: we build the same application source (and thus the same dependencies) for a number of different GOOS/GOARCH pairs concurrently during our release process.

We are currently doing this with gox, which is a wrapper around go build that deals with templating the executable output paths to produce a systematic result. A number of other projects at HashiCorp, and I believe also elsewhere, are using this tool in the same way.

It sounds like it may be viable for gox to simply run an additional single command to completion before it starts its parallel work to ensure that all of the dependencies are available. Would that be go list ./..., or similar?

[wsc1]

We have multiple builds at once at zikichombo.org as well. It is a set of libraries, each a module with very heterogenous versioning projected. We are counting on the module system to allow independent releases, but internally have a need to download and build the whole set of libraries as part of work flow.