Skip to content

buildGoModule: building Go binaries in two phases#57080

Merged
kalbasit merged 13 commits intoNixOS:masterfrom
kalbasit:nixpkgs_create-build-go-module
Mar 14, 2019
Merged

buildGoModule: building Go binaries in two phases#57080
kalbasit merged 13 commits intoNixOS:masterfrom
kalbasit:nixpkgs_create-build-go-module

Conversation

@kalbasit
Copy link
Copy Markdown
Member

@kalbasit kalbasit commented Mar 8, 2019 •
edited
Loading

Motivation for this change

This change provides a new builder called buildGoModule for building Go packages with go modules support in mind. It does build through two phases:

  1. The first phase consists of building the go-modules derivation: go mod download will be executed within the source directory to download all of the dependencies, and store them at $GOPATH/pkg/mod. After removing the impure directory $GOPATH/pkg/mod/cache/vcs, we persist $GOPATH/pkg/mod as the $out of this derivation.
  2. The second phase consists of building the actual Go module, and this is done by setting GOPROXY to point to the download directory within ${go-modules}/cache/download which instructs Go to skip any network call, and simply use the already-downloaded dependencies.

The benefit of such infrastructure is that we no longer need tools such as dep2nix or vgo2nix to parse the go.mod and go.sum in order to generate Nix dependency file.

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Assured whether relevant documentation is up to date. Rendered.
  • Fits CONTRIBUTING.md.
  • Converted packages
    • Move over hugo.
    • Move over pet.
    • Move over jx.
    • Move over gotools.
    • Move over terminal-parrot.
    • Move over todoist.
    • Move over hetzner-kube.
  • Create new packages
    • Create mod.

cc @ehmry @lethalman @fpletz @Mic92 @nlewo

nix run -f https://github.com/kalbasit/nur-packages/archive/master.tar.gz nix-verify -c nix-verify -I nixpkgs=`pwd` --keep-going \
	-A jx \
	-A pet \
	-A hugo \
    -A gotools \
    -A terminal-parrot \
    -A todoist \
    -A hetzner-kube \
    -A mod

@GrahamcOfBorg GrahamcOfBorg added the 6.topic: golang Go is a high-level general purpose programming language that is statically typed and compiled. label Mar 8, 2019
@kalbasit kalbasit force-pushed the nixpkgs_create-build-go-module branch from 53d9cdf to 53829fb Compare March 8, 2019 19:48
@kalbasit

This comment has been minimized.

Sign in to view
@kalbasit kalbasit force-pushed the nixpkgs_create-build-go-module branch from 53829fb to 1f21995 Compare March 8, 2019 20:01
@GrahamcOfBorg GrahamcOfBorg added 11.by: package-maintainer This PR was created by a maintainer of all the package it changes. 10.rebuild-darwin: 1-10 This PR causes between 1 and 10 packages to rebuild on Darwin. 10.rebuild-linux: 1-10 This PR causes between 1 and 10 packages to rebuild on Linux. labels Mar 8, 2019
@kalbasit kalbasit force-pushed the nixpkgs_create-build-go-module branch 2 times, most recently from 7fc305e to b3437e2 Compare March 8, 2019 23:15
@kalbasit
Copy link
Copy Markdown
Member Author

kalbasit commented Mar 8, 2019

@GrahamcOfBorg build jx pet hugo gotools

@kalbasit kalbasit force-pushed the nixpkgs_create-build-go-module branch from b3437e2 to a1d010c Compare March 8, 2019 23:29
@GrahamcOfBorg GrahamcOfBorg added the 8.has: documentation This PR adds or changes documentation label Mar 9, 2019
@kalbasit kalbasit force-pushed the nixpkgs_create-build-go-module branch from 556fe3f to 6f89cdd Compare March 9, 2019 05:48
@kalbasit kalbasit marked this pull request as ready for review March 9, 2019 06:04
@kalbasit kalbasit requested review from Mic92, fpletz and nlewo March 9, 2019 06:05
@kalbasit
Copy link
Copy Markdown
Member Author

kalbasit commented Mar 9, 2019

@GrahamcOfBorg build jx pet hugo gotools

@kalbasit kalbasit force-pushed the nixpkgs_create-build-go-module branch 2 times, most recently from 45beb9d to 174ba7e Compare March 11, 2019 01:17
@GrahamcOfBorg GrahamcOfBorg added 10.rebuild-darwin: 501+ This PR causes many rebuilds on Darwin and should normally target the staging branches. 10.rebuild-darwin: 501-1000 This PR causes many rebuilds on Darwin and should normally target the staging branches. 10.rebuild-linux: 501+ This PR causes many rebuilds on Linux and should normally target the staging branches. 10.rebuild-linux: 501-1000 This PR causes many rebuilds on Linux and should normally target the staging branches. and removed 10.rebuild-darwin: 1-10 This PR causes between 1 and 10 packages to rebuild on Darwin. 10.rebuild-linux: 1-10 This PR causes between 1 and 10 packages to rebuild on Linux. labels Mar 11, 2019
@Mic92 Mic92 requested a review from adisbladis March 11, 2019 10:14
@kalbasit kalbasit force-pushed the nixpkgs_create-build-go-module branch from 48efa69 to c717237 Compare March 14, 2019 05:42
@sondr3 sondr3 mentioned this pull request Mar 14, 2019
Merged
10 tasks
Mic92
Mic92 reviewed Mar 14, 2019
View reviewed changes
Mic92
Mic92 reviewed Mar 14, 2019
View reviewed changes
@kalbasit kalbasit force-pushed the nixpkgs_create-build-go-module branch from c717237 to 627b088 Compare March 14, 2019 17:30
kalbasit added 11 commits March 14, 2019 10:31
@kalbasit
buildGoModule: function for packaging Go modules
28435e4 
The function buildGoModule builds Go programs managed with Go modules. It builds
a Go module through a two phase build:

- An intermediate fetcher derivation. This derivation will be used to
  fetch all of the dependencies of the Go module.
- A final derivation will use the output of the intermediate derivation
  to build the binaries and produce the final output.
@kalbasit
jx: use buildGoModule instead of buildGoPackage
a516275
@kalbasit
pet: use buildGoModule instead of buildGoPackage
ff11036
@kalbasit
hugo: use buildGoModule instead of buildGoPackage
3c7f228
@kalbasit
gotools: 2019-02-11 -> 2019-03-05
1a4cccd
@kalbasit
gotools: use buildGoModule instead of buildGoPackage
6ee0d36
@kalbasit
terminal-parrot: indent and use version when fetching dependencies
cfbe152
@kalbasit
terminal-parrot: 1.1.0 -> 1.1.1 using buildGoModule
a7586b3
@kalbasit
todoist: use buildGoModule instead of buildGoPackage
554406b
@kalbasit
hetzner-kube: 0.3.2rc1 -> 0.4.1 using buildGoModule
8f5b405
@kalbasit
hetzner-kube: set the version via ldflags
8f8e71c
@kalbasit kalbasit force-pushed the nixpkgs_create-build-go-module branch from af208c4 to 8f8e71c Compare March 14, 2019 17:34
@kalbasit kalbasit changed the title [DO NOT MERGE] buildGoModule: building Go binaries in two phases buildGoModule: building Go binaries in two phases Mar 14, 2019
@kalbasit
Copy link
Copy Markdown
Member Author

kalbasit commented Mar 14, 2019 •
edited
Loading

@Mic92 I've addressed your comments. I have also conducted some tests to verify with certainty that the modules downloaded are not affected by changes to the upstream repository, such as with the release of a new verison. This PR is now clear to be merged as well.

PTAL.

@GrahamcOfBorg GrahamcOfBorg added 8.has: package (new) This PR adds a new package 10.rebuild-darwin: 11-100 This PR causes between 11 and 100 packages to rebuild on Darwin. and removed 10.rebuild-darwin: 1-10 This PR causes between 1 and 10 packages to rebuild on Darwin. labels Mar 14, 2019
@kalbasit kalbasit merged commit a0e4f44 into NixOS:master Mar 14, 2019
@kalbasit kalbasit deleted the nixpkgs_create-build-go-module branch March 14, 2019 22:12
@eonpatapon eonpatapon mentioned this pull request Mar 15, 2019
Open
@kalbasit kalbasit mentioned this pull request Mar 17, 2019
Merged
10 tasks
@elasticdog elasticdog mentioned this pull request Mar 22, 2019
Merged
10 tasks
@hartzell hartzell mentioned this pull request Dec 1, 2019
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Mic92 Mic92 Mic92 approved these changes

@fpletz fpletz Awaiting requested review from fpletz

@nlewo nlewo Awaiting requested review from nlewo

@adisbladis adisbladis Awaiting requested review from adisbladis

Assignees

No one assigned

Labels

6.topic: golang Go is a high-level general purpose programming language that is statically typed and compiled. 8.has: documentation This PR adds or changes documentation 8.has: package (new) This PR adds a new package 10.rebuild-darwin: 11-100 This PR causes between 11 and 100 packages to rebuild on Darwin. 10.rebuild-linux: 11-100 This PR causes between 11 and 100 packages to rebuild on Linux. 11.by: package-maintainer This PR was created by a maintainer of all the package it changes.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@kalbasit @Mic92 @GrahamcOfBorg