example GitHub workflow for for building Spago statically by cdepillabout · Pull Request #882 · purescript/spago

cdepillabout · 2022-05-05T17:53:42Z

Description of the change

wip

Checklist:

Added the change to the "Unreleased" section of the changelog
Added some example of the new feature to the README
Added a test for the contribution (if applicable)

P.S.: the above checks are not compulsory to get a change merged, so you may skip them. However, taking care of them will result in less work for the maintainers and will be much appreciated 😊

cdepillabout · 2022-05-06T07:04:33Z

@f-f This PR is a rough proof-of-concept of adding a GitHub Action job for building Spago statically.

Currently, this PR just adds static job to the build.yaml GitHub Action. I'd like you to take a look at it, and if it generally looks alright, then I'd go though and add more documentation, as well as adding some code to the release.yaml GitHub Action to actually upload the static binary to the GitHub release.

One thing to notice is that building the statically-linked spago binary takes about 20m without using any sort of cache, so that seems basically in-line with the Windows an OSX builds. You mentioned in NixOS/nixpkgs#171604 (comment) about potentially having adding a cachix cache. That's something we could do to speed up the build, but I see it more as a potential future improvement, and not something that is strictly necessary to add in this PR.

.github/workflows/build.yml

f-f · 2022-05-06T07:22:59Z

@cdepillabout this looks amazing! ❤️

I have only one question: what would we do when upgrading Haskell dependencies?
My understanding is that the static building infra from Nixpkgs is not always reliable (or at least it has not been so far, but I'd love to hear that we can trust it nowadays), so bumping the Nixpkgs commit to get the latest Haskell dependencies could be risky?
But then I think that if we'd not do it we'd have to rebuild a whole lot of dependencies? (lengthening the build considerably, at least without cache)

cdepillabout · 2022-05-06T07:26:41Z

default.nix

+  nixpkgs-src = builtins.fetchTarball {
+    # This is one of the last commits from https://github.com/NixOS/nixpkgs/pull/169513.
+    # This commit is used because it is for the `haskell-updates` branch.
+    # Hydra builds a GHC for static linking on this branch, so we can get
+    # most dependencies from the NixOS cache.
+    url = "https://github.com/NixOS/nixpkgs/archive/999a0e87f9b1792e229c53602e80bda6dd52c105.tar.gz";
+    sha256 = "1p0vgmdz884gld4p2zcdy7g0w7xznc4kxj80wf5pk2qdxicbnscy";
+  };


One problem with building from Nixpkgs like this is that we get the Haskell package versions in whatever commit of Nixpkgs we use. So with the above commit, we are building with mainly LTS-19, while the stack.yaml in this repo is still LTS-18.

While this discrepancy doesn't seem to cause any big problems now, it is at least unfortunate that we're not using the same exact package versions.

(haskell.nix has tools for generating a Haskell package set using versions from a stack.yaml and Stackage resolver, but haskell.nix comes with its own set of problems.)

This is somewhat concerning - new lts versions might break behaviour (it has happened in the past), and we wouldn't notice this happening since we don't run tests for this binary.

So I'd like to at least be on the same lts and compiler version, even if the versions of the packages are slightly different.
Would it make sense to generate the package versions in the stack.yaml from the ones in Nixpkgs? (and of course check that they match in CI)
This would mean depending on whatever versions of packages coming from Nixpkgs, which is not the best (because we introduce a dependency) but it's pretty alright in the end. I assume we would/could tolerate some overrides in case we need them for the build to work.

That's a good point, and a good reason for us to make sure we use the same package versions.

Would it make sense to generate the package versions in the stack.yaml from the ones in Nixpkgs? (and of course check that they match in CI)

I think this would make sense, although I don't know of any sort of tool that is able to generate a stack.yaml from a given Nixpkgs commit. I'm not sure how difficult it would be to write something like this.

For now, it would probably be the easiest to just manually keep the Nixpkgs Haskell package set and the resolver from the stack.yaml file the same. This PR uses Nixpkgs at commit NixOS/nixpkgs@999a0e8, which is included in PR NixOS/nixpkgs#169513. At the top of that PR, you can see there is a commit bumping the Nixpkgs Haskell stuff to LTS-19.4.

So maybe I could take a stab at bumping spago to LTS-19.4 (possibly in this PR, or a separate PR). Then we'd be using the same LTS version and GHC version in both stack.yaml and the Nixpkgs Haskell package set.

This would mean depending on whatever versions of packages coming from Nixpkgs, which is not the best (because we introduce a dependency) but it's pretty alright in the end.

I agree this is somewhat unfortunate. Ideally there would be a tool that could generate a Nixpkgs Haskell package set from a stack.yaml file, but we don't currently have something like that. (There used to be a tool https://github.com/input-output-hk/stack2nix that would do this, but it hasn't been maintained over the last few years.)

I assume we would/could tolerate some overrides in case we need them for the build to work.

Yeah, the Nixpkgs Haskell side is somewhat flexible, and it should be possible to add any overrides that we may need. I'd of course be happy to help out with this.

cdepillabout · 2022-05-06T07:27:57Z

default.nix

+                      spagoDrvFromCabal2nix =
+                        final.haskell.lib.overrideCabal
+                          (hfinal.callCabal2nix "spago" src {})
+                          (drv: {
+                            doCheck = false;
+                            postUnpack = (drv.postUnpack or "") + ''
+                              # Spago includes the following two files directly into the binary
+                              # with Template Haskell.  They are fetched at build-time from the
+                              # `purescript-docs-search` repo above.  If they cannot be fetched at
+                              # build-time, they are pulled in from the `templates/` directory in
+                              # the spago source.
+                              #
+                              # However, they are not actually available in the spago source, so they
+                              # need to fetched with nix and put in the correct place.
+                              # https://github.com/spacchetti/spago/issues/510
+                              cp ${docsSearchApp_0_0_10} "$sourceRoot/templates/docs-search-app-0.0.10.js"
+                              cp ${docsSearchApp_0_0_11} "$sourceRoot/templates/docs-search-app-0.0.11.js"
+                              cp ${purescriptDocsSearch_0_0_10} "$sourceRoot/templates/purescript-docs-search-0.0.10"
+                              cp ${purescriptDocsSearch_0_0_11} "$sourceRoot/templates/purescript-docs-search-0.0.11"
+
+                              # For some weird reason, on Darwin, the open(2) call to embed these files
+                              # requires write permissions. The easiest resolution is just to permit that
+                              # (doesn't cause any harm on other systems).
+                              chmod u+w \
+                                "$sourceRoot/templates/docs-search-app-0.0.10.js" \
+                                "$sourceRoot/templates/purescript-docs-search-0.0.10" \
+                                "$sourceRoot/templates/docs-search-app-0.0.11.js" \
+                                "$sourceRoot/templates/purescript-docs-search-0.0.11"
+                            '';
+                          });
+                    in
+                    final.haskell.lib.compose.generateOptparseApplicativeCompletion
+                      "spago"
+                      spagoDrvFromCabal2nix;


These overrides are basically taken directly from Nixpkgs.

cdepillabout · 2022-05-06T07:43:51Z

@f-f

I have only one question: what would we do when upgrading Haskell dependencies? My understanding is that the static building infra from Nixpkgs is not always reliable (or at least it has not been so far, but I'd love to hear that we can trust it nowadays), so bumping the Nixpkgs commit to get the latest Haskell dependencies could be risky? But then I think that if we'd not do it we'd have to rebuild a whole lot of dependencies?

In my opinion, this is basically the biggest problem.

A few months ago, there was a PR that got static-linking working pretty reliably in Nixpkgs for Haskell executables: NixOS/nixpkgs#162374. With this PR, most Haskell executables can be built statically (at least with GHC-9). It seems like there are now enough people interesting in static-linking and cross-compilation in Nixpkgs that the Musl-related infrastructure is pretty solid.

We also have some checks in our CI to make sure that statically-linking Haskell executables is working. This gives us Musl-built GHCs for static linking from the NixOS cache, so end users (or the spago CI) don't have to build GHC themselves. In my opinion, moving forward, relying on the Haskell static-linking facility in Nixpkgs isn't a crazy idea.

However, the bigger problem is what do to when upgrading Haskell dependencies. The Haskell packages in Nixpkgs are generally set as the latest LTS release. There are times when this matches the LTS release that Spago is using, but sometimes Spago lags behind. In the past, this hasn't seemed to prove a big problem maintaining Spago in Nixpkgs, but like I said above, it would be unfortunate for non-Nix-using contributors to be faced with these kinds of errors.

My suggestion would be to move forward with this PR as-is, but keep an eye out for these types of problems in the future. If the Nix-related stuff seems to be turning away too many contributors (or taking up too much of our time fixing things), then we could always consider removing it.

cdepillabout · 2022-05-06T07:48:00Z

default.nix

+                        final.haskell.lib.overrideCabal
+                          (hfinal.callCabal2nix "spago" src {})
+                          (drv: {
+                            doCheck = false;


Since spago uses the network during tests, I've had to disable the tests here.

We end up not really testing this Nix-built spago binary, which is unfortunate. We could potentially add some simple tests to the build.yaml CI file, like running spago version, spago init, spago build, etc.

We do have plans to be able to tolerate network loss - in that case we would refactor the test suite to be able to function without network (and introduce a section that explicitly exercises the network parts). That's some work though, so I think for now it might be easier to concentrate on matching the dependencies

f-f · 2023-05-04T16:40:10Z

I'll close this as we're about to switch to a PureScript-based Spago.

cdepillabout added 2 commits May 6, 2022 02:52

Add default.nix for building spago statically using Nix.

b0a0782

Add example GitHub workflow for building spago statically with Nix.

8ab0276

cdepillabout mentioned this pull request May 5, 2022

spago: 0.20.8 -> 0.20.9 NixOS/nixpkgs#171604

Merged

13 tasks

cdepillabout commented May 6, 2022

View reviewed changes

.github/workflows/build.yml Show resolved Hide resolved

cdepillabout commented May 6, 2022

View reviewed changes

cdepillabout mentioned this pull request May 6, 2022

Static binaries for Linux #528

Closed

f-f closed this May 4, 2023

f-f deleted the static-spago branch October 3, 2023 18:36

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

example GitHub workflow for for building Spago statically#882

example GitHub workflow for for building Spago statically#882
cdepillabout wants to merge 2 commits intomasterfrom
static-spago

cdepillabout commented May 5, 2022 •

edited

Loading

Uh oh!

cdepillabout commented May 6, 2022

Uh oh!

Uh oh!

f-f commented May 6, 2022

Uh oh!

cdepillabout May 6, 2022

Uh oh!

f-f May 6, 2022

Uh oh!

cdepillabout May 7, 2022

Uh oh!

cdepillabout May 6, 2022

Uh oh!

cdepillabout commented May 6, 2022 •

edited

Loading

Uh oh!

cdepillabout May 6, 2022

Uh oh!

f-f May 6, 2022

Uh oh!

f-f commented May 4, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

cdepillabout commented May 5, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of the change

Checklist:

Uh oh!

cdepillabout commented May 6, 2022

Uh oh!

Uh oh!

f-f commented May 6, 2022

Uh oh!

cdepillabout May 6, 2022

Choose a reason for hiding this comment

Uh oh!

f-f May 6, 2022

Choose a reason for hiding this comment

Uh oh!

cdepillabout May 7, 2022

Choose a reason for hiding this comment

Uh oh!

cdepillabout May 6, 2022

Choose a reason for hiding this comment

Uh oh!

cdepillabout commented May 6, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cdepillabout May 6, 2022

Choose a reason for hiding this comment

Uh oh!

f-f May 6, 2022

Choose a reason for hiding this comment

Uh oh!

f-f commented May 4, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

cdepillabout commented May 5, 2022 •

edited

Loading

cdepillabout commented May 6, 2022 •

edited

Loading