Sane cross-compiling through bootstrapping by Ericson2314 · Pull Request #21268 · NixOS/nixpkgs

Ericson2314 · 2016-12-18T20:13:24Z

Ok, this is it, my final and biggest top-level refactor for 2016!

I've factored things into separate PRs, so this builds on #21415 to provide a consistent idiom for bootstrapping which I extend and further utilize for cross compiling, and #21414 which cleans up the cross-compiling tests.

It should also eventually subsume #21388, but I am still getting things unbroken.

This isn't ready to merge (need to unbreak cross compiling completely) but is ready to review---only uninteresting bug fixes should be left.

OP

The first commit introduces a new abstraction for dealing with stdenv bootstrapping. It doesn't really make anything shorter, but at least it enforces a consistent ideom across all of them. This commit works, but I'd love advice on how to make linux and darwin better utilize it. @copumpkin I'm especially curious about your feedback, and whether this will help with your goal of automatically solving for bootstrapping stages (I hope it will! but not sure).

The second commit is just the bare-bones of what I want cross-compiling to look like. The big ideas are

To finally take advantage of bootstrapping rather than have all these *Cross packages and other hacks: "native dependencies" are actually stage n-1 dependencies
Properly distinguish between host and target in the autotools sense.

I'll probably end up splitting this into two PRs to get the first part merged sooner, but wanted to include the second part for now in order to better motivate the first.

Probably closes #14965
Takes a big stab at closing #10874
I shamelessly stole things from #18386 and will steal more!

CC @DavidEGrayson @nbp @shlevy @copumpkin @vcunat

mention-bot · 2016-12-18T20:13:26Z

@Ericson2314, thanks for your PR! By analyzing the history of the files in this pull request, we identified @edolstra, @errge, @gridaphobe, @LnL7 and @copumpkin to be potential reviewers.

DavidEGrayson · 2016-12-18T20:48:53Z

Hey, I'm trying out commit ab9eebbba565932ff92acbcb0a1a541119d95468 on my computer right now and it says:

error: undefined variable ‘crossSystem’ at /home/david/nixpkgs/pkgs/top-level/all-packages.nix:4775:20
(use ‘--show-trace’ to show detailed location information)

Ericson2314 · 2016-12-18T21:00:57Z

@DavidEGrayson yeah almost* every crossSystem needs to be triaged into hostPlatform or targetPlatform. There's a lot of work yet to be done.

*I am (for now) keeping crossSystem as the user-facing argument for nixpkgs, and letting the cross stdenv use the *platform args accordingly.

Ericson2314 · 2016-12-18T21:11:42Z

@DavidEGrayson yes everything is broken. I'm making the WIP now to get the basic idea out there. Fixing things is tedious but fairly easy.

Ericson2314 · 2016-12-18T21:46:20Z

@DavidEGrayson Working on GCC 5 now (also where did your comment go, did I delete it by accident or something?!?). How exactly do cross stdenv.cross current map to host vs target? I see both used in places for --target which is...confusing, to say the least

DavidEGrayson · 2016-12-18T21:56:55Z

I deleted my comment. In my pull request (#18386) I introduced targetCrossSystem and hostCrossSystem variables in the GCC recipe. For a native compiler they are both null. For a cross-compiler, targetCrossSystem will be equal to crossSystem. For a cross-built native compiler, targetCrossSystem and hostCrossSystem will both be equal to crossSystem. There is no support for other combinations yet.

Ericson2314 · 2016-12-18T21:59:51Z

Thanks! that helps a lot

DavidEGrayson · 2016-12-18T22:14:55Z

Hey, I don't know if your pull request does this yet, but the overrides from the vanilla stdenv should get applied to the final stdenv (the one that can cross-compile things). Otherwise, packages like zlib will change just because the crossSystem argument is supplied. I thought I would mention that because it might have a bearing on the architecture you are making, which I have not really taken the time to understand yet.

Ericson2314 · 2016-12-18T22:41:06Z

@DavidEGrayson I think I am doing that? IIRC the idea is if you traverse enough build-time deps from a crossDrv, one gets back to "build = host = target" (or target doesn't matter) first, and those should be overridden just like when not cross compiling and that's the top bootstrap stage.

Ericson2314 · 2016-12-19T00:50:58Z

On interesting complication is currently we like to build libc in the same faux-stage as gcc/clang. But that's wrong as its a runtime dep. Just as we have stdenvNoCC I think we also need a stdenvNoLibc or similar for this.

DavidEGrayson · 2016-12-19T01:17:22Z

libc is a build-time dependency because the gcc/clang compiler needs to know where it is so it can compile programs without the user having to specify a libc. I don't understand what is wrong.

Ericson2314 · 2016-12-19T01:38:04Z

@DavidEGrayson so I'm trying to to make "build time dep" ==> "drawn from stage n-1", but naively doign what you say with libc creates a gaping exception because libc must be built for the cross platform. I could have an additional bootstrapping stage for gccStaticCross, but then normal native deps would need to be "n-2" and it gets awkward.

Similarly, while many packages can meaningfully be built with differently-booted regular compilers, packages either do or don't need libc, so if I did crate another bootstrapping stage it ought to be a completely disjoint set of packages---just those which don't need libc. Again, sort of weird.

Finally, I think it's kind of a smell that libc needs to be treated differently than other libraries---in Rust myself and others have been working to try to make the rust standard library built and used as normally as possible. I'd love to ditch the wrapper script and just make libc a default buildInput--libc of course would disable the default to avoid a cycle.

DavidEGrayson · 2016-12-19T02:41:17Z

I don't understand how the n-1 thing is a good principle; it seems like it would cause you to rebuild some dependencies unnecessarily (e.g. having a zlib at stage k and another one at k+1). My earlier comment is an example where you want to have several dependencies that come from stage n-2. And if we're talking about a normal library and normal program they both would come from the same (final) stage. Also, libc needs to be special because it is part of stdenv.

Ericson2314 · 2016-12-20T06:10:33Z

Ok, I scaled things back leaving the actual hostPlatform targetPlatform rename/refactor as future work. crossSystem vs stdenv.cross is an adequate hack for now to keep the diff down.

Ericson2314 · 2016-12-24T19:38:26Z

OK this is ready to be reviewed!

I haven't tested cross-compiling as much---it ought to have a much better foundation than before, but not necessarily be less broken. I have however tested that linux hashes are unchanged and the travis script passes locally.

Once this is reviewed and merged, I'll do the big crossSystem => hostSystem stdenv.cross => targetSystem rename as a separate PR as it's very big and very boring. I'd like to simultaneously assist with rebasing @DavidEGrayson's work, and use that PR to actually ensure cross compiling works, and is tested so it won't break again.

This effectively reverts commit d27403b.

For the wrapper script changes, see the big comment at the top of `pkgs/top-level/build-wrappers.nix`, which explains that file's motivation. Most of the churn in this commit is because those wrappers are now only available in `buildPackages`. As to the removal of the `*Cross` derivations, the basic idea is further utilizing proper bootstrapping stages. We can simply always pass `crossSystem` or `stdenv.cross or null` and rely on those expressions being null for native packages. Furthermore, we have enough bootstrapping stages so that every variant of a package we need can be found in some current or previous stage. Essentially, the `*Cross` derivations were a poor attempt at squashing multiple stages together. gccCross* alone among the *Cross derivations remains for now to be dealt with in a later commit. Also, the unwrapped GCCs should not be passed a libc even when cross compiling. That too remains to be dealt with.

…iling Previously, all overrides from the final native stage were chucked. This is sound but caused unnecessary rebuilds. It would be tempting to just keep all overrides, but then pkgs like gcc and binutils would not target the correct platform--they would still target the (native) build platform. Now, each native final stage responsibly just overrides such packages of the target platform is that used to build the overriding packages.

Ericson2314 · 2017-02-16T19:33:10Z

Heads up, in https://github.com/obsidiansystems/nixpkgs/tree/ghc-android I have a huge number of hash-breaking, but tested improvements. The git history is unclean however. I'm thinking I clean that up and combine it with the few parts of this I haven't already incorporated into this.

After that, I can make a staging which will really clean things up. This will be big, but actually quite simple mainly making most buildPlatform != targetPlatform unconditional true, and with which I think cross compiling can be considered solved 😀.

FRidh · 2019-03-03T08:27:24Z

@Ericson2314 what should happen with this PR? It hasn't been touched in a long while, and it for sure won't make 19.03.

Ericson2314 · 2019-03-03T14:18:18Z

@FRidh I've basically been keeping it open as a TODO to triage everything in it has either been done or shouldn't be done. Removed the milestone.

matthewbauer · 2019-03-14T21:25:56Z

pkgs/top-level/stage.nix

+  # catch improperly *using* wrappers from `pkgs` instead of `buildPackages`.
+  buildWrappers = self: super:
+    if targetPackages == {}
+    then {};


matthewbauer · 2019-03-14T21:29:23Z

pkgs/top-level/all-packages.nix

-    else if name == "libSystem" then darwin.xcode
-    else throw "Unknown libc";
+  libc =
+    if crossSystem != null


We shouldn't need crossSystem here. Why not just use stdenv.hostPlatform.libc? libc should come from pkgsTargetTarget right?

stale · 2020-06-02T05:10:34Z

Thank you for your contributions.

This has been automatically marked as stale because it has had no activity for 180 days.

If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity.

Here are suggestions that might help resolve this more quickly:

Search for maintainers and people that previously touched the related code and @ mention them in a comment.
Ask on the NixOS Discourse.
Ask on the #nixos channel on irc.freenode.net.

Ericson2314 · 2020-06-14T19:49:43Z

Everything in here I think is done or won't be done.

In particular the "wrappers in next stage" idea isn't being done---better to just keep on using targetPackages to get libc etc until #26004 solves the problem for real.

shlevy self-requested a review December 18, 2016 22:46

Ericson2314 force-pushed the host-target branch 4 times, most recently from 3f3203a to 6eb71f5 Compare December 20, 2016 06:09

Ericson2314 force-pushed the host-target branch 3 times, most recently from 19e5bc5 to c29e7ec Compare December 24, 2016 19:29

Ericson2314 requested a review from nbp December 24, 2016 19:30

Ericson2314 changed the title ~~[WIP] Normalize Stdenv bootstrapping and overhaul cross-compiling~~ Normalize Stdenv bootstrapping and overhaul cross-compiling Dec 24, 2016

This was referenced Dec 24, 2016

libiconv: Correct cross-building #15021

Closed

Make cross-compiling with the crossSystem argument work (v2) #21388

Closed

Ericson2314 force-pushed the host-target branch from c29e7ec to 1b98554 Compare December 24, 2016 20:23

Ericson2314 self-assigned this Dec 24, 2016

Ericson2314 added this to the 17.03 milestone Dec 24, 2016

Ericson2314 added 4 commits February 7, 2017 22:56

libiconv: Restore proper semantics of building library for host system

688bf2a

This effectively reverts commit d27403b.

ccwrapper: Fewer cross args

08a1855

Ericson2314 force-pushed the host-target branch from 5bd3168 to e63cd62 Compare February 8, 2017 20:03

globin modified the milestones: 17.09, 17.03 Mar 15, 2017

Ericson2314 mentioned this pull request Apr 18, 2017

[WIP] Cross compilation changes, with enphasis on (haskell) GHC HEAD #24983

Closed

3 tasks

Ericson2314 modified the milestone: 17.09 Aug 16, 2017

Ericson2314 mentioned this pull request Jun 26, 2018

wrapCC should use buildPackages #42601

Closed

matthewbauer modified the milestones: 17.09, 18.09, 19.03 Oct 1, 2018

FRidh modified the milestones: 19.03, 19.09 Mar 3, 2019

matthewbauer reviewed Mar 14, 2019

View reviewed changes

alyssais added the 2.status: merge conflict This PR has merge conflicts with the target branch label Mar 14, 2020

stale bot added the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Jun 2, 2020

stale bot removed the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Jun 14, 2020

Ericson2314 closed this Jun 14, 2020

Uh oh!

Conversation

Ericson2314 commented Dec 18, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mention-bot commented Dec 18, 2016

Uh oh!

DavidEGrayson commented Dec 18, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Ericson2314 commented Dec 18, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Ericson2314 commented Dec 18, 2016

Uh oh!

Ericson2314 commented Dec 18, 2016

Uh oh!

DavidEGrayson commented Dec 18, 2016

Uh oh!

Ericson2314 commented Dec 18, 2016

Uh oh!

DavidEGrayson commented Dec 18, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Ericson2314 commented Dec 18, 2016

Uh oh!

Ericson2314 commented Dec 19, 2016

Uh oh!

DavidEGrayson commented Dec 19, 2016

Uh oh!

Ericson2314 commented Dec 19, 2016

Uh oh!

DavidEGrayson commented Dec 19, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Ericson2314 commented Dec 20, 2016

Uh oh!

Ericson2314 commented Dec 24, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Ericson2314 commented Feb 16, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

FRidh commented Mar 3, 2019

Uh oh!

Ericson2314 commented Mar 3, 2019

Uh oh!

matthewbauer Mar 14, 2019

Choose a reason for hiding this comment

Uh oh!

matthewbauer Mar 14, 2019

Choose a reason for hiding this comment

Uh oh!

stale bot commented Jun 2, 2020

Uh oh!

Ericson2314 commented Jun 14, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

11 participants

Ericson2314 commented Dec 18, 2016 •

edited

Loading

DavidEGrayson commented Dec 18, 2016 •

edited

Loading

Ericson2314 commented Dec 18, 2016 •

edited

Loading

DavidEGrayson commented Dec 18, 2016 •

edited

Loading

DavidEGrayson commented Dec 19, 2016 •

edited

Loading

Ericson2314 commented Dec 24, 2016 •

edited

Loading

Ericson2314 commented Feb 16, 2017 •

edited

Loading