Somewhat saner cross-compiling through bootstrapping #22107
Merged
Ericson2314 merged 14 commits intoNixOS:masterfrom Jan 24, 2017
Merged
Somewhat saner cross-compiling through bootstrapping #22107Ericson2314 merged 14 commits intoNixOS:masterfrom
Ericson2314 merged 14 commits intoNixOS:masterfrom
Conversation
[N.B., this package also applies to the commits that follow it in the same
PR.]
In most cases, buildPackages = pkgs so things work just as before. For
cross compiling, however, buildPackages is resolved as the previous
bootstrapping stage. This allows us to avoid the mkDerivation hacks cross
compiling currently uses today.
To avoid a massive refactor, callPackage will splice together both package
sets. Again to avoid churn, it uses the old `nativeDrv` vs `crossDrv` to do
so. So now, whether cross compiling or not, packages with get a `nativeDrv`
and `crossDrv`---in the non-cross-compiling case they are simply the same
derivation. This is good because it reduces the divergence between the
cross and non-cross dataflow. See `pkgs/top-level/splice.nix` for a comment
along the lines of the preceding paragraph, and the code that does this
splicing.
Also, `forceNativeDrv` is replaced with `forceNativePackages`. The latter
resolves `pkgs` unless the host platform is different from the build
platform, in which case it resolves to `buildPackages`. Note that the
target platform is not important here---it will not prevent
`forcedNativePackages` from resolving to `pkgs`.
--------
Temporarily, we make preserve some dubious decisions in the name of preserving
hashes:
Most importantly, we don't distinguish between "host" and "target" in the
autoconf sense. This leads to the proliferation of *Cross derivations
currently used. What we ought to is resolve native deps of the cross "build
packages" (build = host != target) package set against the "vanilla
packages" (build = host = target) package set. Instead, "build packages"
uses itself, with (informally) target != build in all cases.
This is wrong because it violates the "sliding window" principle of
bootstrapping stages that shifting the platform triple of one stage to the
left coincides with the next stage's platform triple. Only because we don't
explicitly distinguish between "host" and "target" does it appear that the
"sliding window" principle is preserved--indeed it is over the reductionary
"platform double" of just "build" and "host/target".
Additionally, we build libc, libgcc, etc in the same stage as the compilers
themselves, which is wrong because they are used at runtime, not build
time. Fixing this is somewhat subtle, and the solution and problem will be
better explained in the commit that does fix it.
Commits after this will solve both these issues, at the expense of breaking
cross hashes. Native hashes won't be broken, thankfully.
--------
Did the temporary ugliness pan out? Of the packages that currently build in
`release-cross.nix`, the only ones that have their hash changed are
`*.gcc.crossDrv` and `bootstrapTools.*.coreutilsMinimal`. In both cases I
think it doesn't matter.
1. GCC when doing a `build = host = target = foreign` build (maximally
cross), still defines environment variables like `CPATH`[1] with
packages. This seems assuredly wrong because whether gcc dynamically
links those, or the programs built by gcc dynamically link those---I
have no idea which case is reality---they should be foreign. Therefore,
in all likelihood, I just made the gcc less broken.
2. Coreutils (ab)used the old cross-compiling infrastructure to depend on
a native version of itself. When coreutils was overwritten to be built
with fewer features, the native version it used would also be
overwritten because the binding was tight. Now it uses the much looser
`BuildPackages.coreutils` which is just fine as a richer build dep
doesn't cause any problems and avoids a rebuild.
So, in conclusion I'd say the conservatism payed off. Onward to actually
raking the muck in the next PR!
[1]: https://gcc.gnu.org/onlinedocs/gcc/Environment-Variables.html
The long term goal is a big replace:
{ inherit system platform; } => buildPlatform
crossSystem => hostPlatform
stdenv.cross => targetPlatform
And additionally making sure each is defined even when not cross compiling.
This commit refactors the bootstrapping code along that vision, but leaves
the old identifiers with their null semantics in place so packages can be
modernized incrementally.
Semantics should be unchanged
…form` This is an implementation detail of how the bootstrapping chain is chosen, and thus need not be in the manual.
Do so in the stdenv section where the other two are discussed. This can be done without brining up cross-compilation by talking about build-time vs run-time.
|
@Ericson2314, thanks for your PR! By analyzing the history of the files in this pull request, we identified @edolstra, @vcunat, @viric, @LnL7 and @copumpkin to be potential reviewers. |
Member
|
@Ericson2314 Besides docs has anything changed since my last review? |
Member
Author
|
@shlevy just docs and a tiny bit of comment formatting. |
Member
Author
|
Travis is backlogged, but I ran (the normal) tests locally and they pass. https://hydra.nixos.org/jobset/nixpkgs/sonarpulse-cross-tepid also checks out (no regressions). |
Member
|
:LGTM: |
shlevy
referenced
this pull request
Feb 6, 2017
This makes the disk usage footprint of building the kernel smaller in 3 ways: 1) There is no separate kernel source derivation 2) Rather than using the entire build tree, only the output of make modules_prepare is kept in the $dev output (plus the module symbol versioning file generated during the build) 3) Only the subset of the source tree known to be needed for external builds is kept in $dev Note that while 2) is supported by official kernel documentation, I couldn't find any source describing what we need to keep for 3). I've started with the bare minimum (the main Makefile is called by the Makefile generated by make modules_prepare) and we can/should add more as needed for kernelPackages. Signed-off-by: Shea Levy <[email protected]>
Ericson2314
added a commit
to Ericson2314/nixpkgs
that referenced
this pull request
Feb 7, 2017
…gh bootstrapping Merge commit '5ad696b06763b7c5c6e1aa7ed848a50bff78c8d5' into release-cross-darwin
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
#21915 got closed by mistake due to a force push. @shlevy I wrote docs so this should be ready to go unless we want someone to review the docs.
The docs have some explicit "to be written" sections. I consider that fine, and the docs rightfully WIP, because some stuff will change with future PRs and there is little point writing it just to rewrite it.