Skip to content

{nixos-container,lsb-release}: use substitute rather than substituteAll, fix cross-compilation#300322

Closed
philiptaron wants to merge 2 commits intoNixOS:stagingfrom
philiptaron:toplevel-substituteAll
Closed

{nixos-container,lsb-release}: use substitute rather than substituteAll, fix cross-compilation#300322
philiptaron wants to merge 2 commits intoNixOS:stagingfrom
philiptaron:toplevel-substituteAll

Conversation

@philiptaron
Copy link
Contributor

@philiptaron philiptaron commented Mar 30, 2024

Description of changes

This change fixes cross-compilation of the lsb-release and nixos-container scripts, all of which used the stdenv environment variable shell instead of runtimeShell string as the shebang... leading to a cross-compilation mess.

There's no other change in produced output. Rationale for using substitute instead of substituteAll is here:

Previously:

Things done

  • Built on platform(s)
    • x86_64-linux
    • aarch64-linux
    • x86_64-darwin
    • aarch64-darwin
  • Tested, as applicable:
  • Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
  • Tested basic functionality of all binary files (usually in ./result/bin/)
  • 24.05 Release Notes (or backporting 23.05 and 23.11 Release notes)
    • (Package updates) Added a release notes entry if the change is major or breaking
    • (Module updates) Added a release notes entry if the change is significant
    • (Module addition) Added a release notes entry if adding a new NixOS module
  • Fits CONTRIBUTING.md.

@philiptaron philiptaron requested a review from roberth March 30, 2024 21:06
@ofborg ofborg bot requested review from Artturin and primeos March 30, 2024 21:29
@ofborg ofborg bot added 10.rebuild-darwin: 11-100 This PR causes between 11 and 100 packages to rebuild on Darwin. 10.rebuild-linux: 501+ This PR causes many rebuilds on Linux and should normally target the staging branches. 10.rebuild-linux: 2501-5000 This PR causes many rebuilds on Linux and should target the staging branches. labels Mar 30, 2024
@philiptaron philiptaron changed the base branch from master to staging April 1, 2024 19:18
@philiptaron

This comment was marked as outdated.

@roberth
Copy link
Member

roberth commented Apr 2, 2024

The shell substitutions look like the value would come from buildPlatform, not hostPlatform (the default) like the other substitutions. Does it really exist? hello.drvPath doesn't have it. Otherwise I wouldn't be surprised if these packages were broken when cross-compiled.

@philiptaron
Copy link
Contributor Author

The shell substitutions look like the value would come from buildPlatform, not hostPlatform (the default) like the other substitutions. Does it really exist? hello.drvPath doesn't have it. Otherwise I wouldn't be surprised if these packages were broken when cross-compiled.

Right you are. I did the following on master:

nix-build -A pkgsCross.aarch64-multiplatform-musl.deterministic-uname

Then looked at the shebang at the top of the result (which is where @shell@ expands):

/nix/store/a1s263pmsci9zykm5xcdf7x9rv26w6d5-bash-5.2p26/bin/bash

Then used file:

$ file /nix/store/a1s263pmsci9zykm5xcdf7x9rv26w6d5-bash-5.2p26/bin/bash
/nix/store/a1s263pmsci9zykm5xcdf7x9rv26w6d5-bash-5.2p26/bin/bash: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /nix/store/ddwyrxif62r8n6xclvskjyy6szdhvj60-glibc-2.39-5/lib/ld-linux-x86-64.so.2, BuildID[sha1]=1098c4d83aff81fd6ddd5bc06fb9442388f4f51d, for GNU/Linux 3.10.0, not stripped

Definitely wrong.

It looks like the correct thing to do is use runtimeShell.

@philiptaron philiptaron changed the title nixos-container, deterministic-uname, lsb-release: use substitute rather than substituteAll nixos-container, deterministic-uname, lsb-release: use substitute rather than substituteAll, fix cross-compilation Apr 2, 2024
@ghost
Copy link

ghost commented Apr 3, 2024

while substituteAll implementation may be terrible, the interface is much nicer than using substitute. couldn't substitueAll be re implemented so that the key / value pairs just passed to substitute using the --subst-var-by flag? like:

# some nix code 
substituteAllish {
 src = ./the/file;
 substitutions = {
   inherit coreutils;
   stuff = someComputation;
   etc.
 };
};
# substituteAllish mock implementation
{ lib, src, substitutions):
let
subs =  lib.flatten ( map (pair:  [ "--subst-var-by" pair.name pair.value ]) (lib.attrsets.attrsToList substitutions ));
in
substitute { inherit src; substitutions = subs; etc. }

?

@philiptaron
Copy link
Contributor Author

philiptaron commented Apr 3, 2024

I've been cooking exactly that patch @a-n-n-a-l-e-e. My hope is that I'll get through enough of these substituteAll refactorings to get a sense of how often there's a defect embedded in them to recommend a secondary approach. So far, there's basically a defect per patch.

Edit: made a PR with the cooked patch: #301350

@philiptaron
Copy link
Contributor Author

Let's collaborate on what better would look like on #237216.

@philiptaron philiptaron mentioned this pull request Apr 3, 2024
4 tasks
@wegank wegank added the 2.status: merge conflict This PR has merge conflicts with the target branch label Jul 4, 2024
@philiptaron philiptaron force-pushed the toplevel-substituteAll branch from 5ac45d5 to c89f4ad Compare July 26, 2024 22:47
@philiptaron philiptaron changed the title nixos-container, deterministic-uname, lsb-release: use substitute rather than substituteAll, fix cross-compilation {nixos-container,lsb-release}: use substitute rather than substituteAll, fix cross-compilation Jul 26, 2024
@philiptaron philiptaron removed the 2.status: merge conflict This PR has merge conflicts with the target branch label Jul 26, 2024
@ofborg ofborg bot added 6.topic: cross-compilation Building packages on a different platform than they will be used on 10.rebuild-darwin: 1-10 This PR causes between 1 and 10 packages to rebuild on Darwin. 10.rebuild-linux: 101-500 This PR causes between 101 and 500 packages to rebuild on Linux. and removed 10.rebuild-darwin: 11-100 This PR causes between 11 and 100 packages to rebuild on Darwin. 10.rebuild-linux: 501+ This PR causes many rebuilds on Linux and should normally target the staging branches. 10.rebuild-linux: 2501-5000 This PR causes many rebuilds on Linux and should target the staging branches. labels Jul 27, 2024
@wegank wegank added the 2.status: merge conflict This PR has merge conflicts with the target branch label Nov 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

2.status: merge conflict This PR has merge conflicts with the target branch 6.topic: cross-compilation Building packages on a different platform than they will be used on 10.rebuild-darwin: 1-10 This PR causes between 1 and 10 packages to rebuild on Darwin. 10.rebuild-linux: 101-500 This PR causes between 101 and 500 packages to rebuild on Linux.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants