Skip to content

Sync with upstream 2.29.0#67

Merged
cole-h merged 592 commits intodetsys-mainfrom
sync-2.29
May 22, 2025
Merged

Sync with upstream 2.29.0#67
cole-h merged 592 commits intodetsys-mainfrom
sync-2.29

Conversation

@edolstra
Copy link
Collaborator

Motivation

Context

edolstra and others added 30 commits April 14, 2025 15:18
Mention BLAKE3 in the Nix 2.27 release notes
`CONTENT_ADDRESSED` -> `NIX_TESTS_CA_BY_DEFAULT`
Now, both the unit and functional tests relating to derivation options
are tested both ways -- with input addressing and content-addressing
derivations.
Tests are updated accordingly.
This moves us towards getting rid of `ParsedDerivation` and just having
`DerivationOptions`.

Co-Authored-By: HaeNoe <[email protected]>
Only a much smaller `StructuredAttrs` remains, the rest is is now moved
to `DerivationOptions`.

This gets us quite close to `std::optional<StructuredAttrs>` and
`DerivationOptions` being included in `Derivation` as fields.
This requires that we refer to the `sourceInfo` instead of the
`result`. However, `sourceInfo` does not create a chain of basedir
resolution, so we add that back with `flakeDir`.
Test derivation options with content-addressing too
…s-exportReferencesGraph

Move `exportReferencesGraph` to `DerivationOptions`
Rename FlakeCache to InputCache and move it to libfetchers
Fix issue NixOS#13018, `sourceInfo` strict in `outputs`
…alues

We had fields set to the same values before in our test data. This is
not a problem per-se, but does mean we wouldn't catch certain mixups.
Now, the fields are set to distinct values (where possible), which makes
the test more robust.
This was almost a bug! It wasn't simply because another assignment would
clobber it later.
libexpr: fix UB in builtins.ceil and builtins.floor
Improve derivation "advanced attrs" test, fix almost-bug
The other parameters it took were somewhat implementation-specific.
This is unreleated to the other commits in this PR.
Now, most of it is in two new functions:
`LocalDerivationGoal::{,un}repareBuild`.

This might seems like a step backwards from coroutines --- now we have
more functions, and are stuck with class vars --- but I don't think it
needs to be.

There's a few options here:

- (Re)introduce coroutines for the isolated building logic. We could use the
  same coroutines types, or simpler ones specialized to this use-case.
  The `tryLocalBuild` caller can still use `Goal::Co`, and just will
  manually "pump" this inner coroutine.

- Return closures from each step. This is sort of like coroutines by
  hand, but it still allows us to stop writing down the local variables
  in each type.

  Being able to fully-use RAII again would be very nice!

- Keep top-level first-order functions like now, but make more
  functional. Instead of having one state object (`DerivationBuilder`)
  for all steps (setup, run, teardown), we can have separate structs for
  the live variables at each point we consume and return.

  This at least avoids "are these variables active at this time?"
  questions, but doesn't give us the full benefit of RAII as we must
  manually ensure FIFO create/destroy orders still.

One thing to note is that by keeping the `outputLock` unlocking in
`tryLocalBuild`, we are arguably uncovering a rebuild scheduling vs
building distinction, as the output locks are pretty squarely a
scheduling concern. It's nice that the builder doesn't need to know
about them at all.
We have a new `DerivationBuilder` struct, and `DerivationBuilderParams`
`DerivationBuilderCallbacks` supporting it.

`LocalDerivationGoal` doesn't subclass any of these, so we are ready to
now move them out to a new file!
…goal-encapsulation

Separate derivation building from the scheduler
This is done to prior to splitting, just like
05cc5a8 for
68f4c72.
The building logic is now free of the scheduling logic!

(The interface between them is just what is in the new header. This
makes it much easier to audit, and shrink over time.)
It seems reasonable to add both `$HOME/.profile/bin` and
`@localstatedir@/nix/profiles/default/bin` to `$PATH` for both user
local and daemon based nix execution.  Nix daemon execution mode does
not affect these path.
There seems to be no good reason for `nix-profile.fish` to set
`$MANPATH` while it being unset when `nix-profile-daemon.fish` is used.
While it seems unlikely that `$USER` will be unset while `$HOME` is set,
as `$USER` is not used in the script and as `nix-profile-daemon.fish` is
not checking `$USER`, it seems better to remove this check.

`nix-profile.fish` and `nix-profile-daemon.fish` now become identical.
…st-USER

nix-profile.fish: Do not check $USER
Mostly because they're causing an evaluation error in `nix flake
check` on macOS, and we don't need them right now for Determinate Nix.
@github-actions github-actions bot temporarily deployed to pull request May 16, 2025 16:27 Inactive
Ericson2314 and others added 4 commits May 16, 2025 12:36
…/pr-13212

docs: remove repeated "allowedReferences" and other lexical illusion (backport NixOS#13212)
In rendered form:

```diff
-Add references to the manual using ..
+Add references to the manual using [links like this](@docroot@/example.md)
```

(cherry picked from commit 147a34c)
…/pr-13221

doc: Render verbatim `@docroot@` on contributing page (backport NixOS#13221)
Need to investigate why store->getFSAccessor() breaks a test.
@github-actions github-actions bot temporarily deployed to pull request May 18, 2025 08:13 Inactive
@github-actions github-actions bot temporarily deployed to pull request May 18, 2025 11:21 Inactive
xokdvium and others added 4 commits May 18, 2025 19:46
As it turns out using `std::regex` is actually the bottleneck
for root discovery. Just substituting `std::` -> `boost::`
makes root discovery twice as fast (3x if counting only userspace time).

Some rather ad-hoc measurements to motivate the switch:

(On master)

```
nix build github:nixos/nix/1e822bd4149a8bce1da81ee2ad9404986b07914c#nix-cli --out-link result-1e822bd4149a8bce1da81ee2ad9404986b07914c
taskset -c 2,3 hyperfine "result-1e822bd4149a8bce1da81ee2ad9404986b07914c/bin/nix store gc --dry-run --max 0"
Benchmark 1: result-1e822bd4149a8bce1da81ee2ad9404986b07914c/bin/nix store gc --dry-run --max 0
  Time (mean ± σ):     481.6 ms ±   3.9 ms    [User: 336.2 ms, System: 142.0 ms]
  Range (min … max):   474.6 ms … 487.7 ms    10 runs
```

(After this patch)

```
taskset -c 2,3 hyperfine "result/bin/nix store gc --dry-run --max 0"
Benchmark 1: result/bin/nix store gc --dry-run --max 0
  Time (mean ± σ):     254.7 ms ±   9.7 ms    [User: 111.1 ms, System: 141.3 ms]
  Range (min … max):   246.5 ms … 281.3 ms    10 runs
```

`boost::regex` is a drop-in replacement for `std::regex`, but much faster.
Doing a simple before/after comparison doesn't surface any change in behavior:

```
result/bin/nix store gc --dry-run -vvvvv --max 0 |& grep "got additional" | wc -l
result-1e822bd4149a8bce1da81ee2ad9404986b07914c/bin/nix store gc --dry-run -vvvvv --max 0 |& grep "got additional" | wc -l
```

(cherry picked from commit 3a1301c)
This reduces the closure size on master by 40MiB.

```
$ nix build github:nixos/nix/1e822bd4149a8bce1da81ee2ad9404986b07914c#nix-store --out-link closure-on-master
$ nix build .#nix-store -L --out-link closure-without-icu
$ nix path-info --closure-size -h ./closure-on-master
/nix/store/8gwr38m5h6p7245ji9jv28a2a11w1isx-nix-store-2.29.0pre  124.4 MiB
$ nix path-info --closure-size -h ./closure-without-icu
/nix/store/k0gwfykjqpnmaqbwh23nk55lhanc9g24-nix-store-2.29.0pre   86.6 MiB
```

(cherry picked from commit f3090ef)
…/pr-13142

libstore: Use `boost::regex` for GC root discovery (backport NixOS#13142)
@github-actions github-actions bot temporarily deployed to pull request May 19, 2025 03:54 Inactive
@github-actions github-actions bot temporarily deployed to pull request May 19, 2025 14:10 Inactive
I am on a newer different nixpkgs branch, so I am getting this error

(cherry picked from commit 1290b7e)
@github-actions github-actions bot temporarily deployed to pull request May 19, 2025 14:26 Inactive
…/pr-13228

libutil-tests/json-utils: fix -Werror=sign-compare error (backport NixOS#13228)
Tagging release 2.29.0
@edolstra edolstra marked this pull request as ready for review May 22, 2025 12:53
@github-actions github-actions bot temporarily deployed to pull request May 22, 2025 13:00 Inactive
@cole-h cole-h added this pull request to the merge queue May 22, 2025
Merged via the queue into detsys-main with commit ea2f882 May 22, 2025
26 checks passed
@cole-h cole-h deleted the sync-2.29 branch May 22, 2025 20:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.