2.18 as a baseline was what was previously discussed (with you and @Ma27, I believe?). It seems reasonable to me. I don’t know if we want to set a particular point version or not; I believe we’re on 2.3.bignumber because a bunch of things got backported specifically so they could be used in Nixpkgs.

It’s unfortunate that 2.18 is no longer in Nixpkgs so we can’t use it directly for eval checks, but since we check parsing against Lix already it seems sensible to me to use it for baseline eval checks too, and in any case having a CI baseline that gets security patches seems like a good idea. Maybe we can move nixVersions.minimal outside of nixVersions to solve the jurisdictional issues here and call it nixForEvalBaseline or something.

I know @flokli said he still considers 2.3 evaluation compatibility important the last time we talked about it on Matrix. That was before the vulnerabilities, though. If Tvix/Snix match 2.3, I suppose we could add those to CI? But IIRC they can’t evaluate all of Nixpkgs properly yet, so we might not want to make it load‐bearing. I could be wrong though.

The PR title is inaccurate.

It’s unfortunate that 2.18 is no longer in Nixpkgs so we can’t use it directly for eval checks, but since we check parsing against Lix already it seems sensible to me to use it for baseline eval checks too, and in any case having a CI baseline that gets security patches seems like a good idea.

To be clear: We only do eval with Nix 2.29 right now. We do parse checks with Nix 2.3, 2.29 and Lix.

Do I understand correctly, that you're suggesting to just remove Nix 2.3 from that, and only do parse checks with 2.29 and Lix?

Aka: Do we consider Lix to still reflect "2.18", even if it advances?

The PR title is inaccurate.

This was written while I still assumed Lix to be something like 2.91.x, in which case it would have been accurate. I'm still not sure how to deal with the above.

To be clear: We only do eval with Nix 2.29 right now. We do parse checks with Nix 2.3, 2.29 and Lix.

Do I understand correctly, that you're suggesting to just remove Nix 2.3 from that, and only do parse checks with 2.29 and Lix?

Right. I thought we might do some other checks against the minimal version too. I think it would be good to have more thorough checks of eval against what we claim as our baseline – especially, checks that no derivation/output hashes change! – but I assume it would probably have to wait for merge queues.

Aka: Do we consider Lix to still reflect "2.18", even if it advances?

My understanding is that Lix do not intend to introduce any new language‐accessible functionality without explicit feature gating or versioning – at least out of “obviously unstable” edge cases? – but I can’t speak with confidence there; maybe that only applies to syntax rather than built‐ins? Hopefully someone from the Lix team can comment more authoritatively. But there are definitely things in newer versions of Nix that Lix does not implement and I think does not intend to, so I think it’s sensible as a “keep us honest” choice for 2.18 compatibility.

Note that 24.05 shipped with Nix 2.18 as the default. So I think we do actually care about compatibility with Nix 2.18, and 2.24 would be too much; yeah, 24.05 is long EOL, but I don’t feel good about the idea that we could break even evaluating a new version to upgrade after you take a machine that hasn’t been touched in under a year out of storage. Unfortunately we have no way to validate that directly in CI.

Right. I thought we might do some other checks against the minimal version too. I think it would be good to have more thorough checks of eval against what we claim as our baseline – especially, checks that no derivation/output hashes change! – but I assume it would probably have to wait for merge queues.

I wonder whether we should do all our Eval with the version that is closest to the minimum, aka 2.3 now or 2.18 after this PR. And then only do the additional parse checks with latest versions.

In theory, we could also run one version on push events (the target branch) and one version on pull requests. Then, if we get "fake rebuilds", aka some where we didn't actually change the code, but still got different hashes, we'd know something is up.

In practice, this should only happen when introducing a new Nix/Lix version, which would only happen when updating the pinned nixpkgs for CI.

(theoretically, it could happen when somebody introduces code to trigger an edge-case, that we didn't hit before... but that's unlikely, I think - also these would appear as rebuilds in every PR, so should be quite visible if suddenly a comment-only-change causes rebuilds)

My understanding is that Lix do not intend to introduce any new language‐accessible functionality without explicit feature gating or versioning – at least out of “obviously unstable” edge cases? – but I can’t speak with confidence there; maybe that only applies to syntax rather than built‐ins? Hopefully someone from the Lix team can comment more authoritatively. But there are definitely things in newer versions of Nix that Lix does not implement and I think does not intend to, so I think it’s sensible as a “keep us honest” choice for 2.18 compatibility.

Lix cares about being hash equivalent to Nix 2.18 (i.e. drvs equivalence, outpath equivalence, etc.) & cares about language stability including with respect to laziness behaviors. All experimental features are out of scope, obviously.

Anything else is a bug.

Lix probably cannot be relied upon long term to never ever invent new language features that are on by default, but in general our position is "we're not doing that until there's a versioning mechanism and if CppNix forces our hand by not having a stability policy and adding features that are relied upon then we will take it case by case".

Currently one known feature Lix doesn't have is builtins.warn (but we will add it, and adding builtins is.. mostly considered acceptable assuming they have a fully defined contract). In general the attitude is "you don't add something that cannot be detected at runtime. if you don't like that then build a feature detection mechanism". See the Lix builtins.features issue on said feature detection mechanism.

The other thing that Lix does is ban bad features that are sufficiently unused (thanks @piegamesde for pushing on this evolution!). The most notable ones so far are:

• integer overflow (just gone. there's no replacement)
• url literals (opt-out hard error)
• legacy let {} syntax (opt-out hard error)

Those bans aren't a problem for nixpkgs because it just forces nixpkgs to stay honest with actually never adding them back.

I wonder whether we should do all our Eval with the version that is closest to the minimum, aka 2.3 now or 2.18 after this PR. And then only do the additional parse checks with latest versions.

I think it’s good to ensure all of (minimum versions, default versions, latest versions) work. That would be too much, though, so we have to pick and choose. I don’t have a strong opinion on what we should pick if we can only have one except that it seems best to be conservative about unnecessary changes in this PR since it’s already likely to be a matter of some controversy. My hope is that merge queues would allow us to cover at least both the minimum version and the default or latest, by punting all but one eval into the queue job; we could then also ensure hash stability between versions, which is important for Nixpkgs integrity and also I believe something that has regressed in the past (due to implementation bugs rather than Nixpkgs ones).

(Modulo available CI resources, checking eval against Lix seems like a sensible idea in general given a lot of contributors and deployments use it, even aside from how we encode the minimum version stuff.)

By the way you could probably write such an evaluation test that's slow as a nightly CI job and it wouldn't lose very much aside from not being able to kick out the bad code from the merge train, even without a merge queue.

I think it’s good to ensure all of (minimum versions, default versions, latest versions) work. That would be too much, though, so we have to pick and choose.

If we just compare minimum vs latest, then:

• If we check only latest in Nixpkgs, we risk using features that older versions don't understand.
• If we check only minimum in Nixpkgs, we can still rely on both Nix and Lix having a strong incentive to be able to evaluate older code.

I'd say the chance for breakage is smaller in the second case, although #428076 (comment) explicitly mentions removal of "integer overflow". So we'd need to make sure to at least test with a version that throw for this as well.

I don’t have a strong opinion on what we should pick if we can only have one except that it seems best to be conservative about unnecessary changes in this PR since it’s already likely to be a matter of some controversy.

Absolutely, I don't intend to make any changes that way in this PR. But it's important to understand which direction to take wrt to nixVersions.minimum.

My current gut feel is going towards: We should define both a minimum Nix and a minimum Lix version and test against both of these. But only these, not stable or latest - that would use too many resources.

Also, since Lix is removing support for a few things, I would run Lix in PRs, to catch somebody using the removed features early. Then run Nix on pushes and then compare the results. This should give us really good coverage already.

Since new versions are only introduced when updating the pinned nixpkgs, we could also add a matrix of eval jobs only when ci/pinned.json changes - and then test eval for all nix/lix versions at once, comparing hashes between them for consistency.

I guess the thing is, Lix is the one that can be relied upon for having the strictest semantics, but there's been historically many bugs found in CppNix immediately after a version becomes nixpkgs default, so it winds up being somewhat part of their QA which is a reason it probably needs to be tested also (I used to be involved in pkgs.nix upgrades for about two to three years).

The integer overflow example is actually a happy accident caused by Lix running hardening flags in prod, not noticing we removed integer overflow, not learning we crashed on it for months, then being like "damn we can just ban it with a good error". I also backported that one to CppNix so the nix language simply doesn't have integer overflow in either implementation period. The other language features you could largely just catch with a parse check (or a linter! the reason we banned them was because they were unacceptable in nixpkgs for ages).

--option eval-system, thus we'd have to run this eval on 4 different GitHub actions runners instead of ubuntu-24.04-arm. This is not going to work well, especially for darwin.

Why are we setting eval-system? Nixpkgs can be called purely and I'd say any place that uses builtins.currentSystem instead of the appropriate localSystem/crossSystem/*Platform is arguably a bug.

I have not compared full nixpkgs to Nix 2.3, yet.

I have tried to do this now: It's not easily possible, if at all.

Nix 2.3's nix-env / nix-instantiate does not support:

* `--option eval-system`, thus we'd have to run this eval on 4 different GitHub actions runners instead of `ubuntu-24.04-arm`. This is not going to work well, especially for darwin.

* `--out-path` + `--json` - that was only implemented in 2.6: [NixOS/nix@6525761](https://github.com/NixOS/nix/commit/65257614ea77437a75447c3b5936f240e581ede9) - we'd need to rewrite the script to not produce and consume JSON anymore.

* `--option restrict-eval true` + `-I <path-to-nixpkgs>` behaves differently and just throws an error the way we use it for the attrpaths step right now. We could remove this for an initial verification, but not keep it that way, I think.

It would require significant changes to our Eval workflow to even be able to run it with Nix 2.3. And then we'd still need to get it from somewhere, if it's not available from within the pinned nixpkgs we have.

This does not mean that Nixpkgs actually evals differently, far from it - but the tooling we have just doesn't allow checking it.

With the exceptions mentioned in #428076 (comment) Nixpkgs currently still evaluates the same with the different Lix and Nix versions available in pkgs. We can keep validating this with the ratchet-style workflow that I will open a PR for soon. (Edit: #427724)

TLDR for this PR:

* Nix 2.3 is not supposed to be used by users, thus the package itself can be removed.

* We can only validate Nixpkgs evaluating properly with Nix 2.3 with a lot of work for the various reasons outlined above and earlier in this PR.

* Raising minver to 2.18 should not be a practical problem for any upgrades: NixOS 23.11 was already on Nix 2.18 by default. If somebody was still running something older and would like to update, they can do that step by step.

* The theoretical concern whether we are actually testing minver or not is not worsened by this PR at all, in fact the opposite is true: The gap between what we currently test and what minver is at, is narrowing with this PR.

At this point we should just call it a day and merge this PR.

We can then still discuss in a follow up:

* Do we want to keep or replace the concept of "minver"?

* If we keep "minver": Do we need to Eval _with the exact minimum version_, at least sometimes?

Barring objections, I'd like to merge this PR within the next week. If you, the reader, would like to object, feel free to downvote this comment for quick feedback, but please eventually elaborate on:

* What exactly you're objecting to and

* What you're proposing to do instead and

* Whether you'd be ready to implement your proposal yourself or not.

Sounds good.

Why are we setting eval-system? Nixpkgs can be called purely and I'd say any place that uses builtins.currentSystem instead of the appropriate localSystem/crossSystem/*Platform is arguably a bug.

IIUC the dependency on builtins.currentSystem is in the top-level/release* code, not actual Nixpkgs. --eval-system was introduced here, for more context: #378922. It could still be a bug, sure, but from the perspective of CI it makes sense to make sure that we're evaluating the same thing as if we were actually on that other platform. The Eval check is not primarily a check for purity, but for "does it eval on that platform".

Also extracted pythonix commits into its own PR: #433036

Also extracted pythonix commits into its own PR: #433036

Thanks, that makes sense to split these. Rebased after the merge of that PR.

Since there were no objections to this PR anywhere, I will proceed with it now that #427724 is merged, too.

We can then still discuss in a follow up:

• Do we want to keep or replace the concept of "minver"?

• If we keep "minver": Do we need to Eval with the exact minimum version, at least sometimes?

The follow up with my proposal on how to evolve "minver" towards a feature-based approach is in #433057.

Also two more PRs with cleanups possible after bumping minver: #433055 and #433056.

+1 on this PR. Unfortunately this means people who where still reluctant to flakes, and that might be for good reasons get their stability guarantees on lib dropped.

One the other hand i understand that backporting all fixes to 2.3 is also not long term maintainable.

I like this because it finally frees up the use of 'nix-unit' for our eval tests. which hasnt been backported yet to 2.3

So this inherently resolves the biggest blocker for my planned lib test migration. 😄