Skip to content

Hash modulo meta#14686

Open
roberth wants to merge 7 commits intoNixOS:masterfrom
roberth:hash-modulo-meta
Open

Hash modulo meta#14686
roberth wants to merge 7 commits intoNixOS:masterfrom
roberth:hash-modulo-meta

Conversation

@roberth
Copy link
Member

@roberth roberth commented Dec 1, 2025 •
edited
Loading

Motivation

meta is lost at the derivation level, but it doesn't need to be that way, thanks to quotient hashing aka hash modulo.

This change makes use of the existing "discrepancy" between derivation hashes and output hashes to allow more information to be stored in derivations without causing rebuilds.

The core of the change is fairly simple; see the hashDerivationModulo hunk.

Changes

  • Add system feature derivation-meta improving hash quotient logic to support storing meta
  • Adjust JSON formats to make meta a first class top level attr instead of the ATerm embedding

Notes

  • Must be opted in to by derivations/expressions, as usual, like ca derivations, structuredAttrs
  • Not supported in non-structuredAttrs. Didn't seem worth the added complexity. Users should move to structuredAttrs instead and afaik Nixpkgs is on track to do so.

Example

$ nix run github:roberth/nix/hash-modulo-meta -- \
  build --store ~/stores/tmp \
    --extra-experimental-features derivation-meta \
    --impure --expr \
    'with import (builtins.fetchTarball { url = "https://github.com/roberth/nixpkgs/archive/derivation-meta.tar.gz"; }) { config.derivationMeta = true; }; python3'
...
<progress bar, substituting from cache.nixos.org>
...
$ nix run github:roberth/nix/hash-modulo-meta -- derivation show --store ~/stores/tmp --extra-experimental-features derivation-meta --impure --expr 'with import (builtins.fetchTarball { url = "https://github.com/roberth/nixpkgs/archive/derivation-meta.tar.gz"; }) { config.derivationMeta = true; }; python3'
{
  "vcllzyc6d87wxlncjcwxlgy82lna4h4v-python3-3.13.9.drv": {
    "args": ...
    ...
    "meta": {
      "available": true,
      "broken": false,
      "changelog": "https://docs.python.org/release/3.13.9/whatsnew/changelog.html",
...
  }
}

Context

Add 👍 to pull requests you find important.

The Nix maintainer team uses a GitHub project board to schedule and track reviews.

@github-actions github-actions bot added documentation with-tests Issues related to testing. PRs with tests have some priority store Issues and pull requests concerning the Nix store labels Dec 1, 2025
@roberth roberth mentioned this pull request Dec 1, 2025
Draft
13 tasks
@roberth roberth force-pushed the hash-modulo-meta branch 3 times, most recently from 846a9c9 to b2c772b Compare December 2, 2025 06:02
@edolstra
Copy link
Member

edolstra commented Dec 3, 2025

I didn't see this mentioned explicitly, but I assume __meta is not passed to the builder?

It's not entirely obvious how derivation meta data would be used in practice, especially since derivers are kind of useless (e.g. cache.nixos.org paths do have a deriver, but you have no way to get to them, so you wouldn't be able to get to the metadata). Perhaps in conjunction with #11749, the metadata could be propagated into the provenance records of store paths (which are already JSON).

@roberth
Copy link
Member Author

roberth commented Dec 3, 2025

I didn't see this mentioned explicitly, but I assume __meta is not passed to the builder?

Correct, and this is now also covered by a functional test.

not entirely obvious how derivation meta data would be used in practice

You can instantiate your stuff and inspect the metadata.
If you don't have the expression sources for what you've deployed, you probably should(!), but if you don't provenance could connect that back, either by eval reproduction, by reference or by value.

  • eval reproduction: provenance has a pinned flakeref, and you can instantiate that to acquire derivations
  • by reference: maybe you actually do have derivations in a store
  • by value: meta could be duplicated into the provenance structures, and it will benefit from the infrastructure that's set up around this:
    • having a reliable interface between Nix and the expression world (ie derivation { __meta })
      (meta attribute is not reliable because not the whole derivation closure occurs as packages that are seen by the CLI)
    • that interface getting used, pkgs: Add config.derivationMeta nixpkgs#466932

So this is a good step towards making #11749 more effective.

@roberth
Add derivation-meta feature for no-rebuild meta stored in derivation
d8923d5
@roberth
Translate between __meta and meta in JSON derivation
bb3224e
@roberth
Only activate derivation-meta when __meta is an object
e2eaf9a
@roberth
Update derivation-options-v1 schema for derivation-meta
2c01ebc
@roberth
Ensure backward compatibility for meta field in DerivationOptions JSON
063906f 
Make the meta field optional in DerivationOptions JSON deserialization to maintain backward compatibility with JSON that predates the derivation-meta feature.

Changes:
- Use optionalValueAt instead of valueAt for meta field deserialization
- Mark meta as optional in derivation-options-v1 schema
- Add unit test for backward compatibility
@roberth
Extract __meta to top-level meta field in Derivation JSON format
cbebe0b 
When the derivation-meta experimental feature is enabled, extract the
__meta field from structuredAttrs to a top-level meta field in the
Derivation JSON format (used by `nix derivation show/add`). This makes
the format consistent with DerivationOptions JSON and provides better
visibility of metadata without it being buried in structuredAttrs.

The implementation:
- Creates derivationToJson() helper accepting ExperimentalFeatureSettings
- Extracts __meta to top-level meta field when feature is enabled
- Requires experimental feature when deserializing JSON with meta field
- Reconstructs __meta in structuredAttrs during deserialization
- Maintains backward compatibility with JSON lacking meta field

Tests verify both serialization and deserialization with mocked
experimental feature settings, and functional tests ensure the
experimental feature requirement is enforced.
@roberth
Add functional test that __meta doesn't make it into the sandbox
1806b86
@roberth roberth mentioned this pull request Dec 25, 2025
Open
@roberth roberth mentioned this pull request Feb 6, 2026
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Ericson2314 Ericson2314 Awaiting requested review from Ericson2314 Ericson2314 is a code owner

@edolstra edolstra Awaiting requested review from edolstra edolstra is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

documentation store Issues and pull requests concerning the Nix store with-tests Issues related to testing. PRs with tests have some priority

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Have our cake and eat it too derivation metadata

2 participants

@roberth @edolstra