Skip to content

Should we allow spitting the derivation output map per output? #4344

@Ericson2314

Description

@Ericson2314

The SQL database necessary stores rows individually, but we have a choice for other types of stores. #4330 currently stores a file per derivation × output in the binary cache, but we might also consider storing one file for the entire derivation and all its outputs.

The big idea in doing so is that:

  • Ultimately, after all the trusting and sharing of mappings is "unwound" a mapping comes from a build
  • All outputs must be built at once, so the "original" mappings should always be available for all outputs if they exist for any outputs at all
  • Mixing outputs from different non-deterministic builds might cause issues.

Now, this does clash a bit with the important ability we have to GC individual outputs, and exchange them individually as needed between stores. We certainly don't want to loose that ability, nor have to mutate binary cache to add or remove entries from an existing file.

The crux of the solution is to ensure that a mapping for any output must come with, and keep alive, the mappings for all outputs. The slight challenge is to implement this with the local store and SQLite DB. Perhaps we'll need a single-column DrvHash table, with foreign keys from the build mappings to that table, and a restriction that one cannot delete an individual mapping unless one deletes all the mappings for the derivation?

Metadata

Metadata

Assignees

No one assigned

    Labels

    ca-derivationsDerivations with content addressed outputs

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions