Properly store the outputs of CA derivations#4315
Closed
thufschmitt wants to merge 2 commits intoNixOS:masterfrom
Closed
Properly store the outputs of CA derivations#4315thufschmitt wants to merge 2 commits intoNixOS:masterfrom
thufschmitt wants to merge 2 commits intoNixOS:masterfrom
Conversation
thufschmitt
force-pushed
the
ca/properly-store-outputs
branch
from
39bbea9 to
3cb7378
Compare
Member
Author
|
I've reworked this a little to
|
thufschmitt
force-pushed
the
ca/properly-store-outputs
branch
4 times, most recently
from
3051b84 to
dfeccb1
Compare
For each known realisation, store: - its output - its output path This comes with a set of needed changes: - New `realisations` module declaring the types needed for describing these mappings - New `Store::registerDrvOutput` method registering all the needed informations about a derivation output (also replaces `LocalStore::linkDeriverToPath`) - new `Store::queryDrvOutputInfo` method to retrieve the informations for a derivations This introcudes some redundancy on the remote-store side between `wopQueryDerivationOutputMap` and `wopQueryDrvOutputInfo`. However we might need to keep both (regardless of backwards compat) because we sometimes need to get some infos for all the outputs of a derivation (where `wopQueryDerivationOutputMap` is handy), but all the stores can't implement it − because listing all the outputs of a derivation isn't really possible for binary caches where the server doesn't allow to list a directory.
Add a new table for tracking the derivation output mappings. We used to hijack the `DerivationOutputs` table for that, but (despite its name), it isn't a really good fit: - Its entries depend on the drv being a valid path, making it play badly with garbage collection and preventing us to copy a drv output without copying the whole drv closure too; - It dosen't guaranty that the output path exists; By using a different table, we can experiment with a different schema better suited for tracking the output mappings of CA derivations. (incidentally, this also fixes NixOS#4138)
thufschmitt
force-pushed
the
ca/properly-store-outputs
branch
from
dfeccb1 to
a898bfa
Compare
Member
Author
|
Rebased on top of latest master to fix a conflict |
Member
Author
|
Closing in favor of #4330 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
For each “known” derivation output, store:
(morally what we want is two families of mappings: One mapping
unresolved derivations to their resolved version and one mapping
resolved derivations to their output path, but it's both simpler and
more efficient to combine them)
This comes with a set of needed changes:
drv-output-infomodule declaring the types needed for describingthese mappings
Store::registerDrvOutputmethod registering all the needed informationsabout a derivation output (also replaces
LocalStore::linkDeriverToPath)Store::queryDrvOutputInfomethod to retrieve the informations for aderivations
This introcudes some redundancy on the remote-store side between
wopQueryDerivationOutputMapandwopQueryDrvOutputInfo.However we might need to keep both (regardless of backwards compat)
because we sometimes need to get some infos for all the outputs of a
derivation (where
wopQueryDerivationOutputMapis handy), but all thestores can't implement it − because listing all the outputs of a
derivation isn't really possible for binary caches where the server
doesn't allow to list a directory.
For the local store, this uses a different db tables that will only be added to the schema if the
ca-derivationsexperimental flag is on, meaning that people not using it won't be affected by the schema change