Skip to content

Add a way to get all the outputs of a derivation with their label#3678

Merged
edolstra merged 1 commit intoNixOS:masterfrom
tweag:remote-query-outputs
Jul 1, 2020
Merged

Add a way to get all the outputs of a derivation with their label#3678
edolstra merged 1 commit intoNixOS:masterfrom
tweag:remote-query-outputs

Conversation

@thufschmitt
Copy link
Member

Generalize queryDerivationOutputNames and queryDerivationOutputs by adding a queryDerivationOutputMap that returns the map outputName=>outputPath.

This is needed by #3528 as it needs a way to get the path of a specific derivation output, which (afaik) isn't possible with the current API

(not that this is not equivalent to merging the results of queryDerivationOutputs and queryDerivationOutputNames as sets don't preserve the order, so we would end up with an incorrect mapping).

return outputPaths;
}

StringSet Store::queryDerivationOutputNames(const StorePath & path)
Copy link
Member

@Ericson2314 Ericson2314 Jun 11, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe add a comment saying these are obsolete / deprecated ?


typedef std::set<StorePath> StorePathSet;
typedef std::vector<StorePath> StorePaths;
typedef std::map<string, StorePath> StorePathMap;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I might call this OutputPathMap instead, as StorePathMap sounds to me like std::map<StorePath, ???>.

Copy link
Member Author

@thufschmitt thufschmitt Jun 17, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Indeed, that makes sense. Changed in 53edff2

case wopQueryReferences:
case wopQueryReferrers:
case wopQueryValidDerivers:
case wopQueryDerivationOutputs: {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you know what the policy is on supporting on protocol versions? I don't!

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The daemon should support older clients. So we can't remove APIs like this.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Woops sorry, I'll restore them

@edolstra
Copy link
Member

Do we actually need these functions? The outputs of a derivation are also available by doing readDerivation(drvPath). IIRC, the only reason why we have LocalStore::queryDerivationOutputs() was to make the garbage collector more efficient (since getting this stuff from SQLite is faster than reading a lot of .drvs). queryDerivationOutputNames() doesn't look like it was ever necessary from a performance perspective...

@edolstra
Copy link
Member

As a cleanup I've removed queryDerivationOutputNames() (045b072).

@Ericson2314
Copy link
Member

I agree having 3 of them is unnecessary, but this new on seams good to me? In #3523 (comment) you talk about not wanting to rely on derivations existing on disk as much. Avoiding readDerivation with these functions would seem to me in service of that goal.

@thufschmitt
Copy link
Member Author

Do we actually need these functions? The outputs of a derivation are also available by doing readDerivation(drvPath). IIRC, the only reason why we have LocalStore::queryDerivationOutputs() was to make the garbage collector more efficient (since getting this stuff from SQLite is faster than reading a lot of .drvs). queryDerivationOutputNames() doesn't look like it was ever necessary from a performance perspective...

Atm yes, but that won't be true anymore in a CA setting where the output paths don't match what's in the drv (which is the reason why I needed this in the first place)

@thufschmitt thufschmitt force-pushed the remote-query-outputs branch from 4913e9e to 53edff2 Compare June 17, 2020 07:58
@thufschmitt thufschmitt force-pushed the remote-query-outputs branch from 53edff2 to e23910e Compare June 24, 2020 08:56
@thufschmitt thufschmitt mentioned this pull request Jun 24, 2020
@thufschmitt
Copy link
Member Author

I've rebased it on top of master. No idea what the OSX failure is. It seems totally unrelated

@Ericson2314
Copy link
Member

Yes, we currently fetch the latest nixos-20.03-small, but I think that might be a bit too fast non-deterministism, as it opens the door to all these issues on Darwin deps.

Generalize `queryDerivationOutputNames` and `queryDerivationOutputs` by
adding a `queryDerivationOutputMap` that returns the map
`outputName=>outputPath`

(not that this is not equivalent to merging the results of
`queryDerivationOutputs` and `queryDerivationOutputNames` as sets don't
preserve the order, so we would end up with an incorrect mapping).

squash! Add a way to get all the outputs of a derivation with their label

Rename StorePathMap to OutputPathMap
@thufschmitt thufschmitt force-pushed the remote-query-outputs branch from e23910e to d38f860 Compare June 24, 2020 18:38
@edolstra edolstra merged commit 86a4aba into NixOS:master Jul 1, 2020
@edolstra edolstra deleted the remote-query-outputs branch July 1, 2020 13:32
@edolstra
Copy link
Member

edolstra commented Jul 1, 2020

Thanks, merged!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants