-
-
Notifications
You must be signed in to change notification settings - Fork 1.8k
Description
Describe the bug
__unsafeDiscardOutputDependency used with exportReferencesGraph cause a build failure when the output paths of the drv File used with cannot be found in the sandbox because they are not yet built.
Steps To Reproduce
TODO make exact example; it can be used as a test.
Expected behavior
drv outputs are ignored, only the drv file itself matters.
Additional context
-
__unsafeDiscardOutputDependencyis not unsafe at all, it is just used to make a derivation depend on a drv file itself and not its outputs. (Just depending on the drv itself should probably be the default, but isn't for historical reasons.) How does it work? It removes any=the string context starts with. Implemented innix/src/libexpr/primops/context.cc
Lines 34 to 44 in 62960f3
static void prim_unsafeDiscardOutputDependency(EvalState & state, const PosIdx pos, Value * * args, Value & v) { PathSet context; auto s = state.coerceToString(pos, *args[0], context); PathSet context2; for (auto & p : context) context2.insert(p.at(0) == '=' ? std::string(p, 1) : p); v.mkString(*s, context2); } -
In
theLines 1202 to 1211 in 62960f3
if (path.at(0) == '=') { /* !!! This doesn't work if readOnlyMode is set. */ StorePathSet refs; state.store->computeFSClosure(state.store->parseStorePath(std::string_view(path).substr(1)), refs); for (auto & j : refs) { drv.inputSrcs.insert(j); if (j.isDerivation()) drv.inputDrvs[j] = state.store->readDerivation(j).outputNames(); } } =is handled. It causes the newly-built derivation to have far more dependencies -- in addition to the drv file itself, all its drv file dependencies and all their outputs are depended upon. -
exportReferencesGraphis implemented in large part byStore::exportReferencesin. If a derivation is encountered, it also adds its outputs to the closure.Lines 823 to 836 in 62960f3
for (auto & j : paths2) { if (j.isDerivation()) { Derivation drv = derivationFromPath(j); for (auto & k : drv.outputsAndOptPaths(*this)) { if (!k.second.second) /* FIXME: I am confused why we are calling `computeFSClosure` on the output path, rather than derivation itself. That doesn't seem right to me, so I won't try to implemented this for CA derivations. */ throw UnimplementedError("exportReferences on CA derivations is not yet implemented"); computeFSClosure(*k.second.second, paths); } } }
It seems that the export references code is wrong:
-
It is assuming all drv paths got the
=treatmeant but this is not necessarily the case. For ones that didn't (using__unsafeDiscoardOutputDependency), we cannot start processing their outputs because those are not necessarily dependencies of the derivation. -
The code is unnecessary, because the treatment of
=in the evaluator will already ensure all those paths are ininputPaths/inputDrvs, andensures that the closure of those is already innix/src/libstore/build/derivation-goal.cc
Lines 532 to 560 in 62960f3
/* TODO (impure derivations-induced tech debt): Tracking input derivation outputs statefully through the goals is error prone and has led to bugs. For a robust nix, we need to move towards the `else` branch, which does not rely on goal state to match up with the reality of the store, which is our real source of truth. However, the impure derivations feature still relies on this fragile way of doing things, because its builds do not have a representation in the store, which is a usability problem in itself */ if (auto outPath = get(inputDrvOutputs, { depDrvPath, j })) { worker.store.computeFSClosure(*outPath, inputPaths); } else { auto outMap = worker.evalStore.queryDerivationOutputMap(depDrvPath); auto outMapPath = outMap.find(j); if (outMapPath == outMap.end()) { throw Error( "derivation '%s' requires non-existent output '%s' from input derivation '%s'", worker.store.printStorePath(drvPath), j, worker.store.printStorePath(depDrvPath)); } worker.store.computeFSClosure(outMapPath->second, inputPaths); } } } } /* Second, the input sources. */ worker.store.computeFSClosure(drv->inputSrcs, inputPaths); inputPaths
As such we should be able to do this:
-
Write test confirming the current failure in
tests/export-graph.nix. See there is one for adrvPathwithout__unsafeDiscardOutputDependencyaready. -
Delete the code in
Store::exportReferencesthat seems pointless, and confirm the bug is fixed.
See 2897286 for what I think introduced the bug.