Skip to content

__unsafeDiscardOutputDependency used with exportReferencesGraph does not work #7330

@Ericson2314

Description

@Ericson2314

Describe the bug

__unsafeDiscardOutputDependency used with exportReferencesGraph cause a build failure when the output paths of the drv File used with cannot be found in the sandbox because they are not yet built.

Steps To Reproduce

TODO make exact example; it can be used as a test.

Expected behavior

drv outputs are ignored, only the drv file itself matters.

Additional context

  1. __unsafeDiscardOutputDependency is not unsafe at all, it is just used to make a derivation depend on a drv file itself and not its outputs. (Just depending on the drv itself should probably be the default, but isn't for historical reasons.) How does it work? It removes any = the string context starts with. Implemented in

    static void prim_unsafeDiscardOutputDependency(EvalState & state, const PosIdx pos, Value * * args, Value & v)
    {
    PathSet context;
    auto s = state.coerceToString(pos, *args[0], context);
    PathSet context2;
    for (auto & p : context)
    context2.insert(p.at(0) == '=' ? std::string(p, 1) : p);
    v.mkString(*s, context2);
    }

  2. In

    nix/src/libexpr/primops.cc

    Lines 1202 to 1211 in 62960f3

    if (path.at(0) == '=') {
    /* !!! This doesn't work if readOnlyMode is set. */
    StorePathSet refs;
    state.store->computeFSClosure(state.store->parseStorePath(std::string_view(path).substr(1)), refs);
    for (auto & j : refs) {
    drv.inputSrcs.insert(j);
    if (j.isDerivation())
    drv.inputDrvs[j] = state.store->readDerivation(j).outputNames();
    }
    }
    the = is handled. It causes the newly-built derivation to have far more dependencies -- in addition to the drv file itself, all its drv file dependencies and all their outputs are depended upon.

  3. exportReferencesGraph is implemented in large part by Store::exportReferences in

    for (auto & j : paths2) {
    if (j.isDerivation()) {
    Derivation drv = derivationFromPath(j);
    for (auto & k : drv.outputsAndOptPaths(*this)) {
    if (!k.second.second)
    /* FIXME: I am confused why we are calling
    `computeFSClosure` on the output path, rather than
    derivation itself. That doesn't seem right to me, so I
    won't try to implemented this for CA derivations. */
    throw UnimplementedError("exportReferences on CA derivations is not yet implemented");
    computeFSClosure(*k.second.second, paths);
    }
    }
    }
    . If a derivation is encountered, it also adds its outputs to the closure.

It seems that the export references code is wrong:

  1. It is assuming all drv paths got the = treatmeant but this is not necessarily the case. For ones that didn't (using __unsafeDiscoardOutputDependency), we cannot start processing their outputs because those are not necessarily dependencies of the derivation.

  2. The code is unnecessary, because the treatment of = in the evaluator will already ensure all those paths are in inputPaths/inputDrvs, and

    /* TODO (impure derivations-induced tech debt):
    Tracking input derivation outputs statefully through the
    goals is error prone and has led to bugs.
    For a robust nix, we need to move towards the `else` branch,
    which does not rely on goal state to match up with the
    reality of the store, which is our real source of truth.
    However, the impure derivations feature still relies on this
    fragile way of doing things, because its builds do not have
    a representation in the store, which is a usability problem
    in itself */
    if (auto outPath = get(inputDrvOutputs, { depDrvPath, j })) {
    worker.store.computeFSClosure(*outPath, inputPaths);
    }
    else {
    auto outMap = worker.evalStore.queryDerivationOutputMap(depDrvPath);
    auto outMapPath = outMap.find(j);
    if (outMapPath == outMap.end()) {
    throw Error(
    "derivation '%s' requires non-existent output '%s' from input derivation '%s'",
    worker.store.printStorePath(drvPath), j, worker.store.printStorePath(depDrvPath));
    }
    worker.store.computeFSClosure(outMapPath->second, inputPaths);
    }
    }
    }
    }
    /* Second, the input sources. */
    worker.store.computeFSClosure(drv->inputSrcs, inputPaths);
    ensures that the closure of those is already in inputPaths

As such we should be able to do this:

  1. Write test confirming the current failure in tests/export-graph.nix. See there is one for a drvPath without __unsafeDiscardOutputDependency aready.

  2. Delete the code in Store::exportReferences that seems pointless, and confirm the bug is fixed.

See 2897286 for what I think introduced the bug.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions