verify-backends inherits ACQ_EXTRA_KITS from the caller, so a run is not hermetic (and the ADR-0031 picker path is unreachable) #544

Summary

scripts/verify-backends goes to real trouble to be hermetic — it redirects the
acq config, secret store, state, and provenance directories into a throwaway
$VERIFY_STATE so a run "should not read the user's default backend, acq-owned
secret store, provenance cache, or port/ssh state" (its own comment at
scripts/verify-backends:194-195).

But it does not neutralize ACQ_EXTRA_KITS. It appends to whatever the
caller already exported:

export ACQ_EXTRA_KITS="${ACQ_EXTRA_KITS:+${ACQ_EXTRA_KITS} }${VOLKIT_DIR}"

So a developer with ACQ_EXTRA_KITS set in their shell gets those kits applied
to every throwaway sandbox the verifier creates, and the run is no longer
measuring the pinned built-in bundle alone.

Observed

A live ./scripts/verify-backends --only sbx -x run applied an unreleased
paseo kit alongside the four built-ins:

acq[debug]: fetch: https://github.com/GSA-TTS/agentic-coding-patterns.git
  ref=5dc3a6e6466902c038f4fba225d623d245dda38a
  dir=integrations/isolation/acq-kits/paseo

5dc3a6e ("test(paseo): fix project listing for 0.9 verify") is not an
ancestor of v1.11.0 — it is not in any patterns release. The other five kit
fetches in the same run correctly used the pinned ref. The sandbox also ended up
with an extra published port (Published paseo: localhost:49152 -> 6767/tcp)
and an extra startup command, neither of which the verifier asserts on.

Why this matters beyond extra kits

Setting ACQ_EXTRA_KITS has a second, less obvious effect. acq.backends/common.sh
captures provenance at load time:

if [ -n "${ACQ_EXTRA_KITS:-}" ]; then
  ACQ_EXTRA_KITS_FROM_ENV=1
else
  ACQ_EXTRA_KITS_FROM_ENV=""
fi

and create_time_kit_picker returns early when that flag is set
([ -z "${ACQ_EXTRA_KITS_FROM_ENV:-}" ] || return 0), because env is
authoritative over the interactive picker per ADR-0031.

Since the verifier always ends up with a non-empty ACQ_EXTRA_KITS (it
appends its own volumes fixture), ACQ_EXTRA_KITS_FROM_ENV is always 1
during a verify run. The create-time kit picker can therefore never be exercised
by verify-backends — the live gate cannot cover one of the two code paths
ADR-0031 introduced.

Impact

• A verify run is not reproducible across developers: the result depends on the
  caller's shell environment.
• "verify-backends is green" does not mean "the pinned bundle is green" — it may
  mean "the pinned bundle plus whatever I had exported is green".
• Conversely, a failure caused by a stray personal kit looks like a defect in
  the PR under test.
• The ADR-0031 create-time picker path is structurally unreachable from the live
  verifier.

This matters more than usual because verify-backends is the designated live
gate for the ADR-0030 migration sequencing (see
docs/BACKEND_GUIDE.md, "ADR-0030 migration sequencing"), which instructs
reviewers to run it before/during/after the migration.

Suggested direction

Not prescribing an implementation, but the shape of the fix:

1. Start from a clean slate rather than appending. Set
   ACQ_EXTRA_KITS="$VOLKIT_DIR" (dropping any inherited value) so the run
   always exercises exactly the pinned built-ins plus the verifier's own
   fixture.
2. Offer an explicit opt-in for the case where someone genuinely wants to
   verify with extra kits — e.g. a --extra-kit REF flag or a
   VERIFY_EXTRA_KITS variable that the script reads deliberately, instead of
   silently inheriting ACQ_EXTRA_KITS.
3. Audit the other acq-reading env vars the verifier does not clear. At
   minimum ACQ_IMAGE, ACQ_WORKSPACE, ACQ_BACKEND, and ACQ_CLI_KITS are
   consumed by acq.backends/common.sh and could likewise change what a run
   measures. ACQ_IMAGE in particular would silently swap the base image out
   from under every check.
4. Decide how the create-time picker gets live coverage, given that the
   volumes fixture makes ACQ_EXTRA_KITS_FROM_ENV permanently truthy. Options
   include delivering the fixture through a different channel (e.g. --kit),
   or adding a dedicated sub-check that unsets the env for one create.

Item 4 is the part that needs a design call, not just a patch.

Related

• scripts/verify-backends:194-242 — the hermetic-state block and the
  ACQ_EXTRA_KITS append
• acq.backends/common.sh — ACQ_EXTRA_KITS_FROM_ENV capture and
  create_time_kit_picker's early return
• ADR-0031 (interactive acq configure) — env > config > interactive precedence
• ADR-0023 (neutral volumes: vocabulary) — why the fixture kit exists
• docs/BACKEND_GUIDE.md "ADR-0030 migration sequencing" — why this script is a
  release gate

Found while running the live sbx verification for
#474. Two separate blockers in the same script
were fixed on that PR (dfc8ff9 first-run prompt hang, d27d84f scope-blind
sbx usai seed guard); this one is a design question rather than a clear bug, so
it is filed separately against main instead.

AI-assisted (OpenCode).