fix(vault): unify init() key resolution with resolve_master_key()#5074
Conversation
) The MCP OAuth auth_start handler stores PKCE state across three sequential vault_set calls (pkce_verifier → pkce_state → redirect_uri). The first call lazy-creates the vault via init() + set(); every subsequent call walks unlock() + set() against a fresh CredentialVault instance constructed inside KernelOAuthProvider::vault_set. Pre-fix the second call's unlock() failed with aead::Error because init() duplicated the env / keyring lookup code from resolve_master_key(). The two sites could resolve different master keys on container hosts where DBus is partially available or env mutation races with the vault unlock, leaving a file the same process could not decrypt one call later. Two changes: 1. init() now resolves the master key through resolve_master_key() directly. The random-key fallback only kicks in when resolve_master_key() returns VaultLocked (no env, no keyring), and that branch stores the generated key into the keyring so subsequent resolves on fresh instances find the same value. By construction the two code paths can no longer diverge. 2. init() now performs a post-write verification: it constructs a sibling CredentialVault::new instance and calls unlock() on the freshly-written file. If unlock fails, init() rolls back the file and returns an actionable Vault error pointing at the divergence rather than letting the next vault_set caller fail downstream with an opaque aead::Error. This also catches an AAD path-binding regression (issue hypothesis #2) at the source. Regression coverage: - vault::tests::init_then_set_then_reopen_unlock_via_env_key pins the init() + set() -> fresh-instance unlock() round trip via the env-resolved master key, mirroring the failing auth_start sequence. - mcp_oauth_provider::tests::vault_set_twice_round_trips_via_env_key exercises the same flow through KernelOAuthProvider::vault_set -- the exact API the auth_start handler calls -- and reads every entry back through a fresh provider instance to confirm no entry is lost across the re-open boundary. Verification: - cargo check --workspace --lib - cargo clippy -p librefang-extensions -p librefang-kernel --lib --tests -- -D warnings - cargo test -p librefang-extensions (87 passed) - cargo test -p librefang-kernel --lib vault (10 passed, including cross-test interaction with vault_cache_reuses_unlocked_handle_across_calls) - cargo test -p librefang-api --test mcp_auth_routes_extra_test (8 passed)
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
houko
left a comment
There was a problem hiding this comment.
Independent review — APPROVE-WITH-NITS
Root cause assessment
The hypothesis ("init() duplicated key resolution → divergence") is plausible but unproven. With LIBREFANG_VAULT_KEY set and stable, the OLD init() and resolve_master_key() are byte-identical paths — they can only diverge if the env var is mutated between init's read and the next unlock's read. This PR does not identify any concrete mutator. The issue's hypothesis #2 (AAD path canonicalization) and #3 (salt-flush timing) are NOT cured by routing through resolve_master_key() alone. The post-write verification is the actual load-bearing change.
Fix robustness
Unification is safe and cannot make things worse. The post-save verification is the real fix: it constructs a fresh CredentialVault::new and runs the exact path the next vault_set walks, so any drift surfaces synchronously with a rollback. Residual TOCTOU: if LIBREFANG_VAULT_KEY is mutated between save and verify-unlock, init rolls back even though the daemon's subsequent vault_set (post-mutation value) might also fail — strictly better than the pre-fix opaque aead::Error.
WARNING
- Tests do not pin the regression. Mentally simulating
init_then_set_then_reopen_unlock_via_env_keyagainstorigin/main: env set beforeinit()runs, never mutated,VAULT_NO_KEYRING=1. Oldinit()reads env →decode_master_key→ identical bytes to what nextunlock()'sresolve_master_keyreads. Old code passes both new tests. They exercise the contract but don't reproduce the failure mode the issue describes. PR text "two new regression tests pin the exact failing sequence" overstates. - PR body acknowledges manual container repro is "still recommended". Combined with the above, the test suite cannot prove the bug is fixed — only that the happy path doesn't regress. Human verification in the original Lazycat/Docker environment remains a true gate.
NIT
init()doc-comment guess about "trailing whitespace tolerance / Zeroizing-vs-String temporary" is speculative — neither path strips whitespace, neither is optimizer-sensitive at env-read. Tone down to "duplicated paths could diverge under env mutation".let _ = std::fs::remove_file(...)silences rollback errors — anEACCESleaves a broken file the nextinit()rejects with "Vault already exists". Debug-log on failed unlink would help operators.drop(verify);is redundant — block-scope drop runs unconditionally one line later.
Follow-ups (in-scope per "fix what you found"):
- Align new
vault_set_twice_round_trips_via_env_keyto#[serial_test::serial(librefang_vault_key)]to match othermcp_oauth_providertests. The current state — two disjoint serial groups in the same crate touching the same process-global env — is exactly the race the commit message warns about. Agent flagged + didn't fix — that's a punt. - Add a regression test that mutates
LIBREFANG_VAULT_KEYbetweeninit()and the verify-unlock — that's the actual failure mode the issue describes, and it would prove the post-save verification catches it.
Addresses code review on #5074. - new vault::tests::init_env_mutation_during_save_rolls_back_and_surfaces_typed_error test: a worker thread mutates LIBREFANG_VAULT_KEY between init's resolve_master_key() and init's post-write verify-unlock, proving the verify-unlock catches the divergence, rolls back the corrupt vault.enc, and surfaces a typed ExtensionError::Vault(_) naming the divergence (not a bare aead::Error). Verified failing on origin/main (pre-fix init() returns Ok silently and the test's expect_err panics) and passing on this branch. - align serial_test groups: vault_set_twice_round_trips_via_env_key (mcp_oauth_provider), vault_cache_reuses_unlocked_handle_across_calls and install_integration_writes_through_cached_vault_handle (kernel::tests) all move to #[serial_test::serial(librefang_vault_key)] so every test in librefang-kernel that mutates the process-global LIBREFANG_VAULT_KEY sits in one named group. Two disjoint serial groups in the same crate could otherwise race the env var. - tone down init() doc-comment speculation about whitespace tolerance and Zeroizing-vs-String temporary lifetimes — neither path strips whitespace and neither is optimizer-sensitive at env-read. Replace with the concrete failure mode: env mutation between init's save and the next unlock's read. - log on rollback unlink failure with tracing::warn (path, error). EACCES would otherwise silently leave a corrupt vault.enc that the next init() rejects with "Vault already exists"; operators need visibility into the residue. - drop redundant drop(verify) — the block-scope drop runs anyway.
Adapted from upstream PR librefang#5074 — minimal port without the startup- sentinel changes that custom doesn't carry yet. Two divergences caused the aead::Error from MCP OAuth vault_set on the container deploy: 1. resolve_master_key() ordered "keyring → env → error", while init() ordered "env → keyring → random". When a host had a partially- available DBus keyring with a stale entry, init() encrypted vault.enc with the explicit env key, but the next unlock() walked resolve_master_key() and got the stale keyring value instead — surfacing as aead::Error. 2. init() duplicated the env/keyring lookup code from resolve_master_key(). Even with matching backends the two sites could diverge under env mutation between init's save and the next unlock's read. Two changes: - resolve_master_key() now prefers env over keyring. Explicit operator override (LIBREFANG_VAULT_KEY baked into lzc-manifest for container deploy) wins deterministically; keyring becomes the unattended-host fallback. - init() now calls resolve_master_key() directly. The random-key fallback only kicks in when resolve_master_key() returns VaultLocked (no env, no keyring), and that branch persists the generated key into the keyring so subsequent resolves on fresh instances find the same value. By construction the two code paths can no longer diverge. - init() now performs a post-write verification: constructs a sibling CredentialVault::new instance and calls unlock() on the freshly- written file. If unlock fails, init() rolls back the file and returns an actionable Vault error pointing at the divergence rather than letting the next vault_set caller fail downstream with an opaque aead::Error. Verification: - cargo check --workspace --lib (clean) - cargo test -p librefang-extensions --lib (60 passed) - cargo clippy -p librefang-extensions --lib --tests -- -D warnings (clean)
Closes #5069
Root cause
KernelOAuthProvider::vault_setconstructs a freshCredentialVaultfor every call. The MCP OAuth
auth_starthandler calls it three timesin a row to stash PKCE state (
pkce_verifier→pkce_state→redirect_uri). The first call walksinit() + set()(file absent);every subsequent call walks
unlock() + set()against the freshly-written file.
Pre-fix
init()duplicated the env / keyring lookup code fromresolve_master_key(). The two code paths were not byte-identicalunder every container condition — DBus partially available, env
mutation races from other code paths (e.g. the boot-time vault
unlock, the cached
vault_handle()inkernel/accessors.rs, thedotenv loader). When they diverged,
init()wrote a file with keyK_init while the next
unlock()resolved K_unlock, and AES-GCMauthentication failed with the opaque
aead::Errorthe operatorreport cites. The CLI never hit this because each
librefang vault …subcommand is a fresh process and only ever touches the vault once.
Fix
Minimal, two-part change in
crates/librefang-extensions/src/vault.rs:Unify the key-resolution code path.
init()now resolves themaster key by calling
self.resolve_master_key(). The "generate arandom key" branch only kicks in when resolve returns
VaultLocked(no env, no keyring) and that branch still stores thegenerated key into the keyring so the next
resolve_master_key()on a fresh instance finds the same value. By construction the two
sites can no longer diverge.
Post-write verification. After
save()returns,init()constructs a sibling
CredentialVault::newinstance and callsunlock()on the just-written file — exactly the code path thenext
KernelOAuthProvider::vault_setwill walk. On failure, thefile is rolled back and an actionable
ExtensionError::Vaulterroris returned pointing at the divergence rather than letting the
downstream caller fail with an opaque AEAD error. This also catches
issue hypothesis Bump docker/setup-buildx-action from 3 to 4 #2 (AAD path-binding regression) at the source.
No new dependencies, no new abstractions, no surrounding refactor.
Verification
cargo check --workspace --libcargo clippy -p librefang-extensions -p librefang-kernel --lib --tests -- -D warningscargo test -p librefang-extensions— 87 passed (40 vault unit + 47 sibling) + 4 vault integration tests intests/vault_roundtrip.rs.cargo test -p librefang-kernel --lib vault— 10 passed, including cross-test withvault_cache_reuses_unlocked_handle_across_calls(the kernel-side cached vault handle that also touchesLIBREFANG_VAULT_KEY).cargo test -p librefang-api --test mcp_auth_routes_extra_test— 8 passed (the integration tests that exercise theauth_starthandler the bug report cites).Two new regression tests pin the exact failing sequence:
vault::tests::init_then_set_then_reopen_unlock_via_env_key(extensions, vault layer): explicit
init()+set()thendrop-and-reopen
unlock()via the env-resolved master key, threesequential entries written to mirror the three-entry PKCE stash.
mcp_oauth_provider::tests::vault_set_twice_round_trips_via_env_key(kernel, OAuth provider layer): identical pattern but through the
KernelOAuthProvider::vault_setAPI the auth_start handler calls,plus a round-trip read of every entry through a fresh provider
instance.
Notes for reviewers
unlock()perinit()call (one Argon2id KDF run, one file read).
init()runs once pervault lifetime on a fresh install; the cost is paid once, never on
the hot path.
design — operators hitting this in production should see "init() and
resolve_master_key() resolved different master keys" rather than
having to dig through the issue tracker to interpret
aead::Error.still recommended (the
auth_start → POST /api/mcp/servers/<n>/auth/startflow). Unit + integration tests cover the vault layer's contract
but cannot fully simulate a Lazycat/Render-style container where
the operator-reported divergence first surfaced.