Skip to content

fix(vault): unify init() key resolution with resolve_master_key()#5074

Merged
houko merged 2 commits into
mainfrom
fix/5069-vault-unlock
May 16, 2026
Merged

fix(vault): unify init() key resolution with resolve_master_key()#5074
houko merged 2 commits into
mainfrom
fix/5069-vault-unlock

Conversation

@houko

@houko houko commented May 16, 2026

Copy link
Copy Markdown
Contributor

Closes #5069

Root cause

KernelOAuthProvider::vault_set constructs a fresh CredentialVault
for every call. The MCP OAuth auth_start handler calls it three times
in a row to stash PKCE state (pkce_verifierpkce_state
redirect_uri). The first call walks init() + set() (file absent);
every subsequent call walks unlock() + set() against the freshly-
written file.

Pre-fix init() duplicated the env / keyring lookup code from
resolve_master_key(). The two code paths were not byte-identical
under every container condition — DBus partially available, env
mutation races from other code paths (e.g. the boot-time vault
unlock, the cached vault_handle() in kernel/accessors.rs, the
dotenv loader). When they diverged, init() wrote a file with key
K_init while the next unlock() resolved K_unlock, and AES-GCM
authentication failed with the opaque aead::Error the operator
report cites. The CLI never hit this because each librefang vault …
subcommand is a fresh process and only ever touches the vault once.

Fix

Minimal, two-part change in crates/librefang-extensions/src/vault.rs:

  1. Unify the key-resolution code path. init() now resolves the
    master key by calling self.resolve_master_key(). The "generate a
    random key" branch only kicks in when resolve returns
    VaultLocked (no env, no keyring) and that branch still stores the
    generated key into the keyring so the next resolve_master_key()
    on a fresh instance finds the same value. By construction the two
    sites can no longer diverge.

  2. Post-write verification. After save() returns, init()
    constructs a sibling CredentialVault::new instance and calls
    unlock() on the just-written file — exactly the code path the
    next KernelOAuthProvider::vault_set will walk. On failure, the
    file is rolled back and an actionable ExtensionError::Vault error
    is returned pointing at the divergence rather than letting the
    downstream caller fail with an opaque AEAD error. This also catches
    issue hypothesis Bump docker/setup-buildx-action from 3 to 4 #2 (AAD path-binding regression) at the source.

No new dependencies, no new abstractions, no surrounding refactor.

Verification

  • cargo check --workspace --lib
  • cargo clippy -p librefang-extensions -p librefang-kernel --lib --tests -- -D warnings
  • cargo test -p librefang-extensions — 87 passed (40 vault unit + 47 sibling) + 4 vault integration tests in tests/vault_roundtrip.rs.
  • cargo test -p librefang-kernel --lib vault — 10 passed, including cross-test with vault_cache_reuses_unlocked_handle_across_calls (the kernel-side cached vault handle that also touches LIBREFANG_VAULT_KEY).
  • cargo test -p librefang-api --test mcp_auth_routes_extra_test — 8 passed (the integration tests that exercise the auth_start handler the bug report cites).

Two new regression tests pin the exact failing sequence:

  • vault::tests::init_then_set_then_reopen_unlock_via_env_key
    (extensions, vault layer): explicit init() + set() then
    drop-and-reopen unlock() via the env-resolved master key, three
    sequential entries written to mirror the three-entry PKCE stash.
  • mcp_oauth_provider::tests::vault_set_twice_round_trips_via_env_key
    (kernel, OAuth provider layer): identical pattern but through the
    KernelOAuthProvider::vault_set API the auth_start handler calls,
    plus a round-trip read of every entry through a fresh provider
    instance.

Notes for reviewers

  • The post-write verification adds one extra unlock() per init()
    call (one Argon2id KDF run, one file read). init() runs once per
    vault lifetime on a fresh install; the cost is paid once, never on
    the hot path.
  • The error message in the verification failure branch is verbose by
    design — operators hitting this in production should see "init() and
    resolve_master_key() resolved different master keys" rather than
    having to dig through the issue tracker to interpret aead::Error.
  • Manual end-to-end repro in the original container deployment is
    still recommended (the auth_start → POST /api/mcp/servers/<n>/auth/start
    flow). Unit + integration tests cover the vault layer's contract
    but cannot fully simulate a Lazycat/Render-style container where
    the operator-reported divergence first surfaced.

)

The MCP OAuth auth_start handler stores PKCE state across three
sequential vault_set calls (pkce_verifier → pkce_state → redirect_uri).
The first call lazy-creates the vault via init() + set(); every
subsequent call walks unlock() + set() against a fresh CredentialVault
instance constructed inside KernelOAuthProvider::vault_set. Pre-fix the
second call's unlock() failed with aead::Error because init() duplicated
the env / keyring lookup code from resolve_master_key(). The two sites
could resolve different master keys on container hosts where DBus is
partially available or env mutation races with the vault unlock, leaving
a file the same process could not decrypt one call later.

Two changes:

1. init() now resolves the master key through resolve_master_key()
   directly. The random-key fallback only kicks in when
   resolve_master_key() returns VaultLocked (no env, no keyring), and
   that branch stores the generated key into the keyring so subsequent
   resolves on fresh instances find the same value. By construction the
   two code paths can no longer diverge.

2. init() now performs a post-write verification: it constructs a
   sibling CredentialVault::new instance and calls unlock() on the
   freshly-written file. If unlock fails, init() rolls back the file
   and returns an actionable Vault error pointing at the divergence
   rather than letting the next vault_set caller fail downstream with
   an opaque aead::Error. This also catches an AAD path-binding
   regression (issue hypothesis #2) at the source.

Regression coverage:

- vault::tests::init_then_set_then_reopen_unlock_via_env_key pins the
  init() + set() -> fresh-instance unlock() round trip via the
  env-resolved master key, mirroring the failing auth_start sequence.
- mcp_oauth_provider::tests::vault_set_twice_round_trips_via_env_key
  exercises the same flow through KernelOAuthProvider::vault_set --
  the exact API the auth_start handler calls -- and reads every entry
  back through a fresh provider instance to confirm no entry is lost
  across the re-open boundary.

Verification:
- cargo check --workspace --lib
- cargo clippy -p librefang-extensions -p librefang-kernel --lib --tests -- -D warnings
- cargo test -p librefang-extensions (87 passed)
- cargo test -p librefang-kernel --lib vault (10 passed, including
  cross-test interaction with vault_cache_reuses_unlocked_handle_across_calls)
- cargo test -p librefang-api --test mcp_auth_routes_extra_test (8 passed)
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

@github-actions github-actions Bot added area/kernel Core kernel (scheduling, RBAC, workflows) size/L 250-999 lines changed area/security Security systems and auditing labels May 16, 2026

@houko houko left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent review — APPROVE-WITH-NITS

Root cause assessment

The hypothesis ("init() duplicated key resolution → divergence") is plausible but unproven. With LIBREFANG_VAULT_KEY set and stable, the OLD init() and resolve_master_key() are byte-identical paths — they can only diverge if the env var is mutated between init's read and the next unlock's read. This PR does not identify any concrete mutator. The issue's hypothesis #2 (AAD path canonicalization) and #3 (salt-flush timing) are NOT cured by routing through resolve_master_key() alone. The post-write verification is the actual load-bearing change.

Fix robustness

Unification is safe and cannot make things worse. The post-save verification is the real fix: it constructs a fresh CredentialVault::new and runs the exact path the next vault_set walks, so any drift surfaces synchronously with a rollback. Residual TOCTOU: if LIBREFANG_VAULT_KEY is mutated between save and verify-unlock, init rolls back even though the daemon's subsequent vault_set (post-mutation value) might also fail — strictly better than the pre-fix opaque aead::Error.

WARNING

  • Tests do not pin the regression. Mentally simulating init_then_set_then_reopen_unlock_via_env_key against origin/main: env set before init() runs, never mutated, VAULT_NO_KEYRING=1. Old init() reads env → decode_master_key → identical bytes to what next unlock()'s resolve_master_key reads. Old code passes both new tests. They exercise the contract but don't reproduce the failure mode the issue describes. PR text "two new regression tests pin the exact failing sequence" overstates.
  • PR body acknowledges manual container repro is "still recommended". Combined with the above, the test suite cannot prove the bug is fixed — only that the happy path doesn't regress. Human verification in the original Lazycat/Docker environment remains a true gate.

NIT

  • init() doc-comment guess about "trailing whitespace tolerance / Zeroizing-vs-String temporary" is speculative — neither path strips whitespace, neither is optimizer-sensitive at env-read. Tone down to "duplicated paths could diverge under env mutation".
  • let _ = std::fs::remove_file(...) silences rollback errors — an EACCES leaves a broken file the next init() rejects with "Vault already exists". Debug-log on failed unlink would help operators.
  • drop(verify); is redundant — block-scope drop runs unconditionally one line later.

Follow-ups (in-scope per "fix what you found"):

  • Align new vault_set_twice_round_trips_via_env_key to #[serial_test::serial(librefang_vault_key)] to match other mcp_oauth_provider tests. The current state — two disjoint serial groups in the same crate touching the same process-global env — is exactly the race the commit message warns about. Agent flagged + didn't fix — that's a punt.
  • Add a regression test that mutates LIBREFANG_VAULT_KEY between init() and the verify-unlock — that's the actual failure mode the issue describes, and it would prove the post-save verification catches it.

Addresses code review on #5074.

- new vault::tests::init_env_mutation_during_save_rolls_back_and_surfaces_typed_error
  test: a worker thread mutates LIBREFANG_VAULT_KEY between init's
  resolve_master_key() and init's post-write verify-unlock, proving
  the verify-unlock catches the divergence, rolls back the corrupt
  vault.enc, and surfaces a typed ExtensionError::Vault(_) naming the
  divergence (not a bare aead::Error). Verified failing on origin/main
  (pre-fix init() returns Ok silently and the test's expect_err panics)
  and passing on this branch.
- align serial_test groups: vault_set_twice_round_trips_via_env_key
  (mcp_oauth_provider), vault_cache_reuses_unlocked_handle_across_calls
  and install_integration_writes_through_cached_vault_handle
  (kernel::tests) all move to #[serial_test::serial(librefang_vault_key)]
  so every test in librefang-kernel that mutates the process-global
  LIBREFANG_VAULT_KEY sits in one named group. Two disjoint serial
  groups in the same crate could otherwise race the env var.
- tone down init() doc-comment speculation about whitespace tolerance
  and Zeroizing-vs-String temporary lifetimes — neither path strips
  whitespace and neither is optimizer-sensitive at env-read. Replace
  with the concrete failure mode: env mutation between init's save
  and the next unlock's read.
- log on rollback unlink failure with tracing::warn (path, error).
  EACCES would otherwise silently leave a corrupt vault.enc that the
  next init() rejects with "Vault already exists"; operators need
  visibility into the residue.
- drop redundant drop(verify) — the block-scope drop runs anyway.
@houko
houko merged commit bb595bd into main May 16, 2026
30 checks passed
@houko
houko deleted the fix/5069-vault-unlock branch May 16, 2026 06:02
f-liva pushed a commit to f-liva/librefang that referenced this pull request May 16, 2026
Adapted from upstream PR librefang#5074 — minimal port without the startup-
sentinel changes that custom doesn't carry yet.

Two divergences caused the aead::Error from MCP OAuth vault_set on the
container deploy:

1. resolve_master_key() ordered "keyring → env → error", while init()
   ordered "env → keyring → random". When a host had a partially-
   available DBus keyring with a stale entry, init() encrypted vault.enc
   with the explicit env key, but the next unlock() walked
   resolve_master_key() and got the stale keyring value instead —
   surfacing as aead::Error.

2. init() duplicated the env/keyring lookup code from
   resolve_master_key(). Even with matching backends the two sites could
   diverge under env mutation between init's save and the next unlock's
   read.

Two changes:

- resolve_master_key() now prefers env over keyring. Explicit operator
  override (LIBREFANG_VAULT_KEY baked into lzc-manifest for container
  deploy) wins deterministically; keyring becomes the unattended-host
  fallback.

- init() now calls resolve_master_key() directly. The random-key
  fallback only kicks in when resolve_master_key() returns VaultLocked
  (no env, no keyring), and that branch persists the generated key into
  the keyring so subsequent resolves on fresh instances find the same
  value. By construction the two code paths can no longer diverge.

- init() now performs a post-write verification: constructs a sibling
  CredentialVault::new instance and calls unlock() on the freshly-
  written file. If unlock fails, init() rolls back the file and returns
  an actionable Vault error pointing at the divergence rather than
  letting the next vault_set caller fail downstream with an opaque
  aead::Error.

Verification:
- cargo check --workspace --lib (clean)
- cargo test -p librefang-extensions --lib (60 passed)
- cargo clippy -p librefang-extensions --lib --tests -- -D warnings (clean)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/kernel Core kernel (scheduling, RBAC, workflows) area/security Security systems and auditing size/L 250-999 lines changed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

vault: init() + subsequent unlock() in daemon process fail with aead::Error despite identical LIBREFANG_VAULT_KEY env

1 participant