Skip to content

Commit 5aed49b

Browse files
wpfleger96Duncan
andauthored
feat(buzz-agent): add DatabricksAuthCoordinator single-flight OAuth (#5545)
## What Consolidates all Databricks OAuth acquisition behind one coordinator on `PkceOAuthTokenSource`. Every entry point — the four `TokenSource` methods (`bearer`, `bearer_no_browser`, `refresh_now`, `interactive_login`) plus the public `acquire_with_intent` — routes through a single `acquire()`/`acquire_locked()` core that owns browser and cooldown policy. Before this, acquisition logic was scattered across those methods with no coordination: concurrent callers (Desktop discovery, the saved-agent model picker, managed-runtime inference) could each pop their own browser, and a just-denied attempt would immediately re-prompt on the next passive read. ## How - **Intent policy.** `AuthIntent::{Auto, UserInitiated, Headless}` decides whether a caller may open a browser and whether it honors the cooldown. `Headless` never browses; `Auto` browses but honors an unexpired cooldown; `UserInitiated` browses and bypasses+clears the cooldown. - **Two-layer single-flight per cache key.** An in-process registry (`INFLIGHT`) coalesces same-key, same-intent callers onto one leader's attempt before the file lock. The slot key is `(lock_path, AuthIntent)`, so a `UserInitiated` sign-in never inherits an `Auto` leader's result. A joined result is revalidated against the waiter's own contract. Across processes, callers serialize on a `flock`-based advisory lock and share success through the on-disk cache. RAII `Drop` releases both lock and leader slot. - **Joiner credential-state reconciliation.** `SlotPublish` carries the full `CachedToken` on success. Each joiner reconciles its own independent `state` cell under `state.lock().await` before returning: adopt when absent, expired, or matching the joiner's rejected credential; preserve any distinct newer usable credential. On a matching shared failure, neutralize the joiner's in-memory rejected entry under lock via `expire_rejected_memory` — durable disk mutation is reserved for `acquire_locked` under the cross-process file lock. Without this, a joining source's state remains stale or empty and subsequent plain `bearer()` calls resurface the rejected or absent credential. - **Validate-before-persist boundary.** `finish()` is the candidate-token persistence boundary for refresh and browser results. Before a token is written to cache or the cooldown is cleared, a bearer equal to the caller's rejected bytes yields a typed failure. - **Token neutralization.** When `acquire_locked` enters with `rejected = Some(bytes)`, it calls `expire_rejected()` under the state lock before any cache check. - **Cross-process failure single-flight.** An `AttemptRecord` sidecar records a monotonically-increasing generation, intent, result code, and SHA-256 digest of the completing caller's rejected token. Adoption is temporal (pre-queue snapshot predates current generation) and digest-matched. - **Typed outcomes.** `AuthError` with stable `code()`/`from_code()` replaces display-text matching. - **Durable cooldown sidecar.** Every failed browser attempt is recorded next to the cache key. 5-minute expiry. - **Windows disk persistence disabled.** On non-Unix platforms, `persist()` is a no-op. Lock, cooldown, and attempt sidecars are active on all platforms. Tests that seed or assert on the on-disk token cache are `#[cfg(unix)]`-gated. - **Injected browser opener** invoked while the localhost callback listener is live. ## Tests - `crates/buzz-agent/tests/databricks_auth_coordinator.rs`: browser/cooldown/classification acceptance matrix with a scripted `BrowserOpener` and stub OIDC provider. P1 regressions exercise the full `finish()` → `acquire_locked()` → `acquire_leader()` → `LeaderGuard::complete()` → joiner wiring: - `test_inprocess_joiner_reconciles_stale_state_after_shared_success` (Unix): two real sources both loaded locally-fresh-but-rejected X; after shared success Y, subsequent plain `bearer()` on both returns Y, not X. - `test_inprocess_joiner_neutralizes_rejected_on_matching_shared_failure` (Unix): B's matching rejected X is force-expired in memory after shared `RefreshRejected`; subsequent read cannot return X. - `test_inprocess_joiner_populates_empty_state_no_second_acquisition` (non-Unix): empty A/B join a browser success; B's subsequent headless read returns Y without a second browser (no disk fallback on non-Unix exposes the regression). - `test_crossprocess_userinitiated_waiter_adopts_predecessor_denial` (snapshot-marker barrier replacing an earlier sleep for deterministic generation ordering). - `auth.rs` in-crate tests: lock-primitive edges, disk-recheck on shared failure (`#[cfg(unix)]`), and: - `test_joiner_reconciliation_blocked_until_state_lock_released`: deterministic direct-poll proof that awaited reconciliation requires `state.lock().await`. The test task holds B's state mutex and manually polls a pinned real `acquire()` future — Poll 2 (slot published, mutex still held) must return `Pending` because `lock().await` blocks; with `try_lock` instead, Poll 2 returns `Ready`, failing the assertion. - `test_joiner_preserve_distinct_newer_credential`: deterministic direct polling parks B at `slot.wait()`, then writes Z directly into B's state in the same task, then publishes Y and awaits completion. B must return Y but leave state == Z. Mutation check: unconditional adoption overwrites Z with Y, failing the state assertion. - `test_joiner_shared_failure_recovers_disk_replacement` (Unix): the matching-failure joiner enters the recovery branch — after `expire_rejected_memory` (in-memory, empty state no-op) it reads a sibling-written disk replacement via `usable_from_disk` and returns it. Mutation check: removing the `usable_from_disk` recovery branch returns `Err(RefreshRejected)`. - `test_joiner_failure_does_not_write_disk` (Unix): byte-for-byte disk-invariance regression — a matching-failure joiner calls `expire_rejected_memory` and must not touch the on-disk cache. An independent process C may write a valid replacement between A's failure and B's reconciliation; this guard ensures B's unfenced in-memory neutralization cannot overwrite C's concurrent disk write. Mutation check: reverting to `expire_rejected` rewrites the file (`expires_at = 0`), changing the bytes and failing the assertion. - `test_lock_timeout_leaves_cooldown_sidecar_byte_for_byte_untouched`: a waiter past its deadline returns `LockTimeout` before entering `acquire_locked`; the cooldown sidecar bytes are unchanged. Documents a pre-lock limitation: `LockTimeout` callers do not neutralize state or sidecars. ## Scope / follow-ups - **Runtime 401 handling is deferred.** This PR owns acquisition single-flight and policy. - **Desktop wiring is Phase 2** (not in this PR's boundary). Confined to `crates/buzz-agent/`. - **Windows DACL** is a follow-up once the `windows-sys` binding is available. ## Stack Built on [#5534](#5534) (`hayt/databricks-oauth-cache-hardening`), now merged. Retargeted to `main`. --------- Signed-off-by: Will Pfleger <[email protected]> Signed-off-by: Duncan <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz> Co-authored-by: Duncan <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz>
1 parent e5a7e26 commit 5aed49b

10 files changed

Lines changed: 5952 additions & 236 deletions

File tree

‎.github/workflows/ci.yml‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1170,6 +1170,20 @@ jobs:
11701170
# Serial: windows_resolver_tests mutate process-global env
11711171
# (BUZZ_SHELL/GIT_BASH/SystemRoot) that SharedState::new reads.
11721172
run: cargo test -p buzz-dev-mcp --target $env:TARGET -- --test-threads=1
1173+
- name: Test (buzz-agent auth coordinator)
1174+
# The auth coordinator single-flights on an OS advisory lock, which is
1175+
# LockFileEx on Windows; this integration suite drives real second
1176+
# processes on the same lock file, so it only exercises the Windows
1177+
# lock runtime if it runs ON Windows. Every other job compiles it but
1178+
# never executes it. Tests exercised on Windows: lock serialization
1179+
# (two coordinators race for the same key), cooldown sidecar sharing
1180+
# across processes, attempt-sidecar adoption (UserInitiated waiter
1181+
# adopts a predecessor's denial), and the in-process single-flight for
1182+
# same-key coalescing. Tests that are UNIX-ONLY and NOT executed here:
1183+
# crash-release (flock drop on SIGKILL, guarded by #[cfg(unix)]) and
1184+
# cross-process cache success/race (on-disk token handoff, also
1185+
# #[cfg(unix)]).
1186+
run: cargo test -p buzz-agent --target $env:TARGET --test databricks_auth_coordinator
11731187
# Smoke-test the new host-prereq contract: Git for Windows (which provides
11741188
# bash) is available on the runner, a shell command round-trips, and bash
11751189
# does NOT resolve from System32 (so WSL's launcher is never picked up).

‎Cargo.lock‎

Lines changed: 11 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎Justfile‎

Lines changed: 15 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -391,14 +391,21 @@ test-unit:
391391
# because nothing in CI runs `cargo test --workspace` — workspace
392392
# membership alone buys clippy/check, not a single executed test.
393393
cargo nextest run -p buzz-backend-kubernetes
394-
# buzz-agent model-capabilities corpus: the Rust half of the
395-
# cross-language drift guard. `model_capabilities.rs` embeds
396-
# scripts/model-capabilities.json + scripts/normative-corpus.json via
397-
# include_str! and replays the full locked corpus as pure in-process tests (no
398-
# infra). Enumerated explicitly because nothing in CI runs
399-
# `cargo test --workspace`; without this step a manifest edit that
400-
# diverges Rust from the corpus ships green.
401-
cargo nextest run -p buzz-agent --lib
394+
# buzz-agent: two infra-free concerns run together by executing the
395+
# whole crate (lib + integration tests), because nothing in CI runs
396+
# `cargo test --workspace`, so without this stanza neither the crate's
397+
# library tests nor its integration tests execute remotely.
398+
# * model-capabilities corpus (lib): the Rust half of the
399+
# cross-language drift guard. `model_capabilities.rs` embeds
400+
# scripts/model-capabilities.json + scripts/normative-corpus.json via
401+
# include_str! and replays the full locked corpus as pure in-process
402+
# tests; without it a manifest edit that diverges Rust from the
403+
# corpus ships green.
404+
# * OAuth auth coordinator (lib concurrency matrix + databricks
405+
# integration tests): lock single-flight, cooldown, cross-process
406+
# crash recovery — infra-free via a stub OIDC provider and an
407+
# injected browser opener, no network or Postgres.
408+
cargo nextest run -p buzz-agent
402409
# Admin API auth-boundary tests (api::admin in buzz-relay): the NIP-98
403410
# duplicate-tag rejections, the Host/Origin replay-ordering causal pair,
404411
# the admin.localhost origin/advertisement/canonical-URL pins, and the

‎crates/buzz-agent/Cargo.toml‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,24 @@ path = "src/main.rs"
2424
name = "fake-mcp"
2525
path = "tests/bin/fake_mcp.rs"
2626

27+
# Test-only lock holder: a real second process that takes the coordinator's
28+
# cross-process advisory lock, so the auth tests can prove genuine
29+
# inter-process single-flight and crash-release rather than same-process
30+
# handles. Tiny; only used by the databricks auth integration tests.
31+
[[bin]]
32+
name = "lock-holder"
33+
path = "tests/bin/lock_holder.rs"
34+
35+
# Test-only auth worker: a real second process that runs the PUBLIC auth
36+
# coordinator API (`acquire_with_intent`) with a scripted browser opener and a
37+
# shared temp cache, so the auth tests can prove the cross-process single-flight
38+
# contract end-to-end — durable cooldown sharing and one-grant/one-cache races
39+
# across a genuine process boundary, not two in-process handles. Only used by
40+
# the databricks auth integration tests.
41+
[[bin]]
42+
name = "auth-worker"
43+
path = "tests/bin/auth_worker.rs"
44+
2745
[dependencies]
2846
tokio = { workspace = true, features = ["rt-multi-thread", "macros", "io-std", "io-util", "sync", "process", "time", "net"] }
2947
serde = { workspace = true }
@@ -45,6 +63,11 @@ url = { workspace = true }
4563
urlencoding = "2"
4664
webbrowser = "1"
4765
dirs = "6"
66+
# Cross-process advisory file lock (flock on Unix, LockFileEx on Windows) for
67+
# the auth coordinator's single-flight. Kept off std's `File::try_lock` so the
68+
# crate stays buildable on the repo's declared 1.88 MSRV (those std APIs are
69+
# 1.89+).
70+
fs2 = "0.4"
4871

4972
[target.'cfg(unix)'.dependencies]
5073
nix = { version = "0.31", default-features = false, features = ["signal", "process"] }

0 commit comments

Comments
 (0)