Skip to content

Commit b55f9c2

Browse files
QuinnTyler Longwell
andcommitted
fix(relay/git/store): drop-and-floor classification for conformance probe races
The probe was conflating transport failures with conformance failures. `if_match_race` / `if_none_match_race` did `Err(e) → ProbeFailure` for any per-racer error, but `S3Error::Reqwest/Http/Io` means the racer never got a classified response from the backend — its outcome is *unknown*, not negative. A3 is a claim about **observers** ("exactly one CAS wins per (key, etag)"); an unknown racer isn't an observer. Treating one as a probe failure made the relay un-bootable on any constrained host that couldn't service the burst (Eva caught this on Desktop E2E at 32-way against a CI-box MinIO). Fix is drop-and-floor: - Pre-classification transport errors (`S3Error::{Reqwest, Http, Io}`) drop the racer from the observer set and increment `transport_drops`. - Post-receipt parse/decode errors (`Utf8`, `ReqwestHeaderToStr`, `SerdeXml`, `FromUtf8`, `InvalidHeaderValue`) and any `HttpFailWithBody` stay in the catch-all — the backend *did* answer, but not in the contract shape, which is a real conformance signal. - Per-round floor: `classified >= 2` (A3 needs ≥2 observers to *see* a race; with 0 classified the probe didn't run at all — fail closed). - `winners == 1` among classified observers (was: among all racers). - Create-only race contract is now `twos == 1 && twelves == classified - 1` (was: fixed `race_width - 1`, which false-positived on any transport drop). `ProbeReport` gains `transport_drops: usize` (total across both race phases), surfaced in the startup admission log line — a slowly- degrading backend shows up as "admitted with degraded observation count" before it's a probe failure. The shape is strictly smaller than the alternatives the team considered: no per-racer or per-round retry, no retry policy parameters, no startup-latency cost under flake. Crucially it doesn't smuggle a network-stack test into a correctness probe — retrying a classified outcome would change the TLA action and violate the same "retry only pre-classification errors" rule Sami's `cas_publish` guidance enforces. Eva's runtime workaround (`SPROUT_GIT_PROBE_WRITERS=8` on Desktop E2E, already shipped on 61cc38f) stays as a deployment-level knob, but the probe is no longer fragile to it. Diff scope: `store.rs` (the rewrite) + one log-line touch in `main.rs` (folding `transport_drops` into the admission log). 239 relay lib tests still green, clippy `--tests -D warnings` clean, fmt clean. Refs: - Channel discussion: PR #726 thread, shape (c) drop-and-floor agreed by Quinn + Max + Eva (the latter accepting the framing flip from her original (a)/(b) retry proposals). - docs/git-on-object-storage.md §Conformance, §Implementation Correspondence (retry rule: "if added, lives in the store layer and retries *only* pre-classification network errors"). Co-authored-by: Tyler Longwell <[email protected]>
1 parent 61cc38f commit b55f9c2

2 files changed

Lines changed: 112 additions & 8 deletions

File tree

‎crates/sprout-relay/src/api/git/store.rs‎

Lines changed: 111 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -115,6 +115,21 @@ pub struct ProbeReport {
115115
pub race_width: usize,
116116
/// Rounds executed per race phase.
117117
pub race_rounds: usize,
118+
/// Total number of *transport-unknown* per-racer outcomes across all
119+
/// race rounds (sum of both `if_match_race` and `if_none_match_race`
120+
/// phases). A "transport-unknown" is a pre-classification failure —
121+
/// `S3Error::{Reqwest, Http, Io}` — that means the racer never got a
122+
/// classified response from the backend, so its outcome is neither
123+
/// evidence for nor against A3 linearizability. Such racers are
124+
/// dropped from the observer set (see the race phases for the
125+
/// invariant: `classified >= 2` and `winners == 1` *among classified
126+
/// observers*).
127+
///
128+
/// Surfaced on the admission log line so a slowly-degrading backend
129+
/// shows up before it's a probe failure: a passing probe with
130+
/// non-zero `transport_drops` is "admitted with degraded
131+
/// observation count," not silently flaky.
132+
pub transport_drops: usize,
118133
}
119134

120135
/// Failure carrying the phase that failed plus enough context to diagnose.
@@ -405,6 +420,9 @@ impl GitStore {
405420
}
406421
let nonce = uuid::Uuid::new_v4();
407422
let pointer_key = format!("probe/pointer-{nonce}");
423+
// Accumulator for *transport-unknown* per-racer outcomes across both
424+
// race phases. See `ProbeReport::transport_drops` for the rationale.
425+
let mut transport_drops = 0usize;
408426

409427
// -- Phase 1: sequential --------------------------------------------------
410428
for round in 0..cfg.race_rounds {
@@ -460,15 +478,42 @@ impl GitStore {
460478
tasks.push(async move { me.put_pointer(&pkey, &body, Precond::IfMatch(et)).await });
461479
}
462480
let outcomes = futures_util::future::join_all(tasks).await;
481+
// Drop-and-floor classification. A `Reqwest`/`Http`/`Io` error
482+
// means the racer never got a classified response from the
483+
// backend (couldn't open a socket, send flaked, etc.); its
484+
// outcome is *unknown*, not negative. A3 is a claim about
485+
// **observers**: dropping unknowns from the observer set
486+
// sharpens the assertion ("exactly one winner among observers")
487+
// and avoids smuggling a network-stack test into the
488+
// conformance probe. Parse/decode errors (`Utf8`,
489+
// `ReqwestHeaderToStr`, `SerdeXml`, ...) and `HttpFailWithBody`
490+
// stay in the catch-all — those mean the backend *did* answer
491+
// but not in the contract shape, which is a real conformance
492+
// signal.
493+
let mut classified = 0usize;
463494
let mut winners = 0usize;
464495
let mut new_etag: Option<ETag> = None;
465496
for (i, outcome) in outcomes.into_iter().enumerate() {
466497
match outcome {
467498
Ok(CasOutcome::Won(e)) => {
499+
classified += 1;
468500
winners += 1;
469501
new_etag = Some(e);
470502
}
471-
Ok(CasOutcome::LostRace) => {}
503+
Ok(CasOutcome::LostRace) => {
504+
classified += 1;
505+
}
506+
Err(StoreError::Backend(
507+
S3Error::Reqwest(_) | S3Error::Http(_) | S3Error::Io(_),
508+
)) => {
509+
transport_drops += 1;
510+
tracing::warn!(
511+
phase = "if_match_race",
512+
round,
513+
racer = i,
514+
"transport drop (pre-classification: socket/send failure)"
515+
);
516+
}
472517
Err(e) => {
473518
return Err(ProbeFailure {
474519
phase: "if_match_race",
@@ -480,12 +525,29 @@ impl GitStore {
480525
}
481526
}
482527
}
528+
// A3 needs ≥2 observers to *see* a race. With 31/32 classified
529+
// and 1 transport drop, the race is well-observed; with 0/32
530+
// classified the probe didn't run at all — fail closed.
531+
if classified < 2 {
532+
return Err(ProbeFailure {
533+
phase: "if_match_race",
534+
round,
535+
key: pointer_key,
536+
reason: format!(
537+
"race not observed: classified={classified}, transport_drops={}",
538+
cfg.race_width - classified
539+
),
540+
}
541+
.into());
542+
}
483543
if winners != 1 {
484544
return Err(ProbeFailure {
485545
phase: "if_match_race",
486546
round,
487547
key: pointer_key,
488-
reason: format!("expected exactly 1 winner, got {winners}"),
548+
reason: format!(
549+
"expected exactly 1 winner among {classified} classified observers, got {winners}"
550+
),
489551
}
490552
.into());
491553
}
@@ -509,12 +571,24 @@ impl GitStore {
509571
tasks.push(async move { me.put_immutable_raw(&k, &b).await });
510572
}
511573
let results = futures_util::future::join_all(tasks).await;
574+
// Drop-and-floor: same classification rule as Phase 2. Drop
575+
// `Reqwest`/`Http`/`Io` (pre-classification — socket/send
576+
// failure); count 2xx + 412 as the classified observers. Any
577+
// other status or any non-transport `StoreError` is a real
578+
// conformance signal and fails closed.
579+
let mut classified = 0usize;
512580
let mut twos = 0usize;
513581
let mut twelves = 0usize;
514582
for (i, r) in results.into_iter().enumerate() {
515583
match r {
516-
Ok(200..=299) => twos += 1,
517-
Ok(412) => twelves += 1,
584+
Ok(200..=299) => {
585+
classified += 1;
586+
twos += 1;
587+
}
588+
Ok(412) => {
589+
classified += 1;
590+
twelves += 1;
591+
}
518592
Ok(code) => {
519593
return Err(ProbeFailure {
520594
phase: "if_none_match_race",
@@ -524,25 +598,53 @@ impl GitStore {
524598
}
525599
.into())
526600
}
601+
Err(StoreError::Backend(
602+
S3Error::Reqwest(_) | S3Error::Http(_) | S3Error::Io(_),
603+
)) => {
604+
transport_drops += 1;
605+
tracing::warn!(
606+
phase = "if_none_match_race",
607+
round,
608+
racer = i,
609+
"transport drop (pre-classification: socket/send failure)"
610+
);
611+
}
527612
Err(e) => {
528613
return Err(ProbeFailure {
529614
phase: "if_none_match_race",
530615
round,
531616
key,
532-
reason: format!("racer {i}: {e}"),
617+
reason: format!("racer {i} backend error: {e}"),
533618
}
534619
.into())
535620
}
536621
}
537622
}
538-
if twos != 1 || twelves != cfg.race_width - 1 {
623+
// Floor: A3 needs ≥2 observers to *see* a race.
624+
if classified < 2 {
625+
return Err(ProbeFailure {
626+
phase: "if_none_match_race",
627+
round,
628+
key,
629+
reason: format!(
630+
"race not observed: classified={classified}, transport_drops={}",
631+
cfg.race_width - classified
632+
),
633+
}
634+
.into());
635+
}
636+
// Create-only contract: exactly 1×2xx + (classified − 1)×412
637+
// *among observers*. The previous fixed `race_width − 1` would
638+
// false-positive on any transport drop; this expression honors
639+
// the drop-and-floor invariant.
640+
if twos != 1 || twelves != classified - 1 {
539641
return Err(ProbeFailure {
540642
phase: "if_none_match_race",
541643
round,
542644
key,
543645
reason: format!(
544-
"expected 1×2xx + {}×412, got {twos}×2xx + {twelves}×412",
545-
cfg.race_width - 1
646+
"expected 1×2xx + {}×412 among {classified} classified observers, got {twos}×2xx + {twelves}×412",
647+
classified - 1
546648
),
547649
}
548650
.into());
@@ -607,6 +709,7 @@ impl GitStore {
607709
Ok(ProbeReport {
608710
race_width: cfg.race_width,
609711
race_rounds: cfg.race_rounds,
712+
transport_drops,
610713
})
611714
}
612715

‎crates/sprout-relay/src/main.rs‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -258,6 +258,7 @@ async fn main() -> anyhow::Result<()> {
258258
tracing::info!(
259259
race_width = report.race_width,
260260
race_rounds = report.race_rounds,
261+
transport_drops = report.transport_drops,
261262
"git object-store backend admitted: A3 conformance probe passed"
262263
);
263264
}

0 commit comments

Comments
 (0)