Repository navigation
Commit b55f9c2
fix(relay/git/store): drop-and-floor classification for conformance probe races
The probe was conflating transport failures with conformance failures.
`if_match_race` / `if_none_match_race` did `Err(e) → ProbeFailure` for
any per-racer error, but `S3Error::Reqwest/Http/Io` means the racer
never got a classified response from the backend — its outcome is
*unknown*, not negative. A3 is a claim about **observers** ("exactly
one CAS wins per (key, etag)"); an unknown racer isn't an observer.
Treating one as a probe failure made the relay un-bootable on any
constrained host that couldn't service the burst (Eva caught this on
Desktop E2E at 32-way against a CI-box MinIO).
Fix is drop-and-floor:
- Pre-classification transport errors (`S3Error::{Reqwest, Http, Io}`)
drop the racer from the observer set and increment `transport_drops`.
- Post-receipt parse/decode errors (`Utf8`, `ReqwestHeaderToStr`,
`SerdeXml`, `FromUtf8`, `InvalidHeaderValue`) and any
`HttpFailWithBody` stay in the catch-all — the backend *did* answer,
but not in the contract shape, which is a real conformance signal.
- Per-round floor: `classified >= 2` (A3 needs ≥2 observers to *see* a
race; with 0 classified the probe didn't run at all — fail closed).
- `winners == 1` among classified observers (was: among all racers).
- Create-only race contract is now `twos == 1 && twelves == classified
- 1` (was: fixed `race_width - 1`, which false-positived on any
transport drop).
`ProbeReport` gains `transport_drops: usize` (total across both race
phases), surfaced in the startup admission log line — a slowly-
degrading backend shows up as "admitted with degraded observation
count" before it's a probe failure.
The shape is strictly smaller than the alternatives the team
considered: no per-racer or per-round retry, no retry policy
parameters, no startup-latency cost under flake. Crucially it doesn't
smuggle a network-stack test into a correctness probe — retrying a
classified outcome would change the TLA action and violate the same
"retry only pre-classification errors" rule Sami's `cas_publish`
guidance enforces.
Eva's runtime workaround (`SPROUT_GIT_PROBE_WRITERS=8` on Desktop E2E,
already shipped on 61cc38f) stays as a deployment-level knob, but the
probe is no longer fragile to it.
Diff scope: `store.rs` (the rewrite) + one log-line touch in `main.rs`
(folding `transport_drops` into the admission log). 239 relay lib
tests still green, clippy `--tests -D warnings` clean, fmt clean.
Refs:
- Channel discussion: PR #726 thread, shape (c) drop-and-floor
agreed by Quinn + Max + Eva (the latter accepting the framing flip
from her original (a)/(b) retry proposals).
- docs/git-on-object-storage.md §Conformance, §Implementation
Correspondence (retry rule: "if added, lives in the store layer and
retries *only* pre-classification network errors").
Co-authored-by: Tyler Longwell <[email protected]>1 parent 61cc38f commit b55f9c2
2 files changed
Lines changed: 112 additions & 8 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
115 | 115 | | |
116 | 116 | | |
117 | 117 | | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
118 | 133 | | |
119 | 134 | | |
120 | 135 | | |
| |||
405 | 420 | | |
406 | 421 | | |
407 | 422 | | |
| 423 | + | |
| 424 | + | |
| 425 | + | |
408 | 426 | | |
409 | 427 | | |
410 | 428 | | |
| |||
460 | 478 | | |
461 | 479 | | |
462 | 480 | | |
| 481 | + | |
| 482 | + | |
| 483 | + | |
| 484 | + | |
| 485 | + | |
| 486 | + | |
| 487 | + | |
| 488 | + | |
| 489 | + | |
| 490 | + | |
| 491 | + | |
| 492 | + | |
| 493 | + | |
463 | 494 | | |
464 | 495 | | |
465 | 496 | | |
466 | 497 | | |
467 | 498 | | |
| 499 | + | |
468 | 500 | | |
469 | 501 | | |
470 | 502 | | |
471 | | - | |
| 503 | + | |
| 504 | + | |
| 505 | + | |
| 506 | + | |
| 507 | + | |
| 508 | + | |
| 509 | + | |
| 510 | + | |
| 511 | + | |
| 512 | + | |
| 513 | + | |
| 514 | + | |
| 515 | + | |
| 516 | + | |
472 | 517 | | |
473 | 518 | | |
474 | 519 | | |
| |||
480 | 525 | | |
481 | 526 | | |
482 | 527 | | |
| 528 | + | |
| 529 | + | |
| 530 | + | |
| 531 | + | |
| 532 | + | |
| 533 | + | |
| 534 | + | |
| 535 | + | |
| 536 | + | |
| 537 | + | |
| 538 | + | |
| 539 | + | |
| 540 | + | |
| 541 | + | |
| 542 | + | |
483 | 543 | | |
484 | 544 | | |
485 | 545 | | |
486 | 546 | | |
487 | 547 | | |
488 | | - | |
| 548 | + | |
| 549 | + | |
| 550 | + | |
489 | 551 | | |
490 | 552 | | |
491 | 553 | | |
| |||
509 | 571 | | |
510 | 572 | | |
511 | 573 | | |
| 574 | + | |
| 575 | + | |
| 576 | + | |
| 577 | + | |
| 578 | + | |
| 579 | + | |
512 | 580 | | |
513 | 581 | | |
514 | 582 | | |
515 | 583 | | |
516 | | - | |
517 | | - | |
| 584 | + | |
| 585 | + | |
| 586 | + | |
| 587 | + | |
| 588 | + | |
| 589 | + | |
| 590 | + | |
| 591 | + | |
518 | 592 | | |
519 | 593 | | |
520 | 594 | | |
| |||
524 | 598 | | |
525 | 599 | | |
526 | 600 | | |
| 601 | + | |
| 602 | + | |
| 603 | + | |
| 604 | + | |
| 605 | + | |
| 606 | + | |
| 607 | + | |
| 608 | + | |
| 609 | + | |
| 610 | + | |
| 611 | + | |
527 | 612 | | |
528 | 613 | | |
529 | 614 | | |
530 | 615 | | |
531 | 616 | | |
532 | | - | |
| 617 | + | |
533 | 618 | | |
534 | 619 | | |
535 | 620 | | |
536 | 621 | | |
537 | 622 | | |
538 | | - | |
| 623 | + | |
| 624 | + | |
| 625 | + | |
| 626 | + | |
| 627 | + | |
| 628 | + | |
| 629 | + | |
| 630 | + | |
| 631 | + | |
| 632 | + | |
| 633 | + | |
| 634 | + | |
| 635 | + | |
| 636 | + | |
| 637 | + | |
| 638 | + | |
| 639 | + | |
| 640 | + | |
539 | 641 | | |
540 | 642 | | |
541 | 643 | | |
542 | 644 | | |
543 | 645 | | |
544 | | - | |
545 | | - | |
| 646 | + | |
| 647 | + | |
546 | 648 | | |
547 | 649 | | |
548 | 650 | | |
| |||
607 | 709 | | |
608 | 710 | | |
609 | 711 | | |
| 712 | + | |
610 | 713 | | |
611 | 714 | | |
612 | 715 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
258 | 258 | | |
259 | 259 | | |
260 | 260 | | |
| 261 | + | |
261 | 262 | | |
262 | 263 | | |
263 | 264 | | |
| |||
0 commit comments