Repository navigation
[DV-05] [P1] Keep bridge response backpressure inside the bounded worker lifecycle #565
Description
Priority: P1
Area: Bridge transport
Repository / target branch: wavect/semaprax / wavect/v090
Reviewed commit: 5428b0805fa3cf35859c843a53e30c3252ef378c
Review date: 6 October 2026
Defect
A bridge/invoke worker calls registry.settle before it writes its reply. Settlement removes the request from the table that enforces MAX_IN_FLIGHT = 4. The worker then takes a shared output mutex and performs a blocking writeln!/flush.
If the client keeps sending requests but stops reading stdout, the first response eventually blocks on the full pipe and later response workers block on the output mutex. Their registry entries have already been removed, so the serving loop can continue spawning new workers for additional calls. The nominal four-call gate no longer bounds outstanding response workers or their retained result/ID buffers. A slow or stalled consumer can cause unbounded thread/memory growth; actual OS thread exhaustion need not be induced to demonstrate the escape.
EOF/shutdown also cannot finish while these workers are blocked. EndOfSession cancels only the still-live registry entries, which are empty for these workers; the Rust scope then waits for all response writers to return. A successful but blocked write does not set broken_flag, so the error-only cancellation logic does not help.
Executed reproduction
The test pre-handshakes a real Server, then calls the production serve_with with a barrier-controlled Write implementation representing a consumer that stops draining stdout. It sends 12 sequential invokes with invalid params, using each registry settlement as a barrier before sending the next request. No adapter, external service or slow provider is involved.
Observed on Linux:
MAX_IN_FLIGHT: 4.- Settled invocations while the writer remains blocked: 12.
registry.live(): 0.- Process thread count: 4 before, 16 during.
- EOF does not complete the session until the test releases the output writer.
After releasing the writer, every fixture worker and session exits. The test avoids exhausting the machine and makes no claim that a real production process was crashed. The production source was copied byte-for-byte with only a test module appended.
Smallest useful change
Account for each spawned worker through its response-delivery lifetime, including queued/blocked output. Releasing the provider-call slot may remain separate, but it cannot release all ownership that bounds a still-running reply worker. Introduce a small bounded response capacity/permit or reuse existing admission ownership; do not allow completed replies to accumulate unboundedly behind the output mutex.
For the supported stdio entry point, make stalled output an observable, bounded transport state so session stop can cancel live adapter work and finish according to a defined policy. Preserve response correlation and already-dispatched side-effect uncertainty if delivery fails. A generic arbitrary Write can block forever; do not merely advertise a bounded guarantee over such an object without a cancellable/bounded contract or an enforceable stdio implementation. Avoid adding an unrelated transport framework.
Acceptance criteria
- A blocked output consumer cannot create more outstanding worker/response ownership than the documented bound.
- Repeated fast refusals and successful replies are both covered; a provider-invoke limit alone is insufficient.
- Stalled/failed output triggers the defined session-end policy and does not prevent cancellation/settlement of unrelated live adapter work.
- The installed stdio path has a real pipe-backpressure regression with finite outer watchdog and test-controlled cleanup.
- Generic
serve_withembedding behavior clearly states or enforces its output cancellation/bounding requirements. - Responses retain the correct IDs, terminal states and dispatch uncertainty; stopping output never silently makes a possibly executed operation retryable.
- Healthy slow readers can resume within the supported allowance without duplicate execution or unbounded buffering.
Existing issue boundary
#476 handles transport operations that return an error, ensuring cancellation happens before joining workers. This case has no I/O error: Write remains blocked after the registry slot has been released. #454 required bounded concurrent cancellation/pipe handling but did not supply this response-worker ownership behavior. The existing early cancellation guard and frame-size limits should remain unchanged.
Verification handoff
Read AGENTS.md and docs/HARNESS-BRIDGE-V1.md. Extend the existing bridge lifecycle tests and use a barrier/capacity-controlled writer before a real bounded pipe test. Record the fixing commit, executed counts and ordinary/error/EOF cases. Keep tests local and bounded rather than forcing machine-wide resource exhaustion.
Activity
harness: bounded bridge backpressure, idle quarantine, strict chunkin…
wsdt commentedon Oct 6, 2026
Fixed in b5e3f37 (pushed to main). Root cause: registry.settle freed the in-flight slot before the blocking reply write, so nothing bounded response workers. Slots are now owned until the reply is delivered or fails (MAX_IN_FLIGHT, SPX-HPN012); an output-stall policy (default 30s, SEMAPRAX_BRIDGE_OUTPUT_STALL_MS) ends a stalled session with TimedOut; the installed --stdio path writes through a bounded detached writer on a dup'd fd. Evidence: bridge_lifecycle::backpressure tests (at most MAX_IN_FLIGHT+1 frames read while output blocked; exact one reply per id after resume; serve_with and serve_detached TimedOut; real --stdio child exits within a watchdog when its client stops reading); cargo test -p semaprax-harness green, clippy -D warnings clean.