fix: backpressure bounded queues instead of destroying the session/sidecar + centralized queue tracker#123
Merged
Conversation
railway-app
Bot
temporarily deployed
to
secure-exec / secure-exec-pr-123
June 24, 2026 16:55
Destroyed
railway-app
Bot
temporarily deployed
to
rivet-frontend / secure-exec-pr-123
June 24, 2026 16:55
Destroyed
NathanFlurry
force-pushed
the
fix/queue-backpressure-and-tracker
branch
from
June 24, 2026 17:14
1e5676c to
3034194
Compare
railway-app
Bot
temporarily deployed
to
secure-exec / secure-exec-pr-123
June 24, 2026 17:14
Destroyed
railway-app
Bot
temporarily deployed
to
rivet-frontend / secure-exec-pr-123
June 24, 2026 17:14
Destroyed
|
🚅 Deployed to the secure-exec-pr-123 environment in rivet-frontend
🚅 Deployed to the secure-exec-pr-123 environment in secure-exec
|
NathanFlurry
force-pushed
the
fix/queue-backpressure-and-tracker
branch
from
June 24, 2026 17:43
3034194 to
75779aa
Compare
railway-app
Bot
temporarily deployed
to
secure-exec / secure-exec-pr-123
June 24, 2026 17:44
Destroyed
railway-app
Bot
temporarily deployed
to
rivet-frontend / secure-exec-pr-123
June 24, 2026 17:44
Destroyed
NathanFlurry
force-pushed
the
fix/queue-backpressure-and-tracker
branch
from
June 24, 2026 17:53
75779aa to
5e447f8
Compare
railway-app
Bot
temporarily deployed
to
secure-exec / secure-exec-pr-123
June 24, 2026 17:53
Destroyed
railway-app
Bot
temporarily deployed
to
rivet-frontend / secure-exec-pr-123
June 24, 2026 17:53
Destroyed
NathanFlurry
force-pushed
the
fix/queue-backpressure-and-tracker
branch
from
June 24, 2026 18:42
5e447f8 to
11a992d
Compare
railway-app
Bot
temporarily deployed
to
secure-exec / secure-exec-pr-123
June 24, 2026 18:42
Destroyed
railway-app
Bot
temporarily deployed
to
rivet-frontend / secure-exec-pr-123
June 24, 2026 18:42
Destroyed
NathanFlurry
force-pushed
the
fix/queue-backpressure-and-tracker
branch
from
June 24, 2026 19:33
11a992d to
506fc46
Compare
railway-app
Bot
temporarily deployed
to
secure-exec / secure-exec-pr-123
June 24, 2026 19:33
Destroyed
railway-app
Bot
temporarily deployed
to
rivet-frontend / secure-exec-pr-123
June 24, 2026 19:33
Destroyed
NathanFlurry
force-pushed
the
fix/queue-backpressure-and-tracker
branch
from
June 24, 2026 19:50
506fc46 to
f7cc68e
Compare
railway-app
Bot
temporarily deployed
to
secure-exec / secure-exec-pr-123
June 24, 2026 19:50
Destroyed
railway-app
Bot
temporarily deployed
to
rivet-frontend / secure-exec-pr-123
June 24, 2026 19:50
Destroyed
NathanFlurry
force-pushed
the
fix/queue-backpressure-and-tracker
branch
from
June 25, 2026 03:11
f7cc68e to
275a863
Compare
railway-app
Bot
temporarily deployed
to
secure-exec / secure-exec-pr-123
June 25, 2026 03:11
Destroyed
railway-app
Bot
temporarily deployed
to
rivet-frontend / secure-exec-pr-123
June 25, 2026 03:11
Destroyed
NathanFlurry
force-pushed
the
fix/queue-backpressure-and-tracker
branch
from
June 25, 2026 07:52
275a863 to
9a65695
Compare
railway-app
Bot
temporarily deployed
to
rivet-frontend / secure-exec-pr-123
June 25, 2026 07:52
Destroyed
railway-app
Bot
temporarily deployed
to
secure-exec / secure-exec-pr-123
June 25, 2026 07:52
Destroyed
NathanFlurry
force-pushed
the
fix/queue-backpressure-and-tracker
branch
from
June 25, 2026 08:21
9a65695 to
a2e7648
Compare
railway-app
Bot
temporarily deployed
to
secure-exec / secure-exec-pr-123
June 25, 2026 08:21
Destroyed
railway-app
Bot
temporarily deployed
to
rivet-frontend / secure-exec-pr-123
June 25, 2026 08:21
Destroyed
The V8->host event channel and the sidecar stdout frame queue both reacted catastrophically when full instead of applying backpressure: - send_javascript_event: a single try_send returning Full called v8_session.destroy() -> Shutdown -> Exited(1), tearing the session down and truncating the event stream. - send_output_frame: try_send returning Full became a BrokenPipe error that propagated up and exited the whole sidecar process (code 1), taking every session with it. Both fire only when the host consumer is briefly slow (e.g. a chatty tool/skill turn outpacing a host that is momentarily not draining) — a transient, recoverable condition that should never be fatal. Changes: - Event channel and stdout frame queue now block (clean backpressure) until the consumer drains. Both producers run where a blocking wait is safe (a dedicated OS thread / a current-thread runtime whose writer drains on an independent thread); a genuinely dead consumer still terminates via Closed/ Disconnected rather than blocking forever. The oversized-single-event guard (a frame too large to ever send) still destroys, since it is unsendable. - Defense-in-depth headroom: raise JAVASCRIPT_EVENT_CHANNEL_CAPACITY 64->512 and MAX_STDOUT_FRAME_QUEUE 128->4096 so realistic bursts are absorbed before backpressure engages at all. - New centralized queue tracker (secure_exec_bridge::queue_tracker): every bounded queue registers a QueueGauge in a process-global registry; depth and high-water are tracked, a single edge-triggered warn! fires at >=80% fill (re-arming below 50%) so a slow consumer is visible before it stalls a session, and queue_snapshot() exposes live usage for debugging. Wired the event channel, the stdin frame queue (sampled via Tokio channel capacity), and the stdout frame queue (TrackedSyncSender/Receiver). Tests: - bridge: queue_tracker unit tests (depth/high-water, edge-triggered warn with hysteresis, registry snapshot + auto-prune). - execution: send_javascript_event now backpressures instead of destroying (unit) + an integration test (javascript_v8 suite) that bursts 1000 guest logs past the channel with a delayed host drain and asserts a clean exit with every line delivered (vs lines=capacity + torn-down session before the fix). - sidecar: stdout frame queue accepts > capacity frames under a slow drainer and only errors when the writer is gone. Defense-in-depth for the Zid integration's adapter-stall hang; the primary fix is host-side (non-blocking buffered adapter stderr). Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> Also documents the standard in CLAUDE.md (new "Limits, Bounds & Observability" section): bounded-by-default + config-wired/audited + registered-for-warn + clear-typed-error + no catastrophic reaction to transient fullness. Also fixes two more instances of the same destroy/silent-loss anti-pattern: - v8_host.rs v8_session_frames (1024): try_send Full -> destroy_session is now a blocking_send via tracked_sync_channel (backpressures the V8 runtime; only a dropped consumer is terminal). This per-session channel feeds the event channel, completing the end-to-end backpressure chain. (Note: a console.log repro can't fill it because the sync-RPC bridge serializes guest output one frame at a time; it shares the TrackedSyncSender mechanism the bridge unit tests cover.) - service.rs completed_sidecar_responses (10k): silent FIFO eviction is now LOUD — warns on the 80% approach and on every dropped unretrieved response, so a host that stops fetching responses is diagnosable instead of silently losing them. Also raise sidecar logging from ERROR-only to WARN (SECURE_EXEC_LOG-overridable) and redirect the subscriber to stderr — without this the near-limit/backpressure warnings added here were swallowed, and logging to stdout would corrupt the wire protocol (stdout is the framed channel). Enrich the tracker into a limits registry: LimitCategory (queue/resource/memory), a structured LimitWarning + a process-global warning sink (set_limit_warning_handler) that fires on the SAME edge-triggered/hysteresis-gated boundary as the log — the foundation for a host onLimitWarning hook — and category in snapshots. register_limit() lets saturating resources share the queue machinery. Register the per-VM saturating kernel resource limits (processes, open fds, pipes, ptys, sockets, connections, socket buffered bytes, datagram queue) with the limit registry, sampled in ResourceAccountant::snapshot(). These had ZERO approach warnings before — now they emit the same edge-triggered ~80% warning and appear in the registry snapshot, alongside the bounded queues. Forward limit-registry warnings to the host: the run_async event loop drains the warning sink into a StructuredEvent (name="limit_warning", detail={limit,category, observed,capacity,fillPercent}) emitted per active connection — reusing the existing wire StructuredEvent variant (no protocol schema change). Backs the agentOS onLimitWarning hook. Add ~80% APPROACH warnings (edge-triggered) for the CPU/time budgets, complementing the terminal exhaustion warnings: register a registry gauge sampled in the CPU-budget watchdog poll (v8_cpu_time_ms), an 80% timer in the wall-clock guard (v8_wall_clock_ms), and per-poll sampling of the WASM execution budget (wasm_fuel_ms). Memory (v8_heap_bytes) stays terminal-only via V8's near-heap-limit callback — a V8 isolate cannot be safely sampled from the watchdog thread. Docs updated to state approach-vs-terminal per category. Address PR review findings: - Emit each (process-global) limit_warning ONCE to a single active connection instead of fanning out a copy per connection (active_connections has no stale entries: the stdio transport only removes connections at teardown). - Bound the send_request frame write by the caller's deadline (was a blocking send that ignored the timeout when stdout stalled); add TrackedSyncSender::try_send. - Register gauges + ~80% approach warnings for the three remaining bounded VecDeques (pending_process_events, pending_sidecar_responses, outbound_sidecar_requests) so they meet the new standard; doc warning-names table updated. - Disarm the limit_warning select! branch on a closed receiver (no busy-loop), never break. - Observe the filesystem-bytes/inode gauges only on the success path (a rejected over-limit write no longer latches a spurious near-capacity warning). - record_dequeue now re-evaluates so std-channel gauges re-arm as they drain (+ test). - dispatch_warning clones the handler Arc out and drops the registry mutex before invoking it (avoid running the handler under the kernel fd_tables lock order). - Fix stale '64-slot' comments (now 512) and pin the round-trip test to the real emitted limit name (javascript_event_channel).
NathanFlurry
force-pushed
the
fix/queue-backpressure-and-tracker
branch
from
June 25, 2026 08:28
a2e7648 to
b870388
Compare
railway-app
Bot
temporarily deployed
to
secure-exec / secure-exec-pr-123
June 25, 2026 08:28
Destroyed
railway-app
Bot
temporarily deployed
to
rivet-frontend / secure-exec-pr-123
June 25, 2026 08:28
Destroyed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Two bounded queues in the guest→host path reacted catastrophically when full instead of applying backpressure:
crates/execution/src/javascript.rs) —send_javascript_eventdidtry_send, and onFullcalledv8_session.destroy()→Shutdown→Exited(1). A transient backlog tore the whole session down with a truncated event stream.crates/sidecar/src/stdio.rs) —send_output_framedidtry_send, and onFullreturned aBrokenPipeerror that propagated up and exited the entire sidecar process (code 1), taking every session with it.Both only fire when the host consumer is briefly slow (e.g. a chatty tool/skill turn outpacing a host that is momentarily not draining stdout) — a transient, recoverable condition that should never be fatal. This is the defense-in-depth half of the Zid adapter-stall investigation (the primary fix is host-side: a non-blocking buffered adapter-stderr WriteStream instead of per-line sync I/O).
Fix
Backpressure instead of self-destruct. Both producers now block until the consumer drains a slot:
blocking_send.SyncSender::sendbackpressures without deadlock. A genuinely dead consumer still terminates the producer viaClosed/Disconnected(e.g. the writer thread detecting a real broken pipe) rather than blocking forever.Headroom. Raise
JAVASCRIPT_EVENT_CHANNEL_CAPACITY64→512 andMAX_STDOUT_FRAME_QUEUE128→4096 so realistic bursts are absorbed before backpressure engages at all.Centralized queue tracker (
secure_exec_bridge::queue_tracker) — the explicit ask for observability:QueueGaugein a process-globalQueueRegistry.warn!fires at ≥80% fill (re-arming below 50% for hysteresis) so a slow consumer is visible before it stalls a session.queue_snapshot()/log_queue_snapshot()expose live usage (name/depth/high-water/capacity/fill%) for debugging.max_capacity - capacity), and the stdout frame queue (via aTrackedSyncSender/TrackedReceiverwrapper that records enqueue/dequeue).Tests
queue_trackerunit tests — depth/high-water tracking, edge-triggered warn with hysteresis, registry snapshot + auto-prune of dropped queues.send_javascript_eventnow backpressures instead of destroying (unit, with a slow drainer) + an integration test in thejavascript_v8suite that bursts 1000 guest logs past the channel with a delayed host drain and asserts a clean exit with every line delivered. Without the fix the same test reproduces the bug exactly:lines=64/1000(the old capacity) and a torn-down session.BrokenPipe).All green:
cargo build --workspaceclean (no warnings), bridge / sidecar-lib / execution unit +javascript_v8integration /limits_auditsuites pass.