Skip to content

fix: backpressure bounded queues instead of destroying the session/sidecar + centralized queue tracker#123

Merged
NathanFlurry merged 1 commit into
mainfrom
fix/queue-backpressure-and-tracker
Jun 25, 2026
Merged

fix: backpressure bounded queues instead of destroying the session/sidecar + centralized queue tracker#123
NathanFlurry merged 1 commit into
mainfrom
fix/queue-backpressure-and-tracker

Conversation

@NathanFlurry

@NathanFlurry NathanFlurry commented Jun 24, 2026

Copy link
Copy Markdown
Member

Problem

Two bounded queues in the guest→host path reacted catastrophically when full instead of applying backpressure:

  1. Event channel (crates/execution/src/javascript.rs) — send_javascript_event did try_send, and on Full called v8_session.destroy()ShutdownExited(1). A transient backlog tore the whole session down with a truncated event stream.
  2. Stdout frame queue (crates/sidecar/src/stdio.rs) — send_output_frame did try_send, and on Full returned a BrokenPipe error that propagated up and exited the entire sidecar process (code 1), taking every session with it.

Both only fire when the host consumer is briefly slow (e.g. a chatty tool/skill turn outpacing a host that is momentarily not draining stdout) — a transient, recoverable condition that should never be fatal. This is the defense-in-depth half of the Zid adapter-stall investigation (the primary fix is host-side: a non-blocking buffered adapter-stderr WriteStream instead of per-line sync I/O).

Fix

Backpressure instead of self-destruct. Both producers now block until the consumer drains a slot:

  • The event channel runs on a dedicated OS thread → blocking_send.
  • The stdout queue producer is a current-thread runtime whose writer drains on an independent thread, so a blocking SyncSender::send backpressures without deadlock. A genuinely dead consumer still terminates the producer via Closed/Disconnected (e.g. the writer thread detecting a real broken pipe) rather than blocking forever.
  • The oversized-single-event guard (a frame too large to ever send) still destroys — it's genuinely unsendable, not backpressure.

Headroom. Raise JAVASCRIPT_EVENT_CHANNEL_CAPACITY 64→512 and MAX_STDOUT_FRAME_QUEUE 128→4096 so realistic bursts are absorbed before backpressure engages at all.

Centralized queue tracker (secure_exec_bridge::queue_tracker) — the explicit ask for observability:

  • Every bounded queue registers a QueueGauge in a process-global QueueRegistry.
  • Depth + high-water are tracked; a single edge-triggered warn! fires at ≥80% fill (re-arming below 50% for hysteresis) so a slow consumer is visible before it stalls a session.
  • queue_snapshot() / log_queue_snapshot() expose live usage (name/depth/high-water/capacity/fill%) for debugging.
  • Wired: the V8 event channel and the stdin frame queue (depth sampled from the Tokio channel's max_capacity - capacity), and the stdout frame queue (via a TrackedSyncSender/TrackedReceiver wrapper that records enqueue/dequeue).

Tests

  • bridge: queue_tracker unit tests — depth/high-water tracking, edge-triggered warn with hysteresis, registry snapshot + auto-prune of dropped queues.
  • execution: send_javascript_event now backpressures instead of destroying (unit, with a slow drainer) + an integration test in the javascript_v8 suite that bursts 1000 guest logs past the channel with a delayed host drain and asserts a clean exit with every line delivered. Without the fix the same test reproduces the bug exactly: lines=64/1000 (the old capacity) and a torn-down session.
  • sidecar: stdout frame queue accepts > capacity frames under a slow drainer (backpressure) and only errors when the writer is gone (BrokenPipe).

All green: cargo build --workspace clean (no warnings), bridge / sidecar-lib / execution unit + javascript_v8 integration / limits_audit suites pass.

@railway-app
railway-app Bot temporarily deployed to secure-exec / secure-exec-pr-123 June 24, 2026 16:55 Destroyed
@railway-app
railway-app Bot temporarily deployed to rivet-frontend / secure-exec-pr-123 June 24, 2026 16:55 Destroyed
@NathanFlurry
NathanFlurry force-pushed the fix/queue-backpressure-and-tracker branch from 1e5676c to 3034194 Compare June 24, 2026 17:14
@railway-app
railway-app Bot temporarily deployed to secure-exec / secure-exec-pr-123 June 24, 2026 17:14 Destroyed
@railway-app
railway-app Bot temporarily deployed to rivet-frontend / secure-exec-pr-123 June 24, 2026 17:14 Destroyed
@railway-app

railway-app Bot commented Jun 24, 2026

Copy link
Copy Markdown

🚅 Deployed to the secure-exec-pr-123 environment in rivet-frontend

Service Status Web Updated (UTC)
secure-exec 😴 Sleeping (View Logs) Jun 25, 2026 at 8:34 am

🚅 Deployed to the secure-exec-pr-123 environment in secure-exec

Service Status Web Updated (UTC)
secure-exec 😴 Sleeping (View Logs) Web Jun 25, 2026 at 8:37 am

@NathanFlurry
NathanFlurry force-pushed the fix/queue-backpressure-and-tracker branch from 3034194 to 75779aa Compare June 24, 2026 17:43
@railway-app
railway-app Bot temporarily deployed to secure-exec / secure-exec-pr-123 June 24, 2026 17:44 Destroyed
@railway-app
railway-app Bot temporarily deployed to rivet-frontend / secure-exec-pr-123 June 24, 2026 17:44 Destroyed
@NathanFlurry
NathanFlurry force-pushed the fix/queue-backpressure-and-tracker branch from 75779aa to 5e447f8 Compare June 24, 2026 17:53
@railway-app
railway-app Bot temporarily deployed to secure-exec / secure-exec-pr-123 June 24, 2026 17:53 Destroyed
@railway-app
railway-app Bot temporarily deployed to rivet-frontend / secure-exec-pr-123 June 24, 2026 17:53 Destroyed
@NathanFlurry
NathanFlurry force-pushed the fix/queue-backpressure-and-tracker branch from 5e447f8 to 11a992d Compare June 24, 2026 18:42
@railway-app
railway-app Bot temporarily deployed to secure-exec / secure-exec-pr-123 June 24, 2026 18:42 Destroyed
@railway-app
railway-app Bot temporarily deployed to rivet-frontend / secure-exec-pr-123 June 24, 2026 18:42 Destroyed
@NathanFlurry
NathanFlurry force-pushed the fix/queue-backpressure-and-tracker branch from 11a992d to 506fc46 Compare June 24, 2026 19:33
@railway-app
railway-app Bot temporarily deployed to secure-exec / secure-exec-pr-123 June 24, 2026 19:33 Destroyed
@railway-app
railway-app Bot temporarily deployed to rivet-frontend / secure-exec-pr-123 June 24, 2026 19:33 Destroyed
@NathanFlurry
NathanFlurry force-pushed the fix/queue-backpressure-and-tracker branch from 506fc46 to f7cc68e Compare June 24, 2026 19:50
@railway-app
railway-app Bot temporarily deployed to secure-exec / secure-exec-pr-123 June 24, 2026 19:50 Destroyed
@railway-app
railway-app Bot temporarily deployed to rivet-frontend / secure-exec-pr-123 June 24, 2026 19:50 Destroyed
@NathanFlurry
NathanFlurry force-pushed the fix/queue-backpressure-and-tracker branch from f7cc68e to 275a863 Compare June 25, 2026 03:11
@railway-app
railway-app Bot temporarily deployed to secure-exec / secure-exec-pr-123 June 25, 2026 03:11 Destroyed
@railway-app
railway-app Bot temporarily deployed to rivet-frontend / secure-exec-pr-123 June 25, 2026 03:11 Destroyed
@NathanFlurry
NathanFlurry force-pushed the fix/queue-backpressure-and-tracker branch from 275a863 to 9a65695 Compare June 25, 2026 07:52
@railway-app
railway-app Bot temporarily deployed to rivet-frontend / secure-exec-pr-123 June 25, 2026 07:52 Destroyed
@railway-app
railway-app Bot temporarily deployed to secure-exec / secure-exec-pr-123 June 25, 2026 07:52 Destroyed
@NathanFlurry
NathanFlurry force-pushed the fix/queue-backpressure-and-tracker branch from 9a65695 to a2e7648 Compare June 25, 2026 08:21
@railway-app
railway-app Bot temporarily deployed to secure-exec / secure-exec-pr-123 June 25, 2026 08:21 Destroyed
@railway-app
railway-app Bot temporarily deployed to rivet-frontend / secure-exec-pr-123 June 25, 2026 08:21 Destroyed
The V8->host event channel and the sidecar stdout frame queue both reacted
catastrophically when full instead of applying backpressure:

- send_javascript_event: a single try_send returning Full called
  v8_session.destroy() -> Shutdown -> Exited(1), tearing the session down and
  truncating the event stream.
- send_output_frame: try_send returning Full became a BrokenPipe error that
  propagated up and exited the whole sidecar process (code 1), taking every
  session with it.

Both fire only when the host consumer is briefly slow (e.g. a chatty
tool/skill turn outpacing a host that is momentarily not draining) — a
transient, recoverable condition that should never be fatal.

Changes:
- Event channel and stdout frame queue now block (clean backpressure) until
  the consumer drains. Both producers run where a blocking wait is safe (a
  dedicated OS thread / a current-thread runtime whose writer drains on an
  independent thread); a genuinely dead consumer still terminates via Closed/
  Disconnected rather than blocking forever. The oversized-single-event guard
  (a frame too large to ever send) still destroys, since it is unsendable.
- Defense-in-depth headroom: raise JAVASCRIPT_EVENT_CHANNEL_CAPACITY 64->512
  and MAX_STDOUT_FRAME_QUEUE 128->4096 so realistic bursts are absorbed before
  backpressure engages at all.
- New centralized queue tracker (secure_exec_bridge::queue_tracker): every
  bounded queue registers a QueueGauge in a process-global registry; depth and
  high-water are tracked, a single edge-triggered warn! fires at >=80% fill
  (re-arming below 50%) so a slow consumer is visible before it stalls a
  session, and queue_snapshot() exposes live usage for debugging. Wired the
  event channel, the stdin frame queue (sampled via Tokio channel capacity),
  and the stdout frame queue (TrackedSyncSender/Receiver).

Tests:
- bridge: queue_tracker unit tests (depth/high-water, edge-triggered warn with
  hysteresis, registry snapshot + auto-prune).
- execution: send_javascript_event now backpressures instead of destroying
  (unit) + an integration test (javascript_v8 suite) that bursts 1000 guest
  logs past the channel with a delayed host drain and asserts a clean exit with
  every line delivered (vs lines=capacity + torn-down session before the fix).
- sidecar: stdout frame queue accepts > capacity frames under a slow drainer
  and only errors when the writer is gone.

Defense-in-depth for the Zid integration's adapter-stall hang; the primary
fix is host-side (non-blocking buffered adapter stderr).

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Also documents the standard in CLAUDE.md (new "Limits, Bounds & Observability" section): bounded-by-default + config-wired/audited + registered-for-warn + clear-typed-error + no catastrophic reaction to transient fullness.

Also fixes two more instances of the same destroy/silent-loss anti-pattern:
- v8_host.rs v8_session_frames (1024): try_send Full -> destroy_session is now a
  blocking_send via tracked_sync_channel (backpressures the V8 runtime; only a
  dropped consumer is terminal). This per-session channel feeds the event channel,
  completing the end-to-end backpressure chain. (Note: a console.log repro can't
  fill it because the sync-RPC bridge serializes guest output one frame at a time;
  it shares the TrackedSyncSender mechanism the bridge unit tests cover.)
- service.rs completed_sidecar_responses (10k): silent FIFO eviction is now LOUD —
  warns on the 80% approach and on every dropped unretrieved response, so a host
  that stops fetching responses is diagnosable instead of silently losing them.

Also raise sidecar logging from ERROR-only to WARN (SECURE_EXEC_LOG-overridable)
and redirect the subscriber to stderr — without this the near-limit/backpressure
warnings added here were swallowed, and logging to stdout would corrupt the wire
protocol (stdout is the framed channel).

Enrich the tracker into a limits registry: LimitCategory (queue/resource/memory),
a structured LimitWarning + a process-global warning sink (set_limit_warning_handler)
that fires on the SAME edge-triggered/hysteresis-gated boundary as the log — the
foundation for a host onLimitWarning hook — and category in snapshots. register_limit()
lets saturating resources share the queue machinery.

Register the per-VM saturating kernel resource limits (processes, open fds, pipes,
ptys, sockets, connections, socket buffered bytes, datagram queue) with the limit
registry, sampled in ResourceAccountant::snapshot(). These had ZERO approach
warnings before — now they emit the same edge-triggered ~80% warning and appear in
the registry snapshot, alongside the bounded queues.

Forward limit-registry warnings to the host: the run_async event loop drains the
warning sink into a StructuredEvent (name="limit_warning", detail={limit,category,
observed,capacity,fillPercent}) emitted per active connection — reusing the existing
wire StructuredEvent variant (no protocol schema change). Backs the agentOS
onLimitWarning hook.

Add ~80% APPROACH warnings (edge-triggered) for the CPU/time budgets, complementing
the terminal exhaustion warnings: register a registry gauge sampled in the CPU-budget
watchdog poll (v8_cpu_time_ms), an 80% timer in the wall-clock guard (v8_wall_clock_ms),
and per-poll sampling of the WASM execution budget (wasm_fuel_ms). Memory (v8_heap_bytes)
stays terminal-only via V8's near-heap-limit callback — a V8 isolate cannot be safely
sampled from the watchdog thread. Docs updated to state approach-vs-terminal per category.

Address PR review findings:
- Emit each (process-global) limit_warning ONCE to a single active connection
  instead of fanning out a copy per connection (active_connections has no stale
  entries: the stdio transport only removes connections at teardown).
- Bound the send_request frame write by the caller's deadline (was a blocking
  send that ignored the timeout when stdout stalled); add TrackedSyncSender::try_send.
- Register gauges + ~80% approach warnings for the three remaining bounded
  VecDeques (pending_process_events, pending_sidecar_responses, outbound_sidecar_requests)
  so they meet the new standard; doc warning-names table updated.
- Disarm the limit_warning select! branch on a closed receiver (no busy-loop), never break.
- Observe the filesystem-bytes/inode gauges only on the success path (a rejected
  over-limit write no longer latches a spurious near-capacity warning).
- record_dequeue now re-evaluates so std-channel gauges re-arm as they drain (+ test).
- dispatch_warning clones the handler Arc out and drops the registry mutex before
  invoking it (avoid running the handler under the kernel fd_tables lock order).
- Fix stale '64-slot' comments (now 512) and pin the round-trip test to the real
  emitted limit name (javascript_event_channel).
@NathanFlurry
NathanFlurry force-pushed the fix/queue-backpressure-and-tracker branch from a2e7648 to b870388 Compare June 25, 2026 08:28
@railway-app
railway-app Bot temporarily deployed to secure-exec / secure-exec-pr-123 June 25, 2026 08:28 Destroyed
@railway-app
railway-app Bot temporarily deployed to rivet-frontend / secure-exec-pr-123 June 25, 2026 08:28 Destroyed
@NathanFlurry
NathanFlurry merged commit 0c92ac6 into main Jun 25, 2026
3 of 4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant