Skip to content

Instantly share code, notes, and snippets.

@zhumengzhu
Created April 15, 2026 13:26
Show Gist options
  • Select an option

  • Save zhumengzhu/091076a04491762659c3cd210d02cee7 to your computer and use it in GitHub Desktop.

Select an option

Save zhumengzhu/091076a04491762659c3cd210d02cee7 to your computer and use it in GitHub Desktop.
opencode SSE memory leak: local reproduction attempt and signal timing test (PR #22552)

Local Reproduction Attempt: SSE CLOSE_WAIT Memory Leak

Supporting material for PR #22552.

Short version: Two reproduction methods were attempted on an unpatched dev build. Neither produced memory growth — not because the bug doesn't exist, but because the test server accidentally had Bun's default idle timeout active. The production code explicitly sets idleTimeout: 0, which disables that safety net entirely. That single line is the root cause of unbounded memory growth in production.


Test environment

  • OS: Linux 6.17.0-20-generic (Ubuntu 24.04)
  • Bun: 1.3.11
  • Hono: 4.10.7 (packages/opencode/node_modules/hono)
  • nginx: 1.24.0
  • Server branch: dev at commit 7cbe1627e (unpatched)

Method 1: iptables FIN drop

Idea: Drop the client's FIN packet at the kernel level so the server never learns the client closed. Expected to produce CLOSE_WAIT on the server side.

sudo iptables -A INPUT -p tcp --dport 19876 --tcp-flags FIN,ACK FIN,ACK -j DROP
python3 bad-client.py 20 19876   # connect → HTTP GET /event → SHUT_WR → hold 60s

Result: Server-side connections showed as ESTAB, not CLOSE_WAIT. Gone within ~30s. Memory flat.

Why it failed: An unacknowledged FIN triggers exponential TCP retransmits. On loopback, the retransmit timeout is ~30s, after which the kernel sends RST to force-close. Bun receives the RST and fires onAbort — cleanup runs, no leak. CLOSE_WAIT never forms because the server eventually gets a hard reset, not silence.


Method 2: nginx proxy_ignore_client_abort on

Idea: This directive tells nginx to keep the upstream connection open after the downstream client disconnects — the application-layer pipe closes while the TCP connection stays alive. This is the closest local approximation of the Electron shell scenario described in #22198.

nginx config (proxying 19877 → 19876):

server {
    listen 19877;
    location /event {
        proxy_pass                http://127.0.0.1:19876;
        proxy_http_version        1.1;
        proxy_buffering           off;
        proxy_cache               off;
        proxy_ignore_client_abort on;
        proxy_read_timeout        3600s;
        proxy_set_header          Connection "";
    }
}

Procedure: Start unpatched server → open 20 SSE connections via nginx → kill all curl clients → sample VmRSS and upstream connection count every 10s for 120s.

Results:

Time Server RSS Upstream connections (nginx→bun)
Baseline 436,128 kB —
Before disconnect — 40 (20 connections × 2 directions)
t+10s 436,496 kB 40 — still alive
t+20s 436,856 kB 0 — all gone
t+30s–t+120s ~436,000 kB 0

Net RSS change: −252 kB. No leak.

This was unexpected. Connections disappeared at t+20s and memory did not grow. The next section explains why.


Why cleanup ran: Bun's idle timeout

Reading Hono's source (node_modules/hono/dist/) and instrumenting a timing test server (see signal-timing-test.md) revealed the full picture.

Hono source: two relevant facts

Fact 1 — req.raw.signal listener is version-gated (hono/dist/helper/streaming/sse.js):

var isOldBunVersion = () => {
  const v = Bun.version;   // "1.3.11" in this environment
  return v.startsWith("1.1") || v.startsWith("1.0") || v.startsWith("0.");
  // → false: the listener below is NOT registered on Bun 1.3.11
};

if (isOldBunVersion()) {
  c.req.raw.signal.addEventListener("abort", () => {
    if (!stream.closed) stream.abort();
  });
}

This gate was introduced in Hono PR #3859 by the Bun author, on the assumption that Bun 1.2+ reliably calls ReadableStream.cancel() on disconnect. On Bun 1.3.11 this listener is not registered.

Fact 2 — StreamingApi.write() silently swallows all errors (hono/dist/utils/stream.js):

async write(input) {
  try {
    await this.writer.write(input);
  } catch {
    // empty — writeSSE never throws regardless of connection state
  }
}

The actual cleanup trigger is responseReadable.cancel():

this.responseReadable = new ReadableStream({
  cancel: () => {
    this.abort();  // → fires all onAbort subscribers including stop()
  }
});

When the downstream consumer (nginx) stops reading, cancel() fires → StreamingApi.abort() → stream.onAbort(stop) → cleanup.

The actual trigger: Bun's built-in idle timeout

Timing logs from the instrumented server (full methodology in signal-timing-test.md):

[+0ms]     connected
[+10002ms] heartbeat push
[Bun.serve]: request timed out after 10 seconds. Pass `idleTimeout` to configure.
[+20003ms] >>> req.raw.signal ABORT fired  (onAbort fired: false)
[+20003ms] >>> stream.onAbort fired        (signal fired: true)
[+20003ms] cleanup done

Bun has a 10-second idle timeout by default. When no data is written for 10s, Bun aborts Request.signal. The t+20s timing in the nginx test follows directly:

t=0s:   connection established, heartbeat timer starts, idle timer starts (10s)
t=2s:   kill curl → nginx enters CLOSE_WAIT (nginx←curl)
        nginx→bun upstream stays ESTAB (proxy_ignore_client_abort on)
t=10s:  heartbeat fires → writeSSE → data enters nginx TCP buffer
        nginx has no downstream reader but the buffer accepts the write
        → Bun's idle timer resets
t=20s:  10 more seconds of silence
        → Bun idle timeout fires → Request.signal aborted
        → req.raw.signal "abort" fires  (PR #22552 listener)
        → responseReadable.cancel() → StreamingApi.abort()
        → stream.onAbort(stop) fires
        → cleanup complete

The first heartbeat write at t+10s resets the idle timer, pushing cleanup to t+20s. Without a heartbeat, cleanup would have fired at t+12s (2s connect + 10s idle).

On a direct connection (no nginx): killing curl immediately aborts Request.signal (~0ms). Both req.raw.signal and stream.onAbort fire within 2ms, with req.raw.signal first.


Root cause: idleTimeout: 0 in production

The local tests appeared clean only because the test server ran Bun.serve without any idleTimeout setting — leaving Bun's default 10s active. The production code does not have this backstop. From packages/opencode/src/server/adapter.bun.ts:

const args = {
  fetch: app.fetch,
  hostname: opts.hostname,
  idleTimeout: 0,   // disables idle timeout entirely
  websocket: ws.websocket,
} as const

idleTimeout: 0 disables Bun's idle timeout entirely. With it set to zero, the complete production leak chain is:

idleTimeout: 0
  → Bun never aborts Request.signal due to inactivity
  → heartbeat writes every 10s "succeed" (data enters proxy TCP buffer)
  → Bun sees the connection as alive, no cleanup signal fires
  → responseReadable.cancel() never called → stream.onAbort never fires
  → req.raw.signal never aborts
  → stop() never executes
  → setInterval keeps ticking, Bus subscriptions stay alive
  → AsyncQueue grows without bound (14 MB/s, peak 24.5 GB)

Assessment of PR #22552's fix layers

With the full picture:

Layer With idleTimeout: 0 (production) With idleTimeout: 120 (fix)
idleTimeout: 0 → 120 — Root fix. Active connections reset the timer on every 10s heartbeat and are never killed. Dead connections are cleaned up within 120s of the proxy buffer filling
req.raw.signal explicit listener Request.signal never aborts — does not fire Fires before stream.onAbort on direct connections (~2ms earlier); independent path when responseReadable.cancel() chain breaks
writeSSE try/catch Hono write() has empty catch — never fires Same — ineffective on Hono 4.x. Kept for forward compatibility if a future Hono version propagates write errors
AsyncQueue capacity limit Always effective — independent of all signals Always effective
setMaxListeners(100) Fixes #22422 independently Fixes #22422 independently

Signal Timing Test: req.raw.signal vs stream.onAbort

This test instruments a minimal SSE server to measure exactly when req.raw.signal and stream.onAbort fire relative to client disconnect — both with a direct connection and through an nginx reverse proxy with proxy_ignore_client_abort on.

Run this to reproduce the findings in local-reproduction-attempt.md.


Prerequisites

  • Bun installed
  • nginx installed and running
  • opencode repo checked out (for Hono dependency resolution)
  • nginx test config deployed (see below)

Step 1: Deploy nginx config for test server

Create /etc/nginx/sites-available/sse-signal-test:

server {
    listen 19879;
    location /event {
        proxy_pass                http://127.0.0.1:19878;
        proxy_http_version        1.1;
        proxy_buffering           off;
        proxy_cache               off;
        proxy_ignore_client_abort on;
        proxy_read_timeout        3600s;
        proxy_set_header          Connection "";
    }
}
sudo cp sse-signal-test.nginx.conf /etc/nginx/sites-available/sse-signal-test
sudo ln -sf /etc/nginx/sites-available/sse-signal-test /etc/nginx/sites-enabled/sse-signal-test
sudo nginx -t && sudo nginx -s reload

Step 2: Test server source

Save as test-signal-timing.ts inside packages/opencode/ (required for Hono module resolution):

/**
 * Minimal SSE server that logs when req.raw.signal and stream.onAbort fire,
 * and which fires first. Run from packages/opencode/ directory.
 *
 * Ports:
 *   19878 — Bun server (direct)
 *   19879 — nginx proxy → 19878 (proxy_ignore_client_abort on)
 */
import { Hono } from "hono"
import { streamSSE } from "hono/streaming"

const app = new Hono()

app.get("/event", async (c) => {
  c.header("Cache-Control", "no-cache")
  c.header("X-Accel-Buffering", "no")

  return streamSSE(c, async (stream) => {
    const t0 = Date.now()
    const ts = () => `+${Date.now() - t0}ms`

    console.log(`[${ts()}] connected`)

    let signalFired = false
    let onAbortFired = false

    c.req.raw.signal.addEventListener("abort", () => {
      signalFired = true
      console.log(`[${ts()}] >>> req.raw.signal ABORT  (onAbort fired before: ${onAbortFired})`)
    })

    stream.onAbort(() => {
      onAbortFired = true
      console.log(`[${ts()}] >>> stream.onAbort        (signal fired before: ${signalFired})`)
    })

    const heartbeat = setInterval(() => {
      console.log(`[${ts()}] heartbeat push`)
      stream.writeSSE({ data: JSON.stringify({ type: "heartbeat" }) })
    }, 10_000)

    // Wait until either signal fires
    await new Promise<void>((resolve) => {
      c.req.raw.signal.addEventListener("abort", resolve)
      stream.onAbort(resolve)
    })

    clearInterval(heartbeat)
    console.log(`[${ts()}] cleanup done`)
  })
})

console.log("Listening on :19878  (nginx proxy on :19879)")
Bun.serve({ fetch: app.fetch, port: 19878 })

Step 3: Run the tests

cd packages/opencode
bun run test-signal-timing.ts

In a second terminal:

Test A: Direct connection

# Connect directly to Bun (no nginx)
curl -sN --no-buffer http://127.0.0.1:19878/event > /dev/null &
CURL_PID=$!
sleep 2
kill $CURL_PID

Expected output:

[+0ms]    connected
[+~2000ms] >>> req.raw.signal ABORT  (onAbort fired before: false)
[+~2002ms] >>> stream.onAbort        (signal fired before: true)
[+~2002ms] cleanup done

req.raw.signal fires first, stream.onAbort follows ~2ms later. Both fire immediately after curl is killed (~0ms delay from kill to signal).

Test B: Via nginx (proxy_ignore_client_abort on)

# Connect via nginx
curl -sN --no-buffer http://127.0.0.1:19879/event > /dev/null &
CURL_PID=$!
sleep 2      # let connection stabilize
kill $CURL_PID
# wait ~20s for cleanup

Expected output:

[+0ms]     connected
[+10002ms] heartbeat push
[Bun.serve]: request timed out after 10 seconds. Pass `idleTimeout` to configure.
[+20003ms] >>> req.raw.signal ABORT  (onAbort fired before: false)
[+20003ms] >>> stream.onAbort        (signal fired before: true)
[+20003ms] cleanup done

req.raw.signal fires first again, stream.onAbort ~0ms later. Both fire at connect+20s.


What the timing reveals

Direct connection

Client FIN → Bun receives it immediately → Request.signal aborts → cleanup runs in ~0ms.

Via nginx with proxy_ignore_client_abort on

t=0:    connection established, heartbeat timer starts
t=2s:   curl killed → nginx CLOSE_WAIT (nginx←curl)
        nginx→bun stays ESTAB (proxy_ignore_client_abort on)
t=10s:  heartbeat fires → writeSSE → data written to nginx TCP buffer
        nginx has no downstream reader but buffer accepts the write
        → Bun's idle timer RESET (data was written)
t=20s:  10 more seconds of silence
        → Bun idleTimeout fires → Request.signal aborted
        → req.raw.signal "abort" event fires
        → responseReadable.cancel() → StreamingApi.abort()
        → stream.onAbort fires
        → cleanup runs

Key insight: the first heartbeat write (t=10s) resets Bun's idle timer, delaying cleanup by one full heartbeat interval. If connections had no heartbeat, cleanup would happen at t=12s (2s connect + 10s idle). With a 10s heartbeat, cleanup happens at t=20s.

The writeSSE try/catch situation

Hono 4.10.7's StreamingApi.write() has an empty catch block — writeSSE never throws even when the connection is dead. The writeSSE try/catch added in PR #22552 cannot fire on Hono 4.10.7. The timing logs confirm this: in the test environment (default idleTimeout: 10s), cleanup is triggered by idleTimeout → Request.signal, not by a write exception.

Note: this test server does not set idleTimeout: 0. The production opencode server does (adapter.bun.ts), which means the idleTimeout-based cleanup seen here does not occur in production — making the AsyncQueue capacity limit the only unconditionally effective protection.

Why req.raw.signal still matters

Hono only registers req.raw.signal for isOldBunVersion() (Bun 1.0/1.1); on Bun 1.2+ this listener is absent (see Hono PR #3859). PR #22552 registers it unconditionally, which:

  1. On direct connections: fires ~2ms before stream.onAbort — a faster, more direct path
  2. On nginx proxy: fires at the same instant as stream.onAbort but via an independent code path — if responseReadable.cancel() chain is broken, the signal path still runs

Cleanup

# Remove test file
rm packages/opencode/test-signal-timing.ts

# Remove nginx config
sudo rm /etc/nginx/sites-enabled/sse-signal-test
sudo rm /etc/nginx/sites-available/sse-signal-test
sudo nginx -s reload
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment