|
| 1 | +# GET /event O(N×M) fan-out: verdict + minimal-fix design |
| 2 | + |
| 3 | +**Date:** 2026-06-23 |
| 4 | +**Author:** swarm worker 2 (qjk4 territory) |
| 5 | +**Round constraint:** DESIGN/VERIFY ONLY — no implementation, no patch edits. |
| 6 | +**Scope:** the `/event` fan-out half of `workstation-qjk4` (M3.1 filter-before-queue, |
| 7 | +M3.3 setMaxListeners). The ProjectCopy half (M3.2) is owned by worker 1. |
| 8 | + |
| 9 | +--- |
| 10 | + |
| 11 | +## Verdict: RESIDUAL GAP (structurally defused, not eliminated) |
| 12 | + |
| 13 | +The catastrophic single-core O(N×M) event storm from |
| 14 | +`OPENCODE-SERVE-MULTICORE-INVESTIGATION.md §1.A` is **structurally defused** by the |
| 15 | +K-serve pool, but the two handler-level fixes qjk4 scoped (M3.1 filter-before-queue, |
| 16 | +M3.3 setMaxListeners) were **NOT implemented**. They remain real (if smaller and |
| 17 | +currently-dormant) gaps. |
| 18 | + |
| 19 | +### Evidence |
| 20 | + |
| 21 | +**The deployed `/event` handler still offers every event to every client's queue.** |
| 22 | +After `event-session-scope.patch` (#7) and `event-cold-start-directory.patch` (#11), |
| 23 | +the listener line is byte-for-byte the base v1.17.7 line: |
| 24 | + |
| 25 | +```ts |
| 26 | +const unsubscribe = yield* events.listen((event) => Effect.sync(() => Queue.offerUnsafe(queue, event))) |
| 27 | +``` |
| 28 | + |
| 29 | +Both the directory filter AND the `?session_ids=` filter run **post-queue**, inside |
| 30 | +`Stream.fromQueue(queue).pipe(Stream.filter(...))`. The cold-start patch merely *merged* |
| 31 | +the directory filter into the same post-queue `Stream.filter` and added session-aggregate |
| 32 | +gating — it did not move anything ahead of the queue offer. So per published event, every |
| 33 | +connected client still: |
| 34 | +1. fires its `events.listen` callback (EventV2 keeps listeners in a plain array, iterated |
| 35 | + for all N on every publish — `core/src/event.ts:421`), and |
| 36 | +2. allocates a `Queue.offerUnsafe`, |
| 37 | +…and only *then* is the event filtered before SSE serialization. The doc's §5 Step 1 |
| 38 | +("filter **before** offering to the queue", reduce to O(M_local)) was **not done**. |
| 39 | + |
| 40 | +**`?session_ids=` is DEPLOYED-BUT-UNUSED by real clients.** The TUI SSE path |
| 41 | +(`tui/src/context/sdk.tsx`, via `attach-route-resolve.patch`) and the bash clients |
| 42 | +(`oc-auto-attach`, `opencode-launch`) all use pigeon `GET /route` only to extract |
| 43 | +`.apiBase` and *attach to the owning serve*; the actual subscription is a bare |
| 44 | +`sdk.global.event({ signal, sseMaxRetryAttempts: 0 })` with **no** `session_ids` param. |
| 45 | +The only `?session_ids=` occurrences in 1.1M log lines are manual `probe`/grep commands, |
| 46 | +never TUI traffic. Session isolation is therefore achieved **structurally** (each serve is |
| 47 | +a separate process whose in-memory GlobalBus only carries its own sessions' events), not by |
| 48 | +the filter. The filter is dead weight on the hot path today — and being post-queue, it |
| 49 | +wouldn't kill the fan-out even if clients did use it. |
| 50 | + |
| 51 | +**`setMaxListeners` is never raised.** Zero `setMaxListeners` references in base v1.17.7 or |
| 52 | +in any patch; `bus/global.ts` is unmodified (default limit = 10). Each `/event` client |
| 53 | +registers one `GlobalBus.on("event")` listener for `server.instance.disposed` detection |
| 54 | +(`event.ts` `disposed` sub-stream), so a single serve with ≥11 concurrent clients emits |
| 55 | +`MaxListenersExceededWarning`. Live evidence: **0 occurrences in 1.1M log lines** → on the |
| 56 | +current 4-serve pool (ports 4096–4099, confirmed running + pigeon) no single serve has hit |
| 57 | +11 concurrent clients. The warning is **dormant, not fixed** — a serve owning 11+ |
| 58 | +sessions/TUIs would still emit it. |
| 59 | + |
| 60 | +### Why the storm is defused (the structural win, for the record) |
| 61 | + |
| 62 | +| | Pre-pool (the §1.A scenario) | Deployed K=4 pool | |
| 63 | +|---|---|---| |
| 64 | +| Processes / event loops | 1 | 4 (separate GlobalBus each) | |
| 65 | +| N (clients per bus) | all TUIs (11–20+) | only that serve's owners (single digits) | |
| 66 | +| M (events per bus) | events from all active sessions | only that serve's sessions' events | |
| 67 | +| Fan-out work | O(N×M) on one core | ~O((N/K)×(M/K)) spread over K cores | |
| 68 | + |
| 69 | +Worst-case fan-out drops ~K² *and* moves off the single hot core. That is the real reason |
| 70 | +the production warning/CPU-spin is gone — `serve-lease.patch` (HRW session sharding) + |
| 71 | +`attach-route-resolve.patch` (TUI → owning serve), **not** the `?session_ids=` filter. |
| 72 | + |
| 73 | +--- |
| 74 | + |
| 75 | +## Minimal fix design (the two still-open qjk4 items) |
| 76 | + |
| 77 | +Both are small, low-risk, and independent. Recommend shipping M3.3 unconditionally and |
| 78 | +M3.1 as a cheap defense-in-depth for "hot serve" cases. |
| 79 | + |
| 80 | +### M3.3 — `GlobalBus.setMaxListeners` (trivial, unambiguous) |
| 81 | + |
| 82 | +`packages/opencode/src/bus/global.ts`, one line after construction: |
| 83 | + |
| 84 | +```ts |
| 85 | +export const GlobalBus = new GlobalBusEmitter() |
| 86 | +GlobalBus.setMaxListeners(0) // 0 = unlimited; or 100 per the investigation doc |
| 87 | +``` |
| 88 | + |
| 89 | +Rationale: the disposed-listener-per-client pattern is legitimate fan-out, not a leak. `0` |
| 90 | +(unlimited) is preferable to `100` because a busy multi-session serve has a genuinely |
| 91 | +unbounded client count and we never want this false-positive to mask a *real* leak warning |
| 92 | +elsewhere. Matches doc §5 Step 2. New patch file (e.g. `globalbus-maxlisteners.patch`), |
| 93 | +applied any time after base; touches only `global.ts`. |
| 94 | + |
| 95 | +### M3.1 — move the existing predicate ahead of the queue offer (yl00-safe) |
| 96 | + |
| 97 | +The key correctness constraint: do **not** re-introduce the yl00 cold-start directory race. |
| 98 | +That race is already handled by the cold-start predicate (session-scoped subs gate on |
| 99 | +session-aggregate membership, directory-independent). The fix is simply to evaluate that |
| 100 | +**same** predicate in the `listen` callback instead of in a downstream `Stream.filter`: |
| 101 | + |
| 102 | +```ts |
| 103 | +const shouldForward = (event: EventV2.Payload) => { |
| 104 | + const sid = (event.data as Record<string, unknown> | undefined)?.["sessionID"] |
| 105 | + if (sessionIds !== undefined && typeof sid === "string") return sessionIds.has(sid) |
| 106 | + return matchesDirectory(event) |
| 107 | +} |
| 108 | +const unsubscribe = yield* events.listen((event) => |
| 109 | + shouldForward(event) ? Effect.sync(() => Queue.offerUnsafe(queue, event)) : Effect.void, |
| 110 | +) |
| 111 | +// ...the downstream Stream.filter becomes redundant; drop it (or keep as defense-in-depth). |
| 112 | +``` |
| 113 | + |
| 114 | +Why this is safe and sufficient: |
| 115 | +- It is the **identical predicate** already in `event-cold-start-directory.patch`, just |
| 116 | + evaluated earlier — so yl00 semantics are preserved exactly (session subs stay |
| 117 | + directory-independent; non-session subs keep directory scoping they always had). |
| 118 | +- Non-matching events no longer allocate a queue offer or wake the client's stream fiber → |
| 119 | + per-serve fan-out becomes O(M_local) instead of O(N_serve × M_serve). This matters for a |
| 120 | + *hot serve* (e.g. a swarm coordinator + workers colocated on one serve via HRW). |
| 121 | +- The `disposed` GlobalBus sub-stream is a separate path and is untouched. |
| 122 | + |
| 123 | +This would land as an edit to `event-cold-start-directory.patch` (it owns the final shape of |
| 124 | +that block) **or** a new follow-on patch that must apply after it. No client change required. |
| 125 | + |
| 126 | +--- |
| 127 | + |
| 128 | +## Open decision for the coordinator / human |
| 129 | + |
| 130 | +M3.3 is unambiguous, ~free, and currently-dormant-but-unmitigated → recommend ship. |
| 131 | + |
| 132 | +M3.1 is the judgment call: the pool already defuses the *catastrophic* case, so the |
| 133 | +remaining win is bounded (hot-serve only) and the change, while small, edits a patch that |
| 134 | +two beads (qjk4 + yl00) already depend on. Decision to escalate: **ship M3.1 now as cheap |
| 135 | +insurance, or close qjk4-M3.1 as "obviated by the pool" and keep only M3.3?** I lean |
| 136 | +ship-both (both are low-risk and qjk4 is still OPEN scoping all three), but the |
| 137 | +"obviated by pool" close is defensible given zero production warnings today. |
| 138 | + |
| 139 | +No implementation performed this round. |
0 commit comments