perf: issue #108 backend cleanup — alias-collision warn, SSE idle backoff, public-URL cache (items 3+4+5)#113
Conversation
Backend-only cleanup bundle for GitHub issue #108 (items 1+2 are architectural/frontend and out of scope). TDD throughout; behaviour is preserved at every call site — these are cost/observability improvements, not policy changes. Item 3 — tool_aliases collision policy (WARN, never reject): canonicalize_tool_name folds dotted→wire lossily, so two distinct spellings can collide with no detection. Add detect_alias_collisions() (canonical → the >=2 distinct source spellings) + warn_alias_collisions() to corlinman_agent.tool_aliases, and call it where the two security gates build their allow-sets: the skill allowed-tools gate (agent_servicer._skill_allowed_tools_block) and the subagent allowlist escalation gate (subagent/runner._filter_tools_for_child). Each distinct collision logs ONE structured warning (log key tool_aliases.collision, deduped per gate for the process lifetime) — a hard reject could silently break a working dotted allowlist on a security path, so first-registered still wins and allow/deny outcomes are unchanged. Item 4 — /admin/subagents overview SSE idle backoff: the live-overview stream ran store.list_all()+merge per client on a fixed 2s tick forever. Extract the pure _next_poll_interval(changed, current) helper (base 2.0s, ×1.5 backoff, 10.0s cap) and drive the loop with it: an unchanged tick backs the interval off toward the cap; any change snaps straight back to 2s so liveness is never traded away. Keepalive now rides wall-clock elapsed time so idle proxies keep the same ~10s byte rhythm. Item 5 — public-URL disk-read caching: _resolve_public_base_url runs once per media-bearing tool result and, with CORLINMAN_PUBLIC_URL unset, re-read BOTH backing files every call. Add a module-level cache keyed on a cheap stat() signature (mtime_ns, size) of the py-config drop and the learned public_origin file: a hit skips the two reads; a mtime/size change or either file appearing/disappearing invalidates; a transient stat() error yields a never-equal signature so we re-read. The env short-circuit is unchanged and never touches the cache. Tests (all TDD, red→green): - test_tool_aliases.py: detector finds {a.b, a_b}, dedupes identical spellings, no false positive on the shipped tool set; warn logs one structured event, deduped; runner gate warns without changing outcome. - test_fix_LANE_A_servicer.py: skill gate logs the collision warning without changing allow/deny. - test_subagents.py: _next_poll_interval grows to the cap when unchanged, resets to base on change. - test_public_url_cache.py: cache hit avoids re-read (counters), mtime bump invalidates, appear/disappear invalidate, env still wins and short-circuits before any disk read.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: cce089aad5
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| await asyncio.sleep(_poll) | ||
| ticks += 1 | ||
| if ticks * _poll >= _SUBAGENT_SSE_HEARTBEAT_SECONDS: | ||
| since_keepalive += _poll | ||
| if since_keepalive >= _SUBAGENT_SSE_HEARTBEAT_SECONDS: | ||
| yield b": keepalive\n\n" | ||
| ticks = 0 | ||
| since_keepalive = 0.0 |
There was a problem hiding this comment.
Keep heartbeats on the 10s cadence during backoff
When this SSE stream is idle, _poll grows 2 → 3 → 4.5 → 6.75 before it reaches the 10s cap. Because the heartbeat check only runs after sleeping the entire current interval and then resets since_keepalive to zero, the first idle keepalive after the initial snapshot is delayed to about 16.25s instead of _SUBAGENT_SSE_HEARTBEAT_SECONDS (10s). In deployments where proxies or clients rely on the documented 10s SSE heartbeat, an otherwise idle /admin/subagents/events/live connection can be closed; clamp the next sleep to the remaining heartbeat time or emit a keepalive before taking a long backoff sleep.
Useful? React with 👍 / 👎.
… (Codex #113) The idle backoff let since_keepalive accumulate a full backed-off interval before the heartbeat check ran — the first idle keepalive after a growth sequence (2→3→4.5→6.75) landed at ~16.25s, past the documented ~10s SSE rhythm proxies rely on. The poll sleep is now split into heartbeat-bounded slices via a pure _sleep_slices helper: keepalives fire exactly at rhythm crossings mid-sleep while the scan cadence still honours the full poll interval (idle cost unchanged). 5 new unit tests (no-crossing, boundary cut, multi-crossing, entry-at- rhythm, cadence-preservation sweep). 23 passed; ruff+mypy clean.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 91cd580543
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| await asyncio.sleep(slice_s) | ||
| since_keepalive += slice_s |
There was a problem hiding this comment.
Measure SSE heartbeat from real elapsed time
Even with the new slicer, this keepalive clock is advanced only by the requested sleep duration; the fresh evidence is that time spent in the subsequent await _combined() scan, JSON serialization, or blocked yields is not counted. In the exact high-load/slow-store case this backoff is meant to help, an unchanged scan that takes a few seconds lets the first idle : keepalive arrive well after the documented 10s cadence, so strict proxies can still drop /admin/subagents/events/live; track a time.monotonic() last-byte timestamp instead of accumulating scheduled sleeps.
Useful? React with 👍 / 👎.
| for slice_s, emit_keepalive in _sleep_slices( | ||
| _poll, since_keepalive, _SUBAGENT_SSE_HEARTBEAT_SECONDS |
There was a problem hiding this comment.
Preserve 2s detection for the first change after idle
For /admin/subagents/events/live, once an unchanged panel has backed _poll off toward 10s, a new subagent or activity update that occurs just after this sleep starts cannot be observed until the whole backed-off interval finishes and _combined() runs. The reset to the 2s cadence happens only after that late scan, so the first change after an idle period regresses from the previous ~2s visibility to as much as ~10s; use an event wakeup or keep the scan sleep bounded by the liveness target.
Useful? React with 👍 / 👎.
…dex #113 round 2) Address 2 confirmed Codex #113 round-2 findings on the subagents overview SSE loop (stream_subagents_overview), replacing the _sleep_slices machinery with one cohesive three-clock redesign: (a) 3520699516 — the keepalive clock accumulated only REQUESTED sleep, so time in _combined()/serialization/blocked yields went uncounted and the first idle keepalive could land past the 10s rhythm under load. The loop now measures real elapsed time: last_byte = time.monotonic() is updated after every yield (data frames AND keepalives) and a keepalive fires whenever now - last_byte >= HEARTBEAT. (b) 3520699518 — after backing off toward 10s, the first change following an idle period was not observed until the whole backed-off sleep elapsed (detection regressed ~2s → up to 10s). A new cheap O(1) probe, _overview_change_signature(store, registry) = (registry.revision, store _persist_path st_mtime_ns), is sampled at the base cadence; a diff from the last full scan's signature forces an immediate scan (+ backoff reset via _next_poll_interval), so a change surfaces within ~2s again. LiveSubagentRegistry gains a monotonically increasing `revision` int bumped on every genuine row create/update/remove (_apply_spawned/_apply_child_event/ _apply_completed/_prune_terminal); the expensive list_all()+merge full scan now runs only on the backed-off next_full_scan deadline or a detected change. Tests: revision increments per mutation + is stable on no-op observes; _overview_change_signature tracks store mtime + registry revision and tolerates a missing/absent path (None component). _next_poll_interval backoff tests kept; the 5 _sleep_slices tests deleted with the helper.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b495e128a4
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| if now >= next_full_scan or sig != last_scan_sig: | ||
| seen_ids: set[str] = set() | ||
| changed = False | ||
| for row in await _combined(): |
There was a problem hiding this comment.
Send keepalive before long overview scans
When the overview stream is idle and next_full_scan wins just before the 10s heartbeat deadline, the loop enters await _combined() before checking the heartbeat again; if store.list_all()/merge/serialization takes more than the remaining heartbeat budget, the connection can still go silent for >10s and strict proxies may close /admin/subagents/events/live. Fresh evidence after the patch is that the new monotonic clock is checked only before this full scan, so the scan itself can still overrun the documented heartbeat cadence; emit a keepalive before starting a scan when the heartbeat is nearly due or split the scan so heartbeats can interleave.
Useful? React with 👍 / 👎.
The monotonic-clock loop checks the heartbeat only before a full scan, so a _combined()+merge that outruns the remaining heartbeat budget could still silence /admin/subagents/events/live past the 10s rhythm and a strict proxy would drop it. Now, once silence passes halfway to the deadline, a keepalive is emitted BEFORE the scan (last_byte resets), so post-scan silence is bounded by the scan duration alone. Decision extracted as the pure _prescan_keepalive_due helper + 2 unit tests. 23 passed; ruff+mypy clean.
…parse/subagent allowlist) Each is a DIFFERENT code path than the earlier same-class fixes: 1. shutdown() now reaps each running task's whole group via the shared _reap — a daemonized child that outlived the wrapper would otherwise keep the pipe open and block the awaited pump indefinitely. 2. The shell_task_output dispatcher's own int(offset) parse catches OverflowError too — a non-finite JSON offset (1e10000 → inf) raises there BEFORE registry.read()'s seek, so the round-3 seek-side catch didn't cover it; now degrades to offset 0 instead of builtin_tool_failed. 3. The subagent runner's _filter_tools_for_child applies the same run_shell⇒shell_task_output/kill implication the skill gate got in r4 (only for task tools the parent actually holds, so the child never exceeds its ceiling) — a subagent narrowed to run_shell can now poll and terminate its bg tasks. Also merges origin/main (both #113's alias-collision-warn and this PR's run_shell skill implication live in _skill_allowed_tools_block — kept both). 4 new tests. 220 passed; ruff+mypy clean.
…ell + #108 backend) Closes claude-code parity Wave 4, merged via #111 (Dim 2), #112 (Dim 4), #113 (#108 items 3/4/5). - Dim 2: summary-LLM compaction cooldown + failure/low-savings breaker. - Dim 4: run_shell run_in_background + shell_task_output/shell_task_kill, with session-ownership isolation, 64KiB paged has_more reads, unified process-group _reap across all 6 terminal paths (incl. daemonized grandchildren + atexit), lifecycle watchdog, evicted-log cleanup, a task-control permission model that tracks the start-grant (allow/log/ask across scoped/wildcard-deny/default-deny/strict/plan) confined by the ownership gate, non-bool/overflow never-raise hardening, and subagent safety (bg refused + controls stripped unless explicitly granted). - #108 items 3/4/5: alias-collision warn, adaptive three-clock overview SSE with pre-scan keepalive, public-URL mtime cache.
Summary
The backend-only bundle from issue #108's deferral list (items 3, 4, 5 — verified all still live at HEAD by adversarial-checked triage). Items 1 (live-registry process-wide feed, architectural) and 2 (in-progress user bubble, ui/) deliberately remain open.
Orchestrated build: Opus triage (13 claims, 0 refuted) → Opus implementation in an isolated worktree → independently re-validated before push.
What shipped
detect_alias_collisions+warn_alias_collisionsintool_aliases.py(structlog keytool_aliases.collision, process-lifetime dedup). Wired at both security gates — skill allowed-tools and subagent allowlist escalation — as a pure side-effect: allow/deny outcomes unchanged (verified in review), first-registered-wins documented. Policy rationale: no live collision exists today (latent), and a hard reject could break working dotted allowlists at security choke points._next_poll_interval(changed, current)helper — 2.0s base, ×1.5 on unchanged ticks up to 10.0s cap, instant reset to 2.0s on any change (row change OR stale-id reap), so liveness on change is preserved while N idle chat tabs no longer each drive a 2s fulllist_all()+merge scan. Keepalive switched to wall-clock accounting._resolve_public_base_urlnow caches on a(mtime_ns, size)signature of the py-config drop + learned-origin file. EnvCORLINMAN_PUBLIC_URLshort-circuit unchanged; mtime change/appear/disappear invalidates; stat errors force re-read. Eliminates up to 2 disk reads per media-bearing tool result on env-unset deploys.Testing
Per-item TDD: collision detector + no-false-positive-on-shipped-set + gates-warn-without-outcome-change (caplog); interval helper growth/cap/reset; cache hit/mtime-invalidate/env-wins. 84 passed across the touched suites; agent+server mypy + ruff clean — independently re-verified.
Refs #108 (items 3, 4, 5 — do not close; items 1, 2 remain).