Skip to content

Investigate T3 performance with priority on fork additions and upstream compatibility #135

New issue
New issue
Open
Open
Investigate T3 performance with priority on fork additions and upstream compatibility#135
Labels
discoveryRequires investigation before implementation dispatchin-progressOwned active workspecA delivery or discovery outcome

Description

@pechhe
pechhe
opened on Oct 18, 2026

Goal and request

Henry wants T3 Code to remain fast during real daily use with multiple agents, large histories, many SlipJay workspaces, desktop/web clients and remote environments. Investigate why it has become progressively slower, then recommend and prioritize measured improvements. Give particular attention to features and integrations added in our fork, while keeping upstream T3 and the interaction between both in scope.

This issue is a report and investigation brief for Opus 5.5 and Fable. Henry will hand it to them; creating this brief does not launch either advisor or create a schedule now. The requested outcome includes establishing the single daily routine described below after the initial investigation and baseline. A broad rewrite still requires a justified proposal. Start with independent assessments of the same evidence, distinguish agreement from disagreement, and resolve disagreements with a decisive measurement.

Confirmed incidents and evidence

1. Released-workspace loop and identifier explosion — repaired

  • Stop released-workspace preparation loops and recover oversized identifiers #99: fork released-workspace preparation repeatedly reused an empty/released folder. Derived command/effect IDs recursively embedded their predecessors.
  • One chain produced identifiers hundreds of thousands of characters long. Events, receipts, outbox rows and indexes expanded the database to 74.22 GiB.
  • Fix: 54b2aa17c69c11cf0553a5f8d4523618aa699978. Recovered the database offline to 3.79 GiB, with SQLite integrity/foreign key checks and zero differences across all 18 application state tables. The fresh recovery preserved 34,023 messages. The untouched original DB/WAL/SHM were compressed and byte-verified before deleting the uncompressed backup.
  • On 18 October the live database was 6.1 GiB. A read-only check found zero new event IDs over 4096 bytes after recovery boundary sequence 773762. This rules against the old identifier explosion; it does not establish that all subsequent growth or database costs are healthy.

2. Unbounded landing-triggered VCS refresh — concrete fork defect

  • Before the fix, every relevant landing stage change swept every SlipJay checkout the VCS cache had ever held, including checkouts no client was viewing. Each refresh spawned several jj/Git processes. Historical cache size therefore amplified live work.
  • Fork introduction: 6d1274008e7 and 14c9c0cfa94, 15 October. Earlier Refresh only the checkouts a SlipJay landing change concerns #42 limited refreshes by repository, but did not bound them by current viewers.
  • Fix independently landed in 46c567bfd5fe6fcc6782e4be0aebaba97664652d, 18 October, 19:15 BST. It reference-counts live status streams, refreshes watched checkouts and evicts unused cached entries so their next read reloads fresh. It includes cases for an unwatched entry, a closed stream and remote-state reloading.
  • Installed build at incident time: 0.1.1-nightly.20261018.3315, source 3da5d2e273, so it did not contain this fix. The rebuilt app uses the same upstream version string: verify the source/build receipt and compiled fix, not just the version. See Keep T3 responsive during concurrent local agent and compiler work #134 for activation and post-install evidence.

3. Host saturation and request stalls — observed, not a complete root cause

At approximately 19:12 BST on 18 October:

  • 8 logical CPUs, 0% CPU idle, load averages around 170 / 117 / 76, ~105 runnable processes.
  • ~31 GiB RAM used, ~17 GiB compressed, ~3.6 GiB swap used.
  • T3 durable run state showed 8 running, 4 starting and 1 queued. Four compiler processes had 26, 20, 23 and 41 OS threads; not all threads were proven CPU-active. Compilers alone reached several GiB each.
  • T3 event-loop records showed 2.9–3.4 second stalls and utilization 1.0. Client-observed RPC spans reached ~152 seconds for dispatch and ~146 seconds for provider refresh. These are client request durations, not proof the server handler spent that entire time executing.
  • SQLite spans reached ~34 seconds, mostly annotated writer-lock wait, not query execution. Trace lock ownership and work held inside transactions rather than assuming the database simply needs vacuuming.
  • A shell snapshot query over 1,742 non-deleted threads reached ~11.6 seconds during the incident. The same SELECT against the stopped live database completed in ~1.35 seconds once load eased. Data counts then: 1,750 threads, 5,721 runs, 203,382 turn items, 45,862 messages. Its query plan used the expected per-thread indexes; no claim that it is optimally cheap.
  • Slow repository status calls reached ~61 seconds, ref listing ~108 seconds and editor discovery ~50 seconds. Long-lived subscriptions were excluded when identifying request latency.

Henry first closed/quitted T3, then saw its app icon still report background activity and right-clicked to kill it again. Some detached Claude-rooted compiler jobs survived the first stop. After the second stop, RAM use fell to 12–13 GiB, unused RAM rose to 18–20 GiB, compressed RAM fell to ~1.7 GiB, swap fell to ~1 GiB, runnable processes fell to ~12, and CPU idle reached ~28%. That shows T3-associated background work contributed to contention; it does not prove that ordinary agent concurrency alone was the cause or that T3 had no regression.

4. Related SlipJay runner investigation — evidence from another thread

Read Diagnose and Fix Slow Landings. It is a SlipJay investigation, not a second name for T3's VCS-cache defect.

The thread reports: a ledger query with an OR prevented indexed per-operation reads; ~150 operation reads per queue snapshot; each connected watcher rebuilding snapshots after events and 15-second heartbeats; ~20 watchers; snapshot cost ~1.2 seconds before the query change and ~95 ms after; payload reduction ~470 KB to ~259 KB; shared snapshot work intended to occur once per event rather than once per watcher. These are reported measurements, not re-run in this investigation. Verify their present release/runner activation before treating them as fixed. That thread was cancelled and contains unfinished follow-up work. #133 tracks queue/run timer accuracy; a long displayed queue timer is not itself evidence a check is stuck.

Investigation priorities

  1. Fork work amplification: SlipJay landing/watch integrations; shared versus per-client snapshots; polling, heartbeats, invalidation and cache eviction; work that scales with all historical workspaces instead of current activity. Measure server and runner separately.
  2. Database and history: writer-lock holders; read-only work using the writer; full projection reloads and JSON decoding; fork/lineage history expansion; replay/idempotency records and retention. Preserve event-sourced semantics and recovery evidence.
  3. Lifecycle: closing a window versus Cmd+Q versus killing an app; local/background server lifetime; detached provider process groups; shutdown deadlines; restart recovery and duplicate providers/checks. Determine what should survive before changing behavior.
  4. Interactive path: client/server time split, connection queues, cancellation, request acknowledgement and reconnect; renderer work for long chats and streamed tool output; desktop versus web behavior. A slow toast is a symptom, not a diagnosis.
  5. Resources: overlapping compiler/build/test work and thread counts; memory compression/swap; process and filesystem churn. Separate T3-owned overhead from intentional tool work and unrelated services. Do not impose an agent cap as the diagnosis or kill another thread's work.
  6. Observability cost: use existing traces, SQL lock-wait attribution, event-loop stall history and resource telemetry; verify that measurement itself remains bounded. Existing upstream idle telemetry and history optimizations must be checked before reinventing them.

Method and deliverables

  • Pin the running source/build, current integration source, latest merged upstream baseline, data shape and client/provider configuration. Re-read Keep T3 responsive during concurrent local agent and compiler work #134 because activation may have advanced after this brief was written.
  • Use disposable, isolated data with schedules, pending work, auth and live provider sessions removed. Never run a diagnostic server against Henry's installed database. Read-only inspection is allowed. Do not copy relay credentials or other secrets into fixtures.
  • Compare the fixed fork, the prior installed fork and an appropriate upstream baseline with equivalent safe data/workloads. If an upstream baseline cannot represent a fork feature or schema, say so and use a common-path fixture; do not call an invalid comparison a pass.
  • Measure idle, one active agent, several active agents, multiple clients, landing-stage bursts, long histories and repeated open/close/reconnect cycles. Record p50/p95/p99 interactive latency, event-loop delays, process spawn rate, CPU, memory/swap, SQL execution versus lock wait, payload volume and cost per watcher/event.
  • Produce a ranked finding list: observed/traced/suspected; fork/upstream/integration boundary; exact source path and revision; repro; user effect; smallest fix; behavior proof; expected benefit; compatibility and future merge cost. Include negative results and unresolved alternatives.
  • Prioritize bounded work and removal of duplicate work before tunable caps. Prove scaling with history size, watcher count and event rate, not just one fast fresh-start run. Recommend justified performance budgets and a repeatable regression benchmark.
  • After changes, verify the actual installed build and representative user actions under sustained activity. Separate changed, tested, landed, installed, active and owner-accepted claims. Do not close the performance investigation merely because restarting temporarily helps.

Daily performance, test and DevOps champion — required deliverable

Henry also requested a daily scheduled task that checks all SlipJay landings, notices test/runtime bloat, and continues improving performance. Include creation or configuration of that routine in the work; do not stop at recommending that someone schedule it.

  • First inventory existing schedules and the task in thread 237b6586-7fe9-43ec-828a-bfb6c1bdef19. Its temporary ten-minute runner check has a different purpose and must not be mistaken for the permanent daily champion. Reuse or extend an existing daily routine where appropriate; create exactly one canonical task, not one per advisor.
  • Run once daily in Europe/London. Use an existing agreed daily slot if present; otherwise default to 09:00 local time, preserving local time across BST/GMT changes. The routine runs unattended under the normal repository authority and delivery rules.
  • Inspect the last 24 hours of landings across configured SlipJay repositories, compared with a rolling seven-day baseline. Separate queue time from execution and stage time; cover install, typecheck, tests and publication, infrastructure failures, flaky checks, retries, actual check selection and increasing work/data volume. Include sample counts so a small or changed workload is not called a regression on timing alone.
  • Include T3 health where telemetry is available: installed source/build identity, interactive latency and event-loop stalls, process/watcher amplification, SQL execution versus lock wait, and DB/history growth. Keep Mac and VPS measurements separate. Continue prioritizing fork additions while considering upstream causes and compatible upstream fixes.
  • Use existing metrics and bounded queries. Do not re-run every suite, scan all journals/workspaces, start expensive profiles, or generate a new load problem on every daily run. Run focused checks only when needed to establish or verify a concrete improvement.
  • Persist a small trend/history record and give Henry a concise daily digest: what changed, observed regressions and evidence, fixes actually landed/installed, and anything blocked. Reuse the existing reporting and telemetry stores rather than introducing another monitoring system without a concrete need.
  • Investigate and deliver bounded, obvious performance/check-selection fixes through the owning repository's normal workflow; do not weaken behavioral coverage to make checks look faster. Record larger or uncertain changes as Issues/Specs with evidence and a smallest next step. Never cancel another thread's active work or silently change agent concurrency to make a measurement pass.
  • Keep one owner and prevent overlapping daily runs. The handoff must name the task ID, timezone/cadence, durable instruction/entrypoint and its current enabled state, and include one observed bounded run proving the task reads the intended evidence and produces its report. Distinguish configured, enabled and actually executed.

Fork versus upstream and maintenance constraint

Henry wants deep improvements without making future upstream updates prohibitively difficult. Prefer fixes in existing shared services and public extension points; keep provider/UI transports thin; avoid a parallel architecture or permanent compatibility layers. Check for upstream fixes and adopt compatible ones first. For each proposal explain whether it could be upstreamed, what overlap it creates with upstream files/contracts, and the expected rebase/test cost. A larger redesign may still be justified by measured benefit; justify it explicitly rather than excluding it or assuming it is necessary.

For orientation only: against the latest upstream merge's second parent 9c161462c94f7d9d6d9169da5c841457cbadf213, fork source 46c567bfd5fe differs in 782 files, +95,164 / -7,996 lines. This includes tests and other artifacts; line count is not a quality or performance metric. Recompute against the then-current upstream baseline.

Evidence locations and related work

This brief records evidence and asks for recommendations. It does not claim that every performance cause has been fixed or authorize disruption of other active work.

Activity

  1. pechhe
    added
    specA delivery or discovery outcome
    discoveryRequires investigation before implementation dispatch
    in-progressOwned active work
    on Oct 18, 2026

  2. pechhe commented on Oct 18, 2026

    @pechhe
    pechhe
    on Oct 18, 2026
    OwnerAuthor

    Activation update, 18 October 19:30 BST: Mac installer receipt confirms installation at 19:29:44 BST, and the running desktop/server started at 19:29:45–48 BST. The installed app archive contains the new watchedRef code; the staged directory is gone. VPS was already updated to 46c567b. A thread read completed in 1.23 s. Early post-restart spans show SQL/transactions up to ~1.6–1.9 s and VCS status up to ~7.4 s; no new stall-history entries were present since the earlier incident. Host CPU still sampled at 0% idle, although RAM use was 17–18 GiB with 14 GiB unused and ~1 GiB swap. The concrete checkout-cache bug is now deployed, but these early samples do NOT establish that all bloat is gone or that sustained performance is acceptable.

  3. pechhe
    added a commit that references this issue on Oct 18, 2026

    perf(server): a landing watch reconnect refreshes each watched checko…

    0b9b1fb

  4. pechhe commented on Oct 18, 2026

    @pechhe
    pechhe
    on Oct 18, 2026
    OwnerAuthor

    Findings, round 1 (18 Oct, 19:20–19:45 BST)

    Two independent assessments of the same evidence: Opus 5.5 (lead) and GPT-6 Astra (advisor; Henry asked for Astra instead of Fable). The source is pinned at t3peach b12e0997a18d. The upstream baseline is 9c161462c94f. The installed app (installed 19:23 BST, version string still 0.1.1-nightly.20261018.3315) contains the #134 fix 46c567bfd5fe, confirmed from its watchedRef code in app.asar. Evidence: server traces covering 19:21:55–19:32:02 BST (193k spans), event-loop-stalls.ndjson, and cheap read-only counts on the live DB. I started no diagnostic server and wrote nothing to ~/.t3/userdata.

    Data shape: 998k events (max sequence 1.15M), 29k receipts, 1,764 threads, 6.6 GB, 0 free pages.

    Agreement and disagreement

    • Both found: reads holding or waiting on the writer connection are the dominant SQLite cost; the settlement sweep revisits all history every minute; trace volume is very high; and the fork's rollover/experiment lookups run on every usage event.
    • Disagreement, now resolved: Astra counted the landing fan-out as fixed by 46c567bfd5fe. The trace shows a residual amplification: 283 serial checkout refreshes over 5.5 minutes after the 19:23 install. The cause is changes × watched checkouts (finding 1). It was fixed and landed in this round (0b9b1fb350fc).
    • Only Opus found: the upstream startup compaction full-history scan (finding 3).
    • Only Astra found: per-subscriber shell rebuilding (finding 5) and the usage-limit recovery scan (part of finding 2).

    Ranked findings

    # Finding Kind Boundary Effect / scaling Smallest fix Merge cost
    1 A landing-watch connect refreshes every watched checkout once per retained landing. SlipJayLandings reports every retained landing (≤200) on connect; VcsStatusBroadcaster refreshed each watched SlipJay checkout of the repo per change, serially. Observed: 283 refreshes, 245 s of jj/git, 19:23:47–19:29:16. Sidebar rows watch their checkout while visible, so watched ≈ visible workspace rows. Observed Fork (6d1274008e7, 14c9c0cfa94; bounded by 46c567bfd5fe) Grows as landings × visible workspace rows on every app start or watch reconnect; also on overlapping stage changes Landed 0b9b1fb350fc (19:52 BST, ticket review clean): buffered changes are taken as one batch (Stream.runForEachArray), one refresh per checkout per pass. Not yet installed None (fork-only file)
    2 Reads queue on the writer connection. In 10 min: 57k writer statements, 229 s, of which 148 s was writer-lock wait. Mostly plain SELECTs: node/thread/subagent lookups, scheduled_tasks polling (18.5 s, all wait), usage-limit recovery candidates (3.2 s span, 26 ms of work). Astra traced readCanonicalProjection decoding inside writer transactions (895 ms hold) and settlement candidate decoding inside one (1.68 s hold). Traced Upstream (nodeSqliteClient defaults to writer; de343914273e) Every background read delays commands; worse with history and event rate Route non-mutating reads to the existing reader; decode after releasing the transaction Medium: central persistence; upstreamable
    3 Startup compaction scans the whole event history on the writer. ProjectionMaintenance.compactEventStore starts 1 min after each launch (then every 6 h) and pages ~1M events 50 at a time with json_extract: 37,818 queries, 71 s, 19:23:41–19:31:40. Each app restart (frequent with fork installs) repeats it. Observed Upstream Linear in total events; competes with every write for ~8 min after launch Persist a compaction high-water mark so later passes scan only new events; run discovery on the reader Low–medium; upstreamable
    4 Trace volume erases the evidence it exists to keep. ~10 MB/min (~19k spans/min). 10×10 MB retention holds ~10 minutes. 55% of bytes are sqlite spans; 67% of spans are under 1 ms (e.g. settleThread: 8 µs of work per span, 1.4k/min). Serialization is synchronous. Observed (latency effect suspected) Upstream config; fork adds span sources Grows with background work; incidents become unrecoverable Drop or aggregate successful fast SQL/helper spans; keep failures, slow spans and lock waits Low; upstreamable
    5 Shell refresh rebuilds per subscriber. ws.ts builds shells inside each subscription; skipUnchangedThreadShells runs after the DB work. 2,210 projectShellItems spans, p95 202 ms (Astra). Traced mechanism; duplication not yet measured Upstream (de343914273e, c138163c8cc3) Grows as clients × events × history Share shell computation per thread revision Medium–high (replay ordering)
    6 The settlement sweep visits every thread every minute. ~1,440 settleThread + ~700 pullRequestFor per sweep, 0.1–4.6 s per sweep. Observed Upstream (fork diff empty) Linear in non-deleted threads Schedule by change or deadline instead of a full rescan; reader-route candidates Medium; upstreamable
    7 Rollover reacts to every usage report with run reload and experiment resolution. ContextRolloverReactor.onUsage ran 638×/10 min, 32 s total, max 3.6 s; it loads all the thread's runs and resolves experiment assignments (the OR predicate is a full table scan per Astra's EXPLAIN; only 70 rows today). Traced Fork (84f77f9f5181, 16034c70f300) Grows with usage rate × runs per thread Cache the active run/assignment per attempt; use direct predicates Low (fork-only)
    8 Renderer ~26–48% CPU and GPU ~20–30% during streaming. Not attributed. Suspected Unknown Battery/GPU; may compete with agents Profile one streaming session in a renderer (needs browser permission) n/a

    Negative results

    • Identifier explosion (Stop released-workspace preparation loops and recover oversized identifiers #99): not recurring (as reported in Keep T3 responsive during concurrent local agent and compiler work #134).
    • No event-loop stall over 2 s since 19:13 BST, including during the post-install refresh burst and compaction. Current costs are lock contention and throughput, not blocking.
    • Incremental vacuum (DatabaseStorage.reclaim, 1,425 one-page steps per 10 min) is upstream, deliberately yields, and cost ~4 s. Not a priority.
    • AgentAwarenessRelay.publishThread (545 per 10 min, network-bound) is upstream and has no CPU hotspot.
    • Claude transcript tailing is bounded (512 KiB, 10 s throttle). Remote VCS pollers are reference-counted. Subscription spans measure lifetime, not latency (Astra).

    Not yet done (keeps this investigation open)

    • An isolated replay benchmark that varies history size, subscriber count and event rate, recording writer hold vs wait, SQL per event, RPC p50/p95/p99, and spawn rate. Compare fixed fork, prior install and upstream. This is the decisive measurement for findings 2, 5 and 6.
    • Lifecycle (Cmd+Q vs kill, detached compiler process groups) and the client/server time split: not investigated in this round.
    • After installing finding 1, confirm one refresh per watched checkout on the next reconnect in the installed app's trace.

  5. pechhe commented on Oct 18, 2026

    @pechhe
    pechhe
    on Oct 18, 2026
    OwnerAuthor

    Follow-ups filed from the findings above: #137 (reads on the writer, finding 2), #138 (compaction full rescan, finding 3), #139 (trace retention, finding 4), #140 (rollover per-usage reload, fork, finding 7). Findings 5, 6 and 8, plus the isolated replay benchmark, lifecycle and client/server split, stay open here. Next decisive step: the isolated replay benchmark varying history, subscriber count and event rate.

  6. pechhe
    removed
    in-progressOwned active work
    on Oct 18, 2026

  7. pechhe commented on Oct 18, 2026

    @pechhe
    pechhe
    on Oct 18, 2026
    OwnerAuthor

    Round 2 follow-ups landed in a4bc496cd85d (t3peach), and #137–#140 are closed:

    • Server trace retains only ~10 minutes under normal load #139 trace retention: successful spans under 1 ms are dropped unless they waited on a lock for at least 5 ms or logged a warning. Spans record parentSpanName. The production default keeps 60 trace files, and the diagnostics RPC reads at most 10. Traces should now cover hours instead of about 10 minutes.
    • Route non-mutating projection reads to the SQLite reader #137 reads off the writer: background reads (recovery, settlement, duplicate-node and subagent lookups, the scheduled-task poll) and client thread snapshots now use the reader. Canonical and settlement payloads are decoded after the transaction ends. Command decisions stay on the writer on purpose. Live traces show reader transactions normally take about 140 ms, but one PR-sync shell snapshot held the reader for 13.8 s, and commands must not queue behind that. The isolated benchmark was not run; once the build is installed, compare writer db.lock_wait_ms in the retained traces instead.
    • Event-store compaction rescans all history after every launch #138 compaction: discovery reads from the reader. Completion is persisted, so a restart no longer triggers a rescan within 6 h, and a failed pass waits a full interval before retrying. Partly met: a pass still scans all history.
    • Context rollover reloads runs and experiment assignments on every usage report #140 rollover: caches the active run per attempt, re-enrols only when its inputs change, and assignment queries use index-backed predicates.

    Finding 8 (renderer), static analysis. I couldn't profile it live, because the integrated Browser panel was unavailable and Phi rejects evaluate. The likely costs:

    • Fork "live" response streaming (8fb3bdb71f74): it runs a requestAnimationFrame reveal loop plus a 300 ms .chat-stream-height height transition, which forces layout every frame. It also re-renders the whole message's markdown for each sentence. Per-word fade only applies to the last 2 blocks.
    • DotMatrixSpinner: it animates 25 dots at 16 fps.
    • Indicator animations: these are already stepped and paused when not visible upstream.

    The measured installed renderer averaged about 24% CPU and the GPU about 17%, in bursts. The decisive test is an A/B of Settings → response streaming "live" vs "paragraph" while one thread streams, sampling renderer and GPU %CPU.

    Still open here: shared shell computation per subscriber (finding 5), settlement sweep scheduling (finding 6), an isolated replay benchmark, lifecycle, and the client/server split.

  8. pechhe
    added
    in-progressOwned active work
    on Oct 18, 2026

  9. pechhe commented on Oct 19, 2026

    @pechhe
    pechhe
    on Oct 19, 2026
    OwnerAuthor

    Landed da2b87f: perf(server): serve overlapping reads from a pool of SQLite readers. After #137 moved client thread snapshots onto the single reader, opening a thread waited a median 4.9 s (max 7.5 s) for it, behind PR sync, settlement sweeps and archived shell snapshots that hold it for 2–15 s. Clients showed "Syncing messages..." meanwhile. Reads now lease the lowest free of up to four reader workers, which start only when reads overlap, and every connection wait is interruptible. Two review rounds: round 1 found a non-cancellable wait, which is fixed and has a regression test. Not yet installed: the fork build is blocked on the upstream v0.1.1-nightly.20261019.3348 merge (33 conflicted files). After it's installed, compare db.lock_wait_ms on reader sql.transaction spans under ws.orchestrationV2.threadSnapshotThenLive.

    The background reads themselves are still slow: ThreadPullRequestServiceV2.synchronize runs 1.4–1.8 s queries, and archived shell snapshots take about 5 s of statements. That is findings 5 and 6.

  10. pechhe commented on Oct 19, 2026

    @pechhe
    pechhe
    on Oct 19, 2026
    OwnerAuthor

    Status 2026-10-19 14:10 BST. The reader-pool fix (da2b87f) is installed in 0.1.1-nightly.20261019.3348. Traces since 12:13 BST show thread-open reads wait under 1 ms for the database (up to about 18 s before). About 1 in 20 thread loads still take 1–5 s, all in query execution, because the Mac is overloaded (load average around 85, disk 94% full). The main cause is SlipJay spare provisioning cloning node_modules, tracked in pechhe/slipjay#124, with background deletion in #123; both are being worked on in the Workspace pool cleanup thread. #112 is landed.

    Still open here: findings 5 and 6 (PR-sync synchronize queries take 1.4–1.8 s and archived shell snapshots about 5 s), finding 8 (renderer, static analysis only), the isolated replay benchmark, lifecycle and the client/server split, and confirming the daily champion routine against the deliverable above.

  11. pechhe
    added a commit that references this issue on Oct 19, 2026

    perf(server): pull request discovery reads thread records, not shells

    228fcfe

  12. pechhe commented on Oct 19, 2026

    @pechhe
    pechhe
    on Oct 19, 2026
    OwnerAuthor

    Landed 228fcfe: perf(server): pull request discovery reads thread records, not shells. Every minute, ThreadPullRequestServiceV2.synchronize built full sidebar shells for every unsettled thread, 752 on the Mac. Since the 12:13 BST install, 149 shell reads under it took p50 7 ms, p95 662 ms and max 3.7 s. Single-thread refreshes also read on the writer, with up to 3.6 s of lock wait. Now one reader transaction reads only the unarchived thread records that have a branch or a detected pull request: 8 threads in about 20 ms on the same data, timed read-only. The dispatch guard is unchanged. Review was clean on both axes; tests 60/60, server typecheck clean, landing gate passed. Not yet installed: it ships with the next fork build.

    Correction to my 14:10 BST comment: these background reads are not the table's findings 5 and 6. Finding 5 (shells rebuilt per subscriber) and finding 6 (the settlement sweep visiting every thread each minute) are still open. The archived-thread read I mentioned is user-triggered (9 calls in 6 h). Its 7–18 s cases all predate the reader-pool install, and the one call since then took 0.7 s, so the reader pool resolved it and it needs no change.

    Daily champion: there is no T3 performance routine yet. pechhe/slipjay#107 is building the single daily test-and-dev-ops champion (first run tracked in slipjay#120). I've asked that thread to add a bounded T3 runtime-health section to it (installed build, thread-open and shell latency, writer lock wait, top statements, event-loop stalls, DB growth, Mac and VPS kept separate), so there is one canonical task rather than two.

  13. pechhe commented on Oct 19, 2026

    @pechhe
    pechhe
    on Oct 19, 2026
    OwnerAuthor

    The daily champion now covers T3 runtime health. It is the one canonical daily performance routine for this Issue; no second task was created.

    • Scheduled task: scheduled-task:command:mcp:d97f42b5-feec-479b-b23f-48b7e2b65543:schedule-task:sl-107-champion-schedule-1 ("Daily test champion"). It runs daily at 06:00 Europe/London in a fresh thread; the first run with this section is 06:00 BST on 20 Oct.
    • Skill: skills/routines/test-champion/SKILL.md §4 "T3 runtime health", with collector t3-health.ts, landed in agent-config 9c71c6f5. It reads only: bun:sqlite opened read-only with query_only, and it never starts a server against ~/.t3/userdata.
    • Per host, Mac and peach-box separately, over the last 24 h: the builds from fork-nightly.sh status (selected, staged, active-mac, active-vps); p50/p95 of threadSnapshotThenLive and projectShellItems; total db.lock_wait_ms on writer transactions; the top 5 sqlite statements by total time, each with the span that issued it (resolved through its sql.transaction); event-loop stalls over 1 s; and statev2.sqlite size with thread (orchestration_v2_projection_threads) and event (orchestration_events) counts.
    • Trend file: ~/.local/state/test-champion/t3-health.ndjson on the Mac, keeping 30 days. A doubling against the host's previous record, or a new top-5 statement, is commented here.

    Baseline recorded 19 Oct, 14:40 BST:

    Mac peach-box
    threadSnapshotThenLive p50/p95 11.1 s / 40.5 s (16) no samples
    projectShellItems p50/p95 2.8 / 64.5 ms (36,661) 1.6 / 2.8 ms (218)
    writer lock wait, total 818 s 0.2 s
    stalls over 1 s 91 (max 11.6 s) 1 (2.2 s)
    statev2.sqlite 7.97 GB, 1,994 threads, 1.19 M events 0.51 GB, 276 threads, 77 k events

    The Mac's top statements by time: projection_nodes payload reads under ServerRuntimeStartup.startEffectWorkerWithRelay (1,259 s), projection_subagents reads under ProviderEventIngestor.mirrorSubagentTokens (837 s), ProjectionStore.getLimitRecoveryCandidates (598 s), ThreadPullRequestServiceV2.synchronize (595 s) and ScheduledTaskService.listDueTasks (577 s). The Mac's 24-hour window includes the hours before the reader-pool install, so tomorrow's run is the first clean comparison.

  14. pechhe
    added 3 commits that reference this issue on Oct 19, 2026

    perf(server): settlement sweep skips threads whose timer is not due

    da9f9ff

    test(server): memory fixture provides the turn-start services it now …

    a9c39ae

    docs(fork): record measured renderer cost of live response streaming

    b3a3a73
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    discoveryRequires investigation before implementation dispatchin-progressOwned active workspecA delivery or discovery outcome

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions