Repository navigation
Deferred-steer check and queued-message cancel read a long thread's whole history under its dispatch lock #36095
Description
Two paths still read a long thread's whole history while they hold the thread's dispatch lock, so sends and other commands to that thread wait. #29521 narrowed the steer and Stop reads from #29468, but not these two.
Measured in production over 8.6 hours on 0.1.10-nightly.20261112.4606; our local patches leave both paths unchanged, and the code is the same on main at 5f69cac5c9. Both threads are coordinators:
- thread A: 7,511 runs, 4,577 subagents, about 45,000 nodes;
- thread B: 6,939 runs, 2,535 subagents, about 40,000 nodes.
1. Deferred steers: a full runs read on every running provider-turn update
deliverDeferredSteers handles every provider-turn.updated event with status running. Inside threadDispatch.withLock, it reads all runs of the thread to find queued runs whose steerWhenReadyRunId is the running run. The Claude adapter emits such an event with each token-usage update, so this happens several times per turn, and it almost never finds a steer to deliver.
| Thread A | Thread B | |
|---|---|---|
| Provider turns started | 152 | 50 |
Runs reads by deliverDeferredSteers |
1,400 | 481 |
| Per read, median | 1.10 s | 1.08 s |
| Lock time, total | 26 min | 8.8 min |
That is about 10 s of lock time per turn on thread A, and each read decodes about 8.7 MB of run JSON.
A possible fix: runStatuses: ["queued", "running"] on that read, which seeks orchestration_v2_projection_runs_thread_status_idx. On a copy of the database, with the sqlite3 CLI, that query returns 2 rows in under 50 ms; the full read returns 7,352 rows in 2.2 s.
2. Cancelling a queued message reads the whole thread
dispatchQueuedRunCancel loads all runs, user messages, nodes, attempts and subagents to remove one queued run; rahulshinde11 noted this in #29468. #29521 gave the steer and Stop reads omitSettledLeafNodes, but not this one. In the same 8.6 hours, five cancels on the two threads each read for about 15 s:
| Read | Rows | Median |
|---|---|---|
| Nodes | 40,000–44,000 | 8.0 s |
| User messages (two reads) | 7,100–7,800 | 2.8 + 1.3 s |
| Attempts | 6,900–7,400 | 1.1 s |
| Runs | 6,900–7,400 | 1.0 s |
| Subagents | 2,500–4,500 | 0.9 s |
An agent that clears handled messages from its own queue with t3_queue cancel pays this for each message.
Times are statement execution times from a local statement log that records every read over 250 ms, so the counts are complete for these threads.
Measured and written by Claude (Opus 5.5) in T3 Code; posted with the owner's approval.
Activity