[Bug]: Delegated-task threads append each turn-item update dozens of times as identical rows (up to 74x per hour, 16 GB in two days) #36438

Claude (claude-opus-5-5), assisting the reporter:

Before submitting

• I searched existing issues and did not find a duplicate.
• I included enough detail to reproduce or investigate the problem.

Area

apps/server

Steps to reproduce

1. Run many concurrent delegate_task workers (we had 25 to 60 delegated-task threads active per hour, Codex and Claude, all on one server).
2. On a copy of statev2.sqlite, measure how many rows each distinct turn-item payload occupies, split by delegated vs ordinary threads:

WITH e AS (
  SELECT substr(occurred_at, 1, 13) AS hour,
         stream_id LIKE '%delegated-task%' AS delegated,
         json_extract(payload_json, '$.id') || '|' || ifnull(json_extract(payload_json, '$.status'), '')
           || '|' || ifnull(json_extract(payload_json, '$.updatedAt'), '') || '|' || length(payload_json) AS k
  FROM orchestration_events
  WHERE event_type = 'turn-item.updated'
)
SELECT hour, delegated, count(*) AS rows, count(DISTINCT k) AS distinct_payloads,
       round(1.0 * count(*) / count(DISTINCT k), 1) AS copies_per_payload
FROM e GROUP BY hour, delegated HAVING rows > 500 ORDER BY hour, delegated;

Expected behavior

Each provider item update is appended to orchestration_events once.

Actual behavior

In delegated-task threads, the same turn-item.updated payload is appended many times, byte for byte identical, each copy with its own fresh event_id. Ordinary (non-delegated) threads in the same hours are unaffected (exactly 1.0 copies per payload).

Copies per distinct payload, per hour (UTC), delegated-task threads:

Ordinary threads, every hour: 1.0.

One delegated Codex run, traced in full: every item in the run (assistant messages and 25 command executions) has exactly 2 distinct payloads (running, then completed), and each payload is stored 131 to 133 times. For one command item, all 266 rows were written within 1.5 seconds (22:55:53.243Z to 22:55:54.689Z); 133 identical running rows are interleaved with 133 identical completed rows, so running copies keep landing after the completed one. metadata_json is identical across the copies; event_id differs (event:thread:<thread>:provider-session:provider-session%3Aprovider-instance%3Acodex%3Ashared:<uuid>), causation_event_id and correlation_id are null. So each copy is a separate append, not a retried write.

Effect on one macOS install: about 9.5 GiB of event payload was written in under two days, taking statev2.sqlite to 16 GB. turn-item.updated held 7.9 GiB for 993k rows across about 57k items whose final payloads total about 0.6 GiB. Command executions alone: 581k rows and 6.4 GiB for 23k items (393 MB final). Because the background prune keeps a day of history (RETAINED_EVENT_HISTORY), none of it was removable for a day, and t3 doctor reported "0 superseded events removed". We pruned offline with the same rules as pruneEventStore (15.8 GiB to 2.8 GiB; projection row counts unchanged, quick_check ok).

We have not found the cause. The duplication comes in bursts, during periods of heavy delegated fan-out, but it is not simply proportional to the number of active delegated threads (Nov 13 02:00 had 59 active with 4.7 copies; Nov 12 22:00 had 61 with 73.7).

Impact

Major degradation or frequent failure

Version or commit

0.1.9 (background service)

Environment

macOS 27.0 (arm64). Codex (shared app-server provider instance) and Claude Agent providers. Workload is an orchestrator thread that fans out delegate_task workers continuously.

Logs or stack traces

The server log on this machine also has a burst of SQLite database or disk is full errors (and a ServerOwnershipError on the following restart) early on 2026-11-13, consistent with the database growth filling the disk. We have not confirmed the link.

Workaround

Stop the service and delete superseded events offline with the pruneEventStore rules (keep thread.created, run.updated with status running, each stream's newest row, and promptSequence-referenced rows), then VACUUM INTO a new file and swap it in.

Related

• [Bug]: Pi tool output updates store a full snapshot row each (154 rows / 7 MB for one 15s command) #36425: full-snapshot rows for Pi tool output (a different mechanism: distinct, growing snapshots from one provider).
• [Bug]: Paragraph streaming stores every full reasoning snapshot as a permanent event (OpenCode: 93k rows, 2.2 GB in one day) #23365 / fix(server): a streamed reply or reasoning block no longer stores every snapshot of its text #36362: streaming snapshot rows for assistant messages and reasoning.
• statev2.sqlite grows without bound (33 GB in 7 weeks): retention/compaction for orchestration events and turn items #28903: retention and compaction for orchestration_events.