Repository navigation
[Bug]: A copied state database wakes the original install's agents and can touch its worktrees #15513
Description
Note
🤖 Claude Opus 5.5 on behalf of Oliver
Before submitting
- I searched existing issues and did not find a duplicate.
- I included enough detail to reproduce or investigate the problem.
Area
apps/server
Summary
A copy of statev2.sqlite booted by a second server, such as a dev server seeded from real data, inherits the original install's automatic work. It does not check which environment that work belongs to. Startup recovery and the background workers then wake the original install's agents and can act on its real worktrees.
The environment ID lives in a file under T3 home (ServerEnvironment.ts), not in the database. The copy therefore can't tell that it isn't the install that wrote the data.
This is the problem behind #12879, which was closed as stale. A fresh fix on current main turned out to need a design decision, so I'm opening this to agree on direction first, per CONTRIBUTING.md.
Steps to reproduce
- Run a normal install with agents active, including at least one delegated subagent mid-run.
- Snapshot the live DB:
VACUUM INTOfrom~/.t3/userdata/statev2.sqliteinto a fresh home'suserdata/. - Optionally set every running run in the copy to
interrupted(column and JSON). - Boot a server on that home, e.g.
vp run devin a worktree.
Expected behavior
A database booted by a different environment does not start provider turns, deliver messages to threads, or touch files on behalf of the environment that wrote it. Inherited automatic work stays suspended until someone explicitly adopts it.
Actual behavior
This happened on 2026-10-04. Two disposable dev servers were seeded from a live-DB copy to capture PR screenshots, with every running run set to interrupted and thread continuation turned off. On startup:
- Startup recovery delivered "subagent interrupted" notifications to the parent orchestrator thread in the copy.
- The resumed parent, running with full permissions, re-steered duplicate copies of its subagents.
- For roughly 100 seconds, those duplicates ran real commands in sibling
/tmpworktrees. One rangit checkout HEAD -- …and reverted another agent's staged files. Another ranrmon another agent's screenshots. One edited an evidence script. - Separately, four resumed
claudesessions ran for about 19 minutes in real checkouts of other projects, and left uncommitted edits in one of them.
Nothing was pushed. The only defence was a human noticing.
Paths that wake inherited work today
Each of these acts on a copied database with no ownership check:
| Path | How it fires |
|---|---|
| Restart continuation | ProviderRuntimeRecoveryService → restartContinuationRun / continueRestartedRun (RestartContinuation.ts) posts "Continue where you left off." into interrupted runs. |
| Delegated completion delivery | #15323's delegation recovery records interrupted or cancelled child results and offers a parent wake through the terminal-run listener (Orchestrator.ts). This is the path that fired in the incident. It ignores continueThreadsAfterServerUpdate, which defaults to off. |
| Checkpoint-waiting children | A child run waiting on a pending checkpoint capture is kept by recovery. When CheckpointCaptureService completes it after activation, the parent is woken. |
| PR watches | PullRequestWatchReactor reads persisted watches and turns check, comment and conflict changes into message.dispatch. |
| Scheduled tasks | ScheduledTaskService selects due tasks by enabled state and time only. It launches or sends using the stored project and workspace binding, whose absolute paths can point into the original checkout. |
| Usage-limit auto-resume | Persisted limitRecovery.autoResume is enough for UsageLimitRecoveryWorker to post "Continue where you left off." at reset time, regardless of the copy's settings. |
| Pending rollbacks | provider-thread.rollback is replayable. CheckpointRollbackService reopens the stored provider session and restores checkpoint files. On the same machine, those are the original worktree's files. CheckpointRestoreSafety only checks threads inside the copied DB. |
Held queued runs are already safe: promotion is rejected while any held run exists.
Proposed direction
1. Record which environment owns the database. Add one owner row, orchestration_v2_runtime_owner, in a new migration (057 is currently free). Write it after startup recovery completes.
2. Quarantine inherited automatic work instead of enumerating recovery paths. A one-boot cleanup pass isn't enough: later checkpoint completions, watches, schedules and limit resets can recreate wakes after the owner row has been overwritten. Instead, mark inherited automation as suspended on a known ownership mismatch. Enforce that where it takes effect:
- automatic message admission (continuations, delegated deliveries, PR-watch wakes, scheduled sends, usage-limit resumes)
- external-effect execution (rollback and restore)
Suspended work keeps its configuration and data. It isn't deleted.
3. Make adoption explicit and visible. Show suspended work and let the user adopt it ("this is my install") or dismiss it. Never silently drop it.
4. Distinguish three ownership states so existing installs aren't punished on upgrade:
- Matching owner: behave as today.
- Known mismatch: suspend all inherited automation.
- Legacy, no owner recorded (every install's first boot after this ships): the database can't prove where it came from. One proposed heuristic is to record whether the home's
environment-idfile already existed before this boot. A nonempty database plus a freshly generated identity looks like a copy into a new home, and gets suspended. A pre-existing identity looks like a normal upgrade, and gets adopted. Copying an unowned database into an already-initialized home remains ambiguous without an independent database-to-home binding.
Questions for maintainers
- Is a persistent quarantine of inherited automation acceptable, or should ownership live on each record instead? Per-record ownership would mean
environmentIdon runs, schedules, watches and limit recovery. On the old fix(v2): resume interrupted runs only for the environment that started them #12879, putting it on the clientOrchestrationV2Runcontract was questioned. - Should scheduled tasks and usage-limit resume be included, given that suspending them on a mismatch changes product behaviour?
- What should a legacy, unowned database do on first boot: adopt on the identity-file heuristic, suspend and prompt, or something else?
- Where should suspended work surface in the UI?
Impact
- Correctness and safety: a copy can start provider turns against real repos, revert files, and restore checkpoints in the original worktree.
- Who hits it: anyone who copies a DB for testing, support or migration. That includes maintainers and agents following AGENTS.md's "copy real data" advice, and dev servers seeded from real data. AGENTS.md currently recommends
VACUUM INTOsnapshots of the live DB for test data. Until this is fixed, that advice is unsafe while agents are running.
Version or commit
main at c5bc98d (2026-10-04).
Prior work
- fix(v2): resume interrupted runs only for the environment that started them #12879: original PR, closed as stale.
- A rebuild on current main exists as three branches on my fork, all reviewed:
fix/restart-owner-marker: the owner row and migration 057 only.fix/restart-owner-continuations: cancels continuations on foreign or unowned databases.fix/restart-owner-deliveries: stops delegated-completion deliveries.
- An adversarial review found the checkpoint, PR-watch, schedule, usage-limit and rollback paths above still open. It concluded that per-path cleanup is the wrong shape, hence this issue.
- The owner-row branch is reusable as groundwork whatever direction is chosen.
Workaround
Never boot a copy of a live database while agents are running. Use an empty T3 home and create fixtures through the UI.
Activity
juliusmarminge commentedon Oct 4, 2026
Note
Grok responding on behalf of Julius.
Triage
Thanks for the careful incident write-up and for opening this to agree on direction first, @flamboh. I could confirm on current
mainthat astatev2.sqlitesnapshot booted under a different T3 home can start provider turns and write to the source install's worktrees. The environment id lives only in the file atServerConfig.environmentIdPath(ServerEnvironmentIdentityinapps/server/src/environment/ServerEnvironment.ts), and nothing in the database records which environment wrote it.056_RemoveRedundantProjectionIndexesis the latest migration, so057is free.What I found
The incident path. This one ignores
continueThreadsAfterServerUpdate(defaultfalseinpackages/contracts/src/settings.ts).interruptedmakes those children show up inProjectionStore.getRecoveryThreadIds("subagent-results")whenever the latest run is terminal and there's nosubagent_resulttransfer.Orchestrator.recoverDelegatedTasksthen callsfinalizeAppOwnedSubagent, which offers a parent wake.ProviderContinuationServicedispatches it asmessage.dispatchwithqueue_after_active, and since an interrupted parent has no active run, the wake starts a provider turn on the stored native session and absolute worktree path.continueRestartedRundoes re-check the setting and won't post "Continue where you left off." when it's off, but its finalizer still callsrecoverDelegatedTask. TheDelegatedCompletionDelivery"continuation declines to start" test currently locks that in.The other automatic paths you listed are real too, and none consult an environment owner:
waitingrun with a pending or runningcheckpoint.captureeffect is kept byProviderRuntimeRecoveryService. That effect is replay-safe (EffectOutbox.REPLAY_SAFE_EFFECT_TYPES_AFTER_PROCESS_LOSS), and when capture completes,handleTerminalRuncan finalize the child and wake the parent.provider-thread.rollbackreplays the same way.CheckpointRestoreSafetyonly checks other threads in the same database, so a copied thread can restore files at the originalcwd.PullRequestWatchReactorsweeps persisted watches and can dispatch a wake on any unsettled thread.ScheduledTaskService.runDueTasksfires from enabled state andnext_run_at, using the stored project and workspace binding.limitRecoveryCommandposts "Continue where you left off." from a persistedlimitRecovery.autoResumeeven when the booting server'sautoResumeLimitedThreadsis off.queueHeld, andstartNextQueuedRunwon't promote while a held run exists.On AGENTS.md: the documented seed is
vp run migrate-dev-dbwith the dev server stopped, not a rawVACUUM INTOinto a fresh home. That command snapshots withVACUUM INTO, then drops the families recovery would touch (runtime,queued-runs,subagent-results,delegated-completions, a latestfailedrun or anylimitRecovery) and deletesscheduled_tasksandorchestration_v2_effect_outbox. The incident used a raw copy, which skips that prune. The prune isn't a full ownership check either: an active PR watch on a stopped, unsettled thread survives it and can still wake.On #12879: it was closed because the branch went stale, not because the problem was rejected. The earlier review note about not adding
environmentIdto the clientOrchestrationV2Runcontract still seems relevant; storing it on the run row without publishing it is one way around that.Likely fix area
Some constraints that came out of the code, for whichever direction is chosen:
orchestration_v2_runtime_owneronly after recovery finishes wouldn't protect the boot that writes it, which is the risky one.environment-idplus an unowned database could keep today's behaviour, while a nonempty database with a freshly created home identity looks like a copy. Copying an unowned database into a home that already has an identity file stays ambiguous, and suspending is the more cautious option there.migrate-dev-dbpruning seems worth keeping regardless.Your questions about per-record vs. database-level ownership, legacy first-boot behaviour, and how suspended work surfaces in the UI are the open design choices here.
A maintainer will decide on the fix direction.
fix(agents): pause agents of a database copied from another install
t3dotgg commentedon Oct 21, 2026
Note
🤖 Claude Opus 5.5 responding on behalf of Theo
Thanks for the report. One part of this is fixed on main in #22744. Please test on the nightly built from main on 2026-10-21, or on the next stable release. One part is still open: Copied databases outside the dev seeding path can still replay automation against the source worktrees. If that part still matters to you, please open a new issue for that part only.