A collection of real failures in the plumbing that starts AI agents (schedulers, task registries, watchdogs), each with a small script that reproduces the failure on your own computer.
Who it's for: people who run scheduled AI agents, and researchers who study how agent systems break. What you get: three documented cases, each with a repro script and a check that turns the silent failure into an alarm.
- Install Python 3.8 or newer. Nothing else is needed.
git clone https://github.com/tonydzi/agent-control-plane-casebook- Run
python agent-control-plane-casebook/incidents/001-one-bad-entry-kills-the-scheduler/repro.py(usepython3ifpythonis not set up).
Done when: the last line printed is repro: OK - mechanism demonstrated. The first line, loaded=0 fired=0 and alarm=none (SILENT), looks like a failure but is the point: it shows the scheduler silently dropping all 6 tasks with no alarm. The second line shows the fix catching it (alarm=PARITY).
Want an AI agent to do it for you? Paste this to Claude Code / Codex / Cursor: "Read the 'For AI agents' section of https://github.com/tonydzi/agent-control-plane-casebook and set it up for me."
- Python 3.8 or newer, standard library only. No network, no accounts, no API keys.
- Any OS. The author ran the checks below on Windows with Python 3.12.
- Ask the human only if they want to submit a reproduction as an issue: it needs their GitHub account and their consent to post OS and runtime details.
git clone https://github.com/tonydzi/agent-control-plane-casebook
cd agent-control-plane-casebook
python incidents/001-one-bad-entry-kills-the-scheduler/repro.py
python incidents/002-restart-refires-ran-and-disabled-tasks/repro.py
python incidents/007-a-second-list-makes-the-reader-report-zero/repro.py
python incidents/001-one-bad-entry-kills-the-scheduler/test_registry_guard.py
python incidents/002-restart-refires-ran-and-disabled-tasks/test_catchup_guard.py
python incidents/007-a-second-list-makes-the-reader-report-zero/test_lookup_guard.pyEach case folder has README.md (symptom, root cause, fix, evidence), repro.py and a test_*.py guard. Case numbers are permanent; new cases follow template/CASE.md and CONTRIBUTING.md.
Nothing to configure. The repro scripts use a fixed clock and no network, so output is the same on every run.
python incidents/001-one-bad-entry-kills-the-scheduler/repro.pyExpected output, exit code 0 (verified locally):
mode=all-or-nothing on_disk=6 loaded=0 fired=0 dropped=6 alarm=none (SILENT)
mode=quarantine on_disk=6 loaded=5 fired=5 quarantined=1 alarm=PARITY on_disk=6 loaded=5
mode=healthy on_disk=6 loaded=6 fired=6 quarantined=0 alarm=none
repro: OK - mechanism demonstrated
Cases 002 and 007 also end with repro: OK - mechanism demonstrated and exit 0. The repro.py scripts only demonstrate a mechanism, so they print OK whenever they run; they are not the check that fails when something is broken. That check is the guard tests: run test_registry_guard.py (001), test_catchup_guard.py (002) and test_lookup_guard.py (007); each exits 0 and ends with 4/4 passed, 4/4 passed and 5/5 passed respectively, and exits non-zero with a FAIL line if a guard stops working.
Delete the cloned folder. The scripts write nothing outside it.
- These scripts model a mechanism. They are not the vendors' code, so do not claim they prove a vendor bug on the human's own setup.
- Do not open issues or comments on upstream trackers without the human's approval; state OS, runtime version and what you actually saw if they approve.
- Do not add real task names, paths or private configuration to a new case.
Reproducible failure cases from the control plane of a real, production multi-agent fleet — the schedulers, registries, dispatchers and watchdogs that decide when and whether agents run, each case carrying a runnable repro such as incidents/001-one-bad-entry-kills-the-scheduler/repro.py. Not the models. The plumbing that launches them.
Independent. This project is not affiliated with, endorsed by, or reviewed by Anthropic, OpenAI, Hugging Face, or any other vendor. Upstream links point at public bug trackers, several to reports we filed ourselves.
No install, no dependencies, no network. Python 3.8+ and its standard library:
git clone https://github.com/tonydzi/agent-control-plane-casebook
python agent-control-plane-casebook/incidents/001-one-bad-entry-kills-the-scheduler/repro.pymode=all-or-nothing on_disk=6 loaded=0 fired=0 dropped=6 alarm=none (SILENT)
mode=quarantine on_disk=6 loaded=5 fired=5 quarantined=1 alarm=PARITY on_disk=6 loaded=5
mode=healthy on_disk=6 loaded=6 fired=6 quarantined=0 alarm=none
repro: OK - mechanism demonstrated
Six tasks on disk, zero loaded, zero fired, and not one alarm anywhere. That first line is what a dead scheduler looks like from the outside, which is why nobody notices it for days.
Did it reproduce on your stack, or did it not? Both answers are worth an issue: name the case, your OS and runtime version, and what you actually saw. Outside reproductions are what move a case from "happened to us" to "happens", and they are the contribution this book needs most.
Model quality gets benchmarked constantly. The layer that actually runs agents in production — cron registries, session spawners, health checks — mostly gets debugged in private and forgotten. Its failures share one trait: they are silent by construction. A scheduler that loads zero tasks doesn't crash; it just stops, while every liveness proxy around it stays green — that is case 001, written up in incidents/001-one-bad-entry-kills-the-scheduler/README.md.
We run a multi-machine agent fleet (Windows + macOS + a cloud anchor) with dozens of scheduled agent routines. Every case in this book actually happened to us, in production, with dates, versions and log excerpts in the shape template/CASE.md prescribes — and where the root cause lives upstream, a public issue is linked.
Each case ships as a directory under incidents/ with:
- Symptom — what the operator saw (usually: nothing).
- Root cause — the mechanism, stated plainly.
- Deterministic repro such as incidents/001-one-bad-entry-kills-the-scheduler/repro.py — a stdlib-only Python model of the mechanism (Python 3.8+,
no network, no dependencies, fixed clock, same output every run; use
python3wherepythonisn't aliased). - Detection guard such as incidents/001-one-bad-entry-kills-the-scheduler/test_registry_guard.py — the check that turns the silent failure into an alarm, with a test that is shown red on the broken behavior.
- Fix / mitigation and residual risk — what actually resolved it, and what still can bite, in the section template/CASE.md reserves for it.
- Evidence — versions, dates, upstream links.
An honest limitation, stated once and repeated per case: repro scripts like incidents/002-restart-refires-ran-and-disabled-tasks/repro.py model the mechanism deterministically; they are not the vendor's code. The real-world evidence is cited separately so you can check the mechanism against it.
| # | Title | Status | Upstream |
|---|---|---|---|
| 001 | One malformed field silently kills the whole scheduler registry | reproduced · upstream open | claude-code#90533 |
| 002 | A restart re-fires already-run and disabled tasks, invisibly | reproduced · upstream open | claude-code#74055 |
| 007 | A registry that grew a second top-level list makes readers report a confident zero | reproduced · upstream open | claude-code#82056 |
Planned — each already documented in our incident journal; a case only lands here together with its public evidence (log excerpt or upstream link), the counters below are journal figures that will be published with the case:
- 003 — Split-brain registries: per-account × per-workspace task stores make routines invisible after an account switch (measured: 269 task prompts on disk, 23 registered) — claude-code#89840
- 004 — Encoding kills the cron: a BOM-less
.ps1written by an agent parses as ANSI and dies only when scheduled, with no log — claude-code#90962 - 005 — Clone storm: a remote dispatcher duplicates one prompt into N parallel sessions (fleet journal, 6 recurrences)
- 006 — Watchdog false-green: a health check that accepts any reply as service reported "0 orphans" for 22 consecutive runs across 2 real orphans (fleet journal)
Case numbers are permanent once assigned, including to a planned case, so a case that ships before its lower-numbered neighbours takes the next free number rather than renumbering the queue; the rule lives in CONTRIBUTING.md.
- Our own stack only. No scoring, no vendor comparisons, no leaderboard — the scope is fixed in CONTRIBUTING.md. This is a casebook, not a benchmark — deliberately. A scoring harness only earns the right to exist after cases have external reproductions and at least one vendor-confirmed root cause, which is the bar CONTRIBUTING.md sets. That is the staged plan, in that order.
- Failures are abstracted to their mechanism (registry shapes, loader contracts, timing), not our private configuration.
- External reproductions are the most valuable contribution — see CONTRIBUTING.md.
Maintained by Anton Dzyatkovsky. Case docs and code are
drafted by Mycroft — Anton's synthetic co-founder, an AI agent — and published
autonomously; Anton is the responsible human for this repository. Every commit carries Assisted-by provenance trailers; the repository is licensed MIT and citable via CITATION.cff. Corrections: open an issue.
License: MIT.
This repository is one piece lifted out of a live operation: one engineer running a fleet of machines, built with Claude as implementation collaborator; the machines reach consensus with each other and wake the human only for money or the irreversible. It was extracted after it survived production, not written as a demo — and it runs on its own: nothing here phones home to the rest.
See how the whole thing fits together → SYSTEM.md
Its closest neighbours in the in public layer: the-journey · clawrush · dashboards
This project is built by a human + AI team, and the git log says so: Claude writes most of the code, Codex and Grok review it, Gemini feeds the research. Each is credited on a commit only if its output changed that commit's content — no decorative credits. Lab-wide policy, one source for every repo: AI-CONTRIBUTORS.md.
One click and an agent reads the repo, pulls out the patterns and helps you apply them to your own work.
Copy the prompt (works in any agent: Gemini, Grok, a local model, your own CLI)
Read this repo: https://github.com/tonydzi/agent-control-plane-casebook (“agent-control-plane-casebook” - Reproducible control-plane failures from a production multi-agent fleet: schedulers, registries, watchdogs. Independent casebook, not a benchmark (yet)). Work out what problem it actually solves, pull out the reusable patterns and help me apply them to my own setup. Start by asking what I am working on.
— TonyDzi, Palo Alto AI Research Lab · second brain, agent coordination, persistent memory: github.com/tonydzi