Agent skills for the correctness problems that only show up in production: retries that double-charge, replays that drift a ledger, partial failures that leave two systems disagreeing.
Most agent skill packs cover the development lifecycle — testing, review, refactoring, docs. This one covers the part that gets discovered by finance three months later.
Distilled from fifteen years building commerce, pricing, payments, and fulfillment systems at scale — including idempotent payout pipelines serving 500M+ daily users and order pipelines at 20K TPM.
Core correctness
| Skill | Covers | Status |
|---|---|---|
idempotency-and-exactly-once |
Retry-safe side effects, dedup stores, atomic claims, reconciliation | ✅ Ready |
failure-mode-analysis |
Enumerating partial failures before writing the happy path; the three-outcome table; blast radius; degradation ladder | ✅ Ready |
resilience-patterns |
Timeouts, backoff, jitter, circuit breakers, bulkheads, backpressure, load shedding | ✅ Ready |
distributed-data-consistency |
Sagas, outbox pattern, compensations, read-after-write, dual-write elimination | ✅ Ready |
money-movement-correctness |
Ledger invariants, double-entry, settlement windows, dispute handling, audit trails | ✅ Ready |
multi-region-and-replication-lag |
Read classification, read-your-writes mechanisms, lag budgets, failover thresholds, write conflicts | ✅ Ready |
observability-for-correctness |
Invariants as queries, age-based backlog alarms, job heartbeats, per-state gauges, directional reconciliation | ✅ Ready |
Coordination and flow
| Skill | Covers | Status |
|---|---|---|
distributed-locking-and-leases |
Fencing tokens, lease revalidation, CAS over object storage, backend capability checks | ✅ Ready |
queue-semantics-and-replay |
Delivery guarantees, ack placement, ordering, poison messages, DLQ replay safety | ✅ Ready |
rate-limiting-and-quota-correctness |
Shared counters, algorithm choice, fail-open vs fail-closed, quota vs throughput | ✅ Ready |
clock-and-time-correctness |
Which clock a timestamp came from, skew-proof lease expiry, monotonic durations, daylight-saving arithmetic | ✅ Ready |
Change and state
| Skill | Covers | Status |
|---|---|---|
data-migration-and-backfill |
Expand/migrate/contract, resumable backfills, verification before contraction | ✅ Ready |
caching-and-staleness |
Staleness budgets, invalidation, stampede protection, what must never be cached | ✅ Ready |
api-compatibility-and-versioning |
Hyrum's Law, contract evolution, deprecation windows, breaking-change detection | ✅ Ready |
feature-flags-and-rollout-safety |
Deciding a flag once per unit of work, kill switches above experiments, reversible ramps, fail-safe lookups | ✅ Ready |
See #16 for the full roadmap, what is deliberately out of scope, and candidates still under consideration.
Each skill follows the same contract:
- Actionable, not advisory. A process with steps and code, not a list of concepts to be aware of.
- Verifiable. Every skill ends with checks that fail concretely — "if you can't point at a test that breaks when this logic is removed, the logic is undefended."
- Rationalization-aware. Each skill names the excuses used to skip it, with rebuttals. Agents and engineers reach for the same ones.
- Cross-referenced, not duplicated. Skills point at each other rather than restating.
Some material is worth carrying into a review on its own, separately from the skill that explains it. Those artifacts live under references/:
| Reference | From | Use it for |
|---|---|---|
| Idempotency review checklist | idempotency-and-exactly-once |
A pre-merge gate for any change with a retryable side effect. Paste into a PR description or review comment. |
Copy the skill directory into your agent's skills path:
# Claude Code
cp -r skills/idempotency-and-exactly-once ~/.claude/skills/
# Or install the whole pack
git clone https://github.com/abhisheksharma2411/distributed-systems-skills
cp -r distributed-systems-skills/skills/* ~/.claude/skills/Each skill ships an eval case under evals/cases/ defining positive triggers, negative triggers (routed to the skill that should own them), and behavioral expectations, backed by a real fixture under evals/fixtures/. See evals/README.md for the format and the minimum bar.
The format is deliberately compatible with addyosmani/agent-skills, so skills developed here can be submitted upstream unchanged. idempotency-and-exactly-once has been submitted there as PR #479, where it passes that repo's full validator suite (validate-skills, validate-reference-links, validate-artifact-paths, validate-commands, and run-evals --min-rank1 80) with zero errors and zero warnings.
The skills above are the knowledge layer. The capability layer — an MCP server that inspects a running system for these properties rather than advising about them — is in progress: reading dedup stores, checking retry and timeout configuration, and replaying an effect ledger to answer "did this agent double-charge anyone?"
MIT