Summary

Adds rocky run --idempotency-key <KEY> — caller-supplied opaque key that dedups a run against prior runs with the same key. All three phases of the implementation plan ship together so reviewers see the full mechanism in one surface:

• Phase 1 (Option A — local + valkey + tiered): redb write txn inside state.redb.lock on local; Valkey SET NX EX on {prefix}:idempotency:<key> on Valkey/Tiered.
• Phase 2 (s3): object_store::PutMode::Create → If-None-Match: \"*\" on the S3 PutObject call.
• Phase 3 (gcs): same primitive, x-goog-if-generation-match: 0 precondition.

Three outcomes per invocation:

Defence-in-depth below Dagster's run_key. Catches pod retries, Kafka re-delivery, webhook duplicates, cron races. Also usable from non-Dagster callers (CI, cron, webhooks, custom orchestrators).

Design decisions locked

Per the plan in ~/Developer/rocky-plans/plans/rocky-fr-004-idempotency-key.md:

• Verbatim key storage (debuggability over privacy; CLI help + Dagster docstring carry the "no secrets in keys" warning).
• dedup_on = \"success\" default (failed runs delete their entry so retries claim cleanly; \"any\" opts in to dedup on any terminal status).
• 30-day retention default, swept during the existing state_sync::upload_state cadence — no new cron.
• 24h in_flight_ttl_hours — reader encountering a stale InFlight adopts via AdoptStale.
• In-flight lock via each backend's native primitive — one source of truth per backend, no new services.

Schema cascade (just codegen)

• RunStatus (rocky-core/src/state.rs) gains SkippedIdempotent + SkippedInFlight variants; rippled to Pydantic (run_schema.py) and TypeScript (run.ts).
• RunOutput (rocky-cli/src/output.rs) gains status: RunStatus, skipped_by_run_id: Option<String>, idempotency_key: Option<String>.
• [state.idempotency] TOML block with retention_days, dedup_on, in_flight_ttl_hours.
• redb schema version 4 → 5 (new IDEMPOTENCY_KEYS table; NOT in LOCAL_ONLY_TABLE_NAMES — replicates across pods for the tiered backend).

Guardrails

• --idempotency-key + --resume{,-latest} is a hard error at flag parse — resume is an explicit override and must never be short-circuited.
• S3/GCS object paths use sha256(key) for safety + bounded length; original key stored verbatim inside the payload.
• DAG sub-runs ignore the outer key to avoid sibling pipelines short-circuiting on a single stamp.

Dagster

run() / run_streaming() / run_pipes() all gain idempotency_key: str | None = None. _build_run_args threads --idempotency-key to the subprocess. Docstring carries the secrets warning.

Tests

• 1046 rocky-core tests pass, including 24 new tests covering:
  • Unit: IdempotencyCheck classification, DedupPolicy branching, object-store path stability, backend label resolution.
  • Integration (redb-backed): first claim proceeds, repeat succeeded skips, repeat in-flight-within-TTL skips, stale in-flight adopts, finalize-succeeded keeps entry, finalize-failed + Success policy deletes, finalize-failed + Any policy keeps, finalize with mismatched run_id is a no-op, sweep removes past-retention + stale-InFlight entries, concurrent claim is deterministic.
  • Object-store primitive: put_if_not_exists returns Created on first call, AlreadyExists on second.
• 374 dagster pytest tests pass.
• vscode tsc --noEmit clean.
• cargo clippy --all-targets -- -D warnings clean.
• cargo fmt --check clean.
• just codegen idempotent (no drift after rerun).

POC

examples/playground/pocs/05-orchestration/09-idempotency-key/ — self-contained DuckDB POC demonstrating the flag acceptance + short-circuit. ./run.sh verifies:

• Run 1 (fresh key) claims the entry.
• Run 2 (same key) short-circuits with status = SkippedInFlight / SkippedIdempotent + prior run_id in skipped_by_run_id.
• Run 3 (new key) claims independently.

Follows the POC-04 pattern (DuckDB POCs exit non-zero on "no discovery adapter" — orthogonal to the dedup primitive being validated).

Known follow-up

F1 in the plan §11 — today a run that errors before persist_run_record() leaves its InFlight entry until the 24h in_flight_ttl_hours sweep reaps it. Worst-case latency for retrying a crashed run with the same key is 24h. Tighten by extracting run_inner() (so the outer run() can catch errors and finalize) or adding a tokio::spawn-based drop guard on IdempotencyCtx. Dedicated follow-up PR.

Test plan

• CI green (engine-ci, dagster-ci, vscode-ci, codegen-drift)
• cargo test -p rocky-core — 1046 pass
• pytest integrations/dagster/tests/ — 374 pass
• Manual: run examples/playground/pocs/05-orchestration/09-idempotency-key/run.sh, confirm Run 2 short-circuits with a skip status and surfaces Run 1's run_id

References

• FR-004 source: ~/Developer/rocky-plans/feature-requests/gold-fr-004-rocky-run-idempotency-key.md
• Implementation plan: ~/Developer/rocky-plans/plans/rocky-fr-004-idempotency-key.md
• Backlog reprioritization: ~/Developer/rocky-plans/archive/backlog-reprioritization-2026-04-22.md