Role: Tech Lead · Inputs: RFC-0001 + RFC-0001a (source of truth) · Principle: every phase ends with the repository in a working, demoable, tested state. No phase leaves a broken build or a half-wired subsystem.
- Vertical slices, not horizontal layers. Phase 1 already runs a durable workflow end-to-end (even if trivial). We do not build "all of the backend" then "all of the UI."
- Each phase is releasable. Tag
v0.1…v0.9. A user could clone at any tag and run something real. - The durability proof is front-loaded. By end of Phase 3 you can kill a worker mid-run and watch recovery — that is the whole pitch, so it comes early.
- Difficulty legend: S ≈ ≤1 day · M ≈ 2–4 days · L ≈ 1–2 weeks · XL ≈ multi-week/multi-issue.
Goal: an empty-but-real monorepo where docker compose up starts Temporal + Postgres + a FastAPI health endpoint + a Next.js shell, and CI is green.
- Deliverables: monorepo scaffold; Compose stack (Temporal, Postgres, Redis, API, web); CI (lint, type-check, test, build); pre-commit;
ancoraCLI stub; ADR/RFC docs wired. - Components:
services/api-gateway,web/,packages/sdk-python,packages/cli,deploy/docker. - Technologies: Python 3.11, FastAPI, Pydantic v2,
uv/Poetry, Temporal dev server, Postgres 15, Redis 7, Next.js 14, TypeScript, Tailwind, shadcn/ui, GitHub Actions. - APIs:
GET /healthz,GET /v1/version. - DB changes: initial migration framework (Alembic);
org,project,usertables. - UI changes: app shell, nav, dark theme, health page hitting the API.
- Testing: CI runs unit tests + lint +
tsc; one API test; one web smoke test (Playwright) that loads the shell. - Definition of Done:
docker compose up→ dashboard shell loads,/healthzgreen, CI green onmain, README quickstart works on a clean clone.
Goal: start a real Temporal workflow from the API and watch it complete. No Ray yet; activities run inline.
- Deliverables: Temporal worker service; minimal SDK (
@workflow,self.call,@activity); start/query/list runs via API; a "hello" workflow with 3 sequential activities. - Components:
services/workflow-workers,services/workflow-service,packages/sdk-python(core). - Technologies: Temporal Python SDK.
- APIs:
POST /v1/workflows/{name}/runs,GET /v1/runs/{id},GET /v1/runs,POST /v1/runs/{id}/cancel. - DB changes:
workflow_def,workflow_version,workflow_run(projection, populated crudely by polling for now — replaced in Phase 4). - UI changes: run list page; run detail (status, input/output).
- Testing: integration test spins Temporal dev server, starts a workflow, asserts completion; replay test harness introduced.
- Definition of Done:
POSTa run → it executes 3 activities →Completed; visible in UI; replay test passes; killing the worker mid-run and restarting it resumes to completion (durability smoke test).
Goal: activities dispatch to Ray; long activities use async completion; worker lifecycle is real.
- Deliverables: activity-worker service; Ray integration; async activity completion path; heartbeat-checkpointing; cooperative cancellation; graceful drain; worker registration/discovery; capability→queue routing (gpu/cpu/io queues exist).
- Components:
services/activity-workers, worker registry (inplugin-registry/workflow-service), Ray head in Compose. - Technologies: Ray, Temporal async completion (
complete_activity), Temporal heartbeats. - APIs:
GET /v1/workers,GET /v1/queues. - DB changes:
workerregistry table;node_execution(still crude projection). - UI changes: Worker view (pools, utilization, registered capabilities); queue depth widget.
- Testing: integration test: long activity → async-handed-off → Ray completes → workflow resumes; kill activity worker (inline) → retry; kill activity worker (async) → no re-run; checkpoint resume test.
- Definition of Done: a workflow with a 30s "GPU-ish" activity runs via async completion; dispatcher slot is freed during compute (asserted); worker drains cleanly on SIGTERM; killing workers mid-run recovers with the correct dup-safety per §8 scenarios 1–3.
Goal: the durability proof is complete and the node library is usable.
- Deliverables: Scheduler service (admission, capability routing, per-provider rate-limit via Redis token buckets, backpressure via queue watermark → defer, fair-queue weights; budget/deadline stubbed with interfaces); built-in nodes LLM, HTTP, Python, Database, Approval; inbox idempotency table; retry policies per node class; provider fallback chain for LLM.
- Components:
services/scheduler, node library inpackages/sdk-python/nodes. - Technologies: Redis (token buckets, locks), provider SDKs (mock provider for CI).
- APIs:
POST /v1/runs/{id}/signals/{name},GET /v1/approvals,POST /v1/approvals/{id}/decision,POST /v1/chaos/inject(basic),GET /v1/runs/{id}/cost(from returned costs). - DB changes:
inbox(idempotency),retry_attempt,approval_gate,cost_ledger. - UI changes: DAG view (React Flow) with node states; node inspector (logs/input/output/retries); Approval inbox + approve/reject.
- Testing: rate-limit governor prevents 429 storm (scenario 9); backpressure defers under load; idempotent side-effect test (HTTP node fired twice → one effect); human-gate durable wait test (scenario 12); LLM fallback test.
- Definition of Done: end-to-end research-agent example runs (search→summarize×N→synthesize→approve→publish); rate-limits/backpressure demonstrably work; kill any worker at any node → correct resume, zero dup effects (the north-star metric from RFC-0001 §1).
Goal: you can see everything live and after the fact.
- Deliverables: interceptor→Redis-Stream→consumer projection pipeline (replaces crude polling); OTel tracing with context propagated across the Ray boundary; Prometheus metrics catalog; Grafana dashboards; reconnect-safe WebSocket via Redis Streams; timeline/history replay UI; cost breakdown; critical-path bottleneck view.
- Components:
services/event-consumer,observability/, WS layer in API gateway. - Technologies: OpenTelemetry, Prometheus, Grafana, Redis Streams, Temporal interceptors.
- APIs:
WS /v1/stream/runs/{id},WS /v1/stream/workers,GET /v1/runs/{id}/history,POST /v1/runs/{id}/replay. - DB changes: projections now authoritative for UI reads; reconciler job; visibility search wiring (Advanced Visibility optional).
- UI changes: live-animating DAG; history scrubber + replay; timeline; cost view; worker live metrics; DAG virtualization + minimap.
- Testing: trace spans span workflow→activity→ray→provider (asserted unbroken); reconnect replays missed events; projection reconciler rebuilds from history; bottleneck critical-path correctness test.
- Definition of Done: open a run mid-flight → DAG animates in real time; drop the WS → reconnect → no gaps; one trace links UI+logs+metrics+cost; Grafana shows RED/USE + cost.
Goal: third parties can add nodes; chaos is a first-class, asserting feature.
- Deliverables: plugin contract + registry; isolation tiers T0/T1 (Ray runtime-env dep isolation); resource limits + capability manifest; Sigstore signing + verification; Chaos engine with scenario library (1–9) + invariant assertions (0 lost state / 0 dup effects) + RTO measurement.
- Components:
services/plugin-registry,services/chaos-controller. - Technologies: Ray runtime environments, cosign/Sigstore, subprocess sandbox.
- APIs:
POST /v1/plugins,GET /v1/plugins,POST /v1/chaos/inject,GET /v1/chaos/experiments. - DB changes:
plugin,node_type,chaos_event. - UI changes: Plugin registry view (versions, tier, signature status, schema); Chaos Lab (scenario picker, blast-radius, live recovery timeline, expected-vs-actual RTO, pass/fail).
- Testing: custom plugin (cross-encoder rerank) registers + runs sandboxed with declared resources; unsigned plugin rejected; each chaos scenario 1–9 runs and asserts its invariant in CI (chaos regression suite).
- Definition of Done: a signed third-party node runs end-to-end in a workflow; Chaos Lab kills a Ray node mid-run and shows measured recovery + a green invariant assertion.
Goal: run it on Kubernetes with autoscaling and a real security posture.
- Deliverables: Helm charts; KubeRay operator integration + Ray autoscaler; KEDA activity-worker autoscaling on queue backlog; Postgres HA + Redis; secrets manager integration; OIDC/SSO login + RBAC (org/project/run scopes); tenant isolation (namespaces + queues); network policies; TLS/mTLS.
- Components:
deploy/helm,deploy/terraform(optional),services/api-gateway(auth). - Technologies: Kubernetes, Helm, KubeRay, KEDA, OIDC, cert-manager, external-secrets.
- APIs: auth on all endpoints;
POST /v1/auth/*; API-key management. - DB changes:
api_key, RBAC tables; row-level project scoping. - UI changes: login, org/project switcher, RBAC-gated actions, settings.
- Testing: Helm install on kind/k3d in CI; autoscale-up under load and scale-down with warm-model retention; authz matrix tests; secret never appears in history (asserted).
- Definition of Done:
helm installon a fresh cluster → full stack healthy; load spike autoscales Ray + activity workers; unauthorized access denied; secrets sourced from manager, absent from history.
Goal: the v2 developer surface and full scheduler.
- Deliverables: declarative DAG SDK +
g.compile()→dag_spec; LangGraph adapter (compile a LangGraph to an Ancora workflow); budget + deadline scheduling fully enabled; retrieval + embedding nodes with Ray auto-batching; run comparison/diff UI (eval). - Testing: a LangGraph graph runs durably on Ancora unchanged in behavior; budget hard-stop test; deadline cancellation test; auto-batching throughput test.
- Definition of Done: the same workflow authored declaratively and imperatively produce identical histories; budgets/deadlines enforced; a LangGraph example runs on Ancora.
Goal: production credibility.
- Deliverables: load/soak tests; failure-mode runbooks; full API reference (generated OpenAPI); SDK docs + tutorials; example gallery; upgrade/versioning guide; SBOM + release signing; contribution guide + governance.
- Testing: 24h soak with periodic chaos injection; chaos regression suite gating releases; performance benchmarks published.
- Definition of Done: soak passes with zero lost workflows; docs cover quickstart→production;
v1.0-rctagged; chaos suite green in CI.
- Trunk-based with short-lived PRs; every PR: CI green + review + no drop in coverage on changed lines.
- Every subsystem ships with a replay test (durability) and, where it touches failure, a chaos assertion.
- Migrations are forward-only and reversible-tested; no destructive migration without a backfill.
- Feature flags for anything half-built so
mainis always releasable. - Docs-as-code: the relevant RFC is merged before the phase's first implementation PR.
- Observability is not a phase you bolt on — every service emits structured logs + metrics + traces from its first commit (Phase 4 makes them pretty, not present).
flowchart LR
P0[Phase 0\nFoundation] --> P1[Phase 1\nDurable core]
P1 --> P2[Phase 2\nRuntime + Ray]
P2 --> P3[Phase 3\nScheduler + Nodes]
P3 --> P4[Phase 4\nObservability]
P3 --> P5[Phase 5\nPlugins + Chaos]
P4 --> P6[Phase 6\nDeploy + Security]
P5 --> P6
P6 --> P7[Phase 7\nDeclarative SDK]
P7 --> P8[Phase 8\nHardening + 1.0]