Status: Draft · Author: Platform Engineering · Reviewers: TBD · Target: Open-source flagship (Apache-2.0) Document type: Combined PRD + Technical Design Document + Architecture Specification Codename:
Ancora(Italian anchor — the thing that keeps a ship from drifting). Final product name TBD; every occurrence of Ancora is a placeholder.
- Executive Summary
- Vision
- Problem Statement
- Competitive Analysis
- System Architecture
- Component Breakdown
- Database Schema
- Event Flow
- Workflow Lifecycle
- Failure Recovery Lifecycle
- State Machine
- API Design
- SDK Design
- Plugin Architecture
- UI Wireframes
- Folder Structure
- Deployment Architecture
- Scaling Strategy
- Security Model
- Future Roadmap
- Engineering Tradeoffs
- Risks
- Open Questions
- Stretch Goals
Ancora is a durable execution runtime for AI workloads. It fuses two proven systems — Temporal for durable, event-sourced workflow orchestration and Ray for distributed, GPU-aware compute — behind a single programming model and control plane purpose-built for LLM agents and multi-step AI pipelines.
The thesis is narrow and deliberate: AI applications are non-deterministic, long-running, expensive, and failure-prone, yet the orchestration layer beneath them is usually a for loop in a Python process. When that process dies — an OOM on a GPU worker, a 90-second LLM timeout, a provider rate-limit storm, a pod eviction — the entire multi-thousand-token, multi-dollar computation is lost. Ancora makes that loss structurally impossible.
Ancora is not an agent framework. It is the runtime underneath agent frameworks. LangGraph, CrewAI, AutoGen, or a hand-rolled orchestration loop should be able to target Ancora as their execution substrate the way a web app targets a Kubernetes cluster.
What we deliver:
- A workflow engine with DAGs, conditional branches, parallel fan-out/fan-in, typed retry policies, checkpointing, deterministic replay, and workflow versioning — inherited from and enforced by Temporal's event-sourced core.
- An execution engine that offloads heavy, parallel, and GPU-bound work to Ray, with dynamic worker allocation, resource constraints, queueing, and autoscaling.
- A first-class AI node library (LLM, embedding, retrieval, tool-call, HTTP, DB, shell, human-approval) with built-in idempotency, cost accounting, and streaming.
- Deep observability: live workflow graph, per-node timelines, event stream replay, worker health, queue depth, GPU/CPU utilization, OpenTelemetry traces, Prometheus metrics.
- Human-in-the-loop gates that keep a workflow durably suspended (zero compute cost) for minutes, days, or weeks awaiting approval.
- A Chaos Mode control surface that injects real failures (kill worker, drop provider, timeout, GPU crash) so teams can watch recovery, not just trust it.
Target users: platform teams building agentic products, ML infra engineers running batch inference and evaluation pipelines, and framework authors who want durability without reimplementing event sourcing.
Success metric for MVP: a developer can express a 10-node agentic workflow in <50 lines of SDK code, kill any worker at any time, and observe the workflow resume to correct completion with no duplicated side effects and no lost state.
"Kubernetes made stateless services trivial to run reliably. Ancora makes stateful, long-running, expensive AI computations trivial to run reliably."
Three-year vision:
- The default substrate for production agents. When a team ships an AI agent to production and asks "what happens when this fails at step 7 of 12?", the answer is "Ancora already handled it."
- A portable programming model. Workflows authored against the Ancora SDK run identically on a laptop (single-node dev mode), a single VM (Docker Compose), and a 500-GPU cluster (Kubernetes + Ray) with no code change.
- An observability standard for AI. The Ancora dashboard becomes to AI workflows what the Temporal Web UI is to microservice orchestration and what the Ray dashboard is to distributed compute — the place you look when something breaks.
We measure the vision against a single north-star property: durability is not a feature you opt into; it is the default and only mode of execution.
A representative production agent — "research a topic, call three tools, synthesize, get human sign-off, publish" — touches every unreliable component in modern infrastructure at once:
| Failure class | Real-world trigger | Typical blast radius today |
|---|---|---|
| LLM call failure | 500 from provider, content-filter block, malformed JSON | Whole run aborts; partial work discarded |
| Timeout | Slow model, cold GPU, 120s+ generation | Client library gives up; retry re-runs everything |
| Rate limiting | 429 during traffic spike | Cascading retries amplify the storm |
| Worker crash | GPU OOM, pod eviction, node preemption | In-memory agent state gone |
| Network partition | Transient DNS/VPC blip | Half-committed side effects (email sent twice?) |
| Process restart | Deploy, autoscale-down, OOM-kill | Every in-flight workflow lost |
| Human latency | Approval takes 3 days | Process can't afford to hold a thread/socket that long |
The industry response has been to bolt retries onto individual calls. That is insufficient because retries at the call level are not durable at the workflow level. If the process holding the loop dies, no amount of tenacity decorators saves you. The state of the computation — which steps completed, what they returned, what side effects fired — lives in RAM and dies with the process.
- Agent frameworks (LangGraph, CrewAI, AutoGen) optimize for authoring agent logic. Their persistence is a checkpoint bolt-on, not an event-sourced source of truth, and they have no distributed compute or resource scheduling story.
- Durable orchestrators (Temporal, Prefect, Dagster, Airflow) solve durability beautifully but are compute-agnostic: they schedule tasks, they don't run GPU inference. You still bring your own Ray/K8s and glue it together.
- Compute engines (Ray) solve distribution and GPU scheduling but have no durable, event-sourced workflow state. A Ray job that dies mid-DAG restarts from scratch.
Nobody owns the seam. Ancora is that seam: durable state (Temporal) and distributed compute (Ray) unified behind one AI-native programming model.
A workflow is a deterministic program whose every non-deterministic and side-effecting step is recorded as an immutable event. Re-executing the program by replaying its event history reconstructs exact state without re-incurring side effects. Heavy steps run on elastic distributed workers. The developer writes ordinary-looking Python; the runtime provides the guarantees.
quadrantChart
title Durability vs. AI/Compute-Native
x-axis "Compute-agnostic" --> "GPU/Distributed-native"
y-axis "Best-effort state" --> "Durable / event-sourced"
quadrant-1 "Ancora's wedge"
quadrant-2 "Durable orchestrators"
quadrant-3 "Schedulers"
quadrant-4 "Compute engines"
Temporal: [0.20, 0.92]
Prefect: [0.30, 0.68]
Dagster: [0.34, 0.66]
Airflow: [0.25, 0.45]
Ray: [0.90, 0.20]
LangGraph: [0.55, 0.40]
CrewAI: [0.50, 0.22]
AutoGen: [0.52, 0.25]
Ancora: [0.82, 0.88]
| Capability | Temporal | Ray | LangGraph | CrewAI / AutoGen | Prefect | Dagster | Airflow | Ancora |
|---|---|---|---|---|---|---|---|---|
| Durable, event-sourced state | ✅ | ❌ | ❌ | ✅ | ||||
| Deterministic replay | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| Distributed / GPU compute | ❌ | ✅ | ❌ | ❌ | ❌ | ✅ | ||
| Autoscaling workers | ❌ | ✅ | ❌ | ❌ | ✅ | |||
| Native LLM/agent nodes | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | ❌ | ✅ |
| Human-in-the-loop (durable wait) | ✅ | ❌ | ❌ | ❌ | ✅ | |||
| Workflow versioning | ✅ | ❌ | ❌ | ❌ | ✅ | ✅ | ||
| Cost/token accounting | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ||
| Chaos/failure injection UX | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| Live DAG + replay UI | ❌ | ❌ | ✅ | ✅ | ✅ | ✅ |
Legend: ✅ first-class ·
- vs. Temporal: Ancora is built on Temporal's durability core but adds the missing compute layer (Ray), an AI-native node library, and cost/token semantics. We are a Temporal complement, not a competitor — much like Airflow-on-K8s vs. raw K8s.
- vs. Ray: Ancora is built on Ray's compute core but adds durable state, replay, and human-in-the-loop. Ray answers "run this fast in parallel"; Ancora answers "run this reliably, forever, and never lose it."
- vs. LangGraph/CrewAI/AutoGen: These are customers, not competitors. Their graphs should compile to Ancora workflows. We deliberately do not ship opinionated agent-authoring ergonomics (prompt templating, memory abstractions, tool routing DSLs).
- vs. Prefect/Dagster/Airflow: These are data-engineering orchestrators optimized for scheduled batch DAGs over data assets. Ancora is optimized for long-lived, interactive, non-deterministic, GPU-bound AI computations with per-step durability and replay.
One-line positioning: Ancora is to AI workflows what Kubernetes is to containers — the runtime, not the app.
flowchart TB
subgraph Client["Client Layer"]
SDK["Ancora Python SDK"]
CLI["ancora CLI"]
UI["Next.js Dashboard"]
end
subgraph ControlPlane["Control Plane"]
API["API Gateway (FastAPI)"]
WFSVC["Workflow Service"]
REG["Plugin / Node Registry"]
AUTH["AuthN/Z + RBAC"]
CHAOS["Chaos Controller"]
end
subgraph Durability["Durability Core — Temporal"]
TFE["Temporal Frontend"]
THIST["History Service (event sourcing)"]
TMATCH["Matching Service (task queues)"]
TWORKER["Workflow Workers (deterministic)"]
end
subgraph Compute["Compute Core — Ray"]
RHEAD["Ray Head / GCS"]
RAUTO["Ray Autoscaler"]
RCPU["CPU Actor Pool"]
RGPU["GPU Actor Pool"]
end
subgraph Activities["Activity Workers"]
ACT["Ancora Activity Workers\n(bridge Temporal ↔ Ray)"]
end
subgraph Data["State & Telemetry"]
PG[("PostgreSQL\nTemporal + App state")]
REDIS[("Redis\ncache / pub-sub / rate-limit")]
OTEL["OTel Collector"]
PROM["Prometheus"]
GRAF["Grafana"]
OBJ[("Object Store\nlarge payloads")]
end
SDK --> API
CLI --> API
UI <-->|REST + WebSocket| API
API --> WFSVC & REG & AUTH & CHAOS
WFSVC --> TFE
TFE --> THIST --> PG
TFE --> TMATCH
TMATCH --> TWORKER
TWORKER -->|schedule activity| ACT
ACT -->|ray.remote| RHEAD
RHEAD --> RAUTO --> RCPU & RGPU
ACT -->|large payloads| OBJ
ACT --> REDIS
TWORKER -.->|history| THIST
ControlPlane & Activities & Compute --> OTEL --> PROM --> GRAF
CHAOS -.->|fault injection| ACT & RGPU
The load-bearing design decision: Temporal workflows are deterministic and must never do I/O directly; Ray is where non-deterministic heavy work happens. We bridge them with a thin, durable Activity layer.
sequenceDiagram
participant WF as Temporal Workflow<br/>(deterministic)
participant Q as Task Queue
participant AW as Ancora Activity Worker
participant RAY as Ray Cluster
participant EXT as External (LLM/API/GPU)
WF->>Q: Schedule Activity("llm_node", input, retry_policy)
Note over WF: Workflow blocks durably.<br/>State is in Temporal history, not RAM.
Q->>AW: Deliver activity task
AW->>RAY: ray.remote(run_node).remote(input)
RAY->>EXT: Execute (GPU inference / API call)
EXT-->>RAY: Result (or failure)
RAY-->>AW: ObjectRef → result
AW-->>Q: CompleteActivityTask(result)
Q-->>WF: Deliver result → append to history
Note over WF: Workflow deterministically resumes.<br/>Survives worker/process death at any point.
Why this shape: Temporal guarantees exactly-once workflow progression via event sourcing; Ray guarantees efficient at-least-once compute. The Activity boundary is the idempotency seam — activities carry an idempotency key so an at-least-once Ray execution never produces a duplicated externally-visible side effect.
flowchart LR
subgraph K8s["Kubernetes Cluster"]
subgraph ns_ctl["ns: ancora-control"]
api["API Gateway (HPA)"]
web["Dashboard"]
end
subgraph ns_temporal["ns: temporal"]
tf["Frontend"]; th["History"]; tm["Matching"]; tw["WF Workers (HPA)"]
end
subgraph ns_ray["ns: ray"]
rh["Ray Head"]; rw["Ray Workers (GPU/CPU pools, autoscaled)"]
end
subgraph ns_data["ns: data"]
pg[("Postgres HA")]; rd[("Redis")]; prom["Prometheus"]; graf["Grafana"]
end
aw["Activity Workers (HPA/KEDA)"]
end
api --> tf --> th --> pg
tm --> tw --> aw --> rh --> rw
aw --> rd
K8s --> prom --> graf
| Component | Responsibility | Tech | Scaling axis |
|---|---|---|---|
| API Gateway | REST + WebSocket entrypoint; request validation; auth; rate limiting; SSE/WS fan-out of live events | FastAPI, Pydantic, uvicorn |
Stateless, HPA on RPS |
| Workflow Service | Translates API calls into Temporal client operations (start/signal/query/terminate); owns workflow metadata | Python, Temporal SDK | Stateless, HPA |
| Node / Plugin Registry | Catalog of node types (built-in + custom); schema validation; versioning; sandbox policy | Python, Postgres-backed | Stateless |
| Temporal Frontend | gRPC entrypoint to durability core; auth passthrough; rate limiting | Temporal (Go) | Stateless |
| History Service | The event-sourced heart: appends immutable events, drives replay, owns durability | Temporal | Sharded by workflow ID |
| Matching Service | Task-queue matchmaking between workflows and workers | Temporal | Partitioned queues |
| Workflow Workers | Execute deterministic workflow code; make orchestration decisions; never do I/O | Ancora SDK on Temporal worker | HPA on task-queue depth |
| Activity Workers | The bridge: receive activity tasks, dispatch to Ray, enforce idempotency, stream progress, report heartbeats | Python, Temporal + Ray clients | KEDA on queue depth |
| Ray Head / GCS | Global control store for cluster; actor/task scheduling; object store metadata | Ray | 1 (HA via GCS fault-tolerance + Redis) |
| Ray Workers | Actual compute: LLM inference, embeddings, data transforms, GPU kernels | Ray actors/tasks | Ray autoscaler on demand |
| Chaos Controller | Injects faults into activity/Ray layer; records fault → recovery timelines | Python | Stateless |
| Postgres | Temporal persistence + Ancora app metadata (workflow catalog, users, plugins, cost ledger) | PostgreSQL 15+ | Read replicas; partitioning |
| Redis | Rate-limit token buckets, WS pub/sub, hot cache, distributed locks | Redis 7 | Cluster mode |
| Object Store | Payloads exceeding Temporal's blob limits (large prompts, embeddings, artifacts) | S3/MinIO | Native |
| OTel Collector / Prometheus / Grafana | Traces, metrics, dashboards | OpenTelemetry, Prometheus, Grafana | Standard |
Temporal workflow workers must be deterministic — the same history must always produce the same decisions on replay. Any I/O, randomness, or wall-clock read inside workflow code breaks replay. Therefore all non-determinism is exiled to activities, and activities are exiled to a tier that can talk to Ray. This is the single most important boundary in the system; violating it silently corrupts durability.
Two logical databases: Temporal's (managed by Temporal, we don't touch its internals) and Ancora's application DB. Below is the Ancora app schema; Temporal's history is the workflow source of truth, so we intentionally do not duplicate event history — we store metadata, projections, and things Temporal doesn't own (users, plugins, cost).
erDiagram
ORG ||--o{ PROJECT : owns
PROJECT ||--o{ WORKFLOW_DEF : contains
WORKFLOW_DEF ||--o{ WORKFLOW_VERSION : versions
WORKFLOW_VERSION ||--o{ WORKFLOW_RUN : executes
WORKFLOW_RUN ||--o{ NODE_EXECUTION : has
NODE_EXECUTION ||--o{ RETRY_ATTEMPT : records
WORKFLOW_RUN ||--o{ APPROVAL_GATE : suspends_at
WORKFLOW_RUN ||--o{ COST_LEDGER : accrues
PROJECT ||--o{ PLUGIN : registers
PLUGIN ||--o{ NODE_TYPE : defines
ORG ||--o{ USER : has
USER ||--o{ API_KEY : holds
WORKFLOW_RUN ||--o{ CHAOS_EVENT : injected_with
-- Workflow definitions & versions ------------------------------------------
CREATE TABLE workflow_def (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
project_id UUID NOT NULL REFERENCES project(id),
name TEXT NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
UNIQUE (project_id, name)
);
CREATE TABLE workflow_version (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
workflow_def_id UUID NOT NULL REFERENCES workflow_def(id),
version INT NOT NULL, -- monotonically increasing
dag_spec JSONB NOT NULL, -- declarative DAG (nodes, edges, policies)
code_hash TEXT NOT NULL, -- content-address of workflow code
determinism_token TEXT NOT NULL, -- Temporal build/version ID for safe replay
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
UNIQUE (workflow_def_id, version)
);
-- Runs (a projection over Temporal history for fast querying) ---------------
CREATE TABLE workflow_run (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
temporal_wf_id TEXT NOT NULL, -- Temporal WorkflowId
temporal_run_id TEXT NOT NULL, -- Temporal RunId (changes on continue-as-new)
workflow_version_id UUID NOT NULL REFERENCES workflow_version(id),
status TEXT NOT NULL, -- see state machine §11
input_ref TEXT, -- object-store pointer if large
output_ref TEXT,
started_at TIMESTAMPTZ,
closed_at TIMESTAMPTZ,
UNIQUE (temporal_wf_id, temporal_run_id)
);
CREATE INDEX ix_run_status ON workflow_run(status);
CREATE INDEX ix_run_version ON workflow_run(workflow_version_id);
-- Per-node execution projection --------------------------------------------
CREATE TABLE node_execution (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
run_id UUID NOT NULL REFERENCES workflow_run(id),
node_id TEXT NOT NULL, -- stable node id within DAG
node_type TEXT NOT NULL, -- llm / embedding / http / ...
status TEXT NOT NULL, -- pending/running/completed/failed/retrying/waiting
idempotency_key TEXT NOT NULL,
ray_object_ref TEXT,
input_ref TEXT,
output_ref TEXT,
started_at TIMESTAMPTZ,
ended_at TIMESTAMPTZ,
duration_ms INT,
UNIQUE (run_id, node_id)
);
CREATE TABLE retry_attempt (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
node_execution_id UUID NOT NULL REFERENCES node_execution(id),
attempt INT NOT NULL,
error_type TEXT, -- timeout/rate_limit/oom/provider_5xx/...
error_detail TEXT,
backoff_ms INT,
occurred_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
-- Human-in-the-loop gates ---------------------------------------------------
CREATE TABLE approval_gate (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
run_id UUID NOT NULL REFERENCES workflow_run(id),
node_id TEXT NOT NULL,
status TEXT NOT NULL DEFAULT 'waiting', -- waiting/approved/rejected/expired
payload JSONB, -- what the human reviews
decided_by UUID REFERENCES "user"(id),
decided_at TIMESTAMPTZ,
expires_at TIMESTAMPTZ,
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
-- Cost/token accounting -----------------------------------------------------
CREATE TABLE cost_ledger (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
run_id UUID NOT NULL REFERENCES workflow_run(id),
node_id TEXT NOT NULL,
provider TEXT, -- openai/anthropic/local/...
model TEXT,
input_tokens INT,
output_tokens INT,
usd_cost NUMERIC(12,6),
occurred_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
-- Plugins / node types ------------------------------------------------------
CREATE TABLE plugin (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
project_id UUID NOT NULL REFERENCES project(id),
name TEXT NOT NULL,
version TEXT NOT NULL,
entrypoint TEXT NOT NULL, -- import path
schema JSONB NOT NULL, -- input/output JSON schema
sandbox_policy JSONB NOT NULL, -- resources, network, capabilities
signature TEXT, -- supply-chain signing
UNIQUE (project_id, name, version)
);
-- Chaos experiments ---------------------------------------------------------
CREATE TABLE chaos_event (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
run_id UUID REFERENCES workflow_run(id),
fault_type TEXT NOT NULL, -- kill_worker/drop_api/timeout/gpu_oom/...
target TEXT,
injected_at TIMESTAMPTZ NOT NULL DEFAULT now(),
recovered_at TIMESTAMPTZ,
recovery_ms INT
);Design note — projection, not source of truth: workflow_run and node_execution are read-optimized projections rebuilt from Temporal history by an event-consumer. Temporal's history is authoritative; if Postgres projections are lost we can rebuild them. This keeps the durability guarantee anchored in one place and makes the app DB replaceable.
Everything is event-sourced. Temporal produces a canonical event history; Ancora consumes it to drive projections, the live UI, metrics, and the cost ledger.
flowchart LR
subgraph Temporal["Temporal History (source of truth)"]
E1["WorkflowExecutionStarted"]
E2["ActivityTaskScheduled"]
E3["ActivityTaskStarted"]
E4["ActivityTaskCompleted / Failed"]
E5["TimerStarted / Fired"]
E6["WorkflowExecutionSignaled (approval)"]
E7["WorkflowExecutionCompleted"]
end
CONSUMER["Event Consumer\n(Temporal Visibility + custom)"]
subgraph Sinks
PROJ["Postgres Projections"]
WS["WebSocket Fan-out (live UI)"]
COST["Cost Ledger"]
OTELS["OTel Spans"]
METRICS["Prometheus Counters"]
end
E1 & E2 & E3 & E4 & E5 & E6 & E7 --> CONSUMER
CONSUMER --> PROJ & WS & COST & OTELS & METRICS
Live update path: activity workers emit heartbeats and progress via Temporal activity heartbeats and a Redis pub/sub channel keyed by run_id. The API Gateway subscribes and pushes to connected dashboard clients over WebSocket, so the DAG animates in real time without polling Temporal on the hot path.
sequenceDiagram
autonumber
participant Dev
participant API as API Gateway
participant WFSVC as Workflow Service
participant T as Temporal
participant WW as Workflow Worker
participant AW as Activity Worker
participant Ray
Dev->>API: POST /workflows/{name}/runs (input)
API->>WFSVC: validate + authz
WFSVC->>T: StartWorkflowExecution(id, version, input)
T-->>WFSVC: run_id
WFSVC-->>API: 202 Accepted {run_id}
T->>WW: Deliver workflow task
WW->>WW: Deterministically evaluate DAG → next nodes
WW->>T: Command: ScheduleActivity(node_A)
T->>AW: Activity task (node_A)
AW->>Ray: dispatch compute (idempotency_key)
Ray-->>AW: result
AW->>T: CompleteActivity(result)
T->>WW: Deliver result (append to history)
WW->>WW: Evaluate branches; fan-out node_B, node_C in parallel
Note over WW,Ray: repeat until terminal
WW->>T: CompleteWorkflow(output)
T-->>API: (via query/webhook) status=Completed
Key properties enforced across the lifecycle:
- Determinism: workflow code only orchestrates; all effects go through activities.
- Checkpointing is implicit: every completed activity is a durable checkpoint in history — no explicit save/load.
- Resume is free: if any worker dies, Temporal re-delivers the workflow task to another worker, which replays history to reconstruct exact state, then continues.
This is the product's core value. Recovery is handled at two layers with different guarantees.
stateDiagram-v2
[*] --> Executing
Executing --> ActivityFailed: exception / timeout / non-2xx
ActivityFailed --> Classify
Classify --> Retryable: transient (5xx, 429, timeout, OOM)
Classify --> NonRetryable: bad input, auth, content policy
Retryable --> Backoff: exponential + jitter, capped
Backoff --> Executing: retry attempt (same idempotency key)
Retryable --> Escalate: retries exhausted
NonRetryable --> Escalate
Escalate --> Compensate: run saga/rollback activities
Escalate --> HumanReview: route to approval gate
Compensate --> Failed
HumanReview --> Executing: operator retries/overrides
HumanReview --> Failed: operator aborts
Executing --> [*]: Completed
Failed --> [*]
| Injected failure | Detected by | Automatic response | Idempotency safeguard |
|---|---|---|---|
| Worker crash (activity worker dies mid-task) | Missed heartbeat / task timeout | Temporal re-queues task to another worker; Ray result may be re-fetched by ObjectRef or recomputed |
Idempotency key + result cache dedupes side effects |
| API timeout | Activity start_to_close_timeout |
Retry policy fires with backoff | Cached partial results keyed by attempt |
| LLM failure (5xx / malformed) | Exception classification | Retry; optional model fallback chain (primary → secondary) | Deterministic prompt hash key |
| Network interruption | gRPC/HTTP error | Retry; workflow state untouched (in history) | N/A — no side effect committed yet |
| GPU OOM | Ray OutOfMemoryError / OOM-kill |
Ray reschedules on a larger-memory node; autoscaler may add capacity; optional batch-size downshift | Recompute is safe (pure activity) |
| Rate limit (429) | Provider header / status | Redis token-bucket throttle + retry-after aware backoff |
Global limiter prevents retry storm |
| Process restart / deploy | Worker disconnect | All in-flight workflows replay on new workers | Full — durability core |
| Database restart | Temporal persistence blip | Temporal buffers & retries internally; workflows pause, then resume | Full |
Guarantee statement: For pure activities (no external side effect), Ancora provides effectively exactly-once completion via at-least-once execution + result caching. For side-effecting activities (send email, charge card), Ancora provides exactly-once via idempotency keys that the developer must supply or the SDK derives deterministically — the runtime enforces the check but the developer owns the key's semantic meaning.
stateDiagram-v2
[*] --> Queued
Queued --> Running: worker picks up
Running --> WaitingApproval: hits human gate
WaitingApproval --> Running: approved
WaitingApproval --> Cancelled: rejected/expired
Running --> Retrying: recoverable failure
Retrying --> Running: backoff elapsed
Running --> Paused: operator/chaos pause
Paused --> Running: resume
Running --> Completed: success
Running --> Failed: unrecoverable
Running --> Cancelled: operator cancel
Running --> ContinuedAsNew: history too large → fresh run
ContinuedAsNew --> Running
Completed --> [*]
Failed --> [*]
Cancelled --> [*]
Pending → Running → (Completed | Failed | Retrying | WaitingApproval); Retrying → Running on backoff; WaitingApproval → Running on signal. These map 1:1 to the UI node badges (§15).
Note on ContinuedAsNew: long-running or high-cardinality workflows accumulate large histories, which slow replay. Temporal's continue-as-new atomically starts a fresh execution carrying forward summarized state. Ancora surfaces this transparently — the logical run appears continuous in the UI even though run_id changes underneath.
REST for control operations; WebSocket for live streams. Versioned under /v1. All mutating endpoints require an idempotency key header. OpenAPI spec is generated from FastAPI/Pydantic models.
# Workflow definitions
POST /v1/workflows # register/version a workflow definition
GET /v1/workflows # list definitions
GET /v1/workflows/{name} # get definition + versions
# Runs
POST /v1/workflows/{name}/runs # start a run → 202 {run_id}
GET /v1/runs/{run_id} # run status + summary
GET /v1/runs/{run_id}/nodes # per-node execution projection
GET /v1/runs/{run_id}/history # full event history (replayable)
POST /v1/runs/{run_id}/signals/{name} # send signal (e.g. approval)
POST /v1/runs/{run_id}/cancel # graceful cancel
POST /v1/runs/{run_id}/terminate # hard terminate
POST /v1/runs/{run_id}/replay # deterministic replay (debug)
# Human-in-the-loop
GET /v1/approvals?status=waiting # pending gates
POST /v1/approvals/{gate_id}/decision # {approve|reject, comment}
# Observability
GET /v1/runs/{run_id}/cost # token/$ ledger
GET /v1/workers # worker health, pools, utilization
GET /v1/queues # task-queue depths
# Chaos
POST /v1/chaos/inject # {fault_type, target, run_id?}
GET /v1/chaos/experiments # fault→recovery timelines
# Plugins
POST /v1/plugins # register custom node type
GET /v1/plugins # list node types + schemasWS /v1/stream/runs/{run_id} → node state transitions, logs, progress, cost deltas
WS /v1/stream/workers → worker health, GPU/CPU/mem, queue depth
WS /v1/stream/events → org-wide event firehose (filterable)
Two authoring styles, one runtime. Both compile to the same durable Temporal workflow.
from ancora import workflow, activity, RetryPolicy
from ancora.nodes import LLMNode, HTTPNode, ApprovalGate
@workflow
class ResearchAgent:
@workflow.run
async def run(self, topic: str) -> dict:
# Each `call` is a durable activity dispatched to Ray.
sources = await self.call(
HTTPNode(url=f"https://search/?q={topic}"),
retry=RetryPolicy(max_attempts=5, backoff="exponential"),
)
# Parallel fan-out — runs on distributed Ray workers.
summaries = await self.gather(*[
self.call(LLMNode(model="claude-opus-4-8", prompt=summarize(s)))
for s in sources
])
draft = await self.call(LLMNode(model="claude-opus-4-8", prompt=synthesize(summaries)))
# Durable human gate — process holds ZERO resources while waiting.
decision = await self.await_approval(
ApprovalGate(payload=draft, expires_in="3d")
)
if not decision.approved:
return {"status": "rejected"}
return {"status": "published", "draft": draft}from ancora import Graph, node
g = Graph("research_agent", version=3)
search = g.add(node.HTTP(id="search", url="https://search/?q={{topic}}"))
summ = g.add(node.LLM(id="summarize", model="claude-opus-4-8"), map_over=search.output)
synth = g.add(node.LLM(id="synthesize", model="claude-opus-4-8"), depends_on=[summ])
gate = g.add(node.Approval(id="review", depends_on=[synth], expires_in="3d"))
g.edge(gate.on("approved") >> node.HTTP(id="publish", method="POST", url="https://cms"))
g.compile() # → validated dag_spec JSON, registered via API- Determinism guardrails: the SDK ships a static analyzer + runtime sandbox that flags forbidden operations in workflow code (
datetime.now(),random, direct network I/O) and points to the activity equivalent. - Typed I/O: every node declares Pydantic input/output; mismatches fail at compile, not at 2am.
- Local-first:
ancora devruns an in-process Temporal + local Ray so the identical code runs on a laptop. - Idempotency by construction:
self.callderives a deterministic idempotency key from(node_id, attempt-invariant input hash); side-effecting nodes can override.
Nodes are the extension unit. A node is a typed, sandboxed, versioned unit of work that runs as a Ray task/actor and is invoked via a Temporal activity.
flowchart TB
subgraph Contract["Node Contract"]
SCHEMA["Input/Output JSON Schema"]
EXEC["execute(input, ctx) -> output"]
POLICY["Resource + Sandbox Policy"]
META["Cost/metrics hooks"]
end
subgraph Registry
REG["Plugin Registry (Postgres)"]
SIGN["Signature verification"]
end
subgraph Runtime
AW["Activity Worker"]
SBX["Sandbox (subprocess / container / gVisor)"]
RAY["Ray actor pool"]
end
Contract --> REG --> SIGN
AW -->|resolve node_type + version| REG
AW --> SBX --> RAY
| Node | Purpose | Runs on | Notes |
|---|---|---|---|
| LLMNode | Chat/completion with any provider | GPU (local) or CPU (API) | Streaming, fallback chains, token accounting |
| EmbeddingNode | Vectorize text | GPU/CPU batch | Auto-batching on Ray |
| RetrievalNode | Vector/keyword search | CPU | Pluggable backends (pgvector, external) |
| ToolCallNode | Execute a declared tool | CPU | Schema-validated args |
| PythonNode | Arbitrary Python fn | CPU/GPU | Sandboxed |
| HTTPNode | REST call | CPU | Idempotency, retry-after aware |
| DatabaseNode | Parameterized SQL | CPU | Connection pooling, read/write split |
| WebhookNode | Emit event | CPU | At-least-once with dedupe |
| ShellNode | Run command | Sandboxed CPU | gVisor/container isolation |
| ApprovalGate | Human-in-the-loop | none | Durable wait, no compute cost |
| CustomComputeNode | User-defined Ray actor | CPU/GPU | Full resource control |
from ancora.plugin import node, NodeContext
from pydantic import BaseModel
class RerankIn(BaseModel):
query: str
docs: list[str]
class RerankOut(BaseModel):
ranked: list[str]
@node(
name="cross_encoder_rerank",
version="1.0.0",
resources={"num_gpus": 1, "memory_mb": 8000},
sandbox={"network": "deny", "filesystem": "read_only"},
)
async def rerank(inp: RerankIn, ctx: NodeContext) -> RerankOut:
model = ctx.ray_get_or_create_actor("cross-encoder") # cached across calls
scores = await model.score.remote(inp.query, inp.docs)
ctx.record_cost(gpu_seconds=ctx.elapsed()) # cost hook
return RerankOut(ranked=[d for _, d in sorted(zip(scores, inp.docs), reverse=True)])Isolation tiers: trusted first-party (in-process), semi-trusted (subprocess with resource limits), untrusted (container/gVisor with network deny). The registry's sandbox_policy selects the tier; the activity worker enforces it before dispatch to Ray.
Design language: dense, dark-first, keyboard-navigable — Temporal Cloud × Vercel × Ray Dashboard × GitHub Actions. Next.js + TypeScript + Tailwind + shadcn/ui + React Flow + WebSockets.
┌───────────────────────────────────────────────────────────────────────────┐
│ Ancora Workflows Workers History Chaos [org ▾] [⌘K] ◐ │
├───────────────────────────────────────────────────────────────────────────┤
│ Running 42 Completed 1.2k Failed 7 Waiting 3 $Today $184.20 │
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ ┌──────────────────────┐ │
│ │ Queue depth │ │ GPU util 78% │ │ Worker health│ │ p95 node latency │ │
│ │ ▁▂▅▇▆▃ 128 │ │ ███████░░ │ │ 18/20 ● │ │ ▂▃▂▅▃▂ 2.4s │ │
│ └──────────────┘ └──────────────┘ └──────────────┘ └──────────────────────┘ │
│ Recent runs │
│ ● research_agent #01J… Running 3/10 nodes 00:42 $0.87 [view] │
│ ⏸ invoice_review #01J… Waiting approval 2d $0.12 [view] │
│ ✖ batch_embed #01J… Failed GPU OOM →recovered $4.10 [view] │
└───────────────────────────────────────────────────────────────────────────┘
┌──────────────────────────────── run #01J… research_agent v3 ─────────────┐
│ [Replay ▷] [Pause ⏸] [Cancel ✖] status: Running $0.87 │
│ │
│ ┌─────────┐ ┌──────────────┐ ┌───────────┐ │
│ │ search │─────▶│ summarize │───┐ │ synthesize│ │
│ │ ●done │ │ ●running x3 │ ├──▶│ ○pending │ │
│ └─────────┘ │ (fan-out) │ │ └───────────┘ │
│ └──────────────┘ │ │ │
│ │ ▼ │
│ ┌───────────┐ ┌───────────┐ │
│ │ review │ │ publish │ │
│ │ ⏸waiting │ │ ○pending │ │
│ └───────────┘ └───────────┘ │
│ Legend: ●done ◍running ○pending ⟳retrying ⏸waiting ✖failed │
├─────────────────────────── node inspector: summarize[1] ──────────────────┤
│ Status running · Worker gpu-07 · Attempt 1 · 1.9s · $0.03 · 812 tok │
│ Input ▸ {"text":"…"} Output ▸ (streaming…) Retries ▸ none │
│ Logs ▸ 12:04:01 dispatch→ray 12:04:02 model=claude-opus-4-8 … │
└────────────────────────────────────────────────────────────────────────────┘
┌─────────────────────────────── Workers ──────────────────────────────────┐
│ Pool: GPU (12) CPU (8) Autoscaler: ● enabled target 82% util │
│ ┌ gpu-07 ─────────────┐ ┌ gpu-08 ─────────────┐ ┌ cpu-03 ──────────────┐ │
│ │ ● healthy │ │ ● healthy │ │ ● healthy │ │
│ │ GPU ███████░ 74% │ │ GPU █████░░░ 51% │ │ CPU ███░░ 38% │ │
│ │ MEM ██████░ 22/32GB │ │ MEM ████░░ 14/32GB │ │ tasks 4 · idle │ │
│ │ tasks: summarize×3 │ │ tasks: embed×8 │ │ │ │
│ └─────────────────────┘ └─────────────────────┘ └──────────────────────┘ │
└────────────────────────────────────────────────────────────────────────────┘
┌───────────────────────── History — run #01J… ────────────────────────────┐
│ ◀ ▏▎▍▌▋▊▉█ ▶ [scrub timeline] [Replay from selected ⟲] │
│ t+0.0s WorkflowStarted input {topic:"…"} │
│ t+0.3s ActivityScheduled search │
│ t+1.1s ActivityCompleted search → 5 sources │
│ t+1.2s ActivityScheduled ×3 summarize (fan-out) │
│ t+3.0s ActivityFailed summarize[2] rate_limit(429) → retry │
│ t+4.4s ActivityCompleted ×3 summarize │
│ t+4.5s TimerStarted approval expires in 3d │
│ … WorkflowSignaled approval=approved by alice │
└────────────────────────────────────────────────────────────────────────────┘
┌──────────────────────────────── Chaos Lab ───────────────────────────────┐
│ Inject fault into: [ run #01J… ▾ ] │
│ ┌ Kill worker ┐ ┌ Drop API ┐ ┌ Network timeout ┐ ┌ GPU crash ┐ ┌ DB blip┐│
│ │ [inject] │ │ [inject] │ │ [inject] │ │ [inject] │ │[inject]││
│ └─────────────┘ └──────────┘ └──────────────────┘ └───────────┘ └────────┘│
│ Live recovery timeline │
│ 12:10:00 💥 killed gpu-07 (running summarize×3) │
│ 12:10:01 ⟳ Temporal re-queued 3 activities │
│ 12:10:03 ➕ autoscaler added gpu-09 │
│ 12:10:06 ✅ all activities completed on gpu-08/gpu-09 — recovery 6.0s │
│ Result: 0 lost state · 0 duplicated side effects · workflow on track │
└────────────────────────────────────────────────────────────────────────────┘
ancora/
├── docs/
│ └── RFC-0001-durable-ai-runtime.md
├── packages/
│ ├── sdk-python/ # ancora SDK (workflow/activity/nodes)
│ │ └── ancora/
│ │ ├── workflow.py
│ │ ├── activity.py
│ │ ├── nodes/ # built-in node library
│ │ ├── plugin/ # plugin contract + registry client
│ │ └── determinism.py # static analyzer + guardrails
│ └── cli/ # `ancora` CLI (dev, deploy, run, logs)
├── services/
│ ├── api-gateway/ # FastAPI: REST + WS
│ ├── workflow-service/ # Temporal client orchestration
│ ├── activity-workers/ # Temporal ↔ Ray bridge
│ ├── workflow-workers/ # deterministic workflow execution
│ ├── event-consumer/ # history → projections/metrics/cost
│ ├── plugin-registry/ # node catalog + sandbox policy
│ └── chaos-controller/ # fault injection
├── web/ # Next.js dashboard
│ ├── app/ # routes: dashboard, workflow, workers, history, chaos
│ ├── components/ # shadcn/ui + React Flow DAG
│ └── lib/ws/ # WebSocket client
├── deploy/
│ ├── docker/ # Dockerfiles + docker-compose.yml (single-node)
│ ├── helm/ # Kubernetes charts
│ └── terraform/ # cloud infra (v2)
├── migrations/ # Postgres schema (app DB)
├── observability/
│ ├── otel-collector-config.yaml
│ ├── prometheus/
│ └── grafana/dashboards/
├── examples/ # sample workflows (research agent, batch eval, …)
└── tests/
├── unit/
├── integration/ # Temporal + Ray in-loop
└── chaos/ # automated failure-injection suite
Three deployment tiers, identical code:
flowchart LR
subgraph Dev["1 · Local dev (`ancora dev`)"]
D1["In-process Temporal (dev server)"]
D2["Local Ray (local_mode)"]
D3["SQLite/embedded PG"]
end
subgraph Compose["2 · Single node (Docker Compose)"]
C1["Temporal + Postgres"]
C2["Ray head+worker"]
C3["API + Web + Prom/Grafana"]
end
subgraph K8s["3 · Production (Kubernetes + Helm)"]
K1["Temporal cluster (HA)"]
K2["KubeRay operator + autoscaled pools"]
K3["HPA/KEDA on all stateless tiers"]
K4["Postgres HA + Redis + object store"]
end
Dev --> Compose --> K8s
- Local dev: single command, no external deps, sub-second start — critical for adoption.
- Docker Compose: the "clone and
docker compose up" experience; full stack on one machine for demos and small workloads. - Kubernetes: Helm chart wires Temporal (Helm or Temporal Cloud), KubeRay operator for autoscaled GPU/CPU pools, KEDA for queue-driven activity-worker scaling, and the observability stack. Cloud-native, 12-factor, horizontally scalable.
| Dimension | Bottleneck | Strategy |
|---|---|---|
| Workflow throughput | Temporal history service | Shard by workflow ID (native); scale history/matching; multiple task queues |
| Activity throughput | Activity worker count | KEDA autoscale on task-queue depth; separate queues per node class (GPU vs CPU) |
| Compute | Ray cluster capacity | Ray autoscaler adds/removes nodes on pending-task pressure; distinct GPU/CPU pools; placement groups for locality |
| Large payloads | Temporal blob size limits | Offload to object store; pass references, not bytes, through history |
| Hot workflows | Single-workflow serialization (Temporal is single-writer per workflow) | Fan work out to child workflows / activities; continue-as-new to bound history |
| Read/query load | Postgres projections | Read replicas; partition node_execution by time; cache hot runs in Redis |
| Live UI fan-out | WebSocket connections | Redis pub/sub + horizontally scaled gateway; per-run channels |
| Rate-limited providers | External API quotas | Global Redis token buckets per provider/model; queue + backpressure |
Principle: the durability core scales by sharding (Temporal), the compute core scales by elasticity (Ray autoscaler), and the stateless tiers scale by replication (HPA/KEDA). No single component is a monolithic bottleneck.
flowchart TB
subgraph Edge
TLS["mTLS / TLS everywhere"]
WAF["Rate limit + WAF"]
end
subgraph Identity
OIDC["OIDC / SSO"]
RBAC["RBAC: org → project → run scopes"]
KEYS["API keys + short-lived tokens"]
end
subgraph Isolation
NS["Namespace/tenant isolation"]
SBX["Node sandboxing (gVisor/container)"]
NETPOL["Network policies (default-deny)"]
end
subgraph Data
ENC["Encryption at rest + in transit"]
SECRETS["Secrets manager (no secrets in history)"]
REDACT["PII redaction in logs/traces"]
AUDIT["Immutable audit log (event-sourced)"]
end
Edge --> Identity --> Isolation --> Data
Key controls:
- Tenant isolation via Temporal namespaces + per-project task queues + Ray runtime-env boundaries.
- Untrusted node execution runs in gVisor/container sandboxes with default-deny networking and declared resource ceilings; the shell node is always sandboxed.
- Secrets never enter workflow history — activities fetch from a secrets manager at execution time; history stores references only. This matters because history is durable and replayable, so a leaked secret would be leaked forever.
- Auditability is free: the event-sourced history is a tamper-evident audit log. Every state change, approval, and retry is recorded with actor and timestamp.
- Supply chain: plugins are signature-verified before registration; SBOM + provenance for images.
timeline
title Ancora Roadmap
MVP : Durable DAG engine on Temporal+Ray : Built-in node library (LLM/HTTP/Python/Approval) : Live DAG dashboard + replay : Chaos mode : Docker Compose deploy
v2 : Autoscaling GPU pools (KubeRay) : Declarative DAG SDK : Cost budgets + guardrails : Vector/retrieval nodes : Helm/K8s production deploy
v3 : Multi-region durability : Workflow marketplace/templates : Streaming-first LLM nodes : Fine-grained RBAC + SSO : Managed cloud (beta)
Enterprise : SOC2/HIPAA compliance : VPC/self-hosted control plane : SLA-backed multi-tenant cloud : Advanced governance + audit exports : Priority provider failover
- Durable DAG execution (branches, parallel, retries, checkpoint, resume) on Temporal.
- Ray bridge for compute; single GPU/CPU pool.
- Built-in nodes: LLM, HTTP, Python, Database, Approval.
- Dashboard: dashboard, workflow DAG, worker view, history/replay, chaos mode.
- Deterministic replay + idempotency enforcement.
- Docker Compose +
ancora dev. - OTel + Prometheus + Grafana.
- KubeRay autoscaling GPU pools; distinct queues per node class.
- Declarative DAG SDK + framework-target compilers (LangGraph adapter).
- Cost budgets, per-run/$ ceilings, provider fallback chains.
- Retrieval/embedding nodes with auto-batching.
- Production Helm chart; KEDA-driven activity scaling.
- Multi-region / DR durability.
- Workflow template marketplace.
- Streaming-first LLM UX end-to-end.
- SSO, fine-grained RBAC, org governance.
- Managed cloud beta.
- SOC2 Type II, HIPAA, ISO 27001.
- Self-hosted control plane / BYOC.
- SLA-backed multi-tenant cloud, dedicated capacity.
- Audit exports, DLP, governance policies.
Every major decision and why:
| Decision | Chosen | Rejected alternative | Why |
|---|---|---|---|
| Durability engine | Build on Temporal | Roll our own event-sourced engine | Temporal has a decade of hardening on the exact problem (deterministic replay, history, timers, signals). Reimplementing it is a multi-year distraction from our actual value: the AI/compute fusion. |
| Compute engine | Build on Ray | Kubernetes Jobs / Celery / custom | Ray uniquely offers Python-native distributed compute with GPU scheduling, actors, and object store. Celery lacks GPU/actor model; raw K8s Jobs lack fine-grained scheduling and shared object memory. |
| Bridge model | Temporal activity → Ray task | Run Ray inside workflow code | Workflow code must be deterministic; Ray calls are non-deterministic I/O. The activity boundary is the only correct seam. Violating it corrupts replay. |
| State source of truth | Temporal history; Postgres = projection | Postgres as primary state | One durability guarantee, one place. Projections are rebuildable and swappable; two sources of truth invite split-brain. |
| Idempotency | Developer-supplied/derived keys, runtime-enforced | Fully automatic exactly-once | True exactly-once for arbitrary external side effects is impossible; we give exactly-once for pure work and enforced idempotency keys for effects. Honest guarantee > magic that silently double-charges. |
| Not an agent framework | Runtime only | Ship agent ergonomics too | Owning the runtime layer is a durable moat and keeps us un-opinionated so frameworks target us. Bundling agent DSLs would make us a LangChain competitor and shrink the market. |
| Language | Python backend | Go/Rust for perf | Our users, the AI ecosystem, and Ray are Python. Determinism-critical hot paths live in Temporal (Go) already; our glue doesn't need to be. |
| Large payloads | Object-store references | Inline in history | Temporal history has size limits and replay cost scales with history size; big blobs would bloat and slow every replay. |
| UI transport | WebSocket + Redis pub/sub | Poll Temporal | Polling the durability core for live UI would add load to the critical path; a projection + pub/sub decouples read scaling from durability. |
| Human-in-loop | Durable Temporal timers/signals | Hold a thread/DB row + poller | A durable workflow can wait days at zero compute cost and survive restarts — a held thread or cron-poller cannot. |
| Risk | Likelihood | Impact | Mitigation |
|---|---|---|---|
| Operational complexity — running Temporal and Ray and the app is a lot of moving parts | High | High | ancora dev + Compose hide it in dev; Helm + managed options in prod; strong defaults; eventually a managed cloud |
| Determinism footguns — users write non-deterministic workflow code and corrupt replay | High | High | Static analyzer, runtime sandbox, clear errors, extensive docs, examples that model correct patterns |
| Two-system version skew — Temporal and Ray upgrade independently and break the bridge | Medium | Medium | Pin + integration-test the matrix; abstraction layer at the bridge; CI against version combos |
| Latency overhead — activity round-trips add ms vs. a raw in-process loop | Medium | Medium | Batch/fan-out on Ray; local caching; the durability win dwarfs the overhead for the target workloads |
| Ray head SPOF | Medium | High | GCS fault tolerance + Redis-backed HA; graceful degradation; workflows survive via Temporal even if Ray blips |
| Adoption — "why not just Temporal or just Ray?" | Medium | High | Lead with the demo: kill a worker mid-agent, watch it recover. The chaos UX is the pitch |
| Cost of GPUs in demos/CI | Medium | Low | CPU-only default nodes; GPU tests gated; mock LLM provider for CI |
| Scope creep toward agent framework | Medium | Medium | Charter discipline: we ship the runtime, not the agent DSL |
- Determinism enforcement strength — how aggressive should the static analyzer be? Hard-fail on any suspicious call, or warn-and-sandbox? Too strict frustrates; too lax corrupts state.
- Ray-inside vs. Ray-adjacent — should activity workers be Ray actors (co-located) or external clients dispatching to Ray? Co-location cuts a hop but couples lifecycles.
- Result caching semantics — where does the pure-activity result cache live (Ray object store vs. Redis vs. object store), and what is its TTL/eviction policy under replay?
- Multi-tenancy depth — namespace-level isolation for MVP vs. full per-tenant Ray clusters? Cost vs. isolation tradeoff.
- Workflow versioning UX — how do we present Temporal's
patched/build-ID versioning to users authoring in a DAG DSL without leaking Temporal internals? - Streaming through durable boundaries — LLM token streams are inherently non-durable; how much of a stream do we persist vs. treat as ephemeral UI-only?
- Cost attribution accuracy — provider token counts vs. our estimates; how do we reconcile, and who's authoritative for budget enforcement?
- Backpressure policy — when Ray is saturated and Temporal keeps scheduling activities, what's the queuing/rejection/priority model?
- Time-travel debugger — step through a workflow's history in the UI, inspect state at any event, and fork a replay from any point with modified inputs.
- Speculative execution — pre-warm likely branches (e.g. both sides of a conditional) on idle GPU capacity, discard the loser; trade cost for latency.
- Workflow-as-a-tool — expose any durable workflow as a callable tool to an LLM, so agents can invoke durable sub-workflows.
- Cross-workflow memory — durable, queryable long-term memory store as a first-class node backed by the object store + vector index.
- Auto-compensation synthesis — infer saga/rollback steps from node declarations so failure compensation is generated, not hand-written.
- Deterministic LLM replay — record provider responses in history so a replay reproduces the exact model outputs for debugging non-deterministic agents.
- Budget-aware scheduler — the scheduler treats
$andGPU-secondsas first-class constraints, degrading model quality to stay within budget. - Multi-language SDKs — TypeScript and Go SDKs targeting the same runtime, so non-Python teams can author workflows.
- Federated / edge execution — run compute nodes on edge/on-prem workers while durability stays in the cloud control plane.
End of RFC-0001. This document is intended to guide implementation of a production-grade, open-source durable AI workflow runtime. Sections are versioned independently; propose changes via follow-up RFCs (RFC-0002+).