Temporal Order Lifecycle

An Order → Payment → Shipping workflow on Temporal's open-source SDK and dev server. Postgres holds the state, FastAPI drives it. Every business step calls the take-home's flaky_call(), which randomly throws or sleeps for 300 seconds, and the whole thing still finishes in about two seconds. The 15-second limit ended up with plenty of room to spare, though getting there took some digging (design notes at the bottom).

Architecture

docker compose
├── temporal          open-source dev server (gRPC :7233, Web UI :8233)
├── postgres          application DB: orders, payments, events
├── worker-order      polls "order-tq"     -> OrderWorkflow + its activities
├── worker-shipping   polls "shipping-tq"  -> ShippingWorkflow + its activities
└── api               FastAPI: start / signals / status (:8000)

OrderWorkflow (order-tq)

receive → validate → [manual-review gate] → charge → ShippingWorkflow → mark shipped

• Signals: cancel_order, update_address, approve_order, dispatch_failed.
• The manual-review gate is a durable timer racing the human approve_order signal (workflow.wait_condition(..., timeout=…)). Approve and it moves on; let the timer win and the order cancels.
• Cancelling before shipment is honored. If payment already went through, the terminal state records the refund compensation.
• status() is a query that returns the live step, the flags, the shipping attempt count, and the last error.

ShippingWorkflow (shipping-tq) is a child workflow on its own task queue.

prepare_package → dispatch_carrier

If dispatch_carrier runs out of retries, the child signals the parent with dispatch_failed(reason) and fails. The parent then restarts the child up to MAX_SHIPPING_ATTEMPTS before giving up and compensating.

Two workers on two queues is the whole point of the queue-isolation requirement. The child physically runs in a different process, so shipping could be scaled or deployed on its own.

Quick start

All you need installed is Docker with Compose. There's no Python, local Postgres, or Temporal CLI to set up.

Start the whole stack with a single command. The first run builds the image and pulls Temporal and Postgres, so give it a minute; it brings up five services — the Temporal server, Postgres, the two workers, and the API.

docker compose up -d --build

Give it a few seconds to settle, then confirm it's ready. Wait for temporal and postgres to report healthy in the first command, and for the API to answer the second.

docker compose ps
curl localhost:8000/healthz     # {"status":"ok"} once the API is up

Now run your first order. Passing auto_approve lets it clear the manual-review gate without a second call, so it runs start to finish on its own.

OID=demo1
curl -X POST localhost:8000/orders/$OID/start \
  -H 'content-type: application/json' \
  -d '{"payment_id":"pay-'$OID'","auto_approve":true}'

sleep 3
curl localhost:8000/orders/$OID/status | python3 -m json.tool

You should see "run_status": "COMPLETED" and "step": "completed". That order just ran receive → validate → approve → charge → ship, surviving the random failures and hangs baked into every step, in a couple of seconds.

There are three ways to watch what actually happened, which is where the observability lives. The Temporal Web UI at http://localhost:8233 is the clearest: open the workflow order-demo1 and read its event history top to bottom, including every activity, every retry, and the child shipping workflow linked off it. For a live view, docker compose logs -f worker-order worker-shipping streams the retries, signals, and state changes as they happen. And the GET /orders/{id}/status endpoint you just called reports the current step alongside live per-activity retry counts and recent errors.

When you're done, docker compose down stops everything (add -v to also wipe the Postgres data).

Try the signals (manual review, address, cancel)

Start an order without auto_approve and it parks at the manual-review gate, waiting for a human:

OID=demo2
curl -X POST localhost:8000/orders/$OID/start \
  -H 'content-type: application/json' -d '{"payment_id":"pay-'$OID'"}'

curl localhost:8000/orders/$OID/status          # step == awaiting_review

# change the address while it waits, then approve it:
curl -X POST localhost:8000/orders/$OID/signals/address \
  -H 'content-type: application/json' -d '{"address":{"city":"Austin","zip":"78701"}}'
curl -X POST localhost:8000/orders/$OID/signals/approve   # proceeds to payment + shipping

# ...or cancel instead of approving:
curl -X POST localhost:8000/orders/$OID/signals/cancel \
  -H 'content-type: application/json' -d '{"reason":"changed mind"}'

Prefer clicking to curl? The API has interactive docs at http://localhost:8000/docs — every endpoint above has a "Try it out" button.

API

Method & Path	Description
POST /orders/{id}/start	Start OrderWorkflow. Body: {"payment_id": "...", "auto_approve": false}
POST /orders/{id}/signals/approve	Approve the manual-review gate
POST /orders/{id}/signals/cancel	CancelOrder. Body: {"reason": "..."}
POST /orders/{id}/signals/address	UpdateAddress. Body: {"address": {...}}
GET /orders/{id}/status	Live state: current step, run status, per-activity retry counts, last errors
GET /healthz	Liveness

GET /status stitches together two views: the workflow's own status() query and Temporal's describe(). So you get the logical step and the live pending-activity retry counts and failures from a single call.

Persistence

The schema lives in db/init.sql and runs automatically the first time Postgres boots:

• orders(id, state, address_json, items_json, created_at, updated_at)
• payments(payment_id PRIMARY KEY, order_id, status, amount, created_at)
• events(id, order_id, type, payload_json, ts), an append-only audit trail for debugging.

Payment idempotency is the interesting bit here, and it lives entirely in the database. payment_id is the primary key, and the charge runs INSERT ... ON CONFLICT (payment_id) DO NOTHING RETURNING payment_id (app/db.py::upsert_payment). If the row is already there, the charge is a duplicate and we skip it. The key collision decides it, not any bookkeeping in the app, which is what makes a retried charge safe.

One ordering detail matters: every business function calls flaky_call() before it writes to the DB. So a try that fails, times out, or gets cancelled never leaves a half-written row behind.

Poke at the state directly:

docker exec trellis-postgres-1 psql -U trellis -d trellis -c \
  "SELECT id, state FROM orders ORDER BY updated_at DESC LIMIT 10;"

Tests

docker compose up -d                     # postgres + temporal must be running
docker compose run --rm -v "$PWD":/work -w /work api python -m pytest -q

tests/test_workflows.py covers the control plane on Temporal's time-skipping test server, with fake activities standing in for the real ones: happy path, cancel, manual-review timeout, and the child dispatch-failure that bubbles back up to a parent retry. tests/test_idempotency.py hits real Postgres and proves the payment upsert charges once and no-ops on duplicates.

Meeting the 15-second constraint (design notes)

This was the actual puzzle. Six activities run in sequence, and each one fails or hangs roughly two times out of three. Two things came out of measuring it.

First, flaky_call() as written had a bug. The sleep(300) branch was indented under the raise, so it was dead code and the timeout path never ran at all. The obvious intent is three roughly-equal outcomes, so I fixed the indentation in app/business.py and left a comment saying so. Nothing else changed, and every step still calls it.

Second, and this is where the time went, the dev server is slow in two ways I didn't expect. Reading the event history showed that server-side start_to_close took about 1.8s to give up on a 300-second hang, and each Temporal-level activity retry cost around 0.8s in round-trip overhead. Do the arithmetic on ~18 retries and you're already over budget before anything useful happens.

The fix is two levels of retry. The fast one lives inside the activity (app/resilience.py): each try wraps the business call in asyncio.wait_for, so a hang gets cancelled in ~250ms (DB writes take about 5ms, so there's no real risk of cutting off good work) and there's no Temporal round-trip on the common failures. Temporal's own retry policy stays on top as the outer net, for worker death or the rare case where the inner loop exhausts itself. Every inner try gets logged, so you can still see the retries happening. And the workflow run timeout is pinned to 15s at start, so Temporal enforces the requirement rather than me just hoping.

After that, 8 out of 8 runs completed, averaging about 1.7s, worst case 3.5s.

Observability

The Web UI on :8233 gives you the full event history per workflow, child workflows and activity attempts included. docker compose logs -f worker-order worker-shipping streams structured logs: each activity's Temporal attempt, every inner retry, the signals, and the state transitions. And GET /status gives the live step plus per-activity retry counts and last errors without leaving the terminal.

Notes and shortcuts

I used temporalio/temporal server start-dev, which bundles the server and the Web UI in one image, instead of the FAQ's temporalio/auto-setup that wants a separately wired datastore. Same open-source server, much less to set up.

Worth being clear about the two databases: the Temporal dev server keeps its own internal store for workflow history, and the postgres service here is purely application state (orders, payments, events). They're unrelated.

The python:3.12-slim base image ships with some system-library CVEs. For a local exercise I left those alone.

Repo layout

app/
  config.py           connections, queue names, timeouts (single source of truth)
  db.py               asyncpg pool + persistence helpers (idempotent upsert)
  business.py         flaky_call + the 6 stubs, wired to the DB
  resilience.py       in-activity retry loop (the 15s fix)
  activities.py       thin Temporal activities
  workflows.py        OrderWorkflow + ShippingWorkflow (all the primitives)
  workers/            order_worker.py, shipping_worker.py (two queues)
  api.py              FastAPI control surface
db/init.sql           schema / migration
tests/                workflow + idempotency tests
docker-compose.yml    the whole local stack