CommitsQuorum 3/5 ✓
Write to Majority
Commits
Three of five nodes acknowledge the entry. The transition commits durably into immutable history.
Systems Architecture & Product Engineering
Architecting systems that scale & endure.
Product systems for fintech, identity, and operations.
Deterministic State MachinesExplainable ArchitectureVerifiable Correctness
Majority Quorum 3 of 5
State Durable- Leader NodeConsensus Coordinator
- Replica AQuorum Participant
- Replica BQuorum Participant
Minority Partition 2 of 5
Writes Stalled- Replica CPartitioned Node
- Replica DPartitioned Node
UnknownQuorum 2/5 ✗
Write to Minority
Unknown
Two nodes cannot form quorum. The state engine stalls rather than presenting an unverified illusion of success.
- Product systems
- Fintech
- Identity
- Operations
- Payments
- Cloud delivery
- Reliability
- Observability
Selected work
Each system had one thing it could not get wrong.
Property operations, identity and payments, and reconciliation. In each one the work was keeping a single version of the truth across product, services, and the release that carries them.
What happens when a release goes out
Build
The artifact is signed and versioned.
Canary
One percent of traffic takes the change first.
Rollout
Half the fleet is on the new version right now.
Health gate
Reads the traces, metrics, and logs the deploy already emits.
On regression the gate halts the release and returns it to build. The rollout stays half-finished and visible until someone decides.
One product. One operating truth.
A multi-surface property platform: billing, maintenance, people, devices, releases. The work was keeping one operating truth across product and infrastructure.
- The constraint
- Keep several mobile roles and backend services coherent while releases, live events, and operational visibility move underneath them.
- The decision
- Put release health in the product path: blue/green rollout, health checks, traces, metrics, and logs ship with the change instead of after an incident. Operators see the truth the deploy pipeline already has.
Shipping is a moment. Knowing whether the change is healthy is the work.
What happens when the provider answers
Provider responds
A redirect moves the browser. It proves nothing about the payment.
Record the transition
Every external answer becomes an event with an identity, so applying it twice changes nothing.
Reconcile
The user view is checked against the business record before anything moves.
Then the journey continues on one of two paths
- VerifiedValue may move.
- UnknownWaits for a query or a callback. Not a failure.
Verify before value moves.
Identity and payment flows where provider callbacks, retries, and user status must agree before value moves.
- The constraint
- Keep external responses, user status, and business state aligned through delayed answers, retries, redirects, and partial failure.
- The decision
- Never call a redirect success. Record each external response as a state transition; reconcile the user view with the business record before the next action. Timeouts stay unknown until a query or callback closes the loop.
A redirect changes a browser. Verification changes the record.
Where a number comes from
Command accepted
A return, a split payment, a stock adjustment, an actor and a reason.
Event appended
Written once, in order, and never edited. Five events here; the same shape at any number.
Projection
The balance or stock total you actually read is rebuilt from those events.
Audit trail
Who moved it, why, and when — kept beside the projection, not reconstructed after a complaint.
If the total and the story disagree the events are the tiebreaker. The projection is rebuilt; the history is never rewritten.
Every number deserves a history.
A history-first model for stock, balances, returns, and split payments. Every number keeps a traceable reason.
- The constraint
- Preserve an explainable history for every stock, balance, return, adjustment, and split-payment change.
- The decision
- Choose an explainable history over mutable convenience. Summaries can be rebuilt; the events that justify them must stay intact. Audit trails are product requirements, not afterthoughts for compliance season.
The total is an answer. The history makes that answer believable.
Operating truths
Rules that cost something to learn.
01
State and invariants
Name what must stay true before you write the code.
Deterministic software starts with explicit boundaries: balances reconcile, identity can be re-proven, and state does not drift.
02
Commerce and trust
Certainty in the interface is not certainty in the record.
A callback is a message, not a settlement. Read the provider state before it is allowed to move a balance, and keep the screen honest about which of the two it is showing.
03
Release and drift
Shipping without evidence is just hoping in production.
Release health belongs in the pipeline. Canary gates and traces ship with the code, not after an incident.
Practice
What has to stay true, and what checks it.
Each discipline below is one rule I hold to. No benchmarks, no measured numbers — just the thing that has to stay true when a gateway answers late, a phone loses signal, or a release goes sideways.
What has to hold in the product
01 — Integrations
Reliable integrations and fault tolerance
Webhooks arrive late, twice, or not at all. Each one is an event with an identity, and applying it twice does nothing.
02 — Client architecture
Offline-first state synchronisation
A phone on a weak connection is still a correct client. Local state reconciles to the server deterministically on reconnect.
03 — System boundaries
Boundaries that survive change
Dependencies point inward. The domain never imports a framework, so the rules outlive the tools.
04 — Storage
A data layer that stays fast under load
Index for the query you actually run, and partition before the table is the problem rather than after.
05 — Quality
Guardrails that catch regressions
A change that breaks a promise fails the build. The pipeline is where you find out, not the customer.
06 — Security
Least privilege, verified at the edge
Nothing mutates state until its signature is verified. Trust is a check that runs, not a header that is trusted.
Field notes
Writing about the parts that bite.
01
02
03