Skip to content

feat: state backplane — clustered StateAppEvents/StateApp/StateSess (event-log model)#93

Merged
joaomdsg merged 40 commits into
mainfrom
design/state-backplane
Jun 5, 2026
Merged

feat: state backplane — clustered StateAppEvents/StateApp/StateSess (event-log model)#93
joaomdsg merged 40 commits into
mainfrom
design/state-backplane

Conversation

@joaomdsg

@joaomdsg joaomdsg commented Jun 5, 2026

Copy link
Copy Markdown
Member

Makes server-owned reactive state durable and convergent across N processes without prescribing infrastructure — closing the two long-standing non-goals (cluster runtime + restart survivability). Purely additive: a nil backplane is byte-for-byte the existing single-pod behavior.

What it adds

  • StateAppEvents[E,V] — an event-log-backed sibling of StateApp: append immutable events, declare a pure Fold, read the projection. Per-(pod,key) projector is the sole fold path; cross-pod convergence by tailing the log.
  • Backplane seam (Store CAS-cell + ordered resumable EventLog + optional Compactor) with via.InMemory() as the base impl nil resolves to — so the interface is exercised on every single-pod run.
  • vianats — NATS JetStream reference backend (KV + stream), passes the parameterized conformance suite (release-gating).
  • Clustered StateApp/StateSess — Store is the source of truth; pods converge via a changes feed + periodic reconcile sweep; full-sid fail-closed isolation. Sticky sessions no longer required for state correctness.
  • Versioning — self-describing envelope + decode-only upcaster chain, drop-on-undecodable, forward-incompat halt, epoch/offset-space-reset detection.
  • Snapshots + compaction — disposable cache for uncompacted keys; durable-genesis + seeded migration for compacted keys; consumer-aware + monotonic floor.
  • OnEvent side-effect consumers (at-least-once, offset-committed, idempotency-key).
  • GDPR crypto-shredKeyStore + per-subject AES-256-GCM payloads + EraseDataSubject (key drop + snapshot gen-invalidation; compacted+erasure halts rather than truncates).
  • WithFoldVerify — runtime fold-determinism gate that refuses to compact a non-deterministic fold; per-fold divergence canary.
  • Reconnect-rehydrate — projector + consumer re-subscribe on a transient drop, exit on graceful Shutdown.

Hardened by a multi-node load benchmark

The load test surfaced a real cross-pod projection-truncation bug (a fast pod compacting the shared log silently truncated a lagging peer). Fixed: a lagging projector re-seeds from the snapshot on a compaction gap (or halts, never diverges), and snapshot writes are monotonic in CoveredOffset. Verified: convergence-with-compaction hammered 90× + -race.

Verification

  • Full suite + backplanetest conformance + vianats green under -race (both modules); go vet clean.
  • Parameterized backplane conformance, adversarial memevents.Faulty (redelivery/disconnect/reorder), two-pods-one-backplane convergence keystone, fuzz + subprocess fold-purity gates.

Docs

design/state-backplane.md (spec, DECIDED), design-council.md (full deliberation, 46 ticks), operator metrics catalogue + load profile in docs/production.md, new-adopter pass on docs/why-via.md.

Residuals are documented and fail-safe (eventual read-your-write, live-projection erasure clears on cold-start, compaction×erasure halts pending snapshot re-encryption, value-tailer reconnect symmetry).

joaomdsg added 30 commits June 3, 2026 23:38
Add design/state-backplane.md: a scoping/design proposal for cluster
fan-out + restart survivability as an ADDITIVE feature, not a core/API
rewrite. Value state (StateApp/StateSess) keeps its API and rides a durable
CAS Store + a value-less Change on an ordered Log; a new opt-in sibling
StateAppLog[E, V] models high-churn shared state as append-events-and-fold,
killing the hot-key CAS retry-storm. One Store + Log (+ optional Compactor)
+ Codec interface backs NATS JetStream / Redis Streams / Postgres / Kafka.

Captures the recommended interfaces (godoc-load-bearing), resolutions to all
seven open questions, the residual fold-determinism risk, a terse
four-architect comparison, and a seven-phase additive rollout. Derived from
two multi-architect design passes; supersedes the earlier lossy-Bus sketch
(see "Design evolution").
…2-GO-4)

Design-council ticks 5-6 over the converged Via state backplane design.

design/state-backplane.md:
- Tick 5: apply the naming-council rename — StateAppLog -> StateAppEvents,
  Log interface/concept -> EventLog (Backplane = Store + EventLog), chat field
  Log/r.Log -> Events/r.Events, metrics via.log.* -> via.events.*. Bare "Log"
  left only on the lines describing today's StateAppSlice chat (current code).
- Tick 6: reconcile the body with the converged T2-GO-4 resolution. Fixed 4
  places that asserted "snapshot = pure cache, evolving V free, no snapshot
  upcasters" with no compacted-key caveat. The snapshot is a disposable cache
  only for uncompacted keys; once a key compacts the deleted prefix is
  unrecoverable, so the snapshot is durable genesis state (typed Codec[V],
  versioned Checkpoint, seeded migration on mismatch, retained-event floor,
  WithFoldVerify mandatory before compaction).

design-council.md: the council's running record (ticks 1-6: convergence at
tick 4, in-mem-backplane and naming addenda, then the doc-reconciliation ticks).
…-path, projector)

Council ticks 7-9 propagating the converged decisions from design-council.md
into design/state-backplane.md, where the body still described the pre-council
design in several load-bearing places.

- Tick 7 (T1-GO-2/T2-GO-4): replace the untyped Codec (Encode(any)/Decode()any)
  with the generic Codec[T] — event Codec[E] + version-tagged snapshot Codec[V];
  kills the public any and carries the seeded compacted-key migration.
- Tick 8 (T3-SRE-1/T1-SRE-5/T4-SRE-1): value (StateApp/StateSess) state is no
  longer pod-local. The durable Store cell is the single source of truth, the
  per-pod kvStore is an L1 cache; every pod incl. the writer re-pulls to Store
  HEAD on each Change (gated storeRev >= change.rev, monotone L1 gate) backed by
  a periodic reconcile sweep so correctness never depends on a notify. Removes
  the sticky-session requirement for state. Decision bullet + Phase 3 rewritten.
- Tick 9 (T1-SRE-1/T1-SRE-2): the per-(pod,key) projector is the sole fold path
  on every pod incl. the writer. Append no longer folds or renders locally;
  broadcastRender is the intra-pod leg, the projector tailing EventLog.Subscribe
  is the inter-pod leg. Cross-pod read-your-write is eventual (WaitFor for
  strict). nil-ctx panic reframed as the AUTH gate, not the render driver.

design-council.md records ticks 5-9 (the rename + the four reconciliation steps).
…convergence

Council ticks 10-17. Completes propagating the converged decisions into
design/state-backplane.md and audits the whole doc for internal consistency.

Reconciliation (ticks 10-12):
- T1-GO-1: drop E.Zero(); fold seed = `var zero V` (kills the pointer-E
  nil-panic), across the EventReducer interface, Read, both examples, cold-start.
- T1-SEC-1/2: add standalone Multi-tenant & session isolation and GDPR
  crypto-shred sections (physical namespace + mTLS boundary, full-sid
  exact-match fail-closed, AUTH-1, tier-0 trust; KeyStore DropKey ->
  ErrUndecodable, snapshot invalidation, audit-class vs PII-class).
- header Status -> CONVERGED; nil backplane resolves internally to
  via.InMemory()/memevents.Backplane (one code path); Phase 1/2 in-mem-default
  shift (memevents.Faulty + parameterized conformance, release-gating real NATS).

Coherence audit (ticks 13-17, rounds 1-5):
- Store godoc: disambiguate the two source-of-truth roles (val:<key> cell vs
  the EventLog + its snapshot) so the value/log SoT statements don't conflict.
- T1-GO-3: Offset end-to-end on Append/AppendIf/ReadAt/OnEvent (no bare uint64).
- v1-surface demarcation: mark AppendIf/ReadAt/OnEvent as Advanced (post-v1).
- T1-SRE-3: Epoch newtype on the delivered Record and the Head return
  (offset-space-reset detection).
- T1-DX-2: StateAppCounter specialization replaces the raw counter example.
- declare ErrForwardIncompatible and ErrEpochUnbridgeable (were referenced but
  undefined).

design-council.md records ticks 10-17; the design deliberation converged at
tick 17 (a full coherence sweep found nothing new).
First implementation slice of the state backplane. Purely additive type
detection + wire-key binding — nil backplane / zero observable change.

- StateAppEvents[E EventReducer[E,V], V any] handle: holds the wire key,
  binds via the existing scopeBinder/bindWireKey path, carries its own
  distinct isStateAppEvents() marker so the walker can later start a
  per-key projector only for log keys.
- EventReducer[E,V] constraint (Fold(acc V, ev E) V) with the determinism
  contract documented.
- walker: roleStateAppEvents + cached marker type + isStateAppEventsType +
  classifyField branch, folded into the existing scope-slot case.

Read/Update/projector/app-binding are deferred to Phase 1 (their first
consumer); scopeKind + bindApp land there too.

Tests: default-key + via-tag override through the real Mount->render path;
the pair defeats any hardcoded Key(). go test -race ./... green.
The base in-process Backplane the design's nil-backplane resolves to. A
standalone, race-tested unit — no via-runtime wiring yet.

- backplane.go: public contract — Offset/Rev/Epoch, Record, Store
  (LoadSnapshot/CAS), EventLog (Append/Subscribe/Head), Backplane
  (Store + EventLog + io.Closer), InMemory(), ErrCASConflict/ErrClosed.
- inmemory.go: per-key CAS cell + per-key append log. Subscribe replays
  Offset>from in per-key total order then live-tails via a broadcast
  channel closed+replaced under the log mutex on each append; the
  subscriber goroutine unwinds on ctx-cancel or Close. Append re-checks
  the per-log closed flag under its mutex so an Append racing a
  concurrent Close cannot commit a lost write.

Guarantees covered by tests (-race): monotone per-key offsets, Head
high-water-mark, resumable+gap-free Subscribe with a real live-tail
block, per-key independence, CAS conflict semantics, ctx-cancel unwind,
Close (ErrClosed + subscriber channels close + idempotent), record-byte
ownership.

Design: records T1-GO-6 — the base in-memory impl lives in package via
(not a separate memevents package) to avoid a via<->memevents import
cycle; memevents stays the Phase 2 home for the Faulty decorator +
conformance suite. AppendIf deferred (post-v1, with StateAppEvents.AppendIf).
App-level dependency injection for the state backplane.

- config.go: WithBackplane(b Backplane) Option + a config field. A nil
  backplane is the documented default.
- app.go: New() resolves the config backplane, defaulting nil to
  InMemory(), so the runtime always drives one Backplane code path
  (no nil-special-case).
- server.go: App.Shutdown gracefully drains the backplane via Close()
  after disposing ctxs and stopping sweeps.

Test: a WithBackplane'd backplane returns ErrClosed after Shutdown
(proves it was stored and drained); a default app shuts down cleanly
with the new drain step. The nil->InMemory resolution itself becomes
black-box observable once Read/Append on the handle land (P1.1b).
… 1.1b)

The heart of Phase 1: StateAppEvents now works end to end in-process.

- Append (stateappevents.go): json.Marshal(ev) -> backplane.Append. No
  local fold, no broadcast — it only commits the fact. Panics on nil ctx
  (the AUTH gate, parity with StateApp.Update).
- Per-(pod,key) projector (applog.go): the SOLE fold path (T1-SRE-2).
  Tails backplane.Subscribe from genesis, folds each record (decode E +
  E.Fold) into a cached projection in offset order, then broadcastRenders
  to subscribed tabs. Started once per key (sync.Once); exits when the
  Subscribe channel closes on Shutdown.
- Read: returns the cached projection (O(1)) and subscribes the ctx via
  trackRead, identical to StateApp.Read. Text mirrors StateApp.Text.
- Binding seam completed: scopeSlot gains a kind flag (value vs log);
  bindScopeKeys binds the App into log handles via a new appBinder; newCtx
  threads the *App. The typed bindApp method registers a seed + fold
  closure with the App, bridging the reflection-detected, type-erased
  field to the generic E/V fold (T1-GO-8) — the App holds only `any`.

Tests (-race): empty log projects zero V; an appended event folds and
reaches a live SSE subscriber; the projection is app-scoped and outlives
the writer (two appends -> "hello,hello", also catching double-fold);
Append panics on nil ctx; Text renders the projection.

Codec is plain encoding/json for now; the versioned Codec envelope is
Phase 4. AppendIf/ReadAt/OnEvent remain post-v1.
The shipped specialization for a monotonic shared counter: just Inc +
Read, no user-defined event type or Fold.

- StateAppCounter embeds StateAppEvents[counterTick, int64], reusing the
  same embed-and-promote path as StateAppNum/StateApp — the walker detects
  and binds it with no walker/runtime change.
- counterTick is unexported (Fold = +1), so Inc is the only append path:
  a caller cannot name the event to Append it directly.
- Unlike StateApp[int] (a CAS cell that retry-storms under churn),
  increments are conflict-free appends; the value is the fold.

Tests (-race): reads 0 before any Inc; three Inc calls accumulate to 3
for a fresh session (app-scoped, exact +1 fold); an Inc reaches a live
SSE subscriber; Inc panics on a nil ctx.
The chat example now models its message log as an event log instead of a
value-shaped StateAppSlice with a read-modify-write trim.

- Room.Log is via.StateAppEvents[Posted, []Message]. Send appends an
  immutable Posted event; the pure Fold derives (and bounds) the rendered
  list. The old Op.Append + trim-Update read-modify-write is gone.
- Fold copies acc before appending (replay determinism) and trims to
  recentWindow; the event log itself is bounded later by snapshot +
  compaction, not by rewriting history.

Behavior is preserved (live cross-session fan-out, bounded render). The
existing fan-out test stays green; a new test proves messages accumulate
in send order in a fresh reader's view (the fold appends, app-scoped).

Closes Phase 1 of the state backplane: the in-process StateAppEvents core
ships green and the flagship example demonstrates it.
A reusable, backend-agnostic contract suite any Backplane implementation
must pass — the executable spec for ordering, resumability, CAS, and
lifecycle.

- New package backplanetest with RunConformance(t, newBackplane). Nine
  subtests: increasing-and-Head-tracking offsets (empty Head = 0), genesis
  replay in order, resume strictly after a returned offset, live-tail,
  per-key independence, concurrent appends get distinct offsets with Head
  at the max (the conflict-free total-order guarantee), independent
  subscribers see the same stream, CAS must-not-exist/conflict/advance,
  Close -> ErrClosed.
- Offsets are read back from Append and asserted only as
  strictly-increasing / resumable, never hardcoded — so a non-dense
  Kafka/JetStream/Postgres backend conforms.
- TestInMemoryConformance runs it against via.InMemory(); the suite will
  also gate the Faulty decorator (2b) and the NATS backend (2d).

Lives in a normal (non-_test) file like fstest/iotest so external backend
packages can import it.
…e conformance (Phase 2b)

Proves the Backplane contract (and the conformance suite) is robust to the
at-least-once redelivery a real network log exhibits.

- memevents.Faulty wraps any via.Backplane and re-emits each Subscribe
  record Redeliver+1 times, in order, via one goroutine that exits on the
  underlying channel close or ctx cancel (no leak). Store/Append/Head/Close
  delegate through the embedded interface.
- backplanetest.RunConformance is now at-least-once-tolerant: a
  collectDistinct helper dedupes by offset and asserts first-delivery in
  per-key order, so the SAME suite gates exactly-once (InMemory) and
  redelivering backends (Faulty, and later NATS) alike. This corrects an
  over-strict exactly-once assumption in the Phase 2a suite (T1-TEST-2).

Tests: conformance passes against Faulty{Redeliver:1}; redelivery actually
duplicates; Redeliver:0 is exact passthrough; the wrapper unwinds on ctx
cancel; Subscribe propagates the underlying ErrClosed.
…hase 2c)

The test that proves the projector — not a local fold — is the cluster
mechanism. Two independent via.App instances ("pods") share one
via.InMemory() backplane:

- An event appended through pod A's StateAppEvents surfaces, folded, in
  pod B's live SSE frame — B's per-key projector tails the shared log.
- After appends on both pods, a fresh reader on either pod converges to
  the same identically-ordered folded feed.

A local-fold-in-Append implementation would pass every single-pod test
yet fail here (A's projection would update but B would never see it),
because Append does no broadcast and each App's broadcastRender only
reaches its own ctx registry. Two Apps sharing one in-memory backplane is
the infra-free stand-in for two pods sharing one NATS — validating the
design's central correctness claim (#3/#7 closed by resumable Subscribe,
no census).
… (T-GO-9)

Phase 3 rewrites the core value-path (StateApp/StateSess), so this is a
convergence + design-feedback tick before the build.

- T-GO-9: making the backplane Store cell (val:<key>) the source of truth
  means StateApp[T]/StateSess[T] values serialize (default JSON) on
  Update->CAS, since Store moves []byte. The L1 kvStore keeps the live T
  so reads stay zero-serialization. Documented at the Store godoc: the
  "v0.4.0 byte-for-byte" promise is identical observable behavior for
  serializable T (the universal shared-state case); a non-serializable T
  in a pod-local kvStore is the one narrow break.
- Recorded the P3 sub-slice plan: P3a StateApp cross-pod value convergence
  (Store CAS + L1 read-cache + value-less Change feed + re-pull tailer,
  existing single-pod suite the green guard), P3b periodic reconcile sweep,
  P3c StateSess with full-sid Change matching.

Phase 2 infra-free work (conformance, Faulty, cross-pod keystone) is done;
the NATS reference backend (2d) remains blocked on a maintainer decision.
… harness (Phase 2d-1)

Maintainer green-lit the NATS reference backend, as a nested module to keep
the core via module dependency-free.

- New ./vianats nested module (own go.mod, replace => ../). Deps: nats.go
  v1.52.0, nats-server v2.14.2. Core via go.mod is untouched, so via
  consumers never pull NATS.
- Embedded JetStream test harness (startEmbeddedJetStream): an in-process
  JetStream server on a random port with a temp store, shut down at test
  end — conformance needs no external container or CI service.
- Smoke test proves the primitives the backend rests on work in-sandbox:
  KV create + CAS-by-revision + Get, and stream publish (monotone sequence)
  + subject-filtered ordered consume.

Next: implement the vianats.Backplane (Store over JetStream KV, EventLog
over a subject-per-key stream) and gate it with backplanetest.RunConformance.
… RELEASE-GATING)

The real ordered/durable/resumable reference backend, proving the Backplane
abstraction holds against a network log — not just the forgiving in-memory one.

- vianats.JetStream(nc, WithPrefix): Store over a JetStream KV bucket
  (LoadSnapshot=Get; CAS = Create for rev 0 else Update(expectedRev), both
  conflict modes mapped to via.ErrCASConflict via ErrKeyExists), EventLog
  over one subject-per-key stream (Append=Publish->Sequence, Head=
  GetLastMsgForSubject, Subscribe=OrderedConsumer with DeliverAll or
  DeliverByStartSequence(from+1) drained into a channel that unwinds on
  ctx-cancel or Close). Close marks closed + stops live subscriptions;
  it does not close the caller's connection. sanitize() maps arbitrary
  wire keys to safe, collision-free subject/KV tokens.
- backplanetest.RunConformance runs against a fresh uniquely-prefixed
  backplane per subtest on an embedded nats-server: all nine guarantees
  (ordering, gap-free resume, live-tail, per-key independence, concurrent
  appends, multi-subscriber agreement, CAS conflict, Close) pass -race.

Closes Phase 2: the backplane is proven across in-memory, fault-injected,
cross-pod, and a real NATS backend under one contract suite (#3/#7 closed).
The vianats nested module keeps the core via module dependency-free.
…e 3a)

StateApp[T] keeps its API but the backplane Store cell val:<key> is now
the single source of truth, so two pods sharing a backplane converge —
while single-pod behavior stays byte-for-byte identical.

- Update is a compare-and-swap retry loop on the Store cell (LoadSnapshot
  -> decode -> fn -> Marshal -> CAS, retrying on ErrCASConflict): no lost
  updates across ctxs or pods, no shared-cell mutex. On success it sets
  the shared per-pod L1 cell synchronously (so every session on this pod
  sees it at once, as before) and appends a value-less Change{key,rev}
  liveness hint to a shared feed.
- A per-App changes-tailer re-pulls the named Store cell to HEAD on each
  Change, gated storeRev>=hint.rev (drop stale replica reads, T1-SRE-5)
  and storeRev>l1Rev (monotone, redelivery/out-of-order safe, T3-SRE-1).
- Read hits the L1 cell (O(1), zero-serialization). A silent (sync-off)
  Update writes the Store but suppresses the hint, so SyncOff still fans
  nothing out (T3-SRE-2).
- appStore is gone; bindScopeKeys now binds the App into any scope handle
  that needs it (value + log), and the dead scopeKind flag is removed.

T must be JSON-serializable once a backplane is in play (T-GO-9; nil
resolves to InMemory, so single-pod too) — the universal case for shared
state. Tests: the existing StateApp suite stays green (byte-for-byte) +
a two-Apps cross-pod convergence test + an internal test locking the
stale-drop / monotone / undecodable-snapshot reconcile gates. -race green.
Makes the changes feed a pure latency optimization: each pod periodically
re-pulls its registered value keys to the Store HEAD, so it converges even
when no Change hint reached it — a pod that joined after the write, a crash
between the CAS and the hint append (T4-SRE-1), or a silent Update that
suppressed the hint (T3-SRE-2).

- WithReconcileInterval(d) (default 5s; 0 disables → changes-feed-only).
- The sweep reuses the existing stopSweep lifecycle (started in New,
  stopped on Shutdown). reconcileKey re-pulls under the monotone gate
  (storeRev > l1Rev) and broadcasts ONLY when L1 actually advances, so a
  steady-state tick is a silent no-op, not a render storm.

Tests: a peer converges via the sweep alone when a silent write emitted no
hint; the changes-feed-only mode (interval 0) still converges via the
tailer; an internal test locks reconcileKey's advance / no-regress /
poison-survival / absent-cell gates. -race green.
Maintainer-approved: a pod now adopts a presented via_session sid it never
issued, so the same logical session is served by any pod a client visits —
the no-sticky-sessions guarantee the clustered value path needs.

- validSessionID: strict 64-lowercase-hex (exactly genSecureID's format), so
  a malformed or attacker-chosen token is never adopted as a session key.
- getOrCreateSession adopts only an unknown, well-formed sid via adoptSession
  (a LoadOrStore under the lock, so concurrent adopters converge on one
  *session). The known-sid path is unchanged (returns the existing session,
  never clobbered); a malformed cookie still mints a fresh session.

The 256-bit sid is the bearer credential (standard distributed-session
model); apps close session fixation with Session.Rotate on privilege change.
via_tab/CSRF and Rotate are unaffected. Internal tests cover the format
gate, adoption, known-sid-unchanged, idempotent re-check, malformed-reject,
and 16-racer convergence; -race green.
…e 3c)

StateSess[T] keeps its API but the backplane Store cell val:s:<sid>:<key>
is now the source of truth, so a session's value spans pods (sessions are
adoptable, so any pod serves them) — while single-pod behavior stays
byte-for-byte.

- Update CAS-retries the per-session Store cell (full sid in the key, no
  aliasing), sets the session L1 synchronously, records a per-session
  monotone rev, and appends a value-less change{Sid,Key,Rev} hint unless
  the action is silent.
- The shared changes-tailer routes session changes to applySessionChange,
  which is SECURITY-CRITICAL and fail-closed: a change for a sid this pod
  does not hold is DROPPED before any Store read or broadcast — a session
  write can never leak into or wake another session. For a held session it
  re-pulls under the stale-replica + per-session monotone gates and
  broadcasts scoped to that one session.
- The reconcile sweep also covers (session x key), so a session converges
  even when a hint was missed (silent write, cold pod).

Tests: existing StateSess suite stays green (byte-for-byte single-pod,
cross-session no-fanout); internal tests lock the fail-closed-no-load drop,
the full-sid key, stale-replica/monotone/poison gates, and the sweep path.
-race green.

Closes Phase 3: the value path (StateApp + StateSess) is cluster-survivable
and cross-pod convergent with the Store as source of truth and no sticky
sessions.
…-incompat halt (Phase 4a)

Every StateAppEvents event now rides a self-describing version envelope
{t,v,d}, so events can be evolved later (an event appended without one can
never be upcast). v1 auto-derives the type tag from E's Go type — no new
user-facing API; the VERSION drives evolution.

- Append wraps the event payload in the envelope; foldBytes decodes it:
  a corrupt/undecodable record -> ErrUndecodable, a record from a newer
  binary (v > this binary's max) -> ErrForwardIncompatible.
- The projector handles the three outcomes: a good record folds and
  re-renders; a poison record is SKIPPED (cursor advances so it is never
  retried forever) and counts via.events.undecodable; a forward-
  incompatible record HALTS that key (projection frozen, cursor NOT
  advanced so a roll-forward redeploy resumes correctly) and counts
  via.events.forward_incompatible. The projector keeps draining its
  subscription when halted, so a frozen key cannot leak the backend's
  sender goroutine.

A poison or future-version record thus can never panic the pod or wedge a
key for its peers. Adds ErrUndecodable/ErrForwardIncompatible. Internal
tests lock the four decode outcomes and the projector's skip-advance /
halt-freeze cursor invariants + metrics; the existing StateAppEvents,
chat, and counter suites stay green (envelope round-trips end to end).
Stored older-version events now decode into the current event shape via
registered upcasters, so a reshape (rename/split/type-change) never strands
old immortal records — and Fold only ever sees the current shape.

- RegisterEvent[E](fromVersion, upcast) registers a single-step vN->vN+1
  migration for event type E (keyed by its Go type). The current version of
  E becomes 1+max(fromVersion); with no upcaster it stays 1, so unregistered
  types (the common case) behave exactly as before — additive-first, JSON
  already tolerates added/missing fields.
- Append stamps the type's current version; the projector's decode runs the
  registered chain stored-version -> current before unmarshal. A version with
  no path to current, or a failing upcaster, yields ErrUndecodable and is
  dropped (never mis-folded); a newer-than-this-binary version stays
  ErrForwardIncompatible (halt).
- runUpcasters snapshots the needed steps under the registry read-lock before
  invoking them, so a concurrent RegisterEvent can't race the steps map.

Tests (-race): single + multi-step (v1->v2->v3) chains fold correctly,
missing-step/failing-upcaster -> ErrUndecodable, current = 1+max(fromVersion)
order-independent, unregistered types unchanged, concurrent register+upcast
race-clean.
…Phase 4c)

A key's underlying stream can be recreated, trimmed to empty, or restored —
restarting its offset space at 1 under a new epoch. A bare offset
high-water-mark would then skip every new record (their offsets are <= the
old cursor), silently freezing the projection.

- The projector now tracks the per-key epoch. The first record sets the
  baseline; a later record on a different epoch means the offset space
  reset, so it re-snapshots from genesis (projection back to the seed,
  cursor to 0, re-fold) and emits via.events.epoch_reset. Any epoch change
  resets — a restore can roll the epoch backward too.
- The per-record fold logic is extracted into projectRecord, the single
  shared fold path (the goroutine calls it and broadcasts on advance,
  outside the lock). A halted (forward-incompatible) key stays halted across
  a reset — roll-forward-only.

Behavior is unchanged for the in-memory backplane (epoch always 0, never
resets) and every existing StateAppEvents test now runs through
projectRecord. Internal test drives crafted epoch changes (forward and
backward) that the in-memory backplane never produces. -race green.

Closes Phase 4: StateAppEvents is version-hardened end to end (envelope,
upcasters, drop-on-undecodable, forward-incompat halt, epoch reset).
joaomdsg added 10 commits June 4, 2026 18:18
StateAppEvents projectors now persist the fold projection as a durable
checkpoint and seed from it on cold start, so a long log replays only the
tail instead of re-folding from genesis.

- WithSnapshotInterval(n): write a snapshot every n folds (default 64);
  <=0 disables. The snapshot is a disposable cache — a missing or
  stale-codec checkpoint just re-folds from offset 0, so evolving the
  projection type V is free (codecHash invalidates the cache).
- applogsnap.go: checkpoint{Epoch,CoveredOffset,CodecHash,V}, snapKey,
  maybeSnapshot (interval gate, off the per-event hot path), writeSnapshot
  (CAS outside the projection lock; a peer's snapshot is benign — refresh
  rev and skip).
- startProjector cold-starts via LoadSnapshot(snapKey): seed
  projection/cursor/epoch when CodecHash matches, then Subscribe from the
  covered offset; otherwise Subscribe from 0.

Keeps the event log's no-CAS-per-append win (snapshots are periodic, not
per-event). Store-only: no Compactor and no compacted-key migration yet
(P5b/P5c).
Adds an optional, type-asserted Compactor capability so a backend can
reclaim the log prefix a durable snapshot already covers. Backends without
native compaction omit it and run snapshot-only.

- Compactor{Compact(ctx,key,beforeOffset)}: discard records below
  beforeOffset; retained offsets stay UNCHANGED (the log keeps a front
  hole, Subscribe(from:0) resumes at the lowest retained offset, never
  renumbered). Idempotent, monotone, clamped to the committed head.
- in-memory backplane implements it via a per-key base offset; append
  continues monotonically from the head and Subscribe indexing accounts
  for the compacted prefix.
- The projector compacts only after a snapshot CAS succeeds
  (snapshot-FIRST, compact-SECOND) and the floor lags one snapshot
  generation, so the current durable snapshot's covered offset is never
  truncated — a cold start always resumes — and a generation of tail
  events survives for in-flight subscribers.

Store-only durable-genesis migration for a compacted key on codec-hash
mismatch is the next slice (P5c).
Completes the v1 state backplane. Once a key has compacted, its event
prefix is unrecoverable, so the snapshot is no longer a disposable cache
but durable genesis state — a codec-hash mismatch can no longer be
resolved by re-folding from offset 0 (that would silently truncate the
value to whatever events happen to survive).

- RegisterSnapshotMigration[V](fromCodecHash, migrate): teaches the
  runtime to decode an old snapshot into the current V. Keyed by the
  previous V codec hash. Uncompacted keys never consult it — their
  snapshot stays a pure cache, so evolving V is free in the common case.
- checkpoint gains a Compacted flag, set durably BEFORE the prefix is
  dropped (snapshot-FIRST), so a cold start never sees a discarded prefix
  behind a Compacted:false checkpoint.
- Cold start is now three-way: codec match seeds from the snapshot; an
  uncompacted mismatch re-folds from genesis; a compacted mismatch runs
  the registered seeded migration (decode old V, fold the retained tail
  on top) or, with no/failing migration, HALTS the projector
  (roll-forward-only) and emits via.snapshot.unbridgeable rather than
  truncate.
Adds StateAppEvents.OnEvent — a named, offset-tracked, side-effecting
consumer over the event log. Effects (send an email, charge a card) live
here, never in Fold, so Fold stays pure and deterministic.

- OnEvent(name, fn func(ctx context.Context, ev E, off Offset) error):
  a separate per-pod tailer whose committed offset is durable in the Store
  cell "consumer:<name>:<wireKey>", advanced only after the handler
  returns nil. A restart resumes from the committed offset. The handler
  receives a context.Context (not a via *Ctx — a background tailer has no
  tab/session); the offset seeds an idempotency key.
- At-least-once: a handler error does not advance (the record is retried,
  head-of-line, preserving order); a poison record is skipped; a
  forward-incompatible record blocks (roll-forward-only) rather than
  silently dropping the effect; the commit CAS loser adopts the peer's
  offset.
- A registered consumer clamps the Compactor floor to its committed
  offset, so compaction never discards an event it has not yet processed.
- Factors the version-envelope decode into a shared decodeEvent[E] used by
  both the projection fold and the consumer.
The design council rated design/state-backplane per-lens (mean 7.4); this
closes every open finding, each via the TDD-rygba cycle. Full suite +
backplanetest conformance + vianats green under -race on both modules.

Correctness/SRE:
- vianats: derive a real per-stream Epoch (StreamInfo.Created) and stamp it on
  Head + every Record, so offset-space-reset detection actually fires on NATS
  (was hard-coded Epoch(0)).
- projector + OnEvent consumer reconnect-rehydrate on a transient mid-stream
  disconnect, distinguished from graceful Shutdown via App.backplaneDone; exit
  cleanly on Close.
- fold-divergence canary: emit via.fold.offset + via.fold.digest per fold.

Safety/GDPR:
- WithFoldVerify(): re-fold each record, emit via.fold.divergence and refuse to
  compact a key whose fold is non-deterministic (never crystallize bad genesis).
- crypto-shred erasure: KeyStore (KeyFor/Key/DropKey) + InMemoryKeyStore,
  DataSubject opt-in, per-subject AES-256-GCM payloads, ErrErased drop-no-op,
  App.EraseDataSubject (DropKey + erasure-generation snapshot invalidation),
  compacted+erasure halts rather than truncating surviving subjects.

Testing:
- memevents.Faulty gains Disconnect + Reorder modes.
- fold-purity fuzz + cross-process two-replay convergence gate.
- fix 3 pre-existing flakes (consumer/compaction timing; helper racing Close).

Go-idioms/DX/docs:
- snapMigrations registry → typed snapshotMigration.
- chat example field Log→Messages; StateAppEvents/.Op()/Append/OnEvent godoc.
- spec + design-council: mTLS/per-pod-creds/namespacing MANDATORY, reconciled
  GDPR + isolation sections, documented residuals & new metrics.
…c snapshots

A multi-node load benchmark surfaced a cross-pod projection-truncation bug. Under
load with compaction (the default), a fast pod compacts the shared log while a
peer projector lags >1 snapshot-generation behind; the lagging pod's subscriber
falls below the compacted base, the backend clamps to base+1, and the projector
SILENTLY SKIPS the dropped records — its projection diverges from peers
permanently, with no error and no metric. Backend-agnostic (JetStream purge +
DeliverByStartSequence behaves identically).

Two root causes, both fixed:

1. Live projector ignored gaps. The projector folded any record with
   offset>cursor straight onto its projection. Now applyRecord detects a gap
   (rec.Offset > cursor+1 — including a fresh cursor=0 projector whose first
   delivered record has offset>1 because the prefix was compacted before its
   subscriber first read) and RE-SEEDS from the durable snapshot (which covers
   the gap) before folding; if no snapshot bridges the whole gap, it HALTS
   (via.events.compaction_gap_halt) rather than fold onto a stale projection.
   reseedFromSnapshot only seeds when the snapshot moves forward AND covers up to
   rec.Offset-1. Extends the cold-start-from-snapshot recovery to the live case.

2. Snapshot writes could regress. The shared snapshot cell is CAS'd by every pod;
   a lagging pod's rev-CAS could legitimately succeed and overwrite a leader's
   higher-covered snapshot with a lower one, breaking the "snapshot covers the
   compacted prefix" invariant that compaction and gap-reseed both rely on.
   writeSnapshot is now MONOTONIC: it reads the current cell and skips the write
   unless it advances CoveredOffset (CAS against the freshly-read rev).

New metrics: via.events.compaction_reseed, via.events.compaction_gap_halt.
Tests: gap-reseed, halt-without-bridging-snapshot, contiguous-no-reseed,
reseed-then-dedup boundary, fresh-cursor0-post-compaction, snapshot-no-regress.
Verified: convergence WITH compaction hammered 90x + -race with zero divergence;
full suite green -race on both modules.
…gence, durable fanout)

Adds the benchmark suite used to load-test the clustered state backplane (and
which surfaced the compaction-truncation bug fixed in the previous commit):

- backplanebench_internal_test.go (in-memory, multi-pod runtime ceiling):
  - InMemoryAppendThroughput: raw append ceiling (~4.7M/s contended, ~6.2M/s sharded).
  - CrossPodConvergence: P pods share one backplane; times full cross-pod
    convergence of b.N events. Aggregate fold throughput scales with pod count
    (~0.6M→2.1M folds/s, core-bound at ~16) while per-event input rate drops
    inversely (every pod re-folds every event — fan-out). GC relaxed during the
    timed region (the in-mem log is unbounded in heap; production bounds it via
    compaction). Barrier polls infrequently so it doesn't perturb the projectors.
  - ConvergenceFeatureCost: baseline vs WithFoldVerify (~2x per-fold) vs
    compaction-on (converges correctly via the gap-reseed fix).
- vianats/bench_test.go (durable backend, embedded JetStream = a floor):
  AppendThroughput (~139k/s publish+ack) and Fanout delivery (scales then
  plateaus on the single embedded server).
- vianats/embedded_test.go: widen startEmbeddedJetStream to testing.TB so
  benchmarks can reuse it.
…pass, staleness sweep

- docs/production.md: rebuild the metrics catalogue (it predated the backplane) —
  all via.events.*/fold.*/snapshot.*/consumer.* + the new compaction_reseed /
  compaction_gap_halt, fix the stale via.sse.disconnect label, add alerting hints
  and a "state backplane under load" profile.
- docs/why-via.md: council pass from a new-adopter POV (skeptical/clarity/
  positioning/trust lenses, iterated to convergence) — plain-English opener +
  code hook, corrected comparison table (Hotwire transport, templ+HTMX row, fair
  build-step), honest clustering (preview/eventual, not shipped-as-fact), expanded
  non-goals (Datastar coupling, auth, payload, no-JS/SEO), SSE-per-tab cost model,
  decision rule, softened recovery claims to match production.md.
- design/state-backplane.md: staleness sweep — drop brittle file:line cites and
  journal/"shipped" framing, restate the load-test fix as timeless invariants.
- README.md: align the cluster + recovery lines with why-via/production.
- design-council.md: Tick 46 (multi-node load benchmark + cross-pod truncation fix).
gofmt -w the accumulated formatting drift (7 files) the branch's first CI run
flagged, and check the fmt.Fprintf return in the fold-purity test (errcheck).
The branch's first CI run surfaced accumulated lint debt. Add t.Parallel() to
the 27 independent tests; //nolint:paralleltest (with reason) for tests that
mutate a process-global registry (snapshot-migration / event-version) or are
timing-sensitive consumer/compaction convergence tests where added concurrency
would reintroduce load-induced flakes. Apply the staticcheck quickfixes
(De Morgan simplifications in backplanetest/conformance.go; Posted→Message
conversion in the chat example).

Verified: golangci-lint clean (root + vianats), go test -race -count=3 green,
gofmt clean, previously-flaky set hammered -race -count=10 clean.
@joaomdsg
joaomdsg merged commit 77fc82c into main Jun 5, 2026
6 checks passed
@joaomdsg
joaomdsg deleted the design/state-backplane branch June 5, 2026 12:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant