feat: state backplane — clustered StateAppEvents/StateApp/StateSess (event-log model)#93
Merged
Conversation
Add design/state-backplane.md: a scoping/design proposal for cluster fan-out + restart survivability as an ADDITIVE feature, not a core/API rewrite. Value state (StateApp/StateSess) keeps its API and rides a durable CAS Store + a value-less Change on an ordered Log; a new opt-in sibling StateAppLog[E, V] models high-churn shared state as append-events-and-fold, killing the hot-key CAS retry-storm. One Store + Log (+ optional Compactor) + Codec interface backs NATS JetStream / Redis Streams / Postgres / Kafka. Captures the recommended interfaces (godoc-load-bearing), resolutions to all seven open questions, the residual fold-determinism risk, a terse four-architect comparison, and a seven-phase additive rollout. Derived from two multi-architect design passes; supersedes the earlier lossy-Bus sketch (see "Design evolution").
…2-GO-4) Design-council ticks 5-6 over the converged Via state backplane design. design/state-backplane.md: - Tick 5: apply the naming-council rename — StateAppLog -> StateAppEvents, Log interface/concept -> EventLog (Backplane = Store + EventLog), chat field Log/r.Log -> Events/r.Events, metrics via.log.* -> via.events.*. Bare "Log" left only on the lines describing today's StateAppSlice chat (current code). - Tick 6: reconcile the body with the converged T2-GO-4 resolution. Fixed 4 places that asserted "snapshot = pure cache, evolving V free, no snapshot upcasters" with no compacted-key caveat. The snapshot is a disposable cache only for uncompacted keys; once a key compacts the deleted prefix is unrecoverable, so the snapshot is durable genesis state (typed Codec[V], versioned Checkpoint, seeded migration on mismatch, retained-event floor, WithFoldVerify mandatory before compaction). design-council.md: the council's running record (ticks 1-6: convergence at tick 4, in-mem-backplane and naming addenda, then the doc-reconciliation ticks).
…-path, projector) Council ticks 7-9 propagating the converged decisions from design-council.md into design/state-backplane.md, where the body still described the pre-council design in several load-bearing places. - Tick 7 (T1-GO-2/T2-GO-4): replace the untyped Codec (Encode(any)/Decode()any) with the generic Codec[T] — event Codec[E] + version-tagged snapshot Codec[V]; kills the public any and carries the seeded compacted-key migration. - Tick 8 (T3-SRE-1/T1-SRE-5/T4-SRE-1): value (StateApp/StateSess) state is no longer pod-local. The durable Store cell is the single source of truth, the per-pod kvStore is an L1 cache; every pod incl. the writer re-pulls to Store HEAD on each Change (gated storeRev >= change.rev, monotone L1 gate) backed by a periodic reconcile sweep so correctness never depends on a notify. Removes the sticky-session requirement for state. Decision bullet + Phase 3 rewritten. - Tick 9 (T1-SRE-1/T1-SRE-2): the per-(pod,key) projector is the sole fold path on every pod incl. the writer. Append no longer folds or renders locally; broadcastRender is the intra-pod leg, the projector tailing EventLog.Subscribe is the inter-pod leg. Cross-pod read-your-write is eventual (WaitFor for strict). nil-ctx panic reframed as the AUTH gate, not the render driver. design-council.md records ticks 5-9 (the rename + the four reconciliation steps).
…convergence Council ticks 10-17. Completes propagating the converged decisions into design/state-backplane.md and audits the whole doc for internal consistency. Reconciliation (ticks 10-12): - T1-GO-1: drop E.Zero(); fold seed = `var zero V` (kills the pointer-E nil-panic), across the EventReducer interface, Read, both examples, cold-start. - T1-SEC-1/2: add standalone Multi-tenant & session isolation and GDPR crypto-shred sections (physical namespace + mTLS boundary, full-sid exact-match fail-closed, AUTH-1, tier-0 trust; KeyStore DropKey -> ErrUndecodable, snapshot invalidation, audit-class vs PII-class). - header Status -> CONVERGED; nil backplane resolves internally to via.InMemory()/memevents.Backplane (one code path); Phase 1/2 in-mem-default shift (memevents.Faulty + parameterized conformance, release-gating real NATS). Coherence audit (ticks 13-17, rounds 1-5): - Store godoc: disambiguate the two source-of-truth roles (val:<key> cell vs the EventLog + its snapshot) so the value/log SoT statements don't conflict. - T1-GO-3: Offset end-to-end on Append/AppendIf/ReadAt/OnEvent (no bare uint64). - v1-surface demarcation: mark AppendIf/ReadAt/OnEvent as Advanced (post-v1). - T1-SRE-3: Epoch newtype on the delivered Record and the Head return (offset-space-reset detection). - T1-DX-2: StateAppCounter specialization replaces the raw counter example. - declare ErrForwardIncompatible and ErrEpochUnbridgeable (were referenced but undefined). design-council.md records ticks 10-17; the design deliberation converged at tick 17 (a full coherence sweep found nothing new).
First implementation slice of the state backplane. Purely additive type detection + wire-key binding — nil backplane / zero observable change. - StateAppEvents[E EventReducer[E,V], V any] handle: holds the wire key, binds via the existing scopeBinder/bindWireKey path, carries its own distinct isStateAppEvents() marker so the walker can later start a per-key projector only for log keys. - EventReducer[E,V] constraint (Fold(acc V, ev E) V) with the determinism contract documented. - walker: roleStateAppEvents + cached marker type + isStateAppEventsType + classifyField branch, folded into the existing scope-slot case. Read/Update/projector/app-binding are deferred to Phase 1 (their first consumer); scopeKind + bindApp land there too. Tests: default-key + via-tag override through the real Mount->render path; the pair defeats any hardcoded Key(). go test -race ./... green.
The base in-process Backplane the design's nil-backplane resolves to. A standalone, race-tested unit — no via-runtime wiring yet. - backplane.go: public contract — Offset/Rev/Epoch, Record, Store (LoadSnapshot/CAS), EventLog (Append/Subscribe/Head), Backplane (Store + EventLog + io.Closer), InMemory(), ErrCASConflict/ErrClosed. - inmemory.go: per-key CAS cell + per-key append log. Subscribe replays Offset>from in per-key total order then live-tails via a broadcast channel closed+replaced under the log mutex on each append; the subscriber goroutine unwinds on ctx-cancel or Close. Append re-checks the per-log closed flag under its mutex so an Append racing a concurrent Close cannot commit a lost write. Guarantees covered by tests (-race): monotone per-key offsets, Head high-water-mark, resumable+gap-free Subscribe with a real live-tail block, per-key independence, CAS conflict semantics, ctx-cancel unwind, Close (ErrClosed + subscriber channels close + idempotent), record-byte ownership. Design: records T1-GO-6 — the base in-memory impl lives in package via (not a separate memevents package) to avoid a via<->memevents import cycle; memevents stays the Phase 2 home for the Faulty decorator + conformance suite. AppendIf deferred (post-v1, with StateAppEvents.AppendIf).
App-level dependency injection for the state backplane. - config.go: WithBackplane(b Backplane) Option + a config field. A nil backplane is the documented default. - app.go: New() resolves the config backplane, defaulting nil to InMemory(), so the runtime always drives one Backplane code path (no nil-special-case). - server.go: App.Shutdown gracefully drains the backplane via Close() after disposing ctxs and stopping sweeps. Test: a WithBackplane'd backplane returns ErrClosed after Shutdown (proves it was stored and drained); a default app shuts down cleanly with the new drain step. The nil->InMemory resolution itself becomes black-box observable once Read/Append on the handle land (P1.1b).
… 1.1b) The heart of Phase 1: StateAppEvents now works end to end in-process. - Append (stateappevents.go): json.Marshal(ev) -> backplane.Append. No local fold, no broadcast — it only commits the fact. Panics on nil ctx (the AUTH gate, parity with StateApp.Update). - Per-(pod,key) projector (applog.go): the SOLE fold path (T1-SRE-2). Tails backplane.Subscribe from genesis, folds each record (decode E + E.Fold) into a cached projection in offset order, then broadcastRenders to subscribed tabs. Started once per key (sync.Once); exits when the Subscribe channel closes on Shutdown. - Read: returns the cached projection (O(1)) and subscribes the ctx via trackRead, identical to StateApp.Read. Text mirrors StateApp.Text. - Binding seam completed: scopeSlot gains a kind flag (value vs log); bindScopeKeys binds the App into log handles via a new appBinder; newCtx threads the *App. The typed bindApp method registers a seed + fold closure with the App, bridging the reflection-detected, type-erased field to the generic E/V fold (T1-GO-8) — the App holds only `any`. Tests (-race): empty log projects zero V; an appended event folds and reaches a live SSE subscriber; the projection is app-scoped and outlives the writer (two appends -> "hello,hello", also catching double-fold); Append panics on nil ctx; Text renders the projection. Codec is plain encoding/json for now; the versioned Codec envelope is Phase 4. AppendIf/ReadAt/OnEvent remain post-v1.
The shipped specialization for a monotonic shared counter: just Inc + Read, no user-defined event type or Fold. - StateAppCounter embeds StateAppEvents[counterTick, int64], reusing the same embed-and-promote path as StateAppNum/StateApp — the walker detects and binds it with no walker/runtime change. - counterTick is unexported (Fold = +1), so Inc is the only append path: a caller cannot name the event to Append it directly. - Unlike StateApp[int] (a CAS cell that retry-storms under churn), increments are conflict-free appends; the value is the fold. Tests (-race): reads 0 before any Inc; three Inc calls accumulate to 3 for a fresh session (app-scoped, exact +1 fold); an Inc reaches a live SSE subscriber; Inc panics on a nil ctx.
The chat example now models its message log as an event log instead of a value-shaped StateAppSlice with a read-modify-write trim. - Room.Log is via.StateAppEvents[Posted, []Message]. Send appends an immutable Posted event; the pure Fold derives (and bounds) the rendered list. The old Op.Append + trim-Update read-modify-write is gone. - Fold copies acc before appending (replay determinism) and trims to recentWindow; the event log itself is bounded later by snapshot + compaction, not by rewriting history. Behavior is preserved (live cross-session fan-out, bounded render). The existing fan-out test stays green; a new test proves messages accumulate in send order in a fresh reader's view (the fold appends, app-scoped). Closes Phase 1 of the state backplane: the in-process StateAppEvents core ships green and the flagship example demonstrates it.
A reusable, backend-agnostic contract suite any Backplane implementation must pass — the executable spec for ordering, resumability, CAS, and lifecycle. - New package backplanetest with RunConformance(t, newBackplane). Nine subtests: increasing-and-Head-tracking offsets (empty Head = 0), genesis replay in order, resume strictly after a returned offset, live-tail, per-key independence, concurrent appends get distinct offsets with Head at the max (the conflict-free total-order guarantee), independent subscribers see the same stream, CAS must-not-exist/conflict/advance, Close -> ErrClosed. - Offsets are read back from Append and asserted only as strictly-increasing / resumable, never hardcoded — so a non-dense Kafka/JetStream/Postgres backend conforms. - TestInMemoryConformance runs it against via.InMemory(); the suite will also gate the Faulty decorator (2b) and the NATS backend (2d). Lives in a normal (non-_test) file like fstest/iotest so external backend packages can import it.
…e conformance (Phase 2b)
Proves the Backplane contract (and the conformance suite) is robust to the
at-least-once redelivery a real network log exhibits.
- memevents.Faulty wraps any via.Backplane and re-emits each Subscribe
record Redeliver+1 times, in order, via one goroutine that exits on the
underlying channel close or ctx cancel (no leak). Store/Append/Head/Close
delegate through the embedded interface.
- backplanetest.RunConformance is now at-least-once-tolerant: a
collectDistinct helper dedupes by offset and asserts first-delivery in
per-key order, so the SAME suite gates exactly-once (InMemory) and
redelivering backends (Faulty, and later NATS) alike. This corrects an
over-strict exactly-once assumption in the Phase 2a suite (T1-TEST-2).
Tests: conformance passes against Faulty{Redeliver:1}; redelivery actually
duplicates; Redeliver:0 is exact passthrough; the wrapper unwinds on ctx
cancel; Subscribe propagates the underlying ErrClosed.
…hase 2c)
The test that proves the projector — not a local fold — is the cluster
mechanism. Two independent via.App instances ("pods") share one
via.InMemory() backplane:
- An event appended through pod A's StateAppEvents surfaces, folded, in
pod B's live SSE frame — B's per-key projector tails the shared log.
- After appends on both pods, a fresh reader on either pod converges to
the same identically-ordered folded feed.
A local-fold-in-Append implementation would pass every single-pod test
yet fail here (A's projection would update but B would never see it),
because Append does no broadcast and each App's broadcastRender only
reaches its own ctx registry. Two Apps sharing one in-memory backplane is
the infra-free stand-in for two pods sharing one NATS — validating the
design's central correctness claim (#3/#7 closed by resumable Subscribe,
no census).
… (T-GO-9) Phase 3 rewrites the core value-path (StateApp/StateSess), so this is a convergence + design-feedback tick before the build. - T-GO-9: making the backplane Store cell (val:<key>) the source of truth means StateApp[T]/StateSess[T] values serialize (default JSON) on Update->CAS, since Store moves []byte. The L1 kvStore keeps the live T so reads stay zero-serialization. Documented at the Store godoc: the "v0.4.0 byte-for-byte" promise is identical observable behavior for serializable T (the universal shared-state case); a non-serializable T in a pod-local kvStore is the one narrow break. - Recorded the P3 sub-slice plan: P3a StateApp cross-pod value convergence (Store CAS + L1 read-cache + value-less Change feed + re-pull tailer, existing single-pod suite the green guard), P3b periodic reconcile sweep, P3c StateSess with full-sid Change matching. Phase 2 infra-free work (conformance, Faulty, cross-pod keystone) is done; the NATS reference backend (2d) remains blocked on a maintainer decision.
… harness (Phase 2d-1) Maintainer green-lit the NATS reference backend, as a nested module to keep the core via module dependency-free. - New ./vianats nested module (own go.mod, replace => ../). Deps: nats.go v1.52.0, nats-server v2.14.2. Core via go.mod is untouched, so via consumers never pull NATS. - Embedded JetStream test harness (startEmbeddedJetStream): an in-process JetStream server on a random port with a temp store, shut down at test end — conformance needs no external container or CI service. - Smoke test proves the primitives the backend rests on work in-sandbox: KV create + CAS-by-revision + Get, and stream publish (monotone sequence) + subject-filtered ordered consume. Next: implement the vianats.Backplane (Store over JetStream KV, EventLog over a subject-per-key stream) and gate it with backplanetest.RunConformance.
… RELEASE-GATING) The real ordered/durable/resumable reference backend, proving the Backplane abstraction holds against a network log — not just the forgiving in-memory one. - vianats.JetStream(nc, WithPrefix): Store over a JetStream KV bucket (LoadSnapshot=Get; CAS = Create for rev 0 else Update(expectedRev), both conflict modes mapped to via.ErrCASConflict via ErrKeyExists), EventLog over one subject-per-key stream (Append=Publish->Sequence, Head= GetLastMsgForSubject, Subscribe=OrderedConsumer with DeliverAll or DeliverByStartSequence(from+1) drained into a channel that unwinds on ctx-cancel or Close). Close marks closed + stops live subscriptions; it does not close the caller's connection. sanitize() maps arbitrary wire keys to safe, collision-free subject/KV tokens. - backplanetest.RunConformance runs against a fresh uniquely-prefixed backplane per subtest on an embedded nats-server: all nine guarantees (ordering, gap-free resume, live-tail, per-key independence, concurrent appends, multi-subscriber agreement, CAS conflict, Close) pass -race. Closes Phase 2: the backplane is proven across in-memory, fault-injected, cross-pod, and a real NATS backend under one contract suite (#3/#7 closed). The vianats nested module keeps the core via module dependency-free.
…e 3a)
StateApp[T] keeps its API but the backplane Store cell val:<key> is now
the single source of truth, so two pods sharing a backplane converge —
while single-pod behavior stays byte-for-byte identical.
- Update is a compare-and-swap retry loop on the Store cell (LoadSnapshot
-> decode -> fn -> Marshal -> CAS, retrying on ErrCASConflict): no lost
updates across ctxs or pods, no shared-cell mutex. On success it sets
the shared per-pod L1 cell synchronously (so every session on this pod
sees it at once, as before) and appends a value-less Change{key,rev}
liveness hint to a shared feed.
- A per-App changes-tailer re-pulls the named Store cell to HEAD on each
Change, gated storeRev>=hint.rev (drop stale replica reads, T1-SRE-5)
and storeRev>l1Rev (monotone, redelivery/out-of-order safe, T3-SRE-1).
- Read hits the L1 cell (O(1), zero-serialization). A silent (sync-off)
Update writes the Store but suppresses the hint, so SyncOff still fans
nothing out (T3-SRE-2).
- appStore is gone; bindScopeKeys now binds the App into any scope handle
that needs it (value + log), and the dead scopeKind flag is removed.
T must be JSON-serializable once a backplane is in play (T-GO-9; nil
resolves to InMemory, so single-pod too) — the universal case for shared
state. Tests: the existing StateApp suite stays green (byte-for-byte) +
a two-Apps cross-pod convergence test + an internal test locking the
stale-drop / monotone / undecodable-snapshot reconcile gates. -race green.
Makes the changes feed a pure latency optimization: each pod periodically re-pulls its registered value keys to the Store HEAD, so it converges even when no Change hint reached it — a pod that joined after the write, a crash between the CAS and the hint append (T4-SRE-1), or a silent Update that suppressed the hint (T3-SRE-2). - WithReconcileInterval(d) (default 5s; 0 disables → changes-feed-only). - The sweep reuses the existing stopSweep lifecycle (started in New, stopped on Shutdown). reconcileKey re-pulls under the monotone gate (storeRev > l1Rev) and broadcasts ONLY when L1 actually advances, so a steady-state tick is a silent no-op, not a render storm. Tests: a peer converges via the sweep alone when a silent write emitted no hint; the changes-feed-only mode (interval 0) still converges via the tailer; an internal test locks reconcileKey's advance / no-regress / poison-survival / absent-cell gates. -race green.
…aintainer decision)
Maintainer-approved: a pod now adopts a presented via_session sid it never issued, so the same logical session is served by any pod a client visits — the no-sticky-sessions guarantee the clustered value path needs. - validSessionID: strict 64-lowercase-hex (exactly genSecureID's format), so a malformed or attacker-chosen token is never adopted as a session key. - getOrCreateSession adopts only an unknown, well-formed sid via adoptSession (a LoadOrStore under the lock, so concurrent adopters converge on one *session). The known-sid path is unchanged (returns the existing session, never clobbered); a malformed cookie still mints a fresh session. The 256-bit sid is the bearer credential (standard distributed-session model); apps close session fixation with Session.Rotate on privilege change. via_tab/CSRF and Rotate are unaffected. Internal tests cover the format gate, adoption, known-sid-unchanged, idempotent re-check, malformed-reject, and 16-racer convergence; -race green.
…e 3c)
StateSess[T] keeps its API but the backplane Store cell val:s:<sid>:<key>
is now the source of truth, so a session's value spans pods (sessions are
adoptable, so any pod serves them) — while single-pod behavior stays
byte-for-byte.
- Update CAS-retries the per-session Store cell (full sid in the key, no
aliasing), sets the session L1 synchronously, records a per-session
monotone rev, and appends a value-less change{Sid,Key,Rev} hint unless
the action is silent.
- The shared changes-tailer routes session changes to applySessionChange,
which is SECURITY-CRITICAL and fail-closed: a change for a sid this pod
does not hold is DROPPED before any Store read or broadcast — a session
write can never leak into or wake another session. For a held session it
re-pulls under the stale-replica + per-session monotone gates and
broadcasts scoped to that one session.
- The reconcile sweep also covers (session x key), so a session converges
even when a hint was missed (silent write, cold pod).
Tests: existing StateSess suite stays green (byte-for-byte single-pod,
cross-session no-fanout); internal tests lock the fail-closed-no-load drop,
the full-sid key, stale-replica/monotone/poison gates, and the sweep path.
-race green.
Closes Phase 3: the value path (StateApp + StateSess) is cluster-survivable
and cross-pod convergent with the Store as source of truth and no sticky
sessions.
…-incompat halt (Phase 4a)
Every StateAppEvents event now rides a self-describing version envelope
{t,v,d}, so events can be evolved later (an event appended without one can
never be upcast). v1 auto-derives the type tag from E's Go type — no new
user-facing API; the VERSION drives evolution.
- Append wraps the event payload in the envelope; foldBytes decodes it:
a corrupt/undecodable record -> ErrUndecodable, a record from a newer
binary (v > this binary's max) -> ErrForwardIncompatible.
- The projector handles the three outcomes: a good record folds and
re-renders; a poison record is SKIPPED (cursor advances so it is never
retried forever) and counts via.events.undecodable; a forward-
incompatible record HALTS that key (projection frozen, cursor NOT
advanced so a roll-forward redeploy resumes correctly) and counts
via.events.forward_incompatible. The projector keeps draining its
subscription when halted, so a frozen key cannot leak the backend's
sender goroutine.
A poison or future-version record thus can never panic the pod or wedge a
key for its peers. Adds ErrUndecodable/ErrForwardIncompatible. Internal
tests lock the four decode outcomes and the projector's skip-advance /
halt-freeze cursor invariants + metrics; the existing StateAppEvents,
chat, and counter suites stay green (envelope round-trips end to end).
Stored older-version events now decode into the current event shape via registered upcasters, so a reshape (rename/split/type-change) never strands old immortal records — and Fold only ever sees the current shape. - RegisterEvent[E](fromVersion, upcast) registers a single-step vN->vN+1 migration for event type E (keyed by its Go type). The current version of E becomes 1+max(fromVersion); with no upcaster it stays 1, so unregistered types (the common case) behave exactly as before — additive-first, JSON already tolerates added/missing fields. - Append stamps the type's current version; the projector's decode runs the registered chain stored-version -> current before unmarshal. A version with no path to current, or a failing upcaster, yields ErrUndecodable and is dropped (never mis-folded); a newer-than-this-binary version stays ErrForwardIncompatible (halt). - runUpcasters snapshots the needed steps under the registry read-lock before invoking them, so a concurrent RegisterEvent can't race the steps map. Tests (-race): single + multi-step (v1->v2->v3) chains fold correctly, missing-step/failing-upcaster -> ErrUndecodable, current = 1+max(fromVersion) order-independent, unregistered types unchanged, concurrent register+upcast race-clean.
…Phase 4c) A key's underlying stream can be recreated, trimmed to empty, or restored — restarting its offset space at 1 under a new epoch. A bare offset high-water-mark would then skip every new record (their offsets are <= the old cursor), silently freezing the projection. - The projector now tracks the per-key epoch. The first record sets the baseline; a later record on a different epoch means the offset space reset, so it re-snapshots from genesis (projection back to the seed, cursor to 0, re-fold) and emits via.events.epoch_reset. Any epoch change resets — a restore can roll the epoch backward too. - The per-record fold logic is extracted into projectRecord, the single shared fold path (the goroutine calls it and broadcasts on advance, outside the lock). A halted (forward-incompatible) key stays halted across a reset — roll-forward-only. Behavior is unchanged for the in-memory backplane (epoch always 0, never resets) and every existing StateAppEvents test now runs through projectRecord. Internal test drives crafted epoch changes (forward and backward) that the in-memory backplane never produces. -race green. Closes Phase 4: StateAppEvents is version-hardened end to end (envelope, upcasters, drop-on-undecodable, forward-incompat halt, epoch reset).
StateAppEvents projectors now persist the fold projection as a durable
checkpoint and seed from it on cold start, so a long log replays only the
tail instead of re-folding from genesis.
- WithSnapshotInterval(n): write a snapshot every n folds (default 64);
<=0 disables. The snapshot is a disposable cache — a missing or
stale-codec checkpoint just re-folds from offset 0, so evolving the
projection type V is free (codecHash invalidates the cache).
- applogsnap.go: checkpoint{Epoch,CoveredOffset,CodecHash,V}, snapKey,
maybeSnapshot (interval gate, off the per-event hot path), writeSnapshot
(CAS outside the projection lock; a peer's snapshot is benign — refresh
rev and skip).
- startProjector cold-starts via LoadSnapshot(snapKey): seed
projection/cursor/epoch when CodecHash matches, then Subscribe from the
covered offset; otherwise Subscribe from 0.
Keeps the event log's no-CAS-per-append win (snapshots are periodic, not
per-event). Store-only: no Compactor and no compacted-key migration yet
(P5b/P5c).
Adds an optional, type-asserted Compactor capability so a backend can
reclaim the log prefix a durable snapshot already covers. Backends without
native compaction omit it and run snapshot-only.
- Compactor{Compact(ctx,key,beforeOffset)}: discard records below
beforeOffset; retained offsets stay UNCHANGED (the log keeps a front
hole, Subscribe(from:0) resumes at the lowest retained offset, never
renumbered). Idempotent, monotone, clamped to the committed head.
- in-memory backplane implements it via a per-key base offset; append
continues monotonically from the head and Subscribe indexing accounts
for the compacted prefix.
- The projector compacts only after a snapshot CAS succeeds
(snapshot-FIRST, compact-SECOND) and the floor lags one snapshot
generation, so the current durable snapshot's covered offset is never
truncated — a cold start always resumes — and a generation of tail
events survives for in-flight subscribers.
Store-only durable-genesis migration for a compacted key on codec-hash
mismatch is the next slice (P5c).
Completes the v1 state backplane. Once a key has compacted, its event prefix is unrecoverable, so the snapshot is no longer a disposable cache but durable genesis state — a codec-hash mismatch can no longer be resolved by re-folding from offset 0 (that would silently truncate the value to whatever events happen to survive). - RegisterSnapshotMigration[V](fromCodecHash, migrate): teaches the runtime to decode an old snapshot into the current V. Keyed by the previous V codec hash. Uncompacted keys never consult it — their snapshot stays a pure cache, so evolving V is free in the common case. - checkpoint gains a Compacted flag, set durably BEFORE the prefix is dropped (snapshot-FIRST), so a cold start never sees a discarded prefix behind a Compacted:false checkpoint. - Cold start is now three-way: codec match seeds from the snapshot; an uncompacted mismatch re-folds from genesis; a compacted mismatch runs the registered seeded migration (decode old V, fold the retained tail on top) or, with no/failing migration, HALTS the projector (roll-forward-only) and emits via.snapshot.unbridgeable rather than truncate.
Adds StateAppEvents.OnEvent — a named, offset-tracked, side-effecting consumer over the event log. Effects (send an email, charge a card) live here, never in Fold, so Fold stays pure and deterministic. - OnEvent(name, fn func(ctx context.Context, ev E, off Offset) error): a separate per-pod tailer whose committed offset is durable in the Store cell "consumer:<name>:<wireKey>", advanced only after the handler returns nil. A restart resumes from the committed offset. The handler receives a context.Context (not a via *Ctx — a background tailer has no tab/session); the offset seeds an idempotency key. - At-least-once: a handler error does not advance (the record is retried, head-of-line, preserving order); a poison record is skipped; a forward-incompatible record blocks (roll-forward-only) rather than silently dropping the effect; the commit CAS loser adopts the peer's offset. - A registered consumer clamps the Compactor floor to its committed offset, so compaction never discards an event it has not yet processed. - Factors the version-envelope decode into a shared decodeEvent[E] used by both the projection fold and the consumer.
The design council rated design/state-backplane per-lens (mean 7.4); this closes every open finding, each via the TDD-rygba cycle. Full suite + backplanetest conformance + vianats green under -race on both modules. Correctness/SRE: - vianats: derive a real per-stream Epoch (StreamInfo.Created) and stamp it on Head + every Record, so offset-space-reset detection actually fires on NATS (was hard-coded Epoch(0)). - projector + OnEvent consumer reconnect-rehydrate on a transient mid-stream disconnect, distinguished from graceful Shutdown via App.backplaneDone; exit cleanly on Close. - fold-divergence canary: emit via.fold.offset + via.fold.digest per fold. Safety/GDPR: - WithFoldVerify(): re-fold each record, emit via.fold.divergence and refuse to compact a key whose fold is non-deterministic (never crystallize bad genesis). - crypto-shred erasure: KeyStore (KeyFor/Key/DropKey) + InMemoryKeyStore, DataSubject opt-in, per-subject AES-256-GCM payloads, ErrErased drop-no-op, App.EraseDataSubject (DropKey + erasure-generation snapshot invalidation), compacted+erasure halts rather than truncating surviving subjects. Testing: - memevents.Faulty gains Disconnect + Reorder modes. - fold-purity fuzz + cross-process two-replay convergence gate. - fix 3 pre-existing flakes (consumer/compaction timing; helper racing Close). Go-idioms/DX/docs: - snapMigrations registry → typed snapshotMigration. - chat example field Log→Messages; StateAppEvents/.Op()/Append/OnEvent godoc. - spec + design-council: mTLS/per-pod-creds/namespacing MANDATORY, reconciled GDPR + isolation sections, documented residuals & new metrics.
…c snapshots A multi-node load benchmark surfaced a cross-pod projection-truncation bug. Under load with compaction (the default), a fast pod compacts the shared log while a peer projector lags >1 snapshot-generation behind; the lagging pod's subscriber falls below the compacted base, the backend clamps to base+1, and the projector SILENTLY SKIPS the dropped records — its projection diverges from peers permanently, with no error and no metric. Backend-agnostic (JetStream purge + DeliverByStartSequence behaves identically). Two root causes, both fixed: 1. Live projector ignored gaps. The projector folded any record with offset>cursor straight onto its projection. Now applyRecord detects a gap (rec.Offset > cursor+1 — including a fresh cursor=0 projector whose first delivered record has offset>1 because the prefix was compacted before its subscriber first read) and RE-SEEDS from the durable snapshot (which covers the gap) before folding; if no snapshot bridges the whole gap, it HALTS (via.events.compaction_gap_halt) rather than fold onto a stale projection. reseedFromSnapshot only seeds when the snapshot moves forward AND covers up to rec.Offset-1. Extends the cold-start-from-snapshot recovery to the live case. 2. Snapshot writes could regress. The shared snapshot cell is CAS'd by every pod; a lagging pod's rev-CAS could legitimately succeed and overwrite a leader's higher-covered snapshot with a lower one, breaking the "snapshot covers the compacted prefix" invariant that compaction and gap-reseed both rely on. writeSnapshot is now MONOTONIC: it reads the current cell and skips the write unless it advances CoveredOffset (CAS against the freshly-read rev). New metrics: via.events.compaction_reseed, via.events.compaction_gap_halt. Tests: gap-reseed, halt-without-bridging-snapshot, contiguous-no-reseed, reseed-then-dedup boundary, fresh-cursor0-post-compaction, snapshot-no-regress. Verified: convergence WITH compaction hammered 90x + -race with zero divergence; full suite green -race on both modules.
…gence, durable fanout)
Adds the benchmark suite used to load-test the clustered state backplane (and
which surfaced the compaction-truncation bug fixed in the previous commit):
- backplanebench_internal_test.go (in-memory, multi-pod runtime ceiling):
- InMemoryAppendThroughput: raw append ceiling (~4.7M/s contended, ~6.2M/s sharded).
- CrossPodConvergence: P pods share one backplane; times full cross-pod
convergence of b.N events. Aggregate fold throughput scales with pod count
(~0.6M→2.1M folds/s, core-bound at ~16) while per-event input rate drops
inversely (every pod re-folds every event — fan-out). GC relaxed during the
timed region (the in-mem log is unbounded in heap; production bounds it via
compaction). Barrier polls infrequently so it doesn't perturb the projectors.
- ConvergenceFeatureCost: baseline vs WithFoldVerify (~2x per-fold) vs
compaction-on (converges correctly via the gap-reseed fix).
- vianats/bench_test.go (durable backend, embedded JetStream = a floor):
AppendThroughput (~139k/s publish+ack) and Fanout delivery (scales then
plateaus on the single embedded server).
- vianats/embedded_test.go: widen startEmbeddedJetStream to testing.TB so
benchmarks can reuse it.
…pass, staleness sweep - docs/production.md: rebuild the metrics catalogue (it predated the backplane) — all via.events.*/fold.*/snapshot.*/consumer.* + the new compaction_reseed / compaction_gap_halt, fix the stale via.sse.disconnect label, add alerting hints and a "state backplane under load" profile. - docs/why-via.md: council pass from a new-adopter POV (skeptical/clarity/ positioning/trust lenses, iterated to convergence) — plain-English opener + code hook, corrected comparison table (Hotwire transport, templ+HTMX row, fair build-step), honest clustering (preview/eventual, not shipped-as-fact), expanded non-goals (Datastar coupling, auth, payload, no-JS/SEO), SSE-per-tab cost model, decision rule, softened recovery claims to match production.md. - design/state-backplane.md: staleness sweep — drop brittle file:line cites and journal/"shipped" framing, restate the load-test fix as timeless invariants. - README.md: align the cluster + recovery lines with why-via/production. - design-council.md: Tick 46 (multi-node load benchmark + cross-pod truncation fix).
gofmt -w the accumulated formatting drift (7 files) the branch's first CI run flagged, and check the fmt.Fprintf return in the fold-purity test (errcheck).
The branch's first CI run surfaced accumulated lint debt. Add t.Parallel() to the 27 independent tests; //nolint:paralleltest (with reason) for tests that mutate a process-global registry (snapshot-migration / event-version) or are timing-sensitive consumer/compaction convergence tests where added concurrency would reintroduce load-induced flakes. Apply the staticcheck quickfixes (De Morgan simplifications in backplanetest/conformance.go; Posted→Message conversion in the chat example). Verified: golangci-lint clean (root + vianats), go test -race -count=3 green, gofmt clean, previously-flaky set hammered -race -count=10 clean.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Makes server-owned reactive state durable and convergent across N processes without prescribing infrastructure — closing the two long-standing non-goals (cluster runtime + restart survivability). Purely additive: a
nilbackplane is byte-for-byte the existing single-pod behavior.What it adds
StateAppEvents[E,V]— an event-log-backed sibling ofStateApp: append immutable events, declare a pureFold, read the projection. Per-(pod,key) projector is the sole fold path; cross-pod convergence by tailing the log.Backplaneseam (StoreCAS-cell + ordered resumableEventLog+ optionalCompactor) withvia.InMemory()as the base implnilresolves to — so the interface is exercised on every single-pod run.vianats— NATS JetStream reference backend (KV + stream), passes the parameterized conformance suite (release-gating).StateApp/StateSess— Store is the source of truth; pods converge via a changes feed + periodic reconcile sweep; full-sid fail-closed isolation. Sticky sessions no longer required for state correctness.OnEventside-effect consumers (at-least-once, offset-committed, idempotency-key).KeyStore+ per-subject AES-256-GCM payloads +EraseDataSubject(key drop + snapshot gen-invalidation; compacted+erasure halts rather than truncates).WithFoldVerify— runtime fold-determinism gate that refuses to compact a non-deterministic fold; per-fold divergence canary.Hardened by a multi-node load benchmark
The load test surfaced a real cross-pod projection-truncation bug (a fast pod compacting the shared log silently truncated a lagging peer). Fixed: a lagging projector re-seeds from the snapshot on a compaction gap (or halts, never diverges), and snapshot writes are monotonic in
CoveredOffset. Verified: convergence-with-compaction hammered 90× +-race.Verification
backplanetestconformance +vianatsgreen under-race(both modules);go vetclean.memevents.Faulty(redelivery/disconnect/reorder), two-pods-one-backplane convergence keystone, fuzz + subprocess fold-purity gates.Docs
design/state-backplane.md(spec, DECIDED),design-council.md(full deliberation, 46 ticks), operator metrics catalogue + load profile indocs/production.md, new-adopter pass ondocs/why-via.md.Residuals are documented and fail-safe (eventual read-your-write, live-projection erasure clears on cold-start, compaction×erasure halts pending snapshot re-encryption, value-tailer reconnect symmetry).