Skip to content

feat(onevent): bounded-retry + opt-in poison-record skip#120

Merged
joaomdsg merged 2 commits into
mainfrom
feat/onevent-poison
Jun 10, 2026
Merged

feat(onevent): bounded-retry + opt-in poison-record skip#120
joaomdsg merged 2 commits into
mainfrom
feat/onevent-poison

Conversation

@joaomdsg

Copy link
Copy Markdown
Member

Finding (critic #5)

An OnEvent side-effect consumer whose handler permanently fails retried head-of-line forever on a flat 10ms backoff. That wedges the consumer and pins the Compactor floor at the poison record's offset, so the event log grows unboundedly — a single bad record can take down a pod's disk over time.

Fix

  • Exponential backoff + full jitter (default 10ms → 30s) replaces the flat 10ms retry. Per-pod attempt counter, reset to 0 on any advance.
  • Block-forever stays the default. WithMaxAttempts(0) (the default) preserves today's zero-data-loss semantics: a permanently-failing handler keeps the record head-of-line and never drops the side effect. Skipping a poison record is strictly opt-in — dropping a side effect by default is the wrong default.
  • Loud while blocking. Each retry emits a via.consumer.stuck gauge (the attempt count); a one-shot WARN fires once attempts cross a threshold, so the floor-pin is observable rather than silent.

New options (variadic — existing 2-arg call sites still compile)

  • WithMaxAttempts(n int) — after n handler-error attempts on a record, treat it as poison: emit via.consumer.poisoned + WARN, dead-letter it (if a hook is set) or drop it, then commit/advance past it. Un-wedges the consumer and unpins the floor.
  • WithRetryBackoff(base, max time.Duration) — backoff bounds (default 10ms → 30s).
  • WithDeadLetter(fn) — invoked just before a record is poisoned. Returns nil → advance; returns an error → do not advance, keep retrying (a record the operator opted to DLQ is never lost when the sink is down).

deliver() was refactored to return a three-way outcome (advance / retry / block) so a forward-incompatible record always blocks and is never counted toward poison; an undecodable record still skips via the decode path (via.consumer.undecodable), distinct from the poison path.

Tests (red→green, internal pkg, -race)

  • default maxAttempts=0: always-erroring handler never advances; via.consumer.error + via.consumer.stuck keep firing; never poisons.
  • WithMaxAttempts(N): advances past the record after N attempts, emits via.consumer.poisoned, then delivers the next record (proves un-wedged).
  • errors-then-succeeds: advances, attempts reset (no premature poison on a later record).
  • WithDeadLetter returning nil: hook gets correct key/off/data/cause, then advances + via.consumer.poisoned.
  • WithDeadLetter returning err: does not advance (stays retrying).
  • forward-incompatible still blocks under WithMaxAttempts (never poisoned); undecodable still skips (not poisoned).

Docs

docs/production.md metrics catalogue: added via.consumer.poisoned and via.consumer.stuck rows, amended the via.consumer.error note (bounded-retry/backoff + WithMaxAttempts opt-in skip). docs/distributed-state.md OnEvent section documents the new options.

ci-check.sh passes (gofmt, vet, golangci v2.12.2, govulncheck, -race, alloc benches).

joaomdsg added 2 commits June 10, 2026 19:35
A side-effect consumer (OnEvent) whose handler permanently fails retried
head-of-line forever on a flat 10ms backoff, wedging the consumer and pinning
the Compactor floor at the poison record's offset -> unbounded log growth
(critic finding #5).

Changes:
- Exponential backoff with full jitter (default 10ms -> 30s) replaces the flat
  10ms retry, with a per-pod attempt counter reset on any advance.
- Variadic ConsumerOption on OnEvent (existing 2-arg call sites still compile):
  - WithMaxAttempts(n) -- default 0 = BLOCK FOREVER (zero-data-loss preserved);
    n>0 opts into treating a record as poison after n handler-error attempts.
  - WithRetryBackoff(base, max) -- tune the backoff bounds.
  - WithDeadLetter(fn) -- archive a record before skipping; returning an error
    keeps retrying so an opted-in DLQ record is never lost when the sink is down.
- Loud-by-default while blocking: via.consumer.stuck gauge per retry + a one-shot
  WARN once attempts cross a threshold, so a floor-pin is observable not silent.
- On poison-skip: via.consumer.poisoned counter + WARN, then commit/advance.
- deliver() now returns a three-way outcome (advance / retry / block) so a
  forward-incompatible record always blocks and is NEVER counted toward poison.

The default keeps today's semantics exactly: a permanently-failing handler still
blocks forever and drops no side effect; skipping is strictly opt-in.

Tests cover: block-forever default, poison-after-N + un-wedge, attempt reset on
recovery, dead-letter receives the record then advances, dead-letter error keeps
retrying without advancing, forward-incompatible still blocks under WithMaxAttempts,
and undecodable still skips (not poisoned).
@joaomdsg
joaomdsg merged commit 3b3d4f4 into main Jun 10, 2026
6 checks passed
@joaomdsg
joaomdsg deleted the feat/onevent-poison branch June 11, 2026 15:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant