feat(onevent): bounded-retry + opt-in poison-record skip#120
Merged
Conversation
A side-effect consumer (OnEvent) whose handler permanently fails retried head-of-line forever on a flat 10ms backoff, wedging the consumer and pinning the Compactor floor at the poison record's offset -> unbounded log growth (critic finding #5). Changes: - Exponential backoff with full jitter (default 10ms -> 30s) replaces the flat 10ms retry, with a per-pod attempt counter reset on any advance. - Variadic ConsumerOption on OnEvent (existing 2-arg call sites still compile): - WithMaxAttempts(n) -- default 0 = BLOCK FOREVER (zero-data-loss preserved); n>0 opts into treating a record as poison after n handler-error attempts. - WithRetryBackoff(base, max) -- tune the backoff bounds. - WithDeadLetter(fn) -- archive a record before skipping; returning an error keeps retrying so an opted-in DLQ record is never lost when the sink is down. - Loud-by-default while blocking: via.consumer.stuck gauge per retry + a one-shot WARN once attempts cross a threshold, so a floor-pin is observable not silent. - On poison-skip: via.consumer.poisoned counter + WARN, then commit/advance. - deliver() now returns a three-way outcome (advance / retry / block) so a forward-incompatible record always blocks and is NEVER counted toward poison. The default keeps today's semantics exactly: a permanently-failing handler still blocks forever and drops no side effect; skipping is strictly opt-in. Tests cover: block-forever default, poison-after-N + un-wedge, attempt reset on recovery, dead-letter receives the record then advances, dead-letter error keeps retrying without advancing, forward-incompatible still blocks under WithMaxAttempts, and undecodable still skips (not poisoned).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Finding (critic #5)
An
OnEventside-effect consumer whose handler permanently fails retried head-of-line forever on a flat 10ms backoff. That wedges the consumer and pins the Compactor floor at the poison record's offset, so the event log grows unboundedly — a single bad record can take down a pod's disk over time.Fix
WithMaxAttempts(0)(the default) preserves today's zero-data-loss semantics: a permanently-failing handler keeps the record head-of-line and never drops the side effect. Skipping a poison record is strictly opt-in — dropping a side effect by default is the wrong default.via.consumer.stuckgauge (the attempt count); a one-shotWARNfires once attempts cross a threshold, so the floor-pin is observable rather than silent.New options (variadic — existing 2-arg call sites still compile)
WithMaxAttempts(n int)— afternhandler-error attempts on a record, treat it as poison: emitvia.consumer.poisoned+WARN, dead-letter it (if a hook is set) or drop it, then commit/advance past it. Un-wedges the consumer and unpins the floor.WithRetryBackoff(base, max time.Duration)— backoff bounds (default 10ms → 30s).WithDeadLetter(fn)— invoked just before a record is poisoned. Returnsnil→ advance; returns an error → do not advance, keep retrying (a record the operator opted to DLQ is never lost when the sink is down).deliver()was refactored to return a three-way outcome (advance / retry / block) so a forward-incompatible record always blocks and is never counted toward poison; an undecodable record still skips via the decode path (via.consumer.undecodable), distinct from the poison path.Tests (red→green, internal pkg,
-race)maxAttempts=0: always-erroring handler never advances;via.consumer.error+via.consumer.stuckkeep firing; never poisons.WithMaxAttempts(N): advances past the record after N attempts, emitsvia.consumer.poisoned, then delivers the next record (proves un-wedged).WithDeadLetterreturning nil: hook gets correct key/off/data/cause, then advances +via.consumer.poisoned.WithDeadLetterreturning err: does not advance (stays retrying).WithMaxAttempts(never poisoned); undecodable still skips (not poisoned).Docs
docs/production.mdmetrics catalogue: addedvia.consumer.poisonedandvia.consumer.stuckrows, amended thevia.consumer.errornote (bounded-retry/backoff +WithMaxAttemptsopt-in skip).docs/distributed-state.mdOnEvent section documents the new options.ci-check.shpasses (gofmt, vet, golangci v2.12.2, govulncheck,-race, alloc benches).