Skip to content

[FEATURE] tsdb: New parallel-wal-decode feature#18806

Open
nicolastakashi wants to merge 10 commits into
prometheus:mainfrom
nicolastakashi:feat/parallel-wal-decode
Open

[FEATURE] tsdb: New parallel-wal-decode feature#18806
nicolastakashi wants to merge 10 commits into
prometheus:mainfrom
nicolastakashi:feat/parallel-wal-decode

Conversation

@nicolastakashi

@nicolastakashi nicolastakashi commented May 27, 2026

Copy link
Copy Markdown
Contributor

Summary

WAL replay's producer side, byte read, snappy decompress, varint decode runs on a single goroutine. On large heads the per-shard sample workers are starved by it; profiling a 19M-series production replay showed ~3.3 of 16 worker cores active. Add a flag-gated decoder pool that splits the work across K decoder goroutines, with a small reorder buffer preserving WAL byte order at the router.

Off by default. Pool size defaults to min(max(GOMAXPROCS/4, 2), 4) when enabled.

Benchmark results

go test -count=6 -benchtime=10x -benchmem, compared with benchstat, on Apple M1 Pro, GOMAXPROCS=8.

BenchmarkLoadWLs — parallel decode vs serial:

500k series × 240 samples (~120M samples), snappy-compressed WAL, no exemplars, no missing series; 60 raw samples per K.

                  │   K=0 (serial)  │           K=4 (parallel)         │
                  │     sec/op      │     sec/op     vs base           │
500k×240/snappy   │   6.129 ± 17%   │   4.568 ± 20%  -25.48% (p=0.004) │

                  │   K=0 (serial)  │           K=4 (parallel)         │
                  │      B/op       │      B/op      vs base           │
500k×240/snappy   │  1011.6Mi ± 9%  │   984.1Mi ± 8%       ~ (p=0.132) │

                  │   K=0 (serial)  │           K=4 (parallel)         │
                  │   allocs/op     │   allocs/op    vs base           │
500k×240/snappy   │   6.553M ± 8%   │   6.632M ± 0%        ~ (p=0.065) │

The bench shape sits below the regime this change targets (≥1M series); production-sized heads should see a larger win.

Regression check — flag off, existing shapes:

BenchmarkLoadWLs re-run on this branch vs a baseline captured at the merge-base commit. 624 raw measurements per side, -count=6. Geomean across all shapes:

                  │    baseline    │            this branch           │
                  │                │                vs base           │
geomean sec/op    │    641.1m      │    649.7m       +1.34%           │
geomean B/op      │    290.1Mi     │    291.4Mi      +0.44%           │
geomean allocs/op │     1.821M     │     1.821M      +0.02%           │

Within bench variance, no regression.

[FEATURE] tsdb: New --enable-feature=parallel-wal-decode flag enables a pool of decoder goroutines during WAL replay, improving head startup time on large heads. Off by default.

nicolastakashi and others added 9 commits May 27, 2026 14:43
NewDecoder previously ignored its *labels.SymbolTable argument and built
a fresh ScratchBuilder via labels.NewScratchBuilder(0). Under the
dedupelabels build tag this means each Decoder constructs its own
private SymbolTable, so callers that intend to share symbol state
across decoder instances (e.g. tsdb/wlog/watcher.go, which already
documents "One table per WAL segment means it won't grow indefinitely")
silently get fragmented per-decoder tables instead.

Thread the argument through via labels.NewScratchBuilderWithSymbolTable
when non-nil; fall back to the historical behaviour when nil. For the
stringlabels and slicelabels build tags this is a no-op since their
SymbolTable type is a zero-byte struct. Under dedupelabels the
caller-supplied table is now actually used and shared across whatever
Decoders the caller constructs against it.

Add a test under -tags=dedupelabels that asserts the passed symbol
table is populated by decode and that a second Decoder constructed
against the same table reuses existing symbols rather than allocating
new ones.

```release-notes
[ENHANCEMENT] tsdb/record: NewDecoder now uses the supplied
*labels.SymbolTable; under -tags=dedupelabels this preserves symbol
sharing across decoder instances as callers already expected.
```

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Signed-off-by: Nicolas Takashi <[email protected]>
The decoder goroutine in loadWAL inlines a ~95-line switch that
allocates a per-record-type pooled slice, dispatches to the right
record.Decoder method, and wraps any decoder error in a
*wlog.CorruptionErr with the current segment/offset. Pull that body
out into a new method:

  (*Head).decodeWALRecord(rec []byte, typ record.Type,
                          dec *record.Decoder,
                          segment int, offset int64) (any, error)

It returns:
  - (decoded slice, nil) for handled record types,
  - (nil, nil)           for types loadWAL does not act on,
  - (nil, *wlog.CorruptionErr) on decode failure (segment/offset are
                          embedded in the error so callers do not
                          need a back-channel to the reader).

The function reads only from its arguments and writes only to the
goroutine-safe sync.Pools on Head, so it is safe to call from
multiple goroutines concurrently. That property is not exercised
here — the existing single-goroutine pipeline is preserved verbatim
— but it is the shape required by the follow-up commits that
introduce a decoder pool.

No behaviour change.

```release-notes
NONE
```

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Signed-off-by: Nicolas Takashi <[email protected]>
BenchmarkLoadWLs currently runs every shape against a WAL written
with compression.None. Real Prometheus deployments default to snappy,
where r.Next() does the decompression on the producer-side hot path
inside the WAL decoder goroutine. Replay benchmarks that skip
compression therefore understate the producer-side CPU cost.

Add a `compress` field to the case struct (zero value treated as
compression.None so the existing cases are unchanged) and one new
case sized for the regime parallel-WAL-decode work targets:
batches=200, seriesPerBatch=2500, samplesPerSeries=240 — 500k series
x 240 samples = ~120M samples, written with compression.Snappy.

Subtest names now include `compress=<type>` so benchstat can
distinguish the new shape from the existing baseline.

```release-notes
NONE
```

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Signed-off-by: Nicolas Takashi <[email protected]>
Add two new fields to HeadOptions:

  - ParallelWALDecode (bool, default false) — represents the
    'parallel-wal-decode' feature flag.
  - ParallelWALDecodeConcurrency (int, default
    min(max(GOMAXPROCS/4, 2), 4) when ParallelWALDecode is enabled)
    — the size of the decoder pool.

Normalise the concurrency default in NewHead alongside the existing
WALReplayConcurrency handling, so a positive explicit value is
preserved and zero/negative gets clamped.

Wire a flag-gated entry point at the top of loadWAL:

  if h.opts.ParallelWALDecode {
      return h.loadWALParallel(...)
  }

For now loadWALParallel is a stub that returns an error: the byte
reader, decoder pool, and reorder buffer described in the design doc
land in the follow-up commit. Returning an explicit error keeps the
behaviour-when-flag-off path bit-identical and prevents the flag from
silently no-op'ing WAL replay if it is enabled prematurely (e.g. by a
test that gets ahead of the implementation).

The CLI wiring for --enable-feature=parallel-wal-decode arrives in a
later commit; until then the flag is only settable directly on
HeadOptions, which is sufficient for tests and benchmarks.

```release-notes
NONE
```

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Signed-off-by: Nicolas Takashi <[email protected]>
The decoder goroutine in loadWAL inlines the per-record read/decode/
push loop directly. Pull that body out into a separate method:

  (*Head).serialDecodeWALRecords(r *wlog.Reader,
                                 syms *labels.SymbolTable,
                                 decoded chan<- any) error

The function reads from r sequentially, calls the existing
decodeWALRecord for each record, and pushes typed values onto decoded
in WAL byte order. It returns when r drains or when decodeWALRecord
returns a *wlog.CorruptionErr.

It deliberately does NOT close decoded; loadWAL keeps the deferred
close on the launching goroutine. That contract keeps the close-once
responsibility local to the call site rather than spreading it across
the producer family — which matters once the parallel path adds a
producer with a different goroutine topology (a byte reader plus a
decoder pool plus a reorder buffer, where the reorder buffer is the
last writer but is not the function the caller invokes directly).

No behaviour change. loadWAL is byte-equivalent under the flag-off
path; tests covering WAL replay, corruption handling, and OOO replay
pass unchanged.

```release-notes
NONE
```

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Signed-off-by: Nicolas Takashi <[email protected]>
loadWAL has ~340 lines of worker-pool setup, exemplar processor, and
router/finalisation that the parallel decode path needs to share verbatim
— only the decoder goroutine differs between the serial and the
upcoming parallel pipeline. Pull the shared body out into a new method:

  (*Head).runWALReplay(r, syms, multiRef, mmappedChunks, oooMmappedChunks,
                       produce func(*wlog.Reader, *labels.SymbolTable,
                                    chan<- any) error) (err error)

The producer is launched inside runWALReplay's own goroutine, which
captures any returned error into decodeErr and closes the decoded
channel via defer. The producer therefore must not close decoded
itself — the close-once contract is centralised in runWALReplay so the
parallel path (whose internal goroutine topology differs) can plug in
without re-implementing channel lifecycle.

loadWAL is now a small dispatcher:

  if h.opts.ParallelWALDecode {
      return h.loadWALParallel(...)
  }
  return h.runWALReplay(..., h.serialDecodeWALRecords)

The 340-line body did not move — it stayed in place under the new
function name and signature — so the diff is the wrapper plus the
producer parameter plus the one-line change in the decoder goroutine.
This keeps the refactor easy to verify by inspection.

No behaviour change. loadWAL with the flag off is bit-identical to
the previous commit; WAL replay, corruption handling, and OOO replay
tests pass unchanged.

```release-notes
NONE
```

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Signed-off-by: Nicolas Takashi <[email protected]>
Replace the loadWALParallel stub with the byte-reader → decoder-pool →
reorder-buffer pipeline described in the design doc. All new code
lives in a new file, tsdb/head_wal_parallel.go, so head_wal.go does
not grow further.

Pipeline shape:

  1. Byte reader (1 goroutine) reads from wlog.Reader, captures
     type / segment / offset, copies the record bytes into a pooled
     []byte (rawRecordBufPool), and dispatches a rawRecord{seq, ...}
     on a buffered channel sized K*4. Unknown record types are
     filtered out before they consume a seq.
  2. Decoder pool (K = ParallelWALDecodeConcurrency goroutines): each
     constructs its own record.Decoder with the shared
     *labels.SymbolTable (so dedupelabels symbol state is shared
     across the pool — that is what the earlier FIXME-fix commit on
     record.NewDecoder enables), consumes rawRecords, calls
     decodeWALRecord, returns the pooled bytes, and emits a
     decodedRecord{seq, val} on decodedOOO. On decode failure the
     goroutine reports the error via walDecodeFirstErr and emits a
     nil-val sentinel so the reorder buffer can advance.
  3. Reorder buffer (runs inline in the parallelDecodeWALRecords
     function): maintains a min-heap by seq, skips sentinels, and
     emits onto the existing `decoded` channel in WAL byte order so
     the router's contract is unchanged. Once the expected-seq
     pointer reaches the seq of the first reported error, the
     buffer stops emitting and only drains — preserving the serial
     path's behaviour where records past the first corruption are
     not applied to the head.

parallelDecodeWALRecords is the producer passed to runWALReplay; the
worker pool, exemplar processor, and router are shared verbatim with
the serial path. The reorder buffer does NOT close `decoded` —
runWALReplay's launching goroutine does that on return via defer.

walDecodeFirstErr is a tiny mutex-guarded min-seq tracker. The serial
path's contract is "fail on the first offending record in WAL byte
order"; multiple decoders can hit corruption concurrently, so we
reduce by min seq rather than wall-clock order.

Add an end-to-end equivalence smoke test: write a WAL containing
series, samples, a tombstone, and an exemplar, replay it twice (flag
off, then K=4), and assert NumSeries / MinTime / MaxTime match
between the two runs. Race-clean under -race. More granular unit
tests for the reorder buffer, first-error scenarios, and back-
pressure interactions land in follow-up commits.

The CLI flag (--enable-feature=parallel-wal-decode) wires up in a
later commit; for now the feature is exercisable only via
HeadOptions, which is sufficient for tests and the benchmark suite.

```release-notes
NONE
```

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Signed-off-by: Nicolas Takashi <[email protected]>
The end-to-end smoke test in the previous commit verifies the
parallel pipeline produces equivalent head state to the serial path
on a small mixed WAL. This commit fills out the test coverage on the
load-bearing internals:

Reorder buffer:
  - Emits records in seq order regardless of arrival order.
  - Skips nil-val sentinels without breaking ordering of surrounding
    records.
  - Buffers out-of-order arrivals in the heap and emits contiguously
    once the low-seq lands.
  - Stops emitting once the expected-seq pointer reaches the seq of
    the first reported error (preserving the serial path's contract
    that records past the first corruption are not applied).

walDecodeFirstErr:
  - Retains the lowest-seq error across multiple concurrent reports.
  - Reports (0, nil) when no error has been recorded.

Extended end-to-end equivalence:
  - Replays a WAL containing histograms, float histograms, and
    metadata in addition to series and samples; asserts NumSeries /
    MinTime / MaxTime match between serial and parallel runs.

All tests pass under -race.

```release-notes
NONE
```

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Signed-off-by: Nicolas Takashi <[email protected]>
…rency

To measure the parallel WAL decode delta with benchstat, both
BenchmarkLoadWLs and BenchmarkLoadRealWLs need to iterate over the K
values we care about. Add that, with two compatibility concerns
addressed up front:

  - BenchmarkLoadWLs: the K dimension is opt-in per shape via a new
    parallelDecodeKs []int field on the case struct. Shapes that
    don't set it (every existing case) keep their old subtest names
    and only run the serial path, so older baseline.txt files
    captured before this commit remain comparable with benchstat.
    The new snappy stress shape opts in with K ∈ {0, 2, 4, 8}.
  - BenchmarkLoadRealWLs: subtest names now include
    parallel_decode=K. This does break compatibility with any
    pre-existing real-WAL baseline files, but real-WAL benches
    require BENCHMARK_LOAD_REAL_WLS_DIR so the population of
    affected baselines is small, and getting the K comparison is
    exactly the point.

The K iteration runs each K as its own b.Run subtest with its own
b.Loop, so b.Loop's timing model is preserved per K. The WAL is
written once per outer subtest and re-read for every K.

```release-notes
NONE
```

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Signed-off-by: Nicolas Takashi <[email protected]>
@nicolastakashi nicolastakashi changed the title Feat/parallel wal decode [WIP] - Feat/parallel wal decode May 27, 2026
@nicolastakashi nicolastakashi changed the title [WIP] - Feat/parallel wal decode [FEATURE] tsdb: New parallel-wal-decode feature May 27, 2026
@nicolastakashi

Copy link
Copy Markdown
Contributor Author

Depends on: #18803

@nicolastakashi
nicolastakashi force-pushed the feat/parallel-wal-decode branch from 709ed21 to 2359df1 Compare May 27, 2026 19:08
Plumb the parallel WAL decode feature through the public surface:

  - tsdb.Options gains a ParallelWALDecode bool. The TSDB Open path
    copies it onto HeadOptions.ParallelWALDecode, where the existing
    normalisation logic sets the concurrency default
    (min(max(GOMAXPROCS/4, 2), 4)).
  - cmd/prometheus/main.go's tsdbOptions gains the same field and
    maps it through ToTSDBOptions.
  - The --enable-feature switch grows a parallel-wal-decode case
    matching the existing pattern: set the option, log that the
    experimental feature is enabled.
  - docs/feature_flags.md gets a new section describing the flag, the
    default pool size, and the workload shape where the win is
    expected to materialise.

The concurrency knob (HeadOptions.ParallelWALDecodeConcurrency) is
intentionally not exposed as its own CLI flag for this first release —
the auto-default is the only safe value pending production data, and
keeping the surface small lets us evolve it later without a
deprecation window. Library consumers can still override it via
tsdb.Options if they construct it directly.

```release-notes
[FEATURE] tsdb: New --enable-feature=parallel-wal-decode flag enables
a pool of decoder goroutines during WAL replay. Improves head startup
time on large heads where the WAL byte reader / record decoder
goroutine is the producer-side bottleneck. Off by default.
```

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Signed-off-by: Nicolas Takashi <[email protected]>
@nicolastakashi
nicolastakashi force-pushed the feat/parallel-wal-decode branch from 2359df1 to d64a279 Compare May 27, 2026 19:27
@nicolastakashi
nicolastakashi marked this pull request as ready for review May 28, 2026 09:51
Comment thread tsdb/head_wal_parallel.go

// Push and Pop satisfy heap.Interface and operate on the underlying
// slice via the pointer receiver.
func (h *decodedRecordHeap) Push(x any) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does it make sense to use any in here? We know the backing storage will be a decodedRecord object, and teh casting adds allocations and processing to the hot path whenever chunks are unordered.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are using any here, because we are using the heap implementation from go std library, to have a typed heap we need to write our own.

Before going on this direction, I'd like a review from one of the maintainers and their opnions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants