[EXPERIMENT] feat(chunk,st): add chunk format that supports start timestamp by bwplotka · Pull Request #17475 · prometheus/prometheus

bwplotka · 2025-11-04T21:35:45Z

This PR is to play with different implementations of float chunk that persists ST/CT information as per PROM-60.

Goals

Understand the ST in chunk perf impact
See if we can have near zero overhead for constant ST cases (typical case for counters and gauges)

Implementations so far:

This PR has many various implementations, but interesting ones are below:

XOROptST:

This is essentially a DOD for ST just with optimization for typical constant ST and ST == 0 cases.

It performs pretty well, perhaps good enough.

2B(numSamples), 1B(stHeader), ?varint(st), varint(t), xor(v), ?varuint(stDelta), varuint(tDelta), xor(v), ?classicvarbitint(stDod), classicvarbitint(tDod), xor(v), ...

stHeader: 1b(firstSTKnown), 7b(firstSTChangeOn)

XORCST:

It's my funky variation of XOROptST where instead of DOD, when we see ST we encode either 0 (ST same as before) or special varbitints, but in relation to the timestamp delta/dod.

Playing with it, results are promising (Iterator/vt=constant/st=delta-2 case is questionable in practice - prod data would tell).

 2B(numSamples), 1B(stHeader), varint(t), ?varint(st), xor(v), varuint(tDelta), ?stDiff(st), xor(v), classicvarbitint(tDod),?stDiff(st),  xor(v), ...

 stHeader: 1b(firstSTKnown), 7b(firstSTChangeOn)

XORBuffered:

"...Buffered" type of algorithms are not iteratively appending or reading, they read and write chunk fully into memory. This is good for cases where you need all samples in a chunk, but not if you want only a few. It also optimizes for latency, less for space, even less for mem allocs.

2B(mixedHeader), DOD(sts), DOD(ts), XOR(values)
mixedHeader: 1b(isSTConstant), 15b(numSamples)

ALPBuffered (lossy float!):

Using tscodec-go buffered, lossy float compression (a lot currently, up to 1e-6 in my tests)

2B(numSamples), DOD(sts), DOD(ts), ALP(values)

Skipped benchmarks in this iteration, but TL;DR it's fast but bit wider (and buffered and lossy) (TODO redo)

Benchmarks (so far!)

Writes

 benchstat -col /fmt bw.bench/append.txt        
goos: darwin
goarch: arm64
pkg: github.com/prometheus/prometheus/tsdb/chunkenc
cpu: Apple M1 Pro
                                                       │ XOR_(ST_ignored) │              XOROptST               │               XORCST                │
                                                       │      sec/op      │    sec/op     vs base               │    sec/op     vs base               │
Appender/vt=constant/st=0-2                                  1.123µ ±  4%   1.186µ ±  0%   +5.66% (p=0.002 n=6)   1.204µ ±  3%   +7.26% (p=0.002 n=6)
Appender/vt=constant/st=cumulative-2                         1.120µ ±  9%   1.227µ ±  3%   +9.56% (p=0.011 n=6)   1.225µ ±  8%   +9.42% (p=0.006 n=6)
Appender/vt=constant/st=delta-2                              1.111µ ±  1%   1.369µ ±  1%  +23.23% (p=0.002 n=6)   1.487µ ±  2%  +33.90% (p=0.002 n=6)
Appender/vt=random_steps/st=0-2                              2.758µ ±  1%   2.854µ ±  4%   +3.46% (p=0.002 n=6)   2.847µ ±  2%   +3.23% (p=0.002 n=6)
Appender/vt=random_steps/st=cumulative-2                     2.412µ ±  1%   2.630µ ± 11%   +9.06% (p=0.002 n=6)   2.565µ ±  0%   +6.37% (p=0.002 n=6)
Appender/vt=random_steps/st=delta-2                          2.831µ ±  2%   3.477µ ±  1%  +22.82% (p=0.002 n=6)   3.381µ ±  7%  +19.45% (p=0.002 n=6)
Appender/vt=random_0-1/st=0-2                                3.618µ ±  0%   3.727µ ±  1%   +3.03% (p=0.002 n=6)   3.794µ ±  5%   +4.88% (p=0.002 n=6)
Appender/vt=negrandom_0-1/st=0-2                             3.996µ ±  7%   4.126µ ±  0%        ~ (p=0.065 n=6)   4.078µ ±  1%        ~ (p=0.119 n=6)
Appender/vt=random_0-1/st=cumulative-2                       3.601µ ± 10%   3.749µ ±  6%        ~ (p=0.056 n=6)   3.675µ ±  1%        ~ (p=0.058 n=6)
Appender/vt=random_0-1/st=cumulative-periodic-resets-2       3.579µ ±  1%   5.653µ ±  3%  +57.94% (p=0.002 n=6)   4.876µ ± 18%  +36.24% (p=0.002 n=6)
Appender/vt=random_0-1/st=cumulative-periodic-zeros-2        3.493µ ±  1%   5.423µ ±  3%  +55.27% (p=0.002 n=6)   4.828µ ±  3%  +38.22% (p=0.002 n=6)
Appender/vt=random_0-1/st=delta-2                            3.467µ ±  6%   4.384µ ±  1%  +26.47% (p=0.002 n=6)   4.031µ ±  5%  +16.28% (p=0.002 n=6)
geomean                                                      2.499µ         2.931µ        +17.31%                 2.854µ        +14.22%

                                                       │ XOR_(ST_ignored) │              XOROptST              │               XORCST                │
                                                       │     B/chunk      │   B/chunk    vs base               │   B/chunk    vs base                │
Appender/vt=constant/st=0-2                                    48.00 ± 0%    49.00 ± 0%   +2.08% (p=0.002 n=6)    49.00 ± 0%    +2.08% (p=0.002 n=6)
Appender/vt=constant/st=cumulative-2                           48.00 ± 0%    55.00 ± 0%  +14.58% (p=0.002 n=6)    55.00 ± 0%   +14.58% (p=0.002 n=6)
Appender/vt=constant/st=delta-2                                48.00 ± 0%    73.00 ± 0%  +52.08% (p=0.002 n=6)   176.00 ± 0%  +266.67% (p=0.002 n=6)
Appender/vt=random_steps/st=0-2                                455.0 ± 0%    456.0 ± 0%   +0.22% (p=0.002 n=6)    456.0 ± 0%    +0.22% (p=0.002 n=6)
Appender/vt=random_steps/st=cumulative-2                       428.0 ± 0%    435.0 ± 0%   +1.64% (p=0.002 n=6)    435.0 ± 0%    +1.64% (p=0.002 n=6)
Appender/vt=random_steps/st=delta-2                            435.0 ± 0%    679.0 ± 0%  +56.09% (p=0.002 n=6)    563.0 ± 0%   +29.43% (p=0.002 n=6)
Appender/vt=random_0-1/st=0-2                                  837.0 ± 0%    838.0 ± 0%   +0.12% (p=0.002 n=6)    838.0 ± 0%    +0.12% (p=0.002 n=6)
Appender/vt=negrandom_0-1/st=0-2                               842.0 ± 0%    843.0 ± 0%   +0.12% (p=0.002 n=6)    843.0 ± 0%    +0.12% (p=0.002 n=6)
Appender/vt=random_0-1/st=cumulative-2                         842.0 ± 0%    849.0 ± 0%   +0.83% (p=0.002 n=6)    849.0 ± 0%    +0.83% (p=0.002 n=6)
Appender/vt=random_0-1/st=cumulative-periodic-resets-2         846.0 ± 0%   1287.0 ± 0%  +52.13% (p=0.002 n=6)   1056.0 ± 0%   +24.82% (p=0.002 n=6)
Appender/vt=random_0-1/st=cumulative-periodic-zeros-2          837.0 ± 0%   1344.0 ± 0%  +60.57% (p=0.002 n=6)   1157.0 ± 0%   +38.23% (p=0.002 n=6)
Appender/vt=random_0-1/st=delta-2                              849.0 ± 0%   1093.0 ± 0%  +28.74% (p=0.002 n=6)    977.0 ± 0%   +15.08% (p=0.002 n=6)
geomean                                                        349.7         419.9       +20.08%                  428.1        +22.44%

                                                       │ XOR_(ST_ignored) │               XOROptST                │                 XORCST                 │
                                                       │       B/op       │     B/op      vs base                 │     B/op      vs base                  │
Appender/vt=constant/st=0-2                                    248.0 ± 0%     248.0 ± 0%        ~ (p=1.000 n=6) ¹     248.0 ± 0%         ~ (p=1.000 n=6) ¹
Appender/vt=constant/st=cumulative-2                           248.0 ± 0%     248.0 ± 0%        ~ (p=1.000 n=6) ¹     248.0 ± 0%         ~ (p=1.000 n=6) ¹
Appender/vt=constant/st=delta-2                                248.0 ± 0%     248.0 ± 0%        ~ (p=1.000 n=6) ¹     504.0 ± 0%  +103.23% (p=0.002 n=6)
Appender/vt=random_steps/st=0-2                               1016.0 ± 0%    1016.0 ± 0%        ~ (p=1.000 n=6) ¹    1016.0 ± 0%         ~ (p=1.000 n=6) ¹
Appender/vt=random_steps/st=cumulative-2                      1016.0 ± 0%    1016.0 ± 0%        ~ (p=1.000 n=6) ¹    1016.0 ± 0%         ~ (p=1.000 n=6) ¹
Appender/vt=random_steps/st=delta-2                           1016.0 ± 0%    1912.0 ± 0%  +88.19% (p=0.002 n=6)      1912.0 ± 0%   +88.19% (p=0.002 n=6)
Appender/vt=random_0-1/st=0-2                                1.867Ki ± 0%   1.867Ki ± 0%        ~ (p=1.000 n=6) ¹   1.867Ki ± 0%         ~ (p=1.000 n=6) ¹
Appender/vt=negrandom_0-1/st=0-2                             1.867Ki ± 0%   1.867Ki ± 0%        ~ (p=1.000 n=6) ¹   1.867Ki ± 0%         ~ (p=1.000 n=6) ¹
Appender/vt=random_0-1/st=cumulative-2                       1.867Ki ± 0%   1.867Ki ± 0%        ~ (p=1.000 n=6) ¹   1.867Ki ± 0%         ~ (p=1.000 n=6) ¹
Appender/vt=random_0-1/st=cumulative-periodic-resets-2       1.867Ki ± 0%   3.242Ki ± 0%  +73.64% (p=0.002 n=6)     3.242Ki ± 0%   +73.64% (p=0.002 n=6)
Appender/vt=random_0-1/st=cumulative-periodic-zeros-2        1.867Ki ± 0%   3.242Ki ± 0%  +73.64% (p=0.002 n=6)     3.242Ki ± 0%   +73.64% (p=0.002 n=6)
Appender/vt=random_0-1/st=delta-2                            1.867Ki ± 0%   3.242Ki ± 0%  +73.64% (p=0.002 n=6)     3.242Ki ± 0%   +73.64% (p=0.002 n=6)
geomean                                                        979.7        1.158Ki       +21.00%                   1.228Ki        +28.37%
¹ all samples are equal

                                                       │ XOR_(ST_ignored) │              XOROptST               │               XORCST                │
                                                       │    allocs/op     │ allocs/op   vs base                 │ allocs/op   vs base                 │
Appender/vt=constant/st=0-2                                    5.000 ± 0%   4.000 ± 0%  -20.00% (p=0.002 n=6)     4.000 ± 0%  -20.00% (p=0.002 n=6)
Appender/vt=constant/st=cumulative-2                           5.000 ± 0%   4.000 ± 0%  -20.00% (p=0.002 n=6)     4.000 ± 0%  -20.00% (p=0.002 n=6)
Appender/vt=constant/st=delta-2                                5.000 ± 0%   4.000 ± 0%  -20.00% (p=0.002 n=6)     5.000 ± 0%        ~ (p=1.000 n=6) ¹
Appender/vt=random_steps/st=0-2                                7.000 ± 0%   6.000 ± 0%  -14.29% (p=0.002 n=6)     6.000 ± 0%  -14.29% (p=0.002 n=6)
Appender/vt=random_steps/st=cumulative-2                       7.000 ± 0%   6.000 ± 0%  -14.29% (p=0.002 n=6)     6.000 ± 0%  -14.29% (p=0.002 n=6)
Appender/vt=random_steps/st=delta-2                            7.000 ± 0%   7.000 ± 0%        ~ (p=1.000 n=6) ¹   7.000 ± 0%        ~ (p=1.000 n=6) ¹
Appender/vt=random_0-1/st=0-2                                  8.000 ± 0%   7.000 ± 0%  -12.50% (p=0.002 n=6)     7.000 ± 0%  -12.50% (p=0.002 n=6)
Appender/vt=negrandom_0-1/st=0-2                               8.000 ± 0%   7.000 ± 0%  -12.50% (p=0.002 n=6)     7.000 ± 0%  -12.50% (p=0.002 n=6)
Appender/vt=random_0-1/st=cumulative-2                         8.000 ± 0%   7.000 ± 0%  -12.50% (p=0.002 n=6)     7.000 ± 0%  -12.50% (p=0.002 n=6)
Appender/vt=random_0-1/st=cumulative-periodic-resets-2         8.000 ± 0%   8.000 ± 0%        ~ (p=1.000 n=6) ¹   8.000 ± 0%        ~ (p=1.000 n=6) ¹
Appender/vt=random_0-1/st=cumulative-periodic-zeros-2          8.000 ± 0%   8.000 ± 0%        ~ (p=1.000 n=6) ¹   8.000 ± 0%        ~ (p=1.000 n=6) ¹
Appender/vt=random_0-1/st=delta-2                              8.000 ± 0%   8.000 ± 0%        ~ (p=1.000 n=6) ¹   8.000 ± 0%        ~ (p=1.000 n=6) ¹
geomean                                                        6.880        6.133       -10.85%                   6.248        -9.18%
¹ all samples are equal

Reads

benchstat -col /fmt bw.bench/iter.txt     
goos: darwin
goarch: arm64
pkg: github.com/prometheus/prometheus/tsdb/chunkenc
cpu: Apple M1 Pro
                                                       │ XOR_(ST_ignored) │              XOROptST               │               XORCST                │
                                                       │      sec/op      │    sec/op     vs base               │    sec/op     vs base               │
Iterator/vt=constant/st=0-2                                   951.9n ± 6%    995.6n ± 1%        ~ (p=0.065 n=6)    985.8n ± 2%        ~ (p=0.145 n=6)
Iterator/vt=constant/st=cumulative-2                          914.3n ± 1%   1021.0n ± 4%  +11.67% (p=0.002 n=6)   1002.5n ± 1%   +9.65% (p=0.002 n=6)
Iterator/vt=constant/st=delta-2                               919.4n ± 1%   1357.5n ± 6%  +47.64% (p=0.002 n=6)   1812.5n ± 8%  +97.13% (p=0.002 n=6)
Iterator/vt=random_steps/st=0-2                               2.105µ ± 5%    2.116µ ± 1%        ~ (p=0.195 n=6)    2.101µ ± 6%        ~ (p=0.937 n=6)
Iterator/vt=random_steps/st=cumulative-2                      1.999µ ± 2%    2.024µ ± 0%        ~ (p=0.058 n=6)    2.043µ ± 2%   +2.23% (p=0.004 n=6)
Iterator/vt=random_steps/st=delta-2                           2.070µ ± 5%    3.173µ ± 0%  +53.29% (p=0.002 n=6)    3.100µ ± 1%  +49.76% (p=0.002 n=6)
Iterator/vt=random_0-1/st=0-2                                 2.285µ ± 8%    2.332µ ± 0%        ~ (p=0.058 n=6)    2.329µ ± 1%        ~ (p=0.061 n=6)
Iterator/vt=negrandom_0-1/st=0-2                              2.478µ ± 2%    2.492µ ± 2%        ~ (p=0.288 n=6)    2.474µ ± 8%        ~ (p=0.905 n=6)
Iterator/vt=random_0-1/st=cumulative-2                        2.226µ ± 1%    2.237µ ± 1%        ~ (p=0.102 n=6)    2.225µ ± 0%        ~ (p=0.727 n=6)
Iterator/vt=random_0-1/st=cumulative-periodic-resets-2        2.165µ ± 1%    3.284µ ± 0%  +51.72% (p=0.002 n=6)    2.992µ ± 1%  +38.23% (p=0.002 n=6)
Iterator/vt=random_0-1/st=cumulative-periodic-zeros-2         2.195µ ± 1%    3.280µ ± 1%  +49.44% (p=0.002 n=6)    3.119µ ± 3%  +42.11% (p=0.002 n=6)
Iterator/vt=random_0-1/st=delta-2                             2.191µ ± 1%    3.030µ ± 4%  +38.27% (p=0.002 n=6)    2.783µ ± 1%  +27.00% (p=0.002 n=6)
geomean                                                       1.765µ         2.114µ       +19.76%                  2.113µ       +19.73%

                                                       │ XOR_(ST_ignored) │              XOROptST              │               XORCST                │
                                                       │     B/chunk      │   B/chunk    vs base               │   B/chunk    vs base                │
Iterator/vt=constant/st=0-2                                    48.00 ± 0%    49.00 ± 0%   +2.08% (p=0.002 n=6)    49.00 ± 0%    +2.08% (p=0.002 n=6)
Iterator/vt=constant/st=cumulative-2                           48.00 ± 0%    55.00 ± 0%  +14.58% (p=0.002 n=6)    55.00 ± 0%   +14.58% (p=0.002 n=6)
Iterator/vt=constant/st=delta-2                                48.00 ± 0%    73.00 ± 0%  +52.08% (p=0.002 n=6)   176.00 ± 0%  +266.67% (p=0.002 n=6)
Iterator/vt=random_steps/st=0-2                                455.0 ± 0%    456.0 ± 0%   +0.22% (p=0.002 n=6)    456.0 ± 0%    +0.22% (p=0.002 n=6)
Iterator/vt=random_steps/st=cumulative-2                       428.0 ± 0%    435.0 ± 0%   +1.64% (p=0.002 n=6)    435.0 ± 0%    +1.64% (p=0.002 n=6)
Iterator/vt=random_steps/st=delta-2                            435.0 ± 0%    679.0 ± 0%  +56.09% (p=0.002 n=6)    563.0 ± 0%   +29.43% (p=0.002 n=6)
Iterator/vt=random_0-1/st=0-2                                  837.0 ± 0%    838.0 ± 0%   +0.12% (p=0.002 n=6)    838.0 ± 0%    +0.12% (p=0.002 n=6)
Iterator/vt=negrandom_0-1/st=0-2                               842.0 ± 0%    843.0 ± 0%   +0.12% (p=0.002 n=6)    843.0 ± 0%    +0.12% (p=0.002 n=6)
Iterator/vt=random_0-1/st=cumulative-2                         842.0 ± 0%    849.0 ± 0%   +0.83% (p=0.002 n=6)    849.0 ± 0%    +0.83% (p=0.002 n=6)
Iterator/vt=random_0-1/st=cumulative-periodic-resets-2         846.0 ± 0%   1287.0 ± 0%  +52.13% (p=0.002 n=6)   1056.0 ± 0%   +24.82% (p=0.002 n=6)
Iterator/vt=random_0-1/st=cumulative-periodic-zeros-2          837.0 ± 0%   1344.0 ± 0%  +60.57% (p=0.002 n=6)   1157.0 ± 0%   +38.23% (p=0.002 n=6)
Iterator/vt=random_0-1/st=delta-2                              849.0 ± 0%   1093.0 ± 0%  +28.74% (p=0.002 n=6)    977.0 ± 0%   +15.08% (p=0.002 n=6)
geomean                                                        349.7         419.9       +20.08%                  428.1        +22.44%

                                                       │ XOR_(ST_ignored) │              XOROptST              │               XORCST               │
                                                       │       B/op       │    B/op     vs base                │    B/op     vs base                │
Iterator/vt=constant/st=0-2                                  0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=6) ¹   0.000 ± 0%       ~ (p=1.000 n=6) ¹
Iterator/vt=constant/st=cumulative-2                         0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=6) ¹   0.000 ± 0%       ~ (p=1.000 n=6) ¹
Iterator/vt=constant/st=delta-2                              0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=6) ¹   0.000 ± 0%       ~ (p=1.000 n=6) ¹
Iterator/vt=random_steps/st=0-2                              0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=6) ¹   0.000 ± 0%       ~ (p=1.000 n=6) ¹
Iterator/vt=random_steps/st=cumulative-2                     0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=6) ¹   0.000 ± 0%       ~ (p=1.000 n=6) ¹
Iterator/vt=random_steps/st=delta-2                          0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=6) ¹   0.000 ± 0%       ~ (p=1.000 n=6) ¹
Iterator/vt=random_0-1/st=0-2                                0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=6) ¹   0.000 ± 0%       ~ (p=1.000 n=6) ¹
Iterator/vt=negrandom_0-1/st=0-2                             0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=6) ¹   0.000 ± 0%       ~ (p=1.000 n=6) ¹
Iterator/vt=random_0-1/st=cumulative-2                       0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=6) ¹   0.000 ± 0%       ~ (p=1.000 n=6) ¹
Iterator/vt=random_0-1/st=cumulative-periodic-resets-2       0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=6) ¹   0.000 ± 0%       ~ (p=1.000 n=6) ¹
Iterator/vt=random_0-1/st=cumulative-periodic-zeros-2        0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=6) ¹   0.000 ± 0%       ~ (p=1.000 n=6) ¹
Iterator/vt=random_0-1/st=delta-2                            0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=6) ¹   0.000 ± 0%       ~ (p=1.000 n=6) ¹
geomean                                                                 ²               +0.00%               ²               +0.00%               ²
¹ all samples are equal
² summaries must be >0 to compute geomean

                                                       │ XOR_(ST_ignored) │              XOROptST              │               XORCST               │
                                                       │    allocs/op     │ allocs/op   vs base                │ allocs/op   vs base                │
Iterator/vt=constant/st=0-2                                  0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=6) ¹   0.000 ± 0%       ~ (p=1.000 n=6) ¹
Iterator/vt=constant/st=cumulative-2                         0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=6) ¹   0.000 ± 0%       ~ (p=1.000 n=6) ¹
Iterator/vt=constant/st=delta-2                              0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=6) ¹   0.000 ± 0%       ~ (p=1.000 n=6) ¹
Iterator/vt=random_steps/st=0-2                              0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=6) ¹   0.000 ± 0%       ~ (p=1.000 n=6) ¹
Iterator/vt=random_steps/st=cumulative-2                     0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=6) ¹   0.000 ± 0%       ~ (p=1.000 n=6) ¹
Iterator/vt=random_steps/st=delta-2                          0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=6) ¹   0.000 ± 0%       ~ (p=1.000 n=6) ¹
Iterator/vt=random_0-1/st=0-2                                0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=6) ¹   0.000 ± 0%       ~ (p=1.000 n=6) ¹
Iterator/vt=negrandom_0-1/st=0-2                             0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=6) ¹   0.000 ± 0%       ~ (p=1.000 n=6) ¹
Iterator/vt=random_0-1/st=cumulative-2                       0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=6) ¹   0.000 ± 0%       ~ (p=1.000 n=6) ¹
Iterator/vt=random_0-1/st=cumulative-periodic-resets-2       0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=6) ¹   0.000 ± 0%       ~ (p=1.000 n=6) ¹
Iterator/vt=random_0-1/st=cumulative-periodic-zeros-2        0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=6) ¹   0.000 ± 0%       ~ (p=1.000 n=6) ¹
Iterator/vt=random_0-1/st=delta-2                            0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=6) ¹   0.000 ± 0%       ~ (p=1.000 n=6) ¹
geomean                                                                 ²               +0.00%               ²               +0.00%               ²
¹ all samples are equal
² summaries must be >0 to compute geomean

Does this PR introduce a user-facing change?

NONE

Signed-off-by: bwplotka <[email protected]>

krajorama · 2026-01-08T10:33:10Z

trivial rebase

Related to prometheus/proposals#60 Signed-off-by: bwplotka <[email protected]> Signed-off-by: György Krajcsovits <[email protected]>

Signed-off-by: bwplotka <[email protected]>

Signed-off-by: György Krajcsovits <[email protected]>

Stores whether the chunk has ST in the size header as 1 bit. Stores whether the chunk has constant ST in the size header as 1 bit. This limits chunk size to 16K samples from 64K. If there is a change in the ST in the middle of the chunk, we recode the chunk to include ST for all samples, otherwise it's only kept for the first sample or not at all. Signed-off-by: György Krajcsovits <[email protected]>

krajorama · 2026-01-09T08:52:02Z

Added my idea for the chunk encoding.

Stores whether the chunk has ST in the size header as 1 bit.
Stores whether the chunk has constant ST in the size header as 1 bit.
This limits chunk size to 16K samples from 64K.

If there is a change in the ST in the middle of the chunk, we recode
the chunk to include ST for all samples, otherwise it's only kept for
the first sample or not at all.

goos: linux
goarch: amd64
pkg: github.com/prometheus/prometheus/tsdb/chunkenc
cpu: Intel(R) Core(TM) Ultra 7 155U
                                                               │ bench_xor.txt │       bench_xor_recoding.txt        │
                                                               │    sec/op     │    sec/op     vs base               │
Appender/fmt=XOR/vt=constant/st=0-2                               1.146µ ±  2%   1.220µ ± 15%   +6.46% (p=0.002 n=6)
Appender/fmt=XOR/vt=constant/st=cumulative-2                      1.155µ ±  3%   1.413µ ±  8%  +22.35% (p=0.002 n=6)
Appender/fmt=XOR/vt=constant/st=delta-2                           1.167µ ±  3%   1.907µ ±  2%  +63.52% (p=0.002 n=6)
Appender/fmt=XOR/vt=random_steps/st=0-2                           3.002µ ±  2%   3.658µ ±  2%  +21.87% (p=0.002 n=6)
Appender/fmt=XOR/vt=random_steps/st=cumulative-2                  2.600µ ±  2%   3.131µ ±  5%  +20.44% (p=0.002 n=6)
Appender/fmt=XOR/vt=random_steps/st=delta-2                       3.015µ ±  4%   4.504µ ±  3%  +49.37% (p=0.002 n=6)
Appender/fmt=XOR/vt=random_0-1/st=0-2                             4.143µ ±  3%   5.005µ ±  3%  +20.82% (p=0.002 n=6)
Appender/fmt=XOR/vt=negrandom_0-1/st=0-2                          4.979µ ± 14%   5.196µ ±  4%        ~ (p=0.065 n=6)
Appender/fmt=XOR/vt=random_0-1/st=cumulative-2                    4.882µ ±  3%   4.934µ ±  2%        ~ (p=0.818 n=6)
Appender/fmt=XOR/vt=random_0-1/st=cumulative-periodic-resets-2    4.846µ ±  4%   7.037µ ±  3%  +45.23% (p=0.002 n=6)
Appender/fmt=XOR/vt=random_0-1/st=cumulative-periodic-zeros-2     4.706µ ±  5%   7.382µ ±  4%  +56.86% (p=0.002 n=6)
Appender/fmt=XOR/vt=random_0-1/st=delta-2                         4.706µ ±  8%   5.405µ ±  3%  +14.85% (p=0.002 n=6)
geomean                                                           2.925µ         3.677µ        +25.71%

                                                               │ bench_xor.txt │         bench_xor_recoding.txt         │
                                                               │     B/op      │     B/op      vs base                  │
Appender/fmt=XOR/vt=constant/st=0-2                                 248.0 ± 0%     248.0 ± 0%         ~ (p=1.000 n=6) ¹
Appender/fmt=XOR/vt=constant/st=cumulative-2                        248.0 ± 0%     248.0 ± 0%         ~ (p=1.000 n=6) ¹
Appender/fmt=XOR/vt=constant/st=delta-2                             248.0 ± 0%     504.0 ± 0%  +103.23% (p=0.002 n=6)
Appender/fmt=XOR/vt=random_steps/st=0-2                            1016.0 ± 0%    1016.0 ± 0%         ~ (p=1.000 n=6) ¹
Appender/fmt=XOR/vt=random_steps/st=cumulative-2                   1016.0 ± 0%    1016.0 ± 0%         ~ (p=1.000 n=6) ¹
Appender/fmt=XOR/vt=random_steps/st=delta-2                        1016.0 ± 0%    1912.0 ± 0%   +88.19% (p=0.002 n=6)
Appender/fmt=XOR/vt=random_0-1/st=0-2                             1.867Ki ± 0%   1.867Ki ± 0%         ~ (p=1.000 n=6) ¹
Appender/fmt=XOR/vt=negrandom_0-1/st=0-2                          1.867Ki ± 0%   1.867Ki ± 0%         ~ (p=1.000 n=6) ¹
Appender/fmt=XOR/vt=random_0-1/st=cumulative-2                    1.867Ki ± 0%   1.867Ki ± 0%         ~ (p=1.000 n=6) ¹
Appender/fmt=XOR/vt=random_0-1/st=cumulative-periodic-resets-2    1.867Ki ± 0%   2.992Ki ± 0%   +60.25% (p=0.002 n=6)
Appender/fmt=XOR/vt=random_0-1/st=cumulative-periodic-zeros-2     1.867Ki ± 0%   4.742Ki ± 0%  +153.97% (p=0.002 n=6)
Appender/fmt=XOR/vt=random_0-1/st=delta-2                         1.867Ki ± 0%   3.242Ki ± 0%   +73.64% (p=0.002 n=6)
geomean                                                             979.7        1.259Ki        +31.62%
¹ all samples are equal

                                                               │ bench_xor.txt │        bench_xor_recoding.txt        │
                                                               │   allocs/op   │  allocs/op   vs base                 │
Appender/fmt=XOR/vt=constant/st=0-2                                 5.000 ± 0%    4.000 ± 0%  -20.00% (p=0.002 n=6)
Appender/fmt=XOR/vt=constant/st=cumulative-2                        5.000 ± 0%    4.000 ± 0%  -20.00% (p=0.002 n=6)
Appender/fmt=XOR/vt=constant/st=delta-2                             5.000 ± 0%    5.000 ± 0%        ~ (p=1.000 n=6) ¹
Appender/fmt=XOR/vt=random_steps/st=0-2                             7.000 ± 0%    6.000 ± 0%  -14.29% (p=0.002 n=6)
Appender/fmt=XOR/vt=random_steps/st=cumulative-2                    7.000 ± 0%    6.000 ± 0%  -14.29% (p=0.002 n=6)
Appender/fmt=XOR/vt=random_steps/st=delta-2                         7.000 ± 0%    7.000 ± 0%        ~ (p=1.000 n=6) ¹
Appender/fmt=XOR/vt=random_0-1/st=0-2                               8.000 ± 0%    7.000 ± 0%  -12.50% (p=0.002 n=6)
Appender/fmt=XOR/vt=negrandom_0-1/st=0-2                            8.000 ± 0%    7.000 ± 0%  -12.50% (p=0.002 n=6)
Appender/fmt=XOR/vt=random_0-1/st=cumulative-2                      8.000 ± 0%    7.000 ± 0%  -12.50% (p=0.002 n=6)
Appender/fmt=XOR/vt=random_0-1/st=cumulative-periodic-resets-2      8.000 ± 0%   11.000 ± 0%  +37.50% (p=0.002 n=6)
Appender/fmt=XOR/vt=random_0-1/st=cumulative-periodic-zeros-2       8.000 ± 0%   12.000 ± 0%  +50.00% (p=0.002 n=6)
Appender/fmt=XOR/vt=random_0-1/st=delta-2                           8.000 ± 0%    8.000 ± 0%        ~ (p=1.000 n=6) ¹
geomean                                                             6.880         6.637        -3.53%
¹ all samples are equal

Signed-off-by: György Krajcsovits <[email protected]>

bwplotka · 2026-01-29T04:49:24Z

Replaced with #17909

krajorama reviewed Nov 10, 2025

View reviewed changes

Comment thread tsdb/chunkenc/experiment.go Outdated

krajorama self-assigned this Nov 11, 2025

bwplotka force-pushed the st-tsdbchunk1 branch 4 times, most recently from 39c93a9 to 7b20f74 Compare November 19, 2025 11:36

bwplotka force-pushed the st-tsdbchunk1 branch 2 times, most recently from 88ebf26 to 7f3f8fc Compare November 21, 2025 06:27

bwplotka mentioned this pull request Dec 9, 2025

Create microbenchmark for new Record class with StartTime included #17650

Closed

ywwg mentioned this pull request Jan 5, 2026

Implement ST-per-sample in WAL & Head #17790

Closed

3 tasks

krajorama mentioned this pull request Jan 7, 2026

Implement ST-per-sample in tsdb blocks #17791

Closed

bwplotka added 4 commits January 8, 2026 11:25

chore: cleanup tsdb/chunkenc benchmarking suite

790163c

Signed-off-by: bwplotka <[email protected]>

kill hist bench cleanup for now

cb21e37

Signed-off-by: bwplotka <[email protected]>

move to separate benches per op

4816c47

Signed-off-by: bwplotka <[email protected]>

common foreach

b6ee4d2

Signed-off-by: bwplotka <[email protected]>

krajorama force-pushed the st-tsdbchunk1 branch from 66796ed to 93e9f9a Compare January 8, 2026 10:32

krajorama and others added 11 commits January 8, 2026 11:37

feat(chunk,ct): add chunk format that supports start timestamp

4f051a6

Related to prometheus/proposals#60 Signed-off-by: bwplotka <[email protected]> Signed-off-by: György Krajcsovits <[email protected]>

further experiments

ab879fa

Signed-off-by: bwplotka <[email protected]>

xorOptSTv2

a575803

Signed-off-by: bwplotka <[email protected]>

xorOptSTv3

f210863

Signed-off-by: bwplotka <[email protected]>

xorSTConst is not possible; added xorBuffered

2e5824d

Signed-off-by: bwplotka <[email protected]>

xorBuffered is now using SIMD enabled dod.Enc/Dec

14dcff8

Signed-off-by: bwplotka <[email protected]>

xorOptST optimizations

4e6ea90

Signed-off-by: bwplotka <[email protected]>

xorOptST different varbit

e9f1c45

Signed-off-by: bwplotka <[email protected]>

xorOptCST

d2382c8

Signed-off-by: bwplotka <[email protected]>

xorOptCST optimization

efa07ed

Signed-off-by: bwplotka <[email protected]>

fix copyrights

60d6159

Signed-off-by: György Krajcsovits <[email protected]>

krajorama force-pushed the st-tsdbchunk1 branch from 93e9f9a to 60d6159 Compare January 8, 2026 10:38

add unit test for put/read ST diff

2a04d04

Signed-off-by: György Krajcsovits <[email protected]>

add a few more tests for xoroptst

e939e53

Signed-off-by: György Krajcsovits <[email protected]>

krajorama mentioned this pull request Jan 12, 2026

feat(tsdb): new Append parameter and AtST interface for chunks #17840

Merged

krajorama mentioned this pull request Jan 22, 2026

feat(tsdb/chunkenc): add float chunk format with start timestamp support #17909

Merged

bwplotka closed this Jan 29, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[EXPERIMENT] feat(chunk,st): add chunk format that supports start timestamp#17475

[EXPERIMENT] feat(chunk,st): add chunk format that supports start timestamp#17475
bwplotka wants to merge 18 commits into
mainfrom
st-tsdbchunk1

bwplotka commented Nov 4, 2025 •

edited

Loading

Uh oh!

Uh oh!

krajorama commented Jan 8, 2026

Uh oh!

krajorama commented Jan 9, 2026

Uh oh!

bwplotka commented Jan 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

bwplotka commented Nov 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Goals

Implementations so far:

Next

Benchmarks (so far!)

Writes

Reads

Does this PR introduce a user-facing change?

Uh oh!

Uh oh!

krajorama commented Jan 8, 2026

Uh oh!

krajorama commented Jan 9, 2026

Uh oh!

bwplotka commented Jan 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

bwplotka commented Nov 4, 2025 •

edited

Loading