storeapi: enable batching of series per SeriesResponse by adezxc · Pull Request #8623 · thanos-io/thanos

adezxc · 2026-01-06T09:08:03Z

This PR adds a batch field to the SeriesResponse message and allows for sending/receiving
multiple Series per request.

The size of the batch is set using query.series-response-batch-size flag and is included in SeriesRequest.
If the StoreAPI is running an older version it will just send series one-by-one, this way keeping backwards-compatibility.

Original suggestion: #7929

I added CHANGELOG entry for this change.

Changes

Add SeriesBatch batch field to SeriesResponse
Create batchableServer struct that accumulates forwarded responses into an array.
Update RespSet and *seriesServer.Send functions to handle batches correctly.

Verification

Some tests, benchmarks are included. We are running this change in production for a few weeks and the network savings are big with query.series-response-batch-size=50, queriers also seem to use a bit less CPU.

adezxc · 2026-01-06T09:12:07Z

Benchmark results with fake, repeating labels on my laptop:

$ benchstat -row ".name /series /sample_count_per_series" -col "/batch_size" ~/playground/benchmark_results.txt
goos: linux
goarch: amd64
pkg: github.com/thanos-io/thanos/pkg/query
cpu: AMD Ryzen 7 PRO 8840U w/ Radeon 780M Graphics
                              │      1       │                  10                  │                 100                  │                 1000                 │                10000                │
                              │    sec/op    │    sec/op     vs base                │    sec/op     vs base                │    sec/op     vs base                │   sec/op     vs base                │
GRPCServerBatching 100 100      771.0µ ±  1%   448.5µ ±  3%  -41.83% (p=0.000 n=10)   535.0µ ± 10%  -30.61% (p=0.000 n=10)   483.8µ ± 24%  -37.24% (p=0.000 n=10)   530.3µ ± 6%  -31.22% (p=0.000 n=10)
GRPCServerBatching 100 240      920.7µ ±  5%   646.9µ ±  3%  -29.73% (p=0.000 n=10)   769.5µ ±  3%  -16.42% (p=0.000 n=10)   739.0µ ±  3%  -19.73% (p=0.000 n=10)   731.2µ ± 3%  -20.58% (p=0.000 n=10)
GRPCServerBatching 1000 100     6.797m ± 14%   2.657m ±  5%  -60.90% (p=0.000 n=10)   3.134m ±  2%  -53.88% (p=0.000 n=10)   3.622m ±  3%  -46.71% (p=0.000 n=10)   3.557m ± 5%  -47.67% (p=0.000 n=10)
GRPCServerBatching 1000 240     7.950m ±  4%   4.720m ±  3%  -40.63% (p=0.000 n=10)   4.261m ±  5%  -46.40% (p=0.000 n=10)   6.270m ±  2%  -21.14% (p=0.000 n=10)   6.420m ± 3%  -19.25% (p=0.000 n=10)
GRPCServerBatching 10000 100    65.57m ±  3%   23.86m ±  3%  -63.61% (p=0.000 n=10)   27.01m ±  3%  -58.80% (p=0.000 n=10)   24.74m ±  5%  -62.26% (p=0.000 n=10)   35.06m ± 7%  -46.54% (p=0.000 n=10)
GRPCServerBatching 10000 240    79.53m ±  6%   41.26m ± 10%  -48.12% (p=0.000 n=10)   36.85m ± 12%  -53.66% (p=0.000 n=10)   45.24m ±  1%  -43.11% (p=0.000 n=10)   65.08m ± 4%  -18.16% (p=0.000 n=10)
GRPCServerBatching 100000 100   635.9m ±  6%   227.7m ±  4%  -64.19% (p=0.000 n=10)   269.1m ±  5%  -57.68% (p=0.000 n=10)   282.1m ± 13%  -55.64% (p=0.000 n=10)   280.1m ± 7%  -55.95% (p=0.000 n=10)
GRPCServerBatching 100000 240   750.3m ±  6%   451.1m ± 25%  -39.88% (p=0.000 n=10)   407.8m ± 12%  -45.65% (p=0.000 n=10)   455.3m ± 29%  -39.32% (p=0.000 n=10)   555.5m ± 3%  -25.96% (p=0.000 n=10)
geomean                         23.58m         11.77m        -50.08%                  12.51m        -46.93%                  13.60m        -42.33%                  15.39m       -34.73%

                              │       1       │                  10                   │                 100                  │                 1000                 │                 10000                 │
                              │     B/op      │     B/op       vs base                │     B/op      vs base                │     B/op      vs base                │     B/op       vs base                │
GRPCServerBatching 100 100      1052.4Ki ± 1%    585.2Ki ± 1%  -44.39% (p=0.000 n=10)   602.0Ki ± 2%  -42.80% (p=0.000 n=10)   604.6Ki ± 3%  -42.55% (p=0.000 n=10)   600.9Ki ±  3%  -42.91% (p=0.000 n=10)
GRPCServerBatching 100 240      1189.4Ki ± 1%   1114.4Ki ± 1%   -6.30% (p=0.000 n=10)   932.1Ki ± 2%  -21.63% (p=0.000 n=10)   919.5Ki ± 3%  -22.69% (p=0.000 n=10)   936.0Ki ±  5%  -21.30% (p=0.000 n=10)
GRPCServerBatching 1000 100      8.338Mi ± 1%    4.404Mi ± 1%  -47.19% (p=0.000 n=10)   5.550Mi ± 2%  -33.44% (p=0.000 n=10)   4.949Mi ± 3%  -40.64% (p=0.000 n=10)   4.966Mi ±  3%  -40.45% (p=0.000 n=10)
GRPCServerBatching 1000 240      9.373Mi ± 0%    7.845Mi ± 1%  -16.30% (p=0.000 n=10)   8.428Mi ± 3%  -10.08% (p=0.000 n=10)   8.498Mi ± 2%   -9.34% (p=0.000 n=10)   8.426Mi ±  4%  -10.11% (p=0.000 n=10)
GRPCServerBatching 10000 100     68.02Mi ± 0%    36.25Mi ± 1%  -46.71% (p=0.000 n=10)   40.88Mi ± 2%  -39.89% (p=0.000 n=10)   38.19Mi ± 2%  -43.86% (p=0.000 n=10)   45.98Mi ± 11%  -32.40% (p=0.000 n=10)
GRPCServerBatching 10000 240     78.89Mi ± 0%    62.62Mi ± 0%  -20.63% (p=0.000 n=10)   65.20Mi ± 1%  -17.36% (p=0.000 n=10)   67.44Mi ± 2%  -14.51% (p=0.000 n=10)   88.11Mi ±  3%  +11.69% (p=0.002 n=10)
GRPCServerBatching 100000 100    611.8Mi ± 0%    342.6Mi ± 0%  -44.00% (p=0.000 n=10)   338.4Mi ± 1%  -44.68% (p=0.000 n=10)   336.4Mi ± 1%  -45.02% (p=0.000 n=10)   405.5Mi ±  3%  -33.73% (p=0.000 n=10)
GRPCServerBatching 100000 240    762.0Mi ± 0%    589.5Mi ± 0%  -22.64% (p=0.000 n=10)   573.6Mi ± 0%  -24.72% (p=0.000 n=10)   682.8Mi ± 6%  -10.39% (p=0.000 n=10)   888.6Mi ±  5%  +16.61% (p=0.000 n=10)
geomean                          26.36Mi         17.75Mi       -32.68%                  18.37Mi       -30.33%                  18.41Mi       -30.17%                  20.63Mi        -21.76%

                              │      1      │                 10                  │                 100                 │                1000                 │                10000                │
                              │  allocs/op  │  allocs/op   vs base                │  allocs/op   vs base                │  allocs/op   vs base                │  allocs/op   vs base                │
GRPCServerBatching 100 100      6.219k ± 0%   2.060k ± 0%  -66.88% (p=0.000 n=10)   1.577k ± 0%  -74.64% (p=0.000 n=10)   1.577k ± 0%  -74.64% (p=0.000 n=10)   1.577k ± 0%  -74.64% (p=0.000 n=10)
GRPCServerBatching 100 240      6.895k ± 0%   2.730k ± 0%  -60.41% (p=0.000 n=10)   2.214k ± 0%  -67.89% (p=0.000 n=10)   2.213k ± 0%  -67.90% (p=0.000 n=10)   2.214k ± 0%  -67.89% (p=0.000 n=10)
GRPCServerBatching 1000 100     59.18k ± 0%   17.97k ± 0%  -69.64% (p=0.000 n=10)   13.22k ± 0%  -77.66% (p=0.000 n=10)   12.52k ± 0%  -78.85% (p=0.000 n=10)   12.52k ± 0%  -78.85% (p=0.000 n=10)
GRPCServerBatching 1000 240     66.62k ± 0%   24.61k ± 0%  -63.06% (p=0.000 n=10)   19.60k ± 0%  -70.58% (p=0.000 n=10)   18.83k ± 0%  -71.74% (p=0.000 n=10)   18.83k ± 0%  -71.74% (p=0.000 n=10)
GRPCServerBatching 10000 100    584.8k ± 0%   176.4k ± 0%  -69.84% (p=0.000 n=10)   129.2k ± 0%  -77.90% (p=0.000 n=10)   122.2k ± 0%  -79.11% (p=0.000 n=10)   121.2k ± 0%  -79.27% (p=0.000 n=10)
GRPCServerBatching 10000 240    662.3k ± 0%   242.1k ± 0%  -63.45% (p=0.000 n=10)   192.8k ± 0%  -70.89% (p=0.000 n=10)   185.3k ± 0%  -72.03% (p=0.000 n=10)   184.6k ± 0%  -72.13% (p=0.000 n=10)
GRPCServerBatching 100000 100   5.750M ± 0%   1.757M ± 0%  -69.45% (p=0.000 n=10)   1.288M ± 0%  -77.59% (p=0.000 n=10)   1.218M ± 0%  -78.83% (p=0.000 n=10)   1.210M ± 0%  -78.95% (p=0.000 n=10)
GRPCServerBatching 100000 240   6.617M ± 0%   2.414M ± 0%  -63.52% (p=0.000 n=10)   1.924M ± 0%  -70.92% (p=0.000 n=10)   1.849M ± 0%  -72.05% (p=0.000 n=10)   1.845M ± 0%  -72.12% (p=0.000 n=10)
geomean                         199.3k        67.87k       -65.95%                  52.29k       -73.77%                  50.44k       -74.69%                  50.32k       -74.75%

Signed-off-by: adezxc <[email protected]>

GiedriusS

We run this in prod and on the most network intensive machine in sum it reduced network traffic by 10x. It also saved about 5 cores on each machine that has 128 cores. I didn't capture any other metrics but memory usage should also drop because the newest grpc-go version reuses buffers.

Not merging this yet because someone might have some review comments.

We will also need to inevitably enable batching because of grpc/grpc-go#8697 - grpc-go internally uses quite big buffers for decompressing so the messages also need to be bigger. It's probably better to fix this on our side instead of grpc-go where it is unlikely that this will be changed.

fpetkovski · 2026-01-09T09:54:26Z

Is the CPU reduction seen in receivers or queriers?

adezxc · 2026-01-09T10:50:15Z

Queriers, for the main part. I checked the average CPU load for a querier node with/without the batching enabled (1 week interval) and the reduction is around 30%. Query hosts are not capped so Giedrius was only talking about the used core count. A very similar reduction in CPU time is seen for receivers as well

Query sum(rate(node_cpu_seconds_total)[5m])

Before receivers:

Before queriers:

After receivers:

After queriers:

xBazilio · 2026-06-23T10:02:59Z

Hello!
This feature mentioned in 0.41.0 release notes. But it isn't here: https://github.com/thanos-io/thanos/blob/v0.41.0/cmd/thanos/query.go
there is no mention of query.series-response-batch-size parameter.
Am I missing something?

EDIT:
found it f3d3972

now it default to 64 and no prameter is needed.

pull-request-size Bot added the size/XL label Jan 6, 2026

adezxc force-pushed the store_batchapi branch from 8cf80dd to 54b9ab7 Compare January 6, 2026 09:10

adezxc force-pushed the store_batchapi branch 3 times, most recently from 8d67a44 to ce4289c Compare January 6, 2026 09:36

adezxc added 2 commits January 6, 2026 11:40

Create proto definitions

d05e90a

Signed-off-by: adezxc <[email protected]>

create Changelog entry

3851e28

Signed-off-by: adezxc <[email protected]>

adezxc force-pushed the store_batchapi branch 2 times, most recently from 9b3be60 to e629f33 Compare January 6, 2026 09:52

adezxc marked this pull request as draft January 6, 2026 10:46

adezxc force-pushed the store_batchapi branch 3 times, most recently from 33af256 to d8b3ac0 Compare January 6, 2026 12:13

create batchable StoreSeries server

44c0fda

Signed-off-by: adezxc <[email protected]>

adezxc force-pushed the store_batchapi branch from d8b3ac0 to 44c0fda Compare January 6, 2026 12:32

adezxc marked this pull request as ready for review January 6, 2026 13:58

GiedriusS approved these changes Jan 8, 2026

View reviewed changes

GiedriusS merged commit 4359ab0 into thanos-io:main Jan 9, 2026
22 checks passed

yeya24 mentioned this pull request Jan 10, 2026

Store Gateway: Enable series response batching cortexproject/cortex#7202

Closed

adezxc mentioned this pull request Jan 12, 2026

query: batch query gRPC responses #8627

Merged

2 tasks

BrewTestBot mentioned this pull request Feb 12, 2026

thanos 0.41.0 Homebrew/homebrew-core#267159

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

storeapi: enable batching of series per SeriesResponse#8623

storeapi: enable batching of series per SeriesResponse#8623
GiedriusS merged 3 commits into
thanos-io:mainfrom
adezxc:store_batchapi

adezxc commented Jan 6, 2026 •

edited

Loading

Uh oh!

adezxc commented Jan 6, 2026

Uh oh!

GiedriusS left a comment •

edited

Loading

Uh oh!

Uh oh!

fpetkovski commented Jan 9, 2026

Uh oh!

adezxc commented Jan 9, 2026 •

edited

Loading

Uh oh!

xBazilio commented Jun 23, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

adezxc commented Jan 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Verification

Uh oh!

adezxc commented Jan 6, 2026

Uh oh!

GiedriusS left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

fpetkovski commented Jan 9, 2026

Uh oh!

adezxc commented Jan 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

xBazilio commented Jun 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

adezxc commented Jan 6, 2026 •

edited

Loading

GiedriusS left a comment •

edited

Loading

adezxc commented Jan 9, 2026 •

edited

Loading

xBazilio commented Jun 23, 2026 •

edited

Loading