Skip to content

bug: fix possible race condition when re-allocating batch#8629

Merged
yeya24 merged 1 commit into
thanos-io:mainfrom
adezxc:fix_batchable_allocation
Jan 12, 2026
Merged

bug: fix possible race condition when re-allocating batch#8629
yeya24 merged 1 commit into
thanos-io:mainfrom
adezxc:fix_batchable_allocation

Conversation

@adezxc

@adezxc adezxc commented Jan 12, 2026

Copy link
Copy Markdown
Contributor

Modifying the batch while it is being sent may cause a race condition

From grpc-go [1]:

It is not safe to modify the message after calling SendMsg. Tracing
libraries and stats handlers may use the message lazily.

[1] - https://github.com/grpc/grpc-go/blob/629ef39c3da583f56463213975f191661ac40dd2/stream.go#L135-L136

@fpetkovski fpetkovski left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah the gRPC framework owns the data once you call Send, and it's responsible for its lifetime.

@yeya24 yeya24 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks

Comment thread pkg/store/batchable.go
@MichaHoffmann

Copy link
Copy Markdown
Contributor

While we are at it; wdyt of making batch size part of the client request?.that way it's forwards and backwards compatible

@yeya24
yeya24 merged commit 49dde50 into thanos-io:main Jan 12, 2026
22 checks passed
@adezxc

adezxc commented Jan 12, 2026

Copy link
Copy Markdown
Contributor Author

While we are at it; wdyt of making batch size part of the client request?.that way it's forwards and backwards compatible

Sorry, could you elaborate? It is part of SeriesRequest right now [1] and it's propagated through NewQueryableCreator

[1] -

// response_batch_size is used to batch Series messages into one SeriesResponse
// If set to 0 or 1, each Series is sent as a separate response
int64 response_batch_size = 16;

[2] -

thanos/cmd/thanos/query.go

Lines 459 to 467 in 49dde50

queryableCreator = query.NewQueryableCreator(
logger,
extprom.WrapRegistererWithPrefix("thanos_query_", reg),
seriesProxy,
maxConcurrentSelects,
queryTimeout,
deduplicationFunc,
seriesResponseBatchSize,
)

@MichaHoffmann

Copy link
Copy Markdown
Contributor

While we are at it; wdyt of making batch size part of the client request?.that way it's forwards and backwards compatible

Sorry, could you elaborate? It is part of SeriesRequest right now [1] and it's propagated through NewQueryableCreator

[1] -

// response_batch_size is used to batch Series messages into one SeriesResponse
// If set to 0 or 1, each Series is sent as a separate response
int64 response_batch_size = 16;

[2] -

thanos/cmd/thanos/query.go

Lines 459 to 467 in 49dde50

queryableCreator = query.NewQueryableCreator(
logger,
extprom.WrapRegistererWithPrefix("thanos_query_", reg),
seriesProxy,
maxConcurrentSelects,
queryTimeout,
deduplicationFunc,
seriesResponseBatchSize,
)

Ah nice nevermind I brainfarted

coleenquadros pushed a commit to coleenquadros/thanos that referenced this pull request Mar 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants