tsdb: fix pendingCommit condition for classic histogram append by krajorama · Pull Request #18838 · prometheus/prometheus

krajorama · 2026-06-02T07:43:43Z

Picks the fix from #18400 by @cuiweixie (preserving authorship and sign-off) and adds the regression test that @bboreham asked for in review.

Why

headAppender.AppendHistogram set s.pendingCommit = true when appendableHistogram returned a non-nil error, while every other site in tsdb/head_append.go (Append, AppendSTZeroSample, the float histogram branch, AppendHistogramSTZeroSample, and all V2 paths in tsdb/head_append_v2.go) sets it only when err == nil. The integer histogram branch was the lone outlier.

When the check fails (e.g. ErrOutOfOrderSample, ErrTooOldSample), the function returns without adding the series to the appender's batch, so Commit/Rollback never clears pendingCommit for that series. Head GC at tsdb/head.go treats series.pendingCommit == true as part of the "still in use" condition and won't remove the series — a small leak that can persist until the next successful append on it.

What's in this PR

The original one-line fix from tsdb: fix pendingCommit condition for V1 histogram append #18400, with @cuiweixie's authorship and sign-off preserved.
TestAppendHistogramErrorDoesNotSetPendingCommit covering both the integer and float histogram branches in V1. The integer subtest fails on main without the fix; both pass with it.
TestHeadAppenderV2_HistogramErrorDoesNotSetPendingCommit covering the same property for V2. V2 already had the correct condition; the test is a lock-in and was confirmed to fail when the V2 integer condition is inverted locally.

Supersedes #18400 — happy to close that one once this lands, or for the maintainers to prefer the test be pushed onto @cuiweixie's branch instead, whichever path is easier.

[BUGFIX] TSDB: Do not leak head series when an integer histogram append is rejected (e.g. out-of-order).

AppendHistogram used err != nil when deciding to set pendingCommit for integer histograms, while the float histogram branch uses err == nil. Align the classic histogram path so pendingCommit is set only after a successful appendableHistogram check, matching appendableFloatHistogram. Signed-off-by: Weixie Cui <[email protected]>

Add TestAppendHistogramErrorDoesNotSetPendingCommit (V1) and TestHeadAppenderV2_HistogramErrorDoesNotSetPendingCommit (V2), each covering the integer and float histogram branches. The integer V1 branch previously set s.pendingCommit on the error path, which left the flag stuck on existing series whenever an append was rejected (e.g. ErrOutOfOrderSample). Because the failed sample is never added to the appender's batch, Commit/Rollback never clears pendingCommit for that series, and head GC at tsdb/head.go treats it as still in use. The V1 integer subtest fails on main without the prior commit; both subtests pass with it. The V2 paths already use err == nil and the V2 test is a lock-in; inverting the V2 condition locally confirms the test would catch a similar regression there. Signed-off-by: György Krajcsovits <[email protected]>

…➔ v3.13.0) (#1360) This PR contains the following updates: | Package | Update | Change | |---|---|---| | [quay.io/prometheus/prometheus](https://github.com/prometheus/prometheus) | minor | `v3.12.0` → `v3.13.0` | --- ### Release Notes <details> <summary>prometheus/prometheus (quay.io/prometheus/prometheus)</summary> ### [`v3.13.0`](https://github.com/prometheus/prometheus/releases/tag/v3.13.0): 3.13.0 / 2026-07-01 [Compare Source](prometheus/prometheus@v3.12.0...v3.13.0) This is a Long Term Support [LTS](https://prometheus.io/docs/introduction/release-cycle/) release. - \[SECURITY] UI: Bump `sanitize-html` to fix a cross-site scripting vulnerability (CVE-2026-44990). [#18697](prometheus/prometheus#18697) - \[CHANGE] UI: Third-party npm dependency licenses are now embedded in the Prometheus binary and served at `/assets/third-party-licenses.txt`, replacing the `npm_licenses.tar.bz2` archive previously shipped in release tarballs and container images. [#18997](prometheus/prometheus#18997) - \[CHANGE] API: Use SHA-256 instead of SHA-1 to generate rule group pagination tokens. [#18927](prometheus/prometheus#18927) - \[CHANGE] HTTP clients: Credentials (Authorization header, basic auth, bearer token, OAuth2, configured headers) are no longer forwarded when following a redirect to a different host; affects scraping, remote read/write, alerting, and service discovery. Via prometheus/common v0.69.0 (CVE-2025-4673 CVE-2023-45289). [#18949](prometheus/prometheus#18949) - \[CHANGE] promtool: Relative file paths in the file passed to `--http.config.file` are now resolved relative to that config file's directory instead of its parent directory. Via prometheus/common v0.69.0. [#18949](prometheus/prometheus#18949) - \[CHANGE] PromQL: Rename the `min()` and `max()` duration-expression functions (experimental feature flag `experimental-duration-expr`) to `min_of()` and `max_of()` to avoid confusion with the `min` and `max` aggregate operators. [#18687](prometheus/prometheus#18687) - \[FEATURE] API: Add experimental search endpoints to search metric names, label names, and label values. [#18573](prometheus/prometheus#18573) - \[FEATURE] Discovery/AWS: Add ability to filter RDS instances. [#18859](prometheus/prometheus#18859) - \[FEATURE] PromQL: Add `min_of(a, b)` and `max_of(a, b)` scalar experimental functions, returning the smaller or larger of two scalar values. [#18687](prometheus/prometheus#18687) - \[FEATURE] PromQL: Add support for smoothed/anchored rate with native histograms. [#18564](prometheus/prometheus#18564) - \[FEATURE] PromQL: Expose per-query `samplesRead` (and `samplesReadPerStep` with `stats=all` and the `promql-per-step-stats` feature flag) in the query stats response, and add the `prometheus_engine_query_samples_read_total` engine counter. `samplesRead` reflects storage I/O distinct from `totalQueryableSamples`, which counts samples loaded into the evaluator (and so over-counts when a sample is reused across multiple range-vector windows). [#18081](prometheus/prometheus#18081) - \[FEATURE] Scrape: Add `__convert_classic_histograms_to_nhcb__` internal label to allow per-target override of `convert_classic_histograms_to_nhcb` scrape configuration via relabeling. [#18840](prometheus/prometheus#18840) - \[FEATURE] TSDB: Add `storage.tsdb.chunk_encoding.floats` configuration field to select float chunk encoding (`xor` or `xor2`) at runtime, independently of the `--enable-feature=xor2-encoding` flag. [#18769](prometheus/prometheus#18769) - \[FEATURE] remote\_write: Add Certificate support for ingesting data into an Azure Monitor Workspace. [#18217](prometheus/prometheus#18217) - \[FEATURE] Scrape: Add `__always_scrape_classic_histograms__` and `__scrape_native_histograms__` internal labels to allow per-target override of the `always_scrape_classic_histograms` and `scrape_native_histograms` scrape configuration via relabeling. [#18929](prometheus/prometheus#18929) - \[ENHANCEMENT] Release: Container images are now also published to the GitHub Container Registry (ghcr.io). [#18791](prometheus/prometheus#18791) - \[ENHANCEMENT] PromQL: Prettify `fill_left(x) fill_right(x)` as `fill(x)` when both fill values are equal. [#18851](prometheus/prometheus#18851) - \[ENHANCEMENT] UI: Improve autocompletion after closing a function bracket. [#18894](prometheus/prometheus#18894) - \[PERF] Labels: Add case-insensitive prefix matching to speed up evaluation of long case-insensitive regular expressions (up to \~2x faster). [#18540](prometheus/prometheus#18540) - \[PERF] TSDB: Reduce per-sample overhead in chunk population, speeding up affected queries by \~12-15% in benchmarks. [#18699](prometheus/prometheus#18699) - \[PERF] TSDB: Eliminate unnecessary heap allocations in the V2 histogram WAL decoder, reducing allocations by up to 50% and memory by up to 10% for deployments using native histograms with created-timestamp storage enabled (`--enable-feature=created-timestamp-zero-ingestion`). [#18813](prometheus/prometheus#18813) - \[BUGFIX] Discovery/AWS: Fix failure when processing an AWS RDS cluster without instances. [#18845](prometheus/prometheus#18845) - \[BUGFIX] Fix race condition in initTime that could cause ErrOutOfBounds. [#18629](prometheus/prometheus#18629) - \[BUGFIX] PromQL: A range query whose `end` was not aligned to `step` caused subqueries inside it to evaluate past the parent's last actual step, inflating `peakSamples` in the query stats and against the `query.max-samples` limit, and wasting storage I/O reading samples that were never used in the result. [#18081](prometheus/prometheus#18081) - \[BUGFIX] PromQL: A range query containing an at-modifier-unsafe function over a range-vector with an `@` modifier (e.g. `predict_linear(metric[60s] @ T, X)`) silently under-counted `totalQueryableSamples` for steps after step 0. [#18081](prometheus/prometheus#18081) - \[BUGFIX] PromQL: Fix `fill_left`/`fill_right` producing missing samples in range queries when using `group_left`/`group_right`. [#18850](prometheus/prometheus#18850) - \[BUGFIX] PromQL: Fix for resets() and changes() in anchored range extenders with histograms. [#18906](prometheus/prometheus#18906) - \[BUGFIX] PromQL: Fix panic on `1[5m] smoothed` and similar expressions when extended range selectors are enabled. [#18764](prometheus/prometheus#18764) - \[BUGFIX] PromQL: Fix panic when a `smoothed` instant vector selector produces no samples for a series. [#18943](prometheus/prometheus#18943) - \[BUGFIX] PromQL: Fix panic when using a parenthesised plain number as an offset (e.g. `foo offset -(5)`). [#18768](prometheus/prometheus#18768) - \[BUGFIX] promtool: Fix panic when parsing exposition text containing empty braces `{}`. Via prometheus/common v0.69.0. [#18949](prometheus/prometheus#18949) - \[BUGFIX] Promtool: Fix `check healthy` and `check ready` when `--url` ends with a trailing slash. [#18854](prometheus/prometheus#18854) - \[BUGFIX] Rules: Close PromQL query after each rule evaluation to ensure resources are released. [#18733](prometheus/prometheus#18733) - \[BUGFIX] Scaleway SD: Resolve VPC/IPAM-only instances that have no legacy `private_ip` or `public_ip` field, but do have private NICs attached. [#18772](prometheus/prometheus#18772) - \[BUGFIX] TSDB: Do not leak head series when an integer histogram append is rejected (e.g. out-of-order). [#18838](prometheus/prometheus#18838) - \[BUGFIX] UI: Escape label values offered by PromQL autocomplete. [#18658](prometheus/prometheus#18658) - \[BUGFIX] TSDB: Fix chunk snapshot encoding for EncXOR2 chunks, preventing corruption on TSDB restart when EncXOR2-encoded series were present. [#18739](prometheus/prometheus#18739) - \[BUGFIX] TSDB: Store a millisecond timestamp (not a WAL segment number) in walExpiries when a series is evicted via CompactStaleHead/CompactSelectedSeries, so the series's label record is correctly retained in the next WAL checkpoint and replays cleanly. [#18847](prometheus/prometheus#18847) - \[BUGFIX] TSDB: Prevent loss of samples at the chunk-range boundary when CompactSelectedSeries (and CompactStaleHead) evict the series — the per-slice compaction loop now runs one more iteration so the boundary timestamp is captured in a block before the in-memory copy is removed. [#18849](prometheus/prometheus#18849) </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about these updates again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate).  Reviewed-on: https://git.erwanleboucher.dev/eleboucher/homelab/pulls/1360

##### [\`v3.13.0\`](https://github.com/prometheus/prometheus/releases/tag/v3.13.0) This is a Long Term Support [LTS](https://prometheus.io/docs/introduction/release-cycle/) release. - \[SECURITY] UI: Bump `sanitize-html` to fix a cross-site scripting vulnerability (CVE-2026-44990). [#18697](prometheus/prometheus#18697) - \[CHANGE] UI: Third-party npm dependency licenses are now embedded in the Prometheus binary and served at `/assets/third-party-licenses.txt`, replacing the `npm_licenses.tar.bz2` archive previously shipped in release tarballs and container images. [#18997](prometheus/prometheus#18997) - \[CHANGE] API: Use SHA-256 instead of SHA-1 to generate rule group pagination tokens. [#18927](prometheus/prometheus#18927) - \[CHANGE] HTTP clients: Credentials (Authorization header, basic auth, bearer token, OAuth2, configured headers) are no longer forwarded when following a redirect to a different host; affects scraping, remote read/write, alerting, and service discovery. Via prometheus/common v0.69.0 (CVE-2025-4673 CVE-2023-45289). [#18949](prometheus/prometheus#18949) - \[CHANGE] promtool: Relative file paths in the file passed to `--http.config.file` are now resolved relative to that config file's directory instead of its parent directory. Via prometheus/common v0.69.0. [#18949](prometheus/prometheus#18949) - \[CHANGE] PromQL: Rename the `min()` and `max()` duration-expression functions (experimental feature flag `experimental-duration-expr`) to `min_of()` and `max_of()` to avoid confusion with the `min` and `max` aggregate operators. [#18687](prometheus/prometheus#18687) - \[FEATURE] API: Add experimental search endpoints to search metric names, label names, and label values. [#18573](prometheus/prometheus#18573) - \[FEATURE] Discovery/AWS: Add ability to filter RDS instances. [#18859](prometheus/prometheus#18859) - \[FEATURE] PromQL: Add `min_of(a, b)` and `max_of(a, b)` scalar experimental functions, returning the smaller or larger of two scalar values. [#18687](prometheus/prometheus#18687) - \[FEATURE] PromQL: Add support for smoothed/anchored rate with native histograms. [#18564](prometheus/prometheus#18564) - \[FEATURE] PromQL: Expose per-query `samplesRead` (and `samplesReadPerStep` with `stats=all` and the `promql-per-step-stats` feature flag) in the query stats response, and add the `prometheus_engine_query_samples_read_total` engine counter. `samplesRead` reflects storage I/O distinct from `totalQueryableSamples`, which counts samples loaded into the evaluator (and so over-counts when a sample is reused across multiple range-vector windows). [#18081](prometheus/prometheus#18081) - \[FEATURE] Scrape: Add `__convert_classic_histograms_to_nhcb__` internal label to allow per-target override of `convert_classic_histograms_to_nhcb` scrape configuration via relabeling. [#18840](prometheus/prometheus#18840) - \[FEATURE] TSDB: Add `storage.tsdb.chunk_encoding.floats` configuration field to select float chunk encoding (`xor` or `xor2`) at runtime, independently of the `--enable-feature=xor2-encoding` flag. [#18769](prometheus/prometheus#18769) - \[FEATURE] remote\_write: Add Certificate support for ingesting data into an Azure Monitor Workspace. [#18217](prometheus/prometheus#18217) - \[FEATURE] Scrape: Add `__always_scrape_classic_histograms__` and `__scrape_native_histograms__` internal labels to allow per-target override of the `always_scrape_classic_histograms` and `scrape_native_histograms` scrape configuration via relabeling. [#18929](prometheus/prometheus#18929) - \[ENHANCEMENT] Release: Container images are now also published to the GitHub Container Registry (ghcr.io). [#18791](prometheus/prometheus#18791) - \[ENHANCEMENT] PromQL: Prettify `fill_left(x) fill_right(x)` as `fill(x)` when both fill values are equal. [#18851](prometheus/prometheus#18851) - \[ENHANCEMENT] UI: Improve autocompletion after closing a function bracket. [#18894](prometheus/prometheus#18894) - \[PERF] Labels: Add case-insensitive prefix matching to speed up evaluation of long case-insensitive regular expressions (up to \~2x faster). [#18540](prometheus/prometheus#18540) - \[PERF] TSDB: Reduce per-sample overhead in chunk population, speeding up affected queries by \~12-15% in benchmarks. [#18699](prometheus/prometheus#18699) - \[PERF] TSDB: Eliminate unnecessary heap allocations in the V2 histogram WAL decoder, reducing allocations by up to 50% and memory by up to 10% for deployments using native histograms with created-timestamp storage enabled (`--enable-feature=created-timestamp-zero-ingestion`). [#18813](prometheus/prometheus#18813) - \[BUGFIX] Discovery/AWS: Fix failure when processing an AWS RDS cluster without instances. [#18845](prometheus/prometheus#18845) - \[BUGFIX] Fix race condition in initTime that could cause ErrOutOfBounds. [#18629](prometheus/prometheus#18629) - \[BUGFIX] PromQL: A range query whose `end` was not aligned to `step` caused subqueries inside it to evaluate past the parent's last actual step, inflating `peakSamples` in the query stats and against the `query.max-samples` limit, and wasting storage I/O reading samples that were never used in the result. [#18081](prometheus/prometheus#18081) - \[BUGFIX] PromQL: A range query containing an at-modifier-unsafe function over a range-vector with an `@` modifier (e.g. `predict_linear(metric[60s] @ T, X)`) silently under-counted `totalQueryableSamples` for steps after step 0. [#18081](prometheus/prometheus#18081) - \[BUGFIX] PromQL: Fix `fill_left`/`fill_right` producing missing samples in range queries when using `group_left`/`group_right`. [#18850](prometheus/prometheus#18850) - \[BUGFIX] PromQL: Fix for resets() and changes() in anchored range extenders with histograms. [#18906](prometheus/prometheus#18906) - \[BUGFIX] PromQL: Fix panic on `1[5m] smoothed` and similar expressions when extended range selectors are enabled. [#18764](prometheus/prometheus#18764) - \[BUGFIX] PromQL: Fix panic when a `smoothed` instant vector selector produces no samples for a series. [#18943](prometheus/prometheus#18943) - \[BUGFIX] PromQL: Fix panic when using a parenthesised plain number as an offset (e.g. `foo offset -(5)`). [#18768](prometheus/prometheus#18768) - \[BUGFIX] promtool: Fix panic when parsing exposition text containing empty braces `{}`. Via prometheus/common v0.69.0. [#18949](prometheus/prometheus#18949) - \[BUGFIX] Promtool: Fix `check healthy` and `check ready` when `--url` ends with a trailing slash. [#18854](prometheus/prometheus#18854) - \[BUGFIX] Rules: Close PromQL query after each rule evaluation to ensure resources are released. [#18733](prometheus/prometheus#18733) - \[BUGFIX] Scaleway SD: Resolve VPC/IPAM-only instances that have no legacy `private_ip` or `public_ip` field, but do have private NICs attached. [#18772](prometheus/prometheus#18772) - \[BUGFIX] TSDB: Do not leak head series when an integer histogram append is rejected (e.g. out-of-order). [#18838](prometheus/prometheus#18838) - \[BUGFIX] UI: Escape label values offered by PromQL autocomplete. [#18658](prometheus/prometheus#18658) - \[BUGFIX] TSDB: Fix chunk snapshot encoding for EncXOR2 chunks, preventing corruption on TSDB restart when EncXOR2-encoded series were present. [#18739](prometheus/prometheus#18739) - \[BUGFIX] TSDB: Store a millisecond timestamp (not a WAL segment number) in walExpiries when a series is evicted via CompactStaleHead/CompactSelectedSeries, so the series's label record is correctly retained in the next WAL checkpoint and replays cleanly. [#18847](prometheus/prometheus#18847) - \[BUGFIX] TSDB: Prevent loss of samples at the chunk-range boundary when CompactSelectedSeries (and CompactStaleHead) evict the series — the per-slice compaction loop now runs one more iteration so the boundary timestamp is captured in a block before the in-memory copy is removed. [#18849](prometheus/prometheus#18849)

cuiweixie and others added 2 commits June 2, 2026 09:15

krajorama requested review from a team, bwplotka, codesome and jesusvazquez as code owners June 2, 2026 07:43

krajorama requested a review from bboreham June 2, 2026 07:47

bboreham approved these changes Jun 2, 2026

View reviewed changes

bboreham merged commit 87866e0 into main Jun 2, 2026
59 of 60 checks passed

bboreham deleted the tsdb/fix-histogram-pending-commit-condition branch June 2, 2026 09:32

bboreham mentioned this pull request Jun 2, 2026

tsdb: fix pendingCommit condition for V1 histogram append #18400

Closed

github-actions Bot mentioned this pull request Jul 24, 2026

Update prom/prometheus Docker tag to v3.13.1 claytono/infra#2595

Merged

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

tsdb: fix pendingCommit condition for classic histogram append#18838

tsdb: fix pendingCommit condition for classic histogram append#18838
bboreham merged 2 commits into
mainfrom
tsdb/fix-histogram-pending-commit-condition

krajorama commented Jun 2, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

krajorama commented Jun 2, 2026

Why

What's in this PR

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants