promql: Implement </ and >/ operators for trimming native histograms by linasm · Pull Request #17904 · prometheus/prometheus

linasm · 2026-01-21T12:47:01Z

This PR implements the </ and >/ operators for trimming observations from native histograms.

</ (TRIM_UPPER): Removes all observations above a threshold value
>/ (TRIM_LOWER): Removes all observations below a threshold value

Note that this PR continues where #16330 left off as I have no way of pushing into the branch of the original PR.

Which issue(s) does the PR fix:

Does this PR introduce a user-facing change?

[FEATURE]  PromQL: Add `</` and `>/` operators for trimming observations from native histograms.

This implements the TRIM_UPPER (</) and TRIM_LOWER (>/) operators that allow removing observations below or above a threshold from a histogram. The implementation zeros out buckets outside the desired range. It also recalculates the sum, including only bucket counts within the specified threshold range. Fixes prometheus#14651. Signed-off-by: sujal shah <[email protected]>

Signed-off-by: Linas Medziunas <[email protected]>

linasm · 2026-01-22T14:07:30Z

Rebased, the build is green. Ready for review.

Signed-off-by: Linas Medziunas <[email protected]>

krajorama

Thank you for taking this on! I think at this stage we need to clean up and finalize the testcases so we're all on the same page as to what the expectations are for the function.

From previous conversations, my understanding is that when the trim value falls into a bucket that has at least one infinite bound, then we either drop or keep the whole bucket instead of interpolation, as there is no way to actually estimate the correct number with any certainty .

The one exception is [-Inf, some positive number] bucket, where we may assume [0, some positive number] in accordance with the special case in https://prometheus.io/docs/prometheus/latest/querying/functions/#histogram_quantile

perebaj · 2026-01-25T16:44:39Z

I was developing this feature here #17544, but I forgot to link my work on the original PR. I think your effort is more advanced. For this reason, I will discontinue my PR. 🙃

beorn7 · 2026-01-25T21:49:24Z

Thanks for all your collective effort so far.

My current guess is that we should consider all samples in an "infinity bucket" to be "infinitely far away" from any finite value (as this makes most sense statistically). Modulo the special case that @krajorama already mentioned for a [-Inf, some positive number] bucket.

My plan was to also check the test cases so far and see if they make sense. I ran out of steam before the weak ended, so I'll try to do that next week. Once we are all on the same page WRT the test cases, the rest should fall into place pretty easily, given the work already done.

Signed-off-by: Linas Medziunas <[email protected]>

linasm · 2026-01-27T07:39:00Z

Added more HNCB test cases, had to implement handling of a couple additional edge cases.
Looks like I still have to add some edge case tests for Exponential histograms.

Signed-off-by: Linas Medziunas <[email protected]>

beorn7 · 2026-02-13T22:19:23Z

Very cool. Looks like the tests are complete and correct now. I'll have a closer look ASAP (hopefully during the weekend).

beorn7

Tests look perfect to me now.

I only have a few comments about comments.

As said, I think the actual code is in the best hands between @linasm and @krajorama and I won't even look at it.

beorn7 · 2026-02-17T19:21:51Z

Now that we have settled on how the operators work precisely, we also need to include documentation in https://github.com/prometheus/prometheus/blob/main/docs/querying/operators.md (presumably in a separate sub-section underneath "binary operators", maybe titled "histogram trim operators".

beorn7 · 2026-02-17T19:35:03Z

And one more tiny thing: The operators also work between histogram vectors and float vectors. However, we do not have a test for that. Could you add just one simple test (which like three histograms and three floats of which two match each other or something).

Signed-off-by: Linas Medziunas <[email protected]>

linasm · 2026-02-18T13:50:41Z

Updated test comments as suggested above. Added a test case with join (histogram series vs float series).

beorn7

Approving just for tests.

I leave the "real" code review to @krajorama.

@krajorama please feel free to merge once you are happy with the code – and ideally there is documentation in https://github.com/prometheus/prometheus/blob/main/docs/querying/operators.md .

Signed-off-by: Linas Medziunas <[email protected]>

linasm · 2026-02-20T13:16:02Z

ideally there is documentation in https://github.com/prometheus/prometheus/blob/main/docs/querying/operators.md

I've just pushed my first shot at documenting trim operators. I chose to go with a rather high level description here, without diving into any of those ugly edge cases. Feedback is welcome.

beorn7 · 2026-02-24T13:25:35Z

Documentation looks good to me. I think it is consistent with the level of detail we usually provide on the website.

I'm just wondering if we should document all the edgecases somewhere beyond what's in the tests. I'm willing to do that in the spec, i.e. over in prometheus/docs#2823 . But before doing so, I would like to hear your thoughts.

krajorama

LGTM as first iteration. I feel there should be some ways to simplify the code, but this is a good starting point with the tests settled. Also I'd like to test this to shadow our current SLO alerts that use histogram_fraction * histogram_count trick. And get feedback from others as well.

krajorama · 2026-02-24T16:11:57Z

Documentation looks good to me. I think it is consistent with the level of detail we usually provide on the website.

I'm just wondering if we should document all the edgecases somewhere beyond what's in the tests. I'm willing to do that in the spec, i.e. over in prometheus/docs#2823 . But before doing so, I would like to hear your thoughts.

If you would add the edge cases to the spec, that would be greatly appreciated.

linasm · 2026-02-24T19:20:53Z

Thanks for landing this @krajorama . There was a TODO to extract trimming logics from the already over bloated engine.go to floag_histogram.go where IMHO it fits naturally. In case you would agree with this statement, I have a PR ready (no changes in logic, just moving the code around): #18185

##### [\`v3.11.0\`](https://github.com/prometheus/prometheus/releases/tag/v3.11.0) - \[CHANGE] Hetzner SD: The `__meta_hetzner_datacenter` label is deprecated for the role `robot` but kept for backward compatibility, use the `__meta_hetzner_robot_datacenter` label instead. For the role `hcloud`, the label is deprecated and will stop working after the 1 July 2026. [#17850](prometheus/prometheus#17850) - \[CHANGE] Hetzner SD: The `__meta_hetzner_hcloud_datacenter_location` and `__meta_hetzner_hcloud_datacenter_location_network_zone` labels are deprecated, use the `__meta_hetzner_hcloud_location` and `__meta_hetzner_hcloud_location_network_zone` labels instead. [#17850](prometheus/prometheus#17850) - \[CHANGE] Promtool: Redirect debug output to stderr to avoid interfering with stdout-based tool output. [#18346](prometheus/prometheus#18346) - \[FEATURE] AWS SD: Add Elasticache Role. [#18099](prometheus/prometheus#18099) - \[FEATURE] AWS SD: Add RDS Role. [#18206](prometheus/prometheus#18206) - \[FEATURE] Azure SD: Add support for Azure Workload Identity authentication method. [#17207](prometheus/prometheus#17207) - \[FEATURE] Discovery: Introduce `prometheus_sd_last_update_timestamp_seconds` metric to track the last time a service discovery update was sent to consumers. [#18194](prometheus/prometheus#18194) - \[FEATURE] Kubernetes SD: Add support for node role selectors for pod roles. [#18006](prometheus/prometheus#18006) - \[FEATURE] Kubernetes SD: Introduce pod-based labels for deployment, cronjob, and job controller names: `__meta_kubernetes_pod_deployment_name`, `__meta_kubernetes_pod_cronjob_name` and `__meta_kubernetes_pod_job_name`, respectively. [#17774](prometheus/prometheus#17774) - \[FEATURE] PromQL: Add `</` and `>/` operators for trimming observations from native histograms. [#17904](prometheus/prometheus#17904) - \[FEATURE] PromQL: Add experimental `histogram_quantiles` variadic function for computing multiple quantiles at once. [#17285](prometheus/prometheus#17285) - \[FEATURE] TSDB: Add `storage.tsdb.retention.percentage` configuration to configure the maximum percent of disk usable for TSDB storage. [#18080](prometheus/prometheus#18080) - \[FEATURE] TSDB: Add an experimental `fast-startup` feature flag that writes a `series_state.json` file to the WAL directory to track active series state across restarts. [#18303](prometheus/prometheus#18303) - \[FEATURE] TSDB: Add an experimental `st-storage` feature flag. When enabled, Prometheus stores ingested start timestamps (ST, previously called Created Timestamp) from scrape or OTLP in the TSDB and Agent WAL, and exposes them via Remote Write 2. [#18062](prometheus/prometheus#18062) - \[FEATURE] TSDB: Add an experimental `xor2-encoding` feature flag for the new TSDB block float sample chunk encoding that is optimized for scraped data and allows encoding start timestamps. [#18062](prometheus/prometheus#18062) - \[ENHANCEMENT] HTTP client: Add AWS `external_id` support for sigv4. [#17916](prometheus/prometheus#17916) - \[ENHANCEMENT] Kubernetes SD: Deduplicate deprecation warning logs from the Kubernetes API to reduce noise. [#17829](prometheus/prometheus#17829) - \[ENHANCEMENT] TSDB: Remove old temporary checkpoints when creating a Checkpoint. [#17598](prometheus/prometheus#17598) - \[ENHANCEMENT] UI: Add autocomplete support for experimental `first_over_time` and `ts_of_first_over_time` PromQL functions. [#18318](prometheus/prometheus#18318) - \[ENHANCEMENT] Vultr SD: Upgrade govultr library from v2 to v3 for continued security patches and maintenance. [#18347](prometheus/prometheus#18347) - \[PERF] PromQL: Improve performance and reduce heap allocations in joins (VectorBinop)/And/Or/Unless. [#17159](prometheus/prometheus#17159) - \[PERF] PromQL: Partially address performance regression in native histogram aggregations due to using `KahanAdd`. [#18252](prometheus/prometheus#18252) - \[PERF] Remote write: Optimize WAL watching used for RW sending to reuse internal buffers. [#18250](prometheus/prometheus#18250) - \[PERF] TSDB: Optimize LabelValues intersection performance for matchers. [#18069](prometheus/prometheus#18069) - \[PERF] UI: Skip restacking on hover in stacked series charts. [#18230](prometheus/prometheus#18230) - \[BUGFIX] AWS SD: Fix EC2 SD ignoring the configured `endpoint` option, a regression from the AWS SDK v2 migration. [#18133](prometheus/prometheus#18133) - \[BUGFIX] AWS SD: Fix panic in EC2 SD when DescribeAvailabilityZones returns nil ZoneName or ZoneId. [#18133](prometheus/prometheus#18133) - \[BUGFIX] Agent: Fix memory leak caused by duplicate SeriesRefs being loaded as active series. [#17538](prometheus/prometheus#17538) - \[BUGFIX] Alerting: Fix alert state incorrectly resetting to pending when the FOR period is increased in the config file. [#18244](prometheus/prometheus#18244) - \[BUGFIX] Azure SD: Fix system-assigned managed identity not working when `client_id` is empty. [#18323](prometheus/prometheus#18323) - \[BUGFIX] Consul SD: Fix filter parameter not being applied to health service endpoint, causing Node and Node.Meta filters to be ignored. [#17349](prometheus/prometheus#17349) - \[BUGFIX] Kubernetes SD: Fix duplicate targets generated by `*DualStack` EndpointSlices policies. [#18192](prometheus/prometheus#18192) - \[BUGFIX] OTLP: Fix ErrTooOldSample being returned as HTTP 500 instead of 400 in PRW v2 histogram write paths, preventing infinite client retry loops. [#18084](prometheus/prometheus#18084) - \[BUGFIX] OTLP: Fix exemplars getting mixed between incorrect parts of a histogram. [#18056](prometheus/prometheus#18056) - \[BUGFIX] PromQL: Do not skip histogram buckets in queries where histogram trimming is used. [#18263](prometheus/prometheus#18263) - \[BUGFIX] Remote write: Fix `prometheus_remote_storage_sent_batch_duration_seconds` measuring before the request was sent. [#18214](prometheus/prometheus#18214) - \[BUGFIX] Rules: Fix alert state restoration when rule labels contain Go template expressions. [#18375](prometheus/prometheus#18375) - \[BUGFIX] Scrape: Fix panic when parsing bare label names without an equal sign in brace-only metric notation. [#18229](prometheus/prometheus#18229) - \[BUGFIX] TSDB: Fail early when `use-uncached-io` feature flag is set on unsupported environments. [#18219](prometheus/prometheus#18219) - \[BUGFIX] TSDB: Fall back to CLI flag values when retention is removed from config file. [#18200](prometheus/prometheus#18200) - \[BUGFIX] TSDB: Fix memory leaks in buffer pools by clearing reference fields before returning buffers to pools. [#17895](prometheus/prometheus#17895) - \[BUGFIX] TSDB: Fix missing mmap of histogram chunks during WAL replay. [#18306](prometheus/prometheus#18306) - \[BUGFIX] TSDB: Fix storage.tsdb.retention.time unit mismatch in file causing retention to be 1e6 times longer than configured. [#18200](prometheus/prometheus#18200) - \[BUGFIX] Tracing: Fix missing traceID in query log when tracing is enabled, previously only spanID was emitted. [#18189](prometheus/prometheus#18189) - \[BUGFIX] UI: Fix tooltip Y-offset drift when using multiple graph panels. [#18228](prometheus/prometheus#18228) - \[BUGFIX] UI: Update retention display in runtime info when config is reloaded. [#18200](prometheus/prometheus#18200)

#### What this PR does Implement histogram trim operators in MQE. Most of the actual logic is delegated to `TrimBuckets()` in prometheus. #### Which issue(s) this PR fixes or relates to Brings prometheus/prometheus#17904 into MQE. #### Checklist - [x] Tests updated. - [ ] Documentation added. - [x] `CHANGELOG.md` updated - the order of entries should be `[CHANGE]`, `[FEATURE]`, `[ENHANCEMENT]`, `[BUGFIX]`. If changelog entry is not needed, please add the `changelog-not-needed` label to the PR. - [ ] [`about-versioning.md`](https://github.com/grafana/mimir/blob/main/docs/sources/mimir/configure/about-versioning.md) updated with experimental features.  --- > [!NOTE] > **Medium Risk** > Changes affect PromQL evaluation correctness for native histogram queries across binary ops and optimizations, though logic is largely delegated to Prometheus and covered by extensive testdata. > > **Overview** > **MQE** gains PromQL native histogram trim operators **`</`** (trim upper) and **`>/`** (trim lower), with bucket math delegated to Prometheus **`TrimBuckets()`**. > > Evaluation wires **`TRIM_UPPER` / `TRIM_LOWER`** into vector–scalar binary ops (histogram on the left, float on the right only); invalid type mixes emit info annotations, and scalar–scalar trim returns a query error instead of panicking. **`promqlext.RetainsMetricName`** centralizes label behavior so trim ops keep **`__name__`** like filter comparisons— affecting grouped joins, vector–scalar metadata, and when **`DeduplicateAndMerge`** is inserted in planning. > > The query plan adds **`BINARY_TRIM_*`** op types, and the skip-histogram-buckets optimization is disabled under trim expressions because buckets must be decoded. **CHANGELOG** and broad **PromQL testdata** (including previously skipped upstream trim cases) cover behavior and **`group_left`** metric-name retention. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit a897013. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup>  --------- Co-authored-by: Charles Korn <[email protected]>

linasm marked this pull request as ready for review January 21, 2026 12:49

linasm requested review from Nexucis, juliusv and roidelapluie as code owners January 21, 2026 12:49

linasm mentioned this pull request Jan 21, 2026

promql: Implement </ and >/ operators for trimming native histograms. #16330

Closed

krajorama self-assigned this Jan 21, 2026

krajorama requested review from beorn7 and krajorama January 21, 2026 13:08

linasm changed the title ~~promql: Implement </ and >/ operators for trimming native histogramsTrim histogram~~ promql: Implement </ and >/ operators for trimming native histograms Jan 21, 2026

sujalshah-bit added 2 commits January 22, 2026 15:22

linasm force-pushed the trim_histogram branch from f276a7f to dbb6540 Compare January 22, 2026 13:23

Address PR comments

25bd93f

Signed-off-by: Linas Medziunas <[email protected]>

linasm force-pushed the trim_histogram branch from dbb6540 to 25bd93f Compare January 22, 2026 13:41

Nexucis reviewed Jan 22, 2026

View reviewed changes

Comment thread web/ui/module/codemirror-promql/src/parser/parser.ts Outdated

linasm force-pushed the trim_histogram branch from ad69b77 to 8b66349 Compare January 23, 2026 08:05

Fix web UI validation

e326490

Signed-off-by: Linas Medziunas <[email protected]>

linasm force-pushed the trim_histogram branch from 8b66349 to e326490 Compare January 23, 2026 08:08

linasm requested a review from Nexucis January 23, 2026 08:11

krajorama reviewed Jan 25, 2026

View reviewed changes

perebaj reviewed Jan 25, 2026

View reviewed changes

Comment thread promql/engine.go

Comment thread promql/promqltest/testdata/native_histograms.test

beorn7 mentioned this pull request Jan 25, 2026

promql: Implement </ and >/ operators for trimming native histograms #17544

Closed

linasm requested review from krajorama and perebaj January 26, 2026 13:09

Handle infinity buckets conservatively

001fc1b

Signed-off-by: Linas Medziunas <[email protected]>

linasm force-pushed the trim_histogram branch from a10897b to 001fc1b Compare January 27, 2026 07:37

linasm added 2 commits February 13, 2026 16:21

Fix for nhcb [-Inf; -x) bucket

c657e85

Signed-off-by: Linas Medziunas <[email protected]>

Fix for nhcb +Inf bucket

86248a5

Signed-off-by: Linas Medziunas <[email protected]>

linasm force-pushed the trim_histogram branch from 0671a29 to 86248a5 Compare February 13, 2026 14:43

linasm requested a review from beorn7 February 13, 2026 14:46

beorn7 suggested changes Feb 17, 2026

View reviewed changes

linasm added 2 commits February 18, 2026 15:38

Update/fix test comments

c753252

Signed-off-by: Linas Medziunas <[email protected]>

Add test case with join

38a7bef

Signed-off-by: Linas Medziunas <[email protected]>

beorn7 approved these changes Feb 18, 2026

View reviewed changes

beorn7 mentioned this pull request Feb 18, 2026

Histogram spec: Rework and amend operators section prometheus/docs#2823

Merged

Documentation

fbb706b

Signed-off-by: Linas Medziunas <[email protected]>

krajorama approved these changes Feb 24, 2026

View reviewed changes

krajorama merged commit 5d3f9ee into prometheus:main Feb 24, 2026
53 of 54 checks passed

linasm mentioned this pull request Feb 24, 2026

chore(histogram): Move histogram trimming code out of engine.go #18185

Merged

This was referenced Feb 25, 2026

[main] Merge upstream prometheus/main to c317f9254e09 grafana/mimir-prometheus#1116

Merged

[main] Update mimir-prometheus to 7c22e95a1b6f grafana/mimir#14488

Merged

Idea: query sharding of native histogram trim operators grafana/mimir#14491

Open

heliapb mentioned this pull request Apr 6, 2026

feat: add support for trim op for native histograms + histogram_quantiles perses/promql-builder#38

Merged

github-actions Bot mentioned this pull request Apr 20, 2026

Update prom/prometheus Docker tag to v3.11.0 claytono/infra#2051

Merged

1 task

zenador mentioned this pull request Jun 16, 2026

MQE: Implement histogram trim operators grafana/mimir#15708

Merged

4 tasks

Uh oh!

Conversation

linasm commented Jan 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue(s) does the PR fix:

Does this PR introduce a user-facing change?

Uh oh!

linasm commented Jan 22, 2026

Uh oh!

Uh oh!

krajorama left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

perebaj commented Jan 25, 2026

Uh oh!

Uh oh!

Uh oh!

beorn7 commented Jan 25, 2026

Uh oh!

linasm commented Jan 27, 2026

Uh oh!

beorn7 commented Feb 13, 2026

Uh oh!

beorn7 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

beorn7 commented Feb 17, 2026

Uh oh!

beorn7 commented Feb 17, 2026

Uh oh!

linasm commented Feb 18, 2026

Uh oh!

beorn7 left a comment

Choose a reason for hiding this comment

Uh oh!

linasm commented Feb 20, 2026

Uh oh!

beorn7 commented Feb 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

krajorama left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

krajorama commented Feb 24, 2026

Uh oh!

Uh oh!

linasm commented Feb 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

linasm commented Jan 21, 2026 •

edited

Loading

beorn7 commented Feb 24, 2026 •

edited

Loading

krajorama left a comment •

edited

Loading