Essentially implementing logic similar to PROM-31 reading via PromQL option 3 but as a standalone Prometheus PromQL feature flag.

Motivation

NHCB was created to allow custom bucket use cases while providing some benefits of the exponential native histogram like improved TSDB and WAL efficiency, transactionality for exporting/importing and consistent new PromQL syntax.

Unfortunately that PromQL syntax change feels like the main friction for users and it's hard to have Prometheus with nhcb benefits for storage and remote write without rewriting user queries.

For this reason promq-nhcb-as-classic feature flag is proposed that will cause all nhcb to look as normal classic histogram on the PromQL side. This is relatively easy logic that can sit in the middleware between TSDB and PromQL (storage interface), similar to #16841 work.

With such a flag, Prometheus operators will be able to switch to nhcb without users noticing, literally means replacing classic histograms with native ones under the hood, while making sure queries still work.

Just to state the obvious: It's not possible to present exponential native histograms as classic histograms, because that would defeat the benefits of the resolution conversion and it would be unreadable (too many buckets).

Risks

• One argument is that this work is futile, given the user has to eventually switch the syntax to the new histograms anyway; in fact we might demotivate people from doing so. The counter argument is that I don't see users switching to nhcb alone without that, because some the friction of changing PromQL is just too high and too scary. Furthermore the motivation for the new syntax should be the ability to NOT manually specify buckets and auto resolution change of exponential buckets, which nhcb's won't give.
• Query performance impact. For the main use case being Prometheus operators switching to nhcb without users noticing, I don't see any huge query overhead for the simple algorithm below on a single Prometheus box -- it's quite rare to ingest metric_bucket{} and metric{} (e.g. a gauge) in the same time. One interesting case is the migration option of exposing both native and classic histogram - but in this case no one will use the promql-nhcb-as-classic and convert_classic_histograms_to_nhcb options and if they do, they do the performance would be on par with querying nhcb using native PromQL syntax.
• Further ecosystem adoption impact (Thanos, Mimir, Cortex) -- those does not need to implement this flag, although transactionality and efficiency might encourage them - there's no requirement though.

Potential algorithm

• Direct series selection of metric_bucket{} would have to look up metric_bucket{} and metric{}; ignore metric{} samples that are not nhcb and return the float buckets decoded directly or from the custom histogram.
• Direct series selection of metric_count{} similar to buckets, look up both metric{} and metric_count{}. Ignore metric{} samples that are not nhcb.
• Direct series selection of metric_sum{} similar to buckets, look up both metric{} and metric_sum{}. Ignore metric{} samples that are not nhcb.

Querying metric{} should work on nhcb as well as it is now.

This can happen behind the select interface. In future a hint could be passed to allow further systems (e.g. TSDB blocks to know which type to return).

cc @krajorama @beorn7 WDYT? I remember we discussed this at some point and we rejected this as too much work for the benefit, but within Prometheus only scope alon, given hard requirements on Remote Write transactionality for some receivers (Otel and Google), and relatively easy algorithm and implementation, perhaps it's challenable? (: