Current implementation of the RW2 sending does not implement exemplars correctly. It keeps the RW1 way of exemplars, where they are sent in a separate TimeSeries message with series labels, exemplar details and without samples. This represents the way Prometheus V1 Appender looks like and the current exemplar in-mem (and WAL) storage. We could call this model as "Exemplars per series".

However RW 2.0 spec follows more reliable way of exporting exemplars, so have exemplar per sample, similar to how our scrape protocols and OTLP have it. This is reflected in the following statement in 2.0 spec:

At least one element in samples or in histograms MUST be provided. A TimeSeries MUST NOT include both samples and histograms. For series which (rarely) would mix float and histogram samples, a separate TimeSeries message MUST be used.

However, the current implementation does not represent this logic as it takes incoming exemplars from WAL and send it as it is, in separate TimeSeries messages without samples. With sharding this might even mean totally separate RW request vs corresponding samples.

Seems this change is something that has been missed during RW2 sending implementation. This won't be easy to implement as well. Feels one solution is to do attack this problem similar to the metadata problem e.g fetch time-relevant exemplars (or simply latest?) from the storage (what if storage is turned off)?

Acceptance Criteria

• Fix exemplars RW2 sending to match the 2.0 spec
• Update RW 2.0 to be more clear on this important change -- it's easy to miss

Alternatives

This missed change is something that likely current early adopters of RW2 missed too (e.g. Mimir, Otel Collector cc @ArthurSens ). It's late to change spec, but we could consider reverting this change RW 2.0 (make exemplars per series). I would vote for NOT doing this given:

• We wanted to limit spec changes at this point especially that big to 2.0
• It gets RW2 back to the stateful receiving for backends that store exemplars per sample (e.g. Google, OTLP).
• For receivers that implement per series, switching RW2 implementation to be according to spec (exemplar per sample) should be noop.

The risk is that this issue will be for longer, so Prometheus sending RW2 won't support exemplars well until then.