Drop old inmemory samples by marctc · Pull Request #13002 · prometheus/prometheus

marctc · 2023-10-18T08:25:48Z

This PR adds a new way to configure the queue config to drop samples whose age exceed
the specified threshold. This useful when recovering from network outages in the case of
the used remote write storage rejects samples that are too old.

The filtering happens in two places:

After decoding samples from the WAL and appended to be sent to the remote write storage.
During the retry attempts when building a write request to the remote write. This adds small overhead
when building the write request as we need to check the age of the samples on every retry.

Fixes #7912
Related #8549

cstyan

No objections to adding this as a feature but I think we need some documentation and potentially a blog post that using this flag is potentially a foot gun.

Some users have compliance requirements for all data being stored, and this configuration option when combined with something like a long MaxBackoff value or long time range of out of order acceptance on the receiving end could lead to unexpected behaviour (for users).

Additionally it would be nice to see benchmark comparisons for the existing benchmarks in storage/remote of before/after this PR. I don't expect any major changes but lets just be sure.

cstyan · 2023-10-20T23:15:06Z

docs/configuration/configuration.md

+  # If set, any sample that is older than  sample_age_limit given 
+  # will not be sent to the remote storage.
+  [ sample_age_limit: <duration> | default = 0s ]


we should clarify here that 0s is essentially disabling this feature, ie samples will always be sent regardless of age

marctc · 2023-10-23T11:42:29Z

Thanks for the review @cstyan!

No objections to adding this as a feature but I think we need some documentation and potentially a blog post that using this flag is potentially a foot gun.

Where do you suggest to do that, besides what we already added?

Additionally it would be nice to see benchmark comparisons for the existing benchmarks in storage/remote of before/after this PR. I don't expect any major changes but lets just be sure.

Do you mean run here prombench or run benchmarks manually?

cstyan · 2023-10-23T23:02:00Z

Where do you suggest to do that, besides what we already added?

I think for now some additional details in the config doc plus a warn level log line when the feature is being used is reasonable. On top of that we could write a draft of a blog post that could go out with this feature, which could outline problems that remote write outages might cause (back of samples leading to a thundering herd/rate limit issues, which other new-ish config options like max retries and retry after headers can also help with).

Do you mean run here prombench or run benchmarks manually?

For now just the existing go benchmarks in storage/remote, maybe use perflock just to ensure we get accurate results.

tpaschalis · 2023-10-24T13:23:12Z

For now just the existing go benchmarks in storage/remote, maybe use perflock just to ensure we get accurate resu

We've run the benchmarks for the 'normal' code path (no filtering happening). Before the optimization in 5461977, we used to see Appending be slower, and allocate a lot more.

$ ~/go/bin/benchstat bench.old bench.new
goos: linux
goarch: amd64
pkg: github.com/prometheus/prometheus/storage/remote
cpu: DO-Premium-AMD
                        │     bench.old      │                 bench.new                  │
                        │       sec/op       │       sec/op        vs base                │
SampleSend-2                    6.882m ±  3%         8.274m ±  9%  +20.22% (p=0.000 n=20)
Startup-2                 0.000001000n ± 30%   0.000001300n ± 77%  +30.00% (p=0.031 n=20)
StreamReadEndpoint-2            54.24µ ±  4%         57.85µ ±  3%   +6.65% (p=0.000 n=20)
RemoteWritehandler-2            4.341µ ±  3%         4.606µ ±  3%   +6.12% (p=0.000 n=20)
RemoteWriteOOOSamples-2        0.1410n ±  5%        0.1491n ±  5%   +5.71% (p=0.027 n=20)
geomean                         46.96n               53.23n        +13.33%

                        │   bench.old    │                bench.new                │
                        │      B/op      │     B/op      vs base                   │
SampleSend-2              1.037Mi ± 4%     2.354Mi ± 2%  +127.00% (p=0.000 n=20)
Startup-2                   0.000 ± 0%       0.000 ± 0%         ~ (p=1.000 n=20) ¹
StreamReadEndpoint-2      23.95Ki ± 0%     23.95Ki ± 0%         ~ (p=1.000 n=20)
RemoteWritehandler-2      2.498Ki ± 1%     2.513Ki ± 1%    +0.59% (p=0.020 n=20)
RemoteWriteOOOSamples-2     0.000 ± 0%       0.000 ± 0%         ~ (p=1.000 n=20) ¹
geomean                                ²                  +17.95%                ²
¹ all samples are equal
² summaries must be >0 to compute geomean

                        │   bench.old   │               bench.new               │
                        │   allocs/op   │  allocs/op   vs base                  │
SampleSend-2              1.498k ± 3%     1.732k ± 5%  +15.69% (p=0.000 n=20)
Startup-2                  0.000 ± 0%      0.000 ± 0%        ~ (p=1.000 n=20) ¹
StreamReadEndpoint-2       239.0 ± 0%      239.0 ± 0%        ~ (p=1.000 n=20) ¹
RemoteWritehandler-2       23.00 ± 0%      23.00 ± 0%        ~ (p=1.000 n=20) ¹
RemoteWriteOOOSamples-2    0.000 ± 0%      0.000 ± 0%        ~ (p=1.000 n=20) ¹
geomean                               ²                 +2.96%                ²
¹ all samples are equal
² summaries must be >0 to compute geomean

After the optimization, the difference is much smaller; the only extra allocation are due to the slice we're keeping the indexes to remove in.

$ ~/go/bin/benchstat bench.old bench.new.1
goos: linux
goarch: amd64
pkg: github.com/prometheus/prometheus/storage/remote
cpu: DO-Premium-AMD
                        │     bench.old      │                bench.new.1                │
                        │       sec/op       │       sec/op        vs base               │
SampleSend-2                    6.882m ±  3%         6.879m ±  4%       ~ (p=0.947 n=20)
Startup-2                 0.000001000n ± 30%   0.000001000n ± 10%       ~ (p=0.594 n=20)
StreamReadEndpoint-2            54.24µ ±  4%         54.94µ ±  4%  +1.28% (p=0.046 n=20)
RemoteWritehandler-2            4.341µ ±  3%         4.415µ ±  3%       ~ (p=0.289 n=20)
RemoteWriteOOOSamples-2        0.1410n ±  5%        0.1420n ±  9%       ~ (p=0.703 n=20)
geomean                         46.96n               47.31n        +0.74%

                        │   bench.old    │              bench.new.1              │
                        │      B/op      │     B/op      vs base                 │
SampleSend-2              1.037Mi ± 4%     1.106Mi ± 3%  +6.68% (p=0.000 n=20)
Startup-2                   0.000 ± 0%       0.000 ± 0%       ~ (p=1.000 n=20) ¹
StreamReadEndpoint-2      23.95Ki ± 0%     23.95Ki ± 0%       ~ (p=1.000 n=20)
RemoteWritehandler-2      2.498Ki ± 1%     2.502Ki ± 1%       ~ (p=0.440 n=20)
RemoteWriteOOOSamples-2     0.000 ± 0%       0.000 ± 0%       ~ (p=1.000 n=20) ¹
geomean                                ²                 +1.34%                ²
¹ all samples are equal
² summaries must be >0 to compute geomean

                        │   bench.old   │             bench.new.1              │
                        │   allocs/op   │  allocs/op   vs base                 │
SampleSend-2              1.498k ± 3%     1.490k ± 3%       ~ (p=0.941 n=20)
Startup-2                  0.000 ± 0%      0.000 ± 0%       ~ (p=1.000 n=20) ¹
StreamReadEndpoint-2       239.0 ± 0%      239.0 ± 0%       ~ (p=1.000 n=20) ¹
RemoteWritehandler-2       23.00 ± 0%      23.00 ± 0%       ~ (p=1.000 n=20) ¹
RemoteWriteOOOSamples-2    0.000 ± 0%      0.000 ± 0%       ~ (p=1.000 n=20) ¹
geomean                               ²                -0.11%                ²
¹ all samples are equal
² summaries must be >0 to compute geomean

We'll run the benchmarks for a case where some actual filtering is happening to see the kind of difference it makes.

bboreham · 2023-10-30T16:46:42Z

Note CI fail on DCO check.

marctc · 2023-11-17T14:52:53Z

Looks like running the same benchmark but setting sample_age_limit to 60s makes the execution of benchmark slightly faster. This is likely due to the fact that it is discarding samples, resulting in less data to process later on.

goos: linux
goarch: amd64
pkg: github.com/prometheus/prometheus/storage/remote
cpu: 11th Gen Intel(R) Core(TM) i7-1185G7 @ 3.00GHz
             │   without    │              with              │
             │    sec/op    │    sec/op     vs base          │
SampleSend-8   5.218m ± 52%   3.153m ± 14%  ~ (p=0.355 n=20)

             │    without    │                  with                  │
             │     B/op      │     B/op       vs base                 │
SampleSend-8   703.4Ki ± 47%   4323.2Ki ± 0%  +514.64% (p=0.000 n=20)

             │   without    │                  with                  │
             │  allocs/op   │  allocs/op    vs base                  │
SampleSend-8   1.038k ± 46%   49.852k ± 0%  +4705.06% (p=0.000 n=20)

bboreham

Suggest you find out what is causing massively increased memory allocation in your benchmark run.

bboreham · 2023-11-17T15:02:14Z

storage/remote/queue_manager.go

+		// If sampleAgeLimit is unset, then we never skip samples due to their age.
+		return false
+	}
+	limitTs := time.Now().Add(-sampleAgeLimit)


time.Now() is potentially expensive; recommend not calling it multiple times like this.

we have to get the current time every time we run isSampleOld. How do you suggest to improve this?

Pass it in from the caller. Or the caller's caller depending.

yep, could name it baseTime and pass it from isTimeSeriesOldFilter

storage/remote/queue_manager.go

bboreham · 2023-11-17T15:04:20Z

storage/remote/queue_manager.go

 	attemptStore := func(try int) error {
+		if s.qm.cfg.SampleAgeLimit != 0 {
+			// This will filter out old samples during retries.
+			req, _, err := buildWriteRequest(


What is going on here? Why it is it before the start of the span?

The idea here is to filter out batches that might get old during multiple attempts of sending samples. There's no reason of putting before the start of the span (probably missed it). Do we want to send the span even if errors happen?

Why does the comment say "during retries" when it is called every time?
In the typical case, is it repeating the work done at the start of sendSamplesWithBackoff? That doesn't seem appealing.

Indeed. The 'during retries' refers to subsequent calls to attemptStore.

The original idea here is that some of the timeseries might be crossing the 'too old' threshold during retries, so they should be filtered out on these subsequent request by re-building the write request when an age limit has been set.

Given the output of buildWriteRequest is the request itself, I believe it would be too expensive to unmarshall, filter out and marshal back and the only alternative here is that we don't filter on subsequent requests; samples are dropped as too old only when the request is built, leading to more samples slipping through to be rejected by the remote write.

Do you feel that's a preferred route?

Disagree this is "the only alternative".
You have a try parameter which could be used to implement the "during retries" idea.
Or you could pass in an "oldest sample" time which could be checked against the current time to see if anything will be filtered.

+1 for Or you could pass in an "oldest sample" time which could be checked against the current time to see if anything will be filtered., if we know the oldest timestamp in the prev. write request is still not old enough to be filtered we can skip building the write request an additional time.

That's a cool idea! The latest approach stores the lowest timestamp encountered in buildWriteRequest and uses that to decide whether we need to re-build it, let us know what you think. If you believe it's the way to go we can find some way to test it (even if it might not be super straightforward).

storage/remote/queue_manager.go

marctc · 2023-11-20T11:25:08Z

Suggest you find out what is causing massively increased memory allocation in your benchmark run.

I missed completly the step of memory increase, my bad...thanks for checking and PR comments. I belive things
start to look better after the last changes:

goos: linux
goarch: amd64
pkg: github.com/prometheus/prometheus/storage/remote
cpu: 11th Gen Intel(R) Core(TM) i7-1185G7 @ 3.00GHz
             │    base2     │                drop2                │
             │    sec/op    │   sec/op     vs base                │
SampleSend-8   9172.8µ ± 4%   727.5µ ± 1%  -92.07% (p=0.000 n=20)

             │      base2      │                drop2                 │
             │      B/op       │     B/op      vs base                │
SampleSend-8   1244.840Ki ± 5%   6.268Ki ± 4%  -99.50% (p=0.000 n=20)

             │    base2     │               drop2                │
             │  allocs/op   │ allocs/op   vs base                │
SampleSend-8   1871.00 ± 6%   24.00 ± 0%  -98.72% (p=0.000 n=20)

cstyan · 2023-11-20T18:26:58Z

storage/remote/queue_manager.go

 	var highest int64
-	var removeIdx []int

+	writeIdx := 0


can we use a more descriptive name? removeIdx was good before, maybe keepIdx now?

cstyan · 2023-11-20T18:36:20Z

storage/remote/queue_manager.go

+		// If sampleAgeLimit is unset, then we never skip samples due to their age.
+		return false
+	}
+	limitTs := time.Now().Add(-sampleAgeLimit)


yep, could name it baseTime and pass it from isTimeSeriesOldFilter

cstyan · 2023-11-20T18:39:35Z

storage/remote/queue_manager.go

 outer:
 	for _, s := range samples {
+		if isSampleOld(time.Duration(t.cfg.SampleAgeLimit), s.T) {
+			t.metrics.droppedSamplesTotal.Inc()


the metrics here are

Total number of samples which were dropped after being read from the WAL before being sent via remote write, either via relabelling or unintentionally because of an unknown reference ID.

We should probably update the help text to include the too old reason, or use a new metric. I think also one level.Debug line per type and one total per buildWriteRequest call for the total # of each data type dropped could be useful. Debug so we don't flood logs by default, I saw you had before but removed a level.Info line.

We've gone with updating the help text for the original metric.

Furthermore, we've gone with a level.Debug per sample for now. Is the suggestion that we have a counter and log on every call to the QueueManager's Append?

Sort of, I think the log line here is still to noisy to keep even on debug. So instead we would not log on every call to Append, I think the correct thing to do is to add a reason label to the dropped.

We could keep a debug or warn log for the end of buildWriteRequest/buildTimeSeries that logs once with the # of samples/exemplars/etc. dropped due to being too old.

I think the correct thing to do is to add a reason label to the dropped.

That means making the metric a CounterVec, right?

We could keep a debug or warn log for the end of buildWriteRequest/buildTimeSeries that logs once with the # of samples/exemplars/etc. dropped due to being too old.

Ok, on it, will check how we can propagate this information upwards here.

cc @cstyan For the first part, it's implemented in c0f1ef9.

The second part means passing either passing the logger a few layers below and keep track of each dropped signal before it's marshalled into a request, or pollute the signature of buildWriteRequest to return these three extra parameters (that are ignored for metadata) just to log this one line.

IMHO, this is an intrusive change for small gain, so I was thinking that maybe it's not worth it? Would like your feedback on it; I've went ahead and implemented this on <>, but I can revert it if you wish so.

cstyan · 2023-11-20T18:41:41Z

storage/remote/queue_manager.go

 	attemptStore := func(try int) error {
+		if s.qm.cfg.SampleAgeLimit != 0 {
+			// This will filter out old samples during retries.
+			req, _, err := buildWriteRequest(


+1 for Or you could pass in an "oldest sample" time which could be checked against the current time to see if anything will be filtered., if we know the oldest timestamp in the prev. write request is still not old enough to be filtered we can skip building the write request an additional time.

cstyan · 2023-11-20T18:43:15Z

storage/remote/queue_manager_test.go

+	newSamples := make([]record.RefSample, 0, numSamples)
+	samples := make([]record.RefSample, 0, numSamples)
+	series := make([]record.RefSeries, 0, numSeries)
+	b := labels.ScratchBuilder{}


can we name this labelBuilder, or even lb, in a test file if I see just b I assume it's testing.B 😂

cstyan · 2023-11-20T18:50:04Z

Looking good @tpaschalis @marctc, left some comments. Some of which are just reiterating Bryan's existing points. Might also be interesting to double check that perf. isn't worse via the scripts/remotewrite11-bench/run.sh stuff @npazosmendez setup. BenchmarkSampleSend and BenchmarkBuildWriteRequest should be indicative of any performance changes, but the Append... calls are on the hot path for remote write so it would be good to confirm.

Also just a note, please try to avoid rebasing and force pushing during review cycles. Makes it harder to know which commits have come in since the last time we reviewed for you, so comments like I belive things start to look better after the last changes: are bit harder to follow if I don't know which commits are the latest changes.

Co-authored-by: Paschalis Tsilias <[email protected]> Signed-off-by: Paschalis Tsilias <[email protected]> Signed-off-by: Marc Tuduri <[email protected]>

Signed-off-by: Paschalis Tsilias <[email protected]> Signed-off-by: Marc Tuduri <[email protected]>

Signed-off-by: Marc Tuduri <[email protected]>

marctc · 2023-11-21T15:34:28Z

Also just a note, please try to avoid rebasing and force pushing during review cycles. Makes it harder to know which commits have come in since the last time we reviewed for you, so comments like I belive things start to look better after the last changes: are bit harder to follow if I don't know which commits are the latest changes.

I haven't done any rebase but I did push force for the sign-off. How come you lost track of the reviewed commits?

cstyan · 2023-11-21T18:01:27Z

@marctc

I haven't done any rebase but I did push force for the sign-off. How come you lost track of the reviewed commits?

I think it's just a limited functionality of github, I guess it depends how far back you modified the commit messages to add the sign off message. Each commit when you rebase, even interactive to go back and change just commit messages, is reapplied in order which results in different commit SHAs. I think githubs UI only tracks the SHA for which the reviewer last submitted comments, and if the SHAs are still in history it will show a nice changes since your last review option.

For example right now I see this

In this case because it was two separate pushes the UI is able to differentiate luckily.

Signed-off-by: Paschalis Tsilias <[email protected]>

cstyan · 2023-12-15T23:07:22Z

storage/remote/queue_manager.go

 		lbls, ok := t.seriesLabels[e.Ref]
 		if !ok {
-			t.metrics.droppedExemplarsTotal.Inc()
+			t.metrics.droppedExemplarsTotal.WithLabelValues(reasonDroppedSeries).Inc()


now that we have the reason label this line needs to move later (and in all similar boxes) so we can set the reason as reasonDroppedSeries or reasonUnintentionalDroppedSeries, something like that:

if _, ok := t.droppedSeries[e.Ref]; !ok { level.Info(t.logger).Log("msg", "Dropped exemplar for series that was not explicitly dropped via relabelling", "ref", e.Ref) }

You were thinking of something along the lines of 25ec6b6?

Signed-off-by: Paschalis Tsilias <[email protected]>

tpaschalis · 2023-12-20T12:39:04Z

storage/remote/queue_manager.go

+	droppedSamplesTotal    *prometheus.CounterVec
+	droppedExemplarsTotal  *prometheus.CounterVec
+	droppedHistogramsTotal *prometheus.CounterVec


FYI @cstyan the use of a *prometheus.CounterVec here makes for a lot of difference in allocations.

For the following benchmarks, I'm using this commit from this branch.

To showcase, here's the difference of running BenchmarkSampleSend on the current 7dee9d0 PR HEAD, and versus the same HEAD but with a simple prometheus.Counter instead.

$ benchstat head-7dee9d009-no-counter-vec.txt head-7dee9d009-counter-vec.txt ... │ head-7dee9d009-no-counter-vec.txt │ head-7dee9d009-counter-vec.txt │ │ allocs/op │ allocs/op vs base │ SampleSend-8 12.00 ± 8% 10030.00 ± 0% +83483.33% (p=0.000 n=20)

There is already a related fix out on client_golang for this, but will be released on the next v1.18.0 version and it's not currently being used by Prometheus.

When I updated client_golang to include the fix, it dramatically decreased allocations, making it on par with using the simple counter and is going a lot faster when it's discarding all samples.

$ go get -u github.com/prometheus/client_golang@51714a5a30c011624485bd78137378ff0d635383 $ ... $ benchstat head-current-client-golang.txt head-new-client-golang.txt goos: darwin goarch: arm64 pkg: github.com/prometheus/prometheus/storage/remote │ head-current-client-golang.txt │ head-new-client-golang.txt │ │ sec/op │ sec/op vs base │ SampleSend-8 911.1µ ± 1% 735.2µ ± 1% -19.31% (p=0.000 n=20) │ head-current-client-golang.txt │ head-new-client-golang.txt │ │ B/op │ B/op vs base │ SampleSend-8 164.173Ki ± 0% 6.366Ki ± 2% -96.12% (p=0.000 n=20) │ head-current-client-golang.txt │ head-new-client-golang.txt │ │ allocs/op │ allocs/op vs base │ SampleSend-8 10030.00 ± 0% 24.00 ± 4% -99.76% (p=0.000 n=20) $ benchstat main-no-filter.txt head-new-client-golang.txt goos: darwin goarch: arm64 pkg: github.com/prometheus/prometheus/storage/remote │ main-no-filter.txt │ head-new-client-golang.txt │ │ sec/op │ sec/op vs base │ SampleSend-8 1881.6µ ± 11% 735.2µ ± 1% -60.93% (p=0.000 n=20) │ main-no-filter.txt │ head-new-client-golang.txt │ │ B/op │ B/op vs base │ SampleSend-8 250.674Ki ± 8% 6.366Ki ± 2% -97.46% (p=0.000 n=20) │ main-no-filter.txt │ head-new-client-golang.txt │ │ allocs/op │ allocs/op vs base │ SampleSend-8 405.00 ± 6% 24.00 ± 4% -94.07% (p=0.000 n=20)

On the other hand, I'm happy to see that there's

a) No performance regression for people not using the flag
b) No performance regression for people using the flag in normal operation, i.e. when all samples go through
c) A significant throughput gain when discarding unwanted samples

Given that this is only impacting cases where we're throwing away samples (either via relabeling, due to their age or unintentionally), do you see this as a blocker?

@cstyan a new v1.18.0 release of client_golang is out, which fixes the performance overhead. Do you think it's ok to bump as part of this release, or should we make it a different PR altogether?

https://github.com/prometheus/client_golang/releases/tag/v1.18.0

@tpaschalis prometheus should be updating that package soon, there's an automated PR open already for it. Lets just double check benchmark results in your PR after the package update is merged and then we can merge here as well.

@cstyan I could only find this automated PR, but it only bumped it in another subpackage and not the main go.mod file.

I've opened the following to manually bump it for the main go.mod and will re-run the benchmarks. #13373

client_golang is now updated on main, and I've merged main into this branch to run the benchmarks once more.

So after updating client_golang, there's no impact on people not using the feature

$ benchstat main.txt head-nofilter.txt goos: darwin goarch: arm64 pkg: github.com/prometheus/prometheus/storage/remote │ main.txt │ head-nofilter.txt │ │ sec/op │ sec/op vs base │ SampleSend-8 2.021m ± 10% 1.999m ± 3% ~ (p=0.841 n=20) │ main.txt │ head-nofilter.txt │ │ B/op │ B/op vs base │ SampleSend-8 276.4Ki ± 6% 279.6Ki ± 6% ~ (p=0.862 n=20) │ main.txt │ head-nofilter.txt │ │ allocs/op │ allocs/op vs base │ SampleSend-8 438.5 ± 4% 440.5 ± 6% ~ (p=0.507 n=20)

as well as a speedup when samples are thrown away

$ benchstat head-nofilter.txt head-filter.txt goos: darwin goarch: arm64 pkg: github.com/prometheus/prometheus/storage/remote │ head-nofilter.txt │ head-filter.txt │ │ sec/op │ sec/op vs base │ SampleSend-8 1999.5µ ± 3% 723.4µ ± 5% -63.82% (p=0.000 n=20) │ head-nofilter.txt │ head-filter.txt │ │ B/op │ B/op vs base │ SampleSend-8 279.571Ki ± 6% 6.304Ki ± 3% -97.75% (p=0.000 n=20) │ head-nofilter.txt │ head-filter.txt │ │ allocs/op │ allocs/op vs base │ SampleSend-8 440.50 ± 6% 24.00 ± 4% -94.55% (p=0.000 n=20)

Signed-off-by: Paschalis Tsilias <[email protected]>

…op-old-inmemory-samples

Signed-off-by: Paschalis Tsilias <[email protected]>

@Nexucis

* consoles: exclude iowait and steal from CPU Utilisation 'iowait' and 'steal' indicate specific idle/wait states, which shouldn't be counted into CPU Utilisation. Also see prometheus-operator/kube-prometheus#796 and kubernetes-monitoring/kubernetes-mixin#667. Per the iostat man page: %idle Show the percentage of time that the CPU or CPUs were idle and the system did not have an outstanding disk I/O request. %iowait Show the percentage of time that the CPU or CPUs were idle during which the system had an outstanding disk I/O request. %steal Show the percentage of time spent in involuntary wait by the virtual CPU or CPUs while the hypervisor was servicing another virtual processor. Signed-off-by: Julian Wiedmann <[email protected]> * tsdb: shrink txRing with smaller integers 4 billion active transactions ought to be enough for anyone. Signed-off-by: Bryan Boreham <[email protected]> * tsdb: create isolation transaction slice on demand When Prometheus restarts it creates every series read in from the WAL, but many of those series will be finished, and never receive any more samples. By defering allocation of the txRing slice to when it is first needed, we save 32 bytes per stale series. Signed-off-by: Bryan Boreham <[email protected]> * add cluster variable to Overview dashboard Signed-off-by: Erik Sommer <[email protected]> * promql: simplify Native Histogram arithmetics Signed-off-by: Linas Medziunas <[email protected]> * Cut 2.49.0-rc.0 (#13270) * Cut 2.49.0-rc.0 Signed-off-by: bwplotka <[email protected]> * Removed the duplicate. Signed-off-by: bwplotka <[email protected]> --------- Signed-off-by: bwplotka <[email protected]> * Add unit protobuf parser Signed-off-by: Arianna Vespri <[email protected]> * Go on adding protobuf parsing for unit Signed-off-by: Arianna Vespri <[email protected]> * ui: create a reproduction for #13292 Signed-off-by: machine424 <[email protected]> * Get conditional right Signed-off-by: Arianna Vespri <[email protected]> * Get VM Scale Set NIC (#13283) Calling `*armnetwork.InterfacesClient.Get()` doesn't work for Scale Set VM NIC, because these use a different Resource ID format. Use `*armnetwork.InterfacesClient.GetVirtualMachineScaleSetNetworkInterface()` instead. This needs both the scale set name and the instance ID, so add an `InstanceID` field to the `virtualMachine` struct. `InstanceID` is empty for a VM that isn't a ScaleSetVM. Signed-off-by: Daniel Nicholls <[email protected]> * Cut v2.49.0-rc.1 Signed-off-by: bwplotka <[email protected]> * Delete debugging lines, amend error message for unit Signed-off-by: Arianna Vespri <[email protected]> * Correct order in error message Signed-off-by: Arianna Vespri <[email protected]> * Consider storage.ErrTooOldSample as non-retryable Signed-off-by: Daniel Kerbel <[email protected]> * scrape_test.go: Increase scrape interval in TestScrapeLoopCache to reduce potential flakiness Signed-off-by: machine424 <[email protected]> * Avoid creating string for suffix, consider counters without _total suffix Signed-off-by: Arianna Vespri <[email protected]> * build(deps): bump github.com/prometheus/client_golang Bumps [github.com/prometheus/client_golang](https://github.com/prometheus/client_golang) from 1.17.0 to 1.18.0. - [Release notes](https://github.com/prometheus/client_golang/releases) - [Changelog](https://github.com/prometheus/client_golang/blob/main/CHANGELOG.md) - [Commits](prometheus/client_golang@v1.17.0...v1.18.0) --- updated-dependencies: - dependency-name: github.com/prometheus/client_golang dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> * build(deps): bump actions/setup-node from 3.8.1 to 4.0.1 Bumps [actions/setup-node](https://github.com/actions/setup-node) from 3.8.1 to 4.0.1. - [Release notes](https://github.com/actions/setup-node/releases) - [Commits](actions/setup-node@5e21ff4...b39b52d) --- updated-dependencies: - dependency-name: actions/setup-node dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> * scripts: sort file list in embed directive Otherwise the resulting string depends on find, which afaict depends on the underlying filesystem. A stable file list make it easier to detect UI changes in downstreams that need to track UI assets. Signed-off-by: Jan Fajerski <[email protected]> * Fix DataTableProps['data'] for resultType string Signed-off-by: Kevin Mingtarja <[email protected]> * Fix handling of scalar and string in isHeatmapData Signed-off-by: Kevin Mingtarja <[email protected]> * build(deps): bump github.com/influxdata/influxdb Bumps [github.com/influxdata/influxdb](https://github.com/influxdata/influxdb) from 1.11.2 to 1.11.4. - [Release notes](https://github.com/influxdata/influxdb/releases) - [Commits](influxdata/influxdb@v1.11.2...v1.11.4) --- updated-dependencies: - dependency-name: github.com/influxdata/influxdb dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> * build(deps): bump github.com/prometheus/prometheus Bumps [github.com/prometheus/prometheus](https://github.com/prometheus/prometheus) from 0.48.0 to 0.48.1. - [Release notes](https://github.com/prometheus/prometheus/releases) - [Changelog](https://github.com/prometheus/prometheus/blob/main/CHANGELOG.md) - [Commits](v0.48.0...v0.48.1) --- updated-dependencies: - dependency-name: github.com/prometheus/prometheus dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> * Bump client_golang to v1.18.0 (#13373) Signed-off-by: Paschalis Tsilias <[email protected]> * Drop old inmemory samples (#13002) * Drop old inmemory samples Co-authored-by: Paschalis Tsilias <[email protected]> Signed-off-by: Paschalis Tsilias <[email protected]> Signed-off-by: Marc Tuduri <[email protected]> * Avoid copying timeseries when the feature is disabled Signed-off-by: Paschalis Tsilias <[email protected]> Signed-off-by: Marc Tuduri <[email protected]> * Run gofmt Signed-off-by: Paschalis Tsilias <[email protected]> Signed-off-by: Marc Tuduri <[email protected]> * Clarify docs Signed-off-by: Marc Tuduri <[email protected]> * Add more logging info Signed-off-by: Marc Tuduri <[email protected]> * Remove loggers Signed-off-by: Marc Tuduri <[email protected]> * optimize function and add tests Signed-off-by: Marc Tuduri <[email protected]> * Simplify filter Signed-off-by: Marc Tuduri <[email protected]> * rename var Signed-off-by: Marc Tuduri <[email protected]> * Update help info from metrics Signed-off-by: Marc Tuduri <[email protected]> * use metrics to keep track of drop elements during buildWriteRequest Signed-off-by: Marc Tuduri <[email protected]> * rename var in tests Signed-off-by: Marc Tuduri <[email protected]> * pass time.Now as parameter Signed-off-by: Marc Tuduri <[email protected]> * Change buildwriterequest during retries Signed-off-by: Marc Tuduri <[email protected]> * Revert "Remove loggers" This reverts commit 54f91df. Signed-off-by: Marc Tuduri <[email protected]> * use log level debug for loggers Signed-off-by: Marc Tuduri <[email protected]> * Fix linter Signed-off-by: Paschalis Tsilias <[email protected]> * Remove noisy debug-level logs; add 'reason' label to drop metrics Signed-off-by: Paschalis Tsilias <[email protected]> * Remove accidentally committed files Signed-off-by: Paschalis Tsilias <[email protected]> * Propagate logger to buildWriteRequest to log dropped data Signed-off-by: Paschalis Tsilias <[email protected]> * Fix docs comment Signed-off-by: Paschalis Tsilias <[email protected]> * Make drop reason more specific Signed-off-by: Paschalis Tsilias <[email protected]> * Remove unnecessary pass of logger Signed-off-by: Paschalis Tsilias <[email protected]> * Use snake_case for reason label Signed-off-by: Paschalis Tsilias <[email protected]> * Fix dropped samples metric Signed-off-by: Paschalis Tsilias <[email protected]> --------- Signed-off-by: Paschalis Tsilias <[email protected]> Signed-off-by: Marc Tuduri <[email protected]> Signed-off-by: Paschalis Tsilias <[email protected]> Co-authored-by: Paschalis Tsilias <[email protected]> Co-authored-by: Paschalis Tsilias <[email protected]> * fix(discovery): allow requireUpdate util to timeout in discovery/file/file_test.go. The loop ran indefinitely if the condition isn't met. Before, each iteration created a new timer channel which was always outpaced by the other timer channel with smaller duration. minor detail: There was a memory leak: resources of the ~10 previous timers were constantly kept. With the fix, we may keep the resources of one timer around for defaultWait but this isn't worth the changes to make it right. Signed-off-by: machine424 <[email protected]> * Merge pull request #13371 from kevinmingtarja/fix-isHeatmapData ui: fix handling of scalar and string in isHeatmapData * tsdb/{index,compact}: allow using custom postings encoding format (#13242) * tsdb/{index,compact}: allow using custom postings encoding format We would like to experiment with a different postings encoding format in Thanos so in this change I am proposing adding another argument to `NewWriter` which would allow users to change the format if needed. Also, wire the leveled compactor so that it would be possible to change the format there too. Signed-off-by: Giedrius Statkevičius <[email protected]> * tsdb/compact: use a struct for leveled compactor options As discussed on Slack, let's use a struct for the options in leveled compactor. Signed-off-by: Giedrius Statkevičius <[email protected]> * tsdb: make changes after Bryan's review - Make changes less intrusive - Turn the postings encoder type into a function - Add NewWriterWithEncoder() Signed-off-by: Giedrius Statkevičius <[email protected]> --------- Signed-off-by: Giedrius Statkevičius <[email protected]> * Cut 2.49.0-rc.2 Signed-off-by: bwplotka <[email protected]> * build(deps): bump actions/setup-go from 3.5.0 to 5.0.0 in /scripts (#13362) Bumps [actions/setup-go](https://github.com/actions/setup-go) from 3.5.0 to 5.0.0. - [Release notes](https://github.com/actions/setup-go/releases) - [Commits](actions/setup-go@6edd440...0c52d54) --- updated-dependencies: - dependency-name: actions/setup-go dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * build(deps): bump github/codeql-action from 2.22.8 to 3.22.12 (#13358) Bumps [github/codeql-action](https://github.com/github/codeql-action) from 2.22.8 to 3.22.12. - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](github/codeql-action@407ffaf...012739e) --- updated-dependencies: - dependency-name: github/codeql-action dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * put @Nexucis has a release shepherd (#13383) Signed-off-by: Augustin Husson <[email protected]> * Add analyze histograms command to promtool (#12331) Add `query analyze` command to promtool This command analyzes the buckets of classic and native histograms, based on data queried from the Prometheus query API, i.e. it doesn't require direct access to the TSDB files. Signed-off-by: Jeanette Tan <[email protected]> --------- Signed-off-by: Jeanette Tan <[email protected]> * included instance in all necessary descriptions Signed-off-by: Erik Sommer <[email protected]> * tsdb/compact: fix passing merge func Fixing a very small logical problem I've introduced :(. Signed-off-by: Giedrius Statkevičius <[email protected]> * tsdb: add enable overlapping compaction This functionality is needed in downstream projects because they have a separate component that does compaction. Upstreaming https://github.com/grafana/mimir-prometheus/blob/7c8e9a2a76fc729e9078889782928b2fdfe240e9/tsdb/compact.go#L323-L325. Signed-off-by: Giedrius Statkevičius <[email protected]> * Cut 2.49.0 Signed-off-by: bwplotka <[email protected]> * promtool: allow setting multiple matchers to "promtool tsdb dump" command. (#13296) Conditions are ANDed inside the same matcher but matchers are ORed Including unit tests for "promtool tsdb dump". Refactor some matchers scraping utils. Signed-off-by: machine424 <[email protected]> * Fixed changelog Signed-off-by: bwplotka <[email protected]> * tsdb/main: wire "EnableOverlappingCompaction" to tsdb.Options (#13398) This added the #13393 "EnableOverlappingCompaction" parameter to the compactor code but not to the tsdb.Options. I forgot about that. Add it to `tsdb.Options` too and set it to `true` in Prometheus. Copy/paste the description from #13393 (comment) Signed-off-by: Giedrius Statkevičius <[email protected]> * Issue #13268: fix quality value in accept header Signed-off-by: Kumar Kalpadiptya Roy <[email protected]> * Cut 2.49.1 with scrape q= bugfix. Signed-off-by: bwplotka <[email protected]> * Cut 2.49.1 web package. Signed-off-by: bwplotka <[email protected]> * Restore more efficient version of NewPossibleNonCounterInfo annotation (#13022) Restore more efficient version of NewPossibleNonCounterInfo annotation Signed-off-by: Jeanette Tan <[email protected]> --------- Signed-off-by: Jeanette Tan <[email protected]> * Fix regressions introduced by #13242 Signed-off-by: Marco Pracucci <[email protected]> * fix slice copy in 1.20 (#13389) The slices package is added to the standard library in Go 1.21; we need to import from the exp area to maintain compatibility with Go 1.20. Signed-off-by: tyltr <[email protected]> * Docs: Query Basics: link to rate (#10538) Co-authored-by: Julien Pivotto <[email protected]> * chore(kubernetes): check preconditions earlier and avoid unnecessary checks or iterations Signed-off-by: machine424 <[email protected]> * Examples: link to `rate` for new users (#10535) * Examples: link to `rate` for new users Signed-off-by: Ted Robertson [email protected] Co-authored-by: Bryan Boreham <[email protected]> * promql: use natural sort in sort_by_label and sort_by_label_desc (#13411) These functions are intended for humans, as robots can already sort the results however they please. Humans like things sorted "naturally": * https://blog.codinghorror.com/sorting-for-humans-natural-sort-order/ A similar thing has been done to Grafana, which is also used by humans: * grafana/grafana#78024 * grafana/grafana#78494 Signed-off-by: Ivan Babrou <[email protected]> * TestLabelValuesWithMatchers: Add test case Signed-off-by: Arve Knudsen <[email protected]> * remove obsolete build tag Signed-off-by: tyltr <[email protected]> * Upgrade some golang dependencies for resty 2.11 Signed-off-by: Israel Blancas <[email protected]> * Native Histograms: support `native_histogram_min_bucket_factor` in scrape_config (#13222) Native Histograms: support native_histogram_min_bucket_factor in scrape_config --------- Signed-off-by: Ziqi Zhao <[email protected]> Signed-off-by: Björn Rabenstein <[email protected]> Co-authored-by: George Krajcsovits <[email protected]> Co-authored-by: Björn Rabenstein <[email protected]> * Add warnings for histogramRate applied with isCounter not matching counter/gauge histogram (#13392) Add warnings for histogramRate applied with isCounter not matching counter/gauge histogram --------- Signed-off-by: Jeanette Tan <[email protected]> * Minor fixes to otlp vendor update script Signed-off-by: Goutham <[email protected]> * build(deps): bump github.com/hetznercloud/hcloud-go/v2 Bumps [github.com/hetznercloud/hcloud-go/v2](https://github.com/hetznercloud/hcloud-go) from 2.4.0 to 2.6.0. - [Release notes](https://github.com/hetznercloud/hcloud-go/releases) - [Changelog](https://github.com/hetznercloud/hcloud-go/blob/main/CHANGELOG.md) - [Commits](hetznercloud/hcloud-go@v2.4.0...v2.6.0) --- updated-dependencies: - dependency-name: github.com/hetznercloud/hcloud-go/v2 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> * Enhanced visibility for `promtool test rules` with JSON colored formatting (#13342) * Added diff flag for unit test to improvise readability & debugging Signed-off-by: Rewanth Tammana <[email protected]> * Removed blank spaces Signed-off-by: Rewanth Tammana <[email protected]> * Fixed linting error Signed-off-by: Rewanth Tammana <[email protected]> * Added cli flags to documentation Signed-off-by: Rewanth Tammana <[email protected]> * Revert unrrelated linting fixes Signed-off-by: Rewanth Tammana <[email protected]> * Fixed review suggestions Signed-off-by: Rewanth Tammana <[email protected]> * Cleanup Signed-off-by: Rewanth Tammana <[email protected]> * Updated flag description Signed-off-by: Rewanth Tammana <[email protected]> * Updated flag description Signed-off-by: Rewanth Tammana <[email protected]> --------- Signed-off-by: Rewanth Tammana <[email protected]> * storage: skip merging when no remote storage configured Prometheus is hard-coded to use a fanout storage between TSDB and a remote storage which by default is empty. This change detects the empty storage and skips merging between result sets, which would make `Select()` sort results. Bottom line: we skip a sort unless there really is some remote storage configured. Signed-off-by: Bryan Boreham <[email protected]> * Remove csmarchbanks from remote write owners (#13432) I have not had the time to keep up with remote write and have no plans to work on it in the near future so I am withdrawing my maintainership of that part of the codebase. I continue to focus on client_python. Signed-off-by: Chris Marchbanks <[email protected]> * add more context cancellation check at evaluation time Signed-off-by: Ben Ye <[email protected]> * Optimize label values with matchers by taking shortcuts (#13426) Don't calculate postings beforehand: we may not need them. If all matchers are for the requested label, we can just filter its values. Also, if there are no values at all, no need to run any kind of logic. Also add more labelValuesWithMatchers benchmarks Signed-off-by: Oleg Zaytsev <[email protected]> * Add automatic memory limit handling Enable automatic detection of memory limits and configure GOMEMLIMIT to match. * Also includes a flag to allow controlling the reserved ratio. Signed-off-by: SuperQ <[email protected]> * Update OSSF badge link (#13433) Provide a more user friendly interface Signed-off-by: Matthieu MOREL <[email protected]> * SD Managers taking over responsibility for registration of debug metrics (#13375) SD Managers take over responsibility for SD metrics registration --------- Signed-off-by: Paulin Todev <[email protected]> Signed-off-by: Björn Rabenstein <[email protected]> Co-authored-by: Björn Rabenstein <[email protected]> * Optimize histogram iterators (#13340) Optimize histogram iterators Histogram iterators allocate new objects in the AtHistogram and AtFloatHistogram methods, which makes calculating rates over long ranges expensive. In #13215 we allowed an existing object to be reused when converting an integer histogram to a float histogram. This commit follows the same idea and allows injecting an existing object in the AtHistogram and AtFloatHistogram methods. When the injected value is nil, iterators allocate new histograms, otherwise they populate and return the injected object. The commit also adds a CopyTo method to Histogram and FloatHistogram which is used in the BufferedIterator to overwrite items in the ring instead of making new copies. Note that a specialized HPoint pool is needed for all of this to work (`matrixSelectorHPool`). --------- Signed-off-by: Filip Petkovski <[email protected]> Co-authored-by: George Krajcsovits <[email protected]> * doc: Mark `mad_over_time` as experimental (#13440) We forgot to do that in #13059 Signed-off-by: beorn7 <[email protected]> * Change metric label for Puppetdb from 'http' to 'puppetdb' Signed-off-by: Paulin Todev <[email protected]> * mirror metrics.proto change & generate code Signed-off-by: Ziqi Zhao <[email protected]> * TestHeadLabelValuesWithMatchers: Add test case (#13414) Add test case to TestHeadLabelValuesWithMatchers, while fixing a couple of typos in other test cases. Also enclosing some implicit sub-tests in a `t.Run` call to make them explicitly sub-tests. Signed-off-by: Arve Knudsen <[email protected]> * update all go dependencies (#13438) Signed-off-by: Augustin Husson <[email protected]> * build(deps): bump the k8s-io group with 2 updates (#13454) Bumps the k8s-io group with 2 updates: [k8s.io/api](https://github.com/kubernetes/api) and [k8s.io/client-go](https://github.com/kubernetes/client-go). Updates `k8s.io/api` from 0.28.4 to 0.29.1 - [Commits](kubernetes/api@v0.28.4...v0.29.1) Updates `k8s.io/client-go` from 0.28.4 to 0.29.1 - [Changelog](https://github.com/kubernetes/client-go/blob/master/CHANGELOG.md) - [Commits](kubernetes/client-go@v0.28.4...v0.29.1) --- updated-dependencies: - dependency-name: k8s.io/api dependency-type: direct:production update-type: version-update:semver-minor dependency-group: k8s-io - dependency-name: k8s.io/client-go dependency-type: direct:production update-type: version-update:semver-minor dependency-group: k8s-io ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * build(deps): bump the go-opentelemetry-io group with 1 update (#13453) Bumps the go-opentelemetry-io group with 1 update: [go.opentelemetry.io/collector/semconv](https://github.com/open-telemetry/opentelemetry-collector). Updates `go.opentelemetry.io/collector/semconv` from 0.92.0 to 0.93.0 - [Release notes](https://github.com/open-telemetry/opentelemetry-collector/releases) - [Changelog](https://github.com/open-telemetry/opentelemetry-collector/blob/main/CHANGELOG-API.md) - [Commits](open-telemetry/opentelemetry-collector@v0.92.0...v0.93.0) --- updated-dependencies: - dependency-name: go.opentelemetry.io/collector/semconv dependency-type: direct:production update-type: version-update:semver-minor dependency-group: go-opentelemetry-io ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * build(deps): bump actions/upload-artifact from 3.1.3 to 4.0.0 (#13355) Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact) from 3.1.3 to 4.0.0. - [Release notes](https://github.com/actions/upload-artifact/releases) - [Commits](actions/upload-artifact@a8a3f3a...c7d193f) --- updated-dependencies: - dependency-name: actions/upload-artifact dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * build(deps): bump bufbuild/buf-push-action (#13357) Bumps [bufbuild/buf-push-action](https://github.com/bufbuild/buf-push-action) from 342fc4cdcf29115a01cf12a2c6dd6aac68dc51e1 to a654ff18effe4641ebea4a4ce242c49800728459. - [Release notes](https://github.com/bufbuild/buf-push-action/releases) - [Commits](bufbuild/buf-push-action@342fc4c...a654ff1) --- updated-dependencies: - dependency-name: bufbuild/buf-push-action dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Labels: Add DropMetricName function, used in PromQL (#13446) This function is called very frequently when executing PromQL functions, and we can do it much more efficiently inside Labels. In the common case that `__name__` comes first in the labels, we simply re-point to start at the next label, which is nearly free. `DropMetricName` is now so cheap I removed the cache - benchmarks show everything still goes faster. Signed-off-by: Bryan Boreham <[email protected]> * tsdb: simplify internal series delete function (#13261) Lifting an optimisation from Agent code, `seriesHashmap.del` can use the unique series reference, doesn't need to check Labels. Also streamline the logic for deleting from `unique` and `conflicts` maps, and add some comments to help the next person. Signed-off-by: Bryan Boreham <[email protected]> * otlptranslator/update-copy.sh: Fix sed command lines Signed-off-by: Arve Knudsen <[email protected]> * Rollback k8s.io requirements (#13462) Rollback k8s.io Go modules to v0.28.6 to avoid forcing upgrade of Go to 1.21. This allows us to keep compatibility with the currently supported upstream Go releases. Signed-off-by: SuperQ <[email protected]> * Make update-copy.sh work for both OSX and GNU sed Signed-off-by: Arve Knudsen <[email protected]> * Name @beorn7 and @krajorama as maintainers for native histograms I have been the de-facto maintainer for native histograms from the beginning. So let's put this into MAINTAINERS.md. In addition, I hereby proposose George Krajcsovits AKA Krajo as a co-maintainer. He has contributed a lot of native histogram code, but more importantly, he has contributed substantially to reviewing other contributors' native histogram code, up to a point where I was merely rubberstamping the PRs he had already reviewed. I'm confident that he is ready to to be granted commit rights as outlined in the "Maintainers" section of the governance: https://prometheus.io/governance/#maintainers According to the same section of the governance, I will announce the proposed change on the developers mailing list and will give some time for lazy consensus before merging this PR. Signed-off-by: beorn7 <[email protected]> * ui/fix: correct url handling for stacked graphs (#13460) Signed-off-by: Yury Moladau <[email protected]> * tsdb: use cheaper Mutex on series Mutex is 8 bytes; RWMutex is 24 bytes and much more complicated. Since `RLock` is only used in two places, `UpdateMetadata` and `Delete`, neither of which are hotspots, we should use the cheaper one. Signed-off-by: Bryan Boreham <[email protected]> * Fix last_over_time for native histograms The last_over_time retains a histogram sample without making a copy. This sample is now coming from the buffered iterator used for windowing functions, and can be reused for reading subsequent samples as the iterator progresses. I would propose copying the sample in the last_over_time function, similar to how it is done for rate, sum_over_time and others. Signed-off-by: Filip Petkovski <[email protected]> * Implementation NOTE: Rebased from main after refactor in #13014 Signed-off-by: Danny Kopping <[email protected]> * Add feature flag Signed-off-by: Danny Kopping <[email protected]> * Refactor concurrency control Signed-off-by: Danny Kopping <[email protected]> * Optimising dependencies/dependents funcs to not produce new slices each request Signed-off-by: Danny Kopping <[email protected]> * Refactoring Signed-off-by: Danny Kopping <[email protected]> * Rename flag Signed-off-by: Danny Kopping <[email protected]> * Refactoring for performance, and to allow controller to be overridden Signed-off-by: Danny Kopping <[email protected]> * Block until all rules, both sync & async, have completed evaluating Updated & added tests Review feedback nits Return empty map if not indeterminate Use highWatermark to track inflight requests counter Appease the linter Clarify feature flag Signed-off-by: Danny Kopping <[email protected]> * Fix typo in CLI flag description Signed-off-by: Marco Pracucci <[email protected]> * Fixed auto-generated doc Signed-off-by: Marco Pracucci <[email protected]> * Improve doc Signed-off-by: Marco Pracucci <[email protected]> * Simplify the design to update concurrency controller once the rule evaluation has done Signed-off-by: Marco Pracucci <[email protected]> * Add more test cases to TestDependenciesEdgeCases Signed-off-by: Marco Pracucci <[email protected]> * Added more test cases to TestDependenciesEdgeCases Signed-off-by: Marco Pracucci <[email protected]> * Improved RuleConcurrencyController interface doc Signed-off-by: Marco Pracucci <[email protected]> * Introduced sequentialRuleEvalController Signed-off-by: Marco Pracucci <[email protected]> * Remove superfluous nil check in Group.metrics Signed-off-by: Marco Pracucci <[email protected]> * api: Serialize discovered and target labels into JSON directly (#13469) Converted maps into labels.Labels to avoid a lot of copying of data which leads to very high memory consumption while opening the /service-discovery endpoint in the Prometheus UI Signed-off-by: Leegin <[email protected]> * api: Serialize discovered labels into JSON directly in dropped targets (#13484) Converted maps into labels.Labels to avoid a lot of copying of data which leads to very high memory consumption while opening the /service-discovery endpoint in the Prometheus UI Signed-off-by: Leegin <[email protected]> * Add ShardedPostings() support to TSDB (#10421) This PR is a reference implementation of the proposal described in #10420. In addition to what described in #10420, in this PR I've introduced labels.StableHash(). The idea is to offer an hashing function which doesn't change over time, and that's used by query sharding in order to get a stable behaviour over time. The implementation of labels.StableHash() is the hashing function used by Prometheus before stringlabels, and what's used by Grafana Mimir for query sharding (because built before stringlabels was a thing). Follow up work As mentioned in #10420, if this PR is accepted I'm also open to upload another foundamental piece used by Grafana Mimir query sharding to accelerate the query execution: an optional, configurable and fast in-memory cache for the series hashes. Signed-off-by: Marco Pracucci <[email protected]> * storage/remote: document why two benchmarks are skipped One was silently doing nothing; one was doing something but the work didn't go up linearly with iteration count. Signed-off-by: Bryan Boreham <[email protected]> * Pod status changes not discovered by Kube Endpoints SD (#13337) * fix(discovery/kubernetes/endpoints): react to changes on Pods because some modifications can occur on them without triggering an update on the related Endpoints (The Pod phase changing from Pending to Running e.g.). --------- Signed-off-by: machine424 <[email protected]> Co-authored-by: Guillermo Sanchez Gavier <[email protected]> * Small improvements, add const, remove copypasta (#8106) Signed-off-by: Mikhail Fesenko <[email protected]> Signed-off-by: Jesus Vazquez <[email protected]> * Proposal to improve FPointSlice and HPointSlice allocation. (#13448) * Reusing points slice from previous series when the slice is under utilized * Adding comments on the bench test Signed-off-by: Alan Protasio <[email protected]> * lint Signed-off-by: Nicolás Pazos <[email protected]> * go mod tidy Signed-off-by: Nicolás Pazos <[email protected]> --------- Signed-off-by: Julian Wiedmann <[email protected]> Signed-off-by: Bryan Boreham <[email protected]> Signed-off-by: Erik Sommer <[email protected]> Signed-off-by: Linas Medziunas <[email protected]> Signed-off-by: bwplotka <[email protected]> Signed-off-by: Arianna Vespri <[email protected]> Signed-off-by: machine424 <[email protected]> Signed-off-by: Daniel Nicholls <[email protected]> Signed-off-by: Daniel Kerbel <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: Jan Fajerski <[email protected]> Signed-off-by: Kevin Mingtarja <[email protected]> Signed-off-by: Paschalis Tsilias <[email protected]> Signed-off-by: Marc Tuduri <[email protected]> Signed-off-by: Paschalis Tsilias <[email protected]> Signed-off-by: Giedrius Statkevičius <[email protected]> Signed-off-by: Augustin Husson <[email protected]> Signed-off-by: Jeanette Tan <[email protected]> Signed-off-by: Bartlomiej Plotka <[email protected]> Signed-off-by: Kumar Kalpadiptya Roy <[email protected]> Signed-off-by: Marco Pracucci <[email protected]> Signed-off-by: tyltr <[email protected]> Signed-off-by: Ted Robertson [email protected] Signed-off-by: Ivan Babrou <[email protected]> Signed-off-by: Arve Knudsen <[email protected]> Signed-off-by: Israel Blancas <[email protected]> Signed-off-by: Ziqi Zhao <[email protected]> Signed-off-by: Björn Rabenstein <[email protected]> Signed-off-by: Goutham <[email protected]> Signed-off-by: Rewanth Tammana <[email protected]> Signed-off-by: Chris Marchbanks <[email protected]> Signed-off-by: Ben Ye <[email protected]> Signed-off-by: Oleg Zaytsev <[email protected]> Signed-off-by: SuperQ <[email protected]> Signed-off-by: Ben Kochie <[email protected]> Signed-off-by: Matthieu MOREL <[email protected]> Signed-off-by: Paulin Todev <[email protected]> Signed-off-by: Filip Petkovski <[email protected]> Signed-off-by: beorn7 <[email protected]> Signed-off-by: Augustin Husson <[email protected]> Signed-off-by: Yury Moladau <[email protected]> Signed-off-by: Danny Kopping <[email protected]> Signed-off-by: Leegin <[email protected]> Signed-off-by: Mikhail Fesenko <[email protected]> Signed-off-by: Jesus Vazquez <[email protected]> Signed-off-by: Alan Protasio <[email protected]> Signed-off-by: Nicolás Pazos <[email protected]> Co-authored-by: Julian Wiedmann <[email protected]> Co-authored-by: Bryan Boreham <[email protected]> Co-authored-by: Erik Sommer <[email protected]> Co-authored-by: Linas Medziunas <[email protected]> Co-authored-by: Bartlomiej Plotka <[email protected]> Co-authored-by: Arianna Vespri <[email protected]> Co-authored-by: machine424 <[email protected]> Co-authored-by: daniel-resdiary <[email protected]> Co-authored-by: Daniel Kerbel <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Jan Fajerski <[email protected]> Co-authored-by: Kevin Mingtarja <[email protected]> Co-authored-by: Paschalis Tsilias <[email protected]> Co-authored-by: Marc Tudurí <[email protected]> Co-authored-by: Paschalis Tsilias <[email protected]> Co-authored-by: Giedrius Statkevičius <[email protected]> Co-authored-by: Augustin Husson <[email protected]> Co-authored-by: Björn Rabenstein <[email protected]> Co-authored-by: zenador <[email protected]> Co-authored-by: gotjosh <[email protected]> Co-authored-by: Ben Kochie <[email protected]> Co-authored-by: Kumar Kalpadiptya Roy <[email protected]> Co-authored-by: Marco Pracucci <[email protected]> Co-authored-by: tyltr <[email protected]> Co-authored-by: Ted Robertson <[email protected]> Co-authored-by: Julien Pivotto <[email protected]> Co-authored-by: Matthias Loibl <[email protected]> Co-authored-by: Ivan Babrou <[email protected]> Co-authored-by: Arve Knudsen <[email protected]> Co-authored-by: Israel Blancas <[email protected]> Co-authored-by: Ziqi Zhao <[email protected]> Co-authored-by: George Krajcsovits <[email protected]> Co-authored-by: Björn Rabenstein <[email protected]> Co-authored-by: Goutham <[email protected]> Co-authored-by: Rewanth Tammana <[email protected]> Co-authored-by: Chris Marchbanks <[email protected]> Co-authored-by: Ben Ye <[email protected]> Co-authored-by: Oleg Zaytsev <[email protected]> Co-authored-by: Matthieu MOREL <[email protected]> Co-authored-by: Paulin Todev <[email protected]> Co-authored-by: Filip Petkovski <[email protected]> Co-authored-by: Yury Molodov <[email protected]> Co-authored-by: Danny Kopping <[email protected]> Co-authored-by: Leegin <[email protected]> Co-authored-by: Guillermo Sanchez Gavier <[email protected]> Co-authored-by: Mikhail Fesenko <[email protected]> Co-authored-by: Alan Protasio <[email protected]>

…support + metadata in WAL) (#14395) * Remote Write 1.1: e2e benchmarks (#13102) * Remote Write e2e benchmarks Signed-off-by: Nicolás Pazos <[email protected]> * Prometheus ports automatically assigned Signed-off-by: Nicolás Pazos <[email protected]> * make dashboard editable + more modular to different job label values Signed-off-by: Callum Styan <[email protected]> * Dashboard improvements * memory stats * diffs look at counter increases Signed-off-by: Nicolás Pazos <[email protected]> * run script: absolute path for config templates Signed-off-by: Nicolás Pazos <[email protected]> * grafana dashboard improvements * show actual values of metrics * add memory stats and diff Signed-off-by: Nicolás Pazos <[email protected]> * dashboard changes Signed-off-by: Callum Styan <[email protected]> --------- Signed-off-by: Nicolás Pazos <[email protected]> Signed-off-by: Callum Styan <[email protected]> Co-authored-by: Callum Styan <[email protected]> * replace snappy encoding library Signed-off-by: Callum Styan <[email protected]> Signed-off-by: Nicolás Pazos <[email protected]> * add new proto types Signed-off-by: Callum Styan <[email protected]> Signed-off-by: Nicolás Pazos <[email protected]> * add decode function for new write request proto Signed-off-by: Callum Styan <[email protected]> Signed-off-by: Nicolás Pazos <[email protected]> * add lookup table struct that is used to build the symbol table in new write request format Signed-off-by: Callum Styan <[email protected]> Signed-off-by: Nicolás Pazos <[email protected]> * Implement code paths for new proto format Signed-off-by: Callum Styan <[email protected]> Signed-off-by: Nicolás Pazos <[email protected]> * update example server to include handler for new format Signed-off-by: Callum Styan <[email protected]> Signed-off-by: Nicolás Pazos <[email protected]> * Add new test client Signed-off-by: Callum Styan <[email protected]> Signed-off-by: Nicolás Pazos <[email protected]> * tests and new -> original proto mapping util Signed-off-by: Nicolás Pazos <[email protected]> * add new proto support on receiver end Signed-off-by: Nicolás Pazos <[email protected]> * Fix test Signed-off-by: Nicolás Pazos <[email protected]> * no-brainer copypaste but more performance write support Signed-off-by: Nicolás Pazos <[email protected]> * remove some comented code Signed-off-by: Nicolás Pazos <[email protected]> * fix mocks and fixture Signed-off-by: Nicolás Pazos <[email protected]> * add basic reduce remote write handler benchmark Signed-off-by: Nicolás Pazos <[email protected]> * refactor out common code between write methods Signed-off-by: Nicolás Pazos <[email protected]> * fix: queue manager to include float histograms in new requests Signed-off-by: Nicolás Pazos <[email protected]> * add sender-side tests and fix failing ones Signed-off-by: Nicolás Pazos <[email protected]> * refactor queue manager code to remove some duplication Signed-off-by: Nicolás Pazos <[email protected]> * fix build Signed-off-by: Nicolás Pazos <[email protected]> * Improve sender benchmarks and some allocations Signed-off-by: Nicolás Pazos <[email protected]> * Use github.com/golang/snappy Signed-off-by: Nicolás Pazos <[email protected]> * cleanup: remove hardcoded fake url for testing Signed-off-by: Nicolás Pazos <[email protected]> * Add 1.1 version handling code Signed-off-by: Nicolás Pazos <[email protected]> * Remove config, update proto Signed-off-by: Nicolás Pazos <[email protected]> * gofmt Signed-off-by: Nicolás Pazos <[email protected]> * fix NewWriteClient and change new flags wording Signed-off-by: Nicolás Pazos <[email protected]> * fields rewording in handler Signed-off-by: Nicolás Pazos <[email protected]> * remote write handler to checks version header Signed-off-by: Nicolás Pazos <[email protected]> * fix typo in log Signed-off-by: Nicolás Pazos <[email protected]> * lint Signed-off-by: Nicolás Pazos <[email protected]> * Add minmized remote write proto format Co-authored-by: Marco Pracucci <[email protected]> Signed-off-by: Callum Styan <[email protected]> Signed-off-by: Nicolás Pazos <[email protected]> * add functions for translating between new proto formats symbol table and actual prometheus labels Co-authored-by: Marco Pracucci <[email protected]> Signed-off-by: Callum Styan <[email protected]> Signed-off-by: Nicolás Pazos <[email protected]> * add functionality for new minimized remote write request format Signed-off-by: Callum Styan <[email protected]> Signed-off-by: Nicolás Pazos <[email protected]> * fix minor things Signed-off-by: Callum Styan <[email protected]> Signed-off-by: Nicolás Pazos <[email protected]> * Make LabelSymbols a fixed32 Signed-off-by: Nicolás Pazos <[email protected]> * remove unused proto type Signed-off-by: Callum Styan <[email protected]> Signed-off-by: Nicolás Pazos <[email protected]> * update tests Signed-off-by: Callum Styan <[email protected]> Signed-off-by: Nicolás Pazos <[email protected]> * fix build for stringlabels tag Signed-off-by: Nicolás Pazos <[email protected]> * Use two uint32 to encode (offset,leng) Signed-off-by: Nicolás Pazos <[email protected]> * manually optimize varint marshaling Signed-off-by: Nicolás Pazos <[email protected]> * Use unsafe []byte->string cast to reuse buffer Signed-off-by: Nicolás Pazos <[email protected]> * fix writeRequestMinimizedFixture Signed-off-by: Nicolás Pazos <[email protected]> * remove all code from previous interning approach the 'minimized' version is now the only v1.1 version Signed-off-by: Nicolás Pazos <[email protected]> * minimally-tested exemplar support for rw 1.1 Signed-off-by: Nicolás Pazos <[email protected]> * refactor new version flag to make it easier to pick a specific format instead of having multiple flags, plus add new formats for testing Signed-off-by: Callum Styan <[email protected]> Signed-off-by: Nicolás Pazos <[email protected]> * use exp slices for backwards compat. to go 1.20 plus add copyright header to test file Signed-off-by: Callum Styan <[email protected]> Signed-off-by: Nicolás Pazos <[email protected]> * fix label ranging Signed-off-by: Callum Styan <[email protected]> Signed-off-by: Nicolás Pazos <[email protected]> * Add bytes slice (instead of slice of 32bit vars) format for testing Co-authored-by: Nicolás Pazos <[email protected]> Signed-off-by: Callum Styan <[email protected]> Signed-off-by: Nicolás Pazos <[email protected]> * test additional len and lenbytes formats Co-authored-by: Nicolás Pazos <[email protected]> Signed-off-by: Callum Styan <[email protected]> Signed-off-by: Nicolás Pazos <[email protected]> * remove mistaken package lock changes Signed-off-by: Callum Styan <[email protected]> Signed-off-by: Nicolás Pazos <[email protected]> * remove formats we've decided not to use Signed-off-by: Callum Styan <[email protected]> Signed-off-by: Nicolás Pazos <[email protected]> * remove more format types we probably won't use Signed-off-by: Callum Styan <[email protected]> Signed-off-by: Nicolás Pazos <[email protected]> * More cleanup Signed-off-by: Callum Styan <[email protected]> Signed-off-by: Nicolás Pazos <[email protected]> * use require instead of assert in custom marshal test Signed-off-by: Callum Styan <[email protected]> Signed-off-by: Nicolás Pazos <[email protected]> * cleanup; remove some unused functions Signed-off-by: Callum Styan <[email protected]> Signed-off-by: Nicolás Pazos <[email protected]> * more cleanup, mostly linting fixes Signed-off-by: Callum Styan <[email protected]> Signed-off-by: Nicolás Pazos <[email protected]> * remove package-lock.json change again Signed-off-by: Callum Styan <[email protected]> Signed-off-by: Nicolás Pazos <[email protected]> * more cleanup, address review comments Signed-off-by: Callum Styan <[email protected]> Signed-off-by: Nicolás Pazos <[email protected]> * fix test panic Signed-off-by: Callum Styan <[email protected]> Signed-off-by: Nicolás Pazos <[email protected]> * fix minor lint issue + use labels Range function since it looks like the tests fail to do `range labels.Labels` on CI Signed-off-by: Callum Styan <[email protected]> Signed-off-by: Nicolás Pazos <[email protected]> * new interning format based on []string indeces Co-authored-by: bwplotka <[email protected]> Signed-off-by: Nicolás Pazos <[email protected]> * remove all new rw formats but the []string one also adapt tests to the new format Signed-off-by: Nicolás Pazos <[email protected]> * cleanup rwSymbolTable Signed-off-by: Nicolás Pazos <[email protected]> * add some TODOs for later Signed-off-by: Nicolás Pazos <[email protected]> * don't reserve field 3 for new proto and add TODO Signed-off-by: Nicolás Pazos <[email protected]> * fix custom marshaling Signed-off-by: Nicolás Pazos <[email protected]> * lint Signed-off-by: Nicolás Pazos <[email protected]> * additional merge fixes Signed-off-by: Nicolás Pazos <[email protected]> * lint fixes Signed-off-by: Nicolás Pazos <[email protected]> * fix server example Signed-off-by: Nicolás Pazos <[email protected]> * revert package-lock.json changes Signed-off-by: Nicolás Pazos <[email protected]> * update example prometheus version Signed-off-by: Nicolás Pazos <[email protected]> * define separate proto types for remote write 2.0 Signed-off-by: Nicolás Pazos <[email protected]> * lint Signed-off-by: Nicolás Pazos <[email protected]> * rename new proto types and move to separate pkg Signed-off-by: Nicolás Pazos <[email protected]> * update prometheus version for example Signed-off-by: Nicolás Pazos <[email protected]> * make proto Signed-off-by: Nicolás Pazos <[email protected]> * make Metadata not nullable Signed-off-by: Nicolás Pazos <[email protected]> * remove old MinSample proto message Signed-off-by: Nicolás Pazos <[email protected]> * change enum names to fit buf build recommend enum naming and lint rules Signed-off-by: Callum Styan <[email protected]> * remote: Added test for classic histogram grouping when sending rw; Fixed queue manager test delay. (#13421) Signed-off-by: bwplotka <[email protected]> * Remote write v2: metadata support in every write request (#13394) * Approach bundling metadata along with samples and exemplars Signed-off-by: Paschalis Tsilias <[email protected]> * Add first test; rebase with main Signed-off-by: Paschalis Tsilias <[email protected]> * Alternative approach: bundle metadata in TimeSeries protobuf Signed-off-by: Paschalis Tsilias <[email protected]> * update go mod to match main branch Signed-off-by: Callum Styan <[email protected]> * fix after rebase Signed-off-by: Callum Styan <[email protected]> * we're not going to modify the 1.X format anymore Signed-off-by: Callum Styan <[email protected]> * Modify AppendMetadata based on the fact that we be putting metadata into timeseries Signed-off-by: Callum Styan <[email protected]> * Rename enums for remote write versions to something that makes more sense + remove the added `sendMetadata` flag. Signed-off-by: Callum Styan <[email protected]> * rename flag that enables writing of metadata records to the WAL Signed-off-by: Callum Styan <[email protected]> * additional clean up Signed-off-by: Callum Styan <[email protected]> * lint Signed-off-by: Callum Styan <[email protected]> * fix usage of require.Len Signed-off-by: Callum Styan <[email protected]> * some clean up from review comments Signed-off-by: Callum Styan <[email protected]> * more review fixes Signed-off-by: Callum Styan <[email protected]> --------- Signed-off-by: Paschalis Tsilias <[email protected]> Signed-off-by: Callum Styan <[email protected]> Co-authored-by: Paschalis Tsilias <[email protected]> * remote write 2.0: sync with `main` branch (#13510) * consoles: exclude iowait and steal from CPU Utilisation 'iowait' and 'steal' indicate specific idle/wait states, which shouldn't be counted into CPU Utilisation. Also see https://github.com/prometheus-operator/kube-prometheus/pull/796 and https://github.com/kubernetes-monitoring/kubernetes-mixin/pull/667. Per the iostat man page: %idle Show the percentage of time that the CPU or CPUs were idle and the system did not have an outstanding disk I/O request. %iowait Show the percentage of time that the CPU or CPUs were idle during which the system had an outstanding disk I/O request. %steal Show the percentage of time spent in involuntary wait by the virtual CPU or CPUs while the hypervisor was servicing another virtual processor. Signed-off-by: Julian Wiedmann <[email protected]> * tsdb: shrink txRing with smaller integers 4 billion active transactions ought to be enough for anyone. Signed-off-by: Bryan Boreham <[email protected]> * tsdb: create isolation transaction slice on demand When Prometheus restarts it creates every series read in from the WAL, but many of those series will be finished, and never receive any more samples. By defering allocation of the txRing slice to when it is first needed, we save 32 bytes per stale series. Signed-off-by: Bryan Boreham <[email protected]> * add cluster variable to Overview dashboard Signed-off-by: Erik Sommer <[email protected]> * promql: simplify Native Histogram arithmetics Signed-off-by: Linas Medziunas <[email protected]> * Cut 2.49.0-rc.0 (#13270) * Cut 2.49.0-rc.0 Signed-off-by: bwplotka <[email protected]> * Removed the duplicate. Signed-off-by: bwplotka <[email protected]> --------- Signed-off-by: bwplotka <[email protected]> * Add unit protobuf parser Signed-off-by: Arianna Vespri <[email protected]> * Go on adding protobuf parsing for unit Signed-off-by: Arianna Vespri <[email protected]> * ui: create a reproduction for https://github.com/prometheus/prometheus/issues/13292 Signed-off-by: machine424 <[email protected]> * Get conditional right Signed-off-by: Arianna Vespri <[email protected]> * Get VM Scale Set NIC (#13283) Calling `*armnetwork.InterfacesClient.Get()` doesn't work for Scale Set VM NIC, because these use a different Resource ID format. Use `*armnetwork.InterfacesClient.GetVirtualMachineScaleSetNetworkInterface()` instead. This needs both the scale set name and the instance ID, so add an `InstanceID` field to the `virtualMachine` struct. `InstanceID` is empty for a VM that isn't a ScaleSetVM. Signed-off-by: Daniel Nicholls <[email protected]> * Cut v2.49.0-rc.1 Signed-off-by: bwplotka <[email protected]> * Delete debugging lines, amend error message for unit Signed-off-by: Arianna Vespri <[email protected]> * Correct order in error message Signed-off-by: Arianna Vespri <[email protected]> * Consider storage.ErrTooOldSample as non-retryable Signed-off-by: Daniel Kerbel <[email protected]> * scrape_test.go: Increase scrape interval in TestScrapeLoopCache to reduce potential flakiness Signed-off-by: machine424 <[email protected]> * Avoid creating string for suffix, consider counters without _total suffix Signed-off-by: Arianna Vespri <[email protected]> * build(deps): bump github.com/prometheus/client_golang Bumps [github.com/prometheus/client_golang](https://github.com/prometheus/client_golang) from 1.17.0 to 1.18.0. - [Release notes](https://github.com/prometheus/client_golang/releases) - [Changelog](https://github.com/prometheus/client_golang/blob/main/CHANGELOG.md) - [Commits](https://github.com/prometheus/client_golang/compare/v1.17.0...v1.18.0) --- updated-dependencies: - dependency-name: github.com/prometheus/client_golang dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> * build(deps): bump actions/setup-node from 3.8.1 to 4.0.1 Bumps [actions/setup-node](https://github.com/actions/setup-node) from 3.8.1 to 4.0.1. - [Release notes](https://github.com/actions/setup-node/releases) - [Commits](https://github.com/actions/setup-node/compare/5e21ff4d9bc1a8cf6de233a3057d20ec6b3fb69d...b39b52d1213e96004bfcb1c61a8a6fa8ab84f3e8) --- updated-dependencies: - dependency-name: actions/setup-node dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> * scripts: sort file list in embed directive Otherwise the resulting string depends on find, which afaict depends on the underlying filesystem. A stable file list make it easier to detect UI changes in downstreams that need to track UI assets. Signed-off-by: Jan Fajerski <[email protected]> * Fix DataTableProps['data'] for resultType string Signed-off-by: Kevin Mingtarja <[email protected]> * Fix handling of scalar and string in isHeatmapData Signed-off-by: Kevin Mingtarja <[email protected]> * build(deps): bump github.com/influxdata/influxdb Bumps [github.com/influxdata/influxdb](https://github.com/influxdata/influxdb) from 1.11.2 to 1.11.4. - [Release notes](https://github.com/influxdata/influxdb/releases) - [Commits](https://github.com/influxdata/influxdb/compare/v1.11.2...v1.11.4) --- updated-dependencies: - dependency-name: github.com/influxdata/influxdb dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> * build(deps): bump github.com/prometheus/prometheus Bumps [github.com/prometheus/prometheus](https://github.com/prometheus/prometheus) from 0.48.0 to 0.48.1. - [Release notes](https://github.com/prometheus/prometheus/releases) - [Changelog](https://github.com/prometheus/prometheus/blob/main/CHANGELOG.md) - [Commits](https://github.com/prometheus/prometheus/compare/v0.48.0...v0.48.1) --- updated-dependencies: - dependency-name: github.com/prometheus/prometheus dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> * Bump client_golang to v1.18.0 (#13373) Signed-off-by: Paschalis Tsilias <[email protected]> * Drop old inmemory samples (#13002) * Drop old inmemory samples Co-authored-by: Paschalis Tsilias <[email protected]> Signed-off-by: Paschalis Tsilias <[email protected]> Signed-off-by: Marc Tuduri <[email protected]> * Avoid copying timeseries when the feature is disabled Signed-off-by: Paschalis Tsilias <[email protected]> Signed-off-by: Marc Tuduri <[email protected]> * Run gofmt Signed-off-by: Paschalis Tsilias <[email protected]> Signed-off-by: Marc Tuduri <[email protected]> * Clarify docs Signed-off-by: Marc Tuduri <[email protected]> * Add more logging info Signed-off-by: Marc Tuduri <[email protected]> * Remove loggers Signed-off-by: Marc Tuduri <[email protected]> * optimize function and add tests Signed-off-by: Marc Tuduri <[email protected]> * Simplify filter Signed-off-by: Marc Tuduri <[email protected]> * rename var Signed-off-by: Marc Tuduri <[email protected]> * Update help info from metrics Signed-off-by: Marc Tuduri <[email protected]> * use metrics to keep track of drop elements during buildWriteRequest Signed-off-by: Marc Tuduri <[email protected]> * rename var in tests Signed-off-by: Marc Tuduri <[email protected]> * pass time.Now as parameter Signed-off-by: Marc Tuduri <[email protected]> * Change buildwriterequest during retries Signed-off-by: Marc Tuduri <[email protected]> * Revert "Remove loggers" This reverts commit 54f91dfcae20488944162335ab4ad8be459df1ab. Signed-off-by: Marc Tuduri <[email protected]> * use log level debug for loggers Signed-off-by: Marc Tuduri <[email protected]> * Fix linter Signed-off-by: Paschalis Tsilias <[email protected]> * Remove noisy debug-level logs; add 'reason' label to drop metrics Signed-off-by: Paschalis Tsilias <[email protected]> * Remove accidentally committed files Signed-off-by: Paschalis Tsilias <[email protected]> * Propagate logger to buildWriteRequest to log dropped data Signed-off-by: Paschalis Tsilias <[email protected]> * Fix docs comment Signed-off-by: Paschalis Tsilias <[email protected]> * Make drop reason more specific Signed-off-by: Paschalis Tsilias <[email protected]> * Remove unnecessary pass of logger Signed-off-by: Paschalis Tsilias <[email protected]> * Use snake_case for reason label Signed-off-by: Paschalis Tsilias <[email protected]> * Fix dropped samples metric Signed-off-by: Paschalis Tsilias <[email protected]> --------- Signed-off-by: Paschalis Tsilias <[email protected]> Signed-off-by: Marc Tuduri <[email protected]> Signed-off-by: Paschalis Tsilias <[email protected]> Co-authored-by: Paschalis Tsilias <[email protected]> Co-authored-by: Paschalis Tsilias <[email protected]> * fix(discovery): allow requireUpdate util to timeout in discovery/file/file_test.go. The loop ran indefinitely if the condition isn't met. Before, each iteration created a new timer channel which was always outpaced by the other timer channel with smaller duration. minor detail: There was a memory leak: resources of the ~10 previous timers were constantly kept. With the fix, we may keep the resources of one timer around for defaultWait but this isn't worth the changes to make it right. Signed-off-by: machine424 <[email protected]> * Merge pull request #13371 from kevinmingtarja/fix-isHeatmapData ui: fix handling of scalar and string in isHeatmapData * tsdb/{index,compact}: allow using custom postings encoding format (#13242) * tsdb/{index,compact}: allow using custom postings encoding format We would like to experiment with a different postings encoding format in Thanos so in this change I am proposing adding another argument to `NewWriter` which would allow users to change the format if needed. Also, wire the leveled compactor so that it would be possible to change the format there too. Signed-off-by: Giedrius Statkevičius <[email protected]> * tsdb/compact: use a struct for leveled compactor options As discussed on Slack, let's use a struct for the options in leveled compactor. Signed-off-by: Giedrius Statkevičius <[email protected]> * tsdb: make changes after Bryan's review - Make changes less intrusive - Turn the postings encoder type into a function - Add NewWriterWithEncoder() Signed-off-by: Giedrius Statkevičius <[email protected]> --------- Signed-off-by: Giedrius Statkevičius <[email protected]> * Cut 2.49.0-rc.2 Signed-off-by: bwplotka <[email protected]> * build(deps): bump actions/setup-go from 3.5.0 to 5.0.0 in /scripts (#13362) Bumps [actions/setup-go](https://github.com/actions/setup-go) from 3.5.0 to 5.0.0. - [Release notes](https://github.com/actions/setup-go/releases) - [Commits](https://github.com/actions/setup-go/compare/6edd4406fa81c3da01a34fa6f6343087c207a568...0c52d547c9bc32b1aa3301fd7a9cb496313a4491) --- updated-dependencies: - dependency-name: actions/setup-go dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * build(deps): bump github/codeql-action from 2.22.8 to 3.22.12 (#13358) Bumps [github/codeql-action](https://github.com/github/codeql-action) from 2.22.8 to 3.22.12. - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](https://github.com/github/codeql-action/compare/407ffafae6a767df3e0230c3df91b6443ae8df75...012739e5082ff0c22ca6d6ab32e07c36df03c4a4) --- updated-dependencies: - dependency-name: github/codeql-action dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * put @nexucis has a release shepherd (#13383) Signed-off-by: Augustin Husson <[email protected]> * Add analyze histograms command to promtool (#12331) Add `query analyze` command to promtool This command analyzes the buckets of classic and native histograms, based on data queried from the Prometheus query API, i.e. it doesn't require direct access to the TSDB files. Signed-off-by: Jeanette Tan <[email protected]> --------- Signed-off-by: Jeanette Tan <[email protected]> * included instance in all necessary descriptions Signed-off-by: Erik Sommer <[email protected]> * tsdb/compact: fix passing merge func Fixing a very small logical problem I've introduced :(. Signed-off-by: Giedrius Statkevičius <[email protected]> * tsdb: add enable overlapping compaction This functionality is needed in downstream projects because they have a separate component that does compaction. Upstreaming https://github.com/grafana/mimir-prometheus/blob/7c8e9a2a76fc729e9078889782928b2fdfe240e9/tsdb/compact.go#L323-L325. Signed-off-by: Giedrius Statkevičius <[email protected]> * Cut 2.49.0 Signed-off-by: bwplotka <[email protected]> * promtool: allow setting multiple matchers to "promtool tsdb dump" command. (#13296) Conditions are ANDed inside the same matcher but matchers are ORed Including unit tests for "promtool tsdb dump". Refactor some matchers scraping utils. Signed-off-by: machine424 <[email protected]> * Fixed changelog Signed-off-by: bwplotka <[email protected]> * tsdb/main: wire "EnableOverlappingCompaction" to tsdb.Options (#13398) This added the https://github.com/prometheus/prometheus/pull/13393 "EnableOverlappingCompaction" parameter to the compactor code but not to the tsdb.Options. I forgot about that. Add it to `tsdb.Options` too and set it to `true` in Prometheus. Copy/paste the description from https://github.com/prometheus/prometheus/pull/13393#issuecomment-1891787986 Signed-off-by: Giedrius Statkevičius <[email protected]> * Issue #13268: fix quality value in accept header Signed-off-by: Kumar Kalpadiptya Roy <[email protected]> * Cut 2.49.1 with scrape q= bugfix. Signed-off-by: bwplotka <[email protected]> * Cut 2.49.1 web package. Signed-off-by: bwplotka <[email protected]> * Restore more efficient version of NewPossibleNonCounterInfo annotation (#13022) Restore more efficient version of NewPossibleNonCounterInfo annotation Signed-off-by: Jeanette Tan <[email protected]> --------- Signed-off-by: Jeanette Tan <[email protected]> * Fix regressions introduced by #13242 Signed-off-by: Marco Pracucci <[email protected]> * fix slice copy in 1.20 (#13389) The slices package is added to the standard library in Go 1.21; we need to import from the exp area to maintain compatibility with Go 1.20. Signed-off-by: tyltr <[email protected]> * Docs: Query Basics: link to rate (#10538) Co-authored-by: Julien Pivotto <[email protected]> * chore(kubernetes): check preconditions earlier and avoid unnecessary checks or iterations Signed-off-by: machine424 <[email protected]> * Examples: link to `rate` for new users (#10535) * Examples: link to `rate` for new users Signed-off-by: Ted Robertson [email protected] Co-authored-by: Bryan Boreham <[email protected]> * promql: use natural sort in sort_by_label and sort_by_label_desc (#13411) These functions are intended for humans, as robots can already sort the results however they please. Humans like things sorted "naturally": * https://blog.codinghorror.com/sorting-for-humans-natural-sort-order/ A similar thing has been done to Grafana, which is also used by humans: * https://github.com/grafana/grafana/pull/78024 * https://github.com/grafana/grafana/pull/78494 Signed-off-by: Ivan Babrou <[email protected]> * TestLabelValuesWithMatchers: Add test case Signed-off-by: Arve Knudsen <[email protected]> * remove obsolete build tag Signed-off-by: tyltr <[email protected]> * Upgrade some golang dependencies for resty 2.11 Signed-off-by: Israel Blancas <[email protected]> * Native Histograms: support `native_histogram_min_bucket_factor` in scrape_config (#13222) Native Histograms: support native_histogram_min_bucket_factor in scrape_config --------- Signed-off-by: Ziqi Zhao <[email protected]> Signed-off-by: Björn Rabenstein <[email protected]> Co-authored-by: George Krajcsovits <[email protected]> Co-authored-by: Björn Rabenstein <[email protected]> * Add warnings for histogramRate applied with isCounter not matching counter/gauge histogram (#13392) Add warnings for histogramRate applied with isCounter not matching counter/gauge histogram --------- Signed-off-by: Jeanette Tan <[email protected]> * Minor fixes to otlp vendor update script Signed-off-by: Goutham <[email protected]> * build(deps): bump github.com/hetznercloud/hcloud-go/v2 Bumps [github.com/hetznercloud/hcloud-go/v2](https://github.com/hetznercloud/hcloud-go) from 2.4.0 to 2.6.0. - [Release notes](https://github.com/hetznercloud/hcloud-go/releases) - [Changelog](https://github.com/hetznercloud/hcloud-go/blob/main/CHANGELOG.md) - [Commits](https://github.com/hetznercloud/hcloud-go/compare/v2.4.0...v2.6.0) --- updated-dependencies: - dependency-name: github.com/hetznercloud/hcloud-go/v2 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> * Enhanced visibility for `promtool test rules` with JSON colored formatting (#13342) * Added diff flag for unit test to improvise readability & debugging Signed-off-by: Rewanth Tammana <[email protected]> * Removed blank spaces Signed-off-by: Rewanth Tammana <[email protected]> * Fixed linting error Signed-off-by: Rewanth Tammana <[email protected]> * Added cli flags to documentation Signed-off-by: Rewanth Tammana <[email protected]> * Revert unrrelated linting fixes Signed-off-by: Rewanth Tammana <[email protected]> * Fixed review suggestions Signed-off-by: Rewanth Tammana <[email protected]> * Cleanup Signed-off-by: Rewanth Tammana <[email protected]> * Updated flag description Signed-off-by: Rewanth Tammana <[email protected]> * Updated flag description Signed-off-by: Rewanth Tammana <[email protected]> --------- Signed-off-by: Rewanth Tammana <[email protected]> * storage: skip merging when no remote storage configured Prometheus is hard-coded to use a fanout storage between TSDB and a remote storage which by default is empty. This change detects the empty storage and skips merging between result sets, which would make `Select()` sort results. Bottom line: we skip a sort unless there really is some remote storage configured. Signed-off-by: Bryan Boreham <[email protected]> * Remove csmarchbanks from remote write owners (#13432) I have not had the time to keep up with remote write and have no plans to work on it in the near future so I am withdrawing my maintainership of that part of the codebase. I continue to focus on client_python. Signed-off-by: Chris Marchbanks <[email protected]> * add more context cancellation check at evaluation time Signed-off-by: Ben Ye <[email protected]> * Optimize label values with matchers by taking shortcuts (#13426) Don't calculate postings beforehand: we may not need them. If all matchers are for the requested label, we can just filter its values. Also, if there are no values at all, no need to run any kind of logic. Also add more labelValuesWithMatchers benchmarks Signed-off-by: Oleg Zaytsev <[email protected]> * Add automatic memory limit handling Enable automatic detection of memory limits and configure GOMEMLIMIT to match. * Also includes a flag to allow controlling the reserved ratio. Signed-off-by: SuperQ <[email protected]> * Update OSSF badge link (#13433) Provide a more user friendly interface Signed-off-by: Matthieu MOREL <[email protected]> * SD Managers taking over responsibility for registration of debug metrics (#13375) SD Managers take over responsibility for SD metrics registration --------- Signed-off-by: Paulin Todev <[email protected]> Signed-off-by: Björn Rabenstein <[email protected]> Co-authored-by: Björn Rabenstein <[email protected]> * Optimize histogram iterators (#13340) Optimize histogram iterators Histogram iterators allocate new objects in the AtHistogram and AtFloatHistogram methods, which makes calculating rates over long ranges expensive. In #13215 we allowed an existing object to be reused when converting an integer histogram to a float histogram. This commit follows the same idea and allows injecting an existing object in the AtHistogram and AtFloatHistogram methods. When the injected value is nil, iterators allocate new histograms, otherwise they populate and return the injected object. The commit also adds a CopyTo method to Histogram and FloatHistogram which is used in the BufferedIterator to overwrite items in the ring instead of making new copies. Note that a specialized HPoint pool is needed for all of this to work (`matrixSelectorHPool`). --------- Signed-off-by: Filip Petkovski <[email protected]> Co-authored-by: George Krajcsovits <[email protected]> * doc: Mark `mad_over_time` as experimental (#13440) We forgot to do that in https://github.com/prometheus/prometheus/pull/13059 Signed-off-by: beorn7 <[email protected]> * Change metric label for Puppetdb from 'http' to 'puppetdb' Signed-off-by: Paulin Todev <[email protected]> * mirror metrics.proto change & generate code Signed-off-by: Ziqi Zhao <[email protected]> * TestHeadLabelValuesWithMatchers: Add test case (#13414) Add test case to TestHeadLabelValuesWithMatchers, while fixing a couple of typos in other test cases. Also enclosing some implicit sub-tests in a `t.Run` call to make them explicitly sub-tests. Signed-off-by: Arve Knudsen <[email protected]> * update all go dependencies (#13438) Signed-off-by: Augustin Husson <[email protected]> * build(deps): bump the k8s-io group with 2 updates (#13454) Bumps the k8s-io group with 2 updates: [k8s.io/api](https://github.com/kubernetes/api) and [k8s.io/client-go](https://github.com/kubernetes/client-go). Updates `k8s.io/api` from 0.28.4 to 0.29.1 - [Commits](https://github.com/kubernetes/api/compare/v0.28.4...v0.29.1) Updates `k8s.io/client-go` from 0.28.4 to 0.29.1 - [Changelog](https://github.com/kubernetes/client-go/blob/master/CHANGELOG.md) - [Commits](https://github.com/kubernetes/client-go/compare/v0.28.4...v0.29.1) --- updated-dependencies: - dependency-name: k8s.io/api dependency-type: direct:production update-type: version-update:semver-minor dependency-group: k8s-io - dependency-name: k8s.io/client-go dependency-type: direct:production update-type: version-update:semver-minor dependency-group: k8s-io ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * build(deps): bump the go-opentelemetry-io group with 1 update (#13453) Bumps the go-opentelemetry-io group with 1 update: [go.opentelemetry.io/collector/semconv](https://github.com/open-telemetry/opentelemetry-collector). Updates `go.opentelemetry.io/collector/semconv` from 0.92.0 to 0.93.0 - [Release notes](https://github.com/open-telemetry/opentelemetry-collector/releases) - [Changelog](https://github.com/open-telemetry/opentelemetry-collector/blob/main/CHANGELOG-API.md) - [Commits](https://github.com/open-telemetry/opentelemetry-collector/compare/v0.92.0...v0.93.0) --- updated-dependencies: - dependency-name: go.opentelemetry.io/collector/semconv dependency-type: direct:production update-type: version-update:semver-minor dependency-group: go-opentelemetry-io ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * build(deps): bump actions/upload-artifact from 3.1.3 to 4.0.0 (#13355) Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact) from 3.1.3 to 4.0.0. - [Release notes](https://github.com/actions/upload-artifact/releases) - [Commits](https://github.com/actions/upload-artifact/compare/a8a3f3ad30e3422c9c7b888a15615d19a852ae32...c7d193f32edcb7bfad88892161225aeda64e9392) --- updated-dependencies: - dependency-name: actions/upload-artifact dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * build(deps): bump bufbuild/buf-push-action (#13357) Bumps [bufbuild/buf-push-action](https://github.com/bufbuild/buf-push-action) from 342fc4cdcf29115a01cf12a2c6dd6aac68dc51e1 to a654ff18effe4641ebea4a4ce242c49800728459. - [Release notes](https://github.com/bufbuild/buf-push-action/releases) - [Commits](https://github.com/bufbuild/buf-push-action/compare/342fc4cdcf29115a01cf12a2c6dd6aac68dc51e1...a654ff18effe4641ebea4a4ce242c49800728459) --- updated-dependencies: - dependency-name: bufbuild/buf-push-action dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Labels: Add DropMetricName function, used in PromQL (#13446) This function is called very frequently when executing PromQL functions, and we can do it much more efficiently inside Labels. In the common case that `__name__` comes first in the labels, we simply re-point to start at the next label, which is nearly free. `DropMetricName` is now so cheap I removed the cache - benchmarks show everything still goes faster. Signed-off-by: Bryan Boreham <[email protected]> * tsdb: simplify internal series delete function (#13261) Lifting an optimisation from Agent code, `seriesHashmap.del` can use the unique series reference, doesn't need to check Labels. Also streamline the logic for deleting from `unique` and `conflicts` maps, and add some comments to help the next person. Signed-off-by: Bryan Boreham <[email protected]> * otlptranslator/update-copy.sh: Fix sed command lines Signed-off-by: Arve Knudsen <[email protected]> * Rollback k8s.io requirements (#13462) Rollback k8s.io Go modules to v0.28.6 to avoid forcing upgrade of Go to 1.21. This allows us to keep compatibility with the currently supported upstream Go releases. Signed-off-by: SuperQ <[email protected]> * Make update-copy.sh work for both OSX and GNU sed Signed-off-by: Arve Knudsen <[email protected]> * Name @beorn7 and @krajorama as maintainers for native histograms I have been the de-facto maintainer for native histograms from the beginning. So let's put this into MAINTAINERS.md. In addition, I hereby proposose George Krajcsovits AKA Krajo as a co-maintainer. He has contributed a lot of native histogram code, but more importantly, he has contributed substantially to reviewing other contributors' native histogram code, up to a point where I was merely rubberstamping the PRs he had already reviewed. I'm confident that he is ready to to be granted commit rights as outlined in the "Maintainers" section of the governance: https://prometheus.io/governance/#maintainers According to the same section of the governance, I will announce the proposed change on the developers mailing list and will give some time for lazy consensus before merging this PR. Signed-off-by: beorn7 <[email protected]> * ui/fix: correct url handling for stacked graphs (#13460) Signed-off-by: Yury Moladau <[email protected]> * tsdb: use cheaper Mutex on series Mutex is 8 bytes; RWMutex is 24 bytes and much more complicated. Since `RLock` is only used in two places, `UpdateMetadata` and `Delete`, neither of which are hotspots, we should use the cheaper one. Signed-off-by: Bryan Boreham <[email protected]> * Fix last_over_time for native histograms The last_over_time retains a histogram sample without making a copy. This sample is now coming from the buffered iterator used for windowing functions, and can be reused for reading subsequent samples as the iterator progresses. I would propose copying the sample in the last_over_time function, similar to how it is done for rate, sum_over_time and others. Signed-off-by: Filip Petkovski <[email protected]> * Implementation NOTE: Rebased from main after refactor in #13014 Signed-off-by: Danny Kopping <[email protected]> * Add feature flag Signed-off-by: Danny Kopping <[email protected]> * Refactor concurrency control Signed-off-by: Danny Kopping <[email protected]> * Optimising dependencies/dependents funcs to not produce new slices each request Signed-off-by: Danny Kopping <[email protected]> * Refactoring Signed-off-by: Danny Kopping <[email protected]> * Rename flag Signed-off-by: Danny Kopping <[email protected]> * Refactoring for performance, and to allow controller to be overridden Signed-off-by: Danny Kopping <[email protected]> * Block until all rules, both sync & async, have completed evaluating Updated & added tests Review feedback nits Return empty map if not indeterminate Use highWatermark to track inflight requests counter Appease the linter Clarify feature flag Signed-off-by: Danny Kopping <[email protected]> * Fix typo in CLI flag description Signed-off-by: Marco Pracucci <[email protected]> * Fixed auto-generated doc Signed-off-by: Marco Pracucci <[email protected]> * Improve doc Signed-off-by: Marco Pracucci <[email protected]> * Simplify the design to update concurrency controller once the rule evaluation has done Signed-off-by: Marco Pracucci <[email protected]> * Add more test cases to TestDependenciesEdgeCases Signed-off-by: Marco Pracucci <[email protected]> * Added more test cases to TestDependenciesEdgeCases Signed-off-by: Marco Pracucci <[email protected]> * Improved RuleConcurrencyController interface doc Signed-off-by: Marco Pracucci <[email protected]> * Introduced sequentialRuleEvalController Signed-off-by: Marco Pracucci <[email protected]> * Remove superfluous nil check in Group.metrics Signed-off-by: Marco Pracucci <[email protected]> * api: Serialize discovered and target labels into JSON directly (#13469) Converted maps into labels.Labels to avoid a lot of copying of data which leads to very high memory consumption while opening the /service-discovery endpoint in the Prometheus UI Signed-off-by: Leegin <[email protected]> * api: Serialize discovered labels into JSON directly in dropped targets (#13484) Converted maps into labels.Labels to avoid a lot of copying of data which leads to very high memory consumption while opening the /service-discovery endpoint in the Prometheus UI Signed-off-by: Leegin <[email protected]> * Add ShardedPostings() support to TSDB (#10421) This PR is a reference implementation of the proposal described in #10420. In addition to what described in #10420, in this PR I've introduced labels.StableHash(). The idea is to offer an hashing function which doesn't change over time, and that's used by query sharding in order to get a stable behaviour over time. The implementation of labels.StableHash() is the hashing function used by Prometheus before stringlabels, and what's used by Grafana Mimir for query sharding (because built before stringlabels was a thing). Follow up work As mentioned in #10420, if this PR is accepted I'm also open to upload another foundamental piece used by Grafana Mimir query sharding to accelerate the query execution: an optional, configurable and fast in-memory cache for the series hashes. Signed-off-by: Marco Pracucci <[email protected]> * storage/remote: document why two benchmarks are skipped One was silently doing nothing; one was doing something but the work didn't go up linearly with iteration count. Signed-off-by: Bryan Boreham <[email protected]> * Pod status changes not discovered by Kube Endpoints SD (#13337) * fix(discovery/kubernetes/endpoints): react to changes on Pods because some modifications can occur on them without triggering an update on the related Endpoints (The Pod phase changing from Pending to Running e.g.). --------- Signed-off-by: machine424 <[email protected]> Co-authored-by: Guillermo Sanchez Gavier <[email protected]> * Small improvements, add const, remove copypasta (#8106) Signed-off-by: Mikhail Fesenko <[email protected]> Signed-off-by: Jesus Vazquez <[email protected]> * Proposal to improve FPointSlice and HPointSlice allocation. (#13448) * Reusing points slice from previous series when the slice is under utilized * Adding comments on the bench test Signed-off-by: Alan Protasio <[email protected]> * lint Signed-off-by: Nicolás Pazos <[email protected]> * go mod tidy Signed-off-by: Nicolás Pazos <[email protected]> --------- Signed-off-by: Julian Wiedmann <[email protected]> Signed-off-by: Bryan Boreham <[email protected]> Signed-off-by: Erik Sommer <[email protected]> Signed-off-by: Linas Medziunas <[email protected]> Signed-off-by: bwplotka <[email protected]> Signed-off-by: Arianna Vespri <[email protected]> Signed-off-by: machine424 <[email protected]> Signed-off-by: Daniel Nicholls <[email protected]> Signed-off-by: Daniel Kerbel <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: Jan Fajerski <[email protected]> Signed-off-by: Kevin Mingtarja <[email protected]> Signed-off-by: Paschalis Tsilias <[email protected]> Signed-off-by: Marc Tuduri <[email protected]> Signed-off-by: Paschalis Tsilias <[email protected]> Signed-off-by: Giedrius Statkevičius <[email protected]> Signed-off-by: Augustin Husson <[email protected]> Signed-off-by: Jeanette Tan <[email protected]> Signed-off-by: Bartlomiej Plotka <[email protected]> Signed-off-by: Kumar Kalpadiptya Roy <[email protected]> Signed-off-by: Marco Pracucci <[email protected]> Signed-off-by: tyltr <[email protected]> Signed-off-by: Ted Robertson [email protected] Signed-off-by: Ivan Babrou <[email protected]> Signed-off-by: Arve Knudsen <[email protected]> Signed-off-by: Israel Blancas <[email protected]> Signed-off-by: Ziqi Zhao <[email protected]> Signed-off-by: Björn Rabenstein <[email protected]> Signed-off-by: Goutham <[email protected]> Signed-off-by: Rewanth Tammana <[email protected]> Signed-off-by: Chris Marchbanks <[email protected]> Signed-off-by: Ben Ye <[email protected]> Signed-off-by: Oleg Zaytsev <[email protected]> Signed-off-by: SuperQ <[email protected]> Signed-off-by: Ben Kochie <[email protected]> Signed-off-by: Matthieu MOREL <[email protected]> Signed-off-by: Paulin Todev <[email protected]> Signed-off-by: Filip Petkovski <[email protected]> Signed-off-by: beorn7 <[email protected]> Signed-off-by: Augustin Husson <[email protected]> Signed-off-by: Yury Moladau <[email protected]> Signed-off-by: Danny Kopping <[email protected]> Signed-off-by: Leegin <[email protected]> Signed-off-by: Mikhail Fesenko <[email protected]> Signed-off-by: Jesus Vazquez <[email protected]> Signed-off-by: Alan Protasio <[email protected]> Signed-off-by: Nicolás Pazos <[email protected]> Co-authored-by: Julian Wiedmann <[email protected]> Co-authored-by: Bryan Boreham <[email protected]> Co-authored-by: Erik Sommer <[email protected]> Co-authored-by: Linas Medziunas <[email protected]> Co-authored-by: Bartlomiej Plotka <[email protected]> Co-authored-by: Arianna Vespri <[email protected]> Co-authored-by: machine424 <[email protected]> Co-authored-by: daniel-resdiary <[email protected]> Co-authored-by: Daniel Kerbel <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Jan Fajerski <[email protected]> Co-authored-by: Kevin Mingtarja <[email protected]> Co-authored-by: Paschalis Tsilias <[email protected]> Co-authored-by: Marc Tudurí <[email protected]> Co-authored-by: Paschalis Tsilias <[email protected]> Co-authored-by: Giedrius Statkevičius <[email protected]> Co-authored-by: Augustin Husson <[email protected]> Co-authored-by: Björn Rabenstein <[email protected]> Co-authored-by: zenador <[email protected]> Co-authored-by: gotjosh <[email protected]> Co-authored-by: Ben Kochie <[email protected]> Co-authored-by: Kumar Kalpadiptya Roy <[email protected]> Co-authored-by: Marco Pracucci <[email protected]> Co-authored-by: tyltr <[email protected]> Co-authored-by: Ted Robertson <[email protected]> Co-authored-by: Julien Pivotto <[email protected]> Co-authored-by: Matthias Loibl <[email protected]> Co-authored-by: Ivan Babrou <[email protected]> Co-authored-by: Arve Knudsen <[email protected]> Co-authored-by: Israel Blancas <[email protected]> Co-authored-by: Ziqi Zhao <[email protected]> Co-authored-by: George Krajcsovits <[email protected]> Co-authored-by: Björn Rabenstein <[email protected]> Co-authored-by: Goutham <[email protected]> Co-authored-by: Rewanth Tammana <[email protected]> Co-authored-by: Chris Marchbanks <[email protected]> Co-authored-by: Ben Ye <[email protected]> Co-authored-by: Oleg Zaytsev <[email protected]> Co-authored-by: Matthieu MOREL <[email protected]> Co-authored-by: Paulin Todev <[email protected]> Co-authored-by: Filip Petkovski <[email protected]> Co-authored-by: Yury Molodov <[email protected]> Co-authored-by: Danny Kopping <[email protected]> Co-authored-by: Leegin <[email protected]> Co-authored-by: Guillermo Sanchez Gavier <[email protected]> Co-authored-by: Mikhail Fesenko <[email protected]> Co-authored-by: Alan Protasio <[email protected]> * remote write 2.0 - follow up improvements (#13478) * move remote write proto version config from a remote storage config to a per remote write configuration option Signed-off-by: Callum Styan <[email protected]> * rename scrape config for metadata, fix 2.0 header var name/value (was 1.1), and more clean up Signed-off-by: Callum Styan <[email protected]> * address review comments, mostly lint fixes Signed-off-by: Callum Styan <[email protected]> * another lint fix Signed-off-by: Callum Styan <[email protected]> * lint imports Signed-off-by: Callum Styan <[email protected]> --------- Signed-off-by: Callum Styan <[email protected]> * go mod tidy Signed-off-by: Nicolás Pazos <[email protected]> * Added commmentary to RW 2.0 protocol for easier adoption and explicit semantics. (#13502) * Added commmentary to RW 2.0 protocol for easier adoption and explicit semantics. Signed-off-by: bwplotka <[email protected]> * Apply suggestions from code review Co-authored-by: Nico Pazos <[email protected]> Signed-off-by: Callum Styan <[email protected]> --------- Signed-off-by: bwplotka <[email protected]> Signed-off-by: Callum Styan <[email protected]> Co-authored-by: Callum Styan <[email protected]> Co-authored-by: Nico Pazos <[email protected]> * prw2.0: Added support for "custom" layouts for native histogram proto (#13558) * prw2.0: Added support for "custom" layouts for native histogram. Result of the discussions: * https://github.com/prometheus/prometheus/issues/13475#issuecomment-1931496924 * https://cloud-native.slack.com/archives/C02KR205UMU/p1707301006347199 Signed-off-by: bwplotka <[email protected]> * prw2.0: Added support for "custom" layouts for native histogram. Result of the discussions: * https://github.com/prometheus/prometheus/issues/13475#issuecomment-1931496924 * https://cloud-native.slack.com/archives/C02KR205UMU/p1707301006347199 Signed-off-by: bwplotka <[email protected]> # Conflicts: # prompb/write/v2/types.pb.go * Update prompb/write/v2/types.proto Co-authored-by: George Krajcsovits <[email protected]> Signed-off-by: Bartlomiej Plotka <[email protected]> * Addressed comments, fixed test. Signed-off-by: bwplotka <[email protected]> --------- Signed-off-by: bwplotka <[email protected]> Signed-off-by: Bartlomiej Plotka <[email protected]> Co-authored-by: George Krajcsovits <[email protected]> * first draft of content negotiation Signed-off-by: Alex Greenbank <[email protected]> * Lint Signed-off-by: Alex Greenbank <[email protected]> * Fix race in test Signed-off-by: Alex Greenbank <[email protected]> * Fix another test race Signed-off-by: Alex Greenbank <[email protected]> * Almost done with lint Signed-off-by: Alex Greenbank <[email protected]> * Fix todos around 405 HEAD handling Signed-off-by: Alex Greenbank <[email protected]> * Changes based on review comments Signed-off-by: Alex Greenbank <[email protected]> * Update storage/remote/client.go Co-authored-by: Bartlomiej Plotka <[email protected]> Signed-off-by: Alex Greenbank <[email protected]> * Latest updates to review comments Signed-off-by: Alex Greenbank <[email protected]> * latest tweaks Signed-off-by: Alex Greenbank <[email protected]> * remote write 2.0 - content negotiation remediation (#13921) * Consolidate renegotiation error into one, fix tests Signed-off-by: Alex Greenbank <[email protected]> * fix metric name and actuall increment counter Signed-off-by: Alex Greenbank <[email protected]> --------- Signed-off-by: Alex Greenbank <[email protected]> * Fixes after main sync. Signed-off-by: bwplotka <[email protected]> * [PRW 2.0] Moved rw2 proto to the full path (both package name and placement) (#13973) undefined * [PRW2.0] Remove benchmark scripts (#13949) See rationales on https://docs.google.com/document/d/1Bpf7mYjrHUhPHkie0qlnZFxzgqf_L32kM8ZOknSdJrU/edit Signed-off-by: bwplotka <[email protected]> * rw20: Update prw commentary after Callum spec review (#14136) * rw20: Update prw commentary after Callum spec review Signed-off-by: Bartlomiej Plotka <[email protected]> * Update types.proto Signed-off-by: Bartlomiej Plotka <[email protected]> --------- Signed-off-by: Bartlomiej Plotka <[email protected]> * [PRW 2.0] Updated spec proto (2.0-rc.1); deterministic v1 interop; to be sympathetic with implementation. (#14330) * [PRW 2.0] Updated spec proto (2.0-rc.1); deterministic v1 interop; to be sympathetic with implementation. Signed-off-by: bwplotka <[email protected]> * update custom marshalling Signed-off-by: bwplotka <[email protected]> * Removed confusing comments. Signed-off-by: bwplotka <[email protected]> --------- Signed-off-by: bwplotka <[email protected]> * [PRW-2.0] (chain1) New Remote Write 2.0 Config options for 2.0-rc.1 spec. (#14335) NOTE: For simple review this change does not touch remote/ packages, only main and configs. Spec: https://prometheus.io/docs/specs/remote_write_spec_2_0 Supersedes https://github.com/prometheus/prometheus/pull/13968 Signed-off-by: bwplotka <[email protected]> * [PRW-2.0] (part 2) Removed automatic negotiation, updates for the latest spec semantics in remote pkg (#14329) * [PRW-2.0] (part2) Moved to latest basic negotiation & spec semantics. Spec: https://github.com/prometheus/docs/pull/2462 Supersedes https://github.com/prometheus/prometheus/pull/13968 Signed-off-by: bwplotka <[email protected]> # Conflicts: # config/config.go # docs/configuration/configuration.md # storage/remote/queue_manager_test.go # storage/remote/write.go # web/api/v1/api.go * Addressed comments. Signed-off-by: bwplotka <[email protected]> --------- Signed-off-by: bwplotka <[email protected]> * lint Signed-off-by: Nicolás Pazos <[email protected]> * storage/remote tests: refactor: extract function newTestQueueManager To reduce repetition. Signed-off-by: Bryan Boreham <[email protected]> Signed-off-by: Nicolás Pazos <[email protected]> * use newTestQueueManager for test Signed-off-by: Nicolás Pazos <[email protected]> * go mod tidy Signed-off-by: Nicolás Pazos <[email protected]> * [PRW 2.0] (part3) moved type specific conversions to prompb and writev2 codecs. Signed-off-by: bwplotka <[email protected]> * Added test for rwProtoMsgFlagParser; fixed TODO comment. Signed-off-by: bwplotka <[email protected]> * Renamed DecodeV2WriteRequestStr to DecodeWriteV2Request (with tests). Signed-off-by: bwplotka <[email protected]> * Addressed comments on remote_storage example, updated it for 2.0 Signed-off-by: bwplotka <[email protected]> * Fixed `--enable-feature=metadata-wal-records` docs and error when using PRW 2.0 without it. Signed-off-by: bwplotka <[email protected]> * Addressed Callum comments on custom*.go Signed-off-by: bwplotka <[email protected]> * Added TODO to genproto. Signed-off-by: bwplotka <[email protected]> * Addressed Callum comments in remote pkg. Signed-off-by: bwplotka <[email protected]> * Added metadata validation to write handler test; fixed ToMetadata. Signed-off-by: bwplotka <[email protected]> * Addressed rest of Callum comments. Signed-off-by: bwplotka <[email protected]> * Fixed writev2.FromMetadataType (was wrongly using prompb). Signed-off-by: bwplotka <[email protected]> * fix a few import whitespaces Signed-off-by: Callum Styan <[email protected]> * add a default case with an error to the example RW receiver Signed-off-by: Callum Styan <[email protected]> * more minor import whitespace chagnes Signed-off-by: Callum Styan <[email protected]> * Apply suggestions from code review Signed-off-by: Bartlomiej Plotka <[email protected]> * Update storage/remote/queue_manager_test.go Signed-off-by: Bartlomiej Plotka <[email protected]> --------- Signed-off-by: Nicolás Pazos <[email protected]> Signed-off-by: Callum Styan <[email protected]> Signed-off-by: bwplotka <[email protected]> Signed-off-by: Paschalis Tsilias <[email protected]> Signed-off-by: Julian Wiedmann <[email protected]> Signed-off-by: Bryan Boreham <[email protected]> Signed-off-by: Erik Sommer <[email protected]> Signed-off-by: Linas Medziunas <[email protected]> Signed-off-by: Arianna Vespri <[email protected]> Signed-off-by: machine424 <[email protected]> Signed-off-by: Daniel Nicholls <[email protected]> Signed-off-by: Daniel Kerbel <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: Jan Fajerski <[email protected]> Signed-off-by: Kevin Mingtarja <[email protected]> Signed-off-by: Paschalis Tsilias <[email protected]> Signed-off-by: Marc Tuduri <[email protected]> Signed-off-by: Paschalis Tsilias <[email protected]> Signed-off-by: Giedrius Statkevičius <[email protected]> Signed-off-by: Augustin Husson <[email protected]> Signed-off-by: Jeanette Tan <[email protected]> Signed-off-by: Bartlomiej Plotka <[email protected]> Signed-off-by: Kumar Kalpadiptya Roy <[email protected]> Signed-off-by: Marco Pracucci <[email protected]> Signed-off-by: tyltr <[email protected]> Signed-off-by: Ted Robertson [email protected] Signed-off-by: Ivan Babrou <[email protected]> Signed-off-by: Arve Knudsen <[email protected]> Signed-off-by: Israel Blancas <[email protected]> Signed-off-by: Ziqi Zhao <[email protected]> Signed-off-by: Björn Rabenstein <[email protected]> Signed-off-by: Goutham <[email protected]> Signed-off-by: Rewanth Tammana <[email protected]> Signed-off-by: Chris Marchbanks <[email protected]> Signed-off-by: Ben Ye <[email protected]> Signed-off-by: Oleg Zaytsev <[email protected]> Signed-off-by: SuperQ <[email protected]> Signed-off-by: Ben Kochie <[email protected]> Signed-off-by: Matthieu MOREL <[email protected]> Signed-off-by: Paulin Todev <[email protected]> Signed-off-by: Filip Petkovski <[email protected]> Signed-off-by: beorn7 <[email protected]> Signed-off-by: Augustin Husson <[email protected]> Signed-off-by: Yury Moladau <[email protected]> Signed-off-by: Danny Kopping <[email protected]> Signed-off-by: Leegin <[email protected]> Signed-off-by: Mikhail Fesenko <[email protected]> Signed-off-by: Jesus Vazquez <[email protected]> Signed-off-by: Alan Protasio <[email protected]> Signed-off-by: Alex Greenbank <[email protected]> Co-authored-by: Nicolás Pazos <[email protected]> Co-authored-by: Callum Styan <[email protected]> Co-authored-by: Nicolás Pazos <[email protected]> Co-authored-by: alexgreenbank <[email protected]> Co-authored-by: Marco Pracucci <[email protected]> Co-authored-by: Paschalis Tsilias <[email protected]> Co-authored-by: Julian Wiedmann <[email protected]> Co-authored-by: Bryan Boreham <[email protected]> Co-authored-by: Erik Sommer <[email protected]> Co-authored-by: Linas Medziunas <[email protected]> Co-authored-by: Arianna Vespri <[email protected]> Co-authored-by: machine424 <[email protected]> Co-authored-by: daniel-resdiary <[email protected]> Co-authored-by: Daniel Kerbel <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot…

* Drop old inmemory samples Co-authored-by: Paschalis Tsilias <[email protected]> Signed-off-by: Paschalis Tsilias <[email protected]> Signed-off-by: Marc Tuduri <[email protected]> * Avoid copying timeseries when the feature is disabled Signed-off-by: Paschalis Tsilias <[email protected]> Signed-off-by: Marc Tuduri <[email protected]> * Run gofmt Signed-off-by: Paschalis Tsilias <[email protected]> Signed-off-by: Marc Tuduri <[email protected]> * Clarify docs Signed-off-by: Marc Tuduri <[email protected]> * Add more logging info Signed-off-by: Marc Tuduri <[email protected]> * Remove loggers Signed-off-by: Marc Tuduri <[email protected]> * optimize function and add tests Signed-off-by: Marc Tuduri <[email protected]> * Simplify filter Signed-off-by: Marc Tuduri <[email protected]> * rename var Signed-off-by: Marc Tuduri <[email protected]> * Update help info from metrics Signed-off-by: Marc Tuduri <[email protected]> * use metrics to keep track of drop elements during buildWriteRequest Signed-off-by: Marc Tuduri <[email protected]> * rename var in tests Signed-off-by: Marc Tuduri <[email protected]> * pass time.Now as parameter Signed-off-by: Marc Tuduri <[email protected]> * Change buildwriterequest during retries Signed-off-by: Marc Tuduri <[email protected]> * Revert "Remove loggers" This reverts commit 54f91df. Signed-off-by: Marc Tuduri <[email protected]> * use log level debug for loggers Signed-off-by: Marc Tuduri <[email protected]> * Fix linter Signed-off-by: Paschalis Tsilias <[email protected]> * Remove noisy debug-level logs; add 'reason' label to drop metrics Signed-off-by: Paschalis Tsilias <[email protected]> * Remove accidentally committed files Signed-off-by: Paschalis Tsilias <[email protected]> * Propagate logger to buildWriteRequest to log dropped data Signed-off-by: Paschalis Tsilias <[email protected]> * Fix docs comment Signed-off-by: Paschalis Tsilias <[email protected]> * Make drop reason more specific Signed-off-by: Paschalis Tsilias <[email protected]> * Remove unnecessary pass of logger Signed-off-by: Paschalis Tsilias <[email protected]> * Use snake_case for reason label Signed-off-by: Paschalis Tsilias <[email protected]> * Fix dropped samples metric Signed-off-by: Paschalis Tsilias <[email protected]> --------- Signed-off-by: Paschalis Tsilias <[email protected]> Signed-off-by: Marc Tuduri <[email protected]> Signed-off-by: Paschalis Tsilias <[email protected]> Co-authored-by: Paschalis Tsilias <[email protected]> Co-authored-by: Paschalis Tsilias <[email protected]>

marctc force-pushed the drop-old-inmemory-samples branch from 3f7225b to b81bf40 Compare October 19, 2023 10:00

marctc marked this pull request as ready for review October 19, 2023 10:00

marctc requested review from bwplotka, csmarchbanks, cstyan and tomwilkie as code owners October 19, 2023 10:00

marctc mentioned this pull request Oct 20, 2023

Improve remote write outage recovery by discarding too-old samples grafana/agent#4651

Closed

cstyan reviewed Oct 20, 2023

View reviewed changes

bboreham reviewed Nov 17, 2023

View reviewed changes

marctc force-pushed the drop-old-inmemory-samples branch from 810bba9 to dad045a Compare November 20, 2023 11:26

cstyan reviewed Nov 20, 2023

View reviewed changes

marctc and others added 12 commits November 21, 2023 16:32

Drop old inmemory samples

443511b

Co-authored-by: Paschalis Tsilias <[email protected]> Signed-off-by: Paschalis Tsilias <[email protected]> Signed-off-by: Marc Tuduri <[email protected]>

Avoid copying timeseries when the feature is disabled

5780421

Signed-off-by: Paschalis Tsilias <[email protected]> Signed-off-by: Marc Tuduri <[email protected]>

Run gofmt

d5460dd

Signed-off-by: Paschalis Tsilias <[email protected]> Signed-off-by: Marc Tuduri <[email protected]>

Clarify docs

39edc4b

Signed-off-by: Marc Tuduri <[email protected]>

Add more logging info

255a6c0

Signed-off-by: Marc Tuduri <[email protected]>

Remove loggers

3da411a

Signed-off-by: Marc Tuduri <[email protected]>

optimize function and add tests

818ac24

Signed-off-by: Marc Tuduri <[email protected]>

Simplify filter

f0c0129

Signed-off-by: Marc Tuduri <[email protected]>

rename var

19f3275

Signed-off-by: Marc Tuduri <[email protected]>

Update help info from metrics

1f7d6f9

Signed-off-by: Marc Tuduri <[email protected]>

use metrics to keep track of drop elements during buildWriteRequest

97cc3ce

Signed-off-by: Marc Tuduri <[email protected]>

rename var in tests

3620ce5

Signed-off-by: Marc Tuduri <[email protected]>

use log level debug for loggers

d3775fc

Signed-off-by: Marc Tuduri <[email protected]>

marctc force-pushed the drop-old-inmemory-samples branch from 4e47676 to d3775fc Compare November 21, 2023 15:33

tpaschalis and others added 5 commits December 14, 2023 11:38

Merge branch 'main' into drop-old-inmemory-samples

a99fc24

Signed-off-by: Paschalis Tsilias <[email protected]>

Fix linter

047a101

Signed-off-by: Paschalis Tsilias <[email protected]>

Remove noisy debug-level logs; add 'reason' label to drop metrics

c0f1ef9

Signed-off-by: Paschalis Tsilias <[email protected]>

Remove accidentally committed files

0b1c94e

Signed-off-by: Paschalis Tsilias <[email protected]>

Propagate logger to buildWriteRequest to log dropped data

0ecfa07

Signed-off-by: Paschalis Tsilias <[email protected]>

cstyan reviewed Dec 15, 2023

View reviewed changes

tpaschalis added 3 commits December 18, 2023 12:02

Fix docs comment

488e1ff

Signed-off-by: Paschalis Tsilias <[email protected]>

Make drop reason more specific

25ec6b6

Signed-off-by: Paschalis Tsilias <[email protected]>

Remove unnecessary pass of logger

7dee9d0

Signed-off-by: Paschalis Tsilias <[email protected]>

tpaschalis reviewed Dec 20, 2023

View reviewed changes

Use snake_case for reason label

722825d

Signed-off-by: Paschalis Tsilias <[email protected]>

tpaschalis mentioned this pull request Jan 4, 2024

Bump client_golang to v1.18.0 #13373

Merged

tpaschalis added 2 commits January 4, 2024 16:04

Merge branch 'main' of ssh://github.com/prometheus/prometheus into dr…

0ae385c

…op-old-inmemory-samples

Fix dropped samples metric

ec3e740

Signed-off-by: Paschalis Tsilias <[email protected]>

marctc mentioned this pull request Jan 5, 2024

Add sample_age_limit to remote_write config to drop old samples grafana/agent#6052

Merged

4 tasks

cstyan approved these changes Jan 5, 2024

View reviewed changes

cstyan merged commit 78c5ce3 into prometheus:main Jan 5, 2024

npazosmendez mentioned this pull request Feb 1, 2024

remote write 2.0: sync with main branch #13510

Merged

bboreham mentioned this pull request Feb 8, 2024

Ability to disable remote write retries #11089

Closed

chenrui333 mentioned this pull request Feb 22, 2024

prometheus 2.50.0 Homebrew/homebrew-core#163774

Merged

dongjiang1989 mentioned this pull request Feb 23, 2024

feat: support sample_age_limit for QueueConfig prometheus-operator/prometheus-operator#6326

Merged

5 tasks

tpaschalis mentioned this pull request Jan 9, 2024

Disable remote writing through agent configuration grafana/alloy#344

Open

FUSAKLA mentioned this pull request May 10, 2024

Fix data corruption in remote write if max_sample_age is applied #14078

Merged

Conversation

marctc commented Oct 18, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cstyan left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

marctc commented Oct 23, 2023

Uh oh!

cstyan commented Oct 23, 2023

Uh oh!

tpaschalis commented Oct 24, 2023

Uh oh!

bboreham commented Oct 30, 2023

Uh oh!

marctc commented Nov 17, 2023

Uh oh!

bboreham left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

marctc commented Nov 20, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cstyan commented Nov 20, 2023

Uh oh!

marctc commented Nov 21, 2023

Uh oh!

cstyan commented Nov 21, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

marctc commented Oct 18, 2023 •

edited

Loading

cstyan commented Nov 21, 2023 •

edited

Loading