For tests only, we had various ways of opening DB. Reduced to one
instead of:

* Open
* newTestDB
* newTestDBOpts
* openTestDB

This so #17629 is smaller
and bit easier. Also for test maintainability and consistency.

Signed-off-by: bwplotka <[email protected]>

For tests only, we had various ways of opening DB. Reduced to one
instead of:

* Open
* newTestDB
* newTestDBOpts
* openTestDB

This so #17629 is smaller
and bit easier. Also for test maintainability and consistency.

Signed-off-by: bwplotka <[email protected]>

For tests only, we had various ways of opening DB. Reduced to one
instead of:

* Open
* newTestDB
* newTestDBOpts
* openTestDB

This so #17629 is smaller
and bit easier. Also for test maintainability and consistency.

Signed-off-by: bwplotka <[email protected]>

For tests only, we had various ways of opening DB. Reduced to one
instead of:

* Open
* newTestDB
* newTestDBOpts
* openTestDB

This so #17629 is smaller
and bit easier. Also for test maintainability and consistency.

Signed-off-by: bwplotka <[email protected]>

For tests only, we had various ways of opening DB. Reduced to one
instead of:

* Open
* newTestDB
* newTestDBOpts
* openTestDB

This so #17629 is smaller
and bit easier. Also for test maintainability and consistency.

Signed-off-by: bwplotka <[email protected]>

* promql: fix histogram_fraction issue when lower falls within the first bucket (#17424)

Signed-off-by: Mohammad Alavi <[email protected]>

* prepare release 3.8.0-rc.0

Signed-off-by: Jan Fajerski <[email protected]>

* test: skip TestRemoteWrite_ReshardingWithoutDeadlock temporarily as flaky (#17534) (#17543)

(cherry picked from commit 35c3232)

Signed-off-by: machine424 <[email protected]>
Signed-off-by: Jan Fajerski <[email protected]>
Co-authored-by: Ayoub Mrini <[email protected]>

* chore(deps): bump prometheus/promci from 0.4.7 to 0.5.0

Signed-off-by: Jan Fajerski <[email protected]>

* chore(deps): bump prometheus/promci from 0.5.0 to 0.5.1

Signed-off-by: Jan Fajerski <[email protected]>

* chore(deps): bump prometheus/promci from 0.5.1 to 0.5.2

Signed-off-by: Jan Fajerski <[email protected]>

* chore(deps): bump prometheus/promci from 0.5.2 to 0.5.3

Signed-off-by: Jan Fajerski <[email protected]>

* prw2: Move Remote Write 2.0 CT to be per Sample; Rename to ST (start timestamp) (#17411)

Relates to
prometheus/prometheus#16944 (comment)

Signed-off-by: bwplotka <[email protected]>
(cherry picked from commit cefefc6)

* chore: prepare 3.8.0-rc.1 entry

Signed-off-by: bwplotka <[email protected]>

* [chore]: bump common dep to support RFC7523 3.1

Signed-off-by: Jorge Turrado <[email protected]>

* Update Prometheus Agent doc (#17591)

* Add a nav title to fix docs website generator.
* Make it more clear that "Prometheus Agent" is a mode, not a seaparate
  service.
* Add to index.
* Cleanup some wording.
* Add a downsides section.

Signed-off-by: SuperQ <[email protected]>
(cherry picked from commit d0d2699)

* chore(deps): bump github.com/prometheus/common from 0.67.3 to 0.67.4 (#17594)

Signed-off-by: Jan Fajerski <[email protected]>

* prepare release v3.8.0-rc.1

Signed-off-by: Jan Fajerski <[email protected]>

* prepare release v3.8.0

Signed-off-by: Jan Fajerski <[email protected]>

* chore: Fix function name typo in createBatchSpan comment

Signed-off-by: zjumathcode <[email protected]>

* feat: Add flag that blocks lvl 1 compactions until upload is confirmed in an external JSON file (#17435)

* Delay compactions until Thanos uploads all blocks

Using Thanos sidecar with Prometheus requires us to disable TSDB compactions on Prometheus side by setting --storage.tsdb.min-block-duration and --storage.tsdb.max-block-duration to the same value. See https://thanos.io/tip/components/sidecar.md. The main problem this avoids is that Prometheus might compact given block before Thanos uploads it, creating a gap in Thanos metrics. Thanos does not upload compacted blocks because that would upload the same sample multiple times. You can tell Thanos to upload compacted blocks but that is aimed at one time migrations. This patch creates a bridge between Thanos and Prometheus by allowing Prometheus to read the shipper file Thanos creates, where it tracks which blocks were already uploaded, and using that data delays compaction of blocks until they are marked as uploaded by Thanos. Thanks to this both services can coordinate with each other (in a way) and we can stop disabling compaction on Prometheus side when Thanos uploads are enabled.

The reason to have this is that disabling compactions have very dramatic performance cost. Since most time series exist for longer than a single block duration (2h by default) large chunks of block index will reference the same series, so 10 * 2h blocks will each have an index that is usually fairly big and is almost the same for all 10 blocks. Compaction de-duplicates the index so merging 10 blocks together would leave us with a single index that is around the same size as each of these 10 2h blocks would have (plus some extra for series that only exists in some blocks, but not all). Every range query that iterates over all 10 blocks would then have to read each index and so we're doing 10x more work then if we had a single compacted block.

Signed-off-by: Lukasz Mierzwa <[email protected]>

* Rename structs and functions to make this more generic

Signed-off-by: Lukasz Mierzwa <[email protected]>

* Address review comments

Signed-off-by: Lukasz Mierzwa <[email protected]>

* Cache UploadMeta for 1 minute

Signed-off-by: Lukasz Mierzwa <[email protected]>

---------

Signed-off-by: Lukasz Mierzwa <[email protected]>

* RW2: Allow custom scope in azuread (#17483)

Signed-off-by: Ben Edmunds <[email protected]>

* docs: Describe how time() is set to start at 0 in unit tests

The return value of functions relating to the current time, e.g. time(),
is set by promtool to start at timestamp 0 at the start of a test's
evaluation.

This has the very nice consequence that tests can run reliably without
depending on when they are run.

It does, however, mean that tests will give out results that can be
unexpected by users.

If this behaviour is documented, then users will be empowered to write
tests for their rules that use time-dependent functions.

(Closes: prometheus/docs#1464)

Signed-off-by: Gabriel Filion <[email protected]>

* refactor(tsdb): use one test newTestDB constructor (#17638)

For tests only, we had various ways of opening DB. Reduced to one
instead of:

* Open
* newTestDB
* newTestDBOpts
* openTestDB

This so prometheus/prometheus#17629 is smaller
and bit easier. Also for test maintainability and consistency.

Signed-off-by: bwplotka <[email protected]>

* Add start_timestamp field for unit tests.

This commit adds support for configuring a custom start timestamp
for Prometheus unit tests, allowing tests to use realistic timestamps
instead of starting at Unix epoch 0.

Signed-off-by: Julien Pivotto <[email protected]>

* Fix serialization for empty `ignoring()` in combination with `group_x()`

Currently both the backend and frontend printers/formatters/serializers
incorrectly transform the following expression:

```
up * ignoring() group_left(__name__) node_boot_time_seconds
```

...into:

```
up * node_boot_time_seconds
```

...which yields a different result (including the metric name in the result
vs. no metric name).

We need to keep empty `ignoring()` modifiers if there is a grouping modifier
present.

Signed-off-by: Julius Volz <[email protected]>

* Simplify StartTime assignment in unit test setup.

Remove redundant IsZero check since promqltest.LazyLoader already
handles zero StartTime by defaulting to Unix epoch.

Signed-off-by: Julien Pivotto <[email protected]>

* Update golangci-lint and add modernize check (#17640)

* add modernize check

Signed-off-by: dongjiang1989 <[email protected]>

* fix golangci lint

Signed-off-by: dongjiang1989 <[email protected]>

---------

Signed-off-by: dongjiang1989 <[email protected]>

* fix lint

---------

Signed-off-by: Mohammad Alavi <[email protected]>
Signed-off-by: Jan Fajerski <[email protected]>
Signed-off-by: machine424 <[email protected]>
Signed-off-by: bwplotka <[email protected]>
Signed-off-by: Jorge Turrado <[email protected]>
Signed-off-by: SuperQ <[email protected]>
Signed-off-by: zjumathcode <[email protected]>
Signed-off-by: Lukasz Mierzwa <[email protected]>
Signed-off-by: Ben Edmunds <[email protected]>
Signed-off-by: Gabriel Filion <[email protected]>
Signed-off-by: Julien Pivotto <[email protected]>
Signed-off-by: Julius Volz <[email protected]>
Signed-off-by: dongjiang1989 <[email protected]>
Co-authored-by: Mohammad Alavi <[email protected]>
Co-authored-by: Jan Fajerski <[email protected]>
Co-authored-by: Jan Fajerski <[email protected]>
Co-authored-by: Ayoub Mrini <[email protected]>
Co-authored-by: Bartlomiej Plotka <[email protected]>
Co-authored-by: Jorge Turrado <[email protected]>
Co-authored-by: Ben Kochie <[email protected]>
Co-authored-by: zjumathcode <[email protected]>
Co-authored-by: Łukasz Mierzwa <[email protected]>
Co-authored-by: Ben Edmunds <[email protected]>
Co-authored-by: Julien <[email protected]>
Co-authored-by: Gabriel Filion <[email protected]>
Co-authored-by: Julius Volz <[email protected]>
Co-authored-by: dongjiang <[email protected]>
Co-authored-by: Jeanette Tan <[email protected]>