discovery/aws: handle rds clusters without instances by Intuinewin · Pull Request #18845 · prometheus/prometheus

Intuinewin · 2026-06-03T07:40:03Z

This PR updates the AWS SD RDS role so that it no longer returns an error when processing an empty cluster (i.e. a cluster without instances).

In my opinion, the cluster should be ignored, no target should be generated, and no error should be reported.

I've also added the corresponding test.

Let me know what you think cc @matt-gp

Release notes for end users (ALL commits must be considered).

Reviewers should verify clarity and quality.

[BUGFIX] Discovery/AWS: Fix failure when processing an AWS RDS cluster without instances #18845

Signed-off-by: Antoine Labarussias <[email protected]>

matt-gp · 2026-06-03T19:02:07Z

Cool, out of curiosity why do you have clusters without instances? As in what is the use case?

Intuinewin · 2026-06-03T19:23:14Z

I was trying to use the RDS role with the filters configuration option (this was before I read the doc stating that filters only work with ec2 role)

So it discovered an empty cluster and I saw this weird error that prevented the discovery of other valid clusters.

Do you plan to make the filters configuration available for other roles (rds, elasticache, msk)? That would be helpful. I could take a look if I find the time.

matt-gp · 2026-06-03T20:27:03Z

Yeh issue is that a lot of the other roles use API's which don't support filtering so you would have to pull all the resources and then filter then after.

I suppose hypothetically if you run a number of clusters (say 10's or 100's) but you can't discover all these clusters because you have a cluster with no instances and this stops discovering the other instances. And filtering would've helped as you could've used that to only find the clusters with instances??

matt-gp · 2026-06-03T20:43:45Z

Actually looking at the RDS Api that is used, it does look like it supports filtering
https://github.com/aws/aws-sdk-go-v2/blob/91eca463daf932474778dc4a984c41ecfcd9dc3c/service/rds/api_op_DescribeDBInstances.go#L71

I think adding filtering support would be good as we should be able to make a generic filter instead of only having the EC2 one, then it won't be an additional field in the config.

prometheus/discovery/aws/aws.go

Line 86 in 14b9a0e

Filters []*EC2Filter `yaml:"filters,omitempty"`

I think changing this to a generic filter (from a dedicated ec2) one would be a good idea as we can use it for any other roles where we want filtering, as long as their api supports it (I don't think elasticache/msk do, so it might just be ec2/rds for the time being).

@sysadmind thoughts?

Intuinewin · 2026-06-03T21:16:54Z

Hmm, you're right, msk and elasticache don't support filters. The RDS API does, but they're very limited compared to EC2.

I think it can be useful to filter by engine in order to use an exporter with multi-target support (for example with mysqld_exporter you may want to discovery only MySQL clusters).
Also maybe AWS will add more filters later 🤞🏼

To get back to this PR, I still think empty clusters should not result in a refresh failure and should just be ignored.

sysadmind · 2026-06-04T01:43:31Z

The EC2Filters type is very generic right now, so I could see us repurposing that for more types. I was originally concerned about changing the type and having a breaking change, but I don't think that's currently a concern.

As far as this PR goes, it's okay with me. I didn't spend that long looking for how these empty clusters get treated downstream, but I do think it's a good idea for us to not fail all of service discovery just because we get one bad result back.

matt-gp · 2026-06-04T08:07:30Z

Cool, so I think maybe we should keep this PR to just not failing on empty clusters. Then we can explore the prospect of allowing the filters to other roles in another PR, providing the API supports it.

To be clear when I say make it generic, I basically mean rename it (so it doesn't specify ec2) and move it to the aws.go.

matt-gp

lgtm

matt-gp · 2026-06-04T19:26:46Z

Raised #18859 for the rds filters

…➔ v3.13.0) (#1360) This PR contains the following updates: | Package | Update | Change | |---|---|---| | [quay.io/prometheus/prometheus](https://github.com/prometheus/prometheus) | minor | `v3.12.0` → `v3.13.0` | --- ### Release Notes <details> <summary>prometheus/prometheus (quay.io/prometheus/prometheus)</summary> ### [`v3.13.0`](https://github.com/prometheus/prometheus/releases/tag/v3.13.0): 3.13.0 / 2026-07-01 [Compare Source](prometheus/prometheus@v3.12.0...v3.13.0) This is a Long Term Support [LTS](https://prometheus.io/docs/introduction/release-cycle/) release. - \[SECURITY] UI: Bump `sanitize-html` to fix a cross-site scripting vulnerability (CVE-2026-44990). [#18697](prometheus/prometheus#18697) - \[CHANGE] UI: Third-party npm dependency licenses are now embedded in the Prometheus binary and served at `/assets/third-party-licenses.txt`, replacing the `npm_licenses.tar.bz2` archive previously shipped in release tarballs and container images. [#18997](prometheus/prometheus#18997) - \[CHANGE] API: Use SHA-256 instead of SHA-1 to generate rule group pagination tokens. [#18927](prometheus/prometheus#18927) - \[CHANGE] HTTP clients: Credentials (Authorization header, basic auth, bearer token, OAuth2, configured headers) are no longer forwarded when following a redirect to a different host; affects scraping, remote read/write, alerting, and service discovery. Via prometheus/common v0.69.0 (CVE-2025-4673 CVE-2023-45289). [#18949](prometheus/prometheus#18949) - \[CHANGE] promtool: Relative file paths in the file passed to `--http.config.file` are now resolved relative to that config file's directory instead of its parent directory. Via prometheus/common v0.69.0. [#18949](prometheus/prometheus#18949) - \[CHANGE] PromQL: Rename the `min()` and `max()` duration-expression functions (experimental feature flag `experimental-duration-expr`) to `min_of()` and `max_of()` to avoid confusion with the `min` and `max` aggregate operators. [#18687](prometheus/prometheus#18687) - \[FEATURE] API: Add experimental search endpoints to search metric names, label names, and label values. [#18573](prometheus/prometheus#18573) - \[FEATURE] Discovery/AWS: Add ability to filter RDS instances. [#18859](prometheus/prometheus#18859) - \[FEATURE] PromQL: Add `min_of(a, b)` and `max_of(a, b)` scalar experimental functions, returning the smaller or larger of two scalar values. [#18687](prometheus/prometheus#18687) - \[FEATURE] PromQL: Add support for smoothed/anchored rate with native histograms. [#18564](prometheus/prometheus#18564) - \[FEATURE] PromQL: Expose per-query `samplesRead` (and `samplesReadPerStep` with `stats=all` and the `promql-per-step-stats` feature flag) in the query stats response, and add the `prometheus_engine_query_samples_read_total` engine counter. `samplesRead` reflects storage I/O distinct from `totalQueryableSamples`, which counts samples loaded into the evaluator (and so over-counts when a sample is reused across multiple range-vector windows). [#18081](prometheus/prometheus#18081) - \[FEATURE] Scrape: Add `__convert_classic_histograms_to_nhcb__` internal label to allow per-target override of `convert_classic_histograms_to_nhcb` scrape configuration via relabeling. [#18840](prometheus/prometheus#18840) - \[FEATURE] TSDB: Add `storage.tsdb.chunk_encoding.floats` configuration field to select float chunk encoding (`xor` or `xor2`) at runtime, independently of the `--enable-feature=xor2-encoding` flag. [#18769](prometheus/prometheus#18769) - \[FEATURE] remote\_write: Add Certificate support for ingesting data into an Azure Monitor Workspace. [#18217](prometheus/prometheus#18217) - \[FEATURE] Scrape: Add `__always_scrape_classic_histograms__` and `__scrape_native_histograms__` internal labels to allow per-target override of the `always_scrape_classic_histograms` and `scrape_native_histograms` scrape configuration via relabeling. [#18929](prometheus/prometheus#18929) - \[ENHANCEMENT] Release: Container images are now also published to the GitHub Container Registry (ghcr.io). [#18791](prometheus/prometheus#18791) - \[ENHANCEMENT] PromQL: Prettify `fill_left(x) fill_right(x)` as `fill(x)` when both fill values are equal. [#18851](prometheus/prometheus#18851) - \[ENHANCEMENT] UI: Improve autocompletion after closing a function bracket. [#18894](prometheus/prometheus#18894) - \[PERF] Labels: Add case-insensitive prefix matching to speed up evaluation of long case-insensitive regular expressions (up to \~2x faster). [#18540](prometheus/prometheus#18540) - \[PERF] TSDB: Reduce per-sample overhead in chunk population, speeding up affected queries by \~12-15% in benchmarks. [#18699](prometheus/prometheus#18699) - \[PERF] TSDB: Eliminate unnecessary heap allocations in the V2 histogram WAL decoder, reducing allocations by up to 50% and memory by up to 10% for deployments using native histograms with created-timestamp storage enabled (`--enable-feature=created-timestamp-zero-ingestion`). [#18813](prometheus/prometheus#18813) - \[BUGFIX] Discovery/AWS: Fix failure when processing an AWS RDS cluster without instances. [#18845](prometheus/prometheus#18845) - \[BUGFIX] Fix race condition in initTime that could cause ErrOutOfBounds. [#18629](prometheus/prometheus#18629) - \[BUGFIX] PromQL: A range query whose `end` was not aligned to `step` caused subqueries inside it to evaluate past the parent's last actual step, inflating `peakSamples` in the query stats and against the `query.max-samples` limit, and wasting storage I/O reading samples that were never used in the result. [#18081](prometheus/prometheus#18081) - \[BUGFIX] PromQL: A range query containing an at-modifier-unsafe function over a range-vector with an `@` modifier (e.g. `predict_linear(metric[60s] @ T, X)`) silently under-counted `totalQueryableSamples` for steps after step 0. [#18081](prometheus/prometheus#18081) - \[BUGFIX] PromQL: Fix `fill_left`/`fill_right` producing missing samples in range queries when using `group_left`/`group_right`. [#18850](prometheus/prometheus#18850) - \[BUGFIX] PromQL: Fix for resets() and changes() in anchored range extenders with histograms. [#18906](prometheus/prometheus#18906) - \[BUGFIX] PromQL: Fix panic on `1[5m] smoothed` and similar expressions when extended range selectors are enabled. [#18764](prometheus/prometheus#18764) - \[BUGFIX] PromQL: Fix panic when a `smoothed` instant vector selector produces no samples for a series. [#18943](prometheus/prometheus#18943) - \[BUGFIX] PromQL: Fix panic when using a parenthesised plain number as an offset (e.g. `foo offset -(5)`). [#18768](prometheus/prometheus#18768) - \[BUGFIX] promtool: Fix panic when parsing exposition text containing empty braces `{}`. Via prometheus/common v0.69.0. [#18949](prometheus/prometheus#18949) - \[BUGFIX] Promtool: Fix `check healthy` and `check ready` when `--url` ends with a trailing slash. [#18854](prometheus/prometheus#18854) - \[BUGFIX] Rules: Close PromQL query after each rule evaluation to ensure resources are released. [#18733](prometheus/prometheus#18733) - \[BUGFIX] Scaleway SD: Resolve VPC/IPAM-only instances that have no legacy `private_ip` or `public_ip` field, but do have private NICs attached. [#18772](prometheus/prometheus#18772) - \[BUGFIX] TSDB: Do not leak head series when an integer histogram append is rejected (e.g. out-of-order). [#18838](prometheus/prometheus#18838) - \[BUGFIX] UI: Escape label values offered by PromQL autocomplete. [#18658](prometheus/prometheus#18658) - \[BUGFIX] TSDB: Fix chunk snapshot encoding for EncXOR2 chunks, preventing corruption on TSDB restart when EncXOR2-encoded series were present. [#18739](prometheus/prometheus#18739) - \[BUGFIX] TSDB: Store a millisecond timestamp (not a WAL segment number) in walExpiries when a series is evicted via CompactStaleHead/CompactSelectedSeries, so the series's label record is correctly retained in the next WAL checkpoint and replays cleanly. [#18847](prometheus/prometheus#18847) - \[BUGFIX] TSDB: Prevent loss of samples at the chunk-range boundary when CompactSelectedSeries (and CompactStaleHead) evict the series — the per-slice compaction loop now runs one more iteration so the boundary timestamp is captured in a block before the in-memory copy is removed. [#18849](prometheus/prometheus#18849) </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about these updates again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate).  Reviewed-on: https://git.erwanleboucher.dev/eleboucher/homelab/pulls/1360

##### [\`v3.13.0\`](https://github.com/prometheus/prometheus/releases/tag/v3.13.0) This is a Long Term Support [LTS](https://prometheus.io/docs/introduction/release-cycle/) release. - \[SECURITY] UI: Bump `sanitize-html` to fix a cross-site scripting vulnerability (CVE-2026-44990). [#18697](prometheus/prometheus#18697) - \[CHANGE] UI: Third-party npm dependency licenses are now embedded in the Prometheus binary and served at `/assets/third-party-licenses.txt`, replacing the `npm_licenses.tar.bz2` archive previously shipped in release tarballs and container images. [#18997](prometheus/prometheus#18997) - \[CHANGE] API: Use SHA-256 instead of SHA-1 to generate rule group pagination tokens. [#18927](prometheus/prometheus#18927) - \[CHANGE] HTTP clients: Credentials (Authorization header, basic auth, bearer token, OAuth2, configured headers) are no longer forwarded when following a redirect to a different host; affects scraping, remote read/write, alerting, and service discovery. Via prometheus/common v0.69.0 (CVE-2025-4673 CVE-2023-45289). [#18949](prometheus/prometheus#18949) - \[CHANGE] promtool: Relative file paths in the file passed to `--http.config.file` are now resolved relative to that config file's directory instead of its parent directory. Via prometheus/common v0.69.0. [#18949](prometheus/prometheus#18949) - \[CHANGE] PromQL: Rename the `min()` and `max()` duration-expression functions (experimental feature flag `experimental-duration-expr`) to `min_of()` and `max_of()` to avoid confusion with the `min` and `max` aggregate operators. [#18687](prometheus/prometheus#18687) - \[FEATURE] API: Add experimental search endpoints to search metric names, label names, and label values. [#18573](prometheus/prometheus#18573) - \[FEATURE] Discovery/AWS: Add ability to filter RDS instances. [#18859](prometheus/prometheus#18859) - \[FEATURE] PromQL: Add `min_of(a, b)` and `max_of(a, b)` scalar experimental functions, returning the smaller or larger of two scalar values. [#18687](prometheus/prometheus#18687) - \[FEATURE] PromQL: Add support for smoothed/anchored rate with native histograms. [#18564](prometheus/prometheus#18564) - \[FEATURE] PromQL: Expose per-query `samplesRead` (and `samplesReadPerStep` with `stats=all` and the `promql-per-step-stats` feature flag) in the query stats response, and add the `prometheus_engine_query_samples_read_total` engine counter. `samplesRead` reflects storage I/O distinct from `totalQueryableSamples`, which counts samples loaded into the evaluator (and so over-counts when a sample is reused across multiple range-vector windows). [#18081](prometheus/prometheus#18081) - \[FEATURE] Scrape: Add `__convert_classic_histograms_to_nhcb__` internal label to allow per-target override of `convert_classic_histograms_to_nhcb` scrape configuration via relabeling. [#18840](prometheus/prometheus#18840) - \[FEATURE] TSDB: Add `storage.tsdb.chunk_encoding.floats` configuration field to select float chunk encoding (`xor` or `xor2`) at runtime, independently of the `--enable-feature=xor2-encoding` flag. [#18769](prometheus/prometheus#18769) - \[FEATURE] remote\_write: Add Certificate support for ingesting data into an Azure Monitor Workspace. [#18217](prometheus/prometheus#18217) - \[FEATURE] Scrape: Add `__always_scrape_classic_histograms__` and `__scrape_native_histograms__` internal labels to allow per-target override of the `always_scrape_classic_histograms` and `scrape_native_histograms` scrape configuration via relabeling. [#18929](prometheus/prometheus#18929) - \[ENHANCEMENT] Release: Container images are now also published to the GitHub Container Registry (ghcr.io). [#18791](prometheus/prometheus#18791) - \[ENHANCEMENT] PromQL: Prettify `fill_left(x) fill_right(x)` as `fill(x)` when both fill values are equal. [#18851](prometheus/prometheus#18851) - \[ENHANCEMENT] UI: Improve autocompletion after closing a function bracket. [#18894](prometheus/prometheus#18894) - \[PERF] Labels: Add case-insensitive prefix matching to speed up evaluation of long case-insensitive regular expressions (up to \~2x faster). [#18540](prometheus/prometheus#18540) - \[PERF] TSDB: Reduce per-sample overhead in chunk population, speeding up affected queries by \~12-15% in benchmarks. [#18699](prometheus/prometheus#18699) - \[PERF] TSDB: Eliminate unnecessary heap allocations in the V2 histogram WAL decoder, reducing allocations by up to 50% and memory by up to 10% for deployments using native histograms with created-timestamp storage enabled (`--enable-feature=created-timestamp-zero-ingestion`). [#18813](prometheus/prometheus#18813) - \[BUGFIX] Discovery/AWS: Fix failure when processing an AWS RDS cluster without instances. [#18845](prometheus/prometheus#18845) - \[BUGFIX] Fix race condition in initTime that could cause ErrOutOfBounds. [#18629](prometheus/prometheus#18629) - \[BUGFIX] PromQL: A range query whose `end` was not aligned to `step` caused subqueries inside it to evaluate past the parent's last actual step, inflating `peakSamples` in the query stats and against the `query.max-samples` limit, and wasting storage I/O reading samples that were never used in the result. [#18081](prometheus/prometheus#18081) - \[BUGFIX] PromQL: A range query containing an at-modifier-unsafe function over a range-vector with an `@` modifier (e.g. `predict_linear(metric[60s] @ T, X)`) silently under-counted `totalQueryableSamples` for steps after step 0. [#18081](prometheus/prometheus#18081) - \[BUGFIX] PromQL: Fix `fill_left`/`fill_right` producing missing samples in range queries when using `group_left`/`group_right`. [#18850](prometheus/prometheus#18850) - \[BUGFIX] PromQL: Fix for resets() and changes() in anchored range extenders with histograms. [#18906](prometheus/prometheus#18906) - \[BUGFIX] PromQL: Fix panic on `1[5m] smoothed` and similar expressions when extended range selectors are enabled. [#18764](prometheus/prometheus#18764) - \[BUGFIX] PromQL: Fix panic when a `smoothed` instant vector selector produces no samples for a series. [#18943](prometheus/prometheus#18943) - \[BUGFIX] PromQL: Fix panic when using a parenthesised plain number as an offset (e.g. `foo offset -(5)`). [#18768](prometheus/prometheus#18768) - \[BUGFIX] promtool: Fix panic when parsing exposition text containing empty braces `{}`. Via prometheus/common v0.69.0. [#18949](prometheus/prometheus#18949) - \[BUGFIX] Promtool: Fix `check healthy` and `check ready` when `--url` ends with a trailing slash. [#18854](prometheus/prometheus#18854) - \[BUGFIX] Rules: Close PromQL query after each rule evaluation to ensure resources are released. [#18733](prometheus/prometheus#18733) - \[BUGFIX] Scaleway SD: Resolve VPC/IPAM-only instances that have no legacy `private_ip` or `public_ip` field, but do have private NICs attached. [#18772](prometheus/prometheus#18772) - \[BUGFIX] TSDB: Do not leak head series when an integer histogram append is rejected (e.g. out-of-order). [#18838](prometheus/prometheus#18838) - \[BUGFIX] UI: Escape label values offered by PromQL autocomplete. [#18658](prometheus/prometheus#18658) - \[BUGFIX] TSDB: Fix chunk snapshot encoding for EncXOR2 chunks, preventing corruption on TSDB restart when EncXOR2-encoded series were present. [#18739](prometheus/prometheus#18739) - \[BUGFIX] TSDB: Store a millisecond timestamp (not a WAL segment number) in walExpiries when a series is evicted via CompactStaleHead/CompactSelectedSeries, so the series's label record is correctly retained in the next WAL checkpoint and replays cleanly. [#18847](prometheus/prometheus#18847) - \[BUGFIX] TSDB: Prevent loss of samples at the chunk-range boundary when CompactSelectedSeries (and CompactStaleHead) evict the series — the per-slice compaction loop now runs one more iteration so the boundary timestamp is captured in a block before the in-memory copy is removed. [#18849](prometheus/prometheus#18849)

discovery/aws: handle rds clusters without instances

24b1b60

Signed-off-by: Antoine Labarussias <[email protected]>

Intuinewin requested review from a team, matt-gp and sysadmind as code owners June 3, 2026 07:40

matt-gp approved these changes Jun 4, 2026

View reviewed changes

sysadmind approved these changes Jun 4, 2026

View reviewed changes

sysadmind merged commit de17e17 into prometheus:main Jun 4, 2026
60 of 62 checks passed

matt-gp mentioned this pull request Jun 4, 2026

discovery/aws: add filter support for RDS role #18859

Merged

github-actions Bot mentioned this pull request Jul 24, 2026

Update prom/prometheus Docker tag to v3.13.1 claytono/infra#2595

Merged

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

discovery/aws: handle rds clusters without instances#18845

discovery/aws: handle rds clusters without instances#18845
sysadmind merged 1 commit into
prometheus:mainfrom
Intuinewin:rds-fix-empty-cluster

Intuinewin commented Jun 3, 2026 •

edited

Loading

Uh oh!

matt-gp commented Jun 3, 2026

Uh oh!

Intuinewin commented Jun 3, 2026

Uh oh!

matt-gp commented Jun 3, 2026

Uh oh!

matt-gp commented Jun 3, 2026 •

edited

Loading

Uh oh!

Intuinewin commented Jun 3, 2026 •

edited

Loading

Uh oh!

sysadmind commented Jun 4, 2026

Uh oh!

matt-gp commented Jun 4, 2026 •

edited

Loading

Uh oh!

matt-gp left a comment

Uh oh!

Uh oh!

matt-gp commented Jun 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

Intuinewin commented Jun 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Release notes for end users (ALL commits must be considered).

Uh oh!

matt-gp commented Jun 3, 2026

Uh oh!

Intuinewin commented Jun 3, 2026

Uh oh!

matt-gp commented Jun 3, 2026

Uh oh!

matt-gp commented Jun 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Intuinewin commented Jun 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sysadmind commented Jun 4, 2026

Uh oh!

matt-gp commented Jun 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

matt-gp left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

matt-gp commented Jun 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Intuinewin commented Jun 3, 2026 •

edited

Loading

matt-gp commented Jun 3, 2026 •

edited

Loading

Intuinewin commented Jun 3, 2026 •

edited

Loading

matt-gp commented Jun 4, 2026 •

edited

Loading