fix(discovery): apply EC2 SD endpoint and guard refreshAZIDs nils by zteeed · Pull Request #18133 · prometheus/prometheus

zteeed · 2026-02-21T06:01:03Z

Which issue(s) does the PR fix:

N/A

Context (regression)

The endpoint option for EC2 SD stopped being applied after the AWS SDK v2 migration. The v1 code passed Endpoint into the session config; the v2 code builds the client with LoadDefaultConfig but never passes the configured endpoint, so the field was effectively dropped.

The configuration still documents endpoint for ec2_sd_config. Custom endpoints are needed for AWS-compatible APIs (e.g. Outscale, LocalStack, or other EC2-compatible endpoints) so users can point EC2 SD at a custom API URL.

This PR restores endpoint handling by calling awsConfig.WithBaseEndpoint(d.cfg.Endpoint) when endpoint is set, and adds nil-safety in refreshAZIDs: the AWS SDK v2 DescribeAvailabilityZones response can have AvailabilityZones == nil or entries with nil ZoneName/ZoneId, which would panic when building the azToAZID map. We now guard against nil and skip invalid entries.

Does this PR introduce a user-facing change?

Yes. The documented endpoint option in ec2_sd_config is applied again when building the EC2 client, and availability zone handling no longer panics on nil responses.

[BUGFIX] discovery: EC2 SD now respects the configured `endpoint` option when building the AWS client (regression from AWS SDK v2 migration). Also adds nil-safety in availability zone handling to avoid panics when DescribeAvailabilityZones returns nil or nil ZoneName/ZoneId.

zteeed · 2026-02-21T06:04:43Z

Example config used to verify the endpoint fix (e.g. for an AWS-compatible API):

scrape_configs:
  - job_name: 'ec2-fcu'
    ec2_sd_configs:
      - endpoint: https://fcu.eu-west-2.outscale.com
        region: eu-west-2
        access_key: A1B2C3D4E5F6G7H8I9J0
        secret_key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

matt-gp · 2026-02-21T13:58:55Z

Thanks for flagging this.

Looking at the other service discovery methods (msk/ecs) they apply the custom endpoint slightly differently, like the below (msk example).

	d.msk = kafka.NewFromConfig(cfg, func(options *kafka.Options) {
		if d.cfg.Endpoint != "" {
			options.BaseEndpoint = &d.cfg.Endpoint
		}
		options.HTTPClient = client
	})

My view is that we should try and keep the code similar for all of the aws discovery methods as eventually we want to deprecate the dedicated ones (EC2/Lightsail) and opt for just for the AWS one, when that happens then we can simplify a lot of things, so the more common the code between these components, then the better.

So I would opt for updating the EC2 and Lightsail discovery methods to follow this pattern.

Signed-off-by: Aurelien Duboc <[email protected]>

zteeed · 2026-02-22T06:05:17Z

Thanks @matt-gp for the review :)
Does this look good to you?

matt-gp

LGTM

krajorama · 2026-02-24T11:12:44Z

Hello from the bug scrub!

This is as somewhat new process, introduced recently. We're merging on account of the code owner approval.

##### [\`v3.11.0\`](https://github.com/prometheus/prometheus/releases/tag/v3.11.0) - \[CHANGE] Hetzner SD: The `__meta_hetzner_datacenter` label is deprecated for the role `robot` but kept for backward compatibility, use the `__meta_hetzner_robot_datacenter` label instead. For the role `hcloud`, the label is deprecated and will stop working after the 1 July 2026. [#17850](prometheus/prometheus#17850) - \[CHANGE] Hetzner SD: The `__meta_hetzner_hcloud_datacenter_location` and `__meta_hetzner_hcloud_datacenter_location_network_zone` labels are deprecated, use the `__meta_hetzner_hcloud_location` and `__meta_hetzner_hcloud_location_network_zone` labels instead. [#17850](prometheus/prometheus#17850) - \[CHANGE] Promtool: Redirect debug output to stderr to avoid interfering with stdout-based tool output. [#18346](prometheus/prometheus#18346) - \[FEATURE] AWS SD: Add Elasticache Role. [#18099](prometheus/prometheus#18099) - \[FEATURE] AWS SD: Add RDS Role. [#18206](prometheus/prometheus#18206) - \[FEATURE] Azure SD: Add support for Azure Workload Identity authentication method. [#17207](prometheus/prometheus#17207) - \[FEATURE] Discovery: Introduce `prometheus_sd_last_update_timestamp_seconds` metric to track the last time a service discovery update was sent to consumers. [#18194](prometheus/prometheus#18194) - \[FEATURE] Kubernetes SD: Add support for node role selectors for pod roles. [#18006](prometheus/prometheus#18006) - \[FEATURE] Kubernetes SD: Introduce pod-based labels for deployment, cronjob, and job controller names: `__meta_kubernetes_pod_deployment_name`, `__meta_kubernetes_pod_cronjob_name` and `__meta_kubernetes_pod_job_name`, respectively. [#17774](prometheus/prometheus#17774) - \[FEATURE] PromQL: Add `</` and `>/` operators for trimming observations from native histograms. [#17904](prometheus/prometheus#17904) - \[FEATURE] PromQL: Add experimental `histogram_quantiles` variadic function for computing multiple quantiles at once. [#17285](prometheus/prometheus#17285) - \[FEATURE] TSDB: Add `storage.tsdb.retention.percentage` configuration to configure the maximum percent of disk usable for TSDB storage. [#18080](prometheus/prometheus#18080) - \[FEATURE] TSDB: Add an experimental `fast-startup` feature flag that writes a `series_state.json` file to the WAL directory to track active series state across restarts. [#18303](prometheus/prometheus#18303) - \[FEATURE] TSDB: Add an experimental `st-storage` feature flag. When enabled, Prometheus stores ingested start timestamps (ST, previously called Created Timestamp) from scrape or OTLP in the TSDB and Agent WAL, and exposes them via Remote Write 2. [#18062](prometheus/prometheus#18062) - \[FEATURE] TSDB: Add an experimental `xor2-encoding` feature flag for the new TSDB block float sample chunk encoding that is optimized for scraped data and allows encoding start timestamps. [#18062](prometheus/prometheus#18062) - \[ENHANCEMENT] HTTP client: Add AWS `external_id` support for sigv4. [#17916](prometheus/prometheus#17916) - \[ENHANCEMENT] Kubernetes SD: Deduplicate deprecation warning logs from the Kubernetes API to reduce noise. [#17829](prometheus/prometheus#17829) - \[ENHANCEMENT] TSDB: Remove old temporary checkpoints when creating a Checkpoint. [#17598](prometheus/prometheus#17598) - \[ENHANCEMENT] UI: Add autocomplete support for experimental `first_over_time` and `ts_of_first_over_time` PromQL functions. [#18318](prometheus/prometheus#18318) - \[ENHANCEMENT] Vultr SD: Upgrade govultr library from v2 to v3 for continued security patches and maintenance. [#18347](prometheus/prometheus#18347) - \[PERF] PromQL: Improve performance and reduce heap allocations in joins (VectorBinop)/And/Or/Unless. [#17159](prometheus/prometheus#17159) - \[PERF] PromQL: Partially address performance regression in native histogram aggregations due to using `KahanAdd`. [#18252](prometheus/prometheus#18252) - \[PERF] Remote write: Optimize WAL watching used for RW sending to reuse internal buffers. [#18250](prometheus/prometheus#18250) - \[PERF] TSDB: Optimize LabelValues intersection performance for matchers. [#18069](prometheus/prometheus#18069) - \[PERF] UI: Skip restacking on hover in stacked series charts. [#18230](prometheus/prometheus#18230) - \[BUGFIX] AWS SD: Fix EC2 SD ignoring the configured `endpoint` option, a regression from the AWS SDK v2 migration. [#18133](prometheus/prometheus#18133) - \[BUGFIX] AWS SD: Fix panic in EC2 SD when DescribeAvailabilityZones returns nil ZoneName or ZoneId. [#18133](prometheus/prometheus#18133) - \[BUGFIX] Agent: Fix memory leak caused by duplicate SeriesRefs being loaded as active series. [#17538](prometheus/prometheus#17538) - \[BUGFIX] Alerting: Fix alert state incorrectly resetting to pending when the FOR period is increased in the config file. [#18244](prometheus/prometheus#18244) - \[BUGFIX] Azure SD: Fix system-assigned managed identity not working when `client_id` is empty. [#18323](prometheus/prometheus#18323) - \[BUGFIX] Consul SD: Fix filter parameter not being applied to health service endpoint, causing Node and Node.Meta filters to be ignored. [#17349](prometheus/prometheus#17349) - \[BUGFIX] Kubernetes SD: Fix duplicate targets generated by `*DualStack` EndpointSlices policies. [#18192](prometheus/prometheus#18192) - \[BUGFIX] OTLP: Fix ErrTooOldSample being returned as HTTP 500 instead of 400 in PRW v2 histogram write paths, preventing infinite client retry loops. [#18084](prometheus/prometheus#18084) - \[BUGFIX] OTLP: Fix exemplars getting mixed between incorrect parts of a histogram. [#18056](prometheus/prometheus#18056) - \[BUGFIX] PromQL: Do not skip histogram buckets in queries where histogram trimming is used. [#18263](prometheus/prometheus#18263) - \[BUGFIX] Remote write: Fix `prometheus_remote_storage_sent_batch_duration_seconds` measuring before the request was sent. [#18214](prometheus/prometheus#18214) - \[BUGFIX] Rules: Fix alert state restoration when rule labels contain Go template expressions. [#18375](prometheus/prometheus#18375) - \[BUGFIX] Scrape: Fix panic when parsing bare label names without an equal sign in brace-only metric notation. [#18229](prometheus/prometheus#18229) - \[BUGFIX] TSDB: Fail early when `use-uncached-io` feature flag is set on unsupported environments. [#18219](prometheus/prometheus#18219) - \[BUGFIX] TSDB: Fall back to CLI flag values when retention is removed from config file. [#18200](prometheus/prometheus#18200) - \[BUGFIX] TSDB: Fix memory leaks in buffer pools by clearing reference fields before returning buffers to pools. [#17895](prometheus/prometheus#17895) - \[BUGFIX] TSDB: Fix missing mmap of histogram chunks during WAL replay. [#18306](prometheus/prometheus#18306) - \[BUGFIX] TSDB: Fix storage.tsdb.retention.time unit mismatch in file causing retention to be 1e6 times longer than configured. [#18200](prometheus/prometheus#18200) - \[BUGFIX] Tracing: Fix missing traceID in query log when tracing is enabled, previously only spanID was emitted. [#18189](prometheus/prometheus#18189) - \[BUGFIX] UI: Fix tooltip Y-offset drift when using multiple graph panels. [#18228](prometheus/prometheus#18228) - \[BUGFIX] UI: Update retention display in runtime info when config is reloaded. [#18200](prometheus/prometheus#18200)

zteeed requested review from a team, matt-gp and sysadmind as code owners February 21, 2026 06:01

zteeed force-pushed the fix/ec2-sd-custom-endpoint branch from 25f0f64 to 36f4348 Compare February 22, 2026 05:55

fix(discovery): apply EC2 SD endpoint and guard refreshAZIDs nils

1f6c5f1

Signed-off-by: Aurelien Duboc <[email protected]>

zteeed force-pushed the fix/ec2-sd-custom-endpoint branch from 36f4348 to 1f6c5f1 Compare February 22, 2026 05:56

matt-gp approved these changes Feb 22, 2026

View reviewed changes

krajorama merged commit 1a5da4f into prometheus:main Feb 24, 2026
33 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix(discovery): apply EC2 SD endpoint and guard refreshAZIDs nils#18133

fix(discovery): apply EC2 SD endpoint and guard refreshAZIDs nils#18133
krajorama merged 1 commit into
prometheus:mainfrom
zteeed:fix/ec2-sd-custom-endpoint

zteeed commented Feb 21, 2026

Uh oh!

zteeed commented Feb 21, 2026 •

edited

Loading

Uh oh!

matt-gp commented Feb 21, 2026 •

edited

Loading

Uh oh!

zteeed commented Feb 22, 2026 •

edited

Loading

Uh oh!

matt-gp left a comment

Uh oh!

krajorama commented Feb 24, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

zteeed commented Feb 21, 2026

Which issue(s) does the PR fix:

Context (regression)

Does this PR introduce a user-facing change?

Uh oh!

zteeed commented Feb 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

matt-gp commented Feb 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zteeed commented Feb 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

matt-gp left a comment

Choose a reason for hiding this comment

Uh oh!

krajorama commented Feb 24, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

zteeed commented Feb 21, 2026 •

edited

Loading

matt-gp commented Feb 21, 2026 •

edited

Loading

zteeed commented Feb 22, 2026 •

edited

Loading