Deprecate Hetzner Cloud server datacenter labels / add new labels by jooola · Pull Request #17850 · prometheus/prometheus

jooola · 2026-01-13T14:21:58Z

[hcloud.Server.Datacenter] is deprecated and will be removed after 1 July 2026. Use [hcloud.Server.Location] instead.

See https://docs.hetzner.cloud/changelog#2025-12-16-phasing-out-datacenters

Changes to Hetzner meta labels:

__meta_hetzner_datacenter
- is deprecated for the role robot but kept for backward compatibility. Using __meta_hetzner_robot_datacenter is preferred.
- is deprecated for the role hcloud and will stop working after the 1 July 2026.
__meta_hetzner_hcloud_datacenter_location label
- is deprecated but kept for backward compatibility, the same data is available in the hcloud.Server.Location struct.
- using __meta_hetzner_hcloud_location is preferred.
__meta_hetzner_hcloud_datacenter_location_network_zone
- is deprecated but kept for backward compatibility, the same data is available in the hcloud.Server.Location struct.
- using __meta_hetzner_hcloud_location_network_zone is preferred.
__meta_hetzner_hcloud_location
- replacement label for __meta_hetzner_hcloud_datacenter_location
__meta_hetzner_hcloud_location_network_zone
- replacement label for __meta_hetzner_hcloud_datacenter_location_network_zone
__meta_hetzner_robot_datacenter
- replacement label for __meta_hetzner_datacenter with the role robot.

Which issue(s) does the PR fix:

Related to #17762
Fixes #17803

Does this PR introduce a user-facing change?

[CHANGE] Hetzner SD: The `__meta_hetzner_datacenter` label is deprecated for the role `robot` but kept for backward compatibility, use the `__meta_hetzner_robot_datacenter` label instead. For the role `hcloud`, the label is deprecated and will stop working after the 1 July 2026.

[CHANGE] Hetzner SD: The `__meta_hetzner_hcloud_datacenter_location` and `__meta_hetzner_hcloud_datacenter_location_network_zone` labels are deprecated, use the `__meta_hetzner_hcloud_location` and `__meta_hetzner_hcloud_location_network_zone` labels instead.

jooola · 2026-01-13T14:27:49Z

I'll fill the release note once we all agreed on the changes.

bboreham · 2026-01-20T11:35:18Z

Hello from the bug-scrub!

Normally we don't look at PRs in draft mode, but you asked for agreement on the changes.
Who is going to agree them? Since your profile is associated with Hetzner, would you like to be assigned as an owner of this area?

jooola · 2026-01-21T09:32:34Z

Hey @bboreham

This PR comes from the discussion in #17803, in which I was hoping to get some input from @roidelapluie, when they have some time.

I think @roidelapluie is assigned to this ticket, so I would like wait for them to say if they want to take it over or if I should push this PR over the finish line.

Since your profile is associated with Hetzner, would you like to be assigned as an owner of this area?

Maybe this was already done last time I contributed to Prometheus #16512 (comment) ?

My answer still stands, yes, you can assign @jooola and @apricote.

roidelapluie · 2026-01-26T12:44:33Z

LGTM

jooola · 2026-02-09T10:16:36Z

I did my best to describe the important changes in the release notes, let me know if this is too much or not enough.

The release note in this PR now conflict with the release note in #17762.

Could we remove the release note from that older PR?

I rebased this branch on top of main and fixed the tests.

apricote

Looking good from "Hetzner POV". Thanks @jooola

beorn7 · 2026-02-10T14:19:56Z

I'm not really qualified to review this.

@roidelapluie: You LGTM'd this two weeks ago. Maybe you could have a quick look that it still looks fine after the recent pushes?

jooola · 2026-02-25T09:35:24Z

Rebased on top of main.

[hcloud.Server.Datacenter] is deprecated and will be removed after 1 July 2026. Use [hcloud.Server.Location] instead. See https://docs.hetzner.cloud/changelog#2025-12-16-phasing-out-datacenters Changes to Hetzner meta labels: - `__meta_hetzner_datacenter` - is deprecated for the role `robot` but kept for backward compatibility. Using `__meta_hetzner_robot_datacenter` is preferred. - is deprecated for the role `hcloud` and will stop working after the 1 July 2026. - `__meta_hetzner_hcloud_datacenter_location` label - is deprecated but kept for backward compatibility, the same data is available in the [`hcloud.Server.Location`](https://pkg.go.dev/github.com/hetznercloud/hcloud-go/v2/hcloud#Server) struct. - using `__meta_hetzner_hcloud_location` is preferred. - `__meta_hetzner_hcloud_datacenter_location_network_zone` - is deprecated but kept for backward compatibility, the same data is available in the [`hcloud.Server.Location`](https://pkg.go.dev/github.com/hetznercloud/hcloud-go/v2/hcloud#Server) struct. - using `__meta_hetzner_hcloud_location_network_zone` is preferred. - `__meta_hetzner_hcloud_location` - replacement label for `__meta_hetzner_hcloud_datacenter_location` - `__meta_hetzner_hcloud_location_network_zone` - replacement label for `__meta_hetzner_hcloud_datacenter_location_network_zone` - `__meta_hetzner_robot_datacenter` - replacement label for `__meta_hetzner_datacenter` with the role `robot`. Signed-off-by: Jonas Lammler <[email protected]>

jooola · 2026-03-11T11:36:00Z

Rebased on top of main.

apricote

Still looking good after rebase :)

krajorama

I took a quick glance to check if I see malicious / obviously bad style code.
Looks fine. I think given that we bestowed approval rights to SD owners I'm merging the PR.

jooola · 2026-03-19T11:04:48Z

@krajorama Thanks for merging this!

##### [\`v3.11.0\`](https://github.com/prometheus/prometheus/releases/tag/v3.11.0) - \[CHANGE] Hetzner SD: The `__meta_hetzner_datacenter` label is deprecated for the role `robot` but kept for backward compatibility, use the `__meta_hetzner_robot_datacenter` label instead. For the role `hcloud`, the label is deprecated and will stop working after the 1 July 2026. [#17850](prometheus/prometheus#17850) - \[CHANGE] Hetzner SD: The `__meta_hetzner_hcloud_datacenter_location` and `__meta_hetzner_hcloud_datacenter_location_network_zone` labels are deprecated, use the `__meta_hetzner_hcloud_location` and `__meta_hetzner_hcloud_location_network_zone` labels instead. [#17850](prometheus/prometheus#17850) - \[CHANGE] Promtool: Redirect debug output to stderr to avoid interfering with stdout-based tool output. [#18346](prometheus/prometheus#18346) - \[FEATURE] AWS SD: Add Elasticache Role. [#18099](prometheus/prometheus#18099) - \[FEATURE] AWS SD: Add RDS Role. [#18206](prometheus/prometheus#18206) - \[FEATURE] Azure SD: Add support for Azure Workload Identity authentication method. [#17207](prometheus/prometheus#17207) - \[FEATURE] Discovery: Introduce `prometheus_sd_last_update_timestamp_seconds` metric to track the last time a service discovery update was sent to consumers. [#18194](prometheus/prometheus#18194) - \[FEATURE] Kubernetes SD: Add support for node role selectors for pod roles. [#18006](prometheus/prometheus#18006) - \[FEATURE] Kubernetes SD: Introduce pod-based labels for deployment, cronjob, and job controller names: `__meta_kubernetes_pod_deployment_name`, `__meta_kubernetes_pod_cronjob_name` and `__meta_kubernetes_pod_job_name`, respectively. [#17774](prometheus/prometheus#17774) - \[FEATURE] PromQL: Add `</` and `>/` operators for trimming observations from native histograms. [#17904](prometheus/prometheus#17904) - \[FEATURE] PromQL: Add experimental `histogram_quantiles` variadic function for computing multiple quantiles at once. [#17285](prometheus/prometheus#17285) - \[FEATURE] TSDB: Add `storage.tsdb.retention.percentage` configuration to configure the maximum percent of disk usable for TSDB storage. [#18080](prometheus/prometheus#18080) - \[FEATURE] TSDB: Add an experimental `fast-startup` feature flag that writes a `series_state.json` file to the WAL directory to track active series state across restarts. [#18303](prometheus/prometheus#18303) - \[FEATURE] TSDB: Add an experimental `st-storage` feature flag. When enabled, Prometheus stores ingested start timestamps (ST, previously called Created Timestamp) from scrape or OTLP in the TSDB and Agent WAL, and exposes them via Remote Write 2. [#18062](prometheus/prometheus#18062) - \[FEATURE] TSDB: Add an experimental `xor2-encoding` feature flag for the new TSDB block float sample chunk encoding that is optimized for scraped data and allows encoding start timestamps. [#18062](prometheus/prometheus#18062) - \[ENHANCEMENT] HTTP client: Add AWS `external_id` support for sigv4. [#17916](prometheus/prometheus#17916) - \[ENHANCEMENT] Kubernetes SD: Deduplicate deprecation warning logs from the Kubernetes API to reduce noise. [#17829](prometheus/prometheus#17829) - \[ENHANCEMENT] TSDB: Remove old temporary checkpoints when creating a Checkpoint. [#17598](prometheus/prometheus#17598) - \[ENHANCEMENT] UI: Add autocomplete support for experimental `first_over_time` and `ts_of_first_over_time` PromQL functions. [#18318](prometheus/prometheus#18318) - \[ENHANCEMENT] Vultr SD: Upgrade govultr library from v2 to v3 for continued security patches and maintenance. [#18347](prometheus/prometheus#18347) - \[PERF] PromQL: Improve performance and reduce heap allocations in joins (VectorBinop)/And/Or/Unless. [#17159](prometheus/prometheus#17159) - \[PERF] PromQL: Partially address performance regression in native histogram aggregations due to using `KahanAdd`. [#18252](prometheus/prometheus#18252) - \[PERF] Remote write: Optimize WAL watching used for RW sending to reuse internal buffers. [#18250](prometheus/prometheus#18250) - \[PERF] TSDB: Optimize LabelValues intersection performance for matchers. [#18069](prometheus/prometheus#18069) - \[PERF] UI: Skip restacking on hover in stacked series charts. [#18230](prometheus/prometheus#18230) - \[BUGFIX] AWS SD: Fix EC2 SD ignoring the configured `endpoint` option, a regression from the AWS SDK v2 migration. [#18133](prometheus/prometheus#18133) - \[BUGFIX] AWS SD: Fix panic in EC2 SD when DescribeAvailabilityZones returns nil ZoneName or ZoneId. [#18133](prometheus/prometheus#18133) - \[BUGFIX] Agent: Fix memory leak caused by duplicate SeriesRefs being loaded as active series. [#17538](prometheus/prometheus#17538) - \[BUGFIX] Alerting: Fix alert state incorrectly resetting to pending when the FOR period is increased in the config file. [#18244](prometheus/prometheus#18244) - \[BUGFIX] Azure SD: Fix system-assigned managed identity not working when `client_id` is empty. [#18323](prometheus/prometheus#18323) - \[BUGFIX] Consul SD: Fix filter parameter not being applied to health service endpoint, causing Node and Node.Meta filters to be ignored. [#17349](prometheus/prometheus#17349) - \[BUGFIX] Kubernetes SD: Fix duplicate targets generated by `*DualStack` EndpointSlices policies. [#18192](prometheus/prometheus#18192) - \[BUGFIX] OTLP: Fix ErrTooOldSample being returned as HTTP 500 instead of 400 in PRW v2 histogram write paths, preventing infinite client retry loops. [#18084](prometheus/prometheus#18084) - \[BUGFIX] OTLP: Fix exemplars getting mixed between incorrect parts of a histogram. [#18056](prometheus/prometheus#18056) - \[BUGFIX] PromQL: Do not skip histogram buckets in queries where histogram trimming is used. [#18263](prometheus/prometheus#18263) - \[BUGFIX] Remote write: Fix `prometheus_remote_storage_sent_batch_duration_seconds` measuring before the request was sent. [#18214](prometheus/prometheus#18214) - \[BUGFIX] Rules: Fix alert state restoration when rule labels contain Go template expressions. [#18375](prometheus/prometheus#18375) - \[BUGFIX] Scrape: Fix panic when parsing bare label names without an equal sign in brace-only metric notation. [#18229](prometheus/prometheus#18229) - \[BUGFIX] TSDB: Fail early when `use-uncached-io` feature flag is set on unsupported environments. [#18219](prometheus/prometheus#18219) - \[BUGFIX] TSDB: Fall back to CLI flag values when retention is removed from config file. [#18200](prometheus/prometheus#18200) - \[BUGFIX] TSDB: Fix memory leaks in buffer pools by clearing reference fields before returning buffers to pools. [#17895](prometheus/prometheus#17895) - \[BUGFIX] TSDB: Fix missing mmap of histogram chunks during WAL replay. [#18306](prometheus/prometheus#18306) - \[BUGFIX] TSDB: Fix storage.tsdb.retention.time unit mismatch in file causing retention to be 1e6 times longer than configured. [#18200](prometheus/prometheus#18200) - \[BUGFIX] Tracing: Fix missing traceID in query log when tracing is enabled, previously only spanID was emitted. [#18189](prometheus/prometheus#18189) - \[BUGFIX] UI: Fix tooltip Y-offset drift when using multiple graph panels. [#18228](prometheus/prometheus#18228) - \[BUGFIX] UI: Update retention display in runtime info when config is reloaded. [#18200](prometheus/prometheus#18200)

…ocation labels Hetzner Cloud deprecated the `datacenter` field in their API on 2025-12-16 and will remove it after 2026-07-01. Add a top-level `HCloudLocation` struct and `Location` field on `HCloudServer` that maps to the new top-level `location` field in the API response. Emit two new canonical labels sourced from this field: - `__meta_hetzner_hcloud_location` - `__meta_hetzner_hcloud_location_network_zone` Keep the deprecated `__meta_hetzner_hcloud_datacenter_location` and `__meta_hetzner_hcloud_datacenter_location_network_zone` labels for backward compatibility, now sourced from the new `location` field. The `HCloudDatacenterLocation` struct and the nested `location` field inside `HCloudDatacenter` are dropped: the live Hetzner Cloud API already returns the top-level `location` field, so there is no need to retain the old parsing path. The test fixture no longer includes `datacenter.location` to reflect that the code no longer reads it. Mirrors Prometheus PR prometheus/prometheus#17850, released in v3.11.0. See VictoriaMetrics#10909

… label Add `__meta_hetzner_robot_datacenter` as the canonical label for the datacenter name in the `robot` role, matching the Prometheus v3.11.0 change (prometheus/prometheus#17850). The existing `__meta_hetzner_datacenter` label is kept for backward compatibility but is deprecated for this role. See VictoriaMetrics#10909

…ocation labels Hetzner Cloud deprecated the `datacenter` field in their API on 2025-12-16 and will remove it after 2026-07-01. Add a top-level `HCloudLocation` struct and `Location` field on `HCloudServer` that maps to the new top-level `location` field in the API response. Emit two new canonical labels sourced from this field: - `__meta_hetzner_hcloud_location` - `__meta_hetzner_hcloud_location_network_zone` Keep the deprecated `__meta_hetzner_hcloud_datacenter_location` and `__meta_hetzner_hcloud_datacenter_location_network_zone` labels for backward compatibility, now sourced from the new `location` field. The `HCloudDatacenterLocation` struct and the nested `location` field inside `HCloudDatacenter` are dropped: the live Hetzner Cloud API already returns the top-level `location` field, so there is no need to retain the old parsing path. The test fixture no longer includes `datacenter.location` to reflect that the code no longer reads it. Mirrors Prometheus PR prometheus/prometheus#17850, released in v3.11.0. See VictoriaMetrics#10909

… label Add `__meta_hetzner_robot_datacenter` as the canonical label for the datacenter name in the `robot` role, matching the Prometheus v3.11.0 change (prometheus/prometheus#17850). The existing `__meta_hetzner_datacenter` label is kept for backward compatibility but is deprecated for this role. See VictoriaMetrics#10909

…atacenter → location API change On 2025-12-16, Hetzner Cloud deprecated the `datacenter` field in their Servers API and introduced a top-level `location` field carrying the same data. The `datacenter` field will be removed after 2026-07-01. Without this change, `__meta_hetzner_hcloud_datacenter_location`, and `__meta_hetzner_hcloud_datacenter_location_network_zone` would silently become empty for the `hcloud` role after that date. This mirrors the change made in Prometheus v3.11.0 ([prometheus/prometheus#17850](prometheus/prometheus#17850)). ## Changes **`hcloud` role:** - Add `HCloudLocation` struct and `Location` field on `HCloudServer`, mapped to the new top-level `location` API field - Emit two new canonical labels: `__meta_hetzner_hcloud_location` and `__meta_hetzner_hcloud_location_network_zone` - Keep the deprecated `__meta_hetzner_hcloud_datacenter_location` and `__meta_hetzner_hcloud_datacenter_location_network_zone` labels, now sourced from the new `location` field so they continue to work past 2026-07-01 - `__meta_hetzner_datacenter` (the datacenter name, e.g. `fsn1-dc14`) is unaffected for this role — the datacenter name is a distinct concept from location and is kept as-is (this will stop working starting 2026-07-01) **`robot` role:** - Add `__meta_hetzner_robot_datacenter` as the canonical replacement for `__meta_hetzner_datacenter`; the old label is kept for backward compatibility Fixes #10909

jooola force-pushed the deprecated-datacenter branch 2 times, most recently from 82aca5c to 97a0375 Compare January 13, 2026 14:25

jooola mentioned this pull request Jan 13, 2026

Remove __meta_hetzner_datacenter label from discovery/hetzner before next minor release #17803

Closed

bboreham changed the title ~~Deprecate Hetzner Cloud server datacenter labels~~ Deprecate Hetzner Cloud server datacenter labels / add new labels Jan 20, 2026

jooola force-pushed the deprecated-datacenter branch 2 times, most recently from 4a54f28 to 75aa5ce Compare February 9, 2026 10:13

apricote approved these changes Feb 9, 2026

View reviewed changes

jooola marked this pull request as ready for review February 9, 2026 10:44

jooola requested a review from a team as a code owner February 9, 2026 10:44

jooola requested a review from beorn7 February 9, 2026 10:44

beorn7 requested review from roidelapluie and removed request for beorn7 February 10, 2026 14:19

jooola force-pushed the deprecated-datacenter branch from 75aa5ce to 87a6345 Compare February 25, 2026 09:35

jooola force-pushed the deprecated-datacenter branch from 87a6345 to 6c51a12 Compare March 11, 2026 11:23

jooola requested a review from apricote March 11, 2026 11:35

apricote approved these changes Mar 12, 2026

View reviewed changes

krajorama approved these changes Mar 19, 2026

View reviewed changes

krajorama merged commit a9d9095 into prometheus:main Mar 19, 2026
33 checks passed

jooola deleted the deprecated-datacenter branch March 19, 2026 11:04

juliusrickert mentioned this pull request Apr 30, 2026

vmagent: update hetzner_sd_configs for Hetzner Cloud datacenter → location API change VictoriaMetrics/VictoriaMetrics#10909

Closed

juliusrickert mentioned this pull request Apr 30, 2026

lib/promscrape/discovery/hetzner: update hetzner_sd_configs for Hetzner Cloud datacenter → location API change VictoriaMetrics/VictoriaMetrics#10910

Merged

Uh oh!

Conversation

jooola commented Jan 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue(s) does the PR fix:

Does this PR introduce a user-facing change?

Uh oh!

jooola commented Jan 13, 2026

Uh oh!

bboreham commented Jan 20, 2026

Uh oh!

jooola commented Jan 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

roidelapluie commented Jan 26, 2026

Uh oh!

jooola commented Feb 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

apricote left a comment

Choose a reason for hiding this comment

Uh oh!

beorn7 commented Feb 10, 2026

Uh oh!

jooola commented Feb 25, 2026

Uh oh!

jooola commented Mar 11, 2026

Uh oh!

apricote left a comment

Choose a reason for hiding this comment

Uh oh!

krajorama left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jooola commented Mar 19, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

jooola commented Jan 13, 2026 •

edited

Loading

jooola commented Jan 21, 2026 •

edited

Loading

jooola commented Feb 9, 2026 •

edited

Loading