Repository navigation
Health probes move to their own server on :9091, and :9090 serves only /metrics - #2424
Merged
Krisztian F (krisztianfekete) merged 4 commits intoOct 13, 2026
Conversation
ateapi, atelet, atenet-router, atenet-egress, and the kubernetes-secrets credential provider served /readyz and /healthz on the Prometheus /metrics server on :9090. Thus, a change to the metrics endpoint also changed the health checks. A later change lets OTEL_METRICS_EXPORTER turn off /metrics, and that change must not stop the probes. serverboot now has two servers. StartMetricsServer serves only /metrics. StartHealthServer serves /readyz and /healthz, and the process exits if the server cannot start. Each component gets a new flag for the health address, with the default :9091: --health-listen-addr, or --health-address on the credential provider. The probes in the manifests use the new health port. hack/verify-atenet-drain.sh forwards the health port. glutton serves only /metrics on :9090 now. Its wakeup probe goes to /readyz on port 80, so it does not use the readiness endpoint on :9090. atelet no longer has hostPort: 9090. The port opened /metrics on the IP address of each node, and nothing reads the metrics through the node IP.
The credential provider started its health server before it built the Kubernetes client, loaded the namespace policy, and opened the gRPC listener. Thus /readyz returned 200 before the pod could accept calls. The health server now starts after the listener opens. The Cloud SQL proxy sidecar comment now says that ateapi owns both 9090 and 9091, the same as the comment in ate-setup.
Jeff Luo (JeffLuoo)
added
breaking-change
Jeff Luo (JeffLuoo)
changed the title
| EnableHealthz: true, | ||
| }) | ||
| go serverboot.StartMetricsServer(ctx, *metricsListenAddr) | ||
| go serverboot.StartHealthServer(ctx, *healthListenAddr, readiness) |
Collaborator
There was a problem hiding this comment.
/readyz returns 200 here, before atelet opens its gRPC listener at L343. Can /readyz start as not ready and switch to ready after net.Listen? This is fine as a follow up.
Collaborator
Author
There was a problem hiding this comment.
I will do it in a follow-up PR. This PR doesn't close the issue so I will use the same issue to track. I will leave this discussion open.
The metrics and health servers now use an http.Server with a ReadHeaderTimeout of 10 seconds. Without the timeout, a slow client can keep a connection open forever, and the liveness probe can fail. StartMetricsServer now exits the process when the server cannot start, the same as StartHealthServer. The probes no longer use the metrics server, so without the exit a bind failure shows only as one log line. ateom-gvisor and ateom-microvm now call StartHealthServer, which serves /readyz and also /healthz. StartReadinessServer served only /readyz, and no caller uses it now, so it is removed. ateapi starts its metrics server directly after it sets up the meter provider. The probes have their own server, so /metrics can serve during the store connect and the schema migrations. The test for the two servers is now one table with t.Run, so a failure names the server and the path.
The metrics server and the health server both use the timeout, not only the probe server. The name httpServerReadHeaderTimeout matches newHTTPServer, which applies it.
Krisztian F (krisztianfekete)
approved these changes
Oct 13, 2026
Collaborator
Author
|
Krisztian F (@krisztianfekete) Can you help add the change to the merge queue? Thanks! |
This was referenced Oct 13, 2026
Da Huang (git286)
added a commit
to git286/substrate
that referenced
this pull request
Oct 19, 2026
- kubectl ate logs uses the singular 'actor' subcommand; the not-running error names atespace/name; quote the crash log message exactly. - GKE telemetry goes to the GKE managed OTel collector, not straight to GMP. - Workers normally export through the atelet relay, so an endpoint change reaches them when atelet restarts; the controller copy is the fallback. - List why atelet may start no relay (https, conflicting per-signal endpoints or compression, unparsable headers), per-signal endpoints as a fallback destination, and that ate-controller injects the endpoint only when it has one. - The probes moved to :9091 (agent-substrate#2424); agent-substrate#1558 was closed unmerged. - Drop the removed pause from the lifecycle and snapshot-kind text. - The registry table now lists events.yaml. - Kind Prometheus does not show the metrics that ateworker, atecontroller and the credential provider only push over OTLP. - Tracing is head-sampled, and the roadmap metrics already exist.
Da Huang (git286)
added a commit
to git286/substrate
that referenced
this pull request
Oct 19, 2026
- kubectl ate logs uses the singular 'actor' subcommand; the not-running error names atespace/name; quote the crash log message exactly. - GKE telemetry goes to the GKE managed OTel collector, not straight to GMP. - Workers normally export through the atelet relay, so an endpoint change reaches them when atelet restarts; the controller copy is the fallback. - List why atelet may start no relay (https, conflicting per-signal endpoints or compression, unparsable headers), per-signal endpoints as a fallback destination, and that ate-controller injects the endpoint only when it has one. A socket left by an earlier atelet keeps ateworker on the relay, failing every export instead of falling back. - The probes moved to :9091 (agent-substrate#2424); agent-substrate#1558 was closed unmerged. - Drop the removed pause from the lifecycle and snapshot-kind text. - The registry table now lists events.yaml. - Kind Prometheus does not show the metrics that ateworker and atecontroller only push over OTLP; the credential provider does not read ate-otel-config, so its OTLP push never reaches the collector. - Tracing is head-sampled, and the roadmap metrics already exist. Checked on a Kind cluster built from main and on a GKE install.
Da Huang (git286)
added a commit
to git286/substrate
that referenced
this pull request
Oct 20, 2026
- kubectl ate logs uses the singular 'actor' subcommand; the not-running error names atespace/name; quote the crash log message exactly. - GKE telemetry goes to the GKE managed OTel collector, not straight to GMP. - Workers normally export through the atelet relay, so an endpoint change reaches them when atelet restarts; the controller copy is the fallback. - List why atelet may start no relay (https, conflicting per-signal endpoints or compression, unparsable headers), per-signal endpoints as a fallback destination, and that ate-controller injects the endpoint only when it has one. A socket left by an earlier atelet keeps ateworker on the relay, failing every export instead of falling back. - The probes moved to :9091 (agent-substrate#2424); agent-substrate#1558 was closed unmerged. - Drop the removed pause from the lifecycle and snapshot-kind text. - The registry table now lists events.yaml. - Kind Prometheus does not show the metrics that ateworker and atecontroller only push over OTLP; the credential provider does not read ate-otel-config, so its OTLP push never reaches the collector. - Tracing is head-sampled, and the roadmap metrics already exist. Checked on a Kind cluster built from main and on a GKE install.
Scott Funkenhauser (sfunkenhauser)
pushed a commit
to sfunkenhauser/substrate
that referenced
this pull request
Oct 20, 2026
…2886) - kubectl ate logs uses the singular 'actor' subcommand; the not-running error names atespace/name; quote the crash log message exactly. - GKE telemetry goes to the GKE managed OTel collector, not straight to GMP. - Workers normally export through the atelet relay, so an endpoint change reaches them when atelet restarts; the controller copy is the fallback. - List why atelet may start no relay (https, conflicting per-signal endpoints or compression, unparsable headers), per-signal endpoints as a fallback destination, and that ate-controller injects the endpoint only when it has one. A socket left by an earlier atelet keeps ateworker on the relay, failing every export instead of falling back. - The probes moved to :9091 (agent-substrate#2424); agent-substrate#1558 was closed unmerged. - Drop the removed pause from the lifecycle and snapshot-kind text. - The registry table now lists events.yaml. - Kind Prometheus does not show the metrics that ateworker and atecontroller only push over OTLP; the credential provider does not read ate-otel-config, so its OTLP push never reaches the collector. - Tracing is head-sampled, and the roadmap metrics already exist.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ateapi, atelet, atenet-router, atenet-egress, and the kubernetes-secrets credential provider served
/readyzand/healthzon the Prometheus/metricsserver on:9090. A later change letsOTEL_METRICS_EXPORTERturn off/metrics, and that change must not stop the kubelet probes. This PR moves the probes to a separate health server.Breaking change
The health probes move from
:9090to:9091(/readyz,/healthz).:9090now serves only/metrics. If you use your own manifests, change your probes to the newhealthport. Apply the new images and manifests together. atelet no longer hashostPort: 9090.Testing
go test -racepasses forinternal/serverboot,cmd/ate-setup/internal/steps,cmd/ateapi,cmd/atelet,cmd/atenet/internal/router, andcmd/credential-provider/kubernetes-secrets.Part of #2067