Repository navigation
orchestratord: Make environmentd readiness, liveness and startup probes configurable - #40034
Merged
Merged
Conversation
Alphadelta14
approved these changes
Oct 20, 2026
| volume_mounts: Some(volume_mounts), | ||
| liveness_probe: Some(probe.clone()), | ||
| readiness_probe: Some(probe), | ||
| liveness_probe: Some(liveness_probe), |
Member
There was a problem hiding this comment.
we also desperately need a startup probe. This allows for much better granularity during early bootup vs lifecycle (we don't want the liveness probe to be useless in detecting actual issues).
| termination_grace_period_seconds, | ||
| timeout_seconds, | ||
| } = overlay.clone(); | ||
| let has_handler = |
Member
There was a problem hiding this comment.
this looks like it won't reject if two probe options are set, which is very much required
jubrad
changed the title
jubrad
force-pushed
the
environmentd-probes
branch
from
October 20, 2026 20:02
7924019 to
0c0ce48
Compare
jubrad
added a commit
that referenced
this pull request
Oct 20, 2026
### Motivation environmentd's readiness and liveness probes are TCP checks against the external SQL port. environmentd binds that port before it boots but only accepts connections once the coordinator is up, so connections made during boot wait in the kernel accept queue (129 entries on Linux with the current backlog). balancerd and clients connect to the external SQL port. The TCP readiness probe passes as soon as the port is bound, so the pod is marked Ready during boot and receives that traffic. During a long boot, client connections can fill the queue. Linux then drops new connection attempts, TCP probes time out, and liveness kills environmentd after 12 consecutive failures. That restarts the boot, which can turn into a crashloop. ### Description Point both probes at the internal SQL port (`--environmentd-internal-sql-port`, 6877 by default) instead. orchestratord configures the internal SQL listener for every environmentd version it manages, and client traffic does not use it, so the probes' queue only fills from the probes themselves. Readiness and liveness together make about 12 connections a minute, which fills 129 slots after roughly 10 minutes of boot. A larger listen backlog raises that limit further. This changes the generated environmentd StatefulSet. After upgrading the operator, existing environments report pending changes and pick up the new probe port on their next rollout. They are not restarted on their own. This conflicts with the probe configuration in #40034 on the same lines. Whichever lands second rebases. ### Tests No new tests. The orchestratord mzcompose tests already wait for environmentd to become ready, which requires the readiness probe on the new port to pass. ### Release notes This release will make environmentd less likely to be restarted by its liveness probe during a long startup. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <[email protected]>
…urable
Add `environmentdReadinessProbe` and `environmentdLivenessProbe` to both
versions of the Materialize CRD, matching
`--environmentd-default-{readiness,liveness}-probe` orchestratord flags,
and `environmentd.default{Readiness,Liveness}Probe` helm values.
Each layer is overlaid onto the built-in TCP probe against the SQL port,
then the operator default, then the CR. Only fields set in a layer
change, and a handler replaces the previous one as a unit.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
…robes Add `environmentdStartupProbe` to the Materialize CRD, with a matching `--environmentd-default-startup-probe` flag and `environmentd.defaultStartupProbe` helm value. No startup probe runs unless one is configured. Reject a probe that sets more than one handler, at orchestratord startup for the operator defaults and at reconcile time for the CR fields. Co-Authored-By: Claude Opus 5.5 <[email protected]>
…QL port Co-Authored-By: Claude Opus 5.5 <[email protected]>
`validate_probe_handlers` is public, so its doc must not link to the private `environmentd_probe`; the Doctests lint rejects that with `-D warnings`. The built-in probe targets the internal SQL port only when one is configured, so describe it as environmentd's SQL port in the CRD and helm docs. Co-Authored-By: Claude Opus 5.5 <[email protected]>
jubrad
force-pushed
the
environmentd-probes
branch
from
October 23, 2026 03:27
0c0ce48 to
7cd0cd8
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
The environmentd readiness and liveness probes are hardcoded in orchestratord: a TCP check against environmentd's SQL port (the internal one when configured, see #40036 and #40046) with
initialDelaySeconds: 1andfailureThreshold: 12. Self-managed operators cannot tune them, and there is no startup probe, so the liveness probe has to be loose enough to cover a slow boot, which limits how quickly it can detect a stuck environmentd.Description
The readiness and liveness probes can now be configured, and a startup probe can be added, at three layers:
spec.environmentdReadinessProbe,spec.environmentdLivenessProbeandspec.environmentdStartupProbe, of typeProbe.--environmentd-default-readiness-probe,--environmentd-default-liveness-probeand--environmentd-default-startup-probe, as JSON, like--environmentd-default-resources.environmentd.defaultReadinessProbe,environmentd.defaultLivenessProbeandenvironmentd.defaultStartupProbe(default{}), passed through to the flags above.Each layer is overlaid onto the one below it, in this order: the built-in probe, then the operator default, then the CR. Only the fields a layer sets change, so
environmentdLivenessProbe: {failureThreshold: 60}keeps the built-in TCP check without the CR author having to know the operator's port configuration. Setting any handler (exec,grpc,httpGet,tcpSocket) replaces the inherited handler as a unit, because Kubernetes rejects a probe with more than one.There is no built-in startup probe, so none runs unless the operator default or the CR sets one. When one is set, it is overlaid onto the same built-in TCP check as the other two.
A probe that sets more than one handler is rejected: orchestratord fails to start if an operator-default flag has one, and reconciling a CR with one fails with an error naming the field. Each layer is checked, so the merged probe always has exactly one handler.
Rollout behavior:
environmentdResourceRequirements.The CRD description data under
doc/user/data/self_managed/is regenerated withcrd-writer. The diff only adds the new fields.Tests
generation.rs: unit tests for the probe overlay (no overrides, layered fields, replacing the handler as a unit) and for rejecting multiple handlers.bin/orchestratord.rs: a test that the default probe flags reject a probe with two handlers.materialize.rs: a test that unset probes keep the rollout hash, setting them changes it, and both full-object and partial-object v1/v1alpha1 conversions preserve them.tests/deployment_test.yamlfor the three flags with probes set and unset.EnvironmentdProbesmodification intest/orchestratord/mzcompose.pythat checks the probes on the running environmentd pod, including that there is no startup probe unless one is configured.Release notes
This release will allow configuring the environmentd readiness, liveness and startup probes through the Materialize CR (
environmentdReadinessProbe,environmentdLivenessProbe,environmentdStartupProbe) and operator-wide through the helm chart (environmentd.defaultReadinessProbe,environmentd.defaultLivenessProbe,environmentd.defaultStartupProbe).🤖 Generated with Claude Code