Skip to content

kube: switch the Kubernetes client from OpenSSL to rustls with aws-lc-rs - #39316

Merged
jasonhernandez merged 1 commit into
MaterializeInc:mainfrom
jasonhernandez:jason/kube-rustls-tls
Sep 30, 2026
Merged

jasonhernandez merged 1 commit into
MaterializeInc:mainfrom
jasonhernandez:jason/kube-rustls-tls

Conversation

@jasonhernandez

@jasonhernandez jasonhernandez commented Sep 29, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

Removes OpenSSL from the Kubernetes client. kube now uses rustls with the aws-lc-rs provider, like our other rustls users.

Change

  • kube uses rustls-tls and aws-lc-rs instead of openssl-tls. kube-client no longer depends on openssl or hyper-openssl, and adds no path to ring.
  • kube resolves the process-default rustls provider, which panics when both ring and aws-lc-rs are enabled and none is installed. orchestratord now installs aws-lc-rs unconditionally at startup, and mz-debug now installs it too. environmentd and clusterd already did.
  • The workspace hyper-openssl declaration enables client-legacy and tokio explicitly, since environmentd used to get them through kube.
  • hyper-rustls is added to the log ban wrappers in deny.toml.

Risks

  • PR CI builds and unit-tests the kube-using crates but never connects to an API server. In-cluster config connects by IP, so rustls verifies the API server certificate against an IP SAN, and it is stricter than OpenSSL about certificates without SANs. Nightly covers this: cloudtest and the orchestratord jobs run against kind, and the terraform-* jobs run against EKS, GKE and AKS. Self-managed clusters with hand-issued API server certificates are not covered by any CI.
  • Client keys from kubeconfig, for example in mz-debug, are now parsed by rustls.

Release notes

  • Self-managed: the Materialize operator, environmentd and mz-debug now verify the Kubernetes API server certificate with rustls instead of OpenSSL. Verification is stricter: the certificate must carry a subject alternative name (SAN) matching the address used to reach the API server, which is an IP address for in-cluster connections. Certificates that name the host only in the Common Name are rejected. Clusters on EKS, GKE, AKS and kind are unaffected. Clusters with hand-issued API server certificates should confirm they include the right SANs before upgrading.
  • mz-debug reads kubeconfig client keys with rustls. PEM keys in PKCS#8, PKCS#1 and SEC1 format are supported.

Test plan

  • PR CI: build, clippy and unit tests. Cargo.lock has no version changes.
  • Nightly via the ci-nightly label: cloudtest, cloudtest-upgrade, orchestratord-defaults and orchestratord-rolling-upgrade-downtime against kind.

🤖 Generated with Claude Code

Replace kube's `openssl-tls` feature with `rustls-tls` plus `aws-lc-rs`,
so kube-client no longer depends on `openssl` or `hyper-openssl` and adds
no path to `ring`.

kube builds its rustls config with `ClientConfig::builder()`, which
resolves the process-default crypto provider and panics when rustls'
`ring` and `aws-lc-rs` features are both enabled and none is installed.
environmentd and clusterd already install aws-lc-rs at startup.
orchestratord only installed it when the conversion webhook was enabled,
so the install now runs unconditionally at the top of `main`. mz-debug
installed none and now does.

environmentd relied on kube to enable hyper-openssl's `client-legacy`
and `tokio` features. The workspace declaration now enables them
explicitly. hyper-rustls is added to the `log` ban wrappers in
deny.toml, because kube-client enables its `logging` feature.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@jasonhernandez
jasonhernandez marked this pull request as ready for review September 30, 2026 18:53
@jasonhernandez
jasonhernandez requested a review from a team as a code owner September 30, 2026 18:53
@jasonhernandez jasonhernandez added the ci-nightly PR CI control: also trigger Nightly label Sep 30, 2026

@antiguru antiguru left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@jasonhernandez
jasonhernandez merged commit 1b8ce14 into MaterializeInc:main Sep 30, 2026
345 of 349 checks passed
This was referenced Oct 2, 2026
val-materialize added a commit that referenced this pull request Oct 9, 2026
Release notes for the v26.45 train, published as **v26.45.1**. v26.45.0
did not ship, so (as with v26.44) its notes are folded in here and
[#39477](#39477) is
closed unmerged.

- **v26.45.0 content**: 8 Improvements, 2 Agent Skills, 8 Bug Fixes,
copied verbatim from #39477 with anchors renamed to `{#v26.45.1-…}`.
- **v26.45.1 content**: Alerting for Self-Managed and the external
dependency monitoring Improvement (editor-added). The release's only PR,
[#39659](#39659), is
omitted.
- **Dates**: Cloud 2026-10-08, Self-Managed 2026-10-09. The operator
compatibility row is `v26.45.1`. `v26.45.md` (`patch: 1`, no `rc`) is
already on `main`.

⚠️ **Open questions**

- **Group-sync bullet**: `oidc_group_role_sync_enabled` defaults to
`false` at the tag. Is it on in Cloud? If not, cut the "Quieter logins"
bullet.
- **Certificate SAN requirements**
([#39316](#39316),
[#39415](https://github.com/MaterializeInc/materialize/pull/39415)/[#39459](https://github.com/MaterializeInc/materialize/pull/39459))
need action at upgrade time but sit in Improvements. Should they be
called out more loudly? Also, does
[#39351](#39351) (S3
on rustls) impose the same SAN rule on custom S3 endpoints?
- **[#39659](#39659)
(omitted)**: is `enable_upsert_paged_spill` on in production? A draft
bullet is in
[omitted.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.45.1/final/omitted.md#borderline).
- **`EXPLAIN ANALYZE ... CPU`**:
[agent-skills#90](MaterializeInc/agent-skills#90)
says it overstates joins, reductions, and TopK by 1.5x–3x before v26.46.
Not published. Is that a known issue?
- **Metric sinks**:
[#39332](#39332),
[#38910](#38910), and
[#39337](#39337) are
withheld because `enable_metric_sink` is `false`. #39337 fixes a panic
any superuser can reach, so consider publishing it regardless.
- **Possible cuts**:
[#39362](#39362) was
observed only in CI. The latency figures from
[#39179](#39179) were
measured on one noisy machine.
- **Possible adds**:
[#38894](https://github.com/MaterializeInc/materialize/pull/38894)–[#38896](https://github.com/MaterializeInc/materialize/pull/38896)
(MV correction buffer) need an end-to-end number.
[#39149](#39149) fixes
the kind install guide, which crash-loops today.
[#39549](#39549)
(jemalloc page) is omitted because no released version had the
regression.

Snapshot drafts in mz-skills:
[v26.45.0](https://github.com/MaterializeInc/mz-skills/tree/main/data/mz-release-notes/v26.45.0)
(rc.1–rc.4, final),
[v26.45.1](https://github.com/MaterializeInc/mz-skills/tree/main/data/mz-release-notes/v26.45.1/final).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <[email protected]>
Co-authored-by: Valerie Chiang <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci-nightly PR CI control: also trigger Nightly

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants