crypto: switch HTTP clients from native-tls to rustls - #35947

One of several sequenced PRs in the crypto migration toward rustls +
aws-lc-rs for everything.
Specifically:

- **This PR** — mechanical bump from reqwest 0.12 → 0.13. Keeps the
existing `native-tls` backend for now to scope the change to just the
version bump. Does not flip the TLS backend.
- **Next in the reqwest chain** — the HTTP-clients-to-rustls PR (#35947
in the plan) swaps `native-tls` for `rustls` with `aws-lc-rs` across all
reqwest consumers. That's where the real TLS backend change happens.
- **In parallel** — the Azure SDK migration (#36216, stacked on this)
adopts the new `azure_core` 0.35 series, which defaults to
`reqwest_rustls` + aws-lc-rs on its own; that's a concrete step forward
on the Azure path even before the workspace-wide flip.

## Summary
- Bump `reqwest` 0.12.28 → 0.13.2, `reqwest-middleware` 0.4.2 → 0.5.1,
`reqwest-retry` 0.8.0 → 0.9.1.
- `default-features = false` on reqwest so the 0.13 default changes (now
`default-tls = rustls`, new `system-proxy` feature) don't ambush us
mid-bump.
- Explicitly list the feature set we had in 0.12 (including `native-tls`
/ `native-tls-vendored` for now) plus the two newly-opt-in features we
need: `query` (used by `cloud-api` and `frontegg-client`) and
`system-proxy` (was implicit in 0.12).

The `native-tls` pin here is a temporary consequence of keeping this PR
scoped to the bump — the crypto migration's HTTP-clients PR is where the
switch to rustls actually lands.

## Known workarounds for incompatible call sites

Two places still need reqwest 0.12 because their trait ecosystem is
pinned there. Both are narrow and clearly labeled; both go away as the
related migrations land.

1. **`src/persist/src/azure.rs` — Azurite test path only.** The old code
plumbed a custom `reqwest::Client` through `TransportOptions` inside the
`if account == EMULATOR_ACCOUNT` branch to apply short `BlobKnobs`
per-attempt / read / connect timeouts (5s–10s) to the Azurite test
client. `azure_core` 0.21 pins reqwest 0.12 internally, so our 0.13
`Client` no longer implements its `HttpClient` trait.
- **Production impact: zero.** The `else` branch (real Azure Blob
Storage) never set a custom transport — it uses
`BlobServiceClient::new(...)` which gets the SDK's default reqwest
client. That behavior is unchanged.
- **Test impact: minimal.** Azurite runs on localhost, so falling back
to the SDK's default timeouts has no observable effect on test
reliability.
- The outer `knobs.operation_timeout()` is preserved via the retry
policy's `max_total_elapsed` in both branches.
- The Azure SDK 0.35 migration (#36216, stacked) restores the
emulator-path timeout plumbing against the new SDK.
2. **`src/storage-types/src/connections.rs` — `AwsCredentialLoad`
impl.** The `reqsign::AwsCredentialLoad` trait (re-exported from
iceberg) is defined against reqwest 0.12. Added a versioned
`reqwest_0_12` alias dep in `[workspace.dependencies]` and use it only
for the single trait-impl parameter. Removable once iceberg upstream
bumps to a reqsign that supports reqwest 0.13 — tracked as a multi-hop
ecosystem blocker (reqsign 0.17 was a breaking API rewrite; neither
iceberg nor opendal have migrated).

## Cargo.lock

~27 new packages (quinn for HTTP/3, rustls-platform-verifier,
wasm-streams, newer windows-sys). Both reqwest 0.12 and 0.13 coexist in
the tree — the 0.12 copy comes in via azure_core 0.21 and reqsign 0.16
and is needed only for the two workarounds above. Exempted in
`deny.toml`'s skip list.

## Test plan
- [x] `cargo check --workspace --all-targets` — clean
- [x] `cargo --locked deny check bans sources` — `bans ok`
- [x] `bin/lint-cargo` — exit 0
- [x] Full `cargo test` run / CI green
- [ ] Smoke-test at least one TLS client path (`ccsr`,
`frontegg-client`, or similar) against a real endpoint to confirm
native-tls still works at 0.13 before the crypto migration's rustls
switch lands

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Jason Hernandez <[email protected]>
Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>

One of several sequenced PRs in the crypto migration toward rustls +
aws-lc-rs for everything.
Specifically:

- **This PR** — mechanical bump from reqwest 0.12 → 0.13. Keeps the
existing `native-tls` backend for now to scope the change to just the
version bump. Does not flip the TLS backend.
- **Next in the reqwest chain** — the HTTP-clients-to-rustls PR (#35947
in the plan) swaps `native-tls` for `rustls` with `aws-lc-rs` across all
reqwest consumers. That's where the real TLS backend change happens.
- **In parallel** — the Azure SDK migration (#36216, stacked on this)
adopts the new `azure_core` 0.35 series, which defaults to
`reqwest_rustls` + aws-lc-rs on its own; that's a concrete step forward
on the Azure path even before the workspace-wide flip.

- Bump `reqwest` 0.12.28 → 0.13.2, `reqwest-middleware` 0.4.2 → 0.5.1,
`reqwest-retry` 0.8.0 → 0.9.1.
- `default-features = false` on reqwest so the 0.13 default changes (now
`default-tls = rustls`, new `system-proxy` feature) don't ambush us
mid-bump.
- Explicitly list the feature set we had in 0.12 (including `native-tls`
/ `native-tls-vendored` for now) plus the two newly-opt-in features we
need: `query` (used by `cloud-api` and `frontegg-client`) and
`system-proxy` (was implicit in 0.12).

The `native-tls` pin here is a temporary consequence of keeping this PR
scoped to the bump — the crypto migration's HTTP-clients PR is where the
switch to rustls actually lands.

Two places still need reqwest 0.12 because their trait ecosystem is
pinned there. Both are narrow and clearly labeled; both go away as the
related migrations land.

1. **`src/persist/src/azure.rs` — Azurite test path only.** The old code
plumbed a custom `reqwest::Client` through `TransportOptions` inside the
`if account == EMULATOR_ACCOUNT` branch to apply short `BlobKnobs`
per-attempt / read / connect timeouts (5s–10s) to the Azurite test
client. `azure_core` 0.21 pins reqwest 0.12 internally, so our 0.13
`Client` no longer implements its `HttpClient` trait.
- **Production impact: zero.** The `else` branch (real Azure Blob
Storage) never set a custom transport — it uses
`BlobServiceClient::new(...)` which gets the SDK's default reqwest
client. That behavior is unchanged.
- **Test impact: minimal.** Azurite runs on localhost, so falling back
to the SDK's default timeouts has no observable effect on test
reliability.
- The outer `knobs.operation_timeout()` is preserved via the retry
policy's `max_total_elapsed` in both branches.
- The Azure SDK 0.35 migration (#36216, stacked) restores the
emulator-path timeout plumbing against the new SDK.
2. **`src/storage-types/src/connections.rs` — `AwsCredentialLoad`
impl.** The `reqsign::AwsCredentialLoad` trait (re-exported from
iceberg) is defined against reqwest 0.12. Added a versioned
`reqwest_0_12` alias dep in `[workspace.dependencies]` and use it only
for the single trait-impl parameter. Removable once iceberg upstream
bumps to a reqsign that supports reqwest 0.13 — tracked as a multi-hop
ecosystem blocker (reqsign 0.17 was a breaking API rewrite; neither
iceberg nor opendal have migrated).

~27 new packages (quinn for HTTP/3, rustls-platform-verifier,
wasm-streams, newer windows-sys). Both reqwest 0.12 and 0.13 coexist in
the tree — the 0.12 copy comes in via azure_core 0.21 and reqsign 0.16
and is needed only for the two workarounds above. Exempted in
`deny.toml`'s skip list.

- [x] `cargo check --workspace --all-targets` — clean
- [x] `cargo --locked deny check bans sources` — `bans ok`
- [x] `bin/lint-cargo` — exit 0
- [x] Full `cargo test` run / CI green
- [ ] Smoke-test at least one TLS client path (`ccsr`,
`frontegg-client`, or similar) against a real endpoint to confirm
native-tls still works at 0.13 before the crypto migration's rustls
switch lands

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Jason Hernandez <[email protected]>
Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>

…37469)

## Summary

Re-applies the reqwest 0.12 → 0.13 bump (originally #36215, reverted in
#36241 when the Kafka auth schema-registry TLS tests started flaking on
main). Bumps `reqwest` to 0.13.4, `reqwest-middleware` to 0.5.2, and
`reqwest-retry` to 0.9.1. This is only the version bump — the TLS
backend stays native-tls; the switch to rustls is owned separately by
the crypto migration (#35947).

## Why it should stick this time

The April flake was an OpenSSL `PEM routines:get_name:no start line`
error at connection time on the `ccsr` (schema registry) client, on
requests whose TLS handshake should have succeeded. Three things are
different now:

1. **reqwest 0.13.4, not 0.13.2.** Upstream has since fixed several bugs
in the 0.13 TLS config layer: PEM parsing of CRL values plus a
`rustls-platform-verifier` upgrade (0.13.3), native-tls ALPN
combinations (late 0.13.2), and TLS 1.3 for native-tls (0.13.4). Note
the tree already ships reqwest 0.13.3 transitively via `sentry` 0.48, so
reqwest 0.13 is not itself new to production images.
2. **`ccsr` pins `tls_backend_native()` explicitly.** In reqwest 0.13,
rustls becomes the default backend whenever any transitive dependency
compiles the rustls feature in (ours do — the lockfile carries
`rustls-platform-verifier`/`quinn`), while `ccsr`'s certificates and
identities are prepared for native-tls. Silent backend/verifier drift on
this exact client is the most plausible mechanism for the April flake,
and the explicit pin removes that degree of freedom regardless of which
theory is correct.
3. **Cleaner handling of the reqwest-0.12-only ecosystem pins.** The
Azure transport client now comes from the `reqwest_0_12` workspace
alias, keeping the `BlobKnobs` timeout plumbing intact on both the
emulator and production paths (`azure_core` 0.21 implements `HttpClient`
only for reqwest 0.12; April dropped the emulator-path timeouts to work
around this). The alias also covers iceberg's `RequestAuthenticator`,
which gained a second impl (`Sigv4Authenticator`) since April.

## Validation plan

The Kafka auth suite is the regression signal that matters here (it
exercises the exact schema-registry TLS path that flaked). It was
disabled in CI when this PR was first opened (SS-115), but #37470
re-enabled it and merged, and this branch is now rebased on top — so
**Kafka auth 1–3 run on this PR's CI** and directly exercise the April
failure mode.

Because that failure was intermittent, a single green run isn't
sufficient. The `mz-test` skill's flake-handling guidance applies:
re-trigger the build 10–20 times (via
https://ci.dev.materialize.com/trigger, clearing the target branch for
parallel runs) and confirm Kafka auth stays green before merging.

- [x] `cargo check --workspace --all-targets`
- [x] `cargo --locked deny check licenses bans sources`,
`bin/lint-cargo`, `bin/fmt`
- [x] Kafka auth suite green locally (8/8 consecutive runs) on this
stack
- [ ] Kafka auth green across 10–20 CI re-triggers (now runs here
post-rebase, per the `mz-test` skill)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Jason Hernandez <[email protected]>
Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>

### Motivation

Move reqwest 0.13 off OpenSSL (native-tls) onto rustls with the
aws-lc-rs provider, as part of removing OpenSSL and `ring` from the
tree. Prior art: #35947.

### Description

- reqwest uses its `rustls` feature: aws-lc-rs plus
`rustls-platform-verifier`, which trusts the system store on Linux and
macOS.
- sentry switches from `transport` to `reqwest` + `rustls`. `transport`
enables reqwest's native-tls, and reqwest defaults to native-tls
whenever it is compiled in, so every client would otherwise stay on
OpenSSL.
- mz-ccsr's `Identity` holds the PEM key and certificates and passes
them to rustls. Schema registry connections already take PEM (`SSL KEY`,
`SSL CERTIFICATE`). The PKCS#12 archive `Identity` used to build from
them was only an internal step for native-tls, so the SQL surface is
unchanged. `Identity` checks that the key matches the certificate, is
zeroized on drop, and redacts `Debug`. It is built from secrets at
connect time and never persisted.
- testdrive's `--cert` still takes a PKCS#12 keystore and converts it to
PEM itself. No CI job passes `--cert`.
- New `mz_ccsr::tls::TlsError` replaces the native-tls and OpenSSL
variants of `CsrConnectError`. mz-ccsr and mz-storage-types drop
native-tls and openssl.
- Still on OpenSSL after this PR: reqwest 0.12 (forced by the iceberg
fork's defaults and duckdb), and the MySQL source's PEM to PKCS#12
conversion for `mysql_async`.

### Behavior changes

- On Linux, building a client fails fast if no system CA bundle is
present.
- The system trust store is read each time a client is built, which for
ccsr means once per schema registry connection.
- Schema registry TLS errors use rustls text, for example `invalid peer
certificate: UnknownIssuer`. The kafka-auth `.td` files are updated.
- A key that does not match its certificate is rejected with `keys may
not be consistent: KeyMismatch` instead of an OpenSSL PKCS#12 error.

### Verification

Locally: `bin/fmt`, workspace `cargo check`, clippy on the touched
crates, cargo-deny, lint-cargo, and ccsr and testdrive unit tests,
including a new `test_pem_identity`.

CI: PR CI covers `kafka-auth` (schema registry ssl, ssl-basic, mssl,
mssl-basic), `testdrive`, and the environmentd `auth`/`server` and
balancerd `server` tests. Nightly (`ci-nightly`) covers
`testdrive-old-kafka-src-syntax`, `cloudtest` and `orchestratord`.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Jason Hernandez <[email protected]>
Co-authored-by: Claude Opus 5.5 <[email protected]>