Repository navigation
crypto: switch rdkafka from OpenSSL to AWS-LC (ssl-awslc) - #35941
Closed
jasonhernandez wants to merge 3 commits into
Closed
jasonhernandez wants to merge 3 commits into
jasonhernandez wants to merge 3 commits into
Conversation
Contributor
|
Thanks for opening this PR! Here are a few tips to help make the review process smooth for everyone. PR title guidelines
Pre-merge checklist
|
2 tasks
jasonhernandez
force-pushed
the
jh/crypto-pr4-rdkafka-awslc
branch
from
April 15, 2026 03:34
478f73a to
cc6c732
Compare
2 of 3 tasks
Add rustls, aws-lc-rs, hyper-rustls, rcgen, tokio-rustls workspace deps and the mz-ore `crypto` feature with FIPS-aware CryptoProvider helpers. Azure SDK forks patched to support reqwest-rustls-no-provider. Removes the rustls ban from deny.toml. Does NOT bump LaunchDarkly SDK (deferred to end of migration). Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Switch rdkafka from ssl-vendored (vendored OpenSSL) to ssl-awslc (AWS-LC) using the MaterializeInc fork. AWS-LC provides FIPS 140-3 validation and eliminates the vendored OpenSSL dependency. - Workspace: rdkafka/rdkafka-sys fork → jasonhernandez/ssl-awslc branch - kafka-util: rdkafka features ssl-vendored → ssl-awslc - CI: Add Go, ninja-build, build AWS-LC static library in Docker builder - CI: Simplify macOS clippy (fork auto-builds when /opt/aws-lc absent) - Update Kafka TLS and MySQL CDC test expectations Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Adapt the rdkafka -> AWS-LC branch to current main after a ~1155-commit rebase: - Cargo.lock: regenerated via `cargo metadata` from the merged manifest. Only source redirects, no version bumps: rdkafka/rdkafka-sys now pin the `jasonhernandez/ssl-awslc` fork branch, and the five azure-sdk crates pin the `mz/enable-reqwest-rustls-no-provider` fork branch. - src/ore/Cargo.toml: drop `default-features = false` on the inherited `rustls` dependency. Main's workspace root declares `rustls = "0.23.38"` with default features (already aws_lc_rs, no ring), and orchestratord relies on them. Cargo forbids a member from disabling default features the workspace enables, so mz-ore now inherits them and just adds `aws_lc_rs`. - deny.toml: keep only the duplicate-version skips that current main's tree actually hits via the azure fork (core-foundation, openssl-probe, security-framework). Drop the stale skips and the unused hyper-rustls wrapper that no longer match any crate. - test/kafka-auth schema-registry and test/mysql-cdc TLS tests: revert to main. Those paths still use reqwest/native-tls and mysql_async/native-tls in this branch, so their rustls-specific error assertions (UnknownIssuer, invalid peer certificate, failed to parse private key PEM) cannot match. The genuine librdkafka -> AWS-LC error-string changes are retained in the four test-kafka-*.td files. Co-Authored-By: Claude Fable 5 <[email protected]>
jasonhernandez
force-pushed
the
jh/crypto-pr4-rdkafka-awslc
branch
from
July 7, 2026 08:45
cc6c732 to
443a34d
Compare
jasonhernandez
added a commit
that referenced
this pull request
Jul 7, 2026
… 8.x) (#37470) ## Summary Re-enables the Kafka auth test suite, disabled since #36405 (SS-115), and fixes the two breakages that accumulated while it was dark. ## What broke while the suite was disabled The Confluent Platform 7.9.4 → 8.2.0 bump (#36683) landed three weeks after the suite was skipped, and broke its setup in two ways nothing could catch: 1. **Schema registry basic auth returned 401 for everyone.** CP 8.x ships Jetty 12, which moved the JAAS login modules from `org.eclipse.jetty.jaas.spi` to `org.eclipse.jetty.security.jaas.spi`. Our `schema-registry.jaas.config` referenced the old class, so the login module never loaded and every request, correct credentials included, got 401. Verified directly: `curl -u materialize:sekurity` against the container → 401 before, 200 after the one-line class-path fix (wrong/missing creds still 401). 2. **Kafka mssl negative tests asserted TLS 1.2 alert text.** CP 8.x brokers negotiate TLS 1.3, where a missing/untrusted client certificate fails with `tlsv13 alert certificate required` (alert 116) instead of `ssl/tls alert bad certificate` (alert 42). The assertions now check only that the broker rejected the connection with a TLS alert. The exact wording is a function of TLS version and OpenSSL build that we do not control. This is the third exact-text loosening in this suite's history (#30501, #36232), so the loosening deliberately goes all the way rather than chasing the new strings. ## On the original SS-115 flake The `PEM routines:get_name:no start line` flake that got the suite disabled did not reproduce in local full-suite runs against current main images (reqwest 0.12) or against the reqwest 0.13.4 branch (#37469). Timeline evidence from SS-115 itself shows the flake continued after the reqwest 0.13 revert (#36241), so reqwest was likely misattributed. If it resurfaces, the `ci-regexp` in SS-115 will link it; the suite being enabled is what gives us the data to root-cause it. ## Why now Two in-flight workstreams change exactly the surface this suite covers: the reqwest 0.13 re-application (#37469) and the rdkafka → AWS-LC switch (#35941). Landing those with this suite dark is how auth/TLS regressions reach production unobserved. ## Test plan - [x] Full suite green locally against main images (all fixes applied) - [x] Repeated local runs to probe for the SS-115 flake: 8 consecutive local full-suite passes (5-run loop + 3 individual), plus 2 clean runs on the reqwest-0.12 main baseline. The PEM flake did not reproduce. - [ ] Kafka auth 1–3 green in this PR's CI (the un-skip makes them run here) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Jason Hernandez <[email protected]> Co-authored-by: Claude Fable 5 <[email protected]>
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
ssl-vendored(OpenSSL) tossl-awslc(AWS-LC) using MaterializeInc forkPart 4 of 7 in the crypto migration from native-tls/OpenSSL to rustls/aws-lc-rs.
Depends on PR1 (#35940).
Test plan
cargo check --workspacepassescargo build -p rdkafka-syslinks successfully🤖 Generated with Claude Code
Part of SEC-237.