Repository navigation
Kafka Connect: Track Hive distribution runtime dependencies - #18660
kevinjqliu merged 2 commits into
Conversation
9d038d7 to
85dd7fe
Compare
|
@danielcweeks applying the same change in #18019 to the other kakfa connect distribution. See "main vs hive" to see the diff in dependencies bundled. Please take a look when you get a chance! I don't want to update |
| runtimeClasspath { | ||
| // keep the hive distribution a superset of main by resolving shared jars to the same versions | ||
| shouldResolveConsistentlyWith(hive) | ||
| } |
There was a problem hiding this comment.
this + the kafka-connect/kafka-connect-runtime/main/LICENSE and kafka-connect/kafka-connect-runtime/main/runtime-deps.txt change is to make
hive = main + more deps
see 401b6f0
The Kafka Connect runtime ships main and hive distributions with separate LICENSE and NOTICE files, but only the main distribution's runtime dependencies were tracked. Keep a baseline next to each distribution's LICENSE and NOTICE. Generated-by: GitHub Copilot (Claude Opus 5.5) Co-authored-by: Copilot App <[email protected]>
d8bb7f4 to
fff2c91
Compare
fff2c91 to
4d933aa
Compare
kevinjqliu
changed the title
Replace passing options through ext before each apply with a trackRuntimeDeps(configuration, path) function, so the call site is explicit and order-independent. When trackRuntimeDeps is not called, runtimeClasspath is still tracked in runtime-deps.txt. Generated-by: GitHub Copilot (Claude Opus 5.5) Co-authored-by: Copilot App <[email protected]>
4d933aa to
81c7047
Compare
| apply from: "${rootDir}/runtime-deps.gradle" | ||
| trackRuntimeDeps(configurations.runtimeClasspath, 'main/runtime-deps.txt') | ||
| trackRuntimeDeps(configurations.hive, 'hive/runtime-deps.txt') |
There was a problem hiding this comment.
i like this, this is the new way to run runtime-deps.gradle more than once and configurable
|
keeping this pr scoped to just the infra changes to create the 2 Will modify the dependencies and content in #18663 and with the related LICENSE/NOTICE changes |
Fokko
left a comment
Fokko
left a comment
There was a problem hiding this comment.
Thanks for scoping it down, LGTM
|
thanks for the review @Fokko, moving this forward so we can start working on the follow up PR to fix the LICENSE and NOTICE file. @danielcweeks lmk if you have any concerns, I'm happy to make follow up changes |
Adds
runtime-deps.txtto Kafka Connect runtime'shivedistribution. Follow up to #18019 which added the same to themaindistribution.Move existing
maindistribution'sruntime-deps.txtfromkafka-connect-runtime/runtime-deps.txt->kafka-connect-runtime/main/runtime-deps.txtruntime-deps.gradlenow hastrackRuntimeDeps(configuration, path), which kafka-connect calls once per distribution. Modules that don't call it still trackruntimeClasspathinruntime-deps.txt, so other modules are unaffected. The hive check runs as part ofcheckRuntimeDeps, so CI'scheckAllRuntimeDepscovers both.main vs hive
Hive is main plus 95 deps (Hive Metastore, Hadoop client, and transitives like Jetty, Netty, DataNucleus, ZooKeeper, Kerby), and resolves
commons-logging1.3 instead of 1.2 and nostax-api. It also ships bothlogback-classicandslf4j-reload4jas SLF4J bindings. That's pre-existing, just visible now. Dependency and LICENSE/NOTICE changes are in #18663; this PR only tracks both distributions.diff main/runtime-deps.txt hive/runtime-deps.txt
Testing
./gradlew checkAllRuntimeDeps -DallModules=truepasses, and./gradlew generateRuntimeDeps -DallModules=trueleaves every baseline unchanged. Removing a line from either baseline fails that baseline's check../gradlew :iceberg-kafka-connect:iceberg-kafka-connect-runtime:checkpasses, including the 14 integration tests against the installed main distribution.AI Disclosure