Skip to content

Shrink Dependencies#19002

Open
aruraghuwanshi wants to merge 4 commits into
apache:masterfrom
aruraghuwanshi:shrink-dependencies
Open

Shrink Dependencies#19002
aruraghuwanshi wants to merge 4 commits into
apache:masterfrom
aruraghuwanshi:shrink-dependencies

Conversation

@aruraghuwanshi

Copy link
Copy Markdown
Contributor

Fixes #18998.

Description

PR #18990 introduced the requireUpperBoundDeps enforcer rule to catch transitive dependency version inconsistencies. However, it needed 38 exclusions to pass, since many libraries pull in conflicting transitive versions. This PR shrinks that list from 38 down to 18 by bumping dependency versions to satisfy the upper bound constraints.

Bumped dependency versions

The following dependencies were bumped to the highest version required by their transitive dependency tree. All bumps are patch or minor version increments with no expected API breakages:

Dependency Old New
Netty 4 4.2.6.Final 4.2.7.Final
Error Prone annotations 2.35.1 2.36.0
Caffeine 2.8.0 2.9.3
Scala library 2.13.14 2.13.16
Google HTTP Client 1.42.3 1.43.3
Commons IO 2.17.0 2.21.0
Commons Compress 1.27.0 1.28.0
Commons Collections4 4.2 4.4
Commons Codec 1.17.1 1.19.0
Bouncy Castle (bcpkix) 1.81 1.82
Zstd JNI 1.5.2-3 1.5.6-6
HttpClient 4.x 4.5.13 4.5.14

Added new managed dependencies

These transitive dependencies were not previously managed in the root POM, causing version conflicts. Adding them to dependencyManagement centralizes version control and removes the need for exclusions:

  • kotlin-stdlib-jdk8 (1.9.25) — aligns with existing kotlin-stdlib
  • proto-google-common-protos (2.48.0) — version moved from indexing-service/pom.xml
  • jakarta.activation-api (1.2.2) — fixes conflict between docker-java and jetty
  • httpclient5 (5.5.1) — version moved from embedded-tests/pom.xml
  • httpcore5 (5.3.6) — required by httpclient5 5.5.1

Child POM cleanup

Removed hardcoded versions from child modules that are now centrally managed:

  • indexing-service/pom.xml: proto-google-common-protos version removed
  • embedded-tests/pom.xml: httpclient5 version removed

Remaining exclusions

The 18 remaining exclusions all require major version bumps or involve complex dependency trees that need separate, focused work:

  • Jackson 2.20.x — needed by docker-java 3.7.0 and fabric8
  • Guava 33.x — needed by google-cloud libraries
  • Guice 7.0.0 — needed by fabric8
  • Zookeeper 3.9.x — needed by Curator 5.8.0
  • Jedis 7.0.0 — needed by jedis-mock
  • RoaringBitmap 1.2.0, Aircompressor 2.0.2, OkIO 3.x — major version jumps
  • Jakarta Activation 2.x — Jakarta EE 9 namespace migration
  • Reflections 0.10.2 — API breaking changes
  • Confluent stack (snakeyaml, kafka-clients, kafka-schema-registry-client) — Confluent-specific versioning

Each remaining exclusion is documented inline with a comment explaining why it is still needed.

Release note

Upgraded multiple transitive dependency versions (Netty, Commons IO, Commons Compress, Caffeine, Bouncy Castle, and others) and reduced the number of requireUpperBoundDeps enforcer exclusions from 38 to 18. No user-facing behavior changes.


Key changed/added classes in this PR
  • pom.xml — version bumps, new managed dependencies, reduced enforcer exclusions
  • indexing-service/pom.xml — removed hardcoded proto-google-common-protos version
  • embedded-tests/pom.xml — removed hardcoded httpclient5 version

This PR has:

  • been self-reviewed.
  • added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.

@kgyrtkirk

Copy link
Copy Markdown
Member

thank you for taking care of this!
iirc BlockCompressedPayloadWriterReaderTest is broken by the zstd upgrade - that was the tripping pointwhen I started getting more defensive and add that massive number of exclusions to at least enable this thing :)

@aruraghuwanshi

aruraghuwanshi commented Feb 10, 2026

Copy link
Copy Markdown
Contributor Author

Yeah I plan on trying to address these broken checks shortly. You should see updated to this PR soon, and hopefully everything passes 😅

Update licenses.yaml for all 9 bumped dependency versions:
Caffeine 2.9.3, Error Prone 2.36.0, Commons Codec 1.19.0,
Commons IO 2.21.0, Commons Compress 1.28.0, Bouncy Castle 1.82,
Netty 4.2.7.Final, HttpClient 4.5.14, Zstd JNI 1.5.6-6

Addresses static-checks-maven and validate-dist failures.
Partial fix for apache#18998
Comment thread pom.xml
<groupId>com.github.luben</groupId>
<artifactId>zstd-jni</artifactId>
<version>1.5.2-3</version>
<version>1.5.6-6</version>

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Update ZSTD size-sensitive tests for the zstd-jni bump

Bumping zstd-jni to 1.5.6-6 changes the native compressor used by the ZSTD test path, but the PR leaves the exact serialized-size assertions in BlockCompressedPayloadWriterReaderTest and CellWriterReaderTest unchanged. The prepared packet already notes BlockCompressedPayloadWriterReaderTest is broken by this upgrade, so the processing tests remain failing unless the expected sizes are updated or the ZSTD assertions stop depending on exact compressed length.

Comment thread pom.xml
</dependency>
<dependency>
<groupId>com.google.api.grpc</groupId>
<artifactId>proto-google-common-protos</artifactId>

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Update licenses.yaml for newly managed transitive versions

This PR manages proto-google-common-protos at 2.48.0 and kotlin-stdlib-jdk8 at 1.9.25, but licenses.yaml still registers proto-google-common-protos as 2.28.0 and kotlin-stdlib-jdk8 under the 1.8.21 entry. distribution/bin/check-licenses.py keys licenses by exact groupId/artifactId/version, so the apache-release license check can report these new versions as missing unless licenses.yaml is updated.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Shrink down the number of dependencies excluded from version enforcement

3 participants