Skip to content

feat: log total bytes gathered when max scatter-gather bytes limit is reached#18841

Merged
abhishekrb19 merged 1 commit into
apache:masterfrom
TessaIO:feat-log-bytes-gathered-from-historicals
Dec 17, 2025
Merged

feat: log total bytes gathered when max scatter-gather bytes limit is reached#18841
abhishekrb19 merged 1 commit into
apache:masterfrom
TessaIO:feat-log-bytes-gathered-from-historicals

Conversation

@TessaIO

@TessaIO TessaIO commented Dec 12, 2025

Copy link
Copy Markdown
Contributor

Fixes #18840.

Description

log total bytes gathered when max scatter-gather bytes limit is reached

Release note


log total bytes gathered when max scatter-gather bytes limit is reached

This PR has:

  • been self-reviewed.
  • added documentation for new or modified features or behaviors.
  • a release note entry in the PR description.
  • added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
  • added or updated version, license, or notice information in licenses.yaml
  • added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
  • added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
  • added integration tests.
  • been tested in a test Druid cluster.

@abhishekrb19 abhishekrb19 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR!

It looks like jacaco is failing due to insufficient line coverage...

DirectDruidClientTest uses a bunch of mocks. I’m wondering if we could add a new test using TestHttpClient to exercise the internals of this class, including HttpResponseHandler and cover these paths.

If that turns into too much effort, I’m also fine skipping jacaco coverage for this PR since this is primarily a logging change, and we can address the test coverage debt for this class separately.

Please let me know what you think @TessaIO

Comment thread server/src/main/java/org/apache/druid/client/DirectDruidClient.java Outdated
@TessaIO
TessaIO force-pushed the feat-log-bytes-gathered-from-historicals branch from cf52281 to faebda5 Compare December 13, 2025 17:22
@TessaIO

TessaIO commented Dec 13, 2025

Copy link
Copy Markdown
Contributor Author

@abhishekrb19 thanks for the suggestion, I think it would be cleaner if the test coverage is addressed in a separate PR as this one is solely focused on improving the logs. wdyt?

@TessaIO
TessaIO force-pushed the feat-log-bytes-gathered-from-historicals branch from faebda5 to 68b7a6e Compare December 13, 2025 17:24

@abhishekrb19 abhishekrb19 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@TessaIO could you look into the CI test failures? The test client doesn't call responseContext.initialize() - we can replace the code block as follows:

  protected static ResponseContext responseContext()
  {
    return DirectDruidClient.makeResponseContextForQuery();
  }

The condition maxScatterGatherBytes < Long.MAX_VALUE was previously masking this behavior.

Comment thread server/src/main/java/org/apache/druid/client/DirectDruidClient.java
@TessaIO
TessaIO force-pushed the feat-log-bytes-gathered-from-historicals branch from 68b7a6e to ad827a3 Compare December 16, 2025 23:45
@TessaIO
TessaIO requested a review from abhishekrb19 December 16, 2025 23:45
@TessaIO

TessaIO commented Dec 16, 2025

Copy link
Copy Markdown
Contributor Author

@abhishekrb19 thanks for the review, updated!

Comment thread server/src/main/java/org/apache/druid/client/DirectDruidClient.java Outdated
@TessaIO
TessaIO force-pushed the feat-log-bytes-gathered-from-historicals branch from ad827a3 to 9a7761d Compare December 17, 2025 00:38
@TessaIO

TessaIO commented Dec 17, 2025

Copy link
Copy Markdown
Contributor Author

@abhishekrb19 done

@abhishekrb19 abhishekrb19 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lgtm, thanks!

@abhishekrb19
abhishekrb19 merged commit 582d61c into apache:master Dec 17, 2025
55 checks passed
abhishekrb19 added a commit that referenced this pull request Jan 3, 2026
Test-only change:

- Increases test coverage for the DirectDruidClient class from 57%, 35%, 16% to 76%, 73%, 52% for method / line / branch coverage respectively.
- Replaces mocks with more concrete classes and test helpers, allowing the internals of the class, including HttpResponseHandler to be exercised.
- Also added a test for feat: log total bytes gathered when max scatter-gather bytes limit is reached #18841 for which coverage had to be skipped
@kgyrtkirk kgyrtkirk added this to the 36.0.0 milestone Jan 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Log the number of bytes returned from historical when max scatter-gather is reached

3 participants