Add support for streaming ingestion with concurrent replace

---------

Co-authored-by: Kashif Faraz <[email protected]>

This PR addresses a bug with waiting for segments to be loaded. In the case of append, segments would be created with the same version. This caused the number of segments returned to be incorrect.

This PR changes this to keep track of the range of partition numbers as well for each version, which lets the task wait for the correct set of segments. The partition numbers are expected to be continuous since the task obtains the lock for the segment while running.

* Added concurrent compaction switches

…ator (#15145)

Fixes a bug when executing queries with the ordering of arrays

* Separate k8s and druid task lifecycles

* Remove extra log lines

* Fix unit tests

* fix unit tests

* Fix unit tests

* notify listeners on task completion

* Fix unit test

* unused var

* PR changes

* Fix unit tests

* Fix checkstyle

* PR changes

Patch adds an undocumented parameter taskLockType to MSQ so that we can start enabling this feature for users who are interested in testing the new lock types.

…" inside of lists and arrays (#15190)

…5182)

* add native filters for "(filter) is true" and "(filter) is false"

changes:
* add IsTrueDimFilter, IsFalseDimFilter, and abstract IsBooleanDimFilter for native json filter implementations of `(filter) IS TRUE` and `(filter) IS FALSE`
* add IsBooleanFilter for actual filtering logic for these filters, which ignore includeUnknown to always use matches with false for true and !matches with true for false
* fix test incorrectly adjusted to wrong answer in #15058
* add tests for default value mode

…15192)

* Attach user identity to router request logs

* Add test

* More tests

…gnature (#15183)

…ck (#15085)

* Replacing tasks must read segments created before they acquired their locks

This PR:

adds a flag to JsonToParquet to do the fix during conversion
updates the json files to more correct conents
some resultset mismatches were fixed by this
updates parquet to 1.13.1

* Proper value for taskLockType in streaming ingestion with concurrent compaction

* use setup-java everywhere for consistency

* add Java 21 to integration test matrix

* simplify docker build containers script + add Java 21

* fix for Java versions reporting 21-ea

Release notes: https://downloads.apache.org/kafka/3.6.0/RELEASE_NOTES.html
https://kafka.apache.org/blog#apache_kafka_360_release_announcement

…al replace (#15220)

A SegmentTransactionReplaceAction must only update the mapping of tasks with append locks that are running concurrently. To ensure this, we return the supervisor id only if it has the taskLockType as APPEND in its context.

…15216)

for some exotic queries like:

  SELECT
  	'_'||dim1,
    MIN(cast(0 as double)) OVER (),
    MIN(cast((cnt||cnt) as bigint)) OVER ()
  FROM foo
the compilation have resulted in NPE -s mostly because VirtualColumn -s were not handled properly

#15118)

* Filter pending segments upgraded with transactional replace

* Push sequence name filter to metadata query

…g-segment retry bug. (#15260)

* Fix NPE caused by realtime segment closing race, fix possible missing-segment retry bug.

Fixes #12168, by returning empty from FireHydrant when the segment is
swapped to null. This causes the SinkQuerySegmentWalker to use
ReportTimelineMissingSegmentQueryRunner, which causes the Broker to look
for the segment somewhere else.

In addition, this patch changes SinkQuerySegmentWalker to acquire references
to all hydrants (subsegments of a sink) at once, and return a
ReportTimelineMissingSegmentQueryRunner if *any* of them could not be acquired.
I suspect, although have not confirmed, that the prior behavior could lead to
segments being reported as missing even though results from some hydrants were
still included.

* Some more test coverage.

Co-authored-by: Victoria Lim <[email protected]>
Co-authored-by: George Shiqi Wu <[email protected]>
Co-authored-by: 317brian <[email protected]>
Co-authored-by: ythorat2 <[email protected]>
Co-authored-by: Krishna Anandan <[email protected]>
Co-authored-by: Vadim Ogievetsky <[email protected]>
Co-authored-by: Abhishek Radhakrishnan <[email protected]>
Co-authored-by: Karan Kumar <[email protected]>
Co-authored-by: Rishabh Singh <[email protected]>
Co-authored-by: Magnus Henoch <[email protected]>
Co-authored-by: AmatyaAvadhanula <[email protected]>
Co-authored-by: Charles Smith <[email protected]>
Co-authored-by: Yashdeep Thorat <[email protected]>
Co-authored-by: Atul Mohan <[email protected]>
Co-authored-by: Clint Wylie <[email protected]>
Co-authored-by: Gian Merlino <[email protected]>

* Add test to verify sequence name of Kafka and Kinesis tasks

* fix physical memory detection on OSX

* typo

Fixes #15393

…ize (#15409)

Currently when we submit a task to druid and number of currently active tasks has already reached (druid.indexer.queue.maxSize) then 500 ISE is thrown as per shown in the screenshot in #15380.

This fix will return HTTP 429 Too Many Requests(with proper error message) instead of 500 ISE, when we submit a task and queueSize has reached.

#15363)

Patched security vulnerability by updating Ranger libraries to the newest available version.

Changes
- Suppress CVE-2023-36478 as there is no newer Hadoop version available that addresses
- Suppress CVE-2023-31582 in jose4j. Pulled in by Kubernetes/Kafka but not addressed yet.

…odyWriter not found error on Middle Manager (#15412)

Fixes missing task failure error message on Overlord.

The error message was missing since TaskManagementResource#assignTask API wasn't annotated with @produces(MediaType.APPLICATION_JSON) resulting in the response being treated as application/octet-stream, that in turn lead to MessageBodyWriter not found error on the middle manager. The exception is not logged on the middle manager itself since it happens even before entering the assignTask function -- while mapping arg Task -> MSQControllerTask.

…ion (#15421)

This PR fixes an issue where the grouping aggregator wrongly assumes that a key dimension is a virtual column and assigns a wrong name to it. This results in a mismatch between the dimensions that grouping aggregator sees and the dimension names that rows are aggregated on. And finally, grouping aggregator generates wrong result.

#15424)

Changes:
- Fix log `Got end of partition marker for partition [%s] from task [%s] in discoverTasks`
by fixing order of args
- Simplify in-line classes by using lambda
- Update kill task message from `Task [%s] failed to respond to [set end offsets]
 in a timely manner, killing task` to `Failed to set end offsets, killing task`
- Clean up tests

…ew actions being available on the overlord. (#15430)

* Fixing failing compaction/parallel index jobs during upgrade due to new actions not available on the overlord.

* Fixing build

* Removing extra space.

* Fixing json getter.

* Review comments.

…14990)

* Improved time-chart brush and added auto-granularity

* prettier

* added highlight bubble to explore visualizations

* Added licenses and fixes from PR review

* added missing files...

* Use StubServiceEmitter in tests
* Remove unthrown exception from declaration

Adds a result comparision mode of EQUALS_RELATIVE_1000_ULPS ; which accepts floating point differences up-to 1000 units of least precision

This updates the prod section of the release readme for the website steps. Previously, only staging was updated

I think this is a problem as it discards the false return value when the putToKeyBuffer can't store the value because of the limit

Not forwarding the return value at that point may lead to the normal continuation here regardless something was not added to the dictionary like here

* Fix mmless ingestion and index tasks

* Move comment

* remove dup test

* auto but.. with an expected type

* update confluent's dependencies to common, supported version

  Update io.confluent.* dependencies to common, updated version 6.2.12
currently used versions are EOL

* move version definition to the top level pom

…15302)

This PR revives #14978 with a few more bells and whistles. Instead of an unconditional cross-join, we will now split the join condition such that some conditions are now evaluated post-join. To decide what sub-condition goes where, I have refactored DruidJoinRule class to extract unsupported sub-conditions. We build a postJoinFilter out of these unsupported sub-conditions and push to the join.