Repository navigation
Commit 3dc7341
authored
[fix](external) Preserve ES residual filters and repair scanner regression setup (apache#69650)
### What problem does this PR solve?
Related PR: apache#69614
`external_table_p2/es/test_es_file_scanner_v2.groovy` fails on its first
legacy-scanner baseline query with `Table [test1] does not exist in
database [default_db]`. Its interpolated port key is a Groovy `GString`,
while `otherConfigs` is a `Properties` containing `String` keys.
`Properties.get(Object)` returns null even when the port is configured,
producing `http://<es-host>:null`. Convert the key to `String` and fail
explicitly when the configuration is absent. The failure happens before
any V2 result comparison or BE scan.
The coverage review also found a query correctness issue shared by both
scanners: a nested AND can return a partial ES DSL and record an
unsupported child, but the single-expression path ignored that residual
tracking. For `(a = 1 AND local_predicate) OR a = 2`, the engine could
drop the local predicate; `NOT(a = 1 AND local_predicate)` could also
become an unsafe `NOT(a = 1)` at the source. Retain the complete
expression locally whenever conversion records a residual. Fully
pushable expressions and independent top-level AND conjuncts retain
their existing pushdown behavior.
Expand coverage in the same change:
- ES 5/6/7/8 V1/V2 comparisons across source/docvalue scans,
parallel/whole-index scrolling, and batch sizes 1/1024. Add
scalar/date/array/JSON, IN/NOT IN, null, raw ES query, residual
filtering, aggregation and LIMIT cases; restore every changed session
variable.
- An independent equivalent-query oracle for nested residual OR, since a
V1/V2 comparison alone cannot detect a shared FE error. Compare
cardinality for unordered LIMIT rather than document identity across
BEs.
- FE tests for residual tracking, docvalue eligibility and field-count
boundaries, alias/context serialization, shard planning and safe
early-stop parameters.
- BE tests for source-only fallback, missing/null docvalues and empty
arrays, real HTTP empty results, early close and scroll 404/500 cleanup.
Add an opt-in authenticated live-reader smoke test.
### Release note
Preserve local filtering for Elasticsearch compound predicates that
cannot be completely pushed down.
### Check List (For Author)
- Test
- [x] FE: all 127 ES connector tests pass. The two new nested OR/NOT
tests fail before the residual-tracking fix.
- [x] BE: all 27 selected ES ASAN tests pass against authenticated
Elasticsearch 9.0.0 and 9.5.5, including source/docvalue reads with
batch size 1.
- [x] FE Checkstyle for the ES module and its reactor dependencies;
clang-format 16, clang-tidy on changed C++ lines, and BE build-hygiene
checks.
- [x] Groovy 4.0.19: reproduce the null port before the fix; verify all
four port lookups, suite setup/matrix execution, missing configuration,
disabled execution and restoration after an injected query failure. The
setup probe substitutes SQL execution and does not validate SQL results.
- [ ] Full external ES 5/6/7/8 SQL regression run; requires the external
Docker fixtures and a Doris cluster.
- Behavior changed:
- [x] Yes: residual ES predicates remain enforced; regression setup
reads configured ports correctly.
- Does this need documentation?
- [x] No: existing catalog properties and scanner session variables are
unchanged.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label1 parent a626232 commit 3dc7341
5 files changed
Lines changed: 428 additions & 13 deletions
File tree
- be/test/format_v2/table
- fe/fe-connector/fe-connector-es/src
- main/java/org/apache/doris/connector/es
- test/java/org/apache/doris/connector/es
- regression-test/suites/external_table_p2/es
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
21 | 21 | | |
22 | 22 | | |
23 | 23 | | |
| 24 | + | |
| 25 | + | |
24 | 26 | | |
25 | 27 | | |
26 | 28 | | |
| |||
111 | 113 | | |
112 | 114 | | |
113 | 115 | | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
114 | 138 | | |
115 | 139 | | |
116 | 140 | | |
| |||
130 | 154 | | |
131 | 155 | | |
132 | 156 | | |
133 | | - | |
| 157 | + | |
134 | 158 | | |
135 | 159 | | |
136 | 160 | | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
137 | 177 | | |
138 | | - | |
| 178 | + | |
139 | 179 | | |
| 180 | + | |
140 | 181 | | |
141 | 182 | | |
142 | 183 | | |
| |||
537 | 578 | | |
538 | 579 | | |
539 | 580 | | |
| 581 | + | |
| 582 | + | |
| 583 | + | |
| 584 | + | |
| 585 | + | |
| 586 | + | |
| 587 | + | |
| 588 | + | |
| 589 | + | |
| 590 | + | |
| 591 | + | |
| 592 | + | |
| 593 | + | |
| 594 | + | |
| 595 | + | |
| 596 | + | |
| 597 | + | |
| 598 | + | |
| 599 | + | |
| 600 | + | |
| 601 | + | |
| 602 | + | |
| 603 | + | |
| 604 | + | |
| 605 | + | |
| 606 | + | |
| 607 | + | |
| 608 | + | |
| 609 | + | |
| 610 | + | |
| 611 | + | |
| 612 | + | |
| 613 | + | |
| 614 | + | |
| 615 | + | |
| 616 | + | |
| 617 | + | |
| 618 | + | |
| 619 | + | |
| 620 | + | |
| 621 | + | |
| 622 | + | |
| 623 | + | |
| 624 | + | |
| 625 | + | |
| 626 | + | |
| 627 | + | |
| 628 | + | |
| 629 | + | |
| 630 | + | |
| 631 | + | |
| 632 | + | |
| 633 | + | |
| 634 | + | |
| 635 | + | |
| 636 | + | |
| 637 | + | |
| 638 | + | |
| 639 | + | |
| 640 | + | |
| 641 | + | |
| 642 | + | |
| 643 | + | |
| 644 | + | |
| 645 | + | |
| 646 | + | |
| 647 | + | |
| 648 | + | |
| 649 | + | |
| 650 | + | |
| 651 | + | |
| 652 | + | |
| 653 | + | |
| 654 | + | |
| 655 | + | |
| 656 | + | |
| 657 | + | |
| 658 | + | |
| 659 | + | |
| 660 | + | |
| 661 | + | |
| 662 | + | |
| 663 | + | |
| 664 | + | |
| 665 | + | |
| 666 | + | |
| 667 | + | |
| 668 | + | |
| 669 | + | |
| 670 | + | |
| 671 | + | |
| 672 | + | |
| 673 | + | |
| 674 | + | |
| 675 | + | |
| 676 | + | |
| 677 | + | |
| 678 | + | |
| 679 | + | |
| 680 | + | |
| 681 | + | |
| 682 | + | |
| 683 | + | |
| 684 | + | |
| 685 | + | |
| 686 | + | |
| 687 | + | |
| 688 | + | |
| 689 | + | |
| 690 | + | |
| 691 | + | |
| 692 | + | |
| 693 | + | |
| 694 | + | |
| 695 | + | |
| 696 | + | |
| 697 | + | |
| 698 | + | |
| 699 | + | |
| 700 | + | |
| 701 | + | |
| 702 | + | |
| 703 | + | |
| 704 | + | |
| 705 | + | |
| 706 | + | |
| 707 | + | |
| 708 | + | |
| 709 | + | |
| 710 | + | |
| 711 | + | |
| 712 | + | |
| 713 | + | |
| 714 | + | |
| 715 | + | |
| 716 | + | |
| 717 | + | |
| 718 | + | |
| 719 | + | |
| 720 | + | |
| 721 | + | |
| 722 | + | |
| 723 | + | |
| 724 | + | |
| 725 | + | |
| 726 | + | |
| 727 | + | |
| 728 | + | |
| 729 | + | |
| 730 | + | |
| 731 | + | |
| 732 | + | |
| 733 | + | |
| 734 | + | |
| 735 | + | |
| 736 | + | |
| 737 | + | |
| 738 | + | |
| 739 | + | |
| 740 | + | |
| 741 | + | |
| 742 | + | |
| 743 | + | |
| 744 | + | |
| 745 | + | |
540 | 746 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
147 | 147 | | |
148 | 148 | | |
149 | 149 | | |
150 | | - | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
151 | 153 | | |
152 | 154 | | |
153 | 155 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
18 | 18 | | |
19 | 19 | | |
20 | 20 | | |
| 21 | + | |
21 | 22 | | |
22 | 23 | | |
| 24 | + | |
| 25 | + | |
23 | 26 | | |
24 | 27 | | |
| 28 | + | |
| 29 | + | |
25 | 30 | | |
26 | 31 | | |
27 | 32 | | |
28 | 33 | | |
| 34 | + | |
29 | 35 | | |
30 | 36 | | |
31 | 37 | | |
32 | 38 | | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
33 | 88 | | |
34 | 89 | | |
35 | 90 | | |
| |||
0 commit comments