Repository navigation
Commit cc55086
authored
[feature](external) Support Elasticsearch HTTP scans in FileScannerV2 (apache#69614)
### What problem does this PR solve?
Related PR: apache#69601
Problem Summary:
Elasticsearch HTTP scans always use FileScanner v1, even when
`enable_file_scanner_v2` is enabled. Add an ES TableReader/FileReader
pair and route `FORMAT_ES_HTTP` queries through FileScannerV2 when that
session option is enabled.
The reader exposes a file-local schema, uses MaterializedColumnMapper
for projection and predicate localization, and applies residual filters
with v2 row accounting. It reuses the existing ES query builder, HTTP
scroll transport, authentication, SSL settings, catalog-provided
endpoints, and type conversions. Shared host selection and materialized
complex-schema helpers keep the two scanner paths consistent.
Parse the initial cached response during open so early
LIMIT/cancellation can clear its scroll context. Close the stream on
EOF, cancellation, open/read errors, and split changes, and report close
failures once.
The RLE header reach baseline increases by two for the new ES reader and
test translation units, which use the existing table/expr header
closure. The shared parser retains its existing type dispatch, with a
documented complexity suppression on the new projection overload.
### Release note
Support Elasticsearch HTTP queries in FileScannerV2 when
`enable_file_scanner_v2` is enabled. Disabling the option continues to
select FileScanner v1.
### Check List (For Author)
- Test
- [x] Unit Test: 61 ASAN BE tests covering ES v2 projection, doc values,
`_id`, nullable values/arrays, multi-page batching, zero-column row
counting, residual filtering, split advancement, cancellation, error
cleanup, and a real local HTTP scroll server; existing scanner dispatch,
legacy ES, and Remote Doris tests also pass.
- [ ] Regression test: added ES 5/6/7/8 v1/v2 differential coverage with
source/docvalue scans and small batches. Groovy syntax checked; live
external ES regression not run locally.
- [x] clang-format 16, clang-tidy on changed C++ lines, and BE
build-hygiene checks.
- Behavior changed:
- [x] Yes. ES queries honor the v2 session option.
- Does this need documentation?
- [x] No. Uses the existing scanner session option and ES catalog
properties.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label1 parent 708531d commit cc55086
19 files changed
Lines changed: 1182 additions & 115 deletions
File tree
- be
- src
- exec
- operator
- scan
- format_v2
- table
- format/table/es
- test
- exec/scan
- format_v2/table
- build-support
- regression-test/suites/external_table_p2/es
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
224 | 224 | | |
225 | 225 | | |
226 | 226 | | |
227 | | - | |
228 | 227 | | |
229 | 228 | | |
230 | 229 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
60 | 60 | | |
61 | 61 | | |
62 | 62 | | |
| 63 | + | |
63 | 64 | | |
64 | 65 | | |
65 | 66 | | |
| |||
356 | 357 | | |
357 | 358 | | |
358 | 359 | | |
| 360 | + | |
| 361 | + | |
359 | 362 | | |
360 | 363 | | |
361 | 364 | | |
| |||
801 | 804 | | |
802 | 805 | | |
803 | 806 | | |
| 807 | + | |
| 808 | + | |
804 | 809 | | |
805 | 810 | | |
806 | 811 | | |
| |||
1092 | 1097 | | |
1093 | 1098 | | |
1094 | 1099 | | |
| 1100 | + | |
| 1101 | + | |
| 1102 | + | |
1095 | 1103 | | |
1096 | 1104 | | |
1097 | 1105 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
19 | 19 | | |
20 | 20 | | |
21 | 21 | | |
22 | | - | |
23 | | - | |
24 | | - | |
| 22 | + | |
25 | 23 | | |
26 | 24 | | |
27 | 25 | | |
28 | 26 | | |
29 | 27 | | |
30 | 28 | | |
31 | 29 | | |
32 | | - | |
33 | 30 | | |
34 | 31 | | |
35 | 32 | | |
| |||
65 | 62 | | |
66 | 63 | | |
67 | 64 | | |
68 | | - | |
69 | | - | |
70 | | - | |
71 | | - | |
72 | | - | |
73 | | - | |
74 | | - | |
75 | | - | |
76 | | - | |
77 | | - | |
78 | | - | |
79 | | - | |
80 | | - | |
81 | | - | |
82 | | - | |
83 | | - | |
84 | | - | |
85 | | - | |
86 | | - | |
87 | | - | |
88 | | - | |
89 | | - | |
90 | | - | |
| 65 | + | |
91 | 66 | | |
92 | 67 | | |
93 | 68 | | |
| |||
225 | 200 | | |
226 | 201 | | |
227 | 202 | | |
228 | | - | |
229 | | - | |
230 | | - | |
231 | | - | |
232 | | - | |
233 | | - | |
234 | | - | |
235 | | - | |
236 | | - | |
237 | | - | |
238 | | - | |
239 | | - | |
240 | | - | |
241 | | - | |
242 | | - | |
243 | | - | |
244 | | - | |
245 | | - | |
246 | | - | |
247 | | - | |
248 | | - | |
249 | | - | |
250 | | - | |
251 | | - | |
252 | | - | |
253 | | - | |
| 203 | + | |
254 | 204 | | |
255 | 205 | | |
256 | 206 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
19 | 19 | | |
20 | 20 | | |
21 | 21 | | |
| 22 | + | |
| 23 | + | |
22 | 24 | | |
23 | 25 | | |
24 | 26 | | |
| |||
28 | 30 | | |
29 | 31 | | |
30 | 32 | | |
| 33 | + | |
31 | 34 | | |
32 | 35 | | |
33 | 36 | | |
| |||
266 | 269 | | |
267 | 270 | | |
268 | 271 | | |
| 272 | + | |
| 273 | + | |
| 274 | + | |
| 275 | + | |
| 276 | + | |
| 277 | + | |
| 278 | + | |
| 279 | + | |
| 280 | + | |
| 281 | + | |
| 282 | + | |
| 283 | + | |
| 284 | + | |
| 285 | + | |
| 286 | + | |
| 287 | + | |
| 288 | + | |
| 289 | + | |
| 290 | + | |
| 291 | + | |
| 292 | + | |
| 293 | + | |
| 294 | + | |
| 295 | + | |
| 296 | + | |
| 297 | + | |
| 298 | + | |
| 299 | + | |
| 300 | + | |
| 301 | + | |
| 302 | + | |
| 303 | + | |
| 304 | + | |
| 305 | + | |
| 306 | + | |
| 307 | + | |
| 308 | + | |
| 309 | + | |
| 310 | + | |
| 311 | + | |
| 312 | + | |
| 313 | + | |
| 314 | + | |
| 315 | + | |
| 316 | + | |
| 317 | + | |
| 318 | + | |
| 319 | + | |
| 320 | + | |
| 321 | + | |
| 322 | + | |
| 323 | + | |
| 324 | + | |
| 325 | + | |
| 326 | + | |
| 327 | + | |
269 | 328 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
58 | 58 | | |
59 | 59 | | |
60 | 60 | | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
61 | 64 | | |
62 | 65 | | |
63 | 66 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
662 | 662 | | |
663 | 663 | | |
664 | 664 | | |
| 665 | + | |
| 666 | + | |
| 667 | + | |
| 668 | + | |
| 669 | + | |
| 670 | + | |
| 671 | + | |
| 672 | + | |
| 673 | + | |
| 674 | + | |
| 675 | + | |
665 | 676 | | |
666 | 677 | | |
667 | 678 | | |
| |||
681 | 692 | | |
682 | 693 | | |
683 | 694 | | |
684 | | - | |
685 | | - | |
686 | | - | |
| 695 | + | |
| 696 | + | |
| 697 | + | |
687 | 698 | | |
688 | 699 | | |
689 | 700 | | |
| |||
887 | 898 | | |
888 | 899 | | |
889 | 900 | | |
890 | | - | |
891 | | - | |
892 | | - | |
893 | | - | |
| 901 | + | |
| 902 | + | |
| 903 | + | |
| 904 | + | |
894 | 905 | | |
895 | 906 | | |
896 | 907 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
20 | 20 | | |
21 | 21 | | |
22 | 22 | | |
| 23 | + | |
23 | 24 | | |
24 | 25 | | |
25 | 26 | | |
| |||
30 | 31 | | |
31 | 32 | | |
32 | 33 | | |
| 34 | + | |
33 | 35 | | |
34 | 36 | | |
35 | 37 | | |
| |||
42 | 44 | | |
43 | 45 | | |
44 | 46 | | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
45 | 54 | | |
46 | 55 | | |
47 | 56 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
60 | 60 | | |
61 | 61 | | |
62 | 62 | | |
| 63 | + | |
63 | 64 | | |
64 | 65 | | |
65 | 66 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
21 | 21 | | |
22 | 22 | | |
23 | 23 | | |
| 24 | + | |
| 25 | + | |
24 | 26 | | |
| 27 | + | |
25 | 28 | | |
26 | 29 | | |
27 | 30 | | |
| |||
36 | 39 | | |
37 | 40 | | |
38 | 41 | | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
39 | 53 | | |
40 | 54 | | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
41 | 88 | | |
42 | 89 | | |
43 | 90 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
18 | 18 | | |
19 | 19 | | |
20 | 20 | | |
| 21 | + | |
21 | 22 | | |
22 | 23 | | |
23 | 24 | | |
| |||
33 | 34 | | |
34 | 35 | | |
35 | 36 | | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
36 | 41 | | |
37 | 42 | | |
38 | 43 | | |
| |||
0 commit comments