Repository navigation
Commit 7af1754
committed
[fix](external) Preserve source values and alias semantics in ES scans
### What problem does this PR solve?
ES scans can return incorrect SQL results in two cases:
1. **Indexed values differ from document source values.** For a keyword field with `ignore_above=4`, a source value of `elephant` is readable but absent from the index. Pushing `field = 'elephant'` down as a term query loses a matching row before Doris can filter it. Normalizers, null substitutions and copied/runtime values can also produce false positives or incorrect null filtering. The existing docvalue eligibility check does not establish predicate equivalence, and explicit `doc_values=true` can bypass the `ignore_above` read fallback.
2. **Parallel scans replace an alias with its backing index.** This bypasses alias filters. For a two-index filtered alias in the regression fixture, disabling parallel scroll returns four rows, while enabling it returns all eight backing documents. Reusing the original multi-index alias separately for each shard would risk repeated scans when backing indices share shard IDs.
Derive source-value safety from every backing mapping. Keep affected SQL predicates local without sending a narrowing remote predicate, and read the affected fields from `_source`. Cover keyword subfields, `ignore_above`, normalizers, null substitutions, disabled indexing, explicit `ignore_malformed`, copy targets, runtime overrides and source filtering. Keep Boolean residual tracking and the existing early-LIMIT gate intact; explicit `esquery` retains native ES semantics.
When shard discovery resolves the requested name to backing indices, issue one scroll against the original target so Elasticsearch applies its alias filter and search routing. Physical indices retain shard-parallel scans. This conservative fix trades pushdown/docvalue optimization on affected fields and scan parallelism on aliases for correct results.
Add `test_es_source_semantics` with generated output. It compares source predicates against a Doris internal-table baseline and alias results against native ES queries, across both scanner versions, docvalues on/off and parallel scroll on/off. Cover multi-index aliases with different filters and shared shard IDs, routed/filtered aliases, empty results, full row counts, pagination, nested Boolean filters and LIMIT.
### Release note
Fix incorrect Elasticsearch query results caused by source/index mapping differences and filtered aliases in parallel scans. Alias reads now use one scroll against the original alias.
### Check List (For Author)
- Test
- [x] Four new FE test methods fail against the unmodified implementation before the fix.
- [x] ES connector suite: 135 reported tests, 134 passed and one optional live ES 9 compatibility class skipped; no failures or errors.
- [x] FE package build using `build.sh --fe` and Checkstyle for the ES module and reactor dependencies.
- [x] Real Elasticsearch 8.11.3 SQL regression: eight configurations, 288 generated result checks and 208 independent comparisons.
- [x] Unmodified connector reproduces missing keyword rows, normalized/null-value mismatches and alias over-reads against the same live fixtures; the new SQL suite fails on the missing keyword row.
- [x] Regression output generated by `run-regression-test.sh`, then verified by a separate run without output generation.
- Behavior changed:
- [x] Yes: unsafe mappings retain local filters/source reads; alias targets use one coordinated scroll.
- Does this need documentation?
- [x] No new SQL syntax or catalog/session properties.1 parent af9dad5 commit 7af1754
7 files changed
Lines changed: 1558 additions & 2 deletions
File tree
- fe/fe-connector/fe-connector-es/src
- main/java/org/apache/doris/connector/es
- test/java/org/apache/doris/connector/es
- regression-test
- data/external_table_p2/es
- suites/external_table_p2/es
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
19 | 19 | | |
20 | 20 | | |
21 | 21 | | |
| 22 | + | |
22 | 23 | | |
23 | 24 | | |
| 25 | + | |
24 | 26 | | |
25 | 27 | | |
26 | 28 | | |
| |||
39 | 41 | | |
40 | 42 | | |
41 | 43 | | |
| 44 | + | |
| 45 | + | |
42 | 46 | | |
43 | 47 | | |
44 | 48 | | |
| |||
61 | 65 | | |
62 | 66 | | |
63 | 67 | | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
64 | 72 | | |
65 | 73 | | |
66 | 74 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
153 | 153 | | |
154 | 154 | | |
155 | 155 | | |
| 156 | + | |
156 | 157 | | |
157 | 158 | | |
158 | 159 | | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
159 | 223 | | |
160 | 224 | | |
161 | 225 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
46 | 46 | | |
47 | 47 | | |
48 | 48 | | |
| 49 | + | |
49 | 50 | | |
50 | 51 | | |
51 | 52 | | |
| |||
104 | 105 | | |
105 | 106 | | |
106 | 107 | | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
107 | 118 | | |
108 | 119 | | |
109 | 120 | | |
| |||
116 | 127 | | |
117 | 128 | | |
118 | 129 | | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
119 | 134 | | |
120 | 135 | | |
121 | 136 | | |
| |||
143 | 158 | | |
144 | 159 | | |
145 | 160 | | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
146 | 166 | | |
147 | 167 | | |
148 | 168 | | |
| |||
155 | 175 | | |
156 | 176 | | |
157 | 177 | | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
158 | 195 | | |
159 | 196 | | |
160 | 197 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
131 | 131 | | |
132 | 132 | | |
133 | 133 | | |
134 | | - | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
135 | 139 | | |
136 | 140 | | |
137 | 141 | | |
| |||
270 | 274 | | |
271 | 275 | | |
272 | 276 | | |
273 | | - | |
| 277 | + | |
| 278 | + | |
| 279 | + | |
274 | 280 | | |
275 | 281 | | |
276 | 282 | | |
| |||
0 commit comments