Skip to content

Commit 7af1754

Browse files
committed
[fix](external) Preserve source values and alias semantics in ES scans
### What problem does this PR solve? ES scans can return incorrect SQL results in two cases: 1. **Indexed values differ from document source values.** For a keyword field with `ignore_above=4`, a source value of `elephant` is readable but absent from the index. Pushing `field = 'elephant'` down as a term query loses a matching row before Doris can filter it. Normalizers, null substitutions and copied/runtime values can also produce false positives or incorrect null filtering. The existing docvalue eligibility check does not establish predicate equivalence, and explicit `doc_values=true` can bypass the `ignore_above` read fallback. 2. **Parallel scans replace an alias with its backing index.** This bypasses alias filters. For a two-index filtered alias in the regression fixture, disabling parallel scroll returns four rows, while enabling it returns all eight backing documents. Reusing the original multi-index alias separately for each shard would risk repeated scans when backing indices share shard IDs. Derive source-value safety from every backing mapping. Keep affected SQL predicates local without sending a narrowing remote predicate, and read the affected fields from `_source`. Cover keyword subfields, `ignore_above`, normalizers, null substitutions, disabled indexing, explicit `ignore_malformed`, copy targets, runtime overrides and source filtering. Keep Boolean residual tracking and the existing early-LIMIT gate intact; explicit `esquery` retains native ES semantics. When shard discovery resolves the requested name to backing indices, issue one scroll against the original target so Elasticsearch applies its alias filter and search routing. Physical indices retain shard-parallel scans. This conservative fix trades pushdown/docvalue optimization on affected fields and scan parallelism on aliases for correct results. Add `test_es_source_semantics` with generated output. It compares source predicates against a Doris internal-table baseline and alias results against native ES queries, across both scanner versions, docvalues on/off and parallel scroll on/off. Cover multi-index aliases with different filters and shared shard IDs, routed/filtered aliases, empty results, full row counts, pagination, nested Boolean filters and LIMIT. ### Release note Fix incorrect Elasticsearch query results caused by source/index mapping differences and filtered aliases in parallel scans. Alias reads now use one scroll against the original alias. ### Check List (For Author) - Test - [x] Four new FE test methods fail against the unmodified implementation before the fix. - [x] ES connector suite: 135 reported tests, 134 passed and one optional live ES 9 compatibility class skipped; no failures or errors. - [x] FE package build using `build.sh --fe` and Checkstyle for the ES module and reactor dependencies. - [x] Real Elasticsearch 8.11.3 SQL regression: eight configurations, 288 generated result checks and 208 independent comparisons. - [x] Unmodified connector reproduces missing keyword rows, normalized/null-value mismatches and alias over-reads against the same live fixtures; the new SQL suite fails on the missing keyword row. - [x] Regression output generated by `run-regression-test.sh`, then verified by a separate run without output generation. - Behavior changed: - [x] Yes: unsafe mappings retain local filters/source reads; alias targets use one coordinated scroll. - Does this need documentation? - [x] No new SQL syntax or catalog/session properties.
1 parent af9dad5 commit 7af1754

7 files changed

Lines changed: 1558 additions & 2 deletions

File tree

‎fe/fe-connector/fe-connector-es/src/main/java/org/apache/doris/connector/es/EsFieldContext.java‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,8 +19,10 @@
1919

2020
import java.util.ArrayList;
2121
import java.util.HashMap;
22+
import java.util.HashSet;
2223
import java.util.List;
2324
import java.util.Map;
25+
import java.util.Set;
2426

2527
/**
2628
* Holds field-level context derived from ES index mapping analysis.
@@ -39,6 +41,8 @@ public class EsFieldContext {
3941
private final Map<String, String> fetchFieldsContext = new HashMap<>();
4042
private final Map<String, String> docValueFieldsContext = new HashMap<>();
4143
private final List<String> needCompatDateFields = new ArrayList<>();
44+
private final Set<String> sourceValueUnsafeFields = new HashSet<>();
45+
4246
private final Map<String, String> column2typeMap = new HashMap<>();
4347

4448
public Map<String, String> getFetchFieldsContext() {
@@ -61,6 +65,10 @@ public Map<String, String> getColumn2typeMap() {
6165
return column2typeMap;
6266
}
6367

68+
public Set<String> getSourceValueUnsafeFields() {
69+
return sourceValueUnsafeFields;
70+
}
71+
6472
@Override
6573
public String toString() {
6674
return "EsFieldContext{"

‎fe/fe-connector/fe-connector-es/src/main/java/org/apache/doris/connector/es/EsMappingUtils.java‎

Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -153,9 +153,73 @@ public static EsFieldContext resolveFieldContext(List<String> columnNames,
153153
resolveKeywordFields(fieldContext, fieldObject, colName, fieldType);
154154
resolveDocValuesFields(fieldContext, fieldObject, colName, fieldType);
155155
}
156+
resolveSourceValueSafety(fieldContext, indexMapping, mappingType);
156157
return fieldContext;
157158
}
158159

160+
private static void resolveSourceValueSafety(EsFieldContext context, String indexMapping, String mappingType) {
161+
try {
162+
// An alias can span mappings. Every backing index must preserve source semantics before
163+
// a predicate may remove rows or doc_values may replace the value returned to Doris.
164+
for (JsonNode index : MAPPER.readTree(indexMapping)) {
165+
ObjectNode root = getRootSchema((ObjectNode) index.get("mappings"), mappingType, new ArrayList<>());
166+
JsonNode properties = root.path("properties");
167+
JsonNode source = root.path("_source");
168+
boolean filteredSource = !source.path("enabled").asBoolean(true)
169+
|| source.has("includes") || source.has("excludes");
170+
Set<String> copyTargets = new HashSet<>();
171+
collectCopyTargets(properties, copyTargets);
172+
Set<String> columns = new HashSet<>(context.getColumn2typeMap().keySet());
173+
properties.fieldNames().forEachRemaining(columns::add);
174+
for (String column : columns) {
175+
JsonNode field = properties.path(column);
176+
String indexedPath = context.getFetchFieldsContext().get(column);
177+
JsonNode indexedField = field;
178+
if (indexedPath != null) {
179+
indexedField = field.path("fields").path(indexedPath.substring(column.length() + 1));
180+
}
181+
if (filteredSource || field.isMissingNode() || root.path("runtime").has(column)
182+
|| copyTargets.contains(column) || changesSourceValue(field)
183+
|| changesSourceValue(indexedField) || hasUnsafeKeywordSubfield(field)) {
184+
context.getSourceValueUnsafeFields().add(column);
185+
}
186+
}
187+
}
188+
} catch (IOException e) {
189+
throw new DorisConnectorException("Failed to parse ES mapping: " + e.getMessage());
190+
}
191+
// Local residuals must read the original value, including with explicit doc_values=true.
192+
context.getSourceValueUnsafeFields().forEach(context.getDocValueFieldsContext()::remove);
193+
}
194+
195+
private static boolean hasUnsafeKeywordSubfield(JsonNode field) {
196+
for (JsonNode subfield : field.path("fields")) {
197+
if ("keyword".equals(subfield.path("type").asText()) && changesSourceValue(subfield)) {
198+
return true;
199+
}
200+
}
201+
return false;
202+
}
203+
204+
private static boolean changesSourceValue(JsonNode field) {
205+
return field.isMissingNode() || field.has("ignore_above") || field.has("normalizer")
206+
|| field.has("null_value") || field.path("ignore_malformed").asBoolean(false)
207+
|| !field.path("index").asBoolean(true);
208+
}
209+
210+
private static void collectCopyTargets(JsonNode properties, Set<String> targets) {
211+
for (JsonNode field : properties) {
212+
JsonNode copyTo = field.path("copy_to");
213+
if (copyTo.isTextual()) {
214+
targets.add(copyTo.asText());
215+
} else if (copyTo.isArray()) {
216+
copyTo.forEach(target -> targets.add(target.asText()));
217+
}
218+
collectCopyTargets(field.path("properties"), targets);
219+
collectCopyTargets(field.path("fields"), targets);
220+
}
221+
}
222+
159223
private static void resolveDateFields(EsFieldContext context, ObjectNode fieldObject,
160224
String colName, String fieldType) {
161225
if ("date".equals(fieldType)) {

‎fe/fe-connector/fe-connector-es/src/main/java/org/apache/doris/connector/es/EsQueryDslBuilder.java‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -46,6 +46,7 @@
4646
import java.util.Collections;
4747
import java.util.List;
4848
import java.util.Map;
49+
import java.util.Set;
4950

5051
/**
5152
* Converts {@link ConnectorExpression} filter trees into Elasticsearch Query DSL JSON strings.
@@ -104,6 +105,16 @@ public static EsQueryDslResult buildQueryDslWithResult(ConnectorExpression filte
104105
Map<String, String> column2typeMap,
105106
boolean likePushDown,
106107
List<String> needCompatDateFields) {
108+
return buildQueryDslWithResult(filter, fieldsContext, column2typeMap, likePushDown,
109+
needCompatDateFields, Collections.emptySet());
110+
}
111+
112+
public static EsQueryDslResult buildQueryDslWithResult(ConnectorExpression filter,
113+
Map<String, String> fieldsContext,
114+
Map<String, String> column2typeMap,
115+
boolean likePushDown,
116+
List<String> needCompatDateFields,
117+
Set<String> sourceValueUnsafeFields) {
107118
if (filter == null) {
108119
return new EsQueryDslResult(matchAllQuery(), Collections.emptyList());
109120
}
@@ -116,6 +127,10 @@ public static EsQueryDslResult buildQueryDslWithResult(ConnectorExpression filte
116127
List<Integer> notPushedIndices = new ArrayList<>();
117128

118129
for (int i = 0; i < conjuncts.size(); i++) {
130+
if (referencesUnsafeSourceField(conjuncts.get(i), sourceValueUnsafeFields)) {
131+
notPushedIndices.add(i);
132+
continue;
133+
}
119134
List<ConnectorExpression> tempNotPushed = new ArrayList<>();
120135
String dsl = toEsDsl(conjuncts.get(i), tempNotPushed, fieldsContext,
121136
likePushDown, needCompatDateFields, column2typeMap);
@@ -143,6 +158,11 @@ public static EsQueryDslResult buildQueryDslWithResult(ConnectorExpression filte
143158
return new EsQueryDslResult(queryDsl, notPushedIndices);
144159
}
145160

161+
// Dropping a source-value predicate after translating a transformed/missing index value
162+
// cannot be repaired by a residual: matching source documents may already be absent.
163+
if (referencesUnsafeSourceField(filter, sourceValueUnsafeFields)) {
164+
return new EsQueryDslResult(matchAllQuery(), Collections.singletonList(0));
165+
}
146166
// Single expression (not AND)
147167
List<ConnectorExpression> notPushDownList = new ArrayList<>();
148168
String dsl = toEsDsl(filter, notPushDownList, fieldsContext,
@@ -155,6 +175,23 @@ public static EsQueryDslResult buildQueryDslWithResult(ConnectorExpression filte
155175
return new EsQueryDslResult(dsl, Collections.emptyList());
156176
}
157177

178+
private static boolean referencesUnsafeSourceField(ConnectorExpression expression, Set<String> unsafeFields) {
179+
if (expression instanceof ConnectorFunctionCall
180+
&& "esquery".equals(((ConnectorFunctionCall) expression).getFunctionName())) {
181+
// esquery explicitly requests ES index semantics rather than SQL source-value semantics.
182+
return false;
183+
}
184+
if (expression instanceof ConnectorColumnRef) {
185+
return unsafeFields.contains(((ConnectorColumnRef) expression).getColumnName());
186+
}
187+
for (ConnectorExpression child : expression.getChildren()) {
188+
if (referencesUnsafeSourceField(child, unsafeFields)) {
189+
return true;
190+
}
191+
}
192+
return false;
193+
}
194+
158195
/**
159196
* Returns the JSON string for a match-all query: {@code {"match_all":{}}}.
160197
*/

‎fe/fe-connector/fe-connector-es/src/main/java/org/apache/doris/connector/es/EsScanPlanProvider.java‎

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -131,7 +131,11 @@ public List<ConnectorScanRange> planScan(ConnectorSession session, ConnectorScan
131131
List<ConnectorScanRange> ranges = new ArrayList<>();
132132
Map<Integer, List<EsShardRouting>> routingsMap = shardPartitions.getShardRoutings();
133133

134-
if (!enableParallelScroll) {
134+
// Replacing an alias with a backing index loses its filter and search routing. Keep one
135+
// scroll on the original target: per-shard requests on a multi-index alias repeat shard IDs.
136+
boolean resolvesToBackingIndices = routingsMap.values().stream().flatMap(List::stream)
137+
.anyMatch(routing -> !indexName.equals(routing.getIndexName()));
138+
if (!enableParallelScroll || resolvesToBackingIndices) {
135139
// Single query mode: one scan range for the whole index
136140
List<String> hosts = collectAllHosts(routingsMap);
137141
ranges.add(new EsScanRange(
@@ -270,7 +274,9 @@ private EsQueryDslResult buildQueryDsl(Optional<ConnectorExpression> filter,
270274

271275
return EsQueryDslBuilder.buildQueryDslWithResult(
272276
filter.get(), fieldsContext, column2typeMap,
273-
props.isLikePushDown(), needCompatDateFields);
277+
props.isLikePushDown(), needCompatDateFields,
278+
state.getFieldContext() != null
279+
? state.getFieldContext().getSourceValueUnsafeFields() : Collections.emptySet());
274280
}
275281

276282
private EsMetadataState fetchMetadataState(ConnectorSession session, EsTableHandle handle,

0 commit comments

Comments
 (0)