Skip to content

[fix](external) Preserve source values and alias semantics in ES scans - #69672

Merged
Gabriel39 merged 11 commits into
apache:masterfrom
Gabriel39:fix/es-query-correctness
Nov 3, 2026
Merged

Gabriel39 merged 11 commits into
apache:masterfrom
Gabriel39:fix/es-query-correctness

Conversation

@Gabriel39

@Gabriel39 Gabriel39 commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Elasticsearch scans can return values that fail SQL predicates when indexed values differ from the values Doris reads, or when a scan switches all selected columns from docvalues to source. Parallel alias scans can also bypass alias filters/routing, and supported mapping shapes can fail during planning.

For example, projecting a normalized keyword forces a source read for its siblings. A numeric alias then reads NULL although its docvalue is 5; a declared empty ARRAY remains nonnull although ES considers it missing; a date-only string reads local midnight instead of the indexed UTC midnight. Boolean empty strings and coerced integer strings must also remain readable: "" means false and "1.2e3" means 1200. Keeping their indexed predicates while pruning SQL residuals returns incorrect rows.

Execution flow

  1. Select a populated mapping independently of alias response order. Distinguish typeless flags/root objects from legacy type wrappers, including a legacy type named runtime, and preserve array annotations. Accept empty/configuration-only alias backings during safety analysis while retaining standalone schema diagnostics.
  2. Resolve the selected keyword query path and docvalue read path separately on every backing, including filter-only columns. Fetch index mapping defaults with the existing metadata request and honor per-field overrides for every alias backing; inherited ignore_malformed can remove indexed values while retaining source values. Ignore transformations on unused sibling paths and unrelated source filters. Missing/read-changing paths disable the affected docvalue entry. Root enabled:false and disabled source (enabled:false or mode:disabled) restrict every affected column.
  3. Determine the final scan-wide read mode before building DSL. In source mode, STRING/ARRAY predicates and predicates on rounded numbers, aliases, scripted fields and default dates retain local residuals without narrowing the ES query. Eligible pure docvalue projections preserve their indexed-value behavior; scalar _id metadata remains pushable.
  4. Keep searchability separate from value equivalence: supported doc-value-only fields remain queryable with index=false. Require existence for SQL inequalities. Normalize retained equality negation, double negation and Boolean negation inside the connector. Retained ranges keep the original ES query complement with an existence guard: an inverse existential range changes matching on arrays such as [1,2]. This preserves NULL rejection when optimizer NOT simplification is disabled.
  5. Pass residual tracking and the existing early-LIMIT restrictions through the current scan contract. Native esquery remains remote-only: native-only expressions and independent AND conjuncts work; an indivisible Boolean expression requiring local native evaluation fails explicitly during planning.
  6. Use both readers' shared parser for supported source representations. Boolean empty strings become false in scalar and ARRAY paths. Scalar numeric/Boolean/date singleton source arrays read their element; empty and null-only arrays read NULL, while STRING/JSONB and declared ARRAY containers remain intact. JSONB singleton, empty and null-element arrays retain their original JSON container. Declared ARRAYs preserve nullable members; numeric empty strings become SQL NULL in scalar and ARRAY paths, while Boolean empty strings remain false. Integer coercion follows the ES family: byte/short/integer strings use double rounding before truncation, while long strings retain exact decimal/exponent handling and 64-bit precision. Bounds are checked before narrowing. Declared integer ARRAY elements share the family-specific conversion. Numeric floating tokens are converted directly with checked bounds, preserving numeric INT64_MIN and the Java upper-endpoint conversion. Floating strings and narrow integer strings accept Java hexadecimal exponents, type suffixes and outer ASCII whitespace; FLOAT strings round directly to float rather than through double. Epoch-millis source dates accept zero, short/signed decimal strings and fractional numeric values, with checked bounds and second-precision floor semantics. Four-digit default-format years retain their calendar interpretation; pre-epoch formatted docvalues use the same floor.
  7. For a source read involving disabled-source backings, pass an optional read_full_source flag through the existing ES property map. BE requests _source:true: disabled hits have missing values, while enabled alias backings retain their source. Ordinary source scans keep field projection and pure docvalue scans keep docvalue requests.
  8. Scan aliases once by their original name so ES applies alias filters and search routing without duplicate shared-shard scans. Physical indices retain shard-parallel scans.

Effects and tradeoffs

This change preserves source values, supported decoding and SQL filtering across both readers and read modes. Source fallbacks can fetch more candidate rows because unsafe predicates remain local. Mixed aliases containing disabled-source backings can fetch additional source fields to avoid ES rejecting projected-source requests. Alias scans use one coordinating scroll to preserve alias semantics; physical-index parallelism remains. No throughput improvement is claimed. There are no new public SQL/catalog/session properties or SPI/Thrift/Protobuf definitions; the optional internal flag uses the existing flat property map.

Known limitation

Mixed-generation FE/BE execution during smooth upgrades is outside this change's scope. Source plans depending on the updated decoder require the updated BE for correct results; this PR does not add a backend-generation gate.

Release note

Fix Elasticsearch source/index value discrepancies, supported Boolean/floating/integer source decoding, default epoch-millis date decoding, disabled-source reads, legacy/configuration-only mapping handling, retained negation NULL semantics and alias filter/routing bypass. Preserve eligible doc-value-only queries and reject native-query combinations requiring local evaluation during planning.

Check List (For Author)

  • Test:
    • ES connector reactor package: 163 reported tests, 162 passed, one optional ES 9 compatibility class skipped; zero failures/errors. All nine reactor modules passed FE Checkstyle.
    • Standard build.sh --be ASAN build and all 28 EsScrollParserTest unit tests passed. The combined parser/reader run reported 47 tests: 46 passed and one optional environment-dependent live smoke test skipped; the separate live regression run exercised both readers. Clang-format 16, scoped clang-tidy and build hygiene passed for all affected C++ files. The local BE ELF requires no GLIBC version above 2.10.
    • Reproduced the reported alias/script, ARRAY, date-only, disabled-source, boolean, root-disabled, integer and retained-NOT failures on the previous head with live ES 8.11.3. FE/BE unit tests also demonstrated failures before the relevant fixes. Legacy ES6 type compatibility is covered by mapping-response unit tests.
    • Seven live ES 8.11.3 suites passed 1,860 stored checks and 1,764 independent SQL comparisons, preserving 88 expected native planning errors. All 1,780 prior stored checks remain unchanged.
    • test_es_source_semantics: 612 stored / 516 independent checks; test_es_numeric_source_semantics: 264 / 264; test_es_mapping_boundaries: 272 / 272; test_es_source_fallback_types: 304 / 304; test_es_scalar_materialization: 160 / 160; test_es_source_value_boundaries: 168 / 168.
    • New source-boundary coverage reproduces all six review findings: JSONB singleton/empty/null-element containers, numeric-double INT64_MIN and upper endpoint, nullable declared ARRAYs, numeric empty strings, Java numeric grammar, direct FLOAT rounding, and inherited malformed/coercion settings with field overrides and mixed aliases. All eight reader/docvalue/parallel combinations passed. New BE and FE controls failed before their corresponding fixes.
    • test_es_epoch_millis adds 80 stored checks and 80 independent comparisons across both readers, docvalue/parallel modes and UTC/+08:00. It covers normalized-sibling source fallback, zero/short/fractional epoch strings, numeric doubles, negative instants, NULL/empty/singleton arrays, local predicates and pure-docvalue controls. All outputs were generated by the repository runner and verified without generation; all 80 sections also match an independent date/predicate computation. New unit coverage checks legacy/V2 date types, declared arrays, calendar-year precedence and invalid numeric/civil-year bounds.
    • New scalar-materialization coverage includes floating/Boolean/date singleton, empty and null-only arrays; narrow integer rounding boundaries and NaN, 64-bit precision controls, declared integer ARRAYs and retained range NOT on multi-value numeric docvalues. New coverage spans both readers, docvalue modes and parallel modes, source fallback plus pure-docvalue controls, disabled source indices/mixed aliases, populated disabled roots, non-UTC date-only values and retained NOT with optimizer simplification disabled. Session settings are restored. New outputs were generated by the repository runner and verified separately without generation; the final date control avoids dependence on the FE JVM timezone.
    • Original FE package built with build.sh --fe; connector updates packaged through the documented module reactor recipe and deployed with the rebuilt BE for live validation.
  • Behavior changed:
    • Yes: correctness fallbacks, supported mapping/decoding/query handling and native-only planning restrictions above.
  • Does this need documentation?
    • No new public SQL syntax or configuration.

### What problem does this PR solve?

ES scans can return incorrect SQL results in two cases:

1. **Indexed values differ from document source values.** For a keyword field with `ignore_above=4`, a source value of `elephant` is readable but absent from the index. Pushing `field = 'elephant'` down as a term query loses a matching row before Doris can filter it. Normalizers, null substitutions and copied/runtime values can also produce false positives or incorrect null filtering. The existing docvalue eligibility check does not establish predicate equivalence, and explicit `doc_values=true` can bypass the `ignore_above` read fallback.
2. **Parallel scans replace an alias with its backing index.** This bypasses alias filters. For a two-index filtered alias in the regression fixture, disabling parallel scroll returns four rows, while enabling it returns all eight backing documents. Reusing the original multi-index alias separately for each shard would risk repeated scans when backing indices share shard IDs.

Derive source-value safety from every backing mapping. Keep affected SQL predicates local without sending a narrowing remote predicate, and read the affected fields from `_source`. Cover keyword subfields, `ignore_above`, normalizers, null substitutions, disabled indexing, explicit `ignore_malformed`, copy targets, runtime overrides and source filtering. Keep Boolean residual tracking and the existing early-LIMIT gate intact; explicit `esquery` retains native ES semantics.

When shard discovery resolves the requested name to backing indices, issue one scroll against the original target so Elasticsearch applies its alias filter and search routing. Physical indices retain shard-parallel scans. This conservative fix trades pushdown/docvalue optimization on affected fields and scan parallelism on aliases for correct results.

Add `test_es_source_semantics` with generated output. It compares source predicates against a Doris internal-table baseline and alias results against native ES queries, across both scanner versions, docvalues on/off and parallel scroll on/off. Cover multi-index aliases with different filters and shared shard IDs, routed/filtered aliases, empty results, full row counts, pagination, nested Boolean filters and LIMIT.

### Release note

Fix incorrect Elasticsearch query results caused by source/index mapping differences and filtered aliases in parallel scans. Alias reads now use one scroll against the original alias.

### Check List (For Author)

- Test
    - [x] Four new FE test methods fail against the unmodified implementation before the fix.
    - [x] ES connector suite: 135 reported tests, 134 passed and one optional live ES 9 compatibility class skipped; no failures or errors.
    - [x] FE package build using `build.sh --fe` and Checkstyle for the ES module and reactor dependencies.
    - [x] Real Elasticsearch 8.11.3 SQL regression: eight configurations, 288 generated result checks and 208 independent comparisons.
    - [x] Unmodified connector reproduces missing keyword rows, normalized/null-value mismatches and alias over-reads against the same live fixtures; the new SQL suite fails on the missing keyword row.
    - [x] Regression output generated by `run-regression-test.sh`, then verified by a separate run without output generation.
- Behavior changed:
    - [x] Yes: unsafe mappings retain local filters/source reads; alias targets use one coordinated scroll.
- Does this need documentation?
    - [x] No new SQL syntax or catalog/session properties.
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@Gabriel39

Copy link
Copy Markdown
Contributor Author

run buildall

@Gabriel39

Copy link
Copy Markdown
Contributor Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Found two P2 regressions at head 7af1754: empty backing mappings abort otherwise readable alias scans, and mixed native esquery/source predicates become unevaluable local residuals. Both are described inline. No existing inline findings were present; existing_blocking_comment_ids is empty. No additional user focus was supplied.

Critical checkpoint conclusions:

  1. Goal and correctness: source/index divergence is handled through local predicates and source reads, and alias targets are preserved, but the two inline failures leave valid queries broken. FE unit tests and the ES8 regression cover the principal intended cases; neither reported failure is covered.
  2. Scope and reuse: changes are focused on four connector classes and tests, reuse the existing field contexts and scan-range contract, and keep generic planning connector-agnostic.
  3. Concurrency and locks: no new threads or locks; field sets are built synchronously in per-scan metadata state before consumption. No new shared mutation or lock-order issue found.
  4. Lifecycle/static initialization: existing per-statement raw-mapping reuse and per-scan provider state remain intact; reader/scroll cleanup is unchanged. No new static-initialization dependency or resource ownership issue found.
  5. Configuration: no new settings; existing docvalue, keyword, parallel-scroll and scanner options are respected.
  6. Compatibility: no new SPI, storage format or wire contract. The existing DSL overload remains and alias scans reuse the established shard_id=-1 sentinel. Empty backing mappings are a behavioral compatibility defect described inline.
  7. Thrift/protobuf: no changes, so field-ID/enum migration checks are inapplicable.
  8. Parallel paths: traced both legacy and V2 ES readers; both receive the original alias target and share the ES query/scroll implementation. Concrete indices retain shard-parallel scans.
  9. Conditional checks: the alias mismatch condition correctly explains filters, routing and repeated shard IDs. Unsafe-field checks preserve ordinary Boolean residual indices; the native-function exception does not protect a containing OR/NOT, as reported inline.
  10. Test coverage: FE tests cover unsafe attributes, predicate families, mixed ordinary Boolean filters, source filters, copy targets, runtime overrides, a second mapping, aliases and filter columns. The regression covers eight option combinations, pagination, residual LIMIT, NULLs, source filtering, alias filters/routing and counts. Add empty-backing and mixed-native Boolean cases.
  11. Expected results: all 288 output blocks were checked against their fixtures and across eight configurations. Queries are ordered, except LIMIT cases with one matching row. Static inspection cannot establish output-generation provenance.
  12. Observability: existing scan/debug and ES request logs expose target/DSL/mode; no additional metric requirement identified for this planning change.
  13. Transactions/persistence: no EditLog, replay, commit, failover, visible-version or delete-bitmap changes.
  14. Data writes/atomicity: production code changes only reads; transactionality and crash-safe writes are inapplicable.
  15. FE-BE variables: no new transmitted variable; existing es_params/es_properties carry index, shard, DSL and docvalue mode through both reader paths.
  16. Performance/memory: the additional schema pass is linear in backing-mapping size with no new remote fetch. Alias serialization and conservative pushdown/docvalue fallback are intentional correctness tradeoffs. No new per-row allocation or unbounded retry found; performance was not measured.
  17. Other issues/final sweep: checked heterogeneous/missing fields, selected keyword subfields, native functions, empty/routed aliases, slot dependencies, residual indices and early LIMIT. Source-disabled/source-excluded value changes follow this PR's explicit stored-source policy and are not submitted as another defect.

Static review only: no builds, tests, benchmarks, services or source modifications were performed. End-to-end additions target ES8; typed ES5/6 execution and generation of the checked-in output were not independently run. These validation limits remain explicit; no unresolved review candidate or uncovered changed file remains. Review converged in two rounds after independent reviewers checked the unchanged findings and payload.

Review converged after 2 round(s).

…uals

### What problem does this PR solve?

Empty backing mappings made alias safety analysis abort readable scans. Accept empty mappings in safety analysis while retaining strict schema inference and source reads for missing fields.

Field-wide unsafe-mapping rejection also left remote-only esquery calls inside local OR/NOT residuals. Preserve exact keyword equality/IN using literal bounds intersected across all backing mappings. Reject genuinely inexact native Boolean expressions during planning, and keep separate AND conjuncts independently pushable.

### Release note

Allow ES aliases with empty backing mappings and keep exact native-query Boolean combinations remote. Diagnose combinations that cannot be evaluated locally during FE planning.

### Check List (For Author)

- Test: 139 ES FE tests passed, one optional live class skipped; FE Checkstyle passed. Real ES 8.11.3 regression passed across eight configurations with 344 snapshots, 256 independent comparisons and 16 expected planning errors. Both reported failures reproduced before the update.
- Behavior changed: Yes, as described above.
- Does this need documentation: No new configuration or syntax.
@Gabriel39

Copy link
Copy Markdown
Contributor Author

run buildall

@Gabriel39

Copy link
Copy Markdown
Contributor Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Found four P2 issues: bounded keyword pushdown disagrees with source arrays exposed as STRING, and unrelated source filters, unused normalized subfields or searchable doc-value-only keywords reject otherwise exact native Boolean queries. All four are new instances, detailed inline.

The review reached its three-round cap without convergence: the final mapping sweep added the fourth verified finding after the other reviewers audited the previous three-comment set. The expanded final payload has not received another unchanged-set audit. All verified findings are included; no fourth round was started.

Checkpoint conclusions from static inspection:

  • Goal and correctness: scalar source-value safety and alias preservation are implemented; the four inline cases remain. Added tests exercise the intended scalar behavior but omit these cases.
  • Scope and clarity: changes stay within the ES connector and its tests, with no unrelated production changes.
  • Concurrency and locks: the new field context is constructed and consumed per scan on the existing synchronous planning thread; no new threads, locks or lock-order changes.
  • Lifecycle and initialization: per-scan memoization and reader cleanup remain intact; no new global initialization or ownership cycle.
  • Configuration: no new configuration; keyword sniff, docvalues and parallel-scroll settings use existing paths.
  • Compatibility: existing overloads and wire/storage formats are preserved. The planning regressions are described inline.
  • Thrift/protobuf: no definition or field-ID changes.
  • Parallel paths: traced both scanner generations through the shared ES query builder, reader and parser; aliases retain the original target in one scroll, including shared backing shard IDs.
  • Special conditions: minimum bounds, source transformations, missing/empty backings and remote-only esquery guards were checked; the four overly broad exactness/safety decisions are the findings.
  • Test coverage: FE boundary tests and eight SQL configurations cover both readers, docvalue modes and parallel modes, with independent internal-table and native-ES oracles. Add source-array, unaffected-source-field, unused-normalized-subfield and doc-value-only native-OR cases.
  • Expected results: all 344 stored results have consistent case sets/results across the eight configurations and agree with the inspected scalar/alias fixtures. Generation provenance was not independently verified.
  • Observability: existing query/range logging, read profiles and explicit planning diagnostics cover the changed paths; no additional concrete gap found.
  • Transactions and persistence: no EditLog, transaction, failover or persistence changes.
  • Writes and atomicity: read-only scan changes; write atomicity and crash recovery are unaffected.
  • FE/BE propagation: source/docvalue selection and residual/LIMIT facts travel through existing properties for both readers; no new wire variable.
  • Performance: aliases intentionally use one coordinating scroll; physical indices retain shard parallelism. Mapping safety is derived per scan and reused; no further concrete performance issue found.
  • Other issues and limits: no further substantiated introduced defect after the full sweep. No additional user focus was supplied. This was a static-only review: no builds, tests, benchmarks or runtime validation were run.

Revalidated existing threads 4386047138 and 4386047135: their original native-residual and populated-plus-empty-backing instances are addressed at this head, so neither is reposted. There are no existing P0/P1 comments to retain. All seven changed files and assigned risk areas were covered; no unresolved review candidate or source-coverage gap remains.

Review did not converge after 3 rounds. This automated review cannot pass the review check.

Round 3 added verified M4: index-disabled keyword fields with docvalues are still searchable, but the new safety check rejects their supported native Boolean queries. Not all latest-round reviewers returned NO_NEW_VALUABLE_FINDINGS; the final four-comment payload has not received an unchanged-set audit after this addition. The three-round cap prevents a fourth round. All four verified findings are included, all seven changed files and assigned risks were covered, and no unresolved candidate remains. Validation was static only; runtime behavior and generated-output provenance were not independently executed or verified.

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 28482 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit bdc4c2465c288108084e5cd24b42275bde6079b3, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17609	5254	5070	5070
q2	2109	344	254	254
q3	10202	1380	828	828
q4	4672	460	338	338
q5	7432	797	542	542
q6	189	190	153	153
q7	756	798	606	606
q8	9257	1356	1076	1076
q9	5496	4218	4195	4195
q10	6840	1289	1058	1058
q11	439	245	223	223
q12	633	392	289	289
q13	18131	3689	3082	3082
q14	275	278	248	248
q15	q16	726	717	641	641
q17	1345	750	578	578
q18	7000	6169	6125	6125
q19	1099	954	601	601
q20	382	334	228	228
q21	5199	2097	2316	2097
q22	354	303	250	250
Total cold run time: 100145 ms
Total hot run time: 28482 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	5628	5602	5656	5602
q2	720	591	538	538
q3	5211	5375	4908	4908
q4	2038	2166	1583	1583
q5	5274	5180	5722	5180
q6	325	197	151	151
q7	2371	2100	1924	1924
q8	3174	2945	2741	2741
q9	8191	8086	8136	8086
q10	3770	3672	3221	3221
q11	644	445	418	418
q12	682	682	516	516
q13	3695	3802	3273	3273
q14	310	329	303	303
q15	q16	700	710	643	643
q17	8391	7628	6906	6906
q18	13021	12210	12905	12210
q19	902	785	828	785
q20	2125	2151	1931	1931
q21	5751	4735	4842	4735
q22	518	447	416	416
Total cold run time: 73441 ms
Total hot run time: 66070 ms

### What problem does this PR solve?

Related PR: apache#69672

Problem Summary: A keyword length bound cannot establish scalar source equality,
because ES term queries match array elements while Doris may read a JSON STRING.
Remove that exemption and evaluate mapping restrictions on the selected query and
docvalue paths independently. Apply source filters per field, retain supported
docvalue-only keyword queries, and share populated-schema selection across schema
inference and scan planning so empty alias backings work in either response order.

### Release note

Keep affected ES predicates local and reject native-only Boolean expressions that
cannot be evaluated locally. Preserve unaffected query/read paths and alias semantics.

### Check List (For Author)

- Test: ES connector Maven suite and FE Checkstyle; 144 tests passed, one optional
  external class skipped. Real ES8 regression across eight configurations: 488 result
  checks, 400 independent comparisons, and 48 expected planning errors. Generated
  output verified by a separate regression run.
- Behavior changed: Yes, mapping-related fallback and native Boolean planning checks.
- Does this need documentation: No new SQL syntax or properties.
@Gabriel39

Copy link
Copy Markdown
Contributor Author

run buildall

@Gabriel39

Copy link
Copy Markdown
Contributor Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed exact head cff8803bf3a04e17eb7393d31f4808ebb5e2ccc2 using the supplied aggregate diff. One new P2 correctness finding is inline: forcing source reads for one projected column can leave an unchanged sibling's predicate incorrectly pruned. The PR's source-value preservation goal is therefore incomplete.

Existing context was independently checked and deduplicated. The original bounded-literal exemption is removed; affected native OR/NOT expressions fail during planning, separate AND conjuncts remain supported, selected-path safety and per-column source filtering address the reported scalar examples, and the keyword docvalue-only example is fixed. Two broader cases remain within existing threads and are not reposted:

Critical checkpoint conclusions:

  • Goal/data correctness: mapping transformations and original-target alias scans address the intended problems; the new sibling-representation finding and the two known cases above remain.
  • Scope/clarity/reuse: changes stay within the ES connector and its tests; sharing mapping selection with schema inference reduces duplicate parsing logic.
  • Concurrency/locks: no new threads or locks; per-scan state remains confined to synchronous planning, and statement scope shares the immutable raw mapping string.
  • Lifecycle/static initialization: private parsed trees and scan contexts have bounded ownership; candidate probing preserves array metadata with a deep copy. No new lifecycle or initialization-order concern found.
  • Configuration: no configuration items added; existing docvalue, keyword-sniff, scanner and parallel-scroll settings are exercised.
  • Compatibility: no shared SPI, function symbol or persisted format change. Original-target alias scanning is intentional; native combinations needing local evaluation now fail at FE. Typed/typeless and empty-root handling was inspected, with the known flag-only gap above.
  • Thrift/protobuf: no gensrc definitions changed.
  • Parallel paths: both scanners share the query builder and parser and are affected by the new finding. Physical indices retain shard parallelism; aliases use one original-target range, preserving filters/routing and avoiding repeated shard IDs.
  • Conditional checks: source transformations, selected query/read paths, missing backing paths, globs and remote-only Boolean checks have concrete purposes and controls; global read-mode effects remain incomplete.
  • Test coverage: FE controls and the live-regression fixture span both readers, docvalue/parallel modes, Boolean/native cases, aliases, small pages and LIMIT. Missing cases are the sibling singleton-array interaction and the known extensions above. No tests or builds were run in this static-only review.
  • Changed test results: all 488 stored sections were inspected, including canonical values and equality across eight configurations. Values are consistent with the fixture; test execution and output-generation provenance were not independently verified.
  • Observability: existing scan/debug context and FE diagnostics support the changed decisions; no additional required metric was identified.
  • Transactions/persistence/failover: no transaction, EditLog, replay or persistence path changed.
  • Writes/atomicity/crash handling: production changes are read planning only; no data-write or atomicity mechanism changed.
  • FE-to-BE variables: existing flat properties and docvalue contexts carry the changes; no new wire variable or missing serialization path found.
  • Performance: every-backing field analysis adds local parsing work without new remote metadata calls; single alias scrolls deliberately trade parallelism for correct alias semantics. No separate substantiated performance defect found.
  • Other issues/user focus: the full changed-file list, related control flow, tests and special paths were covered. No additional user focus was supplied.

Review converged after 2 rounds: all three agents reported NO_NEW_VALUABLE_FINDINGS in the final round, required static coverage is complete, and all candidates/risk items have verified or duplicate dispositions. No unresolved review question or uncovered assigned scope remains. Runtime/build/test validation was excluded by contract.

Review converged after 2 round(s).

### What problem does this PR solve?

Related PR: apache#69672

Problem Summary: Removing one projected docvalue path switches the entire scan to
source reads. An unchanged keyword sibling can then become a JSON array STRING,
while its ES element predicate is still pushed and its SQL residual removed.
Compute the scan read mode before DSL generation and retain source STRING predicates
locally, including unprojected filter references. Keep numeric, scalar hit metadata
and native-only paths unchanged. Reject native mixed Boolean residuals that cannot
be evaluated locally; independent AND conjuncts remain supported.

### Release note

Fix wrong ES results after projection-driven source fallback. Ordinary string
predicates are evaluated locally in source scans, and native mixed expressions
requiring local evaluation fail during planning.

### Check List (For Author)

- Test: ES connector Maven suite and FE Checkstyle: 147 tests passed, one optional
  external class skipped. Real ES8 tests across eight configurations: 612 stored
  results, 516 independent comparisons and 76 expected planning errors. Output
  generated and verified in a separate regression run.
- Behavior changed: Yes, string predicate retention follows the final scan read mode.
- Does this need documentation: No new syntax or properties.
@Gabriel39

Copy link
Copy Markdown
Contributor Author

run buildall

@Gabriel39

Copy link
Copy Markdown
Contributor Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed all 8 changed files at 74dc1718b9f495033354060fb3635190ea254034 with three reviewers and independent main-agent verification. One new P2 correctness finding is inline: source fallback leaves scalar scaled_float predicates classified as exact, so returned values can violate a pruned SQL predicate. The search/source difference is documented by Elasticsearch's numeric mapping reference.

Critical checkpoint conclusions:

  1. Goal and correctness: original-target alias scans and source-safe string residuals address the intended failures; scalar numeric representation safety remains incomplete as described inline.
  2. Scope and reuse: changes stay within the ES connector and its tests; sharing mapping selection removes duplicate inference logic, and existing SPI residual tracking/request/parser paths are reused.
  3. Concurrency and locking: planning remains synchronous on a per-scan provider; no new threads, shared lock state, heavy locked operations or lock-order changes.
  4. Lifecycle and initialization: mapping safety uses newly parsed JSON and per-scan context; shared statement mapping strings stay immutable. No new ownership/teardown or static initialization dependency.
  5. Configuration: no configuration added. Existing docvalue, keyword-sniff, field-count, scanner and parallel-scroll settings are read through the existing paths.
  6. Compatibility: no incompatible interface or storage-format changes, and no serialized-field definition changes. Existing DSL/mode/context properties feed both readers.
  7. Thrift/Protobuf IDs: no gensrc changes; branch field-ID comparison is inapplicable.
  8. Parallel paths: traced both reader versions and their shared query builder/parser. Alias ranges preserve the original target and omit shard preference; physical-index shard parallelism remains available. Cloud/shared-nothing use the same ES connector path.
  9. Special conditions and failures: per-backing path/source checks and the source-mode gate have concrete semantic reasons. Native-only esquery is rejected during planning when an indivisible Boolean expression would require local evaluation; independent AND conjuncts remain supported. The string-only gate misses the numeric case inline.
  10. Test coverage: added FE controls and end-to-end fixtures cover mapping transformations, selected query/read paths, empty backings, aliases, arrays, filter-only columns, native Boolean boundaries, small pages and LIMIT across eight scanner/docvalue/parallel combinations. Lossy numeric source fallback is missing.
  11. Changed results: all 612 stored output blocks were checked statically against the fixture expectations, including independent SQL/native oracles; no snapshot mismatch found. Test execution and output-generation provenance were not verified in this review.
  12. Observability: existing mode/DSL/range diagnostics and reader profiles remain available; planning failure explains unsupported native residuals. No additional critical metric or hot INFO logging requirement identified.
  13. Transactions and persistence: production changes are external reads; EditLog, failover, visible-version and delete-bitmap rules are unaffected.
  14. Writes and crash atomicity: no production data-write path changes; regression fixture writes are confined to the test setup. No new write atomicity or crash-lifetime issue identified.
  15. FE/BE propagation: mode, DSL, contexts and original-target scan parameters use existing fields in both readers. Materialized filter slots are included before FE's mode choice; residual indices and source LIMIT gating remain aligned.
  16. Performance: local source predicates and one scroll per alias intentionally trade pushdown/parallelism for correctness. Added analysis is per-planning mapping/expression work, with no new per-row operation. No benchmark was run.
  17. Other issues: no further distinct verified finding after the final assigned-scope rechecks. The already-covered mapping variants below were deduplicated rather than posted again.

Existing-thread disposition: the seven original reproductions (empty backing, native residual, bounded keyword arrays, per-column source filtering, selected keyword path, docvalue-only keyword, and STRING sibling fallback) are addressed for their reported cases. Two extensions remain covered by prior P2 threads: docvalue-only numeric/date searchability is still rejected by the keyword-only isSearchable exception, although ES supports numeric docvalue-only queries; and configuration-only backing mappings, such as a root enabled=false without properties, can still reach the typed-mapping BooleanNode cast (supported ES mapping shape). These substantially overlap known comments and are not reposted. No existing P0/P1 finding needs to be carried.

User focus: no additional focus was supplied; the complete PR was reviewed. Coverage limits: static inspection only, including external API semantics checked against primary documentation. No builds, tests, Checkstyle, benchmarks or runtime queries were run, and repository source was not modified. No unresolved review candidate remains after the recorded acceptance/deduplication/dismissal decisions.

Review converged after 3 round(s).

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 28766 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 74dc1718b9f495033354060fb3635190ea254034, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17582	5170	5064	5064
q2	2147	321	257	257
q3	10194	1440	850	850
q4	4682	475	338	338
q5	7448	794	528	528
q6	187	184	148	148
q7	740	790	607	607
q8	9255	1336	1022	1022
q9	5518	4211	4188	4188
q10	6814	1294	1065	1065
q11	438	260	235	235
q12	635	396	276	276
q13	18158	3702	3040	3040
q14	270	273	249	249
q15	q16	732	710	679	679
q17	1195	763	577	577
q18	7042	6171	6117	6117
q19	1105	968	603	603
q20	379	334	230	230
q21	5957	2523	2421	2421
q22	391	329	272	272
Total cold run time: 100869 ms
Total hot run time: 28766 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	6030	5912	6095	5912
q2	735	618	604	604
q3	5559	5802	5398	5398
q4	2021	2155	1564	1564
q5	5828	5844	5602	5602
q6	237	186	142	142
q7	2306	2032	1793	1793
q8	3023	2677	2642	2642
q9	7982	7501	7500	7500
q10	3649	3589	3174	3174
q11	587	425	423	423
q12	672	671	499	499
q13	3287	3633	3063	3063
q14	280	297	264	264
q15	q16	690	679	613	613
q17	7705	7082	7085	7082
q18	12965	12121	12845	12121
q19	898	789	777	777
q20	2121	2131	1949	1949
q21	5629	4687	4752	4687
q22	508	455	423	423
Total cold run time: 72712 ms
Total hot run time: 66232 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 151091 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 74dc1718b9f495033354060fb3635190ea254034, data reload: false

query5	4307	581	437	437
query6	428	209	201	201
query7	4807	446	230	230
query8	325	159	148	148
query9	8769	3123	3126	3123
query10	472	306	254	254
query11	6063	3837	3609	3609
query12	158	90	83	83
query13	1253	452	340	340
query14	6163	3982	3656	3656
query14_1	3410	3469	3403	3403
query15	207	196	183	183
query16	990	419	456	419
query17	852	681	543	543
query18	2415	441	316	316
query19	200	167	138	138
query20	86	85	84	84
query21	211	129	106	106
query22	12965	12976	12784	12784
query23	13479	12762	12561	12561
query23_1	12590	12511	12559	12511
query24	7345	829	477	477
query24_1	484	488	470	470
query25	545	418	373	373
query26	1265	257	138	138
query27	2769	443	281	281
query28	4621	1965	1972	1965
query29	1573	607	445	445
query30	306	217	188	188
query31	1021	924	831	831
query32	152	92	91	91
query33	537	314	249	249
query34	901	843	492	492
query35	746	759	636	636
query36	840	795	725	725
query37	121	100	84	84
query38	1797	1727	1667	1667
query39	800	788	746	746
query39_1	718	727	739	727
query40	220	120	98	98
query41	81	71	68	68
query42	87	89	84	84
query43	342	364	311	311
query44	1276	703	692	692
query45	182	214	163	163
query46	823	948	548	548
query47	3015	3041	2751	2751
query48	303	307	221	221
query49	572	403	301	301
query50	675	272	203	203
query51	10721	10455	10573	10455
query52	79	81	70	70
query53	189	202	148	148
query54	232	194	176	176
query55	77	66	68	66
query56	239	210	201	201
query57	1673	1594	1578	1578
query58	286	257	252	252
query59	2190	2168	1982	1982
query60	285	244	232	232
query61	145	150	151	150
query62	382	325	283	283
query63	186	156	159	156
query64	2763	950	811	811
query65	3316	3271	3233	3233
query66	1766	412	297	297
query67	20497	20262	19909	19909
query68	2964	1013	654	654
query69	380	290	249	249
query70	852	763	789	763
query71	278	231	206	206
query72	2716	2603	2264	2264
query73	511	522	287	287
query74	4634	4515	4371	4371
query75	2342	2295	1924	1924
query76	2305	1014	584	584
query77	335	369	275	275
query78	9852	9776	9200	9200
query79	959	907	508	508
query80	1228	423	348	348
query81	586	317	264	264
query82	849	136	105	105
query83	307	198	203	198
query84	317	122	104	104
query85	1003	424	372	372
query86	403	239	230	230
query87	1992	1952	1804	1804
query88	3675	2704	2686	2686
query89	334	289	241	241
query90	1940	172	169	169
query91	163	147	129	129
query92	99	89	90	89
query93	1009	1012	591	591
query94	645	315	270	270
query95	581	348	309	309
query96	664	477	241	241
query97	2451	2442	2328	2328
query98	165	158	146	146
query99	779	773	649	649
Total cold run time: 236508 ms
Total hot run time: 151091 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 24.46 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 74dc1718b9f495033354060fb3635190ea254034, data reload: false

query1	0.01	0.01	0.01
query2	0.10	0.06	0.05
query3	0.26	0.08	0.08
query4	1.60	0.10	0.10
query5	0.24	0.25	0.25
query6	1.17	0.69	0.67
query7	0.04	0.01	0.01
query8	0.06	0.04	0.04
query9	0.41	0.36	0.36
query10	0.63	0.61	0.59
query11	0.21	0.14	0.14
query12	0.18	0.15	0.15
query13	0.49	0.49	0.48
query14	1.03	1.01	0.99
query15	0.62	0.60	0.60
query16	0.36	0.34	0.35
query17	1.14	1.13	1.14
query18	0.21	0.19	0.17
query19	2.08	1.92	2.00
query20	0.02	0.01	0.01
query21	15.46	0.20	0.13
query22	4.75	0.05	0.06
query23	16.28	0.32	0.12
query24	3.17	0.58	0.46
query25	0.10	0.06	0.06
query26	1.68	0.22	0.15
query27	0.05	0.05	0.05
query28	2.90	0.67	0.36
query29	12.49	4.20	3.29
query30	0.28	0.13	0.11
query31	2.77	0.55	0.32
query32	3.22	0.59	0.50
query33	3.21	3.22	3.27
query34	15.41	4.03	3.41
query35	3.35	3.33	3.36
query36	0.54	0.44	0.48
query37	0.08	0.06	0.06
query38	0.05	0.04	0.04
query39	0.04	0.03	0.03
query40	0.17	0.16	0.15
query41	0.09	0.03	0.03
query42	0.03	0.03	0.03
query43	0.04	0.04	0.03
Total cold run time: 97.02 s
Total hot run time: 24.46 s

@hello-stephen

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage `` 🎉
Increment coverage report
Complete coverage report

@hello-stephen

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 0.00% (0/4) 🎉
Increment coverage report
Complete coverage report

1 similar comment
@hello-stephen

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 0.00% (0/4) 🎉
Increment coverage report
Complete coverage report

@924060929 924060929 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FE runtime validation found a normalizer-triggered wrong-result regression. Reproduction steps and actual merge-base/head results are inline.

### What problem does this PR solve?

Issue Number: N/A

Related PR: apache#69672

Problem Summary: Projecting a normalized keyword switches the entire ES scan to
_source. A scaled_float with scaling_factor=4 keeps 2.34 in source but indexes
2.25. Fully pushing b=2.25 then returns 2.34 and drops the SQL residual.

Collect scaled_float and half_float source-mode restrictions across every
backing mapping, including unprojected filter columns. Apply them after the
final read mode is known, keeping the SQL residual without a narrowing remote
numeric predicate. Preserve docvalue eligibility for numeric-only projections.

### Release note

Fix rounded numeric predicates after Elasticsearch source fallback while
preserving numeric docvalue scans.

### Check List (For Author)

- Test: ES connector unit tests and FE Checkstyle; live ES 8.11.3 source
  semantics regression; numeric regression with an independent internal-table
  oracle across both readers, both scroll settings and three catalog modes.
- Behavior changed: Yes, rounded numeric mappings use local SQL predicates in
  source mode; indivisible mixed esquery expressions follow the existing
  remote-only planning contract.
- Does this need documentation: No new SQL syntax or configuration.
@Gabriel39

Copy link
Copy Markdown
Contributor Author

run buildall

@Gabriel39

Copy link
Copy Markdown
Contributor Author

/review

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 28517 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ba56fc40930e31b0d8df46395b492ef604014b3d, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17596	5139	5078	5078
q2	2136	301	262	262
q3	10254	1372	821	821
q4	4671	463	333	333
q5	7462	797	528	528
q6	190	183	152	152
q7	747	803	601	601
q8	9271	1342	1120	1120
q9	5516	4209	4207	4207
q10	6840	1295	1028	1028
q11	432	256	233	233
q12	651	398	281	281
q13	18117	3589	3009	3009
q14	274	276	248	248
q15	q16	732	727	674	674
q17	1193	740	563	563
q18	6941	6149	6113	6113
q19	1104	958	617	617
q20	386	337	227	227
q21	4791	2162	2341	2162
q22	349	294	260	260
Total cold run time: 99653 ms
Total hot run time: 28517 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	5595	5628	5606	5606
q2	694	572	522	522
q3	5160	5371	4900	4900
q4	1973	2119	1565	1565
q5	5233	5154	5410	5154
q6	269	203	158	158
q7	2323	2098	1830	1830
q8	3022	2759	2734	2734
q9	8002	7909	7918	7909
q10	3736	3680	3406	3406
q11	618	440	410	410
q12	727	685	523	523
q13	3607	3872	3367	3367
q14	324	313	293	293
q15	q16	708	709	645	645
q17	8332	7300	6890	6890
q18	13001	12224	12978	12224
q19	927	823	830	823
q20	2142	2151	1929	1929
q21	5727	4648	4751	4648
q22	510	474	390	390
Total cold run time: 72630 ms
Total hot run time: 65926 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 151382 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ba56fc40930e31b0d8df46395b492ef604014b3d, data reload: false

query5	4307	606	434	434
query6	422	223	191	191
query7	4796	455	229	229
query8	331	170	154	154
query9	8791	3217	3233	3217
query10	486	316	250	250
query11	5958	3865	3593	3593
query12	159	90	89	89
query13	1251	471	318	318
query14	6152	3974	3645	3645
query14_1	3439	3441	3434	3434
query15	199	191	179	179
query16	935	429	387	387
query17	855	647	520	520
query18	2410	416	302	302
query19	194	155	126	126
query20	85	83	81	81
query21	205	127	105	105
query22	12985	13102	12839	12839
query23	13468	12825	12492	12492
query23_1	12484	12544	12511	12511
query24	7333	861	474	474
query24_1	477	478	479	478
query25	514	398	350	350
query26	1265	258	135	135
query27	2791	446	274	274
query28	4622	2044	2010	2010
query29	1561	629	444	444
query30	297	223	181	181
query31	994	927	816	816
query32	146	94	89	89
query33	518	282	230	230
query34	908	870	500	500
query35	742	745	636	636
query36	789	808	735	735
query37	129	99	78	78
query38	1790	1756	1650	1650
query39	781	788	752	752
query39_1	741	730	733	730
query40	216	114	92	92
query41	63	61	61	61
query42	84	85	82	82
query43	360	361	313	313
query44	1390	703	717	703
query45	191	174	168	168
query46	896	1011	607	607
query47	3031	2909	2793	2793
query48	300	308	220	220
query49	578	396	280	280
query50	715	287	215	215
query51	10675	10522	10129	10129
query52	81	87	73	73
query53	198	218	153	153
query54	233	192	200	192
query55	75	65	65	65
query56	229	219	220	219
query57	1650	1634	1585	1585
query58	283	264	258	258
query59	2148	2187	1975	1975
query60	289	260	246	246
query61	168	173	164	164
query62	406	337	291	291
query63	199	166	171	166
query64	2847	1040	854	854
query65	3346	3313	3290	3290
query66	1804	435	317	317
query67	20115	20230	20129	20129
query68	2976	1028	624	624
query69	399	311	257	257
query70	878	783	788	783
query71	309	249	246	246
query72	2826	2714	2224	2224
query73	535	558	299	299
query74	4633	4525	4389	4389
query75	2340	2321	1929	1929
query76	2321	1033	632	632
query77	332	391	284	284
query78	9879	9713	9118	9118
query79	1015	927	548	548
query80	544	445	357	357
query81	523	318	269	269
query82	576	132	108	108
query83	310	200	189	189
query84	312	128	101	101
query85	795	469	375	375
query86	340	237	234	234
query87	2040	1950	1795	1795
query88	3808	2771	2762	2762
query89	334	289	248	248
query90	1882	176	169	169
query91	165	150	130	130
query92	103	91	82	82
query93	1042	1030	592	592
query94	448	297	248	248
query95	595	388	295	295
query96	687	507	242	242
query97	2423	2438	2377	2377
query98	167	151	154	151
query99	764	763	653	653
Total cold run time: 234950 ms
Total hot run time: 151382 ms

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Static review completed against the exact PR head ba56fc4 in two bounded rounds. One new P2 is reported inline.

The changed scan-wide source fallback can make valid default Elasticsearch date values fail materialization. A date mapping without format accepts epoch-millis source strings such as "0" and fractional values such as "1.5". When a projected normalized keyword sibling loses its docvalue path, useDocValueScan switches the complete projection to _source; get_date_value_int only recognizes 13-character positive epoch strings, so these values fall through datetime casting (and a JSON fractional number reaches GetInt64() despite being a double). Both legacy and v2 readers share this failure path. The inline comment requests ES-compatible epoch parsing and regression coverage for both readers.

I revalidated all existing inline threads: none of the existing P0/P1 findings remains applicable, so existing_blocking_comment_ids is empty. Mapping/alias/REST routing, source-filter safety, Boolean residual planning, hidden slots, full-source propagation, LIMIT gating, parser coercion, arrays, NULL handling, and the user focus (no additional focus supplied) were reviewed. No builds, tests, source edits, or runtime validation were performed under the static-only contract.

Review converged after 2 round(s).

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 24.3 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ba56fc40930e31b0d8df46395b492ef604014b3d, data reload: false

query1	0.01	0.01	0.01
query2	0.09	0.05	0.04
query3	0.26	0.09	0.09
query4	1.61	0.10	0.10
query5	0.26	0.25	0.24
query6	1.18	0.68	0.66
query7	0.04	0.01	0.01
query8	0.06	0.04	0.03
query9	0.40	0.35	0.35
query10	0.59	0.60	0.60
query11	0.20	0.15	0.14
query12	0.20	0.15	0.15
query13	0.49	0.49	0.49
query14	1.04	1.02	1.01
query15	0.63	0.60	0.61
query16	0.34	0.35	0.36
query17	1.17	1.14	1.13
query18	0.22	0.20	0.20
query19	2.05	1.92	2.01
query20	0.02	0.01	0.02
query21	15.48	0.20	0.13
query22	4.80	0.05	0.05
query23	16.27	0.33	0.12
query24	3.11	0.57	0.43
query25	0.10	0.08	0.06
query26	1.37	0.21	0.15
query27	0.04	0.03	0.03
query28	3.47	0.81	0.37
query29	12.49	4.16	3.20
query30	0.30	0.13	0.10
query31	2.77	0.56	0.34
query32	3.21	0.60	0.48
query33	3.22	3.20	3.22
query34	15.31	3.95	3.39
query35	3.35	3.36	3.33
query36	0.54	0.46	0.43
query37	0.09	0.06	0.07
query38	0.05	0.04	0.04
query39	0.04	0.03	0.03
query40	0.17	0.15	0.14
query41	0.08	0.03	0.03
query42	0.04	0.03	0.03
query43	0.05	0.03	0.04
Total cold run time: 97.21 s
Total hot run time: 24.3 s

@hello-stephen

Copy link
Copy Markdown
Contributor

BE UT Coverage Report

Increment line coverage 93.62% (220/235) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 67.24% (32566/48429)
Line Coverage 52.15% (345267/662108)
Region Coverage 47.61% (280879/590001)
Branch Coverage 49.32% (132768/269201)

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 93.62% (220/235) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 77.58% (36409/46932)
Line Coverage 62.87% (411867/655078)
Region Coverage 59.38% (349080/587867)
Branch Coverage 60.25% (161806/268549)

@hello-stephen

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 100% (0/0) 🎉
Increment coverage report
Complete coverage report

@924060929 924060929 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Elasticsearch filters indexed values while Doris may materialize different source values. This PR determines the scan-wide read mode before predicate pruning and preserves alias filtering/routing through the original target.

Static FE review completed at ba56fc4, including the necessary shared BE reader/parser boundary. Request changes: the existing epoch-millis date finding remains valid. A normalized sibling can force an otherwise readable date projection into source decoding and cause a format error. Fix the shared decoder and cover both readers; retaining a local predicate does not fix materialization.

Earlier mapping, predicate, numeric and array findings were checked against current code. The supplied current-head P0, compile and unit-test statuses are successful, but the AI review body still contains this unresolved P2. No builds or tests were run during this review. This conclusion covers FE and its affected execution contracts, not an independent full BE review or merge authorization.

[P2] Still requires changes: #69672 (comment)

Automated FE-focused static review (GPT-6.1 Sol, medium). No build or tests were run.

### What problem does this PR solve?

Related PR: apache#69672

Problem Summary: A normalized projected sibling can switch an Elasticsearch scan from docvalues to source. The shared date decoder rejected valid short, zero and fractional epoch-millis strings and read fractional numeric tokens as integers. Parse decimal epoch values with checked numeric conversions and civil-year bounds, floor negative instants at the inferred second precision, and preserve default-format calendar years. Both readers and date ARRAY elements use the shared fix.

### Release note

Fix Elasticsearch source date materialization for zero, short and fractional epoch-millis values, including negative instants and source-fallback scans.

### Check List (For Author)

- Test: Standard ASAN BE build; parser/reader unit tests (46 passed, one optional live smoke skipped); seven live Elasticsearch regression suites (1860 stored checks, 1764 independent SQL comparisons, 88 expected planning errors); clang-format 16, scoped clang-tidy and build hygiene.
- Behavior changed: Yes. Supported epoch-millis source dates decode correctly, and out-of-range values fail before narrowing.
- Does this need documentation: No new public configuration or SQL syntax.
@Gabriel39

Copy link
Copy Markdown
Contributor Author

run buildall

@Gabriel39

Copy link
Copy Markdown
Contributor Author

/review

@github-actions

github-actions Bot commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

Codex automated review failed and did not complete.

Error: Selected model is at capacity. Please try a different model.
Workflow run: https://github.com/apache/doris/actions/runs/41569443498

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 28949 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ad9d044e97e967a5d2866cd765807984892d2138, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17620	5111	5326	5111
q2	2134	324	274	274
q3	10204	1397	836	836
q4	4680	466	343	343
q5	7451	807	532	532
q6	183	182	156	156
q7	745	797	606	606
q8	9278	1320	1140	1140
q9	5511	4238	4207	4207
q10	6829	1294	1021	1021
q11	430	254	231	231
q12	633	412	280	280
q13	18115	3686	3044	3044
q14	275	277	252	252
q15	q16	735	717	648	648
q17	1225	758	587	587
q18	7001	6237	6144	6144
q19	1258	979	606	606
q20	385	344	229	229
q21	5516	2436	2494	2436
q22	382	330	266	266
Total cold run time: 100590 ms
Total hot run time: 28949 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	6052	5918	6423	5918
q2	753	634	593	593
q3	5655	5838	5452	5452
q4	2048	2365	1610	1610
q5	5739	5835	5614	5614
q6	238	190	150	150
q7	2230	2020	1759	1759
q8	3030	2709	2689	2689
q9	7911	7510	7453	7453
q10	3669	3588	3192	3192
q11	590	444	381	381
q12	641	671	519	519
q13	3335	3624	3090	3090
q14	283	292	261	261
q15	q16	657	682	623	623
q17	7813	7003	7010	7003
q18	12977	12174	12947	12174
q19	907	787	774	774
q20	2168	2153	1950	1950
q21	5652	4753	4867	4753
q22	531	449	411	411
Total cold run time: 72879 ms
Total hot run time: 66369 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 151577 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ad9d044e97e967a5d2866cd765807984892d2138, data reload: false

query5	4308	601	420	420
query6	431	209	191	191
query7	4797	467	229	229
query8	321	172	153	153
query9	8789	3179	3204	3179
query10	477	332	269	269
query11	6049	3878	3605	3605
query12	151	89	87	87
query13	1244	441	331	331
query14	6120	4003	3675	3675
query14_1	3431	3521	3409	3409
query15	214	192	180	180
query16	971	451	439	439
query17	859	671	535	535
query18	2418	420	319	319
query19	204	169	136	136
query20	87	89	85	85
query21	216	131	107	107
query22	13009	12951	12788	12788
query23	13553	13360	12407	12407
query23_1	12525	12516	12517	12516
query24	7350	838	483	483
query24_1	476	483	495	483
query25	542	426	357	357
query26	1276	268	143	143
query27	2750	447	275	275
query28	4608	2044	2035	2035
query29	1561	609	464	464
query30	310	230	183	183
query31	1018	939	812	812
query32	155	93	104	93
query33	535	314	245	245
query34	919	880	509	509
query35	736	742	640	640
query36	823	832	741	741
query37	126	96	90	90
query38	1828	1749	1652	1652
query39	803	785	742	742
query39_1	741	737	745	737
query40	215	117	97	97
query41	71	77	72	72
query42	88	85	82	82
query43	348	367	311	311
query44	1328	712	709	709
query45	180	184	169	169
query46	843	980	608	608
query47	2848	2858	2618	2618
query48	299	323	221	221
query49	580	402	295	295
query50	678	271	217	217
query51	10681	10629	10509	10509
query52	79	80	75	75
query53	184	207	156	156
query54	235	194	197	194
query55	73	67	74	67
query56	247	207	221	207
query57	1682	1623	1558	1558
query58	309	267	261	261
query59	2108	2210	1970	1970
query60	283	240	228	228
query61	150	148	148	148
query62	397	337	282	282
query63	185	155	157	155
query64	2754	971	819	819
query65	3339	3261	3269	3261
query66	1778	472	304	304
query67	20282	20258	20025	20025
query68	3225	1044	630	630
query69	408	287	245	245
query70	859	759	807	759
query71	315	243	210	210
query72	2907	2642	2355	2355
query73	527	522	302	302
query74	4684	4543	4341	4341
query75	2392	2296	1930	1930
query76	2413	1006	588	588
query77	356	366	276	276
query78	9910	9873	9224	9224
query79	1027	852	529	529
query80	1228	429	352	352
query81	604	324	281	281
query82	901	139	105	105
query83	299	203	192	192
query84	321	123	102	102
query85	980	442	373	373
query86	401	246	225	225
query87	1998	1978	1804	1804
query88	3847	2830	2792	2792
query89	339	283	240	240
query90	1871	168	168	168
query91	167	157	131	131
query92	99	84	96	84
query93	1013	1015	581	581
query94	656	314	271	271
query95	572	324	302	302
query96	711	490	240	240
query97	2488	2432	2337	2337
query98	156	153	144	144
query99	776	772	657	657
Total cold run time: 237228 ms
Total hot run time: 151577 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 24.4 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ad9d044e97e967a5d2866cd765807984892d2138, data reload: false

query1	0.00	0.00	0.00
query2	0.09	0.04	0.05
query3	0.26	0.09	0.08
query4	1.61	0.08	0.10
query5	0.26	0.25	0.25
query6	1.16	0.69	0.70
query7	0.04	0.01	0.01
query8	0.06	0.04	0.04
query9	0.41	0.35	0.35
query10	0.58	0.60	0.61
query11	0.20	0.14	0.13
query12	0.18	0.15	0.16
query13	0.49	0.50	0.50
query14	1.04	1.02	1.02
query15	0.63	0.62	0.60
query16	0.35	0.36	0.34
query17	1.15	1.09	1.12
query18	0.21	0.20	0.19
query19	2.10	2.01	2.00
query20	0.02	0.01	0.02
query21	15.43	0.21	0.14
query22	4.84	0.06	0.05
query23	16.29	0.34	0.12
query24	2.88	0.60	0.41
query25	0.10	0.07	0.04
query26	1.05	0.23	0.16
query27	0.05	0.04	0.05
query28	3.23	0.77	0.38
query29	12.50	4.19	3.28
query30	0.28	0.13	0.09
query31	2.77	0.56	0.33
query32	3.23	0.60	0.51
query33	3.13	3.18	3.32
query34	15.50	4.01	3.39
query35	3.34	3.34	3.33
query36	0.57	0.42	0.43
query37	0.09	0.06	0.07
query38	0.04	0.04	0.04
query39	0.04	0.03	0.04
query40	0.18	0.14	0.14
query41	0.09	0.03	0.03
query42	0.04	0.03	0.03
query43	0.04	0.04	0.03
Total cold run time: 96.55 s
Total hot run time: 24.4 s

@Gabriel39

Copy link
Copy Markdown
Contributor Author

/review

@924060929 924060929 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Elasticsearch filters indexed values, while Doris may materialize different source values. This PR selects the scan-wide read mode before predicate pruning, preserves alias filters/routing, and updates shared source decoding.

Independent static FE review completed at ad9d044 against merge base af9dad5. No remaining actionable P0/P1/P2 issue was found in FE or its affected BE execution contracts. Mapping safety, residual tracking, LIMIT gating, alias routing, both readers, and decoding boundaries were checked. The previous epoch-millis finding is addressed by the shared parser changes and added coverage. Approve.

No builds or tests were run. The supplied current-head snapshot reports successful P0, compile and external regression checks; unit tests and automated review remain pending. This is an FE-focused code opinion, not a full BE review or merge authorization.

Automated FE-focused static review (gpt-6.1-sol, high). No build or tests were run.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed #69672 at exact head ad9d044 against the supplied merge-base diff and all existing inline threads. The review converged after two bounded rounds.

No new actionable P0, P1, P2, or P3 finding was verified. The FE mapping/type, REST metadata, DSL/residual, scan-plan/property, alias-routing, and BE parser/wire scopes were covered. I traced mapping safety through scan-wide docvalue/source selection, Boolean and NOT/NULL pushdown, native esquery handling, read_full_source propagation, alias single-scroll routing, and both BE readers. Source/date/numeric/Boolean/ARRAY/JSONB boundaries and the changed unit/regression fixtures were inspected statically. A tentative epoch-millis plus-sign candidate was checked against the Elasticsearch formatter grammar and dismissed.

All existing inline threads and replies were revalidated at this head. They are prior P2 discussions with fixes or explicit scope decisions; no existing P0/P1 inline finding remains applicable, so existing_blocking_comment_ids is empty. The user focus file supplies no additional focus.

This was a static review only. I did not run builds, tests, benchmarks, live Elasticsearch reproductions, or runtime validation; author-reported results were treated as claims rather than execution evidence. No Thrift/Protobuf definitions, persistence, transaction, or write paths changed.

Review converged after 2 round(s).

@hello-stephen

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage `` 🎉
Increment coverage report
Complete coverage report

@hello-stephen

Copy link
Copy Markdown
Contributor

BE UT Coverage Report

Increment line coverage 92.14% (293/318) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 67.26% (32084/47704)
Line Coverage 52.12% (337711/647889)
Region Coverage 47.69% (274758/576137)
Branch Coverage 49.51% (129507/261584)

@Gabriel39
Gabriel39 merged commit c7d1df4 into apache:master Nov 3, 2026
42 of 44 checks passed
@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 92.14% (293/318) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 77.68% (35890/46203)
Line Coverage 63.05% (403943/640636)
Region Coverage 59.70% (342210/573261)
Branch Coverage 60.79% (158413/260577)

Gabriel39 added a commit that referenced this pull request Nov 4, 2026
### What problem does this PR solve?

Issue Number: N/A

Related PR: #69672, #69723

External ES regression suites fail during setup or retain expectations
from the previous indexed-value predicate behavior:

- Read the deprecated scanner option through `SELECT
@@enable_file_scanner_v2`, whose compatibility name remains valid when
SHOW prefixes deprecated names.
- On ES 8.3.3, fall back from the specifically unsupported
`_source.mode` mapping syntax to `enabled:false`, retaining the
disabled-source row and alias checks. Other HTTP failures remain fatal;
supporting ES8 versions still exercise the mode syntax.
- Compare LIKE, REGEXP, NULL and `level` predicates against unfiltered
rows materialized into internal tables, then update only the affected
runner-generated golden blocks.
- Make the date-string equality test explicitly use source reads, test
midnight as the matching value, and retain 08:00 as a negative control.
- Check that a residual predicate prevents early termination while a
safe numeric docvalue predicate still permits LIMIT pushdown.

### Release note

None

### Check List (For Author)

- Test
  - [x] Regression test
  - [ ] Unit Test
  - [x] Manual test

Validation uses fresh FE and ASAN BE artifacts built from this PR commit
(`6386f5f83a51b548c86ab3cff83d53cb48fb18df`) with the standard
`./build.sh --fe --be -j64` entry point. The complete build exited
successfully, and both finished artifact versions match that commit. No
production source is changed by this PR.

- On the same fresh binaries and reset ES 5.6.16, 6.8.23, 7.17.5 and
8.3.3 fixtures, the parent suites/golden files reproduced all ten
failures; the final PR suites passed all ten in normal comparison, with
zero failed, fatal or skipped suites. The passing run executed 2,154
golden checks and 1,798 independent query comparisons, without
force-generation or further golden edits.
- The complete `test_es_query` passed all four ES versions and both
scroll settings, including its unchanged complex-array golden blocks
(276 result checks and 20 independent comparisons).
- Independent SQLite/Python evaluation of the unchanged raw fixture JSON
passed 38 comparisons and covered all 17 changed golden blocks. Real ES
requests also confirmed that `ignore_above` omits matching index values
while retaining source strings, and that the constant-keyword index
matches documents whose source `level` is absent.
- Captured ES search requests verified the date modes: the original
four-column projection reads source, returns midnight values and rejects
the 08:00 predicate. A date-only docvalue projection correctly returns
08:00 in the +08:00 session. The revised test explicitly fixes the
source contract and retains the 08:00 negative control.
- The complete fallback suite additionally passed normal comparison on
ES 8.11.3: 304 golden checks and 304 independent comparisons. The actual
mappings and document responses confirmed absent source for both the ES
8.3.3 `enabled:false` fallback and the supported ES 8.11.3
`mode:disabled` branch.
- The existing `test_es_file_scanner_v2` compatibility suite also passed
on the fresh build across all four fixture versions. The deprecated
option is accepted; this does not claim two distinct physical reader
implementations on current master.
- Direct variable probes confirmed the old exact SHOW name returns no
rows, the wildcard SHOW name exposes its deprecated prefix, and the
scalar compatibility name remains readable.
- `git diff --check` passed. ES9 compatibility is not claimed.

- Behavior changed:
- [x] No production behavior change; regression fixtures and
expectations are corrected.
  - [ ] Yes.

- Does this need documentation?
  - [x] No.
  - [ ] Yes.

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
yiguolei pushed a commit that referenced this pull request Nov 5, 2026
…/JSON/WAL fixes (#70185)

### What problem does this PR solve?

Backport the following master PRs to `branch-4.2`, in master merge
order:

1. #69672 — Preserve source values and alias semantics in ES scans.
2. #69971 — Add session controls for Parquet page indexes and data-page
size.
3. #70155 — Repair ES source predicate regression cases.
4. #70079 — Fix ORC condition-cache seeks past filtered stripe suffixes.
5. #70154 — Preserve JSON and WAL rows with empty file projections.

The ES changes are adapted to the existing 4.2 FE planner and BE source
locations. Mapping safety, scan-wide read mode, local residuals, native
`esquery` handling and alias routing retain the upstream behavior
without introducing the master connector framework. Added adapter tests
cover both scan protocols, aliases, missing-value negations and native
residual rejection.

Preserve existing branch-specific ES regression coverage. Bring in the
previously absent predicate suite, adapting its plan assertions to 4.2's
`LOCAL_PREDICATES` output and retaining LIMIT result checks. Parquet
variables use the 4.2 annotation interface and the original optional
Thrift field IDs. The ORC dependency is pinned to the same fixed commit
as #70079.

The ES implementation also preserves source numeric and JSONB predicate
semantics after scan-wide fallback. Numeric-token rounding must not make
a remote predicate discard a row matching the value Doris materializes;
empty JSONB objects remain nonnull even when ES indexes no leaf.
Docvalue eligibility and pushdown are retained.

Mapped date formats and their order are carried per concrete backing
index through the existing property map to both scan protocols.
Calendar/epoch alternatives now distinguish basic_date from
epoch_millis, for strings and source numeric tokens. Numeric docvalues
and older FE numeric requests retain their epoch decoder. Typed alias
selection consistently uses the requested mapping type in schema
inference and planning. The pinned ORC decoder's libdeflate dependency
is included on ARM, matching master.

### Release note

Fix ES source-value filtering, mapped dates and typed alias query
correctness, ORC condition-cache seek crashes/missing rows and its ARM
link dependency, and JSON/WAL row counts for empty file projections. Add
Parquet output page-index and data-page-size session controls.

### Validation

- FE compilation and Checkstyle passed. ES validation: 81 tests (48
datasource/planner tests and 33 legacy ES tests), 0 failures/errors, 1
external-environment skip. The 9 Parquet session-variable tests passed.
- Rebuilt the pinned ORC dependency under AddressSanitizer: 56 focused
tests passed, including 32 filtered-stripe-suffix seek cases.
- Compiled all 10 affected Doris C++ implementation/test translation
units and the ORC fixture generator. The regenerated fixture is
byte-for-byte identical to the committed fixture.
- clang-format 16 checks passed for all affected Doris C++ sources and
headers. Modified Groovy suites compiled successfully.
- The ES reader/parser implementations and test translation units
compiled. The shared DATETIMEV2 date-format algorithm passed 21
standalone ASAN checks. CMake dependency checks passed for aarch64,
arm64 and x86_64; an actual ARM link was not run locally.
- Added `test_es_source_review_cases.groovy` with SQL-to-SQL checks
against an internal reference table for numeric/JSONB source predicates,
LIMIT, epoch-only formats, and explicit legacy DATETIMEV2 basic_date
columns. It covers both scanners and parallel-scroll settings, waits for
legacy metadata readiness, and restores session settings. The suite
compiled; no golden files were handwritten.
- The applicable ES fixes are synchronized to master in #70205. Master
already links libdeflate on every architecture.
- Full Doris BE unit-test execution and live external SQL regression
were not performed locally; CI validation is requested.

### Check List (For Author)

- Test
  - [x] Unit Test
  - [x] Regression tests included
- Behavior changed:
  - [x] Yes; the upstream behavior is backported to 4.2.
- Does this need documentation?
- [x] Parquet controls are documented by the upstream documentation PR:
apache/doris-website#4581

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants