Repository navigation
[fix](external) Preserve source values and alias semantics in ES scans - #69672
Conversation
### What problem does this PR solve?
ES scans can return incorrect SQL results in two cases:
1. **Indexed values differ from document source values.** For a keyword field with `ignore_above=4`, a source value of `elephant` is readable but absent from the index. Pushing `field = 'elephant'` down as a term query loses a matching row before Doris can filter it. Normalizers, null substitutions and copied/runtime values can also produce false positives or incorrect null filtering. The existing docvalue eligibility check does not establish predicate equivalence, and explicit `doc_values=true` can bypass the `ignore_above` read fallback.
2. **Parallel scans replace an alias with its backing index.** This bypasses alias filters. For a two-index filtered alias in the regression fixture, disabling parallel scroll returns four rows, while enabling it returns all eight backing documents. Reusing the original multi-index alias separately for each shard would risk repeated scans when backing indices share shard IDs.
Derive source-value safety from every backing mapping. Keep affected SQL predicates local without sending a narrowing remote predicate, and read the affected fields from `_source`. Cover keyword subfields, `ignore_above`, normalizers, null substitutions, disabled indexing, explicit `ignore_malformed`, copy targets, runtime overrides and source filtering. Keep Boolean residual tracking and the existing early-LIMIT gate intact; explicit `esquery` retains native ES semantics.
When shard discovery resolves the requested name to backing indices, issue one scroll against the original target so Elasticsearch applies its alias filter and search routing. Physical indices retain shard-parallel scans. This conservative fix trades pushdown/docvalue optimization on affected fields and scan parallelism on aliases for correct results.
Add `test_es_source_semantics` with generated output. It compares source predicates against a Doris internal-table baseline and alias results against native ES queries, across both scanner versions, docvalues on/off and parallel scroll on/off. Cover multi-index aliases with different filters and shared shard IDs, routed/filtered aliases, empty results, full row counts, pagination, nested Boolean filters and LIMIT.
### Release note
Fix incorrect Elasticsearch query results caused by source/index mapping differences and filtered aliases in parallel scans. Alias reads now use one scroll against the original alias.
### Check List (For Author)
- Test
- [x] Four new FE test methods fail against the unmodified implementation before the fix.
- [x] ES connector suite: 135 reported tests, 134 passed and one optional live ES 9 compatibility class skipped; no failures or errors.
- [x] FE package build using `build.sh --fe` and Checkstyle for the ES module and reactor dependencies.
- [x] Real Elasticsearch 8.11.3 SQL regression: eight configurations, 288 generated result checks and 208 independent comparisons.
- [x] Unmodified connector reproduces missing keyword rows, normalized/null-value mismatches and alias over-reads against the same live fixtures; the new SQL suite fails on the missing keyword row.
- [x] Regression output generated by `run-regression-test.sh`, then verified by a separate run without output generation.
- Behavior changed:
- [x] Yes: unsafe mappings retain local filters/source reads; alias targets use one coordinated scroll.
- Does this need documentation?
- [x] No new SQL syntax or catalog/session properties.
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
|
run buildall |
|
/review |
There was a problem hiding this comment.
Found two P2 regressions at head 7af1754: empty backing mappings abort otherwise readable alias scans, and mixed native esquery/source predicates become unevaluable local residuals. Both are described inline. No existing inline findings were present; existing_blocking_comment_ids is empty. No additional user focus was supplied.
Critical checkpoint conclusions:
- Goal and correctness: source/index divergence is handled through local predicates and source reads, and alias targets are preserved, but the two inline failures leave valid queries broken. FE unit tests and the ES8 regression cover the principal intended cases; neither reported failure is covered.
- Scope and reuse: changes are focused on four connector classes and tests, reuse the existing field contexts and scan-range contract, and keep generic planning connector-agnostic.
- Concurrency and locks: no new threads or locks; field sets are built synchronously in per-scan metadata state before consumption. No new shared mutation or lock-order issue found.
- Lifecycle/static initialization: existing per-statement raw-mapping reuse and per-scan provider state remain intact; reader/scroll cleanup is unchanged. No new static-initialization dependency or resource ownership issue found.
- Configuration: no new settings; existing docvalue, keyword, parallel-scroll and scanner options are respected.
- Compatibility: no new SPI, storage format or wire contract. The existing DSL overload remains and alias scans reuse the established shard_id=-1 sentinel. Empty backing mappings are a behavioral compatibility defect described inline.
- Thrift/protobuf: no changes, so field-ID/enum migration checks are inapplicable.
- Parallel paths: traced both legacy and V2 ES readers; both receive the original alias target and share the ES query/scroll implementation. Concrete indices retain shard-parallel scans.
- Conditional checks: the alias mismatch condition correctly explains filters, routing and repeated shard IDs. Unsafe-field checks preserve ordinary Boolean residual indices; the native-function exception does not protect a containing OR/NOT, as reported inline.
- Test coverage: FE tests cover unsafe attributes, predicate families, mixed ordinary Boolean filters, source filters, copy targets, runtime overrides, a second mapping, aliases and filter columns. The regression covers eight option combinations, pagination, residual LIMIT, NULLs, source filtering, alias filters/routing and counts. Add empty-backing and mixed-native Boolean cases.
- Expected results: all 288 output blocks were checked against their fixtures and across eight configurations. Queries are ordered, except LIMIT cases with one matching row. Static inspection cannot establish output-generation provenance.
- Observability: existing scan/debug and ES request logs expose target/DSL/mode; no additional metric requirement identified for this planning change.
- Transactions/persistence: no EditLog, replay, commit, failover, visible-version or delete-bitmap changes.
- Data writes/atomicity: production code changes only reads; transactionality and crash-safe writes are inapplicable.
- FE-BE variables: no new transmitted variable; existing es_params/es_properties carry index, shard, DSL and docvalue mode through both reader paths.
- Performance/memory: the additional schema pass is linear in backing-mapping size with no new remote fetch. Alias serialization and conservative pushdown/docvalue fallback are intentional correctness tradeoffs. No new per-row allocation or unbounded retry found; performance was not measured.
- Other issues/final sweep: checked heterogeneous/missing fields, selected keyword subfields, native functions, empty/routed aliases, slot dependencies, residual indices and early LIMIT. Source-disabled/source-excluded value changes follow this PR's explicit stored-source policy and are not submitted as another defect.
Static review only: no builds, tests, benchmarks, services or source modifications were performed. End-to-end additions target ES8; typed ES5/6 execution and generation of the checked-in output were not independently run. These validation limits remain explicit; no unresolved review candidate or uncovered changed file remains. Review converged in two rounds after independent reviewers checked the unchanged findings and payload.
Review converged after 2 round(s).
…uals ### What problem does this PR solve? Empty backing mappings made alias safety analysis abort readable scans. Accept empty mappings in safety analysis while retaining strict schema inference and source reads for missing fields. Field-wide unsafe-mapping rejection also left remote-only esquery calls inside local OR/NOT residuals. Preserve exact keyword equality/IN using literal bounds intersected across all backing mappings. Reject genuinely inexact native Boolean expressions during planning, and keep separate AND conjuncts independently pushable. ### Release note Allow ES aliases with empty backing mappings and keep exact native-query Boolean combinations remote. Diagnose combinations that cannot be evaluated locally during FE planning. ### Check List (For Author) - Test: 139 ES FE tests passed, one optional live class skipped; FE Checkstyle passed. Real ES 8.11.3 regression passed across eight configurations with 344 snapshots, 256 independent comparisons and 16 expected planning errors. Both reported failures reproduced before the update. - Behavior changed: Yes, as described above. - Does this need documentation: No new configuration or syntax.
|
run buildall |
|
/review |
There was a problem hiding this comment.
Found four P2 issues: bounded keyword pushdown disagrees with source arrays exposed as STRING, and unrelated source filters, unused normalized subfields or searchable doc-value-only keywords reject otherwise exact native Boolean queries. All four are new instances, detailed inline.
The review reached its three-round cap without convergence: the final mapping sweep added the fourth verified finding after the other reviewers audited the previous three-comment set. The expanded final payload has not received another unchanged-set audit. All verified findings are included; no fourth round was started.
Checkpoint conclusions from static inspection:
- Goal and correctness: scalar source-value safety and alias preservation are implemented; the four inline cases remain. Added tests exercise the intended scalar behavior but omit these cases.
- Scope and clarity: changes stay within the ES connector and its tests, with no unrelated production changes.
- Concurrency and locks: the new field context is constructed and consumed per scan on the existing synchronous planning thread; no new threads, locks or lock-order changes.
- Lifecycle and initialization: per-scan memoization and reader cleanup remain intact; no new global initialization or ownership cycle.
- Configuration: no new configuration; keyword sniff, docvalues and parallel-scroll settings use existing paths.
- Compatibility: existing overloads and wire/storage formats are preserved. The planning regressions are described inline.
- Thrift/protobuf: no definition or field-ID changes.
- Parallel paths: traced both scanner generations through the shared ES query builder, reader and parser; aliases retain the original target in one scroll, including shared backing shard IDs.
- Special conditions: minimum bounds, source transformations, missing/empty backings and remote-only esquery guards were checked; the four overly broad exactness/safety decisions are the findings.
- Test coverage: FE boundary tests and eight SQL configurations cover both readers, docvalue modes and parallel modes, with independent internal-table and native-ES oracles. Add source-array, unaffected-source-field, unused-normalized-subfield and doc-value-only native-OR cases.
- Expected results: all 344 stored results have consistent case sets/results across the eight configurations and agree with the inspected scalar/alias fixtures. Generation provenance was not independently verified.
- Observability: existing query/range logging, read profiles and explicit planning diagnostics cover the changed paths; no additional concrete gap found.
- Transactions and persistence: no EditLog, transaction, failover or persistence changes.
- Writes and atomicity: read-only scan changes; write atomicity and crash recovery are unaffected.
- FE/BE propagation: source/docvalue selection and residual/LIMIT facts travel through existing properties for both readers; no new wire variable.
- Performance: aliases intentionally use one coordinating scroll; physical indices retain shard parallelism. Mapping safety is derived per scan and reused; no further concrete performance issue found.
- Other issues and limits: no further substantiated introduced defect after the full sweep. No additional user focus was supplied. This was a static-only review: no builds, tests, benchmarks or runtime validation were run.
Revalidated existing threads 4386047138 and 4386047135: their original native-residual and populated-plus-empty-backing instances are addressed at this head, so neither is reposted. There are no existing P0/P1 comments to retain. All seven changed files and assigned risk areas were covered; no unresolved review candidate or source-coverage gap remains.
Review did not converge after 3 rounds. This automated review cannot pass the review check.
Round 3 added verified M4: index-disabled keyword fields with docvalues are still searchable, but the new safety check rejects their supported native Boolean queries. Not all latest-round reviewers returned NO_NEW_VALUABLE_FINDINGS; the final four-comment payload has not received an unchanged-set audit after this addition. The three-round cap prevents a fourth round. All four verified findings are included, all seven changed files and assigned risks were covered, and no unresolved candidate remains. Validation was static only; runtime behavior and generated-output provenance were not independently executed or verified.
TPC-H: Total hot run time: 28482 ms |
### What problem does this PR solve? Related PR: apache#69672 Problem Summary: A keyword length bound cannot establish scalar source equality, because ES term queries match array elements while Doris may read a JSON STRING. Remove that exemption and evaluate mapping restrictions on the selected query and docvalue paths independently. Apply source filters per field, retain supported docvalue-only keyword queries, and share populated-schema selection across schema inference and scan planning so empty alias backings work in either response order. ### Release note Keep affected ES predicates local and reject native-only Boolean expressions that cannot be evaluated locally. Preserve unaffected query/read paths and alias semantics. ### Check List (For Author) - Test: ES connector Maven suite and FE Checkstyle; 144 tests passed, one optional external class skipped. Real ES8 regression across eight configurations: 488 result checks, 400 independent comparisons, and 48 expected planning errors. Generated output verified by a separate regression run. - Behavior changed: Yes, mapping-related fallback and native Boolean planning checks. - Does this need documentation: No new SQL syntax or properties.
|
run buildall |
|
/review |
There was a problem hiding this comment.
Reviewed exact head cff8803bf3a04e17eb7393d31f4808ebb5e2ccc2 using the supplied aggregate diff. One new P2 correctness finding is inline: forcing source reads for one projected column can leave an unchanged sibling's predicate incorrectly pruned. The PR's source-value preservation goal is therefore incomplete.
Existing context was independently checked and deduplicated. The original bounded-literal exemption is removed; affected native OR/NOT expressions fail during planning, separate AND conjuncts remain supported, selected-path safety and per-column source filtering address the reported scalar examples, and the keyword docvalue-only example is fixed. Two broader cases remain within existing threads and are not reposted:
- Empty backing mappings, r4386047135: a populated-first alias with another backing
mappings:{"date_detection":false}still reaches theBooleanNodetoObjectNodecast ingetRootSchemaand aborts planning. The new tests cover{}, typed empty and_meta-only roots, but not flag-only roots permitted by ES mapping parameters. - Docvalue-only searchability, r4386297855:
isSearchableadmits only keyword whenindex=false. A scalarlongwith docvalues enabled is still classified unsafe, soa=1 OR esquery(b,...)is rejected despite supported remote branches. ES also supports docvalue-only numeric, date, boolean and IP queries. No existing P0/P1 finding applies.
Critical checkpoint conclusions:
- Goal/data correctness: mapping transformations and original-target alias scans address the intended problems; the new sibling-representation finding and the two known cases above remain.
- Scope/clarity/reuse: changes stay within the ES connector and its tests; sharing mapping selection with schema inference reduces duplicate parsing logic.
- Concurrency/locks: no new threads or locks; per-scan state remains confined to synchronous planning, and statement scope shares the immutable raw mapping string.
- Lifecycle/static initialization: private parsed trees and scan contexts have bounded ownership; candidate probing preserves array metadata with a deep copy. No new lifecycle or initialization-order concern found.
- Configuration: no configuration items added; existing docvalue, keyword-sniff, scanner and parallel-scroll settings are exercised.
- Compatibility: no shared SPI, function symbol or persisted format change. Original-target alias scanning is intentional; native combinations needing local evaluation now fail at FE. Typed/typeless and empty-root handling was inspected, with the known flag-only gap above.
- Thrift/protobuf: no gensrc definitions changed.
- Parallel paths: both scanners share the query builder and parser and are affected by the new finding. Physical indices retain shard parallelism; aliases use one original-target range, preserving filters/routing and avoiding repeated shard IDs.
- Conditional checks: source transformations, selected query/read paths, missing backing paths, globs and remote-only Boolean checks have concrete purposes and controls; global read-mode effects remain incomplete.
- Test coverage: FE controls and the live-regression fixture span both readers, docvalue/parallel modes, Boolean/native cases, aliases, small pages and LIMIT. Missing cases are the sibling singleton-array interaction and the known extensions above. No tests or builds were run in this static-only review.
- Changed test results: all 488 stored sections were inspected, including canonical values and equality across eight configurations. Values are consistent with the fixture; test execution and output-generation provenance were not independently verified.
- Observability: existing scan/debug context and FE diagnostics support the changed decisions; no additional required metric was identified.
- Transactions/persistence/failover: no transaction, EditLog, replay or persistence path changed.
- Writes/atomicity/crash handling: production changes are read planning only; no data-write or atomicity mechanism changed.
- FE-to-BE variables: existing flat properties and docvalue contexts carry the changes; no new wire variable or missing serialization path found.
- Performance: every-backing field analysis adds local parsing work without new remote metadata calls; single alias scrolls deliberately trade parallelism for correct alias semantics. No separate substantiated performance defect found.
- Other issues/user focus: the full changed-file list, related control flow, tests and special paths were covered. No additional user focus was supplied.
Review converged after 2 rounds: all three agents reported NO_NEW_VALUABLE_FINDINGS in the final round, required static coverage is complete, and all candidates/risk items have verified or duplicate dispositions. No unresolved review question or uncovered assigned scope remains. Runtime/build/test validation was excluded by contract.
Review converged after 2 round(s).
### What problem does this PR solve? Related PR: apache#69672 Problem Summary: Removing one projected docvalue path switches the entire scan to source reads. An unchanged keyword sibling can then become a JSON array STRING, while its ES element predicate is still pushed and its SQL residual removed. Compute the scan read mode before DSL generation and retain source STRING predicates locally, including unprojected filter references. Keep numeric, scalar hit metadata and native-only paths unchanged. Reject native mixed Boolean residuals that cannot be evaluated locally; independent AND conjuncts remain supported. ### Release note Fix wrong ES results after projection-driven source fallback. Ordinary string predicates are evaluated locally in source scans, and native mixed expressions requiring local evaluation fail during planning. ### Check List (For Author) - Test: ES connector Maven suite and FE Checkstyle: 147 tests passed, one optional external class skipped. Real ES8 tests across eight configurations: 612 stored results, 516 independent comparisons and 76 expected planning errors. Output generated and verified in a separate regression run. - Behavior changed: Yes, string predicate retention follows the final scan read mode. - Does this need documentation: No new syntax or properties.
|
run buildall |
|
/review |
There was a problem hiding this comment.
Reviewed all 8 changed files at 74dc1718b9f495033354060fb3635190ea254034 with three reviewers and independent main-agent verification. One new P2 correctness finding is inline: source fallback leaves scalar scaled_float predicates classified as exact, so returned values can violate a pruned SQL predicate. The search/source difference is documented by Elasticsearch's numeric mapping reference.
Critical checkpoint conclusions:
- Goal and correctness: original-target alias scans and source-safe string residuals address the intended failures; scalar numeric representation safety remains incomplete as described inline.
- Scope and reuse: changes stay within the ES connector and its tests; sharing mapping selection removes duplicate inference logic, and existing SPI residual tracking/request/parser paths are reused.
- Concurrency and locking: planning remains synchronous on a per-scan provider; no new threads, shared lock state, heavy locked operations or lock-order changes.
- Lifecycle and initialization: mapping safety uses newly parsed JSON and per-scan context; shared statement mapping strings stay immutable. No new ownership/teardown or static initialization dependency.
- Configuration: no configuration added. Existing docvalue, keyword-sniff, field-count, scanner and parallel-scroll settings are read through the existing paths.
- Compatibility: no incompatible interface or storage-format changes, and no serialized-field definition changes. Existing DSL/mode/context properties feed both readers.
- Thrift/Protobuf IDs: no gensrc changes; branch field-ID comparison is inapplicable.
- Parallel paths: traced both reader versions and their shared query builder/parser. Alias ranges preserve the original target and omit shard preference; physical-index shard parallelism remains available. Cloud/shared-nothing use the same ES connector path.
- Special conditions and failures: per-backing path/source checks and the source-mode gate have concrete semantic reasons. Native-only esquery is rejected during planning when an indivisible Boolean expression would require local evaluation; independent AND conjuncts remain supported. The string-only gate misses the numeric case inline.
- Test coverage: added FE controls and end-to-end fixtures cover mapping transformations, selected query/read paths, empty backings, aliases, arrays, filter-only columns, native Boolean boundaries, small pages and LIMIT across eight scanner/docvalue/parallel combinations. Lossy numeric source fallback is missing.
- Changed results: all 612 stored output blocks were checked statically against the fixture expectations, including independent SQL/native oracles; no snapshot mismatch found. Test execution and output-generation provenance were not verified in this review.
- Observability: existing mode/DSL/range diagnostics and reader profiles remain available; planning failure explains unsupported native residuals. No additional critical metric or hot INFO logging requirement identified.
- Transactions and persistence: production changes are external reads; EditLog, failover, visible-version and delete-bitmap rules are unaffected.
- Writes and crash atomicity: no production data-write path changes; regression fixture writes are confined to the test setup. No new write atomicity or crash-lifetime issue identified.
- FE/BE propagation: mode, DSL, contexts and original-target scan parameters use existing fields in both readers. Materialized filter slots are included before FE's mode choice; residual indices and source LIMIT gating remain aligned.
- Performance: local source predicates and one scroll per alias intentionally trade pushdown/parallelism for correctness. Added analysis is per-planning mapping/expression work, with no new per-row operation. No benchmark was run.
- Other issues: no further distinct verified finding after the final assigned-scope rechecks. The already-covered mapping variants below were deduplicated rather than posted again.
Existing-thread disposition: the seven original reproductions (empty backing, native residual, bounded keyword arrays, per-column source filtering, selected keyword path, docvalue-only keyword, and STRING sibling fallback) are addressed for their reported cases. Two extensions remain covered by prior P2 threads: docvalue-only numeric/date searchability is still rejected by the keyword-only isSearchable exception, although ES supports numeric docvalue-only queries; and configuration-only backing mappings, such as a root enabled=false without properties, can still reach the typed-mapping BooleanNode cast (supported ES mapping shape). These substantially overlap known comments and are not reposted. No existing P0/P1 finding needs to be carried.
User focus: no additional focus was supplied; the complete PR was reviewed. Coverage limits: static inspection only, including external API semantics checked against primary documentation. No builds, tests, Checkstyle, benchmarks or runtime queries were run, and repository source was not modified. No unresolved review candidate remains after the recorded acceptance/deduplication/dismissal decisions.
Review converged after 3 round(s).
TPC-H: Total hot run time: 28766 ms |
TPC-DS: Total hot run time: 151091 ms |
ClickBench: Total hot run time: 24.46 s |
FE UT Coverage ReportIncrement line coverage `` 🎉 |
FE Regression Coverage ReportIncrement line coverage |
1 similar comment
FE Regression Coverage ReportIncrement line coverage |
924060929
left a comment
There was a problem hiding this comment.
FE runtime validation found a normalizer-triggered wrong-result regression. Reproduction steps and actual merge-base/head results are inline.
### What problem does this PR solve? Issue Number: N/A Related PR: apache#69672 Problem Summary: Projecting a normalized keyword switches the entire ES scan to _source. A scaled_float with scaling_factor=4 keeps 2.34 in source but indexes 2.25. Fully pushing b=2.25 then returns 2.34 and drops the SQL residual. Collect scaled_float and half_float source-mode restrictions across every backing mapping, including unprojected filter columns. Apply them after the final read mode is known, keeping the SQL residual without a narrowing remote numeric predicate. Preserve docvalue eligibility for numeric-only projections. ### Release note Fix rounded numeric predicates after Elasticsearch source fallback while preserving numeric docvalue scans. ### Check List (For Author) - Test: ES connector unit tests and FE Checkstyle; live ES 8.11.3 source semantics regression; numeric regression with an independent internal-table oracle across both readers, both scroll settings and three catalog modes. - Behavior changed: Yes, rounded numeric mappings use local SQL predicates in source mode; indivisible mixed esquery expressions follow the existing remote-only planning contract. - Does this need documentation: No new SQL syntax or configuration.
|
run buildall |
|
/review |
TPC-H: Total hot run time: 28517 ms |
TPC-DS: Total hot run time: 151382 ms |
There was a problem hiding this comment.
Static review completed against the exact PR head ba56fc4 in two bounded rounds. One new P2 is reported inline.
The changed scan-wide source fallback can make valid default Elasticsearch date values fail materialization. A date mapping without format accepts epoch-millis source strings such as "0" and fractional values such as "1.5". When a projected normalized keyword sibling loses its docvalue path, useDocValueScan switches the complete projection to _source; get_date_value_int only recognizes 13-character positive epoch strings, so these values fall through datetime casting (and a JSON fractional number reaches GetInt64() despite being a double). Both legacy and v2 readers share this failure path. The inline comment requests ES-compatible epoch parsing and regression coverage for both readers.
I revalidated all existing inline threads: none of the existing P0/P1 findings remains applicable, so existing_blocking_comment_ids is empty. Mapping/alias/REST routing, source-filter safety, Boolean residual planning, hidden slots, full-source propagation, LIMIT gating, parser coercion, arrays, NULL handling, and the user focus (no additional focus supplied) were reviewed. No builds, tests, source edits, or runtime validation were performed under the static-only contract.
Review converged after 2 round(s).
ClickBench: Total hot run time: 24.3 s |
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
FE Regression Coverage ReportIncrement line coverage |
924060929
left a comment
There was a problem hiding this comment.
Elasticsearch filters indexed values while Doris may materialize different source values. This PR determines the scan-wide read mode before predicate pruning and preserves alias filtering/routing through the original target.
Static FE review completed at ba56fc4, including the necessary shared BE reader/parser boundary. Request changes: the existing epoch-millis date finding remains valid. A normalized sibling can force an otherwise readable date projection into source decoding and cause a format error. Fix the shared decoder and cover both readers; retaining a local predicate does not fix materialization.
Earlier mapping, predicate, numeric and array findings were checked against current code. The supplied current-head P0, compile and unit-test statuses are successful, but the AI review body still contains this unresolved P2. No builds or tests were run during this review. This conclusion covers FE and its affected execution contracts, not an independent full BE review or merge authorization.
[P2] Still requires changes: #69672 (comment)
Automated FE-focused static review (GPT-6.1 Sol, medium). No build or tests were run.
### What problem does this PR solve? Related PR: apache#69672 Problem Summary: A normalized projected sibling can switch an Elasticsearch scan from docvalues to source. The shared date decoder rejected valid short, zero and fractional epoch-millis strings and read fractional numeric tokens as integers. Parse decimal epoch values with checked numeric conversions and civil-year bounds, floor negative instants at the inferred second precision, and preserve default-format calendar years. Both readers and date ARRAY elements use the shared fix. ### Release note Fix Elasticsearch source date materialization for zero, short and fractional epoch-millis values, including negative instants and source-fallback scans. ### Check List (For Author) - Test: Standard ASAN BE build; parser/reader unit tests (46 passed, one optional live smoke skipped); seven live Elasticsearch regression suites (1860 stored checks, 1764 independent SQL comparisons, 88 expected planning errors); clang-format 16, scoped clang-tidy and build hygiene. - Behavior changed: Yes. Supported epoch-millis source dates decode correctly, and out-of-range values fail before narrowing. - Does this need documentation: No new public configuration or SQL syntax.
|
run buildall |
|
/review |
|
Codex automated review failed and did not complete. Error: Selected model is at capacity. Please try a different model. Please inspect the workflow logs and rerun the review after the underlying issue is resolved. |
TPC-H: Total hot run time: 28949 ms |
TPC-DS: Total hot run time: 151577 ms |
ClickBench: Total hot run time: 24.4 s |
|
/review |
924060929
left a comment
There was a problem hiding this comment.
Elasticsearch filters indexed values, while Doris may materialize different source values. This PR selects the scan-wide read mode before predicate pruning, preserves alias filters/routing, and updates shared source decoding.
Independent static FE review completed at ad9d044 against merge base af9dad5. No remaining actionable P0/P1/P2 issue was found in FE or its affected BE execution contracts. Mapping safety, residual tracking, LIMIT gating, alias routing, both readers, and decoding boundaries were checked. The previous epoch-millis finding is addressed by the shared parser changes and added coverage. Approve.
No builds or tests were run. The supplied current-head snapshot reports successful P0, compile and external regression checks; unit tests and automated review remain pending. This is an FE-focused code opinion, not a full BE review or merge authorization.
Automated FE-focused static review (gpt-6.1-sol, high). No build or tests were run.
There was a problem hiding this comment.
Reviewed #69672 at exact head ad9d044 against the supplied merge-base diff and all existing inline threads. The review converged after two bounded rounds.
No new actionable P0, P1, P2, or P3 finding was verified. The FE mapping/type, REST metadata, DSL/residual, scan-plan/property, alias-routing, and BE parser/wire scopes were covered. I traced mapping safety through scan-wide docvalue/source selection, Boolean and NOT/NULL pushdown, native esquery handling, read_full_source propagation, alias single-scroll routing, and both BE readers. Source/date/numeric/Boolean/ARRAY/JSONB boundaries and the changed unit/regression fixtures were inspected statically. A tentative epoch-millis plus-sign candidate was checked against the Elasticsearch formatter grammar and dismissed.
All existing inline threads and replies were revalidated at this head. They are prior P2 discussions with fixes or explicit scope decisions; no existing P0/P1 inline finding remains applicable, so existing_blocking_comment_ids is empty. The user focus file supplies no additional focus.
This was a static review only. I did not run builds, tests, benchmarks, live Elasticsearch reproductions, or runtime validation; author-reported results were treated as claims rather than execution evidence. No Thrift/Protobuf definitions, persistence, transaction, or write paths changed.
Review converged after 2 round(s).
FE UT Coverage ReportIncrement line coverage `` 🎉 |
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
### What problem does this PR solve? Issue Number: N/A Related PR: #69672, #69723 External ES regression suites fail during setup or retain expectations from the previous indexed-value predicate behavior: - Read the deprecated scanner option through `SELECT @@enable_file_scanner_v2`, whose compatibility name remains valid when SHOW prefixes deprecated names. - On ES 8.3.3, fall back from the specifically unsupported `_source.mode` mapping syntax to `enabled:false`, retaining the disabled-source row and alias checks. Other HTTP failures remain fatal; supporting ES8 versions still exercise the mode syntax. - Compare LIKE, REGEXP, NULL and `level` predicates against unfiltered rows materialized into internal tables, then update only the affected runner-generated golden blocks. - Make the date-string equality test explicitly use source reads, test midnight as the matching value, and retain 08:00 as a negative control. - Check that a residual predicate prevents early termination while a safe numeric docvalue predicate still permits LIMIT pushdown. ### Release note None ### Check List (For Author) - Test - [x] Regression test - [ ] Unit Test - [x] Manual test Validation uses fresh FE and ASAN BE artifacts built from this PR commit (`6386f5f83a51b548c86ab3cff83d53cb48fb18df`) with the standard `./build.sh --fe --be -j64` entry point. The complete build exited successfully, and both finished artifact versions match that commit. No production source is changed by this PR. - On the same fresh binaries and reset ES 5.6.16, 6.8.23, 7.17.5 and 8.3.3 fixtures, the parent suites/golden files reproduced all ten failures; the final PR suites passed all ten in normal comparison, with zero failed, fatal or skipped suites. The passing run executed 2,154 golden checks and 1,798 independent query comparisons, without force-generation or further golden edits. - The complete `test_es_query` passed all four ES versions and both scroll settings, including its unchanged complex-array golden blocks (276 result checks and 20 independent comparisons). - Independent SQLite/Python evaluation of the unchanged raw fixture JSON passed 38 comparisons and covered all 17 changed golden blocks. Real ES requests also confirmed that `ignore_above` omits matching index values while retaining source strings, and that the constant-keyword index matches documents whose source `level` is absent. - Captured ES search requests verified the date modes: the original four-column projection reads source, returns midnight values and rejects the 08:00 predicate. A date-only docvalue projection correctly returns 08:00 in the +08:00 session. The revised test explicitly fixes the source contract and retains the 08:00 negative control. - The complete fallback suite additionally passed normal comparison on ES 8.11.3: 304 golden checks and 304 independent comparisons. The actual mappings and document responses confirmed absent source for both the ES 8.3.3 `enabled:false` fallback and the supported ES 8.11.3 `mode:disabled` branch. - The existing `test_es_file_scanner_v2` compatibility suite also passed on the fresh build across all four fixture versions. The deprecated option is accepted; this does not claim two distinct physical reader implementations on current master. - Direct variable probes confirmed the old exact SHOW name returns no rows, the wildcard SHOW name exposes its deprecated prefix, and the scalar compatibility name remains readable. - `git diff --check` passed. ES9 compatibility is not claimed. - Behavior changed: - [x] No production behavior change; regression fixtures and expectations are corrected. - [ ] Yes. - Does this need documentation? - [x] No. - [ ] Yes. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label
…/JSON/WAL fixes (#70185) ### What problem does this PR solve? Backport the following master PRs to `branch-4.2`, in master merge order: 1. #69672 — Preserve source values and alias semantics in ES scans. 2. #69971 — Add session controls for Parquet page indexes and data-page size. 3. #70155 — Repair ES source predicate regression cases. 4. #70079 — Fix ORC condition-cache seeks past filtered stripe suffixes. 5. #70154 — Preserve JSON and WAL rows with empty file projections. The ES changes are adapted to the existing 4.2 FE planner and BE source locations. Mapping safety, scan-wide read mode, local residuals, native `esquery` handling and alias routing retain the upstream behavior without introducing the master connector framework. Added adapter tests cover both scan protocols, aliases, missing-value negations and native residual rejection. Preserve existing branch-specific ES regression coverage. Bring in the previously absent predicate suite, adapting its plan assertions to 4.2's `LOCAL_PREDICATES` output and retaining LIMIT result checks. Parquet variables use the 4.2 annotation interface and the original optional Thrift field IDs. The ORC dependency is pinned to the same fixed commit as #70079. The ES implementation also preserves source numeric and JSONB predicate semantics after scan-wide fallback. Numeric-token rounding must not make a remote predicate discard a row matching the value Doris materializes; empty JSONB objects remain nonnull even when ES indexes no leaf. Docvalue eligibility and pushdown are retained. Mapped date formats and their order are carried per concrete backing index through the existing property map to both scan protocols. Calendar/epoch alternatives now distinguish basic_date from epoch_millis, for strings and source numeric tokens. Numeric docvalues and older FE numeric requests retain their epoch decoder. Typed alias selection consistently uses the requested mapping type in schema inference and planning. The pinned ORC decoder's libdeflate dependency is included on ARM, matching master. ### Release note Fix ES source-value filtering, mapped dates and typed alias query correctness, ORC condition-cache seek crashes/missing rows and its ARM link dependency, and JSON/WAL row counts for empty file projections. Add Parquet output page-index and data-page-size session controls. ### Validation - FE compilation and Checkstyle passed. ES validation: 81 tests (48 datasource/planner tests and 33 legacy ES tests), 0 failures/errors, 1 external-environment skip. The 9 Parquet session-variable tests passed. - Rebuilt the pinned ORC dependency under AddressSanitizer: 56 focused tests passed, including 32 filtered-stripe-suffix seek cases. - Compiled all 10 affected Doris C++ implementation/test translation units and the ORC fixture generator. The regenerated fixture is byte-for-byte identical to the committed fixture. - clang-format 16 checks passed for all affected Doris C++ sources and headers. Modified Groovy suites compiled successfully. - The ES reader/parser implementations and test translation units compiled. The shared DATETIMEV2 date-format algorithm passed 21 standalone ASAN checks. CMake dependency checks passed for aarch64, arm64 and x86_64; an actual ARM link was not run locally. - Added `test_es_source_review_cases.groovy` with SQL-to-SQL checks against an internal reference table for numeric/JSONB source predicates, LIMIT, epoch-only formats, and explicit legacy DATETIMEV2 basic_date columns. It covers both scanners and parallel-scroll settings, waits for legacy metadata readiness, and restores session settings. The suite compiled; no golden files were handwritten. - The applicable ES fixes are synchronized to master in #70205. Master already links libdeflate on every architecture. - Full Doris BE unit-test execution and live external SQL regression were not performed locally; CI validation is requested. ### Check List (For Author) - Test - [x] Unit Test - [x] Regression tests included - Behavior changed: - [x] Yes; the upstream behavior is backported to 4.2. - Does this need documentation? - [x] Parquet controls are documented by the upstream documentation PR: apache/doris-website#4581 ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label
What problem does this PR solve?
Elasticsearch scans can return values that fail SQL predicates when indexed values differ from the values Doris reads, or when a scan switches all selected columns from docvalues to source. Parallel alias scans can also bypass alias filters/routing, and supported mapping shapes can fail during planning.
For example, projecting a normalized keyword forces a source read for its siblings. A numeric alias then reads NULL although its docvalue is 5; a declared empty ARRAY remains nonnull although ES considers it missing; a date-only string reads local midnight instead of the indexed UTC midnight. Boolean empty strings and coerced integer strings must also remain readable:
""means false and"1.2e3"means 1200. Keeping their indexed predicates while pruning SQL residuals returns incorrect rows.Execution flow
runtime, and preserve array annotations. Accept empty/configuration-only alias backings during safety analysis while retaining standalone schema diagnostics.ignore_malformedcan remove indexed values while retaining source values. Ignore transformations on unused sibling paths and unrelated source filters. Missing/read-changing paths disable the affected docvalue entry. Rootenabled:falseand disabled source (enabled:falseormode:disabled) restrict every affected column._idmetadata remains pushable.index=false. Require existence for SQL inequalities. Normalize retained equality negation, double negation and Boolean negation inside the connector. Retained ranges keep the original ES query complement with an existence guard: an inverse existential range changes matching on arrays such as [1,2]. This preserves NULL rejection when optimizer NOT simplification is disabled.esqueryremains remote-only: native-only expressions and independent AND conjuncts work; an indivisible Boolean expression requiring local native evaluation fails explicitly during planning.read_full_sourceflag through the existing ES property map. BE requests_source:true: disabled hits have missing values, while enabled alias backings retain their source. Ordinary source scans keep field projection and pure docvalue scans keep docvalue requests.Effects and tradeoffs
This change preserves source values, supported decoding and SQL filtering across both readers and read modes. Source fallbacks can fetch more candidate rows because unsafe predicates remain local. Mixed aliases containing disabled-source backings can fetch additional source fields to avoid ES rejecting projected-source requests. Alias scans use one coordinating scroll to preserve alias semantics; physical-index parallelism remains. No throughput improvement is claimed. There are no new public SQL/catalog/session properties or SPI/Thrift/Protobuf definitions; the optional internal flag uses the existing flat property map.
Known limitation
Mixed-generation FE/BE execution during smooth upgrades is outside this change's scope. Source plans depending on the updated decoder require the updated BE for correct results; this PR does not add a backend-generation gate.
Release note
Fix Elasticsearch source/index value discrepancies, supported Boolean/floating/integer source decoding, default epoch-millis date decoding, disabled-source reads, legacy/configuration-only mapping handling, retained negation NULL semantics and alias filter/routing bypass. Preserve eligible doc-value-only queries and reject native-query combinations requiring local evaluation during planning.
Check List (For Author)
build.sh --beASAN build and all 28EsScrollParserTestunit tests passed. The combined parser/reader run reported 47 tests: 46 passed and one optional environment-dependent live smoke test skipped; the separate live regression run exercised both readers. Clang-format 16, scoped clang-tidy and build hygiene passed for all affected C++ files. The local BE ELF requires no GLIBC version above 2.10.test_es_source_semantics: 612 stored / 516 independent checks;test_es_numeric_source_semantics: 264 / 264;test_es_mapping_boundaries: 272 / 272;test_es_source_fallback_types: 304 / 304;test_es_scalar_materialization: 160 / 160;test_es_source_value_boundaries: 168 / 168.test_es_epoch_millisadds 80 stored checks and 80 independent comparisons across both readers, docvalue/parallel modes and UTC/+08:00. It covers normalized-sibling source fallback, zero/short/fractional epoch strings, numeric doubles, negative instants, NULL/empty/singleton arrays, local predicates and pure-docvalue controls. All outputs were generated by the repository runner and verified without generation; all 80 sections also match an independent date/predicate computation. New unit coverage checks legacy/V2 date types, declared arrays, calendar-year precedence and invalid numeric/civil-year bounds.build.sh --fe; connector updates packaged through the documented module reactor recipe and deployed with the rebuilt BE for live validation.