Repository navigation
Conversation
) ### What problem does this PR solve? Issue Number: N/A Problem Summary: Queries such as: ```sql SELECT COUNT(*) FROM example_catalog.information_schema.schemata; ``` only need database and catalog names. However, `SchemaSchemataScanner` uses the default `getDbNames` RPC, whose FE handler calls `catalog.getAllDbs()` to also return database IDs. For external catalogs, this resolves every database object even though SCHEMATA never reads those IDs. A large namespace list can therefore churn a much smaller database-object cache and allocate substantial temporary metadata while waiting inside a synchronous BE-to-FE RPC. A warm database-name cache does not remove this work. This PR adds the optional `include_db_ids` request field, defaulting to `true`, and lets SCHEMATA set it to `false`. The FE then calls `catalog.getDbNames()` and omits the unused database IDs. Both paths share the existing privilege checks, pattern matching, and schema-name formatting. Callers that need IDs keep the object-enumeration path without additional intermediate lists. The new field uses ID 8; field 7 remains reserved. Definitions were checked on master and branch-3.1/4.0/4.1/4.2. Older FE nodes ignore the optional field and keep serving the existing response; older BE callers retain the default ID-inclusive behavior on a new FE. ### Validation - Added a 100,000-namespace test that rejects any database-object enumeration and verifies the result-array lengths and omitted IDs. - Covered privilege/pattern filtering, qualified schema names, empty/unavailable catalogs, existing ID-returning callers, and binary deserialization of requests without the new field. - Confirmed the new behavior tests fail on the original handler before implementing the change. All 7 targeted FE tests pass, including the existing `FrontendServiceImplTest#testGetDBNames`. ```bash bash run-fe-ut.sh --run 'org.apache.doris.service.FrontendServiceGetDbNamesTest,org.apache.doris.service.FrontendServiceImplTest#testGetDBNames' ``` - FE Checkstyle: passed with zero violations. - Changed BE scanner: clang-format 16 and C++ syntax-only compilation with regenerated Thrift headers passed. An in-process FE-handler benchmark uses the actual `ExternalCatalog` / `TestExternalCatalog` name and object-cache implementations, a synthetic metadata provider, and stubbed catalog routing and privilege checks. It compares the original handler source at master commit `bdb90856e6e468eb495e5bfd67010de90570bfbc` with the patched handler, using the same dependencies and fixture. Each trial starts with a warm name cache and cold object cache, whose capacity is 1,000. There are 100,000 synthetic namespaces plus the two system databases. Settings: JDK 17, 2 GiB heap, 4 active processors; discard one warm-up pair and report medians from three measured trials. | Metric | Original master | Patched SCHEMATA path | | --- | ---: | ---: | | FE handler elapsed time | 2,984.2 ms | 1,331.1 ms | | Cumulative current-thread allocation | 2,117.2 MiB | 726.2 MiB | | Database objects constructed | 100,002 | 0 | | Name-provider invocations | 1 | 1 | | Returned rows | 100,002 | 100,002 | In this fixture, handler time falls by 55.4% (2.24x faster) and cumulative allocation falls by 65.7%. Allocation is total bytes allocated during the call, not peak live heap. This benchmark excludes network transport, Thrift serialization, and SQL execution; it does not measure a deployed PostgreSQL/Iceberg environment. Full BE linking and a deployed end-to-end SQL test were not run locally. ### Release note Improve `information_schema.schemata` enumeration for large catalogs by avoiding unnecessary database-object loading. ### Check List (For Author) - Test - [ ] Regression test - [x] Unit Test - [x] Manual test (FE-handler benchmark described above) - [ ] No need to test or manual test. - Behavior changed: - [x] No. SQL result semantics and existing ID-returning requests are preserved. - [ ] Yes. - Does this need documentation? - [x] No. - [ ] Yes. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label (cherry picked from commit 74b477d)
…apache#70079) ### What problem does this PR solve? Related PR: apache/doris-thirdparty#431 (merge the ORC dependency fix first) A condition-cache granule can start in a stripe's SARG-filtered suffix while containing a matching row in the next stripe. On an initialized stripe, ORC's `seekToRow()` uses the stripe-end sentinel as a row-group index. This can access past the index when the stripe ends on an index boundary, or prematurely return EOF when the last row group is partial. The same-stripe fast path originates in [Apache ORC ORC-1170](apache/orc#1114); the same missing stripe-end handling remains in the current upstream C++ reader. Pin the ORC fix that initializes the next selected stripe before positioning its column readers. Preserve SARG pruning, lazy reads and split boundaries. Add a Doris reader regression and a 1.5 KB synthetic ORC fixture with its generator. The fixture has two 20,000-row stripes, a 10,000-row index stride, and one matching row at the start of each stripe. A cache hit seeks to row 18,432 in the first stripe's filtered suffix. The SQL regression compares cache-disabled and cached results with lazy reads and min/max pruning enabled or disabled, and checks the profile for actual cache hits and skipped rows. ### Release note Fix potential crashes and missing rows when ORC condition-cache seeks reach a SARG-filtered stripe suffix. ### Check List (For Author) - Test: - [x] Unit Test: 247 ORC-related BE tests passed under AddressSanitizer, including the new Doris condition-cache case and 32 dependency boundary cases. - [x] Unit Test: 56 focused standalone ORC tests passed under AddressSanitizer. Reverting the fix makes all 16 initialized-stripe boundary cases fail. - [ ] Regression test: SQL suite added; Groovy compilation checked. The external SQL suite has not been executed locally. - [x] clang-format 16 and BE build-hygiene checks passed. - [ ] Local full-file clang-tidy analysis did not complete within ten minutes; no completed result is available. - Behavior changed: - [x] Yes. Safely advance to the next matching stripe or EOF. - Does this need documentation? - [x] No. (cherry picked from commit 75ab5ba)
…che#70154) ### What problem does this PR solve? Issue Number: close apache#69926 Problem Summary: Hive JSON queries that only need partition columns can return zero rows even when the selected files contain records. For example, `SELECT * ... WHERE dt = 'p1'` returns data while `SELECT COUNT(*) ... WHERE dt = 'p1'` returns zero. The V2 JSON reader derived its row count from the output block, which has no columns after physical field pruning. Writer-backed reader tests reproduce the same row-count loss for empty WAL projections. Track accepted JSON records independently of materialized columns, including batch progress, array expansion, error rollback, and ignore-malformed NULL rows. Preserve JSON-path record selection for empty projections, and let the existing TableReader materialize partition values using the returned row count. For WAL, save the deserialized source row count before materialization transfers column ownership, then pass it through the existing filter pipeline. This preserves the row count when no file columns are requested. The empty-projection reproduction is at the reader interface; production replay reachability of an empty projection remains unverified. Add ten JSON BE tests covering empty file schemas and projections, arrays, JSON roots/paths, malformed input, batch boundaries, EOF, and actual TableReader partition materialization. Add eight writer-backed WAL cases covering empty, single-column, and reordered full projections across multiple blocks, with and without schema discovery, plus repeated EOF reads. Add a WAL SQL regression that forces two original async-load transactions to abort, then verifies row counts, NULLs, aggregates, filters, and actual values after replay. Add a Hive SQL regression using the existing partitioned JSON fixture for COUNT, partition projections, grouping, and file-column filters with external COUNT pushdown enabled and disabled. ### Release note Fix incorrect zero counts and missing rows when querying only partition columns of Hive JSON tables. Preserve logical row counts for empty WAL reader projections. ### Check List (For Author) - Test - [x] Unit Test: 166 ASAN tests passed, including JSON reader, TableReader, WAL V2 reader, WAL reader/writer, and CSV/Text empty-projection controls. Before the JSON fix, nine of ten new tests failed; before the WAL fix, both empty-projection cases failed while six projection controls passed. - [x] WAL SQL regression: `test_group_commit_wal_row_count` passed twice on isolated Doris 4.1.5 with scanner V2 enabled. Generated the `.out` file with `run-regression-test.sh`, then reran for result comparison. Execution on master is pending CI. - [x] Hive SQL regression: added `test_hive_json_partition_only`; Groovy compilation passed. Full execution against the external Hive fixture is pending CI. - Behavior changed: - [x] Yes. Empty JSON and WAL file projections preserve logical row counts. Hive JSON partition-only queries produce the correct rows and aggregates. - Does this need documentation? - [x] No. (cherry picked from commit 4bde8d0)
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
|
run buildall |
|
/review |
There was a problem hiding this comment.
Request changes for one P1: the ORC revision makes libdeflate mandatory on ARM64, but this branch still links it only on x86. The inline finding identifies the missing companion dependency change. No existing blocking inline findings were present.
Reviewed all 17 changed paths and all three commits included by the ORC revision, with independent JSON/WAL, SCHEMATA/ORC-integration and risk-focused coverage. The other intended fixes look consistent with their caller contracts under static inspection.
Critical checkpoints:
- Goal and proof: names-only SCHEMATA avoids database-object enumeration; JSON/WAL preserve logical rows with empty projections; the intended ORC seek fix handles filtered stripe suffixes. Added tests target each behavior; they were inspected, not executed.
- Scope and clarity: local changes are focused; the dependency pin includes two additional decompression changes. The all-architecture libdeflate change needs the companion linkage fix reported inline.
- Concurrency: no new threads, shared mutable state or locking. Reader state remains scanner-owned; catalog name enumeration uses the existing cache/snapshot synchronization.
- Lock cost and order: no new critical sections, heavy work under locks or lock-order changes identified.
- Lifecycle and static initialization: logical row state resets per batch; malformed rollback restores it with columns. WAL captures row count before ownership moves. ORC stripe transitions reset readers/indexes/followers; decompression scratch retains its owning pool. No new ownership cycle or cross-translation-unit initializer dependency.
- Configuration: no configuration items added or dynamic-update contract changed.
- Compatibility: optional Thrift field 8 defaults to true, preserving legacy ID-returning requests; old FEs can ignore it. No storage/WAL format change. ARM64 link compatibility is the reported defect.
- Thrift/protobuf identity: compared the touched definition on master and branches 4.2, 4.1, 4.0 and 3.1. Master/4.2 have the identical optional field 8; older branches have no conflicting field. Reserved field 7 remains intact; no required field or enum change.
- Parallel paths: eager/lazy ORC and SARG on/off are covered by added tests; the dependency affects both reader generations. JSON roots/paths/outer arrays and WAL empty/subset/reordered/full projections were traced through table materialization.
- Conditional checks: JSON path absence, root selection, malformed-null handling and ORC stripe-end/range/EOF conditions preserve their documented boundaries; no additional defect verified.
- Test coverage: inspected FE name/privilege/pattern/default-wire tests; JSON malformed/empty/root/path/batch/partition tests; WAL multiple-block/projection/EOF/replay tests; ORC filtered-suffix, lazy/range/backward-seek and decompression tests. ARM64 linkage needs coverage after fixing the finding.
- Expected results: WAL golden rows, null counts, sums and predicate count match input; ORC expected IDs/payloads match the generator and static fixture stripe metadata. Query comparisons are ordered. Generation/execution of expected artifacts was not repeated.
- Observability: existing row, parse/materialization/filter, ORC seek/cache and I/O counters remain on the affected paths; no additional logging/metrics defect identified.
- Transactions and persistence: no EditLog, visible-version, publish/delete-bitmap or WAL-byte-format change. Replay transaction handling and failover contracts are unchanged by the row-count capture.
- Data writes, atomicity and crash behavior: production edits change enumeration/read accounting, not commit/write atomicity; no new crash-cleanup, leak or deadlock path identified in the changed lifecycle.
- FE-BE propagation: only SCHEMATA opts out of database IDs; its consumer does not read IDs. Other senders retain the default behavior; no further transmitted state introduced.
- Performance and memory: name enumeration avoids per-database object loading; row counting adds constant state. Lazy ORC compressed-input scratch still grows before split-chunk copies and is reused. No additional verified allocation or complexity regression.
- Other issues: the complete dependency comparison exposed the ARM64 link defect; no second actionable issue survived independent rechecking.
No additional user focus was supplied. All four initial risk questions have explicit dispositions in the ledger; there are no unresolved candidates or static coverage gaps. Review converged in round 2. Validation limits: static review only; no builds, tests, benchmarks or external SQL were executed, and submodules were left uninitialized.
Review converged after 2 round(s).
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
|
run buildall |
|
/review |
There was a problem hiding this comment.
Static review of PR #70319 at head 47c2235 found no new actionable P0-P3 findings across all changed paths. The previously reported P1 on contrib/apache-orc (comment 4488893681) is resolved at this head: be/cmake/thirdparty.cmake now adds deflate on every architecture, so it is not reposted.
Critical checkpoints: JSON v2 now tracks accepted logical rows independently of empty file-column blocks; its array/root/path, malformed rollback, batch, EOF, filter, and TableReader paths remain aligned. WAL captures rectangular source-block rows before projection moves ownership, then passes that count through the existing materialized-filter pipeline. The FE name-only request uses optional Thrift field 8 with default true, so old or unset callers retain database IDs; only SCHEMATA opts out, and its non-ID response lists stay aligned with privileges, patterns, display-name formatting, and empty-catalog handling. The ORC revision advances from a filtered stripe suffix to the next selected stripe and the added tests cover initialized/fresh seeks, split ranges, lazy reads, min/max combinations, both columns, EOF, and backward seeks.
No new threads, locks, configuration items, storage/WAL wire formats, persistence paths, or ownership cycles were introduced. Added unit and regression tests were inspected for deterministic assertions and cleanup; they were not executed. This was a static review only: no builds, tests, benchmarks, or external SQL were run, and repository submodules remained uninitialized. No additional user-provided review focus was supplied.
Review converged after 1 round(s).
FE UT Coverage ReportIncrement line coverage |
What problem does this PR solve?
Backport three master fixes to
branch-4.1: avoid unnecessary database-object loading for SCHEMATA scans, prevent ORC condition-cache seek crashes or missing rows, and preserve JSON/WAL row counts when no file columns are projected.The commits retain their original authors and are applied in master merge order:
Branch adaptations:
include_db_idsflag at upstream field 8 with its defaulttrue. Preserve existing privilege checks, pattern matching and schema-name formatting for both enumeration paths.5962bb071d6345cc29c5353d64d85a203525e4d9, which descends from the branch's existing revision.Validation
FrontendServiceImplTest#testGetDBNames, the 100,000-namespace case, filtering and legacy-request compatibility. FE Checkstyle passed.Release note
Improve SCHEMATA enumeration for large catalogs. Fix potential crashes or missing rows during ORC condition-cache seeks and incorrect JSON/WAL row counts with empty file projections.
Check List (For Author)
Check List (For Reviewer who merge this PR)
ARM64 link dependency follow-up
The ORC revision enables libdeflate ZLIB decoding on every architecture. Remove the old x86-only guard around
add_thirdparty(deflate)so ARM64 BE links also receive the required library, matching master. An adjacent comment records this dependency invariant.Validation: probes include each branch's actual
be/cmake/thirdparty.cmakeand selectaarch64,arm64,x86andx86_64dependency configurations. Before the fix, both ARM configurations fail to link with missing libdeflate symbols while both x86 controls pass. After the fix, all eight repository/configuration combinations link and pass a DEFLATE compression/decompression round trip. These probes run natively on x86_64 and validate CMake dependency selection; a native ARM64/full BE build remains for CI. Clang-format 16 passed for the PR-affected C++ files.