You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 1871546
Browse filesBrowse the repository at this point in the historyBrowse files
### What problem does this PR solve?
Issue Number: N/A
Related PR: apache#65921
Problem Summary:
When a fixed-width Parquet predicate column was also projected, Doris
materialized all selected predicate values and compacted the complete
column after filtering. The native decoder had already produced the
values needed by the raw predicate, so this added an avoidable
materialize-and-compact pass. The overhead is visible in TPC-DS Q88.
### What is changed?
Evaluate eligible predicates on decoded fixed-width values and append
only matching values to the projected column in the same decoder pass.
- Generalize the PLAIN-specific consumer and reader APIs into a
fixed-width raw filter-and-project path.
- Support PLAIN and BYTE_STREAM_SPLIT for identity-width INT32, INT64,
FLOAT, and DOUBLE values.
- Support DELTA_BINARY_PACKED for INT32 and INT64 values.
- Prevalidate advertised chunk encodings before consuming definition
levels.
- Reject an unexpected unsupported late-page encoding instead of
attempting a fallback after cursor progress.
- Keep the existing dictionary-id filter and dictionary materialization
path unchanged.
The raw decoder cannot rewind after predicate evaluation. Therefore,
when the predicate column is projected, survivors must be appended
before the encoded values and definition-level cursor are consumed.
Predicate-only columns retain their placeholder behavior. Nested
columns, converted logical types, residual/delete predicates,
unsupported expressions, and unsupported encodings continue to use the
existing materializing fallback.
### Microbenchmark
The reader microbenchmark from apache#65921 was run with a Release build, warm
fixture cache, CPU 8, a one-second minimum time, and 10 repetitions.
The matrix adds BYTE_STREAM_SPLIT and DELTA_BINARY_PACKED coverage with:
- 10% alternating NULLs;
- predicate-only and predicate-projected modes;
- 1%, 10%, 50%, and 90% selectivity.
All 16 combinations completed with the expected raw and selected row
counts.
Projected, 10% selectivity:
| Encoding | CPU ns/raw row, master | CPU ns/raw row, PR | Improvement |
CPU CV master / PR |
|---|---:|---:|---:|---:|
| BYTE_STREAM_SPLIT | 36.41 | 33.43 | 8.2% | 1.25% / 1.47% |
| DELTA_BINARY_PACKED | 38.75 | 36.84 | 4.9% | 5.40% / 2.94% |
The Delta baseline ran under sustained host load, so its result should
be treated as directional.
The original projected PLAIN results remain:
| Selectivity | CPU ns/raw row, master | CPU ns/raw row, PR |
Improvement |
|---:|---:|---:|---:|
| 1% | 32.80 | 30.80 | 6.1% |
| 10% | 33.95 | 31.82 | 6.3% |
| 50% | 37.76 | 35.05 | 7.2% |
| 90% | 42.06 | 37.66 | 10.5% |
### Release note
None
### Check List (For Author)
- Test: Unit Test and Manual test
- 17 targeted ASAN unit tests covering raw expression evaluation,
nullable mapping, projected PLAIN/BSS/Delta scans, unsupported-type
fallback, mixed-page safety, residual predicates, and benchmark matrix
constraints
- Release benchmark target linked successfully
- 16-case BSS/Delta reader microbenchmark matrix
- Before/after projected-filter microbenchmarks
- clang-format 16, check-format, and `git diff --check`
- clang-tidy was attempted but blocked by existing master/toolchain
errors: unmatched `NOLINTEND` in `be/src/core/types.h` and missing
`stddef.h` from the configured toolchain
- Behavior changed: No. This extends the existing raw predicate
optimization to equivalent fixed-width decoder output.
- Does this need documentation: No
0 commit comments