feat: add column projection across profiling paths by QinXi-ai · Pull Request #655 · AndreaBozzo/dataprof

QinXi-ai · 2026-08-31T17:47:36Z

What changed

Add top-level column selection to AnalysisOptions, with duplicate and unknown-name validation, source-schema ordering, and an explicit empty-selection result.
Apply the same projection contract across the Rust facade, adaptive and streaming engines, CSV, JSON/JSONL, database queries, Parquet file/bytes/HTTP paths, and the Python file/bytes/dict/DataFrame/Arrow APIs.
Expand selected top-level Parquet fields to physical leaf indices and pass them through ProjectionMask::leaves, so unselected column chunks are not read and every leaf under a selected nested field is retained.
Withhold the row-level completeness and uniqueness dimensions as None whenever projection is active, because their row-level report fields are not independently optional.
Preserve typed InvalidConfiguration errors through adaptive fallback and add cross-path, nested-field, and physical byte-range regression coverage.

Related issue: Closes #638

How it was verified

$ cargo +1.98 test --workspace --all-features
all unit, integration, and documentation test suites passed; 0 failed

$ cargo +1.98 test -p dataprof --test analysis_option_parity --all-features
11 passed; 0 failed

$ cargo +1.98 test -p dataprof-engines --all-features
43 passed; 0 failed

$ cargo +1.98 clippy --all --all-targets -- -D warnings
passed

$ cargo +1.98 fmt --all -- --check
passed

$ uv run pytest --basetemp=.pytest-tmp-projection python/tests/test_analysis_option_parity.py python/tests/test_python_api.py -q
353 passed

$ uv run ruff format --check python/ .github/scripts/ .claude/skills/dataprof/scripts/
50 files already formatted

$ uv run ruff check python/ .github/scripts/ .claude/skills/dataprof/scripts/
All checks passed

$ uv run ty check python/
All checks passed

Follow-up review validation

$ cargo +1.98 test -p dataprof-db --features sqlite --test column_order
7 passed; 0 failed

$ cargo +1.98 check -p dataprof-db --all-features
passed

$ cargo +1.98 clippy --all --all-targets -- -D warnings
passed

$ python -m pytest -o addopts='' --basetemp D:/CodexCaches/pytest-dataprof-followup python/tests/test_analysis_option_parity.py python/tests/test_python_api.py -q
355 passed

$ ruff format --check ... && ruff check ... && ty check python/
passed

$ python .github/scripts/check_decode_audit.py
passed

The full workspace suite includes projected_reader_never_reads_unselected_column_chunks and top_level_projection_keeps_every_leaf_of_a_nested_column.

Anything reviewers should know

The public selector intentionally accepts top-level report column names. Dotted nested leaf paths are outside this PR; selecting a top-level struct retains all of its physical leaves.
Completeness and uniqueness are absent under projection rather than being reported as zero or silently changing meaning.
The follow-up was pushed as a normal commit without rebasing or rewriting the existing PR history.

Copilot

🟡 Changes recommended

Zero-row database results bypass projection validation, and projected Python column inputs report incomplete source-memory metadata.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds consistent top-level column projection across Rust, Python, database, streaming, and Parquet profiling paths.

Changes:

Centralizes projection validation, ordering, and quality-dimension suppression.
Adds physical nested-aware Parquet pruning.
Expands cross-path tests and Python API support.

File summaries

File	Description
`crates/dataprof-core/src/analysis_options.rs`	Defines projection contract and validation.
`crates/dataprof/src/profiler.rs`	Exposes and routes column selection.
`crates/dataprof-runtime/src/streaming_stats.rs`	Filters streaming column statistics.
`crates/dataprof-runtime/src/report_assembler.rs`	Withholds projection-sensitive quality dimensions.
`crates/dataprof-engines/src/adaptive.rs`	Propagates projection through adaptive routing.
`crates/dataprof-engines/src/streaming/incremental.rs`	Applies projection to incremental CSV profiling.
`crates/dataprof-engines/src/streaming/async_reader.rs`	Applies projection to asynchronous streams.
`crates/dataprof-json/src/lib.rs`	Filters JSON/JSONL profiles.
`crates/dataprof-parquet/src/parser.rs`	Adds physical leaf projection and tests.
`crates/dataprof-parquet/src/async_http.rs`	Projects remote Parquet reads.
`crates/dataprof-parquet/src/arrow_profiler.rs`	Projects columnar CSV batches.
`crates/dataprof-db/src/lib.rs`	Applies projection to query results.
`crates/dataprof-db/src/query_columns.rs`	Adds ordered query-column filtering.
`crates/dataprof-python/src/config.rs`	Adds Python projection configuration.
`crates/dataprof-python/src/columns.rs`	Projects Python-native column sources.
`crates/dataprof-python/src/arrow_export.rs`	Projects DataFrame and Arrow batches.
`python/dataprof/__init__.py`	Exposes projection in Python APIs.
`python/dataprof/__init__.pyi`	Updates public Python type declarations.
`python/dataprof/_dataprof.pyi`	Updates native extension stubs.
`tests/analysis_option_parity.rs`	Adds Rust cross-format parity coverage.
`python/tests/test_analysis_option_parity.py`	Adds Python cross-path projection tests.
`python/tests/test_python_api.py`	Verifies Python signatures and fluent API.

Review details

Files reviewed: 22/22 changed files
Comments generated: 5
Review effort level: Balanced

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

AndreaBozzo · 2026-08-31T18:05:30Z

One small housekeeping item: this PR appears to fully implement #638, but Refs #638 will not close the issue automatically. Please change it to Closes #638 so the ticket is closed when this merges. Thanks again for taking this on!

AndreaBozzo

Thank you so much for this contribution, @QinXi-ai, and especially for the careful, comprehensive follow-up. You addressed every review point with focused regression coverage: zero-row database projection now preserves and validates the schema, source-memory provenance remains stable, the Python positional API stays backward compatible, the output contract is documented, and the decode-audit gate is clean. I also re-ran the focused database tests and decode audit on the current merge commit; everything passes. This is a substantial cross-path improvement—great work!

* docs(python): document column projection `columns` shipped in #655 and reached the type stubs, `Profiler`, and `ProfilerConfig`, but never the Python API guide: the documented `dp.profile()` signature omitted it and no section described it. Record the parameter and the semantics that are easy to guess wrong: projection selects columns and never rows, the report keeps source order rather than the order the names were listed in, unknown and repeated names raise instead of being ignored, `columns=[]` differs from `columns=None`, and quality is assessed over the projection only, so the score answers a narrower question than the whole-dataset score. * docs(python): record projection's async surface and withheld dimensions Two gaps from review of the projection section. The entry-point list omitted `dataprof.asyncio`; all three of its entry points forward `**kwargs` through `ProfilerConfig`, so projection was documented as if it were sync-only. Quality was described as merely narrowing to the projected columns. It also withholds `completeness` and `uniqueness`, whose row-level fields would otherwise report projected measurements under full-row names. That is worth stating outright: it leaves both dimensions `None`, drops the overall score to an average over fewer dimensions — usually raising it, since `completeness` is what penalizes nulls — and yields no quality object at all when a projection is combined with `quality_dimensions` naming only those two.

feat: add column projection across profiling paths

6064f01

QinXi-ai requested a review from AndreaBozzo as a code owner August 31, 2026 17:47

github-actions Bot added area: rust Pull requests that update rust code area: python Python bindings and API area: database Database connectors and SQL area: testing Tests and quality assurance area: streaming Streaming/real-time processing labels Aug 31, 2026

AndreaBozzo requested a balanced review from Copilot August 31, 2026 17:56

Copilot started reviewing on behalf of AndreaBozzo August 31, 2026 17:56

View session

AndreaBozzo assigned QinXi-ai Aug 31, 2026

AndreaBozzo added this to dataprof Aug 31, 2026

AndreaBozzo added this to the 0.12.0 milestone Aug 31, 2026

Copilot AI reviewed Aug 31, 2026

View reviewed changes

Comment thread crates/dataprof-db/src/lib.rs Resolved

Comment thread crates/dataprof-python/src/columns.rs Resolved

Comment thread crates/dataprof/src/profiler.rs Resolved

Comment thread python/dataprof/__init__.py Resolved

Comment thread python/dataprof/__init__.py Resolved

AndreaBozzo reviewed Aug 31, 2026

View reviewed changes

Comment thread crates/dataprof-python/src/config.rs Resolved

AndreaBozzo reviewed Aug 31, 2026

View reviewed changes

Comment thread crates/dataprof-python/src/arrow_export.rs Resolved

AndreaBozzo mentioned this pull request Aug 31, 2026

docs(ci): make the decode-audit rule clear in the contributor path #657

Closed

QinXi-ai added 2 commits September 1, 2026 02:10

fix(python): classify default config audit

edee870

fix: preserve column projection compatibility

4a650bc

AndreaBozzo approved these changes Sep 1, 2026

View reviewed changes

AndreaBozzo merged commit f5ad0b1 into AndreaBozzo:master Sep 1, 2026
23 checks passed

github-project-automation Bot moved this to Done in dataprof Sep 1, 2026

AndreaBozzo mentioned this pull request Sep 7, 2026

docs(python): document column projection #695

Merged

AndreaBozzo mentioned this pull request Sep 7, 2026

Measure physical reads and accumulator work for Parquet metric selections #697

Open

5 tasks

Navigation Menu

Repository navigation

feat: add column projection across profiling paths - #655

feat: add column projection across profiling paths#655
AndreaBozzo merged 3 commits into
AndreaBozzo:masterfrom
QinXi-ai:feature/column-projection-v2

QinXi-ai commented Aug 31, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

AndreaBozzo commented Aug 31, 2026

Uh oh!

AndreaBozzo left a comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

QinXi-ai commented Aug 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changed

How it was verified

Follow-up review validation

Anything reviewers should know

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

AndreaBozzo commented Aug 31, 2026

Uh oh!

AndreaBozzo left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

QinXi-ai commented Aug 31, 2026 •

edited

Loading