Move more parquet functionality behind experimental feature flag (#1032) #1134
Merged
alamb merged 2 commits intoapache:masterfrom Jan 10, 2022
Merged
Move more parquet functionality behind experimental feature flag (#1032) #1134alamb merged 2 commits intoapache:masterfrom
alamb merged 2 commits intoapache:masterfrom
Conversation
tustvold
force-pushed
the
parquet-experimental
branch
from
4c6b9b2 to
1dc30dc
Compare
tustvold
commented
Jan 5, 2022
parquet/src/compression.rs
Outdated
Contributor
Author
There was a problem hiding this comment.
This is necessary because without the experimental feature this will fail to compile complaining about compression being private.
FWIW this module is never documented now, but I opted to preserve the test.
You can see it running with:
$ cargo test --all-features --doc
Finished test [unoptimized + debuginfo] target(s) in 0.04s
Doc-tests parquet
running 19 tests
test src/arrow/array_reader.rs - arrow::array_reader::StructArrayReader::next_batch (line 1092) ... ignored
test src/arrow/array_reader.rs - arrow::array_reader::StructArrayReader::next_batch (line 1098) ... ignored
test src/arrow/array_reader.rs - arrow::array_reader::StructArrayReader::next_batch (line 1103) ... ignored
test src/file/mod.rs - file (line 64) - compile ... ok
test src/compression.rs - compression::CodecType (line 27) - compile ... ok
test src/file/mod.rs - file (line 29) - compile ... ok
test src/file/mod.rs - file (line 81) - compile ... ok
test src/record/api.rs - record::api::Row::get_column_iter (line 62) - compile ... ok
test src/column/mod.rs - column (line 38) - compile ... ok
test src/file/properties.rs - file::properties (line 22) ... ok
test src/schema/types.rs - schema::types::ColumnPath::append (line 674) ... ok
test src/arrow/mod.rs - arrow (line 58) ... ok
test src/schema/types.rs - schema::types::ColumnPath::string (line 663) ... ok
test src/schema/mod.rs - schema (line 22) ... ok
test src/schema/printer.rs - schema::printer (line 23) ... ok
test src/schema/parser.rs - schema::parser (line 24) ... ok
test src/file/statistics.rs - file::statistics (line 23) ... ok
test src/arrow/mod.rs - arrow (line 25) ... ok
test src/arrow/mod.rs - arrow (line 71) ... ok
test result: ok. 16 passed; 0 failed; 3 ignored; 0 measured; 0 filtered out; finished in 3.12s
1dc30dc to
ecf5952
Compare
alamb
force-pushed
the
parquet-experimental
branch
from
ecf5952 to
7e588f0
Compare
Contributor
|
Rebased on top of master |
alamb
approved these changes
Jan 5, 2022
Contributor
Author
|
Looking into test failures |
asayers
added a commit
to asayers/sqlite2parquet
that referenced
this pull request
Jan 28, 2022
This requires adding the "experimental" flag because of apache/arrow-rs#1134
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft as builds on #1133
Which issue does this PR close?
Closes #1032
Rationale for this change
See ticket
What changes are included in this PR?
This hides a number of modules behind an "experimental" feature flag. These largely seem to contain implementation details, and so making them private will allow for faster iteration without having to worry about breaking clients that haven't explicitly opted in to instability.
Are there any user-facing changes?
Yes, this moves a number of APIs to be crate-local. I am skeptical people are using these APIs, but it is possible.