Integrate Record Skipping into Column Reader Fuzz Test#2315
Integrate Record Skipping into Column Reader Fuzz Test#2315tustvold merged 3 commits intoapache:masterfrom
Conversation
|
Got 4 test error 😂 will dive into |
|
There are two kinds of erroe one is parquet/src/arrow/arrow_reader.rs:1260 |
|
as test |
|
Awesome! FWIW it might help debugging to seed the random number generator to get a deterministic test. I've found the fuzz tests are very good at finding bugs, but can be quite hard to then debug 😅 |
Thanks ! i have found the combination args cause bug 😂 |
From the error message, I guess the error is related to the read/convert parquet data to arrow, we need the target arrow type and parquet logical type/convertedtype |
|
@tustvold seems all tests passed. I copy the all test case with page static. 😊 |
| let mut skip_data: Vec<Option<T::T>> = vec![]; | ||
| for select in selections { | ||
| if select.skip { | ||
| without_skip_data.drain(0..select.row_count); |
| min(total_read + opts.record_batch_size, expected_data.len()); | ||
| let batch = maybe_batch.unwrap().unwrap(); | ||
| assert_eq!(end - total_read, batch.num_rows()); | ||
| //TODO remove this after implement https://github.com/apache/arrow-rs/issues/2197 |
There was a problem hiding this comment.
The linked issue has been done?
There was a problem hiding this comment.
Nope, maybe merge this and use this test to check the result
|
Benchmark runs are scheduled for baseline = 38764c2 and contender = 5676c6e. 5676c6e is a master commit associated with this PR. Results will be available as each benchmark for each run completes. |
4 participants
Which issue does this PR close?
Closes #2198.
Rationale for this change
What changes are included in this PR?
Are there any user-facing changes?