Added tests with are writing into parquet files in memory for issue #…#15325
Added tests with are writing into parquet files in memory for issue #…#15325alamb merged 1 commit intoapache:mainfrom
Conversation
alamb
left a comment
alamb
left a comment
There was a problem hiding this comment.
Thank you @pranavJibhakate
FYI @XiangpengHao
| ) | ||
| .unwrap(); | ||
|
|
||
| writer.write(&batch).unwrap(); |
There was a problem hiding this comment.
Thanks @pranavJibhakate !
Would you be willing to add a test that the same data could be read back from the parquet file as well?
There was a problem hiding this comment.
I agree; I think the current code tests the re-exported Parquet functionalities, not touching the DataFusion-related code. Ideally, we should test the end-to-end Parquet reading process.
The process roughly looks like this:
- Create a in-memory object_store, and put the Parquet data you generated into the object_store.
- Register the object_store along with the path to the DataFusion.
- Run a SQL query from the DataFusion side to see if the results can be read back.
A loosely related test can be found here: https://github.com/XiangpengHao/parquet-viewer/blob/main/src/tests.rs#L9
There was a problem hiding this comment.
I filed a ticket to track this work:
|
Thanks again @pranavJibhakate |
3 participants
…15158
Which issue does this PR close?
Rationale for this change
What changes are included in this PR?
Are these changes tested?
Are there any user-facing changes?