-
Notifications
You must be signed in to change notification settings - Fork 1.1k
Description
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
For better or worse arrow-rs makes use of unsafe in various places. Whilst there are ongoing efforts to reduce the use of unsafe, occasionally bugs will slip through. Both arrow-rs and arrow-datafusion run with MIRI in order to catch many of these, however, MIRI is extremely slow and fiddly to setup which acts as a limiting factor on crowd-sourced test coverage. Additionally it will not catch the arguably more common types of logic bug, e.g. inconsistent null counts, non-consecutive array offsets, etc...
Describe the solution you'd like
Add a test_validate feature flag that when enabled will cause various "unchecked" methods to actually perform validation. The most obvious being ArrayDataBuilder::build_unchecked. This will allow projects to contribute to crowd-sourced nasal demon hunting, without imposing on them all the costs of setting up and running MIRI.
Describe alternatives you've considered
This could also be enabled based on debug_assertions being enabled, but this is a global setting that is on by default in debug builds. A feature flag gives people the ability to opt-out of the validation. FWIW this is what the parquet crate currently does.