Skip to content

Add test_validate feature flag + CI check #1239

@tustvold

Description

@tustvold

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

For better or worse arrow-rs makes use of unsafe in various places. Whilst there are ongoing efforts to reduce the use of unsafe, occasionally bugs will slip through. Both arrow-rs and arrow-datafusion run with MIRI in order to catch many of these, however, MIRI is extremely slow and fiddly to setup which acts as a limiting factor on crowd-sourced test coverage. Additionally it will not catch the arguably more common types of logic bug, e.g. inconsistent null counts, non-consecutive array offsets, etc...

Describe the solution you'd like

Add a test_validate feature flag that when enabled will cause various "unchecked" methods to actually perform validation. The most obvious being ArrayDataBuilder::build_unchecked. This will allow projects to contribute to crowd-sourced nasal demon hunting, without imposing on them all the costs of setting up and running MIRI.

Describe alternatives you've considered

This could also be enabled based on debug_assertions being enabled, but this is a global setting that is on by default in debug builds. A feature flag gives people the ability to opt-out of the validation. FWIW this is what the parquet crate currently does.

Metadata

Metadata

Assignees

No one assigned

    Labels

    development-processRelated to development process of arrow-rsenhancementAny new improvement worthy of a entry in the changelog

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions