(DEMO) tests pass when validate_full is forced#987
Closed
alamb wants to merge 5 commits intoapache:masterfrom
Closed
(DEMO) tests pass when validate_full is forced#987alamb wants to merge 5 commits intoapache:masterfrom
validate_full is forced#987alamb wants to merge 5 commits intoapache:masterfrom
Conversation
alamb
commented
Nov 30, 2021
arrow/src/array/data.rs
Outdated
Contributor
Author
There was a problem hiding this comment.
this is the demo -- it is not for merging, but for demonstrating that all the tests pass even when we run validate_full() on them
1 task
alamb
changed the title
validate_full is forcedvalidate_full is forced
alamb
force-pushed
the
alamb/remove_special_checks
branch
from
ca5a429 to
72906d5
Compare
alamb
commented
Nov 30, 2021
| } | ||
|
|
||
| #[test] | ||
| #[ignore] |
Contributor
Author
There was a problem hiding this comment.
This test fails as follows:
---- arrow::arrow_reader::tests::test_read_maps stdout ----
thread 'arrow::arrow_reader::tests::test_read_maps' panicked at 'called `Result::unwrap()` on an `Err` value: InvalidArgumentError("Child type mismatch for Struct([Field { name: \"key\", data_type: Utf8, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: None }, Field { name: \"value\", data_type: Map(Field { name: \"key_value\", data_type: Struct([Field { name: \"key\", data_type: Int32, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: None }, Field { name: \"value\", data_type: Boolean, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: None }]), nullable: false, dict_id: 0, dict_is_ordered: false, metadata: None }, false), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: None }]). Expected Map(Field { name: \"key_value\", data_type: Struct([Field { name: \"key\", data_type: Int32, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: None }, Field { name: \"value\", data_type: Boolean, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: None }]), nullable: false, dict_id: 0, dict_is_ordered: false, metadata: None }, false) but child data had Map(Field { name: \"key_value\", data_type: Struct([Field { name: \"key\", data_type: Int32, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: None }, Field { name: \"value\", data_type: Boolean, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: None }]), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: None }, false)")', arrow/src/array/data.rs:308:34
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
(the issue is the nullability of the child field is declared differently on the Struct's datatype than on the ArrayData which I think is an actual bug in the array reader, and will file a follow on ticket)
Codecov Report
@@ Coverage Diff @@
## master #987 +/- ##
==========================================
- Coverage 82.31% 82.31% -0.01%
==========================================
Files 168 168
Lines 48763 49008 +245
==========================================
+ Hits 40139 40339 +200
- Misses 8624 8669 +45
Continue to review full report at Codecov.
|
Contributor
Author
|
This PR has served its purpose. RIP |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Built on #921
This PR demonstrates that (most) of the tests pass if I (temporarily) force
ArrayData::new_unchecked()to check all inputs, which gives me additional confidence thatvalidate_full()is validating inputs as it supposed to.The only one that fails is related to the arrow_reader not having its nullability set correctly which I think is a bug in the array reader and which I will file separately