Repository navigation
Description
Problem
parquet-testing-runner tracks apache/parquet-testing master at HEAD. ParquetTestingRepoCloner.ensureCloned() shallow-clones with no ref, and target/ is wiped by mvn clean, so CI re-clones on every run.
Enumeration is by directory walk. Utils.parquetTestFiles() walks shredded_variant/ and takes every *.parquet, and VariantShredReassemblerTest uses a case-*.parquet glob. The skip-lists, though, key on exact filenames:
Utils.SKIPPED_FILESlists"case-041.parquet","case-131.parquet","case-138.parquet"VariantShredReassemblerTest.UNREADABLElists the same three
So when a fixture is renamed upstream the walk still finds it, but the skip-lists stop matching it.
How it surfaced
apache/parquet-testing#117 relabels four shredded_variant fixtures whose variant group omits the spec-required value column, following the convention already used by cases 43, 84 and 125:
| old | new |
|---|---|
case-041.parquet |
case-041-INVALID.parquet |
case-131.parquet |
case-131-INVALID.parquet |
case-132.parquet |
case-132-INVALID.parquet |
case-138.parquet |
case-138-INVALID.parquet |
The goldens are renamed to match and cases.json is updated in lockstep with a notes entry. No file contents change.
Once that merges, cases 41, 131 and 138 stop being skipped and start being exercised. Case 132 is in neither skip-list today, so it is unaffected beyond the new name.
I have not run the suite against the renamed corpus, so I will not claim whether those three then pass or fail. Either way the skip-lists no longer do what they say.
Proposed fix
- Update the three names, or move them into
INVALID_FIXTURES, which is where they now belong. - Optionally make it rename-proof: take fixture filenames from
cases.jsonrather than walking the directory, and key the invalid-case skip off the-INVALIDsubstring and thenotesfield. arrow-go does the substring skip, and arrow-rs drives everything fromcases.json, so it needs no change for this PR.
Acceptance criteria
- After bumping to a parquet-testing revision containing the rename, cases 41, 131 and 138 are handled deliberately rather than by a stale filename match.
- Cases 43, 84, 125 and 132 keep their current behaviour.
Related
- Mark shredded_variant cases 41, 131, 132, 138 as INVALID (missing 'value' column) apache/parquet-testing#117
- Variant: report that arrow-rs produces "bad" shredded variant apache/arrow-rs#10306 (strict readers reject a variant group with no
valuecolumn)
Activity
gunnarmorling commentedon Jul 13, 2026
Thanks for logging, good catch. I wasn't aware of
cases.jsonactually, thanks for that pointer! We'll look into this asap.gunnarmorling commentedon Jul 15, 2026
@rionmonster, this one might be interesting to you, perhaps?
rionmonster commentedon Jul 17, 2026
@gunnarmorling
Sounds good! I'll add this one to my queue.
alamb commentedon Jul 22, 2026
FYI we merged the upstrema PR
gunnarmorling commentedon Jul 22, 2026
Thx for the heads-up, @alamb!
gunnarmorling commentedon Jul 22, 2026
@rionmonster, picked this one up as this became urgent now. Fixed via #811.
#783 Key shredded_variant invalid-skip off the -INVALID suffix
#783 Drop the dead rowComparisonSkipReason hook
hardwood-hq#783 Key shredded_variant invalid-skip off the -INVALID su…
hardwood-hq#783 Drop the dead rowComparisonSkipReason hook