Skip to content

parquet-testing skip-lists key on fixture filenames that upstream is renaming #783

Closed

Description

@sdf-jkl

Problem

parquet-testing-runner tracks apache/parquet-testing master at HEAD. ParquetTestingRepoCloner.ensureCloned() shallow-clones with no ref, and target/ is wiped by mvn clean, so CI re-clones on every run.

Enumeration is by directory walk. Utils.parquetTestFiles() walks shredded_variant/ and takes every *.parquet, and VariantShredReassemblerTest uses a case-*.parquet glob. The skip-lists, though, key on exact filenames:

  • Utils.SKIPPED_FILES lists "case-041.parquet", "case-131.parquet", "case-138.parquet"
  • VariantShredReassemblerTest.UNREADABLE lists the same three

So when a fixture is renamed upstream the walk still finds it, but the skip-lists stop matching it.

How it surfaced

apache/parquet-testing#117 relabels four shredded_variant fixtures whose variant group omits the spec-required value column, following the convention already used by cases 43, 84 and 125:

old new
case-041.parquet case-041-INVALID.parquet
case-131.parquet case-131-INVALID.parquet
case-132.parquet case-132-INVALID.parquet
case-138.parquet case-138-INVALID.parquet

The goldens are renamed to match and cases.json is updated in lockstep with a notes entry. No file contents change.

Once that merges, cases 41, 131 and 138 stop being skipped and start being exercised. Case 132 is in neither skip-list today, so it is unaffected beyond the new name.

I have not run the suite against the renamed corpus, so I will not claim whether those three then pass or fail. Either way the skip-lists no longer do what they say.

Proposed fix

  • Update the three names, or move them into INVALID_FIXTURES, which is where they now belong.
  • Optionally make it rename-proof: take fixture filenames from cases.json rather than walking the directory, and key the invalid-case skip off the -INVALID substring and the notes field. arrow-go does the substring skip, and arrow-rs drives everything from cases.json, so it needs no change for this PR.

Acceptance criteria

  • After bumping to a parquet-testing revision containing the rename, cases 41, 131 and 138 are handled deliberately rather than by a stale filename match.
  • Cases 43, 84, 125 and 132 keep their current behaviour.

Related

Activity

  1. gunnarmorling commented on Jul 13, 2026

    @gunnarmorling
    Collaborator

    Thanks for logging, good catch. I wasn't aware of cases.json actually, thanks for that pointer! We'll look into this asap.

  2. added this to the 1.0.1.Final milestone on Jul 13, 2026
  3. gunnarmorling commented on Jul 15, 2026

    @gunnarmorling
    Collaborator

    @rionmonster, this one might be interesting to you, perhaps?

  4. self-assigned this
    on Jul 17, 2026
  5. rionmonster commented on Jul 17, 2026

    @rionmonster
    Contributor

    @gunnarmorling

    Sounds good! I'll add this one to my queue.

  6. alamb commented on Jul 22, 2026

    @alamb
  7. gunnarmorling commented on Jul 22, 2026

    @gunnarmorling
    Collaborator

    Thx for the heads-up, @alamb!

  8. gunnarmorling commented on Jul 22, 2026

    @gunnarmorling
    Collaborator

    @rionmonster, picked this one up as this became urgent now. Fixed via #811.

  9. added a commit that references this issue on Jul 22, 2026
  10. added a commit that references this issue on Jul 22, 2026
  11. added 2 commits that reference this issue on Aug 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions