Skip to content

fix(variant): reject duplicate object keys in strict casts - #12135

Open
Subodh-17 wants to merge 1 commit into
apache:mainfrom
Subodh-17:fix/variant-strict-duplicate-keys
Open

Subodh-17 wants to merge 1 commit into
apache:mainfrom
Subodh-17:fix/variant-strict-duplicate-keys

Conversation

@Subodh-17

Copy link
Copy Markdown

Which issue does this PR close?

Closes #11948.

Rationale for this change

In cast_to_variant, converting Arrow StructArray or MapArray to Parquet Variant in strict mode (options.safe == false) silently overwrote duplicate object keys instead of returning an error. Duplicate keys with null values could also bypass underlying builder checks because null fields are skipped during builder insertion.

In strict mode, duplicate keys within the same object should be rejected with an ArrowError::InvalidArgumentError. Safe mode retains its existing permissive overwrite behavior.

What changes are included in this PR?

  1. Struct conversion

    • Checks struct_array.column_names() for duplicate names in strict mode.
    • Returns an error when a non-null Struct row contains duplicate field names.
    • Null Struct rows continue to produce null variants.
  2. Map conversion

    • Uses a per-row HashSet to detect duplicate keys in strict mode.
    • Detects duplicates even when the later value is null.
    • Keeps keys in separate rows and nested sibling objects independent.
  3. Regression tests

    • Adds 10 targeted tests covering Struct and Map duplicates, null-valued duplicates, differing value types, nested objects, row independence, and safe-mode behavior.

Are these changes tested?

Previously reported verification:

  • cargo test -p parquet-variant-compute: 381 unit tests and 15 doctests passed.
  • cargo test -p parquet-variant --lib: 178 unit tests passed.
  • Strict targeted tests: 9 passed.
  • Safe-mode targeted test: 1 passed.
  • Clippy with -D warnings passed.
  • cargo fmt --check passed.
  • git diff --check passed.

These are results from the implementation and audit stages, not newly executed tests in this PR-creation stage.

Are there any user-facing changes?

Strict cast_to_variant conversions (CastOptions { safe: false, .. }) containing duplicate Struct or Map keys now return ArrowError::InvalidArgumentError instead of silently overwriting earlier values.

Safe conversions (safe: true) retain the existing permissive behavior.


AI Tooling Disclosure: AI tools were used during initial problem reproduction, test scaffolding, and implementation drafting. The contributor reviewed and refined the implementation, error-handling paths, and test assertions, and is responsible for understanding, debugging, and maintaining the submitted changes.

@boring-cyborg boring-cyborg Bot added the parquet-variant parquet-variant* crates label Nov 9, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

parquet-variant parquet-variant* crates

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Variant] Strict cast_to_variant silently overwrites duplicate object keys

1 participant