Skip to content

GH-55503: [C++] Fix Float16 memo table keys built from raw bits in dictionary unification and count_distinct - #55611

Open
AnuragRaut08 wants to merge 1 commit into
apache:mainfrom
AnuragRaut08:fix/float16-dictionary-count-distinct
Open

AnuragRaut08 wants to merge 1 commit into
apache:mainfrom
AnuragRaut08:fix/float16-dictionary-count-distinct

Conversation

@AnuragRaut08

@AnuragRaut08 AnuragRaut08 commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

Rationale for this change

Float16 dictionary unification and count_distinct can produce incorrect results because raw uint16_t storage values are interpreted as numeric values instead of Float16 bit patterns.

For example, the Float16 bit pattern 0x3C00, representing 1.0, can be interpreted as 15360.0 when passed through the converting constructor.

This causes dictionary values to be corrupted during dictionary unification and distinct Float16 values to be incorrectly merged by count_distinct.

What changes are included in this PR?

  • Use Float16::FromBits() when constructing memo table keys from raw Float16 values in DictionaryUnifierImpl::Unify.
  • Fix Float16 handling in the array path of CountDistinctImpl::Consume.
  • Fix Float16 scalar unboxing in UnboxScalar<HalfFloatType>::Unbox to preserve the original bit representation.

Are these changes tested?

Added regression tests:

  • TestDictionaryUnifier.HalfFloat: verifies unified dictionary values and transpose maps.
  • TestCountDistinctKernel.HalfFloat: covers distinct Float16 values, duplicates, nulls, chunked arrays, and scalar input.

Both regression tests were verified against the original implementation, where they reproduced the incorrect behavior.

Validation performed:

  • arrow-array-test
  • arrow-compute-aggregate-test
  • clang-format --dry-run --Werror
  • git diff --check

Are there any user-facing changes?

Fixes incorrect Float16 values returned by dictionary unification and incorrect distinct counts for Float16 arrays.

This PR includes breaking changes to public APIs.

No.

This PR contains a "Critical Fix".

Yes. This fixes incorrect data produced by Float16 dictionary unification and count_distinct.

Was AI used for this PR?

PR code and description written by:

  • Human
  • AI

Reviewed before submission by:

  • Human
  • AI
  • Not reviewed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant