Skip to content

Use RoaringTreemap for MergeRows cardinality tracking #6608

Open

Description

@unikdahal

Problem

Native MergeRowsExec uses HashSet<i64> for cardinality validation, while Spark uses Roaring64Bitmap.

As measured in the #5318 review, for 5M row IDs:

Representation Memory
HashSet<i64> ~75 MB
Roaring, dense ~0.6 MB
Roaring, shuffled ~5–17 MB

Since this state cannot spill, native MERGE can hit ResourcesExhausted where Spark still fits in memory.

Proposed solution

Replace HashSet<i64> with RoaringTreemap while preserving duplicate detection, existing cardinality semantics, and DataFusion memory accounting.

Add coverage for dense/shuffled IDs, cross-batch duplicates, and bounded-memory cases.

Related: #5318, #5122

Activity

  1. unikdahal commented on Oct 4, 2026

    @unikdahal
    ContributorAuthor

    take

  2. unikdahal commented on Oct 12, 2026

    @unikdahal
    ContributorAuthor

    untake

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions