Skip to content

Java and Spark readers retain native file resources until Cleaner processing #11364

Closed

Description

@ChristopherWirt

Closing or exhausting a Spark Vortex reader releases its Arrow stream and Hadoop readable, but the Java DataSource and Scan handles release native ownership only when their Cleaners run. Completed readers can therefore retain per-file native caches until Java GC even when no Arrow batches remain open.

The Cleaner-only handles and reader cleanup are still present on upstream develop at 2e1b05b84.

In local QC experiments, 30 complete one-leaf reads of a wide nested fixture retained 225.3 MiB of native input buffers after closing every reader; explicitly releasing the data source returned that accounting to zero. This establishes delayed reclamation, not a permanent leak or a production OOM claim. Details: QC issue #96.

Provide idempotent explicit close for source and scan handles and use it at EOF, early close, empty splits, footer reads and failures. Preserve returned partitions/batches and Cleaner fallback. Also release prefetched-but-unreturned partitions and make Arrow stream handoff exception-safe.

QC PR #103 implements this. Its unrelated list-view writer normalization is excluded from the upstream port. Session lifetime (#8075) and decode/output buffering (#11127, #11198) are separate concerns.

Activity

  1. added a commit that references this issue on Nov 11, 2026
    7c1972b
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions