Skip to content

HadoopReadable should pass known file length when opening pooled streams #11365

Closed

Description

@ChristopherWirt

HadoopReadable.openStream() calls fs.open(path) for each pooled input stream, even though the readable already knows the file length. On S3A, plain opens can issue a metadata HEAD for every stream. Concurrent positional reads open multiple streams per file, multiplying metadata requests.

This path remains on upstream develop at 2e1b05b84. QC issue #98 reports approximately 139,588 opens and 144,004 metadata HEADs during a scan of about 4,000 files; those are QC observations, not an upstream benchmark.

Use Hadoop's openFile() builder with optional fs.option.openfile.length and fs.option.openfile.read.policy=random hints. Preserve original exception types, restore interruption and close streams that arrive after interruption.

The HEAD reduction depends on Hadoop honoring the length hint. Spark 3.5's Hadoop 3.3.4 ignores these hints, so it keeps its existing behavior; the QC implementation was tested for compatibility on Spark 3.5 and 4.0. Live post-fix S3A request counts have not been measured.

QC PR #106 contains the fix and focused builder/exception/interruption tests. Its unrelated list-view writer normalization is excluded from the upstream port.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions