You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
HadoopReadable should pass known file length when opening pooled streams #11365
HadoopReadable.openStream() calls fs.open(path) for each pooled input stream, even though the readable already knows the file length. On S3A, plain opens can issue a metadata HEAD for every stream. Concurrent positional reads open multiple streams per file, multiplying metadata requests.
This path remains on upstream develop at 2e1b05b84. QC issue #98 reports approximately 139,588 opens and 144,004 metadata HEADs during a scan of about 4,000 files; those are QC observations, not an upstream benchmark.
Use Hadoop's openFile() builder with optional fs.option.openfile.length and fs.option.openfile.read.policy=random hints. Preserve original exception types, restore interruption and close streams that arrive after interruption.
The HEAD reduction depends on Hadoop honoring the length hint. Spark 3.5's Hadoop 3.3.4 ignores these hints, so it keeps its existing behavior; the QC implementation was tested for compatibility on Spark 3.5 and 4.0. Live post-fix S3A request counts have not been measured.
QC PR #106 contains the fix and focused builder/exception/interruption tests. Its unrelated list-view writer normalization is excluded from the upstream port.
Activity
perf(spark): avoid redundant metadata HEADs when opening Hadoop strea…