Skip to content

Avoid metadata HEADs when opening pooled Hadoop streams - #106

Merged
ChristopherWirt merged 3 commits into
developfrom
cwirt/fix-issue-98
Nov 9, 2026
Merged

ChristopherWirt merged 3 commits into
developfrom
cwirt/fix-issue-98

Conversation

@ChristopherWirt

@ChristopherWirt ChristopherWirt commented Nov 9, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

HadoopReadable opens every pooled stream with fs.open(). On S3A each open sends a metadata HEAD, even though scan planning already supplied the file length.

This opens streams through fs.openFile() with the known length, so S3A versions that honor the hint skip the HEAD. Upstream port: vortex-data#11368.

Changes

  • Use fs.openFile() with two optional hints:
    • fs.option.openfile.length: the known length.
    • fs.option.openfile.read.policy=random: matches Vortex's positional range reads.
  • The hints are string keys because Hadoop 3.3.4 lacks the OpenFileOptions constants. They are optional, so other filesystems ignore them.
  • Waiting on the open's future preserves the original exception types, via FutureIO.raiseInnerCause.
  • An interrupted open restores the interrupt flag and closes any stream that arrives later.

Tests

  • HadoopReadableTest covers:
    • supplied and discovered lengths
    • every pooled stream receiving the hints
    • the hints staying optional
    • exceptions passing through unchanged
    • late-stream cleanup after an interrupt

Notes for reviewers

  • The improvement depends on Hadoop support. Spark 3.5's Hadoop 3.3.4 ignores these hints and still sends the HEAD; it only skips it for an S3AFileStatus, which executors don't have. Spark 4.0+ (Hadoop 3.4.x) skips it.
  • On those versions, the random-read hint takes precedence over a cluster-wide fs.s3a.experimental.input.fadvise setting.

Validation

  • HadoopReadableTest (10 tests) passes on Spark 3.5 / Scala 2.12, 3.5 / 2.13 and 4.0 / 2.13.
  • The full Spark 3.5 test suites (53 classes) pass on both Scala versions.
  • Spotless is clean for the changed files.
  • Tests used a prebuilt JNI library. There are no Rust changes.
  • Live S3A request counts were not measured.

AI assistance

AI tools assisted with this change and its description. Christopher Wirt is the responsible contributor.

🤖 Generated with Claude Code

ChristopherWirt and others added 3 commits November 9, 2026 16:00
…t to Hadoop 3.3.5+

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Signed-off-by: Christopher Wirt <[email protected]>
Signed-off-by: Christopher Wirt <[email protected]>
(cherry picked from commit bfbc5bb)
Signed-off-by: Christopher Wirt <[email protected]>
@ChristopherWirt
ChristopherWirt merged commit de970df into develop Nov 9, 2026
1 check passed
ChristopherWirt added a commit that referenced this pull request Nov 10, 2026
## Summary

Fix the shared Linux musl builder failure on `develop` and PRs #103,
#105, #106, #107, and #108 after the upstream lazy-constructor update.

## Changes

Normalize lists rebuilt from list views through the same `reset_offsets`
path as ordinary lists before forwarding their children to layout
writers. Rebuilding a sliced list view can now leave a lazy elements
slice; a flat child writer cannot serialize that expression. This
preserves the existing referenced-element bounds and resolves the slice
without forcing unrelated encodings into canonical arrays.

Keep the existing nullable sliced-list-view regression and add the
non-nullable case.

## CI Failure Analysis

### Status

Pre-existing base-branch regression, independent of the individual PR
changes. The same failure appears on [develop run
42649358621](https://github.com/quantcast/vortex/actions/runs/42649358621)
and every completed failing PR builder inspected (#105–#108).

### Failed Jobs

- `Rust tests (linux-musl)`: test failure in
`layouts::list::reader::tests::sliced_list_chunks_round_trip::case_4_list_view_chunks`.

### Relevant Log Output

```text
Error: Array vortex.slice is not registered for serialization
```

### Correlation With PR Changes

The upstream merge makes array slicing lazy. The list-view branch of
`canonicalize_to_list_parts` forwarded a rebuilt list's lazy elements
directly, while the ordinary-list branch already normalized them. The
Spark PRs do not change this writer; #107 changes separate zone-pruning
code.

### Recommended Next Step

Merge this shared fix into `develop`. The same patch is being applied to
the affected open PR branches so their new builders include the fix
immediately.

## Validation

- Reproduced the exact failing test on unmodified current `develop`
(exit 100, same serialization error).
- `cargo nextest run -p vortex-layout -E 'test(layouts::list::)'`: all
161 tests passed after the fix, including the original failing case and
new non-nullable case.
- `cargo clippy -p vortex-layout --all-targets --all-features -- -D
warnings`: passed.
- Pinned nightly Rust formatting and `git diff --check`: passed.

Local validation ran on macOS; the fresh Linux musl builders will verify
the runner-specific job.

Signed-off-by: Christopher Wirt <[email protected]>
robert3005 pushed a commit to vortex-data/vortex that referenced this pull request Nov 10, 2026
…ms (#11368)

## Summary

- Tracking Issue: #11365
- Fixes #11365

Open pooled Hadoop streams with the known file length so S3A versions
that honor the hint can avoid redundant metadata HEAD requests. Ports
the Hadoop-readable changes from [QC
#106](quantcast#106) onto upstream
`develop` at `2e1b05b84`.

## Changes

- Use `fs.openFile()` with optional known-length and random-read hints;
other filesystems may ignore them.
- Preserve original asynchronous-open exception types, restore
interruption and close any stream produced after an interrupted open.
- Add focused tests for supplied/discovered lengths, every pooled stream
receiving the options, optional keys, exception identity and late-stream
cleanup.

Signed-off-by: Christopher Wirt <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

HadoopReadable opens every pooled stream with fs.open, costing a HEAD per open on S3A

2 participants