HadoopReadable opens every pooled stream with fs.open, costing a HEAD per open on S3A #98

HadoopReadable.openStream() (java/vortex-spark/common/src/main/java/dev/vortex/spark/io/HadoopReadable.java:49) calls fs.open(path). PooledReadable.readFully opens a new stream whenever none is free, so a file ends up with as many streams as Vortex has reads in flight on it.

On S3A every plain open issues a HEAD for the object's metadata, although HadoopReadable already knows the file's length.

Measured on a filtered, projected read of about 4,000 files through the Spark connector, using S3A's own counters summed across executors:

So HEADs were about 41% of all requests. Switching S3A to the classic input stream with random reads doubled the GETs and left the HEADs unchanged, so they come from the open, not the stream.

Proposal. Open through the openFile() builder with the known length and a random read policy:

fs.openFile(path)
    .opt("fs.option.openfile.length", Long.toString(length()))
    .opt("fs.option.openfile.read.policy", "random")
    .build()
    .get();

S3A skips the HEAD when given the length. Other filesystems ignore options they don't know.