You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Spark V2 scans report no input bytes to Spark's metrics #99
Spark V2 Vortex scans leave a stage's inputBytes / bytesRead at 0, so the Spark UI, event logs and cost tooling can't see how much a scan read.
The likely cause: reads run on Vortex's native worker threads through PooledReadable. Spark fills a task's input metrics from Hadoop FileSystem statistics on the task's own thread, so it never sees reads made on other threads.
The connector's tests measure bytes read through a test-only hook (VortexNestedSchemaPruningIoTest.BYTES_READ); nothing reports them to Spark at run time.
Proposal.
Count bytes read per partition reader inside PooledReadable (or VortexIo).
Report them through PartitionReader.currentMetricsValues() as custom metrics, and also add them to the task's inputMetrics when a TaskContext is present, so bytesRead reflects the scan.
Activity