Skip to content

Spark V2 scans report no input bytes to Spark's metrics #99

Closed

Description

@ChristopherWirt

Spark V2 Vortex scans leave a stage's inputBytes / bytesRead at 0, so the Spark UI, event logs and cost tooling can't see how much a scan read.

The likely cause: reads run on Vortex's native worker threads through PooledReadable. Spark fills a task's input metrics from Hadoop FileSystem statistics on the task's own thread, so it never sees reads made on other threads.

The connector's tests measure bytes read through a test-only hook (VortexNestedSchemaPruningIoTest.BYTES_READ); nothing reports them to Spark at run time.

Proposal.

  • Count bytes read per partition reader inside PooledReadable (or VortexIo).
  • Report them through PartitionReader.currentMetricsValues() as custom metrics, and also add them to the task's inputMetrics when a TaskContext is present, so bytesRead reflects the scan.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions