Slow local file reads when using GetResult.into_stream

Describe the bug
See apache/datafusion#21450
Root cause: there's a spawn_blocking call per each 8KiBs read from the file, adding significant context switch overhead

To Reproduce
See apache/datafusion#21446
For the tests I've used a c7a.16xlarge ec2 instance, with a trimmed down version of hits.json to 51G (original has 217 GiB), with a warm cache (by running cat hits_50.json > /dev/null)

Expected behavior

A more efficient implementation (e.g. tokio uses a buffer size of 2MiBs when reading files)

Additional context
apache/datafusion#21478 (comment)

I think the trade-off here depends on which is more precious, memory or CPU/latency. 2MB might be peanuts in an analytics context, but significant in contexts with higher concurrency.

Perhaps we could make this configurable, e.g. as a parameter?

Even without increasing the buffer size, we could reduce the context switch overhead, e.g. by using a single spawn_blocking thread and an mpsc channel, as I mentioned in apache/datafusion#21478 (comment) when considering alternative implementations.

Does that not just move the problem into how big to make the mpsc buffer?

Not really, cause you eliminate the overhead of creating one spawn_blocking task per chunk.

using an mpsc channel with one slot and sending 8KiB at a time, I get this result:

┏━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━┓
┃ Query    ┃ json-test-on-main ┃       HEAD ┃       Change ┃
┡━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━┩
│ QQuery 0 │        2438.00 ms │ 3268.99 ms │ 1.34x slower │
└──────────┴───────────────────┴────────────┴──────────────┘

whereas with the existing into_stream implementation I get:

┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query     ┃ json-test-on-main ┃ test-json-improvement ┃        Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0  │        2938.54 ms │           36468.92 ms │ 12.41x slower │

In which case SGTM 😄

The only problem is that a long-living spawn_blocking thread runs into this sort of thing: apache/datafusion#15654

Navigation Menu

Repository navigation

Slow local file reads when using GetResult.into_stream #693

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Slow local file reads when using GetResult.into_stream #693

Description

Activity

tustvold commented on Apr 14, 2026

ariel-miculas commented on Apr 14, 2026

tustvold commented on Apr 14, 2026

ariel-miculas commented on Apr 14, 2026

ariel-miculas commented on Apr 14, 2026

tustvold commented on Apr 14, 2026

ariel-miculas commented on Apr 14, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions