You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
datafusion-cli fails to run ClickBench queries with 8GB of RAM #18473
The instance type c6a.xlarge only has 8GB RAM (the clickbench dataset is 15GB) and there are a number of issues with it, including
...
OOM errors happens during test execution, with results reported as null for several tests
To Reproduce
Use docker to build datafusion:
cd .devcontainer
docker build -t datafusion-build .cd ..
docker run -m 4G -v `pwd`:/datafusion -it datafusion-build /bin/bash
Now, in the docker container
# build datafusion-clicd /datafusion
cargo install --profile=release-nonlto --path datafusion-cli
# Get benchmark datacd /datafusion/benchmarks
./bench.sh data clickbench_partitioned
cd /datafusion/benchmarks/data
# make symlink to hits so queries can run without modification
ln -s hits_partitioned hits
# run the queriesforqin`ls ../queries/clickbench/queries/*.sql`;do datafusion-cli -f $q;done
Activity
alamb commentedon Nov 3, 2025
One potential solution is to configure the datafusion CLI memory limit via the
-moptionHowever this doesn't seem to work
We could also potentially consider changing the default memory limit now that DataFusion supports better out of core behavior
goutamadwant commentedon Aug 22, 2026
take