Repository navigation
[EPIC] Reduce binary size #24727
Description
Is your feature request related to a problem or challenge?
DataFusion generates a large amount of machine code when compiled, which has a few negative effects:
- WASM size: compiled WASM is large (10MB+), slowing page load times (See [DISCUSSION] Reduce the DataFusion WASM size #16554)
- Build time: more generated code means more work for LLVM and the linker, so compile times (and thus CI and local iteration) suffer. See Building project takes a *long* time (esp compilation time for
datafusioncore crate) #13814 - Binary size: artifact size (aka binary code) directly affects cold start times, and memory footprint in projects like serverless functions and edge deployments
A significant contributor to code size is unnecessary monomorphization, a fancy name for generic functions that get instantiated once per type, but the copies are mostly the same. Common patterns include:
- Generics that only call trait methods: functions like
fn foo<P: ExecutionPlan + ?Sized>(plan: &P)that never usePexcept through the trait. These compile to N identical copies where a single&dynversion would do. - Per-primitive-type instantiation where only the byte width matters: kernels generic over
ArrowPrimitiveTypethat only move, compare-for-equality, or hash native values behave identically for all same-width types (e.g.Int32,UInt32,Date32,Time32are all 4-byte bit patterns) and can dispatch on width at runtime instead of instantiating 30+ copies. - Large default trait method bodies: default methods on widely-implemented traits (e.g.
ScalarUDFImpl, with 100+ implementations indatafusion-functionsalone) are monomorphized per implementing type, duplicating the default body each time.
An example of pattern 1 is check_default_invariants, which is compiled once for each of the 38 ExecutionPlan implementations (~44,000 lines of LLVM IR) even though every copy is identical:
pub fn check_default_invariants<P: ExecutionPlan + ?Sized>(
plan: &P,
check: InvariantLevel,
) -> Result<(), DataFusionError> {
let children_len = plan.children().len();
check_len!(plan, maintains_input_order, children_len);
check_len!(plan, required_input_ordering, children_len);
check_len!(plan, benefits_from_input_partitioning, children_len);
...
}An example of pattern 2 is hash_array_primitive, which is instantiated for all 32 primitive types in every crate that calls create_hashes (~12,000 lines of LLVM IR per crate). It only reads native values as bit patterns and hashes them, so the copies for Int32, UInt32, Date32, and Time32 are byte-identical machine code:
// condensed; the real function also handles nulls and rehashing
fn hash_array_primitive<T>(
array: &PrimitiveArray<T>,
random_state: &impl HashState,
hashes_buffer: &mut [u64],
rehash: bool,
) where
T: ArrowPrimitiveType<Native: HashValue>,
{
for (hash, &value) in hashes_buffer.iter_mut().zip(array.values().iter()) {
*hash = value.hash_one(random_state);
}
...
}Early hints / inspiration
@geoffreyclaude and I have been working to keep the size of the IN LIST implementations down as part of #19241 (he came up with using just bytewidth for FixedSizeBinary in #24102) and here is an example that reduces code size: #24687 (removed 67% of the module's generated code with no measurable runtime cost)
and I think we have figured out a general recipe to find and remove this in the codebase:
- Find candidates with
cargo llvm-lines --release -p <crate> --lib(rank generic functions by total IR lines × instantiation count) - Rewrite the candidate to dispatch at runtime (e.g. per batch, not per row), keeping monomorphized inner loops only where the generated code actually differs
- Verify with
cargo llvm-linesbefore/after and the relevant criterion benchmarks (a size win must not cost measurable performance)
Describe the solution you'd like
A collection of tickets applying the recipe above to the largest offenders.
How to measure improvements
Every PR / issue in this epic should report before/after numbers using both of the measurements below, taken once at the merge-base with main and once on the branch (see example on "Code size" heading of #24687).
1. Generated code: cargo llvm-lines
Counts pre-optimization LLVM IR lines per monomorphized function. Deterministic, and attributes code to specific functions, so it shows exactly which instantiations were removed:
cargo install cargo-llvm-lines
# Run on main and on your branch:
cargo llvm-lines --release -p <crate> --lib > /tmp/llvm-lines-{main,branch}.txt
# Crate-wide totals (lines and copies):
head -3 /tmp/llvm-lines-main.txt
# Total for the module you changed:
awk '/<module_name>/ {gsub(/[(),%]/,"",$1); sum+=$1; n++}
END {print sum, "IR lines across", n, "functions"}' /tmp/llvm-lines-main.txtReport: module IR lines, module instantiation count, and crate totals, before and after.
2. Final binary size: release build
Measures what users actually ship.
# Gold standard: datafusion-cli (slow: fat LTO takes ~30+ minutes)
cargo build --release --bin datafusion-cli
ls -l target/release/datafusion-cli
# Faster per-crate proxy: any bench binary that links the changed crate
cargo bench --no-run -p <crate> --bench <bench_target>
ls -l target/release/deps/<bench_target>-*Build at the merge-base and on the branch, compare byte sizes.
Note that typically the binary delta will be much smaller than the IR delta as LLVM and the linker deduplicate and dead-code-eliminate aggressively.
Ideas
Here are some ideas (from an AI tool)
Candidate list (click to expand)
Initial candidates identified by an llvm-lines survey of seven crates (rough IR line totals; final binary impact is smaller but proportional — the PoC measured ~3.3 bytes of binary per IR line):
-
udaf_default_*display/schema-name helpers indatafusion/expr/src/udaf.rs: generic over<F: AggregateUDFImpl + ?Sized>but only call trait methods; ~145K IR lines across 29 UDAF impls (pattern 1) -
check_default_invariants/check_dynamic_expression_invariantsindatafusion-physical-plan: ~69K IR lines across 38ExecutionPlanimpls (pattern 1) -
INlist primitive filters (BranchlessFilter<T>,PrimitiveHashSetFilter<T>,BitmapFilter<T>): ~130K IR lines across 27 instantiations; equality-only → width dispatch (pattern 2) -
hash_array_primitiveindatafusion-common: ~12K IR lines × 32 instantiations per consuming crate, ~70K workspace-wide (pattern 2) -
first_lastprimitive accumulators indatafusion-functions-aggregate: ~55K IR lines across 25 instantiations (pattern 2) - GROUP BY primitive group values (
PrimitiveGroupValueBuilder<T, NULLABLE>,GroupValuesPrimitive<T>): ~75K IR lines (pattern 2; theconst NULLABLE: boolparameter doubles every instantiation) - TopK aggregate
PrimitiveHashTable<T>: ~22K IR lines across 32 instantiations (pattern 2) -
functions-nestedper-primitive kernels (array_sort,array_min/array_max,array_has): ~120K IR lines; theOffsetSizeTraitdimension doubles many of these (pattern 2) - CASE literal lookup table
PrimitiveIndexMap<T>: ~25K IR lines across 32 instantiations (pattern 2) - Sort-merge primitive cursor streams (
SortPreservingMergeStreamspecializations): ~27K IR lines (pattern 2) -
ScalarUDFImpldefault method bodies (return_field_from_args,schema_name,output_ordering): ~75K IR lines across 119 impls indatafusion-functions, repeated in every crate defining UDFs (pattern 3) - IN LIST: handle dictionary-encoded needles once (rather than in each filter type) to reduce code size #24658
Important caveats
- Avoid making the code more complicated. The goal is less generated code, not cleverer source code. Most of these changes will probably reduce code.
- Float semantics:
f16/f32/f64have distinct equality/hash semantics (NaN,±0.0), so width-based merging applies to integer-semantics types only; floats likely have to keep their own instantiations - Every change must be benchmarked. The rule from the PoC: a size reduction is acceptable only if the relevant criterion benchmarks show no regression beyond noise.
Describe alternatives you've considered
Do nothing
Additional context
- [DISCUSSION] Reduce the DataFusion WASM size #16554
- [EPIC] A collection of tickets for improved WASM support in DataFusion #13815
- Building project takes a *long* time (esp compilation time for
datafusioncore crate) #13814 - IN LIST: reuse primitive filters for FixedSizeBinary #24102
- IN LIST: use runtime dispatch for
FixedSizeBinaryfilter (reduce binary size) #24687
Activity
alamb commentedon Aug 27, 2026
Filed the upstream arrow-rs ticket for the largest non-DataFusion item found in the survey (the
PrimitiveArrayDebugimpl, ~85K IR lines per consuming crate): apache/arrow-rs#10889alamb commentedon Aug 27, 2026
I made one PR that cuts down the binary size (0.25% -- small but real)
udaf_default_*display/schema name helpers non-generic (214 KB smaller binary) #24728crepererum commentedon Aug 28, 2026
I said this before somewhere in this issue tracker, but I think our liberal use of
implin function signatures also contributes to this.Make `udaf_default_*` display/schema name helpers non-generic (214 KB…
IN LIST: use runtime dispatch for `FixedSizeBinary` filter (reduce bi…
array_sort / array_has: monomorphize per native type instead of per A…
comphead commentedon Sep 7, 2026
Considering AI nature of created PRs it prob makes sense to have a CI rule to check numbers on each PR overall in this repo to make PRs more effective 🤔
alamb commentedon Sep 8, 2026
I added this to the issue description:
Are you suggesting I add it to somewhere else?
comphead commentedon Sep 8, 2026
I was actually thinking if it makes sense to have a CI check for any PRs to measure numbers before the merge, and set the binary size metric as the recommendation/requirement. Given that most of PRs are AI generated it might bring some benefit on early capture? WDYT?
alamb commentedon Sep 8, 2026
I think it is a good idea in theory -- however, I worry about the overall amount of additional CI time. For me it takes 15 minutes to build a release build of
datafusion-cli.one thing we could probably do pretty easily is make a script that measured the size for each merge to main, and then posted a report on the PR after the fact (rather than part of CI) 🤔 I bet that woud take like 10 minutes with an agent (and then a few days to run the actual builds 🤔 )
Reduce binary size of `ScalarUDFImpl` default impls (apache#24966)
liningpan commentedon Sep 15, 2026
For some reason,
sqlparseris significantly bloating the datafusion-cli binary and my own project.cargo bloatsuggests that 6.4MiB of the datafusion-cli binary (release build) is attributed to sqlparser (for my own project, it bloats up to 20 MiB).Top functions from sqlparser look like this. I wonder if these are inlined copies. The
Statementstruct is 3.6k largealamb commentedon Sep 15, 2026
Sounds like a good thing to explore. Maybe worth a ticket upstream to sqlparser-rs 🤔
Reduce binary size of `ScalarUDFImpl` default impls (apache#24966)
srv1n commentedon Nov 4, 2026
A data point on sqlparser size. datafusion_expr::Expr embeds AST nodes (Expr::Wildcard → WildcardOptions → PlannedReplaceSelectItem → ReplaceSelectElement → sqlparser::ast::Expr), so its derives reach into the AST.
To measure that, I built one small program (parse a statement, then clone, compare, hash and debug-print an Expr) with datafusion-expr 54.1 with and without its sql feature, keeping sqlparser 0.62 linked in both. Fat LTO, one codegen unit, macOS arm64.
__text went from 3.50 MB to 4.74 MB, and symbols naming sqlparser from 1.7 MB to 3.0 MB. sqlparser's Hash code went from none to 0.2 MB. So in this build the Expr link costs about 1.2 MB, which is real but modest, and removing Expr::Wildcard (per the TODO in expr.rs) should recover some of it. It doesn't explain a 20 MiB figure, though.