plugin cache: SpliceAI chr1/chr2 built from the full source exhaust the 8 GiB tier pool in ExternalSorterMerge

Observation

Building the SpliceAI plugin cache (vepyr-plugins v0.1.1, full 28.5 GB spliceai_scores.masked.snv.ensembl_mane.grch38.110.vcf.gz with its .tbi, strict source verification, variation cache 116_GRCh38_merged) through vepyr build_plugin_cache(chroms=[1..22, X, Y, MT], overwrite=True) at engine 6476619 on a 64 GB macOS host fails on the largest contigs with the default pool:

Resources exhausted: Failed to allocate additional 760.1 KB for ExternalSorterMerge[8]
with 151.6 MB already allocated for this reservation - 601.1 KB remain available for the total pool

With VEPYR_PLUGIN_RETRY_MEMORY_MIB=24576 chr1 (300,634,726 rows after dedup, source_partitions=16) gets through, but chr2 (282,792,329 rows) fails the same way after the fallback:

plugin_cache[spliceai/2]: read+normalize+dedup done, 282792329 rows, source_partitions=16, 199.9s elapsed
plugin_cache[spliceai/2]: final HashJoin plan exhausted the pool (282792329 plugin rows this chrom) -- retrying with DataFusion SortMergeJoin
Resources exhausted: Failed to allocate additional 1024.1 KB for ExternalSorterMerge[13]
with 0.0 B already allocated for this reservation - 981.4 KB remain available for the total pool

VEPYR_PLUGIN_RETRY_MEMORY_MIB=40960 together with VEPYR_PLUGIN_TARGET_PARTITIONS=8 builds chr2 in 497 s:

adaptive join stage=probe target_partitions=8 algorithm=SortMerge reason=MissingStatistics build_rows=Some(154083555)
adaptive join stage=tier  target_partitions=8 algorithm=Hash reason=BuildFits build_rows=Some(27285704) build_data_bytes=Some(328481044) hash_build_bytes=Some(902317941) available_bytes=Some(42949672960)
plugin_cache[spliceai/2]: final HashJoin plan exhausted the pool (282792329 plugin rows this chrom) -- retrying with DataFusion SortMergeJoin
BUILT spliceai chr2 rows/warm/cold=[('chr2', 282792329, 642961, 282149368)] in 497s

chr22 (52.7M rows) never needed any override, and ClinVar, AlphaMissense and dbNSFP (25 contigs each) build fine with the default.

Why it matters

The 8 GiB default is documented as sized so that "the join alone" clears it (DEFAULT_RETRY_SORT_MEMORY_MIB in build.rs), and the hash join does fall back to a sort-merge join when it does not, yet the sort-merge path then exhausts the same pool in ExternalSorterMerge with a zero-byte reservation and ~1 MB left, i.e. the pool is consumed by other reservations of the same plan when the merge starts. A user following the docs (build_plugin_cache on a 64 GB machine) hits a hard failure on the very first large contig, with no hint that two env vars would fix it. The sources provenance work (#228) only pays off if the full source file is the build input, so slicing per contig first (what run_all_plugin_comparisons.sh did for the Aug-27 cache) is not a way out any more.

Things worth checking

Whether the wider total ORDER BY from fix(vep): make plugin shard row order reproducible #230 (tier, start, allele_string, match cols, values) is what pushed the per-partition sorter over the budget; the previously published cache was built per contig on an engine before fix(vep): make plugin shard row order reproducible #230 with the default pool.
Whether ExternalSorterMerge with 16 target partitions reserves per-partition merge buffers that together exceed what FairSpillPool leaves after the SortMergeJoin's own reservations; halving the partitions was part of the working configuration.
Whether the retry path should shrink target_partitions and/or raise the pool ceiling itself when the fallback plan also fails, instead of surfacing the raw DataFusion error.
The error message could name VEPYR_PLUGIN_RETRY_MEMORY_MIB / VEPYR_PLUGIN_TARGET_PARTITIONS.

Repro

import os
os.environ["RUST_LOG"] = "info"
vepyr.build_plugin_cache("spliceai", "v0.1.1",
    source_path=".../spliceai_scores.masked.snv.ensembl_mane.grch38.110.vcf.gz",
    cache_dir=".../116_GRCh38_merged", plugin_cache_root=OUT, chroms=["2"],
    plugins_repo=".../vepyr-plugins", verify_source="strict")

Fails with the default pool and with VEPYR_PLUGIN_RETRY_MEMORY_MIB=24576; passes with VEPYR_PLUGIN_RETRY_MEMORY_MIB=40960 VEPYR_PLUGIN_TARGET_PARTITIONS=8.

🤖 Generated with Claude Code

https://claude.ai/code/session_015j437TcBbxAqx4XF8tshP3