chore(deps): bump datafusion from 53.0.0 to 55.2.0 in the datafusion group across 1 directory - #8

Fix #3: list_avg handles Int32 natively (sum as i64), avoids full
  f64 cast of flat value arrays for GQ/DP averaging.

Fix #4: skip arrow::compute::filter (8 calls) when all variants
  pass site filtering — use original columns directly.

Fix #5: writer buffers entire variant line into Vec<u8> before a
  single write_all call. Eliminates ~10 tiny write syscalls per
  sample (6M → 300 writes/batch for 2000 samples). Uses itoa for
  zero-alloc integer formatting.

Fix #6: pre-computed lookup table for 10^(-PL/10) replaces 3×
  powf() per sample in dosage computation (~60ns → ~3ns each).

Fix #8: FlatNumeric enum keeps GQ/DP as native Int32, compares
  with f64 thresholds directly — no cast to Float64 then back.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

)

Adds 6 unit-ports to src/partitioned_cache.rs::tests (mirroring the
existing RegFeat unit-ports for the transcript context_type):

- #5  transcript_serializer_label_is_parquet_for_v115_layout
- #6  transcript_context_path_uses_parquet_suffix
- #7  transcript_available_chroms_returns_chrom_list_from_layout
- #8  transcript_context_path_for_chr21_returns_per_chrom_parquet_path
- #9  transcript_context_path_takes_only_chrom_arg
- #10 transcript_context_path_with_empty_chrom_returns_none

Per detailed_plan rows #5-#10. Same v2 paradigm Rule 6 ("parquet IS
vepyr's serializer-equivalent for the same role") applied to the
transcript context_type.

…332)

* chore: bump DataFusion to 55.2.0 and bio-formats to its DF 55 branch

Move the workspace from DataFusion 53.0.0 to 55.2.0 (arrow/parquet 59.3.2,
rust-toolchain and CI on 1.94.0, DataFusion 55.2's MSRV) and pin every
datafusion-bio-format-* dependency to biodatageeks/datafusion-bio-formats#248
(d1eb0c3), which carries the same bump rebased on formats v1.15.0, so it
includes everything the previous e0379fb pin did. pileup and vep share that
one rev so the graph holds one copy of datafusion-bio-format-core and
DataFusion.

Mechanical fallout of the 54/55 API changes:
- `as_any` removed from DataFusion traits: dropped from every impl, the
  `.as_any()` hop removed at plan/expr downcast sites (including the
  multi-line chains in the interval planner), unused `Any` imports
  removed. fastqc's own `QcModule::as_any` is unrelated and kept.
- `ExecutionPlan::apply_expressions` required: leaf implementations for
  the nodes without expressions; `IntervalJoinExec` visits its join keys
  and filter like DataFusion's own `HashJoinExec`.
- `QueryPlanner` / `PhysicalPlanner` take `&dyn Session`, and
  `create_physical_expr` gains the `PhysicalPlanningContext` argument.
- `create_hashes` takes DataFusion's `HashState`; the interval join uses
  `datafusion::common::hash_utils::RandomState` (ahash stays for the
  plain hash maps).
- `Distribution::HashPartitioned` -> `KeyPartitioned`.
- `MemoryPool` now requires `Display` and `name()`: TracingPool and the
  test-only RejectHashPool.
- `MutableArrayData::extend` is deprecated in arrow 59.3: the SIFT page
  gather uses `try_extend` and propagates the overflow error.
- `partition_statistics` returns `Arc<Statistics>` and is deprecated for
  callers; the interval join estimator and the plugin-cache join
  selector read statistics through `StatisticsContext`.

That last item is the one behavioural change. Under DataFusion 55 the
deprecated `partition_statistics` returns unknown statistics for
DataFusion's own nodes, so `choose_for_hash_plan` would have reported
MissingStatistics and silently forced every plugin-cache tier join onto
SortMergeJoin; the adaptive-join tests cover it.

Why: DataFusion 53's external sorter frees its merge headroom and then
re-acquires it from an empty reservation, which starves under
`FairSpillPool` when several partitions sort concurrently (#235).
apache/datafusion#20642 fixes that and shipped in 54.0.0.

Not touched: `research/lance_encoding_sandbox` is outside the workspace
and still pins arrow 58 / datafusion-common 53.

Supersedes #236 and dependabot #8.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* refactor: address review nits on the DataFusion 55.2 bump

- interval join: say why `hash_utils::RandomState::default()` is safe
  (foldhash `FixedState` with a compile-time constant seed, shared by
  both join sides), and fold `partition_statistics_inner` back into
  `partition_statistics` where it lived before the bump, with a note on
  why the partition argument is ignored. Drop the stray blank line left
  by the `Any` import removal.
- plugin-cache join selector: import `StatisticsContext` /
  `StatisticsArgs` instead of spelling the full paths inline.
- the ten leaf `apply_expressions` impls (fastqc, pileup, ranges, vep):
  use imported `Arc` / `PhysicalExpr` / `Result` / `TreeNodeRecursion`
  instead of fully qualified paths.

No behaviour change.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* chore: re-pin datafusion-bio-formats to 99d7f81

Follow biodatageeks/datafusion-bio-formats#248 to its review-nit commit
(test renames, shorter `apply_expressions` signatures, CHANGELOG arrow
version). No API change; vep and pileup stay on one formats rev.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* chore: pin datafusion-bio-formats to the v1.16.0 release

Move every datafusion-bio-format-* dependency (vep and pileup) from the
pre-release rev to tag v1.16.0, which contains the DataFusion 55.2 bump
(biodatageeks/datafusion-bio-formats#248) and the OpenDAL 0.59.5 fix for
remote reads that retried forever under tokio 1.54 (#274). The lockfile
keeps a single copy of datafusion-bio-format-core, datafusion, arrow and
parquet, and drops the OpenDAL git revision.

Also import Session and PhysicalPlanningContext in the ranges planners
instead of spelling out their paths inline (review nit).

Co-Authored-By: Claude Opus 5.5 <[email protected]>

---------

Co-authored-by: Claude Opus 5.5 <[email protected]>