chore: bump DataFusion to 55.0.0 and bio-formats to its DF 55 branch - #236

…x) (#304)

* fix(vep): match VEP 116 frameshift tail and stop-loss del (#223, #227)

* fix(vep): gate CDS endpoint start and stop predicates

* fix(vep): preserve partial RefSeq edit coordinates (#242)

* fix(vep): preserve stop-retained partial-end fallback

* fix(vep): map RefSeq HGVS through unedited exons (#242)

* fix(vep): classify terminal exon runoff as complex indel

* fix(vep): classify edits across RefSeq mapper gaps (#242)

* fix(vep): apply stop fallbacks to complex boundaries

* fix(vep): use spliced span for stop-loss fallback

* fix(vep): retain stop fallbacks for legacy transcript bounds

* fix(vep): preserve partial RefSeq gap coding predictions (#242)

* fix(vep): keep defined partial peptides on stop predicates

* fix(vep): replay RefSeq insertion edit alleles (#242)

* fix(vep): map stop fallback to transcript coordinates

* fix(vep): normalize insertion spans for stop overlap

* fix(vep): exclude terminal stop from clipped frameshift deletion

* fix(vep): omit terminal stop from clipped frameshift deletion

* fix(vep): preserve clipped terminal frameshift HGVS

* fix(vep): replay edited mapper gap HGVS (#242)

* fix(vep): preserve terminal Ter in clipped deletions

* fix(vep): preserve VEP ordering for star-ended HGVSc ranges (#244)

* fix(vep): scope star-negative HGVSc ordering to #244

* fix(vep): handle clamped #244 star-negative ranges

* fix(vep): preserve mapper order for star-negative HGVSc

* test(vep): cover RNA-edit codons with unequal CDS lengths (#242)

* fix(vep): retain star-negative clamped HGVSc order

* fix(vep): keep canonical codons across RNA-edit lengths (#242)

* fix(vep): align terminal insertion stop predicates

* fix(vep): order clamped star-negative HGVSc ranges

* fix(vep): map point insertions in terminal stop fallback

* fix(vep): order star-negative HGVSc insertion flanks

* fix(vep): preserve terminal stop on insertions

* fix(vep): include RefSeq edits in cDNA endpoint length

* fix(vep): stop local peptides at terminal stop

* fix(vep): match cached RefSeq cDNA endpoint length

* fix(vep): restore empty terminal reference residue

* fix(vep): restore empty terminal reference residue

* fix(vep): retain RefSeq HGVSc beyond terminal exon

* fix(vep): preserve terminal stop with partial tail

* fix(vep): retain terminal residue after X trim

* fix(vep): order star-negative transcript coordinates

* fix(vep): retain terminal residue after X trim

* fix(vep): retain terminal insertion stop with partial tail

* fix(vep): retain RefSeq variants within translated CDS

* perf(vep): borrow RefSeq CDS length for boundary checks

* fix(vep): map shifted RefSeq dups through exon boundaries (#236)

* fix(vep): keep shifted RefSeq insertions intronic (#236)

* fix(vep): classify RNA-edit MNVs from genomic alleles (#155)

* fix(vep): pass genomic reference to RNA-edit HGVS (#155)

* fix(vep): preserve RNA-edit HGVSc reference display (#155)

* fix(vep): scope RNA-edit genomic HGVS to MNVs

* fix(vep): map HGVSc insertion flanks in VEP order

* fix(vep): classify terminal insertion stop after CDS

* fix(vep): preserve VEP slice order for clamped insertions

* fix(vep): preserve independent RefSeq splice consequences

* test(vep): cover shifted RefSeq dup exon boundaries

* fix(vep): classify RNA-edited HGVS after clamping

* fix(vep): preserve independent RefSeq splice consequences

* fix(vep): preserve independent RefSeq splice consequences

* test(vep): cover shifted RefSeq dup exon boundaries

* fix(vep): retain stop before terminal stop codon

* fix(vep): map HGVSc in VEP transcript-slice order

* test(vep): cover clamped HGVSc insertions on both strands

* fix(vep): map exon-boundary insertion for stop fallback

* fix(vep): preserve splice-region semantics for edited RefSeq endpoints

* fix(vep): map insertion VF span before HGVS sorting

* fix(vep): retain reference codons for edited gap replacements

* fix(vep): preserve shifted HGVSc insertion VF order

* fix(vep): honor VEP stop fallback endpoint guards

* fix(vep): preserve slice order for clamped HGVS

* fix(vep): construct clamped HGVS in slice order

* fix(vep): exclude downstream stop from insertion peptide

* fix(vep): preserve legacy RefSeq stop-loss fallback

* fix(vep): match VEP terminal peptide splice rules

* fix(vep): preserve legacy stop-loss fallback

* test(vep): tighten HGVSc ordering review coverage

* fix(vep): preserve clipped HGVSc insertion gap

* fix(vep): retain clipped insertion cDNA order

* test(vep): cover clipped insertion output coordinate bypass

* test(vep): isolate clipped insertion shortcut case

* fix(vep): use mapped insertion peptide for stop predicates

* fix(vep): gate ambiguous ALT peptides and HGVS

* fix(vep): suppress ambiguous codon-derived amino acids

* fix(vep): retain independent RefSeq output offsets

* fix(vep): preserve ambiguous insertion domain boundaries

* fix(vep): gate RefSeq offsets for ambiguous ALT

* fix(vep): preserve V115 ambiguous allele semantics

* fix(vep): gate partial-codon stop-retained shortcut

* docs(vep): clarify terminal stop shortcut guard

* fix(vep): preserve V115 domain insertion semantics

* test(vep): cover full V116 IUPAC reverse complement

* merge: retain mapped RefSeq codons at insertion boundaries

* docs(vep): attach domain docs to lookup_domains

* fix(vep): derive RefSeq insertion frame from mapped CDS

* fix(vep): preserve mapped domain insertion geometry

…332)

* chore: bump DataFusion to 55.2.0 and bio-formats to its DF 55 branch

Move the workspace from DataFusion 53.0.0 to 55.2.0 (arrow/parquet 59.3.2,
rust-toolchain and CI on 1.94.0, DataFusion 55.2's MSRV) and pin every
datafusion-bio-format-* dependency to biodatageeks/datafusion-bio-formats#248
(d1eb0c3), which carries the same bump rebased on formats v1.15.0, so it
includes everything the previous e0379fb pin did. pileup and vep share that
one rev so the graph holds one copy of datafusion-bio-format-core and
DataFusion.

Mechanical fallout of the 54/55 API changes:
- `as_any` removed from DataFusion traits: dropped from every impl, the
  `.as_any()` hop removed at plan/expr downcast sites (including the
  multi-line chains in the interval planner), unused `Any` imports
  removed. fastqc's own `QcModule::as_any` is unrelated and kept.
- `ExecutionPlan::apply_expressions` required: leaf implementations for
  the nodes without expressions; `IntervalJoinExec` visits its join keys
  and filter like DataFusion's own `HashJoinExec`.
- `QueryPlanner` / `PhysicalPlanner` take `&dyn Session`, and
  `create_physical_expr` gains the `PhysicalPlanningContext` argument.
- `create_hashes` takes DataFusion's `HashState`; the interval join uses
  `datafusion::common::hash_utils::RandomState` (ahash stays for the
  plain hash maps).
- `Distribution::HashPartitioned` -> `KeyPartitioned`.
- `MemoryPool` now requires `Display` and `name()`: TracingPool and the
  test-only RejectHashPool.
- `MutableArrayData::extend` is deprecated in arrow 59.3: the SIFT page
  gather uses `try_extend` and propagates the overflow error.
- `partition_statistics` returns `Arc<Statistics>` and is deprecated for
  callers; the interval join estimator and the plugin-cache join
  selector read statistics through `StatisticsContext`.

That last item is the one behavioural change. Under DataFusion 55 the
deprecated `partition_statistics` returns unknown statistics for
DataFusion's own nodes, so `choose_for_hash_plan` would have reported
MissingStatistics and silently forced every plugin-cache tier join onto
SortMergeJoin; the adaptive-join tests cover it.

Why: DataFusion 53's external sorter frees its merge headroom and then
re-acquires it from an empty reservation, which starves under
`FairSpillPool` when several partitions sort concurrently (#235).
apache/datafusion#20642 fixes that and shipped in 54.0.0.

Not touched: `research/lance_encoding_sandbox` is outside the workspace
and still pins arrow 58 / datafusion-common 53.

Supersedes #236 and dependabot #8.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* refactor: address review nits on the DataFusion 55.2 bump

- interval join: say why `hash_utils::RandomState::default()` is safe
  (foldhash `FixedState` with a compile-time constant seed, shared by
  both join sides), and fold `partition_statistics_inner` back into
  `partition_statistics` where it lived before the bump, with a note on
  why the partition argument is ignored. Drop the stray blank line left
  by the `Any` import removal.
- plugin-cache join selector: import `StatisticsContext` /
  `StatisticsArgs` instead of spelling the full paths inline.
- the ten leaf `apply_expressions` impls (fastqc, pileup, ranges, vep):
  use imported `Arc` / `PhysicalExpr` / `Result` / `TreeNodeRecursion`
  instead of fully qualified paths.

No behaviour change.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* chore: re-pin datafusion-bio-formats to 99d7f81

Follow biodatageeks/datafusion-bio-formats#248 to its review-nit commit
(test renames, shorter `apply_expressions` signatures, CHANGELOG arrow
version). No API change; vep and pileup stay on one formats rev.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* chore: pin datafusion-bio-formats to the v1.16.0 release

Move every datafusion-bio-format-* dependency (vep and pileup) from the
pre-release rev to tag v1.16.0, which contains the DataFusion 55.2 bump
(biodatageeks/datafusion-bio-formats#248) and the OpenDAL 0.59.5 fix for
remote reads that retried forever under tokio 1.54 (#274). The lockfile
keeps a single copy of datafusion-bio-format-core, datafusion, arrow and
parquet, and drops the OpenDAL git revision.

Also import Session and PhysicalPlanningContext in the ranges planners
instead of spelling out their paths inline (review nit).

Co-Authored-By: Claude Opus 5.5 <[email protected]>

---------

Co-authored-by: Claude Opus 5.5 <[email protected]>