Lance row groups currently report memory_size as a placeholder value of 1, which gives cache and scheduling callers no useful idea of how much decoded memory a chunk may occupy. We need a best-effort estimate that also works for externally referenced Lance files, without relying on extra information being recorded when the file is written.

This change derives the estimate from the existing schema and footer metadata. Fixed-width leaf columns use the logical row count multiplied by their value width. Variable-width and nested columns use Lance page metadata: completed pages follow the 8 MiB writer page target, the final page is prorated from its row count, and encoded page bytes are only used as a lower bound or single-page fallback instead of being mistaken for decoded memory. Deletions are handled by applying the live-to-physical row ratio once, and physical columns belonging to nested fields are grouped back into their top-level schema columns.

The Rust bridge now provides estimates for each top-level column and keeps the fragment-level API as the saturating sum of those values. LanceTableReader distributes the fragment estimate across logical chunks according to their row ranges, so the chunk values add back up to the fragment total and remain consistent across projections and cached reader creation. Estimation stays best effort on the C++ side, with a zero fallback when metadata cannot be estimated, including unsupported legacy files.

The estimator resolves both normal dataset files and files referenced through external base paths. It only reads schema, column, and page metadata from the footer and never reads or decompresses data pages. It also relies on Lance's normalized ColumnInfo and PageInfo representation for storage formats 2.1 through 2.3, avoiding encoding-specific logic that would need to be maintained whenever the file format changes. The current Lance version does not expose get_file_metadata publicly, so the implementation temporarily uses the public read_all_metadata path without sharing Lance's metadata cache; after upgrading to a version containing lance-format/lance#7820, this can switch to cached metadata and remove the duplicate footer I/O.

Signed-off-by: jiaqizho <[email protected]>

Make the full and indexed file metadata lookup methods public so callers
can reuse the fragment's dataset-level metadata cache instead of loading
metadata independently through FileReader.

I ran into this while trying to inspect the file footer of a fragment
that had already been opened. The obvious option was to call
FileReader::read_all_metadata, but that API reads directly from storage
and does not reuse the dataset’s file metadata cache. This can result in
another metadata I/O even though the fragment has already loaded the
same information.

FileFragment already has cache-aware helpers for loading the full
metadata or metadata index. This change makes get_file_metadata and
get_file_metadata_index public so other fragment-level operations can
reuse the existing cached metadata instead of reading it again through
FileReader.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Exposed file metadata and metadata index access for broader
integration and tooling.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

(cherry picked from commit f23b8d6)