The kerchunk struct-dtype fix for FITS binary tables merged upstream as
zarr-developers/VirtualiZarr#1037, so drop the emfdavid fork pin. That rev
(8a1563f) was never pushed to the fork remote and only ever resolved from
uv's local cache -- it was unresolvable from a clean checkout. Pin to
zarr-developers main (f46902d), verified to carry the merged fix as an
ancestor. The fix ships in v2.7.1, so a git pin is still required; it
becomes a plain virtualizarr>=2.7.1 once that releases.

Moving to current upstream deprecated two APIs the astronomy build-time
path uses:

  - .virtualize accessor -> .vz
  - virtualizarr.registry.ObjectStoreRegistry -> obspec_utils.registry

Both are confined to the four FITS index-build sites (two tests, two
examples); nothing in src/ or on the hot path. obspec-utils was already
present as a VirtualiZarr dependency, so declaring it in [astronomy]
adds no package (uv still resolves 202) -- it just stops us importing
through a re-export scheduled for removal.

Astronomy tests pass under -W error::DeprecationWarning.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

The SDSS/spectra astronomy example reads FITS binary tables (structured
dtypes) end-to-end, which needs two upstream fixes not yet in a release:

  - zarr struct-dtype byte-order handling in the bytes codec
    (zarr-developers/zarr-python#4142, merged, past v3.2.1)
  - VirtualiZarr kerchunk struct-dtype refs for FITS binary tables
    (zarr-developers/VirtualiZarr#1037, open on the emfdavid fork)

Pin both to git commits via [tool.uv.sources]. These are dev-environment
only -- uv strips them from the published wheel, so the abstract zarr>=3.0
requirement downstream is unchanged. Drop each pin once a release ships.

Add an `astronomy` optional-dependency extra (virtualizarr/kerchunk/astropy/
s3fs/icechunk) that formalizes the FITS build-time index stack for both the
Hubble and SDSS examples, replacing the ad-hoc `uv run --with ...` install;
point the Hubble README at it.

Verified: uv sync resolves, the struct-dtype endianness repro passes on the
pinned env, and the full suite is green (168 passed / 10 skipped) with the
git-pinned dev-zarr.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

…eshard)

An astroML cross-domain companion to the Hubble example, and the decode-
amortization counterpart to it. astroML's spectral-PCA workflow first
downloads the raw archive and resamples every spectrum onto a common grid
into one spec4000.npz -- the download-and-reshard step insitubatch argues
against. examples/sdss indexes a real SDSS spPlate frame on data.sdss.org
(HTTPS) as virtual references: one plate holds all ~640 fibers on a common
log-wavelength grid as a single (fiber, wave) image, which _rechunk_fibers
splits along the fiber axis into contiguous virtual chunks by pure byte
arithmetic -- no pixels moved. sample_axis=0 makes each fiber one sample and,
because a chunk is many fibers, this is the O(chunks) decode-amortization
regime (contrast Hubble's one image per chunk). Verified byte-identical to
astropy.

The task mirrors astroML's spectral reconstruction: a 1-D conv autoencoder
trained by SGD over the streamed mini-batches beats a PCA baseline at the same
latent dim, because galaxy spectra are translation-structured (redshift shifts
the lines, which a fixed-dim linear basis reconstructs poorly). The offline
synthetic default (redshift-shifted spectra) carries the beats-PCA claim and
backs tests/test_sdss.py; the real run is a same-pipeline, real-archive demo.
Only one plate per store: plates cover slightly different wavelength ranges, so
concatenating them would require resampling -- the reshard this example avoids.

SDSS spectra also ship as FITS binary tables (structured big-endian dtypes).
tests/test_fits_bintable.py validates that path end-to-end through insitubatch
(kerchunk -> VirtualiZarr -> Icechunk -> insitubatch) against astropy, and pins
the VirtualiZarr fix it needs (zarr-developers/VirtualiZarr#1037) -- released
VirtualiZarr raises on the list-valued kerchunk field specs. Endianness is
dropped in the VZ/icechunk struct metadata, so consumers reinterpret each field
with .view('>f4') at the numpy boundary (the zarr #4142 codec fix does not
engage in this path, so it is not pinned).

Co-Authored-By: Claude Opus 4.8 <[email protected]>

The kerchunk struct-dtype fix for FITS binary tables merged upstream as
zarr-developers/VirtualiZarr#1037, so drop the emfdavid fork pin. That rev
(8a1563f) was never pushed to the fork remote and only ever resolved from
uv's local cache -- it was unresolvable from a clean checkout. Pin to
zarr-developers main (f46902d), verified to carry the merged fix as an
ancestor. The fix ships in v2.7.1, so a git pin is still required; it
becomes a plain virtualizarr>=2.7.1 once that releases.

Moving to current upstream deprecated two APIs the astronomy build-time
path uses:

  - .virtualize accessor -> .vz
  - virtualizarr.registry.ObjectStoreRegistry -> obspec_utils.registry

Both are confined to the four FITS index-build sites (two tests, two
examples); nothing in src/ or on the hot path. obspec-utils was already
present as a VirtualiZarr dependency, so declaring it in [astronomy]
adds no package (uv still resolves 202) -- it just stops us importing
through a re-export scheduled for removal.

Astronomy tests pass under -W error::DeprecationWarning.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

The SDSS/spectra astronomy example reads FITS binary tables (structured
dtypes) end-to-end, which needs two upstream fixes not yet in a release:

  - zarr struct-dtype byte-order handling in the bytes codec
    (zarr-developers/zarr-python#4142, merged, past v3.2.1)
  - VirtualiZarr kerchunk struct-dtype refs for FITS binary tables
    (zarr-developers/VirtualiZarr#1037, open on the emfdavid fork)

Pin both to git commits via [tool.uv.sources]. These are dev-environment
only -- uv strips them from the published wheel, so the abstract zarr>=3.0
requirement downstream is unchanged. Drop each pin once a release ships.

Add an `astronomy` optional-dependency extra (virtualizarr/kerchunk/astropy/
s3fs/icechunk) that formalizes the FITS build-time index stack for both the
Hubble and SDSS examples, replacing the ad-hoc `uv run --with ...` install;
point the Hubble README at it.

Verified: uv sync resolves, the struct-dtype endianness repro passes on the
pinned env, and the full suite is green (168 passed / 10 skipped) with the
git-pinned dev-zarr.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

…eshard)

An astroML cross-domain companion to the Hubble example, and the decode-
amortization counterpart to it. astroML's spectral-PCA workflow first
downloads the raw archive and resamples every spectrum onto a common grid
into one spec4000.npz -- the download-and-reshard step insitubatch argues
against. examples/sdss indexes a real SDSS spPlate frame on data.sdss.org
(HTTPS) as virtual references: one plate holds all ~640 fibers on a common
log-wavelength grid as a single (fiber, wave) image, which _rechunk_fibers
splits along the fiber axis into contiguous virtual chunks by pure byte
arithmetic -- no pixels moved. sample_axis=0 makes each fiber one sample and,
because a chunk is many fibers, this is the O(chunks) decode-amortization
regime (contrast Hubble's one image per chunk). Verified byte-identical to
astropy.

The task mirrors astroML's spectral reconstruction: a 1-D conv autoencoder
trained by SGD over the streamed mini-batches beats a PCA baseline at the same
latent dim, because galaxy spectra are translation-structured (redshift shifts
the lines, which a fixed-dim linear basis reconstructs poorly). The offline
synthetic default (redshift-shifted spectra) carries the beats-PCA claim and
backs tests/test_sdss.py; the real run is a same-pipeline, real-archive demo.
Only one plate per store: plates cover slightly different wavelength ranges, so
concatenating them would require resampling -- the reshard this example avoids.

SDSS spectra also ship as FITS binary tables (structured big-endian dtypes).
tests/test_fits_bintable.py validates that path end-to-end through insitubatch
(kerchunk -> VirtualiZarr -> Icechunk -> insitubatch) against astropy, and pins
the VirtualiZarr fix it needs (zarr-developers/VirtualiZarr#1037) -- released
VirtualiZarr raises on the list-valued kerchunk field specs. Endianness is
dropped in the VZ/icechunk struct metadata, so consumers reinterpret each field
with .view('>f4') at the numpy boundary (the zarr #4142 codec fix does not
engage in this path, so it is not pinned).

Co-Authored-By: Claude Opus 4.8 <[email protected]>

The kerchunk struct-dtype fix for FITS binary tables merged upstream as
zarr-developers/VirtualiZarr#1037, so drop the emfdavid fork pin. That rev
(8a1563f) was never pushed to the fork remote and only ever resolved from
uv's local cache -- it was unresolvable from a clean checkout. Pin to
zarr-developers main (f46902d), verified to carry the merged fix as an
ancestor. The fix ships in v2.7.1, so a git pin is still required; it
becomes a plain virtualizarr>=2.7.1 once that releases.

Moving to current upstream deprecated two APIs the astronomy build-time
path uses:

  - .virtualize accessor -> .vz
  - virtualizarr.registry.ObjectStoreRegistry -> obspec_utils.registry

Both are confined to the four FITS index-build sites (two tests, two
examples); nothing in src/ or on the hot path. obspec-utils was already
present as a VirtualiZarr dependency, so declaring it in [astronomy]
adds no package (uv still resolves 202) -- it just stops us importing
through a re-export scheduled for removal.

Astronomy tests pass under -W error::DeprecationWarning.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

The SDSS/spectra astronomy example reads FITS binary tables (structured
dtypes) end-to-end, which needs two upstream fixes not yet in a release:

  - zarr struct-dtype byte-order handling in the bytes codec
    (zarr-developers/zarr-python#4142, merged, past v3.2.1)
  - VirtualiZarr kerchunk struct-dtype refs for FITS binary tables
    (zarr-developers/VirtualiZarr#1037, open on the emfdavid fork)

Pin both to git commits via [tool.uv.sources]. These are dev-environment
only -- uv strips them from the published wheel, so the abstract zarr>=3.0
requirement downstream is unchanged. Drop each pin once a release ships.

Add an `astronomy` optional-dependency extra (virtualizarr/kerchunk/astropy/
s3fs/icechunk) that formalizes the FITS build-time index stack for both the
Hubble and SDSS examples, replacing the ad-hoc `uv run --with ...` install;
point the Hubble README at it.

Verified: uv sync resolves, the struct-dtype endianness repro passes on the
pinned env, and the full suite is green (168 passed / 10 skipped) with the
git-pinned dev-zarr.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

…eshard)

An astroML cross-domain companion to the Hubble example, and the decode-
amortization counterpart to it. astroML's spectral-PCA workflow first
downloads the raw archive and resamples every spectrum onto a common grid
into one spec4000.npz -- the download-and-reshard step insitubatch argues
against. examples/sdss indexes a real SDSS spPlate frame on data.sdss.org
(HTTPS) as virtual references: one plate holds all ~640 fibers on a common
log-wavelength grid as a single (fiber, wave) image, which _rechunk_fibers
splits along the fiber axis into contiguous virtual chunks by pure byte
arithmetic -- no pixels moved. sample_axis=0 makes each fiber one sample and,
because a chunk is many fibers, this is the O(chunks) decode-amortization
regime (contrast Hubble's one image per chunk). Verified byte-identical to
astropy.

The task mirrors astroML's spectral reconstruction: a 1-D conv autoencoder
trained by SGD over the streamed mini-batches beats a PCA baseline at the same
latent dim, because galaxy spectra are translation-structured (redshift shifts
the lines, which a fixed-dim linear basis reconstructs poorly). The offline
synthetic default (redshift-shifted spectra) carries the beats-PCA claim and
backs tests/test_sdss.py; the real run is a same-pipeline, real-archive demo.
Only one plate per store: plates cover slightly different wavelength ranges, so
concatenating them would require resampling -- the reshard this example avoids.

SDSS spectra also ship as FITS binary tables (structured big-endian dtypes).
tests/test_fits_bintable.py validates that path end-to-end through insitubatch
(kerchunk -> VirtualiZarr -> Icechunk -> insitubatch) against astropy, and pins
the VirtualiZarr fix it needs (zarr-developers/VirtualiZarr#1037) -- released
VirtualiZarr raises on the list-valued kerchunk field specs. Endianness is
dropped in the VZ/icechunk struct metadata, so consumers reinterpret each field
with .view('>f4') at the numpy boundary (the zarr #4142 codec fix does not
engage in this path, so it is not pinned).

Co-Authored-By: Claude Opus 4.8 <[email protected]>

The kerchunk struct-dtype fix for FITS binary tables merged upstream as
zarr-developers/VirtualiZarr#1037, so drop the emfdavid fork pin. That rev
(8a1563f) was never pushed to the fork remote and only ever resolved from
uv's local cache -- it was unresolvable from a clean checkout. Pin to
zarr-developers main (f46902d), verified to carry the merged fix as an
ancestor. The fix ships in v2.7.1, so a git pin is still required; it
becomes a plain virtualizarr>=2.7.1 once that releases.

Moving to current upstream deprecated two APIs the astronomy build-time
path uses:

  - .virtualize accessor -> .vz
  - virtualizarr.registry.ObjectStoreRegistry -> obspec_utils.registry

Both are confined to the four FITS index-build sites (two tests, two
examples); nothing in src/ or on the hot path. obspec-utils was already
present as a VirtualiZarr dependency, so declaring it in [astronomy]
adds no package (uv still resolves 202) -- it just stops us importing
through a re-export scheduled for removal.

Astronomy tests pass under -W error::DeprecationWarning.

Co-Authored-By: Claude Opus 4.8 <[email protected]>