Repository navigation
Implement scale/offset codec in the EOPF data model #134
Description
Context
The scale/offset codec extension has been merged in the zarr-extensions repository, providing a standardized Zarr V3 array-to-array codec for scale/offset transformation. This closes the specification gap documented in zarr-developers/zarr-extensions#42.
The next step is implementation in zarr-python, which would make the codec transparently available through xarray and our downstream stack (data pipeline, TiTiler).
What this enables
With a standardized and implemented scale/offset codec, our GeoZarr data arrays can:
- Store reflectance data efficiently as
uint16(0–10000) while transparently presentingfloat32values (0.0–1.0) to users - Follow CF conventions for physical quantities without requiring users to manually apply
scale_factor/add_offsetattributes - Decouple storage encoding from data presentation — the Zarr V3 design intent
This directly addresses the reflectance data convention discussion with EOPF CPM, where both sides agree on integer storage with codec-based float presentation.
Tasks
- Confirm zarr-python implementation status — @d-v-b: is the scale/offset codec from zarr-extensions#43 being implemented in zarr-python? If so, this is an activity we can do in this project.
- Update the data pipeline to use the codec in the converter and the data pipeline @lhoupert
- Validate downstream compatibility and confirm that TiTiler correctly decode data through the codec without additional handling @vincentsarago
- Document the convention in the data model spec @emmanuelmathot
Related issues
- FixedScaleOffset codec not preserved during zarr v2 to v3 conversion #106
- consistently export data as float64 #108
- Request for Standardized FixedScaleOffset Codec Extension zarr-developers/zarr-extensions#42
- scale-offset codec zarr-developers/zarr-extensions#43
- https://gitlab.cpm.eopf.copernicus.eu/eopf/eopf-cpm/-/issues/985 — Data presentation convention for reflectance values
- Support for fixedscaleoffset manzt/numcodecs.js#49
- Implement fixedscaleoffset codec manzt/zarrita.js#312
- Add V3 cast_value and scale_offset codecs manzt/zarrita.js#395
Activity
emmanuelmathot commentedon Mar 16, 2026
@d-v-b @lhoupert @vincentsarago, please create the relevant sub-tasks or related tasks in the appropriate repository.
d-v-b commentedon Mar 16, 2026
yes, I have 2 working implementations. One is pure python + numpy, in this PR against zarr-python: zarr-developers/zarr-python#3774. If this is hard to merge, we can factor it out into a separate package.
Because I was concerned about the performance of python + numpy and because I anticipated the need to support a javascript implementation, I commissioned a rust-based implementation: https://github.com/d-v-b/zarr-cast-value. This is ~4 - 7x faster, according to some rough benchmarks.
for context, the
cast_valueoperation is performance-sensitive because it requires iterating over each array scalar and potentially replacing that scalar with an arbitrary element from the destination data type. This was necessary to generically express numpy casting behavior.My recommendation: if we don't have fast reviewer activity on that PR, or people think this functionality is too complicated to exist in zarr-python, then we split this off into a separate python package. And in that package, we can depend on rust bindings if we feel like the performance is worth it.