feat!: one scale/offset encoding switch by emmanuelmathot · Pull Request #287 · EOPF-Explorer/data-model

emmanuelmathot · 2026-09-28T14:21:16Z

Closes #281. Closes #246.

Summary

One switch now selects how packed arrays (for example S2 reflectance: uint16, scale_factor 0.0001, add_offset -0.1, nodata 0) are stored. The decoded float32 output is gone.

Mode	Option	On disk	Metadata
Zarr codec	`scale_offset_codec=True` / `--scale-offset-codec`	Source integer dtype, packed by the Zarr `scale_offset` + `cast_value` codecs (from #154); logical dtype `float32`, fill value NaN	No CF scale attributes; no STAC scale fields
Default: ESA layout	None	Source integer dtype, no scale codecs	CF `scale_factor`, `add_offset`, `_FillValue` on arrays; `raster:scale`, `raster:offset`, `nodata` on the STAC `reflectance` asset (#267)

Changes

Encoding (create_uniform_encoding): packing_of() finds the CF packing in .encoding (decoded input) or .attrs (raw input, as in the CPM path). Each packed array is first normalised to decoded float32 with NaN for nodata, so both input forms give the same output. Then:
- the default mode adds the scale_offset + cast_value filters
- the ESA mode writes the CF keys
Arrays without packing (masks, classification, angles) keep their integer dtype and fill value in both modes. The scale_factor 1 / add_offset 0 pair that Zarr v3 CPM products declare on SCL is dropped, so SCL stays uint8 and is not decoded to float.
Pyramid levels:
- decoded levels are averaged with NaN skipped
- raw integer levels are now rounded instead of truncated
- a fill value of 0 is no longer treated as "no fill value"
- finer bands are decoded before injection, so nodata is excluded from the averages even when the input was not masked (decoded Zarr v3 input declares nodata only in the EOPF fill_value attribute)
API and CLI:
- convert_s2_optimized(scale_offset_codec=...) and create_multiscale_from_datatree(scale_offset_codec=...) (default True) replace keep_scale_offset
- convert-s2-optimized --no-scale-offset-codec replaces --keep-scale-offset
- --experimental-scale-offset-codec is removed, because the codec is now the default
- the OLCI converter and CLI lose their unused keep_scale_offset; OLCI already writes raw uint16 with CF attributes
CPM writer: new scale_offset_codec option (default True) and eopf convert-geozarr --no-scale-offset-codec. keep_scale_offset is still accepted for one release with a DeprecationWarning: keep_scale_offset=True is scale_offset_codec=False (ESA layout).
Docs: an Encoding section in docs/converter.md, a short note in README.md.

Breaking changes

The default output of convert-s2-optimized, convert (S2 route) and the CPM writer changes from decoded float32 to codec-packed integers. Decoded values are the same within one packing step.
keep_scale_offset is removed from convert_s2_optimized, create_multiscale_from_datatree, create_uniform_encoding, convert_olci_optimized and the CLI. The CPM writer keeps it for one release.
The CLI flag --experimental-scale-offset-codec is removed. The S2 convert templates in platform-deploy still pass it, so they must drop it before they use this release (their output stays the same, because the codec is now the default).

Tests

test_create_measurements_encoding: both modes × raw and decoded input. This replaces the add_offset check that never ran (@lhoupert's comment in feat!: reintegrate cpm-geozarr into main #277).
test_create_multiscale_from_datatree_behavior: both modes × raw (CPM v3 style, nodata only in fill_value) and decoded input. At every level r10m–r720m it checks dtype, codecs, CF attributes, decoded values within one packing step, and that a nodata block stays NaN.
test_inject_missing_bands_skips_unmasked_nodata: new test.
CPM writer: default codec output, scale_offset_codec=False with its STAC fields, and the keep_scale_offset=True deprecation.
Snapshots regenerated (REGENERATE_SNAPSHOTS=1). Only the reflectance arrays and their group metadata change: float32 logical dtype, NaN fill, codecs nested in the sharding codec, no CF scale attributes. The snapshot comparison now uses JSON, because a NaN fill value never equals itself.
Locally: full suite on Python 3.12, CPM tests on 3.13 with --extra cpm, and ruff/pyright all pass.

🤖 Generated with Claude Code

Packed arrays keep their source integer dtype. By default the Zarr scale_offset + cast_value codecs store them; scale_offset_codec=False (--no-scale-offset-codec) writes the ESA layout with CF scale_factor, add_offset and _FillValue, and STAC raster:scale/raster:offset/nodata on the reflectance asset. The decoded float32 output is removed. Raw (CPM, mask_and_scale=False) and decoded input give the same output. Raw integer pyramid levels are rounded instead of truncated, a 0 fill value is honoured, finer bands are decoded before injection so unmasked nodata is not averaged in, and the trivial scale 1 / offset 0 on SCL is dropped. BREAKING CHANGE: keep_scale_offset is removed from convert_s2_optimized, create_multiscale_from_datatree, create_uniform_encoding, convert_olci_optimized and the CLI; the CLI flag --experimental-scale-offset-codec is removed. The CPM writer still accepts keep_scale_offset for one release with a DeprecationWarning. Closes #281 Co-Authored-By: Claude Opus 5.5 <[email protected]>

lhoupert

Thanks, this covers my comments from #277 and #281:

One non-blocking detail picked up by Claude:

packing_of also looks at var.encoding["fill_value"], but xarray (2026.4) doesn't set that key for Zarr v3 arrays. So a v3 input with nodata only in the Zarr fill_value (no EOPF attribute) isn't masked, and DN 0 is packed as a valid -0.1. CPM products aren't affected. Either read the Zarr array's fill_value, or drop that candidate and the "or the Zarr fill_value" in the docstring.

Is it worth raising an issue for it?

- Updated `test_convert_command_routes_s2_to_optimized` to reflect changes in default encoding from Zarr scale-offset codecs to ESA layout with CF attributes. - Renamed `test_write_s2_default_uses_scale_offset_codecs` to `test_write_s2_default_writes_esa_layout` and adjusted assertions to verify the ESA layout. - Removed the test for writing without scale offset codec and added a new test `test_write_s2_scale_offset_codec` to validate the behavior when `scale_offset_codec=True`. - Updated the deprecation warning test for `keep_scale_offset` to reflect the new codec handling logic.

emmanuelmathot · 2026-09-30T07:39:49Z

@claxn @shollendEODC can be merged as soon as you review it

shollendEODC · 2026-09-30T08:20:55Z

I am currently checking the code and am verifying everything works as expected.

shollendEODC · 2026-09-30T12:08:44Z

Alright, I had a quick look and applied the conversion to S2B_MSIL2A_20260721T100559_N0512_R022_T33UWQ_20260721T143508.SAFE with both 'scale_offset_codec': False and 'scale_offset_codec': True.

For either options, the measurements and injected bands (as they have decoding attributes) get fed through _decode_packed(..) which automatically decodes them to float32, even if they are later saved again as uint16 through the default option 'scale_offset_codec': False. This seems like an unnecessary cast to me, and just increases memory consumption. From my POV the arrays should stay as uint16 and not get casted at all.. This would also need to be considered in the injected bands, as they automatically get casted to float32 which leads to both float32 and uint16 bands being present in the bands (until the other measurements get casted as well).

Of course, a casting already occurs during the coarsening of variables to calculate the multiscales (no data handling/masking), but as this would only afflict certain products (and only the coarsened bands themselves), and as we aim to generalize and harmonize the product-specific functions into the utilities I would refrain from implementing a float cast for all products.

I am not fully aware of the codec implementations and the introduces casting, these observations are based on my personal (development)/user perspective.

shollendEODC · 2026-10-01T08:28:25Z

As this PR should be resolved before merging #292 I wanted to ask for your input on my previous comment @emmanuelmathot? many thanks

claxn

Thanks @emmanuelmathot for the quick work - I left two small remarks, which would be good to be addressed/discussed before merging.

claxn · 2026-10-01T13:49:53Z

+            fill_value = var_data.attrs.get("fill_value")
+            if fill_value is None:
+                fill_value = var_data.encoding.get("_FillValue")


Would it be possible to replace those lines with a new function, e.g. find_fill_value(), which is based on the following lines:

data-model/src/eopf_geozarr/s2_optimization/s2_multiscale.py

Lines 100 to 106 in 8beecb3

candidates = (

var.encoding.get("_FillValue"),

var.attrs.get("_FillValue"),

var.attrs.get("fill_value"),

var.encoding.get("fill_value"),

)

fill = next((value for value in candidates if value is not None), None)

?

claxn · 2026-10-01T14:02:50Z

+    for var_name in list(dataset.data_vars):
+        packing = packing_of(dataset[var_name])
+        if packing is not None:
+            dataset[var_name] = _decode_packed(dataset[var_name], packing)


As @shollendEODC already mentioned this call is executed no matter how scale_offset_codec is set. I would suggest to include scale_offset_codec here in this if statement and only execute _decode_packed if scale_offset_codec=True (and we need to verify that this does not cause any issues further down the pipeline, e.g. when computing the multiscales)

…tiscale processing

emmanuelmathot · 2026-10-01T20:09:39Z

Sorry, i was busy whole day. Great catches. Fixing both.

github-actions · 2026-10-01T20:10:39Z

Security Audit Report

View workflow run

Bandit — Static Security Analysis (Security tab)

12 issue(s) found: 12 low

✅ No issues at or above HIGH severity.

12 low issue(s) below threshold not shown in table.

pip-audit — Dependency Vulnerabilities (Security tab)

Package	Version	ID	Fix Versions	Description
urllib3	2.7.0	PYSEC-2026-4177	2.8.0	## Impact urllib3's streaming API is d
urllib3	2.7.0	PYSEC-2026-4176	2.8.0	### Impact urllib3's streaming API is designed for the efficient handling of large HTTP responses by reading content in
urllib3	2.7.0	PYSEC-2026-4175	2.8.0	## Impact urllib3 supports configuring TLS independently for an HTTPS proxy and the target server. `proxy_ssl_context`
tornado	6.5.8	GHSA-chx6-46f5-w4vp	6.5.9	An unbounded memory accumulation (decompression bomb) in `tornado.curl_httpclient.CurlAsyncHTTPClient` — the client-side
tornado	6.5.8	GHSA-c2m8-h5v5-343r	6.5.9	### Summary StaticFileHandler allows an unauthenticated attacker to read arbitrary files from the server's filesystem by
tornado	6.5.8	GHSA-3hv7-mjh2-fv65	6.5.9	## Summary `HTTPServerRequest.__init__` in `tornado/httputil.py` parses the URL query string via `parse_qs_bytes()` wit
virtualenv	21.7.9	PYSEC-2026-4011	21.7.12	`download_wheel()` runs `pip download` and hands the result straight to the seeder, with nothing checking the bytes it g
virtualenv	21.7.9	PYSEC-2026-4012	21.7.11	`pyvenv.cfg` is a line-based format with no escape syntax. `PyEnvCfg.write()` wrote values verbatim, while `PyEnvCfg._re
virtualenv	21.7.9	PYSEC-2026-4014	21.7.12	`BatchActivator.quote()` returned its input unchanged, the only activator with no escaping at all. `--prompt`, the `VIRT
virtualenv	21.7.9	PYSEC-2026-4013	21.7.13	The generated `activate` (bash/zsh) and `activate.fish` scripts interpolate a `shlex.quote`-ed value into a position tha
virtualenv	21.7.9	PYSEC-2026-4011	21.7.12	`download_wheel()` runs `pip download` and hands the result straight to the seeder, with nothing checking the bytes it g
virtualenv	21.7.9	PYSEC-2026-4012	21.7.11	### Summary `pyvenv.cfg` is a line-based format with no escape syntax. `PyEnvCfg.write()` wrote values verbatim, while
virtualenv	21.7.9	PYSEC-2026-4013	21.7.13	### Impact The generated `activate` (bash/zsh) and `activate.fish` scripts interpolate a `shlex.quote`-ed value into a
virtualenv	21.7.9	PYSEC-2026-4014	21.7.12	`BatchActivator.quote()` returned its input unchanged, the only activator with no escaping at all. `--prompt`, the `VIRT

14 vulnerability/vulnerabilities found (14 fixable) across 3 package(s).

Result: ❌ Blocking issues found — see details above.

…c parameterization

emmanuelmathot · 2026-10-01T21:24:12Z

merged to unblock #292. If some items remain, let's open new issues.

Base automatically changed from revert/281-direct-push to main September 28, 2026 14:25

emmanuelmathot requested review from lhoupert and shollendEODC September 28, 2026 14:26

emmanuelmathot force-pushed the feat/281-scale-offset-codec branch from 211eb8c to ce7ae54 Compare September 28, 2026 14:26

This was referenced Sep 28, 2026

docs: present eopf-geozarr as the GeoZarr driver for EOPF CPM #289

Merged

Extension of functionalities to all other modes #248

Open

During multiscale calculation no data values are not masked #246

Closed

lhoupert approved these changes Sep 29, 2026

View reviewed changes

This was referenced Sep 30, 2026

Integrate/cpm geozarr main feat generic conversion s1slc #292

Merged

One scale/offset encoding switch: CF + STAC by default, Zarr codec as opt-in #281

Closed

emmanuelmathot changed the title ~~feat!: one scale/offset encoding switch, Zarr codec by default~~ feat!: one scale/offset encoding switch Sep 30, 2026

emmanuelmathot mentioned this pull request Sep 30, 2026

GeoZarr source: apply CF scale_factor / add_offset (and _FillValue) per band openlayers/openlayers#17649

Open

claxn requested changes Oct 1, 2026

View reviewed changes

feat: add fill value handling and normalize packed encoding in S2 mul…

652b288

…tiscale processing

feat: enhance test for injecting missing bands with scale/offset code…

d955871

…c parameterization

emmanuelmathot merged commit 5406ac3 into main Oct 1, 2026
7 of 8 checks passed

github-actions Bot mentioned this pull request Sep 29, 2026

chore: release 0.13.0 #288

Merged

shollendEODC mentioned this pull request Oct 2, 2026

Comparison of metadata changes due to codec changes #302

Closed

lhoupert mentioned this pull request Oct 27, 2026

docs: present eopf-geozarr as the GeoZarr driver for EOPF CPM #322

Merged

Navigation Menu

Repository navigation

feat!: one scale/offset encoding switch - #287

feat!: one scale/offset encoding switch#287
emmanuelmathot merged 4 commits into
mainfrom
feat/281-scale-offset-codec

emmanuelmathot commented Sep 28, 2026 •

edited

Loading

Uh oh!

lhoupert left a comment

Uh oh!

emmanuelmathot commented Sep 30, 2026

Uh oh!

shollendEODC commented Sep 30, 2026

Uh oh!

shollendEODC commented Sep 30, 2026

Uh oh!

shollendEODC commented Oct 1, 2026

Uh oh!

claxn left a comment

Uh oh!

claxn Oct 1, 2026

Uh oh!

claxn Oct 1, 2026

Uh oh!

emmanuelmathot commented Oct 1, 2026

Uh oh!

github-actions Bot commented Oct 1, 2026 •

edited

Loading

Uh oh!

Uh oh!

emmanuelmathot commented Oct 1, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

	candidates = (
	var.encoding.get("_FillValue"),
	var.attrs.get("_FillValue"),
	var.attrs.get("fill_value"),
	var.encoding.get("fill_value"),
	)
	fill = next((value for value in candidates if value is not None), None)

Conversation

emmanuelmathot commented Sep 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Breaking changes

Tests

Uh oh!

lhoupert left a comment

Choose a reason for hiding this comment

Uh oh!

emmanuelmathot commented Sep 30, 2026

Uh oh!

shollendEODC commented Sep 30, 2026

Uh oh!

shollendEODC commented Sep 30, 2026

Uh oh!

shollendEODC commented Oct 1, 2026

Uh oh!

claxn left a comment

Choose a reason for hiding this comment

Uh oh!

claxn Oct 1, 2026

Choose a reason for hiding this comment

Uh oh!

claxn Oct 1, 2026

Choose a reason for hiding this comment

Uh oh!

emmanuelmathot commented Oct 1, 2026

Uh oh!

github-actions Bot commented Oct 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Security Audit Report

Bandit — Static Security Analysis (Security tab)

pip-audit — Dependency Vulnerabilities (Security tab)

Uh oh!

Uh oh!

emmanuelmathot commented Oct 1, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

emmanuelmathot commented Sep 28, 2026 •

edited

Loading

github-actions Bot commented Oct 1, 2026 •

edited

Loading