Skip to content

Commit 39b06d6

Browse files
revert: undo #281 work pushed to main by mistake (#286)
776d064 (single scale/offset encoding switch, #281) was pushed straight to main from a branch that tracked origin/main, without review. Revert it so main matches release 0.12.0 again; the work comes back through a normal PR. This reverts commit 776d064.
1 parent bedb88a commit 39b06d6

14 files changed

Lines changed: 2208 additions & 6519 deletions

‎README.md‎

Lines changed: 3 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -292,13 +292,6 @@ layout (`convert-s2-optimized`). The three native resolution groups (10 m, 20 m
292292
60 m) are reused as-is and coarser overviews (120 m, 360 m, 720 m) are computed
293293
via /2 downsampling.
294294

295-
Every level keeps the packing of the source product (for example `uint16` for
296-
reflectance). By default the Zarr `scale_offset` + `cast_value` codecs store it,
297-
so Zarr readers return decoded values. Pass `--no-scale-offset-codec` to write the ESA
298-
layout instead: packed integers with CF `scale_factor`, `add_offset` and
299-
`_FillValue`, and STAC `raster:scale`/`raster:offset`/`nodata` (see
300-
[docs/converter.md](docs/converter.md#encoding)).
301-
302295
### Sentinel-3 OLCI L1 EFR
303296

304297
Sentinel-3 OLCI (Ocean and Land Colour Instrument) Level-1 EFR (Full Resolution)
@@ -331,6 +324,7 @@ Key flags:
331324
- `--min-dimension` — stop generating /2 overview levels once either spatial
332325
dimension would drop below this value (default: 256)
333326
- `--enable-sharding` — accepted but not yet wired into encoding (follow-up task)
327+
- `--keep-scale-offset` — accepted but not yet wired into encoding (follow-up task)
334328
- `--output-grid` — `native` (default) preserves the instrument swath geometry; any other value is parsed as a CRS (e.g. `EPSG:4326`) and the swath is warped once onto a regular grid
335329

336330
#### What is converted
@@ -352,9 +346,8 @@ Key flags:
352346
> **Note:** OLCI support is initial/measurements-focused (v1). Tie-point grid
353347
> groups (`conditions/geometry`, `meteorology`, `instrument`) are copied through but
354348
> not converted to GeoZarr convention. Encoding wiring for `--enable-sharding`,
355-
> `--spatial-chunk` and `--compression-level` is accepted but scheduled as a
356-
> follow-up task. Radiance is written with the source packing (`uint16` with CF
357-
> `scale_factor` and `_FillValue`).
349+
> `--spatial-chunk`, `--compression-level`, and `--keep-scale-offset` is accepted
350+
> but scheduled as a follow-up task.
358351
359352
## Architecture
360353

‎docs/converter.md‎

Lines changed: 3 additions & 32 deletions
Original file line numberDiff line numberDiff line change
@@ -198,35 +198,6 @@ dt_optimized = convert_s2_optimized(
198198

199199
The result is a space-efficient multiscale pyramid: `/measurements/reflectance/{r10m, r20m, r60m, r120m, r360m, r720m}` where the native resolutions are preserved as-is and only the coarser levels are computed.
200200

201-
### Encoding
202-
203-
Every level of the reflectance pyramid keeps the packing of the source product
204-
(for example `uint16`, `scale_factor` 0.0001, `add_offset` -0.1, nodata 0 for
205-
S2 L2A). One option selects how that packing is stored:
206-
207-
| Mode | Option | On disk | Metadata |
208-
|---|---|---|---|
209-
| Default: Zarr codec | none | Source integer dtype, packed by the Zarr `scale_offset` + `cast_value` codecs; the array's logical dtype is `float32` and its fill value is NaN | No CF scale attributes and no STAC scale fields: Zarr readers return decoded values |
210-
| ESA layout | `--no-scale-offset-codec` / `scale_offset_codec=False` | Source integer dtype, no scale codecs | CF `scale_factor`, `add_offset` and `_FillValue` on the arrays; `raster:scale`, `raster:offset` and `nodata` on the STAC `reflectance` asset |
211-
212-
What a reader needs:
213-
214-
- Default mode: a Zarr library that supports the `scale_offset` and
215-
`cast_value` codecs, for example zarr-python with the `cast-value-rs` extra,
216-
or zarrita (used by the OpenLayers `GeoZarr` source).
217-
- ESA layout: a reader that applies the CF attributes. xarray does this by
218-
default (`mask_and_scale=True`); zarrita does not.
219-
220-
The input can be opened raw (`mask_and_scale=False`, as in the CPM path) or
221-
decoded (as in the CLI). Both give the same output. Arrays without a packing,
222-
such as the classification and quality masks, keep their integer dtype and
223-
fill value in both modes.
224-
225-
In the CPM writer (`eopf convert-geozarr --no-scale-offset-codec` or
226-
`target_store_kwargs={"scale_offset_codec": False}`), the former
227-
`keep_scale_offset` option is still accepted for one release with a
228-
`DeprecationWarning`: `keep_scale_offset=True` selects the ESA layout.
229-
230201
## Sentinel-3 OLCI L1 EFR Conversion
231202

232203
Sentinel-3 OLCI (Ocean and Land Colour Instrument) Level-1 EFR (Full Resolution)
@@ -262,6 +233,7 @@ eopf-geozarr convert-s3-olci-optimized S3A_OL_1_EFR.zarr output.zarr \
262233
| `--compression-level` | 3 | Blosc/zstd compression level (1–9) |
263234
| `--min-dimension` | 256 | Minimum spatial dimension for overview levels |
264235
| `--enable-sharding` | off | Accepted but not yet wired into encoding (follow-up task) |
236+
| `--keep-scale-offset` | off | Accepted but not yet wired into encoding (follow-up task) |
265237
| `--output-grid` | native | `native` preserves instrument geometry; any CRS string (e.g. `EPSG:4326`) warps onto a regular grid |
266238

267239
### Output layout
@@ -291,9 +263,8 @@ a regular grid and is dropped (it remains in the source product).
291263
> **Note:** OLCI support is initial/measurements-focused (v1). Tie-point grid
292264
> groups in `conditions/geometry`, `meteorology`, and `instrument` are copied
293265
> through but not converted to GeoZarr convention. Encoding wiring for
294-
> `--enable-sharding`, `--spatial-chunk` and `--compression-level` is accepted
295-
> but scheduled as a follow-up task. Radiance is written with the source
296-
> packing (`uint16` with CF `scale_factor` and `_FillValue`).
266+
> `--enable-sharding`, `--spatial-chunk`, `--compression-level`, and
267+
> `--keep-scale-offset` is accepted but scheduled as a follow-up task.
297268
298269
## Error Handling
299270

‎src/eopf_geozarr/cli.py‎

Lines changed: 23 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -229,6 +229,7 @@ def convert_command(args: argparse.Namespace) -> None:
229229
spatial_chunk=args.spatial_chunk,
230230
compression_level=3,
231231
validate_output=True,
232+
keep_scale_offset=False,
232233
max_retries=args.max_retries,
233234
)
234235
# Opt-out flag first: skip the structural OLCI detection (a full
@@ -1179,7 +1180,7 @@ def create_parser() -> argparse.ArgumentParser:
11791180
description=(
11801181
"Convert EOPF dataset to GeoZarr compliant format. Sentinel-2 inputs are "
11811182
"auto-detected and converted with the optimized flat multiscale layout "
1182-
"(equivalent to convert-s2-optimized with default options); for "
1183+
"(equivalent to convert-s2-optimized with keep_scale_offset disabled); for "
11831184
"those inputs the per-group options --groups, --crs-groups, --gcp-group and "
11841185
"--min-dimension do not apply. Sentinel-3 OLCI inputs are likewise "
11851186
"auto-detected and converted with the OLCI swath converter (equivalent to "
@@ -1412,14 +1413,17 @@ def add_s2_optimization_commands(subparsers: argparse._SubParsersAction) -> None
14121413
s2_parser.add_argument("--skip-validation", action="store_true", help="Skip output validation")
14131414
s2_parser.add_argument("--verbose", action="store_true", help="Enable verbose output")
14141415
s2_parser.add_argument(
1415-
"--no-scale-offset-codec",
1416-
dest="scale_offset_codec",
1417-
action="store_false",
1418-
help=(
1419-
"Write packed reflectance as in the ESA product: source integers with CF "
1420-
"scale_factor/add_offset/_FillValue and STAC raster:scale/raster:offset/nodata. "
1421-
"By default the Zarr scale_offset + cast_value codecs pack it."
1422-
),
1416+
"--keep-scale-offset",
1417+
action="store_true",
1418+
help="""
1419+
Preserve scale-offset encoding. Default is False, in which case arrays stored with
1420+
scale-offset encoding will be re-saved as the decoded data type, i.e. floating point values.
1421+
""",
1422+
)
1423+
s2_parser.add_argument(
1424+
"--experimental-scale-offset-codec",
1425+
action="store_true",
1426+
help="Push CF scale-offset encoding into zarr codec pipeline instead of decoding to float.",
14231427
)
14241428
s2_parser.add_argument(
14251429
"--dask-cluster",
@@ -1449,7 +1453,7 @@ def convert_s2_optimized_command(args: argparse.Namespace) -> None:
14491453
spatial_chunk=args.spatial_chunk,
14501454
compression_level=args.compression_level,
14511455
validate_output=not args.skip_validation,
1452-
scale_offset_codec=args.scale_offset_codec,
1456+
keep_scale_offset=args.keep_scale_offset,
14531457
)
14541458

14551459
log.info("S2 optimization completed", output_path=args.output_path)
@@ -1498,6 +1502,14 @@ def add_s3_olci_optimization_commands(subparsers: argparse._SubParsersAction) ->
14981502
default=256,
14991503
help="Minimum overview dimension (default: 256)",
15001504
)
1505+
p.add_argument(
1506+
"--keep-scale-offset",
1507+
action="store_true",
1508+
help=(
1509+
"Preserve scale-offset encoding instead of decoding to float "
1510+
"(not yet applied; output is currently always raw integer)"
1511+
),
1512+
)
15011513
p.add_argument(
15021514
"--output-grid",
15031515
type=str,
@@ -1522,6 +1534,7 @@ def convert_s3_olci_optimized_command(args: argparse.Namespace) -> None:
15221534
spatial_chunk=args.spatial_chunk,
15231535
compression_level=args.compression_level,
15241536
min_dimension=args.min_dimension,
1537+
keep_scale_offset=args.keep_scale_offset,
15251538
output_grid=args.output_grid,
15261539
)
15271540
log.info("S3 OLCI optimization completed", output_path=args.output_path)

‎src/eopf_geozarr/cpm/writer.py‎

Lines changed: 5 additions & 32 deletions
Original file line numberDiff line numberDiff line change
@@ -36,7 +36,6 @@
3636
from __future__ import annotations
3737

3838
import shutil
39-
import warnings
4039
from pathlib import Path
4140
from typing import TYPE_CHECKING, Any
4241

@@ -111,8 +110,7 @@ def write(
111110
gcp_group: str | None = None,
112111
min_dimension: int = 256,
113112
compression_level: int = 3,
114-
scale_offset_codec: bool = True,
115-
keep_scale_offset: bool | None = None,
113+
keep_scale_offset: bool = False,
116114
validate_output: bool = False,
117115
output_grid: str = "native",
118116
**kwargs: Any,
@@ -169,15 +167,9 @@ def write(
169167
level.
170168
compression_level
171169
S2 optimized pipeline only: blosc-zstd compression level.
172-
scale_offset_codec
173-
S2 optimized pipeline only: pack reflectance with the Zarr
174-
``scale_offset`` + ``cast_value`` codecs (default). ``False``
175-
writes it as in the ESA product: source integers with CF
176-
``scale_factor``/``add_offset``/``_FillValue``, and STAC
177-
``raster:scale``/``raster:offset``/``nodata``.
178170
keep_scale_offset
179-
Deprecated, use ``scale_offset_codec``: ``keep_scale_offset=True``
180-
is ``scale_offset_codec=False`` (ESA layout).
171+
S2 optimized pipeline only: preserve CF scale/offset encoding
172+
instead of decoding to float.
181173
validate_output
182174
S2 optimized pipeline only: run output validation after writing.
183175
output_grid
@@ -202,14 +194,6 @@ def write(
202194
compute=compute,
203195
**kwargs,
204196
)
205-
if keep_scale_offset is not None:
206-
warnings.warn(
207-
"keep_scale_offset is deprecated and will be removed in the next release; "
208-
"use scale_offset_codec instead (keep_scale_offset=True is scale_offset_codec=False).",
209-
DeprecationWarning,
210-
stacklevel=2,
211-
)
212-
scale_offset_codec = scale_offset_codec and not keep_scale_offset
213197
target_str = self._check_target(filename_or_obj)
214198
if mode == "w-" and Path(target_str).exists():
215199
raise EOStoreProductAlreadyExistsError(f"Product already exists in {target_str}")
@@ -252,7 +236,7 @@ def write(
252236
spatial_chunk=resolved_spatial_chunk,
253237
compression_level=compression_level,
254238
validate_output=validate_output,
255-
scale_offset_codec=scale_offset_codec,
239+
keep_scale_offset=keep_scale_offset,
256240
max_retries=max_retries,
257241
)
258242

@@ -264,6 +248,7 @@ def write(
264248
spatial_chunk=resolved_spatial_chunk,
265249
compression_level=compression_level,
266250
min_dimension=min_dimension,
251+
keep_scale_offset=keep_scale_offset,
267252
output_grid=output_grid,
268253
)
269254

@@ -302,7 +287,6 @@ def validate_write_options(
302287
"gcp_group",
303288
"min_dimension",
304289
"compression_level",
305-
"scale_offset_codec",
306290
"keep_scale_offset",
307291
"validate_output",
308292
"output_grid",
@@ -496,14 +480,6 @@ def get_cli_command() -> click.Command:
496480
"geometry; any other value is a CRS string (e.g. 'EPSG:4326') to warp onto."
497481
),
498482
)
499-
@click.option(
500-
"--no-scale-offset-codec",
501-
is_flag=True,
502-
help=(
503-
"S2 optimized pipeline: write packed reflectance as in the ESA product (CF "
504-
"scale_factor/add_offset/_FillValue) instead of the default Zarr scale-offset codecs."
505-
),
506-
)
507483
@click.option(
508484
"--stage-source",
509485
is_flag=True,
@@ -526,7 +502,6 @@ def convert_geozarr(
526502
no_s2_optimized: bool,
527503
no_s3_olci_optimized: bool,
528504
output_grid: str,
529-
no_scale_offset_codec: bool,
530505
stage_source: bool,
531506
stage_output: bool,
532507
) -> None:
@@ -550,8 +525,6 @@ def convert_geozarr(
550525
target_store_kwargs["s3_olci_optimized"] = False
551526
if output_grid != "native":
552527
target_store_kwargs["output_grid"] = output_grid
553-
if no_scale_offset_codec:
554-
target_store_kwargs["scale_offset_codec"] = False
555528
convert(
556529
source_path=source_path,
557530
target_path=target_path,

‎src/eopf_geozarr/s2_optimization/s2_converter.py‎

Lines changed: 6 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,7 @@
1919
from eopf_geozarr.data_api.s1 import Sentinel1Root
2020
from eopf_geozarr.data_api.s2 import Sentinel2Root
2121

22-
from .s2_multiscale import create_multiscale_from_datatree, packing_of
22+
from .s2_multiscale import create_multiscale_from_datatree
2323

2424
if TYPE_CHECKING:
2525
from collections.abc import Hashable, Mapping
@@ -174,6 +174,7 @@ def convert_s2(
174174
output_group=zarr.open_group(output_path),
175175
spatial_chunk=spatial_chunk,
176176
enable_sharding=enable_sharding,
177+
keep_scale_offset=False,
177178
)
178179

179180
log.info("Created multiscale pyramids", num_groups=len(datasets))
@@ -215,7 +216,7 @@ def convert_s2_optimized(
215216
spatial_chunk: int,
216217
compression_level: int,
217218
validate_output: bool,
218-
scale_offset_codec: bool = True,
219+
keep_scale_offset: bool,
219220
max_retries: int = 3,
220221
) -> xr.DataTree:
221222
"""
@@ -228,10 +229,7 @@ def convert_s2_optimized(
228229
spatial_chunk: Spatial chunk size
229230
compression_level: Compression level 1-9
230231
validate_output: Whether to validate the output
231-
scale_offset_codec: Pack reflectance with the Zarr `scale_offset` +
232-
`cast_value` codecs (default). False writes it as in the ESA
233-
product: packed integers with CF `scale_factor` / `add_offset` /
234-
`_FillValue` and STAC `raster:scale` / `raster:offset` / `nodata`.
232+
keep_scale_offset: Whether to preserve scale-offset encoding of the source data.
235233
max_retries: Maximum number of retries for network operations
236234
237235
Returns:
@@ -264,16 +262,14 @@ def convert_s2_optimized(
264262
spatial_chunk=spatial_chunk,
265263
enable_sharding=enable_sharding,
266264
crs=crs,
267-
scale_offset_codec=scale_offset_codec,
265+
keep_scale_offset=keep_scale_offset,
268266
)
269267

270268
log.info("Created multiscale pyramids", num_groups=len(datasets))
271269

272270
# Step 3: Root-level consolidation
273271
log.info("Step 3: Final root-level metadata consolidation")
274-
simple_root_consolidation(
275-
output_path, datasets, dt_input, crs=crs, scale_offset_codec=scale_offset_codec
276-
)
272+
simple_root_consolidation(output_path, datasets, dt_input, crs=crs)
277273

278274
# Step 4: Validation
279275
if validate_output:
@@ -298,7 +294,6 @@ def simple_root_consolidation(
298294
datasets: Mapping[str, object],
299295
dt_input: xr.DataTree | None = None,
300296
crs: CRS | None = None,
301-
scale_offset_codec: bool = True,
302297
) -> None:
303298
"""Simple root-level metadata consolidation with proper zarr group creation."""
304299
# create missing intermediary groups (/conditions, /quality, etc.)
@@ -383,14 +378,6 @@ def simple_root_consolidation(
383378
base = datasets.get("/measurements/reflectance/r10m")
384379
if isinstance(base, xr.Dataset):
385380
reflectance_asset["proj:shape"] = [base.sizes["y"], base.sizes["x"]]
386-
# In codec mode the Zarr reader already returns decoded values, so the
387-
# asset must not declare a scale or clients would apply it twice.
388-
packing = next(filter(None, map(packing_of, base.data_vars.values())), None)
389-
if not scale_offset_codec and packing is not None:
390-
reflectance_asset["raster:scale"] = packing.scale_factor
391-
reflectance_asset["raster:offset"] = packing.add_offset
392-
if packing.fill_value is not None:
393-
reflectance_asset["nodata"] = packing.fill_value
394381

395382
stac.setdefault("assets", {})["reflectance"] = reflectance_asset
396383

0 commit comments

Comments
 (0)