Skip to content

Commit f749368

Browse files
lhoupertclaudegithub-actions[bot]d-v-bemmanuelmathot
authored
feat(stac): add build_s1_rtc_stac_item and generate-stac-s1 CLI (#173)
* feat(stac): add build_s1_rtc_stac_item and generate-stac-s1 CLI command Adds eopf_geozarr.stac.s1_rtc.build_s1_rtc_stac_item, which reads a consolidated S1 GRD RTC Zarr V3 store and returns a pystac.Item with SAR/SAT/projection extensions, WGS84 bbox derived via pyproj, and vv/vh asset sub-paths. Ascending orbit is preferred when both are present. Also adds the generate-stac-s1 CLI subcommand that prints the item as JSON. 8 unit tests cover all acceptance criteria from the plan. Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * fix: support s3:// URIs in S1Tiling discovery and ingest (#175) Four functions in s1_ingest.py silently failed for s3:// paths: - discover_s1tiling_acquisitions / discover_s1tiling_conditions used pathlib.Path.glob(), which normalises s3:// to s3:/ and returns 0 matches - ingest_s1tiling_acquisition / ingest_s1tiling_conditions coerced paths via Path(), corrupting s3:// URIs, and called .exists() which always returns False for S3 (existence checks now use path_exists() from fs_utils) Adds three private helpers: - _list_tifs(): uses s3fs.S3FileSystem.glob for s3:// prefixes - _coerce_input_path(): preserves str for s3://, returns Path otherwise - _rasterio_env(): rasterio.Env(AWSSession) context for s3:// paths; strips https:// from AWS_ENDPOINT_URL since GDAL expects hostname only Tested: rasterio.open("s3://...") confirmed working on OVH S3 with AWS_S3_ENDPOINT or AWS_ENDPOINT_URL set; hostname-only required by GDAL. Refs #139 Co-authored-by: Claude Sonnet 4.6 <[email protected]> * fix(stac): point vv/vh asset hrefs to orbit group root; add tile_matrix_set Per Emmanuel's review on #173: - vv/vh asset hrefs now point to {store}/{orbit} (the zarr group that carries multiscales), matching how S2 reflectance assets point to measurements/reflectance. TiTiler reads tile_matrix_set from that group's multiscales attributes. - create_s1_store now writes tile_matrix_set into the orbit group's multiscales so TiTiler can discover the tiling scheme without error. - Updated test_asset_hrefs to assert the new orbit-group href. Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * fix(ingest): add proj:code to resolution group attrs for TiTiler compatibility TiTiler's _validate_zarr checks ds.rio.crs on the native resolution group (r10m). Without proj:code in the resolution group's attrs, rioxarray returns None and the group is excluded → groups=[] → bounds unpack error. S2 zarr already has proj:code at the resolution group level; mirror that for S1. Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * fix(s1): write CF grid_mapping, drop tile_matrix_set from RTC store (#176) The deployed titiler-eopf reader resolves a group's CRS via rioxarray, which reads a CF spatial_ref/crs_wkt coordinate -- not the GeoZarr proj:code attr. S1 GRD RTC stores carried only proj:code, so rio.crs was None and every multiscale group was rejected (HTTP 500 on render/info). The S2 converter already writes a spatial_ref grid-mapping; this brings S1 in line. Also stop writing tile_matrix_set into the orbit-group multiscales: the data-model owner confirmed it is not part of the S1 GRD RTC data model, and the data_api reader already treats it as optional (MISSING default). - add _add_grid_mapping() using pyproj CRS.to_cf() (the same source rioxarray uses, so no hard-coded projection) and call it for every resolution level (both store-creation paths) and the conditions group - remove _create_tile_matrix_set() and its multiscales entry - tests: assert no tile_matrix_set, spatial_ref present + rio.crs resolves to the native EPSG, conditions group carries grid_mapping This fixes at the source what data-pipeline currently works around in ingest_v1_s1_rtc.py (_patch_cf_grid_mapping / _patch_tile_matrix_limits). Co-authored-by: Claude Opus 4.8 <[email protected]> * Merge #172 (deprecate v0, fix fill values, sanitize attrs) into s1-rtc (#178) * chore: release 0.9.0 * feat: implement scale-offset and data type casting via codecs * fix: fix dependency declaration * chore: use latest version of cast value * chore: make cast-value a project dependency * test: expand test coverage * fix: upgrade pytest to 9.0.3 (CVE-2025-71176) Fixes insecure /tmp directory handling on UNIX systems. Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * - Use zarr-python's presumptive implementation of scale-offset - use scalar map for handling NaN - ensure that downsampled arrays use scalar map + cast value - improve tests across parametrization of relelvant functions * feat: add store-root spatial:bbox and tighten minispec requirements (#164) * feat: add store-root spatial:bbox and tighten minispec requirements Introduces a GeoZarr "Store Root" layer in the minispec so clients can read a summary footprint without walking into child groups, and tightens the multiscale profile so `spatial:bbox` at the root and `spatial:transform` + `spatial:shape` on every layout entry are mandatory. Adds a new `geozarr.store` pydantic module (`GeoZarrStoreAttrs`, `GeoZarrMultiscaleGroupAttrs`, `GeoZarr`) enforcing the tightened profile, and updates the S2 converter to union child-group bboxes into an EPSG:4326 footprint written at the store root. Closes #156. Addresses the clear parts of #163; the array-level and non-multiscale-group parts of that issue need further clarification. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * feat: update GeoZarr store model and optimize S2 converter for improved bbox handling * Addresses review feedback on #164: - Adds a "GeoZarr Hierarchy & Identification" subsection to the Store Root section of the minispec, codifying d-v-b's proposed rules: single root, root prefix ends with `.zarr`, suffix occurs at most once in the hierarchy, and explicit terminal-path conditions. - Promotes the store-root CRS to mandatory: at least one of `proj:code`, `proj:wkt2`, `proj:projjson` MUST be set; there is no implicit EPSG:4326 default. Per @vincentsarago's review. - Promotes `zarr_conventions` declaration at the store root from RECOMMENDED to required. - Converter now writes `proj:code: "EPSG:4326"` at the root alongside `spatial:bbox`. - Pydantic `GeoZarrStoreAttrs` enforces the new CRS-required rule. Cross-links the new hierarchy + spatial:extent follow-ups upstream: zarr-developers/geozarr-spec#132 (hierarchy & root identification, also addresses #124 URL parsing) and #133 (STAC-style spatial:extent). --------- Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]> * chore: uv.lock * refactor: use zarr-python 3.2.0 * chore: lockfile * fix: pin fill value based on data type instead of relying on xarray * test: update tests to consistently use nan fill value for floats * chore: bump urllib3 to 2.7.0 * chore: skip quicklook groups (#165) * chore: group dependabot updates for actions and pip (#160) * chore: group dependabot updates for actions and pip - Group all GitHub Actions bumps (minor, patch, major) into one PR - Group minor and patch pip bumps into one PR; major bumps remain ungrouped for individual review Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * chore: change dependabot schedule day to wednesday * ci: add comment_on option to security auditing action * ci: switch dependabot Python ecosystem from pip to uv uv ecosystem reads uv.lock directly, enabling Dependabot to raise PRs for lockfile-pinned versions including security patches. Co-Authored-By: Claude Sonnet 4.6 <[email protected]> --------- Co-authored-by: Claude Sonnet 4.6 <[email protected]> * fix: derive coarse spatial transforms from coordinates (#168) * fix: derive coarse spatial transforms from coordinates * refactor: improve function definitions for clarity and consistency * chore: release 0.10.0 (#162) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * fix(converter): align convert layout to S2, strip _eopf_attrs, fix _FillValue (#171) - Remove TMS layout and /0 /1 /2 numeric overview groups from the general 'convert' CLI; produce S2-style sibling r{2**level} overviews instead. 'convert-s2-optimized' continues to emit the same layout. - Drop the '--tile-width' CLI option. - Sanitize array attrs in all converters: strip stale '_eopf_attrs' and source-only '_FillValue' / 'dtype' / 'valid_min' / 'valid_max' on decoded floats; rewrite 'units: digital_counts' -> 'units: 1'. The sanitizer now returns a new dict and callers reassign 'attrs', so stale keys are actually removed (a previous '.update()' pattern left them in place). - Sanitize coord attrs too (datetime coords in /conditions/meteorology were leaking '_eopf_decode_datetime64' via '_eopf_attrs'). - Set '_FillValue' properly for float measurements so xarray's CF encoder produces the base64-NaN representation needed for 'use_zarr_fill_value_as_mask=True' round-trip (xarray#11345). Fixes #171. - Remove dead 'utils.encode_cf_fill_value' (no callers). - Tests: open each r{N} multiscale level separately; migrate 'test_multiscales_round_trip' from deprecated tms.Multiscales to zcm.Multiscales (with tuplify_json to handle extra='allow' fields). - Regenerate snapshot fixtures; verified 0 '_eopf_attrs', 'tile_matrix*', or 'digital_counts' markers across S2A/S2B/S2C. * test: add guardrails for converter output attrs (#171) Walks the snapshotted GroupSpec JSON fixtures and asserts: - no '_eopf_attrs' anywhere; - no 'tile_matrix*' markers (TMS removal); - no 'units: digital_counts' on float arrays (decoded scale/offset); - every float array under '/measurements/' has '_FillValue' set (required for CF NaN-mask round-trip, xarray#11345). * docs: deprecate v0 references, document r{N} overview layout (#171) - Drop 'tile_width' parameter from API examples (option removed from CLI/API). - Replace '/measurements/r10m/{0,1,2}' nested-pyramid examples with the current flat 'r{2**level}' sibling-group layout. - Remove the V0 vs V1 split in converter.md / examples.md / quickstart.md; the general 'convert' command now produces the same flat layout as 'convert-s2-optimized'. The S2-optimized section is reframed as a feature description rather than a 'V1 vs V0' comparison. - Update architecture.md multiscales snippet to the new model ('layout' / 'asset' / 'derived_from' / 'transform'). - Update faq.md inspection snippet to walk the new layout. * refactor: clean up code formatting and enhance attribute handling in geozarr conversion * fix(cli-e2e): drop deprecated --tile-width flag from tests and example * refactor: remove deprecated v0 layout, sanitize attributes, and fix _FillValue handling * test: add _FillValue masking roundtrip test (#172) Addresses @vincentsarago's PR review: assert float arrays written with the converter's _FillValue convention round-trip through xarray's use_zarr_fill_value_as_mask=True so NaN cells come back masked. * test: drive _FillValue masking test through create_geozarr_dataset (#172) Replace the standalone xarray write with a real converter run: build a small float DataTree, invoke create_geozarr_dataset, then reopen the output with use_zarr_fill_value_as_mask=True and assert masking on the nodata patch. Exercises the converter's _FillValue + encoding path end-to-end, per @vincentsarago's review. * fix: green CI for #172 — S2 convert flat layout + idna CVE (#177) * fix(security): bump idna 3.13 -> 3.17 to clear CVE-2026-45409 pip-audit in the Security Audit workflow blocks on idna 3.13 (CVE-2026-45409, incomplete fix of CVE-2024-3651; fixed in 3.15). Bump the transitive pin via uv lock. pip-audit now reports no known vulnerabilities. Co-Authored-By: Claude Opus 4.8 <[email protected]> * fix(cli): route Sentinel-2 convert through the optimized flat layout Generic `convert` emitted GeoZarr-0.4 multiscales with native data at the group root and overviews as nested r{2**N} sub-groups. That tree cannot be opened by xr.open_datatree (an overview child's x/y conflict with the parent's inherited x/y), so the `info` and `validate` CLI commands crashed on the converter's own Sentinel-2 output. Detect Sentinel-2 inputs and delegate to the existing, tested convert_s2_optimized path, which emits flat sibling r{N}m levels (the layout PR #172 documents). Non-S2 inputs keep the generic create_geozarr_dataset path. Detection is best-effort and never aborts conversion. Regenerate the geozarr_examples structure snapshots to the flat layout. Co-Authored-By: Claude Opus 4.8 <[email protected]> * test: make CLI e2e assertions debuggable and fix multiscale round-trip - The CLI logs errors to stdout via structlog; assert on stdout+stderr in test_cli_e2e so a non-zero exit shows the actual error instead of an empty stderr. - test_multiscale_attrs_round_trip tuplified only one side of its equality, so flat-layout level groups carrying list-valued attrs (e.g. spatial:bbox) broke a check that is only about JSON list/tuple normalisation. Tuplify both sides; the model still round-trips losslessly. Co-Authored-By: Claude Opus 4.8 <[email protected]> --------- Co-authored-by: Claude Opus 4.8 <[email protected]> --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Davis Vann Bennett <[email protected]> Co-authored-by: Claude Sonnet 4.6 <[email protected]> Co-authored-by: Emmanuel Mathot <[email protected]> --------- Co-authored-by: Claude Sonnet 4.6 <[email protected]> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Davis Vann Bennett <[email protected]> Co-authored-by: Emmanuel Mathot <[email protected]>
1 parent 1e891e5 commit f749368

52 files changed

Lines changed: 26821 additions & 51123 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/dependabot.yml‎

Lines changed: 18 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,11 @@
11
version: 2
22
updates:
33
# Enable version updates for Python dependencies
4-
- package-ecosystem: "pip"
4+
- package-ecosystem: "uv"
55
directory: "/"
66
schedule:
77
interval: "weekly"
8-
day: "monday"
8+
day: "wednesday"
99
time: "09:00"
1010
open-pull-requests-limit: 10
1111
reviewers:
@@ -17,13 +17,20 @@ updates:
1717
include: "scope"
1818
cooldown:
1919
default-days: 7
20+
groups:
21+
uv-minor-patch:
22+
update-types:
23+
- "minor"
24+
- "patch"
25+
patterns:
26+
- "*"
2027

2128
# Enable version updates for GitHub Actions
2229
- package-ecosystem: "github-actions"
2330
directory: "/"
2431
schedule:
2532
interval: "weekly"
26-
day: "monday"
33+
day: "wednesday"
2734
time: "09:00"
2835
open-pull-requests-limit: 5
2936
reviewers:
@@ -35,3 +42,11 @@ updates:
3542
include: "scope"
3643
cooldown:
3744
default-days: 7
45+
groups:
46+
actions:
47+
update-types:
48+
- "minor"
49+
- "patch"
50+
- "major"
51+
patterns:
52+
- "*"

‎.github/workflows/security.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,5 +22,6 @@ jobs:
2222
- uses: developmentseed/action-python-security-auditing@8ebea22ea75dfba2244ed9883c2aa6cb4df8d9a9 # v0.6.0
2323
with:
2424
package_manager: uv
25+
comment_on: blocking
2526
bandit_scan_dirs: 'src/eopf_geozarr/'
2627
artifact_name: security-reports

‎.release-please-manifest.json‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,3 @@
11
{
2-
".": "0.8.0"
2+
".": "0.10.0"
33
}

‎CHANGELOG.md‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,40 @@ All notable changes to this project will be documented in this file.
55
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
66
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
77

8+
## 0.10.0 (2026-05-12)
9+
10+
## What's Changed
11+
* fix: upgrade pytest to 9.0.3 (CVE-2025-71176) by @lhoupert in https://github.com/EOPF-Explorer/data-model/pull/161
12+
* feat: add store-root spatial:bbox and tighten minispec requirements by @emmanuelmathot in https://github.com/EOPF-Explorer/data-model/pull/164
13+
* feat: implement scale-offset and data type casting via codecs by @d-v-b in https://github.com/EOPF-Explorer/data-model/pull/154
14+
* chore: bump urllib3 to 2.7.0 by @d-v-b in https://github.com/EOPF-Explorer/data-model/pull/166
15+
* chore: skip quicklook groups by @d-v-b in https://github.com/EOPF-Explorer/data-model/pull/165
16+
* chore: group dependabot updates for actions and pip by @lhoupert in https://github.com/EOPF-Explorer/data-model/pull/160
17+
* fix: derive coarse spatial transforms from coordinates by @emmanuelmathot in https://github.com/EOPF-Explorer/data-model/pull/168
18+
19+
20+
**Full Changelog**: https://github.com/EOPF-Explorer/data-model/compare/v0.9.0...v0.10.0
21+
22+
## 0.9.0 (2026-04-02)
23+
24+
## What's Changed
25+
* Add site_url to the mkdocs config by @maxrjones in https://github.com/EOPF-Explorer/data-model/pull/121
26+
* Update GeoZarr mini-spec to EOPF V1 by @emmanuelmathot in https://github.com/EOPF-Explorer/data-model/pull/129
27+
* use zarr-cm for defining zarr conventions metadata by @d-v-b in https://github.com/EOPF-Explorer/data-model/pull/131
28+
* chore/fill value consistency by @d-v-b in https://github.com/EOPF-Explorer/data-model/pull/135
29+
* Set the minimum supported python version to 3.12 by @d-v-b in https://github.com/EOPF-Explorer/data-model/pull/141
30+
* add titiler integration test by @d-v-b in https://github.com/EOPF-Explorer/data-model/pull/137
31+
* Pin GitHub Actions to commit SHAs (coordination#239) by @lhoupert in https://github.com/EOPF-Explorer/data-model/pull/143
32+
* ci: add permissions block and make security checks blocking by @lhoupert in https://github.com/EOPF-Explorer/data-model/pull/145
33+
* enh: clarify CF metadata scope in GeoZarr mini-spec and update attribute definitions by @emmanuelmathot in https://github.com/EOPF-Explorer/data-model/pull/133
34+
* feat: include b08 in resolution groups by @d-v-b in https://github.com/EOPF-Explorer/data-model/pull/152
35+
36+
## New Contributors
37+
* @maxrjones made their first contribution in https://github.com/EOPF-Explorer/data-model/pull/121
38+
* @lhoupert made their first contribution in https://github.com/EOPF-Explorer/data-model/pull/143
39+
40+
**Full Changelog**: https://github.com/EOPF-Explorer/data-model/compare/v0.8.0...v0.9.0
41+
842
## 0.8.0 (2026-01-21)
943

1044
## What's Changed

‎docs/api-reference.md‎

Lines changed: 11 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,6 @@ def create_geozarr_dataset(
1616
output_path: str,
1717
spatial_chunk: int = 4096,
1818
min_dimension: int = 256,
19-
tile_width: int = 256,
2019
max_retries: int = 3,
2120
**storage_kwargs
2221
) -> xr.DataTree
@@ -29,7 +28,6 @@ def create_geozarr_dataset(
2928
- `output_path` (str): Output path for the GeoZarr dataset (local or S3)
3029
- `spatial_chunk` (int, optional): Target spatial chunk size. Default: 4096
3130
- `min_dimension` (int, optional): Minimum dimension size for processing. Default: 256
32-
- `tile_width` (int, optional): Tile width for multiscale levels. Default: 256
3331
- `max_retries` (int, optional): Maximum retry attempts for operations. Default: 3
3432
- `**storage_kwargs`: Additional storage options (S3 credentials, etc.)
3533

@@ -201,14 +199,20 @@ def write_geozarr_group(
201199

202200
### create_geozarr_compliant_multiscales
203201

204-
Creates multiscales metadata compliant with GeoZarr specification.
202+
Creates the overview pyramid (`r{2**level}` sibling groups) and writes the
203+
`multiscales` metadata onto the parent group.
205204

206205
```python
207206
# test: skip
208207
def create_geozarr_compliant_multiscales(
209-
datasets: Dict[str, xr.Dataset],
210-
tile_width: int = 256
211-
) -> List[Dict[str, Any]]
208+
ds: xr.Dataset,
209+
output_path: str,
210+
group_name: str,
211+
min_dimension: int = 256,
212+
spatial_chunk: int = 4096,
213+
ds_gcp: xr.Dataset | None = None,
214+
enable_sharding: bool = False,
215+
) -> dict[str, Any]
212216
```
213217

214218
## Utility Functions
@@ -340,22 +344,7 @@ def _setup_grid_mapping(ds: xr.Dataset, grid_mapping_var_name: str) -> None
340344
def _add_geotransform(ds: xr.Dataset, grid_mapping_var: str) -> None
341345
```
342346

343-
### CRS and Tile Matrix
344-
345-
```python
346-
# test: skip
347-
def create_native_crs_tile_matrix_set(
348-
crs: Any,
349-
transform: Any,
350-
width: int,
351-
height: int,
352-
tile_width: int = 256
353-
) -> Dict[str, Any]
354-
```
355-
356-
Creates a tile matrix set for native CRS (non-Web Mercator).
357-
358-
## Overview Generation
347+
### Overview Generation
359348

360349
### calculate_overview_levels
361350

‎docs/architecture.md‎

Lines changed: 19 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -240,28 +240,29 @@ grid_mapping_attrs = {
240240

241241
### 3. Multiscales Metadata
242242

243-
GeoZarr-compliant multiscales structure:
243+
The converter writes the [multiscales convention](https://github.com/zarr-conventions/multiscales) attributes on the parent group. Each level entry points at a sibling subgroup (`r{2**level}`) carrying the downsampled dataset:
244244

245245
```python
246246
# test: skip
247-
multiscales = [{
248-
'version': '0.4',
249-
'name': group_name,
250-
'type': 'reduce',
251-
'metadata': {
252-
'method': 'mean',
253-
'version': '0.1.0'
254-
},
255-
'datasets': [
256-
{'path': '0', 'pixels_per_tile': tile_width},
257-
{'path': '1', 'pixels_per_tile': tile_width},
258-
{'path': '2', 'pixels_per_tile': tile_width}
247+
multiscales = {
248+
"resampling_method": "mean",
249+
"layout": [
250+
{
251+
"asset": "r10m", # native, group root
252+
"transform": {"scale": [1, 1], "translation": [0, 0]},
253+
},
254+
{
255+
"asset": "r20m", # overview
256+
"derived_from": "r10m",
257+
"transform": {"scale": [2, 2], "translation": [0, 0]},
258+
},
259+
{
260+
"asset": "r40m",
261+
"derived_from": "r10m",
262+
"transform": {"scale": [4, 4], "translation": [0, 0]},
263+
},
259264
],
260-
'coordinateSystem': {
261-
'wkid': crs_epsg,
262-
'wkt': crs.to_wkt()
263-
}
264-
}]
265+
}
265266
```
266267

267268
## Performance Considerations

‎docs/converter.md‎

Lines changed: 19 additions & 48 deletions
Original file line numberDiff line numberDiff line change
@@ -59,7 +59,6 @@ dt_geozarr = create_geozarr_dataset(
5959
output_path="path/to/output/geozarr.zarr",
6060
spatial_chunk=4096,
6161
min_dimension=256,
62-
tile_width=256,
6362
max_retries=3
6463
)
6564
```
@@ -82,7 +81,6 @@ dt_geozarr = create_geozarr_dataset(
8281
output_path="s3://my-bucket/output.zarr",
8382
spatial_chunk=4096,
8483
min_dimension=256,
85-
tile_width=256,
8684
max_retries=3
8785
)
8886
```
@@ -95,55 +93,30 @@ The converter ensures proper chunk alignment to optimize storage and prevent dat
9593

9694
### Multiscale Support
9795

98-
The converter supports multiscale datasets, creating overview levels with /2 downsampling logic. Each level is stored as a sibling group (e.g., `/0`, `/1`, `/2`).
96+
The converter writes the native resolution arrays at the group root and adds
97+
factor-of-two overviews as sibling subgroups named `r{2**level}` (`r2`, `r4`,
98+
`r8`, ...). Each overview is a complete dataset with its own coordinates and
99+
`spatial:` / `proj:` attributes; the parent group's `multiscales` metadata
100+
records each level via `asset`, `derived_from`, and `transform`.
99101

100102
### Native CRS Preservation
101103

102104
The converter maintains the native coordinate reference system (CRS) of the dataset, avoiding reprojection to Web Mercator.
103105

104-
## Sentinel-2 Optimized Conversion (V1)
106+
## Sentinel-2 Optimized Conversion
105107

106-
The Sentinel-2 optimized converter (`convert_s2_optimized`) represents the refined approach (V1) that creates an efficient multiscale pyramid by **reusing the original multi-resolution data** (r10m, r20m, r60m) without duplication, and adding coarser overview levels (r120m, r360m, r720m) for efficient visualization at lower resolutions.
108+
The Sentinel-2 optimized converter (`convert-s2-optimized` / `convert_s2_optimized`)
109+
builds an efficient multiscale pyramid by **reusing the original multi-resolution
110+
data** (r10m, r20m, r60m) without duplication, and adding coarser overview
111+
levels (r120m, r360m, r720m) for visualization at lower resolutions.
107112

108-
### V0 vs V1 Converter: Key Differences
113+
The general `convert` command emits the same flat `r{N}` sibling layout for
114+
any input, but only the S2-optimized command takes advantage of S2's native
115+
multi-resolution structure.
109116

110-
Understanding the structural differences between the old (V0) and new (V1) converter approaches:
117+
### Layout
111118

112-
#### V0 Approach (Deprecated - `create_geozarr_dataset`)
113-
114-
Creates **pyramids within each resolution group**:
115-
116-
```
117-
output.zarr/
118-
└── measurements/
119-
└── reflectance/
120-
├── r10m/
121-
│ ├── 0/ # Native 10m data
122-
│ ├── 1/ # Downsampled to 20m
123-
│ ├── 2/ # Downsampled to 40m
124-
│ ├── 3/ # Downsampled to 80m
125-
│ ├── 4/ # Downsampled to 160m
126-
│ └── 5/ # Downsampled to 320m
127-
├── r20m/
128-
│ ├── 0/ # Native 20m data
129-
│ ├── 1/ # Downsampled to 40m
130-
│ ├── 2/ # Downsampled to 80m
131-
│ ├── 3/ # Downsampled to 160m
132-
│ └── 4/ # Downsampled to 320m
133-
└── r60m/
134-
├── 0/ # Native 60m data
135-
├── 1/ # Downsampled to 120m
136-
└── 2/ # Downsampled to 240m
137-
```
138-
139-
**Issues with V0:**
140-
- Creates redundant data at overlapping resolutions (e.g., r10m/1 ≈ r20m/0)
141-
- Inefficient storage due to duplication
142-
- Complex hierarchy with nested levels within each resolution group
143-
144-
#### V1 Approach (Current - `convert_s2_optimized`)
145-
146-
Creates a **consolidated pyramid** by reusing native resolutions and adding coarser levels:
119+
Both converters produce a flat pyramid where each level is a sibling group:
147120

148121
```
149122
output.zarr/
@@ -169,10 +142,10 @@ The resolution levels are chosen to balance data preservation with storage optim
169142

170143
This approach maintains the integrity of ESA's original multi-resolution data while adding computationally efficient overview levels for performance at coarser scales.
171144

172-
**Benefits of V1:**
173-
- No data duplication - native resolutions are reused directly
174-
- More efficient storage
175-
- Simpler, flatter hierarchy
145+
**Benefits:**
146+
- No data duplication — native resolutions are reused directly
147+
- Efficient storage
148+
- Simple, flat hierarchy
176149
- Natural fit for Sentinel-2's multi-resolution data model
177150

178151
### Key Capabilities
@@ -182,8 +155,6 @@ This approach maintains the integrity of ESA's original multi-resolution data wh
182155
- **Variable-Aware Processing**: Applies appropriate resampling methods for different data types (reflectance, classification, quality masks, probabilities)
183156
- **Efficient Testing**: Improved test infrastructure for faster local development
184157

185-
> **Note:** The V0 converter (`create_geozarr_dataset`) is deprecated and will be removed in future versions. All new projects should use the V1 converter (`convert_s2_optimized`).
186-
187158
### Usage Example
188159

189160
```python

‎docs/examples.md‎

Lines changed: 5 additions & 29 deletions
Original file line numberDiff line numberDiff line change
@@ -76,22 +76,19 @@ for group_name in dt_geozarr.groups:
7676

7777
### Sentinel-2 Band Analysis
7878

79-
> **Note on V0 vs V1:** This example shows both V0 (deprecated) and V1 (current) approaches. See [converter documentation](converter.md#v0-vs-v1-converter-key-differences) for structural differences.
80-
81-
#### V1 Approach (Recommended - `convert_s2_optimized`)
82-
83-
Access bands from the consolidated pyramid structure:
79+
Access bands from the consolidated pyramid structure produced by
80+
`convert_s2_optimized`:
8481

8582
```python
8683
import xarray as xr
8784
import matplotlib.pyplot as plt
8885
from eopf_geozarr.s2_optimization.s2_converter import convert_s2_optimized
8986

90-
# Convert using V1 optimizer (recommended)
87+
# Convert using the S2-optimized converter
9188
dt_input = xr.open_datatree("s2_l2a_input.zarr", engine="zarr")
9289
dt = convert_s2_optimized(
9390
dt_input=dt_input,
94-
output_path="s2_l2a_v1.zarr",
91+
output_path="s2_l2a.zarr",
9592
spatial_chunk=256
9693
)
9794

@@ -130,27 +127,6 @@ plt.tight_layout()
130127
plt.show()
131128
```
132129

133-
#### V0 Approach (Deprecated - `create_geozarr_dataset`)
134-
135-
For reference, the V0 structure with nested pyramid levels:
136-
137-
```python
138-
import xarray as xr
139-
import matplotlib.pyplot as plt
140-
141-
# Open V0 converted GeoZarr dataset (deprecated structure)
142-
dt = xr.open_datatree("s2_l2a_v0.zarr", engine="zarr")
143-
144-
# Access 10m resolution with nested pyramid levels
145-
ds_10m_native = dt["/measurements/r10m/0"].ds # Level 0: native 10m
146-
ds_10m_level1 = dt["/measurements/r10m/1"].ds # Level 1: downsampled to ~20m
147-
ds_10m_level2 = dt["/measurements/r10m/2"].ds # Level 2: downsampled to ~40m
148-
149-
# Note: This creates redundant data since r10m/1 ≈ r20m/0
150-
```
151-
152-
> **Migration Note:** V0 is deprecated. Use V1 (`convert_s2_optimized`) for new projects.
153-
154130
## Cloud Storage Examples
155131

156132
### AWS S3 Integration
@@ -433,7 +409,7 @@ dt_geozarr = create_geozarr_dataset(
433409
)
434410

435411
# Extract metadata for STAC
436-
ds = dt_geozarr["/measurements/r10m/0"].ds
412+
ds = dt_geozarr["/measurements/r10m"].ds
437413
spatial_ref = ds.get('spatial_ref', ds.get('crs', None))
438414

439415
# Create basic STAC item

0 commit comments

Comments
 (0)