Repository navigation
Commit f749368
feat(stac): add build_s1_rtc_stac_item and generate-stac-s1 CLI (#173)
* feat(stac): add build_s1_rtc_stac_item and generate-stac-s1 CLI command
Adds eopf_geozarr.stac.s1_rtc.build_s1_rtc_stac_item, which reads a
consolidated S1 GRD RTC Zarr V3 store and returns a pystac.Item with
SAR/SAT/projection extensions, WGS84 bbox derived via pyproj, and
vv/vh asset sub-paths. Ascending orbit is preferred when both are present.
Also adds the generate-stac-s1 CLI subcommand that prints the item as JSON.
8 unit tests cover all acceptance criteria from the plan.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
* fix: support s3:// URIs in S1Tiling discovery and ingest (#175)
Four functions in s1_ingest.py silently failed for s3:// paths:
- discover_s1tiling_acquisitions / discover_s1tiling_conditions used
pathlib.Path.glob(), which normalises s3:// to s3:/ and returns 0 matches
- ingest_s1tiling_acquisition / ingest_s1tiling_conditions coerced paths via
Path(), corrupting s3:// URIs, and called .exists() which always returns
False for S3 (existence checks now use path_exists() from fs_utils)
Adds three private helpers:
- _list_tifs(): uses s3fs.S3FileSystem.glob for s3:// prefixes
- _coerce_input_path(): preserves str for s3://, returns Path otherwise
- _rasterio_env(): rasterio.Env(AWSSession) context for s3:// paths;
strips https:// from AWS_ENDPOINT_URL since GDAL expects hostname only
Tested: rasterio.open("s3://...") confirmed working on OVH S3 with
AWS_S3_ENDPOINT or AWS_ENDPOINT_URL set; hostname-only required by GDAL.
Refs #139
Co-authored-by: Claude Sonnet 4.6 <[email protected]>
* fix(stac): point vv/vh asset hrefs to orbit group root; add tile_matrix_set
Per Emmanuel's review on #173:
- vv/vh asset hrefs now point to {store}/{orbit} (the zarr group that carries
multiscales), matching how S2 reflectance assets point to measurements/reflectance.
TiTiler reads tile_matrix_set from that group's multiscales attributes.
- create_s1_store now writes tile_matrix_set into the orbit group's multiscales
so TiTiler can discover the tiling scheme without error.
- Updated test_asset_hrefs to assert the new orbit-group href.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
* fix(ingest): add proj:code to resolution group attrs for TiTiler compatibility
TiTiler's _validate_zarr checks ds.rio.crs on the native resolution group
(r10m). Without proj:code in the resolution group's attrs, rioxarray returns
None and the group is excluded → groups=[] → bounds unpack error.
S2 zarr already has proj:code at the resolution group level; mirror that for S1.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
* fix(s1): write CF grid_mapping, drop tile_matrix_set from RTC store (#176)
The deployed titiler-eopf reader resolves a group's CRS via rioxarray,
which reads a CF spatial_ref/crs_wkt coordinate -- not the GeoZarr
proj:code attr. S1 GRD RTC stores carried only proj:code, so rio.crs
was None and every multiscale group was rejected (HTTP 500 on
render/info). The S2 converter already writes a spatial_ref
grid-mapping; this brings S1 in line.
Also stop writing tile_matrix_set into the orbit-group multiscales: the
data-model owner confirmed it is not part of the S1 GRD RTC data model,
and the data_api reader already treats it as optional (MISSING default).
- add _add_grid_mapping() using pyproj CRS.to_cf() (the same source
rioxarray uses, so no hard-coded projection) and call it for every
resolution level (both store-creation paths) and the conditions group
- remove _create_tile_matrix_set() and its multiscales entry
- tests: assert no tile_matrix_set, spatial_ref present + rio.crs
resolves to the native EPSG, conditions group carries grid_mapping
This fixes at the source what data-pipeline currently works around in
ingest_v1_s1_rtc.py (_patch_cf_grid_mapping / _patch_tile_matrix_limits).
Co-authored-by: Claude Opus 4.8 <[email protected]>
* Merge #172 (deprecate v0, fix fill values, sanitize attrs) into s1-rtc (#178)
* chore: release 0.9.0
* feat: implement scale-offset and data type casting via codecs
* fix: fix dependency declaration
* chore: use latest version of cast value
* chore: make cast-value a project dependency
* test: expand test coverage
* fix: upgrade pytest to 9.0.3 (CVE-2025-71176)
Fixes insecure /tmp directory handling on UNIX systems.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
* - Use zarr-python's presumptive implementation of scale-offset
- use scalar map for handling NaN
- ensure that downsampled arrays use scalar map + cast value
- improve tests across parametrization of relelvant functions
* feat: add store-root spatial:bbox and tighten minispec requirements (#164)
* feat: add store-root spatial:bbox and tighten minispec requirements
Introduces a GeoZarr "Store Root" layer in the minispec so clients can read
a summary footprint without walking into child groups, and tightens the
multiscale profile so `spatial:bbox` at the root and `spatial:transform` +
`spatial:shape` on every layout entry are mandatory.
Adds a new `geozarr.store` pydantic module (`GeoZarrStoreAttrs`,
`GeoZarrMultiscaleGroupAttrs`, `GeoZarr`) enforcing the tightened profile,
and updates the S2 converter to union child-group bboxes into an EPSG:4326
footprint written at the store root.
Closes #156. Addresses the clear parts of #163; the array-level and
non-multiscale-group parts of that issue need further clarification.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* feat: update GeoZarr store model and optimize S2 converter for improved bbox handling
* Addresses review feedback on #164:
- Adds a "GeoZarr Hierarchy & Identification" subsection to the Store Root
section of the minispec, codifying d-v-b's proposed rules: single root,
root prefix ends with `.zarr`, suffix occurs at most once in the
hierarchy, and explicit terminal-path conditions.
- Promotes the store-root CRS to mandatory: at least one of `proj:code`,
`proj:wkt2`, `proj:projjson` MUST be set; there is no implicit EPSG:4326
default. Per @vincentsarago's review.
- Promotes `zarr_conventions` declaration at the store root from
RECOMMENDED to required.
- Converter now writes `proj:code: "EPSG:4326"` at the root alongside
`spatial:bbox`.
- Pydantic `GeoZarrStoreAttrs` enforces the new CRS-required rule.
Cross-links the new hierarchy + spatial:extent follow-ups upstream:
zarr-developers/geozarr-spec#132 (hierarchy & root identification, also
addresses #124 URL parsing) and #133 (STAC-style spatial:extent).
---------
Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>
* chore: uv.lock
* refactor: use zarr-python 3.2.0
* chore: lockfile
* fix: pin fill value based on data type instead of relying on xarray
* test: update tests to consistently use nan fill value for floats
* chore: bump urllib3 to 2.7.0
* chore: skip quicklook groups (#165)
* chore: group dependabot updates for actions and pip (#160)
* chore: group dependabot updates for actions and pip
- Group all GitHub Actions bumps (minor, patch, major) into one PR
- Group minor and patch pip bumps into one PR; major bumps remain
ungrouped for individual review
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
* chore: change dependabot schedule day to wednesday
* ci: add comment_on option to security auditing action
* ci: switch dependabot Python ecosystem from pip to uv
uv ecosystem reads uv.lock directly, enabling Dependabot to raise PRs
for lockfile-pinned versions including security patches.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
---------
Co-authored-by: Claude Sonnet 4.6 <[email protected]>
* fix: derive coarse spatial transforms from coordinates (#168)
* fix: derive coarse spatial transforms from coordinates
* refactor: improve function definitions for clarity and consistency
* chore: release 0.10.0 (#162)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* fix(converter): align convert layout to S2, strip _eopf_attrs, fix _FillValue (#171)
- Remove TMS layout and /0 /1 /2 numeric overview groups from the general
'convert' CLI; produce S2-style sibling r{2**level} overviews instead.
'convert-s2-optimized' continues to emit the same layout.
- Drop the '--tile-width' CLI option.
- Sanitize array attrs in all converters: strip stale '_eopf_attrs' and
source-only '_FillValue' / 'dtype' / 'valid_min' / 'valid_max' on decoded
floats; rewrite 'units: digital_counts' -> 'units: 1'. The sanitizer now
returns a new dict and callers reassign 'attrs', so stale keys are
actually removed (a previous '.update()' pattern left them in place).
- Sanitize coord attrs too (datetime coords in /conditions/meteorology
were leaking '_eopf_decode_datetime64' via '_eopf_attrs').
- Set '_FillValue' properly for float measurements so xarray's CF
encoder produces the base64-NaN representation needed for
'use_zarr_fill_value_as_mask=True' round-trip (xarray#11345). Fixes #171.
- Remove dead 'utils.encode_cf_fill_value' (no callers).
- Tests: open each r{N} multiscale level separately; migrate
'test_multiscales_round_trip' from deprecated tms.Multiscales to
zcm.Multiscales (with tuplify_json to handle extra='allow' fields).
- Regenerate snapshot fixtures; verified 0 '_eopf_attrs',
'tile_matrix*', or 'digital_counts' markers across S2A/S2B/S2C.
* test: add guardrails for converter output attrs (#171)
Walks the snapshotted GroupSpec JSON fixtures and asserts:
- no '_eopf_attrs' anywhere;
- no 'tile_matrix*' markers (TMS removal);
- no 'units: digital_counts' on float arrays (decoded scale/offset);
- every float array under '/measurements/' has '_FillValue' set
(required for CF NaN-mask round-trip, xarray#11345).
* docs: deprecate v0 references, document r{N} overview layout (#171)
- Drop 'tile_width' parameter from API examples (option removed from CLI/API).
- Replace '/measurements/r10m/{0,1,2}' nested-pyramid examples with the
current flat 'r{2**level}' sibling-group layout.
- Remove the V0 vs V1 split in converter.md / examples.md / quickstart.md;
the general 'convert' command now produces the same flat layout as
'convert-s2-optimized'. The S2-optimized section is reframed as a
feature description rather than a 'V1 vs V0' comparison.
- Update architecture.md multiscales snippet to the new model
('layout' / 'asset' / 'derived_from' / 'transform').
- Update faq.md inspection snippet to walk the new layout.
* refactor: clean up code formatting and enhance attribute handling in geozarr conversion
* fix(cli-e2e): drop deprecated --tile-width flag from tests and example
* refactor: remove deprecated v0 layout, sanitize attributes, and fix _FillValue handling
* test: add _FillValue masking roundtrip test (#172)
Addresses @vincentsarago's PR review: assert float arrays written with
the converter's _FillValue convention round-trip through xarray's
use_zarr_fill_value_as_mask=True so NaN cells come back masked.
* test: drive _FillValue masking test through create_geozarr_dataset (#172)
Replace the standalone xarray write with a real converter run: build a
small float DataTree, invoke create_geozarr_dataset, then reopen the
output with use_zarr_fill_value_as_mask=True and assert masking on the
nodata patch. Exercises the converter's _FillValue + encoding path
end-to-end, per @vincentsarago's review.
* fix: green CI for #172 — S2 convert flat layout + idna CVE (#177)
* fix(security): bump idna 3.13 -> 3.17 to clear CVE-2026-45409
pip-audit in the Security Audit workflow blocks on idna 3.13
(CVE-2026-45409, incomplete fix of CVE-2024-3651; fixed in 3.15).
Bump the transitive pin via uv lock. pip-audit now reports no
known vulnerabilities.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
* fix(cli): route Sentinel-2 convert through the optimized flat layout
Generic `convert` emitted GeoZarr-0.4 multiscales with native data at the
group root and overviews as nested r{2**N} sub-groups. That tree cannot be
opened by xr.open_datatree (an overview child's x/y conflict with the
parent's inherited x/y), so the `info` and `validate` CLI commands crashed
on the converter's own Sentinel-2 output.
Detect Sentinel-2 inputs and delegate to the existing, tested
convert_s2_optimized path, which emits flat sibling r{N}m levels (the layout
PR #172 documents). Non-S2 inputs keep the generic create_geozarr_dataset
path. Detection is best-effort and never aborts conversion.
Regenerate the geozarr_examples structure snapshots to the flat layout.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
* test: make CLI e2e assertions debuggable and fix multiscale round-trip
- The CLI logs errors to stdout via structlog; assert on stdout+stderr in
test_cli_e2e so a non-zero exit shows the actual error instead of an
empty stderr.
- test_multiscale_attrs_round_trip tuplified only one side of its equality,
so flat-layout level groups carrying list-valued attrs (e.g. spatial:bbox)
broke a check that is only about JSON list/tuple normalisation. Tuplify
both sides; the model still round-trips losslessly.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
---------
Co-authored-by: Claude Opus 4.8 <[email protected]>
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Davis Vann Bennett <[email protected]>
Co-authored-by: Claude Sonnet 4.6 <[email protected]>
Co-authored-by: Emmanuel Mathot <[email protected]>
---------
Co-authored-by: Claude Sonnet 4.6 <[email protected]>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Davis Vann Bennett <[email protected]>
Co-authored-by: Emmanuel Mathot <[email protected]>1 parent 1e891e5 commit f749368
52 files changed
Lines changed: 26821 additions & 51123 deletions
File tree
- .github
- workflows
- docs
- examples
- src/eopf_geozarr
- codecs
- conversion
- data_api/geozarr
- s2_optimization
- stac
- tests
- _test_data
- geozarr_examples
- optimized_geozarr_examples
- test_data_api
- test_geozarr
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
3 | 3 | | |
4 | | - | |
| 4 | + | |
5 | 5 | | |
6 | 6 | | |
7 | 7 | | |
8 | | - | |
| 8 | + | |
9 | 9 | | |
10 | 10 | | |
11 | 11 | | |
| |||
17 | 17 | | |
18 | 18 | | |
19 | 19 | | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
20 | 27 | | |
21 | 28 | | |
22 | 29 | | |
23 | 30 | | |
24 | 31 | | |
25 | 32 | | |
26 | | - | |
| 33 | + | |
27 | 34 | | |
28 | 35 | | |
29 | 36 | | |
| |||
35 | 42 | | |
36 | 43 | | |
37 | 44 | | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
22 | 22 | | |
23 | 23 | | |
24 | 24 | | |
| 25 | + | |
25 | 26 | | |
26 | 27 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | | - | |
| 2 | + | |
3 | 3 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
5 | 5 | | |
6 | 6 | | |
7 | 7 | | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
8 | 42 | | |
9 | 43 | | |
10 | 44 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
16 | 16 | | |
17 | 17 | | |
18 | 18 | | |
19 | | - | |
20 | 19 | | |
21 | 20 | | |
22 | 21 | | |
| |||
29 | 28 | | |
30 | 29 | | |
31 | 30 | | |
32 | | - | |
33 | 31 | | |
34 | 32 | | |
35 | 33 | | |
| |||
201 | 199 | | |
202 | 200 | | |
203 | 201 | | |
204 | | - | |
| 202 | + | |
| 203 | + | |
205 | 204 | | |
206 | 205 | | |
207 | 206 | | |
208 | 207 | | |
209 | | - | |
210 | | - | |
211 | | - | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
212 | 216 | | |
213 | 217 | | |
214 | 218 | | |
| |||
340 | 344 | | |
341 | 345 | | |
342 | 346 | | |
343 | | - | |
344 | | - | |
345 | | - | |
346 | | - | |
347 | | - | |
348 | | - | |
349 | | - | |
350 | | - | |
351 | | - | |
352 | | - | |
353 | | - | |
354 | | - | |
355 | | - | |
356 | | - | |
357 | | - | |
358 | | - | |
| 347 | + | |
359 | 348 | | |
360 | 349 | | |
361 | 350 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
240 | 240 | | |
241 | 241 | | |
242 | 242 | | |
243 | | - | |
| 243 | + | |
244 | 244 | | |
245 | 245 | | |
246 | 246 | | |
247 | | - | |
248 | | - | |
249 | | - | |
250 | | - | |
251 | | - | |
252 | | - | |
253 | | - | |
254 | | - | |
255 | | - | |
256 | | - | |
257 | | - | |
258 | | - | |
| 247 | + | |
| 248 | + | |
| 249 | + | |
| 250 | + | |
| 251 | + | |
| 252 | + | |
| 253 | + | |
| 254 | + | |
| 255 | + | |
| 256 | + | |
| 257 | + | |
| 258 | + | |
| 259 | + | |
| 260 | + | |
| 261 | + | |
| 262 | + | |
| 263 | + | |
259 | 264 | | |
260 | | - | |
261 | | - | |
262 | | - | |
263 | | - | |
264 | | - | |
| 265 | + | |
265 | 266 | | |
266 | 267 | | |
267 | 268 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
59 | 59 | | |
60 | 60 | | |
61 | 61 | | |
62 | | - | |
63 | 62 | | |
64 | 63 | | |
65 | 64 | | |
| |||
82 | 81 | | |
83 | 82 | | |
84 | 83 | | |
85 | | - | |
86 | 84 | | |
87 | 85 | | |
88 | 86 | | |
| |||
95 | 93 | | |
96 | 94 | | |
97 | 95 | | |
98 | | - | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
99 | 101 | | |
100 | 102 | | |
101 | 103 | | |
102 | 104 | | |
103 | 105 | | |
104 | | - | |
| 106 | + | |
105 | 107 | | |
106 | | - | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
107 | 112 | | |
108 | | - | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
109 | 116 | | |
110 | | - | |
| 117 | + | |
111 | 118 | | |
112 | | - | |
113 | | - | |
114 | | - | |
115 | | - | |
116 | | - | |
117 | | - | |
118 | | - | |
119 | | - | |
120 | | - | |
121 | | - | |
122 | | - | |
123 | | - | |
124 | | - | |
125 | | - | |
126 | | - | |
127 | | - | |
128 | | - | |
129 | | - | |
130 | | - | |
131 | | - | |
132 | | - | |
133 | | - | |
134 | | - | |
135 | | - | |
136 | | - | |
137 | | - | |
138 | | - | |
139 | | - | |
140 | | - | |
141 | | - | |
142 | | - | |
143 | | - | |
144 | | - | |
145 | | - | |
146 | | - | |
| 119 | + | |
147 | 120 | | |
148 | 121 | | |
149 | 122 | | |
| |||
169 | 142 | | |
170 | 143 | | |
171 | 144 | | |
172 | | - | |
173 | | - | |
174 | | - | |
175 | | - | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
176 | 149 | | |
177 | 150 | | |
178 | 151 | | |
| |||
182 | 155 | | |
183 | 156 | | |
184 | 157 | | |
185 | | - | |
186 | | - | |
187 | 158 | | |
188 | 159 | | |
189 | 160 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
76 | 76 | | |
77 | 77 | | |
78 | 78 | | |
79 | | - | |
80 | | - | |
81 | | - | |
82 | | - | |
83 | | - | |
| 79 | + | |
| 80 | + | |
84 | 81 | | |
85 | 82 | | |
86 | 83 | | |
87 | 84 | | |
88 | 85 | | |
89 | 86 | | |
90 | | - | |
| 87 | + | |
91 | 88 | | |
92 | 89 | | |
93 | 90 | | |
94 | | - | |
| 91 | + | |
95 | 92 | | |
96 | 93 | | |
97 | 94 | | |
| |||
130 | 127 | | |
131 | 128 | | |
132 | 129 | | |
133 | | - | |
134 | | - | |
135 | | - | |
136 | | - | |
137 | | - | |
138 | | - | |
139 | | - | |
140 | | - | |
141 | | - | |
142 | | - | |
143 | | - | |
144 | | - | |
145 | | - | |
146 | | - | |
147 | | - | |
148 | | - | |
149 | | - | |
150 | | - | |
151 | | - | |
152 | | - | |
153 | | - | |
154 | 130 | | |
155 | 131 | | |
156 | 132 | | |
| |||
433 | 409 | | |
434 | 410 | | |
435 | 411 | | |
436 | | - | |
| 412 | + | |
437 | 413 | | |
438 | 414 | | |
439 | 415 | | |
| |||
0 commit comments