Repository navigation
Check downstream impact of the chunk layout from #277 (1024 chunks, single-chunk overviews) #282
Description
Check downstream impact of the chunk layout from #277 (1024 chunks, single-chunk overviews)## Context
In #277, @lhoupert raised that overview chunks now come from the parent level's dask chunks, not from spatial_chunk:
Sharded (as data-pipeline runs it), r120m is a single 915×915 chunk, so a 64×64 tile read decodes ~2.5 MB instead of 102 KB on main. Unsharded, r720m ends up as 64 chunks of 21×21.
@shollendEODC replied that EODC production requires a uniform 1024 × 1024 chunking. _rechunk_ds() rechunks the whole tree before the multiscale step, so the overviews inherit it.
We expect little or no downstream impact, but nobody has checked. This issue records Loïc's point and the check.
Layout today
S2 L2A reflectance, sharded. Values are from the snapshot fixtures: main (test with spatial_chunk=256) against #277 (test with spatial_chunk=1024):
| Level | Shape | main: shard / inner chunk |
#277: shard / inner chunk |
|---|---|---|---|
| r10m | 10980² | 10980² / 244² | 11264² / 1024² |
| r20m | 5490² | 5490² / 183² | 6144² / 1024² |
| r60m | 1830² | 1830² / 183² | 2048² / 1024² |
| r120m | 915² | 915² / 183² | 915² / 915² (one chunk) |
| r360m | 305² | 256² / 256² | 305² / 305² (one chunk) |
| r720m | 152² | 152² / 152² | 152² / 152² |
Notes:
- The layout depends on
spatial_chunk, not only on feat!: reintegrate cpm-geozarr into main #277. Our production S2 template (platform-deploy,eopf-explorer-convert-v1-s2-template.yaml) does not pass--spatial-chunk, so it uses the CLI default of 256. EODC runs with 1024. - With 1024, one uint16 inner chunk is 2 MiB before compression, compared with about 116 KiB for 244² and 65 KiB for 183². The smallest read unit is the inner chunk, including inside a shard.
- Without sharding, the overviews can end up with many very small chunks (Loïc: 64 chunks of 21² at r720m).
What to check
For each consumer, compare tile latency, bytes read and memory, for spatial_chunk=256 against 1024, at native zoom (r10m) and at overview zooms (r120m–r720m):
- titiler-eopf: XYZ tiles (256 and 512 px) and a
rescaletrue-colour request. Reads go through zarr-python; check bytes fetched per tile and response time. - Viewer, OpenLayers
GeoZarrsource via zarrita: reads in the browser, so larger inner chunks mean more download and more decode per displayed tile. Check network bytes per tile and time to first render at low zoom (r120m and r360m become one chunk). - titiler-openeo / openEO Studio: processing graphs over a small AOI; check bytes read.
- Notebooks and the foss4g workshop (zarrita exercise): only check that nothing assumes a chunk size.
Expected outcome
- If no consumer degrades in a way that matters, close this issue with the measurements.
- If one does, feed the results into the chunk policy work that @shollendEODC takes on (Discussion: output chunk-layout policy for optimized GeoZarr (single-chunk policy, sharding, inner-tile size) #222, Keep input chunking pattern instead of overwriting #243). For example: overview inner chunks limited to a tile-friendly size (256 or 512) while shards stay large, or a different chunk size per level.
Activity
shollendEODC commentedon Sep 28, 2026
Sounds good, we are also starting our own internal benchmarking (including read-times a comparison of
consolidated_metadatainrootandmeasurements). We can report here or in another issue, depending on the resultsemmanuelmathot commentedon Sep 28, 2026
Here is perfect. Thx @shollendEODC 🚀
lhoupert commentedon Sep 28, 2026
Hi all, I asked Claude to do a quick check on one granule that exists in both layouts on the same OVH S3 endpoint:
S2B_MSIL2A_20260920T112109_N0512_R037_T29SPB_20260920T151606insentinel-2-l2aand insentinel-2-l2a-samples-zarr3-ovh. I just ran it on band b04 only, median of 4 reads, from a laptop:Some findings from Claude:
@shollendEODC: could the overview inner chunks be capped at 512 or less? And is 1024 at r10m–r60m a hard requirement? If not, could we use 512?
emmanuelmathot commentedon Sep 29, 2026
@lhoupert 1024 px has been agreed early in the project. That call has been made by ESA as the best trade-off between data requets size and roundtrips. We used 256px for exploration but we knew 1024px chunk would be the default. I believe the trade-off upside is less requests made. On the other hand, sharding was not an option at that time (zarr v2) but may now be a trade-off new parameter to decide the best chunking size.
shollendEODC commentedon Sep 29, 2026
Yeah 1024 (or 2048, but thats discarded) is defined in our project requirements, so we would stick with it. 3dim arrays are chunked along the polarization (1, 1024, 1024) for S1 but sharded together in a single file.
lhoupert commentedon Sep 29, 2026
Thanks both that settled it.
For the viewer: with 1024-px chunks, OpenLayers (10.10) draws the map in small 64-px tiles and has to read and unpack the whole 1024-px chunk for each tile. So the same chunk can be unpacked up to 256 times. I'll check whether the Explorer draws these items this way or through titiler, and follow up on the viewer side.
OpenLayers
mainalready fixes this (openlayers/openlayers#17602), but it isn't released yet. cc @ahocevarm-mohr commentedon Sep 30, 2026
Quick oite that ol-stac and STAC Browser (latest released versions of each) already use the unreleased GeoZarr source. So no issues in those tools.