Skip to content

Extension of functionalities to all other modes #248

Open

Description

@shollendEODC

I did some tests with more dataproducts to check if and how they are working (X not working, Y working). From these results, I deduced some tasks/recommendation for aligning the workflows:

Status Product Mission
X S1A GRDH IW SDV Sentinel-1A
X S1A GRDH IW SSH Sentinel-1A
X S1B GRDM EW SDV Sentinel-1B
X S1A GRDH IW SSH Sentinel-1A
X S1A SLC IW SDV Sentinel-1A
X S1C OCN IW SDV Sentinel-1C
Y S3 OLC EFR Sentinel-3
X S3 SL RBT Sentinel-3
Y S3 SL LST Sentinel-3
Y S3 OLC ERR Sentinel-3

General recommendations:

  • harmonize the code to outsource functions used by all modes into a utils module
  • chunking
  • sharding
  • ... everything else that can be simplified

Ideas for implementing S1:

  • remove bugs preventing running of the code (tbi) -> maybe alignment problem with cpm 3.0.0
  • create own S1 submodule

Ideas for implementing S3:

  • extend to more mode than OLCI -> SLSTR
  • extend functionalities of sharding and chunking

As these would be rather big changes, we would need to align who will implement which features.

Activity

  1. shollendEODC commented on Aug 25, 2026

    @shollendEODC
    CollaboratorAuthor

    @emmanuelmathot @d-v-b
    would appreciate feedback/alignment on how to tackle it

  2. added
    enhancementNew feature or request
    sentinel-1Sentinel-1 products (GRD, SLC, OCN, GRD RTC)
    sentinel-3Sentinel-3 products (OLCI, SLSTR)
    sample-serviceRoadmap with EODC for the EOPF Sentinel Zarr Samples Service (#277)
    on Sep 28, 2026
  3. emmanuelmathot commented on Sep 28, 2026

    @emmanuelmathot
    Contributor

    Thanks @shollendEODC for the test matrix, and sorry for the late answer. The plan that @claxn proposed in #277 answers most of this, so I suggest we use this issue to track the product coverage, with one PR per product into main, as agreed there.

    Where we are after #277 (0.12.0)

    Product Status on main Next step Owner
    S2 MSI L1C / L2A ✅ optimized pipeline, CPM driver routing encoding switch in #287 @emmanuelmathot / @lhoupert
    S3 OLCI EFR / ERR ✅ optimized pipeline (native swath or --output-grid) check multiscales generation @shollendEODC
    S3 SLSTR RBT / LST ❌ / ✅ in your tests (generic path) SLSTR support after the generic chunking and sharding utilities @shollendEODC
    S1 GRDH / GRDM (IW, EW; SDV, SSH) ❌ in your tests first find why they fail (see below), then an S1 submodule @shollendEODC
    S1 SLC IW ❌ after the generic parts are in place @shollendEODC
    S1 OCN ❌ after the generic parts are in place @shollendEODC
    S1 GRD RTC (S1Tiling / OTB COGs) ✅ standalone ingest-s1 commands (#216) — @emmanuelmathot / @lhoupert

    Sentinel-1 GRD failures. On main, S1 GRD goes through the generic pipeline with ground control points (groups=["measurements"], gcp_group="conditions/gcp"). Our tests cover this, but only with our fixtures. Could you share the errors (or a log) for the failing GRD products, and whether you ran them through the CPM driver with CPM 3.0.0? That tells us whether it is a CPM 3.0.0 product-structure change or a bug in the generic path.

    Order of work, following #277:

    1. Generic chunking and sharding utilities (Discussion: output chunk-layout policy for optimized GeoZarr (single-chunk policy, sharding, inner-tile size) #222, Keep input chunking pattern instead of overwriting #243), which all products need.
    2. Classification pyramids for S2 (Make classification layers (SCL, cloud masks) a full multiscale pyramid #274, with GeoZarr specification dissimilarities #245 and fix(s2): write geo metadata for conditions/mask groups so SCL is readable #262).
    3. Sentinel-1: fix GRD, then SLC and OCN, as their own submodule.
    4. Sentinel-3: all OLCI and SLSTR products.
    5. The optional "overviews" multiscale utility, and removing the reprojection under measurements for S1 GRD.

    For each product PR, please include:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or requestsample-serviceRoadmap with EODC for the EOPF Sentinel Zarr Samples Service (#277)sentinel-1Sentinel-1 products (GRD, SLC, OCN, GRD RTC)sentinel-3Sentinel-3 products (OLCI, SLSTR)

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions