[{"content":"I am a cloud engineer at Development Seed, based in Durham, UK. Most of my work is about making Earth observation data easier to use: building the pipelines that turn satellite archives into analysis-ready datasets, and contributing to the open-source tools people use to read them.\nWhat I care about #I am a strong believer in open source and open data. What attracted me to Development Seed is that its projects make scientific data more accessible, build on open standards and open-source tools, and contribute to environmental monitoring, climate research and open science.\nWhen a tool I depend on has a problem, I would rather fix it upstream than work around it. I have contributed to various projects and tools in the STAC and eoAPI ecosystem, and I started a supply-chain security effort for Development Seed’s open-source repositories.\nI also like sharing what I learn. Preparing a talk or a workshop is one of the fastest ways I know to find the gaps in my own understanding, and giving it starts conversations that would never have happened otherwise. In 2026 I spoke at FOSS4G Europe and the UK Earth Observation Conference, and helped run an eoAPI + STAC workshop (more on the talks page). I write about what I learn on this blog.\nWhat I work on #Since joining Development Seed in November 2025, I have mostly worked on two projects for European agencies.\nFor ESA’s EOPF Sentinel Zarr Explorer, I worked on the pipelines that converted Sentinel products to GeoZarr and published them in a STAC catalogue, and on the automation that kept a large catalogue in order over time (storage tiers, retention). There is more on the project page.\nFor EUMETSAT’s Destination Earth Data Lake, I co-wrote the first architecture decision record and the initial plan for onboarding the data, then built a large part of the chain that turns Meteosat imagery into Zarr and Icechunk datacubes. I also wrote the quickstart notebooks that users start from.\nOn both projects, a good part of the work happens before the pipelines are built: agreeing formats and conventions with the agencies that own the data and the teams who build on it, and choosing designs from benchmarks on the real data.\nHow I got here #I spent about ten years (2010\u20132021) as a physical oceanographer. I did an MSc at Universit\u00e9 Pierre et Marie Curie (now Sorbonne Universit\u00e9), a PhD at the Universit\u00e9 de Perpignan on deep convection and the upper ocean in the Mediterranean, and a short postdoc at LOCEAN in Paris. I then spent four years at the Scottish Association for Marine Science (SAMS) in Oban and two at the National Oceanography Centre in Southampton, working on the North Atlantic Current with gliders and moorings as part of the international OSNAP programme.\nA lot of that work was making observations usable by other scientists. I merged 13 hydrographic databases into a Mediterranean climatology and published it as an open dataset. At SAMS I helped plan and run our part of OSNAP, from the cruises to the archived data: I published the UK-OSNAP glider datasets at the British Oceanographic Data Centre, and adapted and maintained the mooring processing toolbox that SAMS still uses on its mooring cruises. The glider and mooring data went into the first OSNAP overturning results, published in Science in 2019, which I co-authored. Along the way I published 30+ peer-reviewed papers, spent about 200 days at sea on 12 research cruises, and learnt a lot about planning and teamwork under pressure.\nIn 2021 I left research. As a scientist I relied on data systems every day, and I had become more and more curious about how they worked and how to fix them, rather than just reporting issues. I learnt the software side at OSE Engineering, packaging routing algorithms into Python libraries for a delivery-robot prototype, and at the UK Department for Work and Pensions, where I co-architected a secure container platform for data scientists and analysts and mentored junior engineers. At Development Seed, the two sides meet again.\nA selection of my papers and datasets is on my CV, and the full list is on Google Scholar.\nOutside work #I run a small homelab with self-hosted services, and I explore the UK and Europe in a campervan with my family \ud83d\ude90.\n","date":"8 October 2026","permalink":"https:\/\/lhoupert.fr\/about\/","section":"Lo\u00efc Houpert","summary":"","title":"About Me"},{"content":"\nI am a cloud engineer at Development Seed. I build data pipelines and contribute to open-source tools that make Earth observation data more accessible, so it can be used for environmental monitoring, climate research and open science. I am a strong believer in open source and open data, and I like sharing what I learn through talks, workshops and this blog.\nBefore that, I spent about ten years as a physical oceanographer, working with gliders and moorings in the Mediterranean and the subpolar North Atlantic, and publishing open datasets for other scientists. In between, I learnt the software side at OSE Engineering and at the UK Department for Work and Pensions.\n","date":null,"permalink":"https:\/\/lhoupert.fr\/","section":"Lo\u00efc Houpert","summary":"","title":"Lo\u00efc Houpert"},{"content":"I started in physical oceanography, where I spent about ten years (2010\u20132021) observing the ocean with gliders, moorings and research ships, writing processing code and publishing open datasets so others could use those observations. I then moved into software and cloud engineering, and since November 2025 I’ve been building Earth observation data systems at Development Seed.\nCareer Evolution Timeline # \ud83c\udf0d Cloud Engineer 2025 \u2192 Present #Development Seed | Remote, UK\nEarth observation data systems\nWorked on the Sentinel-to-GeoZarr conversion and STAC registration pipelines for ESA’s EOPF Sentinel Zarr Explorer (Kubernetes, Argo Workflows), and built most of the storage lifecycle and retention automation Added STAC validation before publishing and GeoZarr conformance checks across readers (GDAL, TiTiler, OpenLayers), reporting validator gaps upstream Co-wrote the first architecture decision record and MVP plan for onboarding EUMETSAT data into the Destination Earth Data Lake, then built a large part of the chain that turns satellite products into HEALPix Zarr and Icechunk datacubes, including the MSG SEVIRI and MTG FCI cubes, with STAC metadata and quickstart notebooks for users Turned throughput benchmarks into a capacity plan for the MVP, and chose datacube chunk layouts from read benchmarks on the live stores Started Development Seed’s supply-chain security effort: a reusable security-auditing GitHub Action and pinning\/Dependabot fixes across Development Seed and community geospatial repositories Upstream contributions to OpenLayers (GeoZarr dimension selection) and fixes in fsspec and the STAC\/eoAPI tooling; talks at FOSS4G Europe and UKEO 2026, and helped run an eoAPI + STAC workshop \ud83d\ude80 Senior Cyber Platform Engineer 2023 \u2192 2025 #Department for Work and Pensions - Digital | Remote, UK\nSecure container platform for cyber data science\nCo-architected and built a container platform on AWS (ECS Fargate, Terraform modules) used to deploy the analytics and AI applications that cyber data scientists and analysts rely on Added automated security scanning (Semgrep, KICS, Trivy) to the CI\/CD pipelines Moved container images to minimal Chainguard base images Increased deployment frequency from monthly to weekly through CI\/CD automation Raised test coverage from 10% to 85% on two production web apps, with a containerised integration-test environment Mentored four junior engineers through pair programming and knowledge sharing \ud83d\udcbb Software Engineer 2022 \u2192 2023 #Department for Work and Pensions - Digital | Newcastle, UK\nMachine learning prototypes in a regulated environment\nBuilt the machine-learning component of a prototype that classifies handwritten correspondence: a zero-shot classifier (BART-large-MNLI) whose candidate labels I tuned against test datasets I built, on AWS SageMaker, deployed with AWS CDK in TypeScript Built container-based serverless applications with Python and AWS Lambda, with Docker-based CI Helped establish Python coding standards across development teams \ud83d\udd27 R&D Software Engineer 2021 \u2192 2022 #OSE Engineering | France (remote)\nPython libraries and a product prototype\nPackaged vehicle-routing optimisation methods into Python libraries, with documentation and tutorials, for a delivery-robot fleet management prototype Built the full-stack prototype (Django, with live data collection and visualisation; WebSocket\/WebRTC real-time communication handling 20+ concurrent robot connections) and its Docker development environment Wrote the team’s development best-practice guides \ud83c\udf0a Research Scientist 2019 \u2192 2021 #National Oceanography Centre | Southampton, UK\nSubpolar North Atlantic circulation and transport\nPublished the first four years of continuous North Atlantic Current volume transport measurements from the OSNAP mooring array in the Rockall Trough (JGR Oceans, 2020), using an ocean reanalysis to fill a gap at the shelf edge. Quantified the uncertainty with Monte Carlo propagation of instrument errors and with sampling tests on repeat ship sections, and released the time series openly; a 2026 study by the SAMS and NOC team builds on it for a decade-long record Co-authored a study of Nordic Seas overturning, and the associated heat flux, over the past 70\u2013100 years (GRL, 2020), and contributed to EU Blue-Action and ATLAS project reports, including the glider volume-transport section of a Blue-Action report on heat transport toward the Arctic Extended the OSNAP mooring processing toolbox to NOC’s Iceland Basin moorings and co-created the resulting BODC dataset Moved my analysis to Python (xarray, pandas), with reproducible and containerised workflows (conda, Docker, Binder, CI) Helped train PhD students and research staff in computational methods \ud83d\udd2c Research Project Scientist 2014 \u2192 2018 #Scottish Association for Marine Science | Oban, UK\nUK-OSNAP: gliders and moorings in the eastern subpolar North Atlantic\nHelped plan and run SAMS’s part of OSNAP on the eastern boundary (Rockall Trough moorings and Seaglider missions over the Rockall Plateau), from the cruises to the archived data Co-planned and co-piloted nine Seaglider missions; sailed on four OSNAP mooring cruises (2015\u20132018) with international teams, ran calibration casts for the moored CTD sensors against the ship’s CTD, and edited or co-authored the 2014\u20132017 cruise reports Wrote a MATLAB toolbox for glider processing, quality control and gridding; led the adaptation of the RAPID-MOC mooring processing toolbox for OSNAP and maintained it (SAMS still uses it on its mooring cruises) Published the UK-OSNAP glider datasets (near-real-time and quality-controlled) at the British Oceanographic Data Centre and co-created the Rockall Trough mooring datasets; these data went into the array-wide OSNAP overturning results I co-authored (Lozier et al. 2019, Science) Quantified North Atlantic Current volume transport over the Rockall Plateau from 16 repeat glider sections, with Monte Carlo uncertainty estimates for each section (JGR Oceans, 2018) Across my research years: 12 research cruises in the North Atlantic and Mediterranean (2010\u20132018), about 200 days at sea \ud83d\udcda PhD Fellow & Postdoc 2010 \u2192 2014 #CNRS \/ Universit\u00e9 de Perpignan & LOCEAN Paris | France\nMediterranean deep convection and upper-ocean heat storage\nMerged 13 hydrographic databases into a quality-controlled set of more than 140,000 temperature profiles, and published the resulting Mediterranean climatology as an open dataset with a DOI (Progress in Oceanography, 2015) Analysed 2007\u20132013 observations of open-ocean deep convection in the north-western Mediterranean (JGR Oceans, 2016) Helped run the LION deep-convection mooring and wrote its sensor intercalibration report Postdoc at LOCEAN: quantified the space and time scales of winter mixing with statistical and wavelet methods Took part in Mediterranean research cruises with gliders, moorings and ship-based measurements Selected publications and open data #The full list is on Google Scholar and ORCID.\nFirst-author papers\nHoupert, L., et al. (2020). Observed variability of the North Atlantic Current in the Rockall Trough from 4 years of mooring measurements. JGR: Oceans. doi:10.1029\/2020JC016403 Houpert, L., et al. (2018). Structure and transport of the North Atlantic Current in the eastern subpolar gyre from sustained glider observations. JGR: Oceans. doi:10.1029\/2018JC014162 Houpert, L., et al. (2016). Observations of open-ocean deep convection in the northwestern Mediterranean Sea: Seasonal and interannual variability of mixing and deep water masses for the 2007\u20132013 period. JGR: Oceans. doi:10.1002\/2016JC011857 Houpert, L., et al. (2015). Seasonal cycle of the mixed layer, the seasonal thermocline and the upper-ocean heat storage rate in the Mediterranean Sea derived from observations. Progress in Oceanography, 132, 333\u2013352. doi:10.1016\/j.pocean.2014.11.004 Close collaborations\nDurrieu de Madron, X., Houpert, L., et al. (2013). Interaction of dense shelf water cascading and open-sea convection in the northwestern Mediterranean during winter 2012. Geophysical Research Letters. doi:10.1002\/grl.50331 Somot, S., Houpert, L., et al. (2018). Characterizing, modelling and understanding the climate variability of the deep water formation in the North-Western Mediterranean Sea. Climate Dynamics. doi:10.1007\/s00382-016-3295-0 Koman, G., Johns, W. E., Houk, A., Houpert, L., & Li, F. (2022). Circulation and overturning in the eastern North Atlantic subpolar gyre. Progress in Oceanography. doi:10.1016\/j.pocean.2022.102884 OSNAP and ocean-observing programme papers\nLozier, M. S., et al. (2019). A sea change in our view of overturning in the subpolar North Atlantic. Science. doi:10.1126\/science.aau6592 Li, F., et al. (2021). Subpolar North Atlantic western boundary density anomalies and the Meridional Overturning Circulation. Nature Communications. doi:10.1038\/s41467-021-23350-2 Lozier, M. S., et al. (2017). Overturning in the Subpolar North Atlantic Program: A new international ocean observing system. Bulletin of the American Meteorological Society. doi:10.1175\/BAMS-D-16-0057.1 Testor, P., et al. (2019). OceanGliders: A component of the integrated GOOS. Frontiers in Marine Science. doi:10.3389\/fmars.2019.00422 Also on sustained ocean observing and instrument uncertainty: McCarthy et al. 2020, Sustainable observations of the AMOC: methodology and technology (doi:10.1029\/2019RG000654); Hopkins et al. 2019, Transport variability of the Irminger Sea Deep Western Boundary Current from a mooring array (doi:10.1029\/2018JC014730); and Cheng et al. 2016, XBT science: assessment of instrumental biases and errors (doi:10.1175\/BAMS-D-15-00031.1).\nOpen data, code and reports\nGridded climatology of the mixed layer, seasonal thermocline and upper-ocean heat storage rate for the Mediterranean Sea, 1969\u20132013 (SEANOE). doi:10.17882\/46532 UK OSNAP delayed-mode glider dataset, 2014\u20132018 (BODC). doi:10.5285\/79fdab65-0ce9-56ef-e053-6c86abc08912 OSNAP mooring processing toolbox (MATLAB), now maintained by SAMS: ScotMarPhys\/m_moorproc_toolbox Glider processing and quality-control toolbox (MATLAB): lhoupert\/m_oceanglider Sentinel-1 RTC GeoZarr data model and STAC builders in eopf-geozarr (contributor) Houpert, L. (2014). Technical report on the intercalibration of the sensors on the LION mooring line. doi:10.13140\/RG.2.1.4787.3044 ","date":"6 October 2026","permalink":"https:\/\/lhoupert.fr\/cv\/","section":"Lo\u00efc Houpert","summary":"","title":"CV"},{"content":"The new format for all Sentinel products: EOPF Zarr \u2014 FOSS4G:UK 2026, Leeds, 13 October 2026. Lightning talk. A five-minute introduction to EOPF Zarr, the Zarr-based format ESA is exploring for Sentinel products: its design, current status, and the open-source tools growing around it.\neoAPI with STAC for Earth Data at scale \u2014 FOSS4G:UK 2026, Leeds, 12 October 2026. Workshop, co-run with Ciaran Sweet. A hands-on session on cataloguing, discovering, visualising and analysing Earth observation data with eoAPI and STAC, working with its metadata, raster and vector services.\nVisualising Zarr-encoded Sentinel data in the browser \u2014 UK Earth Observation Conference 2026, University of Warwick, 15 September 2026. How the EOPF Sentinel Zarr Explorer lets people browse Sentinel data converted to GeoZarr straight in the browser, without downloads, and the open tools behind it (STAC, GeoZarr, TiTiler). Presented on behalf of the EOPF Explorer team.\neoAPI + STAC workshop \u2014 FOSS4G Europe 2026, Timi\u0219oara, 2 July 2026. Workshop, co-run with Felix Delattre. A hands-on introduction to standing up a STAC API and browsing Earth observation data with eoAPI.\nFrom Cron Job to Self-Healing Pipeline \u2014 FOSS4G Europe 2026, Timi\u0219oara, 29 June 2026. Using Argo and STAC to build Earth observation ingestion pipelines that recover from silent failures instead of hiding them. Slides.\nBefore moving into engineering I spent about ten years in oceanography, where I co-authored more than 50 conference presentations, including 12 invited talks. This page tracks the engineering era; the research years are covered in my CV.\n","date":"6 October 2026","permalink":"https:\/\/lhoupert.fr\/talks\/","section":"Lo\u00efc Houpert","summary":"","title":"Talks & Workshops"},{"content":"","date":null,"permalink":"https:\/\/lhoupert.fr\/tags\/argo-workflows\/","section":"Tags","summary":"","title":"Argo-Workflows"},{"content":"Welcome to my blog! I document here my learning journey as a Cloud Engineer at Development Seed, working on cloud-native pipelines and datacubes for Earth observation. Beyond documenting my own learning, I hope these posts help others exploring the intersection of scientific computing, geospatial technologies, and modern cloud infrastructure. My background spans oceanographic research, platform engineering, and cloud infrastructure.\nWhat I write about # Learning in Public: Building expertise in STAC, Zarr, Kubernetes, and AI orchestration for geospatial systems\nGeospatial Technologies: Cloud-native formats, Python libraries, and infrastructure for Earth observation\nOpen Source Journey: Contributing to community projects and participating in standards development\nCareer Evolution: Navigating transitions from scientific computing to cloud-native engineering\n","date":null,"permalink":"https:\/\/lhoupert.fr\/posts\/","section":"Blog","summary":"","title":"Blog"},{"content":"","date":null,"permalink":"https:\/\/lhoupert.fr\/tags\/data-pipelines\/","section":"Tags","summary":"","title":"Data-Pipelines"},{"content":"","date":null,"permalink":"https:\/\/lhoupert.fr\/tags\/earth-observation\/","section":"Tags","summary":"","title":"Earth-Observation"},{"content":"","date":null,"permalink":"https:\/\/lhoupert.fr\/tags\/foss4g\/","section":"Tags","summary":"","title":"Foss4g"},{"content":"When my colleague Felix introduced me to the FOSS4G conferences and mentioned FOSS4G-Europe in Timi\u0219oara, I spent a bit of time brainstorming about what would be an interesting talk, and I thought that a good candidate could be to share what I learnt this year when building Earth Observation data pipelines (see my other post on it)!\nI started working with data pipeline back in the days of my PhD (2010-2013). For my first scientific article, I built a consolidated database of upper ocean temperature measurements in order to quantify and understand the seasonal variability of the upper ocean heat content in the Mediterranean Sea. The main job was to merge thirteen separate hydrographic databases into one usable climatology: more than 140,000 quality-controlled individual ocean profiles, spanning 44 years of measurements. It was published in Houpert et al. 2015, Progress in Oceanography 132, 333\u2013352 (if you are really interested by the details you can download it from here) . At that time the “data-pipeline” was stitched together by hand with linux cron jobs calling MATLAB scripts…. that I had to re-run myself whenever something looked off \ud83d\ude22 . I made It work, but it was very fragile, and needed someone constantly watching over it!\nWhen I started working on data pipelines at Development Seed at the end of last year, I was not familiar with the tooling but I had a good understanding of the problems we were trying to solve. I was relatively new to Kubernetes (I never used it in prod environment, only tinkering with it on my homelab) and I never used Argo Workflows before. But to my surprise, it went faster than I expected, mostly because I was already familiar with the concepts and I just had to adapt to the new tool. I still remember where I got a bit stuck during my learning journey, which give me the idea of the talk I gave at FOSS4G and its companion repo co-created with Claude, argo-stac-eo-pipeline .\nTo structure the talk, I placed the setups I had used or built over the years on a small maturity ladder with different rungs. Rung 0 is the most basic step, the crontab line I started with during my PhD. Rung 4, the last one presented in the talk, represents a pipeline that repairs itself and tells me every morning what it did. Each rung takes a bit more of the babysitting work off my hands. In the companion repo each rung is a stage (stages\/00-cron, stages\/01-argo-retries …), which is why the demo commands below are make demo STAGE=01, STAGE=02 and so on.\nMy version of the maturity ladder for EO data pipeline (rungs 0 to 4). Source: my FOSS4G 2026 talk, CC BY 4.0 Deep-dive into the demo #All the demo is built so it can run on a laptop. Once the companion repo is cloned, you can just run make check; make up to spin a small kind cluster with MinIO for object storage, pgSTAC and a STAC API for the catalog, stac-browser to look at it, and Argo Workflows to run everything.\nIt ingests synthetic Earth-observation data across two STAC collections, synthetic-aurora-veil and synthetic-tidal-glass. None of it is a real satellite data, I wanted to keep that lightweight and put the focus on the data ingestion problems instead: daily acquisitions, occasional gaps, a backlog that needs backfilling.\nThe same docker image runs at every rung: eo-ingest:dev. It never changes from rung 0 to rung 4, what changes instead is what wraps it.\nRung 0: the cron job #This is where I started when I built my first data pipeline during my PhD, a crontab line starting a shell script regularly with a log file which store the output of the script.\n# trimmed from stages\/00-cron\/crontab 0 3 * * * ...\/stages\/00-cron\/run.sh >> \/tmp\/eo-rung0.log 2>&1 The demo repo reproduces this situation with a single script. This is the only rung you run by hand (and the only one that works without the cluster):\n.\/stages\/00-cron\/run.sh # ingest the default day If we go into more details, the script simply starts a throwaway MinIO in docker, so there is a local S3 to write into, and then runs the same eo-ingest:dev image against it.\nIn some cases, this simple set up may be enough to get the job done, but I remember that when I was in academia I was spending a lot of time “babysitting” my pipelines (checking the outputs in the morning or re-running things by hand whenever a number looked wrong, or a network issue broke the pipeline) …\nRung 1: retries and a first logbook #The first step towards removing this babysitting work is to move the crontab line into into Argo Workflows. The ingest script itself does not change at all, but we add two things in the orchestration around it:\na retry policy, so I am no longer the one re-running things when the pipeline fail, and a STAC catalog acting as a logbook of what has landed, so there is finally a place where to find processed data (and their metadata). # trimmed from stages\/01-argo-retries\/workflows\/ingest.yaml - name: ingest retryStrategy: {limit: "2", retryPolicy: Always} # retry up to twice, on any failure container: image: eo-ingest:dev command: ["python", "-m", "eo_ingest.ingest"] env: - {name: FAIL_ONCE, value: "1"} - {name: STAC_URL, value: http:\/\/stac-api} If we go into more details of the code block above, the FAIL_ONCE variable exist just for the demo: it creates the first fail attempt on purpose to simulate the kind of transient failure I used to fix by hand (a network issue, an upstream service having a bad moment …). Argo then notices the failure, starts a fresh pod, and the second attempt succeeds: in the Argo UI you can watch ingest(0) fail and ingest(1) succeed, and the day is recovered without anyone touching a keyboard. You can see it by yourself by running these commands:\nmake ui # 1st terminal \u2014 open the Argo UI and leave it running make demo STAGE=01 # 2nd terminal \u2014 submit rung 1, watch it fail once then succeed make browse # afterwards \u2014 the ingested item now appears in the catalog The automatic retry recovering a failed ingest in the Argo UI. Source: my FOSS4G 2026 talk, CC BY 4.0 The CronWorkflow that schedules this processing is very similar to the crontab line it replaces:\n# trimmed from stages\/01-argo-retries\/cronworkflow.yaml spec: schedule: "0 3 * * *" # 03:00 daily \u2014 the same time the rung-0 crontab fired concurrencyPolicy: Forbid workflowSpec: { ... } # the workflow above, unchanged Now all the changes are wrapped around the original code in two small YAML files.\nI think this is the rung that bring most values and saves the most human time on a real project!\nRung 2: fanning out the backfill #Retries protect a single run, but at some point we want to “scale things up” especially when working on backfilling historical data. Rung 2 keeps exactly the same ingest step and runs it across a whole time window instead of one day at a time.\n# trimmed from stages\/02-fanout\/workflows\/backfill.yaml spec: parallelism: 10 # at most 10 ingests run at the same time templates: - name: main dag: tasks: - name: backfill template: ingest-day arguments: parameters: [{name: day, value: "{{item}}"}] withItems: ["2026-03-01", "2026-03-02", "\u2026", "2026-03-30"] withItems fans the same step across thirty days, capped by the parallelism limit. Source: my FOSS4G 2026 talk, CC BY 4.0 withItems is doing all the work here: you give Argo a list, and it runs the same template once per item. In the demo, it turns a 30-day backfill from 30 sequential runs into waves of up to ten pods at a time, and you can watch it happen by yourself:\nmake demo STAGE=02 # backfill 30 days; watch ~10 ingest pods run at a time make browse # afterwards \u2014 the whole month appears in the logbook at once I was curious to measure the actual speedup on my laptop, so I ran the same workflow twice, once with the cap pinned to parallelism: 1 and once as above: about 311 seconds for the sequential version against about 50 seconds fanned out, so roughly a 6\u00d7 speedup. It is not a 10\u00d7 speedup because ten pods starting up on a single small kind node lose some time before any of them reach the actual work.\nThe parallelism: 10 line is as important as the fan-out itself: the cap should be sized to what the upstream services can absorb, not to how many cores happen to be available in the cluster.\nThis is also the rung I benefited from the most outside the demo. On a real pipeline, I had estimated a reprocessing job at more than 30 hours with the old monolithic approach, and it finished in less than 4 hours once I restructured it in the way I just described: one query to find what needs doing, then one worker per item. The main work for this step is to identify the part of the job that is embarrassingly parallel and to make sure no state is shared between items; the orchestrator did the rest.\nThe 30-day backfill fanning out, capped at ten ingests at a time. Source: my FOSS4G 2026 talk, CC BY 4.0 Rung 3: asking the logbook what is missing #Until now, I was always the one telling the pipeline what to do: run this day, backfill this window. Rung 3 is where the pipeline starts working out by itself what needs doing, and it is probably my favourite rung. The idea fits in a few lines:\n# the idea, as I put it on a slide; the real find_gaps in src\/eo_ingest\/logbook.py handles the edge cases def find_gaps(config, collection, start, end): present = {day(f) for f in query(collection, start, end)} # what the logbook HAS return [d for d in window(start, end) if d not in present] # what it's MISSING The set of days that should be present, minus the set of days actually present in the catalog, gives the list of days to fetch. If we go into more details, the real find_gaps function is a bit longer than this because it has to handle edge cases (items with a null datetime, catalog pagination …), but it stays cheap: it is a single query over metadata that is already in the database, not a re-scan of a bucket or a re-download of anything.\nfind_gaps feeds only the missing days back into the fan-out. Source: my FOSS4G 2026 talk, CC BY 4.0 The result feeds directly into the fan-out from rung 2, with withParam replacing the hand-written list of days:\n# trimmed from stages\/03-stac-logbook\/workflows\/repair.yaml - name: close-gaps template: ingest-day arguments: parameters: [{name: day, value: "{{item}}"}] withParam: "{{tasks.find-gaps.outputs.result}}" If the gap list is empty, Argo fans out over nothing and the whole workflow is a no-op. This is what makes it safe to leave on a schedule: it will not double-ingest days that are already there. In the demo you can see both behaviours back to back:\nmake seed # plant two collections with deliberate, reproducible gaps make demo STAGE=03 # find-gaps prints the missing days, close-gaps fills exactly them make demo STAGE=03 # run it again \u2192 find-gaps prints [], zero ingest pods start find_gaps reporting the missing days and close-gaps ingesting only them, nothing else. Source: my FOSS4G 2026 talk, CC BY 4.0 Interestingly, this rung felt very familiar to me as I was used to work on similar issues when reprocessing glider data archive during my previous life in oceanography. At that time I did not have a separate log of what had been processed, the NetCDF file naming convention was the record. If a file existed under the expected name, the day was done, and if not, it was a gap and someone (usually me) had to go and get the data. Twelve years later, here I am building the same idea again, but that time using a STAC catalog instead of a directory listing!\nRung 4: a daily report to see what happened #By rung 3 the pipeline “repairs itself”, but I still could not easily see what it had done from one day to another, and I did not want to fall back into my old habit of checking everything by hand every morning. Rung 4 does not touch the ingest step at all; it runs the same image one more way, as a daily report:\nmake demo STAGE=04 # render the report in-cluster argo logs @latest -n eo # the gap heatmap appears in the pod logs The daily gap report rendered in the pod logs, one cell per day per collection. Source: my FOSS4G 2026 talk, CC BY 4.0 This simple demo report has two parts, matching the two kinds of repair happening underneath, and each part has its own data source:\nthe item-level summary (e.g. number of failed attempts and retries) is built by querying the Argo Workflows API: the same argo-server that ran the workflows keeps their run history, so the report simply asks it for the recent runs, and the gap heatmap, one cell per day and per collection (filled when the day landed, empty when it is still missing), is built by calling the same find_gaps function as rung 3 against the STAC catalog. Nothing is exported or stored in advance for the report, no metrics files, no Prometheus or Grafana: everything is re-derived at render time from these two live sources, and the markdown ends up captured as an Argo output parameter, so it can also be retrieved from the finished workflow.\nThe gap heatmap from the daily report: one cell per day and per collection. Source: my FOSS4G 2026 talk, CC BY 4.0 On a real multi-mission pipeline, I built the same idea with a scheduled GitHub Action instead (thank you Claude for the assistance): every morning it pulled the item metrics that the ingest pods had been writing to S3, rendered a self-contained HTML report, and opened it as a labelled GitHub issue. The prerequisites are a bit different in that case: the metrics only exist because the ingest pods were instrumented to write them to S3 in the first place, and the Action needs read read access to that bucket (plus the rights to open issues on the repo), but that is mostly all, there is still no metrics server or dashboard to keep running. One morning, this report showed 1,308 HTTP errors in 24 hours. The report did not fix anything by itself but it pointed me straight at the cause: a token-reuse bug where every parallel pod was re-authenticating from scratch instead of reusing a valid session.\nTwo levels of self-correction #Looking back at this whole ladder, the ingest step never changed, only what wraps it changed. But “self-healing” actually covers two quite different mechanisms, and it may be useful to clearly separate them:\nLevel Failure Who fixes it Rung Item a single run fails transiently Argo retries it, automatically 1 System a whole day never lands the logbook detects and refills it; the report surfaces it 3 + 4 The item-level healing is completely automatic. If a transient failure happens, Argo retries it, and in most cases nobody even needs to know about it. The system-level healing is only half automatic: the logbook can detect and refill a missing day on its own, but it has no way to understand why the day was missing. If an upstream source has been down for a whole week, the repair workflow will happily keep retrying the same gaps every night; it takes a human reading the report to recognise the pattern and go verify what is wrong upstream.\nWhat I’m still learning #I am still early in my Argo Workflows journey, and there is plenty I don’t fully grasp yet! To give a few examples:\nI have been stuck more than once because of semaphores… For example, I have already had to clear “phantom” semaphore holders (ghost locks that block queued workflows, with nothing visible in the Argo UI \ud83d\ude22) without really understanding the controller internals that create them. Capacity planning is another one: in rung 2 the parallelism cap is there to protect the upstream services, but on a real production cluster there is a second limit, the cluster itself. If you launch enough workflows in parallel, the nodes run out of CPU and memory, pods pile up in Pending, and workflows start queueing behind each other. I am still learning how to estimate what a cluster can absorb before launching the work, rather than discovering it when things get stuck… I have also only just started with event-driven triggering (Argo Events) as an alternative to a cron schedule. My first sensor is deployed, but I am still very much at the “copy and adapt” stage. I am very thankful to my colleagues who patiently answered all the questions I had when I started working with Argo Workflows and Kubernetes (a very big thank you Emmanuel and Ciaran!). They really helped me become comfortable with this new tech stack, work more and more independently, and deepen my knowledge much faster than if I had been learning on my own.\nAfter my FOSS4G presentation, I also had very interesting discussions on Temporal, a workflow engine I had never used before. It gave me ideas for small projects I could build to understand how it compares with Argo Workflows, and assess what additional values Temporal could bring to our projects.\nWrapping up #I wrote this post mostly to write down and share the main things I learnt during my first year working on Earth Observation data pipelines as a cloud engineer, but also to show that, with the right architecture and tools, making a data pipeline more reliable is not that difficult, even when you are new to the tools, like I was.\nI find that writing things down also has a lot of value on its own! It is one of the fastest ways I know to find out which parts I only half understood (it happened more than once while writing this post \ud83d\ude05), and, as I experienced at FOSS4G, sharing what you learnt is a great way to start conversations that would never have happened otherwise!\nEverything I presented here is available in the companion repo argo-stac-eo-pipeline: make check; make up brings the whole demo up on a laptop, and you can then walk the rungs one by one with the commands I gave along the way. The full guided tour, with what to expect at each step, is in docs\/walkthrough.md, and the slides of the talk are at lhoupert.fr\/foss4g2026-talk.\nThe same patterns run in production for the EOPF Explorer project, a service available through the Copernicus Data Space Ecosystem.\nThis post is based on the talk “From Cron Job to Self-Healing Pipeline” I gave at FOSS4G Europe 2026 in Timi\u0219oara. It was my first FOSS4G and it was really a great experience, and it definitely makes me want to go to more FOSS4Gs (I already submitted an abstract for FOSS4G-UK happening in October 2026 \ud83d\ude05)!\n","date":"27 July 2026","permalink":"https:\/\/lhoupert.fr\/posts\/04_cron_to_self_healing\/","section":"Blog","summary":"When I was working as a marine scientist I used to babysit my data pipelines, checking the outputs every morning and re-running things by hand. Here is what I learnt in my first year building reliable Earth observation pipelines.","title":"From Cron Job to Self-Healing Pipeline, what I learnt in my first year building EO data pipelines"},{"content":"","date":null,"permalink":"https:\/\/lhoupert.fr\/tags\/stac\/","section":"Tags","summary":"","title":"Stac"},{"content":"","date":null,"permalink":"https:\/\/lhoupert.fr\/tags\/","section":"Tags","summary":"","title":"Tags"},{"content":"","date":null,"permalink":"https:\/\/lhoupert.fr\/tags\/technical\/","section":"Tags","summary":"","title":"Technical"},{"content":"","date":null,"permalink":"https:\/\/lhoupert.fr\/tags\/argo\/","section":"Tags","summary":"","title":"Argo"},{"content":"What it is #Since November 2025 I’ve worked on ESA’s EOPF Explorer, building the pipelines that turn raw Sentinel data into analysis-ready, cloud-native datasets. My work has mostly been in three areas: converting Sentinel-2 products into GeoZarr at scale, the operational machinery that keeps a large catalogue healthy without manual intervention, and extending the pipeline to Sentinel-1.\nSentinel-2: GeoZarr ingestion at scale #The backbone of the project was a pipeline that converted Sentinel-2 L2A products into GeoZarr (CF-compliant metadata, multiscale overviews, tuned sharding and compression) and registered each product in a STAC catalogue, so downstream users could search and open exactly the pixels they need without a bulk download step.\nKeeping the catalogue healthy #The pipelines ran on Kubernetes, with Argo Workflows doing the heavy processing and Argo Events triggering runs as new data landed rather than on a fixed schedule. Around that core sat the lifecycle automation a large catalogue needs: automated storage-tier management to keep recent data fast and older data cheap, expiry-driven retention that cleans up transient products on a schedule, and observability tuned so the system only reprocesses what actually changed.\nI gave a talk at FOSS4G Europe 2026 on this “self-healing” design: From Cron Job to Self-Healing Pipeline. A companion repository with a runnable version of the pattern is on GitHub: lhoupert\/argo-stac-eo-pipeline.\nSentinel-1: radiometric terrain correction #The newest strand was an RTC pipeline for Sentinel-1 GRD imagery, built as an MVP to show how to produce multidimensional GeoZarr datacubes (time x latitude x longitude) per tile. It was built on the Orfeo ToolBox and S1Tiling, fetched Copernicus DEM tiles on demand, and was triggered by new acquisitions. The MVP covered an area of interest spanning mainland France and the Alps (~160 tiles), appending each acquisition into per-tile datacubes served through TiTiler. I contributed its GeoZarr data model and STAC builders to the open-source eopf-geozarr library.\nStack: Kubernetes, Argo Workflows & Events, Zarr, STAC, TiTiler\/eoAPI, Python.\n","date":"23 July 2026","permalink":"https:\/\/lhoupert.fr\/projects\/eopf-explorer-sentinel-pipelines\/","section":"Technical Projects","summary":"Cloud-native Sentinel pipelines for ESA’s EOPF Explorer: Sentinel GeoZarr ingestion at scale, self-healing lifecycle automation.","title":"EOPF Explorer: Sentinel Pipelines for ESA"},{"content":"","date":null,"permalink":"https:\/\/lhoupert.fr\/tags\/esa\/","section":"Tags","summary":"","title":"Esa"},{"content":"","date":null,"permalink":"https:\/\/lhoupert.fr\/tags\/sentinel-1\/","section":"Tags","summary":"","title":"Sentinel-1"},{"content":"","date":null,"permalink":"https:\/\/lhoupert.fr\/tags\/sentinel-2\/","section":"Tags","summary":"","title":"Sentinel-2"},{"content":"A place to write up some of the projects I’ve worked on in more depth than a repository README allows \u2014 from secure container platforms to cloud-native Earth observation pipelines. More are on the way as I find time to document them.\n","date":null,"permalink":"https:\/\/lhoupert.fr\/projects\/","section":"Technical Projects","summary":"","title":"Technical Projects"},{"content":"","date":null,"permalink":"https:\/\/lhoupert.fr\/tags\/zarr\/","section":"Tags","summary":"","title":"Zarr"},{"content":"","date":null,"permalink":"https:\/\/lhoupert.fr\/tags\/conferences\/","section":"Tags","summary":"","title":"Conferences"},{"content":"","date":null,"permalink":"https:\/\/lhoupert.fr\/tags\/geospatial\/","section":"Tags","summary":"","title":"Geospatial"},{"content":"Late June I went to my first FOSS4G conference! FOSS4G Europe 2026 in Timi\u0219oara, and on the first day of talks I gave one myself: From Cron Job to Self-Healing Pipeline, using Argo and STAC for Earth observation data ingestion. I’ll write a separate post going into more detail on the talk and the companion repo I built.\nWith my Development Seed colleagues Felix Delattre and Daniel da Silva at FOSS4G Europe 2026 in Timi\u0219oara. Before the main conference started, I put most of my energy into getting the eoAPI + STAC workshop ready. Preparing it was a really fast way to find out which parts of my own understanding of eoAPI were still fuzzy! I liked running the workshop so much that I submitted an abstract for an eoAPI workshop at FOSS4G-UK, happening in Leeds later this year in October.\nI also really enjoyed participating in other workshops such as the EOEPCA+ workshop run by Richard Conway and James Hinton from Telespazio. I went in mostly to get a good overview of the project as EOEPCA+ is a platform with a lot of components and I wanted to get oriented before some of them show up in my own work. In the end I found the hands-on part the most useful, more than the overview itself.\nThe second day of the conference I mostly followed the European track. I liked Stefanie Lumnitz’s keynote, “CONTRIBUTING.md for Europe: Time to fork the policy repo?”. It described EU policy-making the way you would describe an open-source project’s contribution process (where the entry points are, who reviews what). It was the first time I’d heard EU policy explained that way. The talk Jody Garnett co-presented with Antonio Cerciello on the Cyber Resilience Act deadline did something similar for a regulation I had been vaguely aware of and never actually read. The joint talk by Stefanie Lumnitz and Marco Minghini on open source and digital sovereignty was really eye-opening for me! Coming from the technical side, it changed how I think about the landscape we build in every day. None of these talks were aimed at a cloud engineer who spends his days on ingestion pipelines, but I got a lot from each of them.\nAfter my talk at FOSS4G, someone took the time to show me Temporal, a workflow orchestration tool I had not used before, close enough to what I have been building with Argo Workflows that we had plenty to talk about. I don’t have a comparison to offer yet, but this chat gave me ideas for small projects I could do to learn it and understand its benefits for example for application development, or to streamline how we deploy data pipelines across different cloud environments. But it is a great example of how the whole week went: someone recognizes the shape of a problem they have also had, and is happy to start talking with you about their own experience and what they learned along the way.\nIf you want to know more about “self-healing” geospatial data pipelines, my talk is here: From Cron Job to Self-Healing Pipeline, and the companion repo is on GitHub: lhoupert\/argo-stac-eo-pipeline. I’ll write more about how I built it in a future post.\n","date":"20 July 2026","permalink":"https:\/\/lhoupert.fr\/posts\/03_first_foss4g\/","section":"Blog","summary":"What I got out of my first FOSS4G \u2014 giving a talk, helping run the eoAPI + STAC workshop, and the policy talks and hallway conversations that stuck with me.","title":"My first FOSS4G: a conference full of people who love open source and sharing knowledge"},{"content":"","date":null,"permalink":"https:\/\/lhoupert.fr\/tags\/open-source\/","section":"Tags","summary":"","title":"Open-Source"},{"content":"","date":null,"permalink":"https:\/\/lhoupert.fr\/tags\/cloud-native\/","section":"Tags","summary":"","title":"Cloud-Native"},{"content":"","date":null,"permalink":"https:\/\/lhoupert.fr\/tags\/learning-in-public\/","section":"Tags","summary":"","title":"Learning-in-Public"},{"content":" Ocean observing systems including moored buoys, profiling floats, gliders and satellites. Source: Whitt et. al, Future Vision for Autonomous Ocean Observations, Frontiers in Marine Science, 2020 For a big part of my career, I built data processing systems for oceanographic observations. I designed pipelines that ingested data from underwater buoys, research vessels, and satellite instruments. Working with observational data at scale, I developed strong opinions about data formats. NetCDF became my default because it is a robust, well-documented format that handles multi-dimensional scientific arrays with complete metadata.\nSo when I transitioned to cloud engineering and started working with Earth observation data infrastructure, I made what seemed like a reasonable assumption: Zarr would be NetCDF for the cloud. Same conceptual model, just optimized for object storage instead of POSIX filesystems.\nThat assumption lasted about a week \ud83d\ude05\nWhat Actually Happened #I started my Zarr learning with Lindsey Nield’s excellent explainer “What is Zarr?” on the Earthmover blog. It’s a really nice introduction, very well written, that presents all the key points about Zarr. The article compared Zarr to other file formats, talked about chunks and compression, showed storage layouts. It all made sense on paper, but something in my understanding was still missing. I couldn’t quite connect the technical details to why everyone seemed so enthusiastic about it.\nThen I read Davis Bennett’s presentation from the resources published in the 2025 Zarr Summit recap. In his talk “What is Zarr (and what isn’t it)”, Davis explains something that reframed completely how I understood Zarr: Zarr is an API or protocol, not a conventional file format. For my scientist friends who are struggling in grasping the meaning of an API, it can be summarized as a set of rules and definitions that let software systems communicate with each other.\nI had been approaching Zarr like it was a container format with some clever cloud optimizations. But I had been missing the point… Zarr is fundamentally about storing arrays as collections of named chunks. Each chunk gets a unique identifier (a key), and when you request specific data, for example, reading a time slice or spatial subset, Zarr figures out which chunk keys to fetch. It doesn’t care where those chunks actually live. The keys can point to files on disk, objects in cloud object storage, or entries in a database.\nThis is an important distinction compared to traditional file formats, and it completely changed how I think about designing data pipelines.\nZarr hierarchical structure showing groups and arrays. Source: Zarr Documentation An Oceanographic Sensor Network Analogy #In my oceanographic work, we operated networks of underwater instruments. Autonomous gliders, moored buoys, profiling floats, each collecting continuous measurements. When analysing large amount of data, I learned quickly that “download everything first, then analyze” doesn’t scale well. At that time I wished there was an efficient way to support selective queries (give me temperature profiles between these dates at these depths), lazy evaluation (don’t fetch data you won’t use), distributed access (multiple researchers querying different aspects simultaneously), and partial reads (get monthly averages at a specific location without downloading daily samples of an entire region).\nZarr’s key-value architecture enables exactly these patterns for array data.\nDifferent Ways to Access Data #Working through one of the EOPF tutorial on Zarr (Earth Observation Processing Framework) changed how I was thinking about the problem. The tutorial shows different ways to access the same Sentinel-1 radar data stored in Zarr format. At first this felt like API complexity. Then I realized each pattern represents a different mental model, and both are equally valid depending on your workflow.\nHere’s the first approach, opening the entire hierarchical structure:\nimport xarray as xr # Open the complete data tree dt = xr.open_datatree(url, engine='zarr', chunks={}) This gives you the full organizational structure. Groups, subgroups, the whole hierarchy. It’s exploratory. You’re getting oriented in unfamiliar data. In NetCDF terms, this would be like opening a file and running ncdump -h to see what variables exist and how they’re organized.\nThe second approach jumps directly to a specific group. You can do it in two different ways:\n# Access a specific measurement group directly ds = xr.open_dataset( url, engine='zarr', group='measurements\/VV', chunks={} ) or\n# Navigate to a group, then convert to a dataset ds = dt["measurements\/VV"].to_dataset() You already know what you want. You know the data structure. Just get that specific variable without loading anything else. This is the “I have worked with this dataset before” pattern. In my OSNAP mooring array processing, this would be equivalent to querying just the current meter data from a specific mooring location without pulling temperature and salinity measurements from all the other moorings across the North Atlantic.\nSomething crucial about how both of these are accessing data: there is no download step, it is over HTTP from a remote URL. No local caching is needed before you start working. Zarr figures out which chunks to fetch based on what you are actually requesting.\nThe chunks={} parameter preserves the original chunking from the metadata. The idea behind it is to respect the intent of whoever organized the data. They chose those chunk dimensions for a reason, probably based on common access patterns, and using chunks={} means “trust those decisions”.\nSchematic of the OSNAP mooring array. Source: OSNAP website Chunks as Architecture, Not Optimization #In NetCDF, chunks are a performance tuning option. You can use them to optimize read patterns, or you can ignore them entirely. The format still works.\nIn Zarr, chunks define the data structure itself. Davis’s other presentation makes this clear: Zarr stores large arrays as separate sub-arrays. Each chunk is an independent unit that can be read, written, or compressed separately. The format is lazy: it only creates data chunks when values are written.\nThis maps very well to distributed scientific workflows. When I was processing data from the OSNAP mooring array, we didn’t collect uniform spatial grids across the North Atlantic. We had moorings at specific locations spanning from Labrador to Scotland, each equipped with multiple instruments at different depths measuring temperature, salinity, and velocity. Each mooring was its own data unit, each sensor type its own stream. The observational data was inherently chunked by location, by instrument, by depth.\nZarr formalizes this approach. Chunks themselves become the fundamental units and the chunking strategy can become an expression of your data’s structure and your access patterns.\nWhat This Changes #The remote access pattern from the EOPF tutorials reflects how Zarr fundamentally works. You point to a URL, describe what you want, and the system fetches the relevant chunks. There is no need to download the data before running a processing workflow, or manage local copies of the data.\nFor Earth observation workflows, this matters a lot! Satellite data archives are massive. A single Sentinel-1 scene can be gigabytes but an analysis might only need a specific polarization, a particular time range, and\/or a subset of the spatial extent. Similarly, climate model outputs can be terabytes spanning decades of simulations across global grids. A researcher studying regional precipitation patterns doesn’t need global temperature data at every vertical level! While NetCDF-4 with OPeNDAP servers can enable remote subsetting, it requires specialized server infrastructure and sequential access patterns. Zarr makes this pattern native and works directly with standard object storage over HTTP. You fetch only the chunks that matter for your specific query, no special middleware is required. Plus, Zarr’s independent chunks enable parallel reads, which means multiple chunks can be fetched simultaneously, dramatically improving performance compared to OPeNDAP’s sequential file access!\nI think that is why everyone building cloud-native geospatial infrastructure is converging on Zarr. It provides both a performance gain and a better underlying data model to work with Earth observation data.\nCopernicus Sentinel-1 satellite. Source: Copernicus What I’m Still Learning #I am in my 2nd week into understanding Zarr, and there is still plenty I don’t fully grasp! The differences between Zarr V2 and V3, differences between Zarr and GeoZarr, differences between Zarr conventions and Zarr extensions and how they relate to the core specification, or details of chunk encoding and compression options. But the fundamental mental model shift has happened: I have stopped thinking about files and have started thinking about array protocols over key-value stores.\nIf you are coming from NetCDF, this requires unlearning some assumptions (file handles, sequential reads, monolithic containers). These concepts don’t map cleanly to Zarr because I think Zarr isn’t trying to be a better file format but it operates on different principles.\nIn my opinion, this difference should matter for scientists and data engineers building systems to handle large-scale Earth observation. There is definitely a learning curve, but I think the mental model shift is worth it, especially as Zarr is becoming one of the data standards to work with climate data at the Exascale.\n","date":"25 November 2025","permalink":"https:\/\/lhoupert.fr\/posts\/02_1st_weeks_zarr\/","section":"Blog","summary":"Why understanding Zarr as an API rather than a file format changed my way of thinking about cloud-native geospatial pipelines","title":"When Zarr Stopped Being a File Format: Notes from my first weeks at Development Seed!"},{"content":"","date":null,"permalink":"https:\/\/lhoupert.fr\/tags\/announcement\/","section":"Tags","summary":"","title":"Announcement"},{"content":"","date":null,"permalink":"https:\/\/lhoupert.fr\/tags\/career\/","section":"Tags","summary":"","title":"Career"},{"content":"","date":null,"permalink":"https:\/\/lhoupert.fr\/tags\/development-seed\/","section":"Tags","summary":"","title":"Development-Seed"},{"content":" I am excited to announce that I will be joining Development Seed next week as a Cloud Engineer, working on cloud-native geospatial pipelines and AI-driven data tools for partners such as the European Space Agency (ESA) and the European Center for Medium-Range Weather Forecasts (ECMWF)! On top of that, I am very thrilled about Development Seed’s approach to building open-source solutions by default!\nFrom Ocean Observations to Cloud Engineering #My path into Earth observation data has been a bit unusual. I started by collecting oceanographic measurements at sea; I spent more than 200 days on research vessels where we used various instruments to measure ocean physical parameters such as temperature, salinity, and velocity. I also used a lot of underwater gliders: autonomous, propeller-less robots that use changes in buoyancy and wings to move through the water column and collect oceanographic data. Their multi-month operational autonomy allows coverage of remote areas, supports long-term monitoring, and allows for consistent measurements across vast distances.\nMy observationalist work involved integrating these observations with other sources of climate data such as satellite observations and ocean models, developing quality control workflows for these multi-platform observations, implementing statistical and physics-based analysis methods for spatial variability studies, and publishing my results in scientific articles.\nSchematic of the most common type of measurement techniques used by oceanographers to observe the oceans. Source: Marine Copernicus In 2021, after more than a decade in academic research, I decided to leave my research career to take a research engineer position at OSE Engineering, a fully remote role. One of the main reasons why I decided to quit academia is that I became increasingly curious about the engineering challenges behind the data systems I used regularly. As a researcher, I relied on these systems for oceanographic analysis, but I wanted to understand how they actually work from the inside: how data services handle performance issues, how infrastructure is managed when systems fail. When processing pipelines encountered problems or performed poorly, I wanted to understand why and learn how to fix them rather than just reporting issues.\nThis shift required learning backend development, system architecture, and software engineering best practices. Over the past 4 years, I have built Python-based backend systems and co-managed AWS infrastructure using Terraform. The technical stack I used evolved significantly since my researcher days! I went from MATLAB scientific computing to production-grade Python services and from research-focused workflows to scalable cloud architectures. My expertise also broadened significantly, evolving from single-domain expertise to cross-functional engineering approaches.\nComing Full Circle #Now I am back in the field of Earth observation! This time I will be working with satellite data and geospatial services from an engineering and product perspective, rather than a user perspective. At Development Seed, I will be combining scientific domain knowledge with cloud infrastructure expertise to make Earth observation data more accessible through modern engineering practices. I will work with partners such as the European Space Agency and the European Center for Medium-Range Weather Forecasts. My background in Earth system physics, environmental data collection, and cloud-native engineering all come together in this role! I think I’ll be well-prepared for the technical challenges ahead.\nRange of ESA satellites used for Earth observation missions. Source: ESA Why Development Seed and These Projects #While I was going through the interview process, I discovered the vast range of Development Seed projects and I was very impressed by both their diversity and their depth! It really matches the kind of projects I would like to contribute to. Here are some examples of such projects:\nGlobal Nature Watch #Global Nature Watch is “an open, AI-powered system that transforms groundbreaking land monitoring data into intelligence to understand Earth’s landscapes”. The platform combines 80+ peer-reviewed datasets into a conversational AI interface that makes environmental intelligence accessible across 170 languages. The technical approach uses AI for orchestration rather than content generation which ensures reliable, reproducible outputs grounded in peer-reviewed science. Indigenous communities in the Philippines are using this technology to defend ancestral territories, and Brazilian cooperatives are designing finance products for smallholder farmers. This is exactly the type of real-world impact that motivates me when building software products!\nZarr Ecosystem Development #Zarr ecosystem development shows Development Seed’s commitment to advancing core infrastructure. The work on VirtualiZarr and Zarr v3 directly addresses challenges I faced in oceanography when trying to work with massive climate datasets efficiently. The collaboration with ESA to adopt Zarr for Sentinel missions means analysts can stream only the data they need instead of downloading large zip files.\nObstore #Obstore shows the kind of engineering fundamentals that excite me! The goal is to build a stateless, Rust-backed Python library for object storage access. It might seem like infrastructure work, but it directly impacts python performance when working in cloud-native solution as it enables everything else to run faster and more reliably. The focus on speed, and clarity\/interoperability corresponds to the kind of engineering quality I value.\nCloud-Native Data Transformation #Cloud-native data transformation projects like the Spatial Access Lab work demonstrate how modern tools can make public data truly accessible. The project converts Canada’s Spatial Access Measures from legacy CSVs to GeoParquet, then builds an entirely browser-based analytical interface using DuckDB-WASM and deck.gl.\neoAPI #eoAPI brings together several state-of-the-art projects to create a complete Earth Observation API framework! The integration of pgSTAC, STAC API, TiTiler, and other components shows the power of combining existing open-source tools into coherent systems. This modular, standards-based approach really matches the way I think about building sustainable infrastructure.\nAll these projects share common threads: they make scientific data more accessible and they build on open standards and open-source tools. In addition, they contribute to environmental monitoring, climate action, and open science. But these projects also face substantial (and interesting!) technical challenges such as working with planetary-scale datasets, building AI systems that remain grounded in scientific rigor, or creating infrastructure that runs efficiently across multiple cloud providers. This combination of technical depth and real-world impact is exactly what attracted me to Development Seed!\nTrue color image of the Earth from space. This image is a composite image collected over 16 days by the MODIS sensor on NASA\u2019s Terra satellite. Source: NASA\/GSFC\/Reto St\u00f6ckli, Nazmi El Saleous, and Marit Jentoft-Nilsen Looking Ahead #I think I’m starting from a solid technical foundation: I have experience in Python backend development, cloud-native infrastructure with AWS and Terraform, and with scientific computing libraries like Xarray for multi-dimensional data analysis. That said, there are several technology areas I’ll need to learn:\nThe geospatial data ecosystem (STAC, Zarr, Cloud-Optimized GeoTIFFs, and libraries like GDAL and rio-tiler) represents new formats and standards for making Earth observation data accessible at scale. Kubernetes in production environments will require moving beyond my ECS experience. Modern AI-driven tools for building natural language interfaces to scientific data represent a different approach from the traditional machine learning models I have deployed previously. Now that I am working for an organisation that is open by default, I have no excuse not to document my learning journey \ud83d\ude04. I’ll do this through regular blog posts. I plan to focus them on technical discoveries and challenges encountered. Beyond documenting my own learning, I hope these posts will help others exploring the intersection of scientific computing, geospatial technologies, and modern cloud infrastructure.\n","date":"13 November 2025","permalink":"https:\/\/lhoupert.fr\/posts\/01_joining_development_seed\/","section":"Blog","summary":"Excited to announce I’m joining Development Seed as a Cloud Engineer to work on geospatial infrastructure for ESA and ECMWF. From oceanographic fieldwork to satellite data engineering, my journey with Earth observations continues!","title":"Joining Development Seed to Build Earth Observation Infrastructure"},{"content":" On this page I tried to provide a snapshot of my tools and practices: the Earth observation data I work with and how I go about it, the infrastructure I run things on, how I work with teams, and the language I write it all in.\nData & geospatial STAC \u00b7 Zarr \/ GeoZarr \u00b7 Icechunk \u00b7 TiTiler & eoAPI How I work & teams ADRs before code \u00b7 dry runs and canaries \u00b7 in-depth reviews \u00b7 runbooks \u00b7 workshops Infrastructure Kubernetes \u00b7 Argo Workflows & Events \u00b7 Flux & Helm \u00b7 Terraform \u00b7 GitHub Actions Python uv \u00b7 ruff \u00b7 pyright \u00b7 pytest \u00b7 FastAPI \u00b7 Xarray \ud83d\uddfa\ufe0f Geospatial data engineering #In oceanography I processed data from gliders, moorings, research ships and satellite altimetry, and published some of the results as open datasets. The problems carry over to Earth observation almost unchanged. You still need quality control and metadata that a stranger can understand, only now the data lives in object storage and the formats have to let people read just the part they need.\nFormats & catalogues STAC \u00b7 Zarr v3 \u00b7 GeoZarr \u00b7 Icechunk \u00b7 HEALPix \u00b7 NetCDF Services TiTiler \u00b7 eoAPI Ocean data then, Earth observation now\nFormats Ocean: NetCDF, HDF5\nEarth observation: STAC, Zarr, cloud-optimized geospatial formats Processing Ocean: time-series and spectral analysis, gridding, quality control, uncertainty quantification\nEarth observation: data pipelines Scale Ocean: ocean model output and reanalyses, plus 140,000+ quality-controlled profiles merged from 13 databases\nEarth observation: terabyte-scale satellite archives Metadata Ocean: CF conventions, standardized vocabularies, documented datasets with DOIs\nEarth observation: STAC catalogs, data-model version recorded in each item Quality Ocean: calibration of moored CTD sensors against ship casts, sensor intercalibration, automatic and manual QC\nEarth observation: STAC validation before publishing, GeoZarr conformance checks across readers (GDAL, TiTiler, OpenLayers) \ud83e\udded How I work #A good part of my job doesn’t show on the commit graph: working out what to build before building it, and how to change a system people already depend on without breaking it.\nArchitecture decisions Architecture decision records and MVP breakdowns written before the code, with open questions left explicit for the team Capacity & data layout Capacity plans from measured throughput; chunk and shard layouts chosen from read benchmarks on the live data Standards STAC, Zarr\/GeoZarr and CF conventions Releases & versioning Automated semantic releases and pinned versions; the data-model version recorded in the catalogue; each data release tied to a tagged snapshot Production changes Dry runs and canaries before full production runs, run limits enforced inside the tool, and an undo path rehearsed before the big runs Working with coding agents Coding agents for exploring code, investigating and reviewing; changes are not shipped until I can explain them; adversarial agents to challenge and test architecture design and decisions \ud83c\udfd7\ufe0f Infrastructure & DevOps #I discovered DevOps practices out of necessity: managing research data across multiple environments taught me that manual processes don’t scale, and inconsistency often leads to problems. Now I find myself gravitating toward automation not just for efficiency and repeatability but because it forces clarity in thinking. “If you can’t automate it, you probably don’t understand it well enough yet.”\nOrchestration Kubernetes (OVH\/OpenStack) \u00b7 Argo Workflows (batch pipelines) \u00b7 Argo Events (data-driven triggers) \u2192 data-driven Earth observation pipelines CI\/CD GitHub Actions \u00b7 Conventional Commits (GitLab CI previously) \u2192 automated, reproducible releases GitOps & IaC Flux (GitOps reconciliation) \u00b7 Helm (packaging) \u00b7 Terraform (multi-environment) \u2192 declarative multi-environment deployments Containers Docker \u00b7 multi-stage builds \u00b7 Chainguard \u00b7 minimal base images Security Prowler \u00b7 Trivy \u00b7 zizmor \u00b7 SHA-pinned actions \u00b7 Dependabot \u00b7 Keycloak\/OIDC \u2192 supply-chain and platform hardening Observability Grafana \u00b7 Loki \u00b7 Prometheus \u00b7 Falco \u2192 catching silent failures early AWS boto3 \u00b7 AWS CDK constructs \u00b7 Lambda functions My experience with cloud infrastructure has taught me to appreciate both the technical challenges of system design and the practical impact these systems have on development teams. Container security emerged as a natural extension of my infrastructure work when I began focusing on platform reliability and team productivity. Working with minimal base images has shown me that security practices can actually simplify operations: fewer vulnerabilities mean less time spent on patches and more predictable deployment cycles.\n\ud83d\udc65 Team leadership & enablement # The greatest good you can do for another is not just share your riches, but reveal to them their own.\n\u2014 Benjamin Disraeli Now, at Development Seed\nReviewing colleagues’ and partners’ pull requests in depth before release Writing runbooks and playbooks the team can reuse, for example on container hardening and on registering datacube releases Helping run eoAPI + STAC workshops Earlier, at DWP (2022\u20132025): mentored four junior engineers \u00b7 onboarding guides \u00b7 test coverage from 10% to 85% Mentored four junior engineers through pair programming and knowledge sharing Regular debugging sessions that became teaching moments Wrote onboarding guides for new team members Created reusable Terraform modules and Docker templates encoding best practices Raised test coverage from 10% to 85% on two production web apps Implemented pre-commit hooks and CI\/CD standards balancing guidance with flexibility Earlier, in research (2010\u20132021): trained PhD students \u00b7 OSNAP cruises with teams of 10+ \u00b7 peer review for JGR Oceans and GRL Helped train PhD students and research staff in computational methods Helped plan and run OSNAP mooring cruises with international teams of 10+ people Mentored Master’s students on research cruises and their thesis projects Contributed to the OceanGliders water-transformation task team Reviewed papers for JGR Oceans and Geophysical Research Letters, and research proposals Learning Through Collaboration: Some of my most valuable moments happen when working alongside my teammates on challenging problems. Those debugging sessions often become mutual learning experiences where we both discover better approaches to error handling and system design.\nFreedom Through Structure: I’ve found it interesting how clear technical guidelines can actually increase creativity. When the team doesn’t have to worry about formatting or basic quality checks, I think it creates more mental space to focus on solving the actual problems at hand.\nFrom Solving to Enabling: I shifted my way of thinking from “I know how to fix this” to “How can we build systems so this problem becomes easier for everyone to solve?”\n\ud83d\udc0d Python #I transitioned to Python as my primary language in 2020 after extensive work with MATLAB in research. What began as exploratory data analysis for oceanographic datasets has evolved into building production Flask applications, maintaining reusable Python packages and implementing comprehensive testing frameworks. I’ve found that each domain I worked in taught me something different about writing maintainable, reliable code.\nTooling uv \u00b7 ruff \u00b7 pyright \u00b7 pre-commit hooks Analysis NumPy \u00b7 Pandas \u00b7 Xarray Testing pytest \u00b7 integration testing \u00b7 mocking strategies Web frameworks Flask \u00b7 Django \u00b7 FastAPI Visualization Matplotlib \u00b7 Plotly \u00b7 Cartopy (geospatial) \u00b7 Folium ML & statistics scikit-learn \u00b7 statistical modeling \u00b7 Monte Carlo methods Documentation Sphinx \u00b7 MkDocs \u00b7 automated API docs \ud83c\udfaf My technical evolution #The diagram below shows how I understand my professional development, with Problem Solving as “the driving force”.\nI like the idea of representing my career evolution into three different branches: Scientific Thinking from research years where I learnt to be systematic and question everything, Engineering Practices developed transitioning to software development, and Team Leadership , the area I think I am growing most, as I found sharing knowledge often more impactful than individual work. I think these areas reinforce each other: my research background helps me approach infrastructure methodically, engineering experience makes me a better mentor, and team work drives more systematic thinking.\nThe Continuous Learning foundation seems also to me essential, it is thanks to this that I have been able to developed these three branches, through my various role changes, technologies I learnt, or conversations I had with my colleagues. I think it is the way I approach growth: staying curious, building on what I know, and helping others grow too.\nProblem solving\nScientific thinking\nsystematic \u00b7 rigorous \u00b7 data-driven\nEngineering practices\nautomation \u00b7 security \u00b7 scalability\nTeam leadership\nmentoring \u00b7 pairing \u00b7 standards\nContinuous learning\nWhat I’m currently focussing on\nKnowledge Sharing: Contributing to documentation and learning resources for geospatial data engineering Open Source Participation: Engaging with communities building Earth observation infrastructure Science to Software: Turning scientific data and methods into tools and datasets other people use, from glider and mooring toolboxes in my oceanography years to analysis-ready satellite datacubes today ","date":null,"permalink":"https:\/\/lhoupert.fr\/skills\/","section":"Lo\u00efc Houpert","summary":"","title":"Skills"},{"content":"","date":null,"permalink":"https:\/\/lhoupert.fr\/tags\/ai-ml\/","section":"Tags","summary":"","title":"Ai-Ml"},{"content":"","date":null,"permalink":"https:\/\/lhoupert.fr\/tags\/aws\/","section":"Tags","summary":"","title":"Aws"},{"content":"Project Overview #Co-architected and built a container-based platform for deploying analytics and AI applications for internal teams, focusing on security compliance and developer productivity.\nTechnical Architecture #Infrastructure Foundation\nAWS ECS Fargate for containerized application runtime Terraform modules for reproducible infrastructure deployment Multi-environment support (development, testing, production) Integration with organizational identity management systems Security Implementation\nMigration from standard Debian images to Chainguard minimal containers Implemented security scanning integration within CI\/CD pipelines Network isolation through VPC endpoint configurations New AI\/ML Integration\nAmazon Bedrock integration with custom guardrails RAG architecture using OpenSearch vector storage Support for multiple foundation models with dynamic selection Token usage monitoring and cost optimization Development Methodology #Infrastructure as Code\n# Example Terraform module structure module "multi_user_app" { source = ".\/modules\/ecs-app" app_name = var.app_name user_config = local.user_configurations container_image = "${var.ecr_repository}:${var.image_version}" # Security configurations vpc_config = var.vpc_settings security_groups = var.security_group_ids # Monitoring integration cloudwatch_config = var.monitoring_settings } CI\/CD Automation\nAutomated version tagging using conventional commit standards Container image building and registry publishing Terraform validation and security scanning Integration testing with containerized dependencies Collaborative Aspects #This project involved close coordination with:\nData science teams for application requirements Security architects for compliance validation Infrastructure specialists for networking design Project delivery teams for deployment scheduling Technical Outcomes # Deployment frequency improved from monthly to weekly cycles Developer onboarding time decreased through standardized tooling Infrastructure consistency achieved across multiple environments Learning Points #Working within government compliance requirements provided valuable insights into balancing security constraints with developer productivity. The experience highlighted the importance of early security integration rather than retrofitting compliance measures.\nTechnology Stack: AWS ECS, Terraform, Docker, Python, GitLab CI\/CD\n","date":"1 August 2025","permalink":"https:\/\/lhoupert.fr\/projects\/candl-platform\/","section":"Technical Projects","summary":"Multi-user AI platform with RAG capabilities, serving internal data science teams","title":"CAnDL Platform"},{"content":"","date":null,"permalink":"https:\/\/lhoupert.fr\/tags\/containers\/","section":"Tags","summary":"","title":"Containers"},{"content":"","date":null,"permalink":"https:\/\/lhoupert.fr\/tags\/terraform\/","section":"Tags","summary":"","title":"Terraform"},{"content":"","date":null,"permalink":"https:\/\/lhoupert.fr\/categories\/","section":"Categories","summary":"","title":"Categories"}]