Repository navigation
[V3] v2 -> v3 data migration #1798
Description
We should invest in tools to make the v2 -> v3 conversion simple for people who are motivated to convert their data. A few high-level ideas:
- A simple CLI that converts an array, or a group (recursive or not) from v2 to v3, in a new location.
-
Someone should investigate how complicated in-place conversions would be. On a local filesystem where. V3 is designed to make array conversions easy, requiring only the creation of new metadata.mvis cheap, this could be attractive.
- Documentation of the key differences between
zarr-pythonv2 and v3, and a migration guide. This should have its own page in the docs. -
- We should consider options for people who don't want to re-save their data. I'm not presently a kerchunk user, but I presume that kerchunk could map v2 to v3, for people who don't want to convert their data? cc @martindurant.
Reacted by Joe Hamman and tkakar
Reacted by Josh Moore, Joe Hamman and Peter Sobolewski
Activity
jeromekelleher commentedon Apr 17, 2024
Big +1 on this. I'm working on a conversion tool for large-scale genomics data (100s TB scale) which is usually held in file systems (for the moment, it will probably migrate to object stores later on). A CLI tool that does an in-place migration from v2 to v3 would be a big help. I'm hoping to move to v3 early on, before too many datasets are converted into v2 format and so most users won't ever know about v2.
My assumptions was that the migration was largely a case of writing a new JSON metadata file per-array, and should be possible to do both cheaply and safely?
d-v-b commentedon Apr 17, 2024
Yes, I think this is right. Besides the metadata, which will live in a completely new JSON document (
zarr.json), V3 supports a backwards-compatible layout for the chunksjeromekelleher commentedon Apr 17, 2024
Thanks yes, I've been aiming for v3 forwards compatibility by using "/" as the default dimension separator. Then, iterating over the chunks in the first dimension and renaming to have a "c" prefix should be relatively cheap (I forgot about this difference).
Is there some developer documentation with recommendations for forwards/backwards compatibility?
normanrz commentedon Apr 17, 2024
For most cases, the migration only requires adding
zarr.jsonfiles throughout the hierarchy. There should be no need to even touch the chunk files.zarr.jsonand.zarrayfiles can also live side-by-side. So, why would amvbe needed?Only, when using a non-supported codec or filter, chunks need to be rewritten.
d-v-b commentedon Apr 17, 2024
This is correct. When I wrote up this issue, I forgot about the
v2chunk key encoding supported by v3 🤦d-v-b commentedon Apr 17, 2024
i updated the issue to be more accurate :)
normanrz commentedon Apr 17, 2024
I agree that a CLI tool that can convert an entire hierarchy would be great!
jhamman commentedon Apr 19, 2024
Today I learned that there is a v1 to v2 migrator in the zarr-python codebase:
zarr-python/zarr/storage.py
Lines 1941 to 1956 in 6105ef2
6 remaining items
jhamman commentedon Jan 3, 2025
#2596 was mislabeled as closing this. A migration tool would still be great!
meteoDaniel commentedon Feb 6, 2025
I am intersted, too.
Without the opportunity to migrate Data from v2 to v3, it would be a nightmare to deprecate zarr v2 support.
M
eschechter commentedon Mar 14, 2025
This gist is a first-pass effort at a migration script. My team has begun to use it to explore converting our existing zarr datasets, but it is currently limited in scope to zarrs stored on s3. I'd welcome any feedback or discussion about how this could be useful to other devs.
meteoDaniel commentedon Mar 14, 2025
@eschechter thanks for sharing!
So it is enough to adapt the metadata without touching the data itself?
eschechter commentedon Mar 17, 2025
That's correct @meteoDaniel - the zarr v3 conversion can be done with just the creation of zarr.json metadata files. No need to touch the data.
K-Meech commentedon Jun 25, 2025
Hi all, I'd be happy to put together a draft PR of a v2 > v3 conversion CLI (expanding / extending @eschechter 's gist - which is a great start 😄). Do let me know if anyone is already working on this (happy to collaborate!) + if there are any preferences for structure / location of the CLI in the existing codebase.
normanrz commentedon Jun 30, 2025
That's great! I don't think there are any CLI tools in this codebase already. So, I guess we're open for suggestions on the structure.
I think it would be great, if the conversion functions could also be called from Python directly. A CLI would then be using these functions.
I had good experiences with building CLIs with typer. That should probably go in a new dependency group, e.g.
pip install zarr[cli].I think the CLI should only attempt to do metadata-only conversions. If that is not possible (e.g. due to missing codecs in v3) it should fail.
d-v-b commentedon Apr 23, 2026
with the CLI added and improved documentation, I think this can be closed as completed.