file type discovery for the parquet format #519

Activity

1. martindurant commented on Oct 22, 2024

   

   martindurant

   on Oct 22, 2024

   Member

   More actions

   Given the new manifests coming, I'm not sure if this is a good time to do this.
   A couple of thoughts:

   • we already have a .z metadata file, so versioning information can go in there
   • any extra json (or other) files are against the parquet conventions
   • we actually use the presence of *.json file(s) to assume "not parquet", although that should be in the input path name
2. TomNicholas commented on Oct 22, 2024

   

   TomNicholas

   on Oct 22, 2024

   Contributor

   More actions

   Given the new manifests coming, I'm not sure if this is a good time to do this.

   I actually agree. Changing the file format would only help people who write new references in the kerchunk format, but now that icechunk exists I would recommend that people write references in that format from now on instead.

   👍React with 👍1Reacted by Raphael Hagen
3. martindurant commented on Oct 22, 2024

   

   martindurant

   on Oct 22, 2024

   Member

   More actions

   I would recommend that people write references in that format

   You still need to kerchunk-scan the original files at some point - how would you write these references?
4. keewis commented on Oct 22, 2024

   

   keewis

   on Oct 22, 2024

   Last edited by keewis

   ContributorAuthor

   More actions

   all I'd really need is some way to decide whether a directory named ".parquet" is a "kerchunk reference file".

   If we can rely on .zmetadata to exist, that in combination with the directory suffix may already be enough?

   Otherwise I agree that if we have that we don't really need another file and can just put the version info in there (something like {"format": "kerchunk-parquet", "version": 1, "metadata": ...} would already be sufficient).

   I would recommend that people write references in that format

   I don't think we'll be able to convince everyone to switch immediately, so having good tool support for the existing formats is still important.
5. martindurant commented on Oct 22, 2024

   

   martindurant

   on Oct 22, 2024

   Member

   More actions

   If we can rely on .zmetadata to exist, that in combination with the directory suffix may already be enough?

   Yes, I think so
6. TomNicholas commented on Oct 22, 2024

   

   TomNicholas

   on Oct 22, 2024

   Last edited by TomNicholas

   Contributor

   More actions

   You still need to kerchunk-scan the original files at some point - how would you write these references?

   If you use VirtualiZarr then Kerchunk gets called to do the scanning, then the references are in memory (as wrapped ManifestArrays), then you write them out straight to Icechunk. With that workflow at no point do you write out references using the kerchunk json or parquet formats.

   import virtualizarr as vz
   
   vds = vz.open_virtual_dataset(file.nc)  # uses kerchunk's SingleHDF5ToZarr under the hood, references are now in-memory
   
   vds.virtualize.to_icechunk(icechunkstore)  # writes directly to Icechunk's on-disk format