FileNotFoundError for .zmetadata with MultiZarrToZarr on non-consolidated Zarr sources #553

Activity

1. martindurant commented on Apr 14, 2025

   

   martindurant

   on Apr 14, 2025

   Member

   More actions

   use_consolidated=False is peppered throughout MultiZarrToZarr (used with zarr.open, zarr.open_group, xr.open_dataset, etc.), so it would be good to see the full traceback, to see if maybe there is a call somewhere in the code that has missed it.
2. marstonsward commented on Apr 15, 2025

   

   marstonsward

   on Apr 15, 2025

   Author

   More actions

   @martindurant I will run it again to see if I can get the entire traceback. I thought I had pasted all the tb. Hang tight.
3. marstonsward commented on Apr 15, 2025

   

   marstonsward

   on Apr 15, 2025

   Author

   More actions

   @martindurant

   I'm afraid that is all the tb the notebook spits out.

   2025-04-15 21:05:35.082 | INFO     | kerchunk_create:kerchunk_create:37 - Received 4488 input Zarr stores.
   2025-04-15 21:05:35.083 | INFO     | kerchunk_create:kerchunk_create:38 - First few files: ['/export/sml-data-lake/datastash/47/47b16245-7415-4b92-b52f-98e884caef3c/gfs/gfs_2024_01_01_t00z_f000.zarr', '/export/sml-data-lake/datastash/47/47b16245-7415-4b92-b52f-98e884caef3c/gfs/gfs_2024_01_01_t00z_f001.zarr', '/export/sml-data-lake/datastash/47/47b16245-7415-4b92-b52f-98e884caef3c/gfs/gfs_2024_01_01_t00z_f002.zarr']
   2025-04-15 21:05:35.083 | INFO     | kerchunk_create:kerchunk_create:42 - Generating Kerchunk index for concatenation along 'time'...
   2025-04-15 21:05:35.083 | INFO     | kerchunk_create:kerchunk_create:43 - Identical dimensions: ['latitude', 'longitude']
   2025-04-15 21:05:35.084 | INFO     | kerchunk_create:kerchunk_create:50 - Use MultiZarrToZarr with the provided list of files
   2025-04-15 21:05:35.084 | INFO     | kerchunk_create:kerchunk_create:59 - Generate the dictionary representation of the virtual dataset
   2025-04-15 21:05:35.173 | ERROR    | kerchunk_create:kerchunk_create:97 - An error occurred during Kerchunk processing: [Errno 2] No such file or directory: '/export/sml-data-lake/datastash/47/47b16245-7415-4b92-b52f-98e884caef3c/gfs/gfs_2024_01_01_t00z_f000.zarr/.zmetadata'

   Here's the code:

   def kerchunk_create(
       input_file_list: list,
       output_index_file: str = './combined_dataset.kerchunk.json',
       concat_dimension: str = 'time',
       identical_dimensions: list = ['latitude', 'longitude', 'level'] # Adjust as needed
       ):
       """
       Creates a Kerchunk index file from a list of Zarr stores,
       optimized for local filesystem access (like EFS). Uses Loguru for logging.
   
       Args:
           input_file_list: A list of paths to the input Zarr stores.
           output_index_file: Path to save the generated Kerchunk JSON index.
           concat_dimension: Dimension name along which to concatenate.
           identical_dimensions: List of dimension names expected to be identical
                                 across all input stores.
   
       Returns:
           xarray.Dataset: The dataset opened via the generated Kerchunk index.
           Returns None if index creation fails or no input files are provided.
       """
   
       # --- Step 1: Validate Input File List ---
       if not input_file_list or not all(isinstance(f, str) for f in input_file_list):
           # Using imported loguru logger directly
           logger.error("Input must be a non-empty list of file paths.")
           return None # Or raise ValueError
   
       logger.info(f"Received {len(input_file_list)} input Zarr stores.")
       logger.info(f"First few files: {input_file_list[:3]}")
   
       # --- Step 2: Generate the Kerchunk Index ---
   
       logger.info(f"Generating Kerchunk index for concatenation along '{concat_dimension}'...")
       logger.info(f"Identical dimensions: {identical_dimensions}")
   
       # Since data is on EFS (mounted locally), protocol is 'file'
       protocol = 'file'
       storage_options = {} # No special options needed for local EFS access
   
       try:
           logger.info('Use MultiZarrToZarr with the provided list of files')
           mzz = kerchunk.combine.MultiZarrToZarr(
               input_file_list, # Use the list directly
               concat_dims=[concat_dimension],
               identical_dims=identical_dimensions,
               remote_protocol=None, # Data is local (EFS)
               remote_options=None,  # Data is local (EFS)
           )
   
           logger.info('Generate the dictionary representation of the virtual dataset')
           kerchunk_dict = mzz.translate()
           logger.info("Kerchunk dictionary generated successfully.")
   
           # --- Step 3: Save the Index to a File ---
   
           logger.info(f"Saving Kerchunk index to: {output_index_file}")
           output_dir = os.path.dirname(output_index_file)
           if output_dir and not os.path.exists(output_dir):
               os.makedirs(output_dir, exist_ok=True)
               logger.info(f"Created output directory: {output_dir}")
   
           with open(output_index_file, 'w') as f:
               json.dump(kerchunk_dict, f)
           logger.info("Kerchunk index saved successfully.")
   
   
           # --- Step 4: Open the Virtual Dataset using Xarray ---
   
           logger.info(f"Opening virtual dataset using Xarray from: {output_index_file}")
   
           ds = xr.open_dataset(
               "reference://",
               engine="zarr",
               backend_kwargs={
                   "storage_options": {
                       "fo": output_index_file, # Point to the local JSON index
                       "remote_protocol": None, # Original data is local
                       "remote_options": {}     # Original data is local
                   },
               }
           )
   
           logger.info("Successfully opened dataset via Kerchunk.")
           return ds
   
       except Exception as e:
           logger.error(f"An error occurred during Kerchunk processing: {e}")
           return None # Return None or handle error appropriately

   When I try and pass in consolidated in either the xr.open_dataset() and/or mzz = kerchunk.combine.MultiZarrToZarr() I get the error that argument or kwargs is not recognized.