Reading sharded array with multiprocessing produces wrong result #3221

Activity

1. ziw-liu

   added

   bugPotential issues with the zarr-python libraryPotential issues with the zarr-python library

   on Jul 10, 2025
2. d-v-b commented on Jul 10, 2025

   

   d-v-b

   on Jul 10, 2025

   Last edited by d-v-b

   Contributor

   More actions

   i think sharding invites race conditions if multiple processes attempt to write to the same shard file at the same time. so we need a mechanism to ensure that sharded writes in a multiprocessing context are per-shard, and not per-chunk. I wonder if this is related to #3217
3. dstansby

   removed

   bugPotential issues with the zarr-python libraryPotential issues with the zarr-python library

   on Jul 10, 2025
4. ziw-liu commented on Jul 10, 2025

   

   ziw-liu

   on Jul 10, 2025

   ContributorAuthor

   More actions

   i think sharding invites race conditions if multiple processes attempt to write to the same shard file at the same time.

   The test linked in OP is only reading in parallel but still fails.
5. ziw-liu commented on Jul 10, 2025

   

   ziw-liu

   on Jul 10, 2025

   ContributorAuthor

   More actions

   Here's a script version of the test in case reproducing the dev environment is too much trouble:

   import multiprocessing as mp
   
   import numpy as np
   import zarr
   
   
   def _index_array(arr, index):
       return arr[index]
   
   
   if __name__ == "__main__":
       data = np.arange(100)
       arr = zarr.create_array(
           store=zarr.storage.MemoryStore(), data=data, shards=(20,), chunks=(10,)
       )
       ctx = mp.get_context("forkserver")
       with ctx.Pool() as pool:
           results = pool.starmap(_index_array, [(arr, slice(len(data)))])
       assert all(np.array_equal(r, data) for r in results)

   On Linux it doesn't work when shards=(20,), but passes with shards=(10,) (i.e. equal to chunks).
6. d-v-b commented on Jul 10, 2025

   

   d-v-b

   on Jul 10, 2025

   Contributor

   More actions

   thanks for the reproducer. hopefully this is a simple bug to fix, since it's on the reading side
7. ziw-liu commented on Jul 10, 2025

   

   ziw-liu

   on Jul 10, 2025

   ContributorAuthor

   More actions

   It seems like it's only reading the last chunk. results[0] with chunk size fixed to 10:

   # shard size 20
   [ 0  0  0  0  0  0  0  0  0  0 10 11 12 13 14 15 16 17 18 19  0  0  0  0
     0  0  0  0  0  0 30 31 32 33 34 35 36 37 38 39  0  0  0  0  0  0  0  0
     0  0 50 51 52 53 54 55 56 57 58 59  0  0  0  0  0  0  0  0  0  0 70 71
    72 73 74 75 76 77 78 79  0  0  0  0  0  0  0  0  0  0 90 91 92 93 94 95
    96 97 98 99]
   
   # shard size 30
   [ 0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0 20 21 22 23
    24 25 26 27 28 29  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0
     0  0 50 51 52 53 54 55 56 57 58 59  0  0  0  0  0  0  0  0  0  0  0  0
     0  0  0  0  0  0  0  0 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95
    96 97 98 99]
   
   # shard size 90
   [ 0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0
     0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0
     0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0
     0  0  0  0  0  0  0  0 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95
    96 97 98 99]
   

   And all the workers read the same results independent of the number of workers.
8. ziw-liu

   changed the title [-]How to work with shards and multiprocessing?[/-] [+]Reading sharded array with multiprocessing produces wrong result[/+] on Jul 10, 2025
9. ziw-liu commented on Jul 14, 2025

   

   ziw-liu

   on Jul 14, 2025

   ContributorAuthor

   More actions

   @dstansby can you add the 'bug' label back?
10. dstansby

    added

    bugPotential issues with the zarr-python libraryPotential issues with the zarr-python library

    on Jul 14, 2025
11. dstansby commented on Jul 14, 2025

    

    dstansby

    on Jul 14, 2025

    Contributor

    More actions

    sorry - not sure why I removed it!
12. d-v-b commented on Aug 11, 2025

    

    d-v-b

    on Aug 11, 2025

    Contributor

    More actions

    it would be great to get some help fixing this. I don't have time to look into it this week but I think this is an important bug. I'm adding the "help wanted" tag and also pinging @zarr-developers/python-core-devs
13. d-v-b

    added

    help wantedIssue could use help from someone with familiarity on the topicIssue could use help from someone with familiarity on the topic

    on Aug 11, 2025
14. ziw-liu commented on Sep 19, 2025

    

    ziw-liu

    on Sep 19, 2025

    ContributorAuthor

    More actions

    Appears to be fixed in 3.1.3 by #3299.
15. ziw-liu

    closed this as completedon Sep 19, 2025