Repository navigation
sharding codec use of lru caching fails with numpy void scalars #3054
Description
numpy void scalars are not hashable, which means that a class with a np.void attribute cannot be hashed by lru_cache, which is used by the sharding codec class here.
I ran into this In #2874. Any ideas for fixing this would be appreciated! IMO the ideal fix would be to resolve the basic performance problems we are solving with this caching layer.
Reacted by Sebastian Hoffmann
Activity
nenb commentedon May 19, 2025
It seems like
_get_chunk_spec(the problem function) is internal to the module in question, and is only called a handful of times. Might a practical solution (short-term) be to add another function that parses the inputs to_get_chunk_specprior to calling it? This parser could then check for a void scalar, convert it to something hashable (e.g.(dtype signature, raw bytes), and then recover the original value from the hashable object prior to returning from_get_chunk_spec.I agree that the ideal fix (likely requires larger refactor, longer-term) would to resolve the basic performance problems that are being solved with this caching layer.
sehoffmann commentedon Jul 20, 2026
+1 Also ran across this when trying to do sharding on arrays with structured dtypes (as they have np.void type fill values)
sehoffmann commentedon Jul 20, 2026
In my case, the issue is the default
ArraySpec.__hash__implementation which hashes each field, includingnp.voidfill values. I think this can be fixed locally inArraySpecby manually hashingnp.voidfill values based on their byte content (.tobytes()). If suchnp.voidvalues need to be hashed in different places as well, it might make sense to introduce a dedicated utility function for this.test: add end-to-end regression test for structured-dtype fills in sh…
fix: gate bulk full-shard decode on identity reads; harden shard-inde…
fix(ArraySpec): proper and robust equality semantics for ArraySpec by…
fix: gate bulk full-shard decode on identity reads; harden shard-inde…
fix: gate bulk full-shard decode on identity reads; harden shard-inde…
fix: gate bulk full-shard decode on identity reads; harden shard-inde…