feat: add C-order option for writing sub chunks within shards - #364

Background

While investigating zarrs/zarrs-python#152, I discovered that within-shard reads are random in zarrs (which took way longer than I'd like to admit). Even after merging sub chunks and removing the big copy (#358 + #362) the partial reads were still crazy slow. The reason turned out to be the sub chunk layout on disk did not match my mental image.

Solution

I'm adding subchunk_write_order here as an option on CodecOptions with two possible values as of writing - C-ordering and Randomized (default). Then on calls to ShardingCodec.encode, the correct branch is taken (for only bounded arrays as of writing).

Maybe TODOs (now or open issue for later):

• If the design here makes sense, I'd apply this to encode_unbounded as well especially since it seems like an even lighter refactor there - encoding and writing are already split so we would just slot in between those two steps with this option. I think independent of whether or not the "design" is right (i.e., a global option or CodecOption), encode_unbounded makes sense so I've refactored to implement that.
• Morton order would probably be simple (famous last words) to implement since it seems like we only need to worry about interleaving one data type, u64. But there is also a crate that does seem to be actively maintained as well (see note in CodecOptions: https://docs.rs/morton-encoding/latest/morton_encoding/)
• Benchmarking could be nice (in zarr_benchmarks). I have already tried out the combination of zarr-python write order with perf: merge adjacent subchunk reads #358 and can confirm that it gives (the expected) 20x speedup I have come to expect from doing sequential I/o (in my use case).
• F order for completeness