Skip to content

Do not shard arrays that fit in one chunk #357

Open

Description

@emmanuelmathot

Reported in #349 (Z3). Related to #282.

With --enable-sharding, every array gets one shard that covers the whole array (conversion/utils.py:673-697). For 75 arrays in one product, the shard holds only one inner chunk (for example 9 × 9 meteorology arrays and 23 × 23 angle arrays). A reader then makes two requests (index, then chunk) where one is enough. In 66 arrays the shard is larger than the array (for example 2048 × 2048 for 1830 × 1830). That is valid, but it is not necessary.

Proposal

In build_dataarray_encoding, set shards = None when the array fits in one inner chunk.

Tests

Add a unit test for the shard rule. Regenerate the snapshots.

Activity

  1. changed the title [-]#TBD-04 Do not shard arrays that fit in one chunk[/-] [+]Do not shard arrays that fit in one chunk[/+] on Nov 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    chunkingChunk and shard layoutenhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions