Skip to content

Place random dask views at a random position inside each chunk - #2605

Open
Thanatos9404 wants to merge 1 commit into
glue-viz:mainfrom
Thanatos9404:fix-random-views-dask-2413
Open

Thanatos9404 wants to merge 1 commit into
glue-viz:mainfrom
Thanatos9404:fix-random-views-dask-2413

Conversation

@Thanatos9404

@Thanatos9404 Thanatos9404 commented Oct 19, 2026 •

Copy link
Copy Markdown

Description

random_views_for_dask_array picks a random chunk for each view but always sliced from the start of that chunk. When a view is smaller than the chunk, which is the usual case for compute_statistic and compute_histogram with random_subset, every view covered the same strip of its chunk. That biases the results for data that repeats with the chunk layout, such as tiled imaging data.

Chunk selection and view sizes are unchanged. Each view is now placed at a random offset that keeps it inside its chunk. With the issue's reproducer, the sampled maximum over 200 seeds was always 63 before and is now 2431 to 4095 (true maximum 4095).

Tests: added test_random_views_for_dask_array (the issue's reproducer) and test_random_views_for_dask_array_within_chunks to glue/utils/tests/test_array.py. Both fail on main and pass with this change. The offset is drawn as int64 so it also works on Windows for chunks longer than 2**31 elements, covered by test_random_views_for_dask_array_large_chunk.

When a view covers only part of a chunk, positions near the chunk edges are still sampled a bit less often than the middle. Fixing that needs a different view shape, so I left it for a follow-up.

Fixes #2413

random_views_for_dask_array picked a random chunk for each view but always
sliced it from the start of that chunk, so whenever a view was smaller than
the chunk the same strip of every chunk was sampled. Data with structure
that repeats with the chunk layout then gave biased statistics and
histogram limits. Offset each view by a random amount that keeps it inside
its chunk, drawn as a 64-bit integer so that chunks longer than the int32
range also work, and add regression tests.

Fixes glue-viz#2413

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

random_views_for_dask_array is not so random

1 participant