Repository navigation
Place random dask views at a random position inside each chunk - #2605
Open
Thanatos9404 wants to merge 1 commit into
Open
Thanatos9404 wants to merge 1 commit into
Thanatos9404 wants to merge 1 commit into
Conversation
random_views_for_dask_array picked a random chunk for each view but always sliced it from the start of that chunk, so whenever a view was smaller than the chunk the same strip of every chunk was sampled. Data with structure that repeats with the chunk layout then gave biased statistics and histogram limits. Offset each view by a random amount that keeps it inside its chunk, drawn as a 64-bit integer so that chunks longer than the int32 range also work, and add regression tests. Fixes glue-viz#2413
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
random_views_for_dask_arraypicks a random chunk for each view but always sliced from the start of that chunk. When a view is smaller than the chunk, which is the usual case forcompute_statisticandcompute_histogramwithrandom_subset, every view covered the same strip of its chunk. That biases the results for data that repeats with the chunk layout, such as tiled imaging data.Chunk selection and view sizes are unchanged. Each view is now placed at a random offset that keeps it inside its chunk. With the issue's reproducer, the sampled maximum over 200 seeds was always 63 before and is now 2431 to 4095 (true maximum 4095).
Tests: added
test_random_views_for_dask_array(the issue's reproducer) andtest_random_views_for_dask_array_within_chunkstoglue/utils/tests/test_array.py. Both fail onmainand pass with this change. The offset is drawn as int64 so it also works on Windows for chunks longer than 2**31 elements, covered bytest_random_views_for_dask_array_large_chunk.When a view covers only part of a chunk, positions near the chunk edges are still sampled a bit less often than the middle. Fixing that needs a different view shape, so I left it for a follow-up.
Fixes #2413