Repository navigation
Document that random_shuffle shuffles only locally - #6491
Merged
JanuszL merged 3 commits intoSep 21, 2026
Merged
Conversation
The random_shuffle buffer is a local shuffle: a sample is never returned more than initial_fill positions ahead of where it is read. Readers of container formats (TFRecord, MXNet, Caffe, Caffe2, WebDataset) read the samples in the stored order, so a dataset sorted by label stays mostly sorted with the default buffer. The documentation did not mention this. Add a note to the random_shuffle and initial_fill arguments and explain the behavior in the Sharding and Shuffling guide. Signed-off-by: Max Freedom Pollard <[email protected]>
Contributor
 |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
The buffer only draws samples from the shard that is currently being read, so it is the shard, not the whole dataset, that has to fit into the buffer for a complete shuffle. With one shard the two are the same. Signed-off-by: Max Freedom Pollard <[email protected]>
Signed-off-by: Max Freedom Pollard <[email protected]>
JanuszL
approved these changes
Sep 21, 2026
jantonguirao
approved these changes
Sep 21, 2026
jantonguirao
left a comment
jantonguirao
left a comment
Collaborator
There was a problem hiding this comment.
Documentation-only change. Verified the shuffle-buffer behavior and reader list against the loader implementation — matches. Prior review comments (Greptile, JanuszL) already addressed in the current HEAD.
Contributor
|
!build |
Collaborator
|
CI MESSAGE: [69031371]: BUILD STARTED |
Collaborator
|
CI MESSAGE: [69031371]: BUILD PASSED |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Category:
Other (e.g. Documentation, Tests, Configuration)
Description:
Closes #3088.
random_shuffledraws samples from a buffer ofinitial_fillsamples that is filled sequentially, so it only shuffles locally. Readers of container formats (TFRecord, MXNet, Caffe, Caffe2, WebDataset) read the samples in the order in which they are stored, which means a dataset sorted by label comes out mostly sorted with the default buffer. #3088 ran into this when switching from the file reader to TFRecord, and the reply there suggested proposing wording forloader.cc.This adds a note to the
random_shuffleandinitial_fillarguments, and two paragraphs to the Shuffling section added in #6484. They also explain why the file-based readers are not affected: they shuffle the whole file list once whenrandom_shuffleis set.To check the wording against real behavior, I read a TFRecord with 20 classes of 1000 samples each, stored class after class, using DALI 2.3.0 on CPU with a batch size of 256 (
loader.hhas not changed since that release). With the defaultinitial_fill=1024, the first batch contained 2 of the 20 classes, on average 80% of a batch came from just two classes, and no sample was returned more than 1023 positions ahead of its stored position. Withinitial_fillequal to the dataset size, every batch contained all 20 classes.Additional information:
Affected modules and functionalities:
Documentation only: the
random_shuffleandinitial_fillargument docs in theLoaderBaseschema, which all readers share, anddocs/advanced_topics_sharding.rst.Key points relevant for the review:
The list of readers that read in the stored order. I went through the loaders: every list-based one (file, numpy, fits, coco, sequence, nemo_asr, video) shuffles its whole list once under
if (shuffle_), while the indexed-file (TFRecord), RecordIO, LMDB and WebDataset loaders do not.Tests:
Checklist
Documentation
DALI team only
Requirements
REQ IDs: N/A
JIRA TASK: N/A