Skip to content

Commit ab58267

Browse files
Clarify shuffling of container datasets
Signed-off-by: Max Freedom Pollard <[email protected]>
1 parent b20195c commit ab58267

2 files changed

Lines changed: 8 additions & 4 deletions

File tree

‎dali/operators/reader/loader/loader.cc‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -29,7 +29,9 @@ and then samples are selected randomly to form a batch.
2929
.. note::
3030
The buffer shuffles the data only locally. When it is much smaller than the dataset, samples
3131
that are read far apart rarely end up in the same batch. Readers of container formats, such as
32-
TFRecord, read the samples in the order in which they are stored. If that order is meaningful,
32+
TFRecord, read the samples in their stored order. Such datasets are often shuffled during
33+
creation. Some readers also support shuffling the order of container files between epochs,
34+
without reordering the samples within each file. If the stored sample order is meaningful,
3335
for example the samples are sorted by label, shuffle the dataset when you create it or increase
3436
`initial_fill`.
3537

‎docs/advanced_topics_sharding.rst‎

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -47,9 +47,11 @@ when ``random_shuffle`` is enabled, so the order of the files does not matter. R
4747
formats, such as ``fn.readers.tfrecord``, ``fn.readers.mxnet``, ``fn.readers.caffe``,
4848
``fn.readers.caffe2``, and ``fn.readers.webdataset``, read the samples in the order in which they
4949
are stored. Where such a reader supports ``shuffle_after_epoch``, it changes only the order of the
50-
files. With these readers, shuffle the dataset when you create it or increase ``initial_fill``.
51-
A larger buffer uses more memory and delays the first batch, and a buffer as large as the shard
52-
read by the pipeline shuffles that shard completely.
50+
files, preserving the sample order within each file. Datasets stored in these formats are often
51+
shuffled during creation, so their stored sample order may already be random. If it is meaningful,
52+
shuffle the dataset when you create it or increase ``initial_fill``. A larger buffer uses more
53+
memory and delays the first batch, and a buffer as large as the shard read by the pipeline shuffles
54+
that shard completely.
5355

5456
Framework iterator configuration
5557
--------------------------------

0 commit comments

Comments
 (0)