We realized that in the introduction notebook, the usage examples given for the make_multisource_episode_pipeline did not set the shuffle_buffer_size parameter, which defaults to not shuffling examples within each class.

Two unfortunate consequences we identified in code that would not shuffle examples are:

• Evaluation on the traffic_sign dataset were overly optimistic, since the examples were organized as 30-image sequences of pictures from the same physical sign (successive frames from the same video), leading to support and query examples being more frequently really close.
• Training on small datasets can be worse, since the first examples of a given class would always tend to be support examples, and the later ones would be query examples, reducing the diversity of episodes.

Code using the training loop of Meta-Dataset was not affected, since it gets its shuffle_buffer_size value from a DataConfig object set from a gin configuration that is explicitly passed to Trainer's constructor (in all.gin and imagenet.gin).

We have mitigated the first point by updating the dataset conversion code to shuffle the traffic_sign images once (3512a82), and updated the notebook to show a better practice (c3f62a1), but existing datasets, and code inspired from the notebook (outside of this repository) are still impacted.

Similarly, make_multisource_batch_pipeline does not pass a shuffle_buffer_size, but the impact seems much smaller (batch training should be less sensitive to the order of examples, and the random mixing of different classes adds randomness already).