Training needs much more memory than evaluation, since there are no gradients.

We need to have an optional eval batch size that should ideally be larger than training. Especially for large models with low training batch size eval becomes much slower (on the order of 5-10x).