It is totally possible to implement deterministic training with lua torch avoiding some cunn modules and cudnn algos. cutorch has atomicAdd in THIndexTensor which shouldn't pose a problem.
PyTorch should implement:

• ordered parallel data loading similar to torchnet https://github.com/torchnet/torchnet/blob/master/dataset/paralleldatasetiterator.lua#L53
• cudnn should allow user set deterministic mode somehow