When doing QA each start end logit combination is checked for validity based on multiple conditions.
One bad side effect of this: different runtimes for "relevant" vs "irrelevant" questions
We started vectorizing these checks (they were slow if conditions before) but did not finish completely. E.g. see todo comment here: https://github.com/deepset-ai/FARM/blob/master/farm/modeling/prediction_head.py#L1075