I realised that packets with big payloads get lost after they get fragmented by sixlowpan.
Reproduce:
Ping from a pba-d-01-kw2x board to a samr21-xpro board:
ping6 1 beef::1 59 0 will work
ping6 1 beef::1 60 0 will fail (fragmented)

Surprisingly, enabling DEBUG output of sixlowpan "fixes" that error, so it seems to be a race or timing related thing in the driver.