I diffed the installed package lists for good/bad and I can now repro, the problem comes from the just released datasets==4.0.0 which is the source of this breakage.

if someone needs this to work right away, force datasets==3.6.0

So it's comparing:

def simple_accuracy(preds, labels):
    print(preds, labels)
    print(f"{preds==labels}")
    return float((preds == labels).mean())

data:

Column([1, 0, 0, 1, 1]) Column([1, 0, 0, 1, 0])
False

and fails there as it returns a bool, and bool has no mean

reported it here huggingface/datasets#7679 and huggingface/accelerate#3676

so we can merge the workaround to force datasets==3.6.0 in this PR until upstream fixes it.

update: they are working on fixing the evaluate package.