Today I learnt what grokking means!
Absolutely amazing, by the way. Totally unexpected behaviour, having trained a few.
In 2022, OpenAI researchers found something that broke every rule of machine learning.
Their tiny model trained for 10,000 epochs. It learned absolutely nothing. Validation accuracy was dead stuck at 50%.
Then at epoch 12,000, without warning, it jumped to 99%.
This phenomenon
