We have released our DCLM models on huggingface! To our knowledge these are by far the best performing truly open-source models (open data, open weight models, open training code) 1/5
I am really excited to introduce DataComp for Language Models (DCLM), our new testbed for controlled dataset experiments aimed at improving language models. 1/x
I had an argument with @PreetumNakkiran about MLPs 4 years ago. He said with enough data + compute the MLP/ConvNet gap would go to 0. I was convolution-pilled and convinced this wasn't possible. He was right:
It's a multiple choice exam that covers ~57 subjects. It's generally a good benchmark for capabilities of a model. 90% just means the model got 90% of these questions right. The paper is not a terrible read:
Neural Kernels Without Tangents
arxiv.org/abs/2003.02237
Joint work with Alex Fang, @WSguo, Sara Fridovich-Keil, @lschmidt3, @jrk and @beenwrekt
Taking inspiration from convolutional networks, we construct high performance kernel functions for image classification (1/6)
