Alec Radford (@AlecRad) / X

Alec Radford

565 posts

Alec Radford

@AlecRad

San Francisco, CA

Joined October 2012

Pinned
Alec Radford
@AlecRad
Jun 11, 2018
What I've been working on for the past year! blog.openai.com/p/7fa97c36-611… Inspired by CoVE, ELMo, and ULMFiT we show that a single transformer language model can be finetuned to a wide variety of NLP tasks and performs very well with little tuning/tweaking.
Alec Radford
@AlecRad
Feb 11, 2019
The DL CV community is having a "oh wait, bags of local features are a really strong baseline for classification" moment with the BagNet paper. This has always been clear for text classification due to n-gram baselines. It took an embarrassingly long time for nets to beat them.
Alec Radford
@AlecRad
Feb 11, 2019
Replying to @AlecRad
So nets are stubbornly, begrudgingly, moving in the right direction and we're throwing ever larger amounts of compute and data at them and praying it's enough for them to figure out how to do things "the right way". Will that work? Don't know. Probably still worth checking?
Alec Radford
@AlecRad
Dec 9, 2017
If you were stranded on a deserted island and could only bring one dataset to train models on what dataset would you take?
Alec Radford
@AlecRad
Aug 26, 2015
Moar pixels! Samples from 128x128 generative model of ~700K album covers.
Alec Radford
@AlecRad
Feb 11, 2019
Replying to @AlecRad
Spent two frustrating years between 2013 and 2015 banging my head against this. "Hey Alec you just trained an LSTM for three days on 10 million examples using a $1,000 GPU but there's 20 lines of scikit-learn that beats it in 5 minutes on a single CPU core." NOPE NOT BITTER
Alec Radford
@AlecRad
Feb 17, 2019
By the way - I think a valid (if extreme) take on GPT-2 is "lol you need 10,000x the data, 1 billion parameters, and a supercomputer to get current DL models to generalize to Penn Treebank."
Alec Radford
@AlecRad
Apr 25, 2019
This is a really fun live experiment with twitch chat predictably oscillating between love and hate based on the sample.
Alec Radford
@AlecRad
Nov 19, 2018
Nice discussion of the progress in NLU that's happening with BERT, OpenAI GPT, ULMFiT, ELMo, and more covered by @CadeMetz in the @nytimes I'm super excited to see how far this line of research will be able to get in the next few years!
Finally, a Machine That Can Finish Your Sentence (Published 2018)
From nytimes.com
Alec Radford
@AlecRad
Dec 5, 2017
A replication of Learning To Generate Reviews and Discovering Sentiment by Raul Puri at NVIDIA github.com/NVIDIA/sentime…
Alec Radford
@AlecRad
Jan 2, 2017
Happy new year everyone! Since they look a bit like fireworks - here's some pretty features learned by a fully-connected network.
Alec Radford
@AlecRad
May 17, 2017
Replying to @goodfellow_ian and @karpathy
The mode of the "distance of my head to a GPU when home" distribution is less than 2 ft.
Alec Radford
@AlecRad
Apr 17, 2018
Just stumbled across Wikipedia's vital article lists containing the top 10 -> 100 -> 1,000 -> 10,000 most important articles. Very cool and useful resource from a concept learning perspective! Here's a subset of the top 100:
Alec Radford
@AlecRad
Jul 2, 2018
Eye candy for the day: PCA of the position embeddings (colored by pixel height) over the course of training for an autoregressive Transformer modeling pixel-by-pixel Imagenet
GIF