Shibani Santurkar (@ShibaniSan) / X

Does language supervision (as in CLIP) help vision models transfer better? You might expect a clear-cut answer: 'captions always help' or 'not at all'. But w/ @yanndubs @rtaori13 @percyliang @tatsu_hashimoto, we find that the picture is nuanced.🧵 arxiv.org/abs/2207.07635

❤️

i love the openai team so much

💙💙💙💙💙💙💙

We have reached an agreement in principle for Sam Altman to return to OpenAI as CEO with a new initial board of Bret Taylor (Chair), Larry Summers, and Adam D'Angelo. We are collaborating to figure out the details. Thank you so much for your patience through this.

Auto data selection is comparable to expert curated data for pretraining LMs! The leverage: n-gram overlap between pretrain and downstream predicts downstream acc well (r=0.89). But it's not the whole story - lots to uncover on the effect of pretrain data on downstream tasks.

Sang Michael Xie

@sangmichaelxie

Feb 8, 2023

Replying to @sangmichaelxie

Data selection typically involves filtering a large source of raw data towards some desired target distribution, whether it's high-quality/formal text (e.g., Wikipedia + books) for general-domain LMs like GPT-3 or domain-specific data for specialized LMs like Codex.

💛

I deeply regret my participation in the board's actions. I never intended to harm OpenAI. I love everything we've built together and I will do everything I can to reunite the company.

Come talk to us at our NeurIPS poster from 8:30-10am PT today (now) at spot A2!

Aleksander Madry

@aleks_madry

Dec 3, 2021

Can we perform surgery on the prediction rules of an already trained classifier? It turns out yes (and with only a single example too!) with @ShibaniSan, @tsiprasd, Mahi Elango, David Bau, and Antonio Torralba Paper: arxiv.org/abs/2112.01008 Blog post: gradientscience.org/editing/

So proud!

Congratulations, @tsiprasd ! Extremely well deserved—it was an honor to be a (small) part of your (now, honorable ;) PhD journey.

Jun 16, 2021

Replying to @aleks_madry and @zacharylipton

It's been a blast! Thank you for being an incredible advisor @aleks_madry

🚢 🚢 🚢

ChatGPT with voice is now available to all free users. Download the app on your phone and tap the headphones icon to start a conversation. Sound on 🔊

00:00

Replying to @ShibaniSan

Based on our findings, we design simple interventions to improve CLIP’s ability to leverage web-scraped captions: by filtering them and using GPT-J to perform text data augmentations via paraphrasing.

Replying to @ShibaniSan

(ii) *What is in the caption matters* Given a data budget, CLIP’s performance depends on whether captions directly discuss parts of the image (left) or are complementary to it (right). In fact, one descriptive COCO caption is worth 5x YFCC ones!

Replying to @ShibaniSan

We find that: (i) *Scale is crucial* When the dataset used to train CLIP/SimCLR is fairly large, CLIP >> SimCLR. If not, SimCLR >> CLIP. Also, the transition point between these regimes is dataset dependent (vertical lines).