Pinned
First time I've opened up about the Lightning story, military, PhD, open-source and merging companies!
Excited to share the journey with everyone.
youtu.be/EmvrHnjAySg?si…
00:00
William Falcon ⚡️
2,790 posts
William Falcon ⚡️
@williamfalcon
Founder, CEO @LightningAI, PyTorch Lightning⚡, Former USN, AI PhD student (pretraining, researcher) @AIatMeta @CILVRatNYU w @kchonyc @ylecun
- Most people don't know that Lightning Studios offer: - free persistent storage - free persistent environments - unlimited background execution - VSCode, PyCharm, (any IDE) integration Set up your Studio environment once and reuse it again any time 🤯🤯
- I know this is fake because now that @Microsoft is in the mix, GPT numbers will be: - GPT-3 - GPT-4 - GPT-XP - GPT-2000 - GPT-7 - GPT-9 - GPT-10
- In this new 90 minute lecture, I show how to pretrain a 3B LLM from scratch. No edits. No detail skipped. Companies want you to believe pretraining models is super hard and costly. With the right tools, it's not. - We start by tuning the model on a cheap A10G. - Then we scale
- At this point, if you're still "GPU poor" you just haven't started using Lightning Studios... lightning.ai
- Excited to announce the release of bolts! - linear + Logistic regression on TPUs/GPUs - self-supervised learning - RL - GANs - GPT - Callbacks library - Datamodules library All powered by lightning and rigorously tested. Don't waste more time implementing your own baselines...Bolts is a new Deep Learning research and production toolbox from PyTorch Lightning. Iterate faster with pre-trained models, components, callbacks, and data sets, all modular, tested, and optimized for GPUs/TPUs. Simply subclass, override, and train. medium.com/pytorch/pytorc…
- Excited to announce our new compiler - Thunder! (built in collaboration with NVIDIA). 🤯 🤯 Thunder is a source to source compiler for PyTorch. It speeds up PyTorch models. As an example, it speeds up llama 2 7B by 40%. github.com/Lightning-AI/l…
- Excited to announce the launch of GPT-42. - Half the size of GPT-3 (100 billion parameters) - runs on 775 watts a day (2000 calories) - can do one-shot learning - multi-modal - does NOT require 9,000 GPUs - 300,000 years worth of evolution research! Inference API coming soon!
- LLM? old news. New hotness will be LVMs (Large Vision Models). This time, looks like @GoogleAI is finally staying ahead of @OpenAI. How well does it work? a 🧵-> 1/n arxiv.org/pdf/2302.05442…
- Excited to release our latest paper (with @kchonyc) which establishes a conceptual framework for characterizing contrastive learning methods (SimCLR, BYOL, CPC, AMDIM, Swav). (work done at @facebookai) Btw.. this was the motivation for @PyTorchLightnin bit.ly/yadimp
