The more I learn the more I realize everything in ML is copied from physics
- I wrote a blog post for an introduction to stochastic calculus! I share my perspective and intuition behind Brownian motion, the connections between random walks, Binomial and normal distributions, or Itô and Stratonovich calculus. It includes some visualizations too!
- Wow, Qwen web search led me to some Chinese websites, and there is real gold mine of information on machine learning
- I discovered this very cool application called Obsidian, it's very good for writing math, especially if you install some plugins that can auto-replace text
- I stumbled upon a highly underrated paper It studies the dimensionality self-attention geometry via algebraic methods, focusing on a simplified version without softmax but also a short section with it at the end, they have some conjectures if anyone will can prove it
- I love @bremen79’s explanations on optimization theory, I am inspired to learn more about it. I highly recommend watching his content, lectures and blog posts.
- ML is only hard because people obfuscate, overcomplicate and overengineer everythingPrompt engineering is a real profession because in 2024, prompt engineering can look like this
- I got recommended Terence Tao's YouTube channel created in 2010, where he uploaded his first video just yesterday! He showcases his process of formalizing a proof in Lean 4 with the help of GitHub Copilot and the "canonical" tactic in Lean.
- I suspect LLMs are secretly running on complex numbers and we are not exploiting this enoughReplying to @Ji_Ha_KimAn additional note: complex numbers can be used to represent rotations very simply. Intuitively, RoPE is coupling every 2 consecutive dimensions of the embedding space. Let q = [x1,y1,x2,y2,…] then we update with e^ik(x+iy)=(x cos k - y sin k) + i(x sin k + y cos k), k=ptheta
- Very cool fact: the FFT is based on sampling the roots of unity smartly, typically on C. But it actually works for any field! For example, multiplication of very big integers often uses something called "number-theoretic transform", basically FFT on the finite field Z/pZAt MIT, the only course I ever dropped was signal processing. The DFT math was too intimidating. It’s so easy to just type fft() in MATLAB and move on. Years later, I finally did DFT by hand. ✍️ If you are also afraid of DFT, I hope this helps! ⬇️ Download:
- It’s interesting that transformer dot product self-attention has d_k^2 redundant degrees of freedom. Since setting Q’=QA, K’=K(A^-1)^T gives Q’K’^T=QAA^-1 K^T = QK^T So for d_k = 64 there are 64^2=4096 redundant dimensions since there are dim(GL(n))=n^2 invertible n x n matrices
- The inventor of muTransfer, muP, LoRA and GFlowNets also worked on o1’s inference time reasoning. He’s totally unstoppable. It seems like he recently left OpenAI to co-found a stealth AI company in Woodside, California. Definitely something to follow closelyproud to see what i worked on at OpenAI finally shipped! go 🐢!!
- The theory of operators is very interesting. Another cool trick: eigen-decomposition of the transpose T, since T^2-I=0, λ=±1. Then projectors P₊=(I+T)/2, P₋=(I-T)/2 decompose into symmetric/antisymmetric parts with A=(A+A^T)/2 + (A-A^T)/2An addendum to @3blue1brown’s video, here is the linear algebra proof behind *why* Laplace transform works. For the ML folks who follow me, you can think of Laplace as a PCA on your ODE, where the “principal components” are the values of s at which your transform has a pole
