Pinned
Career update: Iām joining @blocks for the Summer as a DL Research Intern!
Iāll be working on some very cool and novel approaches to generative audio modelling, would be happy to get to know more people in this space!
- An addendum to @3blue1brownās video, here is the linear algebra proof behind *why* Laplace transform works. For the ML folks who follow me, you can think of Laplace as a PCA on your ODE, where the āprincipal componentsā are the values of s at which your transform has a pole
- Hot take: The First Isomorphism Theorem deserves to be called the Fundamental Theorem of Algebra. Itās true for Groups, Rings, Modules, Lie Algebras. It even underpins RankāNullity in Linear Algebra, and quotient spaces in topology. Itās in the DNA of algebraic structure itself
- I absolutely love it when I see people with a strong math background working/doing stuff in ML. Their blogs are always THE BEST If you're like this, please let me know. I'd love to follow you. I'll post a tweet compiling blogs like these some day (and yes I do read them)
- I watched 3B1Bās new vid on the Laplace transform and wanted to say: Laplace transform for ODEs is just the fact that an m-th order ODEās solutions lie in the generalized eigenspace of the ādifferential operatorā d/dt, which is just a consequence of FTA Iāll share a proof tmrw
- Pretty much every piece of coverage on this paper I've seen is a misrepresentation of what it says. For starters, this is a measure-theoretic statement. The probability of something being 0 does NOT mean it never happens. The main result uses a theorem on analytic functions so-
- š„° Really loving this lesser-known gem: a generalization of the Euler Lagrange equations, also known as Euler-Poisson Equation! š Explanation and proof in replies
- As much as I want to do a PhD in deep learning theory, the idea of being a PhD student for 4-5 years and trying to live up to the current "standards" in academia terrifies me I love all the research I've worked on and would wanna do more, but committing to a PhD is v daunting
- š« Lagrange Interpolation š« Given n data points, there is a unique polynomial of degree n+1 that passes through all points! The image below shows the construction of the polynomial š„
- They used an interesting math result here though: If f : R^n -> R is analytic, its zero set is measure 0 This ends up being non-standard (thereās a 4 page arXiv preprint with >200 citations proving it) but I wrote a nice and simple proof that fits in one screenshotPretty much every piece of coverage on this paper I've seen is a misrepresentation of what it says. For starters, this is a measure-theoretic statement. The probability of something being 0 does NOT mean it never happens. The main result uses a theorem on analytic functions so-
- The reason for this is simple: Math is not a spectator sport. There is no free lunch and you learn by thinking through it yourself I always try to work something out myself from a video of his after watching it, this has worked well so farThis post is unavailable.
- Crazy bit of UofT trivia I just found: Ilya Sutskever and the mathematician Jacob Tsimmerman (who solved the Andre-Oort conjecture) were both in the same grad differential topology class in 2005 as undergrads (labeled 37 and 25 respectively)
- I saw this while reading Anthropicās āToy models of superpositionā (transformer-circuits.pub/2022/toy_modelā¦). I was surprised to see J-L mentioned, and their paper shows an analogue of it can be true for NNs! The proof is really a one-liner (āapply J-L from big space to smallā). The details:Pure math Q relevant for LLMs: Unrelated tokens (like "dog" and "sun") should roughly correspond to orthogonal vectors in embedding space. So, In ā^k, whatās the max no. of pairwise orthogonal vectors? what about if they're only "nearly" orthogonal, so ā£viā vjā£<Īµ, ā iā j?
- It's crazy that the key idea behind LoRA is just this. It literally is just exploiting a simple result in linear algebra to give us one of the most effective ways to finetune LLMs
