If you’re bored with today’s neural net architectures, then my advice to you is to start training models to use an external memory, now that RL is finally working.
The normalization scheme that DeepMind researchers came up with for their "linear recurrent unit" (LRU) is a nice example of how it is possible to predictably engineer circuits in artificial neural networks, when you know what you're doing. A thread:
Just read their paper. Looks like they re-invented an existing method known as context distillation (or merely re-branded it for their startup). No mention of prior work, sadly. Links to papers in thread.
Announcing Bread Technologies. We’re building machines that learn like humans.
We raised a $5 million seed round led by Menlo Ventures and have been building in stealth for 10 months.
Today, we rise 🍞
Before you say “this isn’t surprising”…
Yes, it is. We got people to preregister their expectations, and even folks who are extremely in-the-know about AI coding abilities still failed to predict this result.
Your *vibes* are not reliable indicators of productivity effects.
We ran a randomized controlled trial to see how much AI coding tools speed up experienced open-source developers.
The results surprised us: Developers thought they were 20% faster with AI tools, but they were actually 19% slower when they had access to AI than when they didn't.
YES! If you initialize a LoRA layer based on the SVD of the original weight matrix (with its top singular values & vectors), you get significantly better fine-tuning results.
This is a straight-up free lunch, as far as I can tell.
PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models
Significantly improved finetuned perf by simply changing the initialization of LoRA's AB matrix from Gaussian/zero to principal components of W
repo: github.com/GraphPKU/PiSSA
abs:
What excites me most about the rising tide of RNNs/SSMs is that it could let the fields of machine learning and computational neuroscience use the same modeling tools.
Note: sparse coding is an *established* method for disentangling representations. Anthropic did not invent it, nor did they claim to. If their new results seem surprising, now's a great time to revisit the older literature (Olshausen, Kanerva, etc.).
Wow! Papers from two different teams—one from academia and one from Google DeepMind—with the same finding: linear recurrence + local (sliding window) attention is your best bet if you want an efficient alternative to global attention.
Simple linear attention language models balance the recall-throughput tradeoff
Recent work has shown that attention-based language models excel at recall, the ability to ground generations in tokens previously seen in context. However, the efficiency of attention-based models is
Stability changed the name of these models to "Stable Beluga 1/2" and quietly removed the sentence of the blog post that mentioned they used two unnamed LLMs to generate their dataset. (This likely means they used OpenAI models, in clear violation of ToS)
web.archive.org/web/2023072122…
We put the model in a test and then we steer the model away from thinking “I am in a test” and then we steer the model away from introspecting “I am being steered away from thinking «I am in a test»”
Problem: AIs can detect when they are being tested and fake good behavior.
Can we suppress the “I’m being tested” concept & make them act normally?
Yes! In a new paper, we show that subtracting this concept vector can elicit real-world behavior even when normal prompting fails.
A short thread of news 🧵 (1/3)
I’ve joined the policy team at METR!
Rapid AI changes will require measuring & addressing potential new threats far more quickly.
![Goose chasing man meme, with "Which LLMs did you generate those examples with?... Which LLMs generated your dataset, [blurred expletive]?!" and a screenshot from the blog post this is responding to, with a quote displayed reading "With this approach, we generated 500,000 examples with one simpler LLM model and an additional 100,000 with a more sophisticated LLM model."](https://pbs.twimg.com/media/F1ltNlhaYAQ_PZj?format=webp&name=medium)
