In the past three years, I have written 23 posts on AI safety for my research newsletter. This November, I plan to write closer to 30. Why?

As far as I can remember, I always wanted to be some sort of writer; at 9 I started writing some kind of novel. In school, I really enjoyed reading and creative writing; but I got sidetracked by other pursuits.

When I started my PhD, I immediately started a newsletter, first on Twitter (now X) and then on Substack. And I have had many ideas for posts and papers alike; but it is difficult to take these ideas and turn them into a finished product consistently.

As with many things in life that I found difficult, I find that the only way out is through: bulldoze through issues, and force myself to do the difficult thing in much larger quantities than I would like to.

So I am now at Inkhaven, and they will kick me out if I don’t post something that makes sense every day for the month of November.

Scott Alexander, one of the most well-acclaimed writers of the AI world of today, says:

Whenever I see a new person who blogs every day, it’s very rare that that never goes anywhere or they don’t get good.1

My hope is that if I survive this program, it will be a phase transition in my life where writing is going to become a lot easier in general.

Writing and ideation are good skills to develop for the centaur age

In research, I am guided by the belief that AI agents are going to progress faster on verifiable tasks such as coding and math than on fuzzy tasks such as ideation, critique, and project vision. Many ideas are better as well-thought-out Google Docs than research papers, unless there are reasons to do the experiments sooner rather than later.

While pretraining did wonders for writing, as Ilya said in the NeurIPS keynote, “we have but one Internet”, the amount of juice we get out of each token is not increasing fast, and current post-training efforts are by all accounts focused more on tasks such as coding and math than on writing.

LLMs still kind of suck at novel intellectual writing and ideation and perhaps, maybe, they will not improve that soon. This is not the case for software engineering or mathematics, which are my other fields of expertise. We have verifiers there, and quite soon every task with a verifier and a suitable curriculum is going to be quite easy to RL on. The labs don’t mess around. Open-source might catch up eventually.

Writing and coding are both about getting ideas into executable form. Code executes on computers; good writing executes through people2, and the max upside of good write-ups is larger than the upside of any technical research that can be done in the same amount of time.

I don’t mean to say engineering will not have value or that I will stop doing it, far from it. Rather, what I mean is that, due to the rapid decrease in the time costs of writing code, the value of original and clear writing complementary to engineering is going to increase.

How I feel about Inkhaven so far

My desire to do research has been rejuvenated.

The world is full of all these things you can explore; and I have been blessed to work with the rapidly-improving world of LLMs, where you can easily figure out an important thing no one has written about or run experiments for yet.

Throughout my PhD, I have accumulated many ideas that deserve being written down, but have never been really expanded upon. How did they come about?

• me reading way too many papers and taking notes / my own thinking about other people’s research;

• a five-digit number of chats with LLMs;

• conversations with people in the lab, especially Florian;

• experiments or research ideas that never really went anywhere;

• and finally, my own published research.

That’s many ideas! I feel everyone’s PhD deserves a month of getting a fraction of these ideas out of their heads.

Originally I planned to post roughly all writing on my newsletter; but it seems that my audience is there for my fleshed-out takes on AI safety research, and not to be bombarded with every single idea I have. Hence, I started the Random Features newsletter; I needed an outlet for my lower-stakes writing anyway.

On Day 1 I wrote a proposal for a user retention/engagement eval for LLMs; I feel good about it, and I am making progress on a post on red-teaming philosophy today.

Notes from Days 1 and 2 of Inkhaven

There is no place in the world like Lighthaven. This time I am staying on campus.

The Inklings crowd is nice. Most are significantly more experienced writers than me. There are other AI safety researchers around, but not many; the crowd writes about quite a diverse set of topics. Ben Pace sets a great vibe for the whole group through the all-hands meetings every single lunch and dinner.

I saw Gwern Branwen for the first time in person on Friday; he wasn’t busy, but I didn’t want to bother him. His writing influenced my decisions to go and do AI in general (and AI safety later); and has helped out with a paper of mine previously. Somehow I knew right away it was him, even though I had never seen him before.

At the intro session, Gwern gave a bunch of advice on how to find things to write about (paraphrasing from him):

1. You should be constantly trying to predict things to yourself and, every time you are wrong, ask yourself why? There are interesting things in everything you read or observe, but only if you are thinking about them to the point of being surprised.

2. People are constantly saying interesting things and good starting points for a post. The problem is that they immediately forget everything. Resolve to never let a good question or a rant go to waste; just write it down.

3. Rule of three: if you have explained the same thing 3 times, it is time to write that down. It is clearly interesting enough to keep going back to.

Scott Alexander is also around. I chatted with one half3 of Slime Mold Time Mold, and the teeth bacteria guy. Lots of stuff to do!

Predicting how well I will do

Following the trends, I made some Manifold markets for others to express confidence (or the lack of it) in my writing streak. I did research on forecasting, but have never really used prediction markets for anything about my own life.

Feel free to bet! For now, I plan to keep writing until I write all of my good AI research thoughts down. If you want to hear more updates on off days, now or in the future, subscribe to Random Features.