\n“Claude Code running in this mode genuinely feels like a completely different product from regular, default Claude Code.” \u2014 Simon Willison on YOLO mode<\/a><\/p>\n<\/blockquote>\n
He’s right. But these tools can also destroy things<\/a>. So the first thing I did was look into containerizing it.<\/p>\n
Anthropic includes a
Dockerfile<\/code><\/a> for Claude Code. I forked it into a script calledrobotcage<\/code><\/a> that spins up Claude Code in YOLO with only the current folder mounted and network access locked down. It can only destroy what’s in that folder, and it can’t exfiltrate anything.<\/p>\nIn practice, this approach has too much friction:<\/p>\n
\n
- Claude often needs network access for legitimate purposes. I added a dynamic whitelist, but since Claude can run it, it defeats the purpose.<\/li>\n
- The mounted filesystem differs from my native one. Claude would misinterpret pasted commands, or node modules and uv environments would drift out of sync.<\/li>\n
- I’d often need access to folders outside the Docker root, which means running from a shared ancestor, which, again, defeats the purpose.<\/li>\n<\/ul>\n
These frictions added up. I’ve moved away from containerization and now run Claude Code natively on a remote machine with everything version controlled; not bulletproof, but the best mitigation I’ve found.<\/p>\n
One thing I want to keep from this experiment: a dedicated Claude profile with its own SSH keys and Github account. Git blame tells me instantly whether I wrote something or Claude did. When I’m reviewing old code and wondering why something is the way it is, I instantly know: if it’s Claude, I scrutinize differently. PRs show authorship the same way. It also separates credentials. Claude’s SSH keys can be revoked without touching mine. If something goes wrong, the blast radius is contained.<\/p>\n
Embracing Undo<\/h2>\n
The common thread across the next few sections: I stopped trying to prevent damage and started embracing recovery. Assume some damage will happen and make it cheap to undo.<\/p>\n
Notes as Artifacts<\/h3>\n
I use Claude Code to capture notes, plans, and decisions as markdown files in a
notes\/<\/code> folder.<\/p>\nI don’t want to commit these files because a lot of the text is transient, throwaway, or slop. But they’re still important to track. I had my own version of catastrophic data loss in December when I asked Claude to “clean up” a folder and it deleted a month’s worth of notes. I hadn’t realized how valuable they were until they were gone.<\/p>\n
I wanted something like infinite undo, or a git repo that auto-committed on every change. Claude helped me set up a
btrfs<\/code> file system with once-a-minute snapshots. Lightweight, cheap to store, easy to restore.<\/p>\nNotes stay colocated with projects, backed up but not committed. What’s missing: semantic understanding of what a change is.<\/p>\n
Chief of Staff<\/h2>\n
I have a “Chief of Staff” agent: a folder with
claude.md<\/code> and some skills to help organize my projects.<\/p>\nMost Claude Code users are probably familiar with plan vs execute mode, but this serves as a sort of “meta” planning mode across<\/em> projects.<\/p>\n
The value here is cross-project synthesis. It surfaces patterns across projects that I’ve missed; it also provides a single interface to pick off tasks and plan work across projects.<\/p>\n
Coding Style<\/h2>\n
I started out as an incredibly pedantic reviewer with an incredibly pedantic workflow: an issue for each feature, a PR for each issue, detailed comments on every minor thing. I would never<\/em> be this pedantic with a human colleague. I was also strict about which technologies Claude could use.<\/p>\n
Over time I’ve loosened the reins in two ways.<\/p>\n
Technology Choice<\/h3>\n
I’m letting Claude gravitate towards technologies it knows best. React instead of web components, for example, even though I’d prefer the latter.<\/p>\n
My hypothesis: LLMs perform best where training data is richest. I don’t have evidence. It feels true. When Claude struggles with a technology I prefer, I ask whether the battle is worth fighting. Usually no.<\/p>\n
Review Cycle<\/h3>\n
The pedantic cycle costs a lot of time and tokens. Reviewing a PR might take hours of back and forth. Claude isn’t good at extrapolating: if I ask it to separate 3 functions into files, it won’t apply that to similar code elsewhere in the same PR.<\/p>\n
Is it necessary? I want my code a certain way, but with a robust test suite, who cares what’s on the inside?<\/p>\n
This points to preference learning, or evals. More on that later.<\/p>\n
Not Just Code<\/h2>\n
I run Librechat<\/a> locally: Anthropic, Google, and OpenAI in a single chat interface. I’m pretty comfortable throwing a lot of my personal life at an LLM. But I’m surprised by how much more<\/em> I’m throwing at it. I was wary of uploading personal information, but the benefits are so immense I couldn’t help myself.<\/p>\n
The trend here is clear: LLMs are surprisingly capable outside of code. I’ve used it for tax strategy, legal fine print, financial planning. Research that would take hours and cost hundreds in professional fees now takes minutes.<\/p>\n
And Claude Code specifically is a step up from how I was using LLMs before. More powerful models combined with tool use and file system access fit into my workflow more seamlessly than a web UI ever could.<\/p>\n
A Loosening of the Cages<\/h2>\n
This is meant to be a snapshot in time. I’m far from the promised land of knowing how to work effectively with agents. But I can draw some conclusions:<\/p>\n
One, don’t swim upstream. I started out exerting a high degree of control that loosened over time. Instead of running Claude in a container, I embraced undo. Instead of being a pedantic reviewer, I let it do its thing. Give it rope, but protect yourself with good rollback.<\/p>\n
Two, my personal ambition has increased significantly. The range of projects I am considering has ballooned beyond what would have been reasonable last year, and it’s because projects that would’ve otherwise taken 2 weeks now take a day.<\/p>\n
Three, interact to learn. By using Claude Code heavily I’m learning how to work with agents, but also finding that a tighter interactive loop helps when designing software. More akin to play. More akin to a REPL.<\/p>\n
Four, the capability threshold has shifted. If you still believe LLMs are incapable of doing X, where X is some large component of your job, take the latest generation out for a spin. My priors from six months ago are already stale.<\/p>\n
Further Reading<\/h2>\n
Others are reaching similar conclusions:<\/p>\n
Thoughts on Claude Code<\/a><\/strong> (Slava Akhmechet) \u2014 Built a programming language in two weeks; the most detailed practitioner account I’ve found.<\/p>\n
When AI Writes Almost All Code<\/a><\/strong> (Gergely Orosz) \u2014 Frames recent models as crossing a capability threshold.<\/p>\n
Opus 4.5 is not the normal AI agent experience<\/a><\/strong> (Hacker News) \u2014 Practitioner thread on mature Claude setups: custom skills, code review agents, automated workflows.<\/p>\n
I don’t know what being a developer means anymore. I’m figuring it out in real time.<\/p>"},{"title":"LLMs running in the browser","link":"https:\/\/thekevinscott.com\/llms-in-the-browser\/","pubDate":"Fri, 01 Mar 2024 10:00:00 +0000","guid":"https:\/\/thekevinscott.com\/llms-in-the-browser\/","description":"
This post explores various ways to run Language Models (LLMs) in the browser. I will:<\/p>\n
\n
- Review the available frameworks today, and choose the most compelling options<\/li>\n
- Implement a working example for each framework<\/li>\n
- Evaluate them both quantitatively (specifically, speed) and qualitatively (documentation, ease-of-use, etc.)<\/li>\n<\/ol>\n
This is written spring of 2024 and things move quickly; consider this post a snapshot in time.<\/p>\n
Why<\/h2>\n
Before we jump in, you might wonder why<\/em> someone would want to run an LLM in the browser.<\/p>\n
For me, there’s three reasons:<\/p>\n
\n
- Data privacy<\/strong> - keep your data local<\/li>\n
- Latency<\/strong> - avoid round trips to the server (also can support offline mode)<\/li>\n
- Cost<\/strong> - run models locally, save money. This applies both to the user (avoid paying for an API) and operators (run models on the users’ devices and avoid server inference costs)<\/li>\n<\/ul>\n
One very large drawback is that LLMs tend to be huge, measuring from hundreds of MBs to GBs. If your users are on, say, a mobile data plan, a downloading an LLM won’t be a feasible solution. However, for desktop users with robust data connections, including electron apps and Node apps, browser-based LLMs remain a viable approach.<\/p>\n
1. Available options<\/h2>\n
Available options for running LLMs in the browser I’ve found include:<\/p>\n
\n
- Transformers.js<\/a> - a Hugging Face library built on top of ONNX runtime<\/li>\n
- web-llm<\/a> - put out by the MLC team, and running on top of Apache TVM Unity<\/a><\/li>\n
- Candle<\/a> - another Hugging Face library, this one in Rust and converting to WASM<\/li>\n
- Burn<\/a> - another Rust library compiling to WGPU (and Candle)<\/li>\n
- Tensorflow.js<\/a> - a Google library for doing machine learning in Javascript, with CPU, WebGL, WASM, and WebGPU backends<\/li>\n
- ONNX<\/a> - Microsoft library, supports WebGL, WebGPU, and WASM<\/li>\n<\/ul>\n
I chose to evaluate Transformers.js, web-llm, and Candle, and discarded the rest for the following reasons:<\/p>\n
\n
- Burn leverages the Candle backend so an evaluation of Candle should cover both (and Candle offers specific LLM examples).<\/li>\n
- ONNX is leveraged by Transformers.js, so an evaluation of Transformers.js should have implications for ONNX (and Transformers.js offers specific LLM examples)<\/li>\n
- Tensorflow.js doesn’t seem to offer much in the way of cutting edge LLM support, though it’s been around for a long time and boasts great support. I’m keeping my eyes on TFJS in the hopes that cutting-edge LLMs get support soon.<\/li>\n<\/ul>\n
2. Implementations<\/h2>\n
Transformers.js<\/h3>\n
Transformers.js<\/a> is a Javascript library offered by Hugging Face, and is “designed to be functionally equivalent to Hugging Face’s transformers<\/a> python library”.
transformers<\/code> is a key part of the Python ecosystem and often used for running LLMs, so feature parity is a huge plus. In addition to text,Transformers.js<\/code> support vision, audio, and multimodal tasks, so it’s a great choice for supporting a wide variety of modalities.<\/p>\nTransformers.js<\/a> offers an NPM package at
@xenova\/transformers<\/code>. Models can be loaded and used with something like<\/a>:<\/p>\nconst generator = await pipeline('text-generation', 'Xenova\/distilgpt2');\nconst text = 'I enjoy walking with my cute dog,';\nconst output = await generator(text);\n<\/code><\/pre>\nI set up a demo here<\/a>.<\/p>\n
Web-LLM<\/h3>\n
web-llm<\/a> runs on top of TVM. An NPM package is available at
@mlc-ai\/web-llm<\/code>. Models can be loaded with:<\/p>\nconst chat = new webllm.ChatModule();\nawait chat.reload('Phi1.5-q0f16', undefined, {\n"model_list": [\n\/\/ Phi-1.5\n{\n"model_url": "https:\/\/huggingface.co\/mlc-ai\/phi-1_5-q0f16-MLC\/resolve\/main\/",\n"local_id": "Phi1.5-q0f16",\n"model_lib_url": "https:\/\/raw.githubusercontent.com\/mlc-ai\/binary-mlc-llm-libs\/main\/phi-1_5\/phi-1_5-q0f16-ctx2k-webgpu.wasm",\n"vram_required_MB": 5818.09,\n"low_resource_required": false,\n"required_features": ["shader-f16"],\n},\n],\n});\nconst reply = await chat.generate(prompt);\n<\/code><\/pre>\nI set up a demo here<\/a>.<\/p>\n
Candle<\/h3>\n
Candle<\/a> is a Rust-based framework that compiles models into WASM. Like Transformers.js, this is a Hugging Face library, and boasts a larger mandate than just LLMs (namely, supporting training, any model, and a number of compilation targets).<\/p>\n
Candle requires having Rust installed<\/a> and does not offer an NPM package. Once you’ve got the repository cloned and installed, you can run:<\/p>\n
sh build-lib.sh\n<\/code><\/pre>\nto generate the WASM build outputs. Then a model can be loaded and run with:<\/p>\n
import init, { Model } from ".\/build\/m.js";\nawait init();\nconst model = {\nbase_url:\n"https:\/\/huggingface.co\/lmz\/candle-quantized-phi\/resolve\/main\/",\nmodel: "model-q4k.gguf",\ntokenizer: "tokenizer.json",\nconfig: "phi-1_5.json",\nquantized: true,\nseq_len: 2048,\nsize: "800 MB",\n}\nconst weightsURL =\nmodel.model instanceof Array\n? model.model.map((m) => model.base_url + m)\n: model.base_url + model.model;\nconst tokenizerURL = model.base_url + model.tokenizer;\nconst configURL = model.base_url + model.config;\nconst [weightsArrayU8, tokenizerArrayU8, configArrayU8] =\nawait Promise.all([\n\/\/ see demo for these implementations\nconcatenateArrayBuffers(([] as string[]).concat(weightsURL)),\nfetchArrayBuffer(tokenizerURL),\nfetchArrayBuffer(configURL),\n]);\nconst model = new Model(\nweightsArrayU8,\ntokenizerArrayU8,\nconfigArrayU8,\nquantized\n);\nconst firstToken = model.init_with_prompt(\nprompt,\ntemp,\ntop_p,\nrepeatPenalty,\n64,\nBigInt(seed)\n);\nconst seq_len = 2048;\nlet sentence = firstToken;\nlet maxTokens = maxSeqLen ? maxSeqLen : seq_len - prompt.length - 1;\nlet startTime = performance.now();\nlet tokensCount = 0;\nwhile (tokensCount < maxTokens) {\nawait new Promise(async (resolve) => {\nconst token = await model.next_token();\nif (token === "<|endoftext|>") {\nreturn;\n}\nsentence += token;\n});\ntokensCount++;\n}\n<\/code><\/pre>\nI set up a demo here<\/a>.<\/p>\n
3. Evaluations<\/h2>\n
Qualitative Evaluations<\/h3>\n
Transformers.js<\/h4>\n
Transformers.js is really fantastic. It’s simple to get up and running and the code is concise and readable. I’m a big fan of how easily models can be loaded directly from Hugging Face (though models do need to be converted before-hand for the library; Xenova maintains a number of LLMs).<\/p>\n
On the cons side, I found some of the Typescript definitions to be out of date or missing, and the documentation (particularly around callbacks) to be lacking. The other big con is that a large number of models are broken in the browser (including Phi-2) due to an upstream issue with ONNX. The author has a PR fixing it here<\/a> so hopefully this will be fixed soon.<\/p>\n
ONNX, the backend on top of which Tranformers.js runs, supports both CPU and GPU processing<\/a>:<\/p>\n
\nWith onnxruntime-web, you have the option to use webgl or webgpu for GPU processing, and WebAssembly (wasm, alias to cpu) for CPU processing. All ONNX operators are supported by WASM but only a subset are currently supported by WebGL and WebGPU.<\/p>\n<\/blockquote>\n
Web-LLM<\/h4>\n
The output of web-llm are rock solid and its Typescript definitions are great.<\/p>\n
However, I found the code to be overly prescriptive, almost as if specifically designed to support the demo. For example, the parameters for specifying a model configuration are strangely constructed. It was difficult to strip away the demo code to a minimal running example.<\/p>\n
I believe<\/em> web-llm only supports webgpu; it’s not clear to me whether this project supports CPU-only<\/a>.<\/p>\n
Candle<\/h4>\n
Candle is written in Rust. This may or may not be a dealbreaker. You’ll need a Rust-compatible build step, a way to check in WASM artifacts, and more.<\/p>\n
What I find compelling about this compilation-based approach is the resulting size of the models; Burn (another Rust-written framework) writes<\/a>:<\/p>\n
\nThe main differentiating factor of this example’s approach (compiling rust model into wasm) and other popular tools, such as TensorFlow.js, ONNX Runtime JS and TVM Web is the absence of runtime code. The rust compiler optimizes and includes only used burn routines. 1,509,747 bytes out of Wasm’s 1,866,491 byte file is the model’s parameters. The rest of 356,744 bytes contain all the code (including burn’s nn components, the data deserialization library, and math operations).<\/p>\n<\/blockquote>\n
The code for loading a WASM model is considerably more verbose than the other two solutions on offer, and you must host and load the WASM files yourself.<\/p>\n
Candle does not yet support WebGPU<\/a> but it seems to be in active development. It also<\/em> sounds like the Transformers.js maintainer<\/a> will be adding support for Candle as a backend as soon as that lands, so it seems like these two projects will share compatibility, which makes sense since they’re both Hugging Face projects.<\/p>\n
Quantitative Evaluations<\/h3>\n
For each set of measurements, I tried to target the same Phi 1.5 model, for a max of 128 tokens. I ran each evaluation three times and averaged the results. I ran on Chrome on a Macbook M3. I’m measuring model loading from cache (not<\/em> over the network).<\/p>\n
Transformers.js<\/h4>\n
I would love to evaluate the model
Xenova\/phi-1_5_dev<\/code>; however, due to a bug<\/a> that’s broken in the browser, so I won’t include generating time measurements here.<\/p>\n
![\"UpscalerJS\"](\".\/images\/demo.gif\")
<\/p>\n![\"A](\".\/images\/dog-150.jpg\")
<\/a>\n![\"The](\".\/images\/dog-300-pixelated.png\"){kind=tracking}
<\/a>\n<\/div>\n![\"The](\".\/images\/dog-300-bicubic.png\")
<\/a>\n<\/div>\n![\"The](\".\/images\/dog-300-gan.png\")
<\/a>\n<\/div>\n