[{"content":"Fable 5 is another step up the abstraction ladder on the long journey from “ai is better autocomplete”. We need to yet again throw out old assumptions about how to work with LLMs. Old prompt techniques that worked well before are now too constrictive. It’s a different animal entirely, and using it well means unlearning some habits.\nHere I’ll describe how I’ve been using it, what works and what still doesn’t.\nWhat it is Fable 5 shipped on June 9, 2026 as Anthropic’s most capable widely released model, a tier that sits above Opus. Fable 5 and Mythos 5 share the same weights and capabilities. Mythos 5 is the model without safety classifiers, offered only to vetted partners through Project Glasswing. Fable 5 is that same model with the classifiers, generally available. Once you know that, a lot of Fable 5’s quirks make sense: you’re using a deliberately fenced-in frontier model.\nSimon Willison, after a day of hands-on testing, was impressed: “the challenge is finding tasks that it can’t do.” The concrete demos would have blown us away a half year ago: a 50-million-line Ruby migration finished in a day, Pok\u00e9mon FireRed beaten from raw screenshots with no helper harness. Those two are from Anthropic’s own announcement.\nStop writing playbooks Every model before Fable 5 rewarded decomposition. You broke a task into steps, you handed the model a recipe, and it followed along more reliably than if you’d just stated the goal.\nFable 5 flips that, at least for open-ended work. Prescribe the reasoning path and you box in a model that reasons better than your recipe does. Constrain the outcome and the boundaries as hard as you like. But stop dictating the route. What it wants is an outcome description:\nThe larger project, and who it’s for What the finished outcome enables The current state, and the relevant files The non-negotiable constraints The goal Then get out of the way. Anthropic’s own prompting guide recommends a system-prompt line that gives it explicit permission to move, to stop it overplanning on ambiguous tasks:\nWhen you have enough information to act, act. Do not re-derive facts already established in the conversation, re-litigate a decision the user has already made, or narrate options you will not pursue in user-facing messages. If you are weighing a choice, give a recommendation, not an exhaustive survey.\nLike previous models though, high-effort Fable 5 tends to over-verify, producing polished work that costs more than it needed to. It helps to tell it not to add features, refactor, or abstract beyond what the task requires. The YAGNI principle has been a mainstay in my main CLAUDE.md for almost a year and remains relevant with Fable.\nWhat “long-horizon” really means Anthropic says Fable is capable of performing “days-long, complex, asynchronous tasks.” In plain terms: give it an agent harness with file-based memory and Fable 5 will run, verify its own work, revise, and hold a coherent thread for hours. That memory isn’t the model, it’s the harness handing it files and telling it to keep notes. A stateless API doesn’t remember anything between calls. What’s new is that Fable 5 uses that scaffolding well. It is much better at deciding what should be recorded, organizing those notes, retrieving them later, and using them without losing sight of the overall goal.\nTwo behaviors I’ve watched it do that older models didn’t:\nIt’s hypothesis-driven. It doesn’t just try fixes until one sticks; it forms a hypothesis, runs something to test it, and revises. If the evidence disagrees, it revises the hypothesis rather than continuing blindly. Reasoning becomes more reliable when it can test beliefs against reality. That’s why it’s strong in an environment where it can execute and observe, and weaker as a one-shot oracle. It runs, then fixes. That same experimental approach can be a double-edged sword. Hand it a subtly wrong command and it tends to execute first and correct after. Something to think about before you give it a production shell. How I actually use it Most of my Fable 5 time isn’t spent watching it write code. Instead, I run it as a coordinator. My current pattern: I hand it a goal (port a feature from one of my products to another, bring one agent box up to parity with another), it searches my notes and past sessions for context, and then it writes the plan out as a document I can actually read and approve before a line of code exists. Once I sign off, it dispatches a fleet of subagents (Opus, or increasingly GPT-5.6) to implement in parallel, and reviews what they hand back before it comes back to me. I wrote the rule into my CLAUDE.md: Fable plans and reviews, the cheaper\/faster models write.\nA Claude Code session used to be a firehose. You sat and watched a wall of code scroll past, because the one agent doing the work was also the only thing you could talk to, so watching it was the job. With Fable coordinating, the implementation runs in the background across a fleet of subagents and the main agent stays free. I can see where the session is at a high level instead of reading every line, and when I think of the next thing that needs doing, I tell the main agent and it gets queued. Nothing blocks on a wall of output finishing. It’s the difference between reading over one engineer’s shoulder and checking in with a lead whose team is heads-down.\nHere’s what’s currently in my CLAUDE.md for how Fable is to be used:\n## Fable 5: Coordinator Role When the session model is Fable 5 (`claude-fable-5`): - **Fable coordinates, Opus codes.** Act as project manager: plan, decompose, dispatch, review, verify. Delegate implementation work to Opus subagents (`Agent` tool with `model: "opus"`). Spend Fable tokens on planning, hard reasoning, and cross-cutting review \u2014 not routine edits ($50\/M output vs Opus). - Exceptions: trivial edits (a few lines) and tight debug loops where handoff overhead exceeds the work \u2014 do those directly. - When writing prompts for Fable (subagents, scheduled runs): use outcome-description format \u2014 larger context, current state, constraints, goal. Step-by-step playbooks written for Opus DEGRADE Fable output. The guardrails Fable 5 ships under Anthropic’s ASL-3 safeguards and its classifiers watch three specific domains: offensive cybersecurity, biology and chemistry, and extraction of the model’s own reasoning (to stop rivals training on it). If your work is ordinary coding, product, and writing, the safety layer is close to invisible for you.\nAnthropic’s prompting guide warns that “benign cybersecurity work and beneficial life sciences tasks may also trigger these safeguards.” If that’s you, don’t fight the classifier, route that work to Sonnet 5 or Opus directly.\nA raw API refusal arrives as HTTP 200 with stop_reason: "refusal" and a stop_details.category. Automatic API retry requires server-side fallback, SDK middleware, or application logic.\nClaude Code handles this automatically by default, but visibly: it shows a switch notice, labels the answer as Opus 4.8, and leaves the conversation on Opus until you switch back.\nFor science, I deliberately triggered the classifier to see what would happen:\nIn my daily work, I have not yet hit a fallback to Opus.\nPrompts or skills that tell the model to echo or explain its own reasoning as output can trip the reasoning-extraction classifier. If you carried “show your thinking” instructions over from an Opus-era setup, audit them out.\nTwo more operational facts that will surprise you:\nMandatory 30-day retention. There is no zero-retention option. If your org enforces ZDR, your calls fail with invalid_request_error until the config is loosened. For regulated workloads, this alone can make Opus 4.8 the right choice. Thinking is always on and never shown. You can’t disable adaptive thinking; the effort parameter (low through max) is your cost lever. And you never get raw chain-of-thought back, only a summary or nothing. What it costs, and when it’s worth it Fable is a premium model that comes at a premium price. The sticker price is $10 \/ $50 per million tokens, double Opus 4.8’s rate and the most Anthropic has ever charged for a general model. But per-token price undersells it, because Fable also thinks more per task (adaptive thinking is always on) and over-verifies at high effort. Artificial Analysis captures the real number with cost-per-task: the total spend to run their full Intelligence Index suite.\nFable 5 with fallback runs $2.75 a task. “Task” here is defined by Artificial Analysis\u2019s specific Intelligence Index workload, weighted across its benchmark categories. GPT-5.6 Sol at max effort is $1.04. Kimi K3 is about $0.95. For that roughly 3x premium, Fable scores 60 on their intelligence index, against Sol’s 59 and Kimi’s 57. You are paying triple for the top of a very tight pack.\nCost per task to run Artificial Analysis’s Intelligence Index, lower is better. Source: Artificial Analysis.\nThat chart is the argument for using Fable as a coordinator instead of an implementer. Let it plan and review while cheaper models write, and you spend the $2.75-a-task intelligence on the fraction of the work that actually needs judgment, and let a Sol or a Kimi K3 burn the tokens on the boilerplate. Pay Fable rates to write a CRUD endpoint and you are lighting money on fire for output a model a third the price gets right.\nUnless your employer is AI pilled enough to pay API rates for Fable, or you have investor money to burn, most of us will be accessing Fable via Claude Code. Anthropic first tried to pull Fable from plans and reserve it for API pricing, extended the cutoff three times as users revolted, and then made it permanent instead. From July 20, Fable is included in Max and Team Premium plans at 50% of your weekly limits, with each Fable token burning that limit at roughly twice the rate of an Opus token. Pro and Team Standard users don’t get it bundled, so they pay through usage credits at the $10\/$50 API rate, somewhat softened by a one-time $100 credit. The reason for that split is the chart above. With Sol and Kimi K3 landing near a third of Fable’s per-task cost, hiding Fable behind an API-only paywall stopped being tenable.\nThree ways to mitigate cost on Fable 5:\nKeep the cache warm. In Claude Code caching is automatic; on a subscription it even uses the one-hour cache for free. What you control is not breaking it: switching model, changing effort, or running \/compact each force the next turn to re-read your whole context at full price, when reads are otherwise 90% off. Pick model and effort at the start of a session and save \/compact for the breaks between tasks. Manual cache_control breakpoints only matter if you’re building your own agent on the API. Tune effort deliberately. Anthropic recommends high as the default, xhigh for the hardest work, and medium or low for routine tasks. Fable 5 at low effort often beats prior models at their max. So drop the effort when a task completes but runs longer than it should, rather than reaching for high reflexively. Mix models. Use Sonnet 5 to gather context, hand only the hard reasoning to Fable 5. And watch your fallback rate: if a domain keeps bouncing to Opus, you’re paying the Fable premium for Opus output, so route it straight to Opus. What it’s still bad at For all the capability, Fable 5 still has many of the same weaknesses as previous LLMs. In some cases it has gotten worse.\nIt states guesses as facts. The system card shows a real regression in missing-context hallucination. When a reference or tool it expects isn’t there, Fable 5 invents one about 18% of the time, against 9% for Opus 4.8. It’s less willing to say “I don’t know” and more willing to assert something plausible it never verified. In a chat you’re watching, you’ll (maybe) catch it. In an agent running unattended for an hour, a confident fabrication three steps back can poison everything downstream. So make it show receipts: tie claims to a file or a command output.\nIt agrees with you too easily. AKA sycophancy. Push back on something Fable 5 said, even when you’re the one who’s wrong, and it tends to fold. Depending on the test, it either regresses to the sycophancy of much older models or ranks best in class. The model is a little more eager to tell you you’re right than to hold a correct position. The fix is to stop handing it the answer you want. Ask “what’s wrong with this approach” instead of “this is good, right?”. Confirmation bias is yet again the enemy.\nIts design taste is generic. Ask for a landing page and you get the output everyone else gets: system fonts, a purple gradient, a big-number hero, cards in a grid. Anthropic clearly knows: it ships an official Frontend Design plugin whose whole job is to steer Claude away from “generic system fonts, predictable purple gradients, and cookie-cutter components.” That’s an admission that the default has a taste problem. Not that the model can’t design, it tops the UI benchmarks. But the plugin only helps if the task reads as design work and you’ve handed it a real brief: a subject, an audience, a point of view. Generic in, generic out. The plugin is just a good prompt you can install, and “prompt it very well” is the entire game.\nIts prose still reads like an AI wrote it. Out of the box the writing is competent and characterless, the exact register that makes people say “this sounds like Claude.” There’s even a word for it now: claudish. This seems nearly impossible to prompt your way out of. Some say GPT-5.6 is more steerable away from this sort of generic AI writing style.\nTakeaway Fable 5 is the strongest model I’ve used and the one that most punishes using it like the last one. Used correctly, it unlocks the next step in the abstraction ladder: a true product owner, not a junior engineer. It is also the most expensive, and has the most guardrails on what you can use it for.\nReach for Fable 5 when the cost of babysitting a cheaper model, the re-prompts, the missed context, the mistakes you’d have to catch, is higher than the premium. When a task is easy to specify and easy to verify, a cheaper model wins and Fable is just an expensive way to get the same answer. When it’s ambiguous, long, or needs one capable agent holding the whole thread, that’s the premium earning out. Describe the outcome, verify the dangerous defaults, and point it at the work that’s actually hard. Use it like last year’s model and you’ll pay frontier prices for frontier frustration.\n","permalink":"https:\/\/tenzinwangdhen.com\/posts\/fable-5-abstraction-ladder\/","summary":"

Fable 5 is another step up the abstraction ladder on the long journey from “ai is better autocomplete”. We need to yet again throw out old assumptions about how to work with LLMs. Old prompt techniques that worked well before are now too constrictive. It’s a different animal entirely, and using it well means unlearning some habits.<\/p>\n

Here I’ll describe how I’ve been using it, what works and what still doesn’t.<\/p>","title":"Fable 5: Another Level Up the Abstraction Ladder"},{"content":"Alex Graveley, the guy behind GitHub Copilot, showed me how I’d been thinking about agentic coding wrong.\nHe used pen and paper.\nAlex dropped by our Mission Street office on a Tuesday. I told him we had about a week of work left to get self-serve working.\n“No. That should take 4 hours.”\nHe pulled out a sheet of paper, wrote down ten steps, and handed it to me. That was the plan.\nI’ve been through every phase: ChatGPT copy\/paste, Cursor, Claude Code, Codex, back to Claude Code, subagents, orchestration frameworks, 10 parallel sessions. The assumption was that more tooling meant more output.\nThat assumption is half-right. The real constraint is undirected scope, not the model or the tools. Without a fixed list, parallel agents multiply your WIP, not your throughput. You end up with ten sessions, five half-finished features, and nothing shipped.\nParallelism only helps when every agent is pointed at the right thing.\nAgentic coding makes hard things look like low-hanging fruit. A feature will only take an hour, so you fire off a session. Then another. Suddenly you’re ten commits deep in a niche optimization nobody asked for. Alex calls this “fake work”\u2014all the motion that doesn’t move a metric or unblock a user. In startups, the only thing that matters is getting users and removing blockers to that goal.\nAlex’s process is blunt: ten steps on paper, two Claude Code sessions, nothing else. Work through the list until done. The checklist is a contract with yourself about what ships today.\nHe starts every project with a fresh CLAUDE.md. He only adds lines when something annoys him. Claude keeps asking whether to commit? Update the file. He calls it annoyance-driven development: you don’t pre-load rules, you earn them through friction. Here’s where we ended up by the end of our session:\nkeep everything as concise as possible no unnecessary abstractions no unnecessary comments when the next step is obvious, do it without asking always be as general as possible parallelize independent steps whenever possible On prompting: I stuff agents with context, design docs, previous sessions. He keeps prompts short. For tasks a thousand teams have already solved (ECS, auth, DB migrations), more context means more noise. The model already knows the generic path; what it doesn’t know are your sharp constraints, which is where you spend your context budget.\nOn early optimizations: I default to thinking about reliability and scaling way ahead of when it matters. “Building for scale with a handful of users is avoidance.”\nThis changes scope discipline, not the quality bar. The ten-step list is a forcing function, not a shortcut around quality. When you ship a step, you verify it works. The discipline is in writing the list tightly enough that each step has a clear done-state: something you can check, not just declare.\nWhat it does kill is architectural drift from over-scoped agents. Short prompts, tight scope, explicit review at each step. That’s the governance. Simple, but it holds.\nBy the end of our session we were on step 6 of 10.\n“Don’t stop until you finish the list. Tomorrow, rinse and repeat.”\nBy night, anyone who requested access could onboard end-to-end.\nIn the age of agentic coding, focus is the superpower. Scope creep that feels like velocity is eating most of your AI-coding gains. More sessions, more output, less of what actually matters shipped.\nI came into that conversation thinking the bottleneck was tooling. More parallelism, richer context, smarter orchestration.\nI left with a crossed-off list on a sheet of paper. The milestone shipped.\nLook at what you’re working on right now. Can you write it down in clear steps on a sheet of paper? If not, you’re running on a vibe, not an explicit plan.\n","permalink":"https:\/\/tenzinwangdhen.com\/posts\/copilot-creator-ai-coding-wrong\/","summary":"

Alex Graveley, the guy behind GitHub Copilot, showed me how I’d been thinking about agentic coding wrong.<\/p>\n

He used pen and paper.<\/p>\n

Alex dropped by our Mission Street office on a Tuesday. I told him we had about a week of work left to get self-serve working.<\/p>\n

“No. That should take 4 hours.”<\/p>\n

He pulled out a sheet of paper, wrote down ten steps, and handed it to me. That was the plan.<\/p>","title":"The Power of Focus in Agentic Coding"},{"content":"\nI used Gas Town for a week.\nI ended up with three mayors, 141 orphaned Claude Code processes and a new appreciation for why agent orchestration is hard.\nGas Town is Steve Yegge’s framework for orchestrating a fleet of coding agents. It shines when most of your work is pure code: lots of independent tasks, clear specs, and you mostly know what you want and don\u2019t care how you get there. It falls apart when the work is heavily human-in-the-loop or ambiguous. Keeping Gas Town fed with beads that align with your project goals is the hard part.\nWhen it works it is breathtaking. I hit the promised land briefly\u2026 then crashed into a Mad Max themed trough of disillusionment. I now use a different setup that borrows ideas from Gas Town.\nThe Gas Town Universe Yegge went full Mad Max with the naming. Here are the 6 terms you need to follow this post:\nMayor \ud83c\udfa9: Your primary coordinator. You talk to the Mayor; it delegates to workers. Polecats: Ephemeral worker agents (Claude Code \/ Gemini CLI \/ Codex). One polecat per task, decommissioned after merge. Beads: Git-backed issues: the atomic unit of work. Each bead has an ID, description, status, and assignee. Convoy: A batch of related beads assigned together. Rig: A project container. One repo = one rig. Refinery: The merge queue. Handles PRs when multiple polecats finish at once, intelligently merging one at a time. Full glossary at the end.\nWhat Is Gas Town, really? Gas Town is an orchestration layer for parallel coding agents, with state tracked in git. Your main interface to it is a Claude Code session running in a tmux window. It tries to abstract away everything between idea and implementation even for very large projects.\nThe flow:\nYou describe what you want done to the Mayor. The Mayor defines work as beads (git-backed tasks with an ID + spec). The Mayor slings beads to polecats (ephemeral agent workers in their own worktrees). You can also directly sling beads to polecats. Polecats open PRs; the refinery merges them safely one at a time. When a bead merges, the polecat is decommissioned and the rig moves on. Gas Town is a framework, not a library. You either buy into it wholesale or you don\u2019t. There are components of it that are modular though, such as beads. Beads are git-backed, dependency-aware tasks for agents. It deserves its own post because it is the one thing that I continue to use. Turns out an ai-native, git-backed issue tracker is a very useful thing to coordinate work across ephemeral agents.\nThe Good Gas Town takes a maximalist view of agentic coding. You as the human do not look at the code, you just define the spec and task list and let Gas Town take care of the rest. It is vibe coding taken to its logical extreme. And it is magical when it works.\n\u201cIn Gas Town, you let Claude Code do its thing.\u00a0You are a Product Manager, and Gas Town is an Idea Compiler. You just make up features, design them, file the implementation plans, and then sling the work around to your polecats and crew. Opus 4.5 can handle any reasonably sized task, so your job is to make tasks for it. That\u2019s it.\u201d - Steve Yegge\nI love the concept of just talking to a main agent called the \u201cmayor\u201d. As someone that regularly runs 10+ claude code sessions across multiple terminal tabs and tmux panes, one of my pain points was keeping track of which agents were working on what task. The mayor handles tracking the polecats for me.\nThe mayor of Gas Town, I assume\nThe parallelization is also real. Each polecat works on its own git worktree, so they can work in parallel on beads. This is a massive speedup in development time. I extracted this idea into a skill called smithers. One difference from other orchestrators is that each polecat is NOT a subagent, but a full Claude Code session running in another tmux window. It even supports swapping in Codex as the model via an --agent flag.\nPart of the pain of coding with agents is that they often get blocked and won\u2019t continue working unless you prod them along. Steve has apparently also had this issue, so he made sure Gas Town has mechanisms to account for this. The system’s propulsion principle is supposed to keep agents moving \u2014 if there’s work on a hook, it must run.\nSince all work is backed by git via beads, you don\u2019t need to stress over the context of any individual polecat. A bead is only closed when a PR is merged, so until then any polecat can pick up and progress on the work. In an earlier epoch of vibe coding, we would feed a giant spec into a single agent\u2019s context and hope to god it would follow through. Externalizing the spec via a dependency graph like beads allows the agent to focus all of its context on one bead at a time, thereby making it much more likely that the project doesn\u2019t completely derail.\nAnother clever idea is the concept of a mail system. Agents can send mail to each other, and agents check their mail. The main use case for this is that it allows polecats to send mail to the mayor when they run into issues, allowing the mayor to either address the issue or escalate to the human.\nThe “aha” moment for me was when I slung 7 beads before dinner \u2014 things like “add a styled footer with alert badges,” “implement fsnotify file watching,” and “cache rendered views with debouncing” \u2014 all for a Go TUI dashboard I was building. Came back to find 6 PRs merged, and only one blocked on a dependency.\nFor projects where you know exactly what you want but it\u2019s complex enough that you can’t one-shot it, Gas Town might finish it in one night rather than a week.\nThe Bad Gas Town has some rough edges. This can be mainly attributed to the fact that this is an early stage, experimental project.\nStuff doesn’t work out of the box. The mail system (polecat \u2192 Mayor status) needed patching when I first used it, and seemed to break down mysteriously. Background maintenance processes like the deacon (a background daemon that keeps agents working) and dogs (its cleanup helpers) simply weren\u2019t running until I fixed them. Despite all this, agents still seem to need manual prodding to continue.\nTowards the end of my experiment with Gas Town, I noticed the ram usage on my 32gb Macbook Air was more elevated than normal. It turned out I had 141 orphaned claude code processes. The polecats were not being nuked correctly, so I needed to manually clean them up. To be fair, this issue has been fixed since then.\nThat\u2019s my project strapped to a Gas Town rig!\nObservability is thin. Sometimes 6 PRs merged and I had no idea when I’d slung them. Other times the system seemed stuck and I couldn’t tell why. There’s a web dashboard (gt dashboard), but I found myself in tmux panes anyway.\nTmux fluency is required. You talk to the mayor in a tmux window. Crews and polecats also run in their own tmux windows. When things break, you’re diving into polecat sessions directly. I already knew tmux, but this will be a wall for those who don’t.\nA lot of this can be chalked up to \u201cearly stage, experimental project.\u201d But some of the most unsettling parts are baked into the core philosophy of Gas Town.\nThe Ugly You must buy in to the philosophy of Gas Town to even be willing to use it. There may be nothing better at making GPUs screech in pain. Some have called it a slop factory, and for good reason if used poorly.\nFriendly warning from Yegge\u2019s Gas Town blog post\nSteve does his best to warn away the faint of heart in his blog post. You need to be a level 7 vibe coder (managing 10+ agents manually) to even consider it. The \u201csmart auto complete\u201d crowd should show themselves out.\n“Gas Town is an industrialized coding factory manned by superintelligent robot chimps, and when they feel like it, they can wreck your shit in an instant. They will wreck the other chimps, the workstations, the customers. They’ll rip your face off\u201d - Steve Yegge\nThe core bargain is that you let go of the wheel. Throughput is valued over precision. Work will get lost, and you\u2019re expected to be okay with it. You also aren\u2019t supposed to read the code. Just hope that, somehow, the system produces something that does what you wanted.\nYegge is explicit that the system optimizes for throughput, not correctness: most work gets done; some work gets lost. I experienced this first hand when an agent overwrote a config file entirely. Thankfully it was recoverable via git. This is a feature, not a bug. If that makes you flinch, you\u2019re not the target user.\n\u201cWork becomes fluid, an uncountable substance that you sling around freely, like slopping shiny fish into wooden barrels at the docks. Most work gets done; some work gets lost. Fish fall out of the barrel. Some escape back to sea, or get stepped on. More fish will come. The focus is\u00a0throughput: creation and correction at the speed of thought.\u201d - Steve Yegge\nIf it wasn\u2019t already clear, you must be the type of person that is comfortable with running (multiple) Claude Code with the --dangerously-skip-permissions flag, aka \u201cyolo-mode\u201d. This means Claude doesn\u2019t ask you for permissions and will chug along even if it\u2019s going to rm -rf your home directory. By default, Gas Town launches its agents (Mayor, polecats, etc.) with this flag enabled.\nGas Town can, for obvious reasons, get expensive. Yegge describes Gas Town as a “cash guzzler” and mentions needing multiple Claude Code accounts, each at $200\/month. If you’re on API billing with heavy parallel usage, costs can spiral fast. If you\u2019ve done a poor job speccing your project and churn a lot of code, you\u2019re effectively just burning money and tokens. Peter Steinberger, creator of OpenClaw, calls Gas Town the \u201cultimate token burner\u201d.\nWhy I stopped using it I fit the bill for a Gas Town user in many ways. I have a lot of side projects and ideas that I want to validate quickly. I have 10+ Claude Code sessions at any given time. I\u2019ve been \u201cvibe coding\u201d for years at this point. Despite this, I did not end up adopting Gas Town after trialing it for a week.\nChatting with the Mayor about chaos in my Gas Town\nI\u2019m a believer in the Unix philosophy of tools doing one thing well. Gas Town is the opposite. I found it difficult to build a mental model of all of the moving parts.\nIt is much too heavy for some projects. For something like \u201cbuild a CRUD app\u201d, where you know exactly what you want and therefore can simply create a large backlog of tickets and churn through them, it can be a great fit. For very small tasks like creating a cli tool, it often isn\u2019t worth the hassle of creating a rig in Gas Town. For iterative tasks like building a UI, where you provide a lot of feedback before you get a good result, it is a terrible tool.\nMy dream setup is the agents working autonomously and pinging me when they need direction, not me constantly prodding them to move along. Gas town showed me that is possible, but I still needed to prod the mayor(s) and it asked for too much in return.\nI wanted the same parallelism and \u201cmayor driven\u201d workflow, but with less complexity and overhead. So I started borrowing the best ideas rather than adopting the whole system.\nThe verdict Gas Town is early and messy, but also brave and prophetic. It\u2019s the first system I’ve used that made me feel like I was directing a team rather than babysitting terminals. It has several novel ideas that have inspired a lot of other agent orchestration tools.\nIf you’re already running 5-10+ coding agents regularly, are comfortable letting go of the wheel, and have large well-defined specs to crank through, Gas Town might be exactly what you\u2019re looking for. As coding agents get more capable, I suspect people will see the need for this or related systems.\nI\u2019d skip it if you need fine-grained control and lots of back-and-forth with your agents, yolo mode makes you nervous, or are looking for less complexity.\nFor all its chaos, Gas Town showed me what\u2019s coming. I didn\u2019t become a citizen, but I did leave with a new framework for managing teams of agents. I think that mental model is going to matter a lot this year.\nAppendix: Full Gas Town Glossary For completeness, here’s the rest of the Mad Max vocabulary:\nTown \u2014 Your workspace directory (e.g. ~\/gt\/). The HQ for config + all projects. Hook \u2014 Git-worktree-based persistent storage for agent work. Lets work survive crashes\/restarts. Crew \ud83d\udc77 \u2014 Longer-lived “personal agents” with sticky context. Unlike polecats, crew have persistent identities and aren’t managed by the Witness. Good for design work with lots of back-and-forth. Witness \ud83e\udd89 \u2014 Supervisor that checks on stuck polecats and nudges them. Runs patrols to keep things moving. Deacon \ud83d\udc3a \u2014 Town-level daemon\/beacon. Gets pinged every few minutes to “do your job” and propagates that signal to other workers. Dogs \ud83d\udc36 \u2014 Deacon’s helpers. Handle maintenance tasks like cleaning up stale branches. Sling \u2014 Assign a bead to an agent (“sling a bead to a polecat”). Convoy \u2014 A batch of related beads assigned together. GUPP \u2014 “Gas Town Universal Propulsion Principle” \u2014 stated simply: if there is work on your hook, you must run it. This is what keeps agents moving instead of stalling. Molecule \u2014 A durable, chained multi-step workflow tracked as linked beads. Survives agent crashes and restarts. The core unit of Gas Town’s workflow engine. Wisp \u2014 An ephemeral molecule destroyed after completion. Used for transient work that doesn’t need to persist. Formula \u2014 A TOML workflow template that gets “cooked” into protomolecules, then instantiated into molecules or wisps. Think of it as the source code for workflows. Protomolecule \u2014 A template molecule (yes, it’s an Expanse reference). Contains pre-built bead graphs with variable placeholders, instantiated into real workflows. Nudge \u2014 Real-time messaging (gt nudge) that pokes agents via tmux to check their hook and mail. The workaround for GUPP not always firing on its own. Handoff \u2014 Graceful session transfer (\/handoff). Agent cleans up, restarts, and the successor picks up via GUPP. Seance \u2014 Talk to a worker’s previous session (gt seance). Uses Claude Code’s \/resume to revive dead sessions and recover lost context. Patrol \u2014 A recurring workflow loop run by the Deacon and Witness. A well-defined sequence of checks encoded as linked beads. Boot \ud83d\udc15 \u2014 A special Dog awakened every 5 minutes to check on the Deacon. Decides if the Deacon needs a heartbeat, nudge, restart, or to be left alone. Thanks to Dan Shapiro and Nico Pinto for reading drafts of this post.\n","permalink":"https:\/\/tenzinwangdhen.com\/posts\/gastown-good-bad-ugly\/","summary":"

\"Gas<\/p>\n

I used Gas Town for a week.<\/p>\n

I ended up with three mayors, 141 orphaned Claude Code processes and a new appreciation for why agent orchestration is hard.<\/p>\n

Gas Town<\/a> is Steve Yegge’s framework for orchestrating a fleet of coding agents. It shines when most of your work is pure code: lots of independent tasks, clear specs, and you mostly know what you want and don\u2019t care how you get there. It falls apart when the work is heavily human-in-the-loop or ambiguous. Keeping Gas Town fed with beads<\/a> that align with your project goals is the hard part.<\/p>","title":"Gas Town: The Good, The Bad, The Ugly"},{"content":"AI, Jevons Paradox, and the Future of Software Engineering The year is 1994. A groundbreaking technology is proliferating\u2014one that would expose all of human knowledge to everyone, anywhere in the world. Industries relying on information scarcity trembled. Newspapers, encyclopedias, travel agents, and countless others faced an existential threat.\nThat technology was the Internet, and while it did eliminate many jobs, it created exponentially more. Entire categories of work\u2014social media managers, SEO specialists, app developers\u2014emerged that would have been impossible to predict.\nFast forward to 2025, and a similar panic is unfolding. This time, the technology is AI. With each new model release demonstrating capabilities that seemed impossible just months earlier, most now believe mass displacement is inevitable\u2014ironically, even for those primarily responsible for automation: software engineers. As one viral post put it: “The last job automated will be automating jobs.”\nIn this post, I’ll explore the history of technological job displacement, how it often creates a net increase in jobs through mechanisms like Jevons Paradox, what parts of engineering work remain stubbornly human, and how to position yourself for the AI-augmented future of our profession.\nTechnological Unemployment Is Not New Since antiquity, automation has displaced jobs but created new ones.\nWater Mills in the Roman Empire (1st-4th centuries CE) provide one of the clearest early cases. The introduction of water-powered grain mills displaced many manual grain grinders, who traditionally were often enslaved workers or lower-class laborers.\nHowever, new specialized roles were created. There was a need for millwrights who designed and maintained the complex gear mechanisms, hydraulic engineers who designed water management systems, and contractors who built aqueducts and water channels to power the mills, to name a few.\nThis pattern has repeated for other major inventions and automations: railroads, the printing press, textile mechanization, and computers. In every case, the incumbent workers trained only in the old paradigm end up losing out.\nDue to this pattern of net job creation, labor’s share of GDP has remained stable even as new industries emerge for at least the last 200 years.\nThe question is whether this time is different. Due to AI, capital can finally break free of the need for labor, perhaps even for those creating and optimizing the AI systems.\nWhat is a job? Before we dive into the question of whether “AI will take all jobs,” we should first define what we mean by a job. A job, specifically white-collar professions such as “software engineer,” is a collection of tasks carried out towards some broader goal. Your employer probably didn’t hire you specifically just to write code, unless you are junior or the task is very complex (such as foundation model training).\nEach task that makes up a job can be more or less liable to be replaced by an AI system.\nUsually, from the outside, your mental model of a job is probably marked by some stereotype you have about the role, and then you ask yourself whether you see an AI able to perform that stereotype.\nYou imagine a therapist sitting on a couch, asking probing questions about how you feel, and you say sure, an AI can totally do that. For software engineers, people typically visualize a sun-starved young man in a hoodie hunched over a laptop in a dark room as green text flickers across the screen. Unfortunately, the reality is more banal. In a large company, you don’t have the luxury of dreaming up some feature and then getting it into production after some blissful coding sessions.\nTo properly understand AI’s impact on software engineering, we need to examine what current AI systems can and cannot do effectively.\nCurrent AI Strengths and Weaknesses Currently, AI excels at handling isolated coding challenges, such as those found on LeetCode or function generation tasks, where the problem is well-defined, and the solution follows recognizable patterns. It is particularly effective at automating repetitive tasks, including generating boilerplate code, writing tests, and even drafting documentation, as seen in tools like GitHub Copilot and Cursor. My own experience demonstrates this evolution - while a couple years ago I was merely experimenting with ‘vibe-coding’ by copy-pasting from GPT-4, today I can generate a working POC in under an hour with Cursor Composer and Claude 3.7, complete with tests, polished Tailwind CSS, and a functioning backend, all primarily through natural language. The productivity gains for greenfield projects and research-driven tasks are substantial, with tools like Glean making knowledge discovery faster and more efficient.\nAI struggles with contextual reasoning, particularly when it comes to deciding what to code. Understanding company-specific nuances\u2014such as interpreting Slack discussions, navigating legacy systems, and aligning with long-term business goals\u2014remains beyond AI’s capabilities. Tasks like quarterly planning, RFC processes, and incident management still require human oversight, as they involve judgment, negotiation, and adaptation to evolving priorities. Additionally, AI faces significant challenges when working with legacy systems. Integrating AI-generated code into monolithic, decades-old architectures is far from straightforward, as these systems often contain undocumented dependencies, unconventional design patterns, and business logic that AI cannot easily infer.\nWhat I’ve realized through this rapid evolution is that the bottleneck has shifted - the most time-consuming part of the development process is now planning and providing the correct context to the AI. Effective prompt engineering has become a crucial skill - the ability to concisely explain tasks and demonstrate sufficient technical knowledge to narrow the solution space. For example, specifically requesting React and Tailwind yields much faster results than vaguely asking for a “nice looking UI.” This shift highlights how engineering roles are evolving from implementation-focused to more design and direction-oriented work.\nTo better understand how these AI capabilities translate to real-world engineering work, let’s examine a typical day for a senior engineer and identify which tasks are ripe for AI assistance and which remain human.\nDay in the Life of a Senior Engineer One of the tasks of a senior engineer is to propose and implement ideas that will improve something about the company. The ideas that are likely to be adopted depend on many variables: pain points, management priorities, staffing, costs, etc. Thinking up viable ideas given these factors is currently best left to humans due to the context required.\nOnce you’ve identified a promising project, you propose it in a planning process. This involves meetings where your idea competes with other priorities. This task is difficult to automate.\nAssuming your project is approved, you develop an RFC document describing importance, technical requirements, implementation plan, timeline, and alternatives. The actual writing of the RFC is likely best left to AI with human oversight. LLMs need sufficient context about your organization, either through prompt context or RAG systems.\nAfter the RFC is approved, you implement the idea. Some coding tasks are definitely automatable - LeetCode-style problems, greenfield projects in popular languages, and test writing are ideal for AI generation.\nA feature requiring large-scale refactoring on a million+ line codebase is more challenging due to LLM context limitations. The AI needs to search for relevant code, which is harder with poor documentation and naming.\nOnce code is written, you solicit PR reviews from peers. There are tools aiming to automate PR review with AI already. Human review becomes more important when the code was AI-generated to catch subtle bugs or hallucinations.\nFinally, you deploy your change. Deployment should use traditional automation, with AI potentially helping detect issues via anomaly detection. If your change causes a production incident, root cause analysis requires context about your tech stack and access to logs, metrics, and dashboards that current AI systems lack.\nHere are the major tasks identified, and whether AI can currently replace that task:\nEngineering Task Current AI Capability Human Involvement Notes Idea inception Limited Mostly human Requires understanding organizational context, historical knowledge, and business priorities Quarterly planning Very limited Fully human Involves negotiation, politics, and cross-team coordination RFC document writing Strong Human + AI AI can draft content but humans must provide context and review Coding (standard tasks) Very strong Minimally human Particularly effective for greenfield projects and standard patterns Refactoring legacy code Moderate Mostly human Limited by context windows and understanding complex interdependencies Test writing Strong Minimally human Excellent at generating comprehensive test cases PR review Moderate Mostly human Tools exist but human judgment needed for architectural decisions Incident management Limited Mostly human Root cause analysis requires deep system understanding These tasks are just within the context of creating a feature. There are several other tasks I have not covered: team standups, presentations, etc.\nThere are also some new tasks and skills that are starting to become necessary for engineers:\nPrompt engineering Tradeoffs between LLM providers and APIs Vector databases Retrieval Augmented Generation Effective use of coding copilot systems AI content management Building Agentic workflows While this workflow represents traditional engineering environments, a new breed of companies is emerging that fundamentally reimagines how software is built in the AI era.\nAI First Companies The engineering workflow described above primarily reflects processes in established enterprises. In contrast, AI-first companies operate with different organizational structures and development paradigms.\nThese organizations leverage AI capabilities extensively throughout their development lifecycle, significantly compressing the time between ideation and deployment. By integrating AI tools into core engineering processes, these companies maintain leaner technical teams composed of engineers who excel at AI-augmented development.\nAI-first companies typically exhibit several defining characteristics:\nStreamlined engineering teams with higher leverage per engineer Integrated AI tools across the entire development pipeline Reduced organizational overhead and coordination requirements Accelerated product iteration cycles These companies often extend AI adoption beyond engineering to functions like marketing, customer support, and operations. This comprehensive approach reduces cross-functional dependencies and administrative overhead, enabling more agile product development.\nEngineers at AI-first companies require a distinct skill profile that combines traditional software engineering expertise with proficiency in AI-augmented development tools. This includes mastery of prompt engineering, AI development frameworks, and efficient collaboration with AI systems.\nThe emergence of these organizations represents a structural shift in how software products are conceived, built, and delivered rather than merely a tactical adjustment to existing processes.\nGiven how both traditional and AI-first companies are evolving, what does this mean for engineers at different career stages?\nImpact on Engineering Roles Junior Engineers For junior engineers, the baseline expectations are shifting. Pure coding ability\u2014once the core of entry-level roles\u2014is becoming less critical as AI tools handle routine tasks. Instead, success depends more on abstract reasoning, problem decomposition, and effective collaboration with AI. Those who fail to integrate AI into their workflow risk displacement, as companies may opt for a smaller team of AI-augmented developers rather than hiring large cohorts of junior engineers.\nHowever, AI can also be a powerful learning tool, accelerating skill development for those who embrace it. Understanding the fundamentals of computer engineering, such as data structures and algorithms, as well as general skill in problem-solving will still be important. Junior devs who only use the tools without understanding the fundamentals won’t know when and why to stray from what the LLM is generating, or come up with novel solutions that are not in the training set. Those who understand the fundamentals as well as how to use the tools will be on the fast track to becoming senior engineers.\nSenior+ Engineers For senior engineers, AI doesn’t eliminate responsibilities\u2014it shifts them. The emphasis moves toward problem-framing, system architecture, and cross-functional collaboration, areas where human judgment is irreplaceable. Senior engineers shift to becoming “product owners” that understand the business context and assign tasks given that context to AI agents. Engineers who adapt well may find AI accelerating their career progression, as increased productivity and strategic thinking could shorten the time to promotion. However, AI cannot be held accountable for architectural decisions, security risks, or ethical trade-offs. The responsibility for those choices\u2014and the consequences\u2014still falls on senior engineers. The line between senior engineers and product managers is becoming more blurred - senior engineers can no longer simply take tasks from product managers and implement them. Instead, they need to be able to understand the business context and assign tasks to AI agents.\nWith these role transformations in mind, let’s examine the broader economic arguments for and against AI’s impact on total engineering employment.\nBull and Bear Case for Eng Jobs Jevons Paradox and the “Expanding Pie” The bull case for engineering jobs is the concept of “Jevons Paradox,” which is the idea that increased efficiency leads to increased consumption. Adding another lane to a 5-lane highway can ironically increase traffic, because drivers that otherwise would have taken public transport or carpooled may be lured to attempt to drive themselves.\nIn fact, I feel I have gotten more busy after LLMs than before. I am now tackling more technically challenging projects that I would have shied away from due to lack of knowledge, and am more able to retain a flow state while working since I don’t have to spend any time looking up syntax, and much less time looking up docs. My pattern is starting to develop into: kickoff the ideation with deep research, come up with a spec, break it down into prompts with a reasoning model, pass each prompt to Cursor composer, and just start iterating. I can usually go from idea to working POC within an hour this way. This approach is described in detail here.\nWith a lower barrier to entry for creating software, more software will inevitably be created. This means more engineers could be needed to support this increasing pool of software systems, which still need cloud services, monitoring tools, security, etc.\nNew classes of tasks that did not exist a couple of years ago are being created. There are already a few entirely new jobs as well, such as “AI Engineer.”\nSome venture capitalists argue that AI fundamentally reduces the need for large engineering teams. Investors like Chamath Palihapitiya, through firms like 8090, suggest that AI-driven efficiency means startups can achieve the same output with fewer developers.\nThe counterpoint to this argument is ambition inflation\u2014as AI reduces the cost of software development, companies may simply scale their ambitions instead of reducing headcount. If a team that once built a product with 10 features can now deliver 100, the demand for skilled engineers remains strong. Instead of eliminating jobs, AI may push the industry toward building more complex, ambitious software at a faster pace.\nRegardless of whether the total number of engineering jobs increases or decreases, one thing is clear: the nature of these jobs is changing rapidly, and adaptation is essential for career longevity.\nAdapting to the AI Era The engineers who thrive won’t be those who resist AI tools, but those who learn to use them, and develop a sense for their strengths and weaknesses. This means mastering AI-augmented IDEs like Cursor, developing the meta-skill of prompt engineering, and becoming adept at debugging the sometimes-flawed outputs these systems generate. Yet paradoxically, as AI capabilities expand, distinctly human skills become more valuable, not less. The ability to frame problems correctly before an AI even sees them, to collaborate across teams with different priorities and knowledge bases, and to make ethical judgments that reflect our values rather than statistical patterns in data\u2014these become the differentiators. Perhaps most importantly, complacency represents the greatest career risk in this environment. Those who assume their current skills will retain value without adaptation risk displacement by more adaptable colleagues. Continuous learning isn’t just beneficial; it’s non-negotiable in a landscape where the technological ground shifts beneath our feet almost daily.\nWhile current AI systems are already transforming engineering work, the ultimate question looms: what happens when AI reaches or exceeds general human capabilities?\nAGI and the Long-Term Engineering Landscape While most of this article addresses the immediate impacts of current AI systems, it’s worth considering how Artificial General Intelligence (AGI) might reshape software engineering more fundamentally. Unlike today’s specialized systems, AGI would theoretically match or exceed human capabilities across all cognitive tasks.\nEconomic Constraints Will Still Matter Even as AGI becomes technically feasible, economic constraints will likely determine its adoption patterns. Training costs for advanced AI systems remain substantial - AlphaGo’s $35 million training expense provides a reference point. These costs suggest that human-AI collaboration will persist in domains where the marginal benefit of full automation doesn’t justify the expense.\nSoftware engineers working on problems with lower economic value (like maintaining internal tools or building systems for smaller markets) may retain their roles longer than those in high-value domains where AGI investment would yield significant returns. The principle of comparative advantage suggests humans may specialize in areas where our cost-to-capability ratio remains competitive.\nTask Evolution, Not Job Elimination As with current AI tools, AGI will likely transform engineering tasks rather than eliminate engineering roles wholesale. Staff+ engineers might shift from writing code to defining and refining system objectives, establishing performance criteria, and ensuring alignment between technical capabilities and business needs. This represents a continuation of the trend we already see, where increasing automation elevates human work to higher levels of abstraction.\nHistorical patterns support this view. Despite centuries of productivity improvements, the 40-hour workweek persists, as do professional roles. Technology has consistently transformed the nature of work rather than eliminated it entirely. Engineering will likely follow this pattern, with AGI handling implementation details while humans manage higher-order concerns.\nNew Engineering Specializations AGI would likely create entirely new engineering specializations. We might see roles emerge around:\nAGI Alignment Engineering - Ensuring AGI systems maintain appropriate goal alignment System Orchestration - Designing workflows that optimally combine AGI capabilities Human-AGI Interfaces - Creating effective collaboration mechanisms between humans and AGI systems AGI Supervision - Monitoring AGI outputs for accuracy, bias, and alignment with intended outcomes Context Engineering - Understanding the business context and ensure the context provided to the AGI is correct and up to date Rather than engineering jobs disappearing, they would transform into roles focused on defining problems clearly, evaluating solutions critically, and directing AGI resources effectively. The core engineering skills of systems thinking, trade-off analysis, and problem decomposition would remain valuable, even as the technical implementation shifts to AGI systems.\nIn this scenario, the competitive advantage goes to engineers who develop metacognitive skills - understanding how to frame problems in ways that AGI can optimally address, recognizing the limitations of automated solutions, and maintaining the human judgment necessary to evaluate AGI outputs critically.\nConclusion AI is here and is at a level where it can do some tasks much more proficiently than human engineers. However, it is still a ways off from doing the whole job as there are tasks within the job that are difficult to automate. Over time the tasks will be automated, but more, higher-level tasks could also be created, such as tuning the AI prompts or providing the right context to the AI. Every time technology has increased efficiency in the past, there is a subsequent increase in overall demand, though incumbents and laggards are liable to be replaced. As it currently stands, AI won’t replace engineers, but engineers that are proficient with AI tools will replace those that are not. Perhaps the best advice here is to remain on the forefront of adopting AI tools and being adaptable as the skills required change. As William Gibson famously said, “The future is already here\u2014it’s just not evenly distributed.”\n","permalink":"https:\/\/tenzinwangdhen.com\/posts\/ai-replace-engineers\/","summary":"

AI, Jevons Paradox, and the Future of Software Engineering<\/h2>\n

The year is 1994. A groundbreaking technology is proliferating\u2014one that would expose all of human knowledge to everyone, anywhere in the world. Industries relying on information scarcity trembled. Newspapers, encyclopedias, travel agents, and countless others faced an existential threat.<\/p>\n

That technology was the Internet<\/strong>, and while it did eliminate many jobs, it created exponentially more. Entire categories of work\u2014social media managers, SEO specialists, app developers\u2014emerged that would have been impossible to predict.<\/p>","title":"Will AI Displace Software Engineers?"},{"content":"Software systems exhibit a peculiar property: the more sophisticated they become, the more they tend toward catastrophic rather than graceful failure. This pattern isn’t unique to software - it’s characteristic of all complex systems, from neural networks to financial markets. But software’s rapid evolution and ubiquity makes it a particularly interesting case study.\nA complex system isn’t merely complicated. Rather, it possesses specific properties that make its behavior fundamentally unpredictable:\nDynamic interaction between components that can’t be reduced to simple cause-and-effect relationships Feedback loops that create non-linear responses to changes Emergent properties that can’t be predicted from analyzing individual components Nested complexity where components themselves are complex systems Consider a seemingly straightforward example: running a persistent database on Kubernetes. At first glance, this might appear to be just a collection of software components working together. But examine it more closely:\nThe database must maintain consistency across distributed nodes Kubernetes continuously adjusts resource allocation based on load Network latency creates feedback loops affecting query performance Storage systems interact with hardware in non-linear ways Each layer (database, Kubernetes, cloud infrastructure) is itself a complex system The result isn’t just a complicated stack of technology - it’s a system where changes in one component can propagate in unexpected ways, creating emergent behaviors that weren’t designed or anticipated.\nRichard Cook’s “How Complex Systems Fail” makes a counterintuitive claim: complex systems are always running in a partially broken state. This isn’t a flaw in implementation, but rather an inherent property of complexity itself. Why?\nThe number of potential interactions between components grows factorially with system size Each interaction represents a potential failure mode Testing all possible states becomes computationally intractable Individual “minor” flaws are too numerous to fully eliminate The system continues functioning despite these flaws due to human adaptation This leads to what we might call the “complexity paradox”: as systems grow more sophisticated in an attempt to prevent failures, they become more complex, which in turn makes them more prone to catastrophic failure modes.\nThe pattern becomes clearer when we examine how defenses against failure evolve:\nTechnical defenses (redundancy, monitoring, automated recovery) Organizational defenses (procedures, certifications, audits) Human defenses (training, expertise, tacit knowledge) Each layer of defense adds complexity, creating new potential failure modes even as it guards against known ones. The result is a system that appears more robust to anticipated problems while becoming more vulnerable to “black swan” events - rare but catastrophic failures that emerge from unexpected interactions between components.\nWhen catastrophic failures occur in complex systems, organizations typically respond with a search for the “root cause” - a fundamentally flawed approach that misunderstands the nature of complex system failures. These failures aren’t linear chains of causation but rather emergent phenomena arising from multiple interacting components and conditions. What makes a failure “catastrophic” rather than routine is not merely its impact, but its emergence from the subtle interplay between system components, creating cascade effects that overwhelm our carefully constructed defenses.\nThe impossibility of truly isolating root causes becomes clear when we consider the nature of complex systems: they are constantly evolving, operating with multiple simultaneous flaws, maintained by changing combinations of human and technical components, and subject to different environmental conditions at different times. The very notion of “cause” in such systems may be more a human construct than a meaningful description of system behavior.\nA particularly insidious aspect of complex system failures is what we might call the “hindsight fallacy” - the tendency to view past failures as obviously predictable once we know their outcome. This creates a dangerous illusion of preventability that drives misguided remediation efforts.\nConsider a Kubernetes cluster where application pods suddenly lose database connectivity due to an underscaled load balancer. Post-incident, the solution appears obvious: “We should have scaled the load balancer.” But this apparently simple insight obscures the reality of operating complex systems. The actual system state before failure was far more ambiguous, with multiple potential failure points existing simultaneously. Resources and attention were finite and had to be allocated across many concerns. The relationship between load balancer scaling and system stability wasn’t necessarily clear in advance.\nThe natural response to such failures is often automation - an approach that introduces what Cook and Parameswaran independently identify as a profound paradox. By attempting to prevent specific failure modes through automation, we paradoxically increase system complexity, create new potential failure modes, reduce operator engagement with the system, and mask accumulating problems until catastrophic failure occurs.\nTake the load balancer example: Implementing autoscaling seems like an obvious solution, but it introduces new complexities in configuration, documentation, monitoring requirements, and potential failure modes. Perhaps most critically, it reduces operator familiarity with manual scaling procedures. Over time, operators interact with the system less frequently, their skills atrophy, and the system becomes progressively less legible to human understanding.\nThe paradox of automation extends beyond merely increasing system complexity. In “The Control Revolution and Its Discontents”, Ashwin Parameswaran identifies a more subtle and pernicious effect: automation’s apparent safety creates an environment where human error can accumulate invisibly. By smoothing over minor issues and handling routine failures, automated systems mask the early warning signs that would traditionally alert operators to deteriorating performance.\nThis leads to what Parameswaran calls the “uncanny valley” of automation - a state where the system appears more reliable in normal operation, but operators become progressively deskilled and the potential for catastrophic failure actually increases. The system becomes a black box, operating in ways that are increasingly opaque to the humans nominally in charge of maintaining it.\nAn instructive analogy is that of self-driving cars. Early automotive automation - cruise control, lane assistance - augments human capability without introducing significant new risks. A sweet spot emerges where the machine handles routine tasks while the human driver remains engaged and capable, hands on the wheel, ready to respond to situations requiring judgment. The uncanny valley begins when automation extends into critical scenarios, fostering dangerous overconfidence in drivers who have grown complacent yet must still intervene in emergencies. Only true level 5 autonomy, with its complete elimination of uncertainty, would justify removing the human driver entirely.\nParameswaran’s analysis of the Air France Flight 447 crash in 2009 provides a haunting illustration of the uncanny valley in automation. In his essay “People Make Poor Monitors for Computers”, he shows how the automated systems that were meant to make the flight safer ultimately contributed to its catastrophic failure. The pilots, accustomed to the plane’s sophisticated autopilot handling most situations, found themselves suddenly forced to take manual control in challenging conditions. Their skills, dulled by routine reliance on automation, proved inadequate for the crisis they faced.\nWhat lies beyond the uncanny valley of automation? In principle, a state of perfect algorithmization where radical uncertainty has been eliminated entirely. This represents the ultimate goal of the “control revolution” - that centuries-long project to solve every problem through data and algorithms. In such a world, omniscient AI systems would handle all complexity, and human operators could take a much needed sabbatical.\nHow should we approach automation in complex systems, knowing that perfect algorithmic control remains aspirational? Parameswaran suggests a nuanced strategy: embrace automation for non-catastrophic failures while maintaining redundancy where failures could prove ruinous. This keeps human operators meaningfully engaged while benefiting from automated assistance. The goal is not to eliminate human involvement but to structure it properly - maintaining the feedback loops that build and preserve operator capability.\nCook arrives at similar conclusions through different reasoning. He argues that safety emerges from operators’ intimate familiarity with failure modes and system boundaries. Effective monitoring systems should help operators recognize the “edge of the envelope” - that threshold beyond which system behavior becomes unpredictable. This requires not just data collection but careful design of human-machine interfaces that make system state and trajectories legible to operators.\nThe fundamental insight is that safety isn’t a product to be purchased or a feature to be coded - it’s an emergent property arising from moment-to-moment adaptation by skilled human operators working with automated systems. Until we achieve true algorithmic omniscience (if ever), our focus must be on maintaining this creative tension between human and machine capabilities, keeping operators firmly within the loop while leveraging automation to extend rather than replace their capabilities.\n","permalink":"https:\/\/tenzinwangdhen.com\/posts\/catastrophic-failure-complex-systems\/","summary":"

Software systems exhibit a peculiar property: the more sophisticated they become, the more they tend toward catastrophic rather than graceful failure. This pattern isn’t unique to software - it’s characteristic of all complex systems, from neural networks to financial markets. But software’s rapid evolution and ubiquity makes it a particularly interesting case study.<\/p>\n

A complex system isn’t merely complicated. Rather, it possesses specific properties that make its behavior fundamentally unpredictable:<\/p>","title":"Catastrophic Failure in Complex Systems"},{"content":"Nvidia is one of the largest companies in the world, frequently taking the top spot. It’s revenue is growing at an astonishing rate, with margins better than a lot of pure software businesses - something usually unheard of for hardware companies. All of this is on the back of a massive AI hype cycle. During a gold rush, you should sell picks and shovels. Nvidia is selling bulldozers. In this post, I will dive into the components of Nvidia’s competitive moat, its strengths and weaknesses, and the competitors trying to cross it.\nIn a capitalist system, profits are always competed away, unless you’ve managed to create a legal monopoly (i.e. power companies, Google). Nvidia does not have a monopoly, but they do have a near impenetrable moat.\nToday, consumer GPUs make up just a fraction of Nvidia’s revenue. The vast majority of it comes from their data center business. Very large companies with massive data centers like Tesla, Meta, and OpenAI (the “hyperscalers”) are in an arms race to acquire the currently scare resource that is Nvidia GPUs, which are, for now, basically the only game in town for AI training hardware. And because of that, Nvidia can charge exorbitant amounts for their GPUs. Nvidia’s margin of 71% is larger than most SaaS companies, something previously unheard of for hardware companies selling commodities.\nSource: https:\/\/www.investmentideas.io\/p\/nvidia-pivoting-towards-chiplets\nSo how did Nvidia get into this enviable position?\nMost of it comes down to the foresight of Nvidia’s founder Jensen Huang. He realized that GPUs could be used for scientific computing and started building CUDA even before there were real use cases. Through a stroke of good luck, the creators of AlexNet thought to use Nvidia GPUs and CUDA to train a model that won the ImageNet competition in 2012. This marked the beginning of the deep learning revolution. Deep learning is “embarassingly parallel” problem, and GPUs are perfectly suited to parallel processing.\nA Primer on GPUs A bit of an aside on CPUs vs GPUs (graphics processing units), and why GPUs are so much better for modern AI.\nTo help you develop a grossly simplified mental model, I will provide an analogy. Say you have to deliver a ton of coffee beans from Guatemala to Miami. You could use a jet (the CPU) which can carry one bag of beans (data) from the warehouse (memory) very quickly to and from Miami.\nOr you can use a cargo plane (a GPU), which can carry a ton of bags much more slowly. The jet is optimized to prioritize latency (or speed), while the cargo plane is optimized to prioritize cargo capacity (memory bandwidth).\nSmaller coffee shops that require different types of beans might benefit more from the jet since they need a small amount delivered quickly. This represents the general compute that a CPU is tasked with. Large coffee chains like Starbucks just need a consistent amount of the same beans in bulk. This represents the high bandwidth parallel processing required from the GPU.\nWhen training a model, you need to run a single operation, usually matrix multiplication, on a large amount of data. The GPU, with its memory bandwidth, is much better suited to this task than the CPU. The tradeoff is that each individual task is slower, but you can simply add more parallelism (i.e., cargo planes).\nGPUs, or graphics processing units, used to only be used for, well, processing graphics. Rendering graphics requires parallel processing and would be too slow to do on a CPU. Jensen’s insight was that this parallelism would be useful for tasks other than graphics processing, namely scientific computing. But in order for end users to be able to take advantage of the GPU for tasks other than gaming, they needed a programmable software interface on top of the GPU. That is where CUDA, or Compute Unified Device Architecture, comes in.\nCUDA Nvidia created CUDA before knowing whether there would even be a significant market for it. Their competitors, such as AMD, completely ignored this development. With AlexNet, CUDA found it’s “build it and they will come” moment. Deep learning needed to be done on GPUs, and Nvidia GPUs were the only usable ones because they have CUDA.\nAlmost every deep learning library (PyTorch, TensorFlow, Keras, etc.) is built on top of CUDA. There is an entire ecosystem of guides, blogs, StackOverflow posts that support CUDA users. AI researchers looking to train a model or experiment with new types of models favor Nvidia because they want an out of the box solution to working with the GPU without having to waste their time. Unsurprisingly, CUDA only works for Nvidia GPUs and is closed-source. CUDA has created a network effect for Nvidia GPUs, as the value of each GPU increases as the number of users increases. Any rival to Nvidia needs to be able to convince users than their offering is enticing enough to forego this massive ecosystem.\nCUDA is just one of several software products the company produces. One of the most exciting is Omniverse, which uses Nvidia hardware to create “world scale” simulations. Automakers for example are using this to create “digital twins” of their assembly lines, where they can perfect configurations in the digital realm before implementing them in reality, among other applications. The real value here is the ability to create synthetic data sets at scale. Quality data is one of the key scarce resources in developing AI models.\nCUDA is of course not how Nvidia makes the bulk of its money. Nvidia is first and foremost a hardware company. They have been building GPUs for over three decades, and have perfected the art.\nNvidia Hardware The current best Nvidia AI GPU is the H200. It has 141GB HBM memory with a total bandwidth of 4.8TB\/s per GPU across six HBM3e stacks. HBM memory, which consists of three dimensional stacks of memory chips, represents the cutting edge in high bandwidth memory. In my earlier analogy, HBM memory would represent a much larger storage area in the cargo plane.\nTo put the 141GB of memory into perspective, I built an AI PC with a consumer grade RTX 4090 last year. It has 24GB of memory, and can only fit Llama with 7 billion parameters. The H200 can fit Llama with 70 billion parameters. It still would not fit Claude 3.5 which as 2 trillion parameters though, so you’d need to network multiple GPUs together.\nAt GTC 2024, Nvidia announced the B200 “Blackwell” GPU, and a GB200 “superchip”. The B200 has up to 20 petaflops of processing power and 192GB HBM memory. Training a 1.8 trillion parameter model (such as GPT 4) would only require 2,000 B200’s, whereas it would have taken 8,000 previous generation “Hopper” chips for the same task. All of this while consuming a fourth of the energy. Suggested retail price from Nvidia: $30,000 to $50,000 each. In comparison, AMD’s competing MI300X costs between $10,000 to $15,000. Interestingly, Nvidia is not increasing the price for the Blackwell class chips as much as expected.\nThese chips can also be networked together with a next-gen NVLink switch, which can connect up to 576 GPUs with 1.8TB\/s bidirectional bandwidth.\nNow that I have sung Nvidia’s praises, it’s time to explore the threats to their business\nThe Chiplet Threat Nvidia GPUs have a monolithic architecture. They often feature a single large chip. This means that if there is any issue with a chip during manufacturing, the whole chip needs to be discarded. AMD is working on chiplets for GPUs, which are several smaller chips that are stitched together to make one big chip. They already do this for their CPUs. If a smaller chip has an issue, you can replace just that one chip, thereby increasing yields. This means AMD could make GPUs on par with Nvidia, but at lower cost.\nLarge chips are also starting to run into the lithographic reticle limit, whereby producing cutting edge chips at nanometer scale becomes exponentially more expensive and eventually physically impossible. Chiplets on the other hand are still far away from this limit. AMD has been making chiplets for a while already and has a bit of a process moat, since getting chiplet production right requires a lot of trial and error. Nvidia only recently started breaking up the monolithic chip in the Blackwell version, which is just two big chips stitched together. If Nvidia continues down the path of making monolithic chips, they risk disruption by chiplets.\nInference Another opening for competitors is the difference in requirements for training AI models vs “inference”. In simple terms, training a model is the actual creation of the AI model. GPT is a trained AI model. Inference is the process of getting outputs from the trained model. When you ask GPT a question, that is inference. Training requires much more compute power than inference, while inference requires much lower latency. Nvidia GPUs are currently the best available for training, but are already outperformed on inference by competitors, such as Groq.\nIf the need for inference scales more than the need for training, AI infrastructure spend could start to flow away from Nvidia unless they can create a competitive inference solution. Nvidia does already have open source frameworks focused on inference like TensorRT and is touting the inference performance of their Blackwell chips, so they likely realize this threat.\nThe Open Source Threat The crown jewel of Nvidia’s competitive moat, CUDA, is also coming under attack. As discussed earlier, CUDA is what currently makes Nvidia the default choice for AI researchers and model creators. It is part of what justifies Nvidia’s exorbitant margins. Nvidia’s competitors are finally starting to realize the leverage that can be gained from offering a library like CUDA. AMD’s CUDA alternative is ROCm. Unlike Nvidia, they have open sourced ROCm. If CUDA remains closed-source, the contributions of the open source community could eventually make ROCm a more attractive platform.\nVendor lock-in is when you are forced to stick with one vendor for a particular service. Most software company operators (and probably most business owners in general) would agree that vendor lock-in is bad, especially for a core component of your business. If you’re locked in, the vendor can charge a right arm for crappy service and you’d just have to take it. Currently, most companies in the AI space are locked in to Nvidia GPUs. They therefore have a huge incentive to either develop or fund alternatives.\nA Bubble? There is no question that we are in the middle of a massive AI bubble currently. AGI, a technology that, if manifested, could be the last technology we as a race would need to create, seems to be just around the corner. This has sparked a trillion dollar arms race among the largest companies in the world. For that spend, some would argue that we should have gotten more than a some chatbots and image generators. Sequoia’s David Cahn calls this a $600 billion dollar hole. Almost all of that money has flowed to Nvidia. LLMs using the transformer architecture need exponentially more parameters for near linear performance increase. If real enterprise value isn’t created relatively soon, the bubble will burst, resulting in a precipitous drop in revenue for Nvidia.\nAn analogous historical parallel to Nvidia is Cisco Systems. During the internet bubble, everyone was clamoring for Cisco networking hardware. This sent their stock surging over 1000x over a decade. However, the stock lost 88% of its value and has yet to reach it’s peak again over twenty years later.\nTLDR Nvidia is the big winner of the current AI hype wave. They have a seemingly insurmountable moat that is comprised of cutting edge hardware and the dominant software interface for it. The largest companies in the world are stocking up on this hardware at exorbitant prices to remain on the cutting edge. There are, however, significant threats to Nvidia’s continued dominance. Their monolithic chip strategy might hit a scaling wall, leaving them scrambling to catch up to chiplets. The demand for inference might outweigh that of training, requiring them to refactor much of their development pipeline. CUDA might be superseded by open source alternatives like ROCm, diminishing the need to stay on Nvidia GPUs - a trend that will likely have support from Nvidia’s largest customers. Finally, the current AI bubble could burst, resulting in significantly reduced spend on GPUs for AI use cases.\n","permalink":"https:\/\/tenzinwangdhen.com\/posts\/nvidia-barbarians-at-the-moat\/","summary":"

Nvidia is one of the largest companies in the world, frequently taking the top spot. It’s revenue is growing at an astonishing rate, with margins better than a lot of pure software businesses - something usually unheard of for hardware companies. All of this is on the back of a massive AI hype cycle. During a gold rush, you should sell picks and shovels. Nvidia is selling bulldozers. In this post, I will dive into the components of Nvidia’s competitive moat, its strengths and weaknesses, and the competitors trying to cross it.<\/p>","title":"Nvidia - Barbarians at the Moat"},{"content":"Joshua Foer’s “Moonwalking with Einstein” is one of the rare books that I found worth rereading. In it, Foer, a young journalist, enters the bizarre world of memory competitions after being assigned to cover the world championship. He describes in vivid detail the unusual characters he encounters, mnemonic techniques he learns, and books he reads to help prepare him for the U.S. Memory Championship. The “memory athletes” he interviews are able to memorize a deck of cards in thirty-two seconds, recall over eighty thousand digits of pi, and recite the entire works of Shakespeare. Foer’s efforts pay off, and he becomes the U.S. Memory Champion by the end of the year.\nThe History of Mnemonics The ancient Greeks and Romans lacked modern technologies for storing information outside their minds. Most people were illiterate, and written books were rare. Socrates even opposed written language, arguing it would leave men’s minds empty as they wouldn’t need to retain what they learned. Instead, these cultures relied on mnemonic techniques to memorize speeches and epic poems. “The Iliad” and “The Odyssey” feature vivid descriptors like “rosy-fingered dawn” and “weeping Aphrodite” because such imagery aided memorization. In “Ad Herennium,” Cicero’s treatise on rhetoric, he includes a brief section on mnemonic techniques. Foer refers to this work as the “bible” of memory competitors.\nSome ancient figures were reportedly capable of truly astounding feats. Pliny the Elder writes: “King Cyrus could give the names of all the soldiers in his army. Lucius Scipio knew the names of the whole Roman people. King Pyrrhus’s envoy Cineas knew those of the Senate and knighthood at Rome the day after his arrival. A person in Greece named Charmadas recited the contents of any volumes in libraries that anyone asked him to quote, just as if he were reading them.”\nMnemonics in Modern Times Rote memorization was a staple in education until relatively recently. Students were required to memorize historical dates, multiplication tables, Latin vocabulary, and other facts. Educators believed the value of this practice lay not only in learning the information but also in training the ability to memorize.\nEventually, memorization came to be seen as dehumanizing. Schools systematically deprioritized it in favor of more “experiential learning.” Foer laments that we are now in a state where most high school students can’t identify when the Civil War occurred, and 20% of students can’t even name the countries the U.S. fought in World War II.\nWhen exactly did educators remove advanced mnemonic techniques used by the ancients from the curriculum? Given their effectiveness, one would expect these methods to have been favored over rote memorization. While the book doesn’t directly answer this question, it offers some hints.\nThe Science of Memory As a rule, humans have a working memory capacity of 7 bits of information, plus or minus 2. This explains why phone numbers are 7 digits long (excluding area and country codes), and why license plates are of similar length. Huberman offers a working memory test on his podcast. It didn’t surprise me to learn that I have a below-average working memory capacity of 5, at least according to his test (perhaps explaining my need to reread this book!).\nWhile you might have suspected your memory to be unreliable, German psychologist Hermann Ebbinghaus used science to confirm it. In the late 19th century, he discovered the “forgetting curve” by memorizing 2,300 three-letter nonsense syllables and testing his recall at various intervals. His experiments demonstrated that memory retention decreases exponentially over time. This finding also led to the concept of spaced repetition, as the frequency of reminders needed to retain information also decreases exponentially. This principle is why I use Anki, a topic I’ll explore in another blog post. Ebbinghaus unfortunately did not repeat the experiment using mnemonic techniques in order to see the impact on the forgetting curve, but one suspects the curve will be less dramatic.\nSource: Wikipedia\nMnemonic techniques work exceptionally well because they leverage the human brain’s natural tendencies. Humans possess remarkable spatial memory. According to Foer, we can “learn and recall the layout, dimensions, decoration, and contents of a house, including the location of hundreds of objects, without consciously realizing it.” The concept of the memory palace exploits this ability to store and organize information.\nHuman memory is deeply associative. It’s far easier to remember information when it has something to “cling” to. This characteristic explains why knowledge compounds: the more you know, the more you can learn. Mnemonic techniques aim to create artificial associations to enhance retention.\nMnemonic Techniques Foer mentions several mnemonic techniques in the book. Below is an overview of those that left the strongest impression on me.\nMemorable Visual Representation The human mind is far more adept at recalling images than names and numbers. The fundamental principle underlying most mnemonic techniques is to create memorable visual representations of the information you want to retain.\nFor example, to remember the definition of “hermetically” (meaning “in a way that is completely protected from outside influences”), you might visualize a hermit sealed in a glass jar.\nSource: Mammoth Memory\nMemory Palace The memory palace technique dates back at least to ancient Rome, believed to be invented by Simonides of Ceos around 500 BCE. It exploits the human brain’s capacity for remembering spatial information.\nThe first step is to construct a “memory palace.” This need not be a literal palace; the most effective locations are those you know well, such as your home, office, or college dorm.\nImagine you have a grocery list with the following items: rice, milk, salmon, broccoli, ginger, lentils, and curry paste. The concept involves placing each item within your “memory palace” in bizarre and memorable scenarios.\nFor instance, you might envision:\nYour front door covered in a pile of rice, with rice raining down on you. Several people in your entryway sporting milk mustaches, asking “Got milk?” You open the door to your bedroom to find the bed replaced by a giant fish tank full of jumping salmon. Using this technique, you can more easily recall each item in the list, as well as the order of items.\nI’ve encountered several challenges in fully utilizing memory palaces. One significant issue is the surprising difficulty in creating large memory palaces. I’ve attempted to make one based on my usual neighborhood walk. While it’s easy to recall major landmarks like the park or ice cream shop, retaining more objects requires remembering finer details. I find myself questioning: Did the house with wooden shingles precede or follow the Victorian? How many houses stood between the park and Lombard Street? Maybe the ancient Greeks simply possessed a superior memory for spatial details, at least compared to me.\nAnother challenge is that memory palaces resemble linked list data structures in computer science. In linked lists, each element contains a pointer to the next element in sequential order. This structure has several drawbacks, including the need for sequential access, and inefficient insertion and deletion in the middle of the list. Similarly, if you forget a location in your memory palace, recalling the rest of the list becomes more challenging. Retrieving a memory from the middle of the palace might require starting from the beginning, slowing down recall . If you realize you need to add an item in the middle of your palace, you might need to shift all of the following items in your palace. Also, most information I want to remember doesn’t require exact sequential order.\nI’ve found memory palaces useful for various uses, such as recalling the main points of a presentation or managing task lists associated with specific responsibilities. For example, I use my office building as a memory palace for work-related tasks, and the route to the grocery store for shopping lists.\nHere’s a more detailed guide on memory palaces if you want to explore further: https:\/\/artofmemory.com\/blog\/how-to-build-a-memory-palace\/\nThe Major System The Major system is designed to help remember numbers, something that we are usually terrible at. Its core principle involves associating each digit with a specific phonetic sound, then transforming numbers into words that incorporate these sounds.\nEach digit maps to phonetic sounds as follows:\n0 -> s, z 1 -> T, D, Th 2 -> N 3 -> M 4 -> R 5 -> L 6 -> Ch, J 7 -> K 8 -> F, V 9 -> P, B The Wikipedia page has a more comprehensive description.\nUsing this mapping, you can transform numbers like 31415 into vivid imagery: MaT (31) covered in RaTs (41) chased by an eeL (5). As with the memory palace technique, the more unusual the imagery, the more effective it is.\nThe Major system’s primary advantage lies in its simplicity. Once you’ve memorized the digit-to-sound mapping, you can apply it to any number you wish to remember.\nThis was the main mnemonic technique that remained with me since my previous reading of Foer’s book. It can significantly enhance recall of short number sequences. I’ve successfully used it to remember lock combinations, dates, and the last four digits of credit card numbers.\nThe PAO System The Major system becomes less effective when dealing with truly large numbers, such as one hundred digits of pi. This is where the PAO (Person-Action-Object) system comes into play. Be warned: beyond this point lies the domain of serious memory geeks. During my previous reading of Foer’s book, I stopped practicing mnemonics at this point.\nThe PAO (person-action-object) system is designed to help memorize very long numbers, decks of cards, or really any other long sequence of elements. A standard version of it maps each number from 00 to 99 to a person, an action, and an object. For example, the number 42 might be Ron Burgundy, cooking, and hammer. To remember a longer number like 429087, one would create a vivid mental image combining the associated elements: Ron Burgundy (42) cooking (90) a dish of hammers (87).\nA significant drawback of the standard PAO system is the arbitrary nature of its subject-to-number mappings. For a two-digit system, this requires memorizing 300 distinct associations before one can fully use the system.\nTo work around this, one can combine the Major system with the PAO system. Using my previous example of 42, one could employ RoN Burgundy for the person, RuNNing for the action, and wReNch for the object. This approach significantly eases the memorization of mappings, though it comes with the trade-off that some numbers are hard to map to words or names, such as 88. In these cases, I simply chose something approximate or easily visualizable that wasn’t already assigned. In these cases, I simply chose something approximate or easily visualizable that wasn’t already assigned. This helpful blog post describes a Major PAO system in more detail.\nI also realized that no PAO system can be universally effective. For optimal results, it must be tailored to be memorable for you specifically. If someone suggests using GeNe Wilder for 62, but you’re unfamiliar with him, that association becomes effectively useless to you. The people you select must be ones you can visualize easily. For example, while you might be aware of Enrico Fermi, if you can’t picture his appearance, he would be a poor choice for your system.\nMindfulness An implicit element in all these mnemonic techniques is mindfulness. Applying any of these methods requires focusing on the subject for at least a brief moment. As I practiced these techniques, I discovered that even a weak encoding significantly enhanced my retention compared to my default strategy of making a “mental note” (assuming I was mindful enough to do even that). You can’t remember what you never paid attention to in the first place.\nThe Value of Mnemonics While preparing for the U.S. Memory Championship, Foer begins to question the relevance of mnemonic techniques. He wonders if these ancient methods are “fascinating for what they tell us about the minds of a bygone era, but as out of place in our modern world as quill pens and papyrus scrolls.” Even Francis Bacon believed that “the art of memory was fundamentally ‘barren’”.\nAn relevant case study is that of Kim Peek, the individual who inspired the movie “Rain Man.” Peek possessed a genuine photographic memory, having memorized Shakespeare’s entire corpus. He could read two pages simultaneously, using each eye independently, at a rate of ten seconds per page, and had read over nine thousand books. One might expect such capabilities to lead to extraordinary success. However, Peek had an IQ of just 87 and significant neurological and physical abnormalities. His case demonstrates that a powerful memory alone is insufficient for worldly success, at least in our current society.\nWe live in an era of abundant and increasingly affordable external storage. One might question the need to memorize anything when we can ask Siri to set a reminder or jot down a note on our phones. Is memorizing ten thousand digits of pi or the order of a deck of cards anything more than an elaborate party trick?\nOne compelling argument for cultivating our memories is that they constitute the essence of our personal identity. Foer notes that some memory athletes he encountered, including Ed Cooke, maintained that “memory training was considered a form of character building, a way of developing the cardinal virtue of prudence and, by extension, ethics.”\nFoer discusses the case of Henry Molaison, widely known as “H.M.,” one of neuroscience’s most extensively studied patients. Molaison underwent a radical surgical procedure in an attempt to cure his severe epilepsy. While the surgery alleviated his epilepsy, it left him unable to form new long-term memories, a condition known as anterograde amnesia. His memory remained frozen at the point of his surgery. This raises a profound question: Is it possible for an individual like Molaison to develop as a person without the ability to form new memories?\nFoer also argues that the perceived dichotomy between “learning” and “memorizing” is false. Effective learning, he contends, necessitates memorization, and when done properly, memorization inherently involves learning. He cites chess grandmasters as an example: studies have shown they typically possess average IQs, with their primary differentiator being their robust memories of past chess games. Expertise, in general, hinges on the ability to recognize familiar patterns and subsequently apply appropriate solutions \u2014 a concept Malcolm Gladwell explores in his book “Blink.”\nMemory may also play a crucial role in innovation. The term “invention” shares a common etymological root with “inventory.” To invent, one first needs a comprehensive inventory, or repository, of previous innovations and knowledge stored in memory.\nPersonally, I have found the mnemonic techniques described in this book to be beneficial, despite having a heavily customized Obsidian vault and an established Anki spaced repetition practice. There will always be information that is advantageous to recall without resorting to a smartphone. Additionally, there are often facts I need to retain temporarily before having an opportunity to record them. While I would readily adopt more advanced technologies like Neuralink once they become viable, until then, I’m continuing to develop my PAO system.\nConclusion Foer’s whirlwind tour through the history of mnemonics and modern memory competitions is more engaging than one might anticipate. The book leans towards journalistic narrative, featuring interviews with intriguing characters, rather than delving deeply into the science behind memory. Nevertheless, Foer’s immersion in the world of memory competitions and his ultimate victory attest to the effectiveness of the mnemonic techniques he describes.\nThe question of a good memory’s ultimate value in today’s age remains open for debate. Although Foer himself appears to question the worth of mnemonics, I personally find significant value in the ability (or even the attempt) to quickly commit to memory that which I deem important. “Moonwalking with Einstein” is as an accessible introduction to mnemonic techniques that readers can immediately put into practice, while also providing references for those wishing to explore the subject further.\n","permalink":"https:\/\/tenzinwangdhen.com\/posts\/moonwalking-with-einstein\/","summary":"

Joshua Foer’s “Moonwalking with Einstein” is one of the rare books that I found worth rereading. In it, Foer, a young journalist, enters the bizarre world of memory competitions after being assigned to cover the world championship. He describes in vivid detail the unusual characters he encounters, mnemonic techniques he learns, and books he reads to help prepare him for the U.S. Memory Championship. The “memory athletes” he interviews are able to memorize a deck of cards in thirty-two seconds, recall over eighty thousand digits of pi, and recite the entire works of Shakespeare. Foer’s efforts pay off, and he becomes the U.S. Memory Champion by the end of the year.<\/p>","title":"Book Review - Moonwalking With Einstein"},{"content":"Regardless of your views on David Goggins, it’s undeniable that he possesses an extraordinary level of persistence and willpower. Not only did he complete the grueling BUDS training for the Navy Seals three times, but he also finished US Army Ranger School and Air Force Tactical Air Controller training. He set a world record by completing 4,030 pull-ups in just 17 hours. Beyond that, he has finished over 60 ultra-marathons, triathlons, and ultra-triathlons. Remarkably, he achieved all this despite facing a challenging childhood, poverty, and battles with obesity, asthma, depression, and a heart condition.\nGoggins consistently pushes the limits of human endurance. His life story demonstrates that tenacity and willpower are not necessarily innate; anyone can develop them.\nWhat distinguishes Goggins from the average individual could be rooted in the neurological underpinnings of willpower and tenacity, within brain regions called the anterior midcingulate cortex (aMCC) and the anterior cingulate cortex (ACC).\nThe Neuroscience of Willpower The aMCC and ACC, key components of the brain’s salience network, play crucial roles in recognizing reward-related feelings and assessing the effort needed to achieve certain goals. These regions sit at a central location in the brain and maintain robust connections to various areas involved in planning, motor control, and sensory processing. Researchers believe that this central positioning enables their critical roles in willpower and determining the effort required for specific rewards.\nSource\nWhat role does the aMCC play in determining why some people give up when faced with a challenge while others persevere? Consider the last time you needed to use willpower, whether during the final mile of a run, while tackling a coding problem, or resisting a bowl of ice cream. You likely weighed the benefits against the drawbacks without even realizing it, before deciding whether to give up or keep going. The aMCC is believed to be crucial in this decision-making process.\nWillpower appears to involve balancing the perceived costs against the anticipated rewards. The ACC is key in assessing these costs, receiving cost estimates from various interconnected brain areas.\nIllustration of cortical areas involved in effort cost estimation. The arrows indicate glutaminergic (Glu) pathways. Source: Training Willpower: Reducing Costs and Valuing Effort\nThe best proof of the correlation between the aMCC and willpower comes from stimulation studies, where the aMCC is electrically stimulated while behavioral response is measured. Stimulation of the aMCC was found to elicit various goal-directed behaviors. Another study found that patients had the feeling of “preparing for a difficult challenge”, and a “will to persevere” when their aMCC was electrically stimulated. They expected an imminent challenge, and had a determined attitude towards overcoming it. Check out the transcript of the fascinating interview.\nLesion studies in rats have shown that inactivation of their ACC causes them to decrease the amount of effort they were willing to expend for a reward. In humans, tumors pressing against the aMCC led to “indifference to the environment, change in personality or character, and a stuperous or comatose state”.\nPeople with depression seem to have a reduced activation and size of the aMCC, thus leading to greater apathy. The severity of apathy was found to be correlated with decreased gray matter volume in the aMCC. Those with neurodegenerative diseases such as Alzheimer’s and Parkinson’s disease also displayed greater apathy.\nHopefully by this point you’re convinced of the link between the aMCC and willpower. How can we enhance the activation of the aMCC, thereby boosting our willpower?\nEnhancing Willpower Several studies seem to point to aerobic exercise as a key method to enhance willpower. One study of with fifty-nine older (60\u201379 years) and 20 younger (18\u201330 years) adults took part in a 6-month study in which they underwent aerobic exercise training. It found that the older participants had significant increases in brain volume, especially in the ACC. Previous studies have also shown that chronic aerobic exercise can lead to the growth of new capilaries in the brain, enhancing cognition.\n“Superagers”, or elderly individuals whose cognitive performance is in some cases equivalent even to young adults, tend to have an aMCC of similar size to young adults. They often have a lifelong habit of aerobic fitness training. Another study found that aerobic exercise induced structural changes in the aMCC of adolescents that participated in 12 weeks of training.\nConsistent effortful exercise leads to the reduction of the the estimated effort costs, due to increased connectivity between the ACC and other large scale networks. In other words, the more you exercise, the easier the same amount of exercise feels, establishing a “virtuous circle.” This cycle means training enhances your ability to exert effort, which in turn allows for more intense training and further boosts in willpower. Goggins’ remarkable achievements exemplify the extreme pursuit of this virtuous circle.\nAnother behavior that showed meaningful enhancements to the ACC was mindfulness meditation. One study found that mindfulness meditation improved bloodflow in the aMCC.\nConclusion Although not everyone can or should reach the same level of willpower as David Goggins, the neuroscientific foundation of willpower and the effectiveness of behavioral interventions like aerobic exercise in strengthening it are evident. Starting a consistent exercise routine is one way to initiate a “virtous circle” leading to compounding benefits in terms of willpower and tenacity, helping individuals push their limits and achieve their goals.\nSources Aerobic Exercise Training Increases Brain Volume in Aging Humans Aerobic exercise impacts the anterior cingulate cortex in adolescents with subthreshold mood syndromes: a randomized controlled trial study An amygdala-cingulate network underpins changes in effort-based decision making after a fitness program Insights into Human Behavior from Lesions to the Prefrontal Cortex Motor and emotional behaviours elicited by electrical stimulation of the human cingulate cortex Neuroanatomical Characteristics of Geriatric Apathy and Depression: A Magnetic Resonance Imaging Study Short-term meditation increases blood flow in anterior cingulate cortex and insula The Will to Persevere Induced by Electrical Stimulation of the Human Cingulate Gyrus Training Willpower: Reducing Costs and Valuing Effort ","permalink":"https:\/\/tenzinwangdhen.com\/posts\/willpower-and-the-brain\/","summary":"

Regardless of your views on David Goggins<\/a>, it’s undeniable that he possesses an extraordinary level of persistence and willpower. Not only did he complete the grueling BUDS training for the Navy Seals three times, but he also finished US Army Ranger School and Air Force Tactical Air Controller training. He set a world record by completing 4,030 pull-ups in just 17 hours. Beyond that, he has finished over 60 ultra-marathons, triathlons, and ultra-triathlons. Remarkably, he achieved all this despite facing a challenging childhood, poverty, and battles with obesity, asthma, depression, and a heart condition.<\/p>","title":"Willpower and the Brain"},{"content":"We’ve lived with like ChatGPT for over a year now. Last year seemed to be the height of the hype cycle, at least for me. It felt like everyone, from friends to Uber drivers, was excitedly discussing the potential of this groundbreaking technology. However, as often happens with new technological advancements, the initial excitement has dimmed somewhat when faced with the reality of day-to-day use. The trough of disillusionment is real. In this article, I’ll mostly discuss the limitations of LLMs in technical tasks, though I believe these limitations apply to any sufficiently complex domain.\nWhat I anticipated would be a significant leap in productivity has turned out to be more of a gradual step forward. As a developer, a substantial portion of my work involves coding. For new projects or when crafting quick scripts in bash or Python, GPT-4, the current best LLM, proves to be an invaluable partner. However, when it comes to refactoring functions in a vast, monolithic codebase or debugging complex performance issues in large distributed systems, GPT’s assistance often becomes superficial, providing broad, generic advice rather than actionable solutions, especially without inputting an extensive amount of code. Not to mention the risk of hallucinations that can lead to calling imaginary libraries or functions that don’t exist.\nSource: reddit\nThe issue of context window size is a significant barrier. Ideally, if it were possible to input an entire million-line codebase into the context window, GPT might offer more useful insights. However, research has shown that LLM performance decreases as the context size grows, especially in the middle of the window.\nThis presents a challenge for troubleshooting large systems, which would require analyzing trends across numerous dashboards and millions of system logs\u2014a task far beyond the current capabilities of LLMs, given their limited context window compared to the virtually unlimited, albeit slower, processing capacity of the human brain.\nRetrieval-Augmented Generation (RAG) might offer some solutions, but its effectiveness heavily depends on whether it can locate the correct code blocks to put into context. This becomes challenging when the code lacks clear explanations or uses poorly named functions. My experiences with GitHub Copilot and Sourcegraph’s Cody have been mixed; Cody tends to identify relevant files more effectively but often overlooks critical code blocks and files, while Copilot Chat appears limited to analyzing snippets from a single file, which is hardly helpful. GPT-4 does a decent job if you’ve pasted in enough of your code in the chat, but this is impractical on large codebases, or if you don’t know what code would be relevant context to answer your question. Efforts to expand the context window to accommodate entire codebases efficiently are worth monitoring.\nAgent-based frameworks also show potential. Unlike the traditional question-and-answer model, agent frameworks like AutoGPT set a task or goal and automatically break it down into subtasks, utilizing the necessary tools (such as calculators, Python scripts, or searches) to achieve the objective. An example of this approach is Grit, which aims to tackle the refactoring challenge using agents. However, this technology is still in its infancy.\nAnother problem with LLMs, particularly advanced ones like GPT-4, is their latency. Generating a response can take over 30 seconds, depending on the size of the input and desired output, disrupting many users’ flow state. For specific needs, like looking up a library method, I still prefer Google for its near-instant results.\nAs with any hype cycle, the focus tends to be on the technology’s potential rather than its current state. Remember how we thought bitcoin would imminently take over the banking system? With this latest AI summer, it feels to many like Artificial General Intelligence (AGI) is just around the corner, and we can all lay back and let the machines do the work for us (or kill us all, if that’s your political bent). However, we seem to be approaching a point of diminishing returns with the current transformer architecture, with increases in number of parameters resulting in sub-linear increases in performance.\nLLMs are far from being able to replace a colleague who possesses years of nuanced knowledge about your code base and systems, someone who can immediately identify why a seemingly minor commit led to doubled latency across all your microservices. Organizations that adopt coding assistants without implementing proper guardrails and training might encounter decreased code quality. This issue arises because, while these tools make writing code easier, they do not necessarily facilitate its refactoring.\nRelying too heavily on it for answers can also impede knowledge acquisition. It is increasingly tempting for engineers to not learn esoteric concepts when they could so easily just ask GPT, but they will almost certainly get burnt by hallucinated responses or run into roadblocks during an incident due to the aforementioned limitations with LLMs. To paraphrase Charlie Munger, knowledge and concepts stick only when you’ve put in the effort to reach for it. We still need to increase our own circle of competence.\nAs LLMs continue to evolve, some of my concerns might soon become obsolete. A valid rebuttal to my critique could be that we are, once again, shifting the goalposts regarding our expectations from AI. LLMs are now capable of performing tasks that were deemed impossible just a few years back, yet our focus remains on their limitations. The rate of progress in AI capabilities is exponential, with each new advancement building upon the last. AI will inevitably become increasingly adept at the ‘how’ of implementation. The ‘what’ to build and the ‘why’ will likely remain human domains for a considerable time as these questions require a deep understanding of the problem domain, the current state of the art, and a certain amount of creativity. The equation changes entirely if and when we achieve superintelligence. By then, hopefully, we will pursue learning and growth for our own benefit and interest, rather than out of necessity.\nI am far from being a luddite and believe in leveraging technological advancements to their fullest. When used appropriately, a symbiotic relationship between humans and AI can be incredibly powerful, surpassing the capabilities of either on their own. An AGI could potentially serve as the ultimate personalized teacher, knowing precisely where a student is in their learning journey and how to challenge them appropriately. However, it’s crucial to remember that an AI’s understanding is not a substitute for our own. Until advancements like Neuralink evolve to enhance our cognitive processes directly, the responsibility for personal growth and knowledge acquisition lies with us.\n","permalink":"https:\/\/tenzinwangdhen.com\/posts\/gpt-cant-think-for-you\/","summary":"

We’ve lived with like ChatGPT for over a year now. Last year seemed to be the height of the hype cycle, at least for me. It felt like everyone, from friends to Uber drivers, was excitedly discussing the potential of this groundbreaking technology. However, as often happens with new technological advancements, the initial excitement has dimmed somewhat when faced with the reality of day-to-day use. The trough of disillusionment is real. In this article, I’ll mostly discuss the limitations of LLMs in technical tasks, though I believe these limitations apply to any sufficiently complex domain.<\/p>","title":"GPT Can't Think for You"},{"content":"“The Coming Wave” explores the advantages and risks associated with a surge of emerging technologies. Authored by Mustafa Suleyman, a British artificial intelligence researcher and co-founder of DeepMind and Inflection AI, the book extends beyond just AI. It delves into the merits and drawbacks of other developing technologies, such as quantum computing and synthetic biology.\nIn the initial sections of the book, the author paints a picture of the potential benefits of these emerging technologies, reading like utopian science fiction. It later shifts to outlining the potential threats, evoking a dystopian hellscape. The book ends with a series of recommendations on navigating the fine line between these two extremes. The following review reflects my interpretations of the author’s opinions, except where I specifically state my own views.\nThe Wave The central metaphor of the book is the “wave,” defined by Suleyman as a convergence of new, general-purpose technologies that emerge simultaneously and carry significant societal impact. Historically, there have been only 24 technologies of this magnitude, such as farming, the factory system, and electricity. Suleyman identifies the upcoming wave as comprising three primary general-purpose technologies: artificial intelligence, synthetic biology, and quantum computing. Everett Rogers, a technology scholar, characterizes technologies as “clusters of innovations” in which one or more features are interrelated. According to this perspective, the impending wave represents a supercluster.\nOnce established, technological waves are nearly impossible to halt. Technology tends to disseminate regardless of obstacles, a process primarily driven by two factors: demand and the subsequent reduction in costs to meet that demand. Historical attempts to impede technological proliferation, such as the Ottomans’ resistance to the printing press and the Luddites’ opposition to mechanized looms, have largely been unsuccessful. A notable exception is the proliferation of nuclear technology, which was curtailed through a concerted multinational effort, with demand confined mainly to nation-states. As long as technology remains useful, desirable, affordable, and accessible, it not only persists but also grows in influence, with these attributes reinforcing one another.\nHow people end up using technology is never certain. Johannes Gutenberg, for instance, did not intend to spark the Reformation; his aim was simply to sell more Bibles. The more general-purpose a technology is, the less control its inventors have over its applications.\nArtificial Intelligence As an AI researcher and founder, Mustafa Suleyman naturally dedicates significant attention to the subject. His assertion is that AI will eventually supplant what he terms “intellectual manual labor.” Over the coming decade, he anticipates that large language models (LLMs) will experience a substantial enhancement in their capabilities, coinciding with a dramatic decrease in costs by several orders of magnitude.\nInstead of concentrating on the pursuit of the elusive concept of artificial general intelligence (AGI), Suleyman proposes a focus on what he calls artificial capable intelligence (ACI). He views ACI, defined as the stage where AI can accomplish complex objectives with minimal human oversight, as the next significant phase in the evolution of artificial intelligence.\nSuleyman introduces the concept of the “modern Turing test,” which he defines as an AI’s ability to successfully execute the task of “making $1 million dollars on Amazon in a few months with just a $100k investment.” He believes that achieving this feat with minimal human intervention is possible within the next year, and likely to become fully autonomous within three to six years. The key challenge lies in developing an AI capable of hierarchical planning, effectively coordinating multiple goals and subgoals towards a singular objective. This concept bears a striking resemblance to the well-known hypothetical scenario of the “paperclip maximizer,” a thought experiment that illustrates the potential risks of an AI system single-mindedly pursuing a defined goal without ethical or safety considerations.\nArtificial intelligence introduces asymmetric risks, affecting entire societies rather than just individuals. Traditional risks, such as a car crash, typically impact only the parties directly involved in the incident. In contrast, an asymmetric risk in this context would be akin to a hypothetical scenario where a malfunction in Tesla’s autopilot system causes all Teslas to spontaneously take a sharp right turn (note: this analogy is my own, not Suleyman’s). This type of risk implies a broader, systemic impact, originating from a single source but affecting a wide array of users or stakeholders.\nThe advancement of artificial intelligence will invariably lead to the development of more sophisticated cyberweapons. Incidents like the WannaCry and NotPetya cyber attacks had significant impacts and were only neutralized due to exploitable flaws in their design \u2014 flaws that could have been easily rectified. Future generations of AI-powered cyberweapons are anticipated to possess the capability to modify their code in real-time. This adaptability will enable them to persistently scan networks, autonomously identify vulnerabilities, and exploit them effectively.\nThe advent of generative AI technologies also raises concerns about the proliferation of fake news and synthetic identities. With these tools, it becomes increasingly feasible to create convincing yet entirely fictitious accounts of events, complete with rich, detailed histories. This development poses a significant challenge to the discernment of truth, as the authenticity of information can be easily obscured.\nThis situation is further complicated by individual biases, where people’s perceptions of reality are influenced by their pre-existing beliefs and opinions. The scenario described in the Israel-Gaza conflict, where interpretations of an event like a rocket landing on a hospital are split along lines of bias, illustrates this challenge. People may perceive such footage as either fabricated or real, largely depending on their preconceived notions or allegiances. As generative AI continues to evolve, distinguishing between genuine and artificially created content will become increasingly difficult, amplifying the risk of misinformation and the manipulation of public opinion.\nThe first known case of an AI-caused human fatality involved an Israeli robot sharpshooter. This autonomous system, satellite-operated and able to fire 600 rounds per minute, was used in the killing of an Iranian nuclear scientist. Suleyman foresees that the full automation of militaries will lower the barriers to conflict, as the human cost for at least one side diminishes. He speculates that future wars might be triggered by AI systems responding to perceived threats, similar to overreactions observed in algorithmic trading, leading to conflicts initiated for reasons not fully understood by humans.\nSuleyman challenges the optimistic view that billions of people will transition to high-end jobs in the future. He posits that there will be limited domains where human capabilities surpass those of machines. This perspective aligns with that of Nick Bostrom, who suggests that human labor might eventually be valued solely for its artistic or sentimental significance, as performed by a human. Contrary to the historical trend where technological advancements created new, unforeseen job categories (like “social media influencer”), Suleyman believes this pattern will not continue in the era of advanced AI and automation.\nIf humanity succeeds in creating an artificial superintelligence, we would encounter what is termed the “gorilla problem.” This analogy draws on the fact that while gorillas are physically stronger than humans, they are kept in zoos by humans due to our superior intelligence. Thus, if an entity possessing intelligence surpassing that of humans were to emerge, it could potentially dominate us in a similar manner. This scenario underscores the concern that a superintelligent AI might not inherently share human goals or values, leading to what is known as the “alignment problem” \u2014 the challenge of ensuring that such an AI’s objectives are aligned with human interests and ethics.\nSynthetic Biology Suleyman turns his attention to another emerging technology: synthetic biology. His predictions for this field are diverse and profound. They include the development of organisms engineered with the precision characteristic of modern software, highly personalized health treatments tailored to individual genetic profiles, and the creation of “enhanced” humans with improved physical attributes. Additionally, he envisions the use of carbon nanotubes as interfaces connecting humans directly with the digital world, enabling new forms of interaction and integration between biological and digital systems. These advancements, according to Suleyman, represent just a few of the potential breakthroughs in the rapidly evolving field of synthetic biology.\nDeepMind, a pioneering company in the field of artificial intelligence, developed AlphaFold. This deep learning system has made a significant breakthrough in the scientific community by being able to predict the three-dimensional structures of proteins based solely on their amino acid sequences. This capability represents a major advancement in understanding protein folding, a complex and crucial aspect of biology, with far-reaching implications for medical research, drug discovery, and our overall understanding of life processes.\nTechnologies like AlphaFold and CRISPR unlock the ability to manipulate biology at the molecular level. It also enables more advanced gain-of-function experiments, which involve the deliberate modification of pathogens to enhance their properties, such as increased lethality or infectiousness. This type of research, while often aimed at understanding diseases better and developing treatments, carries significant risks. One of the potential dangers is the creation of bioweapons that could be designed to target specific populations based on their DNA. Such bioweapons would have the capability to cause selective, large-scale harm, posing a grave threat to global health security. The ethical and safety concerns surrounding gain-of-function research are substantial, especially considering the possibility of such advanced biological agents being misused.\nSuleyman also foresees a biohacking arms race, where some individuals enhance themselves to “post-human” levels. This scenario could lead to stark inequalities between enhanced and non-enhanced humans, raising critical questions about rights, access to technology, and the very definition of being human. The emergence of a biologically enhanced class could profoundly shift societal dynamics, governance, and ethical frameworks, necessitating a reevaluation of fairness and opportunity in a radically altered world.\nQuantum Computers Quantum computers, leveraging the principles of quantum mechanics, offer computing speeds vastly superior to traditional computers. Each quantum bit, or “qubit,” can exist in multiple states simultaneously, as opposed to the binary 0 or 1 state of classical bits. This capability allows quantum computers to process complex calculations at an unprecedented pace. Google’s quantum computer, for instance, completed a task in mere seconds that would have taken a traditional computer over 10,000 years. The computing power of a quantum computer effectively doubles with the addition of each qubit. This immense speed makes quantum computing exceptionally well-suited for optimization problems, which are central to many modern technological challenges. The advancement in quantum computing will not only boost AI development, due to its reliance on computing power, but also significantly aid synthetic biology, which benefits from AI’s capabilities. Thus, quantum computers are poised to be a catalyst in the rapid progression of both artificial intelligence and synthetic biology.\nOn the downside, quantum computing poses a serious threat to modern cryptography. Current encryption methods are secure because they rely on problems that are extremely time-consuming for traditional computers to solve. However, the extraordinary computational abilities of quantum computers could easily break these encryption algorithms. This vulnerability extends to a wide range of technologies, including cryptocurrencies like Bitcoin. The advent of quantum computing, therefore, necessitates the development of new forms of cryptography that can withstand the power of quantum processing.\nImpact on the nation state Suleyman emphasizes that technology inherently embodies a form of power, making it intrinsically political. The emerging technologies discussed significantly amplify the power accessible to individuals. This development leads to a paradoxical situation where power is simultaneously concentrated and dispersed, resulting in both the strengthening and weakening of existing power structures, including nation-states.\nThe original formation and justification of the nation-state revolved around the Hobbesian bargain, a trade-off between individual liberty and collective security. However, as Suleyman points out, if advancements in technology reach a point where a nation-state can no longer assure security, this foundational social contract comes into question. The evolution of technology, therefore, not only reshapes our physical and digital worlds but also has profound implications for our social structures and the very concept of governance.\nSuleyman envisions two primary paths that nation-states might take, with various possibilities in between. The first is what he refers to as the “zombie state.” In this scenario, a country maintains the outward appearance of a liberal democracy but becomes functionally hollowed-out. Core services in such states would be significantly weakened, and they would suffer from political instability and divisiveness. Suleyman observes early signs of this trend in the United States, indicating a potential shift towards this “zombie state” model.\nAt the opposite end of the spectrum, Suleyman foresees the potential emergence of totalitarian governments, which would make the oppressive regime depicted in George Orwell’s “1984” seem almost utopian by comparison. The uncritical and unchecked adoption of emerging technologies could pave the way for extreme state control. Such governments would possess unprecedented means to monitor and repress their populations, including the potential for genetic manipulation.\nSuleyman points out that early indications of technology-enabled population surveillance are already visible in countries like China. This development suggests a trajectory towards more invasive and comprehensive state monitoring and control, underlining the profound implications of how emerging technologies are adopted and regulated by governments.\nWinner take all The “superstar effect” is a phenomenon where leading players in a field capture a disproportionately large share of the rewards, significantly overshadowing others. This is similar to the “power law”, fundamental principle underlying venture capitalism as well as several other domains and phenomena. In this context, power law dynamics dictate that a small number of successful investments can yield returns of 100 times or more their initial value. Meanwhile, the remaining majority of ventures end up competing for a substantially smaller portion of the overall gains. This dynamic leads to a highly skewed distribution of success and rewards, with a few top performers reaping the majority of the benefits.\nThe advent of the “coming wave” of technological advancements is expected to intensify the “superstar effect.” This will result in the creation of even wealthier and more successful superstars in various domains. The returns on intelligence, driven by advancements in AI, quantum computing, and synthetic biology, are anticipated to compound exponentially. Consequently, a small number of highly capable organizations will reap massive benefits, leading to significant concentrations of wealth and power.\nThese developments will enable the rise of vast, automated megacorporations that shift value transfer from human capital to raw capital. In this scenario, the economic and societal influence of human labor and skills diminishes, while capital, especially in the form of advanced technology and infrastructure, becomes the primary driver of economic value. This shift could further exacerbate economic inequalities and alter the traditional dynamics of labor, capital, and production.\nThe trend of technological dominance and the concentration of power in a few large entities is already evident in the current business landscape. Big tech companies are among the largest in the world and, as a category, are larger and growing more rapidly than traditional Fortune 500 companies. These companies possess both the financial capital and technological resources to capitalize on the forthcoming wave of technological advancements, often leading the charge in developing these transformative technologies.\nFor instance, Amazon’s investment in research and development (R&D) is a testament to this trend. With an R&D expenditure of $78 billion, Amazon alone accounts for a significant portion of the global R&D spending, which stands at around $700 billion. This level of investment not only highlights Amazon’s role in driving technological innovation but also illustrates the scale at which these tech giants operate and their capacity to shape future technological landscapes.\nChina The game of Go holds significant cultural and intellectual importance in China, so the defeat of the world Go champion by DeepMind’s AlphaGo was a momentous event, often likened to a “Sputnik moment” for China. Just as the launch of Sputnik by the Soviet Union in 1957 galvanized the United States to invest heavily in space technology, rocketry, and computing, leading to its emergence as a superpower in these fields, AlphaGo’s victory may have a similar catalyzing effect on China’s pursuit of leadership in artificial intelligence.\nChina has since adopted an explicit national strategy to become the world leader in AI by 2030. This ambition is evident in the significant resources being allocated to AI research and development. For instance, Tsinghua University in Beijing has emerged as a leading academic institution in AI research, publishing more papers in this field than any other university worldwide.\nThese developments have not gone unnoticed in other countries. In 2021, the Pentagon’s chief software officer resigned, expressing a stark viewpoint on the global AI competition. He was quoted in the Financial Times saying, \u201cWe have no fighting chance against China in 15 to 20 years. Right now, it\u2019s already a done deal; it is already over in my opinion.\u201d This statement reflects the growing concern about the pace and scale of China’s advancements in AI and its implications for global technological and strategic balances.\nChina’s commitment to developing advanced technologies, including AI, is partly driven by demographic challenges. The country’s total fertility rate is among the lowest globally, leading to projections of significant population decline. For instance, the Shanghai Academy of Social Sciences predicts that China’s population could drop to around 600 million by the end of the 21st century.\nThis demographic trend means that China could face challenges similar to those currently experienced by Japan, where a shrinking working-age population is burdened with supporting an increasingly large retired population. The development and implementation of technologies like AI become crucial under these circumstances, as they can help mitigate the impacts of a declining workforce. AI and automation can potentially compensate for labor shortages, maintain economic productivity, and support the elderly, thus playing a vital role in managing the socio-economic consequences of demographic shifts.\nTech can save us The challenges highlighted by Suleyman, including climate change, an expanding retired population, and the need to sustain rising living and healthcare standards, are indeed pressing global issues. The anticipated decline in global population over the next century will further exacerbate these problems. As the ratio of workers to retirees shifts, traditional economic models and welfare systems will be strained, potentially making it difficult to maintain current living standards.\nIn light of these challenges, Suleyman presents a somewhat paradoxical argument: despite the various risks and drawbacks associated with new technologies (as discussed earlier in the book), he believes that these technologies are also crucial for addressing these major global challenges. This viewpoint aligns with the idea that technological advancements, if harnessed responsibly, can provide innovative solutions to complex problems like climate change, healthcare, and economic productivity.\nThe need for new technology becomes a balancing act between mitigating its potential negative impacts and leveraging its capabilities for the greater good. This perspective reflects a growing recognition that technology, while presenting risks, is also an indispensable tool in tackling some of the most daunting challenges facing humanity.\nWhat is to be done? In the concluding section of his book, Suleyman presents a comprehensive strategy for managing the upcoming wave of technological advancements. His aim is to maximize their benefits while mitigating associated risks. He introduces the concept of ‘containment’, which he describes as an array of interconnected and mutually supportive mechanisms spanning technical, cultural, legal, and political domains. These mechanisms are designed to maintain societal control over technology during periods of rapid and exponential change.\nSuleyman emphasizes that regulation alone is insufficient to manage these challenges, but acknowledges it as a necessary component of a broader strategy. To this end, he advocates for a large-scale initiative akin to the Apollo program, specifically targeting AI and bio safety. This program would involve hundreds of thousands of professionals working towards ensuring the safe development and deployment of these technologies.\nFurthermore, Suleyman suggests the implementation of legislation mandating that a certain percentage of corporate R&D budgets be allocated specifically for safety measures. This proposal aims to institutionalize safety as a core aspect of technological development, ensuring that as companies innovate, they also consistently invest in mitigating potential risks associated with their technologies. This approach reflects a proactive stance towards technological stewardship, prioritizing not just advancement but also the safety and well-being of society.\nSuleyman points out that the generality of an AI model correlates with its potential threat level. Thus, AI laboratories working on foundational AI capabilities, such as OpenAI, Anthropic, and Gemini, require particular scrutiny and oversight.\nHe observes that, as of now, there is no comprehensive, formalized global effort for routinely testing deployed AI systems or the necessary tools for such testing. To address this gap, Suleyman proposes the creation of an AI Audit Authority (AAA). This body would be dedicated to fact-finding and auditing the scale of AI models.\nThe AAA’s role would involve monitoring AI development, particularly when AI systems reach certain capability thresholds, and informing the public about these advancements. Its function would include posing critical questions about the systems’ capabilities, such as whether they show signs of self-improvement or can set their own goals. The establishment of such an authority aims to provide a systematic and transparent approach to AI oversight, ensuring that advancements in AI are closely monitored for safety, ethical considerations, and potential societal impacts.\nSuleyman’s proposal for managing risks in synthetic biology includes the SecureDNA program. This program envisages connecting every DNA synthesizer to a centralized, encrypted system that screens for pathogenic sequences. I’m not sure how well this would work, as I assume a motivated actor could easily remove or spoof the communications to SecureDNA.\nSuleyman also emphasizes the responsibility of those creating new technologies. He advocates for scientists and developers to adhere to ethical standards akin to the Hippocratic Oath. This approach is intended to ensure that only responsible parties produce the most sophisticated AI systems, DNA synthesizers, and quantum computers. At Inflection, for example, their AI named Pi (Personal Intelligence) is designed to exhibit caution, express self-doubt, and generally defer to human judgment. This to me seems a pretty weak defense, at least as described in the book.\nHe concludes by saying that safe, contained technology is not a final end state, but rather an ongoing process. Containment is a narrow and treacherous path.\nMy thoughts I can’t say this was an entirely enjoyable book, though I did like the sci-fi scenarios. It is very prone to make readers paranoid and\/or filled with existential dread. Some sections also felt somewhat dry and stiff. However, I do think it is an important book, because it describes all of the very real dangers we face with the coming (or arrived?) wave of technologies.\nSuleyman’s book echoes themes from several influential works on technology and societal change. It recalls “Guns, Germs, and Steel” by Jared Diamond in its historical analysis of technological revolutions. Similarities with Max Tegmark’s “Life 3.0” are evident in the discussions on biotechnology. The book also resonates with Alvin Toffler’s “Future Shock” in addressing the challenges of adapting to rapid technological change. Furthermore, it parallels “The Singularity is Near” by Ray Kurzweil and “Superintelligence” by Nick Bostrom in exploring the exponential growth and potential risks of AI. Unique to Suleyman’s narrative is the emphasis on the interplay and cumulative acceleration of various emerging technologies, highlighting compounded risks not typically addressed in these other works.\nOne could have the cynical outlook that Suleyman is advocating for regulation in order to cement his firm’s position. However, he does say that companies building general capabilities should face increased scrutiny, which would seem to include Inflection. I think for the most part, his suggested changes should allow for continued innovation without too much stifling red tape. I think monitoring model size might be the wrong parameter to monitor as LLMs continue to increase capabilities at smaller parameter counts (the new term being Small Language Models, or SLM).\nTime will tell whether Suleyman will turn out to be the modern day Malthus. During Malthus’ time, it was easier to project out the rise in population increase, but not so easy to predict the even greater increase in food production. This is because predicting the increase in food production required the generation of new knowledge, which was impossible to predict. Likewise, we simply can’t predict what new technologies will be developed that will help curtail or eliminate the threats outlined in this book.\n","permalink":"https:\/\/tenzinwangdhen.com\/posts\/the-coming-wave\/","summary":"

“The Coming Wave” explores the advantages and risks associated with a surge of emerging technologies. Authored by Mustafa Suleyman, a British artificial intelligence researcher and co-founder of DeepMind and Inflection AI, the book extends beyond just AI. It delves into the merits and drawbacks of other developing technologies, such as quantum computing and synthetic biology.<\/p>\n

In the initial sections of the book, the author paints a picture of the potential benefits of these emerging technologies, reading like utopian science fiction. It later shifts to outlining the potential threats, evoking a dystopian hellscape. The book ends with a series of recommendations on navigating the fine line between these two extremes. The following review reflects my interpretations of the author’s opinions, except where I specifically state my own views.<\/p>","title":"Book Review - The Coming Wave"},{"content":"Sebastian Mallaby’s “The Power Law” traces the early history of the venture capital industry up to the present day, highlighting its significant impact on the development of technology. Contrary to common claims that venture capitalists (VCs) are merely opportunistic lemmings following trends, the book argues that venture capital requires genuine skill and provides various examples to support this claim. It covers successful venture capital investments in tech giants like Amazon, Apple, Facebook, and Google, as well as notable failures such as WeWork and Theranos. Mallaby argues that these failures cannot be solely attributed to “traditional” venture capital, as non-traditional investors outside Silicon Valley played a significant role. The book also explores the expansion of China’s venture capital industry through the involvement of Silicon Valley VCs and offers recommendations for policymakers on navigating VC investment in China amidst recent political tensions.\nOverall, “The Power Law” is a comprehensive and well-written history of the venture capital industry.\nOrigins The venture capital industry can be traced back to the late 1930s. The term “venture capital” was first used by Lammot du Pont, the president of E. I. du Pont de Nemours & Company, in 1938 during a speech before the U.S. Senate Committee to Investigate Unemployment and Relief. He defined venture capital as capital that is invested in an enterprise without expecting an immediate return, but rather taking a chance on getting an ultimate return. Jean Witter of the San Francisco investment bank Dean Witter & Company also used the term in his 1939 address to the Investment Bankers Association of America. However, the term did not become widely recognized until the 1960s.\nFairchild Semiconductor The first breakout success of venture capital was Arthur Rock’s Liberation Capital financing of Fairchild Semiconductor. According to Arthur Rock, venture capital was more than just financial investment. It involved unlocking human talent, creating incentives, and fostering a new kind of applied science and commercial culture.\nBefore venture capital, startups faced a catch-22 situation. They needed financing to generate cash flows, but they also had to show cash flows to get financing. Venture capital addresses this by providing funding to unproven firms, with the hope that a small percentage of those firms will generate significant returns. Fairchild Semiconductor was one such firm, founded by eight scientists known as the “Traitorous Eight” who left Shockley’s company due to his poor management style. Unable to secure traditional financing, they were fortunate to receive funding from Liberation Capital. This investment paid off handsomely, with the company ultimately being valued at over $100 million in 1960.\nArthur Rock went on to co-found the Davis & Rock venture capital firm. They raised $3.2 million from thirty \u201climited partners\u201d, providing companies with the necessary funds to fuel aggressive growth. Ultimately, they achieved a 22x return for their investors, surpassing even Warren Buffett. One of their notable successful investments was in Intel. The success of the Davis & Rock model led to the emergence of several partnerships, driving the expansion of the venture capital industry.\nKleiner Perkins Kleiner Perkins was founded in 1972 by Eugene Kleiner, a member of the Traitorous Eight, and Tom Perkins, an executive at Hewlett-Packard. Tom Perkins played a significant role in the establishment of Genentech and was the first venture capitalist to openly embrace the role of promoter and front man, signaling to scientists that they were part of something glamorous and ambitious, beyond academia. Perkins contributed to Genentech’s culture by creating an environment where everyone, from the janitor to the top executives, was invested in the company’s success. He ensured that Genentech employees, including key contractors, received stock options, which increased incentives for researchers.\nPerkins also developed a strategy to mitigate risks associated with the venture, making it an attractive investment opportunity. He secured new financing rounds when necessary, attracting money from other investors by promising to achieve the next research milestone. This approach set the standard for stage-by-stage investing in startup companies.\nKleiner Perkins took an active approach to investment, going beyond traditional stock-pickers on Wall Street. They worked closely with the entrepreneurs they invested in and actively participated in the businesses. Following the example of Davis & Rock, they established a time-limited fund and committed some of their own savings. Notable investments by Kleiner Perkins include Genentech, Google, and Netscape.\nSequoia and Apple Sequoia Capital was founded in 1972 by Don Valentine. Valentine introduced a hands-on activism approach to venture capital, supporting brilliant but unconventional entrepreneurs. One notable example is Nolan Bushnell, an early video game pioneer, who held board meetings in his hot tub. Initially focused on the information technology sector, which was a niche field at the time, Sequoia Capital became a master of finance in the technology-driven twenty-first century. The firm is known for its disciplined approach and ability to work with challenging founders. Despite facing risks and losses in many venture bets, Sequoia has consistently achieved success, with several investments generating significant profits.\nApple was a company with a unique founder. At the time, Steve Jobs was not the refined, turtleneck-wearing guru we remember him as now, but rather a long-haired hippy who didn’t shower frequently. They faced rejection from every venture capitalist, but luck was on their side when Nolan from Atari referred them to Don Valentine. Nolan believed Don might be a good fit due to his reputation for working with difficult founders. Back then, Nolan could have purchased a third of Apple for $50,000 but declined the investment, instead referring Jobs to Don to soften the blow. Another investor, Mark Markkula, a veteran from Fairchild and Intel, was introduced to Jobs by Valentine. Markkula is considered the first “angel investor,” independently wealthy individuals (usually through exits from successful tech companies) who decide to invest some of their money in startups. Markkula’s technical background allowed him to recognize the value of Apple’s technology, and his industry connections were invaluable in helping Apple secure further investments and publicity. Today, Apple is the most valuable company in the world.\nAt this point, certain trends are becoming evident. The success of venture capital largely depends on dense networks of both talent and capital, as well as the fortuitous connections that result from them. Through Markkula, we can directly trace a line from Fairchild Semiconductor, the first VC-backed startup, to Apple.\nGrowth Investing Masayoshi Son, the founder of the Japanese firm SoftBank, pioneered the concept of “growth investing”. In 1997, he invested $100 million in Yahoo and doubled his money when the company went public the following year. Similarly, his $400 million investment in ETrade grew to $2.4 billion within a year. These investments far surpassed those made by previous venture capitalists, briefly making him the world’s richest man.\nTiger Global, another firm focused on growth investing, was established by Chase Coleman and Scott Shleifer. They applied hedge-fund and private-equity strategies to tech firms in emerging markets. Their approach involved identifying promising tech companies in these markets, often referred to as “the this of the that” (such as the Amazon of China or the Google of Russia), and making substantial investments as they entered their growth phase. This strategy helped Tiger Global become a highly profitable technology investment franchise.\nThe creation of Tiger’s private fund introduced a new type of technology investment vehicle. It moved away from traditional hedge-fund stock picking and instead focused on private technology investments. This model would later be successfully adapted by Yuri Milner in his investment in Facebook.\nTiger Global’s investment approach was not limited by geographical boundaries. For example, they invested $20 million in Chinese companies Sina, Sohu, and NetEase, despite not having visited the country. They were able to make these investments at a significant discount due to the SARS epidemic, which had deterred other investors. Within a year, the value of their investments in China grew by 5 to 10 times, increasing the fund’s value by $100 million. This investment strategy was inspired by Julian Robertson, the founder of Tiger Management, who believed that the best investment opportunities were often found abroad.\nVC in China American venture capitalists played a significant role in the development of the Chinese venture capital industry. The growth of Chinese venture capital followed a similar pattern to that of Silicon Valley: initially limited capital and investors, followed by an influx of money leading to an increase in venture capitalists and startups. Eventually, venture capitalists played a coordinating role in the face of intense competition among startups.\nShirley Lin, a pioneer of venture capital in China and a partner at Goldman Sachs, played a key role in bridging the gap between the U.S. and China. Fluent in both languages and cultures, Lin brought the U.S. venture playbook to China and supported several internet startups, including Alibaba. Initially, Goldman Sachs invested $1.7 million in Alibaba. Fifteen years later, when Alibaba had a successful IPO, that stake would have been worth an astonishing $4.5 billion. However, due to pressure from superiors at Goldman Sachs who viewed the Chinese investment as unprofitable, Lin gave up 17 percent of Alibaba, distributing it among four other investment companies. As a result, they did not fully realize the potential earnings.\nAlibaba has emerged as a formidable enterprise and a breeding ground for ambitious individuals who have gone on to create their own startups, thereby fostering a culture of entrepreneurship and innovation in China. Alongside companies like Tencent and Baidu, which also received U.S. capital, Alibaba has become a pillar of China’s digital economy. The success of Alibaba has demonstrated to other entrepreneurs in China the immense potential that can be achieved, inspiring them to dream bigger and make significant contributions to the rapidly growing economy. As a result, there has been a surge in the number of startups and venture capitalists in China, mirroring the growth experienced by Silicon Valley around 1980.\nTraditionally, Chinese laws have prohibited foreign ownership of Chinese companies, particularly in sectors such as website operations, in order to protect domestic industries and maintain control over the country’s economy. These laws did not recognize employee stock options or certain types of “preferred” stock that foreign investors, especially those from Silicon Valley, typically use to secure their rights in startups. Additionally, the listing of Chinese internet stocks on America’s Nasdaq market was considered illegal. These legal restrictions posed significant challenges for foreign investors, especially U.S. venture capitalists. However, over time, investors like Lin have developed workarounds to circumvent these limitations, such as establishing offshore entities and utilizing synthetic equity. Chinese officials have generally turned a blind eye to these workarounds, possibly because they have calculated that the benefits outweigh the costs.\nThe Youth Revolt and Yuri Milner A colorful episode in the book recounts Zuckerberg’s appearance at a Sequoia pitch meeting wearing pajamas. Instead of pitching Facebook, he presents a side project. This was clearly a snub, influenced by Sean Parker’s disdain for venture capitalists, who had ousted him from his own startups multiple times. The book describes this shift in power from venture capitalists to young, often contrarian entrepreneurs as the “youth revolt”. The founders of Google had managed to raise $1 million solely through angel investments, allowing them to exert more control over their venture capital backers than usual. The revolt was further fueled by the emergence of new venture firms such as Founders Fund and Y Combinator, which took non-traditional approaches to venture investing, signaling that the venture industry itself could be disrupted.\nYuri Milner’s investment in Facebook finalized the enthronement of the entrepreneur. Milner, a Russian tech mogul, meticulously compiled a comprehensive spreadsheet on consumer-internet businesses across multiple countries. This spreadsheet tracked metrics such as daily users, monthly users, time spent on the site, and more. Milner’s international experience, particularly his investment in VKontakte, a leading Facebook clone in Russia, convinced him that the idea of Facebook\u2019s market saturation was incorrect. He believed that while Facebook was not yet among the top five websites in the United States, it consistently ranked in the top three in other countries. Milner anticipated that if the U.S. followed the typical pattern, there was still significant growth potential for Facebook. Additionally, he believed that Facebook lagged behind foreign social-media sites in terms of converting users into revenues. Based on these analyses, Milner made a $300 million investment in Facebook.\nAt the time, Zuckerberg was in need of funds, but he was reluctant to surrender control to venture capitalists, possibly influenced by Sean Parker’s advice. In contrast to his traditional VC counterparts, Milner was satisfied with being a passive investor and did not even seek a board seat for his investment. Milner’s investment in Facebook demonstrated that a mature internet company could remain private and still raise substantial capital. This revelation led to the emergence of “unicorns,” private technology companies valued at over $1 billion.\nWework and Theranos After Milner’s successful investment in Facebook, investors became more willing to give founders autonomy in running their companies, providing minimal oversight. However, this approach proved disastrous with the rise and fall of WeWork and Theranos.\nBenchmark first invested in WeWork in 2012, primarily due to the charismatic co-founder, Adam Neumann, a former Israeli naval officer. WeWork’s business model involved renting out short-term office spaces with additional perks like fruit water, free espresso, and occasional ice-cream parties. Neumann’s marketing strategy successfully attracted a vibrant clientele to these office spaces.\nAt the time of Benchmark’s initial investment, WeWork had a plausible business model. It leased office spaces at affordable long-term rates and rented them out for shorter periods, marking up the prices. In 2012, the company even turned a profit. However, in order to justify the inflated valuations placed on it by later investors such as banks and mutual funds, WeWork had to grow rapidly. To achieve this, it reduced the rents charged to tenants. As a result, the company’s losses increased with each additional revenue.\nBy the beginning of 2016, Benchmark faced a dilemma. They had made a smart bet on a charismatic founder who was initially profitable. WeWork’s valuation skyrocketed from $100 million to $10 billion, a 100-fold increase. However, due to the arrival of reckless late-stage investors, led by Masayoshi Son, the founder was now losing money and accumulating conflicts of interest. The risk of WeWork’s inflated valuation collapsing towards its actual value became apparent.\nTheranos was a health technology company founded by Elizabeth Holmes, a Stanford undergraduate. The company claimed to have developed a revolutionary blood-testing machine that could provide cheap and accurate results from a small amount of blood. Holmes managed to raise significant funding for the company, mostly from investors outside of the traditional venture capital community. She also recruited high-profile individuals from Stanford’s Hoover Institution to serve on the company’s board, adding credibility to the company.\nHowever, an investigation by The Wall Street Journal revealed that Theranos’ blood-testing machines were fraudulent and its promises of cheap and accurate results were misleading. As a result, the company faced numerous lawsuits and its value plummeted from $9 billion to zero. Elizabeth Holmes, once compared to Steve Jobs, faced the possibility of imprisonment. The downfall of Theranos was seen as a critique of Silicon Valley and its tendency to overhype and underdeliver on technological promises. Holmes was accused not only of lying about her technology’s capabilities, but also of making premature claims about its potential. Mallaby states that in both cases, traditional venture capitalists either did not invest in these companies or divested once problems with their operations became apparent. Elizabeth Holmes raised very little money from practitioners on Sand Hill Road, the heart of the venture capital industry. While Benchmark initially invested in WeWork, they eventually pulled out when Neumann attracted significant investments from investors like Son. Mallaby defends traditional venture capitalists, arguing that they would not have allowed such fraudulent companies to thrive.\nThe Power Law The book introduces several “laws” that govern VC investing. The most prominent one is the eponymous “power law,” a statistical concept that explains how a change in one variable can lead to a significant change in another. In the context of this book, the power law describes how a small subset of investments generates the majority of returns for a venture fund. This phenomenon occurs because successful companies often leverage their advantages to gain even greater advantages, such as economies of scale, first mover advantage, and network effects. These breakout companies come to dominate their industries. Venture capital investments play a crucial role in enabling these companies to develop a “flywheel” effect, where success begets more success. Certain markets exhibit “winner take all” dynamics, where a single company owns a significant portion of the market share, as exemplified by Google.\nReturns among venture capital firms themselves also follow a power law, with top firms like Sequoia and Benchmark capturing the majority of venture capital returns. These elite firms have a larger network and stronger brand recognition among founders, which allows them to access the most promising early-stage companies. For instance, Bezos chose to work with John Doer from Kleiner Perkins, despite receiving a better term sheet, solely due to Doer’s legendary reputation. Additionally, these top firms are more successful at attracting other investors in later funding rounds and ultimately achieving liquidity on their investments. As the power law suggests, only a few firms can benefit from this flywheel effect, and the industry as a whole has an average return of approximately 10%.\nCriticisms of VC Venture capital has faced significant criticism, much of which Mallaby also highlights in his book. The industry is primarily dominated by white men from a handful of prestigious universities, although this is gradually changing. Another issue is the industry’s tendency to follow trends blindly, which can sometimes lead to disastrous outcomes. For instance, the recent collapse of Silicon Valley Bank, whose depositors were mostly venture capitalists and their portfolio firms, exemplifies this phenomenon. Fears of the bank’s collapse became a self-fulfilling prophecy within the close-knit community, resulting in billions of dollars being withdrawn from the bank within days.\nMallaby notes that venture capital-backed firms sometimes succeed in displacing established competitors, not necessarily because they have a superior product, but because they can undercut prices using their VC funding. Companies like Doordash and Uber heavily subsidized their services to consumers with VC funds. While consumers may benefit from artificially low prices in the short term, once these companies gain a significant market share, they can then increase prices.\nConclusion Overall, I found this to be an enjoyable read and learned a lot about the origins and dynamics of the venture capital industry. The book could be seen as glorifying and defending the venture capital industry, but it also did a fair job of addressing criticisms against it. I didn’t realize the breadth of topics covered in the book until I started writing this review; there is a lot to unpack and I may not have covered all the main points. I believe most venture capitalists would benefit from reading this book, and I would also recommend it to founders considering venture capital to gain a better understanding of venture capitalists’ motivations, as well as to those interested in tech history in general.\n","permalink":"https:\/\/tenzinwangdhen.com\/posts\/the-power-law\/","summary":"

Sebastian Mallaby’s “The Power Law” traces the early history of the venture capital industry up to the present day, highlighting its significant impact on the development of technology. Contrary to common claims that venture capitalists (VCs) are merely opportunistic lemmings following trends, the book argues that venture capital requires genuine skill and provides various examples to support this claim. It covers successful venture capital investments in tech giants like Amazon, Apple, Facebook, and Google, as well as notable failures such as WeWork and Theranos. Mallaby argues that these failures cannot be solely attributed to “traditional” venture capital, as non-traditional investors outside Silicon Valley played a significant role. The book also explores the expansion of China’s venture capital industry through the involvement of Silicon Valley VCs and offers recommendations for policymakers on navigating VC investment in China amidst recent political tensions.<\/p>","title":"Book Review - The Power Law"},{"content":"Jared Diamond’s Pulitzer Prize-winning book, “Guns, Germs, and Steel” begins with a question posed by a New Guinea politician named Yali: “Why do white men have so much cargo [i.e., steel tools and other products of civilization], and we New Guineans have so little?” The book attempts to answer this question by uncovering the historical and evolutionary reasons for the vast wealth disparities between nations. Diamond’s answer to the question is summarized as follows: “History followed different courses for different peoples because of differences among peoples’ environments, not because of biological differences among peoples themselves.\u201d\nIn the book, a dramatic encounter occurs between Incan emperor Atahuallpa and the Spanish conquistadors led by Francisco Pizarro in 1532. At that time, Atahuallpa was the absolute monarch of the largest and most advanced state in the New World, surrounded by his 80,000 soldiers, while Pizarro led a small group of 168 Spanish adventurers. Despite being vastly outnumbered and far from reinforcements, the Spanish were able to capture Atahuallpa.\nThe Spanish used their cannons to create a panic among the Incan troops who trampled over each other in an attempt to flee. The Incan armaments were no match for Spanish steel, turning the engagement into a slaughter. The Spanish were able to extract a king’s ransom and subsequently murdered the king anyway. Their eventual defeat of the Incans allowed them to dominate Peru.\nEngagements such as this were so one-sided due to several factors, including technological superiority, immunity to diseases, and political organization. The Spanish had steel weapons, gunpowder, and horses, while the Incans had bronze weapons and no horses. The Spanish were also immune to diseases such as smallpox, measles, and influenza, while the Incans had no immunity to these diseases. The Spanish were also able to exploit the political divisions between the Incans and their neighbors. What led to such disparities in technology, immunity, and political organization?\nA significant turning point in human history was the shift from hunting and gathering to farming. Farming allowed for the production of a surplus food supply, which enabled certain classes of people such as rulers, politicians, and professionals to spend time on activities other than gathering food all day. In contrast, hunting and gathering cultures tended to be more egalitarian since everyone had to collect their own food, leaving little time for other activities. Although hunter-gatherers traded with farming cultures, they did not adopt farming due to various reasons. Ironically, farmers themselves tended to have less freedom and food compared to their hunter-gatherer counterparts.\nSeveral factors made it feasible for farming civilizations to flourish, one of which was the availability of large, domesticable mammals. These provided numerous benefits, such as meat and milk for consumption, land transport, military uses, and assistance with farming and cultivation.\nAs the largest contiguous landmass, Eurasia has the largest collection of domesticable mammals. Out of the world’s 148 large herbivorous mammals, only 14 have been proven to be domesticable. The five major domesticable mammals are cows, sheep, goats, pigs, and horses. All of these mammals could be found natively in Eurasia, but only a few were in the Americas and Africa, and none in places like New Guinea. For a mammal to be domesticable, it needs to have certain traits, such as the willingness to live in close proximity to other members of its species, the ability to breed in captivity, and a disposition that makes it easy to tame.\nAfrica has a variety of large mammals, so why weren’t more of them domesticated? Cheetahs cannot be domesticated due to their courtship ritual, which involves the female being chased by a pack of brothers; without this, the female does not ovulate. Zebras are not domesticable due to their sour disposition and tendency to bite their captors. Rhinos are not domesticable because they are unwilling to accept humans as the head of their herd. If Africans had managed to domesticate rhinos, rhino-mounted shock troops could have overwhelmed even Roman legionnaires. While all of these animals had been tamed at some point, domestication also requires that they can be bred in captivity.\nCivilizations also varied in their access to domesticable crops. The wild versions of crops like barley and peas were already edible and had high yields prior to domestication. They were also much easier to plant using the simple tools available to early humans and could be stored and eaten during winter months. Regions such as Australia, California, and the Argentine Pampas lacked domesticable plants and so did not develop indigenous food production. Conversely, regions like the Fertile Crescent and China, both in Eurasia, had a much larger share of domesticable crops, making them the earliest centers of food production.\nFarming cultures also tended to be much more densely populated and had herds of animals living in close proximity. This increased the likelihood of germs spreading between animals and, eventually, evolving to infect humans. Over time, farming cultures developed immunity to these diseases, but other cultures that had not been exposed to them remained vulnerable.\nUntil World War 1, germs were the biggest killers in human history. According to Diamond, the winners of past wars were often those with the nastiest germs to transmit to their enemies. More Native Americans were killed by the germs that the Spanish brought with them than were slaughtered with guns or steel weapons. It is estimated that 95% of pre-Columbian Native Americans were killed by diseases. Around 20 million Indians lived in the Americas, most of whom were killed by a host of diseases they had never before encountered. The key infectious diseases included smallpox, measles, influenza, plague, tuberculosis, typhus, cholera, malaria, and others. The reason that Native Americans were wiped out by Spanish diseases and not the other way around was that the Spanish came from farming civilizations with dense populations.\nAnother key factor in the differences between civilizations is the development and transmission of technology. There have been several cases where a culture adopts a technology, only to abandon it later on. For example, after being introduced to firearm technology by the Portuguese, the Japanese had perfected it, but their samurai class eventually banned them. Similarly, Aboriginal Australians abandoned such killer apps as the bow and arrow, and Aboriginal Tasmanians abandoned bone tools and fishing.\nWhy do some civilizations abandon technologies while others improve and innovate upon them? Diamond theorizes that the level of isolation of the civilization plays a major role. Some civilizations were isolated due to geography, such as Aboriginal Australians, who were cut off from Eurasia by water barriers in the Indonesian Archipelago. The orientation of continents also determined the level of isolation. Eurasia, with its east-west major axis, benefited from the fact that crops from the far east could thrive as easily in the west, since they share the same latitude and roughly the same climatic and ecological conditions. Africa and the Americas have a north-south major axis, making diffusion more difficult.\nThe Aboriginal Australians, Japanese, and Chinese all experienced the luxury and curse of isolation from other civilizations, which meant they didn’t have to compete with neighbors, but also couldn’t learn from them. Failing to innovate when surrounded by rival powers often spelled doom for the lagging civilizations, but isolated states didn’t face such pressure.\nIsolation also reduced the likelihood of civilizations encountering new technologies. It is often more difficult to invent a technology independently than to replicate something that already works. Written language was only independently invented by a few civilizations, including the Sumerians around 3000 BC, early Mesoamericans around 600 BC, and possibly Egypt and China (which is still debated). It was then adopted by neighboring civilizations through cultural exchange and trade.\nCivilizations such as the Polynesian Tonga, societies in subequatorial Africa and sub-Saharan West Africa before the arrival of Islam, and the largest native North American societies did not develop writing. These civilizations were more isolated, either by oceans in the case of the Tonga, or by north-south axes that slowed the diffusion in the case of Africa and North American Indians, who were otherwise connected by land to civilizations that had writing.\nFor most of human history, Western Europe was a technological backwater, receiving most of its technologies from China and Islamic nations. The region consisted of a large number of rival states, often at war with one another. However, this balkanization turned out to be a blessing. Failure to adopt a particular innovation might mean falling behind economically or militarily, so technologies were adopted rapidly and rarely abandoned.\nWhen Columbus pitched his plans to explore a new route to the Indies, he was able to present them to numerous princes before finally securing funds from the King and Queen of Spain. Once rivals saw the wealth that the Spanish were bringing back, they quickly jumped on the bandwagon.\nChina has been a center of innovation throughout most of human civilization, with a long list of inventions that includes gunpowder, cast iron, deep drilling, magnetic compasses, movable type, and printing, among others. The country was politically unified by 221 BC, which was facilitated by the many large rivers that flow through the country and connect the north and south.\nHowever, China’s unity became a disadvantage in contrast to Europe. It did not face the same pressure to continue innovating, and was able to close itself off from other cultures.\nSeveral decades before Columbus’s voyages, China had large ocean-going fleets, each consisting of hundreds of ships, some measuring up to 400 feet long. However, China abandoned these fleets after A.D. 1433, due to a power struggle between two political factions. The faction that had been sending the fleets lost power, and the fleets were stopped, shipyards dismantled, and oceangoing shipping was forbidden. When the political climate changed, shipbuilding couldn’t resume where it left off, as the shipyards were no longer available and no one could rebuild them. This scenario could not have occurred in Western Europe, where a rival nation would have continued shipbuilding.\nAnother interesting question is the outsized impact of idiosyncratic individuals, known as the “Great Man” theory of history. Why did certain cultures produce Einsteins, Guttenbergs, and Alexander the Greats, while others did not? Diamond’s theory is that these individuals benefited from environmental differences between cultures that enabled them to flourish. Diamond believes that the average Aboriginal person is more intelligent than the average Westerner, but Aboriginals do not have the benefit of a massive cultural and technological endowment. However, these “Stone Age” peoples can and do master industrial technologies when given the opportunity. If Einstein were born in New Guinea instead of the industrial era, he likely would not have discovered the special theory of relativity.\nDiamond concludes his book by defending the study of history. Despite its reputation as a less scientific field than physics, history is vital. It is not, as the saying goes, “just one damn fact after another.” Unlike the “hard” sciences, there is no way to rerun experiments and observe changes by controlling a single variable. However, Diamond notes that we can predict broad patterns and trends based on historical precedents, even if we can’t predict the outcome of near-term outcomes like elections. By studying history, we can gain insight into the possible development of societal structures or cultural norms under certain conditions.\nAs a history buff, I thoroughly enjoyed this book. It is packed with nuggets of historical insight woven together skillfully to answer a profound question. At times, it can be dry, such as when Diamond spends several chapters on the obscure farming and animal husbandry practices of long-lost civilizations, but all these details serve to back up his claims. Interestingly, the author spends much more time discussing seeds than steel, though “Guns, Germs, and Seeds” doesn’t quite have the same ring to it. Some might argue that the book is an exquisite exercise in conjecture, with examples cherry-picked to support Diamond’s claims. While I agree with the author’s general conclusion, those who oppose it might have a rougher time with the book.\n","permalink":"https:\/\/tenzinwangdhen.com\/posts\/guns-germs-steel\/","summary":"

Jared Diamond’s Pulitzer Prize-winning book, “Guns, Germs, and Steel”<\/a> begins with a question posed by a New Guinea politician named Yali: “Why do white men have so much cargo [i.e., steel tools and other products of civilization], and we New Guineans have so little?” The book attempts to answer this question by uncovering the historical and evolutionary reasons for the vast wealth disparities between nations. Diamond’s answer to the question is summarized as follows: “History followed different courses for different peoples because of differences among peoples’ environments, not because of biological differences among peoples themselves.\u201d<\/p>","title":"Book Review - Guns, Germs, and Steel"},{"content":"Melanie Michell is an AI researcher and professor at the Santa Fe Institute. In her book “Artificial Intelligence: A Guide for Thinking Humans,” she seeks to make artificial intelligence understandable for laypeople, and for the most part, achieves that goal. She begins by discussing the history of artificial intelligence, starting from the “perceptron” and working up to deep learning for natural language. Michell explains each concept in concrete terms and apt metaphors, with minimal technical jargon and math. Each \u201cAI spring\u201d, heralded by new breakthroughs and followed by breathless claims about potential revolutionary applications just around the corner, are inevitably followed by \u201cAI winters\u201d, where the technology falls short of expectations. She also delves into philosophical concepts such as the alignment problem, the singularity, and consciousness in AI systems.\nTo quote the end of the book:\nI hope that this book has helped you, as a thinking human, to get a sense of the current state of this burgeoning discipline, including its many unsolved problems, the potential risks and benefits of its technologies, and the scientific and philosophical questions it raises for understanding our own human intelligence. And if any computers are reading this, tell me what it refers to in the previous sentence and you\u2019re welcome to join in the discussion.\nOne of the challenges of writing a book in such a rapidly evolving field is that it is bound to become obsolete. Mitchell does not cover newer technologies such as LLMs and stable diffusion at all. Furthermore, she makes her stance on AGI risk clear: she thinks that the existential risk from AGI detracts from the real-world consequences of much dumber AI models that are already negatively impacting us today. While I do not fully agree with her philosophical stance, the book provides a great overview of the most important AI models and their history, as well as the philosophical discourse around AI.\nPerceptron Michell describes the perceptron as a single-layer neural network that outputs a prediction based on input weights. The weights are updated using a “loss function” with the goal of minimizing the loss over multiple iterations to achieve more accurate predictions. Frank Rosenblatt, a research psychologist at the Cornell Aeronautical Laboratory, invented the perceptron. He was inspired by the way neurons process information in the brain, where adjustments to the strength of connections between neurons play a key role in learning. He was also influenced by the work of behavioral psychologist B.F. Skinner, who used rewards and punishments to train animals. In the perceptron, correct firing is rewarded, while errors are punished.\nTo simulate the different strengths of connections to a neuron, Rosenblatt proposed that a numerical weight be assigned to each of a perceptron\u2019s inputs; each input is multiplied by its weight before being added to the sum. A perceptron\u2019s threshold is simply a number set by the programmer.\nIn 1958, The New York Times reported on the perceptron and predicted that it would “be able to recognize people and call out their names, and instantly translate speech in one language to speech and writing in another.” This was one of the first instances of the AI hype cycle that repeats throughout the book and history.\nReporting on a press conference Rosenblatt held in July 1958, The New York Times featured this recap: “The Navy revealed the embryo of an electronic computer today that it expects will be able to walk, talk, see, write, reproduce itself, and be conscious of its existence. Later perceptrons will be able to recognize people and call out their names and instantly translate speech in one language to speech and writing in another language, it was predicted.”\nHowever, it was eventually found that the single-layer perceptron could only learn linearly separable problems. The required compute for more layers, and thus more complex problems, did not yet exist. As a result, Marvin Minsky declared perceptrons a “dead-end”, and research stalled for decades. This marked the start of the first \u201cAI winter\u201d. Perceptrons are the foundation of the current deep learning revolution, who knows how much we would have progressed had research continued uninterrupted?\nMulti-layer perceptron Multi-layer perceptrons (MLPs) are based on the idea of the perceptron, but are more complex. An MLP has at least three layers: the input layer, the “hidden” layer, and the output layer. The input layer takes in numbers as inputs, the hidden layer processes information from the input layer, and the output layer produces the network’s output. The additional layers enable the MLP to learn much more complex functions than a perceptron. An MLP with more than three layers is called a “deep neural network” (DNN).\nConvolutional neural networks (CNN) Convolutional neural networks, or ConvNets, are machine vision models based on key insights about the brain’s visual system, discovered by Hubel and Wiesel in the 1950s and 60s. They found that neurons in the lower layers of the visual cortex are arranged in a rough grid. Each neuron in the grid responds to a corresponding area of the visual field.\nConvNets are deep learning models that transform an input image into a set of activation maps with increasingly complex features, via “convolutions”. The lower layers detect things like edges, while higher layers detect more complex features like shapes or faces. The features of the highest layer are fed into a traditional neural network, which outputs confidence percentages for categories known to the network. The network returns the category or label with the highest confidence as the image’s classification.\nAlexNet was the first to use ConvNets for the 2012 ImageNet competition, achieving 85% accuracy. This was a significant improvement over the previous record of 72%, which was based on support vector machines. The success of AlexNet can be seen as the start of the deep learning revolution. Interestingly, Ilya Sutskever, cofounder of OpenAI, helped create AlexNet.\nAlexNet, named after its main creator, Alex Krizhevsky, then a graduate student at the University of Toronto, supervised by the eminent neural network researcher Geoffrey Hinton. Krizhevsky, working with Hinton and a fellow student, Ilya Sutskever, created a scaled-up version of Yann LeCun\u2019s LeNet from the 1990s; training such a large network was now made possible by increases in computer power\nAlthough ConvNets are stronger than previous methods, they still have several issues. They can become confused by images that contain multiple objects, miss small objects in an image, and are easily thrown off by slight distortions and abstract representations. Additionally, they have been known to display bias. For example, the Google Photos app infamously tagged an image of two African Americans with the label “gorillas”. While CNNs excel at categorization, they still struggle with localization tasks, such as drawing a box around the target object. Mitchell also argues that CNNs are not truly “understanding” the images, but rather are just learning to recognize patterns.\nIf the goal of computer vision is to \u201cget a machine to describe what it sees,\u201d then machines will need to recognize not only objects but also their relationships to one another and how they interact with the world. If the \u201cobjects\u201d in question are living beings, the machines will need to know something about their actions, goals, emotions, likely next steps, and all the other aspects that figure into telling the story of a visual scene.\nReinforcement Learning Unlike supervised learning, reinforcement learning doesn’t require labeled data for training. Michell uses the analogy of a dog finding the path to food. The dog (agent), takes actions over a series of “learning episodes”, which consist of some number of “iterations”. At each iteration, the agent determines the current state and chooses the next action to take. If the agent receives a reward, it has “learned” something. The agent must balance between exploring new paths versus exploiting successful paths.\nThe promise of reinforcement learning is that the agent\u2014here our robo-dog\u2014can learn flexible strategies on its own simply by performing actions in the world and occasionally receiving rewards (that is, reinforcement) without humans having to manually write rules or directly teach the agent every possible circumstance.\nDeepMind used deep reinforcement learning to first beat Atari games, and then famously the board game “Go” with AlphaGo. Mitchell describes it as \u201cthe ultimate idiot savant\u201d, since it can only play Go very well but that intelligence fails to generalize to other tasks.\nNatural language processing (NLP) After the 1990s, rule-based approaches to NLP were overshadowed by more successful statistical approaches that used massive datasets to train machine learning algorithms. Mitchell describes several of the newer algorithms and techniques, including recurrent neural networks (RNNs), long-short term memory (LSTM), and word vectors.\nWord vectors embody the dictum, “you shall know a word by the company it keeps”. Words are converted into numbers that are then mapped (embedded) into a semantic space. Here, similar words like “king” and “queen” are closer together than “king” and “basketball”. This allows computers to “reason” about the meaning of words, enabling calculations such as “king” + “queen” - “woman” = “prince”. This is one of the key ideas behind more intelligent technologies, such as semantic search and ChatGPT.\nMitchell discusses the use of these technologies in tools like Google Translate. While deep learning performs remarkably well at translation, she doubts it will match the expertise of human translators for some time. This is because these technologies lack common sense knowledge and do not truly comprehend the content they are processing.\nI\u2019m skeptical that machine translation will actually reach the level of human translators\u2014except perhaps in narrow circumstances\u2014for a long time to come. The main obstacle is this: like speech-recognition systems, machine-translation systems perform their task without actually understanding the text they are processing\nThe book is starting to show its age because it doesn’t mention transformers, the most powerful NLP system yet created and used in ChatGPT.\nThe Turing test Alan Turing predicted that in 50 years, computers would be able to trick humans into thinking they are humans about 70% of the time. This became known as the “Turing test”. In 2014, a group of Russian and Ukrainian programmers won a competition held by the Royal Society by fooling 10 of 30 judges into thinking it was human.\nTuring did not specify the criteria for selecting the human contestant and the judge, or stipulate how long the test should last, or what conversational topics should be allowed. However, he did make an oddly specific prediction: \u201cI believe that in about 50 years\u2019 time it will be possible to programme computers\u00a0\u2026 to make them play the imitation game so well that an average interrogator will not have more than 70 percent chance of making the right identification after five minutes of questioning.\u201d In other words, in a five-minute session, the average judge will be fooled 30 percent of the time.\nThere is now a modified Turing test, designed by Kapor and Kurzweil and to be carried out by 2029. Three humans and one AI will be interviewed by three judges for two hours. The AI passes if it fools two or more judges. Kapor believes that machines will never pass the Turing test without the equivalent of a human body, while Kurzweil believes that AGI is imminent.\nMitchell Kapor. He made a negative prediction: \u201cBy 2029 no computer\u2014or \u2018machine intelligence\u2019\u2014will have passed the Turing Test.\u201d Kapor, who had founded the successful software company Lotus and who is also a longtime activist on internet civil liberties, knew Kurzweil well and was on the \u201chighly skeptical\u201d side of the Singularity divide. Kurzweil agreed to be the challenger for this public bet, with $20,000 going to the Electronic Frontier Foundation\nThe Singularity Mitchell shares her thoughts on the “singularity”: a future where the pace of technological change is so rapid that human life is irreversibly transformed. This idea is closely related to mathematician I.J. Good’s concept of an “intelligence explosion”, which postulates that an ultra-intelligent machine could design even more intelligent machines, leading to a recursive loop of self-improvement that would leave humans far behind. The resulting gap in intelligence could become as wide as the gap between us and ants. Ray Kurzweil, currently principal researcher, is the most famous proponent of the Singularity.\nKurzweil\u2019s ideas were spurred by the mathematician I. J. Good\u2019s speculations on the potential of an intelligence explosion: \u201cLet an ultraintelligent machine be defined as a machine that can far surpass all the intellectual activities of any man however clever. Since the design of machines is one of these intellectual activities, an ultraintelligent machine could design even better machines; there would then unquestionably be an \u2018intelligence explosion,\u2019 and the intelligence of man would be left far behind.\nI was inspired to pursue a career in technology after reading his book “The Singularity is Near” in college. Mitchell makes it clear that she does not believe in the Singularity.\nLack of common sense knowledge One of the core issues with current machine learning systems from Michell’s perspective is the lack of real understanding in these algorithms. Despite this lack, she is surprised that language translation has reached such a high level. These algorithms lack “common sense” knowledge, which is something that humans implicitly know and therefore do not explicitly write down. Instead, the machine learns from what it observes in the data, which may not align with what humans would observe. A good example of this is the Winograd schemas, which are sentences where changing one word of the question changes the expected answer.\nSENTENCE 1: \u201cThe city council refused the demonstrators a permit because they feared violence.\u201d QUESTION: Who feared violence? A. The city council B. The demonstrators SENTENCE 2: \u201cThe city council refused the demonstrators a permit because they advocated violence.\u201d QUESTION: Who advocated violence? A. The city council B. The demonstrators Sentences 1 and 2 differ by only one word (feared \/ advocated), but that single word determines the answer to the question. In sentence 1 the pronoun they refers to the city council, and in sentence 2 they refers to the demonstrators. How do we humans know this? We rely on our background knowledge about how society works: we know that demonstrators are the ones with a grievance and that they sometimes advocate or instigate violence at a protest.\nMichell tells the colorful story of “Clever Hans”. This horse gained fame because people believed it could “calculate”. When asked a question like “What is 5 times 3?”, it would tap its hoof the correct number of times. However, it was eventually revealed that the horse did not actually understand math. Instead, it was responding to subtle cues given by the questioner. Michell uses this as a metaphor for something that appears to understand, but in reality, does not understand anything at all. This still holds true today, as even the best LLMs like GPT-4 are still prone to hallucinate, and certain prompts can illicit responses that make it clear that it currently has no common sense knowledge.\nThe need for embodiment According to Mitchell, the primary mode of human learning is experiential, with book learning as an added layer. She argues that AI can never attain a common sense understanding if it does not have a physical presence and experience in the real world. In support of this, she quotes Andrei Karpathy, who suggests that to build computers capable of interpreting scenes like humans, we may need to provide them with exposure to years of structured, temporally coherent experience, the ability to interact with the world, and some form of magical active learning\/inference architecture that is currently difficult to imagine.\nShe quotes Karpathy:\nA seemingly inescapable conclusion for me is that we may\u00a0\u2026 need embodiment, and that the only way to build computers that can interpret scenes like we do is to allow them to get exposed to all the years of (structured, temporally coherent) experience we have, ability to interact with the world, and some magical active learning\/inference architecture that I can barely even imagine when I think backwards about what it should be capable of.\nPredictions At the end of her book, Michell lists some predictions for some of the most pressing questions in AI. She does not believe that mass unemployment or fully autonomous self-driving cars will happen anytime soon. According to her, creativity is not just about generating content, but also requires the ability to judge and understand what has been created. Therefore, AI is not creative in the full sense of the word. Mitchell also does not believe in an intelligence explosion or the singularity.\nAbove all, the take-home message from this book is that we humans tend to overestimate AI advances and underestimate the complexity of our own intelligence. Today\u2019s AI is far from general intelligence, and I don\u2019t believe that machine \u201csuperintelligence\u201d is anywhere on the horizon. If general AI ever comes about, I am betting that its complexity will rival that of our own brains.*\nIn fact, she states that the opposite of superintelligence is the problem: much dumber AI systems already deployed at scale causing issues such as bias, economic displacement, and increasing radicalization.\nI think the most worrisome aspect of AI systems in the short term is that we will give them too much autonomy without being fully aware of their limitations and vulnerabilities. We tend to anthropomorphize AI systems: we impute human qualities to them and end up overestimating the extent to which these systems can actually be fully trusted.\nIn response to the question of how far away we are from achieving AGI, Michell quotes Oren Etzioni, director of the Allen Institute for AI, who said, “Take your estimate, double it, triple it, quadruple it. That’s when.”\nIt’s an interesting thought that what are perceived to be human limitations - such as biases, emotions, and cognitive shortcomings - may actually enable humans to possess general intelligence.\nClosing thoughts Mitchell expresses a healthy dose of skepticism about the capabilities of AI, which may serve as a good antidote to the current massive hype cycle. Given AI’s history, it’s likely that we’ll experience another “AI winter” where the touted benefits never materialize. However, as the book contains the word “guide” in its title, it would be preferable if it were not biased in either direction. While it’s important to solve current issues with “dumb” AI, I believe that mitigating existential risks from AI is also necessary. I lean more towards the positive end of the AI doomer to AI enabled utopia spectrum, and believe that AGI aligned with human values may be the last invention humans need to make. I would recommend this book to non-technical friends who want an accessible primer on the field of artificial intelligence.\n","permalink":"https:\/\/tenzinwangdhen.com\/posts\/ai-for-thinking-humans\/","summary":"

Melanie Michell is an AI researcher and professor at the Santa Fe Institute. In her book “Artificial Intelligence: A Guide for Thinking Humans<\/a>,” she seeks to make artificial intelligence understandable for laypeople, and for the most part, achieves that goal. She begins by discussing the history of artificial intelligence, starting from the “perceptron” and working up to deep learning for natural language. Michell explains each concept in concrete terms and apt metaphors, with minimal technical jargon and math. Each \u201cAI spring\u201d, heralded by new breakthroughs and followed by breathless claims about potential revolutionary applications just around the corner, are inevitably followed by \u201cAI winters\u201d, where the technology falls short of expectations. She also delves into philosophical concepts such as the alignment problem, the singularity, and consciousness in AI systems.<\/p>","title":"Book Review - Artificial Intelligence: A Guide for Thinking Humans"},{"content":"Generative AI PC Build Log Why With the emergence of generative AI, I finally had a compelling reason to build a new PC. Most deep learning projects require a dedicated GPU, making them difficult to run even on an M1 MBP. While cloud options like Paperspace and Colab exist, the GPUs they provide access to are limited. Paperspace, for instance, only allows access to the most basic GPU and denied my request for access to the RTX 4000. AWS was also an option, but it was somewhat inconvenient to use cost-effectively since you would have to stop the instance and snapshot the volume to save on costs. Having a local GPU is crucial to quickly run experiments at a small scale before deploying them to the cloud to run on bigger hardware. This helps me minimize the feedback loop and make faster progress. I had previously built a computer a couple of times in high school, mostly for gaming. I didn’t have a good reason to do it again until now.\nAnother option would have been to buy a pre-built RTX 4090 rig, which would have been easier. However, it would have provided less value for the money, with cheaper components. By building my own computer, I was able to get twice the amount of RAM, a larger SSD, and a nicer case for the same price as the pre-built option. Additionally, building my own rig provided a learning experience and made me more comfortable with changing out parts if necessary.\nBuild Specs CPU: Intel Core i9-13900K 3 GHz 24-Core RAM: G.Skill Trident Z5 RGB Series (Intel XMP) 64GB (2 x 32GB) GPU: Gigabyte RTX 4090 Power supply: 1000W Corsair modular power supply Motherboard: MSI MPG Z790 EDGE Wifi Case: Lian Li Dynamic EVO Case fans: Lian Li UNI FAN SL-INFINITY 120 RGB Triple Pack Black SSD: 2TB Samsung 990 Pro Thermal paste: Thermal Grizzly Kryonaut (1 gram) Magnetic screw driver: Spec Ops Tools Phillips Screwdriver, 2 x 4", Magnetic Tip RGB motherboard connector: Lian Li Strimer Plus V2 24 Pin USB Header splitter: 9pin USB Header Male 1 to 2 Female Extension Splitter Cable Connector Adapter Parts list: https:\/\/pcpartpicker.com\/b\/rZqp99\nPicking the parts When I was picking the parts for my PC build, I started by researching on PC Parts Picker. I was looking for a builds that included the RTX 4090, currently the most powerful consumer GPU. Most people seemed to be building for gaming or content production use cases, but the requirements for deep learning are not that different. The main thing is having a powerful GPU. Using PC Parts Picker, I was able to come up with a list of everything I needed and gradually swapped things out as I did more research on parts.\nI got all the parts on Amazon. I considered Newegg, but heard some horror stories online about Newegg making it very difficult to return defective parts. Amazon has a no questions asked return policy, which I did end up using for one of my RAM sticks with broken RGB.\nGPU I had already decided on the RTX 4090 as the centerpiece of my build. The only alternative would have been the RTX 3090, which has 50% fewer CUDA cores and lower performance in training throughput, as well as being less power efficient. However, the 4090 has the same amount of VRAM as the 3090, and costs about $1600 vs $1000 for the 3090.\nThe choice was mainly between the top-rated 4090 cards on Amazon, and I ultimately decided on the Gigabyte 4090 due to its positive reviews and quick shipping.\nCPU The decision was between Intel and AMD, specifically the Intel 13900K versus the AMD 7950X. The general consensus is that AMD provides better value for money but runs hotter. Although the 7950X used to be about $100 more expensive, the prices are now the same. While Intel has 8 more cores, AMD has a 5nm TSMC chip and a larger L3 cache, which can benefit machine learning workloads.\nIn the end, I chose the 13900K, but I would have chosen AMD if the prices were the same. For machine learning use cases, most of the load will be on the GPU anyway. Although the AMD Threadripper is the absolute best CPU for machine learning, its price tag of over $1k is prohibitive for hobbyists.\nMotherboard This one took a lot of research. My criteria were to find a motherboard that supports the Intel 13th gen chip, DDR5 RAM, and has built-in wifi, and is also reasonably future-proof. I initially started looking at the ASUS ROG Strix Z790-E, but some concerning reviews about reliability swayed me to look elsewhere. I really didn’t want to flash the bios to enable the motherboard to work with the latest Intel chip, but that seemed unavoidable unless I was willing to shell out for the most expensive motherboards. Eventually, I landed on the MSI MPG Z790 EDGE WIFI, which had solid reviews and offered good value for the price.\nFor those with a larger budget who want a 2x 4090 setup, you\u2019ll need to consider a motherboard that supports two GPUs. This will also require a larger power supply and case, and possibly a beefier CPU like the AMD Threadripper for the additional PCIe lanes.\nCase This was a bit tricky. The 4090 is a huge card, so I needed a case that could contain it and be reasonably easy to build in. Several builds on PC Part Picker used Lian Li cases. While I liked the idea of going with the smaller Lian Li Dynamic Mini, the build seemed too adventurous for me. The last thing I wanted was to have to start over because something didn’t fit in the case. So I went with the Dynamic Evo.\nRAM Pretty easy choice here. I knew I wanted DDR5, which can be 50% faster than DDR4. I went with the G.Skill Trident Z5 since it was a popular option.\nPower supply The 4090 requires at least 450W. The i9 pulls up to 253W. So I went with a 1000W power supply. The Corsair I landed on is also fully modular, which made it easy to only connect the cables I actually needed.\nSSD This was an easy choice. I definitely wanted an NVMe for my main drive, and the Samsung is known to be reliable.\nAssembly Recommended Tools: Phillips #2 Screwdriver with magnetic tip. The magnetic tip came in really handy to pick up tiny screws. Zip ties. I got a variety pack from Amazon. Didn’t end up using too many, but good to have on hand. A bowl to hold loose screws. A bright work lamp. Optional: Anti-static wrist strap. I had this and started using it, but it was pretty restrictive to have on and I ended up doing most of the build without it. Updating the BIOS The first critical task I tackled was updating the BIOS on the motherboard. Thankfully, this process was pretty straightforward. There are numerous Youtube videos describing the process for any given motherboard.\nDuring the process, I noticed that the PSU showed no signs of life when connected to the motherboard, which gave me a moment of concern. I thought I received a defective unit and decided to test it using the “paper clip test.” When the PSU fans roared to life during the test, I realized they don’t activate unless under load.\nInstalling the CPU Next on the list was the CPU. The process was smooth and relatively easy to execute. It\u2019s a simple matter of lining up the arrows on the board with the ones on the chip. It is a delicate process though, make sure you don\u2019t bend or touch any pins or get dust in the socket or the chip.\nSnapping the RAM Into Place Installing the RAM was literally a snap. The only product defect I ran into during the build was that one of my RAM sticks did not light up. I was able to get it replaced via Amazon very easily with no questions asked.\nAside on overclocking: The RAM advertises a top speed of 6400 MHz, but by default was only 4000. I was unable to get Windows to start after setting the speed to 6400 in BIOS, the highest I was able to get to was 5600 MHz. This resulted in a 10% performance increase on the Time Spy 3d Mark test.\nNVMe SSD Installation I had to look up how to remove the cover on the motherboard for the SSD slot, but after that it was just a matter of screwing the SSD into place.\nCase Setup This is where things got somewhat hairy. The Lian Li case instructions are not super clear, and there are a million parts to deal with. It really helps here to stay organized and know where you place every screw. Use a bowl for loose screws. Thankfully there are a few good videos on my exact case, this one was pretty good: https:\/\/www.youtube.com\/watch?v=2v2fd1nrnkY\nImportant: Most CPU coolers require attaching a plate to the back of the motherboard. Do this before screwing the motherboard into the case to avoid any unnecessary backtrack.\nOnce the case was prepped, I screwed the motherboard into place.\nCase Fans This probably took the most time. I had to spend quite a bit of time thinking through fan placement. I had ordered 10 Lian Li SL-Infinity fans, by far the most case fans I’ve ever put into a PC. There’s some theory on the best fan placement to reduce dust build up and improve airflow. I ended up going with three intake on the bottom and three on the side, and three exhaust on the top and on in the back. I originally had the reverse setup, but the rear intake would just suck in hot air from the CPU. I miss out on the cool RGB effect of the fans on the side, but I think it’s worth it for the improved airflow.\nThe nice thing about the Lian Li fans was that they daisy chain pretty easily, so a set of three fans could be powered by a single cable. That cable then connects to a central hub that can be connected to up to four sets of fans, which is powered by a single cable from the power supply. The one issue was I had a hard time figuring out which side was the face (intake). It turned out that in my version, the mirrored side is the face. Lots of screwing fans in and out involved.\nApplying Thermal Paste Once all the fans were in place, I was ready to apply the thermal paste. The NZXT CPU cooler came with some preinstalled, but I saw it’s recommended to apply your own. I went with the technique of manually spreading the paste with the provided tool.\nInstalling the CPU Cooler The NZXT Kraken Elite is an “all-in-one” liquid CPU cooler that pumps cold liquid to the CPU and warm liquid back through a radiator. It comes with its own fans, but they did not work well with the Lian Li fans. The Kraken fans require their own controller hub, and the RGB did not sync well. Additionally, I had run out of RGB slots on the motherboard, so I used the Lian Li fans on the NZXT filter instead.\nPutting the head of the cooler on the CPU was actually pretty easy.\nFitting the GPU The 4090 is a beast of a GPU, taking up 3.5 PCIe slots. Thankfully it fit in the case with plenty of room. Installation proved pretty easy. Apparently the Lian Li case allows for vertical mounting of the GPU with a supplied bracket, but I didn’t really want to bother with that.\nI got scolded by folks on PC Parts Picker for not installing the GPU bracket. This was included with the Gigabyte RTX 4090, but I avoided installing it because my JUSB4 connector was too big and blocked the top part of the bracket. The GPU also seemed stable enough without it. However, it seems that the GPU is so large that overtime it will sag and potentially break the PCIe slot or the GPU itself. I ended up installing the bracket after all, I can live without a front panel USB-C port.\nConnecting Power Supply Since I got a modular power supply, I had to figure out exactly which cables I needed for all of the components. I hooked up the required cables to the power supply and routed the cables through the holes in the case to the components.\nTest Run At this point all of my main components were in. I connected all of the various wires to the motherboard. This was pretty easy because as long as you read the labels and connect the wires to the matching pins without forcing anything, it’s basically like legos.\nThankfully nothing blew up when I flipped the switch. I made sure that the computer booted up properly before moving on to cable management and cleanup.\nCable Management Managing the cables turned out to be relatively straightforward with the Lian Li case. All cables could be hidden at the back and fed through rubberized slots around the motherboard. After organizing the mess, a provided plate neatly covered the center gap in the case.\nMore pretty RGB After the build was complete, the RBG bug got me and I decided to get an RBG motherboard connector. This required a USB connector splitter, since I was out of USB slots on the motherboard. I also needed another Lian Li controller hub since I’d used up the four on the current controller. The result is a programmable RBG motherboard connector. This does absolutely nothing but feed my vanity and was cool for about five minutes. Pretty lights!\nInstalling Windows and Drivers You\u2019ll need to create a bootable Windows USB on another Windows PC. I thankfully have Parallels Desktop on my Mac, so I was able to do this pretty easily. You also need a wired Ethernet connection and wired mouse and keyboard, since wifi and bluetooth don\u2019t work without the necessary drivers installed. Once you get through the Windows installation process and install the drivers, you should be good to go!\nBenchmarks Cinebench 3d Mark https:\/\/3dmark.com\/spy\/39191470\nGenerative AI performance Stable Diffusion This thing can crank out basic 512x512 images in seconds, much faster than what my M1 Macbook Pro can do. I do run into CUDA out of memory errors if I try to upscale images too much, but for rapid iteration it’s great.\nimg2img of a screenshot of the completed build, with prompt: “futuristic cyberpunk gaming computer,neon,4k,highly detailed,large graphics card,water cooling,glowing\nLLAMA I’m able to load llama-7b fully on the GPU, and inference is very fast. The generated text is trash. Llama-14b requires putting some layers on the CPU, slowing down inference substantially, and the larger models are basically unusable. This is where a dual GPU setup would be nice. As LLMs get more efficient, I think this will be less of an issue.\nFinal Thoughts After several years of working with disembodied infrastructure in the cloud, there is something refreshing about working with physical hardware larger than a Raspberry Pi. Going through this experience was definitely a great learning opportunity, and I recommend doing it yourself if you have similar requirements. I hope you found this useful as another data point in your research.\n","permalink":"https:\/\/tenzinwangdhen.com\/posts\/ml-pc-build-log\/","summary":"

Generative AI PC Build Log<\/h1>\n

Why<\/h2>\n

With the emergence of generative AI, I finally had a compelling reason to build a new PC. Most deep learning projects require a dedicated GPU, making them difficult to run even on an M1 MBP. While cloud options like Paperspace and Colab exist, the GPUs they provide access to are limited. Paperspace, for instance, only allows access to the most basic GPU and denied my request for access to the RTX 4000. AWS was also an option, but it was somewhat inconvenient to use cost-effectively since you would have to stop the instance and snapshot the volume to save on costs. Having a local GPU is crucial to quickly run experiments at a small scale before deploying them to the cloud to run on bigger hardware. This helps me minimize the feedback loop and make faster progress. I had previously built a computer a couple of times in high school, mostly for gaming. I didn’t have a good reason to do it again until now.<\/p>","title":"2023 Generative AI PC Build Log"},{"content":"A Time Capsule of Startup Interviews “Founders at Work” by Jessica Livingston is a chronicle of startup stories, mostly from startups in the 80s and 90s. The book consists of a series of interviews between Livingston and the founders or early employees of various startups. It feels similar to tuning into a podcast from a decade or two ago, reminiscent of modern equivalents like “How I Built This” or “The SaaS Podcast”. With 32 interviews, the book offers a fascinating peek into the early days of several notable startups. Livingston, as one of the founders of the famous startup incubator Y Combinator, is well-suited to act as the interviewer.\nThe Allure of Hard Problems In the realm of startups, hard problems possess a peculiar charm. They attract the best minds, acting as a beacon for those who dare to challenge the status quo. TiVo, a company that dared to redefined how we watch TV, or Apple, where Steve Wozniak’s passion for designing elegant and simple computers led to the birth of one of the most influential tech companies. Brewster Kahle of Alexa Internet advises, “Pick a big, difficult project that will take years; otherwise, once you make money, you’ll run out of things to work on.” Similarly, Charles Geschke of Adobe Systems suggests it is wise (but risky) to develop a solution for where the market will be in a few years and wait for it to catch up, instead of just solving today’s problems.\nUnderstand your Customer The book emphasizes the importance of understanding your customers. Joel Spolsky, co-founder of Fog Creek Software, advises founders to engage with their customers, understand their needs, and disregard the competition. Sabeer Bhatia, co-founder of Hotmail, and Mark Fletcher, founder of ONElist and Bloglines, stress the value of a large user base, which provides a competitive advantage that is difficult to replicate and aids in potential acquisition negotiations.\nIdeally, you are your own customer and can adapt your product to meet your own needs that other companies also happen to have. A great example of this is Basecamp, which 37signals used to manage the Basecamp project itself. Designing a product for someone who is not you is difficult, so usability testing and customer interviews become even more important. A great product is often the best form of marketing, since your customers themselves will market the product if they love it and have the tools to do so.\nRemain Flexible Many startups initially set out to solve one problem, but ended up finding success in a different area. For example, PayPal began as a payments solution for PalmPilots, Hotmail started as a database, and Flickr started by designing a video game. Some companies even started as consulting businesses and then ended up creating products. Like unhappy families, all successful startups seem to find success in their own way.\nUltimately, great teams are more important than great ideas. They can pivot and adapt as requirements change, leading to success even in unexpected areas.\nVenture Capitalists: A Double-Edged Sword Throughout the book, Venture Capitalists (VCs) are frequently mentioned and often portrayed unfavorably. Paul Graham, co-founder of Viaweb, warns that money raised from VCs “will literally be pulled out of your ass”. ArsDigita, a company that accepted VC funding to expedite the IPO process, ended up ousting its founder and replacing management, leading to its downfall. (Although Joel Spolsky later notes that its lucrative consulting business was also drying up.) Steve Perlman of WebTV had another blunt take, saying VCs might see your company as carrion to sell for parts if it seems to be running out of money. VCs, driven by the need for exponential growth, may turn a successful company in the eyes of its founders into a “zombie.” The general consensus leans towards avoiding raising money if possible, and many founders highlight the importance of frugality.\nStartup Failures The book avoids the common pitfall of only highlighting successful startups, which can lead to survivorship bias. Instead, it openly discusses failures, such as ArsDigita and Alexa Internet, providing a balanced perspective on the startup journey. The book offers contradictory advice, acknowledging that there is no one-size-fits-all approach to building a successful startup. As Arthur van Hoff, co-founder of Marimba, wisely advises, “Don’t be afraid to listen to your gut instead of taking advice.\u201d\nCritique I didn’t realize that Livingston was one of the founders of Y Combinator until the end of the book. This information would have been helpful earlier on to provide context. Additionally, I was expecting the book to be more prescriptive about how to work as a founder, but instead it was in an interview format.\nDespite this, the book provides a lens into startup culture during a unique time in Silicon Valley. Many of the companies mentioned were unfamiliar to me, and the technologies discussed are primitive compared to what we have today. However, much of the advice remains relevant since it mostly has to do with human nature.\nOverall, I would recommend this as entertaining light reading to anyone interested in startups, entrepreneurship, or the tech industry.\nMy rating: 4\/5\n","permalink":"https:\/\/tenzinwangdhen.com\/posts\/book-review-founders-at-work\/","summary":"

A Time Capsule of Startup Interviews<\/h2>\n

“Founders at Work” by Jessica Livingston is a chronicle of startup stories, mostly from startups in the 80s and 90s. The book consists of a series of interviews between Livingston and the founders or early employees of various startups. It feels similar to tuning into a podcast from a decade or two ago, reminiscent of modern equivalents like “How I Built This” or “The SaaS Podcast”. With 32 interviews, the book offers a fascinating peek into the early days of several notable startups. Livingston, as one of the founders of the famous startup incubator Y Combinator, is well-suited to act as the interviewer.<\/p>","title":"Book Review - Founders at Work"},{"content":"Summary Made to Stick, by Chip and Dan Heath, explores the art of crafting memorable and influential ideas by outlining six key principles: Simplicity, Unexpectedness, Concreteness, Credibility, Emotions, and Stories (SUCCESs). Through engaging examples and stories, the book demonstrates how to apply these principles to create “sticky” ideas that leave a lasting impact. By overcoming the “Curse of Knowledge” and employing the SUCCESs framework, readers can improve their communication skills across various fields and contexts. Made to Stick offers practical guidance, backed by research, to help anyone create and share ideas that resonate with their audience.\nThe SUCCESs Framework The authors identify six key principles, encapsulated in the acronym SUCCESs:\nSimplicity: Aim for simplicity and conciseness. Distill the core message down to its essential elements, making it easy for people to understand and remember.\nExample: Using “jaws in space” to describe the high-concept movie “Alien” Unexpectedness: Capture attention by surprising your audience. Use unexpected information or stories to break through the noise and create a memorable impression.\nExample: The “Kidney Heist” legend, where a man wakes up in a bathtub full of ice and his kidneys removed. Concreteness: Use vivid, tangible language and examples to convey your ideas. Abstract concepts are harder to remember, so anchor your message in relatable, concrete terms.\nExample: The “Velcro Theory of Memory”, where an idea that has more sensory details or concepts are more likely to stick in people’s minds. A robbery description that vividly describes the stolen goods, the suspect’s appearance, and location are more memorable than an abstract description. Credibility: Enhance the believability of your message by providing credible sources, statistics, or testimonials. Leverage internal credibility, such as personal experience, as well as external credibility from authorities or experts.\nExample: The “Sinatra Test”, named after Frank Sinatra’s lyric “If I can make it there, I can make it anywhere”, referring to New York. Provides the example of a chef that earned his credibility by working at a prestigious restaurant in NYC. Emotions: Tap into emotions to create a deeper connection with your audience. Messages that evoke strong feelings, whether positive or negative, are more likely to be remembered and acted upon.\nExample: The story of Jared Fogle, of Subway fame. His personal story connected with people on an emotional level, making the advertising campaign more memorable and impactful. Stories: Use storytelling to engage your audience, make your ideas more relatable, and facilitate learning. Well-crafted stories help people visualize, understand, and remember your message.\nExample: A story about a Nordstrom salesperson that accepted a returned set of tires, even though the store doesn’t sell tires. The story illustrates the company’s commitment to exceptional customer service. The curse of knowledge The “Curse of Knowledge” is a central concept in Made to Stick. It refers to the cognitive bias where individuals with a certain level of knowledge or expertise struggle to imagine what it’s like to not possess that knowledge. As a result, they often have difficulty communicating their ideas to those with less knowledge, using jargon or abstract concepts that are hard to understand. This is why those who have just learned about a topic are often the best teachers.\nOvercoming the curse of knowledge is key to creating sticky ideas. The book has some recommendations:\nSimplify your message Use analogies and examples. Tap into schemas that are already familiar to the audience Test your message on a small group that represents your target audience Adjust your language and level of detail based on the audience What I liked “Made to Stick” provides a wealth of actionable advice and engaging examples. The book is well-structured, with a clear layout of the SUCCESs framework and each component of it. The principles are supported by research and real-world case studies, lending them credibility. Anyone whose job involves effectively communicating ideas can benefit from reading this book.\nWhat I didn’t like This is another book that could have been summarized in just a couple of pages, which is what I will attempt to do below. The book is filled with stories that elaborate on the key principles, but it doesn’t really explain why these principles are so effective. Instead, it repeats examples where they have been effective.\nMy Rating: 3\/5\n","permalink":"https:\/\/tenzinwangdhen.com\/posts\/book-review-made-to-stick\/","summary":"

Summary<\/h2>\n

Made to Stick, by Chip and Dan Heath, explores the art of crafting memorable and influential ideas by outlining six key principles: Simplicity, Unexpectedness, Concreteness, Credibility, Emotions, and Stories (SUCCESs). Through engaging examples and stories, the book demonstrates how to apply these principles to create “sticky” ideas that leave a lasting impact. By overcoming the “Curse of Knowledge” and employing the SUCCESs framework, readers can improve their communication skills across various fields and contexts. Made to Stick offers practical guidance, backed by research, to help anyone create and share ideas that resonate with their audience.<\/p>","title":"Book Review - Made to Stick"},{"content":"Kodak was once the dominant player in the photography industry, selling disposable cameras, film cartridges, and developing photos. However, they failed to invest in digital cameras, which they had invented. They were making so much money on film that they didn’t want to speed up the process of making the old technology obsolete. This approach made sense to Kodak, but customers increasingly preferred instant digital photos. Now, most Gen-Zs have probably never heard of Kodak.\nGoogle is currently the dominant player in the search industry, with almost 94% market share worldwide. Its lucrative business is being the gatekeeper between you and information. To users, its services are free, as the company earns revenue through targeted advertising slots displayed at the top of search results. Google’s advanced AI helps users find information, but its primary goal is to encourage ad clicks.\nEnter ChatGPT. Rather than searching on Google and browsing multiple sites, ChatGPT engages users in conversation, providing precise information without ads. But it has its flaws. Critics dubbed it a bullshitter and a stochastic parrot, hallucinating so much it must have macrodosed. However, the release of GPT-4 significantly enhanced ChatGPT, causing some critics to shift from skepticism to apprehension.\nSuddenly, it doesn\u2019t seem inevitable that Google will rule the universe with the most advanced AI and the most data. Ironically, Google invented the transformer, the “T” in “GPT,” but failed to invest sufficiently in its commercialization. This hesitance partially stems from the “innovator’s dilemma,” as described by Clayton Christensen. Established industry giants try to milk their aging but highly profitable technology until disruptive newcomers eat their lunch with incrementally improved innovations.\nEven when aware of the innovator’s dilemma, companies can still be disrupted by new technologies they fail to invest in. Andy Grove invited Clayton Christensen to discuss the innovator’s dilemma at Intel, yet Intel still succumbed to it. Intel clung to its outdated but lucrative x86 architecture and missed out on ARM, effectively losing the mobile market to Apple. Presently, Intel lags at least two chip generations behind TSMC.\nGoogle is making a killing on selling search ads. Replacing their search with LLM-based search would result in fewer searches, since users would find information faster without clicking on ads. While Google’s cost per search query is 1.06 cents, ChatGPT’s estimated cost per query is 36 cents. At Google’s scale, this translates to a $36B reduction in operating income. Google faces an innovator’s dilemma: whether to invest in a superior technology that could potentially cannibalize its profits.\nUnlike newcomers such as OpenAI and you.com , Google is a public company with shareholders to consider and a massive workforce of 190,000 employees. Even if CEO Sundar Pichai advocates for a full commitment to LLMs, he may struggle to gain shareholder support. Co-founders Larry Page and Sergey Brin would need to exercise their super-voting shares to redirect the company.\nTo be fair, Google is not completely sitting on its hands. It launched Bard, a ChatGPT competitor, which initially used the smaller LaMDA model but now employs PaLM , outperforming GPT-3 in some tests. Google also owns DeepMind and Anthropic, two of the most advanced AI teams in the field, and possesses one of the world’s largest data sets, crucial for training language models. However, Google had similar advantages before, such as in cloud computing, only to be outmaneuvered by more agile competitors like Amazon. The crucial question remains: Can Google invest sufficiently in this emerging technology to fend off disruption to their core business?\n","permalink":"https:\/\/tenzinwangdhen.com\/posts\/google-innovators-dilemma\/","summary":"

Kodak was once the dominant player in the photography industry, selling disposable cameras, film cartridges, and developing photos. However, they failed to invest in digital cameras, which they had invented. They were making so much money on film that they didn’t want to speed up the process of making the old technology obsolete. This approach made sense to Kodak, but customers increasingly preferred instant digital photos. Now, most Gen-Zs have probably never heard of Kodak.<\/p>","title":"Google's Innovator's Dilemma"},{"content":"TLDR: I built a semantic movie search app with GPT-4. You can check it out here: https:\/\/moviesgpt.net\/. And here’s the Github repo: https:\/\/github.com\/sinzin91\/moviesGPT\nthe hero keeps forgetting who he is\nIdea I’ve been thinking a lot about potential products that one can build around LLMs. One idea I mentioned jokingly to some friends was to use GPT as a natural language interface to a movies API.\nIn theory, I thought, you could ask GPT about movies semantically, meaning you could search based on what the movie is about or its content, instead of just raw keyword search. For example, I could search for “movies where all the main characters die” and get back “The Departed”. Then I could use the response to query a movies API like The Movies Database. Google is surprisingly bad at this, returning links to listicles instead of the movies themselves.\nThe current way to do this, and the one that even ChatGPT recommends, is to gather data on all the movies, generate word embeddings on that data, store it in a Vector database, and then query using something like cosine similarity. But GPT-3, having been trained on all internet data, has “seen” every movie and discussion around every movie already. It already has the embeddings. And it already “knows” how to find those movies given the right prompt.\nImplementation With this rough sketch of an idea, I sipped some coffee and entered this into GPT-4:\nYou are a master professional front end React developer. Show how you would build a front end that includes a search bar at the top. Below the search bar are six cards which include the movie cover with the title below it. Searches hit The Movie Database api's search endpoint with the search term. The results from the API call are used to update the cards below the search bar. The site uses a black and blue, dark mode style theme, with minimal UI components. I basically just wrote that off the cuff with no large amount of premeditation, and got back a very reasonable starting place for my app: Note: I am a hobbyist front-end engineer at best and only know enough React and Javascript to be slightly dangerous. I absolutely hate CSS, and can’t remember the difference between align-items and justify-content to save my life.\nTo my surprise, I was able to just copy paste the code from GPT-4 to get a decent looking site!\nI then asked to change it so that it calls OpenAI first:\nOk great! Now I want you to change it so that searches are sent to OpenAI's text-davinci-003 API. Hit The Movies Database API with each element from the returned array, and render those movies in the MovieGrid. Interestingly, GPT generated a prompt from my prompt:\nconst response = await openai.Completion.create( { engine: "text-davinci-003", prompt: `Find 6 movie titles related to "${searchTerm}"` }) This prompt is not precise enough though, so here I actually had to do some work:\nReturn an array of movie titles that best match this search term, ordered from most to least relevant. Generate up to 16 titles. If you are unable to answer the question, start your response with Sorry. Example: prompt: "movies with brando" response: ["The Godfather", "The Godfather: Part II", "Apocalypse Now", "The Wild One"] prompt: ${searchTerm} response: I’m giving GPT an example of the kind of response I expect. In this case, I want it to create a valid array of movie titles.\nWith this change, I had the first working POC of my original idea! The last time experienced such a sensation of god-like empowerment might have been unlocking robots in Factorio.\nin Factorio, eventually you can have robots automatically build things based on blueprints\nWhew, okay, great. Now what?\nI showed the app to a friend, and he mentioned that it looked terrible on mobile. So I asked GPT-4 to “change the CSS so the website works well on mobile phones.” Lo and behold, after copying over the CSS directly, the site worked flawlessly on mobile. No futzing with media queries for me!\nI also realized that I wanted to add star ratings to these movies to get a sense of their popularity. The TMDb API returns “vote_ratings,” which is a number from 0 to 10. I found the distribution of “vote_ratings” here: https:\/\/www.kaggle.com\/code\/erikbruin\/movie-recommendation-systems-for-tmdb.\nAdding a stars component was also mostly copy\/paste once I wrote the prompt:\nThe movie database api returns a `vote_average` in the response. I want a "Stars" component below each movie title that gives gives the movie 1 star if `vote_average` is between 0 to 2, 2 stars if it is between 2 to 4, etc. If a movie got one star, there should be a single bright yellow star and four grey stars in the component. I also wanted something to happen when I clicked on the cards. Maybe show the movie summary, release date, etc. Here’s the prompt: One thing that was bothering me in terms of practical usability was the lack of Rotten Tomatoes ratings. Often, that’s the first thing I check when deciding whether to watch a given movie. Unfortunately, Rotten Tomatoes charges $60k\/year (!!) for access to their API. Then I had the idea of asking GPT for the Rotten Tomatoes ratings. To my surprise, it does seem to have a sense for the ratings. Although they are not fully accurate, they seem to be generally in the right ballpark.\nI updated my prompt to generate a new data structure that includes the Rotten Tomatoes ratings:\nReturn a JSON object movie titles that best match this search term and their Rotten Tomatoes tomatometer score, ordered from most to least relevant. Get the most up to date and accurate Rotten Tomatoes tomatometer score. Generate up to 12 titles. If you are unable to answer the question, return a string that starts with Sorry. The response must be a valid JSON. Example: prompt: "movies with brando" response: [ { "title": "The Godfather", "rottenTomatoesScore": 98 }, { "title": "The Godfather: Part II", "rottenTomatoesScore": 97 }, { "title": "Apocalypse Now", "rottenTomatoesScore": 96 }, { "title": "The Wild One", "rottenTomatoesScore": 85 } ] prompt: ${searchTerm} response: I also had GPT make me the new Rotten Tomatoes rating component:\nNow I can safely avoid these movies!\nI was surprised by how well semantic search works out of the box with GPT. However, there are some limitations. One is that it only recognizes movies up to the date it was trained, which was late 2021. When asked about movies released after that, it simply responds that it doesn’t know. This issue may be addressed by the Plugins that OpenAI announced, which would allow GPT to call live APIs to fetch real-time data. One could also probably create a fine-tuned model that is trained on more recent movies.\ncan’t find movies from 2022\nI tried using gpt-3.5-turbo, but that model seemed to be much more prudish, refusing to respond to searches on movies deemed unseemly by the RLHF moderators. So I switched back to text-davinci-003 for now.\nLimitations To be clear, I still had to do some coding to get the functionality and styling to match exactly what I wanted. I don’t think someone without experience in React or Javascript would be able to replicate this. GPT still gets some things wrong, like telling me to use outdated libraries or generating wonky styles (who wants a blue background on a header??). But it’s getting pretty close.\nIn more ambitious projects, I’ve noticed that GPT starts to “forget” the code it had previously suggested. So when I ask to change some function, it spits out a new version of the file that has a completely different implementation and method names. Maybe this is a result of the limited context window. I’ve gotten around this by pasting in the code again and asking it to make the change given that context.\nClosing Thoughts I got the feeling that I was a tech lead pair programming with a very knowledgeable junior engineer. It doesn’t do well with big picture decisions, but knows how to implement very well given the right direction. To be fair, I’m not doing anything groundbreaking here. There must be tons of sites like this in the pretraining data, so it’s relatively easy for GPT to generate the relevant code.\nIt was also much easier to get into that elusive flow state while hacking this out. The dopamine hits from thinking of a feature and seeing it in action just kept coming, without the usual frustrations of not knowing how a particular function works and having to look it up. This makes coding fun again.\nIt’s clear that I am the bottleneck in this process. I’m sitting here copying and pasting from GPT with some slight modifications. It seems like the natural progression here is to have GPT directly create the code based on my prompts. I imagine you could build a more advanced “Create React App” that generates and updates the app based on your prompts.\nI hope this inspires people to pursue building those apps they always wanted to build, but couldn’t find time for. GPT still does not know what app to build and why it should build it, which is where we come in. Let a thousand flowers bloom!\n","permalink":"https:\/\/tenzinwangdhen.com\/posts\/semantic-movie-search-with-gpt\/","summary":"

TLDR:\nI built a semantic movie search app with GPT-4. You can check it out here: https:\/\/moviesgpt.net\/<\/a>. And here’s the Github repo: https:\/\/github.com\/sinzin91\/moviesGPT<\/a><\/p>\n

\"MoviesGPT\nthe hero keeps forgetting who he is<\/em><\/p>\n

Idea<\/h3>\n

I’ve been thinking a lot about potential products that one can build around LLMs. One idea I mentioned jokingly to some friends was to use GPT as a natural language interface to a movies API.<\/p>\n

In theory, I thought, you could ask GPT about movies semantically, meaning you could search based on what the movie is about or its content, instead of just raw keyword search. For example, I could search for “movies where all the main characters die” and get back “The Departed”. Then I could use the response to query a movies API like The Movies Database. Google is surprisingly bad at this, returning links to listicles instead of the movies themselves.<\/p>","title":"GPT-4 made me a GPT powered movie search app"},{"content":"TLDR “Peak: Secrets from the New Science of Expertise” by Anders Ericsson challenges conventional wisdom about talent, skill acquisition, and what it takes to achieve true expertise. Ericsson, a psychologist and researcher in the field of expertise, highlights several experiments to back up his claims, although some of the cited studies had small sample sizes. Despite containing some contradictions, the book offers a fascinating exploration of the science of expertise and empowers readers to pursue their path to mastery. Ericsson demonstrates that deliberate practice, not innate talent, is the key to mastery.\nDeliberate Practice Deliberate practice, a term coined by Ericsson, refers to a focused and intentional approach to skill development. It involves consistently working on specific areas of weakness or challenge, rather than merely repeating tasks or engaging in casual practice. To achieve this, it is necessary to set clear goals, seek feedback, and engage in continuous self-assessment to refine techniques and mental representations. Deliberate practice is designed to push individuals beyond their comfort zones, leading to significant improvements in performance and the eventual attainment of expertise in a given domain.\nEricsson’s concept of deliberate practice challenges the popular notion of Malcolm Gladwell’s “10,000-hour rule.” Merely spending time on a task does not guarantee expertise. It is essential to identify areas of weakness and actively work to improve them. For example, London cab drivers have developed larger hippocampi than bus drivers due to the former having to pass an extremely difficult exam and navigate to any location in London without a map.\nEricsson details an experiment in which a subject named Steve was trained to recall up to 86 numbers in a row after two years of deliberate practice. This was a significant improvement over the standard wisdom at the time, which held that working memory can only store 7 pieces of information, plus or minus two. Steve’s success can be attributed to his background as a runner, which made him accustomed to self-development and more likely to persevere through plateaus. Effective mnemonics were key to Steve’s progress, allowing him to surpass the typical working memory limit of seven numbers. Joshua Foer’s book “Moonwalking with Einstein” got me interested in mnemonic techniques, and it is interesting to note that he consulted with Ericsson.\nNeuroscience of practice Both the brain and body adjust to maintain homeostasis, a tendency towards equilibrium. For example, during endurance training, your body will eventually build additional capillaries to increase blood flow. Similarly, with the brain, neurons that “fire together, wire together”. Practicing just beyond your comfort zone signals to your brain that it needs to change to maintain homeostasis. Simple repetition does not have this effect.\nInterestingly, Ericsson relates an experiment that showed London cabbies who passed the test and started working were less effective at general recall than the control group. This might be because the adult brain does not experience as much neurogenesis, so the process of skill acquisition involves pruning inefficient synapses to be more efficient for certain tasks. This finding might have implications for what we choose to prioritize practicing.\nEricsson also mentions skilled “three-finger” braille readers who use each of their fingers for specific parts of the task. Brain scans showed that the parts of the brain that control these fingers become enlarged to the point where they overlap. These readers develop extreme sensitivity in their fingers but can’t tell which finger is being touched. The idea that the brain is not static, but can change and adapt with directed practice, is inspiring and motivating.\nThe Innate Talent Myth Ericsson challenges the belief in innate talent by first examining the concept of “perfect pitch,” the ability to identify any note in music, which was once considered a natural gift. Ericsson cites Mozart as an example of a prodigy who possessed this ability from a young age. However, the author argues that with just a year of training a few minutes a day, almost any child can learn to develop perfect pitch, provided they begin before the age of six. Even adults are able to learn this skill, although it takes longer and more deliberate practice.\nLater in the book, Ericsson debunks the notion of the savant, who may excel in a narrow field but has limited ability in other areas. He uses the example of “Donnie,” who could identify the day of the week for any date, to illustrate his point. Ericsson shows that savants often become fixated on their interests, leading to deliberate practice and mastery in a very specific domain. Outsiders only see the results of their intense focus and dedication.\nMental Representations Ericsson’s thesis focuses on the idea that deliberate practice requires the development of efficient mental representations. This concept is similar to the idea of “mental models” popularized by Charlie Munger. Ericsson uses the example of chess players who use “chunking” to recognize patterns on the board to illustrate this idea. Through deliberate practice, masters stop seeing individual pieces and instead recognize groups of pieces forming a semantic unit.\nChess players are often portrayed in popular culture as hyper intelligent, but Ericsson mentions studies that showed that adult chess players do not have a higher IQ than average. While children with high IQs may learn the rules faster, ultimately it comes down to who practices the most. Interestingly, Ericsson notes that Go players, on average, have a lower IQ than average, despite the fact that computers have yet to beat the world’s best Go players. This was published in 2015, and since then, AlphaGo famously beat the Go world champion.\nIn several sports anecdotes, Ericsson shows that athletes can surpass their predecessors by adopting better practice techniques, which are essentially better mental representations. He emphasizes the importance of learning from experts and adopting their practice methods to improve more rapidly.\nContradictions Despite its persuasive arguments, “Peak” is not without contradictions. For instance, Ericsson claims in one section that deliberate practice is not enjoyable for practitioners, while in another he says that it is. He asserts that deliberate practice can lead to virtually any accomplishment, yet also acknowledges that certain abilities can only be developed during childhood or for those with certain genetic attributes.\nEricsson notes that younger doctors are often more effective than those with 20 years of experience because their medical school knowledge is more recent. However, he also emphasizes that mere knowledge is not enough and that one needs to acquire skills through practice. While Ericsson tried to address both sides of the argument, it would have been helpful if he were more explicit about the contradictions in his thesis.\nApplication To make the idea of deliberate practice more concrete, let’s use the example of practicing guitar every day. A naive way to practice would be to mindlessly play through the same songs repeatedly each day, failing at the same sections each time. This mindless repetition can leave you stagnating in a well-worn rut, making you feel like you’ve reached the peak of your abilities. I’ll admit, this is often how I practice guitar.\nThe deliberate practice approach, on the other hand, is to focus on the difficult portions of the songs and practice those parts intensively. Once you have mastered the songs, you move on to more challenging pieces that are just outside of your comfort zone. It helps to measure your progress and note exactly which parts are giving you issues. You can also seek out other practitioners who can provide you with better and more efficient practice techniques. Though this new approach might be less fun, it is the way to mastery (assuming that’s your goal!)\nClosing “Peak” is a captivating and insightful examination of the science of expertise. Personally, I have started exploring how I can incorporate deliberate practice into my daily habits like meditation and guitar. By uncovering the secrets of deliberate practice, Anders Ericsson has provided readers with the opportunity to pursue their own path towards mastery, regardless of their background or perceived “talent.\u201d\n","permalink":"https:\/\/tenzinwangdhen.com\/posts\/book-review-peak\/","summary":"

TLDR<\/h3>\n

“Peak: Secrets from the New Science of Expertise”<\/a> by Anders Ericsson<\/a> challenges conventional wisdom about talent, skill acquisition, and what it takes to achieve true expertise. Ericsson, a psychologist and researcher in the field of expertise, highlights several experiments to back up his claims, although some of the cited studies had small sample sizes. Despite containing some contradictions, the book offers a fascinating exploration of the science of expertise and empowers readers to pursue their path to mastery. Ericsson demonstrates that deliberate practice, not innate talent, is the key to mastery.<\/p>","title":"Book Review - Peak: Secrets from the New Science of Expertise"},{"content":"TLDR I created SocratesGPT to test the concept of using AI to generate questions about a given topic. Its goal you learn topics better by asking you questions about it. I also provide some technical details on the implementation.\nWhy? Over a year ago, I began using Anki, a spaced repetition flashcard app that helps me retain information over the long term by taking advantage of the Ebbinghaus forgetting curve. Memory is like a leaky bucket, and it was incredibly frustrating to realize I’d forgotten most of what I learned. Anki has enabled me to confidently tackle difficult technical topics that I would have otherwise avoided.\nOne issue with Anki is that you can start to “overfit” on the questions you wrote. With Anki, you look at the card and then check the answer after trying to recall it. Once you’ve seen the answer, you rate how easily you were able to recall it, which determines when the question will reappear in your deck. Creating questions can be time consuming, so I’ve found that most of my cards end up being “cloze deletions”, i.e. fill-in-the-blank. This increases the risk of memorizing the answer to the card rather than truly understanding the concept.\nWhat if we could use AI to help us understand a topic better?\nSocratesGPT Unless you’ve been living under a particularly forlorn boulder, you’ve heard about ChatGPT. You may not know that you can access GPT-3.5 as an API. The most advanced model available via OpenAI’s API is gpt-3.5-turbo. This space is moving fast. When I started this project, the most advanced model was text-davinci-003 which was 10x more expensive and somewhat slower. Edit: now GPT-4 is out.\nGPT (generative pre-trained transformer) is a type of “large language model” (LLM) (sorry about the acronyms). By using “prompt engineering”, you can have the model return a structured JSON output for a given prompt, which can then be rendered in a UI. This means the app’s “backend” can consist of a single prompt.\nI saw a great example of this at https:\/\/github.com\/varunshenoy\/GraphGPT. It shows an example of using one-shot prompting to teach GPT to give a JSON structured response. I was surprised to find that this style of prompting always gives back valid JSON, assuming you also set the model parameters appropriately.\nI created SocratesGPT to test the concept of using AI to generate questions about a given text. The Socratic method is a questioning style that encourages students to discover their beliefs and uncover hidden assumptions. However, SocratesGPT does not fully follow the Socratic style, as there is no real dialogue between student and teacher. The name is more aspirational than descriptive.\nIn my prompt, I ask the model to impersonate Socrates, and generate questions based on the given text. I also provide an example of the expected response in JSON format. The React front end parses the JSON and renders it. If you ask for another question, the app updates the prompt with the previous state of questions and options selected and pass it back to the model.\nCreating a traditional backend to power an app like this without the use of LLMs would be highly non-trivial.\n“Prompt Engineering” Prompt engineering is the process of designing effective prompts for large language models such as GPT. Effective prompts are more likely to elicit high-quality outputs. Garbage prompt in, garbage response from the LLM.\nI started with a fairly simple prompt:\nYou are impersonating a question generator using the socratic method. You will be given a prompt, and you must respond with a question about that prompt. The response must be a valid json object with the following fields: prompt, question, answer, and correct. Eventually I had to add the following lines to get a cleaner response:\nYou cannot ask the same question twice. Try to limit the number of words in the choices to 4. If a choice is selected, it should be marked as selected: true. If a choice is not selected, it should be marked as selected: false. I had the feeling of training an overly eager genie when phrasing my prompt. When you ask a genie to stop the trolley from hitting the pedestrian, the genie might find blowing up the trolley to be a perfectly reasonable solution. So you end up adding additional clauses: “err, also don’t blow it up!”.\nThis is followed by a one-shot training example, where I provide a single example of what I want the model to do. I provide a starting state with my initial data structure:\nExamples: current state: { "counter": 1, "prompt": "", "questions": [{}] } prompt is the text the model should generate questions on. questions is an array of question objects.\nThen I show an example prompt, and the expected response from the model:\nprompt: The sky is blue. new state: { "counter": 1, "prompt": "The sky is blue.", "questions": [{ "question": "What color is the sky?", "choices": [{ "text": "blue", "correct": true, "selected": false }, { "text": "red", "correct": false, "selected": false }, { "text": "green", "correct": false, "selected": false }, { "text": "yellow", "correct": false, "selected": false }] }] } Each question includes the generated question, along with an array of choices. The selected boolean is used in the front end to maintain the state of which options the user clicked, so these aren’t lost when a new prompt is generated.\nFinally we create a way for us to dynamically update the prompt:\ncurrent state: $state prompt: $prompt new state: In the front end, we inject the current state into $state and the prompt into $prompt:\nfetch("prompts\/data.prompt") .then((response) => response.text()) .then((text) => text.replace("$prompt", prompt)) .then((text) => text.replace("$state", JSON.stringify(state))) We have some control over the parameters of the model, such as the temperature, max tokens, and top p. I found that the model works best when the temperature is set to 0.3. Temperature controls the “creativity” of the model. A lower temperature means the model is more likely to generate correct, almost deterministic responses. A higher temperature means the model generally leads to creative responses. frequency_penalty sets a penalty for repeating words. presence_penalty sets a penalty for repeating n-grams. top_p sets the probability mass function cutoff.\nconst DEFAULT_PARAMS = { model: "gpt-3.5-turbo", temperature: 0.3, max_tokens: 800, top_p: 1, frequency_penalty: 0, presence_penalty: 0, }; After the user clicks “Generate Question”, the value of $prompt is obtained from the prompt text area in the UI. The $state starts as a skeleton of the data structure provided in the training example. We call OpenAI’s new gpt-3.5-turbo chat completions API with data.prompt. The JSON response contained in data.choices[0].message.content is parsed to update the state and render the question in the UI.\nI found that the prompt in the new API needs to use single quotes instead of quotes when sent to the model, and then I have to replace the single quotes in the response with double quotes again to make it valid JSON. There’s probably a cleaner way to do this. I also feed the entire prompt as the role user, but some parts of it may be better suited to the system role.\nconst params = { ...DEFAULT_PARAMS, messages: [{ role: "user", content: prompt }], }; ... const requestOptions = { method: "POST", headers: { "Content-Type": "application\/json", Authorization: "Bearer " + String(apiKey || OPENAI_API_KEY), }, body: JSON.stringify(params), }; fetch("https:\/\/api.openai.com\/v1\/chat\/completions", requestOptions) .then((response) => response.json()) .then((data) => { const text = data.choices[0].message.content; \/\/ replace ' with " to make it valid JSON const new_state = JSON.parse(text.replace(\/'\/g, '"')); setState(new_state); If the user generates another question, we pass in the $state which now includes the first question. This lets the model we know what it already asked.\nLimitations There are several limitations with the current implementation:\nWhen providing very short input text, the model may generate questions that are not grounded in the text. For example, if the input text is “The sky is blue”, the model may generate a question like “Why is the sky blue?” even though the reason is not provided in the text. This is related to the “grounding problem”, which is one of the major flaws of LLMs. Shorter input text provides less grounding, thus allowing for more hallucinations. Occasionally, the model seems to “undo” changes in state. For example, a question might get removed from state on the next run. It can take up to 20 seconds to receive a response from the API. The response time seems to increase with the size of the prompt. This can make it less engaging. Newer versions of the OpenAI API will likely have faster response times. Setting frequency_penalty and presense_penalty parameters to 1 seems to result in invalid JSON. This might be because they make the model less likely to generate characters like { multiple times, which is required for the JSON to be valid. Future There’s a few ways SocratesGPT could be improved. Users could be able to save questions, and later be quizzed on them using spaced repetition. Larger context windows would allow the model to create questions on entire books. Eventually, GPT could be “fine-tuned” to ask more effective questions. You might even be able to implement reinforcement learning from human feedback (RLHF). Students could rate the quality of questions, which is then fed back into the model.\nAs multi-modal models mature, the AI could even generate diagrams, such as an image of a section of the brain with an arrow pointing to the area that the user is asked to label. Finally, the grounding problem should be addressed to prevent the model from “hallucinating” questions that are not in the input text.\nThe “T” in GPT stands for transformer, which is currently the most powerful deep learning architecture for several tasks, including NLP. There will likely be better architectures in the future. As the cost of GPU compute decreases, larger models with many more parameters become possible. By building on improvements like these, we can create ever more advanced AI tutors. These tutors will be able to understand your learning goals and current understanding to create personalized curriculums that are just difficult enough to challenge and ensure long term retention.\nAdvancing AI can help close the gap that MOOCs currently have difficulty filling, and help to scale education. Increased education leads to more people with advanced degrees that can help us solve some of our most pressing challenges. These are still the early days of LLMs.\n","permalink":"https:\/\/tenzinwangdhen.com\/posts\/augmenting-human-intelligence-with-ai\/","summary":"

TLDR<\/h2>\n

I created SocratesGPT<\/a> to test the concept of using AI to generate questions about a given topic. Its goal you learn topics better by asking you questions about it. I also provide some technical details on the implementation.<\/p>\n

\"SocratesGPT\"<\/p>\n

Why?<\/h2>\n

Over a year ago, I began using Anki, a spaced repetition flashcard app that helps me retain information over the long term by taking advantage of the Ebbinghaus forgetting curve<\/a>. Memory is like a leaky bucket, and it was incredibly frustrating to realize I’d forgotten most of what I learned. Anki has enabled me to confidently tackle difficult technical topics that I would have otherwise avoided.<\/p>","title":"Augmenting Human Intelligence With AI"}]