What I built at Amazon
I joined Amazon in 2001 and earned the title “Master of Disaster.” I was responsible for the availability of every property bearing the Amazon brand. That scope took me across most of Amazon’s teams and systems.
I brought the power of the Incident Command System to Amazon when we desperately needed it to scale. Clear roles, practiced procedures, calm under pressure. I wrote the proposal that created Amazon’s System & Network Operations Center, the 24x7 operations center for the platform, and managed its team. For the 2004 and 2005 holiday seasons I established and led the Q4 WAR Team, the command center of senior leaders and executives responsible for a successful holiday for Amazon’s customers and merchant partners, before handing it off in 2006. I also created the Call Leader program, a rotation of senior leaders who served as the highest level of escalation for high-severity events. Call Leaders were what we called our SRE program. Google had a better name and the cool leather SRE bomber jackets.
I built the GameDay practice of deliberately breaking production systems so teams could practice before real failures hit. The best way to fix major failures was to create them. Adapting the Incident Command System I learned in my training as a firefighter and emergency manager, I also built Amazon’s incident management practice and early versions of practices like blameless postmortems and scalability testing, and put service owners on call for the systems they built. These are key components of what we now call DevOps and SRE practices. My team and I distributed about 4,000 pagers. I’m proud that the tools and culture I helped build at Amazon inspired PagerDuty’s founders, Alex Solomon, Andrew Miklas, and Baskar Puvanathasan, to start such an impactful company. I started advising them after their seed round, helped develop their incident management strategy, including describing PagerDuty as “9-1-1 dispatch” for IT, and helped them raise their Series A. I also worked to bring in Ron Vidal and Rob Schnepp of Blackrock Partners, the fire-service incident commanders who developed PagerDuty’s incident management playbooks. Netflix later adapted GameDay and named their version Chaos Engineering, which is a better name.
I also helped define and deliver Amazon’s architectural shift to always-on. Traditional disaster recovery meant staging cold or warm standby systems and rehearsing failover. We did the opposite, running services in active production across multiple data centers at all times. That changed the problem from recovering from a disaster to managing capacity and risk: a system that could absorb the loss of an entire data center without a recovery event. It became the foundation everything else was built on.
The turning point was watching a junior engineer physically shaking after they triggered an outage, terrified of being blamed. That moment convinced me that the punitive culture around failure was the real reliability problem. I shifted Amazon’s approach from blame to learning, making it safe to experiment, safe to fail, and safe to report problems honestly. You only get to do really big, great things when you are able to take great risks safely.
Further reading
- Prime Day 2017 – Powered by AWS — Jeff Barr, 2017, on GameDay, “which I believe originated with former Amazonian Jesse Robbins”
- Improving Resiliency with Chaos Engineering (DOP309-R1) — the AWS re:Invent 2019 session deck on chaos engineering at Amazon, opening with Jesse Robbins, “Master of Disaster,” and the 2004 origin of GameDay
- What failure looks like in Microsoft’s cloud-first world — Computerworld, 2014, on the GameDay program and the Master of Disaster years at Amazon
- GameDay: Creating Resiliency Through Destruction — USENIX, 2011
- Ex-Amazon ‘Master of Disaster’ Animates Server Chef — The Register, 2010
- The Origins of Amazon’s Cloud Computing — GigaOM, 2010
- Fireside Chat with Kolton Andrus — Failover Conf, 2021
- AWS Well-Architected: Game Day — GameDay is now a formal AWS reliability concept
- AWS Fault Injection Service — the managed service that automates the fault injection I created at Amazon
About Jesse Robbins
Jesse Robbins invests at the seed stage in AI developer tools and infrastructure. He cofounded Chef and the DevOps movement. Learn more about Jesse.