What I built at Amazon

I joined Amazon in 2001 and earned the title “Master of Disaster.” I was responsible for the availability of every property bearing the Amazon brand. That scope took me across most of Amazon’s teams and systems.

I brought the power of the Incident Command System to Amazon when we desperately needed it to scale. Clear roles, practiced procedures, calm under pressure. I wrote the proposal that created Amazon’s System & Network Operations Center, the 24x7 operations center for the platform, and managed its team. For the 2004 and 2005 holiday seasons I established and led the Q4 WAR Team, the command center of senior leaders and executives responsible for a successful holiday for Amazon’s customers and merchant partners, before handing it off in 2006. I also created the Call Leader program, a rotation of senior leaders who served as the highest level of escalation for high-severity events. Call Leaders were what we called our SRE program. Google had a better name and the cool leather SRE bomber jackets.

I built the GameDay practice of deliberately breaking production systems so teams could practice before real failures hit. The best way to fix major failures was to create them. Adapting the Incident Command System I learned in my training as a firefighter and emergency manager, I also built Amazon’s incident management practice and early versions of practices like blameless postmortems and scalability testing, and put service owners on call for the systems they built. These are key components of what we now call DevOps and SRE practices. My team and I distributed about 4,000 pagers. I’m proud that the tools and culture I helped build at Amazon inspired PagerDuty’s founders, Alex Solomon, Andrew Miklas, and Baskar Puvanathasan, to start such an impactful company. I started advising them after their seed round, helped develop their incident management strategy, including describing PagerDuty as “9-1-1 dispatch” for IT, and helped them raise their Series A. I also worked to bring in Ron Vidal and Rob Schnepp of Blackrock Partners, the fire-service incident commanders who developed PagerDuty’s incident management playbooks. Netflix later adapted GameDay and named their version Chaos Engineering, which is a better name.

I also helped define and deliver Amazon’s architectural shift to always-on. Traditional disaster recovery meant staging cold or warm standby systems and rehearsing failover. We did the opposite, running services in active production across multiple data centers at all times. That changed the problem from recovering from a disaster to managing capacity and risk: a system that could absorb the loss of an entire data center without a recovery event. It became the foundation everything else was built on.

The turning point was watching a junior engineer physically shaking after they triggered an outage, terrified of being blamed. That moment convinced me that the punitive culture around failure was the real reliability problem. I shifted Amazon’s approach from blame to learning, making it safe to experiment, safe to fail, and safe to report problems honestly. You only get to do really big, great things when you are able to take great risks safely.

Further reading

About Jesse Robbins

Jesse Robbins invests at the seed stage in AI developer tools and infrastructure. He cofounded Chef and the DevOps movement. Learn more about Jesse.

Related Topics