Conf42: Site Reliability Engineering 2021


Games We Play to Improve on Incident Response

Austin King
Founder @

Austin King's LinkedIn account Austin King's twitter account

Incident Response is a core competency for SRE teams, but how can teams practice and improve?

Inefficient incident response can be costly to a company. It causes lost revenue and destroys customer trust.

We will discuss mainstream games, a conceptual frameworks for creating team specific drills, and finally introduce an innovative research topic - outage simulation. An outage simulator gives SRE teams a tool for practicing incident response.

Core incident response skills are: severity triage, communication, delegation, and system familiarity. Drilling on these increases these skills, knowledge, efficiency, team cohesion and resilience.

SRE Managers and ICs will learn why games such as “Keep Talking and Nobody Explodes” are played by many SRE teams. They will learn in detail how to create their own fire-drills from existing runbook entries.

An overview of chaos testing and gamedays will be mentioned to provide the broader context. We will cover the pros and cons of each of these methods.

Lastly, we will present the concept of an incident response simulator as an open research topic. The hypothesis will be that an incident simulator is a good trade-off between non-domain specific games and full-blown Gamedays.

This is a talk illustrated with slides.

Awesome conferences for

Priority access to all content

Community Discord

Exclusive promotions and giveaways