Many companies run tabletop exercises for security, reliability, and disaster recovery. There is a rich tradition of role playing and tabletop gaming to improve engineering rigor!

That’s why we decided to create a one-shot dungeons and dragons-style game about incident response to play as a team during our engineering offsite. We call it: Runbooks & Rollbacks.

Why play D&D to learn incident response?

Being an on-call engineer is a great experience if nothing goes wrong. But if an incident occurs, even one the team is prepared for, people can get stressed and confused.

Good incident response involves investigation, communication, speed, and kindness. Dungeons and dragons (D&D) is an excellent medium for cultivating those qualities! The format allows players to:

1. Learn in a low-stakes environment

2. Explore new abilities by role-playing

This was probably the nerdiest activity we did during our engineering offsite in Tomales Bay last year and our teammates enjoyed it whether they had extensive D&D experience or none at all. Since then, we have made it an annual tradition, with a new focus topic for each game.

In Runbooks & Rollbacks, we can cook up as intricate and challenging of a fake incident as we wish because it’s pretend and no one will be blamed for the outcome, which is also what we want in real incident management.

It also affords a space for players to branch out. You can either act as yourself or put on a totally new persona with powers that you don’t think you normally have. Imagining the abilities you want and how it would feel to use them is the first step to growth.

At the start of the session, we gave the players three ground rules:

• “Yes and…” → Build off of other peoples’ ideas, we’re creating a shared experience

• “I don’t know, so my question is...” → In a fast-growing software system, everyone has gaps in their knowledge, so ask good clarifying questions and focus on learning

• Chaotic good > chaotic evil → Use your imagination to help the situation, not hurt it

We structured the hour-long session like this:

• Character Creation (10m)

  • Choose who you want to be and fill out your character sheet

  • Does not have to be based on your work!

• Core Gameplay (40m)

  • Discuss as a team, take actions, roll dice

  • Dungeon manager tells you the outcomes

• Retrospective (10m)

  • Five whys + action items

  • Where did we get lucky?

Character creation

Every player filled out a little character sheet like this:

We wanted a diversity of perspectives on the incident teams and a framework that would both encourage creativity and support efficient setup.

To build a character, each player had to choose three things:

• Domain: What kind of engineering work does your character like to focus on?

• Origin: Which does your character like more: building for users or for systems?

• Class: What role does your character like to play in an incident team?

Based on the choices, each player received modifiers for our six core abilities and two skills for their character. Together, there were 108 unique character builds to choose from.

Ambience engineering has many generalists and specialists. We offered nine domains that our projects frequently touch:

• Product: building excellent experiences for clinicians

• Platform: supporting and scaling our products

• Data: analyzing outcomes and measuring ROI

• Mobile: creating a sleek assistant in the clinician’s pocket

• Model: building and testing AI tools and features

• Integrations: connecting with electronic health records and systems

• Security: protecting patient health information and privacy

• Quality: ensuring that our products always work

• Reliability: keeping our services up and performant

We offered two origins, a broad strokes way to describe what gets each character excited:

• Feature engineering: building for clinicians

• Infra engineering: building for engineers who build for clinicians

And we offered six classes, representing the roles needed in an incident:

• Druid: deeply in touch with the system, able to detect and prevent incidents

• Paladin: leaps into action to resolve incidents and mitigate damage to others

• Cleric: skilled in healing, can remediate and reflect for sustained safety

• Bard: positive and clear, keeping the lines of communication open and morale up

• Ranger: watches from afar, providing observability into threats and changes

• Monk: disciplined to maintain a solid core, ensuring reliability even under duress

As players introduced their characters, they could plug their choices into a nifty little Google spreadsheet we made that would output their ability modifiers and skills:

We adapted the D&D abilities into six abilities relevant to incident response:

• Systems (SYS): knowledge of infrastructure and its major characteristics

• Investigation (INV): ability to glean insights and narrow down hypotheses

• Endurance (END): ability to persist even when the trail runs cold

• Context (CTX): knowledge about relevant product and business activity

• Judgement (JDG): knowledge to make good, fast choices under pressure

• Poise (PSE): ability to stay calm and radiate positivity to others

During the game, players would roll checks against their strengths to attempt challenging actions or gain new information. This also gave players a chance to imagine and role play powers that they might not normally think of themselves as being good at. If you can imagine it, you can learn it.

We also assign additional skills to players based on the combination of domain + origin and origin + class. Here are a few examples of the available skills:

• Canary Speech: Detect issues in logging before wider rollout.

• Cloud Spirits: Detect under or over-provisioned system resources.

• Crystal Permutation: Identify key differences between customers.

• Deflake Chant: Test runs and deploys do not flake on re-run.

• Network Prism: Identify valuable client-side request logs.

• Retrospective Oath: Quickly complete all post-incident action items.

• Taming Herd: Spread out high load with backoffs and jitter.

• Vigorous Page: Reduce alert noise or fatigue for allies.

Dungeon managers

We recruited five of our engineers to serve as “dungeon managers.” A dungeon manager, of course, supports a team of ICs (individual creatures).

They trained on the scenario for the quest ahead of time and playtested the game together to improve the experience for the rest of the engineering team. We awarded each dungeon manager with their own dice set to run the game and commemorate the session.

Each dungeon manager took a group of engineers to play through the scenario together. The dungeon manager cheat sheet included tons of context to help adapt to the players’ actions and keep the scenario attached to the learning goals:

• Technical reference for the scenario

  • Terminology definitions

  • System diagram

  • API endpoint pseudocode

  • Database schema migration pseudocode

  • Napkin math for capacity, load, and error rate

• Game management advice

  • Session agenda

  • Evidence and events

  • How to interpret dice rolls for ability checks, action outcomes, and saving throws 

  • Skills and abilities list

Round 1: Triage

We called the scenario “Latency Lagoon” and it began with an immediate quirky piece of interactivity. The dungeon manager hands a player a six-sided die to roll and whatever value they roll becomes the hour in the morning that they get woken up by a critical severity page.

(Roll 1d6 to determine X) You wake up in a cold sweat at X:47 AM PT. Your phone buzzes. Then again. And again. The #issues-internal channel is lighting up with messages from customer success: “[Health system A] and [health system B] reporting slow note generation.” “[Health system C] says notes are taking forever.” “[Health system D] escalated to director level.” PagerDuty alert follows seconds later: “CRITICAL - Increased complaints about latency."

In the first phase of play, players take turns declaring their desired actions and rolling dice to find out the outcomes. This part of the session focuses on triage: asking clarifying questions, gathering information, checking early hypotheses, trying basic remediations, estimating impact, and assigning a severity level.

The dungeon manager can provide additional color or even red herrings, depending on the players’ ideas and a little bit of luck.

This stage is a great diagnostic for how deeply the team understands our product and platform and how effectively they can communicate with each other about the incident. Even the dungeon manager might not be familiar with all the systems or code paths exercised in this fake incident, but the technical reference gives them a clear mental model and standard language to understand where the players are searching and to prompt them to be more precise.

This is exactly why we use system diagrams in our engineering design documents: they provide an appropriate level of abstraction to imagine where things could break and to point to “where on the body” the system hurts.

Round 2: Root cause analysis

As the players move into phase two, they would have isolated the latency increase to a particular API endpoint and perhaps even traced it to a new database table that underwent an interesting migration. There are many ways the team can get to this point. Some players will dutifully navigate metrics, logs, and traces to find their way to the hotspot. Others will call upon their knowledge of recent code changes or feature launches and follow the scent where it leads. The dungeon manager can open up many routes to the root cause, showing the variety of complementary perspectives everyone on the team brings to an investigation.

Reasoning about system degradation in a live discussion can be stressful, which is why we lean on simple napkin math to sanity check our hypotheses. The dungeon manager has napkin math that they can reference if they need to coach the team or save them from having to do calculations on the spot.

This also allows the dungeon manager to model strong on-call behavior. For example, if a player invokes their character’s ability but isn’t clear about how specifically they would use it or how it applies to this situation, the dungeon manager can fill those points in when they narrate the outcome. They can describe in detail what the character does, what tools they use, what they consider, and what they learn, helping players understand the range of powers their character can use and things that they might otherwise only learn through experience or shadowing other on-call engineers.

Round 3: Resolution

By phase three, the players should have a good idea of the underlying cause of the incident and may already have taken some actions to provide immediate relief. This is effectively the “combat” phase of the quest. The team throws everything they have at the incident in order to restore healthy operations for the clinicians we serve.

The dungeon manager can aid them by affirming their wildest ideas and positive dice rolls or can deploy new obstacles in their path by reporting that certain remediation attempts failed, or perhaps even caused knock-on issues.

In the rush to take action and fix the situation, it might be possible to make things worse. The most nefarious incidents can arise when multiple systems and guardrails fail at the same time. This game is a safe, low-stakes environment where the team can explore that and push the limits of their knowledge and composure.

While the triage and root cause analysis phases help engineers learn more about our systems and application logic, the resolution phase helps them learn about our release process, production operations, and customer communication process.

Post-game retrospective

After the game time finishes, the dungeon manager facilitates the team to articulate with high clarity what the impact of the incident was and the five why’s behind how it happened. The team brainstorms action items and since the systems and tools are based on Ambience’s real production architecture, many of these action items can be learning goals or growth ideas for the engineers when they return to regular work.

No single engineer can know everything about a large, rapidly-changing codebase and platform. Continuous curiosity and sharing of experiences helps us become more resilient. This tabletop game is a way to simulate the learnings that usually require many on-call shifts to gain exposure to. And hopefully it makes engineers feel even a bit excited to go on-call.

Whether engineers had been at the company for years or for weeks, they all said that they learned many new things from playing the game. This scenario inspired us to plan a second game that focuses on the kinds of incidents that could happen if the electronic health record systems at a health system experience downtime. We want to prepare for the worst so that we can support clinicians at their best.

Let us know if you want to come over and play Runbooks & Rollbacks some time!