You humans make mistakes, and at some point in time, errors in judgment will occur despite everybody's best sections and domain expertise. Failure and what you do with it can have a transformational effect on your organization. So what do you do when things go wrong? Do you blame people up, ignore indicators of underlying problems, and continue business as usual until the next crisis occurs? Or do you encourage feedback and try find ways to transform failures into learning opportunities? I'll cover some of these fantastic topics and a little bit more in this talk on blameless postmortem culture. So very quickly, what this talk is not let's understand that this is not a lecture on SRE. This is not a lecture on outages and how to resolve them. And I will not be teaching you how to write a postmortem step by step. However, what this talk is, is I'll go deeper into some of these following topics. So what's a good postmortem? Why is it important to write them? And then why isn't writing a good postmortem not easy? I'll also go over how well written postmortem culture be an effective tool to influence culture around you. And then towards the end, I'll cover some possibilities around what you can do after you've written a bunch of good postmortems. So here's a little bit about myself. I am currently a program manager at Google within site reliability engineering, or SRE as we know it. I've been at Google for about four and a half years and I've worked on multiple teams. So mobile client infrastructure, currently firebase SRE. And then I also had some opportunity to work with Gmail, spam abuse and delivery SRE. I have co authored a chapter on blameless postmortem culture in the second Cytre Liability engineering book, which is called the Cytre Liability Workbook. And then I also co authored a report which is titled Engineering Reliable Mobile Applications. Both of these are available on the SRE site for free. And in my previous experience, I juggled multiple roles at a company called Bright Idea. By education, I'm can electrical engineer, and then I've also done a little bit of Bollywood dance instructor, and I was a primary dancer for them. And I love traveling and I love food. 2021 looks very optimistic with respect to vaccines and hopefully some sort of normalcy. So I just cannot wait to board my next flight and go to a new destination soon. All right, so that's enough context about me in terms of philosophy. Let's bring everybody on the same page and talk a little bit about general philosophy behind postmortem culture. Now, whenever an incident occurs, the person who's on call, these first priority is to mitigate the incident, fix the underlying issue, and then ensure that services return back to their normal operating conditions. The next big responsibility is to make sure that the system doesn't break in the same way again. Now, unless we have some sort of a formalized process of learning from these incidents in place, these may reoccur. Left unchecked, incidents can multiply in complexity or they can sometimes cascade. This can overwhelm a system and ultimately impact our users. Now one of the reasons I decided to share this talk with you today is because I want to dispel the myth that postmortem culture is only applicable to big organizations or deep technical problems. I think as long as one understands the basic framework, they can use it to address a variety of failures for personal growth, growth of others, encouraging accountability, sharing knowledge and best practices, and documents facts for the long term so others can learn from it. I know I've personally used the framework to think through a variety of tough situations in my personal life. Also, when things didn't go according to plan. It can also be a very effective tool for leadership to be transparent about popular decisions and document some rationale behind their unpopular choices. In almost all the cases I've seen, people really appreciate when leaders lead by example and are transparent. Now, introducing such a culture can seem intimidating at first, but making this change incrementally is possible, and one can gradually tune the process based on their organization's needs. One of the best ways to make use of this talk today is to think of a recent high impact failure that you experienced and think about how that situation was handled. Then apply the principles and best practices of blameless postmortem culture to that situation and these imagine if things could have been done a little bit differently. So what is things postmortem I keep talking about? This is especially for newer folks. So a postmortem is a written record of an incident, its impact actions taken to mitigate or resolve it, the root cause, and then very importantly, the follow up actions to prevent the incident from reoccurring. Now, let's talk through a simple, high level template. The first section is all about quick executive summary. This can include sections like incident details which could be simple facts such as incident date, postmortem writing date, when it was published, and then who has access to the document. Another subset of these exec updates could be about the problem summary duration of these problem estimated user impact detection mechanisms and resolution after exec updates. These next section is background. This is specifically meant for people who don't have a lot of context around your postmortem culture. Could be new employees or people who've just recently joined your team, or people from a completely different part of the company, or people who think your postmortem in a way could be related to an outage they are dealing with currently. This section is also important because one of the best practices is that once a postmortem is written, ideally everybody should be able to read it throughout the company. Next we go into sections where concrete details are expected so impact of the outage, root cause trigger and these recovery efforts. After that, let's do some deliberate introspection and we document lessons learned so things that went well, things that did not go well and then my personal favorite is where did we get lucky? And things unfortunately happens more than one would like. Next, we document action items and then what we want to get out of it. One important problem with action items is usually they're not very measurable. It's important that they are objectively measurable. They have owners priority and some sort of a tracking id. And then these last sections is glossary again meant for people who don't have a lot of context about your postmodern with respect, two terminology that you're using and may need a quick reference in addition to the basic template. Usually it's encouraged that you over communicate in terms of making the other person understand what your postmortem is all about. All of these sections need careful attention and there are best practices around how to handle each one of them. So now you may be wondering that things sounds like too much work and so do we need to write a postmortem for every single failure? Well, the answer is no, we don't. Teams need to define postmortem criteria before an incident occurs so that everyone knows when a postmortem is necessary. In addition to these objective triggers, 1 may also request a postmortem from another team if they think it's warranted. Personally speaking, over the years, postmortem culture become such an innate part of the SRE culture that we organically know when we can expect one. Another idea could be to group small incidents that have a similar in nature into one postmortem for resolution rather than writing one little postmortem for each incident. Writing postmortem should ideally be looked at as a learning opportunity for the entire company and not as a punishment. It's definitely worth acknowledging that a good postmortem process does present an inherent cost in terms of time and or effort, so one needs to be deliberate in choosing when to write one. So the next question is, when do you write one? Well, depends on your needs and circumstances. One criteria fits all approach doesn't really work here. For example, a team whose primary responsibility is ensuring the reliability of a website's ecommerce infrastructure will have distinctly different success criteria or failure metrics than a team whose primary responsibility is product development. The two teams will also have a distinct set of personalities, making incident management nuances somewhat different. So some example scenarios could include maybe business reasons where x numbers of users are affected, y amount of money is lost in revenue, or there could be loss in user trust due to a marketing error, or some other reasons. There could be product reasons for writing a postmortem. For example, a latest canary shows significant regression in a metric. A risky change was pushed out two production during production freeze. Or there could be people reasons. So let's say a disruptive reorg impacted people's careers. An improper project management overloaded people for a long time. What matters are that these criteria are defined in the first place and periodically revisited, so one knows when to escalate an incident and when it becomes an all hands on deck situation. One can also write a postmortem for personal reasons, for example, that travel plan went completely awry or I'm feeling burnt out, et cetera, in terms of defining a criteria for your personal growth. Also, a solid understanding of your own self will guide you, and then you'll know when to write a postmortem. So, so far we've talked in detail about what we should do. It sounds pretty straightforward. Have some predefined postmortem writing criteria. If an incident occurs, fill out a template, create a return record of an incident, create some action items, and then magically all our problems are solved. But is it really that simple to answer the question? The next thing I want to emphasize on is the importance of ensuring psychological safety. Now, writing a postmortem just for the sake of writing a document is not enough. In fact, if not written well, it can be counterproductive to your culture. It could lead to an atmosphere where incidents or issues are swept under the rug, leading to a greater risk for your company. So to make the best of postmortem culture, it's absolutely imperative to keep them blameless. So a blamelessly written postmortem assumes that everyone involved in an incident had good intentions and did the right thing with these information they had, it focuses on what went wrong instead of who did wrong. This is important because basic human nature makes accepting failure very difficult. Sometimes these mistakes can cost your company millions of dollars or euros. And accepting that something that did caused it can be very embarrassing and distressing. For individuals in a very toxic organization, publicly acknowledging mistakes can sometimes cost people their careers. Now, all of these fears will make it pretty much impossible for a postmortem to be valuable, fact based and objective. Now, the best practice to combat this fear is to keep your postmortems blameless. But why? Now we has humans fear public humiliation. Speaking up under normal circumstances is hard enough. Speaking up under immense pressure, as in when an ongoing incident is happening, is much harder, and people generally tend to avoid the spotlight for the fear of being ridiculed. Now, ironically, a crisis demands for someone to step up, think differently, and also speak. Also remember that perform postmortems are artifacts. So now we've increased the complexity of the problem. Not only does one have to risk isolating themselves in difficult circumstances, they also fear being documented in history. So what if I make a mistake? Everyone will know. Will I be fired? Will I not get promoted? Will my future employees make fun of me? These are all very valid concerns. And then there's the potential to curb innovation and autonomy. Now dreaming big working on revolutionary ideas brings a certain amount of associated risk, which may very well result in failures. If a person doesn't feel psychologically safe in taking calculated risks, they may never act on those ideas and then innovation will probably stagnate. This also takes away from that person's autonomy has they'll tend to just follow instructions from their sections or someone else instead of being creative themselves. We cannot create future visionaries and leaders like things. Now it may seem like a good idea to highlight individuals while describing an outage in a postmortem. Instinctively, it feels like we are assigning ownership to someone, which may then motivate the individual to take responsibility. But the big risk in doing so is individuals would become risk covers because they may fear public humiliation. This can lead to people covering up facts. Risking transparency, which could be critical to understanding, can issue and preventing it from reoccurring. And when it comes to personal circumstances, blaming yourself for, let's say a personal outage is not great either. Have faith in yourself, assume best intent, and know that you acted in the best of your abilities, in the circumstances and the context around you. Blameful behavior can be pretty detrimental from a people perspective, but blameful behavior can be very problematic from business perspective. Also, when mistakes are hidden, systematic fixes are harder and problems are more likely to reoccur. Also, blaming humans sometimes tends to result in fire human as an action item. Now for a moment, even if we ignore that that may not be the right thing to do. It still doesn't prevent reoccurrence of a problem. If the system has set up to allow these human to make the mistake in the first place, there's a higher probability that a newly hired and less experienced human will repeat that outage. So blameful behavior is pretty detrimental from a business perspective also. All right, so we all get the idea of blamelessness. Let's work on an example together and see how we can transform accusations into simple facts. So, in terms of this exercise, remember to focus on what went wrong instead of who did wrong. So let's say there's a hypothetical incident, and then there's a hypothetical postmortem that was written, and this was noted as the root cause. So this says, dylan, one did not both to set up alerting for our storage cluster or check our hard drives manually in case of doubt. Of course, we ran out of space and then this disaster ensued. Now, maybe pause this video, take a couple of minutes, and then think if there's anything wrong with the way this is written in the postmortem. All right, I hope you've thought about it a little bit so I can give my perspective here. So not only is this example very blameful, where clearly one person is being scapegoated, it's also quite dramatic. So phrases such as disaster or of course, that happened add absolutely no value to your document. What it does is erode psychological safety and make your culture toxic. You can be sure that, Dylan, one is not going to speak up when another outage happens, and the company could probably lose or out on some valuable information. A better way to express the gravity of the situation would be to gather facts and document them objectively with metrics. So here's an example of a more objective, blameless version. So the hard drives in one of the trading storage clusters run out of space, and this was not noticed due to a lack of alerting. This led to trades from that cluster being redirected to other trading clusters, which were also almost full. And an action item that could come out of this is to set up alerting for this scenario, and that AI must have an owner and some sort of a tracking id. Let's move on to leadership. Now, I'm assuming we're all leaders here, so let's take a look at some of the ideas we can propose and implement to foster a blameless postmortem culture in our surroundings. Now, one of the best things all of us can do is lead by examples you has. Leaders can start writing a few postmortems yourself. This will give you an opportunity to experiment with what format or template works best for your. This is also an opportunity to set the tone of the postmortem culture you'd like to see. Once you've written with an artifact, share it openly, let the readers digest the content of your documents, and then come back to you with questions and feedback. This is another opportunity to experiment with your postmortem format. Now once people warm up to the concept and see you doing the right thing, they'll probably be encouraged to do some themselves. Now, once folks have accumulated a few postmortems over time, you can use this wealth of knowledge as training material for future leaders. One example is an exercise called the wheel of misfortune. Think of it as disaster roleplay in which a previous postmortem is reenacted with a cast of employees. Playing does as laid out in the postmortem. The aim is to make the experience has real as possible and make the educational portion of ramping up as fun as possible. Now, if you or your senior management think it's appropriate, you can go a step further and create a culture of rewarding people for doing these right things. So some examples could be highlight well written postmortems in company or team meetings. Reward closure of postmortem action items if action items are documented but not acted upon, what's the point? Also acknowledge that making systems more reliable is high impact work, and then reward such work in your performance. Assessments regarding postmortem owners as leaders can be very motivating. Setting up the postmortem owner as the go to person for resolving or mitigating a failure can be very rewarding for some people. If your senior management is on board, it is okay to even publicly celebrate a person who actually was the proximal cause of a big outage, but then who also acted effectively and sensibly to either escalate or troubleshoot or mitigate or resolve that outage, and who honestly then documented system failures that allowed them to make that mistake. This is again to emphasize that even if it's clearly a person who can be held responsible, that it was always a set of conditions and system failures that allowed things to happen in the first place. You can also celebrate people for finding and fixing reliability bugs. Another idea is to invite folks to talk about failures and lessons learned on a high visibility platform. This could help normalization of conversations around failure. All of these ways are some great examples to instill psychological safety in current and especially newer members of the team. You can do even more. So let's talk about that. So remember, things slide we looked at previously. So now you've done all this postmortem work, you're all experts at blamelessness, and then you have a magically blameless orc, but the work is not done yet. We still haven't completed the last step. We still haven't made sure that the same incident never occurs again. Now, as a part of writing your blameless postmortem, you documented many action items. So what happened to those? Remember, the overarching reason we are doing all this work is to prevent incidents from reoccurring. And if action items remain unaddressed, you can be sure that incidents are going to reoccur and maybe on a much bigger scale, since complexity of our systems and processes is not going down anytime soon. So in order to do so, review postmortems and sections items periodically. These action items can be around adding more monitoring, automated rollbacks, automated release gating, or even more refactoring of the existing code base. If a company doesn't have preexisting tooling around postmortem action item management, one can look at third party tools. Also, what process you follow, how simple or hard it is can depend on your team, but it's important that you do it in the first place. Otherwise, what's the point of accumulating these action items in your postmortem? Sometimes reoccurring postmortem reviews can also give you a sneak peek into maybe some themes that are emerging, and you can then combine those themes into a bigger project and then prioritize fixes accordingly. Now, with a growing number of postmortems being written every quarter, the next big stretch goal could be creation of tools to aggregate these postmortems and identify common themes and break for improvement across product boundaries. One can work on machine learning projects where they can teach a model to preempt outages based on past patterns. Encourage folks to keep their trans encourage folks to keep their postmortems accessible by all. As a default, this will encourage transparency and reinforce the concept of using failures or postmortems as a learning opportunity for the company. And then let's not forget that without intentional nurturing, any sort of culture can ultimately fizzle out. The breakdown of postmortem culture may not always be obvious. Following are some common failure patterns and then recommended solutions that might work so lack of time quality postmortem culture time to write when a team is overloaded with other tasks. The quality of postmortem culture so prioritize postmortem work, track its completion and review, and allow teams adequate time to implement the associated plan. If teams are experiencing failures that mirror previous incidents, it could be time to dig a bit deeper. So consider asking questions like are action items taking too long to close? Are we biased towards feature velocity over reliability? Are the right action items been captured in the first place? Or maybe the service is overdue for a refactor? Next point is blameful language while speaking with each other responding when someone uses blameful language can be very challenging, especially if a person senior than you is doing so. One trick is to mitigate the damage by moving the narrative in a more constructive direction, reminding them that investigating the source of misleading information is much more beneficial to the than assigning blame. And these disengaging from the postmortem process is another sign that postmortem culture is starting to fail. If folks seem content with not discussing failures and maybe avoid the issue altogether, it may be time to reinforce blamelessness and ensuring that high visibility postmortem culture reviewed for possible blameful wording. All right, this brings me to my final slide. In case you don't remember anything I said in the last 30 to 40 minutes, I hope these these key takeaways stay with you. So number one, the cost of failure is education. Number two, keep your postmortems blameless. Focus on these system, not the people. And then lastly, when written well, acted upon and widely shared, blameless postmortems can be a very effective tool for driving positive cultural changes and also preventing reoccurring errors. Thank you. Thank.