Hi there. Welcome to my session. It's good to see you. Well, actually, I don't see you, but it's good that you're watching this. And I'm excited that you are, because it shows me that you care about more than just making software. But youre also interested in the delivery part, which is, if you ask me, very crucial. My name is Abel Ferhover. I'm a quality and test automation engineer, and I work for a company called Testcoders. Now, the reason I became a consultant is so that I got to work with many different clients, many different companies and groups and teams and people. And the funny thing is that I did not know before is that all these different groups, all these different teams, they're all struggling with this very same issues, which almost reminds me of parenting. Now, I don't know if you have any kids. I do. I've got two kids. And only yesterday I was grocery shopping with my son, and at some point he decides he's had enough and he does not want to walk any further. So luckily we were almost done. So I had to pick him up, put him under my right arm, and in my left arm I had a heavy grocery bag. So we were walking to the car, back to the car, and there were two women. They were sort of laughing at me. And once I walked past them, they said, well, it's the same everywhere. They were clearly recognizing some of my strategies that they probably experienced themselves. And it's the same in our world of software development and software engineering, we have the same problems and have to deal with the same issues and the same pitfalls. And that's what this session is all about. Now, before we get started, I'd like to do a show of hands, but this is a virtual conference, so I can't do that. But I will run a few scenarios, a few statements by you anyway, because I think you will recognize a few of them. All right, so let's see. Here we go. Let me move over to a different view. Right. Do you recognize this situation specifically? Have you ever had a dedicated sprint to work on bugs or technical debts Specifically? And be honest, I know that I have another one. Do you often find items on your product backlog that date back more than twelve months ago? Oh, yeah. This is a classic one. What you actually see here is a sprint board, a sprint backlog where you'll find a lot of items sitting and waiting in the testing column. So there's a lot of work to be done in that area. This is what we call a bottleneck. So do you recognize a bottleneck in your sprint board? Another great one. Yeah. Canceled retrospectives to finish the almost done work. So instead of doing a valuable retrospective session, we finish that item that was almost done so that we don't have to bring it on into sprint planning for the next sprint. Have you ever done that? This is a classic one. Product owners who do testing. Now, these are actual things I've seen out there, things that I've actually been a part of as well, to be honest. And to me, all of these situations are typical examples of signals and symptoms and characteristics, if you will, of teams that are not performing well, teams typically, that are complaining, for example, about the amount of unaccounted work that comes up during a sprint in the form of bugs or technical debt or whatever. And these are also the teams that struggle with pressure from their business to deliver more stuff that they can't in time. Teams that typically finish less than 90% of what they originally planned for. Now, if youre recognized a few of these scenarios yourself, don't be alarmed. It's not that you are part of a team that's not doing well. It's just that there are some opportunities for you to improve. These opportunities to improve typically revolve around three major areas, so to speak. And the first one is bugs, dealing with bugs, preventing bugs, all that stuff. The second one is legacy. There's quite a lot of legacy out there. I think everyone deals with legacy on a daily basis, so therefore, it's important that you have a way of dealing that. And the third one is technical depth. Many, many teams do not have a way of working that involves the definition of technical depth and how they should deal with technical debt. There's no strategy, there's no clear vision. So first of all, I'd like to align on the definition of these three areas so that we're speaking the same language. I think that makes sense. Let me move over. Let's start with bugs. All right. So to me, a bug is very clear. Let there be no mistake. It is something that we have delivered and that worked before. It's been tested, it's been through the whole software development lifecycle, and now it's not working anymore. So that's very straightforward. All right. Now, the second one, the second area was legacy, and that's a bit more difficult, a bit more tricky. There's no clear definition, but I think there are a few signals you can spot, if you will, that point to a legacy system. And the first one is that the system outdates the current tech stack or team, if you will. So there's no one around anymore who knows how to deal with the system, who has developed the system, has actively worked on it, or it's from a technical framework built in a technical framework or with a technology that you are not no longer using anymore. Another characteristic or symptom of legacy is when youre see around you or you feel it yourself, that people are afraid of touching a system. It's so delicate, it's so fragile. You might even be afraid of cloning a certain repository on youre local machine. Just because you're afraid you accidentally make a commit, merge the whole thing, and then break the system, which is not going to happen, of course, but you get what I'm saying. And another one is that if there is no or zero testing for this project available, so, like, let's say there are only a couple of unit tests. For example, I'm not talking about things as test coverage or code coverage here, like when you would have an 80% coverage and you say, oh, it's a legacy system, because there's 20% uncovered, or that 20% is now considered legacy. Now, not like that. It's a bit more pragmatic, if you will. But if there is no testing, there's no way of learning what the system is supposed to do. And I think that's the whole thing with legacy. You need to want to learn what the system does because you are unfamiliar with it. And tests are a very good way to do that. All right, so the next item was technical debt. And that's the one we'll be going into. A little bit more into debt. Technical debt can be split up into four categories, hence the quadrant. And on one hand, we have prudent actions, careful actions, careful decisions, and on the other hand, we have reckless decisions. Then on the other axis, we've got deliberate choices, deliberate decisions, and we've got the accidental decisions, like not knowingly. So let's go over all four of these categories. So this green block simulates the best form of technical debt, the non harmful, let's say. And that is when you say, okay, I know that I'm taking on technical debt. In this case, it's a deliberate action, and I will deal with it later. I know that I have to deal with this later, otherwise it's not going to go away and it will slow me down. Then there's the reckless form of this, the reckless thing where we say, well, we don't have the time to do it right now, so we take on the technical debt and we'll see what happens after that. There's no plan for paying back that debt. There's no plan for fixing it. Then we have the other one where it's an accidental decision. We did not know it. But we have a careful mindset. We take a careful mindset here. So at some point we figure out, or it comes to our attention that we have taken on technical debt, and at that point we say, oh, wait, hold on, we should have done it better. We shouldn't have done it differently, and we're going to do it that way from now on. And then there's the final one, the one that you should definitely avoid. That is when you did not know that youre did make a mistake or you took on technical debt. And at some point you do know and you still ignore it, right? Someone points out we have taken on technical debt and you say, oh, well, we don't care. We don't know how to do it. We don't know how to fix it. Let's not do it. That's the horrible one. Make sure you're not in that last category. All right, so let's have a look at an example for that. So let's look at a very simple example of technical debt in a mathematical form. Let's say for argument's sake, I want to take a trip. I love holidays. I love taking holidays and going on trips. And this trip that I have in mind is going to cost me €6000. It's good for my health, it's good for my family, it's good for everything, basically. And it's a must have. It's very important to me and it will solve a lot of issues and everyone's going to be really happy after this trip, however, I have only managed to save up €2400, 200 a month. Right? That's what I can save. So I'm short quite a lot, actually. Now I can take on this 3600 as a debt. I can do that. Which makes me having to pay this. Let's for argument's sake, say that I have to pay this back in one year. And on the previous slide, we just learned I can only save 2400 a year, in other words, 200 a month. So I would be short another 1200 after one year. And that doesn't even go to savings. That goes to paying back the debt. How do I get an extra hundred a month? That's the question. So here I've sketched our monthly spending pattern, if you will. So we pay something like mortgage or rent. We have insurances. We have utilities like Wifi and also water and power, but most of all wifi. Of course we have a car that we need to drive around and we need to eat and of course we have that 200 on savings. So we need an extra hundred. And the bank is going to come every month going to charge that other account on. Where is that money going to come from? Where is that hundred going to come from? That's the big question. Are you going to pay less rent? Don't think your landlord would like that. Are you going to stop some insurances? Sounds very, very risky. Pay less for utilities. No, go, stop driving the car. Could be, but that way we can't drive to work, eat less food. Maybe we can find something there. But that's the whole idea. That's what we're doing right now. We're planning to come up with the extra hundred. This is the planning phase. If you do not do this, you will slow down. You will be forced into changing your spending pattern in this case, or the effects will be that you won't be able to deliver as much stuff because you have to deal with the implications of that debt that you did not pay. For example, there could be more bugs on your backlog because you didn't pay back your technical debt and now you have got a temporary dirty fix still out there in production and you have to deal with the consequences. So please, oh please, just pay back your debt. Do not plan to be clever or add more resources. Pay back the debt. Accept the pain. Because yes, it hurts a little at first, but you should spend time every sprint to pay back technical debt, even if your technical debt is minimal every day. I'm sorry, every sprint, you put technical debt on the top of your backlog. That's it. You put technical debt on the top of your backlog every sprint, and you finish it first. 1st thing when a new sprint starts, you work on that. Simple as that. If you fail to do that, if you just ignore your technical debt, it'll become a vicious cycle and it'll drag you down. And I want to sketch a simple example of that from the field of test automation because that's where I'm specialized at. So what I've seen a lot in continuous deployments and continuous delivery pipelines is when there is like failing tests at a late stage, there's perhaps an end to end test or something that is flaky or it gives back a false positive because some technical reason it is failing and it's blocking our build. Right. What I've seen developers do is to turn off that test just to make that build pass. That is the moment youre take on technical debt. And it can be for good reason. It can be, but there are some follow up actions you should do, and if you don't do them, youre build passes, you get a lower test coverage results into more bugs, or at least more bug reports. And bugs result into you having to deploy a hot fix and you forgot to update your tests because you've turned them off. And you get to the same cycle again and again and again. Youre box result into more bugs, result into more bugs, and you get where I'm going at right now, as I said before, I worked in various teams and in my experience, it is the DevOps teams that are typically better in handling technical depth. I think the reason why is because true DevOps teams that contain both developers and operational engineers, true DevOps teams that work together on a daily basis, they feel the pain of technical debt in each phase of the software development lifecycle that you'll see on this image, the DevOps logo. Because now developers do not only feel the pain of technical debt when they have to perform testing or building or releasing or coding, they feel it during the entire lifecycle. And it's the other way around. Ops engineers now feel the pain of technical debt also in the development lifecycle, in the coding phase or the building phase. All right, so another very big thing in DevOps is automation. It's super important, and it's for the simple reason that in DevOps, what we try to achieve, first of all, is flow. It's actually the first way of DevOps is realizing or creating flow. And that means that work flows from left to right, from dev to ops, without having to wait a lot in between. We're not going to make this whole big release window once a week or once a month, where we put all of our commits together, we squash them, and then we give it to an end user and it becomes a very big release. What we try to achieve is really small bits all the time, and maintainable, containable. And in order to do that, you need a lot of automation. You need automating tests an awful lot. You need automate, CI pipelines, deployment processes, release processes, everything you can imagine you want to automate, because you want to be able to release or give stuff to an end user all day long, ten times a day if necessary. And this is something you can pick up yourself, even if you're not practicing DevOps, even if you're not part of a DevOps team, or you don't even have DevOps teams within your organization. Even then you can still pick up on this very important item of automation. And a good way to start is with the definition of done, right? So the definition of done in agile and scrum is a list or a description of stuff that need to be completed, tasks that need to be completed before you call your work done. So if you haven't got a definition of done, I think that's the first thing to do. And if you need inspiration for what to automate, then go look at this list because it typically contains the stuff we don't like to do in my experience, like writing release notes and doing some manual testing, or preferably not actually, but all kinds of quality gates, peer reviews, all of that. See where you can automate and start with that. So another big thing that can slow down a team without them even realizing it is the decision making process. Decisions need to be made on a daily basis, and sometimes they're big decisions, sometimes they're small decisions, typically. The bigger ones, of course, tend to take longer and require more thought and are slowing down our productivity and sometimes even our quality as well. Because we have no way of doing this. It's all gut feeling, discussion, opinions, and DevOps advocates that you practice data driven decision making over gut feeling, as clearly sketched in this funny image. Now this is a very difficult thing, because all decisions ever, always, all the time, no exceptions, are made by emotions, are based on emotions. So how do you quantify that? That's a challenge. All right, so I understand this is not a workshop, but I do want to mention two tools in particular. I found them very useful in the past when having to make a decision as a team, when there are multiple solutions out there. Now, these tools are called the AHP matrix and the pew matrix, and they are complementary to one another. So we use the AHP matrix as a team exercise to define what aspects of the possible solution we find important. And the AHP metrics can be used to quantify that and find out what is most important. Now, we use the output from the AHP matrix in the pew matrix. So in the pew matrix, we ultimately compare the final solutions that we have to pick from. So let me give you a bit of an example of an AHP matrix and what it looks like. I was working with a team that needed to select a contract testing tool to ultimately test API versions and contracts, actually. So there are many different solutions out there, but before we get to the solution, before we choose the best tool, for youre job, we needed to identify what we found important about this tool, and that's where we did the HP matrix. So first we've identified pricing is important, then collaboration and source control is important. It needed to support open API 3.0. We wanted to look at the learning curve, because if it's a very difficult tool to master, that may influence the decision we want to make. It would be nice if there was a mocking service or mock server available within the tool and it needed to support graphQL in the future, even though we weren't now, the exercise goes as following only. The yellow highlighted cells are the ones youre need to fill in, and it is every time we do a compare between two attributes and two only, that's all youre need to discuss with your team. So the first one, I'm highlighting it right here, is collaboration and source control versus pricing versus the column. So row versus column, which do we found more important? And if you find, for example, collaboration more important, you give it 12358 points, basically Fibonacci sequence. And if you find pricing more important, you do the same, but it's like a negative. So this is like, I find pricing important two versus zero or three versus zero. In this case, if I would put in a three, let me select it. Whoops, here we go. Then I would say collaboration is more important than pricing. If I find it way more important, I could use the Fibonacci sequence to say in, for example, eight. That's a lot more important. So you do that for each yellow cell. You don't need to look at the white cells because they, in this case, got automatically filled. Otherwise youre have to do the opposite. If you don't have a spreadsheet that does that, if you don't have the formula, then just fill in the opposite and you'll be fine. So you do that for each and every yellow cell, and what it ultimately does, it comes up with a weight. That's the output of the HP matrix. It's the weight for each property. We will use this in the pew matrix. Yeah, there it is. So we use this in the pew matrix to ultimately compare the different solutions with one another. But first we needed to identify which one we found most important. So, in this case, you can see that we found the second row, which is collaboration and source control and open API 3.0, most important. So if we change this, for example, the learning curve now becomes a lot less important. But then again, this is a team exercise. So you should decide with a team and have discussion on each matter. All right, so let me just copy the values of these properties, the weight that we basically calculated. Let me just copy it in right here. And here we have it. So this is the Q matrix you're looking at. We have a list of properties that we've defined that are important to take into an account when we make a decision. We have the calculated weight that we did from the Ahb matrix. So this is no longer AhB matrix, this is the pew matrix. And now we have three different solutions listed up that we want to select from, that we want to choose from. For contract testing, you can use postman, you can use pactflow, and you can use pact. All right, so I've cleared the values, set them all to zero for all the different solution. Now, the first thing we do is we identify or we select one of the three, in this case, one of the three solutions as a baseline. And that could be any one of them. It doesn't matter, none at all. It could be the current solution. That could be one of the suggested solutions in this case. As you can see, we have selected pact as the baseline. I've highlighted it in yellow. So that means we don't change those values, there's zeros, we don't change those scores. What we are going to do as a team is we going to compare, for each solution, each attribute with the baseline with pact in this case. So as first, we could compare packed flow with packed on pricing, and we give that a score. So, for example, I say packed flow is more expensive than pact and therefore gets minus one, or it's a lot more expensive than packed and it gets minus two. And you do that for each of the attributes. And if they're equal, youre give it a zero. And if they're better or worse, give them a minus or plus number. And the options I have selected here is minus one, minus 20, and plus one, plus two. That's the one you get to select. You can change the math if you think that makes the decision ultimately more clear. So you do this as a team. And let me just randomly put some stuff out there just so that we have a clear winner. Minus one. Actually, learning curve and postman is quite easy. So that I'm going to give that to backflow is going to be a bit more difficult or actually as difficult as pact. There's a mocking service. Yes, they have it. Graphql support yes. But I suppose that's the same with pact. So let's say this is the final output in my case, as you can see, there's a clear winner. Now, I've altered the numbers that way, so that would be for this example. But the clear winner is postman. It could also be that your baseline is the clear winner. And if we would alter the numbers and have to score all the other solution with minus numbers, you can guess that, of course. Let me just tweak it a little bit here. If this would be the output of the pew metrics, the pact, the baseline would be the clear winner. So this is how you can use that. Hopefully you understand that this tool helps you avoid discussions based on opinions. Right? So it's no longer, I think Packflow is better than packed because I've worked with it in the past and it was so good. Now you get to dive deep on the matter, dive deep in the solutions and compare them based on what you have identified that you find important about this decision, in this moment, with this team. Now, I would love to give you more tips and tricks to lead these exercises, these group discussions, and how to put together the right group, for example, put together the right people, how to select them, how to find and identify the criteria that are the basis of this entire exercise. But as I said, we are limited in time and I have to move on. If youre like to learn more, please come visit one of my workshops. I give them at conferences as well. You can check me out on abletohoova.com schedule, or you can book one of these workshops virtually, go to my website and you'll find a submission form and all about this workshop. So let's move on. All right, so there's one final thing I'd like to say about decision making. And if you remember only one thing from this session, let it be this one. And it is good. Is good enough. And what I mean to say by that is I've seen many, many different teams and individual developers struggle with the decision making process because they thought there was more information out there to be gathered to be taken into an account. I've seen developers that had to select, let's say, between three or four solutions, make the entire proof of concept for each solution into an mvp, almost like building the entire thing like four times, and then say, I want that solution because that's the best one. That is going to take ages. You're never going to get things done that way. So please make informed decisions, but stop overanalyzing. Good is good enough. I actually had to learn and understand this principles the hard way. It was about ten years ago, I think, when I was parking my car. Actually, I was locking up my car almost in front of my house when I heard footsteps coming in from the left. And it was a Saturday night, I think it was around eleven ish or twelve, and I had just come home from visiting friends and I heard footsteps coming towards me. But I lived in the city, so people walk, and that's not a thing to pay attention to, I suppose. But all of a sudden, as I'm locking up my car, I hear these footsteps stop right behind me. And just before I pick up on that, I get a hit on the head from behind. I get hit pretty hard and I fell over onto my car. Now I immediately get up and turn around and I see a man standing there, holding a knife in his hand. Now from that moment on, everything went dark at that moment, and it took quite a few months, actually, before I remembered more or less what happened after that moment. So after I had turned around, there was a fight. I immediately jumped onto the knife. I realized it like that, that this is a critical situation and it's a life or death thing. So as that takes about eight milliseconds, youre body jumps into a mode or your mind actually changes into a mode. And luckily my mind chose to fight because I don't think there was actually any other way I would have gotten out of it. There was a fight. Ultimately I survived, obviously, because I wouldn't be here sitting having this talk. And it had a lot of impact after that. It was not the fight, not the injuries. There were injuries, but they were minor, really minor, nothing to be really concerned about, but that had a mental impact on me. So at this point in my life, I was actually in the army, I was in the infantry in the Royal dutch army. And luckily for me, they have got specialists who know very well how to treat PTSD and PTSD symptoms, which is what I was showing now. They treated me and it worked very well, actually. So I was happy to do not have those symptoms anymore. But still, I could not make sense of this whole event. I couldn't make sense out of it because everything that happened that night was very illogical. I mean, someone just doesn't walk up to you and start attacking you if he's after your money or something, because that is what happened during the fight. I got knocked out somehow, and when I woke up, my wallet was gone and it had been found by the police later that very night. But if someone's after your money, they are likely to warn you before they attack you, just so to avoid the whole physical confrontation, because it's risky for the Margaret as well. But that didn't happen. Also, he attacked me with a knife, but he hit me with his fist, I assume first from behind. If you want to attack someone with a knife, you want to inflict damage, you do that with a knife. And this guy, he didn't stab me, really. He cut me, and I got injured, but never fatally injured. Luckily for me, my injuries weren't even close. And this guy clearly did not know how to work the knife or how to do most damage. And if he did, it wasn't his intention, but it seemed like it didn't. So that didn't make any sense either. So what I was left with was a lot of nonsensible facts, and that kept bothering me. At some point, the police stopped the investigation because there were no more leads. I mean, they did some investigation that night, but there were no leads coming in. They never caught the guy. And I was left with a dead end, basically. And I noticed, I was very much bothered by this. I kept analyzing this. I kept analyzing this until the moment someone actually mentioned to me, like, are there any new facts coming in? Is there something you can do or anyone can do to make sense of this all? And the answer to that question was, no, there's nothing we can do. That's the whole reason the police had to close down the investigation. So at that point, I had to say to myself, all right, it's enough. We know what we know. It doesn't make any sense. We have to move on. Don't overanalyze. Good is good enough, even in a horrible or bad situation. Right. So I'd like to wrap up this session and briefly summarize the stuff that we talked about. So we talked a lot about technical debt, and I hope you get that. The point here is to not take on technical debt unless you are willing to pay back the debt. If you're not willing to pay back the debt or you can't, you don't have a plan, do not take it on. Please keep to your own quality standards and do not take shortcuts, because that's essentially what technical debt is. Technical debt goes to the top of the backlog. That's the rule. That's the only way youre can deal with it on a daily basis. So please make sure your sprint backlog always contains one or two or more. Three perhaps. I don't know. It depends on your team items of technical depth, and make sure you handle them first. Very important. I'm afraid this is the only way you can deal with it that is sustainable. Then we talked about how DevOps teams are typically a lot better in handling technical depth because their responsibility for the entire software development lifecycle and how they incorporate automation to do that. Now that's something you can pick up on yourself. And a good way of starting there is to analyze your definition of done and see what tasks there are in there that you could possibly automate. That's where you should start. And then finally we talked about decision making and how data driven decision making is a lot better than gut feeling, and how data driven decision making can offer you a bit more structure. So we talked about the AHP matrix and the pew matrix and how you can easily incorporate them in the decision making when having to select out of multiple solutions with an entire team or group of people, which can be tricky. So thank you for listening. If you have any questions, comments, remarks or anything, please reach out to me on GitHub, LinkedIn or send an old fashioned electronic mail through to ablet at testcoders NL or visit my website awoltforhoova.com and I'd love to get in touch. I'd love to hear your thoughts and ideas for now. Thanks again and enjoy. Conf 42 DevOps 2024 cheers.