Conf42 DevSecOps 2022 - Online

Flow Engineering - Boost velocity, quality and happiness through your entire value stream

Video size:


This is an introduction to 4 valuable mapping techniques and models that build clarity, alignment, and confidence in teams using a combination of collaboration, visibility and measurement. I’ll introduce 4 powerful maps: Outcome, Value Stream, Dependency, and Capability.


  • Flow engineering is how to level up your team one map at a time. We're going to look at constraints in the workflow, what's holding us down. We'll connect those constraints to actual factors inside of our organization. And we'll wrap all that up into what I call flow roadmap.
  • Use whiteboarding to understand the current state of a project. Use value stream mapping to lay out all the activities from start to finish for a given workflow. And then look at investigations as options to figure out how to navigate the landscape.
  • A lot of people struggle with identifying a value stream. Looking at working backwards from that outcome helps us isolate a single stream. Third stage is to look at that constraint, that hotspot, and break it down. How might we improve the workflow around these things?
  • Flow engineering is an excellent method for optimizing collective flow. You really can't do the personal without the collective. We balance collective flow with personal flow. Often, what we'll see is we can do more with less when we optimize a workflow.
  • People are going to start talking about this a lot now because this just showed up at the very top of the thoughtworks radar. Everyone who talks about massive performance improvements talks about looking at the end to end flow. Flow engineering is really going to ride this wave.
  • Mapping is so valuable and so important working together in this way on a visual medium. It's incredibly customizable and it's entirely bespoke for whatever context. It works just as well remotely where everyone's in a different time zone. There's an ebook and an email course that is entirely self guided at flow.


This transcript was autogenerated. To make changes, submit a PR.
Hi, folks. Welcome to flow engineering. I'm going to be taking you through how to level up your team one map at a time. So let's jump in with a classic. So we've got a ton of stuff to do, right? Everybody has this giant backlog of crap that they love to ship, and there's a million things in the way, right? We need to do more with less. We've got staff cuts, people leaving, reorganizations, demands from leadership. We have to always be improving value, getting more out of what we do. We need higher quality, fewer defects. We want to stop firefighting, and we need to keep everybody happy, ideally, make them more happy as time goes on. So the question becomes, how do we do all this and stay sane? How do we constantly level up? How do we constantly improve our performance, not just as individuals, but as teams? So I'm going to take you through an entire end to end process for doing this. First of all, begin with the end in mind, right. We're going to talk about setting a clear target and heading in a direction together. Then we're going to look at constraints in the workflow, what's holding us down, what's really holding us back. Then we'll connect those constraints, those things holding us back to actual factors inside of our organization that we can quantify and understand. And then we're using to connect those dependencies to capability that we can level up, that we can improve in order to mitigate or break those dependencies. And we're going to wrap all that up into what I call flow roadmap, which is essentially a roadmap for your workflow, how you're going to improve what you do on a day to day basis. So let's dive right in. First of all, what we want to do is understand the current state. We want to get everybody together, and this is a variety of stakeholders, right? We've got design, development, leadership, quality, product, all the folks involved in the value stream, everyone that it takes to get something out the door in the same room, and we'll work this out. So first thing we want to understand is pains, goals, questions, ideas, context. What are we dealing with? Right? What are people focused on? What are they concerned about? What do they want to do? We'll get it all out on the page onto a surface that we can all look at. And we use collaboration whiteboarding for this, but we could use postit notes on a wall or a whiteboard. Then what we do is we take all these things and we focus on just a single target. Right? What's one thing that we can agree is more important than everything else to focus on. So in this case, we've got all this out on the board, all this context, all this valuable information. Everybody is on the same page about heading in this direction towards releasing twice as often. So we can take that and we can understand, what does that actually mean? What does that look like? So we place that on the board, right? Clear ourselves a space and understand, why is that important? Why is it important to customers? Why is it important to us? Why is it important to leadership, the business? And we can look at it through all of those lenses to make sure that we all understand why this is important and we can all get behind it. Then we look at obstacles. What are the things that could get in our way? What are the things that will hold us back from achieving that target outcome? And let's get all that out so we can talk about it and see it. And then we want to understand what are the things that we can learn? What are the things that we can test to see if we can avoid obstacles or get to that outcome most effectively. And I have a few of my favorite investigations and I'm going to take you through them. But you can really put anything on this board and you can use this in isolation whenever you want, right? You don't have to go through the entire end to end process. Sometimes you just want to clear the air or get on the same page and you can use outcome mapping to do that. But let's look at that process again from start to finish. So we want to explore and survey the landscape, understand what's around us, right? What are we dealing with? Where are we, where do we want to go? Then we want to understand what's important to us, why is it important and who is it important to? For different reasons, we want to look at obstacles, we want to understand what's ahead of us so we don't trip over something the moment we get going. And then we look at investigations as options. What can we do to make progress? What can we do to learn something, to figure out how to navigate this landscape that's going to bring us to workflow? What are we doing right now and what does that look like? How do we understand our current workflow? So I use value stream mapping for this. I love it. It's extremely valuable for looking at workflow. And what we're doing when we look at a value stream map is laying out all the activities from start to finish for a given workflow or process. In this case, we can look at a release process or a single stories path through a sprint or whatever you want. It can be as high level or as low level as you want. Obviously, you don't want to spend all your time in the weeds, and you don't want to be so high level that it's not very useful to folks. But after we lay out everything that's happening, we want to measure it to understand where should we focus. So the first thing we measure is how long do each of these things take? And then we measure how much time is passing between each of these steps. So we'll look at step timing and delay timing, and then we can layer all kinds of measurements in addition to that. But those basics are going to give us a focal point, right? They're going to tell us that there's one area or a couple of areas that we could prioritize that we should focus on because they're taking up most of the time. And in this case, we've got a hotspot that's taking up way more time than any other area of the value stream. So all of a sudden, we've gone from everything we could possibly focus on to one focal point, and we're all on the same page. So we can take this super deep, too. We can layer on all kinds of different information. You can see here there's calculations of totals represented. We've got multiple hotspots, we've got roles. These little dots represent different players in the process. We have tools represented here. You can really layer on a number of different dimensions to understand the flow of work at whatever detail level you find productive. So let's look at this again, start to finish really quick. We pick a stream, and a great way to pick a stream. A lot of people struggle with identifying a value stream. Pick a single customer outcome, and that customer could be internal or external. But what is one thing that they need to do? And how do you deliver that outcome? Right. Looking at working backwards from that outcome, what makes it happen? That helps us isolate a single stream, and then we can look at what's happening most of the time in that workflow. So we target a stream, we lay out the activities of what's happening. We look at the measurements for how long each stage takes and what happens between each stage. We add any dimensions that are going to help us understand to a greater level of detail what's going on and where we should focus and how we should focus our efforts. And then we gather all that information into hotspots. What are the things that stand out? What do we notice? Where can we focus our attention? So the next thing that we do, the third stage here, is to look at that constraint, that hotspot, and break it down. Understand what's behind this, what's the reason, right? We might have good ideas, maybe some of us know exactly what it is, but there's a good chance that not all of us do. Right? So this gives us an opportunity to look at that in detail, break it down and understand it, so that we can all split up and go tackle different areas or just be on the same page as we continue through this process of understanding flow and understanding what to do about it. So in this case, environment, setup takes forever. Why does it take forever? Well, we have to go to the data team and we have to ask them for stuff. We have to go to the DevOps team, we have to ask them for stuff. They're busy, they're doing other things. So what's really happening there, right? We can dig even deeper than that. So if we zoom in and then we zoom in again, we see that we're getting data refreshed by the data team. They're cloning some data, maybe from production, maybe from another environment. The DevOps team is doing their thing, but now we can take these and dig a little bit deeper. Which one of them is the most problematic? What's the most time consuming? Which one is the biggest pain that they don't want to deal with. So they just kick it down the road and you end up waiting for it to get done. So let's look at that from start to finish. We're going to look at dependencies, we're going to map these dependencies. We can look at the timing of each of these stages, like, how long do each of these take? Where are the hotspots there? And again, we're zeroing in. We're understanding, where is the major constraint, what's causing it, what are the factors involved, so that we can focus and we know what to do about what we see, and we can plot all that through various means. We can use wordly mapping, we can use simple matrix to understand how close it is to us, how complex it is. There's a number of different dimensions we can use to measure dependencies, but what do we do about dependencies? Right? So we can take those capability that we discovered in the dependencies, the activities, the contributions of each of those external teams, and we can plot them in the context of our team, so we can look at how would we take these things over? How might we improve the workflow around these things and could be that we can't do anything about it, really, but maybe we can improve something on our end that makes it easier for that other team to deliver what we need. But the first part about this that really matters is understanding who's doing what, who owns it. And that could be just a liaison, right? The person who always goes out to that team and gets in touch with them and then waits for whatever to come back. Right. But in a lot of cases we don't even have that defined, right? We don't have a single point person. So no wonder nothing happens. Another big reason nothing happens is that we often don't have a backup for those people. So if they're away, if they're on vacation, if they're sick, everything grinds to a halt or people are just scrambling around, we can look at the skill level, where are we at with these things? Are we actually able to take them on? Could we break those dependencies? Do we have that level of capability? And maybe it's a challenge that we don't have sufficient resources. We might not have access to APIs or documentation or tooling, but we can get all that out onto the page as well and look at that as a group. And all of this information can be collected into an aggregate score. So we can use whatever scoring mechanism we want. We just recognize that these are not equal. Some of these might be easier to tackle than others, and we can use that to prioritize. Again, we're just focusing, focusing, focusing, focusing from our target outcome down to what can we actually do to make a huge impact on that outcome? What can we do to get there? How can we actually make it happen? So let's look at capability. First of all, we identify those capabilities, looking at the dependencies, we find out who owns it so that we actually have like an identified point person. Or at least it makes sense that we haven't been able to break the dependency yet. We can look at skill, and that's skill as a team. We're not singling out individuals here because we live or die as a team. And we look at the support mechanisms that we have, what do we have access to? Can we actually break this dependency? Can we manage this dependency? Is this something that we can actually deal with or is it just going to keep dragging us down? And then all of this factors into this score. The score that helps us focus, helps us have really productive conversations about what to do next. So what do we do next? We take all this information and we can plot it with whatever prioritization method you want. But this can be very simple. It can be very complicated. It's really based on whatever you use to prioritize, use whatever you're comfortable with. But a simple two factor matrix for this works really well, helps you focus on the things that end up in the top. Right. So again, focusing, focusing, using. Right. So out of that prioritization, what we can do is we can understand what do we want to do now, what do we want to do next and what are we going to do later? That's, again, focus. Right. What are we going to do tomorrow to make progress and what can wait or what has to wait, and then what are we going to put on the radar, keep on the radar but not worry about for now. Right. We're going to tell people that we're not worrying about it until we've delivered some of the outcomes that we can earlier on. So we look at milestones and actions as a part of this. We look at measures of progress. How are we going to know that we're headed in the right direction? And then who's going to own those things? Right? Who's going to drive it forward? And this is not a case of that person doing all the work, but someone's got to be able to say, here's where we are, here's what we're learning, here's what's happening, here's where we're going next. And so getting that all out on paper onto this shared workspace where everybody can see and be in alignment means that coming out of this, things start to happen. And there's no reason why things shouldn't happen. So we can lay out these things that we're going to address and understand how we're going to measure the success of those actions, those deliverables, and who's going to own those things. And then we can have varying levels of detail going forward. Obviously, we don't have to lay out everything for later because we can tackle that when we get to later or when we get closer to later. So this is really kind of high definition in the short term and then off into the distance. We don't need that detail. We just need to know we're not going to do it right now. And that clarifies. Again, this is clarifying and focusing. So why do we do this? What's the major benefit here? What are the big ideas? We want to achieve a level of collective flow. So the ability for all of us to work together on these complicated workflows, complicated features, code bases products in a way that's productive. And the way that we can connect that to our daily experiences and what it's going to be like to work in that context is to have personal flow as well. We need a combination of both in order to really make significant progress and to really accelerate over time without burning people out or slowing down as we scale. So we balance collective flow with personal flow. And so we're constantly working at varying levels to make sure that everything flows with minimal friction, minimal waste. So, at the collective level, what does that mean? That means we're minimizing delays. We're attacking those areas of the value stream where we're waiting or things aren't happening, we're not moving things forward. And where we can't move things forward, where things are just stuck in a tedious process or an unproductive process, we can target those areas and eliminate them. And then, in a personal context, what we want to do is maximize engagement. We want to make sure that everybody is operating at their peak capacity. And that's not maximum work, it's maximum engagement. It's that sweet spot between challenge and skill where we see progress. We're engaged, we feel like we're growing, we're enjoying ourselves, and we're making the maximum contribution that we can to the team. And then we want to, as well, maximize the outcomes of that engagement. What's actually being delivered? What is the real progress that we're making? And so, through flow engineering and bringing all this together, we're able to optimize both of those dimensions, the collective and the personal. And you really can't do the personal without the collective. You really need to have an optimized team workflow in order for anybody to really be able to optimize their personal sense of flow. Otherwise, they're constantly being interrupted, they're constantly firefighting buried under work. There's a million reasons why the collective really negatively can affect the personal. So we really have to understand both. But personal flow, I believe, starts with collective flow. And this flow engineering is an excellent method for optimizing collective flow. So what happens when we do this right? Why should we do this? Often, what we'll see is we can do more with less, we can serve more people when we optimize a workflow. If we look back at that environment setup stage, we've got usually shared services managing things like infrastructure, and they're completely overworked. They're constantly firefighting, constantly interrupted. If they get a chance to optimize their workflow, if they can automate. If they can understand what other teams are constrained by, then all of a sudden they can do more with less, they can have much greater impact. We see cycle times go down, lead times go down, fixed times go down. So anytime there's a bug, if you have an optimized workflow, it gets fixed faster with less panic, with less stress, and that can have a positive outcome on what actually gets delivered, as that fix could be the difference between duct tape and an actual solution. We see higher deployment frequency, so deploying more often, learning more often, getting things out and getting hopefully benefits from customers. Feedback rewards, a sense of progress and definitely learning and lead time. This is the heartbeat of a company, right? How often are you actually shipping things out to customers that they can use, that they pay you for, that they're happy with you about? So we see this dramatically reduced when you can minimize waste, optimize flow, optimize performance, and target your efforts so you're not just chasing every improvement opportunity. You focus on what's going to make the biggest difference. So let's wrap it all up. First of all, clarity your target by understanding your target outcome. You can use an outcome map for this. Very simple, very effective. Then understand your value stream. What's the current workflow? Where's the constraint? What is being negatively impacted by the current workflow? And then understanding the dependencies behind those things. It could be there's zero dependencies, but often I find the biggest delays are connected to external dependencies. You've got to wait to go and ask for approval or wait for somebody else to do something for you that perhaps you could do self service and then capabilities. What is it going to take to break those dependencies or improve our level of performance with the things that we do have control over? That all comes down to capabilities, capabilities around automation, around creating self service, ux, good documentation, adopting new toolings, stitching things together within integrations, there's all kinds of capabilities that improve flow, and so we can use those capabilities to address dependencies and accelerate. Remove that waste, remove the delays, and all of this. Everything you learn, all the insights from, all the mapping, goes into this flow roadmap, which is not just a roadmap, it's a roadmap for how you ship your real roadmap, right? The roadmap of features that you need to deliver. This is the roadmap for how you're going to ship all those features. So this is the most neglected map in every organization is how are we actually going to do the things that we want to do. Not a map of things that you want to do, but the how of anything that you want to do. And if you unlock this, then you can really tackle anything. So bringing all this insight together, plotting into something that's actionable, is highly effective. So why do I love mapping? And why are people talking about this now? People are going to start talking about this a lot now because this just showed up at the very top of the thoughtworks radar. So the thoughtworks radar has been going on forever, and this is the first year that they've actually added what they're calling path to production mapping, because value stream mapping means different things to different people. But essentially what we're talking about here is end to end flow with mapping, and it showed up at the top of the radar. Now, this is a huge development. It went from not on the radar to the very top, and it's because it is cited by every success story. Everyone who talks about massive performance improvements talks about looking at the end to end flow and understanding where the data was pointing, their efforts, their attention, and making decisions based on that data. So flow engineering is really going to ride this wave, and hopefully we hear a lot more about collaborative mapping and flow focused mapping in the coming months and years. So a little bit more about why mapping is so valuable and so important working together in this way on a visual medium. And again, if it's a whiteboard or postit notes or miro or mural, it really doesn't matter. You could use anything for this, but it's engaging, right? It's really valuable to have everybody sort of participating and working together to build this picture of what everybody sees, everybody's different perspective together. As you're building this, you own it. You feel a sense of attachment to what you've created, what you understand, and most importantly, the next steps. What are you using to do with this? Feeling a sense of ownership over that process and the outcome of the map. The map that you create is really valuable. A lot of people feel like they're not a part of the planning process, they're not a part of decision making. This brings people in. It's extremely inclusive. It also brings people into alignment. Everybody has a different perspective, everybody has a different opinion. So unless you get those out into the open, you're not going to understand that people are pulling in different directions or have very different understandings of where they're headed and how to get there. So this provides an opportunity for folks to really come together. It's entirely customized. This is not a prescriptive framework you will not get the same outcome. You will not make the same maps twice. It's incredibly customizable and it's entirely bespoke for whatever context. You can make the same maps in the same team two weeks apart and have a totally different outcome that represents where they are at that moment. So it's incredibly valuable for representing exactly what you're dealing with, not trying to copy Spotify or Netflix or whoever folks think is doing a good job. This is how you do a good job in your context. It's also something that you remember. You retain the things that you learn because you're a part of interactively learning those things and discovering them yourself along with everybody else. So it sticks. And again, this is all developed over the pandemic, so it's incredibly remote friendly. It works just as well remotely where everyone's in a different time zone or in a room, it works for both. I had very wonderful results with any scenario. It's very flexible and you can deploy this, you can use this however you choose in your own environment. So with that, if you want to learn more, if you're curious about this, you can shoot that QR code. It's going to take you to a bunch of links. But there's an ebook and an email course that is entirely self guided at flow. Visible is it'll take you through start to finish. And I've also launched with a good friend of mine, Andrew Davis, the value stream show, which we're talking about everything. Value streams, mapping, management, flow, organizational dynamics, everything that it takes to achieve that collective and personal flow. And with that, I want to say thank you, really appreciate you taking the time to check this out, and I hope you reach out to me and let me know what your you're doing for flow or struggling with flow in your organizations.

Steve Pereira

Founder - Transforming Software Team Value Delivery @ Visible Value Stream Consulting

Steve Pereira's LinkedIn account Steve Pereira's twitter account

Awesome tech events for

Priority access to all content

Video hallway track

Community chat

Exclusive promotions and giveaways