Conf42 Machine Learning 2025 - Online

- premiere 5PM GMT

Rethinking Test Automation: A Modern Approach to Managing Complexity and Enhancing ROI

Video size:

Abstract

The complexity of modern test automation has escalated, leading to: - Maintenance challenges - Resource drain - Diminished returns on investment (ROI) This session introduces a transformative framework designed to address these issues by integrating a modular, data-driven, and Excel-based test automation model.

The Utility Model: A New Approach to Test Automation

The proposed utility model utilizes a lightweight, keyword-driven design that: - Streamlines test case creation - Reduces script complexity - Enhances reusability

Key Results from the Utility Model:

  • 35% reduction in execution time - 50% less maintenance effort within the first six months

Industry Struggles and Challenges

Recent data highlights significant struggles within the industry: - Only 62% of automated tests pass on their first run - 45% of organizations face tool selection issues - Enterprises spend $100,000 to $300,000 annually on automation infrastructure, yet many report low ROI The utility model addresses these challenges by improving: - Modularity - Optimized code coverage - Seamless integration with CI/CD pipelines

Benefits of the Utility Model:

  • 60% faster debugging cycles - 45% better test script maintainability - 70% reduction in environment-related failures

Overcoming the “White Elephant” Syndrome

In many enterprises, automation initiatives suffer from the “White Elephant” syndrome, where resources are consumed without delivering proportional benefits. The utility model directly counters this by: - Reducing resource consumption by 43% - Enhancing test reliability by 50% - Ensuring 80% better alignment between testing efforts and business goals

Improved Outcomes:

  • 40% improvement in defect detection rates - 55% faster feedback loops in CI/CD pipelines

Practical Insights for Sustainable Test Automation

This session will provide practical strategies for adopting and scaling a sustainable test automation strategy. Attendees will learn how to: - Reduce costs - Improve testing efficiency - Boost quality and ROI Join us to discover how this innovative approach can drive better results in your test automation efforts.

Summary

Transcript

This transcript was autogenerated. To make changes, submit a PR.
So today I'm, I have come up with an unique topic that will help you, to optimize your test automation and be successful in your endeavor. So before going into the topic, I would like to thank Con 42 for giving me this opportunities to exchange our ideas and for common benefit. Okay, now the topic is about rethinking test automation, a utility based, modern approach to manage complexity. Alright, so right now in software engineering and mainly with test automation, since it is not a conventional. Department or con, conventional part of software engineering. There's still, there is lot of maturity needs to be achieved and there is lot of failure rate in this, especially in this area. So this is where I'm going to focus and walk you through how we can rate maximum benefits. Okay. Before getting into the topic, I would like to take you through the evolution of test automation. Early 2000. 2000, we have a, simple and tools like, with the limited maintainability and limited scope it helps to cover application of four to five screens and with it's like a simple, it doesn't have any help, any integrations or APIs or any new s right. And 2010, we have come up with the page object model, open source software to test the application which. Even though we are able to progress a lot with the increase of software in our life day-to-day life, the success rates still stay at the same level as in the original days and even the present day with a lot of technological shifts, advancements, we have advanced systems like Selfing Scripts, ml, visual Recognition Analysis, but still. The success rate remains same. Why? Because these are still evolving and there is lot of scope to evolve in this. So the right approach, we take the right place, we invest our money today helps us to reap the maximum benefits of these technological advances we are going to see in the coming years. Okay, so what are the current day challenges? So current day challenges is the growing complexity. Now we have seen the software is there in every minute of our lives. Every minute we live, we are using, we are depending on some part of the software. So that's how the complexity of it's growing and it needs to be more integrated and more there is no chance of failure because it can result in huge loss and huge customer dissatisfaction. So that is the, significance of software test automation where we have, where it'll help us to test all these bits and pieces of softwares and make sure there is no overhead costs. That is the failure of the co software is going to create us to avoid those failure costs. We have to have it thorough testing, and the test automation helps to achieve that. The technology shift? Yes. The pace at which the technology is changing is unforeseen. Nobody has imagined to this level. And this is just a starting point. We are just still in the first step and right. So there is a lot more to go through. Long way to go in this technology channel. So given the, this scenario, what are the common pitfall pitfalls? Organizations are attempting to automate more than 60% of test cases with first year experience, 78% failure rate teams managing over a thousand test cases spend around four to five hours weekly on maintenance, while 67% of automation product fails due to improper test case selection, right? So what this gives us. This common pitfall is the coverage or the results of automation is not as expected. Like we are expecting a hundred percent, but we are achieving only seven 32%, where 78% is the failure. Second thing, we are spend in huge amount of time in maintaining the software and why we fail, because why we fail in achieving the coverage or in achieving the maintenance health of the software we developed. Why? Because we are failing in automation, test selection. We are failing in identifying the right automation candidate. So that is where that tech strategy comes in, and we have to spend time on this strategy to achieve the results. Okay we have seen theoretically, but practically what is it happening in, on ground is we expect a hundred percent pho rate, but we are achieving 62% in the first run. And as the age we have seen the previous slide, right? So we have to keep on investing money on maintenance. If we don't maintain it, this 62% will fall to 26% very soon. And then the test coverage, we expect 80%. We are achieving 54%. There is a huge gap of 26%. And mainly the CICD integration, we are still in the native primitive standards. We are still not there where we can see. The, our target of achieving a hundred percent success. And same with the defect detection. Why? Because most of our tests failed due to lack of maintenance. There is still, majority of our scripts doesn't get an opportunity to be executed and do identify that defects. So giving this background, what do you see? So we see that, right? There is a huge investment we are putting in, but for less result, less output. So that is what we call it as a white elephant syndrome. So why do we call this? Because it's test automation is a kind of a black hole where it drains all our resources. Because we have to invest tons in building the big elephant. And it's not done by building an elephant. I have to feed it every day. So unless there is a proper approach to use proper environment, eco ecosystem to use this elephant, it is going to eat all my resources. Just imagine you are repeating, you are petting elephant today. What is the use? No, but you have to spend a lot to mainten it. So similarly, the maintenance challenges as we discussed, it's a huge maintenance. Like even, any company in generally small software project has to invest in 2.5 fulltime engineers with each engineer about hundred K 25. Just in maintenance in the resource cost, right? And tools and then failures, and then technology shifting, upgrades migrations. These are all additional, right? So how does the annual cost looks like the breakup, as I told you right, the tools and licenses they cost a huge, actually if I member team or 10 member team roughly, it would cost about one 20 k. And as you discussed, the huge maintenance cost and the personal cost as I told you, 2.5 full-time employees and a direct and opportunity cost. So these are direct end opportunity costs or the failed initiative costs directly impacting roughly around two. For any decent size of the software team with five to 10 testers, and while opportunity costs reach to 300 k, the cost per test case execution raises from two to $3 to eight to $10 within two years. With the raising with the raising raising, the resource cost, personal cost infrastructure tools. All these costs are going to add up and it's going to multiply our costs in near future. So I would say this is the high and that we spend time on the proper strategy and we invest on the tools wisely so that we can be saved or we can sustain in this global market. Okay. What is my proposed solution? So my proposed solution is based. In our model framework here the core strategy comes into play in identifying the testing candidate. So the testing candidate should be in such a way that it is to, it has to help the manual testing speed up their most complex jobs with the simplest script, right? And less maintenance. This is where the strategy comes up. So when you implement this strategy. The manual testing resources, the manual testing personal could be drastically reduced as majority of the heavy weight load. Is taken by these utilities. So why I would say this utility, I can say it as a normal test automation framework test case flow or something I can say, right? Why I'm calling this as utilities. So these utilities will have, everything will have unique entry and exit points, unique capabilities. Like for example I have to test a data a data set of 1 million records. So manual testing, manual, it'll take me five days, six days with sampling technique, right? But this utility technique, we can implement. Forget about the conventional testing approach. So we develop utilities. You take any technology you bring in a new technology that is completely different to your technology stack. For example, Python. You validate that data set and give a result in a standard format, right? So these utilities will help us to solve the main pain areas of the software testing or the manual testers, and these utilities will fit into our overall enterprise framework of keyword driven, and then our standard logging and logging framework. That way the output of this utility will become in input further next utility to carry on to carry upon. So that's how we are decoupling these complex pieces, calling them as utilities, building in its own ecosystem, its own technology stack. And then integrating them with our life framework, which includes the enterprise level logging. You got it. So that's how we are going to scope our test automation. We, we are not going to, with attack the whole hundred per a hundred percent of test cases, testing testing landscape, and pitfall unable to achieve it. So here we are wise enough to choose what is to, what is most required to be automated, where it is going to help us more for that. We take the help of this universal framework and universal login. Okay. So with this approach, what did we achieve? We achieved 85% of maintainability. It's very easy. We have reduced 85% in maintenance costs. And then since these are utilities we focused more on API based testing rather than UI based testing because the major pain or the major failures happen at API are integration level. And then we have implemented separate tools for UI testing alone. That way we have we, we know the strategy, like where we are going to cover what component. That's how we are able to achieve better reliability and then the better reusability. Yes, of course. So if I develop a utility in my team, I can definitely use it anywhere else wherever it is needed. And then faster execution. Since this is not a conventional testing, we are taking every approach to test database EAPI or UA layer separately, disintegration of UA layer. Separately. Our execution is very faster. And main importantly, more importantly, it is about the sanity testing. We are able to san test the sanity of any application within minutes. Sorry. So what are the benefits of this utility approach? As I told you, the, the improved scope management, it's not like I'm planning to cover. And I'm able to achieve 25%, but here I'm sc Descoping my the coverage and then I'm trying to reach, achieve a hundred percent of the redefined coverage. So that way I know what I'm doing, what I'm expecting, and how much money to put. I'm not building resources to automate everything a hundred percent. Just putting resources to automate my 25% and hundred percent of it. And then the tool flexibility. Yes. And then faster implementation, reduced maintenance. Yes. Overall with the technology shifts, new tools coming in, new technologies coming in. Since my standard approach is fixed, my platform is fixed, I just have to, migrate from one technology to another technology. But my strategy, my automation resources are going to stay safe forever. So as in long run, right? In long run since the strategy is good, somebody who knows the strategy, it's just a matter of with the latest we have the GitHub co-pilot or charge GBT. It's very easy to migrate from one to others. And then the implementation strategy. This is like more of generic, established, clear objective. To say, here, establish a clear strategy on how much to automate, what to automate, what component to automate. Just forget about everything. Focus on that component, develop the utility, and then we can integrate with others. So build organizational support. Yeah, do analysis and get the stakeholder buying. Implement gradually. Measure and optimize. So measuring success? Yes. Key performance indicators. So define your key performance indicators and track your written on investment. So in my projects, wherever I have implemented utilities, so they are like independent utilities. Most of the cases where we have achieved return on investment, it's 300% return investment in first few months. And then, and there are some comp, some utilities where we had to fit into the enterprise architecture for the continuity of the flow. So this is like a top down or bottom up approach the conventional testing where we come with list of test cases to be automated while automating it. I develop utilities slowly and then go on right. But here it is like bottom up approach. I, first, I know the utilities where I can help to manual testing to ease up their complexities charts and later if time permits or if I need, I can build on test cases by, by integrating multiple utilities so that way we know where we are going to, how much we have to invest, and then quality first. Teams should always monitor the automated execution results. But still we have the standard logging system and the logging pattern is what integrates from one utility. It very easy to track and. Okay. Thank you friends for giving me this opportunity. So hope you would have got some idea on how these unconventional strategies maverick thing thinking implementations will help ease up our life. In software testing space and especially in software test automation. So if you need any help or future guidance or some case studies of my previous works, I'm happy to share. So please do reach out to me. Thank you. Bye.
...

Sujeeth Manchikanti

Test Architect @ Capgemini

Sujeeth Manchikanti's LinkedIn account



Join the community!

Learn for free, join the best tech learning community for a price of a pumpkin latte.

Annual
Monthly
Newsletter
$ 0 /mo

Event notifications, weekly newsletter

Delayed access to all content

Immediate access to Keynotes & Panels

Community
$ 8.34 /mo

Immediate access to all content

Courses, quizes & certificates

Community chats

Join the community (7 day free trial)