As the software development industry has seen unprecedented levels of digital transformation, the demand for automated testing in the CI/CD pipeline has taken on greater urgency, especially at the early stages. Also, new advancements in AI are helping developers with some of the biggest challenges in testing: test creation, maintenance, and many manual tasks. Many companies have noticed and are spending more on their automated testing initiatives. Strong testing practices have gotten to be so crucial that they’ve become “the main differentiator between companies that are successful and those that aren’t,” according to Guy Arieli, the QA CTO at Digital.ai.
When companies are looking for an automated testing solution, they’re primarily looking for one that will increase the quality of their releases, increase the speed at which they can be done, and be the most cost-efficient. A common approach among enterprise customers is to seek a vendor that satisfies most of their needs while integrating into their CI/CD pipelines.
“We see customers want a unified solution. You don’t want to be using disparate tools for end-to-end testing of different types of clients,” said Dan Belcher, the co-founder of Mabel. “Increasingly, they’re pushing us to add value to those end-to-end tests with insight around things like performance and visual correctness and other kinds of attributes of quality because they’re trying to move from pure quality assurance like ‘did I break this core feature?’ to quality engineering: ‘Is the feature better than it was before? Is it faster? Is it more accessible? Is it visually appealing?'”
RELATED CONTENT
A Guide to automated testing providers
How do vendors help customers with their mechanical testing initiatives?
Chris Haggan, the product management lead at HCL OneTest, said it’s more than just getting the solution with the most features. It’s also about supporting users with the tools they already have and seeing if it fits appropriately with the development organization’s overall approach. Another issue is whether the organization has enough resources to add testing solutions to the mix since they can add complexity.
Where to start?
Organizations must build quality into the application earlier to start with their automated initiatives. Quality has become a core functional necessity, and testing early on is a vital part of that.
“We see people all the time wanting to fully automate everything in weeks. Yes, of course, that is technically possible. Still, it takes time to evaluate what’s important to test, how solutions fit into your CI/CD chain, which generates test data, and so on, according to Kevin Surace, the CEO, and co-founder of Appvance. “While no one wants to hear it, the best automation strategy is laid out over a year,” Surace added.
Building quality comes down to both the organization’s culture and in executing deep code analysis, as well as deep reliability and security at the earliest stages. “If you put it at the end, you can’t accelerate your delivery. You’re always kind of running into a bottleneck at the end of the process,” said Mark Lambert, the vice president of Strategic Initiatives at Parasoft.
This has led to continuous quality and compliance, which need to be tested in the CI/CD pipeline. Mabel’s Belcher said that as more extensive automated testing becomes available, his one concern is that it will create a test sprawl in which it’s so easy to make end-to-end tests and get the coverage you want in a place that perhaps teams will become more complacent about testing.
“Just because it’s easy doesn’t mean it’s right,” Belcher said. “They have to put more thought into, you know, are these tests accomplishing the objectives that I set out? Are we doing only what is necessary? Do we think about the data? Do we have the right environments? And there’s a lot more than just the capability to add lots of requests. We keep score by the quality of what’s in production.”
Organizations must also prioritize tests that need to be automated first to avoid getting overwhelmed. “What I want to achieve is not more and more tests. I want as many tests as possible because that will minimize the maintenance effort and still get the kind of risk coverage I’m looking for,” said Gartner senior director Joachim Herschmann, who is on the App Design and Development team.
In the past, what used to happen is that the organization used to say, “We’ll recruit the developer, and we’ll do the R&D, and then when we need to test it, we’ll send it to India, and then it will be tested there.” Still, organizations realize that this has to be at the core of their R&D organization, Digital.ai’s Arieli explained.
As developers get more involved in quality, the notion of building quality as part of the application has begun to take hold.
“So developers have to think about how do I engender unit testing and more and more of it, and when you reach the total extreme of it, and you’re mature, they start thinking of automation,” said Anand Sundaram, the SVP of Products, UI, Device Cloud, and Performance Testing at SmartBear Software.
Security and performance testing
After quality, there are other aspects of the application for which automated testing can be leveraged: security and performance testing of your APIs and microservices at the developer level before everything comes in for integration testing or the entire application comes together.
“We’ve accepted that test automation is valuable, deep code analysis is valuable, and now we’re starting to say the same thing around security; how can we embed security in each stage to build security into the pipeline,” Parasoft’s Lambert said.
Now, testers are trying to apply the same methods they used to test quality to security. So that means deep code analysis to identify potential runtime exceptions that could go uncaught. As they move up the stack, they’re looking to leverage unit testing to fuzz the underlying code and see how they can utilize API tests for API security testing. Developers can start to build quality and security by taking advantage of those earliest-stage validation techniques, Lambert explained.
While automated testing has received widespread recognition as a must for today’s software development environments, organizations face many challenges when setting up effective testing strategies in their CI/CD pipelines.
At the bottom of the testing pyramid, the struggle with unit testing is that there isn’t a lot of visibility, and it’s difficult to understand how much it covers. The system and UI testing are at the top of the pyramid, which can be the most challenging. On top of that is the service component testing, usually driven by an API.
Implementing all of these levels of testing can be a challenge, especially for legacy systems since these aspects of testing were not initially accounted for when the applications were created, Digital.ai’s Arieli added.
Another challenge in implementing automated testing is finding the staff with the appropriate skill set. Testing complex enterprise applications requires business domain expertise. Also, maintaining test scripts makes it difficult to achieve continuous test automation — as automation requires teams to ensure that testing doesn’t become a bottleneck. Therefore, tests must be designed to minimize disruption to the ongoing testing process. The goal is for test automation teams to build robust and reusable test scripts that don’t require constant attention and maintenance, according to Clinton Sprauve, the director of Product Marketing at Tricentis.
Organizations must also find a way to manage and track test automation efforts across multiple tools through observability and analytics.
“There is a challenge to testing in the sense that we need to do it more frequently, we need to do it for more complex applications, and we need to do it at a larger scale. This is impossible without automation, so test automation is necessary,” Gartner’s Herschmann said.
AI and observability in automated testing
With value being a core tenet of DevOps, managers must see how each decision impacts the user experience, revenue, and business performance. This is why testing providers want to create more intelligent means of testing that can provide analytics.
Intelligent testing can combine data analytics, clever heuristics and algorithms, machine learning, and anything that analyzes data in real time and makes decisions or recommendations that help solve the problem. Developers can then use that need to have instant feedback on where precisely the problem occurred and move much more quickly.
Observability is needed in the pipeline because it gives testers a clue as to where exactly the problem is when the problem occurred and then alerts the tester. In addition to observability, automated testing solutions have also created ways to help developers with many pain points around testing and speed up the process.
“At the beginning of Agile, when you start talking about quarterly releases, you could still kind of fake it, right? You could still handle quality. You would have minimal time to do all your regression testings, but you could build that into a schedule and make it work. When you move to CI/CD where change is continuous and disruptive, you need to find new solutions,” Mabel’s Belcher said. “And so for a few years, as an industry, we turned to, well, let’s just make us another thing that the developers have to worry about and have them write tests that do end-to-end validation. But the problem with that is that those tests relied on the stability of the very thing that was constantly changing.”