Future fit organizations are quickly adopting hybrid and native clouds, automating wherever possible, and bringing their agile and DevOps practices to a more mature and broader level. At the same time, testing tools, testing practices, and testing software IP developed by vendors have all increased the infusion of AI. The consequence is that ruthless or hyper automation gets replaced by smarter automation: Less automation can be more. Clients accelerating their digital journey look for their service providers to support them in whatever it takes to deliver better software faster.

In The Forrester Wave™: Continuous Automation And Testing Services, Q3 2021, we evaluated and compared the 15 most advanced and mature continuous automation and testing (CAT) services providers that can help companies make sure that testing does not become a major obstacle during their digital acceleration.

Leaders, Strong Performers, And Contenders: All Close To The Finish Line

Despite the spread of positioning of vendors in our Wave graphic, in some cases, minor differentiation still exists among them. To emphasize differentiation, scores across the 25 evaluated criteria get normalized to three standard values: scores of 1 reflecting below-average performance among the evaluated providers, scores of 3 within the average range of evaluated providers, and scores of 5 for being above average of the evaluated providers. In other words, a 1 does not mean that the vendor provides a terrible service. It is just telling you that vendors scoring a 3 or 5 did better in our evaluation. Sometimes the average is quite high, so even a 1 could mean that the provider is doing well. Of course — as always — our Forrester Wave methodology allows you to balance the relative weightings of these scores based on your own requirements.

It is also important to note that the evaluation also takes into account the feedback from 68 reference clients that were provided by the vendors and that we interviewed as part of the Wave process. This client feedback often pushed the individual vendors’ scores up higher or down lower compared to our own evaluation. The scores are also influenced by the ability of the vendor to adhere to the Wave process as effectively as possible, just like participating in an important client’s request for information/RFP process.

IBM, Tata Consultancy Services, And Cognizant Lead The Pack, But Others Are Getting Close

The 15 vendors for this Wave were chosen through a rigorous selection process of collecting data from 43 players in the CAT services market that matched our inclusion criteria through the Now Tech: Continuous Automation And Testing Services Providers, Q1 2021 research. Among the chosen 15 vendors, the Wave evaluation process dictated the Leaders, which are IBM, Tata Consultancy Services, and Cognizant, followed by the Strong Performers — EPAM, Hexaware, Wipro, Accenture, Infosys, HCL Technologies, Capgemini, and Cigniti Technologies — and the Contenders, which are Larsen & Toubro Infotech, Qualitest, Tech Mahindra, and NTT DATA (which was a nonparticipant). If you are a Forrester client, you can read more about the details here.

Choose Your CAT Services Partner Based On Comprehensiveness And Completeness

You should look at the comprehensiveness and completeness of your vendor’s CAT services and consulting capabilities. Understand how these companies acquire, groom, and train talent, how they invest in innovation continuously and — last but not least — how they build and add value to services with their own IP in the form of accelerators. But above all, when evaluating your CAT services partner, you should look for vendors that:

  • Make customer intimacy their North Star. Clients expect all provider resources to embody the role of a strategic partner by anticipating customer needs, introducing new practices and technologies, going the extra mile to solve problems beyond contractual constraints, and pushing innovation.
  • Collaborate as true partners with clients to modernize their testing programs. Clients expect providers to support their need for various team sourcing models and testing services execution. Furthermore, they look for transformation programs and services supporting their testing modernization journey with senior and experienced consultants who can coach their resources on the job.
  • Offer a modernized and innovative CAT services portfolio. Technology advances around AI and machine learning are driving customer expectations. Clients expect their providers to infuse AI in their test automation platform accelerators for functional test automation, self-healing of tests, design automation, and more. They need testing consultants to be fluent with machine learning to support them in their strategic AI journey plan, which requires testing, too.

I’d be happy to have an inquiry with you if you are a Forrester client going through your own selection process and want to navigate this crowded and still-evolving market. Contact us through inquiry@forrester.com, and I’d be happy to dive through the details. Have a great summer.