Part 1: Testing Tools Market Landscape: Agile Keeps It Alive, Vital, and Dynamic

I just finished my new report on the Agile testing tools landscape. I’ll point Forrester readers to it as soon as it publishes. But there are few things that have struck me since I took over the software quality and testing research coverage at Forrester and which I would like to share with you in this preview of my findings of the testing tools landscape doc.

My research focus area was initially on software development life cycles (SDLCs) with a main focus on Agile and Lean. In fact, my main contribution in the past 12 months has been to the Forrester Agile and Lean playbook, where all my testing research has also focused. Among other reasons, I took the testing research area because testing was becoming more and more a discipline for software developers. So it all made sense for me to extend my software development research focus with testing. But I was not sure how deep testing was really going to integrate with development. My concern was that I’d have to spend too much time on the traditional testing standards, processes, and practices and little on new and more advanced development and testing practices. After 12 months, I am happy to say that it was the right bet! My published recent research shows the shift testing is making, and so does the testing tool landscape document, and here is why:

  1. Testing and development are getting more and more integrated and becoming an integral part of the SDLC from its very initial phases. Developers are adjusting to this trend; surprisingly, QA managers don’t take it too badly, although for many of them this might mean no longer managing an army of manual testers (the power syndrome of managing large groups of people). As you can see in the figure below, extracted from our Forrsights Developers Survey, Q2 2013, developers use various testing tools every day! 
  2. Developers love open source tools and choose them as a first option. Because developers are getting more involved in testing, specifically in unit and automation (functional and nonfunctional testing), performance and integration testing, they also have a strong say in the tools used for testing. You will see in the testing tools doc research that open source tools are flourishing in all main testing categories: test management, test data management, automation, and performance. In tools, categories like test-driven development, unit testing, code quality, and bug tracking are open source and developers are an entrenched binomial.
  3. New tools (although some might say old wine — SOA testing tools — in a new bottle) are emerging: service virtualization tools. This is an area I am closely monitoring and will be doing more research on going forward. Simulation is a technique frequently used for testing purposes in other domains like engineering, oil and gas upstream, military, manufacturing, etc. — why not in software development? It’s been for many reasons, including management’s low interest in spending money on testing and quality. But with software becoming more strategic to enterprises, software quality and testing are becoming first-class citizens aiming at primary budgets. So simulation in the form of service virtualization (a higher level of abstraction compared with the virtualized test environments and labs you are probably more familiar with) becomes an important enabler for providing a stable testing environment that simulates a real complex software integration landscape. In fact, service virtualization allows for early integration testing, early automation and functional testing, and continuous and incremental testing. The big guys, like IBM, HP, and CA are all heating up their marketing engines to position themselves in this nascent market. Parasoft has a solution in this space too. I am actually seeing an interesting evolution of this whole area into testing optimization, where integration of service virtualization with downstream automation tools provides a complete end-to-end testing and continuous deployment solution  (both IBM’s acquisition of  UrbanCode and CA’s acquisition of Nolio give an indication of this). But this is only the start; be sure that more on this will follow.

I'll stop here. You will read Part 2 of this blog once my research, “Navigating The Agile Testing Vendor Landscape,” publishes, and will add some more takeaways from my research. I’m also completing the “Agile metrics that count” research and will be blogging about that too. BTW, the “agile metrics that count” doc will complete our Agile and Lean playbook!

I’d love to talk to folks or hear from those that are having a good or bad experience with service virtualization tools in Agile contexts.

As always, your comments are welcome!

Diego