A few months ago, I blogged about testing quality@speed in the same way that F1 racing teams do to win races and fans. Last week, I published my F(TA)1 Forrester Wave! It examines the capabilities of nine vendors to evaluate how they support Agile development and continuous delivery teams when it comes to continuous testing: Borland, CA Technologies, HP, IBM, Microsoft, Parasoft, SmartBear, TestPlant, and Tricentis. However, only Forrester clients can attend “the race” to see the leaders.
The market overview section of our evaluation complements the analysis in the underlying model by looking at other providers that either augment FTA capabilities, play in a different market segment, or did not meet one of the criteria for inclusion in the Forrester Wave. These include: 1) open source tools like Selenium and Sahi, 2) test case design and automation tools like Grid-Tools Agile Designer, and 3) other tools, such as Original Software, which mostly focuses on graphical user interface (GUI) and packaged apps testing, and Qualitia and Applitools, which focus on GUI and visualization testing.
We deliberately weighted the Forrester Wave criteria more heavily towards “beyond GUI” and API testing approaches. Why? Because:
Formula One has gotten us all used to amazing speed. In as little as three seconds, F1 pit teams replace all four wheels on a car and even load in dozens of liters of fuel. Pit stops are no longer an impediment to success in F1 — but they can be differentiating to the point where teams that are good at it win and those that aren’t lose.
It turns out that pit stops not only affect speed; they also maintain and improve quality. In fact, prestigious teams like Ferrari, Mercedes-Benz, and Red Bull use pit stops to (usually!) prevent bad things from happening to their cars. In other words, pit stops are now a strategic component of any F1 racing strategy; they enhance speed with quality. But F1 teams also continuously test the condition of their cars and external conditions that might influence the race.
My question: Why can’t we do the same with software delivery? Can fast testing pit stops help? Today, in the age of the customer, delivery teams face a challenge like none before: a business need for unprecedented speed with quality — quality@speed. Release cycle times are plummeting from years to months, weeks, or even seconds — as companies like Amazon, Netflix, and Google prove.
What a strange summer this has been! From Boston to London to Paris to Turin, the weather has offered weekly and even daily reversals, with continuous change from sun to rain, from hot and damp to cool and crisp. I missed a nice spring season. Even today, from 35º-38º Celsius (95º-100º Fahrenheit), we just went to 22º Celsius (71º Fahrenheit) with a perfect storm! A continuous climate and sudden change is quite unusual in some of these countries. Certainly it is where the Azores Anticyclone usually dominates from mid-late June to mid-late August, offering a stable summer. How many times have you had to change plans because you discover weather is about to change!?
You might be thinking, "What does this have to do with this AD&D blog?" It’s about change! I am wondering if, in our daily lives, getting used to unexpected conditions and having to handle continuous change favors a mindset where change is just something we have to deal with and not fight. A new mindset very much needed given the change we see ahead in how we develop, test, and deploy software!
My focus in this blog is testing, although the first change we need to get used to is that we can’t talk any longer about testing in an isolated fashion! Testing is getting more and more interconnected in a continuous feedback loop with development and deployment. (See my colleague Kurt Bittner's report on continuous delivery; I could not agree more with what Kurt says there!)
I just finished my new report on the Agile testing tools landscape. I’ll point Forrester readers to it as soon as it publishes. But there are few things that have struck me since I took over the software quality and testing research coverage at Forrester and which I would like to share with you in this preview of my findings of the testing tools landscape doc.
My research focus area was initially on software development life cycles (SDLCs) with a main focus on Agile and Lean. In fact, my main contribution in the past 12 months has been to the Forrester Agile and Lean playbook, where all my testing research has also focused. Among other reasons, I took the testing research area because testing was becoming more and more a discipline for software developers. So it all made sense for me to extend my software development research focus with testing. But I was not sure how deep testing was really going to integrate with development. My concern was that I’d have to spend too much time on the traditional testing standards, processes, and practices and little on new and more advanced development and testing practices. After 12 months, I am happy to say that it was the right bet! My published recent research shows the shift testing is making, and so does the testing tool landscape document, and here is why:
DevOps is a movement for developers and operations professionals that encourages more collaboration and release automation. Why? To keep up with the faster application delivery pace of Agile. In fact, with Agile, as development teams deliver faster and in shorter cycles, IT operations finds itself unprepared to keep up with the new pace. For operations teams, managing a continuous stream of software delivery with traditional manual-based processes is Mission Impossible. Vendors have responded to DevOps requirements with more automation in their release management, delivery, and deployment tools. However, there is a key process that sits between development and operations that seems to have been given little attention: testing.
In fact, some key testing activities, like integration testing and end-to-end performance testing, are caught right in the middle of the handover process between development and operations. In the Agile and Lean playbook, I’ve dedicated my latest research precisely to Agile testing, because I’ve seen testing as the black beast in many transformations to Agile because it was initially ignored.
Recently, I’ve been getting more inquiries around risk based testing. In addition to agile test methods and test estimation, test teams turning their eyes to risk based testing is just another positive step in integrating quality through out the SDLC. Yes, I still see QA engineers as having to put their evangelist hats on to educate their developer brothers and sisters that quality is more than just testing (don’t get me wrong, consistent unit and integration testing is a beautiful thing), however, any time that business and technology partners can think about impact and dependencies in their approach to a solid, workable application elevates quality to the next level.
Keep asking those questions about risk based testing – and make sure that you’re covering all of the angles. Make sure that you’re covering: