Posted by Diego Lo Giudice on January 16, 2013
DevOps is a movement for developers and operations professionals that encourages more collaboration and release automation. Why? To keep up with the faster application delivery pace of Agile. In fact, with Agile, as development teams deliver faster and in shorter cycles, IT operations finds itself unprepared to keep up with the new pace. For operations teams, managing a continuous stream of software delivery with traditional manual-based processes is Mission Impossible. Vendors have responded to DevOps requirements with more automation in their release management, delivery, and deployment tools. However, there is a key process that sits between development and operations that seems to have been given little attention: testing.
In fact, some key testing activities, like integration testing and end-to-end performance testing, are caught right in the middle of the handover process between development and operations. In the Agile and Lean playbook, I’ve dedicated my latest research precisely to Agile testing, because I’ve seen testing as the black beast in many transformations to Agile because it was initially ignored.
On the other hand successful Agile teams do focus on testing up front: while they continuously develop, continuously integrate, and continously deliver they also continuously test. Actually, they even tend to lead development with testing. In other words, Agile elevates testing to the rank of a first-class citizen, just as development and operations are in DevOps. So I think we need a new buzzword that makes sure proper attention to testing is given. It could be TestDevOps or DevTestOps — a debate I’d like to hear your opinion on. The first new buzzword conveys the concept that testing is highly ranked as development and operations but also leads both; the second that testing is just as important as the other two and deserves more attention, especially to succed with Agile.
But the real important findings are that companies interviewed for this research went through some dramatic changes in the way they do testing. They claim that, to be successful with Agile, you must fully integrate testing practices into your development life cycle, increase automation, and change testers’ and leaders’ missions and day-to-day jobs to bring them closer to the needs of the Agile projects. There will also be change for development and developers. Here’s a summary of the takeaways from the report:
- Testing, business, and development skills are mandatory for succesful Agile testing. Two broad categories of skills are needed for Agile testing: testing expertise (combined with business domain knowledge) and development expertise to support the implementation of good automation.
- Test managers must be hands-on to be coaches and change agents. Test managers have a very different role in an Agile environment.
- Walls have to be torn down to make testers an integral part of development testing. It’s essential to execute test activities at the pace of the Agile development team, with frequent testing iterations and automated regression testing on top of continuous build and code integration.
- Test-driven development approaches have to go beyond unit testing. Increasing unit test coverage is just one necessary step to achieving real test-driven development (TDD). Teams that use TDD require testing and development rigor that helps build quality into the code from the very beginning. Testing truly drives development.
- Pragmatic, usable test automation strategies need to be adopted. Staff testing with highly skilled development resources. Be smart about your automation strategy. For example, don’t automate unstable user stories for regression testing; go beyond GUI automation into API services for business logic and processes for more reuse and less rework.
Focusing just on the last key takeaway, about automation in Agile, here are some recommendations:
- Make sure that your team develops with good design principles. All in all, automation requires better SW design and quality. As one client put it, if you can’t automate, you probably have a design problem.
- Take an architectural approach to automation that encourages modularity and reuse. This will lead to less rework, as changes through your Agile process becomes the norm.
- Get your best developers on Automation or train them to become so. Good-quality SW design requires good quality developers — full stop. Automated test development needs an SDLC process of its own.
- Traditional automation tools alone don’t cut it. open source and smaller innovative tools are making their way. Use them with your flagship testing tools.
- A new category of testing tools, called test virtualization tools, can help Agile deal with complex testing dependencies and support early testing of nonfunctional requirements. Check if you have a good business case to use them; they will help as you scale testing.
Forrester readers can access the full report here. I am wondering what everyone else thinks: Should we include Test in DevOps to make it clearer that testing is a full part and key enabler of the Agile journey? Have you made testing a key area to change in your Agile journey? What successes or challenges have you had? If you don’t want to share this information publicly, email me at firstname.lastname@example.org.
Search Forrester's Blogs
Lead BT Transformation
Develop customer-obsessed strategies to drive growth »
Forrester's CX Index
Predict how actions to improve CX will affect revenue performance.
Measure the customer experiences that matter most »
Forrester's Forum For Technology Leaders
June 2-3, 2015 — Lisbon »
- Anjali Yakkundi (28)
- Art Schoeller (2)
- Boris Evelson (145)
- Claire Schooley (2)
- Clay Richardson (1)
- Diego Lo Giudice (19)
- Gene Cao (1)
- George Lawrie (17)
- Holger Kisker (38)
- Ian Jacobs (7)
- Jeffrey Hammond (27)
- John R. Rymer (45)
- Jost Hoppermann (33)
- Kate Leggett (130)
- Kurt Bittner (4)
- Kyle McNabb (12)
- Margo Visitacion (9)
- Mark Grannan (9)
- Martha Bennett (12)
- Michael Barnes (21)
- Michael Facemire (14)
- Mike Gualtieri (115)
- Noel Yuhanna (10)
- Paul Hamerman (2)
- Phil Murphy (24)
- Philipp Karcher (1)
- Randy Heffner (15)
- Rowan Curran (1)
- Stephen Powers (23)
- Ted Schadler (10)