Early this year, on January 15, I published our first research on testing for the Agile and Lean playbook. Connected to that research, my colleague Margo Visitacion and I also published a self-assessment testing toolkit. The toolkit helps app-dev and testing leaders understand how mature their current testing practices and organization are for Agile and Lean development.
The Agile Testing Self-Assessment Toolkit
So what are the necessary elements to assess Agile testing maturity? Looking to compromise between simplicity and comprehensiveness, we focused on the following:
Testing team behavior. Some of the questions we ask here look at collaboration around testing among all roles in the Scrum teams. We also ask about unit testing: Is it a mandatory task for developers? Are all of the repeititive tests that can be run over and over at each regression testing automated?
Organization. In our earlier Agile testing research, we noticed a change in the way testing gets organized when Agile is being adopted. So here we look at the role test managers are playing: Are they focusing more on being coaches and change agents to accelerate adoption of the new Agile testing practices? Or are managers still operating in a command-and-control regime? Is the number of manual testers decreasing? Are testing centers of excellence (TCOEs) shifting to become testing practice centers of excellence (TPCOEs)?
DevOps is a movement for developers and operations professionals that encourages more collaboration and release automation. Why? To keep up with the faster application delivery pace of Agile. In fact, with Agile, as development teams deliver faster and in shorter cycles, IT operations finds itself unprepared to keep up with the new pace. For operations teams, managing a continuous stream of software delivery with traditional manual-based processes is Mission Impossible. Vendors have responded to DevOps requirements with more automation in their release management, delivery, and deployment tools. However, there is a key process that sits between development and operations that seems to have been given little attention: testing.
In fact, some key testing activities, like integration testing and end-to-end performance testing, are caught right in the middle of the handover process between development and operations. In the Agile and Lean playbook, I’ve dedicated my latest research precisely to Agile testing, because I’ve seen testing as the black beast in many transformations to Agile because it was initially ignored.
One-Size Software Development Methodologies Do Not Fit All
Dozens of software development methodologies exist, from waterfall to Agile to pure anarchy (Agile has always rubbed me wrong). Mark Kennaley speaks the truth when he says that “there is no ‘best’; there’s only contextual fitness for purpose.” Mark is the founder of Software Development Experts, a software development methodology historian, a consultant, and the creator of an expert system that helps organizations determine the best software methodology to use based on 10 factors: development team size, domain complexity, technical complexity, the geographical dispersion of the development team, time-to-market pressure, enterprise specialization, contract relationships, compliance, criticality, and culture. This makes perfect sense, and so does Mark. Unfortunately, entrenched dogma and high ceremony can obscure what really matters.
Composite, Dynamic Software Development Methodologies Are Best
TechnoPolitics speaks with Mark about how firms can choose the best methodology based on the 10 factors that matter. One size does not fit all. Listen to find out why and how to move forward.
Podcast: One-Size Software Development Methodologies Do Not Fit All
Agile and Lean transformations depend to a great extent on cultivating a good sourcing ecosystem. The decisions you make around the partners and providers supporting your transformation and projects will be at the core of a successful strategy. But sourcing strategy needs to go beyond just resource or services providers (read outsourcing) and address a larger ecosystem made of Agile SW development and delivery choices, collaboration and communication capabilities for distributed teams, and teams' physical work spaces, standard equipment, and office layout.
In September, I published a report on how to source your Agile strategy, that describes what the ecosystem looks like and how to navigate it effectively, the document is part of our larger research container on Agile - The Agile and Lean Playbook. The report gives an overview on how large vendors, SIs, and medium to small consulting organizations can (not) help you with your Agile journey but also what you need to do to be successful. Here are some of the takeaways from the research:
What you think about Agile and Lean might not be what your SI thinks. You need to take control of your own destiny with Agile and Lean. Change your application development and delivery sourcing strategy to embed the best talents around the world to help you make it happen. But be careful with the traditional SIs, because Agile is as disruptive to them as it is you, and if they have not been seriously transforming themselves, it will be hard for them to deliver Agile services to you. Some good alternative new fully Agile players exist, including highly specialized external consulting firms. You might want to start testing the ground with these options.
I just finished analyzing our Q3 2012 Global State Of Enterprise Architecture Online Survey, where we asked a number of questions at the end of the survey on how firms identify and introduce new technology – new technology that your firm is counting on for innovation and competitive advantage. The results underscore a conviction that is growing in me: IT’s “one-size-fits-all” approach to standardizing everything and general aversion to risk isn't cutting the mustard. Simply put, opportunities for competitive advantage through technology-fueled disruption get missed, and this means digital extinction. Some data from our survey of 207 enterprise architects:
58% reported that sales and marketing is among the top five most likely organizations to deliver technology innovations, and they are chasing windows of opportunity that close in months. IT typically takes at least a year to do anything.
52% say there is at least some business dissatisfaction with the level of new technology introduction. The top reason, given by 78% of respondents, is that IT is too slow.
70% of respondents admit their firms have trouble reacting to disruptions caused by emerging technology, and 60% admit to difficulty reacting to change in general.
In a recent blog post that I wrote right before the Agile 2012 conference in Dallas, I expressed my hope that the conference sessions would be full of specifics, not generalities. No, thank you, I don't need another presentation explaining what test-driven development is. I'd like to hear someone talk about how they did test-driven development in a large project, or how they made it a standard practice across projects, or whether anyone has had success with it when working with outsourcing partners.
The presentations at this year's conference were a mixed bag, consisting of the following:
Very high-level discussions, often from consultants, about Agile and Lean concepts. These presentations consisted of a lot of name dropping ("Mumblemumble Daniel Pink mumblemumble Geoffrey Moore mumblemumble Taiichi Ohno") and colorful polygons ("You want to be in the two-by-two Grid Of Delight, not the Overlapping Ovals Of Doom"). Unfortunately, the keynote bore a bit too much resemblance to this category.
Emphatic statements of Agile precepts. Sometimes, the arguments were convincing, such as a good workshop-ish session I attended on Agile planning. Others were less persuasive, such as the contention that Scrum should be the governing methodology in every project. (That's exactly the sort of over-the-top pronouncement that gave birth to the phrase "Agile zealot.")
We've seen a number of misconceptions about Agile come and go. For example, the urban myth that Agile is all about velocity gets far less circulation. More people have seen Agile in practice, witnessing first hand the other potential benefits (more chances for mid-course corrections, greater predictability of outcomes, better business/IT alignment, etc.) than just writing code faster. More people are starting projects with these potential benefits in mind, so Agile has clearly moved past the perception that it was some perverse cult of speed. Speed has no intrinsic business value, aside from keeping a twitchy software developer who has consumed way too much Mountain Dew from chewing his own foot off in frustration about delays and obstacles.
Agile Gets To Specifics Quickly
Business value is the goal for Agile transformation; benefits like quality, predictability, and business/IT alignment are either measures of that value or steps needed to achieve that value. Both topics require a lot of clarity or specificity. Otherwise, Agile can look a lot like a road trip gone horribly wrong: we're not sure where we're going, how close we are, or whether we're going the right way at all.
Agile succeeded brilliantly because it started with some very specific practices and values. Don't end a sprint without working code. Keep your backlog prioritized. Build the expectation of new or changing requirements into planning. Don't build your plans on information you don't have. That's a lot more-specific guidance than something like"Achieve this maturity level and you'll be fine" or "Follow this KPI to the ends of the earth."
Agile Practices Create The Need For Other Practices
Agile is a big topic that goes far beyond a set of practices and principles. Change the development cog in the larger software machine, and you change how other parts of the machine operate, too. It’s a deliberately disruptive change that’s supposed to make the software development and delivery machine become more adaptable, produce a higher quality product, or satisfy some other goal that makes people willing to ride the tiger of disruptive change.
Lean helps sustain Agile while driving it in the direction of value, flow, team empowerment, and waste reduction. Other practices, such as continuous delivery and design-driven development, also complement, reinforce, and supercharge Agile transformation.
As analysts, we might treat these developments as an excuse to write research that amounts, more or less, to a list of what’s hot and what’s not. Are more organizations taking a broader approach to DevOps challenges instead of just focusing on continuous delivery? Are Agile-friendly requirements practices, such as visualization, storyboarding, and serious games just a passing fad or permanent additions to the requirements toolkit? How many organizations have adopted test-driven development, and what are the barriers to adoption?
While these questions are important, and the answers more than a little interesting, they’re insufficient. Forrester is in the business of providing practical advice, not academic musings, so that software professionals can apply innovation strategies such as Agile and Lean immediately to the decisions they have to make today.
Last December, I published three things I'd tell your CIO. Since then, I've spent time with dozens and dozens of sourcing and vendor management professionals, CIOs, and leaders of application development and delivery, including last week's Paris Forrester Forums. Most days, I share our ongoing research on what impact today's software-fueled, consumer-led digital disruption has on your ability to meet and exceed the expectations of your customers and the employees serving them. For some folks, software and software development remains a commodity. But for many, the need to deliver great software has taken hold of 2013 planning discussions. With July just around the corner, and as you start 2013 planning, focus on what you need to start delivering great software (remember, software is your business), and keep these three things I'd tell you and your CIO in mind:
One of the core priniciples of Agile is a realistic attitude about the unknown. We might have a rough idea of how much work it will take to complete a project, but we cannot state with the certainty of a papal bull how we're going to get to that destination. Therefore, Agile teams have to embrace Agile principles like loving plans, but abandoning any fetishistic relationship with specific, immutable plans.
Agilists learn to live with uncertainty, but they're far from fatalistic about it. In fact, the opposite is true: the truly good Agile teams assume a very aggressive posture about the management of uncertainty. In this respect, Agile software teams behave a lot like military professionals. First, they accept the inherent unpredictability that they'll face, either on the battlefield or in the backlog. They adopt maxims like, "No plan survives contact with the enemy," or concepts like friction, to describe the nature, sources, and effects of uncertainty. Next, they develop strategies, like the OODA loop (observe, orient, decide, and act), to navigate through the minefields of unexpected outcomes. And finally, they adopt a great deal of rigor and discipline, plus no small amount of self-criticism, to the application of these uncertainty-management practices.