Early this year, on January 15, I published our first research on testing for the Agile and Lean playbook. Connected to that research, my colleague Margo Visitacion and I also published a self-assessment testing toolkit. The toolkit helps app-dev and testing leaders understand how mature their current testing practices and organization are for Agile and Lean development.
The Agile Testing Self-Assessment Toolkit
So what are the necessary elements to assess Agile testing maturity? Looking to compromise between simplicity and comprehensiveness, we focused on the following:
Testing team behavior. Some of the questions we ask here look at collaboration around testing among all roles in the Scrum teams. We also ask about unit testing: Is it a mandatory task for developers? Are all of the repeititive tests that can be run over and over at each regression testing automated?
Organization. In our earlier Agile testing research, we noticed a change in the way testing gets organized when Agile is being adopted. So here we look at the role test managers are playing: Are they focusing more on being coaches and change agents to accelerate adoption of the new Agile testing practices? Or are managers still operating in a command-and-control regime? Is the number of manual testers decreasing? Are testing centers of excellence (TCOEs) shifting to become testing practice centers of excellence (TPCOEs)?
DevOps is a movement for developers and operations professionals that encourages more collaboration and release automation. Why? To keep up with the faster application delivery pace of Agile. In fact, with Agile, as development teams deliver faster and in shorter cycles, IT operations finds itself unprepared to keep up with the new pace. For operations teams, managing a continuous stream of software delivery with traditional manual-based processes is Mission Impossible. Vendors have responded to DevOps requirements with more automation in their release management, delivery, and deployment tools. However, there is a key process that sits between development and operations that seems to have been given little attention: testing.
In fact, some key testing activities, like integration testing and end-to-end performance testing, are caught right in the middle of the handover process between development and operations. In the Agile and Lean playbook, I’ve dedicated my latest research precisely to Agile testing, because I’ve seen testing as the black beast in many transformations to Agile because it was initially ignored.
I just finished analyzing our Q3 2012 Global State Of Enterprise Architecture Online Survey, where we asked a number of questions at the end of the survey on how firms identify and introduce new technology – new technology that your firm is counting on for innovation and competitive advantage. The results underscore a conviction that is growing in me: IT’s “one-size-fits-all” approach to standardizing everything and general aversion to risk isn't cutting the mustard. Simply put, opportunities for competitive advantage through technology-fueled disruption get missed, and this means digital extinction. Some data from our survey of 207 enterprise architects:
58% reported that sales and marketing is among the top five most likely organizations to deliver technology innovations, and they are chasing windows of opportunity that close in months. IT typically takes at least a year to do anything.
52% say there is at least some business dissatisfaction with the level of new technology introduction. The top reason, given by 78% of respondents, is that IT is too slow.
70% of respondents admit their firms have trouble reacting to disruptions caused by emerging technology, and 60% admit to difficulty reacting to change in general.
Agile Practices Create The Need For Other Practices
Agile is a big topic that goes far beyond a set of practices and principles. Change the development cog in the larger software machine, and you change how other parts of the machine operate, too. It’s a deliberately disruptive change that’s supposed to make the software development and delivery machine become more adaptable, produce a higher quality product, or satisfy some other goal that makes people willing to ride the tiger of disruptive change.
Lean helps sustain Agile while driving it in the direction of value, flow, team empowerment, and waste reduction. Other practices, such as continuous delivery and design-driven development, also complement, reinforce, and supercharge Agile transformation.
As analysts, we might treat these developments as an excuse to write research that amounts, more or less, to a list of what’s hot and what’s not. Are more organizations taking a broader approach to DevOps challenges instead of just focusing on continuous delivery? Are Agile-friendly requirements practices, such as visualization, storyboarding, and serious games just a passing fad or permanent additions to the requirements toolkit? How many organizations have adopted test-driven development, and what are the barriers to adoption?
While these questions are important, and the answers more than a little interesting, they’re insufficient. Forrester is in the business of providing practical advice, not academic musings, so that software professionals can apply innovation strategies such as Agile and Lean immediately to the decisions they have to make today.
One of the core priniciples of Agile is a realistic attitude about the unknown. We might have a rough idea of how much work it will take to complete a project, but we cannot state with the certainty of a papal bull how we're going to get to that destination. Therefore, Agile teams have to embrace Agile principles like loving plans, but abandoning any fetishistic relationship with specific, immutable plans.
Agilists learn to live with uncertainty, but they're far from fatalistic about it. In fact, the opposite is true: the truly good Agile teams assume a very aggressive posture about the management of uncertainty. In this respect, Agile software teams behave a lot like military professionals. First, they accept the inherent unpredictability that they'll face, either on the battlefield or in the backlog. They adopt maxims like, "No plan survives contact with the enemy," or concepts like friction, to describe the nature, sources, and effects of uncertainty. Next, they develop strategies, like the OODA loop (observe, orient, decide, and act), to navigate through the minefields of unexpected outcomes. And finally, they adopt a great deal of rigor and discipline, plus no small amount of self-criticism, to the application of these uncertainty-management practices.
As Forrester has pointed out in past research, to move forward in a product-centric way, you must establish a number of capabilities in your organization in a product-centric operating model:
I used to make a living doing product management and running product management organizations, as did a number of people at Forrester. My perspective on product management is somewhat distinct because I started out on the product engineering side, leading product development organizations. But between my first ISV and my second, I saw the contrast between weak, ineffective product management and strong, effective product management. When strong product management is in balance with an effective engineering group, well aligned, it’s a beautiful thing to see. The weak option? You don’t want to know – it was too painful to remember.
Is it hard to focus your software delivery organization on the right things? Do you sometimes deliver the wrong features or give too little priority to the most important features? Are you drowning in the cost of too much redundant software, because stakeholders can’t get on one page about what the business really needs? Do you struggle to make the case for investments you know are essential to your long-term survival but that deliver few short-term benefits? If so, consider the benefits of running your shop more like a business by reorganizing to deliver products (or value streams, in Lean lingo).
It’s been more than two years since we last surveyed software delivery leaders about their increasing tendency to organize to deliver software as products (rather than projects or application functional areas), but even then this trend was well under way:
I have only anecdotal evidence that this trend is continuing to grow, but I’m convinced by hundreds of interactions with top software delivery leaders since we did this survey that it is, especially for people who deliver customer-facing websites and mobile apps. Customer-facing dynamics are also driving this trend for the “Internet of things” among firms focused on Smart Grid and other similar domains that depend on customer adoption to drive success. What are the factors driving this growth?
There is no doubt that Agile growth in the market is significant, and the growing daily number of inquiries I’ve been getting on Agile from end user organizations in 2012 gives me the impression that many are moving from tactical to strategic adoption. Why’s that? Many reasons, and you can read about them in our focused research on Agile transformation on the Forrester website. But I’d like to summarize the top five reasons from my recent research “Determine The Business And IT Impact Of Agile Development” :
Quality was the top — quite astonishing, but both the survey we ran across 205 Agile “professional adopters” and the interviews across some 21 organizations confirmed this. My read is that this is about functional quality.
Change was second to quality. We live in an era where innovation strives and organizations are continuously developing new apps and projects. But your business does not necessarily know what it needs or wants upfront. The business really appreciates the due-course changes that Agile development allows, as they enable the business to experiment and try out various options so it can become more confident about what is really right for the organization. Cutting edge cutting edge systems-of-engagement (Mobile, Web-facing, Social-media, etc) require lots of Change in due course.
Think of a medieval fortress: It was originally used for a small army, it has walls nine meters thick, and it’s surrounded by buildings hundreds of years old. Upon entering, you are confronted with the concept of eternity.
This fortress is located in the smallest state on earth — though it is also perhaps the best-known state in the world. The business housed within the fortress is what many might classify as a SME but with with complexity of a large enterprise, holy but busy, centralized but truly global — its work spans hundreds of countries with hundreds of currencies and hundreds of languages — and it serves very special and demanding clients.
Have a clue yet of where we are?
Zoom on Italy, then zoom on Rome, then zoom on Vatican City, and you can’t miss the round tower (Torrione Sisto V) where the Vatican Bank, or Istituto per le Opere di Religione (IOR ), is located. You won’t be allowed in if you are not a client, an employee, or part of a religious congregation. Change comes hard to institutions this steeped in tradition. To give you a clue, IOR’s previous managing director spent his entire career at IOR — 60 years — and retired at the age of 80. We all know it’s the soft and cultural aspects of transformation that are the hardest part for any organization.
Nevertheless, IOR has been going through a major change since 2008, working to replace its legacy IT system with a modern BT one. The new BT system brings more flexibility for the business, richer business functionality, and greater integration and development capabilities. Enabling fast change is the key driver for IOR’s IT transformation program from IT into BT.
Two weeks have passed since our successful AD&D and BP Forums in Boston. I’m still struck by conversations we held there and continue to hold now with many of you on how your teams can help deliver to your firm’s ever-important customer experience outcomes. Following one tip can help you either get ahead of this issue or catch up to the expectations of your stakeholders…act more like an interactive agency!
Note I didn’t say “transform” into an interactive agency. No, at the end of the day you have responsibilities to your organization the agencies your business peers use often don’t – you have to manage, operate, and maintain what’s been delivered. What I did say was “act” like one, and in doing so you’ll need to:
Revisit your talent. For those of you that haven’t outsourced big portions of development, make sure you have great, creative developers, build a high-performance development team, and up-skill your business analysts by putting personas and customer journey maps into their tool kit. Why? The agencies your peers use have and cultivate these skills. At minimum, you'll be in a better position to manage and maintain what they’ve put in place if you have complementary skills of your own. If you have outsourced development, we can help you make the case to bring back the right pieces.