In a recent blog post that I wrote right before the Agile 2012 conference in Dallas, I expressed my hope that the conference sessions would be full of specifics, not generalities. No, thank you, I don't need another presentation explaining what test-driven development is. I'd like to hear someone talk about how they did test-driven development in a large project, or how they made it a standard practice across projects, or whether anyone has had success with it when working with outsourcing partners.
The presentations at this year's conference were a mixed bag, consisting of the following:
Very high-level discussions, often from consultants, about Agile and Lean concepts. These presentations consisted of a lot of name dropping ("Mumblemumble Daniel Pink mumblemumble Geoffrey Moore mumblemumble Taiichi Ohno") and colorful polygons ("You want to be in the two-by-two Grid Of Delight, not the Overlapping Ovals Of Doom"). Unfortunately, the keynote bore a bit too much resemblance to this category.
Emphatic statements of Agile precepts. Sometimes, the arguments were convincing, such as a good workshop-ish session I attended on Agile planning. Others were less persuasive, such as the contention that Scrum should be the governing methodology in every project. (That's exactly the sort of over-the-top pronouncement that gave birth to the phrase "Agile zealot.")
A common question we analysts hear from our clients is, "How do we scale our Agile efforts?" Now, let's be clear: the question is not how to get Agile to work in a large project. Sure, there are challenges in making Agile work within big teams, but there's a much bigger concern. Organizations invest in Agile, or allow Agile experiments to blossom, and then try to capture the Agile magic in a bottle and mass produce it across the organization.
That's an entirely different challenge, with ambitions and uncertainties that are both Texas-sized. (I'm in Dallas, so I'll use Texas metaphors. So shoot me. Wait, no, I take that back.) The uncertainties have many faces, including (but in no way limited to) issues like:
How many projects or products should employ an Agile approach?
Can we expect our outsourcing partners to use Agile as widely as we do?
Do we need a common tools strategy to support Agile?
How much diversity of Agile approaches within teams is a good thing, and how much is counter-productive?
We've seen a number of misconceptions about Agile come and go. For example, the urban myth that Agile is all about velocity gets far less circulation. More people have seen Agile in practice, witnessing first hand the other potential benefits (more chances for mid-course corrections, greater predictability of outcomes, better business/IT alignment, etc.) than just writing code faster. More people are starting projects with these potential benefits in mind, so Agile has clearly moved past the perception that it was some perverse cult of speed. Speed has no intrinsic business value, aside from keeping a twitchy software developer who has consumed way too much Mountain Dew from chewing his own foot off in frustration about delays and obstacles.
Agile Gets To Specifics Quickly
Business value is the goal for Agile transformation; benefits like quality, predictability, and business/IT alignment are either measures of that value or steps needed to achieve that value. Both topics require a lot of clarity or specificity. Otherwise, Agile can look a lot like a road trip gone horribly wrong: we're not sure where we're going, how close we are, or whether we're going the right way at all.
Agile succeeded brilliantly because it started with some very specific practices and values. Don't end a sprint without working code. Keep your backlog prioritized. Build the expectation of new or changing requirements into planning. Don't build your plans on information you don't have. That's a lot more-specific guidance than something like"Achieve this maturity level and you'll be fine" or "Follow this KPI to the ends of the earth."
Agile Practices Create The Need For Other Practices
Agile is a big topic that goes far beyond a set of practices and principles. Change the development cog in the larger software machine, and you change how other parts of the machine operate, too. It’s a deliberately disruptive change that’s supposed to make the software development and delivery machine become more adaptable, produce a higher quality product, or satisfy some other goal that makes people willing to ride the tiger of disruptive change.
Lean helps sustain Agile while driving it in the direction of value, flow, team empowerment, and waste reduction. Other practices, such as continuous delivery and design-driven development, also complement, reinforce, and supercharge Agile transformation.
As analysts, we might treat these developments as an excuse to write research that amounts, more or less, to a list of what’s hot and what’s not. Are more organizations taking a broader approach to DevOps challenges instead of just focusing on continuous delivery? Are Agile-friendly requirements practices, such as visualization, storyboarding, and serious games just a passing fad or permanent additions to the requirements toolkit? How many organizations have adopted test-driven development, and what are the barriers to adoption?
While these questions are important, and the answers more than a little interesting, they’re insufficient. Forrester is in the business of providing practical advice, not academic musings, so that software professionals can apply innovation strategies such as Agile and Lean immediately to the decisions they have to make today.
One of the core priniciples of Agile is a realistic attitude about the unknown. We might have a rough idea of how much work it will take to complete a project, but we cannot state with the certainty of a papal bull how we're going to get to that destination. Therefore, Agile teams have to embrace Agile principles like loving plans, but abandoning any fetishistic relationship with specific, immutable plans.
Agilists learn to live with uncertainty, but they're far from fatalistic about it. In fact, the opposite is true: the truly good Agile teams assume a very aggressive posture about the management of uncertainty. In this respect, Agile software teams behave a lot like military professionals. First, they accept the inherent unpredictability that they'll face, either on the battlefield or in the backlog. They adopt maxims like, "No plan survives contact with the enemy," or concepts like friction, to describe the nature, sources, and effects of uncertainty. Next, they develop strategies, like the OODA loop (observe, orient, decide, and act), to navigate through the minefields of unexpected outcomes. And finally, they adopt a great deal of rigor and discipline, plus no small amount of self-criticism, to the application of these uncertainty-management practices.
The story of Agile is more than just one chapter in the history of software development. It's also an extremely valuable case study in innovation, an elusive and often humbling process that doesn't work in quite the way that we instinctively think it should.
The trajectory of Agile points toward increasing the value of the software delivered. However, it started with no metric of value and almost no notion of what happened outside the development team. Instead, the first phase of Agile, as I describe in the recent publication "Navigating The Future Of Agile And Lean," focused primarily on changing the behavior and world view of developers working together in a team. Therefore, you won't find anything in Scrum or XP that says, "This is how you know your software is better." Instead, these methodologies told you how to work more effectively as a team. Presumably, better results would follow.
Engineering methodologies have been around for a long time. They've not been noticeable for being terribly successful. They are even less noted for being popular. The most frequent criticism of these methodologies is that they are bureaucratic. There's so much stuff to do to follow the methodology that the whole pace of development slows down.
Revolutions take two forms. The most familiar kind is the noisy, conspicuous, disjunctive event that marks a clean break from the past. Yesterday, George III was our monarch. Today, he's not. The other kind of revolution is a more gradual and subtle event, when multiple forces pointing in the same direction push people into a new world. The shock of Pearl Harbor, the power vacuum left by a devastated Europe and Japan, a reinvigorated economy, and an aggressive superpower adversary made Americans feel, for the first time, that they needed to be far more deeply involved in international affairs than ever before. Without any formal declaration, Americans became internationalists after 1945.
Something like that second kind of revolution has happened with software requirements. Over the past decade or so, organizations grew increasingly worried about the problems that took root in bad requirements. The problems took many forms (portfolios filled with applications no one was using, users unhappy with the software that complicated their lives more than helped them, ideas that no one vetted carefully, etc.) and arose from just as many sources.
All of these discontents pointed in a common direction: Take requirements more seriously. In Forrester's Q1 2011 Application Development And Delivery Organization Structure Online Survey, "improvements of requirements" appeared at the top of the list of initiatives that would improve software development the most.
From: Forrester Analysts Tom Grant and Diego Lo Giudice To: App dev and delivery practitioners, especially ones with Agile experience Re: It’s time for us to take another look at the value adoption, and we’re inviting you to join our survey
For example, Scrum is far and away the most widely adopted flavor of Agile. Scrum focuses on how teams organize themselves and how they organize their work. For teams that have struggled to make accurate estimates or adapt to changes to the backlog, the attraction of Scrum isn’t just velocity.
The boundaries of what we mean by “application life-cycle management” continue to stretch and tear, like Arnold Schwarzenegger stuffed into a toddler’s jumper. While we still have to be careful about defining ALM so broadly that it’s no longer a meaningful category, it’s clear that the traditional list of functionality ― task management, build management, requirements, management, etc., etc.― is at least a couple of sizes too small. In fact, the amount of overlap with product life-cycle management (PLM) is so great that it may be increasingly hard to discuss them separately. They may be surprised to find how closely related they are, like Schwarzenegger and Danny DeVito in Twins, but the connection is definitely there.
Even without PLM tugging at it, ALM is stretching to fit the real development processes it ostensibly manages. As development teams are not indifferent to what happens after they hand off their code to the operations people, ALM has been expanding to include more elements of release and deployment. ALM can’t accommodate everything ops-related without ripping apart at the seams, but it does need some alterations.
PLM is a whole different consideration. Rather than expanding the definition of ALM, it adds another layer on top of it ― primarily to accommodate the realities of embedding software in other products (cars, refrigerators, medical devices, etc.). Because the number of these hybrid hardware/software products expands daily, the urgency of figuring out how ALM and PLM fit together as part of a common ensemble has been increasing.
Complexity is the nemesis of application developers everywhere. While software products' complexity may be, to a great extent, unavoidable, we don't have to complicate software development to match it.
That's the conclusion driving many of the new approaches that app dev teams are embracing. For example, Agile breaks down humongous, complex projects into incremental deliverables that are far easier to scope, build, test, and deliver. Lean practices simplify processes, making it easier for teams to achieve a state of flow. DevOps advocates are trying to eliminate the needless complexities that result from throwing software over the wall from one team to the next.
But there's more to Agile, Lean, and DevOps than just a common enemy. It's no accident that practitioners often speak of this trio in practically the same breath. The Kanban board is the symbol of this convergence of practices. Here's an increasingly common use case: an Agile teams uses a Kanban board to manage work with other groups, including the DevOps team.
The evidence for this convergence is more than just anecdotal. When we survey app dev professionals, they report that they're mixing methodologies all the time, including Agile and Waterfall. For a variety of reasons, Agile teams have made the adjustments necessary to accommodate Waterfall. Not every planning activity at the beginning of a project can be handled in a strictly Agile fashion, and not everyone involved at the end of the project (release management, operations, and business users) is necessarily sold on the idea of rapid, continuous delivery. If you work in a regulated environment, you're required to take extra steps at the beginning and end of a project.