Agile Teaches Us An Important Lesson About Innovation

The story of Agile is more than just one chapter in the history of software development. It's also an extremely valuable case study in innovation, an elusive and often humbling process that doesn't work in quite the way that we instinctively think it should.

The trajectory of Agile points toward increasing the value of the software delivered. However, it started with no metric of value and almost no notion of what happened outside the development team. Instead, the first phase of Agile, as I describe in the recent publication "Navigating The Future Of Agile And Lean," focused primarily on changing the behavior and world view of developers working together in a team. Therefore, you won't find anything in Scrum or XP that says, "This is how you know your software is better." Instead, these methodologies told you how to work more effectively as a team. Presumably, better results would follow.

In fact, the framers of the Agile Manifesto sought to change the whole notion of methodology, according to Martin Fowler:

Engineering methodologies have been around for a long time. They've not been noticeable for being terribly successful. They are even less noted for being popular. The most frequent criticism of these methodologies is that they are bureaucratic. There's so much stuff to do to follow the methodology that the whole pace of development slows down.

Agile methodologies developed as a reaction to these methodologies. For many people the appeal of these agile methodologies is their reaction to the bureaucracy of the engineering methodologies. These new methods attempt a useful compromise between no process and too much process, providing just enough process to gain a reasonable payoff.
The Agile movement started with big ambitions. So why then, after the foundational meeting in Snowbird, did Fowler, Schwaber, Highsmith, et al. then dive into the procedural weeds? What does something like daily stand-ups have to do with changing the concept of a software development methodology? What's the connection between continuous integration and increased value from the standpoint of the customer using the software?
 
Fowler gives part of the answer in that quote. Most methodologies, up to that point, assumed that to get to the desired outcome, you had to take into account everything that would ultimately lead to that outcome. The journey of a thousand miles might start with a single step, but these methodologies tried to tell you what the next step would be, and the next, and the next after that, not just for you the developer, but for everyone involved in software development and delivery. 
 
Fine intentions aside, the results were godawful. Checklists that developers hated to follow and managers hated to enforce. Only a small cadre of the initiated with the certification to prove they could understand Byzantine methodological logic that no one else could follow.
 
And, worst of all, these methodologies allowed practically no room to accommodate changes in the world. In 2001, the year of the Agile Manifesto, software development was very different than it is today. Cloud computing was in its infancy (or maybe even a fetal state). The idea of doing real work through real applications on your phone seemed ludicrous. Social computing was barely out of its BBS phase. And so much more changed, creating new ways to approach software development and delivery and new expectations of what would emerge from it.
 
Therefore, the real genius of Agile is that it started small. It didn't try to chart the entire journey from software development in 2001 to the terra incognita of 2012. It didn't try to change the way everyone in the software value chain worked all at once. Instead, it chose a critical point in that value chain, the development team, and made a highly disruptive change to it. Eventually, people would figure out the next steps that would ultimately lead to better software as delivered (not just as built, as proponents of the DevOps movement will remind us).

Regardless of what sort of innovation we're attempting, we should heed this example. No one is smart enough to figure out every aspect of disruptive innovation from beginning to end. Innovation is a journey, not of a thousand steps, but into an unfamiliar landscape, full of potholes, wrong turns, and dead ends. It's important to start with something that compels you to make the journey at all, and to navigate successfully through the unknown. 

You can see that innovation strategy today in the issues Agilists are talking about. We've moved well past the developer-focused topics, such as sprint planning and test-driven development. Now, the Agile community is grappling with other parts of the value chain beyond the development team. How do we better incorporate customer feedback? How do we collaborate better with UX designers? How can we release as quickly as we build? If there are business imperatives beyond just building software more effectively, how do we alter the Agile timeline, creating something like water-Scrum-fall instead of an endless series of sprints? How do we incorporate Lean to ensure that we're all moving in the right direction?

These are the questions that no one could have answered in 2001, or even in 2006. It took a long time to hone core Agile practices, get a lot of teams to adopt Agile, and then watch the ripple effects. Only then could we get into these sorts of discussions about improving software value.

Comments

Great 2012 Agile Motto

Hi, Tom, good 2012 agile motto, compared to more than a decade ago, when Agile Manifesto was written, today's agile does have much more agile eco-system: the cloud, the social, mobile, dynamic BPM., to create more dynamic working environment and diversified business culture, so hopefully agile as second class methodology, compare to waterfall in last decade, can become mainstream to help improve project success rate and even rejuvenate the organizational culture via its continuous iteration and communication nature. thanks.

Innovation is inevitable for an agile team

Tom, you have brought up a very valid subject.

Innovation is inevitable for any agile team. We have seen the period where technology was very close to business and they were literally face-face while developing mission critical applications. Then we invented processes and moved the development away from business to have sanity and stability. The innovation was more towards processes than the product itself.

Over the last few years we discovered that due to heavy processes, the product quality is suffering and engineering in 'software engineering' has lost its meaning.

Now is a phase where we are re-discovering the 'engineering, and we are sort of 'going back to basics'. here what the agilists are trying to do on a daily basis through innovation is to see how closer we can be to business and how to enable the business, but in a risk-free and reliable fashion, still holding on to high-tech practices.

I have put together a checklist of 24 assessment criterias where one can assess the agility quotient of one's team: http://people10.com/blog/2012/02/19/are-you-really-agile-a-free-assessme...

I am really excited about total agility where we are now able to have continuous deployments and push button releases and the array of open source tools that enable our developers to experience disruptive innovation in an agile enabled world!

Great article, thanks!

Nisha Shoukath
Co-Founder & COO, People10
www.people10.com