The Rise And Fall Of Complexity: The Development Process

[For earlier posts in this series, click here, here, and here.]

Imagine yourself back in childhood, sitting in the back seat of the family station wagon, en route to one of those long, high-stress family vacations that Americans have honed to perfection. Mom and Dad are arguing over what went wrong so far on the trip, and of course, how they could have avoided these mishaps. Why didn't we ask for directions at the last gas station, instead of letting Dad navigate by dead reckoning? Should someone have called ahead to ensure that there wasn't a problem with the hotel reservations? Who was supposed to remember to pack the camera? Was it reasonable to expect a travel rate of 500 miles a day? Quickly, your vision of vacation as a straight line into the heart of fun evaporates. Replacing it is a circuitous flowchart, each serpentine twist representing a different set of decision points, estimates, and possible outcomes. Who knows what might happen next? The wheels will fall off on a desert highway, and someone will have forgotten to renew the AAA membership?

Moving quickly from point A to point B is always desirable. However, even if you replaced the station wagon with a souped-up sports car, the road ahead may still be full of potholes. If you plan your vacation so that you have just enough time, and just enough money, to get to Wally World and back, chances are that something will go wrong, wrecking those plans. The longer the journey, the more opportunities for hitting one of those potholes. (By the way, the power that misfortune has to wreck complex plans is well known and even has a very evocative term: friction.)

Enforced simplicity is one of the chief attractions of Agile, as we've discovered from our surveys of people who have adopted it. During a four-week sprint, you'll run into at least a few minor problems. Maybe you'll run into unexpected performance bottlenecks, or the sainted Voice Of The Customer says that your UI stinks. As unattractive as these possibilities may be, there will be fewer of them in a four-week sprint than a year-long release cycle. By keeping iterations short, Agile reduces the complexity of the final deliverable. The simpler the deliverable, the fewer elements that can go wrong, which has the added benefit of making plans and estimates more accurate.

Adaptability is a crucial benefit of Agile across multiple iterations, not just within a single iteration. A software vendor might still release a major version of a product only once per year. Under the rules of Waterfall, the closer you get to the release date, the more unexpected problems appear, jeopardizing the release date. Eventually, getting the release out the door can take precedence over all issues. Unfortunately, the cost of failing to respond to some of those issues – shifting customer priorities, new competitive threats, poor usability, etc. – can be quite high. Nevertheless, the development engine, straining as it drags its heavy workload forward through months of development, must reach the finish line.

Adaptability isn't the only benefit, beyond velocity, that Agile teams value. Across multiple surveys, Agilists have cited adaptability, predictability, quality, and business/IT alignment as important as speed, and in some cases, more so. Reducing the complexity of the deliverable, by pruning the number of projects down to what can be accomplished within a short sprint, changes the entire development process.

Reducing the size of each iteration has its potential downsides. Long pole projects, compliance documentation, the pressures of constant activity, deep architectural questions – these are just a few of the challenges that newly Agile teams often address. However, this is a trade-off that, if successful with Agile, they'd gladly make. They've been trapped in the back seat of the development process as the grown-ups argued over a plan that was too way complex to succeed.

Comments

Cobblers

What a load of old cobblers. Sorry, for saying it, I hate flamers but, that is just what I think. Here is why:

"
Enforced simplicity is one of the chief attractions of Agile, as we've discovered from our surveys of people who have adopted it.
"

This sentence reads that people heard agile enforces simplicity and so people adopted it. It in no way demonstrates that Agile does enforce simplicity. I would suggest it does 100% the reverse. Architecture enforces simplicity. Look at the structure of a modern sky-scraper and compare that to a medieval cathedral. The former was designed by an architect and the latter was 'just built' by skilled craftsmen.

You actually point out how Agile has problems with architecture. So, in effect Agile bakes in complexity then hides it.

I have seen the product of this effect over and over again. There are three phases:

1) What happens is that the skilled craftsmen and architects forced to work as craftsmen build lots of incremental steps without being able/permitted to look at the big picture. Velocity shoots up and agile is held up as a wonder paradigm.
2) The technical debt in the system caused by lack of architecture and inability to take on large, non decomposable stories reduces productivity.
3) The teams get the something kicked out of them for not producing quickly enough. 'Why is everything taking so long. Why cannot we get back to the rate we managed last year?'

Short sprints and agile teams have their place - BUT - the current trend is that they are the way to develop. That is just like saying bananas are _the_ thing to eat rather than _a_ thing to eat. The latter is good for your health; the former will land you in a pile of that thing referred to above.

Best wishes - AJ

Dr Alexander J Turner

Fair points...

I'd agree with you, except that you've overmade your point. There's a difference between the risk of making mistakes, and actually making them.

"1) What happens is that the skilled craftsmen and architects forced to work as craftsmen build lots of incremental steps without being able/permitted to look at the big picture. Velocity shoots up and agile is held up as a wonder paradigm."

You're right, the backlog is not be a good way of understanding the big picture. In fact, the backlog is a byproduct of the big picture. Often, strict prioritization of the backlog is difficult because the big picture, the ultimate yardstick of what's important, isn't clear to the team members doing the prioritization.

So where's the big picture, in Agile? The answer is not always clear. Scrum identifies the product owner as the keeper of the product vision. In practice, it's often unreasonable to expect the product owner to be deep in the details of development, while still distant enough from the process to fully understand what stakeholders want. Rich Mironov had a great presentation about the unrealistic expectation that product owners can also do the heavy lifting that product managers do.

"2) The technical debt in the system caused by lack of architecture and inability to take on large, non decomposable stories reduces productivity."

Unfortunately, the necessary response to this issue is way bigger than I can stuff into a comment. Heck, it took Scott Ambler an incredibly long blog post to address this question.

http://www.agilemodeling.com/essays/agileArchitecture.htm

It wouldn't take a big post to address if Agilists had the architectural challenges tidily sewn up. I'll agree, this is one place where Agile imposes some risks -- but again, risk is not inevitability.

"3) The teams get the something kicked out of them for not producing quickly enough. 'Why is everything taking so long. Why cannot we get back to the rate we managed last year?'"

I've definitely heard this one before, and the source is usually someone in the management layers above the team. An executive at an ISV may have a one-word understanding of Agile: velocity. However, that's not the only benefit of being Agile, as the survey data I cited indicates. Whether the team oversold the benefits of velocity, or the executive just hasn't taken the time to understand how Agile works, there's a gap of understanding between those doing Agile and those sponsoring it. But not every organization has this problem, and there are ways to minimize the risk. (Which is why there have been a lot of presentations about "Agile for executives" at recent Agile conferences.) There has also been the reverse problem, delivering faster than stakeholders want.

Agile is not a "wonder paradigm." People can do Agile badly, just as they can do any methodology badly. I've made that point to overzealous Agile advocates who did think it was a wonder paradigm. I don't. What I have seen, however, is a principle that extends beyond software development: the more complex the plan, whether it's a major software release or the invasion of Midway, the more likely it will fall victim to "friction."

Have to agree with ...

Tom that you overstate the point, and I'm not sure that the breaking up development into short sprints makes it any harder to anticipate the performance bottlenecks or get the UI right. I concur with Paul Czarnik in his blog post (http://blog.compuware.com/post/apm-coding-for-performance) that coding for performance has to happen from the outset and that "old school" programming sensibilities can be incorporated into the most modern frameworks.

More Fair Points

Tom,

Thanks for the well balanced response. I don't strongly disagree with you on your points. However, I would like to reflect on what you have said.

"
Unfortunately, the necessary response to this issue is way bigger than I can stuff into a comment. Heck, it took Scott Ambler an incredibly long blog post to address this question.
"

Then

"
What I have seen, however, is a principle that extends beyond software development: the more complex the plan, whether it's a major software release or the invasion of Midway, the more likely it will fall victim to "friction."
"

It seems to me that for architecturally complex systems (unlike flattish websites - for which agile was invented) the amount of complexity needed to fix up agile to be fit for purpose is so great the approach will almost certainly fall victim to "friction".

If I was building a web site which had a simple, flat, structure with lots of little features then I would go for agile without a second thought. For a complex architectural product, say and financial trading system, or a compiler, I see agile as a square peg in a round whole. Why force it in with all these complex patches and working practices. Why not just accept that agile is good for something and not others.

I am quite happy to accept relativity is good for big things and quantum mechanics for small things. One day we may well figure out a way to join them up. But, for now, we use both. Why cannot we adopt and equally balanced view to software development?

In Freudian terms, what the agile advocates are doing is being paranoid-schizoid. Splitting the world into good (agile) and bad (everything else). The mental state we normally associate with children. I suspect we need to grow up and accept nothing is good for everything and nothing is back and white.

Please note - that I am not accusing anyone of being truly childish. However, sometimes the pro-agile community (not dissimilarly to the pro-functional community) do fall into child like philosophical stances.

In closing, this is a great blog and it is just fabulous that you have taken the time to respond to my post - thanks.

Best wishes - AJ