Agile Software Is A Cop-Out; Here’s What’s Next

Never has a new trend annoyed me as much as Agile. Right from the get-go, the Agile Manifesto revealed the weaknesses and immaturity of the founding principles. The two most disturbing: “Working software is the primary measure of progress” and “Business people and developers must work together daily throughout the project.” These are cop-outs because:

  • Using “working software as the measure of progress” is narcissistic. Rush to write code is oft the translation of this misguided principle. Software developers want to be measured on what they know — code — instead of what they don’t know: the domain, business requirements, and the user. The idea is that you’ve got nothing if you don’t have working software. The cop-out is thinking that code is the measure of everything. Software is always part of a bigger picture. It works in conjunction directly or indirectly with other software, technologies, the environment, business processes, and people to create customer experience and/or improve operational efficiency. Like most of the Agile principles, this one is developer-centric rather than user-centric.
  • “Business people and developers working together daily” is lazy. This principle is a capitulation by developers who say it is impossible to understand the true requirements. Developers would much rather stick with what they know: typing class definitions, if-then-else statements, and looping constructs into glorified text editors. The absurd Agile solution to getting the requirements right is to meet with the business every day (or frequently). Often the business people who are involved are not the same as the users, and they are not domain experts or design experts. Seriously, can your best people be available to help developers do their job? The cop-out is thinking that great software can be developed through a process of dead-reckoning with business people. This is just another way for software developers to blame the business for challenged or failed software projects and a sure-fire way to propagate uncertainty and mediocrity. It is a false argument to say that developers cannot understand the requirements and anticipate changes. Development teams must understand the business and the users better than they understand themselves.

Yikes. I have used some strong words here, and I could go on and on. I acknowledge the good intentions of the Agile community. But, the lack of empirical evidence for the benefits of Agile methods is telling. My colleague Dave West, who covers Agile for Forrester, has seen the cracks in how Agile is implemented and written about them in a new research paper titled “Water-Scrum-Fall Is The Reality Of Agile For Most Organizations Today. The report also shows Agile adoption at 38.6% in 2010 and notes that interest in Agile remains very, very high. Most of the evidence I have seen is anecdotal — of the “It’s a miracle. I can walk again!” kind. My take is that the Hawthorne effect is hard at work after years of brain-numbing sequential processes such as waterfall. Anything was better than the worst.   

A New Thesis For A Better Approach To Software Development
For the future of application development to be insanely great user experiences, we need a new software development methodology. Agile is not it. Agile is a group of methodologies that came from asking, “How can we fix the software development mistakes of the past?” We need a new approach that asks, “What software development process prepares us to be experience creators?” I call this new approach: parallel, immersive software studio (I am well aware of the purely coincidental acronym P#SS. I just call it STUDIO). This new approach rests on four pillars:

  • Parallel. Mature software development teams can chew gum, walk, juggle, and can learn new stuff as fast as Neo (aka Thomas A. Anderson) learns martial arts. Development teams must have the confidence and insight to do parallel streams of development. Jason Darrow explains parallel development: “As we get a handle on requirements we break off a piece and start work on it in parallel.” For example, if one of the requirements of a new software project is to provide CRUD operations on 27 fields in a legacy IMS database, then get to it. The fastest way to get faster is to run parallel streams of design, development, and testing. When you break the tasks apart, some are sequential and others may be iterative. Walker Royce from IBM is doing groundbreaking work he calls software econometrics to measure the effectiveness of software development. One of his key conclusions is that successful software projects reduce uncertainty. Parallel work streams can reduce uncertainty and the number of costly iterations.
  • Immersive. Software is a means to an end. That end is to deliver applications that exhibit the seven qualities of great software: user experience, availability, performance, scalability, adaptability, security, and economy. All seven qualities are important and interrelated. But if you get the user experience wrong, then nothing else matters. Developers often blindly rely on businesspeople, business analysts, user experience designers, and customers to tell them what will make a great user experience. Writing code is the least of the talents of great software developers. Immersing is a domain that takes time and hard work. Software development leaders must deeply immerse themselves in their users' domain.
  • Software. That’s what we develop. That’s what we deliver. However, the future software development is experience. Blame Apple if you want. Thanks to the mobile revolution, user expectations for software, whether that software is mobile, web, or traditional, have never been higher. People use apps that are amazingly intuitive to use and that make them feel pampered. They quickly start to wonder why all their software experiences can’t be like that. The bar has risen — big time. My view is that software developers are much more than coders. Software developers are experience creators.
  • Studio. Software development is not pure coding, engineering, architecture, management, or design. It is cross-disciplinary. Better yet, it is its own discipline. It is more akin to making a movie than to building automobiles on an assembly line. The studio revolves around talent. Great software talent means renaissance developers who have passion, creativity, discipline, domain knowledge, and user empathy. These traits are backed by architecture, design, and by technical know-how that spans just knowing the technology flavor of the day. Process is the studio; it has structure but is flexible enough to optimize talent and tools. Tools matter in the studio — not just management tools, but software development and design tools that go beyond programming languages such as Java. We need model-based and more-visual tools to make development faster and easy for more people to participate in the creative process of software development.

The parallel, immersive software studio is different from other methodologies in five important ways:

  1. Software is not code. It creates experience.
  2. Development teams are not coders. They are experience creators.
  3. Technical talent is table stakes. Great developers must be design and domain experts.
  4. Process is bankrupt without design. You get what you design, so you better get the design right.
  5. Software is a creative endeavor, not an industrial process like building automobiles. The methodology is structured to support the creative talent.

This post about parallel, immersive software studio (STUDIO) can’t compete with the 1,813 books about Agile Software on Amazon.com right now. It is just a new idea in a sea of many. But, the accelerating demand for applications that “wow” businesses and users will create new winners and losers in the world of software development. Agile-mania is ending. The evidence: the Agile gurus are now telling us what is wrong with Agile and how to fix it. I say: Throw the baby out with the bath water. Now it is time to get serious. Our craft of software development must be based on creating a continuous stream of insanely great user experiences, not based on the problems of our past. Our value as application development professionals depends on it.

As always, I welcome a vibrant discussion!
Mike Gualtieri, Principal Analyst at Forrester Research

Note
I initially wrote about this new approach in December 2008. It is the result of countless conversations with real software development professionals; observations in many different software development shops, including software vendors, enterprises apps, consumer apps, and core technologies; and a large amount of research in adjacent disciplines such as product design, advertising, and movie making. This research is also based on interviews with business people, product managers, and executives as well as my experience as a software developer for 25+ years using a wide range of technologies and methodologies for small and very large applications including Z80 for operating systems and Java for enterprise apps.

Recommended reading:

  1. The Design of Design: Essays from a Computer Scientist by Frederick P. Brooks, Jr., 2010
  2. The Design of Business: Why Design Thinking is the Next Competitive Advantage by Roger L. Martin, 2009
  3. Bringing Design to Software by Terry Winograd, 1996
  4. Built to Love: Creating Products That Captivate Customers by Peter Boatwright & Jonathan Cagan, 2010

Comments

Mike strikes again!

Hey Mike,

Another controversial post! The Product Manager in me violently agrees with you. User experience, user requirements and user empowerment are the pilars of successful software, not code. Clearly we expect it to run with as little bugs as possible but this is not the end goal. The goal is to deliver value for this center role: the business user.

I love the analogy with a studio. Well put.

My only word of caution is on the use of the term "design". It has been abused in the industry. We use it interchangeably when referring to software design (closer to architecture) or UI design (closer to user experience).

The empowerment of talented users -- the room for creativity that you advocate -- is somewhat incompatible with a heavy design in the sense of "modelling". If you need to spend 2 years and an army of analysts to model your business, you are certainly not agile (not in the methodology sense; in the regular "need for flexibility" sense). This is actually typically intimidating for the creative mind -- how could he/she rethink the way the company does business when it requires again such a large effort to change it? What you really need is a fantastic "experience" design that allows for that kind of creativity, one in which you are not intimidated, one in which you understand the impact of your changes, one in which you feel empowered, and possibly one in which you could be challenged to rethink actually.

I do not believe we are saying different things though.

Thanks for another thought-provoking post!

Carole-Ann

The meaning of design is often misunderstood

Hi Carole-Ann, Thanks for your comments! I agree that the word" design" is often misinterpreted as too narrow in scope. To me, design = decisions. There are thousands of decisions that go into a software product (or any product for that matter). Those decisions range from architecture to interaction design to visual design - as you pint out.

I love how you brought in the empowered businessperson into the Studio. Yes! That is why the studio must be filled will all kinds of tools such as Business rules, BPM, content tools - not only programming tools.

Seems you imply at least two

Seems you imply at least two possible approaches towards developing great software experiences: either division of labor with, or investing in, more front end/creative developers (as opposed to backend coders, architects, systems integrators) as part of the experience creation team, which is an art in itself, or, in the more Agile approach, devs get UX/IxD/IA and do that work themselves.

I agree if everyone on the team knew more domain knowledge and owned that domain knowledge, they could potentially disintermediate people like me, who currently act as communication coordinators and synthesizers among users, creatives, strategists, and technologists. And therefore we should get better/greater experiences delivered through products/software faster, because this would speed up the needed requirements integration into the product.

The problem is that it is extremely hard to do both jobs well. It's hard enough to make software well, much less work. It's difficult to do user experience well, since people barely understand the relationship between usability, interaction design and information architecture. We haven't even touched on sensorial design. The number of true hybrid designer-coders I have known in my 15 years in this industry is very rare. And even rarer is that rarest of multi-class characters, the UX/Strategist/Developer/Designer. Yet that is clearly where we should be going.

Have you spoken to the comp sci or design education community, which churns out batches of increasingly unprepared programmers and designers each year?

Rare indeed. You are correct on division of labor.

Thanks for your comments Gene. Yes. The parallel part of this approach recognizes that all developers don't have that hybrid designer-architect-coder talent. So we find the things we do know and then let the more programming focused developers implement in parallel. I think of this like how films are made. You have talented people at all levels below and above the line as they say. The above the line are the renaissance developers. Below the line people do the more routine work.

You are so right about the comp sci and design education communities. They churn out single-disciplinary clones.

I find the ideas you've put

I find the ideas you've put forth intriguing, but I'm struggling to understand how they're dramatically different than what we're doing in an iteration now. In our teams, where our process model is Scrum, and our implementation model is formed around Agile, we're doing exactly what you've discussed. We're spending some time up front determining the over-arching flow with the business executives (SME, Product Manager, whatever you want to call them), and then launching into parallel streams to implement. We have UX, Coder, Database, and BA and QA all working together to create useful and exciting user interfaces that delight our customers while at the same time minimizing the upfront investment so as to maximize our efficiency.

As measures of our success we've seen:
1. Happier employees. Low attrition across the board. People want to work here.
2. Happier customers. We're churning out compelling features faster.
3. Higher quality. Our bug counts have dropped significantly. Literally, 14x.
4. Higher ROI. We're moving more efficiently, which is getting product in our customer's hands faster, and therefore helping us to realize our return on investment at a greater clip. It's both faster and driving new sales.

Perhaps we're just an advanced Agile community, but I fail to see how what you're presenting here is any different than what many agile shops who have moved beyond XP are doing today.

Nothing Incompatible Here

That's because, while there are some great (if not necessarily new) approach-suggestions in this article, none of them are at all incompatible with Agile/Scrum.

Railing about overly simple-minded and poorly executed Agile does nothing to disprove the potential value (and ongoing effectiveness for many) of Agile. Take any one of these perfectly good suggestions and execute on them in a thoughtless, sloppy manner, and you'll experience the same failures. Try infusing these suggestions into a receptive Agile team, and you may see their effectiveness increase without necessarily threatening their basic Agile approach.

BTW, in most of the real world, working software *is* the primary measure of progress. It's a perception thing on the part of most management, and something to which implementation teams often have to answer. Whether that perception is right or wrong is an entirely different question, one that ends up not being all that practically relevant to many implementors...

AAren’t Software Developers the Archetype Knowledge Worker?

Mike,
Great post. Really got me thinking - but as I found myself starting to write an essay I moved it to my blog here:
http://ukelson.wordpress.com/2011/10/13/arent-software-developers-the-ar...

The future is Business Technology = "Agile Software"

I agree the "agile" movement has been overhyped (so what's new) in an effort to placate business who remain cynical about "IT" capability to relate to business. What is needed is a complete change in how we build applications. Why in the 21st century are we still coding business logic that never changes?
To put this into context I had an exchange with your CEO George Colony in 2010 said: “If we don’t get from IT to BT [Business Technology] we’re going to have more disasters like our present mortgage meltdown. Why? Because IT creates impenetrable systems that human beings can’t manage. BT is about human beings back in control.” In 2008 Bill Gates said “future applications should use only around 10% of the code that is used today” seeing this as “the holy grail of development forever”.
First people are the source creators of all information and second there is the need to recognise that people work in relatively small teams to achieve individual and collective outcomes that make any business
This may be a surprise for technical led people but understand that business logic has never changed since commerce started; indeed business is actually quite “simple” if you focus on supporting people at work. The communication technologies to deliver are both complex and challenging but do not change the fundamentals of business; it is about people, internal and external to the organisation and IT is there to support and contribute to efficiency.
When you look at how people work irrespective of the required function there are relatively few work task types, human and system, including the user interface that address all business driven issues? So why repeatedly recode for every function in a business? Business owners ask the questions; “Do we really need to rely on a vendor’s view to be imposed to run “my” organisation? What do coders know about business when I want a custom solution?” In both cases the result is a compromise to the actual operational requirements and by nature both lack flexibility to support the dynamic environment where people work.
Have you “got it” yet? Well 20 years ago R&D in UK based Procession started to address this fundamental issue and solved it. The result is the gap is closed between business and IT and simplicity replaces complexity. See here if you are interested plenty reading. http://bit.ly/qlSUvM
The question I ask of you is why do you (Forrester and the other industry analysts) ignore such step change innovation? All credit to your CEO allowing me to use his quote but Forrester seem incapable of rising above the self interest that holds back the software side of “IT”? Should you not be helping users to become the intelligent buyer? Ask the penetrating questions of vendors and push them to solve such issues as you raise? Agile software is now real and addresses both your CEO’s and Gates’ vision of the future It’s your job to test and help users gat the best from “IT” – so up to you and welcome to come and see!

IT seems to take two steps backward for every step forward

Hi David, Thank you so much for your insightful comments. I am not sure what you mean by "ignore step change innovation". My view is that IT seems to take two steps backward for every step forward because of a myopic focus on new technologies rather than keeping their eye on the business ball.

Step change without technology "drag"?

Mike
Yes totally agree and that's why "IT" is in a mess. 40 years ago I was walking around businesses talking to people, mapping out their processes and looking for weaknesses in order to both focus the audit and make recommendations for change. We have created a new platform that goes back to those fundamentals in business yet removes any technology complexity to deliver exactly what the business needs.
The step change is the result of the core design philosophy not technology - but that is boring and lacks a challenge for technology driven companies.....! We are totally business driven making life easy for developers and users - it is what "BT" is about where "IT" has failed?

re:Agile Software Is A Cop-Out, Here’s What’s Next

Interesting viewpoint. One thing you missed from your analysis of Agile methodologies is risk management.

When a software producer insists on regular iterations of working software, they are also insisting on resolving all of the risks of UI/UX design, schedule, and costs with each iteration, or bringing them to the forefront. It's one thing to write lengthy requirements & design documents describing work flows, user interactions, data models, process models, et cetera, but you never know if the elements of the design are complete and/or mutually exclusive. Only working software will tell you if your design document is right, thus resolving and/or exposing the risks tacitly baked into the design documents.

I have been a project manager, and been project managed in large teams. When project status reviews come around, it's amazing how much optimistic guessing and/or lying takes place. That is usually because you're reporting on software that doesn't work yet. I have also been managed by agile project managers. In the best gig I had, I was responsible for producing new deployed working user stories every Friday. The weekly development review consisted of walking through the deployed software and showing what I built. I didn't need to make unreliable gues-timates about when I would be done. The deployed and working software was the surest indication that I did what I said. The interesting thing about weekly stories is how hard I had to work to resolve all of the risks from design to deployment every week. But by being forced (by weekly deployments of new working software) to continuously resolve end-to-end risks, the overall requirements/development risks of the project were lowered.

Weekly of BiWeekly Sprints

Good post and good discussion. I agree with Jay that having to show working code at the end of a sprint can really help move things along.

The Product Manager also plays a key roll here. Since I just came out being a GM at a Silicon Valley Software firm, I witnessed what using Agile can do. It does take a unified team. I only went to Sprint wrap meetings if the demo was going to be part of the agenda.

So, Agile will evolve. The biggest thing besides UX that is needed is Speed.

Jim Lundy
CEO, Aragon Research
aragonresearch.com

Speed in build with Agile Software

Jim
I hope you read some of our papers. We have now reached the stage where build of end to end applications in measured in man days not man months. The discovery stage involves direct discussion with end users step by step what happens. It is important that thinking is “user centric” and not to be distracted by legacy which of course is recognised as existing information is required or special programs can be called at the appropriate moment. Once this step by step detail is understood you will have a good idea of the number of user forms required.

Count the required forms and that number becomes the number of man days to build the TOTAL solution including the user forms, rules, events, workflow, reporting etc. This build takes place in an easily understood graphical designer by the business analysts the core code never changes with all business requirements prebuilt as generic “task types” of which the user form is the most important.
Each user form is dynamically created for that user with the required role and presents only the information the users require to allow them to make their contribution to the business ready to pass on.

I believe it is the user centric approach and the user interface designed for ease of use that contributes to high levels of efficiency that are being achieved. Not only that, users are encouraged to contribute ideas to improve efficiency as change is readily supported as there is NO code change, code generation or compiling

So what is the challenge – actually it is getting folk like Forrester to take seriously! Someone had to do it just happens to be UK based…….! They should be telling you about such capability?

Doman Knowledge, UX, Fast feedback, Problem solving and Craftsm

That is a good post and I agree with many ideas you've shared.
"However, the future software development is experience" is very true.
I've wrote about future of software development as well, and I think Doman Knowledge, UX, Fast feedback, Problem solving and Craftsmanship are next huge trends.
http://targetprocess.com/rightthing.html

Wow. We are almost brothers

Michael,
We have similar thinking and we are both named Michael. Great work and and love the illustrations.

Typo in one of your sentences?

Hi

In the "Studio" section, you have a sentence -- Process is the studio has structure but it flexible to optimize talent and tools -- that doesn't really make sense. Can you correct or elaborate?

Very nice and an eye opener

Very nice and an eye opener article to managers who see agile a solution for all S/w Developmental work

Excellent! Got all riled, but in a good way.

Mine grew too big for a comment too--posted here: http://pagilista.blogspot.com/2011/10/next-gen-agile-get-rid-of-non-code...

Cheers, Mike!

Reflection is Good

I started to comment about this post, but it turned into a blog entry. :)

http://practicalagility.blogspot.com/2011/10/agile-is-cop-out.html

TL:DR? This has some really good points, and some that I don't agree with. I will, though, defend the Manifesto as a statement of the times (early 2001). I explain why in the blog post.

Wow. I Appreciate the detailed comments

Hi Dave, Thanks for the detailed analysis of my post. I encourage everyone to read it: http://practicalagility.blogspot.com/2011/10/agile-is-cop-out.html

No study methodology - not credible

Hi Hass,

Unfortunately, the report commissioned by Rally Software, a vendor that provides Agile tools, does not show any methodology to conduct the study. To be fair it is really hard to provide empirical evidence for software methodlogy because projects are so different and there are many variables.

But, I object to these unsubstantiated claims by firms or organizations that have a financial stake in the outcome.

Those who code are not the only ones who practice Agile

Bold stuff, which I appreciate. Though, my experience has been quite the opposite. I work with Agile teams that pair developers with UX. *And*, the developers I've worked with are very business-focused. The process facilitates business, customer, process, and technology understanding first, then translates it into working models that can be solved with software. It's not code for code's sake. It's design and code to demonstrate understanding. And for me and the teams that I work with, it has worked beautifully.

That said, I always appreciate being challenged. This conversation increases our awareness of what it takes to be successful. And practitioners of all methodologies need to embrace that.

UX teams and Agile

Hi Bill, Thanks for appreciating the dialog. I appreciate your comments. Mindset more than process results in successful software projects. It sounds like you have a great team that focuses on the right thing - satisfying business needs by satisfying user needs. With right mindset, perhaps any process can work well.

Agile Zealots

When I saw this video, it reminded me of every Agile zealot I've ever met... and I've met a lot!

http://www.youtube.com/watch?v=nvks70PD0Rs

LOL

Great parody of snake oil salesman

Two more critical pillars

Wow, you really seem to have hit a nerve.

Your four pillars are all good, important points but they ignore the business perspective! I have been a user experience advocate forever - since being the user experience advocate for IBM Research in the 90s. User experience is a critical perspective for developers to understand, and critical for the success of any software. But I have learned it isn't enough - software development needs to take into account two more critical business aspects:
•Business Value. Software must deliver business value, and that isn't always completely aligned with user experience or any of the immersive points you list. A application can have a great user experience but bring no value to the business, which means it will either be killed or die from lack of attention. In most cases business value needs to come first, and foremost.
•Platform. Software (even what ends up as applications software) is either a platform itself, or built on (and used to enrich) a platform. This is both a business and technical decision, and can greatly affect the long term success of any software. See a great rant on the subject at https://raw.github.com/gist/933cc4f7df97d553ed89/24386c6a79bb4b31fb818b7...

I wrote more on this and how it relates to BPM here: http://ukelson.wordpress.com/2011/10/14/if-agile-software-is-a-cop-out-i...

I don't know how I can agree

I don't know how I can agree with most of your final points yet completely disagree with your premise.

You state:

"1. Software is not code. It creates experience."

Agreed.

"2. Development teams are not coders. They are experience creators."

Agreed.

"4. Process is bankrupt without design. You get what you design, so you better get the design right."

Agreed.

"5. Software is a creative endeavor, not an industrial process like building automobiles. The methodology is structured to support the creative talent."

Wholly agree!

"3. Technical talent is table stakes. Great developers must be design and domain experts."

Disagree. Great developers can learn enough about any given domain in which they are working to be effective but it is doing real domain experts a disservice to say that their knowledge is so easily garnered that each developer rolling onto the project can just "pic it up". There is such a thing as software development domain knowledge and that is what developers should be fluent in.

I think what it comes down to is your own experience with developers has not been good. How could you have worked with great developers if your view of them is so biased:

"Rush to write code", "Software developers want to be measured on what they know — code", "typing class definitions, if-then-else statements, and looping constructs into glorified text editors"

If this is your view of developers I can see why you jump to the conclusions that you do. Agile methodologies don't work well with the heads-down, dark room, Mountain Dew-swilling, developer archetype which you appear to subscribe to. The developers I work with can communicate effectively to other developers as well as with clients and stakeholders. They can easily jump from view layers and user interactions to domain classes and system architecture discussions. But more than that they can understand and speak to business requirements or, when they don't, they can drive towards the answers they need to keep going, and move quickly. Software development is more about communication that code, but it appears that those you come in contact with are only interested in the code piece.

You stumble upon the truth of it here: "Software development is not pure coding, engineering, architecture, management, or design. It is cross-disciplinary."

This is exactly right but for whatever reason you come to this near the end of your post and don't go back to revise your stereotypes about developers made earlier in the write up.

All too often people think you can change out the process and all your disfunction will be solved just like that. Chances are that if you have poor communicators on your teal in one process, you'll likely have the same scenario with agile. Agile isn't a magin pill. It is a practice that takes discipling AND the right skill set. This sounds elitist but not everyone developer I have met can communicate effectively with clients or deal with the ambiguity that is inherent in an iterative approach such as agile.

My advice to you. Keep the process and find different developers.

re: I don't know how I can agree

"It is a practice that takes discipling..."

Was that a Freudian slip? :)

I do like your comment - I guess I've become somewhat numb to the comments about the stereotypical developers.

I worked as a contract developer from 1991 until I became an Agile Coach full-time a few years back. As such, I've moved from organization to organization, from domain to domain. Each time I had to learn the business as quickly as possible in order to be an effective developer. That said, I most certainly was NOT a domain expert - there were people in each case who were - but I did acquire enough domain knowledge to be able to carry out meaningful conversations with the experts in order to determine the best approaches to take with the systems on which we worked.

Did I need to be an expert in order to be effective? No. Would it have hurt if I did become an expert? Probably not.

Again, though, I'd like to hear Mike's definition of "design". Does he mean external, i.e. user or customer-facing design, or internal structural design.

"It is a practice that takes

"It is a practice that takes discipling..."

Was that a Freudian slip? :)

Too many years as a project manager :)

No panaceas for bad culture

Ben makes a great point: [typos corrected]

"Chances are that if you have poor communicators on your team in one process, you'll likely have the same scenario with agile. Agile isn't a magic pill. It is a practice that takes discipling AND the right skill set. "

There has been A LOT of hype around Agile. The benefits -- improved communication, better project status visibility -- have value, but only when the culture is one of open communication and honestly. As an individual, it's still possible to fudge your status or withhold information, but ideally, you'll be found out faster in an Agile environment. BUT if the general culture is one where open communication does not happen, or is not promoted, Agile is as bad as any other process.

In the end, (as with most things), it all comes down to the people. And while it's not always easy -- as Ben says at the end -- to find different developers, something has to change to get the benefits.

I've written about this problem (a few years back) on my blog.

http://onproductmanagement.net/2008/09/28/agilescrum-reality-check/

Saeed

I really appreciate your

I really appreciate your detail comments Ben. I completely hear you that developers are not the heads-down Mountain-dew swilling variety. I have been a developer for more than 20+ years. I am with you. I object to view of developers as automatons on an industrial assembly line. I think the process gurus (such as Agile) want to say that all developers are created equal. They suggest that you simply do this process and you will have magical miracles happen. By new suggested methodology (STUDIO) calls for a flexible process that supports talented developers - like real studio.

Anyway, I think we are in 100% agreement. I can tell from your comments that you are are a real deal developer. I am with you.

I suppose the usual way of

I suppose the usual way of progress is to abandon a concept even though the original definition has very little to do with the actual execution out in the wild. A prominent example would be Communism. The actual execution on a large scale is a pretty poor implementation of the original idea, but one has to acknowledge that the original idea only seems to work under very special circumstances (small groups of tribe size). The question is whether "agile" suffers from the same problem.

Regardless of this, you are using very specific definitions for some terms stated in the agile manifesto. For example, working software to me is far from narcissistic. Working software to me is that it works for those who use it and that includes a lot more than what working software means to me as a developer. Also “Business people and developers working together daily” is not even a sentence of the agile manifesto, it says "Customer collaboration over contract negotiation". I would think that if you reduce customer collaboration to talking to business stakeholders and analysts you are limiting the original definition.

Either way, I suppose you are attacking common interpretations of agility, that is, the actual execution out there. As such I would think that you are very much in accordance with the original idea(l) which will advice you to "respond to change".

Agile Manifesto

Hi Frank, Thanks much for the comments. The Agile Manifesto does say "Business people and developers working together daily" according to the fourth principle in http://agilemanifesto.org/principles.html

You make a great point about the difference between the original definition and how it is actually implemented. My colleague Dave West has found that Agile is actually implemented as Water-scrum-fall. (See link in my post to this).

I just think Agile is naive, a capitulation to our ability to understand the business domain, and that we can do way better.

The key change in my STUDIO approach is split off the the technical parts of a software development project from the user experience. The technical parts happen in parallel and and the ux happens in a flexible studio environment.

By the way, I love your analogy to Communism to make the point of theory versus execution.

The Issue of Scale

I like your ideas and share your feelings about Agile. My main interest is in methodologies that scale up to large $10M+ systems. To build these kind of systems, we need a reproducible approach to the problem of partitioning that can be validated. If you can't partition effectively, you can't achieve the parallelism that you point out is necessary. The partitioning problem is one of the great failures of Agile, and the reason that Agile can't scale up beyond a small system. However I don't see that Studio has solved this problem.

The partitioning problem is one that I have worked on extensively, and have come to the following conclusions:

- Effective partitioning requires close cooperation between the business and IT (which you also point out.)

- Partitioning is very sensitive to errors, and therefore must be guided by a directed framework, preferably one grounded in a mathematical model for complexity.

- Decompositional approaches to partitioning cannot work because they are "drunkard walk" methodologies, that is, only slightly better than random.

I'll be interested in hearing more about Studio.

Parallel is how STUDIO scales

Hi Roger,
Almost all software projects contain some very well known requirements. In STUDIO, I propose those are peeled off and run in multiple parallel streams of work. That is how STUDIO scales. At the same time the Studio, focuses the unknowns or innovation within a small talented creative group in a studio. The studio is a completely flexible processed centered around the talent. So, the parallel means faster development and the studio means design creativity.

I'll have to do a much bigger post to describe the STUDIO approach in full.

Not sure what you mean by the "drunkard walk". There are well-known requirements that can be spun off and run with great confidence. Those that can't are in the studio.

"Peeling Off"

Thanks Mike. The problem of scalability all centers around the issue of "peeling off" into parallel streams of work. Which requirements get peeled off into which streams with which other requirements? The answer to this question is critical from the perspective of the overall complexity of the resulting workflows.

By "drunkard walk" I mean that the answer is variable, and probably not optimal.

Consider the problem of a system with requirements R1, R2, R3, etc. and Streams S1, S2, etc. Which of these is better:
Choice 1: S1 = R1, R2; S2 = R3, R4
Choice 2: S1 = R1, R2, R3; S2 = R4
etc.

How do you choose? You take you best shot, much like a drunkard takes his best shot of getting from A to B.

I believe this is the wrong approach because it is not scalability. I believe that for large systems (>$10M) you must make the choice through a directed, mathematically grounded methodology.

This is a topic that I covered in excruciating detail in my white paper, "The Mathematics of IT Simplification" at
http://www.objectwatch.com/white_papers.htm#Math

Thanks for the discussion!

- Roger

Wow. Great points on Iterative versus Certainty

What you call the "drunkards walk", I called dead-reckoning in my post. I like your drunkards walk phrase much better, by the way. The drunkard's walk is one of my key criticisms of Agile because as you point out and can be very costly in terms of money and time. We are really talking about iteration here. Iteration is good, but I want as few iterations as possible. Every turn of the iteration wheel costs time and money. The fewer the better.

I will check out your paper on "The Mathematics of IT Simplification" at
http://www.objectwatch.com/white_papers.htm#Math
If your approach can help determine what to peel off in parallel and what work to iteratate it would be a boon to software development. I would like to talk to you about you about this.

Further Discussions

Mike,
I believe my approach not only helps to determine what to peel off in parallel and what work to iterate, but does so with mathematical precision (with a dash of uncertainty thrown it for the human element!) In fact, the whole approach is so novel that I was recently granted a patent for it. As far as I can tell, this is the only patent ever awarded for a mathematically grounded methodology for both business process and IT simplification.

I would love to discuss it with you. I don't know how to contact you, but I am userID roger, domain objectwatch.com. Drop me a note!

- Roger

You sound like

Mike,
Enjoyed your post - love the rabble-rousing. And as someone who lived though the "we can't ship anything on time" 90's, I agree that Agile WAS mainly a way of fixing old problems.

But we depart right from the git-go -

"working software as the measure of progress" is the way the original manifesto folks (who were just following a bunch of other best practices and labeling them) were fixing the old problem. Who defines "working"? Where I've worked (most recently) it's the designers of the software (UX pros) and QA - nobody else.

"Business people and developers working together daily” is lazy" - "Business people" in the original manifesto now means designers in my experience. Companies who have finally discovered design as process (still a minority in my recent experience) will push regular synchronizations (call them scrums, zero-defect vertical milestones, whatever) between coders, designers, and QA, with designers being first amongst equals.

And what I know for sure is that you DON'T want to put any user-design knowledge responsibilities in the hands of coders. I'm a coder. I know better.

Don't read the Agile bible literally or try to channel the founders' intentions via the words in the Constitution. The principle's, in my personal experience, are quite sound when incorporated to the (somewhat newly accepted) discipline of design process.

With all due respect, your suggested solutions were so full of ill-defined terms I had trouble reading them :-)
best,
Joe

I DO want to put design in the hands of software developers

Thanks for your comments Joe. I actually DO want to put UX design in the hands of software developers. I argue that software developers are actually experience creators. If "coders" can't learn this (which I think they can) then perhaps they should be the parallel programmers doing the plumbing part of the project.

I don't think the Agile principles are sound at all because they magically assume process creates great software. My new methodology suggests a Studio approach that values talent and creativity over process. In STUDIO process is flexible.

Agile advocates constantly re-interpret Agile whenever criticism is made. I hear often, "Oh Agile covers that. Oh it does that, etc...."

I will acknowledge that my STUDIO process is hard to explain fully in a short blog post. I'll post a more fuller version.

Process is bankrupt without design, and STUDIO puts design first.

Agile Advocates

"Agile advocates constantly re-interpret Agile whenever criticism is made. I hear often, "Oh Agile covers that. Oh it does that, etc...."

OK, Mike, I want names. Point me to sources. Either show us who is saying these things, or stop making these types of attacks.

I understand that this is a blog and therefore your personal opinion. If you were writing an article for Forrester that had to be vetted by an editor, would that sort of style be permitted?

Using a quote to aggregate sentiment is fair writing style

Dave,
I am just expressing the sentiment I often hear when I discuss the pros/cons of Agile with people who have a financial stake in Agile. What does westborosystems do?

re: Using a quote to aggregate sentiment is fair writing style

We provide Agile coaching and training services. We've all "been around" since long before the term Agile was coined, and like yourself have all been software developers. Hell, I haven't written any code since yesterday. ;) Do *I* have a financial stake in Agile? Currently, yes. As I mentioned in my blog response, though, I really like what I'm seeing in the Lean Startup community. Do I like it because it's hype? Not at all. I like it because it represents refinements and additions to (and some subtractions from) numerous practices that I've seen work and work well since I started in this industry in the late 80's

Hwever, my real point is that you're asking people to simply believe that you've heard these things. Essentially you're saying, "If you don't believe me, just ask me!"

Again, I agree with a LOT of what you say in the article. I don't like a lot of the hype that comes from people in the Agile world, and I specifically referred to Jeff Sutherland and the notion of hyper-productivity.

To give you an idea of my perspective on this, have a look at a post I made to the Extreme Programming Yahoo Group in May 2002: http://tech.groups.yahoo.com/group/extremeprogramming/message/51426

Define a software developer?

Let me explain how it happens with "agile software" and in particular in the context of the UX
First all front and directly linked back office is built by the business analysts skill set. This includes the build of a user form as a task type and does not require programmers to build a custom business form. The core design is once only entry of information and that includes on the form where the original entry made. This requires intelligently linked grids where information need not to be re-entered. Each form is dynamically created for that instance so is customised for the ease of use by that user for quick assimilation of information, logical use and minimal finger work = good UX. This is the functionality that comes as part of the end to end tool kit including the "plumbing". But the BA skills is not about "artistic design" that's a whole different skill set that can be handed over to a web developer if say it is an external landing page where there is an image building requirement. But people at work just want their software to do the support job with minimal effort and of course change quickly as their job changes.

Don't ask the geeks to do design.

Mike - thanks for your always respectful discourse. I love that you rabble rouse and then discuss in a mature manner - keep it up!
We will have to agree to disagree on this programmer-as-designer issue. It's core to our disagreement. After 30 years in an industry where design was dictated by programmers to bad ends, and with the more recent rise of the legitimate profession of UX (as well as working to teach the UX people about programming) I'm more convinced than ever that those twain shall rarely meet. "Us geeks" just don't seem to be able to do human-centric design, and the UX folks need to be grounded in technical reality. it's a team effort - the more x-over of skills the better, but I have to resort to that hackneyed left-brain/right-brain view to suggest that, in my experience barely 10% can do well at both.
I like Agile because the name itself infers an adaptable/flexible process, made to conform to a particular product/company's needs. I'm not a pigs and chickens guy. Every product/team has its own sociological structure that works best. The folks who follow Agile as dogma in my experience, almost inevitably get it wrong. It's about producing human-centric designed software on time with good quality. I think we can all agree on that!
Joe
p.s. ever read "The Dynamics of Software Development"? A 90's view of development as sociology that still holds up even today

re: Pigs & Chickens

"I'm not a pigs and chickens guy."

Me neither. I've never like Ken Schwaber's anti-management attitude that creates more impediments than it solves in my experience.

"The folks who follow Agile as dogma in my experience, almost inevitably get it wrong. It's about producing human-centric designed software on time with good quality. I think we can all agree on that!"

Add me to the list of those who agree! :)

Pigs And Chickens Is So Immature

Hi Dave,
Agreed. Pigs and chickens is so naive (and SCRUM).There are many software development methodology who have never developed software or have developed system software. Many of the original manifesto signers never developer user interfaces. They had an epiphany that users matter. LOL. I am for reality and progress. Let's move ahead!

Merge Geeks and UX, set coders on parallel development

Hi Joe,
I haven't read The Dynamics Of Software Development" but I will check it out. Did you see my recommended reading list at the end of this post. The first book was the Design of Design by Frederick P. Brooks. He is the author of the Mythical Man Month. Good stuff.

I agree that all coders are not good at UX. But, that is partially because they haven't learned how. But, I acknowledge that some don;t just have the affinity for it.

I account for that in STUDIO. I propose that great coders split off and do parallel development on requirements that are very well known and lean toward the technical. The developer/UX/domain expert work in the creative studio environment to innoavte and develop the human interface,

A few other books, and some comment

Mike - You'll likely have a tough time finding "Dynamics of Software Development", as it's been out of print for some years. It is dated in the sense that the industry had not at that time come to understand that usability couldn't just be 'added on', but needs to be designed in from the start.
Two other books that served us well back in the 90's when we were desperately searching for process improvements are also good reads, but also dated. Steve McConnell - well known as the author of 'Code Complete' wrote a process book called "Rapid Development".
Finally, there was an analysis of what Microsoft was attempting to do to solve this problem called "Microsoft Secrets" - which was the first book to mention zero-defect vertical milestones as a way to motivate developers to integrate their efforts towards success in the inevitably multi-tiered segregation of the development of complex product (front end guys, server side guys, API guys - the whole works).

Ironically all three of these books were written about (or by) Microsoft, who was then the only company actually facing these challenges (at a significant scale) and coming up with ideas to solve them (this was the mid-90's). They did, and they, well, I guess we'd all say they kind of went off the rails in some ways since then. But don't let that fool you - those three books were a beginning to what we today call Agile (or whatever).

I have retired from industry and am indulging myself in a sunset career of teaching programming to UX students. I've been involved with the U of Maryland's excellent HCI department. I've analysed the programs out there today - Stanford and Carnegie Mellon are the best I would say. Niether school emphasizes programming as a skill required for design. In fact, although both programs are masters level programs associated with the computer science departments, their students are populated by multi-disciplinary students - BS's in education, humanities, sociology, and of course Cognitive Psych (where the roots of UX design lie). Design, as a discipline, cannot be encapsulated in any way in programming. In fact, excellent design's principles lie only in part in the HCI world - much of what good design is can be completely separate from computers.

So I leave you with this 'rabble-rousing' comment (which I'm sure you'll appreciate since it's your MO too) - there is ZERO evidence that genuinely competent programmers (beyond perhaps 5 to 10%) could possibly 'learn' UX design principles and be any good at them. To say that they could do so is to trivialize what UX design really is. It's a growing discipline that involves mostly non-programmers toward the goal of producing software that is sensible to you and your mom. It can't be learned 'on the job', and IMHO it can't be learned at all by most programmers. As I attempt to teach UX students to program I am reminded every day how diverse our minds are. Please give some credence to each of our skills, and propose a development process that includes all aspects of our different cognitive modalities - something we MUST so to produce excellent product.

Keep stirring up the s**t!!!! I love it!
Joe

The way I see it

In my 20+ years in the software industry, I have worked with them all, waterfall, RAD, iterative, Agile, Scrum, etc. Besides being a developer on all the major platforms like Windows, UNIX, Linux, etc, I have also managed engineering organizations of up to 25 people. For the past few years, I am once again a full-time developer primarily using Ruby on Rails focusing on web development.

Now onto your 2 main points on Agile cop-outs.

First point, "Using “working software as the measure of progress” is narcissistic." In the old waterfall days, developers were just measured on their coding ability and little else. Hence, they usually implemented features blindly thinking well, that's their job. That mindset led to incomprehensible software especially when it comes to user experience. That's where Agile comes in because it gives the developers a better methodology to be involved all through the lifecycle working with product managers (and real users sometimes) so that the user experience doesn't go untested until the end. IMHO, working code is the ultimate end-goal because without it, there is nothing for the user to touch, see or feel hence not allowing them to provide timely feedback.

Second point, "Business people and developers working together daily is lazy". Hardly the case, in fact this type of working relationship spurs innovation and promotes better working relationship. True, business people aren't usually end users but that's no different than product managers writing MRDs and PRDs. In ideal agile teams, the actual end users are involved, otherwise business people acts as proxy users. I found that it's actually much easier for developers to blame it on business people using the waterfall model because well it's all written down and they coded what was in the specs so it didn't meet user needs, it's not the developers' fault. Agile empowers the developers to have meaningful discussions with business people along the way to flesh out details that often won't come to light until implementation instead of waiting till the end of the project to sort out.

That said, there are definitely challenges with implementing Agile in the real world. One of Agile's main challenge is getting everybody on board and trained enough to understand why and how this is different from waterfall. Sadly, the majority of a team attempting agile is poorly trained, or not trained at all, and far more familiar with waterfall and never understands Agile well enough to properly execute it. That's speaking from personal experience after attempting to implement it at a couple of companies.

@whataboutbob