Agile Software Is A Cop-Out; Here’s What’s Next

Never has a new trend annoyed me as much as Agile. Right from the get-go, the Agile Manifesto revealed the weaknesses and immaturity of the founding principles. The two most disturbing: “Working software is the primary measure of progress” and “Business people and developers must work together daily throughout the project.” These are cop-outs because:

  • Using “working software as the measure of progress” is narcissistic. Rush to write code is oft the translation of this misguided principle. Software developers want to be measured on what they know — code — instead of what they don’t know: the domain, business requirements, and the user. The idea is that you’ve got nothing if you don’t have working software. The cop-out is thinking that code is the measure of everything. Software is always part of a bigger picture. It works in conjunction directly or indirectly with other software, technologies, the environment, business processes, and people to create customer experience and/or improve operational efficiency. Like most of the Agile principles, this one is developer-centric rather than user-centric.
  • “Business people and developers working together daily” is lazy. This principle is a capitulation by developers who say it is impossible to understand the true requirements. Developers would much rather stick with what they know: typing class definitions, if-then-else statements, and looping constructs into glorified text editors. The absurd Agile solution to getting the requirements right is to meet with the business every day (or frequently). Often the business people who are involved are not the same as the users, and they are not domain experts or design experts. Seriously, can your best people be available to help developers do their job? The cop-out is thinking that great software can be developed through a process of dead-reckoning with business people. This is just another way for software developers to blame the business for challenged or failed software projects and a sure-fire way to propagate uncertainty and mediocrity. It is a false argument to say that developers cannot understand the requirements and anticipate changes. Development teams must understand the business and the users better than they understand themselves.

Yikes. I have used some strong words here, and I could go on and on. I acknowledge the good intentions of the Agile community. But, the lack of empirical evidence for the benefits of Agile methods is telling. My colleague Dave West, who covers Agile for Forrester, has seen the cracks in how Agile is implemented and written about them in a new research paper titled “Water-Scrum-Fall Is The Reality Of Agile For Most Organizations Today. The report also shows Agile adoption at 38.6% in 2010 and notes that interest in Agile remains very, very high. Most of the evidence I have seen is anecdotal — of the “It’s a miracle. I can walk again!” kind. My take is that the Hawthorne effect is hard at work after years of brain-numbing sequential processes such as waterfall. Anything was better than the worst.   

A New Thesis For A Better Approach To Software Development
For the future of application development to be insanely great user experiences, we need a new software development methodology. Agile is not it. Agile is a group of methodologies that came from asking, “How can we fix the software development mistakes of the past?” We need a new approach that asks, “What software development process prepares us to be experience creators?” I call this new approach: parallel, immersive software studio (I am well aware of the purely coincidental acronym P#SS. I just call it STUDIO). This new approach rests on four pillars:

  • Parallel. Mature software development teams can chew gum, walk, juggle, and can learn new stuff as fast as Neo (aka Thomas A. Anderson) learns martial arts. Development teams must have the confidence and insight to do parallel streams of development. Jason Darrow explains parallel development: “As we get a handle on requirements we break off a piece and start work on it in parallel.” For example, if one of the requirements of a new software project is to provide CRUD operations on 27 fields in a legacy IMS database, then get to it. The fastest way to get faster is to run parallel streams of design, development, and testing. When you break the tasks apart, some are sequential and others may be iterative. Walker Royce from IBM is doing groundbreaking work he calls software econometrics to measure the effectiveness of software development. One of his key conclusions is that successful software projects reduce uncertainty. Parallel work streams can reduce uncertainty and the number of costly iterations.
  • Immersive. Software is a means to an end. That end is to deliver applications that exhibit the seven qualities of great software: user experience, availability, performance, scalability, adaptability, security, and economy. All seven qualities are important and interrelated. But if you get the user experience wrong, then nothing else matters. Developers often blindly rely on businesspeople, business analysts, user experience designers, and customers to tell them what will make a great user experience. Writing code is the least of the talents of great software developers. Immersing is a domain that takes time and hard work. Software development leaders must deeply immerse themselves in their users' domain.
  • Software. That’s what we develop. That’s what we deliver. However, the future software development is experience. Blame Apple if you want. Thanks to the mobile revolution, user expectations for software, whether that software is mobile, web, or traditional, have never been higher. People use apps that are amazingly intuitive to use and that make them feel pampered. They quickly start to wonder why all their software experiences can’t be like that. The bar has risen — big time. My view is that software developers are much more than coders. Software developers are experience creators.
  • Studio. Software development is not pure coding, engineering, architecture, management, or design. It is cross-disciplinary. Better yet, it is its own discipline. It is more akin to making a movie than to building automobiles on an assembly line. The studio revolves around talent. Great software talent means renaissance developers who have passion, creativity, discipline, domain knowledge, and user empathy. These traits are backed by architecture, design, and by technical know-how that spans just knowing the technology flavor of the day. Process is the studio; it has structure but is flexible enough to optimize talent and tools. Tools matter in the studio — not just management tools, but software development and design tools that go beyond programming languages such as Java. We need model-based and more-visual tools to make development faster and easy for more people to participate in the creative process of software development.

The parallel, immersive software studio is different from other methodologies in five important ways:

  1. Software is not code. It creates experience.
  2. Development teams are not coders. They are experience creators.
  3. Technical talent is table stakes. Great developers must be design and domain experts.
  4. Process is bankrupt without design. You get what you design, so you better get the design right.
  5. Software is a creative endeavor, not an industrial process like building automobiles. The methodology is structured to support the creative talent.

This post about parallel, immersive software studio (STUDIO) can’t compete with the 1,813 books about Agile Software on Amazon.com right now. It is just a new idea in a sea of many. But, the accelerating demand for applications that “wow” businesses and users will create new winners and losers in the world of software development. Agile-mania is ending. The evidence: the Agile gurus are now telling us what is wrong with Agile and how to fix it. I say: Throw the baby out with the bath water. Now it is time to get serious. Our craft of software development must be based on creating a continuous stream of insanely great user experiences, not based on the problems of our past. Our value as application development professionals depends on it.

As always, I welcome a vibrant discussion!
Mike Gualtieri, Principal Analyst at Forrester Research

Note
I initially wrote about this new approach in December 2008. It is the result of countless conversations with real software development professionals; observations in many different software development shops, including software vendors, enterprises apps, consumer apps, and core technologies; and a large amount of research in adjacent disciplines such as product design, advertising, and movie making. This research is also based on interviews with business people, product managers, and executives as well as my experience as a software developer for 25+ years using a wide range of technologies and methodologies for small and very large applications including Z80 for operating systems and Java for enterprise apps.

Recommended reading:

  1. The Design of Design: Essays from a Computer Scientist by Frederick P. Brooks, Jr., 2010
  2. The Design of Business: Why Design Thinking is the Next Competitive Advantage by Roger L. Martin, 2009
  3. Bringing Design to Software by Terry Winograd, 1996
  4. Built to Love: Creating Products That Captivate Customers by Peter Boatwright & Jonathan Cagan, 2010

Comments

A well intentioned blog post

A well intentioned blog post but I'm not entirely sure that you are covering things that the Agile methodology doesn't already cover.

First off when you made the statement “working software as the measure of progress” is narcissistic, the next thing that you talked about was code and coding. Then you make the statement "Software is not code. It creates experience". It was as though YOU broke the rules as to what working software is or isn't rather than Agile.

"Development teams must have the confidence and insight to do parallel streams of development." Nothing in Agile is opposed to this as far as I can see. In fact the encouraging a close working environment between business and developers would only serve to increase this confidence.

The statement "Great software talent means renaissance developers who have passion, creativity, discipline, domain knowledge, and user empathy" followed by a rather strange dismissal of the Java programming language could easily be summed up as "Individuals and interactions over processes and tools". This also sounds very similar to "Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done."

In short I think you may be reinventing the wheel with this blog post probably based on meeting people that have taken on Agile without ever even reading the Agile Manifesto (there are loads out there - I've met them).

Laughed my arse off

Reading your blog was the best fun I had all day!

Your P#SS approach has alot of the Agfile values in it. Shame that you cannot understand them:

"The parallel, immersive software studio is different from other methodologies in five important ways:

Software is not code. It creates experience."
- Ya! Remember Agile is about producing Value not code. If you equate Value to Experience (user experience I am guessing) then this is a principle in the Manifestor.
1 - 0

" Development teams are not coders. They are experience creators."
There is nothing in Agile that says that everyone has to be a programmer. In fact, most Agile methods foster a cross-functional organisation not associated with more traditional development methods.
2 - 0

" Technical talent is table stakes. Great developers must be design and domain experts."
Ok. Agreed. Hire good people. Agile isn't for idiots either. Duh!
3 - 0

" Process is bankrupt without design. You get what you design, so you better get the design right."
Agreed. Don't understand how this is a revolutionary idea in terms of the Big Design Up Front and Waterfall methods. Nothing nerw here
4 - 0

" Software is a creative endeavor, not an industrial process like building automobiles. The methodology is structured to support the creative talent."
Ok. Here you read the Toyoto Way and how it was adapted to Agile. What you missed was the fact that the Toyoto Way fostered a creative environment - yes they were making cars - but the fostering of the creative environment is what was picked up by Agile and not how to make automobiles.
5 - 0

Thanks Mike for a good laugh!

LOL

Hey Eamonn. LOL. I'm glad you got a good laugh. I agree that STUDIO (aka P#SS) shares many of the same Agile values. But, it goes further by defining parallel streams of work and developers who rely own their own deep domain knowledge rather than needing incessant interaction with "the business". You can't take a bunch of domain-green developers and just turn the Agile crank to get great software.

By the way,software can learn little from Toyota. Building the same car on an assembly with a very, very long product development cycle is nothing like developing software.

I do appreciate the time you took to add to the discussion. I'm cool with a vibrant discussion about this.

Just a question...

Mike - love these discussions you provoke. But to be honest, I just have to ask: how many software projects have you managed or participated in, in the last 20+ years? It's hard to understand your POV having myself lived thru those challenging years. So much has changed, and 99% (call me a pollyanna) is for the positive. Agile was just a milestone (no pun intended) in a genuine attempt to get control over the software dev process. Its principles are now basically accepted (not the the letter) as machinery to support the most recent movement towards 'Design methodology' meaning human-centric design process that results in a product that successfully meets users' needs.
Turns out these two methodologies work together quite well, IF adopted by the entire organization. For those of us who keep abreast of the best practices of successful software companies (I assume that is you too) it seems like a slam dunk movement in the right direction.

So to my point - instead of slamming some 10 year old manifesto, why not write about what DOES work and why? I'll volunteer to be a source for you.
Joe

Real experience

Hi Joe,
My software development experience is pretty diverse because I have developed operating systems, systems software, enterprise applications, and customer-facing web applications. My first paid programming languages were Pascal, PL/M, and Z80. I developed Windows applications in C and C++. LISP and Prolog kept me busy developing expert systems for air traffic control. although I eventually implemented in C with X-Windows because of performance. I even had a pretty long stint with Dbase IV/Clipper/Foxpro developing logistics systems. The 90's was all about Java and .NET for large enterprises such as AT&T, Bank of America, Avery Dennison, Liberty Mutual. I developed a complex auction billing and revenue distribution system for a firm acquired by Ebay.

I have worked the full spectrum from start-ups to large companies and from software as a product to software as an enterprise app. So, I have been in the trenches and experienced many different processes.

I have observed common threads: 1) There are software developers who really understand the user and the domain 2) There are software developers who focus more on coding. Both roles are very important.

The most disturbing thing about Agile is that is relegates all requirements to "the business". How naive? The busines sis the starting point. But, they do not understand what is possible with technology. Software developers must understand thier requirements and then, ask Emeril would say, "Kick it up a notch".

The STUDIO approach acknowledges that both types of developers are valuable and proposes a way to working to maxi8mize the potential of each.

Did you ever consider that the 'Business' includes users?

Further, "Customer Collaboration" is promoted in the AM. Do you not think that users are customers?

This article smacks of:
* Shock jock yellow journalism
* Do more UX work.

While more UX work is a good thing, it may not be the most valuable thing for all systems. There are many systems where the UX is less important and less fancy. Ever heard of a company called Google?

UX does not have to be seen

Charles,

1. I don't believe this is "yellow journalism" because I have made honest arguments for my analysis and even offered references at the end of the post. Is Agile "yellow journalism" since no empirical evidence is nonexistant or scarce. Please hold Agile advocates to the same standard as you wish to hold me to.
2. Google is a great example of a great user experience because: a) the search results are awesome b) it returns the results really fast. All user experience need not be complex or visually stunning user interfaces. UX is the experience such as that which Google provides in its search.

You did not offer references

1. You did not offer references at the end of the post. You offered "recommended reading." Further, you made no footnotes or endnotes(or whatever they are called -- I don't pretend to know your business because I don't.) to support your blog post from said references.

2. Google is a pretty good experience, and yet, I seriously doubt that they had high priced UX designers, or Immersion efforts like you describe, working on the Google search interface. If you find out differently, please reference your source.

3. "Yellow journalism, or the yellow press, is a type of journalism that presents little or no legitimate well-researched news and instead uses eye-catching headlines to sell more newspapers. 1" Note how wiki footnotes the definition? http://en.wikipedia.org/wiki/Yellow_journalism
You should think about doing the same if you're going to claim that you provided references.

Mike, can you back up your article?

Mike,

I didn't see an answer to this question:

> Did you ever consider that the 'Business' includes users?

How do you propose to get that business to let go of the domain?

In my experience, developers who already have good domain knowledge, would love to push further into the domain. We see it every day. Businesss users, who largely control domain knowledge and access do not want them there. They still want to maintain some control over the output that is delivered. It's simply a fools mission to think that in any organization larger than a startup that the business experts are going to capitulate that responsibility to a development team.

There's a second problem which is that of a regulated environment. In regulated environments you will have no checks or balances between the business and development. If the development team takes on the domain (a good thing) but then takes on the responsibility for final decisions of what happens as a result of that domain knowledge (dangerous) you will lose all credibility in a regulated environment. There has to be some separation. Now, I would argue we could change that mindset, but that lies well beyond the purview of the development process.

Great discussion. Sorry I'm late to the party!

Agile Development versus Business Technology

Hi Mike, in reality we need an AGILE business and not just AGILE development. I propose that agile development does not create an agile business. It just makes the development more user focused but the process in itself holds the business back.

I agree with a lot of things you write, but I feel that the main outcome of this should not be that we need SW developers to be UX designers and domain experts. I have found that this never works. People who write great code are usually withdrawn and in the end care little about the people who will use. The problem has to be solved by NOT HAVING TO WRITE APPLICATIONS in Java.

Look at the impact of mobile client server. Trying to have a dev team on the go to make the same app work on an unlimited set of frontend devices is impossible and hope for this to be agile in anyway is futile.

The future must be application systems (like our Papyrus Platform) that provide a business studio to non-technical developers. What many vendors call a Business Studio today is however a developer tool. Developers need to write application SYSTEMS in which non-developers can model and define the data, content, rules, processes and UX that the business needs. I am still fighting the BPM flowcharting crowd because that is already too rigid for the needed agility. What the business needs is ultimately defined in a Business Architecture. So the platform has to expose the Business Architecture and link goal-oriented, application processes right into it. ONLY that is Business Technology. Coding for user frontend applications is a dead-end. Coding for back-end transaction systems will obviously always be necessary. But that is not holding business agility back.

More here: http://isismjpucher.wordpress.com/2011/10/17/goal-orientation-in-process...

STUDIO methodology for Business and UX needs

Hi Max, Thanks so much for your comments. I only, very partially agree with that software developers struggle to be UX designers and domain experts. My STUDIO methodology proposes that software development projects split into parallel streams: 1) Traditional coders break off in multiple streams to do traditional coding activities whose requirements are well known like database access and integration 2) A developer who are domain experts and UX experts work together on the business process and human interfaces.

A key problem with Agile is that it assumes all developers are created equal. They are not. Some know how to write rote code and others are creative domain experts.

Ideally, development tools will make it easier for domain experts to become developers with understanding the ins and outs of technical programming platforms such as Java or .NET. The 4GL crowd was headed in the right direction in the late 80' and early 90's.The Web changed everything and development tools went medieval.

We need more tools that abstract technology to let creative domain experts do development. One caution: Many business domain experts seek to duplicate what they do now rather than to improve process and customer experience.

Yet another idiotic statement by the author

> A key problem with Agile is that it assumes all developers are created equal.

Mike, this just further proves you have no idea what you are talking about when it comes to Agile.

I'm willing to listen to your argument on this one, but I can already tell you...It's weak!

(I promise to change my view if you prove me wrong)

Show me an Agile method that classifies talent

Charles,
Developers make design decisions (at varied levels of granularity) as well as code. Those design decisions and how they code have consequences on the speed and quality of the project. Anyone who has ever been on software development projects knows that developer talent and experience varies greatly.

Agile places no emphasis on classifying talent and assign work appropriately. Instead Agile methods like SCRUM oversimplify how resources should be assigned to user stories. As I say in my post, this is a very immature approach. Experiences development shops know that talent and experience matter and that work should be done in parallel.

Mike

Further evidence of the Author's lack of Agile knowledge

None of what you just said supports this statement by you:

> A key problem with Agile is that it assumes all developers are created equal.

I think you should try again. Where in the "Agile Manifesto" does it say that we should assume all developers are created equal"?

"Assign work appropriately." Uhh.... that's been tried before in Waterfall, and even though the Agile implementations are relatively weak out there right now(as you correctly point out in your post), mediocre Agile is kicking Waterfall's @ss. See: http://www.mountaingoatsoftware.com/blog/agile-succeeds-three-times-more...

(Cue the part where Mike says that study is bogus)

Why compare Agile to waterfall?

Advocates of Agile often compare it versus waterfall. Waterfall is not widely used in its pure academic form. Agile looks good compared to waterfall which is why so many agile advocates do it. In my opinion, it is a lame comparison since it is not the most widely used SDLC. My post gives the detail of where Agile goes wrong.

What should we compare Agile to, Mike?

What should we compare Agile to, Mike? Can you cite any studies that compare Agile to other software approaches?

The sad truth is that Waterfall and RUP are pretty close to the same. RUP just does small waterfalls for the most part.

You're an analyist, what is "most widely used SDLC" ?

I thought I was the only one!

I really did. May of the techniques used by agile have been used for decades (at least in my experience). The idea of a "manifesto" in regards to an engineering methodology just degrades the whole profession. In addition, agile is really a software production methodology, not a software engineering methodology. There are still applications where there is no user experience, or where that experience MUST be well defined. One concern, especially in reference to design, is that this mania about code delivery can create silos of information, since the agilistas are somewhat hostile to real software architecture.

I was talking to a high level manager of a large company recently and he had some really good realizations about agile. The company had said, six years ago, that all software development would be agile. They did not really know that that meant, but six years later there is still about 10 - 15% of the products put out by this company that are not done in an agile way. He also had some very good observations about what was good in agile and what is not. The biggest problem I have with agile is that they do not like tools. I have foound that using real software engineering tools makes a tremendous difference in the quality and usability of a software product.

Along the same lines, I was at a conference where there were two talks given regarding agile. One was by a true agilista. He really pushed the used of index cards, etc. The other was by a top scientist at IBM Rational (full disclosure, I worked at Rational a couple of decades ago). Of course he was talking tools. He was also asked about user input during a sprint. The answer was no, after the user involvement at the front end of a sprint, the developers begin to do real engineering. This does not require user input. At the end of the sprint the user, of course, should be involved.

Another indicator that agile is perhaps not all it is cracked up to be is that they are now talking kanban in conjunction with scrum. Bringing in an automotive, or factory, production control technique into the software development process shows that there is a real problem from a theoretical point of view.

My impression is that the agilistas do not have a wide range of experiences in the software world and do not have a scientific or engineering backing for their theories. Thus the manifesto. When I have pressed organizations on what agile really gets them, it is generally better people management. As I mentioned above, I have seen many of the pieces of agile used in the past (as long as 35 years ago) in more sophiscated companies. I have also seen waterfall techniques used to good effect and have seen them fail utterly. It is usually a failure of management that causes that.

I could go on, but to summarize, agile pulls from many successful techniques that have a long history in other fields. Agile also attempts to shed many of the advances made in software engineering through their hostility to automated and computer based tools. By using the language of manifesto and of production control techniques agile denigrates the profession. There are some good things here, but sometime it is hard to find.

Now there are two of us

Thanks for your comments Louis. Fantastic insights about the reality of Agile. I agree with everything you have said and particularly like:

"My impression is that the agilistas do not have a wide range of experiences in the software world and do not have a scientific or engineering backing for their theories. Thus the manifesto. When I have pressed organizations on what agile really gets them, it is generally better people management. As I mentioned above, I have seen many of the pieces of agile used in the past (as long as 35 years ago) in more sophiscated companies. I have also seen waterfall techniques used to good effect and have seen them fail utterly. It is usually a failure of management that causes that."

Back to the Future

Get over yourself.

If the Studio model is "what's next", then welcome to the '90's. Many of us were doing multimedia games then, and we borrowed almost exactly this "new" model from animation studios. Software was just another aspect grafted onto the already established, highly parallel, collaborative, design-driven movie-production process. People who know software's history know the difference between these models, and which to use when.

The current hype of Agile is sometimes galling, but the approach has its place, and that place is not small. Besides which, it needed the hype to get traction, because, unlike Gaultier's suggestion, it is actually new.

Yes. Studio borrows from creative processes

Jon, Thanks for your comments. I guess we can argue about what makes something new and whether or not anything is actually new. New is often associated as differing from the norm. I have used a variation of the Studio approach for twenty years. But, technology, tools, and processes do change and methodologies need to change too. So, I think it is too dismissive to merely say that STUDIO is merely a rip-off of the design-driven movie-production process.

I agree that Agile is new. However, it is new in a very bad way as described in this post.

Sounds familiar

Unlike other people, my blog entry here: http://methods-and-music.blogspot.com/2011/03/agile-four-tenets-false-di... was written before I saw this, but I am saying similar things.

X over Y statements

Hi Keith, Great post analyzing agile tenents - X over Y as you call them. I encourage everyone to read http://methods-and-music.blogspot.com/2011/03/agile-four-tenets-false-dichotomies-or.html#!/2011/03/agile-four-tenets-false-dichotomies-or.html

“Business people and developers working together daily” is lazy.

I couldn't disagree with you more on this. You give the impression that any and all sundry of business people are getting together with developers. In reality it is the Product Manager (aka Product Owner) who on behalf of the business has synthesized information from various quarters (competitors, market, technology & cultural trends etc.) and hopefully has speculated the product vision supported by some form of roadmap in order to shape that which needs to be developed on a prioritized basis. To think that product management is an exact science and not a mix of art and science has been the folly of those who think that the business knows "exactly what customers want or need".

So the iterative approach, core to agile practices, as a process appears akin to the navigational process of dead-reckoning allows teams to pre-test the customers as to whether what they are getting is in fact what they really needed as opposed to have wanted but then not needed. I find that you too readily allow "business people", lets just say this is the euphemism for Product Managers, (knowing full well that there are other business function roles in the value stream of product development), an easy way out by claiming "often the business people who are involved are not the same as the users, and they are not domain experts or design experts". Sure they are not design experts, and following agile practices does not mean they have to become experts in this facet of development, though they too can have an informed opinion in this area based on what they are seeing in the broader market. As for not becoming domain experts or having the ability to get into the end users head and assimilate how they are going to use or not use a product is does question how do you expect the product to succeed in the market place? Developers are not infront of customers convincing them to the benefits and value add product features have. They are not the ones who have captured the eco-system and workflow within which the end user is likely to be using the product. In fact Product Managers may not fully understand this until they get infront of the customer and demonstrate the product capabilities, and with some too and fro get a better idea as to what they really needed. This is what agile practices better addresses than does the old, tried and tested waterfall methods.

I do agree that software product development is more akin to making a movie, and as you know in movies these days they audience test many things before the final release, including giving the end user control as to what kind of ending they would like to see. In essence iterative development promotes all this.

If that is what happens, it isn't what most people call agile

Tarang, That might be what happens in reality, but the fact is that most agile methodologists want actual users involved. And this can be a very good thing, if done well. Product Managers are, to many, just a symptom of the sort of bureaucracy that agile so hates. But for involvement by the business people to be relevant, they must be representative, they must be allowed the time needed to be involved and they must be able, emotionally as well as intellectually, to answer questions that can veer from sounding naive to challenging the core of their jobs.

Iterative process good in some cases

Thanks Tarang for your detail comments. For software development companies, I think you are right the the businesspeople are product managers. I also agree it is their job to represent the future of the software product that is being developed. In enterprise IT, the user is often business analysts or a business domain "expert" who may or may not represent the users. So for software development companies, I agree that the product manager represents the roadmap of the software. It is their job to know what the users want and what they will buy or adopt.

Iteration is useful when there are unknowns. But, many, many requirements are absolutely known. In fact, product managers will tell you that they know exactly the features that they need to either be on par with competitors or to beat competitors. In these cases, no iteration is needed to figure out what is actually needed. You know what needs to get done, so you do it in parallel.

In my approach I call STUDIO, I propose that the knowns and unknowns are listed. The knowns are developed in "parallel". The unknowns are developed in the "studio" which is iterative.

I do not think actual reflects the complete reality of software development. Is it an improvement over pure waterfall. Sure. But, Agile suffers from the same problem as waterfall - it an extreme in the opposite direction. That is why I have proposed this new approach.

Very well written and highly

Very well written and highly thought provoking!!

I love the idea of creating code visually & easily!! Then we can truly
create renaissance types of designers-coders who can create magic
in the world!

Renaissance Designer-Developers

Thanks for your comment Shraddha. Yes! If we provide better development tools for domain experts and business people then that will help the renaissance developer emerge. And, if developers make domain expertise a goal then they can use their traditional tools to become renaissance developers. The experts seem to want to silo coders and domain experts. They say they have to work together to get results. I am all for that. But, the next step to make developers and domain experts one in the same. This is a valid approach for many types of applications. Although I fully acknowledge that much software such as operating systems, tools, and platforms must be developed by deeply knowledgeable software developers - that is their domain expertise.

"if" is now and domain knowledge rules!

Mike
As I hope you will find out the tool now exists the developer is the "analysts" not a coder for all business requirements. This "next step" is not just helping the "renaissance developer emerge" it is the start of commoditising business software. As for "operating systems, tools, and platforms" quite right they require clever coding but remain behind the doors of such vendors and released to deliver "commodity" products

Sorry, but all I hear is the

Sorry, but all I hear is the whining of a Analyst looking, once again, to blame developers for their own shortcomings. If Scrum/Agile is a very developer centric process, it's because it has to be. Business actors like yourself constantly blame developers for all the woes of the company "Can't deliver on time, can't meet spec, You're over budget, Didn't make it like I want it". If you want developers to be accountable for delivery fine, you give them the tools that allow them to succeed, not by crippling them with inane archaic processes that are utterly divorced from the reality of software development. Software is not a construction project. I laugh any time anyone tries to use Prince2 because there one thing software certainly isn't and that's predictable.

Here's a bit of truth. Your requirements documentation sucks, I know that even never having read it. Having written 80 pages of prose has already made it obsolete. I know as many unicorns a I do good requirements documents.

As for lumping "developers" with the responsibility of design, I think you need to come back to the 21st century. A modern UI is a complex thing requiring it's own discipline as much as development. Steve Jobs / Jonathan Ives are not developers or engineers, they are designers, it takes a different mind. You might one day find someone who is as skilled a developer as they are a designer, when you do please pay them double and send me their profile, I'd like to introduce them to my friend the Unicorn.

Your Assumption

Craig, Thanks for commenting. First off, I have been an analyst for four years. Prior to joining Forrester I was a developer for 25+ years. I noted that at the end of the post.

I actually agree with you that many "business actors...blame developers for all the woes of the company.". That's a key reason why Agile is dangerous because it assumes that by bringing business people into the iteration mix will solve all problems. You are actually making my point that Agile is used by developers so they don't get blamed. You'll get blamed anyway.

I learned that if you develop software based solely on what business or users say they want, then you will not get paid. You have to look beyond what they are saying and have a deeper understanding of the domain you are serving.

Also, the idea that developers don't understand many of the requirements is hogwash too in my opinion. Some things you understand completely and can develop that in parallel. See my proposed new approach called STUDIO that puts together parallel development and iteration in a studio.

Developers and Designers working together is the key

In the studio for mobile development we created, we didn't even try to make developers understand more about design or vice-versa. What proved to be key to improve our odds of building successful products was the fact that we made them understand they were working towards the same goal. More here: http://www.uxmindset.com/2011/11/lean-ux-agile-development-the-next-big-...

Agile development

I'm too busy writing code to fully formulate a reply to your critique of agile development, but I must take exception to one overarching assertion you make. Great software is developed bt individuals, not teams. Every meeting I'm obliged to attend (of more than three attendees) I regard as time wasted. "Designs" are usually tooconfining to be helpful - just let me contact the individual with a problem, and allow me to iterate with her through a few solutions, and we'll soon develop an elegant, robust, maintainable solution.

It resonates with what we do

Hey Mike, hope all is well!

We spoke over the phone a few days ago and as I told you, I think the post was great to pinpoint some of the struggles business units inside companies will have if they look at Agile as the silver bullet for building successful software. I used RUP a lot, then transitioned to Agile in 2002, then lean, but figured out more recently the only way to build successul software is by leveraging a solution finding process driven by designers and developers working together.

I finally finished my article about the combination between Agile and Lean UX, a process that makes it possible to implement this studio-like "entity" you mention in your article. As a matter of fact, our mobile studio works exactly like you described, with a few minor differences.

Here's the link to the article, hope it's helpful: http://www.uxmindset.com/2011/11/lean-ux-agile-development-the-next-big-...

Marcio I added this to your

Marcio
I added this to your article and I hope it will be Forrester that tells the world the game is changing after all why are we still in the 21st century coding business logic that never changes?

“Trick is to eliminate need for coders so the gap between developer and user is eliminated. Fact is business logic never changes it is people working individually and collectively to create business outcomes by doing something....That is a "new paradigm" - read about it here on one page http://bit.ly/nQOAzE. Users are encouraged to contribute not just at the build but subsequently as change is readily supported as the core code never changes and no code generation or compiling.
On top of that users want easy use of forms to enter the information only once and thus create one version of the truth. Working forms need to be simple i.e. no flashing symbols etc to distract doing the job. If you want designer web pages for external show that is a different job?”

Publish STUDIO ideas in dedicated post

Mike,
I wish you had published STUDIO idea in a separate blog (without Agile criticism) to avoid religious debates. The STUDIO approach looks good and valuable. It can be applied as an 'environment' (or culture) for formal processes, either scrum or iterative or rup etc.
I hope, the STUDIO will live on, do not let it disappear!
Thanks,
--Alex

I agree with Alex!

Agile doesn't need to be dropped for us to see an incremental improvement in software development process. Agile itself was a huge improvement to previously available practices and processes.

In less than one year of practical results, we already see the benefits of working in a studio model where agile teams play an important role on predictability, quality and performance.

Thanks for encouragement - STUDIO post

Hi Alex, Thanks for encourage to write a separate post on just STUDIO. I have to work through some other deadlines, but will publish in a few weeks.

Missing the future

What Agile lacks, and what you failed to mention, is the fact that software you build that has
any long-term value needs to "live". The key problem with all of these technologies is that they
ignore one fundamental of software lifecycle cost, which is that most of the cost is in maintaining
and modifying the code.

We still write code like I did on the PDP11/40 in the 1970s. We create directories of tiny little
sand files that contain only the source code, and possibly some comments. From that code
we are expected to maintain and modify these systems, which are the very lifeblood of the
organization.

What we miss is the story. Why was the code written? What were the design decisions?
What are the tradeoffs?

Imagine a calculus textbook. Create a directory for each chapter. Collect all of the equations
from each chapter and put them in tiny files. Throw away the textbook. Now try to learn
calculus from the files. THAT's what we do today with our code. We throw away the most
valuable part of the problem.

What we need to do is write code that "lives". Literate programming provides the primary
tool. Imagine hiring a new programmer. You give them a copy of the literate program, give
them a couple weeks of vacation time in Hawaii to read the code. When they come back
they are effective programmers on your project. If you can do this, what I call the
"independence test", then you have written a literate program.

Literate programming is not about the tools. You can do it in any language. I have written
an example using straight HTML
http://axiom-developer.org/axiom-webiste/litprog.html

Great analogy!

Tim, just wanted to say that the calculus book is a great analogy. I'll probably steal it myself - if I do, I'll try to remember to credit you!

Programmers are detectives

Hi Tim,
Great point about the need to modify software. Many methodologies, including Agile, don't specifically focus on how to do that. I could not agree with you more that "What we miss is the story. Why was the code written? What were the design decisions?
What are the tradeoffs?".

I have been that programmer many times where I had to figure out huge amounts of code to figure out what it did. I always said that programming is like detective work. But, it shouldn't be.

I'd love to see your example, but your link is broken.

Thanks so much for adding to the discussion.
Mike

Fixed Broken Link

http://axiom-developer.org/axiom-website/litprog.html

Next time you get into a discussion about programming, point out that:
"The best programming language is English". Anything else is just
notation.

Business Software no longer needs custom coding?

Suggest you research here http://bit.ly/qlSUvM the "holy grail of software" has arrived - but not in US! As a result constant change supported in effect "live" software?

Before you criticize, make sure you understand the author.

Agile is not a cop out! It has a time and place.

You state "[Software] WORKS in conjunction directly or indirectly with other software, technologies, the environment, business processes, and people to create customer experience and/or improve operational efficiency."

Absolutely true.

That is why WORKING software IS a measure of progress. If it is not working at full [desired] capacity, i.e., if it is operating in isolation or out of harmony with any of the areas you identified in the above statement, the delivery of capability achieves ** LESS ** progress than it needs to have, and thus its completeness (which can be measured in objects of code) IS an appropriate metric for progress of the effort. It has nothing to do with lines of code except that there is a cost associated with writing a line of code, and thus the number of lines of code is one measure of the level of investment consumed, and thus the actual cost (accounting value) of what was delivered.

As far as business and developers working together daily, it is no different than waterfall development, 6 Sigma, or Lean when those programs are run by a competent project manager and supported by appropriate SMEs & business analysts. Business and IT always need to work together, more frequently in the beginning, perhaps less frequently later, but that depends on the project. What the Agile development brings to the table is a format in which communication can occur on a scheduled basis, without frequent interruptions of the sponsor or other business people, who remain responsible for getting today's work out the door. Expedited feedback that can reduce the development expense with more timely answers from the sponsor or the sponsor's delegate to requirement implementation questions, avoidance alternatives and work around preferences for technical constraints, and resolution of matrixed and cross domain ownership issues that arise during development. The PM still needs to set and manage proper expectations with all project stakeholders if the scope or expectations around chartered deliverables changes as a result of this communication, whether it occurs in a daily standup or via phone call. Again, nothing different than the normal PMI-advocated project management practices.

Unicode Systems possesses a

Unicode Systems possesses a strong team of software developers who specialize in the development of custom software applications.They are expert in the wide range of custom programming skills involving the latest and most effective development technologies like JAVA & PHP and many more. We, at Unicode Systems strive hard to deliver the services that are best in the field fulfil all the parameters of your needs & desires.
The services that we offer are:-

* Client-Server Applications
* Distributed Applications
Unicode Systems

Coding custom business solutions heading for history book!

Well maybe there are some solutions that will need clever coders but business logic is now catered for with new approach where the code never changes yet builds custom solutions. It is agile software a new alternative to COTS and commissioning coders. It will be game changing so Unicode you model will change. See here http://bit.ly/nQOAzE as the customer becomes intelligent as to what exactly they are buying into? Maybe Forrester will one day “get it”?

Interesting article... not quite sure I agree...

The main criticism you have with Agile is that it makes it "developer-centric." I've found in practice that this is not the case. I think your true aim--the complaint about user experience--is placing blame in the wrong place. It isn't just Agile that neglects user experience--these requirements can be missed using any process.

By any measure--working software is better than no software at all, and therefore I will fully agree with that particular tenet. In most cases--where user experience is ignored (again, nothing to do with development methodology) these requirements get tacked onto the project and prioritized alongside of everything else used. Agile's key insight is that "engineering" is the wrong word for developing software--writing is the correct word. It's a process of revision, like writing a play or a novel--NOT like designing and building an office building or skyscraper. Usually even good Business Analysts don't know exactly what they want. Colocation solves that problem. Another critical insight is that by nature--developers are perfectionists. We want to optimize everything, when in 99% of cases, non-optimized code is perfectly acceptable. Agile forces code tweakers to move on to the next requirement instead of wasting money tweaking something to perfection. (No performance requirement? Out of scope.)

The biggest boon to the business/developer relationship is Agile's recognition that all requirements need to be prioritized-->if a new requirement appears, then what OTHER requirement are you going to sacrifice to get it? This isn't handled as explicitly in other methodologies. It gets the product owners realizing that you can't just expect a developer to code EVERYTHING you want in the last 400 developer hours.

Most importantly, I've found that agile projects really do break down a huge communication barrier--it's not that developers can't know business logic--the problem is that we are finite and extremely expensive resources, and anything that prevents us from writing code IS a critical problem. If I have to wait for a phone call or an email for an answer, then guess what, you're probably paying me to twiddle my thumbs, especially if the question is in regards to a critical decision. Colocation is the be-all-end-all in communication.

In conclusion--if you want user experience as part of the software project, don't blame the methodology. Bring users in and colocate them with your analysts and developers--problems identified in projects with colocation (regardless of methodology) will ALWAYS be solved at a faster rate than in the normal "email chase" I've seen happen in projects that don't recognize face to face communication. Agile isn't a panacea, but in my mind it certainly is the most practical "real-world" methodology I've been exposed to.

You all realize that Agile

You all realize that Agile and CMMI and (now) STUDIO are simply mechnisms to entice people to buy consulting services, books, articles, magazines etc., right? There is always a better way to skin a cat, at least someone is always willing to tell you that and sell you that.

There are really only three truisms around this enitre subject:
Good Process Design is important
Good Process Execution is CRITICAL (and people execute process...so find the best people possible to ensure success)
Someone is always looking to take your money if you are willing

Oh...and a fourth...eventually these manifestos, practices, and standards will be adopted by the government (of whatever juristiction) and will drive overburdensome and conflicting contracting requirements that escalate cost and confusion while lowering quality.

Out...

There are always vested

There are always vested interests in software development methodologies that artificially pump up the results to sell consulting services, books, etc.... as you say. You can hardly disagree with many Agilista's because they will try to dismiss any new idea of criticism of Agile (or RUP back in the day). It is particularly hard since it is very, very difficult to run double blind studies on software development projects. Who is going to fund two development efforts and find the causation?

The fact is that methodologies build on each other and vested interests often don't want to let go because it means letting go of profits too.

Anyway, I agree with you that good process design and execution is important. However, I argue in my post the software design is just as important.

Why didn't you mention "good software design" in your three truisms? A huge point of my post is that Agile largely ignores that.

I really appreciate your comments and discussion.

You may have seriously missed the point

Hi Mike,

After reading through your article twice, I can't help but think that you've not really understood the point behind the wording of the Agile manifesto. I do agree with you however that Agile is suffering from "fad-ism". More to the point, Agile is presently suffering from an image and identity crisis on a grand scale, mostly because the majority of people jumping on the Agile bandwagon are treating Agile like it's some sort of validation for complacent development and business practices. That hurts the image and the efforts of those who get what Agile really is, and that's as both a philosophical and a pragmatic approach to the software development process.

The two points that you've used to support your point of view are weak arguments at best, and provide a very narrow point of view. Implying narcissism is as you yourself mentioned a fairly strongly worded view that software developers want everything to focus on them and have everything their own way. The point is that the Agile manifesto is written and intended to be quite the opposite. Using working software as a measure of success isn't about measuring what a software developer knows, but about measuring the success of a project based on whether or not it solves the customer's problem. Which is to say, if the software solves the problem, it works, and therefore it is successful, whereas failure to solve the problem means the software doesn't work, therefore it is unsuccessful.

As far as your point about developers and business people working together being lazy, you hit on something that is both correct and incorrect in a sense. In the spirit of the Agile manifesto, the business people are supposed to be the domain experts. Frequent meetings are intended to provide feedback often, allowing a project team to learn and adapt at a rapid pace. Meeting once a month or at the beginning and end of a project doesn't provide all stakeholders with an opportunity to avoid critical and costly mistakes early, nor does it allow decision making to be deferred to the last possible minute. Meeting frequently, and more importantly acting on the issues raised requires a high degree of engagement and commitment on behalf of all of the parties concerned, and is the opposite of lazy. Meeting twice and leaving it up to everyone else to figure things out for themselves on the other hand is about as lazy as it gets.

Now, here's where you mentioned something that was partly correct. Laziness does occur, regardless of whether you are an Agile team or not. In the case of many teams that call themselves Agile, they don't take the time to learn how to apply a methodology properly so that it will give them the agility that the manifesto represents. Agility isn't about processes or practices, yet so many people think that if they are pair-programming, or if they are using a Kanban board, or if their process is anything except the Waterfall method, that it means they are Agile. Even worse they evangelize, promote their misconceptions. There is a view that being Agile means you don't need to design your software, that design emerges through refactoring. This couldn't be farther from the truth (or from reality). The thing is, that design doesn't all need to be done up front, and that it is actually costly to design everything at the beginning because it commits you to a course of action that becomes much more costly later if change is required. Some design MUST occur in the beginning to be sure, and some design can emerge, but the real point is that you can and should delay design decisions until you actually need to make them, when the cost of commitment will be less because you will have learned more about your system requirements as the project is developed, and can make those decisions based on knowledge rather than speculatively. Mary Poppendieck's "Lean Software Development" book goes into this concept in great detail, and in some ways epitomizes what being Agile is supposed to be about.

Like yourself, I've been in this business for a very long time, and I've experienced all of the methodologies, both applied well, and applied very poorly. When Agile is applied well - and more importantly when it is understood by ALL of the stakeholders - then the software development process works very well. To apply the philosophy well however requires a high degree of discipline and a very open minded approach, and this (in my experience) is where the majority of Agile failures occur. Waterfall, Scrum, and even the STUDIO approach you have mentioned are all simply methods for approaching software development, and they are all largely the same in how they attempt to codify and organize the software development process. What each of the methodologies lack however is a guiding principal that defines what it is the process is intending to achieve, and that is where the Agile manifesto comes in, and what makes it such a powerful statement, not just for the programmers, but for the business people alike. Rather than showing weaknesses, the manifesto shows a maturity of thought that goes way beyond the "methodology wars" that seem to have been occurring in our industry over the last decade.

Where the Agile philosophy really stands out is that it is solution-centric, and not programmer-centric or business-centric as many seem to think it is. To make the issue about which group of people a methodology should focus on is IMHO very narrow-minded. As programmers, we aren't here simply to create user experiences, satisfy business people, or even to simply write software. These are all aspects of the job to be sure, but the real focus of our work is to provide solutions to complex problems, and to use whichever tools are at our disposal to do this to the satisfaction of all of the people who will benefit from the solutions, including the users, and also the people paying us to do the job.

One of the great concerns within the industry is with the analysts and other middle-management types who fear that Agility means they risk losing work, and that their skills will become superfluous. Personally, I see it as more of an opportunity to learn new skills, or to apply existing skills in a way that supplements those methodologies that may seem to eschew the analysts, by learning to apply knowledge and skill in a more iterative manner, and to learn to be more selective in the work-output that these roles would normally produce.

I'll admit that I was pleased to locate an article that offers a criticism of Agile, but I think if you are going to be critical, it would be better to provide a much wider point of view for the betterment of the philosophy itself, and not simply to IMHO incorrectly pick flaws with a small part of the manifesto and assume that the implied flaws invalidate the philosophy while offering another methodology - which can quite easily fit within the Agile paradigm if managed properly - as a counterpoint.

No hard feelings, and I love a lively debate, however in this case I think you essentially got it 'wrong'.