Measure Software Development By Predictability, Not Just Speed

It feels as though the word "value" has appeared in more discussions about software development and delivery than in the previous two decades. We see this increased demand for immediate, tangible value across the entire range of technology producers and consumers. The dubious value of legacy applications, which have grown like kudzu, is the impetus for many painfully difficult cutting and pruning jobs within IT departments. Faster realization of value is driving more applications and infrastructure into the cloud. Software vendors are realizing that, while revenue is vital, the long-term relationship with the customer depends on the mutual value that both parties think they're getting from the relationship.

If we measure software by value, instead of cost, revenue, completeness, or other possible measures, we have to measure the software development process in a complementary way. What characteristic of software development is most likely to generate a valuable result? If your answer is "speed," think again. Predictability is a much better measure.

At the IBM Innovate conference last month, Walker Royce made a very plausible case for valuing predictability over velocity. Here's his keynote address, which is definitely worth watching.

It's easy to operationalize what Royce is talking about, since we've all paid the cost of unpredictability. When a project jumps the rails, the business problem that was the target of that project remains unaddressed. A frequent culprit is the software vendor that fails to deliver on its promised road map. While the salesperson usually takes the beating for slipped release dates, the source of unpredictability lies further within the organization. It could be that the release is ready, but the customer-facing parts of the company (sales, marketing, support, etc.) aren't yet capable of delivering it. Or, the product development team might have honestly tried to make its deadline, but software development being what it is, ran into unforseen problems.

Frequently, unpredictability is not a single event but a chain reaction. Anything that can reduce unpredictability at the beginning of the value chain creates positive results all the way to the end. And that's why, as I strongly agree with Royce, one of the chief benefits of Agile is enhanced predictability.

We've been hearing this refrain for a while. For example, in a 2009 survey, 62% of respondents said that their reason for adopting Agile was greater velocity, and 55% percent cited greater predictability. That's not a huge difference, especially given how the familiar description of Agile stresses velocity, velocity, velocity. I suspect that, if we asked the same question today, velocity and predictability might be even closer together in the ranking of Agile's benefits.

As new methodologies or movements emerge, such as DevOps, I'd advise them to start talking and thinking about predictability. DevOps is at the point where Agile was several years ago, a great idea still coalescing into a more clearly defined methodology, a movement still gathering adherents, and a strategy still building a portfolio of success stories.  And, just as Agile did in its early years, DevOps is talking way too much about velocity.

When DevOps advocates talk about speed of deployment to the exclusion of all other possible benefits, they make their campaign harder than it should be. Sure, it's great that Flickr has overcome its DevOps challenges so that it can now do 10 updates a day without breaking a sweat. Of course, not everyone is like Flickr, or wants to be, or can be. The development and operations teams responsible for applications in highly regulated industries might not be impressed by that statistic. In face, it might turn them off completely, if it seems that DevOps threatens to blow past governance and safeguards.

Actually, by "might turn them off," I mean "does turn them off," since I've already had occasion to dispel some misconceptions about DevOps among IT people. Obviously, they don't want to deploy new code more slowly than they need to, but they also don't want to join some harebrained cult of speed.  DevOps does have other potential benefits, including greater predictability. Now that's a bridge between development and operations that both sides can agree is built on the right foundations. 


Just a bit of misunderstanding....

Some points that should be clear, that run counter to what you're putting forth in your article:

1 - You are confusing terms, and you're not the first one to do so. "DevOps" is *NOT* the same as "Continuous Deployment/Delivery" development methods. One can have a DevOps culture (it's not a method, tool, or process, BTW) without deploying code to production frequently. You can argue that organizations that do practice Continuous Deployment are businesses that enjoy a healthy amount of cooperation and collaboration between Dev and Ops (IMHO, it's a requirement) but they are not the same.

Given that, I think what you really mean to comment on is Continuous Deployment, not DevOps, so I'll add some more things that could help...

2 - Orgs that do practice Continuous Deployment (like Flickr and Etsy) are not doing for SPEED. They're not doing it for bragging rights. They're not doing it to give talks at conferences. They're not doing it to be cool, be hip, or be on some imaginary bleeding edge. They are doing it to support the business, and no other reason. Small and frequent deployments (done well) are intended to provide greater Availability, Peformance, and the word you started with: "value". Be very clear: if the business didn't benefit from deploying small code changes frequently, then those organizations wouldn't do it. (I worked at Flickr, and now work at Etsy). Businesses that are in this scary imaginary "cult of speed" you mention don't exist, and people who believe that successful organizations are doing it for non-business reasons are dismissing out of fear and not taking the time to understand what Continuous Deployment means. The number of deployments per day isn't a goal. Enabling the business to evolve and grow is the goal, and Continuous Deployment is only one of the tools to help that.

3 - I'll argue that "governance and safeguards" exist maybe even more in Continuous Deployment environments, because of how it works. Saying that governance and safeguards don't exist there assumes that deploying small changes frequently happens as part of a WildWest, cowboy, anything-goes process, which couldn't be farther than the truth. Knowing the Who, What (i.e. the detailed diff), and When (down to the second) code changes are deployed is a known part of Continuous Deployment. ITIL, rigorous Change Management, and separation of duties can absolutely co-exist in an organization that practices Continuous Deployment. I'm certainly aware of many organizations who have a lack of those Who/What/Why audit details, and deploy quarterly.

4 - Many companies other than Flickr (or Etsy) have been deploying small and frequent changes in even much tighter security and compliance environments. Amazon just said publicly this year that they deploy production changes (on average) every 11 seconds, and Facebook also have posted a detailed video about how they also hedge their business risk by deploying small changes frequently.

Fair points, John, to a point

Your response to the short section on DevOps is a lot longer than the section itself, so you may be reading things into it that aren't there. I almost made that section longer, in part because every conversation about DevOps seems to require a clarification about what it really means, versus continuous delivery, continuous deployment, etc. My blog posts often run a little too long, but maybe this time it needed to, if the distinctions weren't clear.

No, continuous deployment isn't being done for speed alone. However, that's the one business benefit that gets stressed. The first part of the post says something that you haven't taken issue with, that velocity is not the only business good. Agile from the beginning wasn't only about speed, but it took a while for that message to diffuse outside of the small group of people who really understood Agile. To outsiders, it sometimes did sound like a "cult of speed." Although they may not always have been listening carefully, I wouldn't blame the outsiders completely for that misunderstanding.

So, I'm in vehement agreement with a lot of what you say. Continuous deployment is not a Wild West approach, just as Agile requires a lot more discipline than people sometimes realize. What does concern me, however, is the way that we talk about both continuous deployment and DevOps. For a long time, social media proponents used the same example, Dell, so many times that it started to be counterproductive. Your company may not be like Dell, and you might begin to wonder why you're not hearing about companies other than Dell, if this social media thing is so great. Similarly, we hear a lot about Flickr and Facebook's rapid rate of deployment, and less about other companies. At some point, citing these examples too often will get counterproductive, too. (Not to mention how Facebook suggests sloppy privacy rules that put security and compliance people on their guard.)

One quick word about DevOps: It needs to have a clearly-associated methodology. You might say that Agile is just a set of values, too, but it definitely helps that it has Scrum, XP, et al. immediately and obviously associated with it.

One other quick word about DevOps: Out of curiosity, I checked the Wikipedia page to see what case studies it mentioned. Care to guess?

Yep, you hit upon a hot

Yep, you hit upon a hot button of mine, which is the common misunderstanding that continuous deployment is the Wild West. And yep, I'm aware that Flickr's often mentioned. :) What is sometimes frustrating is that in that 2009 Velocity talk, Paul and I went to pains to point out that you can't say that you deploy 10x per day if you go *down* 10x per day, and that it's about the business.

I don't actually agree that there needs to be an associated methodology with the term "DevOps". I personally think it can be ok that it doesn't have a manifesto and doesn't have any best practices other than the CAMS acronym that Opscode and John Willis has spoken about.