Delivering exceptional customer experiences and product for your business take speed and flexibility. More than ever before, speed and flexibility are required from every part of your organization, business and IT alike. DevOps provides your business leaders, enterprise architects, developers and I&O leaders a philosophy to achieve, not only the velocity that customers desire but also drive innovation and enforces quality. One example is ING. The company is undergoing a major digital transformation in which DevOps is a primary driver supporting their transformation. ING CIO Ron van Kemenade has initiated DevOps as the vehicle to aggressively support ING’s evolving customer needs. At ING, technology is the beating heart of the bank.[i]
DevOps requires a transition from technical silos to product centered teams
Effective DevOps will require the tearing down of the technology based silos within an organization. Instead, teams need to focus on the products (or service) delivered and be empowered to own the complete lifecycle. Key performance metrics such as such as availability, the number of features added are used to measure the speed and quality of how these product centered teams work. In some organizations, the team may even own support of the designed and delivered services. This integrated product team is a fusion of developers, infrastructure & operations, quality assurance, and release managers into a single team that works on the entire pipeline, from commit to deployment. Existing centers of excellence such as DBA’s or security teams will remain and support the DevOps team; in some cases, they might even be allocated to the team for a particular duration. [ii]
Deconstruct silos of automation and replace with full pipeline automation
Much has been written about how artificial intelligence (AI) will put white-collar workers out of a job eventually. Will robots soon be able to do what programmers do best — i.e., write software programs? Actually, if you are or were a developer, you’ve probably already written or used software programs that can generate other software programs. That’s called code generation; in the past, it was done through “next” generation programming languages (such as a second-, third-, fourth-, or even fifth-generation languages), today are called low code IDEs. But also Java, C and C++ geeks have been turning high level graphical models like UML or BPML into code. But that’s not what I am talking about: I am talking about a robot (or bot) or AI software system that, if given a business requirement in natural language, can write the code to implement it — or even come up with its own idea and write a program for it.
Modern application delivery leaders realize that their primary goal is to deliver value to the business and its customers faster. Most of the modern successful change frameworks, like Agile (in its various instantiations), Lean, and Lean Startup, which inspire developers and development shops, put metrics and measurement at the center of improvement and feedback loops. The objective of controlling and governing projects to meet vaguely estimated efforts but precisely defined budgets as well as unrealistic deadlines is no longer on the agenda of leading BT organizations.
The new objective of BT organizations is to connect more linearly the work that app dev teams do and the results they produce to deliver business outcomes. In this context, application development and delivery (AD&D) leaders need a new set of metrics that help them monitor and improve the value they deliver, based on feedback from business partners and customers.
Preproduction metrics. Leading organizations capture preproduction data on activities and milestones through productivity metrics, but they place a growing emphasis on the predictability of the continuous delivery pipeline, quality, and value.
A few months ago, I blogged about testing quality@speed in the same way that F1 racing teams do to win races and fans. Last week, I published my F(TA)1 Forrester Wave! It examines the capabilities of nine vendors to evaluate how they support Agile development and continuous delivery teams when it comes to continuous testing: Borland, CA Technologies, HP, IBM, Microsoft, Parasoft, SmartBear, TestPlant, and Tricentis. However, only Forrester clients can attend “the race” to see the leaders.
The market overview section of our evaluation complements the analysis in the underlying model by looking at other providers that either augment FTA capabilities, play in a different market segment, or did not meet one of the criteria for inclusion in the Forrester Wave. These include: 1) open source tools like Selenium and Sahi, 2) test case design and automation tools like Grid-Tools Agile Designer, and 3) other tools, such as Original Software, which mostly focuses on graphical user interface (GUI) and packaged apps testing, and Qualitia and Applitools, which focus on GUI and visualization testing.
We deliberately weighted the Forrester Wave criteria more heavily towards “beyond GUI” and API testing approaches. Why? Because:
Formula One has gotten us all used to amazing speed. In as little as three seconds, F1 pit teams replace all four wheels on a car and even load in dozens of liters of fuel. Pit stops are no longer an impediment to success in F1 — but they can be differentiating to the point where teams that are good at it win and those that aren’t lose.
It turns out that pit stops not only affect speed; they also maintain and improve quality. In fact, prestigious teams like Ferrari, Mercedes-Benz, and Red Bull use pit stops to (usually!) prevent bad things from happening to their cars. In other words, pit stops are now a strategic component of any F1 racing strategy; they enhance speed with quality. But F1 teams also continuously test the condition of their cars and external conditions that might influence the race.
My question: Why can’t we do the same with software delivery? Can fast testing pit stops help? Today, in the age of the customer, delivery teams face a challenge like none before: a business need for unprecedented speed with quality — quality@speed. Release cycle times are plummeting from years to months, weeks, or even seconds — as companies like Amazon, Netflix, and Google prove.
Within the modern applications era, regardless of whether new software applications are being developed and delivered for mobile, tablets, or the Web, the truly successful app-dev leaders will be those who focus on delivering constant value and incremental improvement to their business. That is a totally different perspective from “I need to keep control of my team’s productivity to make sure that we stick to our estimated costs, scope, and project dates.” Of course, the interest in cost is never going away, but app-dev leaders today have a great chance to enhance their conversation with executives and business stakeholders and add value to the conversation.
However, as the recent research I just published, Agile Metrics That Matter, proves, while some of the most advanced Agile teams do use new progress, quality, efficiency, and value/benefits metrics (these to a lesser degree), some software development industry luminaries have worked and are working on new methods to measure value in software development. But it’s still early days!
I’d like to summarize here some good old practices on establishing metrics that count together with some of the new findings of the research:
What a strange summer this has been! From Boston to London to Paris to Turin, the weather has offered weekly and even daily reversals, with continuous change from sun to rain, from hot and damp to cool and crisp. I missed a nice spring season. Even today, from 35º-38º Celsius (95º-100º Fahrenheit), we just went to 22º Celsius (71º Fahrenheit) with a perfect storm! A continuous climate and sudden change is quite unusual in some of these countries. Certainly it is where the Azores Anticyclone usually dominates from mid-late June to mid-late August, offering a stable summer. How many times have you had to change plans because you discover weather is about to change!?
You might be thinking, "What does this have to do with this AD&D blog?" It’s about change! I am wondering if, in our daily lives, getting used to unexpected conditions and having to handle continuous change favors a mindset where change is just something we have to deal with and not fight. A new mindset very much needed given the change we see ahead in how we develop, test, and deploy software!
My focus in this blog is testing, although the first change we need to get used to is that we can’t talk any longer about testing in an isolated fashion! Testing is getting more and more interconnected in a continuous feedback loop with development and deployment. (See my colleague Kurt Bittner's report on continuous delivery; I could not agree more with what Kurt says there!)
I just finished my new report on the Agile testing tools landscape. I’ll point Forrester readers to it as soon as it publishes. But there are few things that have struck me since I took over the software quality and testing research coverage at Forrester and which I would like to share with you in this preview of my findings of the testing tools landscape doc.
My research focus area was initially on software development life cycles (SDLCs) with a main focus on Agile and Lean. In fact, my main contribution in the past 12 months has been to the Forrester Agile and Lean playbook, where all my testing research has also focused. Among other reasons, I took the testing research area because testing was becoming more and more a discipline for software developers. So it all made sense for me to extend my software development research focus with testing. But I was not sure how deep testing was really going to integrate with development. My concern was that I’d have to spend too much time on the traditional testing standards, processes, and practices and little on new and more advanced development and testing practices. After 12 months, I am happy to say that it was the right bet! My published recent research shows the shift testing is making, and so does the testing tool landscape document, and here is why:
As I write this, I am in seat 1A of United flight 1607 from Philly to Houston. playing on the screen in front of me is CNBC. I make no secret of my disdain for much of the so called "news media" so I won't launch into my usual rant there (there are some superb journalists out there, but Murrow and Cronkite must be rolling in their graves!). I am bristling over the coverage right now that is focused on the 787's latest woes. As usual, the talking heads are clueless and painting a doomsday scenario for Boeing! It's a bunch of finance people who don't understand the engineering realities. They're smart bean counters, but not engineers. I am an old engineer, so let me shed light on what the Wall Street mouths don't know. There is an important lesson here for I&O leaders!
There is no doubt that Agile growth in the market is significant, and the growing daily number of inquiries I’ve been getting on Agile from end user organizations in 2012 gives me the impression that many are moving from tactical to strategic adoption. Why’s that? Many reasons, and you can read about them in our focused research on Agile transformation on the Forrester website. But I’d like to summarize the top five reasons from my recent research “Determine The Business And IT Impact Of Agile Development” :
Quality was the top — quite astonishing, but both the survey we ran across 205 Agile “professional adopters” and the interviews across some 21 organizations confirmed this. My read is that this is about functional quality.
Change was second to quality. We live in an era where innovation strives and organizations are continuously developing new apps and projects. But your business does not necessarily know what it needs or wants upfront. The business really appreciates the due-course changes that Agile development allows, as they enable the business to experiment and try out various options so it can become more confident about what is really right for the organization. Cutting edge cutting edge systems-of-engagement (Mobile, Web-facing, Social-media, etc) require lots of Change in due course.