In the days of old, not very long ago, release cycles were measured in years —organizations were using “on-time” and “on-budget" as the mantra for project efficacy. Business today is compelled to deliver business technology in cycles of hours, or days. Faster cycles render not only tradition “waterfall” processes and silo based IT obsolete, it also renders traditional metrics ineffective! These arcane metrics no longer deliver the visibility and granularity tech pros need to fine-tune their delivery capability. The mission has transitioned to rapidly deliver high quality, high value solutions. For all, this is a significant shift from the past, when the main points of focus were schedule, cost, and efficiency. Modern software metrics — speed, quality, and value — are based on continuous feedback from business partners and customers.
Much has been written about how artificial intelligence (AI) will put white-collar workers out of a job eventually. Will robots soon be able to do what programmers do best — i.e., write software programs? Actually, if you are or were a developer, you’ve probably already written or used software programs that can generate other software programs. That’s called code generation; in the past, it was done through “next” generation programming languages (such as a second-, third-, fourth-, or even fifth-generation languages), today are called low code IDEs. But also Java, C and C++ geeks have been turning high level graphical models like UML or BPML into code. But that’s not what I am talking about: I am talking about a robot (or bot) or AI software system that, if given a business requirement in natural language, can write the code to implement it — or even come up with its own idea and write a program for it.
In the latter half of last year, I started researching mobile application testing tools. My research focused, so far, on functional testing, primarily around mobile app front-end testing. As I began the research, it became clear that the automation capabilities testers needed to validate app UIs was there, but application development and delivery teams felt that device labs were too expensive to be practical. During the research for the Vendor Landscape: Front-End Mobile Testing Tools report, we expected that device labs would be a differentiator among products only to discover that most of the major mobile testing solutions provide them in one way or another. There are differences between vendors when it comes to the flexibility, configurability, and management of their device lab offerings, but if you’re delivering customer-facing mobile apps you can do much of your testing on physical devices (our recommended method).
In earlier reports, we recommended that, because of the cost of on-device testing, development organizations focused their testing efforts on the most important aspect of their apps, letting users find issues in less popular areas of the app for them. With most of the major mobile testing vendors offering device labs plus Amazon and Google’s entry into the device cloud space, competition will drive down cost and make on-device testing the more common option for mobile app testing. Microsoft’s acquisition of Xamarin now gives Microsoft a robust and capable device lab, stuffed with a variety of Android and iOS devices, which adds to the competition in this space as well.
A few months ago, I blogged about testing quality@speed in the same way that F1 racing teams do to win races and fans. Last week, I published my F(TA)1 Forrester Wave! It examines the capabilities of nine vendors to evaluate how they support Agile development and continuous delivery teams when it comes to continuous testing: Borland, CA Technologies, HP, IBM, Microsoft, Parasoft, SmartBear, TestPlant, and Tricentis. However, only Forrester clients can attend “the race” to see the leaders.
The market overview section of our evaluation complements the analysis in the underlying model by looking at other providers that either augment FTA capabilities, play in a different market segment, or did not meet one of the criteria for inclusion in the Forrester Wave. These include: 1) open source tools like Selenium and Sahi, 2) test case design and automation tools like Grid-Tools Agile Designer, and 3) other tools, such as Original Software, which mostly focuses on graphical user interface (GUI) and packaged apps testing, and Qualitia and Applitools, which focus on GUI and visualization testing.
We deliberately weighted the Forrester Wave criteria more heavily towards “beyond GUI” and API testing approaches. Why? Because: