Don’t Discount The Value Of Planning Tools To Your Online Testing Program

In the past 18 months I've spent a lot of time working with Forrester clients on many facets of online testing (that’s a/b and multivariate testing for those of you scoring at home) spanning vendor selection, organizing and developing skills for testing, and building processes to support testing.

One of the general trends in online testing has been the democratization of access to marketing users. I think this is a positive development because successful online testing is a team sport that requires collaboration across multiple departments and skillsets. However, pulling testing outside of the exclusive domain of analysts puts a lot of pressure on vendors to supply tools that are suitable for non-technical audiences. This means providing easy-to-use, guided functionality, collaboration features, campaign preview facilities, extensive object reuse, and modern interface designs. And, to varying degrees, vendors are making progress in the area of user experience to meet these needs.

I have noticed that one of the features that often gets short shrift is test planning tools. In my experience, planning functionality has come forward as a crucial – and underrated – feature in situations where marketers or non-technical users will be involved in the development and deployment of online testing campaigns. To explore this idea further, I just published a new piece of research titled "How CI Professionals Can Plan For Site Optimization Success."

An online testing planning tool is essentially a calculator designed to evaluate the parameters of an experiment to determine the expected duration of a test. Typically, the functionality is available natively within the online testing application, but occasionally may be available as a standalone application from the vendor or as a homegrown macro or spreadsheet. Planning tools provide visibility to help users tune their experiments to complete within an appropriate time window or for specific situations; for example, varying amounts of site traffic (perhaps seasonality or different web properties) or different conversion rates for different types of promotions.

As you look at online testing tools, it is crucial to understand the day-to-day users of the application. Organizations planning to deploy an online testing tool to non-technical or marketing users must take special care to evaluate application usability – including the availability of planning functionality – to ensure that those users will be productive in the testing environment. Poor usability will become an obstacle to overall application adoption and lack of planning tools may impede appropriate test design, both of which can cripple an organization’s ability to scale up a site optimization program.

Comments

I think this is yet another

I think this is yet another solid writeup from Joseph. I don't you should stop there planning tools should be used in combination with or wrapped into a planning and prioritization process as well. There are 2 main reasons for this
- An out-of-the-box tool is never enough each organization will need to plot their own parameters into the calculation of priority.

-Many marketers will still be intimidated by existing planning tools. Wrapping these tools into the planning committee's sprint meetings will help to achieve proper integration and encourage the testing culture to grow.

I'd like to see an improvement in people/process/system integration when it comes to online testing

Thanks for the thoughtful

Thanks for the thoughtful comment Dwayne.
I completely agree that planning functionality needs to operate within the context of the broader testing planning and design process.
And process is absolutely the biggest challenge that organizations face in online testing. It is my hope that users will continue to increase their emphasis on processes, and that vendors will enhance their tools with increased support for planning and workflow.