What’s In Your Test Estimation Framework?

I've been working quite closely with fellow analysts Dave West and Mary Gerush surrounding project estimation. Regardless if you're in the Agile world, testing is factored in (well, unit testing anyway), and if you're in the traditional camp, we've heard the same pain from a number of Forrester customers. No matter what methodology we use, there's not enough time to test. To combat that, testing organizations are attempting to build a livable, usable framework to provide them with information to battle for sufficient testing time. The problem is, there is no one way of looking at it. Nothing new here, estimation has been always been a challenge in project management. What is different, however, is that organizations are starting to take testing seriously; actually, let me expand that to taking quality seriously, and because of that, testing organizations are finding support in developing test estimation frameworks that can help them provide realistic estimates to gain enough time to test what's important. I've heard everything from 1.5 function points of testing for every function point of development to 40% of development time. I think it's time that we share what we know to help our fellow testers. Too often, it depends on:

  • The content of the project,
  • Project methodology
  • Behavior of the development organization
  • Skills of the testing team

Still organizations want something to hang their hats on, as a guide or as way to forecast resources. In order to do that, we've been talking to a number of vendors and customers about their testing practice and their testing tool practices (for upcoming report on the testing tools landscape). It's not surprising that we've not found a silver bullet, but we are starting to see some trends in estimation practices. Among the most common used methods are:

  • Function points – used most often with consulting firms, but there is increased interest for internal use as well.
  • Baseline industry metrics (see my other post "Metrics, Metrics Everywhere - What Do Metrics Tell Us Anyway?")
  • Story points – definitely in unit testing, but now also being more widely adopted for functional and acceptance testing
  • Use cases – can provide context and perspective for estimating test requirements.
  • Percentage of development effort - that depends on who's writing their opinion of it.

Most of these methods provide great foundations for building estimates. Function points do require the steepest learning curve; however, they can provide a good indication of size, which you can then convert to effort. To me, the most problematic is percentage of development effort. You just can't count on that. A very small development effort can require a significant testing effort, especially if the change has many dependent modules. Personally, I think that a combination of practices, such as use cases or story points, knowing your testers capabilities, plus historical effort are the most efficient way of getting a real perspective. So, I put it out there to you – what's in your test estimation framework?