My Favorite Use For Metrics: Improving Day By Day

by Mary Gerush

Mary-Gerush During my first year as a Forrester analyst, one widespread pain point has made itself apparent: IT leaders wrestle with the subject of metrics, particularly in these challenging economic times. Most people I talk with are effective at collecting and reporting the basics: project on-time and on-budget performance, application up time, and help desk call statistics. But other measurements are more difficult to engineer. And there are so many things that can be measured; it’s difficult to figure out which ones really matter and are worth the time needed to collect and report them.

So as an IT leader, how do you decide what constitutes a healthy set of metrics for your organization? Well, you have to start by knowing what your goals are. IT leaders gather metrics for three primary reasons: 1) to show the organization what they’re doing, 2) to manage people, platforms, and applications, and 3) to pinpoint areas for improvement of processes and outcomes. You can read my 2008 report “App Dev Leaders – Got Metrics?” to understand more about the first two drivers. But my personal favorite rationale for collecting and reporting metrics is the last: improving your team’s capabilities to improve your outcomes and increase your value to the business.

I recently talked with PMO manager who was struggling because the majority of her organization’s projects ended well past their planned delivery dates, and this metric of on-time performance was very important in her organization. I learned that they have a well-organized change control process in place and that they set a new project baseline when business or technical requirements change. But other problems exist: tasks are underestimated or missed, and project team members miss their due dates. Slowly, but surely, their projects’ delivery dates slip; what was once expected in May is now projected for September. Her question to me was “is it proper to rebaseline a project and establish a new delivery date as long as everyone, including the stakeholders and sponsors, agrees that it is OK?”

While taking this approach would certainly make their status report look better, there are a number of problems with it. First, it would obscure the truth. Second, it would set a precedent that team members don’t have to meet due dates. And finally, they would lose a key indicator which could help them pinpoint areas for improvement. With a little root cause analysis, this organization can find ways to prevent the slippages instead of covering them up. We identified five potential root causes that – if tracked for all projects that miss their original due dates – can provide them with the insight to get better.

1. A high rate of business requirements change indicates a need to improve business analysis. Focus energy on improving requirements elicitation, analysis, and management techniques within your business analyst and project management communities.
2. A high rate of technical requirements change could have two root causes. Your architecture and design teams may not understand the requirements, or they may not be spending sufficient time drilling into the detailed design. In this case, investigate both the quality of your requirements documentation and the effectiveness of design. Identify the root cause, and seek to improve that area.
3. Extensive evidence of underestimation will drive you to improve your estimation processes. Check out our report “Best Practices: Estimating Development Projects” for helpful hints to improve the quality of your project estimates.
4. If team members are consistently missing due dates, you may have a performance issue. Identify the team members that tend to deliver late. Make sure they are involved in the estimation process. And if you feel they can’t pull their weight, you consider moving them to another position.
5. If slippage due to late external dependencies is common, focus on risk management. When your project depends on something outside its own scope, there is often little you can do to manage its progress. But creating and using strong risk and issue identification and management processes can help your team plan for and react properly in these situations.

By adding one metric to their project management reporting – reason for project delay – this organization can gain the ability to seek out root cause and focus energy on improving their practices in troubled areas. Over time, they should see their report looking more positive, reflecting their improvement. What a powerful tool that one metric can be.

At the IT Forum in Las Vegas, I’ll be leading a session on metrics titled “Metrics: Make Them Meaningful And Manageable.” I hope you will join me there! And feedback is welcome. What metrics do you use to improve your team?



Business Analysis, and grading Projects

"1. A high rate of business requirements change indicates a need to improve business analysis."

Indeed, and such improvements may have the greatest impact, given that getting it right the first time is always better than having to do it over.

Overall, Project Status reporting does need to focus on current execution versus current plan, but I think projects also need to be graded for evaluation purposes, to expose that truth. Certainly the more times a project has to re-plan and re-baseline, the lower grade it should receive, and should receive extra attention during post-implementation reviews.