What are the Next Secret Weapons in the Test Manager’s Toolbox?

As teams become more agile, or, add more agile like practices to traditional
development practices, I’m seeing increasing frustration on the part of test
managers.  Rapid development cycles and scrutinized bottom lines are putting
more pressure on them to deliver comprehensive testing in tighter time
frames.  More and more testing is being taken on by development teams, and while
that is a positive trend to be sure.  More stringent testing performed by
development is a good thing, as a long time QA manager myself, I used to pray
for consistent unit and integration testing, but, ultimately, developers are not
trained to think in the same way that QA does.  Development testing is meant to
ensure that the code, service, or integration performs the way it was conceived;
it doesn’t always cover the assurance that the business process is being met
and it doesn’t replace the end to end perspective that an organization needs to
ensure that the highest quality software is being delivered. Development testing
is faster, to be sure.  End to end testing takes more time, but it’s necessary.
Test managers must do something to prevent testing being co-opted by development
at the expense of business value.

I’ve discussed seeing the trends that show test management becoming a more
strategic function within the application lifecycle – as discussed in The Dawn
of Dynamic SQA
from earlier this year, and as 2009 has passed, this
strategic shift has shown us that traditional test management tools are not
enough.  What we are seeing at Forrester is that the walls between development
and QA continue to crumble – and as well they should – in order to allow the
developers and testers to work more effectively together.  In order for test
managers to be truly effective and to streamline the application development and
delivery process, they must extend their visibility further into the lifecycle –
that means that they need to have insight into planning, requirements, build
changes and potential service issues.  This allows test managers to be able to
anticipate what’s coming that is going to impact their test plans and avoid
being that classic “QA Bottleneck”. 

Extending this visibility means that test managers must take advantage of
tools that provide ability to drill into multiple layers of the application
lifecycle – from planning to deployment.  It’s no longer enough to use just test
management suites, defect management, Excel and Word to plan their test
cycles. test managers must add the following tools to their arsenal:

1. Application lifecycle management (ALM) – these tools provide the ability
to determine build status, configuration changes and monitor requirement
changes.  If there is a problem with a build, the test manager can see how that
will affect his/her test schedules.  If there are requirement errors,
development errors or changes, the test manager can immediately know that tests
may need to be reprioritized, rescheduled, or scrapped.  In a continuous build
cycle, this can significantly reduce the amount of unnecessary testing that is
all too normal in QA environments.  Vendors such as MKS, IBM, HP provide the
ability to integrate test management with ALM to provide this level of

2. Service management or help desk tools – these tools can help the test
manager see what potential problems are coming down the pike and can prepare the
test team with prioritized test cases to quickly test any fixes that are being
planned by development. Together with ALM, test managers can be ready for quick
break/fix cycle updates so that there are no delays in getting a fix out to the
field, nor will they have to rely on only development testing to approve the
fix. I’ve talked with several Forrester customers that are beginning to pull in
service desk information into their test planning for maintenance projects. 
With QA being a critical element of the deployment team, I expect to see

3. Project portfolio management (PPM) – for the really strategic test
manager, this environment will enable test managers to see active collaboration
at the project level, not just at the task or activity level.  If there are
risks, issues and/or scope changes,  test managers can quickly see them and
suggest schedule changes and test plan changes proactively, instead of
scrambling at the last minute to make sure there is coverage.   HP, MKS, Serena
and Microsoft all provide various forms of information at this level and IBM
should be joining the mix soon.  CA and Planview offer requirements management
within their tools, so that if there are requirements changes, the test manager
with access to the team project page can make necessary changes and contribute
to the requirements management as needed.

4. Ad hoc – wikis, SharePoint and a host of other options are out there as
well. Today, I think this is probably the most widely used manner of bringing
test management into the process, however, without linking to artifacts, there
is still the chance of missing critical touch points.

Bottom line, test managers that are forced to scramble to find out time
sensitive information will continue to be squeezed out of the application
lifecycle – it’s time to start tearing down the walls at the tools level, too. 
ALM shares important project artifacts that test manager must use not as an
occasional, but a daily part of their management process.  PPM and service
management tools should also be on the short list to raise the visibility and
cut down on the communication barriers.   I’m interested to hear – what other
tools do you think should be part of the test management toolbox?


re: What are the Next Secret Weapons in the Test Manager’s Tool

Margo, I absolutely agree. Testing has to move closer and managing requirements, testing and deployment has to be business as usual. Which is why our Papyrus Platform has ALM, PPM and the user to help desk service processes embedded.But testing has not only to move closer to R&D but bugfixing has to move closer to the production and to the user. While we are at it, why just bugfixing? Why not dealing with improvements the same way.That's how we do it in Papyrus:1) A first version of an application (a set of processes - what else?) is deployed through PLM after sign off by QA. The end-to-end test proved functional completeness.2) The users do not reject the application because of possible improvements. Today they know if they accept it - that's how it will be for the years to come.3) The users add bugs, wishes and nice-to-haves into the requirements log in PLM.4) R&D prioritizes them with the business management through the work assignment, coding, testing and signoff processes that are part of the platform.5) Once signed off, changes are delivered one by one through our automatic deployment mechanism - most commonly on a weekly basis. With Java, C++, .net that is a three month cycle at least, often six because of deployment.Because so many user wishes are just related to presentation, Papyrus uses the EYE concept, where process and presentation are kept independent and users can change the GUI and content/document presentation without needing a programmer.I believe that this kind of integration is the future of application development.

re: What are the Next Secret Weapons in the Test Manager’s Tool

Requirements management is a critical area, and is evolving to include more assets such as ideas, defects, test cases, feature requests and the entire conversation that goes on during the planning and development process. The big difference over the traditional RM tool or ALM suites, is the trend of being more open, agile and collaborative with customers throughout the process requires tools provide a better platform of social collaboration between the entire team and all the stakeholders. There's a realization that the strengths of RM are extensible across the entire process:1. traceability (connecting everything together)2. change management (assessing impact when change occurs and alerting the related people impacted)3. release management (grouping requirements, defects and other items into manageable sprints/iterations/releases)4. reporting/analysis (automation of specification documents, test coverage reports, etc.)5. collaboration (Web access for all, inline comments, smart alerts, greater visibility upstream and downstream)In addition, the market wants tightly integrated best of breed solutions, the all-in-one ALM suite was a nice idea at one time but in most cases there's 1 core competency and a bunch of padding. The next generation applications are Web-based and naturally collaborative, they're built on open standards and naturally integration-ready with other complimentary tools.As an example, this is the philosophy of the solution, Contour, we're creating with our customers at Jama.http://www.jamasoftware.com/contour/Curious to know Forrester's point of view and others on this trend and the evolution of the tools to support more open/agile/collaborative processes?