The PM job calculator: validation

[Last in this series about the Tech Product Management/Marketing Job And Department Profiler. Previous posts cover its origins, the design decisions that went into it, and the job specializations it profiles.]

From my grad student years, I know that a lot of mathematical models fall flat on their simulated faces because they don't get the testing they should. In the case of the PM job calculator/department profiler, I had to apply three different tests.

The tasks are many and varied
Last year, when I did the survey research on the state of the product management/product marketing job in technology companies, I vetted the list of tasks with a lot of PMs. Using their feedback, I added tasks, rethought others, and even merged a couple. The resulting list represented everything that PMs might be doing, depending on where they work.

That was the easy part.

The tasks define the jobs
A major goal of the Profiler is, as I explained in an earlier post, classifying PMs according to their specialization. A requirements-focused PM is not a product manager, who is not a release manager, who really isn't a product owner, who can't find the spare time to also be a community manager...You get the idea.

Consequently, the next test involved showing these job categories—defined by the tasks these roles perform, and the priority each task receives—to people who run PM departments. That group included the heads of product management teams, product marketing groups, and combination product management/product marketing departments.

Since no one's job description for any of these roles exactly resembles anyone else's, the goal wasn't to find perfect agreeement among job categories as they're defined in particular organizations. Instead, I did a version of "ideal type" analysis: does this definition offend your sense of what's essential about this role? Over the course of several conversations, I managed to develop a set of ideal types that everyone found more or less acceptable.

The jobs define the department
When I started this project, the stretch goal was finding a way of using the job-level results to help define what mix of people you might need at a departmental level. Surprisingly, that stage of the project was easier to complete than the job descriptions.

Perhaps that's because every senior director or VP of a PM organization focuses first on the type of person needed to perform a job, steps back a bit, and then figures out how many people, at what level of seniority, need to appear in the group portrait.

The most important choice was, which variables mattered? There are dozens you might consider, from the type of product to the size of the company. While all of these variables have some influence, they don't all have the same weight in determining what kind of team you need to build.

Once again, tasks proved to be the right starting point for this project, for a variety of reasons. For example, the degree to which someone can focus on their core responsibilities has a definite impact on how many of those people you need.

This last stage of validation revolved around the question, "You put this information into the black box, and out comes this picture of the kind of department you need. Does this picture make sense?" The results had to be good enough, since a perfect calculation is probably not possible. (Again, many variables have relevance, but it's necessary to focus on the ones that matter the most.) Perhaps not all of these people in this hypothetical department, in practice, report into the same group, but they have to exist somewhere in the company to provide adequate support and leadership for the product or service under development.

Don't let the perfect be the enemy of the good
Even though perfect accuracy in the prediction of staffing needs—how many, in which roles, at what level of seniority—may be impossible, the "good enough" picture has definite value. For example, during the validation discussions, some PM department heads were interested in showing the results to their management to (surprise!) justify their staffing needs. Others spent time discussing the things they hadn't considered, such as tuning their organization to better support the sales team. Someone else spent a long time talking with me about where to shift particular tasks, if you take them off the shoulders of PMs.

In other words, the Profiler provides a starting point for a discussion about organizational changes. Without it, you might not be able to have that discussion, since the proverbial devil is truly in the details. Making a particular task, such as maintaining the product roadmap, a higher or lower priority affects other PM responsibilities. It also begs a lot of questions about skills, time management, deliverables, and compensation.

However, if you didn't have some kind of detailed portrait of what PM does and doesn't do, it's difficult to get to these second-order questions, such as professional development. If you don't force yourself to be specific about PM priorities, then there's always a case for something else to get added to the to-do list. Unfortunately, that often leads to the situation in which, if everything is a priority, then nothing is.

[Cross-posted at The Heretech.]

PMs know that requirements are flawed

The workshop next week, "Using Social Media To Make Smarter Product Decisions," was a good occasion for me to jump into the data we've been collecting about requirements in the technology industry. It's one thing to claim that there are problems that need fixing, and that social media can help fix them. It's another to put that claim to the test.

The requirements survey has produced a lot of interesting results that will see the light of day in a research document to be published later this quarter. The questions we asked build a composite picture of both the content of requirements and the process of creating them. For example, we asked, ""How much time do you dedicate to requirements work at each stage of the development cycle?" Doing a weighted average of the responses, here's the answer:

This picture should be familiar to any PM: lots of requirements work at the beginning of the project; less and less as the project proceeds. Which begs the question, how effectively can the organization make mid-course corrections? When does the "after action report" kind of analysis happen, when PMs look back on the work they just did to figure out what they could do better the next time?

While this spike of requirements attention at the beginning of a project makes sense in bureaucratic terms, it doesn't make sense in research terms. You might pretend that the requirements you build at the beginning of a project are everything that the development team needs to proceed, but at best, that's a comforting fiction. You always need to fill in gaps, provide more detailed explanations, and validate the results.

If requirements are the information that technology teams need to avoid making costly mistakes, then requirements gathering and analysis should be an ongoing activity. While that's certainly true in some PM teams that I've met during my time as a Forrester analyst, in the aggregate, these peaks and valleys of activity still define the requirements landscape for many PMs.

There are plenty of other signposts to deficiencies in product requirements. For example, there's a big mismatch between the skills needed to do requirements, and the opportunities to put these skills to work. While the rest of the story will come later, I thought this one result was interesting enough to now, instead of waiting for the final report.

[Cross-posted at The Heretech.]

The PM job calculator: specializations

[Continuing the "behind the scenes" discussion of the Product Management/Marketing Job And Department Profiler, a.k.a. the job calculator. Here are the links to the first two parts, covering the origins of the calculator, and the design decisions behind it.]

As discussed in the last post on this topic, I decided to define product management roles through the tasks that those roles perform. The maxim, "You are what you do," made it easy to define different job categories for product managers and product marketers. A task -based approach also helped make a more convincing argument why particular specializations are necessary.

Tasks—what are people doing, and how much attention or emphasis does each task receive—make the difference between, say, the canonical product manager and the archetypical product marketer clear. Realistically, you never have enough time to do everything that's on the massive List Of Things For Which PM Might Be Responsible, nor are you necessarily good at all these tasks. Plus, if everything is top priority, then nothing is top priority.

From single-cell creatures to complex animals
If you add up the tasks that product managers do to ensure that a particular product or service meets its design and business objectives, you already have a pretty big list. Just working with the development team is going to consume a lot of time, without adding product marketing tasks like developing messaging, or even the project management tasks related to the development cycle, such as ensuring company readiness for a new release.

If you keep pushing the question, What should I be doing, and how important is it that I do it? it's easy to see the further specialization beyond just the product manager/product marketer distinction. I've already alluded to the separate job function of release manager, who's responsible for project management. There are also sales enablement specialists, whose job it is to support the customer- and partner-facing parts of the company. Sure, PMs have a role to play in sales enablement, but it's dangerous to have them spending too much time focused on the field. Even though the average PM crafts product strategy through the media of requirements and roadmaps, a user experience professional is contributing to the product's success in a different way.

In many companies, these roles report into different parts of the org chart. Whether or not they're centralized into the same PM organization is a question of execution.

Evolution creates new differentiations
In recent years, we've seen the emergence of other job specializations. The difference between product owners and product managers may not be completely clear, but the roles aren't exactly the same. (We heard that issue discussed a lot at the Agile 2009 conference. The role of PM in an Agile team turned into a much bigger research project than I ever expected, because there are no clear-cut right and wrong ways of defining it.) The explosion of social media has inspired a new job category, community manager, responsible for product management- or product marketing-like activities exclusively within social media outlets.

During the validation phase of this project, when I showed the calculator to heads of PM groups, no one blinked at these job specializations. It's clear that the day of the generic PM is over. What we call a product manager now is someone who is focused on product and service development. That person often works side-by-side, in the same group, with product marketers, technical product managers, release managers, and in some cases, even sales enablement. Or, these functions are scattered across the organization.

Nevertheless, the increasing specialization of product management and product marketing seems to be a fact of life. The to-do list—how big it is, and how much priority each task receives—has made that conclusion inescapable in any organization serious about the PM role.

[Cross-posted at The Heretech.]

The PM calculator: tasks

[Continuing the discussion of the PM Job And Department Profiler from the previous post.]

The most fundamental decision for the Profiler was the choice of how to depict the PM role. I might have chosen functional areas, such as Pragmatic Marketing's famous diagram of PM responsibilities, or something skill-based, such as Steve Haine's efforts to define product management and product marketing as a corpus of knowledge that people in that profession need to master. Instead, I decided to use tasks to build the picture of what PMs do.

There's nothing inherently wrong in the other depictions of product management, but they weren't the right raw materials for what I wanted to do. If I accomplished nothing else, I wanted to depict the volume of work that PMs do, and the degree to which that work is focused or unfocused.

The Pragmatic Marketing periodic chart of responsibilities tells you how to categorize the work, but the categories don't point directly to the amount of work. A skill-based approach tells you how many different skills you need to acquire, and might even suggest how challenging some of those skills are to learn. However, that doesn't tell you, in hurly-burly of daily work, how much time the tasks that require those skills will take, or (again) how challenging it is to juggle competing priorities.

From zoology to ecology
The closest analogy I can think of is biology. As my graduate adviser said, "Every science begins with classification." That's what the Pragmatic Marketing framework does very well: define the different species of responsibilities that constitute the family tree of product management and product marketing. However useful and necessary classification may be, it doesn't answer all the questions of biology, such as, "Why do these species dominate, and these other ones don't?"

To take this analogy further, job descriptions are like ecological niches. Many different species might fill each niche; not everyone is equally well suited to do so. (By the way, the number of niches keeps expanding, but that's a topic for another post.)

Tasks, therefore, are the essential traits that determine the success of a PM in each niche. If the PM is doing the wrong activities, or is too busy to perform the essential activities, then that PM won't do well within that niche.

The ecology of vital business activities
Classification tells us what species have evolved to fill these niches. Skills, like DNA, define what the species is capable of doing. However, if the question at hand is, What kind of PM should we have? the answer depends on what the PM is actually doing within a particular niche, a.k.a. job category.

Or, to look at it another way, classification of PM job categories is interesting, but it's not a very convincing argument for why a company should adopt one species over another. On the other hand, discussions of skills is sometimes premature, because it begs the question, What's the vital business function for which these skills are necessary? Tasks, as it turned out, were a better starting point for defining the vital business functions in a way that might lead to a fruitful discussion of, We need this many PMs in these specializations to do the job right.

[Cross-posted at The Heretech.]

The PM job calculator: genesis

The Product Management/Marketing Job And Department Profiler is finally, finally available (Forrester account required). This project was unusual in  a lot of respects, so it's worth spending a little time explaining it. You might also get some insight into how research gets done at Forrester, which might be a little mysterious to the outside world.

What is it?
Here's the short version: You enter the priorities for a particular PM, and you get a recommended job description. If you want to go one step further, you can also get a picture of what your department should look like: how many PMs, in which specializations, at which level of seniority for their positions.

In other words, it's a tool designed primarily for the people who run PM groups. The Profiler is also useful for rank-and-file PMs who might want to gauge where their priorities and skills are, and where they should be.

I'll get into more details about the tool and how it works in a later post, but let me first explain why I created it.

What inspired it?
Last  year, I ran a survey to find out what PMs were doing in their jobs, versus what they thought they should be doing. The results, released in the document "Product Managers Were Working On The Wrong Things," were very interesting.

For example, we saw in stark terms how product managers struggled to stay focused on their core work, helping the development team make good products and services, because they were pulled into many tasks that were not directly relevant, such as marketing support and sales enablement. At the same time, product marketers had their own version of "the urgent defeats the important" principle, with too many conflicting priorities and too few resources to address all of them.

Of course, the next question is, "Now what?" The point of this study wasn 't merely to commiserate, but to suggest a course of action to fix what is perhaps the core problem with product management and product marketing in the technology industry: vagueness about roles, responsibilities, skills, and deliverables. Research that dramatized the problem was necessary, but not sufficient to do more than generate broad recommendations (for example, run your department so that product managers can focus on their requirements).

Why this approach?
After having defined a problem area, the standard analyst response is to dig more deeply into each part of it. And, to a large extent, that's what I've been doing. Natural topics for further research included:

• How the development and delivery processes shaped the job description, which led to work on product management for SaaS products, and the role of the product manager in (or adjacent to) development teams that have adopted Agile methods. 
• What PMs produce, which inspired some work on serious gaming and product requirements.
• Where new opportunities to improve product management exist, such as the multi-part series on the role of "inbound" social media as a new source of requirements. Again, in this case, we circled back to the question of how the nature of the work—in this case, collecting information from social media channels—affects the definition of the job.

In fact, nearly everything that I've written points back, to some extent, to the roles and responsibilities of product managers and product marketers. Are they really two different roles, or one? If they're separate, should they be part of the same department, or different ones? What are the important adjustments in a PM's to-do list, based on the type of product they're working? How do these changes dictate the skills necessary to handle these tasks? What kind of authority should PMs have, over which kinds of product-related decisions?

Rough sketch, not paint by numbers
Even if I did the best job imaginable in each research document, I still felt that something was missing. Tech companies want baselines against which to measure their own PM organizations, and they want best practices for getting the most from these teams. Ultimately, however, they want a picture of what the PM team should be.

Individual pieces of research fill out some details. However, these details don't always add up to the complete picture. Or, to look at it another way, you need a rough sketch of the family portrait before you fill in the details for each person.

That rough sketch is vital for answering many common questions, such as the PM's relationship to an Agile team. If you choose the "PM as product owner" option, that won't necessarily tell you everything you need to know about the tasks that person performs, who takes care of the PM-like tasks that are now off that person's plate, and what skills and experiences the hybrid product manager/product owner needs to succeed. While a snapshot of the individual PM might be useful, a family portrait that includes other roles and responsibilities is even more helpful.

The Profiler provides that rough sketch. You won't find many (or perhaps any) tool like it, which is why it took a long time to build. It was worth trying to build one, even if the experiment turned out to be an utter failure.  If we know that you can't create the rough sketch, you'll continue working on separate pieces. It's then up to the reader to build a mosaic of the pieces, in no particular order.

On the other hand, if you can build the rough sketch, you can tell right away which areas will need the most attention, and how they affect adjacent pieces. If Rembrandt hadn't the whole composition of The Mill worked out in advance, he would have wasted time on sections of the canvas that, in the end, make the central subject as striking and interesting as it is.

[Cross-posted at The Heretech.]

Teleconference on Agile adoption next week

Next Monday, August 31, Forrester colleague Dave West and I will be presenting some of our preliminary results from our Agile adoption survey. (Which is still open, if you're interested in participating.) We'll be covering Agile adoption in both IT departments and technology industry companies, including the differences between the two.

Of course, everyone wants to know how many development teams have adopted Agile. My sneak peek for you is, "Substantially more than we had originally estimated." If you want to hear the full answer, here's the link for the teleconference.

[Cross-posted at The Heretech.]

Please spread the word about the requirements survey

We're trying to make our requirements survey as successful as possible, which depends in large part on getting as many people to take the survey as possible. Here's where we humbly beg for your help.

Please distribute this link to the survey to anyone you think has experience with, and opinions about, requirements documents and the requirements process:

http://deploy.ztelligence.com/start/survey/survey_taking.jsp?PIN=15TQ9CMTFTDDZ

Again, all responses remain confidential, and participants get both our thanks and a copy of the final research.

[Cross-posted at The Heretech.]

Best practices for requirements

We're doing a major research project on the state of product requirements in the technology industry.  To make this project as successful as possible, we need your participation in either or both aspects of it, the survey and the examples. (Whichever works for you.)

What's in it for you? When the final research document is done, we'll give you a copy. We also promise to keep any information you provide confidential.

The survey
We just launched the survey at the following link:

http://deploy.ztelligence.com/start/survey/survey_taking.jsp?PIN=15TQ9CMTFTDDZ

If you have approximately 20 minutes to spare, we're interested in hearing how you collect, write, review, and circulate requirements in your organization. Also, please feel free to forward this link to anyone you think would be interested in participating in this survey.

The examples
We've been building a library of examples of requirements documents. (And thanks to everyone who has sent us their samples already.) We don't need to see the actual words, just the structure of the document. In other words, we want to see your template, not the actual contents of a completed requirements document. Fake or redacted data is good, too, if you use a requirements application.

We're being deliberately vague about what a "requirements document" is, since we don't want to lead the witness. Your examples will tell us what you think the fundamental requirements documents are. What do the key stakeholders and consumers in your organization need?

If you're willing, please send your examples directly to me, tgrant@forrester.com.

[Cross-posted at The Heretech.]

The Heretech, episode 14: Israel Gat on Agile adoption

In this episode, Israel Gat illuminates the ways in which Agile adoption depends on organizational and cultural factors. We also muse about Helmut von Moltke, 19^th century military Agilist. (Go look it up!)

Plus, a brief review of an even briefer document about innovation in “knowledge-creating” companies, and a heads-up about some upcoming survey research about product requirements and Agile adoption. Copyright (c) 2009 Tom Grant

The extended family of Agile

While the actual document about Agile usage in the technology industry slowly but majestically navigates through the final editing and production process, I thought I'd share an important bit of data that didn't make it into the report. Shown in the diagram below is the percentage of survey participants who said that they used particular Agile methodologies. We also asked the respondents about other methodologies that often come up in discussions of Agile, either as complementary approaches (for example, Lean), or as points of comparison (Waterfall being the most obvious one).



How many at each table in the Agile seating chart?
As you can tell, there's no clear winner here, other than Scrum, which overshadows every other Agile methodology in adoption. There's another way to look at the same picture: whenever there's a family photo of Agile methodologies, the participants almost always make sure to invite their poor cousins to attend. There are both good and bad sides to that earnestly ecumenical approach.

Someone once told me, "If you want to learn about Agile, read a book." Most of these references have an earnestly ecumenical approach to Agile. For example, Craig Larman's Agile And Iterative Development: A Manager's Guide gives equal time to Scrum, XP, UP, and Evo. Peter Schuh's Integrating Agile Development In The Real World covers a different set of methods, but with equal egalitarianism. Sure, these books are a few years old, but they're representative of what teams interested in Agile will find when they scan the bookshelves.

We don't need to declare a winner. A methodology that's in the minority may shoot forward tomorrow in adoption. Scrum might become so commoditized that it becomes the basis for a new set of tailored approaches, in which case the original meaning of "Scrum" winds up submerged in other things.

However, I can tell you from first-hand experience that many development teams are having  a tough time figuring out if they'll find a healthy community of fellow adopters for particular Agile methods. That's a very legitimate question, particularly since the success of Agile implementations often turns on what lessons learned a team can apply from other people's experiences . (Which is one reason, perhaps, why Agile coaching seems to be as important as it is. Among other things, a coach is a storehouse of vicarious experiences with Agile implementations.)

Everyone is welcome at every table
Actually, the survey results shown in the diagram tell a different story about methodological pluralism. In practice, teams mix and match different methodologies. The pluralism of methods within teams may be just as interesting, and maybe more interesting, than pluralism across teams. My Forrester colleagues Dave West, John Rymer, and Mike Gilpin wrote an excellent piece on using Lean to complement Agile, and vice-versa. The diagram shown here indicates that a lot of other methodologies (Six Sigma, ISO, etc.) occasionally wind up in the mix, too.

My upcoming research document, which covers a lot of other aspects of Agile adoption, provides some more information about this mix-and-match behavior some additional teams. The real topic of the study, however, is how Agile changes the way that groups interact with Development, and the effects these changes have on the company at large. I'll post when it becomes available. There's a second document in the works, too, which I'll talk about later.

[Cross-posted at The Heretech.]