What does a 21st century software development process look like?

During the last three months I have been talking to many people on what a software development process looks like to them in the 21st century — you have seen some of the initial thoughts and ideas posted in this blog.  The community seems full of different views as to what makes a process, or even if process, as we currently think of it, is right for software development teams. 

It is clear that software development is NOT the same as other manufacturing processes, and that organizations that approach software development in that manner do not exploit the opportunity for innovation or develop an efficient process.

In the pursuit of building better software, it has always been the belief of most senior managers that if you have a good process you will get great software; that problems with software can be attributed to problems with process.  Many people are now questioning that traditional mindset, believing that changing the tasks, work products, and responsibilities within the process does not lead to success, but a focus on culture, knowledge, and skills will instead improve the end product.  This change in emphasis is embodied by the Agile methods movement and described nicely in one principle in the Agile manifesto ‘Individuals and interactions over process and tools’.

Initiatives around process improvement such as the CMMI seem to be drying up, with companies either at the level they desire or seeing no real value from such work. A focus on professional development with content such as PMBOK and BABOK are gaining favor with an increased focus on the practices of software rather than the processes.

Are we seeing a tide change?  Is software development process changing focus? A new survey is currently underway to help uncover the truth about process in the 21st century.  If you have 10 minutes please take it, your contribution and support will help shape how we think about process in the future. You will also receive a copy of the report when it is published. PROCESS SURVEY

The Death of Process as a Product

During the last six weeks we have been researching the state of software development processes and a number of very interesting trends have become apparent. These trends and the results of our research will be documented in the research report Best Practices In Software Development Processes slated for Q1 2009. But I just couldn’t wait to share some of the insights and get your reactions to them before then.

Hybrid Processes take the floor

One interesting trend is the number of development leaders saying that they are not following Scrum or RUP, but instead talking about their process as a hybrid. Examples include Scrum and XP, RUP and Scrum and perhaps the most surprising DODAF and Scrum! Traditionally customization has always been a tricky job with organizations finding it hard to decide which process element to include and how these very different processes fit together. 

Process power to the people doing the work

So what has changed? Have organizations suddenly discovered the DNA of process and now can transplant X bit of one process to a Y bit of another process? Our research seems not to indicate that. Organizations are not trying to develop the perfect hybrid process, but instead leaving it to the development teams to select the bits that work for them from either experience or from the latest agile / lean method. Instead of focusing on building a perfect process and then getting all the teams to follow it, organizations are focusing on making teams more educated and process aware. The process police seem to be changing from enforcement to enablement. Here is my first pass at a three step guide to enabling practical process implementation;

• Empower teams to make process decisions – By giving teams the power to decide on the process they are going to follow, that process is much more likely to succeed.
• Educate teams on what they need to do to be compliant – Instead of hiding compliance behind a wall of process steps and artifacts, educate the team on what they need to do, when, to ensure that the project conforms to the company compliance steps.
• Enable teams to learn new processes and techniques – By encouraging a culture of continuous improvement teams are continually looking for new and improved ways to make do their work.

But with ownership comes risk

As with any change there comes a set of risks. By putting process power with the development teams mistakes may be made with time being wasted doing things that are not really valuable or activities being undertaken poorly. I encountered a number of war stories with customers starting a project and finding half way through that they really should have undertaken some work up front in the area of architecture or requirements and now are missing a key aspect of the system. In the case of these customers they highlighted two things for resolving these issues. Firstly having experienced people on the team. These may be outside consultants or internal experienced practitioners. The second tip was in the area of software development lifecycle with the implementation of a light SDLC, which describes a series of milestones and key events that must happen on all projects, but does not prescribe all the activities necessary to deliver those milestones. Another concern was highlighted by the management concerning productivity and efficiency of the teams. Allowing the team an almost free hand in the work they undertake could lead to very inefficient processes. You would have to address this with a set of measures being put in place that encourage efficiency whilst promoting ownership.

I would love to hear your feedback around the idea that the religion of process is going away and being replaced with empowered, educated and enabled teams who consider they can change the way they work. What are you finding?

The Value of Projects – Do we always need them?

When I started in software development writing Cobol, sitting in an office with my humming green monitor. I was part of an IT department that was linked to a business department.  The work I did was in direct response to my customers requests for new features or changes to existing features.  We planned work at the start of the month and worked through that list during the month.  Yes, there were cross department projects, but that was handled by everyone involved sitting in a room and saying ‘if you do this, we can do that’.  It was a simpler time, a time when you could leave your screen unlocked and you had a real lunch hour. 

Ok – I am not just getting nostalgic for the sake of it, I honestly believe there is a need to re-evaluate the way in which we use projects as a mechanism for structuring and managing work and remind ourselves that there are different organizational approaches to developing software.  This realization came to me as I talked to a number of clients about the use of Agile approaches and Lean thinking in their development of software.  Many clients have run into problems when trying to implement one product backlog, stating that they need many project backlogs and then some mechanism to bring them together.  They need a backlog of backlogs (or a scrum of scrums J), and then numerous processes to manage the impact of change across those backlogs.  The process starts getting more and more complex and harder and harder to manage.  Can the application of project structures on work delivery be artificial? 

I consider four things when deciding if work should be grouped up and considered a project.

• Size of development team – When a team is large and work needs to be planned in a large amount of detail to take advantage of these resources then it is important to introduce a project structure and appropriate management and reporting to ensure this is working.
• Location of development team – When a team is spread over many locations it is important to put in place organization constructs to better enable these individuals to work together.
• Large amounts of cross organizational dependencies – When a set of requirements require a large amount of change outside of the domain that is driving the requirements it is important to put in place a project organization that includes all those different groups.
• Maturity of the team – Detailed planning and management is often only required when the team is staffed by practitioners who do not have a large amount of experience either in the organization or the problem domain.  The project organization provides the support necessary to enable them to be effective.

Does anyone have any experience on when not to use the structures of project to organize and manage work? Or do we need different types of projects to better manage the work, with some being lighter than others?