Want Better Quality? Fire Your QA Team.

Seriously. I recently spoke with a client who swears that software quality improved once they got rid of the QA team. Instead of making QA responsible for quality, they put the responsibility squarely on the backs of the developers producing the software. This seems to go against conventional wisdom about quality software and developers: Don't trust developers. Or, borrowing from Ronald Reagan, trust but verify.

This client is no slouch, either. The applications provide real-time market data for financial markets, and the client does more than 40 software releases per year. If the market data produced by an application were unavailable or inaccurate, then the financial market it serves would crumble. Availability and accuracy of information are absolute. This app can't go down, and it can't be wrong.

Why Does This Work?

The client said that this works because the developers know that they have 100% responsibility for the application. If it doesn't work, the developers can't say that "QA didn't catch the problem." There is no QA team to blame. The buck stops with the application development team. They better get it right, or heads will roll.

As British author Samuel Johnson famously put it, "The prospect of being hanged focuses the mind wonderfully."

Can This Work For You?

This can work if the application development team is held accountable for software quality. The consequences for bad quality must be so grave that the application team has no choice other than to focus their minds wonderfully on design, development, and testing. Most of the software I have developed over the years was sans QA team. When you develop mission-critical applications for large enterprises as a consultant, you better deliver quality or heads will indeed roll. You won't get paid.

Responsibility isn't enough. To be successful, app dev teams must create test harnesses, create test plans, and use testing tools. But, because the developers know where the bodies are buried in the code, they can focus their efforts on the most brittle parts of the applications. The level of effort will depend on the type of software. The end result could be better quality and faster release cycles.

Comments

Simply firing the QA team and

Simply firing the QA team and hiring more developers is not enough -- the velocity of the dev team will increase and they will take on more story cards without necessarily improving their quality. Without proper organizational and process changes the quality will degrade -- unless you make it clear that the developers are the final checkpoint! I agree I'd *much* rather have a QA team full of developers than people who can just file defects, but you need to build quality into your development cycle -- automated unit tests, code reviews, security scans, and even a good old fashion manual functional test script go a long way.

Casting Away QA is NOT the Answer

That's saying if we take away the net, then the tightrope walkers will be more careful and not fall. My feeling is that if the financial organization had made development equally accountable for quality as the QA group, without sacrificing QA, they would have achieved similar results. Why? Because each group has specific skill sets that are meant to be a check and balance of each other. We've tried developers as testers - Capers Jones himself notes that the best we can hope for is defect removal efficiency of about 68%. I wouldn't want my customers finding that other 32%. When velocity builds and the schedule gets tight, it's human behavior to think it's "good enough" even with accountability rules in place.

Building great applications isn't the same as testing them. The team that fabricates airplanes aren't test mechanics or test pilots - that's a specific skill. Would you want to get on a plane that hasn't been thoroughly tested by a professional? Delivering software is a team effort - everone is important, from the project managers and BAs to the developers, tester and supporting stakeholders along the way. Firing a group just to make another group step up highlights a management 101 problem. When management smartens up and realizes that responsibilities are equal and the relationship between the groups can be positive and not antagonistic, everybody wins. Instead of firing QA - why not incent them to work together as a seamless team?

not quite

Actually, it's more along the lines of, "if there's no net, the team will build a bridge instead of a tightrope." It forces the team to really eat their own dog food. And who wants to eat dog food?

Wow, I'm really disappointed with this post

You wrote:

"To be successful, app dev teams must create test harnesses, create test plans, and use testing tools. But, because the developers know where the bodies are buried in the code, they can focus their efforts on the most brittle parts of the applications."

Why should there be any "bodies buried in the code" to begin with?

The issue here is about accountability and responsibility. If an organization can't structure and run itself to have responsible people and hold the laggards responsible, that's a much larger problem.

BTW, it seems by the same logic, the entire Marketing team should be fired, and Sales should be held accountable for all the lead gen etc. i.e. right now they can point the finger at Marketing for bad leads or poor coverage, but if they are on the hook the job will be done right, correct?

Saeed

Developers often know where the problems might emerge

What I meant by "bodies buried in the code" is that developers know where there are likely to be problems when they change the code. That way, testing can be focused on those areas. As I say in my post, development teams that are repsonsible for their own quality should use test harness, testing tools, and tracking. Most development teams do this all ready.

I don't think your marketing and sales analogy is comparable to development teams doing their own QA. Application development create software. QA is a function that also helps create software. In constrast, marketing creates demand and leads. Sales closes deals. I am sure you don't think marketing should have a marketing qa team. And, sales should not have a sales qa team.

I agree with you that the issue is accountability and responsibility. That is why I was fascinated to hear this story by the firm that does not have qa team.

Yes. but ..... :-)

Mike

Thanks for the response. WRT to your client, it would be very helpful to find out the back story that led up to the org change. i.e.

- what political dynamics were there between Engineering and QA?
- What was the skill set of the QA team members?
- Were they empowered to get their job done. i.e. authority, time, resources etc.

These and other questions would help us better understand why this change was made, because clearly context is important if others were to consider this same approach.

There was a time when there was no distinct QA function. SW developers did exactly as you describe. They were 100% responsible for the software functionality, quality, performance, optimization etc etc. But over time, as software became more complex, there was a clear realization that it was not a scalable or efficient way to work. Even the most conscientious developers couldn't ensure that their code met every requirement as expected by the client or user, and couldn't test it in all the ways it needed testing.

In short, an independent team was needed who could test, stress and otherwise ensure the code met the necessary requirements, AND who could do it without the inherent assumptions (and biases) of those who wrote the code in the first place.

In the vast majority of software companies, this model works. Even Agilists who want to focus on producing code, do not eliminate the need for QA. In fact, they advocate pairing or teaming of engineers and QA so that there is clear communication and close interaction between the two.

Of course there are cases where developers can do everything, but these are the clear exception to this model.

Sorry for the preamble, but getting back to your post -- as to the "buried bodies", thanks for the clarification. Knowing where there might be impacts is a good thing, and certainly as part of the resolution to problems it's important. But QA's focus is to find the problems not necessarily fix them.

As to my analogy, let me clarify, as I think you are taking it far too literally.

In your post you wrote:

'If it [the software] doesn't work, the developers can't say that "QA didn't catch the problem." There is no QA team to blame.'

This blame game is rampant in the sales/marketing dynamic. Sales teams frequently point the finger at Marketing, using poor quality leads as a reason for lagging sales. i.e. the same finger pointing/blame game dynamic exists because of the separation of the responsibilities -- lead gen vs. closing deal.

But if one team (e.g. Sales) was responsible for both, they could not point the finger anywhere as only their necks are on the line. i.e. "The prospect of being hanged focuses the mind wonderfully."

BTW, Tom Grant wrote a great post refuting an analogous situation at Mahalo, WRT Engineers and Product Managers. It is worth reading.

http://blogs.forrester.com/tom_grant/11-01-17-product_management_you_get...

Saeed

Well... I usually see bad

Well... I usually see bad stuff on internet, but this is just the worst... developers that just work good when they are pressed should be fired... it is a terrible idea that if you fire your qa team the developers will work better knowing that they have full responsability over the product... QA is supposed to make sure the delivered product is ok... developers are supposed to make a good work, but they are human, so errors can happen always... that is the reazon why there is a QA team....

Does QA need a QA team

I see your point. But, QA is human too and makes mistakes. I am not advocating that QA is not needed in all circumstances. But, for many organizations application development teams can be responsible for their own testing and quality of their software.

Instead of firing QA, stop marginalizing them.

Mike - As always, thanks for challenging the status quo and getting these conversations started.

I agree with Margo that this isn't a question of incentive, it's a question of skills, competencies and effective program management. For some organizations I've worked with, QA is the red-headed stepchild of the software development lifecycle - given only marginal resources and a brief amount of time as a percentage of the whole project plan to perform their function appropriately. In my experience, both requirements capture and QA are both often short-changed to make room for additional development time. But if requirements are captured and documented effectively, development should be more efficient and effective and QA more successful.

A few additional points to negate the feasibility of killing QA for most complex IT organizations:
(1) To require developers to become experts at the processes, best practices and methodologies of quality assurance may create more breadth of their expertise, but seriously thin their depth of technical competency.
(2) QA isn't just about ensuring "does the code work". It asks whether the code is effectively delivering on the business requirements provided. The QA team, if properly staffed and trained, should act as a critical liaison between the front-line business process owners that define the employee-, customer-, or partner-facing processes the applications support with the developers that are building these capabilities. These business/IT translation skills- often performed by experienced and tenured business analysts within the organization - help fill the gap between "perfectly good code" and "code that actually addresses business needs".

Thanks for out-of-the-box thinking!

Great post, Mike! I totally agree that software QA is not working the way it should. It mostly causes a lot of delay and it does not assure SW quality or application quality for new code and new applications.

Therefore we are moving code-QA into development and we look at application quality in a so called Solution Center that is purely focused on the application functionality with the customer/users.

But we still have to retain some SW-QA because we are a vendor and we have thousands of customers using our existing SW. What SW-QA is needed for is for regression testing to see if the NEW VERSION of a SW still does what the old versions used to do.

So we can't live without QA, but I totally agree that the responsibility for quality should not be in a singular QA department.

1) Code quality - R&D
2) Regression testing - SWQA
3) Application quality (function & usability) - customer service

Thanks, Max J. Pucher
Chief Architect ISIS Papyrus Software

Qa to catch the outliers

A voice of insight and reason as usual. Developers should be accountable for quality and QA should perform regression testing ti find the outliers.

You have a typo in your

You have a typo in your response, "ti - should be to"

Regards,
SQA Engineer

Always account for Variable change

There's so much more to SQW than to pick typo's up. Agile requires us all to champion Total Quality Management throughout the ALM, not at 'checkout'.

A businessman who doesn't see any value in Marketing will soon hang this sign on his door: "BUSINESS FOR SALE."

I am not sure as how to react

I am not sure as how to react to this, Fire QA and make Dev accountable for quality and fire dev if they dont meet quality.I think you client was more into operational business where he was routinely getting the same input and was giving out the same output with almost same clients with similar behaviour everyday to deal with.

QA is more than the testing,they validate the requirements,ensure what developers build is what customers want etc etc.

"real-time market data for financial markets" this is where I smell bad.I have some experience in knowing as how exactly these things work.No one wants to share that kind of information outside the trusted network.Imagine if someone outside actually knows as how NYSE works,where all it sends feeds,in what format, etc etc..

i feel you still need to ask more questions to the client,same version of software released 40 times with each time some patches or some different softwares etc etc.

The key is that App Developers Will be "Fired" too

I think the key point of this story is that developers must have their jobs on the line to "incent" them to deliver better quality. The existance of the QA team, can make app dev teams lazy because they know the QA team may catch the bugs.

@Mike, I humbly disagree

@Mike,

I humbly disagree personally.I always hate to have QA as a gatekeeper.But again I know you are correct , most dev teams feel that having QA means its job of QA to catch bugs.
However firing is just not a right option.Its upto the leadership teams to come up with right processess where each team adds some definite value and owns responsibility and accountability for their work.

Dont' Fire QA, Embed QA in to Agile Teams

This is a good concept - have the team of engineers responsible for creating the software also be responsible for proving the software works. I agree that you should break down the barrier between a development team and a QA team. I disagree that you should fire your testers. You should Embed them in to Agile teams.

More comments on my blog http://www.allianceglobalservices.com/blog/drader/dont-fire-qa-embed-qa-...

Hidden bodies and false economies

I think it's an interesting conversation, but it's a little scary to me. I have seen situations where organizations have literally gutted their QA group with this kind of crazy belief that QA is just like development, only less so.

I don't insist that every team has to have some number of people on it with "QA" on their business card, but the following functions need to be there:

- an ability to get consensus from groups of stakeholders (maybe it's your product owner, maybe it's your BA, maybe it's your PM/IM, or maybe it's your lead developer, but no matter who it is, it's a person with skills in facilitation and communication)

- an ability to develop software, and automated tests (this must be people with development skills--if you're an old-fashioned "programmer-analyst" working on a small team, maybe it's you!)

- an ability to think of edge cases, design test harnesses, and apply the right level of rigor to testing across unit, functional, integration, performance, and UAT levels, and know what must be regression tested and assure that it is. Additionally, it's helpful if "someone" can be doing open-ended experimental testing. This could be your developers, or dedicated QA people. It could be perfectly fine if the person with the devious mind for edge cases wasn't a programmer, and couldn't actually fully implement all of the test cases. If the developers aren't going to come up with those things, it's still great if they can automate functional and other tests.

If all of these functions are covered by someone on the team, and everyone on the team has enough time to get their work fully done at a sustainable pace, I won't quibble. But I can't agree with the philosophy that only the programmers "know where the bodies are buried." There's a separate skill belonging to people who are expert in quality which involves thinking creatively about where other bodies, to extend the grisly metaphor, might be distributed in pieces or hidden behind walls. That is a specific skill, and it's one we all need to respect.

Cheers,

Elena

Excellent points

Developers are often too close to the code to see the edge cases.

Not going to work

Yeah we got a QA team that often does not rise to the occasion and find all the bugs. But they do find things that were just overlooked by developers. To hope that developers can be more careful if they are held accountable is wishful thinking.

I tell you what though. I would not mind if QA disappeared and my salary was doubled. Maybe then I would produce less buggy code. Honestly though, it would not be the end all.

More money - better quality software?

I hear you. What if I said you would get a bonus based upon fewer bugs?

Worst. Idea. Ever.

Worst. Idea. Ever.

Why this post is correct?

I think we need to spend sometime to understand why this topic has come to the forum at all? It would'nt have come if the QA is team is doing what they are supposed to be doing. Nowadays the trend has been to employ 30% to 40% of dev team's size as QA and most of the times they do only black box testing which most of the automated testing tools are doing now..

Why we are discussing this topic is, QA adds to the project cost heavily but the outcome is so small which is not justfiying the investment. A solid QA team requires in depth knowledege of the application / domain and knack of looking the application from the users point of view. They should also have the ability to forsee requirement changes and problems ahead...

Seriously?

It sounds like your client was finished with the product and the dev team was responsible for minor changes and just took up the task of the QA processes. It's easy to look like a hero at that point.

As a dev, if I work for hours to making button A cause effect B, it's hard for me to look at the results and think "well gee, there is a Z here", because I'm so focused on making it work to spec. You *have* to have someone beat on it and find the problems that you could not foresee as you were mired in the trees and not seeing the forest.

My boss tried the tactic of "beating the dev team until they didn't produce any bugs". I told him if he expects a developer who makes zero mistakes I'll leave and let him find one. Instead we hired a QA person.

1995

It must have been 1995 (!) or so when our QA department was already reduced from 20 to 3 people. For the same reasons. When there is a QA department it is much more difficult to get the reponsibility where it should be: with the application development / maintenance team: designers, developers, testers, team managers.

Whole Team Approach

While I agree 100% that the entire development team is responsible for software quality, and that they need to commit to delivering the best possible quality code and learn how to do that over time, I do not equate this to "fire the testers". I've been a tester on teams with this commitment for many years - agile teams since 2000, but I've been on a good waterfall team too - and the testers have made contributions to quality that are just as valuable as the programmers' contributions. None can work in isolation. It takes a village, a diversity of viewpoints and skill sets.

If you posted this just to stir up discussion, then I guess you succeeded. If you really believe that most development teams do not need professional testers as well as programmers and other roles, that is contrary to my experience. I've met a lot of teams over the years, and very few could do a great job of all aspects of quality (especially understanding what the customers really want and delivering that) without testers.

Developers are not immune from firing

If you fire the QA team, you have to also be willing to fire the app dev team. That is the ultimate accountability. This is hard in practice because it is not so black and white. But, I think in general QA teams give application developers an excuse for being lazy and, dare I say, stupid.

p.s.

The term "developer" applies to each member of the development team, everyone who contributes to developing the software. This means testers, sys admins, BAs, DBAs, UX designers, etc., not only programmers. All of these people need to work together as a team - at least as a virtual team - to produce high quality software that really delivers business value. It's not that hard to produce code without "bugs" if by "bug" you mean null pointers and index out of range type errors. But it IS hard to produce code that really does what the business needs it to do. It takes a long time for teams to master this, and they need the roles and skills they need for their situation.

Agree that quality is more than just bugs

Excellent point that that qualilty spans more than just developers. Forrester defines quality as: Software that meets business requirements, provides a satisfying user experience, and has fewer defects. All contributors must be held accountable.

A certain logic

There is a certain logic to it. 'We fired our testers and all of a sudden we had less defects!"

Developers use QA as a debuging function

Yes. Developers under pressure to write code may cut conrers thinking "Ah. What the heck. If this doesn't work QA will catch it.". If there is no QA, then there is no one to catch it so developers work harder to get it right.

You Missed the Point

Think about the comment a little longer. Hint: it's a little tongue-in-cheek...

LOL

Comic relief. Thanks

I don't think anyone denies

I don't think anyone denies that testing is a good idea. I don't think anyone denies that having more people looking at code and testing systems reduces bugs if they understand the domain and or the coding. Testing is essentially deriving assurance that a system works by validating a sample of possible cases and any reasonably complex system will only be tested on a small proportion of scenarios. If however QA don't understand the domain well enough, or the sampling for the testing is not well enough directed at the the higher risk in terms of impact and frequency of errors then it's more than conceivable that increasing the accountability of the developers will outweigh the impact of the loss of a separate QA function.
Having QA as a separate team results in an us and them scenario and increased time on any release as clearly pointed out by those looking at continuous deployment

I didn't say "have a separate QA team"

I guess I didn't explain myself well. The testers should be integrated into the development team. I have seen situations where a separate test team still made sense, such as when testing embedded software and the dev team doesn't have access to all the systems for production-like testing, but they should work closely with the development team throughout the coding.

As Elisabeth Hendrickson says, testing is not a phase. Testing and coding are two integrated activities of software development.

In general, what software testers do isn't "QA" either, but that's a whole 'nother discussion. When people use the word "QA team" it is often a sign to me that they really don't understand testing very well or have a lot of experience with it.

I agree, because programming is like playwriting!

The software-as-a-profession debate continues largely untouched by each generation's innovations in production methods. Most of us in the 90s thought that formal methods, reuse and enforced modularity would introduce to software some of the hallmarks of real engineering -- predictability, repeatability, measurability and quality. Yet many of the human traits of software-as-a-craft remain with us.

We should listen to radical ideas like Mark Gualtieri’s, rather than maintain a slavish devotion to orthodoxies liek QA. We should recognise that software engineering is inherently different from conventional engineering, because software itself is a different kind of material, and much less amenable to regular governance.

Computer programming is more like playwriting than engineering. In both software and playwriting, structure is almost entirely arbitrary. Because neither obey the laws of physics, the structure of software and plays comes from the act of composition. A good software engineer will know their composition from end to end. But another programmer can always come along and insert their own code as they see fit. It is received wisdom in programming that most bugs arise from imprudent changes to old code. Messing with a carefully written piece of software is fraught with danger, just as it is with a finished play.

Code is so very unlike the stuff of other professions - soil and gravel, metals and alloys, nuts and bolts, electronics, even human flesh and blood - that the metaphor of engineering in the phrase “software engineering” may be dangerously misplaced. By coopting the term we have might have started out on the wrong foot, underestimating the fundamental challenge of forging a software profession. It won't be until software engineering develops the normative tools and standards, culture and patience of a true profession that the software crisis will turn around. And then corporate governance will have something to govern in software development.

Well said. The industrialization of software has failed

Programming is indeed like playwirghting. I would go further and say that it is like producing the the play or the movies. Software is a one off, not making the same Toyota Camry over and over again and then tweaking the process. You are my brother!

Think of QA like an Editor

Even in playwriting (having written a couple myself), there are revisions, refinements and reviews, that not only look for places where the pacing and dialogue fall flat (a defect perhaps?), but also to ensure that the process of playwriting ensures that characters are appropriately developed. Even creative pursuits have checks and balances, guys.

Is app developing an art then?

Where are the logic, math, purpose or objective related to the outcome of any software piece? It's like having to sit on a beautifuly-crafted amazingly-designed unconfortable chair. My confort is what counts.

How is this new/radical?

The idea of an integrated dev team where testing and coding are done together (tests first) has been around a long time. I've been a tester on teams like this for 11 years. Works great. But I think it works great BECAUSE we have testers as well as programmers and others.

QA is not the referee, but instead the goalie on the team

In many large enterprises, QA team are outsourced or insourced. A QA team that is embedded in the app dev organization is, essentially, part of the app dev team. If one of your developers consistently has bugs there will be hell to pay. Rather than QA being a referee on the soccer field, the QA person is the goalie on the team.

I'm not the testing police

Nobody on my team ever has "hell to pay" even if they do accidentally introduce a bug. It's always a team problem, and we learn from mistakes. As pointed out in the original post, it's bad when programmers think "Oh, Lisa's a good tester so she'll catch any bugs, I don't need to do any testing myself". Collaborating produces the best results.

QA bad cop; tester good cop

How did we go from QA to testers as the theme of this thread? Maybe some use the terms interchangeably. I don't. I read Gualtieri's blog as arguing against the efficacy of orthodox Quality Assurance (which I interpret to mean process auditors) but not against testing. He didn't have anything bad to say about testing. I myself take it for granted that multi layered testing is essential (though not actually sufficient) to deliver quality software. Unit testing, integration testing, user acceptance testing, black box testing, white box testing, independent testing, bash testing etc etc.

But to me, QA means those Jiminy Crickets that sit in on meetings and sign off that the SDLC process has been followed every step of the way. They have job descriptions rooted in the traditional disciplines of inspection and manufacturing standards, but these activities don't have proper equivalents in software, mainly for the sorts of reasons I touched on above.

In principle I am actually sympathetic to software QA because really the only thing we can measure in software quality is adherance to process. It is the integrity and care of the composition of software that determines its quality (and not the tools or materials). But in practice I fear that software QA rather too often reduces to ticking boxes, and that's not even necessary for good quality, much less sufficient.

QA used as a misnomer for testing

IME companies call their testing department their "QA department" and their testers "QA testers". I don't know why. They aren't doing QA. There may be some companies somewhere that somehow have a true QA department that is different from development, but it's hard to imagine how that could be.

Since so many ill-informed people use "QA" as a synonym for testing, that is how I read the blog post. If that was not the intention of the blog post, then it might have been good to clarify that the "QA" departments he's talking about have nothing to do with testing.

QA and Testing

I was just about to add a similar reaction on this topic. Thank you for doing that already. To me testers are part of the development team as well. QA is an independent team. They do some random additional reviewing on behalf of the top management. QA should never be responsible for testing the new products or the quality of the new products.

By the way: in a bureaucratic organization a QA department has power, in a customer oriented organization QA is only supportive.

QA - wha?

Let me get this straight - the Quality Assurance team should not be responsible for quality?

Indeed

"Let me get this straight - the Quality Assurance team should not be responsible for quality?"

They should not be responsible for the quality of the individual products and services. The responsibility for Quality should always be within the line organization. QA is staff. They do have the reponsibility to report to the management when products or processes are not according to the expectations/requirements (via internal ISO 9000 audits, for instance). They also have an important (advising/supporting) role in defining the quality system. The management makes the decisions and takes the measures.

I have been a quality manager myself for many years and had to get used to this way of thinking. But as long as QA is responsible for the quality you will find mistakes/deviations in a later stadium or not at all and the developers etc will always point at QA when something has gone wrong instead of taking their own responsibility.

Besides one can not expect that not only the developers know the business processes that are supported by the information systems well, but also the QA staff has enough knowledge of all the business processes of all the systems that are made by all the development teams.

QA and Projectplans

What I forgot to mention: a QA officer can play a very good role in reviewing project plans. However, decisions on starting a project are not can not be made by QA. Steering committees or the Board can make these decisions.

Clarify please...

@Machteld I'm afraid at this point you're just muddying the waters. It sounds to me like you work for a very large company and QA means something quite different to you than it does to most of us working in startups. I'm sure yours is the more accurate terminology but it does nothing for the original argument one way or the other. I'm pretty sure if the original post meant QA in the way that you do, it would be quite the non-issue since the product had been released and was in the process of maintenance and upgrades.

I've worked with good QA professionals (which is what they call themselves, but it sounds like the should call themselves testers?), and while they tend to get grief from developers, I know how valuable they are. They will often find bugs (or lack of functionality) that I just never would have found. They also will test in different operating systems and different browsers (assuming of course this is web dev) which is pretty difficult for a developer to do when he has three months of tasks to do in one month in order to get further funding and make payroll.

What is QA...

Rereading the article, I see that the author links to his own definition of QA: http://blogs.forrester.com/mike_gualtieri/10-07-18-software_quality_more... (do links work?) I think that confirms he is referring more to bug testing and functional requirements than he is to the planning of prototypes and goals and milestones, choosing the equipment, setting up a development, testing, and production server, and the architecture the development process.