Stories about Software


How to Actually Reduce Software Defects

Editorial Note: I originally wrote this post for the SmartBear blog.  You can check out the original here, at their site.  Have a look around while you’re there and see what some of the other authors have written.

As an IT management consultant, probably the most frequent question I hear is some variant of “how can we get our defect count down?” Developers may want this as a matter of professional pride, but it’s the managers and project managers that truly burn to improve on this metric. Our software does thousands of undesirable things in production, and we’d like to get that down to hundreds.


Almost invariably, they’re looking for a percentage reduction, presumably because there is some sort of performance incentive based on the defect count metric. And so they want strategies for reducing defects by some percentage, in the same way that the president of the United States might challenge his cabinet to trim 2% of the unemployment percentage in the coming years. The trouble is, though, that this attitude toward defects is actually part of the problem.

The Right Attitude toward Defects

The president sets a goal of reducing unemployment, but not of eliminating it. Why is that? Well, because having nobody in the country unemployed is simply impossible outside of a planned economy – people will quit and take time off between jobs or get laid off and have to spend time searching for new ones. Some unemployment is inevitable.

Management, particularly in traditional, ‘waterfall’ shops, tends to view defects in the same light. We clearly can’t avoid defects, but if we worked really hard, we could reduce them by half. This attitude is a core part of the problem.

It’s often met with initial skepticism, but what I tell these clients is that they should shoot for having no escaped defects (defects that make it to production, as opposed to ones that are caught by the team during testing). In other words, don’t shoot for a 20% or 50% reduction – shoot for not having defects.

It’s not that shooting for 100% will stretch teams further than shooting for 20% or 50%. There’s no psychological gimmickry to it. Instead, it’s about ceasing to view defects as “just part of writing software.” Defects are not inevitable, and coming to view them as preventable mistakes rather than facts of life is important because it leads to a reaction of “oh, wow, a defect – that’s bad, let’s figure out how that happened and fix it” instead of a reaction of “yeah, defects, what are you gonna do?”

When teams realize and accept this, they turn an important corner on the road to defect reduction.

What Won’t Help

Once the mission is properly set to one of defect elimination, it’s important to understand what either won’t help at all or what will help only superficially. And this set includes a lot of the familiar levers that dev managers like to pull.

First and probably most critical to understand is that the core cause of defects is NOT developers not trying hard enough or taking care. In other words, it’s not as though a developer is sitting at his desk and thinking, “I could make this code I’m writing defect free, but, meh, I don’t feel like it because I want to go home.” It is precisely for this reason that exhortations for developers to work harder or to be more careful won’t work. They already are, assuming they aren’t overworked or unhappy with their jobs, and if those things are true, asking for more won’t work anyway.


And, speaking of overworked, increasing workload in a push to get defect free will backfire. When people are forced to work long hours, the work becomes boring and “grueling and boring” is a breeding ground for mistakes – not a fix for them. Resist the urge to make large, effort-intensive quality pushes. That solution should seem too easy, and, in fact, it is.

Finally, resist any impulse to forgo the carrot in favor of the stick and threaten developers and teams with consequences for defects. This is a desperate gambit, and, simply put, it never works. If developers’ jobs depend on not introducing defects, they will find a way to succeed in not introducing defects, even if it means not shipping software, cutting scope, or transferring to other teams/projects. The road to quality isn’t lined by fear.

Understand Superficial Solutions

Once managers understand that eliminating defects is possible and that draconian measures will be counterproductive, the next danger is a tendency to seize on the superficial. Unlike the ideas in the last section, these won’t be actively detrimental, but the realized gains will be limited.

The first thing that everyone seems to seize on is mandating unit test coverage, since this, presumably, forces the developers to write automated tests, which, in turn, catch issues. The trouble here is that high coverage doesn’t actually mean that the tests are effective, nor does it cover all possible defect scenarios. Hiring or logging additional QA hours will be of limited efficacy for similar reasons.

Another thing folks seem to love is the “bug bash” concept, wherein the team takes a break from delivering features and does their best to break the software and then repair the breaks. While this certainly helps in the short term, it doesn’t actually change anything about the development or testing process, so gains will be limited.

And finally, coding standards to be enforced at code review certainly don’t hurt anything, but they are also not a game changer. To the chagrin of managers everywhere, “here are all the mistakes one could make, so don’t make them” doesn’t arise from the past experience of the tenured developers on the team.

Change the Game

So what does it take to put a serious dent into defect counts and to fundamentally alter the organization’s views about defects? The answers here are more philosophical.

The first consideration is to get integration to be continuous and to make deployments to test and production environments trivial. Defects hide and fester in the speculative world between written code and the environment in which it will eventually be run. If, on the other hand, developers see the effects their code will have on production immediately, the defect count will plummet.

Part and parcel with this tight feedback loop strategy is to have an automated regression and problem detection suite. Notice that I’m not talking about test coverage or even unit tests, but about a broader concept. Your suite will include these things, but it might also include smoke/performance tests or tests to see if resources are starved. The idea is to have automated detection for things that could go wrong: regressions, integration mistakes, performance issues, etc. These will allow you to discover defects instead of customers discovering them.

And, finally, on the code side, you need to reduce or eliminate error prone practices and parts of the code. Is there a file that’s constantly being merged and could lead to errors? Do your developers copy, paste, and tweak? Are there config files that require a lot of careful, confusing attention to detail? Recognize these mistake-inviters for what they are and eliminate them.

But here’s the thing – I can’t possibly enumerate all of the tools in your arsenal. These are some of my most tried and true strategies, but you’ll have to figure what works for you. The key is to recognize that defects are not inevitable and go from there.

  • I largely agree with this, and definitely agree with the sentiment.

    However, I think a grey-area here is the definition of defect. In my experience, once you’re running a fairly tight ship with good continuous integration, many of your “defects” boil down to a misunderstanding of requirements. This could be bad communication from the business, poorly documenting the requirement, or a simple misreading of the same, or some combination of all three. In any case, it’s not what the customer wanted so it’s a classified as a bug.

    But in these cases, no amount of automated testing is going to fix the problem, as the assumptions of those writing the tests are incorrect.

    I think your point about feedback loops is the solution here, too. The question is when is it found? Is your business performing UAT frequently? Even in agile shops this can be a struggle.

    • When I think of defects in the sense I was talking about here, I think in terms of the software doing things that the dev team didn’t intend. So that wouldn’t include requirements misunderstandings, since then the software is still behaving as the team expects.

      Generally speaking, I think of that sort of miscommunication as being a different (though no less serious) problem than the one I’m describing here.

    • Dan Olesen

      Hi Ross – that is an insightful observation; “defects” boil down to a misunderstanding of requirements.
      From my many years in software development, I’ve concluded the same …

      So the question is; what’s the cure ???

      Since all projects are unique in many aspects, it’s necessary to look at what these projects have in common. (From a philosophical point of view; “if they all have bugs to cope with, the root cause is most likely in what they have in common …”)

      The common part are the requirements, and the way they get misunderstood.
      This is not a problem with a small list of requirements; everybody can see the big picture and how each requirement collaborate with other requirements to specify the desired solution to a customers needs.
      As the collection of requests grows larger, it becomes increasingly harder to see if a requirement is in sync with the desired solution; even the project managers can’t get the “helicopter view” on the interaction of the individual requirements in the spec.

      As Mr. Dietrich states in the above article; enforcing good coding standards, continuous integration etc. doesn’t hurt, but doesn’t solve the problem since the root cause is somewhere else. A problem should be dealt with at it’s root …

      The answer is to get the specification right; too often the specification grows out of control, becoming a monster with more than 1000 requirements. When this happens a lot of project managers gets overwhelmed by the size and looses the overview.

      Even if the project has the good practice to do “early testing” of the requirements, it is often too difficult to see the “full picture” when things get that big. Often the projects try to cope with this complexity by “slicing and dicing” the requirement until the size is manageable, and attempts to get the customer to sign off on the reduced requirements.

      This is a mistake – since the remaining requirements may be missing the requirements they should interact with – so now the spec isn’t too big but it’s no longer coherent.

      So why do I go on about the specs?

      Well – I’ve been working as a developer for more than 20 years and then I changed to test.

      What I’ve learned during all those projects is that:
      – at least 85 % of all errors are due to bad specs – not misunderstood specs, but WRONG specs.
      – when the amount of requirements increase to more than 500 it becomes too difficult for most people to keep the requirements coherent and still cover the customers needs.

      – the later an error is found, the more money it will cost the project.

      In order to mitigate these difficulties large specs needs to be broken down in several areas with a clearly defined interface – just like “little black boxes”. In these specs for each area care should be taken to not address any requirement outside the area, so each requirement is owned by exactly one sub-project. This stuff is in all the textbooks on managing SW development, but sometimes it gets forgotten when the projects meet the real world.

      A list of things that can cause the requirements coherence to break is:
      – the project should be delivered yesterday, so we have to finish the spec fast…
      – this is what the customer want’s (they signed off on the spec and even if we’re suspicious about coherence we’re not gonna break a done deal…)
      – these are the requirements that the customer came up with and we signed the contract (Hmmm – in most cases the customers aren’t software developers, so you should have architects and QA look over requirements and maybe prepare a workshop with the customer. That way things can get fixed before they cost too much).

      Once you’ve assured that your requirements are coherent and unambiguous you can start the test planning, the development and all the other activities …
      If things get “jump started” the project will just have to “do it over”; which in the end always cost more than “doing it right the first time” …


  • I like this article.

    • one minor nit: “These will allow you to discover defects instead of customers.” — discovering customers is usually a good thing (i.e., you might want to reword that?)

      • lol… good call. I fixed it.

        • Batroid

          These will allow you to discover defects instead of “the” customers… Bug are every where.

    • Thanks!

  • Kirk W

    We have found that doing code reviews of the specific fixes being applied has helped us to see what the problems are, and have helped to clean up a whole bunch of other errors, as well as reminding developers to get questions answered before making some of the changes requested. (The majority of errors were side-effects of requested code changes. That process had to be reviewed. The developer was not brought in soon enough to ask the correct questions and was being forced to make assumptions)

    • Is it fair to say that the majority of errors were regressions, then? Did the code in question have some kind of automated test suite, or do the reviewers serve in this capacity?

      • Kirk W

        Erik, most of our errors were NOT regressions. They were side-effect errors from recent changes. Sorry for the lack of clarity. Fix A, introduce A’ as a smaller, minor issue (some place on the screen not updated. Or this needed to be on screen 16 as well. More a lack of thoroughness in definition of the issue we are addressing.)

        Not issues with this coming back to life after being fixed.

    • nordlyst

      My experience with code reviews (by peers) is also very positive. I don’t know if the defect rate was much affected, but I am convinced it would in the long run. We emphasized that code should be clear in our reviews – meaning that it should reveal the intentions of its author as clearly as possible and it should be as easy as possible to understand what the code does. Both of these things contribute to making all of the undocumented details of the inner workings of the system more accessible to the people who come after to maintain and extend the code, which reduces the need to make assumptions – and saves a lot of time.

      • I am with Erik on his thesis, and I might suggest that most of the typical agile practices merely allow us to make mistakes faster (and fix them faster), and his approach which is better and faster continuous integration is another way of fixing the mistakes we make faster, faster – and with higher visibility.

        • Well said, thanks. I don’t believe I specifically articulated this in the post, but so much of *escaped* defect prevention isn’t about trying to avoid all mistakes. It’s about creating a working paradigm where you discover them fast, early, and automatically, before people outside of the team are ever exposed to them. CI and a lot of craftsmanship/agile practices seem to drive at speeding up the feedback loop for just this purpose.

    • NitzMan

      With regard to the side effects, having unit tests as part of your continuous integration can help remove these. I’d rather have a build fail because a test didn’t pass than search for possible side effects manually.

  • Joel McIntyre

    Another systematic successful way is to get more brains on the problem, especially ones with different perspectives and thinking styles. Programmers are “make it work” kind of people; get some “make it break” and some “use it like a customer” people, make sure to have big picture and detail oriented people taking a look.

    When a test team works, it’s because it is accomplishing this, but you can do it without “testers” and you can have testers and fail to do this. For example, I’ve seen success when you get the PM/designer role involved in an acceptance sign-off pass.

    This also can help developers and the team go faster through overlapping specializations. A good dev/test team is faster than having the same number of bodies all in development, and produces better quality code with fewer escaped customer facing defects. It doesn’t have to be a big team, either – I was once on a good one with 2 developers (dev manager + me) and 1 tester.

    • I personally think that putting different personas into the mix and giving them an interest in the product is a great idea. It sounds like you have a good recipe for success.

      In shops I’ve consulted with over the last several years, I’ve been part of efforts that install this as something, in the agile community, called “three amigos” discussions. Developer, business analyst and QA person all get together ahead of implementing a feature, agree on what success looks like up front, and design some kind of acceptance test. Agreement on done is then academic.

      The nice thing about automating the acceptance test for any given feature is that it provides a nice, robust acceptance test suite as you go to guard against future regressions.

      I’ve been impressed with how adept some teams get with acceptance test frameworks, such that they can really write the tests in language that the BAs or various stakeholders can discuss in plain English. I’ve seen a lot of requirements misunderstandings averted up front this way.

      • Joel McIntyre

        That’s fantastic, I like the sound of that! “Three amigos” – I’ll have to follow up on that, thanks!

  • JerryGirard

    Where is the word “design” in this article? Before I even read the article, I did a text search for the word “design”. I found none, so I did not bother to read the article.

    • That’s an interesting algorithm.

    • JohnAdams_1796

      Just because the article does not use the word ‘design’ does not mean it does not address design. As you must know as a software developer, all of the measures discussed in this article are part of the design process. The design of a software system is not complete until the last code change is tested and verified.

  • nordlyst

    Well, I read the whole thing and apart from setting no defects as the goal I can’t really say what the author thinks I ought to do!

    The complete lack of any discussion on software design is a huge omission. Even if one isn’t designing new features from scratch, design decisions are being made whenever one decides to make changes to software. And it is the decisions one makes here more than implementing them that tends to cause errors – especially the errors that are likely to escape early detection. This is to be expected, since testing features is always easier than testing if the system operates correctly in all conditions – such as when users perform actions in another order than the expected one, or multiple users are interacting simultaneously. Many mistakes can be avoided by systematically looking at the assumptions baked into the *design* and checking whether they necessarily always hold true.

    • Regarding what you ought to do, the thesis of the piece was that any individual software team participant should consider shifting mindset to one that doesn’t regard defects as inevitable.

      It’s hard for me to speak to the comparably nebulous concept of “design” since my position would be that this is not a different activity from implementation, except, perhaps in a phased development approach (requirements phase, design phase, implementation phase, testing phase). I’m not saying this to be evasive, but because I honestly don’t know where the line is drawn in any given shop.

      Is “design” a question of how different layers of the software communicate? Which design patterns are used? Whether you limit class and method length? Whether you enforce CQRS? And, if it’s any of those things, why is this some kind of special activity that isn’t just “making software?”

      • nordlyst

        Fair enough. I actually did get that it was a matter of mindset, not merely goals. Even so, I don’t think it told me very much about what to concretely do.

        Re: design. Yes. Design is ALL of those things!

        It is a mistake in my opinion to think that just because there is no formal design phase in agile it isn’t worth making a distinction between design and implementation. Every decision you make about what exactly the software should do is ultimately a design decision. Implementation is ideally reduced to actually typing the code, and perhaps choosing how to format it.

        Even in waterfall approaches there would always be a lot of design left to do in the formal “implementation phase”.

        The way I see it, agile is a useful reaction to an earlier approach that unrealistically assumed we could figure everything out in principle and then go do it. But this doesn’t change the fact that in order to do anything you need both a clear idea of what you are attempting to accomplish and some sort of plan (even if it’s only in your head) for how you are going to achieve it. That plan is a design, and the process by which you arrive at it greatly influences the chances of ending up with correct software.

        It isn’t always quite that linear of course. Often you have a only a rough plan and then build part of the system. That is perfectly fine as long as you are sure the part you are building will be ok the way you designed it. I may know that I need a set of methods to perform certain calculations and build that without having figured out how information will be stored or presented.

        I am not saying I have the answer. But then again I didn’t publish an article with a headline saying I do.

        • I’m legitimately interested in the distinction between design and implementation. That is, I view it as nebulous because I don’t know where one begins and the other ends. But that’s not to say clear (and useful) distinctions couldn’t be made.

          It sounds as though (and correct me if I’m wrong) the distinction for you is that conceiving of all moving parts, such as the algorithms, architecture, object graph, iterations, etc, constitute design. The actual typing out of code then is implementation?

          Anyway, what I was reacting to was your observation that I didn’t mention design when I wrote the post. The reason I didn’t is the same reason I never mentioned the word “implementation,” either. I view both as such integral parts of creating software that mentioning them (to me) would have seemed redundant.

          • Kevin


            The reason I didn’t is the same reason I never mentioned the word “implementation,” either. I view both as such integral parts of creating software that mentioning them (to me) would have seemed redundant.

            Aren’t you falling into your own trap. You have made assumptions here that, clearly, are not universally true. If you want to generate defects that slip through the net, base your design/development on assumptions. Communications, ie. validation in this case, is a two way street and continuous.

      • Mark Saleski

        The mindset of not thinking that defects are inevitable is perfectly laudible, but if in the development lifecycle, sufficient effort isn’t put into both detailed requirements and how those requirements are translated into designs…the mindset down at the coding end is doomed to failure.

        You can write the cleanest, most elegant code in the world…but if it’s not a correct implementation of the original intent then defects will certainly follow.

        • I agree with this sentiment. I’d also extend it to say that one can build software with no unexpected runtime behaviors, exactly according to the intended requirements, and the thing could still be a flop (e.g. customers don’t buy it).

          I view all of those concerns as related, but separate issues. There’s the issue (at play when I wrote this post) of “does the software behave the way I, the developer, intended it to behave?” There’s the issue of “does the software do what those paying the bills want it to do?” And there’s the issue of “is the software solving a problem in such a way that someone makes or saves money?”

    • JohnAdams_1796

      “…I can’t really say what the author thinks I ought to do!”

      Erik certainly lists things you should do:

      (1) continuous integration

      (2) make deployments to test and production environments trivial

      (3) an automated regression and problem detection suite

      (4) smoke/performance tests or tests to see if resources are starved

      (5) reduce or eliminate error prone practices and parts of the code

      • JohnAdams_1796

        “The complete lack of any discussion on software design is a huge omission.”

        What items would you add to the above list to address design?

        Perhaps you would recommend a design review for each design decision. However, as you point out, “…design decisions are being made whenever one decides to make changes to software.” I’m sure we both would agree that the majority of such decisions are rather minor or trivial (suggestion: reflect on the last ten code change submissions you have made, and identify the design change beneath each of the code changes.)

        So how does one verify the “correctness” of each and every one of these decisions, all of which combine to constitute the design? Some sort of design “syntax checker”? Some sort of design review? (keeping in mind that this might be impractical because of the nature of software development, namely the tight loop of decisions and code changes.) Pair or mob programming?

  • I find most defects are caught and kept gone by Unit Tests. But most test writers forget Parameter Value Coverage. So even though they have 100% code coverage, they aren’t really 100% tested.

    Once you start taking Parameter Value Coverage into account in your unit tests, your methods will rarely fail. They will almost never behave in an unexpected way because you know exactly what to expect no matter the input.

    • That looks like C#. Have you ever used Pex (or whatever it’s called now) that generates automated tests with “interesting” inputs? Seems like a similar concept. (And a great idea, BTW)

  • My2Cents

    Sorry – I’ve read this three times really trying to find a way to agree. I can also refuse to get my teenage daughter an HPV vaccination assuming she and her future husband are both ‘pure’. As long as humans are a part of any process, you can expect… flaws. It’s who we are. Also how cancer happens, BTW, so it’s really deeper than human, isn’t it. Thank you for the thought exercise, though, Erik.

    • JohnAdams_1796

      Don’t be so hard on Erik. If you reduce the article to its essence, here’s what he’s saying:

      (1) Adopt the attitude that your goal is to have zero defects.

      (2) Overworking or threatening developers won’t get you there, however.

      (3) Superficial measures won’t get you there, either.

      (4) Adopt some measures that are known to reduce defects:

      (a) continuous integration

      (b) make deployments to test and production environments trivial

      (c) an automated regression and problem detection suite

      (d) smoke/performance tests or tests to see if resources are starved

      (e) reduce or eliminate error prone practices and parts of the code

      Do you disagree?

    • I don’t really understand the parallel to vaccination. I’m not saying that if you can’t have zero defects, then don’t bother writing software. I’m just saying that it’s possible to write software without defects. Sure it’s hard, so do the best you can.

    • Andy Bailey

      HPV infection doesn’t just happen through intercourse, there are other sources too and most of them do not involve 2 people.
      Why is that relevant? Well the parallel to Eric’s line of thought is that programming practices that accept defects as inevitable here is thinking that HPV infection is not under certain circumstances. Neither are right, nor safe.

  • cjacja

    The most common defect is that the software works fine but does the wrong thing. You can’t test this because the test will pass and not show as a defect.

    About the only way to find these is to build a firewall between the QA people and the developers. Actually this is what beta testing tries to do for free. Let users see it your code solves their problems.

    OK so far but I develop embedded software. It runs inside a controller inside a car or a camera or whatever. Why we have to do is turn over the car or camera and let others see it the device works. The trouble is that they are not so good at finding corner cases.

    What to do? The only thing that works is “al of the above”. Requirements written with the end user’s point of view, code standards, code review and walk throughs, design reviews, automated unit test, continuous integration, testing the target hardware, “red team” alpha testing and then some magic where we wait for the code to “mature”, like wine we don’t ship two day old code.

    There is not magic one solution. You need to apply “all of the above”. The problem is management wants it done cheap and “now” so puts time pressure on the process.

    • I definitely agree with “all of the above.” It’s certainly not a magic bullet situation.

      For “does the wrong thing” do you automate acceptance/behavior tests? I know some people trying to bring this sort of thing to the embedded space, actually, I think using simulation and other indirect kinds of techniques.

  • Tim Gray

    “have an automated regression and problem detection suite”

    Sure, just “have” one. You must be from the Steve Martin school of software engineering.

    ‘How to make a million dollars: First, get a million dollars…’ Steve Martin

    • That’s clever, I suppose, but it seems like it’d be more appropriate if the post had been titled, “how to build an automated test suite.”

      • Tim Gray

        I am not saying the “how” should be part of this article. I am saying that the complete lack of consideration for the cost and effort involved is short-sighted. How often does the cost of a production defect outweigh the costs of what it would have taken to eliminate it pre-production? Are there any case studies of a large software company using this philosophy to release defect-free software? What do you do when third party libraries are the source of your defect? Do you rewrite that library in-house from scratch? How often have companies for which you have consulted successfully implemented this strategy? What was the average scale of the projects involved? I just think this article is a bit detached from the real world.

        • Cost of what? The automated test suite? If you’ve been doing that from the get-go, the cost is just part of the cost of normal software operations. In general, I’m advocating that shops develop these competencies, but clearly there will be more bang for their buck implementing them on new efforts than retrofitting them on test-resistant codebases. For teams with a norm of writing legacy code in real time, switching to more rigorous practice obviously comes with cost. But if management cares about quality, it has to pay the piper sooner or later. (I’m just usually called in when management is in a panic, and it’s definitely “later” to the point of “perhaps too late”)

          I’ve definitely seen practice pivots go well in the wild, including at an extremely large organization with a huge user base. A team rebooted their operating mode (at great initial expense up front, since the project was heading for massive overruns anyway), adopted a norm of tests, CI, and routine production rolls, and would then go for several consecutive releases with 0 reported issues.

          If you doubt its feasibility, I don’t need to cite case studies. Just do a thought exercise. If you want to make a release without introducing defects, one option would be to, say, change a single line of code, test heavily, and roll to prod. Next release, try 2 lines. And, so on. While operating at this level of extreme wouldn’t be practical, it does illustrate the premise — slow down as much as you need until you get to the point where you’re not pushing dirty mop water around anymore. It’s the same premise that musicians follow when trying to perfect a piece. They slow down until they get every note right, and only speed up gradually, while making sure it stays flawless.

          Generally I see the same thing at shop after shop, which is the opinion that thousands of defects per release is “just the way it goes.” I wouldn’t suggest that you can get to a state where nothing ever goes wrong, but I will say that you can certainly get to a state where the team considers an escaped defect (in the sense of their software exhibiting behaviors they didn’t intend) to be unusual and disappointing.

          • Tim Gray

            It isn’t the number of lines of code you change that is the problem. It is the number of tests you have to manage and the time it takes to develop full regression UI tests. Even for something like this comment section a truly full test suite would include tests for entering control characters, unicode characters, pasted text from a Microsoft product with “smart quotes”, html tags and probably at least a dozen more.

  • Dilton_Dalton

    While the number of defects (odd behavior) found during different test phases can be useful, in general it is one of the least useful software metrics available to management. It is right up there with the amount of sky above an airplane.

    The article threw poop on the idea of unit testing although the argument against it is very weak. If you build a system out of perfect parts, you may not get a perfect system but at least there is some hope. If you build your system out of defective parts, you will never have any hope of ending up with a perfect system. So obviously a strong approach to making sure the parts are as perfect as possible is a good strategy for delivering a reliable system. A criteria for unit test case selection has been around since 1976. The paper on complexity by Thomas McCabe (http://www.literateprogramming.com/mccabe.pdf) states that for a program with a single entry point and a single exit, the number of independent paths through the program is equal to the number of decisions + 1. This is an easily calculated value and mccabe.com offers tools to help with the task.

    But as was mentioned, the number of defects found in a system be various verification methods is not very interesting. What is really interesting is how many defects remain.

    There are several methods of estimating the remaining defects based on test results if adequate data is collected (Number of unique defects, actual test effort to find them). If you plot defects vs test effort for verification of collections of components (inspections of requirements, inspections of designs, inspections of code modules) the results plot as a straight line. If we knew the effectiveness of each process, the total number defects originally present can be calculated. The effectiveness depends on the rapidity of verification. In simple terms, the more effort you spend inspecting or testing a component, the more defects you will find. But the more defects you find, the fewer remain and the rate of discovery declines. Historical data for a given team in a given problem domain is remarkably consistent and can be used to make estimates for new projects or for the remainder of current ones.

    For an integrated system, the defect vs test effort curve plots as an exponential function of the form D = N*(1-e^-at) where D is the number of defects found, N is the number of defects at the start of testing, a is an empirical constant and t is the time spent testing.

    The problem with this method is that if there are enough defects found during testing to make a reasonably accurate estimate, the product is so defect ridden that it will never be reliable. There is simply no economically feasible test program that will find enough defects to make the product acceptably reliable.

    There are methods to make estimates when there are only a few defects found but the problem is that to make them useful, the development team cannot be given the results. This proposal is generally viewed with great alarm and rejected out of hand.

    Using metrics to gather historical data and to make decisions based on that data is the only way that organizations can achieve reliable software. But in order for that to happen, the nature of defects has to be understood. This is part of what I teach in the course I provide on Quality Planning. This course shows teams how to estimate the defects that will be inserted in their work products, how to best plan for their removal and how to track the results to ensure the outcome is what was intended. Using this approach, managers can specify, at the outset of a project, how many defects will be delivered and accurately estimate the effort required at each phase to achieve that target for the least cost.

    • Andy Bailey

      I fail to see where poop was thrown at Unit Testing. Eric quite clearly state a “testing suite” is necessary, these would inevitably include Unit Tests as well as other types of testing.

      • Dilton_Dalton

        The article contains this rather negative bit of slung poop. “The first thing that everyone seems to seize on is mandating unit test
        coverage, since this, presumably, forces the developers to write
        automated tests, which, in turn, catch issues. The trouble here is that
        high coverage doesn’t actually mean that the tests are effective, nor
        does it cover all possible defect scenarios. Hiring or logging
        additional QA hours will be of limited efficacy for similar reasons.”