Finding Bug Clusters with Bug Analysis

Bug analysis finds bugs you didn’t know you had

by Robin Rowe

BEVERLY HILLS, CA (LinuxMovies.org) 2016/6/29 – With good planning, organization, metrics and bug analysis, you can predict how many bugs are in your product and where they will be found before testing even begins.

Finding Bugs Is Good

The purpose of testing is to find bugs. From a tester’s perspective, a test that finds bugs is a successful test. Testing is an activ­ity that is inherently at odds with the mandate of your development team, which is to produce soft­ware without bugs. The goal of testing is to prove that your soft­ware does indeed have bugs. That we find bugs makes bug analy­sis possible.

Tracking Bugs

With a ticketed bug-tracking system such Bugzilla, we can keep a list of bugs and hopefully, their solutions. We also want statistics on our codebase, such as KLOC (thousands of lines of code), and how many files. To use for bug analysis. With hindsight, we will know how many bugs there are per KLOC. And use that to predict the future.

The 80-20 Rule in Testing

Eighty percent of the bugs are in 20% of the code. For example, Microsoft noted that by fixing the top 20% of the most-reported bugs, 80% of the related errors and crashes in a given system would be eliminated.1

Bugs vs. Defects

A bug is a symptom, not an actual defect. Your team should fix defects, not bugs.

Bugs can have four relation­ships to defects:

  1. One-to-one: You fix the de­fect, you fix the bug. This will be tedious if you have many bugs.
  2. Many-to-one: You fix the de­fect, many bugs go away. This happens when you put common code in libraries, as you should.
  3. One-to-many: You fix the defect, but the bug still occurs. This may indicate an unstable component. Go back to your test suite and rework or replace the defective component.
  4. Many-to-many: You can’t fix anything without breaking something else. You have spa­ghetti code. Either start over or be very clever in how you refactor code.

When Is It Not a Bug?

Sometimes your black-box test­ers have a different opinion than your developers on what qualifies as a bug. When your product audi­ence is not programmers, you must side with us­ers. That your black-box testers find the program difficult to use means your clients will to. Take the broadest inter­pretation of what is a bug. Don’t count just code bugs, count design bugs, too.

Testing provides a broad feel for the quality of the code under test. But simply logging bugs is not enough. You must have a record of the actual types of de­fects and their location in the source code (class, function, and line number).

Bug Clusters

Bugs, and the code defects that cause them, do not happen randomly. Look for the deeper issue. Very often the same fundamental problem causes many different defects. Typical fundamental causes are a new programmer, overly clever design, old code, faulty design, sloppy coding, writing code around a bug instead of fixing the cause, and even compiler or hardware errors.

A bug cluster isn’t something delicious covered with chocolate. Groups of defects are caused by the same fundamental problem are a bug cluster. Bug clusters are not neces­sarily all together in the code.

They cluster on the problem, not simply the code. A single bug cluster may appear to be distrib­uted across a number of modules or products.

Without good testing and bug analy­sis, software reuse may mean you’re reusing bugs.

Although bug clusters often manifest as error-prone modules, they are not limited to that par­ticular structure. Bug clusters may be any pattern of errors. For example, consider the bad program­mer. It may be that every mod­ule that a bad programmer worked on has been corrupted to some degree, but only the ones he or she worked on a lot will be error-prone modules. Fixing just the error-prone modules will leave similar problems in other modules that are not known to be error prone.

If you know where the causes of bugs tend to appear in the code, you know where to put the emphasis in your testing and de­sign. You can prevent defects by finding and fixing the root prob­lem. Without a bug zapper, bug clusters may go unnoticed. No one will be looking at all the bugs and finding the reasons be­hind them.

Bug Patterns

  • The code location of the defect
  • Each designer of the program
  • Each programmer who origi­nally wrote the code
  • Each programmer who maintained the code
  • The date the defect was created
  • The type of design of the of­fending code
  • The root cause of the defect
  • Anything else you can think of that may be relevant

Of course none of this does you any good unless you have good bug and defect records.

A tricky question, what is the date that a defect was created? Is that bug that a design-level bug or an analysis-level bug? With testing-driven design, these dates are easier to pin down, since the test suites are created so early in the process.

With large U.S. defense software projects, an average of nine months go by between the insertion of a bug and its discovery.2 Preventing this kind of expensive delay is one reason to use testing-driven design.

Bug Resolution

Based on the feedback from your black-box testers, your program­mers will find and remove the cause of the bug, and your pro­grammer testers will improve their test suites to detect that bug more readily. It is important that you improve the test suites to catch these defects. Often the same bug will reappear later if an­other fix undoes the previous one.

Some of the subtlest bugs can occur when a maintenance programmer finds and fixes a long­standing bug that other program­mers have coded around. This can turn the workarounds into bugs. Bugs that are handled by a work­around should be kept on the bug list in a special workaround cate­gory until the fundamental prob­lem is resolved. Best of all is to fix things right the first time.

Some programmers are slop­pier than others. A clever tech­nique that Microsoft uses to control this is to make the most errant programmer responsible for the next build. Most pro­grammers would rather not be the build master; this type of “punishment” can be a powerful motivator.

What Bugs Should Be Fixed First, When?

Should you stack up all the bugs that you’ve found and devote, say, Friday afternoons to fixing them? No, bugs should be han­dled as soon as possible. Any de­lay increases the chance that the tester will have problems repro­ducing a bug or that the pro­grammer will not remember the logic of the code involved. Anticipate an extra build and test cycle over what would have been necessary had your re­sponse been timely.

It seems self-evident you should fix all high priority bugs first, but this often isn’t true. Certainly you want to concentrate on fix­ing the components with critical defects first, but each component should be fixed completely. It wastes effort to take apart a com­ponent to fix a critical de­fect and then wait until later to fix trivial bugs in the same code. The developer would need to study the code twice. It would mean testing it twice. Given the 80-20 rule, that trivial bug in the same component may not be trivial at all.

Critical bugs corrupt data, break hardware, and can even kill people. At the other end of the bug spectrum is an annoying quirk such as a little-used menu item that can’t be selected. Somewhere in the middle are memory leaks (which cause your software to become more unsta­ble the longer it runs) and re­source faults (which cause your system to crash as you run out of disk space). These mid-level bugs are painful, but you can live with them for a short time.

Your bug zapper should have the attitude that all bugs are worth fixing. Any bug a customer dis­covers in your program under­mines his or her faith in your company. If you found a discrep­ancy of 25 cents in your account­ing department, you would be concerned, even though the amount isn’t worth worrying about. You could lose faith that the accounting system was accu­rate and wonder if more serious

problems remain to be discov­ered.

Knowing when and how far to pursue a bug takes the same kind of judgment as with a 25-cent accounting error. If you can possibly do so, you should track it down and fix it. If your software is so unstable that you don’t dare fix minor bugs, then you must ask yourself how stable it really is. On the other hand, changes do imply renewed test­ing and attendant costs. Your automated test suites will help you control those costs. More importantly, they can prevent a minor fix from escalating into a major bug.

Beta Sites Are Not Testers

Despite considerable hope to the contrary, beta sites are not test­ers. Any responsible software company warns its customers not to depend on beta software. You can’t really blame them for believ­ing you. Many major applica­tions have been distributed in massive beta programs only to have major bugs uncovered after the product shipped. With beta sites, you have no control over the quality of testing. How can you ensure that they even used your beta or that they reported bugs accurately?

One way that you can get some control of the beta process is to send your test suites to your beta sites. This is an excellent way for a small company to achieve con­figuration testing on a wide vari­ety of machines. Typical test suites are very small.

For full-fledged betas, install user interface and code coverage logs. With these log files, you can get a measure of how much the software has actually been used. Your program should keep error logs, too. Don’t count on betas to do stress testing for you, even though they may uncover much for you. You should have your own tools to test low mem­ory and reduced disk space condi­tions.

Remember that your beta is the first impression your custom­ers will have of your new soft­ware. It is also their first impression of how you will re­spond to bugs. Respect the ef­fort your customers are putting into helping you with your prod­uct. There’s no point in having a beta program if you don’t re­spond to the bug reports in a pro­fessional manner. Assign every bug a tracking number and give that number to the person report­ing the bug. If you don’t assign a tracking number as a receipt for each bug reported, you have no way to verify that your bug-track­ing system works. You should go back to your beta sites and verify that all the bugs they re­ported have been addressed.

A Happy Ending

Software testing is vitally important. If you don’t have enough testers within your organization, hiring crowd-testing from a service like Mechanical Turk is an option.

Note: This article is excerpts from an earlier article by the author published in American Programmer in April 1994.

###

1 http://www.crn.com/news/security/18821726/microsofts-ceo-80-20-rule-applies-to-bugs-not-just-features.htm

2 See Kenneth G. Cooper and Thomas W. Mullen, “Swords and Plowshares: The Rework Cycles of Defense and Com-merical Software Development Projects,” American Programmer, Vol. 6, no. 5 (May 1993), p. 45

Leave a Reply

Your email address will not be published. Required fields are marked *