Bug analysis finds bugs you didn’t know you had
by Robin Rowe
BEVERLY HILLS, CA (LinuxMovies.org) 2016/6/29 – With good planning, organization, metrics and bug analysis, you can predict how many bugs are in your product and where they will be found before testing even begins.
Finding Bugs Is Good
The purpose of testing is to find bugs. From a tester’s perspective, a test that finds bugs is a successful test. Testing is an activity that is inherently at odds with the mandate of your development team, which is to produce software without bugs. The goal of testing is to prove that your software does indeed have bugs. That we find bugs makes bug analysis possible.
With a ticketed bug-tracking system such Bugzilla, we can keep a list of bugs and hopefully, their solutions. We also want statistics on our codebase, such as KLOC (thousands of lines of code), and how many files. To use for bug analysis. With hindsight, we will know how many bugs there are per KLOC. And use that to predict the future.
The 80-20 Rule in Testing
Eighty percent of the bugs are in 20% of the code. For example, Microsoft noted that by fixing the top 20% of the most-reported bugs, 80% of the related errors and crashes in a given system would be eliminated.1
Bugs vs. Defects
A bug is a symptom, not an actual defect. Your team should fix defects, not bugs.
Bugs can have four relationships to defects:
- One-to-one: You fix the defect, you fix the bug. This will be tedious if you have many bugs.
- Many-to-one: You fix the defect, many bugs go away. This happens when you put common code in libraries, as you should.
- One-to-many: You fix the defect, but the bug still occurs. This may indicate an unstable component. Go back to your test suite and rework or replace the defective component.
- Many-to-many: You can’t fix anything without breaking something else. You have spaghetti code. Either start over or be very clever in how you refactor code.
When Is It Not a Bug?
Sometimes your black-box testers have a different opinion than your developers on what qualifies as a bug. When your product audience is not programmers, you must side with users. That your black-box testers find the program difficult to use means your clients will to. Take the broadest interpretation of what is a bug. Don’t count just code bugs, count design bugs, too.
Testing provides a broad feel for the quality of the code under test. But simply logging bugs is not enough. You must have a record of the actual types of defects and their location in the source code (class, function, and line number).
Bugs, and the code defects that cause them, do not happen randomly. Look for the deeper issue. Very often the same fundamental problem causes many different defects. Typical fundamental causes are a new programmer, overly clever design, old code, faulty design, sloppy coding, writing code around a bug instead of fixing the cause, and even compiler or hardware errors.
A bug cluster isn’t something delicious covered with chocolate. Groups of defects are caused by the same fundamental problem are a bug cluster. Bug clusters are not necessarily all together in the code.
They cluster on the problem, not simply the code. A single bug cluster may appear to be distributed across a number of modules or products.
Without good testing and bug analysis, software reuse may mean you’re reusing bugs.
Although bug clusters often manifest as error-prone modules, they are not limited to that particular structure. Bug clusters may be any pattern of errors. For example, consider the bad programmer. It may be that every module that a bad programmer worked on has been corrupted to some degree, but only the ones he or she worked on a lot will be error-prone modules. Fixing just the error-prone modules will leave similar problems in other modules that are not known to be error prone.
If you know where the causes of bugs tend to appear in the code, you know where to put the emphasis in your testing and design. You can prevent defects by finding and fixing the root problem. Without a bug zapper, bug clusters may go unnoticed. No one will be looking at all the bugs and finding the reasons behind them.
- The code location of the defect
- Each designer of the program
- Each programmer who originally wrote the code
- Each programmer who maintained the code
- The date the defect was created
- The type of design of the offending code
- The root cause of the defect
- Anything else you can think of that may be relevant
Of course none of this does you any good unless you have good bug and defect records.
A tricky question, what is the date that a defect was created? Is that bug that a design-level bug or an analysis-level bug? With testing-driven design, these dates are easier to pin down, since the test suites are created so early in the process.
With large U.S. defense software projects, an average of nine months go by between the insertion of a bug and its discovery.2 Preventing this kind of expensive delay is one reason to use testing-driven design.
Based on the feedback from your black-box testers, your programmers will find and remove the cause of the bug, and your programmer testers will improve their test suites to detect that bug more readily. It is important that you improve the test suites to catch these defects. Often the same bug will reappear later if another fix undoes the previous one.
Some of the subtlest bugs can occur when a maintenance programmer finds and fixes a longstanding bug that other programmers have coded around. This can turn the workarounds into bugs. Bugs that are handled by a workaround should be kept on the bug list in a special workaround category until the fundamental problem is resolved. Best of all is to fix things right the first time.
Some programmers are sloppier than others. A clever technique that Microsoft uses to control this is to make the most errant programmer responsible for the next build. Most programmers would rather not be the build master; this type of “punishment” can be a powerful motivator.
What Bugs Should Be Fixed First, When?
Should you stack up all the bugs that you’ve found and devote, say, Friday afternoons to fixing them? No, bugs should be handled as soon as possible. Any delay increases the chance that the tester will have problems reproducing a bug or that the programmer will not remember the logic of the code involved. Anticipate an extra build and test cycle over what would have been necessary had your response been timely.
It seems self-evident you should fix all high priority bugs first, but this often isn’t true. Certainly you want to concentrate on fixing the components with critical defects first, but each component should be fixed completely. It wastes effort to take apart a component to fix a critical defect and then wait until later to fix trivial bugs in the same code. The developer would need to study the code twice. It would mean testing it twice. Given the 80-20 rule, that trivial bug in the same component may not be trivial at all.
Critical bugs corrupt data, break hardware, and can even kill people. At the other end of the bug spectrum is an annoying quirk such as a little-used menu item that can’t be selected. Somewhere in the middle are memory leaks (which cause your software to become more unstable the longer it runs) and resource faults (which cause your system to crash as you run out of disk space). These mid-level bugs are painful, but you can live with them for a short time.
Your bug zapper should have the attitude that all bugs are worth fixing. Any bug a customer discovers in your program undermines his or her faith in your company. If you found a discrepancy of 25 cents in your accounting department, you would be concerned, even though the amount isn’t worth worrying about. You could lose faith that the accounting system was accurate and wonder if more serious
problems remain to be discovered.
Knowing when and how far to pursue a bug takes the same kind of judgment as with a 25-cent accounting error. If you can possibly do so, you should track it down and fix it. If your software is so unstable that you don’t dare fix minor bugs, then you must ask yourself how stable it really is. On the other hand, changes do imply renewed testing and attendant costs. Your automated test suites will help you control those costs. More importantly, they can prevent a minor fix from escalating into a major bug.
Beta Sites Are Not Testers
Despite considerable hope to the contrary, beta sites are not testers. Any responsible software company warns its customers not to depend on beta software. You can’t really blame them for believing you. Many major applications have been distributed in massive beta programs only to have major bugs uncovered after the product shipped. With beta sites, you have no control over the quality of testing. How can you ensure that they even used your beta or that they reported bugs accurately?
One way that you can get some control of the beta process is to send your test suites to your beta sites. This is an excellent way for a small company to achieve configuration testing on a wide variety of machines. Typical test suites are very small.
For full-fledged betas, install user interface and code coverage logs. With these log files, you can get a measure of how much the software has actually been used. Your program should keep error logs, too. Don’t count on betas to do stress testing for you, even though they may uncover much for you. You should have your own tools to test low memory and reduced disk space conditions.
Remember that your beta is the first impression your customers will have of your new software. It is also their first impression of how you will respond to bugs. Respect the effort your customers are putting into helping you with your product. There’s no point in having a beta program if you don’t respond to the bug reports in a professional manner. Assign every bug a tracking number and give that number to the person reporting the bug. If you don’t assign a tracking number as a receipt for each bug reported, you have no way to verify that your bug-tracking system works. You should go back to your beta sites and verify that all the bugs they reported have been addressed.
A Happy Ending
Software testing is vitally important. If you don’t have enough testers within your organization, hiring crowd-testing from a service like Mechanical Turk is an option.
Note: This article is excerpts from an earlier article by the author published in American Programmer in April 1994.
2 See Kenneth G. Cooper and Thomas W. Mullen, “Swords and Plowshares: The Rework Cycles of Defense and Com-merical Software Development Projects,” American Programmer, Vol. 6, no. 5 (May 1993), p. 45