Learn Software Development » Code Coverage

Code coverage tools

ashish — Sun, 22 Mar 2009 18:10:22 +0000

So you have decided that you need to ensure that your software development cycle has code coverage, and want to move ahead. In addition to making sure that team members have awareness about the principles and about the methods used, another important decision is about the tools to use. There are a wide variety of tools that are available, and here is a listing of the tools. It is hard to recommend a tool, since the actual tool depends on the targets, depends on the developed language (C++, Python, Java, Web language, etc). Here is a sampling of some of the popular tools.

- LDRA Testbed: Automatic instrumentation techniques
- Cobertura: Cobertura is a free Java tool that calculates the percentage of code accessed by tests. It can be used to identify which parts of your Java program are lacking test coverage. It is based on jcoverage.
- Quilt: Quilt is a Java software development tool that measures coverage , the extent to which unit testing exercises the software under test. It is optimized for use with the JUnit unit test package, the Ant Java build facility, and the Maven project management toolkit.
- Clover: Commercial tool. Clover is designed to measure code coverage in a way that fits seamlessly with your current development environment and practices, whatever they may be. Clover’s IDE Plugins provide developers with a way to quickly measure code coverage without having to leave the IDE. Clover’s Ant and Maven integrations allow coverage measurement to be performed in Automated Build and Continuous Integration systems and reports generated to be shared by the team.
- NUnit: Meant for unit testing. NUnit is a unit-testing framework for all .Net languages. It is written entirely in C# and has been completely redesigned to take advantage of many .NET language features, for example custom attributes and other reflection related capabilities. NUnit brings xUnit to all .NET languages.
- NCover: Commercial tool. NCover 3 includes Symbol and Branch Point Coverage, but it also has two new metrics: Cyclomatic Complexity and Method Visit Coverage. Cyclomatic complexity describes the number of independent paths through your methods. It’s a great indicator of when you should re-factor. Code with a low cyclomatic complexity will help you guarantee that new developers on your teams can get up to speed fairly quickly. While method visit coverage previously existed in NCover Explorer, it only worked when you had symbol points for a method.
- Jester: Jester finds code that is not covered by tests. Jester makes some change to your code, runs your tests, and if the tests pass Jester displays a message saying what it changed. Jester includes a script for generating web pages that show the changes made that did not cause the tests to fail.
- Emma: EMMA is an open-source toolkit for measuring and reporting Java code coverage. EMMA distinguishes itself from other tools by going after a unique feature combination: support for large-scale enterprise software development while keeping individual developer’s work fast and iterative.
- Koalog Code Coverage: Commercial license. Koalog Code Coverage is a code coverage computation application written in the Java programming language. Koalog Code Coverage allows you to measure the efficiency of your tests suite, but also to discover dead code in your project.
- EclEmma: EclEmma is a free Java code coverage tool for Eclipse, available under the Eclipse Public License. Internally it is based on the great EMMA Java code coverage tool, trying to adopt EMMA’s philosophy for the Eclipse workbench.
- GroboCodeCoverage: There are several commercially available code coverage tools for Java, but they all require a large fee to use. This is a 100% Pure Java implementation of a Code Coverage tool. It uses Jakarta’s BCEL platform to post-compile class files to add logging statements for tracking coverage. An old tool.
- Hansel: Hansel is an extension to JUnit that adds code coverage testing to the testing framework. What makes Hansel different from other code coverage tools? Most tools try to generate code coverage reports from a test-run of all available tests. But a much more useful information is how much of the code which a test is supposed to test is covered. Hansel gives you this information.
- CodeCover: CodeCover is a free glass-box testing tool developed in 2007 at the University of Stuttgart. CodeCover measures statement, branch, loop, and MC/DC coverage. CodeCover uses the template engine Velocity.
- rcov: code coverage for Ruby: 20-300 times faster than previous tools. Multiple analysis modes: standard, bogo-profile, “intentional testing”, dependency analysis. Detection of uncovered code introduced since the last run (“differential code coverage”)
- Insure++: a coverage of source code of application tested with functional tests.
- iSYSTEM winIDEA: measures coverage on a wide variety of embedded processors. It works by recoding execution directly on hardware, without instrumenting code or modifying the program and in real-time.
- LDRA Testbed: Measures statement coverage, branch/decision coverage, LCSAJ Coverage, procedure/function call coverage, branch condition coverage, branch condition combination coverage and modified condition decision coverage (MC/DC) for DO-178B Level A.
- VB Watch: Visual Basic code coverage and performance analysis tool
- BullseyeCoverage: C and C++ code coverage tool
- XDebug: PHP debugging tool, including code coverage

If you know of more tools, please add in the comments.

How important is 100% code coverage ?

ashish — Sat, 21 Mar 2009 16:41:37 +0000

It can be hard to achieve 100% code coverage, and there are a number of proponents who do not believe in going the extra bit it takes to achieve that. There are people who do not feel comfortable unless they reach a 100% code coverage, and are passionate about it. What are some of the benefits of trying to attain a 100% code coverage ? What is actually 100% code coverage ?
Let’s start with the meaning of what code coverage is (if you missed my previous post where I explained code coverage in more detail). Code coverage is a measurement that answers the question about how much of the source code for an application is run when the unit tests for the application are run, and more important, how much code is missed by the unit cases. If 100% of the lines of source code are run when the unit tests are run, then you have 100% code coverage. Seems fairly simple, right. Have you ever wondered why Service Level Agreements don’t seem to promise 100%, but do 99%, or even 99.9% ? This example was just a statement thrown in between to demonstrate how getting to 100% is not easy in any field. Let’s take this further.
When you write test cases, the principle behind writing the unit test cases and then executing them is to ensure that an application works as expected. So, if your test cases cover more of the code, then it means that the quality of a higher percentage of the code has been verified through testing. In practice, not everything goes by theory, so while 100% code coverage does not guarantee that an application has no bugs, your confidence in the quality of your code increases significantly the higher your code coverage is.
There are many people who feel that trying to target 95% code coverage is a futile target; at the same time, I believe that targeting 95% code coverage is not the way to go. 95% is a great interim target if you have not done code coverage before, but anything less than 100% is a path that you do not want to take. If you testing of the code is only 95%, you are still risking that 5% of the code is untested – this section of the code may be harmless, but may also have some big bugs that could screw up your happiness later. You allow exceptions to creep in, and then pretty soon, pressure causes you to make more exceptions, and an example has already been set. There will always be a niggling feeling that the untested portions of the code can come back to haunt you later.
Reaching there is not easy. It takes a lot of hard work to get everybody to target the 100% code coverage, and needs determination when the temptation is strong to leave things be in the 90% (even more so when sections of the code are older code that may not have seen many changes).

What is code coverage ?

ashish — Sat, 21 Mar 2009 16:12:26 +0000

People know what software testing is, and most people in the profession can differentiate between white box and black box testing. However, when you get into more details, and look to identify how testing can provide even greater value, the benefit of measures such as code coverage become apparent. At the same time, there would be a large number of software professionals who are not even aware of what code coverage is, and what are its key benefits. So the idea of this article is to try and articulate some of the benefits.
So what is Code Coverage ?
Well, Code coverage is a type of measure used in software testing which tries to answer the questions about the degree to which the entire source code of an application has been tested. Traditional black box testing, with its focus on functional testing cannot even come close to trying to answer this question, although White Box testing does come closer to trying to answer this question. In fact, if you examine code coverage practices in detail, it is possible to say that code coverage is a form of testing that inspects the code directly and is therefore a form of white box testing.
Code coverage techniques were amongst the first techniques invented for answering the question of systematic software testing. For those proponents of the text plan / case based method of testing, code coverage testing works on the principle that it is entirely possible that sections of a software application remain untouched by test data, and as a result, then it is not possible to say, with any degree of certainty, that these sections do not contain residual errors.
How do you go ahead with actually trying to do code coverage ?
Code Coverage requires support from engineering to proceed. Why is this so ? Getting code coverage in practise requires a different procedure from the normal software build – The target application / software is configured to be built with special options / libraries and/or run under a special environment such that every function that is exercised (executed) in the program(s) is mapped back to the function points in the source code. Doing this systematic process (although requires effort and the value does not seem immediately clear to individual developed and QE) allows developers and quality assurance personnel to look for parts of a system that are rarely or never accessed under normal conditions (error handling and the like) and helps reassure test engineers that the most important conditions (function points) have been tested. Once this exercise has been done, the output is further analysed to see what areas of code have not been exercised and the tests are updated to include these areas as necessary. Once this exercise has been completed (and it may need to be done on a regular basis as the code is in the process of being developed), it gives a much higher level of confidence about the overall quality of the code.