Learn Software Development » Code Coverage

What is meant by line coverage/ statement coverage/ code coverage? How is it calculated?

ashish — Mon, 05 Mar 2012 19:01:02 +0000

Line coverage, statement coverage, code coverage!! Though these three terms sound different, they are one and the same thing. So please don’t get confused between them. Throughout this article we will be discussing this, and we will be using the common term “code coverage” for all three. From the term itself we can make out that it is from the field of software testing and describes the level to which a particular piece of code has been tested, be it of some variable or function. It gives coverage of the extent of the testing of the source code of the software system or application. Code coverage involves the direct inspection of the code and therefore, it would not be wrong to say that it is a form of white box testing.
Code coverage is very much needed to implement a systematic software testing plan. Apart from testing, it is also given preference in the field of critical industries such as avionics and others, with regard to their security and safety. Code coverage like any other aspect of testing follows a certain criteria and those criteria are called coverage criteria. In some cases more than one criterion can be used. We have discussed about them below:
1. Basic coverage criteria: These are further sub divided in to many types:
(a) Statement coverage: It determines whether or not every node in the software program code has been executed or not.
(b) Condition coverage: This one is also called predicated coverage. This type of coverage determines whether or not the all the Boolean expressions in the whole program have been evaluated or not. It is not necessary that everywhere the condition coverage should imply the decision coverage. In Pascal language it is not necessary since there is no short circuiting of the Boolean expressions.
(c) Decision coverage: This coverage criterion is not to be confused with the branch coverage. It determines whether or not every single edge in the program has been executed or not. For example if- else, switch case etc.
(d) Condition and decision coverage: This type of coverage implements both the condition and decision coverage.
(e) Function coverage: This coverage determines whether or not every sub routine and function in the program has been called or not at least once.
There should be certain measures to counter check these coverage criteria. Fault injection method is one of such measures. It is required to ensure that adequate coverage has been provided to each branch and condition pertaining to the code of the exception handling program while carrying out the testing. There are some exceptions like avionics equipments where the modification of the condition and decision coverage is allowed to some extent. Such coverage is often called as “MC/ DC” or “modified condition/ decision coverage”. Avionics are somewhat safety critical and before testing them it is mandatory that the MC/ DC are met. This is done in order to ensure that they get 100 percent coverage. There arise some cases where you need to test all the possible combinations of the conditions in a decision. In such a case “multiple condition coverage” is used. There are various other types of coverage criteria but they are rarely used:
1. Loop coverage: Determines whether or not all the possible loops in a program code have been executed or not at least once.
2. Path coverage: Determines whether or not every path through a given code has been used at least once or not.
3. LCSAJ coverage: Checks for linear code sequence and jump.
4. Entry / exit coverage: Checks for possible calls and returns of the functions.
5. J- J path coverage: Checks for jump to jump paths.

Lessons Learned in Software Testing	Software Testing (2nd Edition)	Testing Computer Software, 2nd Edition

Some videos from Youtube regarding Code Coverage:

Advanced Software Testing: Code Coverage

Apex Code Testing & Coverage Best Practices

Code Coverage Metrics and How to Use Them Part 1

Code Coverage Metrics and How to Use Them Part 2

What is meant by branch coverage or decision coverage? How is it calculated?

ashish — Thu, 23 Feb 2012 20:17:50 +0000

Code Coverage is a very important aspect when it comes to testing, although a number of people may not even have heard about it. It should not be ignored, instead attempts should be made to incorporate it as part of processes. There are practical difficulties – it requires a great deal of effort to understand it and implement it properly, especially when you want your team to implement it. Here we have tried our best to keep the concepts as simple as possible so that you can atleast understand the basics of code coverage.
Code coverage is a measure to determine the extent to which you have tested a certain piece of code of your software program which is under testing, and the higher the coverage, the better. In this article we are going to discuss about one particular type of code coverage called “branch coverage” and how it is calculated. If we talk about Code coverage, it is used as a method to complement the unit testing, since the code coverage also tells us how much code remains to be tested and unit testing just tells us that about the performance of the code that has been tested. The target of a tester should be to achieve 100 percent code coverage (although it gets difficult to get to 100%).
In a software development cycle the code coverage is measured in the terms of statements and branches that are to be tested. One point you should always keep in mind is that achieving 100 percent code coverage does not means that your software system or application is free of bugs and error. It still might be having many critical bugs. Code coverage just gives the extent of how much code has been tested and does not eradicate the bugs and errors. Bugs and errors still persist. Code coverage might deceive you by giving you an illusion of false sense of security. You are wrong if you think that 100 percent code coverage achievement is enough to give you a solid and bug free product.
Branch coverage is a means to just enough give you an indication about the big visible errors and bugs and not about the hidden ones. It helps in discovering bugs and errors which lie unexplored and unexposed in the unexecuted part of the program code. Branch coverage has got another drawback; it is not effective in digging out bugs and errors that are hidden in the interactions between the decisions and their structures. For your information, path coverage is mostly preferred since it is very comprehensive and more robust as compared to the branch coverage technique. Being more robust it helps discover errors and bugs within a short span of time. It is a known fact that branch is the result of a decision.
So to put it simply we can say that the branch coverage gives the measure of the extent to which the outcomes of the decisions in a particular piece of executable code of a program have been tested. Though being less effective than the statement coverage, it is far much better than the simple statement coverage. Now, calculating the number of branches is pretty easy. In a program; for making decisions either Boolean decisions are used or switches are implemented. Boolean decisions, as we all know can have only 2 outcomes i.e., either true or false. The switches result in only one value for each particular case. It is not surprising that the number of branches equals the number of the outcomes of the decisions in a certain piece of code. Don’t forget to add the one branch for every straight line code. The branch coverage for discovering all the errors and bugs doesn’t works and carrying out exhaustive testing on each and every branch is not recommended.

Lessons Learned in Software Testing	Software Testing (2nd Edition)	Testing Computer Software, 2nd Edition

Code coverage tools

ashish — Sun, 22 Mar 2009 18:10:22 +0000

So you have decided that you need to ensure that your software development cycle has code coverage, and want to move ahead. In addition to making sure that team members have awareness about the principles and about the methods used, another important decision is about the tools to use. There are a wide variety of tools that are available, and here is a listing of the tools. It is hard to recommend a tool, since the actual tool depends on the targets, depends on the developed language (C++, Python, Java, Web language, etc). Here is a sampling of some of the popular tools.

- LDRA Testbed: Automatic instrumentation techniques
- Cobertura: Cobertura is a free Java tool that calculates the percentage of code accessed by tests. It can be used to identify which parts of your Java program are lacking test coverage. It is based on jcoverage.
- Quilt: Quilt is a Java software development tool that measures coverage , the extent to which unit testing exercises the software under test. It is optimized for use with the JUnit unit test package, the Ant Java build facility, and the Maven project management toolkit.
- Clover: Commercial tool. Clover is designed to measure code coverage in a way that fits seamlessly with your current development environment and practices, whatever they may be. Clover’s IDE Plugins provide developers with a way to quickly measure code coverage without having to leave the IDE. Clover’s Ant and Maven integrations allow coverage measurement to be performed in Automated Build and Continuous Integration systems and reports generated to be shared by the team.
- NUnit: Meant for unit testing. NUnit is a unit-testing framework for all .Net languages. It is written entirely in C# and has been completely redesigned to take advantage of many .NET language features, for example custom attributes and other reflection related capabilities. NUnit brings xUnit to all .NET languages.
- NCover: Commercial tool. NCover 3 includes Symbol and Branch Point Coverage, but it also has two new metrics: Cyclomatic Complexity and Method Visit Coverage. Cyclomatic complexity describes the number of independent paths through your methods. It’s a great indicator of when you should re-factor. Code with a low cyclomatic complexity will help you guarantee that new developers on your teams can get up to speed fairly quickly. While method visit coverage previously existed in NCover Explorer, it only worked when you had symbol points for a method.
- Jester: Jester finds code that is not covered by tests. Jester makes some change to your code, runs your tests, and if the tests pass Jester displays a message saying what it changed. Jester includes a script for generating web pages that show the changes made that did not cause the tests to fail.
- Emma: EMMA is an open-source toolkit for measuring and reporting Java code coverage. EMMA distinguishes itself from other tools by going after a unique feature combination: support for large-scale enterprise software development while keeping individual developer’s work fast and iterative.
- Koalog Code Coverage: Commercial license. Koalog Code Coverage is a code coverage computation application written in the Java programming language. Koalog Code Coverage allows you to measure the efficiency of your tests suite, but also to discover dead code in your project.
- EclEmma: EclEmma is a free Java code coverage tool for Eclipse, available under the Eclipse Public License. Internally it is based on the great EMMA Java code coverage tool, trying to adopt EMMA’s philosophy for the Eclipse workbench.
- GroboCodeCoverage: There are several commercially available code coverage tools for Java, but they all require a large fee to use. This is a 100% Pure Java implementation of a Code Coverage tool. It uses Jakarta’s BCEL platform to post-compile class files to add logging statements for tracking coverage. An old tool.
- Hansel: Hansel is an extension to JUnit that adds code coverage testing to the testing framework. What makes Hansel different from other code coverage tools? Most tools try to generate code coverage reports from a test-run of all available tests. But a much more useful information is how much of the code which a test is supposed to test is covered. Hansel gives you this information.
- CodeCover: CodeCover is a free glass-box testing tool developed in 2007 at the University of Stuttgart. CodeCover measures statement, branch, loop, and MC/DC coverage. CodeCover uses the template engine Velocity.
- rcov: code coverage for Ruby: 20-300 times faster than previous tools. Multiple analysis modes: standard, bogo-profile, “intentional testing”, dependency analysis. Detection of uncovered code introduced since the last run (“differential code coverage”)
- Insure++: a coverage of source code of application tested with functional tests.
- iSYSTEM winIDEA: measures coverage on a wide variety of embedded processors. It works by recoding execution directly on hardware, without instrumenting code or modifying the program and in real-time.
- LDRA Testbed: Measures statement coverage, branch/decision coverage, LCSAJ Coverage, procedure/function call coverage, branch condition coverage, branch condition combination coverage and modified condition decision coverage (MC/DC) for DO-178B Level A.
- VB Watch: Visual Basic code coverage and performance analysis tool
- BullseyeCoverage: C and C++ code coverage tool
- XDebug: PHP debugging tool, including code coverage

If you know of more tools, please add in the comments.

How important is 100% code coverage ?

ashish — Sat, 21 Mar 2009 16:41:37 +0000

It can be hard to achieve 100% code coverage, and there are a number of proponents who do not believe in going the extra bit it takes to achieve that. There are people who do not feel comfortable unless they reach a 100% code coverage, and are passionate about it. What are some of the benefits of trying to attain a 100% code coverage ? What is actually 100% code coverage ?
Let’s start with the meaning of what code coverage is (if you missed my previous post where I explained code coverage in more detail). Code coverage is a measurement that answers the question about how much of the source code for an application is run when the unit tests for the application are run, and more important, how much code is missed by the unit cases. If 100% of the lines of source code are run when the unit tests are run, then you have 100% code coverage. Seems fairly simple, right. Have you ever wondered why Service Level Agreements don’t seem to promise 100%, but do 99%, or even 99.9% ? This example was just a statement thrown in between to demonstrate how getting to 100% is not easy in any field. Let’s take this further.
When you write test cases, the principle behind writing the unit test cases and then executing them is to ensure that an application works as expected. So, if your test cases cover more of the code, then it means that the quality of a higher percentage of the code has been verified through testing. In practice, not everything goes by theory, so while 100% code coverage does not guarantee that an application has no bugs, your confidence in the quality of your code increases significantly the higher your code coverage is.
There are many people who feel that trying to target 95% code coverage is a futile target; at the same time, I believe that targeting 95% code coverage is not the way to go. 95% is a great interim target if you have not done code coverage before, but anything less than 100% is a path that you do not want to take. If you testing of the code is only 95%, you are still risking that 5% of the code is untested – this section of the code may be harmless, but may also have some big bugs that could screw up your happiness later. You allow exceptions to creep in, and then pretty soon, pressure causes you to make more exceptions, and an example has already been set. There will always be a niggling feeling that the untested portions of the code can come back to haunt you later.
Reaching there is not easy. It takes a lot of hard work to get everybody to target the 100% code coverage, and needs determination when the temptation is strong to leave things be in the 90% (even more so when sections of the code are older code that may not have seen many changes).

What is code coverage ?

ashish — Sat, 21 Mar 2009 16:12:26 +0000

People know what software testing is, and most people in the profession can differentiate between white box and black box testing. However, when you get into more details, and look to identify how testing can provide even greater value, the benefit of measures such as code coverage become apparent. At the same time, there would be a large number of software professionals who are not even aware of what code coverage is, and what are its key benefits. So the idea of this article is to try and articulate some of the benefits.
So what is Code Coverage ?
Well, Code coverage is a type of measure used in software testing which tries to answer the questions about the degree to which the entire source code of an application has been tested. Traditional black box testing, with its focus on functional testing cannot even come close to trying to answer this question, although White Box testing does come closer to trying to answer this question. In fact, if you examine code coverage practices in detail, it is possible to say that code coverage is a form of testing that inspects the code directly and is therefore a form of white box testing.
Code coverage techniques were amongst the first techniques invented for answering the question of systematic software testing. For those proponents of the text plan / case based method of testing, code coverage testing works on the principle that it is entirely possible that sections of a software application remain untouched by test data, and as a result, then it is not possible to say, with any degree of certainty, that these sections do not contain residual errors.
How do you go ahead with actually trying to do code coverage ?
Code Coverage requires support from engineering to proceed. Why is this so ? Getting code coverage in practise requires a different procedure from the normal software build – The target application / software is configured to be built with special options / libraries and/or run under a special environment such that every function that is exercised (executed) in the program(s) is mapped back to the function points in the source code. Doing this systematic process (although requires effort and the value does not seem immediately clear to individual developed and QE) allows developers and quality assurance personnel to look for parts of a system that are rarely or never accessed under normal conditions (error handling and the like) and helps reassure test engineers that the most important conditions (function points) have been tested. Once this exercise has been done, the output is further analysed to see what areas of code have not been exercised and the tests are updated to include these areas as necessary. Once this exercise has been completed (and it may need to be done on a regular basis as the code is in the process of being developed), it gives a much higher level of confidence about the overall quality of the code.