Learn Software Development » QE

What is code coverage ?

ashish — Sat, 21 Mar 2009 16:12:26 +0000

People know what software testing is, and most people in the profession can differentiate between white box and black box testing. However, when you get into more details, and look to identify how testing can provide even greater value, the benefit of measures such as code coverage become apparent. At the same time, there would be a large number of software professionals who are not even aware of what code coverage is, and what are its key benefits. So the idea of this article is to try and articulate some of the benefits.
So what is Code Coverage ?
Well, Code coverage is a type of measure used in software testing which tries to answer the questions about the degree to which the entire source code of an application has been tested. Traditional black box testing, with its focus on functional testing cannot even come close to trying to answer this question, although White Box testing does come closer to trying to answer this question. In fact, if you examine code coverage practices in detail, it is possible to say that code coverage is a form of testing that inspects the code directly and is therefore a form of white box testing.
Code coverage techniques were amongst the first techniques invented for answering the question of systematic software testing. For those proponents of the text plan / case based method of testing, code coverage testing works on the principle that it is entirely possible that sections of a software application remain untouched by test data, and as a result, then it is not possible to say, with any degree of certainty, that these sections do not contain residual errors.
How do you go ahead with actually trying to do code coverage ?
Code Coverage requires support from engineering to proceed. Why is this so ? Getting code coverage in practise requires a different procedure from the normal software build – The target application / software is configured to be built with special options / libraries and/or run under a special environment such that every function that is exercised (executed) in the program(s) is mapped back to the function points in the source code. Doing this systematic process (although requires effort and the value does not seem immediately clear to individual developed and QE) allows developers and quality assurance personnel to look for parts of a system that are rarely or never accessed under normal conditions (error handling and the like) and helps reassure test engineers that the most important conditions (function points) have been tested. Once this exercise has been done, the output is further analysed to see what areas of code have not been exercised and the tests are updated to include these areas as necessary. Once this exercise has been completed (and it may need to be done on a regular basis as the code is in the process of being developed), it gives a much higher level of confidence about the overall quality of the code.

Questions regarding software testing and functionality

ashish — Mon, 16 Mar 2009 04:22:50 +0000

A couple of questions related to software testing, especially with regard to functionality ..

What if the application has functionality that wasn’t in the requirements?
One might wonder how it is possible to have a situation where the application has functionality that was not in the requirements; but it is possible. In a badly controlled software project, it is possible that functionality may be included based on direct request from clients; another case is when the project has been partly done by another company or is a migration project. In such cases, the software may have many undocumented functionality. The implications of such functionality are related to the extra effort required for testing, for documentation, for bug fixing, for internationalization.
Given the impact, it is necessary to do the serious effort needed to determine if an application has significant unexpected or hidden functionality, and the very fact that it may be necessary to do this analysis indicates problems in the software development process. What do you do if such functionality is found? If the functionality isn’t necessary to the purpose of the application, a decision should be taken to determine whether it should be removed, as it may have unknown impacts or dependencies that were not taken into account by the designer or the customer.
If not removed, analysis needs to be done to determine added testing needs or regression testing needs (and such extra effort may be non-trivial, or may add to the overall risks). Management should be made aware of any significant added risks as a result of the unexpected functionality. If the functionality only effects areas such as minor improvements in the user interface, for example, it may not be a significant risk. However, in no case should such functionality be taken casually.

What if the software is so buggy it can’t really be tested at all?
One would not like to be in such a situation, but can happen easily enough. Suppose the software cycle has been under a lot of stress in the design and development phase, then code can be real buggy. Further, if the processes related to review and code standards are not implemented, then the software could be real buggy. What does QE do in this case, given that they still need to do their job and complete testing.
The best bet in this situation is for the testers to go through the process of reporting whatever bugs or blocking-type problems initially show up, with the focus being on critical bugs. Since this type of problem can severely affect schedules, and indicates deeper problems in the software development process (such as insufficient unit testing or insufficient integration testing, poor design, improper build or release procedures, etc.) managers should be notified, and provided with some documentation as evidence of the problem. If required, development could be tasked with only bug fixing and no new feature work until a situation is reached where the code is much more bug free; and schedules can also be extended.

What’s the role of documentation in QA?

ashish — Thu, 12 Feb 2009 03:48:10 +0000

Sometimes, for people who are new to the software line (and specifically new to the testing business), there are questions about how important documentation would be to regular Quality work. After all, the major work is about testing some software and getting the bugs resolved, how important could testing be ? Well, the role of documentation in enabling the success of QA activities is critical. Note that documentation can be in electronic form as well, no necessity that it should be only paper, and in fact, recent trends are more towards having electronic forms that can be placed in source control areas in their specific directory locations.
QA practices should be documented such that they are repeatable, and are not dependent on individual people. Software artifacts and processes such as requirement and design specifications, design documents such as architecture, HLD, LLD, business rules, inspection reports, configuration control design and operational documents, code changes, software test plans, test cases, bug reports and decision making on important bugs, user manuals, etc. should all be documented such that these can be referred later (and prove very useful in case the project personnel are changed or should be transitioned to other teams).
As a part of documentation, there needs to be a system for easily finding and obtaining documents and determining what documentation will have a particular piece of information. Change management for documentation should be used in all cases, else you will find later that it is hard to figure out why something changed and what were the reasons behind it.
One of the most common reasons for failures or overruns / delays in a complex software project is to have poorly documented requirements specifications. Requirement specifications are the details describing an application’s externally-perceived functionality and properties. The condition for requirements should be clear, complete, reasonably detailed, cohesive, attainable, and testable. A non-testable requirement would be, for example, ‘user-friendly’ (too subjective). A testable requirement would be something like ‘user needs to enter their date of birth while creating their profile’. Determining and organizing requirements details in a useful and efficient way can be a difficult effort; different methods are available depending on the particular project.
Care should be taken to involve ALL of a project’s significant ‘customers’ and important stakeholders in the requirements process. ‘Customers’ could be in-house personnel or out, and could include end-users, customer acceptance testers, customer contract officers, customer management, future software maintenance engineers, salespeople, etc. Anyone who could later derail the project if their expectations aren’t met should be included if possible. This also helps in ensuring that changes later are minimized (can never be eliminated).
Organizations vary considerably in their handling of requirements specifications. Ideally, the requirements are spelled out in a document with statements such as ‘The product shall…..’. ‘Design’ specifications should not be confused with ‘requirements’; design specifications should be traceable back to the requirements.
In some organizations requirements may end up in high level project plans, functional specification documents, in design documents, or in other documents at various levels of detail. No matter what they are called, some type of documentation with detailed requirements will be needed by testers in order to properly plan and execute tests. Without such documentation, there will be no clear-cut way to determine if a software application is performing correctly.
Test plans need to be documented properly with good change control, since a test plan forms the basis of determining the areas of testing, scope, responsibilities, etc. A test plan forms the first level of generation of confidence in the test strategy for the project.

Properties of a test / QA / QE Manager

ashish — Wed, 28 Jan 2009 18:49:39 +0000

Testing is a vital and critical part of the overall software development process, and is very important that the overall testing environment have the right mix of aggression and thoroughness. A large amount of this attitude comes from the person who leads the testing team. So, what makes a good QA or Test manager?
There are many attributes that a good test, or QA manager should have. Here are some of them:
• The test manager should be very familiar with the software development process. This is the only way that the rest of the testing team can develop the feel for when they should be doing what activity.
• The test manager has be able to ensure that the overall enthusiasm of the team remains high, and promote a positive atmosphere, despite what is a somewhat ‘negative’ process (e.g., looking for or preventing problems). People should be made to feel that they have an important role in ensuring that customers get a software that works well.
• The test manager should be able to promote teamwork to increase productivity. Teamwork between the members of the testing team is critical, given that each of them may handle a separate area, and may have several elements of intersection. In addition, each person can have a different field of specialization, and together they can cover a large area.
• The test manager should be able to promote cooperation between software, test, and QA engineers. This is not so easy sometimes, but is very critical. It is a close interaction between dev and QE that results in a deeper understanding of where software can go wrong.
• The test manager have the diplomatic skills needed to promote improvements in QA processes. Sometimes software and hardware can be expensive, and management may not really understand or appreciate the need for such, and it is in such cases that the test manager can better explain.
• The test manager must have the ability to withstand pressures and say ‘no’ to other managers when quality is insufficient or QA processes are not being adhered to. It is the test manager who is responsible for quality.
• The test manager must have people judgement skills for hiring and keeping skilled personnel
• The test manager must be able to communicate with technical and non-technical people, engineers, managers, and customers.