1. Deciding the target audience for your usability testing: Given that the outcome of the usability testing will help in determining changes to your design strategy, it is absolutely essential that the usability testing be done among people who are good representatives of the target subject; this means that the selection should be done carefully. Avoid the temptation to cut corners, or to select neighbors who seem to represent the target segment. Devise a set of queries that will help determine if the selected people actually are good representatives; similarly, use experts to help you get good target subjects. And, other than a few cases, don’t let them know the name of the company - it may cause bias to appear in their results.

1. a) Different users may specialize is separate workflows, for example, if you want to get testing done of a new shopping site catering to working professionals, you should design an appropriate query form. Review the answers so that you can get an idea of which area a user would move more towards; this may help in deciding the exact set of users.

2. Before actually starting the testing: Remember, your tests will only be good when the people taking part are comfortable with the whole process. They need to feel that the environment in which the test is taking place is similar to that of their home or work environment. And of course, stay easy on the legalese - you may want to make sure that your in-process development is safe and want them to sign NDA’s, but if you may it full of complicated legal terms, they are likely to get confused. If your office is not exactly very accessible, consider doing the testing in a more convenient location.

3. Starting the usability testing: Don’t plunge the users directly into the testing process. Get talking with them, explain what the website is about and what the URL is; get some initial feedbacks about what they expect from such a site. If they mention some phrases or some other such term, then it is good to understand these statements or terms. Getting some words of polite conversation in makes them more comfortable.

4. Decide what you want to get tested: When deciding to select tasks for review by users, it is absolutely essential that you dump the notion about favorite sections of the site. You may have added a great new section that was very difficult to develop, but if it is not critical to the success of the site, then it is not a high priority to get an evaluation done. You should select tasks that are critical for the success of the site.

5. Scenarios are always better for such evaluations: Again, you need to talk in the language of the customer. You may have had great internal debate on the naming of features, but if you asking customers to evaluate some flows, then you should:
- Ask them to try out workflows / use cases (for example, you need to find some white shoes for your kid, and pay for them through the card that you normally use)
- Use simple language (avoid more technical words such as payment processing, and instead use phrases such as payment options)
- Set tasks that have a logical conclusion

6. The actual task execution: It is almost a core logic of usability testing that people should not be tasked very heavily during the process of usability testing. Get them to do one task at a time, and focus on their responses during the execution of the task. In a lot of cases, users may have to be given inputs during the course of the task (say, you want them to test out the convenience of posting videos to YouTube); in such cases, make sure that they have the equipment needed for the task (in this case, they either have a sample of home videos, or they have a camera readily available)

7. The participant is not at fault: Sometimes, you get users struggling during the usability test, or they get stuck at places where you would think that things are very simple. Remember, all their feedback is important, and if they are not able to do some tasks or part of tasks, then it is most likely a reflection of the task rather than any inherent inabilities on the part of the contestants. Further, if they are confused by something, or they have to make a choice, don’t guide them down some path (you would most likely introduce a bias if doing so). If you are asked a direct question, then reply, but not venture an opinion.

8. Don’t get distractions into place: Once the user has started, minimise distractions. Prevent people from coming or going in the location, and this includes a lot of traffic in front of the room where the test is happening. If people need to see, then they should see this on a video conference or some other facility. Test subjects can get conscious if there are too many there.

9. The user has completed their testing: If the user is done, then you need to gather as much information as possible. Ask the user about their impressions, about what worked well and what did not. Ask about what they feel could be done to improve things, and whether what they saw was how they would have done things. This could involve also asking them about what they recall about the software or the site - this helps in highlighting the parts that remain in the minds of the tester.

10. Go through the recording of the tester interaction; this helps in determining where the tester was able to move fast, where there was hesitation, and most importantly, where the tester expected to find something and did not.

11. If you are still not clear after going through with a number of testers, use more ! Your product success depends on getting the flow right, and if this means that you need more usability testing done, then so be it.

1. Usability Test Data Logger tool v5.0 (link to site)
Some features:
# Cross-platform: Datalogger is a PC- or Macintosh-compatible Microsoft Excel file (requires Microsoft Excel to run).
# Customisable: You can enter participant details, task names, task order, pre- and post-test interview questions and include your own satisfaction questionnaire.
# Captures quantitative data: The spreadsheet includes preset task completion scores and includes a built-in timer to record time-on-task.
# Captures qualitative data: Allows data entry of qualitative observations for each participant and each task.
# Provides real-time data analysis: Automatically generates charts illustrating task completion, time-on-task and user satisfaction with the product.

2. Morae Usability Testing for Software and Web Sites (link to site)
From the website:
Morae gives you the tools to:
* Instantly calculate and graph standard usability measurements, so you can focus on understanding results
* Visualize important results in ways that make them more understandable and meaningful
* Present results persuasively and professionally
Morae bundle can be bought for $1495 (link)

3. A website that explains how to use Macromedia Director as a Usability testing tool (link to article)
From the website:
While Director will not eliminate standard development environments or programming languages, it will enhance the prototyping and usability testing experience by allowing developers to gather feedback from prospective clients and users early in product development. Early prototyping will allow developers to identify and fix defects early in development.

4. QUIS: The Questionnaire for User Interaction Satisfaction (link to site)
From the website:
The purpose of the questionnaire is to:
1. guide in the design or redesign of systems,
2. give managers a tool for assessing potential areas of system improvement,
3. provide researchers with a validated instrument for conducting comparative evaluations, and
4. serve as a test instrument in usability labs. Validation studies continue to be run. It was recently shown that mean ratings are virtually the same for paper versus computer versions of the QUIS, but the computer version elicits more and longer open-ended comments.

5. Rational Policy Tester Accessibility Edition (link to site)
From the website:
The Accessibility Edition helps ensure website user accessibility by monitoring for over 170 accessibility checks. It helps determine the site’s level of compliance with government standards and displays results in user-friendly dashboards and reports.
* Improves visitor experience by exposing usability issues that may drive visitors away
* Facilitates compliance with federally-regulated guidelines and accessibility best practices
* Enlarges your market opportunity: over 10 percent of the online population has a disability (750 million people worldwide, 55 million Americans)
* Operating systems supported: Windows

6. Serco service (link to site)
They have a service that covers the following stages:
Planning and Strategy
User needs
Defining concepts
Usability evaluation

7. Web accessibility toolbar (link to site)
From website:
The Web Accessibility Toolbar has been developed to aid manual examination of web pages for a variety of aspects of accessibility. It consists of a range of functions that:
* identify components of a web page
* facilitate the use of 3rd party online applications
* simulate user experiences
* provide links to references and additional resources

8. WAVE 4.0 Beta (link to site)
From website:
WAVE is a free web accessibility evaluation tool provided by WebAIM. It is used to aid humans in the web accessibility evaluation process. Rather than providing a complex technical report, WAVE shows the original web page with embedded icons and indicators that reveal the accessibility information within your page.

9. Readability Test (link to site)
The website provides a service that helps in determining how readable a site is.
From the website:
Gunning Fog, Flesch Reading Ease, and Flesch-Kincaid are reading level algorithms that can be helpful in determining how readable your content is. Reading level algorithms only provide a rough guide, as they tend to reward short sentences made up of short words. Whilst they’re rough guides, they can give a useful indication as to whether you’ve pitched your content at the right level for your intended audience.

If you have feedback on the above, or other tools that have been useful for you, please comment.

Cognitive Walkthrough: Performed continually during the development cycle, cognitive workflows try to evaluate the system from a user’s thought process that help in decision making (these include the ability to reason as well as other factors such as memory load). This will help in terms of understanding how the system will seem like to a infrequent user. The input for such users could be in the form of paper workflows or through a working prototype. More details can be found at this link.

Focus Groups: Focus groups are actually a method used by marketing companies whereby a group of people from the target segment are brought together in a discussion type of format. A person from the product team acts as the moderator, and prepares a list of issues to be discussed. More details from this site (link).

GOMS: Goals, Operators, Methods, and Selection Rules; a slightly difficult mechanism to evaluate and model human task performance.

Prototyping: Prototyping is a very famous method for displaying a system for internal users and for external usability testers. Prototypes can range from paper models to actual almost working types of prototypes.

Walk-Around Review: This type of study is meant to harness the observations of other people in the company who are not part of the engineering team. The idea is to place paper mockups on the walls of the company along with space for comments so that people can write their comments; in addition, early builds can be provided to company people for them to use and provide comments on. Such a method can provide a lot of useful tips (and have an incidental benefit of providing greater visibility to your project).

Field Observation: Representatives of the product team go to the actual user workplaces and observe how they use the software in their daily lives (example, if a digital imaging application for professional photographers is being developed, then a good way is to have multiple days where people tag along with photographers and observe their usage of the application).

Task Analysis: # An analyst determines the user goals and tasks, then makes recommendations aimed at increasing efficiency and user-friendliness. The objective is to determine how the user actually feels comfortable in using software / sites.

Interviews/Observations: These are one-to-one discussions. Do a thorough discussion with them about what they do, and observe their usage at the same time. This will help in doing a realistic determination about their pattern of usage.

Usability Inspection: reviews a system based on a set of usability guidelines. Experts familiar with issues of usability in design perform the usability inspection.

User Testing: Users (sampled from the final target customer segment) use the final software / site. These can be final users from the client side (if the software is developed for a specific client), or can be a sample set of users. Usability experts study their interactions and figure out what are the paint points and improvements needed.

Ex: When something is typed on a word document by pressing a key, same is displayed on the monitor. The process of conversion of user input of the key to machine language and then displaying what you have typed on monitor is performed by batch system.

EVENT CONTROL SYSTEMS : These systems process real time data to provide the user with results for the command that has been given.

Ex: When you press Ctrl + S, it tells the computer to save the document. This communication is provided by the event controls that are pre-defined in the system.

PROCESS CONTROL SYSTEMS : When two systems communicate, the co-ordination or data transfer becomes vital. Process control systems are the one’s that receive data from a system and instruct the sending system to perform specific tasks based on the reply sent by the receiving system.

PROCEDURE CONTROL SYSTEMS : These systems are one’s which control the functions of another system.

MESSAGE PROCESSING SYSTEMS : A simple example to explain these kind of systems is the SMS management software used by the mobile operator which handles the incoming and outgoing messages.

DIAGNOSTIC SOFTWARE SYSTEMS : These systems help in diagnosing the computer hardware components.
Ex: The “NEW HARDWARE FOUND” dialogue seen when you plug in a new device to the computer is an example of these kind of systems.

SENSOR & SIGNAL PROCESSING SYSTEMS : In these systems the computer receives input in form of signals and then transforms the signals to a user understandable output.

Consider an example:

There are four customers: A, B, C and D, which are analogous to four processes.
The credit unit is like the resource
The banker himself is the OS
Assume each credit unit = Rs. 1000.

Not all customers need their maximum credit immediately. Hence only 10 credit units are reserved.
Process Current Max. Free = 10
A 0 6 B 0 5 C 0 4 D 0 7

How does the Algorithm Work?

When a new process (customer) enters the system, it (he) must declare the maximum number of instances of each resource type (credit units) that it (he) may need. This number may not exceed the total number of resources (credit units) in the system. When a user (customer) requests a set of resources (credit unit), the system must determine whether the allocation of these resources will leave the system in a safe state. If it will,
the resources are allocated; otherwise, the process must wait until some other process releases enough resources. Consider current allocation to various processes is as shown below.
Process Current Max. Free = 2
A 1 6 B 1 5 C 2 4 D 4 7
Would the System be in a Safe State?
‘C’ requests 2 additional units and gets them. It then runs to completion and frees all the resources it has.
Process Current Max. Free = 4
A 1 6 B 1 5 C 0 - D 4 7
Now either ‘B’ or ‘D’ can request and run to completion. Assume ‘B’ requests 4 additional units and gets them. It then runs to completion and frees all its resources. Process Current Max. Free = 5
A 1 6 B 0 - C 0 - D 4 7
Now ‘D’ runs and requests 3 additional resources and gets them. It then runs to completion and releases all its resources.Process Current Max. Free = 9
A 1 6 B 0 - C 0 - D 0 -
Finally ‘A’ runs and requests 5 additional resources and gets them. It then runs to completion and releases all its resources. Process Current Max. Free = 10
A 0 - B 0 - C 0 - D 0 -
Here is the complete banker’s algorithm:

Several data structures must be maintained to implement banker’s algorithm. Let n be the number of processes and m be the number of resource types. The data structures needed are:
• Available : A vector of length m indicates number of available resources of each type. If Available[j] = k, there are k instances of resource type Rj available.
• Max : A n*m matrix defines the maximum demand of each process. If Max[i,j]=k, then Pi may request at most k instances of resource type Rj.
• Allocation : An n*m matrix defines the number of resources of each type currently allocated to each process. If Allocation[i,j]=k, then process Pi is currently allocated k instances of resource type rj.
• Need : An n*m matrix indicates the remaining resource need of each process. If Need[i,j]=k, then Pi may need k more instances of resource type Rj to complete its task. Need[i,j]=Max[I,j] – Allocation[I,j].

After defining the data structures, algorithm moves into two phases :
• Safety Algorithm
• Resource Request Algorithm

SAFETY ALGORITHM

The safety algorithm is for finding out whether or not a system is in a safe state. It is described below:

1. Let work and finish be vectors of length m and n
respectively. Initialize work = Available and Finish[I] = false for all I = 1, 2, …, n.
2. Find an I such that both
• Finish[I] = false
• Needi <= work
If no such I exists, go to step 4.
3. Work = work + allocationi
finish[I] = true
go to step 2

4. If finish[I] = true for all I, then the system is in a safe state. This algorithm may require an order of m * n2 operations to decide whether a state is safe.

RESOURCE – REQUEST ALGORITHM

Having determined that the system is safe, this algorithm grants the requested resources to the process. Let Request i be the request vector for process Pi. If Request[j] = k, then process Pi wants k instances of resource type Rj. When this request is made, the following actions are taken:

1. If request I <= need I, then go to step 2. Otherwise raise an error condition because the process has exceeded its maximum claim.
2. If request I <= available, go to step 3. Otherwise, Pi must wait since the resources are not available.
3. Have the system pretend to have allocated the requested resources to process Pi by modifying the state as follows:
a. Available = available – request I
b. Allocation = allocation + request I
c. Need I = Need I – request I
4. Call the Safety algorithm. If the state is safe, then transaction is completed and process Pi is allocated the resources. If the new state is unsafe, then Pi must wait and the old resource allocation state is restored.

In order to prevent deadlock, the system is built in such a way that no deadlock occurs. Make sure that at least one of the four conditions in which deadlock can occur does not exist.

Attacking Hold and wait : In order to ensure that hold and wait condition never occurs, two protocols can be used:
• Make all the requests in the beginning. All or nothing policy.
• A process should request a resource only when it has none.

Attacking Pre-emption : To avoid this situation, the protocol that can be used is if a process that is holding some resources requests another resource that cannot be allocated immediately, then all the resources currently held are released. These released resources gets added to the list of resources for which process is waiting.

Or, if process requests a resource, check their availability, if available allocate them. If not, check if they are allocated to other process that is waiting for additional resources. If so, preempt the desired resources from waiting process and allocate to requesting process. If not available, requesting process should wait.

Attacking Mutual Exclusion : This condition holds for nonsharable resources e.g printer. Sharable resources do not require a mutually exclusive access, and thus cannot be involved in a deadlock.

Attacking Circular Wait Condition : To prevent this condition, use the following protocol:
• Order all the resources.
• Make sure that the requests are issued in such a way so that there are no cycles present in the graph.
• Resources can be requested only in an increasing order i.e. we cannot request a resource whose number is less than any you may be holding.

Deadlock Avoidance

It turns out that there are algorithms you can use to avoid deadlock. The principle is as follows: Whenever you are about to make an allocation, you can run this algorithm and see if making that allocation would lead to a deadlock.This seems like
the ideal method but has some important drawbacks.Let us see what these drawbacks are:
1. The algorithms are not that fast and there is a lot of overhead in running them before every resource allocation. The OS might be allocating resources hundreds of times a second.
2. The algorithms assume that processes know their maximum resource needs a priority. But this is often not the case.
3. They (processes) assume that they know what resources are available in the system. Hardware can go down and resources can become unavailable. This could lead to unpredictable deadlock situations.
In short, in deadlock avoidance method, the OS must be given in advance additional information concerning which resources a process will request and use during its lifetime.

Here are some salient points of the algorithm working nature:
• The deadlock avoidance algorithm dynamically examines the resource allocation state to ensure that there can never be a circular-wait condition.
• A state is safe if the system can allocate resources to each process (up to its maximum) in some order and still avoid deadlock.
• A system is in a safe state only if there exists a safe sequence.
• A safe state is not a deadlocked state but a deadlocked state is an unsafe state.
• An unsafe state may lead to a deadlock.

Safe State

A state is safe if the system can allocate resources to each process in some order and avoid deadlock. A safe sequence is a sequence of processes
for the current allocation state if for each Pi, the resources that Pi can still request can be satisfied by the currently available resources plus the resources held by all Pj with j <= i . In this situation, if the resources that process Pi needs are not immediately available, then Pi can wait until all Pj have finished. When they have finished, Pi can obtain all its needed resources, complete its designated task, return its allocated resources, and terminate. When Pi terminates, Pi+1 can obtain its needed resources and so on. If no such sequence exists, then the system state is said to be unsafe.

Resource Allocation Graph Algorithm

Deadlocks can be described more precisely in terms of directed graphs called a system resource allocation graph. A directed edge Pi -> Rj is called request edge which signifies that process Pi requested an instance of resource Rj. A directed edge Rj->Pi is called assignment edge which signifies that resource Rj is allocated to process Pi. A directed edge Pi–>Rj is called a claim edge which indicates that process Pi can request resource Rj at some time in future. When process Pi requests resource Rj the claim edge becomes a request edge. Similarly, when a resource Rj is released by Pi, the assignment edge is converted to a claim edge.

Suppose process P1 requests resource Rj. Request is granted only if this request edge is converted to an assignment edge, hence not resulting in the formation of a cycle in resource allocation graph. Hence the process has to wait for the resource.

Under the normal mode of operation, a process may utilize a resource in the following sequence:

Request – when a process needs some resource, it requests for it. Process has to wait if resource is busy.
Use – The process should operate on the resource.
Release – After using, process should release the resource.

Conditions In Which Deadlock Can Arise

• Hold and Wait : As the name suggests, a process is holding a resource and also waiting for some other resources that are held by other process.
• No Preemption : A resource can be released only voluntarily by the process holding it, after that process has completed the task.
• Mutual Exclusion : It means only one process can use the resource at a time. If some other process requests that resource, the requesting process has to wait until the resource is released.
• Circular Condition : There must exist a state of waiting processes in which process P0 is waiting for a resource that is held by P1, P1 waiting for resource held by P2, Pn-1 waiting for resource held by Pn an Pn is waiting for resource held by P0.

1. Draw up a list of things to test.

* You can’t test everything. Decide what are the most important tasks you want users to be able to perform for your product or site, and prioritize them as to which are the ones to be done first.

2. Draw up the scenarios that you want the users to test.

* The scenarios should be based on some of the initial customers that you had designed, and should be realistic.
* Keep the tests to be a reasonable time frame. Remember that people take a certain amount of time to get comfortable with a topic, so anything less than 10 minutes could be tricky, One the other hand, if it gets too long, it could make the subjects weary.
* Test the scenarios on yourself before you use them on your test subjects.

3. Write a script for administering the test — this is necessary for consistency. It is particularly important if more than one person will be conducting the test, or if you cannot do the test at some point of time. Also, finalizing a script is a good way of ensuring that key people have signed off on the tests.

* Include in the script an introduction to the test, an explanation of how the test will be conducted, the questions that will be asked, and suggestions for follow-up questions to encourage the subjects to discuss what they are doing and why.
* Allow for some free format answers
* You can ask test subjects to think out loud — a behavior that is not always natural to everyone. You may need to drill this into the subjects periodically, but it is very important - it helps find out what they are thinking.

4. Test the test. Try it on some users — find out if the scenarios are comprehensible; if there are some problems, then you may need to rework the tests.

5. Train the testers and recorders.

* First-time testers should each practice conducting a test with volunteers or on each other - this will help them get more comfortable and gain a bit of experience.
* First-time testers should then discuss the experience of conducting the test.
* Critique each other, appreciate good things, but learn from mistakes and highlight necessary improvements to be made.
* Practice and critique the recording of subjects, comments.

6. Gather volunteers to be tested, using the various sources that you would have (forums, existing customers, etc).

* Expect to provide some form of compensation to your volunteers, it could be something as simple as a copy of the last released version of the software, or maybe some sort of company branded gift item
* Volunteers should be selected so that they are representative for the general user population. Choose a mixture of age, year in school, occupation, etc.
* People who volunteered should be scheduled and sent several follow-up messages about the time and place for their test.
* Even with reminder messages, expect some people to drop out - there may be sudden personal issues or some other such reason.

7. Make sure you have a quiet place to do the testing. It’s a good idea to conduct the practice test in the actual room that you will use for the real testing and that the subjects will not have much distractions in such testing. It will not help if people keep dropping in and out of the place.

8. Conduct the tests.

* Schedule enough time between tests for the tester and the recorder to debrief before the next test begins.
* If you discover that a test question or a design element is not working, it is unwise to continue testing.

Fix what is wrong before you continue testing. Make the change and test the correction. The computer industry calls this “rapid prototyping”.

9. Record the test results as soon as possible after the test is completed. If possible, stream the actual process so that interested folks can see it easily as it is going on.

10. Analyze the results and determine how to correct the design problems. Redesign based upon usability evaluation results.

Brief steps that need to be taken:
- Recruiting (one month in advance - selecting locations than dates and participants
- Travel Planning (one month in advance - in tandem with the recruiting)
- Prototype development and refinement (2-4 weeks in advance depending on the fidelity)
- Task development (2-4 weeks in tandem with the prototype)
- Approvals (2-4 weeks in advance depending on the formality of the study)

For a test engineer, a defect is:
• Any deviation from a given specification.
• Anything that causes user dissatisfaction
• Incorrect output
• Software is not behaving the way it should behave.

Difference between bug, defect and error:-
• Software is said to have a bug if the features deviate from the specifications.
• Software is said to have a defect if it is having some unwanted side effects.
• Software is said to have an error if it is giving an incorrect output.

DEFECT TAXONOMIES

All software defects can be broadly categorized into the below mentioned types:
• Errors of Commission : when something wrong is done.
• Errors of Omission : when something is left out by accident.
• Errors of Clarity & Ambiguity : when there are different interpretations.
• Errors of speed & capacity.
• Conceptual or Design bugs.
• Coding bugs
• Integration bugs.
• User interface errors.
• Functional errors.
• Communication errors.
• Missing commands
• Performance
• Output
• Error handling errors.
• Boundary related errors
• Calculation errors
• Control flow errors.
• Race condition errors.
• Load condition errors.
• Hardware errors.
• Source & Version control errors.
• Documentation errors.
• Testing errors.