Learn Software Development » Testing

Getting team members to attend user feedback sessions, usability being very important (Part 4)..

ashish — Tue, 01 May 2012 03:41:38 +0000

In the previous post (methods for getting team to view user testing), I had talked about the ways in which a development team can view the results (rather, the actual process) of the user testing (sending somebody to the actual location; getting a real time broadcast of the event; running the videos later). However, in this post, I will try to get into more detail of what the user testing actually reveals, and how it more than just a video of the session. Interpreting user testing does require a fair degree of professionalism, and to get the best value out of the user testing, it is recommended that you have an expert in the field of user testing to run through the entire process.
Why do you need an expert to run the user testing process ? Well, because if you don’t run the process properly, you get results that may not reflect the sample of the population you are looking for as customers, and you can make changes in your application based on the results you get, which means that you are now going off on the wrong path, and that can be very expensive for the application or even spell doom.
A user testing expert will work with the experience designer and the product manager to determine the questions that need to be tested, to break up the queries in a form that can be made into test cases for the users, such that you can have a query for the test users who can then take a series of actions that answer the query (something like, for an imaging application: “If you want to remove the red eye from a photo using application xyd, what would you do ?”). In such a query, you would find out how the users react, which menu do they try, do they look baffled or start trying out things and get to the required tool within a few seconds, and so on. The key in this is framing the questions in such a way that the user can understand it quickly, and yet in a way that the response meets the objective of the user testing process.
One of the biggest areas where the user testing expert starts out with is about ensuring that the right people are selected for the user testing. This is not so easy as it sounds. The conditions for selecting the right sample need to be enumerated based on discussion with the Product Manager (and these could be multiple people from the same type of sample, or a broader sample to try to represent more sections of the desired user set (and keep in mind, since user testing can be pretty expensive, in the tens of thousands, you cannot in most cases go for large sets)), and then the right set of people are selected. Part of this is whether the tests are needed to be carried out at the location of the users, or in a controlled location. Using the user location gives the background of the user environment, and can be helpful in determining the amount of time and focused attention that the user gives to the application. To be contd ..

A Common Sense Approach to Web Usability	The FlowingData Guide to Design, Visualization, and Statistics	Designing with the Mind in Mind

Will continue this in Part 5 …

Getting team members to attend user feedback sessions, usability being very important (Part 3)..

ashish — Thu, 26 Apr 2012 19:41:46 +0000

In the previous post of the series (Usability testing and its importance to the team), we talked about how to get the team to understand the value of user testing of new concepts and workflows, especially through the process of taking the team members through cases where the team members are shown an example of the users finding workflows difficult, and the team expected this workflow to be a bit hit.
In this post, I will talk more about how the team can actually get to experience the process of user testing. Here are some methods that depend on the techniques that the company may have adopted for this purpose:
- Actually being there for the user testing. This is easiest when the user testing takes place in the same location as where the engineering team is based. If they are there in the same location, then it is fairly easy to take the team to the room or location where the user testing is happening, and putting them where they can observe the user testing but not disturb the actual place of user testing. I have seen teams watching the proceeding from a place with a glass through which they can see inside but the people inside cannot see outside.
- When the user testing is happening in a different geographic location from the engineering team. In this case, the attempt should be to ensure that as many of the team members can be sent for the actual user testing. This gets difficult when the team members are in a different country and the user testing is in a different geography. Given the need to ensure that the team members understand the importance of user testing, it does make sense to try and make the budget for this trip, and get the people who are the most involved in the design aspects for the new workflows to make the trip.
- When the team members are not able to make the trip (or most of them are not able to do so), the next best option for the remaining members of the team is to view the user testing on a real time basis. This is through using some sort of broadcasting situation where the user testing is broadcasted through the user of a solution such as Cisco Web broadcasting or Adobe Connect, such that the team members can see the user testing. With modern networks, the clarity of such videos can be pretty high, and with proper placements of cameras and audio, the team can see it similar to what the user testing researcher sees.
- However, there are situations where all of the above situations cannot happen. When the team is in a different geography, especially when the team is based in India, the time zones are very different and it would be hard for the team to actually get involved in viewing the presentation real time. In such cases, the videos of the user testing should be shown to the team, and in a setting where the entire team watches the presentation, and are led by either the Product Manager or the User Experience Designer who can walk through the critical parts of the actual user testing.

A Common Sense Approach to Web Usability	The FlowingData Guide to Design, Visualization, and Statistics	Designing with the Mind in Mind

Will continue this in Part 4 (link) …

Getting team members to attend user feedback sessions, usability being very important (Part 2)..

ashish — Tue, 24 Apr 2012 18:51:24 +0000

In the previous post (User feedback and teams), the post was about the importance of getting feedback from users where you are evaluating changes in the workflows in your product, or modifying some of the features. In this post, I will write more of this topic, specifically with reference to the team members and their exposure to the user feedback sessions.
In today’s world, a lot of teams are spread geographically and hence getting all of them to the same location where the user testing is happening is not possible or feasible (it can be done if the budget is there, but given tough economic times and a lot of stress on the budget, it just does not seem feasible to ensure that the team members can all make it to the actual physical location where the user testing is done). In our case, we had team members in New York, in Romania and in Bangalore, and the actual testing was done in London.
So what are the alternatives that can be employed ? We will talk about some techniques that can be used by the team members to view these user testing sessions, but for now, we need to also ensure that these team members see the value in viewing these user testing sessions.
This is a difficult task. How do you get team members to see the benefits of being more involved with user testing, viewing how users evaluate the workflows that have been built or modified. One easy way of getting them more interested is taking one of the most radical feedbacks from the user testing and having the product manager present this feedback to the team (this could be something such as the users not finding the entry point to a feature which was supposed to be the most useful features for users). This should present a huge challenge to the team, since it would not be easy for many of the team members to understand how users could react in such a way.
This is the perfect point to get the team involved, by challenging them through such methods. If the user session has been video recorded, then at this point the product manager or the user experience designer should show the video from this point onwards (after first displaying some information about the user which shows how the user is part of the target population, and if possible, showing their setup where they use the application including the times of the day when the user typically uses the software). An important part is to emphasize the point where the users face issues in the workflow, and with modern techniques, it is even possible to monitor the movement of the eyes of the user. Being able to see this gives a very good insight into how the users react, and can be very surprising for the team in terms of the way that the users react.

A Common Sense Approach to Web Usability	The FlowingData Guide to Design, Visualization, and Statistics	Designing with the Mind in Mind

Will continue this in Part 3 (link) …

Getting user feedback on new designs, but do you go totally by user feedback ?

ashish — Thu, 19 Apr 2012 19:46:11 +0000

This is a tricky topic. Conventional wisdom says that when you are planning a new product, it is very important to get user feedback through the process, multiple times. The logic: When the team is working on a new application or software product, they tend to get too involved with their own thoughts, their rationale and seek to believe that these are exactly what the users want. There are a number of products that were developed without user feedback and testing, and a large number of those do not really set the user community on fire, many of them are downright failures, since the users really are not looking for the workflows that the team thought would be loved by the users.
To try and ensure that the people involved in the application design and development remain close to what the users want, and also get a chance to vet their design and prototypes, typically a restricted user testing is done. A sample of users are taken, and depending on the current state of development of the application, either screen shots, or prototypes, or the working product is shown to the users and they are encouraged to use the application or evaluate whatever has been given to them. Viewing the interaction of users with the software is invaluable to the design team, since it gives them a fair idea of what the users like, what they dislike, and whether they are comfortable with the workflows that are being shown to them.
So far so good; however, over the past few years, there is another thought that is gaining attention. A number of new hardware and software applications have become successful where the application is meeting a need that users may not have thought about, and in such cases, user testing and feedback may not be the sole criteria on which to go by. It would be very much possible to think that some of the great apps on the iPhone, such as Instagram or others were not really designed and launched after user feedback.
Instead, you had a committed set of application developers who knew what they wanted, and were passionate about what they were doing. If some users did not like it, this would not have shaken them. In such cases, what happens is that even very honest user feedback would not have been the right way to go about; and this is the concept that is now shaking development companies around the world. When you are trying to build something in the area of the web / tablets / phones / Facebook app, a lot of stuff that is being worked upon is new, or atleast the workflows being developed are new. In such a case, the companies do not even go in for user feedback, instead working on releasing the software, seeing the amount of user acceptance and then tweaking (or dropping) the application, and go on. This makes life difficult for companies that have worked the traditional way, and which depend on user feedback as a way of validating new stuff that they are doing.

Some great books from Amazon on usability and user feedback:

A Common Sense Approach to Web Usability	The FlowingData Guide to Design, Visualization, and Statistics	Designing with the Mind in Mind

What is UAT (User acceptance testing) and when is it done as part of the product schedule?

ashish — Sat, 31 Mar 2012 18:19:00 +0000

User acceptance testing or UAT as it is more commonly known as, forms a very important and necessary part of the whole software testing cycle. User acceptance testing determines or gives you an indication about how your software system or application will be perceived by the user, as it seeks to check whether or not the software system meets the user stated specifications. Therefore, it becomes extremely important for a tester to know what user acceptance testing exactly is and what is the most appropriate time for employing it. This article is concerned with the user acceptance testing and its details, and focuses on making related queries as clear as possible.
Before starting a project, certain agreements are made between the client and the company executing the project. As a part of this, for the program developers, some specifications and requirements are set for the software system or application which should be met by the software system at the time of delivery to the client. The project is completed based on the agreement, but one of the big variables is how to ensure that the software system or application satisfies all of the requirements and specifications that were agreed up on during the agreement signing? And this in turn plays a big role in deciding whether the project has succeeded or not, and more important, whether the client will be satisfied enough to pay for the project.
At this point, the user acceptance testing comes in to play. So we can define the user acceptance testing as a sort of software testing technique employed for testing whether or not the software system or application meets all the agreement specified requirements, and there is actually no need to emphasize how important this can be.
The client plays the role of SME or subject matter expert during the user acceptance testing and determines after taking a trial of the software system whether it is meeting all the requirements as were decided over the contractual agreement or not. One of the most frequently asked questions is about the time in the schedule when the user acceptance testing must be performed? By the definition of the user acceptance testing we can conclude that it should be carried out at the end i.e., after the completion of the whole project and before it released or handed over to the client.
User acceptance testing gives an assurance to the executing company that the client would accept the software system or application or whether some rework may be required. Many a times, the user acceptance testing can be performed by the users also and the test cases are created based on the specifications and requirements stated in the contractual agreement by the client. The test cases are usually kept formal and each one is assigned a severity level, and in fact, the user acceptance testing should be carried out as seriously as possible, and any problems reported should be checked and logged for inspection and possible rework.
User acceptance testing serves a means for verification of the software system or application, such that it can help verify that the application meets all the specifications and requirements and works accordingly. User acceptance testing also serves as a means for emulating the usage conditions as seen in the real world. The test cases run by the users or clients are not just focused up on the identification of the simple problems (like spelling and grammatical errors, show stopper defects etc.) but, also focus upon quite complex issues, and with execution by users who act like the end users (in some cases, these users may not have the same level of computer expertise even after relevant training).
It is important that the previous stages of testing have been performed effectively since if more defects are discovered during the user acceptance testing, it can harm the reputation of the organization as well as it can bring down the production rate. It is better if most of the issues are identified and fixed in the former stages of the software testing life cycle. These days a more sophisticated version of UAT is available called “quantified user acceptance testing” and it is also way faster and smarter than the traditional user acceptance testing. The quantified user acceptance testing is governed by seven following aspects:
1. Linear testing
2. Recursive testing and
3. Adaptive testing
4. Major systems and services checks (MSC)
5. Internal consistency checks (ISC)
6. Real time regression (RTR)
7. Reactive regression

Handbook of Usability Testing	User Acceptance Test (UAT) Planning	UAT Defined: A Guide to Practical User Acceptance Testing