If something isn’t performing well, you have to go be a detective and figure out why. That’s the essence of doing user testing. Usability testing is a qualitative method of research which is much better suited for answering question about why or how to fix a problem. I have attached a sample template
http://spin.atomicobject.com/2012/02/21/usability-testing-summary-template/usabilitytestresultstemplate/
The only possible answer is: it depends.
It depends on the scope of your system/website how many users you need to elicit all the problems. It depends on how sure you want to be that you have found every problem (this is less crucial when more iterations are foreseen). In my view, usability testing is not about creating generalisable results, but about finding problems. And a problem that has only been found once, may be just as important as a problem that has been found numerous times. Proper classification of problem severity should solve this. Therefore, I'm not really fond of publications that state you should have 10 users. period.
The only possible answer is: it depends.
It depends on the scope of your system/website how many users you need to elicit all the problems. It depends on how sure you want to be that you have found every problem (this is less crucial when more iterations are foreseen). In my view, usability testing is not about creating generalisable results, but about finding problems. And a problem that has only been found once, may be just as important as a problem that has been found numerous times. Proper classification of problem severity should solve this. Therefore, I'm not really fond of publications that state you should have 10 users. period.
Indeed, it depends.
One advice is that one user is better than no user and significant problems identified are a strong evidence that the system should be revised. Having 10 users in a user test may not tell much if the system includes end-users with different roles or interests (e.g., a website for a university may need to tailor its content to lecturers, researchers, students, and other visitors. All of them will look for different content. 10 lecturers in the user test may not tell much as you may not know if the website is working well for all the other end-users.
Sensible numbers can be selected based on what is required and functions/roles explored within the system. The bigger the number the more confident you can generalise your results. Researchers may not claim a problem to influence all users if they had only 1 user.
Just as an example, for some scientific papers we conducted usability-related studies involving a group between 25-35 people.
My 2 cents.
Have a look at this article: http://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/
My recommendation is to test with 12 users. After that you tend to find the same issues coming back again and again. With 12 or so, you tend to get the 80% of your problems.
That said, 1. if you have multiple user types or personae using the same interface you will want a representative sample of around 4 each. 2. Remember that iterative design is not about fixing individual problems (as emerge from testing) but reading between the lines of all the feedback or insights to design a 'new' solution based on gestalt interpretation. Just fixing individual items is likely to continue to fragment your design.
http://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/
I agree with the response of "it depends". The research I'm working on publishing right now, discusses why it depends...:-)
The user characteristics is why...:-)
When Nielsen advocated the use of 5 users I'm assuming (although its not clear) that his assumptions include that their characteristics are all similar. The characteristics I refer to include the ability to identify errors due to awareness of the system and/or content; other characteristics that make them better testers than others and I would also like to add the testing method being used (heuristic evaluations may require more/less than a cognitive walkthrough or even a think aloud).
So I think the answer "it depends" as nebulous as it sounds is appropriate..;-)
I agree with the statement "It depends" (on many factors).
We conducted a usability test (thinking aloud, observation and interview) with more than 150 users, because the SUT (system under test) was a security-relevant embedded device and we had to know what kind of UX problems can occur in the real world. Additional to these user-centered testing methods, we also used formal and scenario-based methods. It turned out to be a good decision to have (much) more than 5 or 12 users.
So, the claims of "five users is enough" or "with 12 or so, you'll get 80% of your problems" are "a little bit" dangerous; they are, from my point of view, misleading statements.
If we are doing product usability tests, we have 8 to 12 users - usually with similar characteristics relevant to the (healthcare) product being tested. We find this provides enough data to feedback themes into the iterative design process.
When it comes to sample sizes you may also want to look into the works of Schmettow (e.g. Sample Size in Usability Studies - doi:10.1145/2133806.2133824). He shows that Nielsens 5 user thumb-rule depends on certain assumptions (especially the homogeneity of defect visibility). These assumptions are not always valid. The comment from Roland Petrasch is an indication, but there are other reports from Jared Spool and analyses by Schmettow which lead into that direction. The problem is that you never know the characteristics of your sample before you have it. So the most reasonable approach would be to start testing and to estimate the “hidden defects” until you reach a predefined limit. The problem is that this approach is difficult when you have to stay within budget. Or even better, you have to negotiate for one with your management. Incremental testing is very effective and in the end an excellent way to get the best balance between cost and benefit. But it is difficult to do in business.
- Badges
- Science topic
- Similar topics
- Developmental Psychology
- Attachment