I often talk about the power gurus have over the intellect of professionals working in usability.
By doing so I hope to give back the willingness and eagerness to think, instead of believing everything these gurus tell, without using our own brain and our own critical minds.
Here’s another example of this cerebral anaesthesia: the 5-tester myth…
In 1993, Jakob Nielsen states in a paper that, according to him, 5 testers are enough to identify 80% of ergonomics problems.
A luring statement for people who work in usability because it allows them to put only a very limited number of people in front of a screen. Furthermore, it speeds things up considerably.
A large number of scientists who have conducted studies aimed at measuring the real impact the number of testers has on the performance of a web site, have raised objections against this theory. And companies have also discovered the limits of this myth.
Here’s an example. During a study conducted by Spool & Schroeder in 2001 (fiveusers.pdf), the first five users only revealed 35% of the ergonomics problems of a website. In this same study, the 13th and 15th tester have identified major issues on the website.
Another test used 18 testers. And they have found more than five new obstacles once the number of testers exceeded the magical number 5 (Perfetti&Landesman, 2002).
Laura Faulkner, who is a scientist working at the University of Texas in Austin, has conducted a study in which 60 testers were present (faulkner_brmic_vol35.pdf). The 60 testers were grouped randomly in groups of 5, 10, …
The results are quite revelatory:
- The 12 groups of 5 testers have found between 55 and 85% of the problems.
- By putting people in groups of 10, the minimum percentage of identified problems raises to 80%.
- By making groups of 20 testers, the minimum percentage of identified problems raises to 95%.
Using 15 users will allow for the optimum balance between costs and reliability. You will indeed discover between 90 and 97% of problems. After more than 150 projects, my field experience confirms these different scientific results.
That leaves the question to use techniques allowing you to gather objective data and to avoid subjectivity. We’ll come back to that later…
Have a good week. Marc