It’s necessary to be careful with eyetracking studies. The Cuil example.

An eyetracking study by Think Eyetracking, which has been widely spread on the Internet, announces the death of Cuil. It is based on ocular data. 

Let’s take some time to analyze this

The objective of a user test consists in gathering data on the behaviour of users while they are performing specific tasks on graphic interfaces of which the tester wants to measure the efficiency. 

When asked to perform a task, different parts of the user’s brain will be activated. The more the task is linked with an interest or a real need felt by the user, the more attention he will dedicate to the task at hand. 

Motivation components

Why is this attention level of test participants so important? 

Recent work by the Professor Eric Knudsen (Knudsen, Eric I, 2007), called “Fundamental Components of Attention”, Annual Review of Neuroscience 30(1): 57–78) has identified the four main tasks of the concept ‘attention’

  1. Attention allows for the storage of relevant data in the working memory. When there is low attention, the stored content will be low as well. 
  2. Attention analyzes the relevance of information stored in the working memory via a process that puts the information that can be found in the interface to the test. In case of low attention, the relevance of the information won’t be evaluated in a thorough manner.
  3. Attention allows for the choice of new relevant information on the site, based on the content that is already stored in the working memory. This top-down mechanism facilitates the addition of new information, on a recurring basis. This is called endogenous attention. In the case of low attention, one risks not to choose new information. 
  4. Attention automatically filters information found on a site that doesn’t match the frequent stimuli of the brain, the so-called exogenous attention. Again, in the case of low attention, this filter will be weak. 

It is equally important to know that the attention I will dedicate to the performance of a task is linked to the interest I have for this task in general. Motivation plays a vital role. 

Let’s take an example :

“Let’s say my car broke down. I go on the Internet to look for a company who can come and help me out as soon as possible.”

In this case, the motivation I have will be much higher compared to my level of motivation, when I asked to google this kind of company when I take the metro every single day. 

In other words, the behaviour you’re analyzing depends on the attention the user will dedicate to the task you ask him to perform. 

And as the level of attention is closely linked to the interest or motivation of the user, his behaviour will also vary in function of the tasks or the users. 

If you want to make relevant conclusions, you need to take into account a number of factors. You need:

  1. users who are interested in or motivated to do the required task.
  2. a clear task that can activate a knowledge network that is sufficiently precise. 

If you can’t meet these two conditions, the generated and analyzed behaviour will automatically be so-called behaviour with a diffuse attention. 

In case of the Cuil search engine, the conclusions are based on the behaviour of users who were asked to use Cuil to find information on the keyword “Oasis”. 

CUIL eyetracking hotspot

I have doubts on the motivation or interest of the 30 users. How interested were they in the term “Oasis” and what was the link between this term and the real task performed by the users?

The level of transparency of the task will not allow the users to focus their attention on one type of content or another and to evaluate the relevance of the content found. 

The probably behaviour: users will mechanically look at the entire interface (diffuse attention) and won’t focus as much as they would do when they asked to perform a task with a high level of motivation. 

This means the results of the Think Eyetracking test shows a usage pattern in diffuse attention mode (almost the entire interface had been visited). 

In order to show a behavioural pattern in which the attention was indeed focused, we have done a user test on Cuil, in which we have asked user to test the following scenario:

“You need to go to Barcelona for a business meeting on 15 October. You want to find a four-star hotel near the city centre”. 

The users can understand the motivation behind the task. Furthermore, it is a clear and well-defined task that allows them to activate their knowledge needed to compare and search for hotels. In other words: attention is focused. 

And, surprise surprise, the results are somewhat different ☺.

It is also important to have a look at the progression of the heat map.

This will show you how the attention evolves over time. Compare for instance the results after 3, 6 and 9 seconds.  

CUIL eyetracking hotspot 3sec

CUIL eyetracking hotspot 6 sec

CUIL eyetracking hotspot 9 sec

On top of this methodological problem, it is impossible to make any conclusions on the efficiency of Cuil based on the heat map and nothing else.

Because one and the same heat map shows just as many diverse types of behaviour. But more on that in the following posts. 

Have a good week!