Combining think aloud methods and eye tracking in usability tests

Web usability Usability testing Retrospective Think Aloud

Learn about different think aloud methods suitable for usability testing and how these methods can be combined with eye tracking.

In website usability testing, participants’ eye movement patterns and fixations can be used to gain an understanding about how they complete a task and how difficult or easy it is to complete. An eye tracker produces objective numerical data as well as visual results that can provide additional insight into the participants’ behavior and the interface tested. Ideally, when evaluating usability, eye tracking data should be combined with other methods and data because eye movements cannot always be interpreted correctly without the participant providing context to the data. For example, longer fixations can mean a participant found a particular area interesting, but it can also mean that he or she found the area difficult to interpret. Hence, it is important to supplement eye tracking data with additional information gained from the participants about their experiences. When using eye tracking in usability studies, it is important to select the most suitable methodology in order to extract relevant and useful data from the participants.

Think aloud methods

Think aloud methods are often used when trying to detect usability problems. The most successful form of think aloud to use together with an eye tracker is a retrospective think aloud methodology (RTA), which means that participants verbalize their thoughts after completing a task or a set of tasks. RTA allows the participant to complete a task on their own and in silence. The other most common form of think aloud is the concurrent think aloud method (CTA) where participants verbalize their thoughts during task completion. Both methodologies have their own sets of limitations and problems, but using CTA in combination with eye tracking has proven to be less suitable as participants produce different eye movements compared with the RTA method. Differences in eye movements are for example caused by users looking away from the screen when describing something to the researcher in the CTA method or by focusing on certain areas of the screen while describing their thought processes regarding that area. Therefore, the RTA method is the recommended method when conducting usability tests where also objective eye movement data will be analyzed.

The cued retrospective think aloud method

RTA is not always a problem‐free method; as the participant is asked to remember their experiences rather than provide their immediate response when doing the task, important things might be forgotten or imperfect memories become the source of fabricated usability problems. In order to aid the participant’s memory, and to minimize the risk of fabrication, cued RTA methods have emerged. In a cued RTA, the participant is presented during the interview with a replay of their user journey while completing the task or tasks. The replay is then used to help cue their think aloud. Examples of commonly used cues are screen shots of the task they completed or a video playback of the participant’s user journey or eye movements made during the same user journey.

The RTA interview can be conducted in two different ways, either directly after the participant has completed a task or after the participant has completed all or a few tasks. Which method you choose depends on, among other things, how much time is usually required to complete the tasks, if the tasks are a part of a larger process and if you want to evaluate the entire process before interviewing. Conducting the interview after every task might also make the participant more aware of being eye tracked and tested which in some cases can have an impact on how the tasks are completed. This is generally a good thing to try out during pilot testing or to think through before starting the real test.

Depending on the purpose of the testing and what kind of stimuli is to be used, different kinds of RTA methods should be considered. The quickest and easiest method is to use RTA without any cue, i.e. to just interview the participant after completing the tasks. However, this method does not encourage the participant to talk and remember problems encountered while completing the tasks. Using a screen recording video as a cue in the RTA interview will encourage the participant to talk more than using no cue at all. Research reveals that when using participants’ own eye movements as a cue, participants tend to talk more and remember more usability problems than when they do not see their own eye movements. In this case an eye gaze video shows a video recording of the participant’s interaction with the page on which the eye movements and fixations are also shown. Using an eye gaze video replay as a cue has proven useful for identifying a broad spectrum of usability issues. As it shows an entire user journey in sequence, it is particularly suitable for evaluating dynamic content or processes. Additionally, as it shows eye movements on the screen as they happened, the participant is presented with context around an interaction with the interface and might therefore recall the reasoning behind an action that they might not have thought of if they only saw the static gaze plot. 

RTA in Tobii Pro Studio

If you are using Tobii Pro Studio Enterprise version, there is an easy‐to‐use RTA function included that allows you to quickly set up the RTA recording of your choice (gaze plot cued, eye movement cued, video cued etc.). By clicking the RTA button you can name the RTA recording and then initiate a screen recording where you can present the participant with whatever cue you want to use from Pro Studio or on the computer. The participant’s reactions and comments about the task they just completed are then video and audio recorded in Pro Studio. Participants can now both discuss the task they completed and answer any pre‐set questions you have created, while it is all recorded by Pro Studio.

If you choose to replay a video of the participant’s eye movements during the RTA interview it is important to consider the video settings. In Pro Tobii Studio the user can control some aspects of the video, such as play/pause, rewind/fast forward and playback speed. These features allow participants to pause if they need more time to say what they want to say or to fast forward if they feel they have said everything they need to during a specific section.