Designing a usability test of apps and web sites on mobile devices or tablets

User experience evaluation Usability testing Usability

Learn more about how to conduct usability studies with eye tracking on mobile devices, smart phones and tablets.

User testing on mobile devices

As more people move to mobile devices, such as smart phones and tablets, as their primary or only internet-connected computer, usability testing of applications and responsive-design websites is seeing an associated increase in interest. Typically, evaluation methods that have been developed for assessing the usability of desktop applications and web sites are also used to evaluate mobile devices and applications, for example eye tracking. However, since mobile devices differ significantly from laptops and desktop computers, these differences need to be taken into consideration when testing and evaluating mobile interfaces or content. The most obvious difference that calls for special considerations is the form factor of mobile devices. Mobile devices tend to be significantly smaller than their stationary relatives. Therefore,

observing the small screen of a mobile device can be physically difficult for an observer. Additionally specialized eye tracking and screen capture software such as Tobii Studio cannot be installed on non-Windows operating systems. Consequently, special equipment, like the Tobii Pro Mobile Device Stand for X2, has been developed to capture the screen of mobile devices and the user interactions.

On this page you can learn more about the what to consider when designing and preparing for a usability study of apps, we pages or other content on mobile devices, smart phones and tablet. To learn more about the topic please download our free whitepaper:

Eye tracking mobile devices

Most eye tracking studies aim to identify and analyze the visual attention patterns of users whilst they are performing specific tasks (e.g. reading, interacting with software, scanning an image, browsing the web, etc.). Similar tasks can also be analyzed when tracking participants who are interacting with a mobile device, mobile applications or web pages. It is however worth keeping in mind that when holding the mobile device at a typical distance of 50 cm, only one eye fixation is necessary for the brain to

get an accurate and clear image of a substantial part of the display (depending on screen size). Hence, using an eye tracker to analyze detailed reading or scanning patterns on a mobile device can be challenging. Eye tracking can however still provide useful information in mobile application or content testing if the chosen test setup is carefully designed. To overcome some of these challenges several different solutions for mobile device usability testing are currently used, like the Tobii Pro Mobile Device Stand for X2.

The Tobii Pro Mobile Device Stand for X2

The Tobii Mobile Device Stand for X2 is designed to be used with Tobii Pro X2-30, X2-60, X3-120 Eye Trackers and a Windows computer running the Tobii Studio Software. The Tobii Mobile Device Testing Solution enables collection of eye tracking data during natural interaction with smart phones, tablets and other devices or objects of similar size such as books or brochures. The tested device (e.g. smart phone or tablet) is firmly attached to a supplied holder. The stand

is delivered with all necessary equipment for eight different configurations optimized for different devices and setups. It includes an adjustable scene camera (that points at and films the mobile device) to enable capturing of user actions and the user interface. Eye tracking data will be superimposed on images and video captured with the scene camera in the analysis software, Tobii Studio.

Designing tasks for testing

Testing the usability of mobile applications or content is somewhat different compared to conventional usability testing. Mobiles can be used almost anywhere at any time. Depending on which aspect of the mobile application is the focus of the testing the setup of the usability test has to be designed accordingly. The different methods used in mobile usability evaluation offer various approaches on how to do this. For example, field usability studies take place in a real world environment such as a shopping mall; while lab studies take place in a controlled lab environment.

It is essential to think about the scenarios that users may want to accomplish while being mobile when planning usability evaluations. Test tasks should be as representative as possible to tasks that users are likely to be trying to

do with their mobile device. Use task scenarios to simulate aspects of a mobile usage context when testing in a laboratory environment. By providing the participant with a short scenario, it is possible to simulate certain aspects such as short session length, single handed operation, and interruptions etc. Formulate task scenarios in a way so that participants are able to complete them in a short amount of time. This is because real-world mobile device usage is usually of a short session length. Do not conduct an eye tracking study by merely asking participants to look at a mobile website or application. People typically do not do this in real life. Tasks determine the way people look at a stimulus. Therefore, always give participants a full task when conducting an eye tracking study.


Eye tracking will on it's own in most cases not answer all the questions you might have about the mobile device, the application or the web page. Always collect additional qualitative data in order to help you interpret the participant's behavior. Do not use concurrent think aloud in eye tracking studies where the data is going to be used for visualizations or statistical analysis since thinking out loud during the task sessions changes the way a participant looks at an interface. Use primarily retrospective think aloud to collect additional qualitative data when working with eye tracking. With the retrospective think

aloud technique, the participant is instructed to work on a set of tasks without the obligation to think out loud. Both a video recording of the device's screen or a gaze replay can be used to stimulate the retrospective interview. Plan for collecting the retrospective verbalizations after all tasks have been completed by the participant. Always prepare a test session script that contains everything you have to do and say to the participants. That way you can make sure that the test results are not influenced by participants receiving different instructions or proceeding through the test session in different ways.

Testing in a lab or in the field

While mobile devices are intended to be used freely held in the hands, this introduces some complexities for the user experience researcher. A function that a person uses whilst on the move should ideally be tested in as real a situation as possible. However, it might not be ideal to test the first prototype of an application in a real setting as such testing often requires a lot of resources (planning, setup, participants, monitoring, analysis time etc). Instead, it is often more beneficial to test the application or the web page first in a lab as this allows the most fundamental usability issues to be identified at an early stage and at a much lower cost.

By far the most commonly used methodology when testing content on mobile devices is laboratory experiments. These tests generally take place in a usability laboratory or some sort of artificial and controlled environment. An important benefit of testing in a lab setting, as compared with various field study methodologies, is the control over different variables, which are replicable in a laboratory environment. Laboratory testing also requires less time and fewer resources and offers easier and more possibilities to record the test sessions. To record the interactions with the device and the eye movements in a laboratory setting a stationary eye tracker like the Tobii Pro X2-30, X2-60 or X3-120 is used together with the Tobii Pro Mobile Device Stand for X2.

Another common methodology is field studies or observational fieldwork, which is a broad class of methods conducted in the users' natural environment. Field studies provide a more realistic environment for the test participants.

Typically, researchers follow participants while they are on the move and ask them to work on prepared tasks. Field studies are often used to gain insight into and more understanding of concepts such as mobility rather than for usability testing. The data collected during field studies is also often very time consuming to analyze and collect. In many cases field testing does not reveal any additional usability issues than those that could have be captured via laboratory experiments. To record the interactions with the device and the eye movements in a field study a wearable or head-mounted eye tracking system like the Tobii Pro Glasses 2 is recommended.

The choice of whether to use one or the other method and approach is, of course, completely up to the researcher and is dependent on the research questions that needs to be answered. It is not uncommon to see both approaches during the design process. We recommend conducting eye tracking studies of mobile interfaces in a laboratory setting. In a laboratory the environment can be controlled and non-mobile recording equipment can be used. Because test participants are typically seated and do not move around, capturing their interaction with the mobile device will be much easier. The benefits of this mounted approach are two-fold. First, a stationary eye tracker like the Tobii Pro X3-120 or X2-60 can deliver higher sampling rates and greater accuracy. Second, the Scene generation tools in Tobii Studio provide for an analytical workflow that has a higher degree of automation and aggregation and greater throughput and efficiency than the Tobii Glasses Analyzer for the Glasses 2 wearable eye tracker.

Recruiting participants for mobile device tests

Recruiting participants for usability testing of a mobile application or a mobile website is much like recruiting for any usability test session. If testing with a specific device is required, try to recruit participants who are familiar with that particular device. Participants cannot be expected to know how to handle a device that differs from the one they own and use regularly. Therefore, their performance during a test may suffer if they have to use an unfamiliar device. If it is impossible to recruit participants who own the type of device that is going to be used for testing, give the participants a practice task before starting the actual test. That way, participants get the chance to acclimate themselves with the device.

Choosing the number of participants to include in a usability test is a disputed question in the usability research community and there is not a single sample

size that is suitable for all studies. All of the factors that influence sample sizes in regular usability testing also apply to eye tracking studies. While qualitative eye tracking studies can be conducted with few participants (usually only a handful), statistically significant quantitative studies that are aimed at explaining general behavior require larger samples (10's or even 100's of participant). In a qualitative usability study, spend the available resources on conducting several rounds of testing with fewer participants and fix the identified usability problems after every round of testing instead of conducting one elaborate test with many participants. When using the Tobii Pro Mobile Device Stand for X2 be prepared to lose a few of the eye tracking recordings due to potential technical reasons or because participants cannot be tracked with acceptable quality. Therefore, over-recruitment is recommended in order to end up with enough good quality recordings for analysis.

Pilot testing

Conducting eye tracking tests with mobile devices requires significantly more time for planning, recording and analysis from the researcher compared to eye tracking studies conducted on a laptop or desktop computer. The setup is also technically more complex and thus the risk for errors and mistakes

is also greater. We recommend that you always run a complete pilot study to test and get acquainted with the setup, the scenarios, all the equipment included, as well as the data collected and the analyzis tools available in Tobii Studio before the real test commence.

Analyzing the data collected

When using eye tracking to study websites on a desktop system, the analysis software, Tobii Studio, is able to superimpose the collected gaze data onto a screen capture (still image) of the whole website. Unfortunately this is not possible when working with a scene camera recordings like in the case of the mobile device stand or when using the Tobii Pro Glasses 2. In this case recordings need to

be divided into "scenes" in Tobii Studio before visualizations can be created. Reviewing the recordings and manually creating scenes in Tobii Studio can be time consuming. Thus using the standard visualizations such as heatmaps and gaze plots are not always suitable for eye tracking studies conducted on mobile devices. Conducting a qualitative analysis based on gaze play is the most effective way to analyze eye tracking sessions on mobile devices.

Learn more about mobile device testing by reading the whitepaper and User manual

Related Articles