Learn how to aggregate eye tracking data across several participants in web recordings
In most web studies the participants are given a task that involves browsing through a website or an application. As they browse, they are not just passively watching stimuli that are presented in a pre-determined order and duration, instead, they change and interact with the stimulus presentation, resulting in a unique experience for each participant. Comparing these individual experiences is not a trivial task and present challenges to the automatic aggregation of eye tracking data across multiple participants.
For example, when browsing a website some participants may stay on a webpage longer than others, navigate to different pages, play and stop videos at different times and for different durations. If one or two participants spend a large amount of time viewing a webpage, their number of fixations will also likely increase, potentially adding a bias to the aggregated data. There can also be some dynamic elements on the page, such as drop-down menus, pop-up windows, and flash content, that may present some challenges to automatic data aggregation.
When performing a web recording it is important to be aware of these different challenges and prepare for the analysis accordingly. In this article, you will learn about the different workflows of Pro Lab that enable you to aggregate eye tracking data across several participants. We will highlight the cases where you will be able to aggregate automatically and those that involve a more manual workflow.
When performing a test with a web element, Pro Lab records a viewport* video of the whole session and simultaneously extracts full-page screenshots of all the unique URLs visited. Together with these screenshots it also stores information about the associated web navigation times of interest (TOIs) and events (that determine the start and end of those TOIs). During the analysis, you can use the video of the session to observe the participant's behavior and find usability problems and possible explanations, or use the screenshots and associated TOIs to segment and aggregate the data to each webpage. For example, when you aggregate the data on a heat map Pro Lab uses the screenshot as the background image and the two TOI events to determine which data is represented on the heatmap.
*Viewport is the portion of the webpage that is displayed on the screen when the participant is browsing.
If during the session the participants visit the same webpage more than once, each visit will generate the same start and end events and will be recorded as different intervals and screenshots belonging to the same TOI. By default, when you select a TOI, Pro Lab aggregates the data of all the intervals under that TOI.
Figure 1. Shows how different visits to a webpage are aggregated into the same web navigation TOI.
Next, you will learn the step by step procedure to create an aggregated data visualization and extract eye tracking metrics based on the web navigation TOIs.
If your study requires you to aggregate data to webpages and there are few or no dynamic elements, you can use the web navigation TOIs and associated screenshots to aggregate your data. However, in situations where you expect to encounter dynamic elements, you will run into three challenges:
** E.g. Flash elements are not currently supported by the Pro Lab Browser.
***The only exception is for fixed elements, Tobii Pro Lab is able to track these elements and map the data correctly. Fixed elements are elements in a website that are positioned relative to the browser window instead of the page itself. When the test participants scroll down on a website with fixed elements, those elements stay in their place while the rest of the content moves. These elements will be given a fixed position in the screenshot. Any gaze on them will be mapped to their position of the screenshot regardless of where the element was positioned when it was actually viewed. This will give a more true representation of the fixations.
The listed challenges above imply that you won’t be able to rely on the web navigation TOIs generated by the system, as there will be portions of gaze data that will be aggregated to the automatic web screenshot that is not directed at the elements displayed by the screenshot. Instead, they belong to the participant’s interaction with dynamic elements. For example, if the participants access a drop-down menu on the website the viewing patterns on that menu will be recorded and aggregated on the screenshot of the webpage, but the menu itself won’t be visible on that screenshot. This situation is true for other dynamic elements, such as pop-ups that do not generate a unique URL or elements that move across the screen.
As previously mentioned, during the test recording, Pro Lab also captures a viewport video of the whole browsing session. This video can be viewed later when you replay the recording. Since this video is essentially a screen recording of the participant's browsing activity, you will be able to view the different dynamic elements and the participant viewing behavior and interaction. During your data analysis, you can use this video to manually create events on the timeline that mark the start and end of the interaction with those elements. Once you created these events you can further use them to generate custom TOIs with a video frame. The custom TOI aggregates the gaze data that corresponds to the viewing activity towards the dynamic element, and the video frame is used to illustrate the webpage with the dynamic element visible. By combining the custom TOI and frame you can aggregate data from different recordings to produce visualizations and create AOIs to extract eye tracking metrics.
In the previous example where the participant interacts with a drop-down menu, you would need to log two events, e.g. drop-down open and drop-down close, and create a custom TOI with a frame using these events as the start and end point of the TOI. The frame needs to be extracted from a time in the video when the drop-down menu is open. I would repeat this process in the remaining recordings, and e.g. create a heatmap by selecting the custom TOI in the Visualization tools. This heat map shows the aggregated viewing activity towards the drop-down menu. Below you will find step by step instructions on how to create a custom TOI with a frame.
After you create your Custom TOIs with an associated frame, you can use these TOIs in the same way as the web navigation TOIs to produce visualizations and extract metrics based on aggregated data of several participants.
A limitation of this analysis workflow is that the dynamic element needs to fit within the viewport. If the participant needs to scroll to interact with the element then this solution won't work, as you will not be able to summarize the viewing behavior on a single video frame.
Dynamic web elements | Pro Lab Analysis elements | |
None | Web navigation TOIs associated with webpage screenshots | Events are logged automatically, and full-page screenshots are captured automatically |
Expandable elements and content that do not require scrolling (fits the viewport) | Custom TOIs associated with video frames | Events need to be logged manually and the frames manually chosen from the video |
Expandable elements and content that require scrolling (does not fit the viewport) | Manual mapping (section 8.2 User manual) | Fixations need to be mapped manually on a visual coding scheme sheet or an imported web page image |
Fixed elements | Web navigation TOIs associated with webpage screenshots OR web navigation custom TOI associated to a web screenshot | Events are logged automatically, and full-page screenshots are captured automatically |
Pop-up elements that generate a new URL | Web navigation TOIs associated with webpage screenshots Or web navigation custom TOI associated to a web screenshot | Events are logged automatically, and full-page screenshots are captured automatically |
Pop-up elements that do not generate a new URL | Custom TOIs associated with video frames | Events need to be logged manually and the frames manually chosen from the video |
Embedded mp4 video content | Not possible (not supported by the browser) | Not visible in the video recording |
Embedded flash video content | Not possible (not supported by the browser) | Not visible in the video recording |
In some cases, certain websites may contain elements that remain constant when the participant visits different pages (i.e. these elements remain in the same location and display the same content). If these elements are relevant to the task that spans accross these pages the researcher may be interested in aggregating the data towards these elements across all the pages they are present. This can be done in Pro Lab by using web navigation custom TOIs. Where the researcher uses an automatically generated screenshot from one page where that element is present, to map data from the other pages.
After you create your web navigation custom TOIs, you can generate visualizations and extract metrics based on the aggregated data of several participants and the grouping of web pages.
Aggregating data across multiple test participants in web studies can be conceptually and technically challenging, particularly on today’s web pages, where dynamic elements are abundant.
Conceptually, it is challenging to compare data from several recordings when the dynamic content makes each interaction with a web page different for each participant. Technically, it is challenging to capture content that changes over time in a useful way to be used in visualizations of eye tracking data—especially if the content only changes on some parts of a page.
When preparing a study, you need to consider these challenges and assess the implications on the analysis workflow. This will allow you to have a more realistic estimate of the time and resources you need to dedicate to your data analysis procedure.