Aggregating eye tracking data across several participants in web recordings

Tobii Pro Lab Screen based projects

Learn how to aggregate eye tracking data across several participants in web recordings

In most web studies the participants are given a task that involves browsing through a website or an application. As they browse, they are not just passively watching stimuli that are presented in a pre-determined order and duration, instead, they change and interact with the stimulus presentation, resulting in a unique experience for each participant. Comparing these individual experiences is not a trivial task and present challenges to the automatic aggregation of eye tracking data across multiple participants.

For example, when browsing a website some participants may stay on a webpage longer than others, navigate to different pages, play and stop videos at different times and for different durations. If one or two participants spend a large amount of time viewing a webpage, their number of fixations will also likely increase, potentially adding a bias to the aggregated data. There can also be some dynamic elements on the page, such as drop-down menus, pop-up windows, and flash content, that may present some challenges to automatic data aggregation.

When performing a web recording it is important to be aware of these different challenges and prepare for the analysis accordingly. In this article, you will learn about the different workflows of Pro Lab that enable you to aggregate eye tracking data across several participants. We will highlight the cases where you will be able to aggregate automatically and those that involve a more manual workflow.

Using automatic web navigation TOIs to aggregate data

When performing a test with a web element, Pro Lab records a viewport* video of the whole session and simultaneously extracts full-page screenshots of all the unique URLs visited. Together with these screenshots it also stores information about the associated web navigation times of interest (TOIs) and events (that determine the start and end of those TOIs). During the analysis, you can use the video of the session to observe the participant's behavior and find usability problems and possible explanations, or use the screenshots and associated TOIs to segment and aggregate the data to each webpage. For example, when you aggregate the data on a heat map Pro Lab uses the screenshot as the background image and the two TOI events to determine which data is represented on the heatmap.


*Viewport is the portion of the webpage that is displayed on the screen when the participant is browsing.


If during the session the participants visit the same webpage more than once, each visit will generate the same start and end events and will be recorded as different intervals and screenshots belonging to the same TOI. By default, when you select a TOI, Pro Lab aggregates the data of all the intervals under that TOI.

Web navigation TOis

Figure 1. Shows how different visits to a webpage are aggregated into the same web navigation TOI.

 

Next, you will learn the step by step procedure to create an aggregated data visualization and extract eye tracking metrics based on the web navigation TOIs.

To aggregate the data on a heat map:

  1. Click on the Visualizations button, either in the Project Overview section or from the Analyze drop-down menu in the top navigation.
  2. Go to the Visualization Type and Settings tool in the tool panel, to the right of the Visualization interface and select Heat Map.
  3. Select what Gaze Filter to use when generating your heat map. Read more about Gaze Filters in the “Appendix B Gaze Filters’ functions and effects” in the Pro Lab user manual.
  4. Go to the Times of Interest tool and select the web navigation TOI that corresponds to the webpage you want to use as the base of your Visualization. To help you select the TOI it displays the name and a thumbnail of the webpage.
  5. Go to the Recordings tool and select or deselect the recordings you wish to aggregate in the heat map.
  6. Select the heat map type - absolute count, relative count, absolute duration or relative duration. (for more information about the different types of heat maps, read the appendix “Appendix C Calculating heat maps” in the Pro Lab user manual.)
  7. The heat map is now generated and displays the aggregated data of multiple recordings.

 

To extract aggregated eye tracking measures:

  1. Create your AOIs, by opening an AOI tab
  2. Select the target webpage screenshot in the Media Selection section
  3. Draw your AOIs on the elements of interest to your study
  4. If you want to aggregate data across multiple AOIs you can also create tags and tag each AOI.
  5. Open a Metrics Export tab.
  6. Select the output format for the export file by clicking the Export format drop-down list box in the Settings panel. Select Interval based TSV file (default) .tsv format to segment your data based on each visit to a webpage or the Excel report for the .xlsx format for a summary report.
  7. Select the metrics you want to include in the export (the selection varies depending on what you chose in step 2).
  8. Select which Gaze Filter you want to apply to the data you will export. Read more about Gaze Filters in the “Appendix B Gaze Filters’ functions and effects” in the Pro Lab user manual.
  9. Select the recordings. Choose to export metrics from All Recordings or only the recordings of your choice.
  10. Select your Participant variables to further segment your data according to participant variables.
  11. Select your web navigation TOIs.
  12. Select the AOIs and AOI Tags created in steps 1-4.
  13. If relevant select Web Navigation Event types, to export event related metrics.
  14. Click Export to export the data.

Using custom TOIs to aggregate data

If your study requires you to aggregate data to webpages and there are few or no dynamic elements, you can use the web navigation TOIs and associated screenshots to aggregate your data. However, in situations where you expect to encounter dynamic elements, you will run into three challenges:

  1. Some dynamic elements are not supported by the browser**
  2. Web screenshots will not display certain dynamic web elements***
  3. All the gaze data captured during the webpage visit is mapped to the screenshot irrespective if the user spent some periods interacting and viewing dynamic elements

** E.g. Flash elements are not currently supported by the Pro Lab Browser.

***The only exception is for fixed elements, Tobii Pro Lab is able to track these elements and map the data correctly. Fixed elements are elements in a website that are positioned relative to the browser window instead of the page itself. When the test participants scroll down on a website with fixed elements, those elements stay in their place while the rest of the content moves. These elements will be given a fixed position in the screenshot. Any gaze on them will be mapped to their position of the screenshot regardless of where the element was positioned when it was actually viewed. This will give a more true representation of the fixations. 

The listed challenges above imply that you won’t be able to rely on the web navigation TOIs generated by the system, as there will be portions of gaze data that will be aggregated to the automatic web screenshot that is not directed at the elements displayed by the screenshot. Instead, they belong to the participant’s interaction with dynamic elements. For example, if the participants access a drop-down menu on the website the viewing patterns on that menu will be recorded and aggregated on the screenshot of the webpage, but the menu itself won’t be visible on that screenshot. This situation is true for other dynamic elements, such as pop-ups that do not generate a unique URL or elements that move across the screen.

As previously mentioned, during the test recording, Pro Lab also captures a viewport video of the whole browsing session. This video can be viewed later when you replay the recording. Since this video is essentially a screen recording of the participant's browsing activity, you will be able to view the different dynamic elements and the participant viewing behavior and interaction. During your data analysis, you can use this video to manually create events on the timeline that mark the start and end of the interaction with those elements. Once you created these events you can further use them to generate custom TOIs with a video frame. The custom TOI aggregates the gaze data that corresponds to the viewing activity towards the dynamic element, and the video frame is used to illustrate the webpage with the dynamic element visible. By combining the custom TOI and frame you can aggregate data from different recordings to produce visualizations and create AOIs to extract eye tracking metrics.


In the previous example where the participant interacts with a drop-down menu, you would need to log two events, e.g. drop-down open and drop-down close, and create a custom TOI with a frame using these events as the start and end point of the TOI. The frame needs to be extracted from a time in the video when the drop-down menu is open. I would repeat this process in the remaining recordings, and e.g. create a heatmap by selecting the custom TOI in the Visualization tools. This heat map shows the aggregated viewing activity towards the drop-down menu. Below you will find step by step instructions on how to create a custom TOI with a frame.

To create a custom TOI:

  1. Open a recording
  2. Create one or two user-generated events to define the start and end points of the TOI, e.g. every time the user opens a drop-down menu opens and closes
  3. Move the cursor to a video frame where the menu is open, then click Create frame icon located to the right of the recording time counter above the timeline and select Create Toi with frame. The frame will have the same function as the web screenshot described in the previous section
  4. On the Create custom Time of Interest window select the start and end point events created in step 1
  5. Rename the Media to a more descriptive name
  6. Click OK to create the TOI.
  7. Repeat steps 1 and 2 for the other recordings

After you create your Custom TOIs with an associated frame, you can use these TOIs in the same way as the web navigation TOIs to produce visualizations and extract metrics based on aggregated data of several participants.

A limitation of this analysis workflow is that the dynamic element needs to fit within the viewport. If the participant needs to scroll to interact with the element then this solution won't work, as you will not be able to summarize the viewing behavior on a single video frame.

Conclusion

Aggregating data across multiple test participants in web studies can be conceptually and technically challenging, particularly on today’s web pages, where dynamic elements are abundant.

Conceptually, it is challenging to compare data from several recordings when the dynamic content makes each interaction with a web page different for each participant. Technically, it is challenging to capture content that changes over time in a useful way to be used in visualizations of eye tracking data—especially if the content only changes on some parts of a page.

When preparing a study, you need to consider these challenges and assess the implications on the analysis workflow. This will allow you to have a more realistic estimate of the time and resources you need to dedicate to your data analysis procedure.

Table 1. Displays a summary of different dynamic web elements and the workflows/features that enable data aggregation across several participants.
Dynamic web elements Pro Lab Analysis elements
None Web navigation TOIs associated with webpage screenshots Events are logged automatically, and full-page screenshots are captured automatically
Expandable elements and content that do not require scrolling (fits the viewport) Custom TOIs associated with video frames Events need to be logged manually and the frames manually chosen from the video
Expandable elements and content that require scrolling (does not fit the viewport) Manual mapping (section 8.2 User manual) Fixations need to be mapped manually on a visual coding scheme sheet or an imported web page image
Fixed elements Web navigation TOIs associated with webpage screenshots Events are logged automatically, and full-page screenshots are captured automatically
Pop-up elements that generate a new URL Web navigation TOIs associated with webpage screenshots Events are logged automatically, and full-page screenshots are captured automatically
Pop-up elements that do not generate a new URL Custom TOIs associated with video frames Events need to be logged manually and the frames manually chosen from the video
Embedded mp4 video content Not possible (not supported by the browser) Not visible in the video recording
Embedded flash video content Not possible (not supported by the browser) Not visible in the video recording

Related Articles