Aggregating eye tracking data across several participants in web recordings

Figure 1. Shows how different visits to a webpage are aggregated into the same web navigation TOI.

Next, you will learn the step by step procedure to create an aggregated data visualization and extract eye tracking metrics based on the web navigation TOIs.

To aggregate the data on a heat map:

1. Click on the Visualizations button, either in the Project Overview section or from the Analyze drop-down menu in the top navigation.
2. Go to the Visualization Type and Settings tool in the tool panel, to the right of the Visualization interface and select Heat Map.
3. Select what Gaze Filter to use when generating your heat map. Read more about Gaze Filters in the “Appendix B Gaze Filters’ functions and effects” in the Pro Lab user manual.
4. Go to the Times of Interest tool and select the web navigation TOI that corresponds to the webpage you want to use as the base of your Visualization. To help you select the TOI it displays the name and a thumbnail of the webpage.
5. Go to the Recordings tool and select or deselect the recordings you wish to aggregate in the heat map.
6. Select the heat map type - absolute count, relative count, absolute duration or relative duration. (for more information about the different types of heat maps, read the appendix “Appendix C Calculating heat maps” in the Pro Lab user manual.)
7. The heat map is now generated and displays the aggregated data of multiple recordings.

To extract aggregated eye tracking measures:

1. Create your AOIs, by opening an AOI tab
2. Select the target webpage screenshot in the Media Selection section
3. Draw your AOIs on the elements of interest to your study
4. If you want to aggregate data across multiple AOIs you can also create tags and tag each AOI.
5. Open a Metrics Export tab.
6. Select the output format for the export file by clicking the Export format drop-down list box in the Settings panel. Select Interval based TSV file (default) .tsv format to segment your data based on each visit to a webpage or the Excel report for the .xlsx format for a summary report.
7. Select the metrics you want to include in the export (the selection varies depending on what you chose in step 2).
8. Select which Gaze Filter you want to apply to the data you will export. Read more about Gaze Filters in the “Appendix B Gaze Filters’ functions and effects” in the Pro Lab user manual.
9. Select the recordings. Choose to export metrics from All Recordings or only the recordings of your choice.
10. Select your Participant variables to further segment your data according to participant variables.
11. Select your web navigation TOIs.
12. Select the AOIs and AOI Tags created in steps 1-4.
13. If relevant select Web Navigation Event types, to export event related metrics.
14. Click Export to export the data.

If your study requires you to aggregate data to webpages and there are few or no dynamic elements, you can use the web navigation TOIs and associated screenshots to aggregate your data. However, in situations where you expect to encounter dynamic elements, you will run into three challenges:

1. Some dynamic elements are not supported by the browser**
2. Web screenshots will not display certain dynamic web elements***
3. All the gaze data captured during the webpage visit is mapped to the screenshot irrespective if the user spent some periods interacting and viewing dynamic elements

** E.g. Flash elements are not currently supported by the Pro Lab Browser.

***The only exception is for fixed elements, Tobii Pro Lab is able to track these elements and map the data correctly. Fixed elements are elements in a website that are positioned relative to the browser window instead of the page itself. When the test participants scroll down on a website with fixed elements, those elements stay in their place while the rest of the content moves. These elements will be given a fixed position in the screenshot. Any gaze on them will be mapped to their position of the screenshot regardless of where the element was positioned when it was actually viewed. This will give a more true representation of the fixations. 

The listed challenges above imply that you won’t be able to rely on the web navigation TOIs generated by the system, as there will be portions of gaze data that will be aggregated to the automatic web screenshot that is not directed at the elements displayed by the screenshot. Instead, they belong to the participant’s interaction with dynamic elements. For example, if the participants access a drop-down menu on the website the viewing patterns on that menu will be recorded and aggregated on the screenshot of the webpage, but the menu itself won’t be visible on that screenshot. This situation is true for other dynamic elements, such as pop-ups that do not generate a unique URL or elements that move across the screen.

As previously mentioned, during the test recording, Pro Lab also captures a viewport video of the whole browsing session. This video can be viewed later when you replay the recording. Since this video is essentially a screen recording of the participant's browsing activity, you will be able to view the different dynamic elements and the participant viewing behavior and interaction. During your data analysis, you can use this video to manually create events on the timeline that mark the start and end of the interaction with those elements. Once you created these events you can further use them to generate custom TOIs with a video frame. The custom TOI aggregates the gaze data that corresponds to the viewing activity towards the dynamic element, and the video frame is used to illustrate the webpage with the dynamic element visible. By combining the custom TOI and frame you can aggregate data from different recordings to produce visualizations and create AOIs to extract eye tracking metrics.

In the previous example where the participant interacts with a drop-down menu, you would need to log two events, e.g. drop-down open and drop-down close, and create a custom TOI with a frame using these events as the start and end point of the TOI. The frame needs to be extracted from a time in the video when the drop-down menu is open. I would repeat this process in the remaining recordings, and e.g. create a heatmap by selecting the custom TOI in the Visualization tools. This heat map shows the aggregated viewing activity towards the drop-down menu. Below you will find step by step instructions on how to create a custom TOI with a frame.

To create a custom TOI:

1. Open a recording
2. Create one or two user-generated events to define the start and end points of the TOI, e.g. every time the user opens a drop-down menu opens and closes
3. Move the cursor to a video frame where the menu is open, then click Create frame icon located to the right of the recording time counter above the timeline and select Create Toi with frame. The frame will have the same function as the web screenshot described in the previous section
4. On the Create custom Time of Interest window select the start and end point events created in step 1
5. Rename the Media to a more descriptive name
6. Click OK to create the TOI.
7. Repeat steps 1 and 2 for the other recordings

After you create your Custom TOIs with an associated frame, you can use these TOIs in the same way as the web navigation TOIs to produce visualizations and extract metrics based on aggregated data of several participants.

A limitation of this analysis workflow is that the dynamic element needs to fit within the viewport. If the participant needs to scroll to interact with the element then this solution won't work, as you will not be able to summarize the viewing behavior on a single video frame.

In some cases, certain websites may contain elements that remain constant when the participant visits different pages (i.e. these elements remain in the same location and display the same content). If these elements are relevant to the task that spans accross these pages the researcher may be interested in aggregating the data towards these elements across all the pages they are present. This can be done in Pro Lab by using web navigation custom TOIs. Where the researcher uses an automatically generated screenshot from one page where that element is present, to map data from the other pages.  

To create a web navigation custom TOI:

1. Open a Recording tab.
2. On the Times of Interest panel, look for the Web navigation custom TOI section then click the "plus" icon to create a new TOI.
3. On the Edit custom time of interest window, give a name to the new web navigation custom TOI.
4. Select your Start and Stop points for your TOI by selecting the relevant URLStart and URLEnd from the different pages you wish to group and aggregate the data.
5. Next, choose a screenshot from the Select screenshot section, by clicking on it. 
6. Click OK when you are done.

After you create your web navigation custom TOIs, you can generate visualizations and extract metrics based on the aggregated data of several participants and the grouping of web pages.

Aggregating data across multiple test participants in web studies can be conceptually and technically challenging, particularly on today’s web pages, where dynamic elements are abundant.

Conceptually, it is challenging to compare data from several recordings when the dynamic content makes each interaction with a web page different for each participant. Technically, it is challenging to capture content that changes over time in a useful way to be used in visualizations of eye tracking data—especially if the content only changes on some parts of a page.

When preparing a study, you need to consider these challenges and assess the implications on the analysis workflow. This will allow you to have a more realistic estimate of the time and resources you need to dedicate to your data analysis procedure.