We all dream of exclaiming Archimedes’ famous quote at the end of an experiment- to express the wonder and satisfaction of discovering something exciting or answering a vital question. To reach this eureka moment when studying behavior, the process calls for an intricate set of steps from start to the finish line. Every researcher should be familiar with the methods involved in behavioral study design to ensure that a modern-day eye tracking study results in robust data, and ultimately, that big eureka moment. This article will show you the key cornerstones of amazing eye tracking studies.
In the beginning, there is a question. This simple remark is crucial to achieve robust results, no matter how big or small the study. After figuring out the question, preliminary results and observations are used to get an idea of the big picture, what is known already, and which aspects still need to be revealed. Based on these observations, a hypothesis is formulated.
As hypotheses are uncovered, they are used to make testable predictions to narrow down possible interpretations of the results and to pigeonhole competing hypotheses. After all this theorizing, it is then finally time to identify behavioral variables for testing. For every testable variable, a suitable recording method is chosen, and, finally, the data collection is launched!
In eye tracking and behavioral research, it is essential to start out right. Hypotheses and pilot studies are tools to ensure a study has a solid start and a successful conclusion.
A hypothesis is an explanation for an observed problem or phenomenon based on previous knowledge or observations. Often called a research question, a hypothesis is basically an idea that must be put to the test.
Research questions should lead to clear, testable predictions. The more specific these predictions are, the easier it is to reduce the number of ways in which the results could be explained. Some problems require a fair amount of information and knowledge before one can formulate useful hypotheses, particularly if the problems are complex in nature.
When designing a study, especially an eye tracking study, it is vital to start out with a sound hypothesis. This was famously demonstrated in “The Hitchhiker’s Guide to the Galaxy,” where 42 was discovered to be the answer to life, the universe, and everything, but the question was unknown. After all, if we don’t know what questions we are asking, how can we find the answers?
Eye tracking studies can deliver a vast amount of data, making it even more important to know and understand the metrics necessary to answer the research question. Another big danger when using eye tracking is to choose the wrong recording method for the behavior you want to study. Choosing metrics, research questions, and answers after a study is finished is not only unscientific, but it might also lead to the realization that the question cannot be answered at all with the data collected. For a successful eye tracking study, the hypothesis, as well as the data and metrics required to accept or reject the hypothesis, must be defined at the very beginning of the study design.
Examples of hypotheses for eye tracking studies are:
Pilot studies are small-scale versions of the main study that are performed to assess the feasibility of a study design. Full-scale studies require a lot of time and money: planning the study, recruiting participants, running the study, analyzing the data, evaluating the results, and compiling the report.
Sometimes, it is impossible to repeat an entire study due to any number of factors, such as involving a special group of participants (i.e. patients with a rare condition), limited time available in the study environment, or project deadlines. A pilot study can uncover any pitfalls, blunders, and bottlenecks, from study design to data analysis, early in the project. A carefully performed and analyzed pilot study can save the investigators time, money, and heartache.
Furthermore, including a pilot study in a project is a crucial aspect of good scientific practice. From UX to neuroscience, marketing to psychology, and reading studies to assessing the behavior of a Formula 1 driver, scientific rigor is essential. Reliable results, good data quality, and publishable discoveries can only be achieved through carefully designed studies, from the beginning to the finish line.
For a pilot study to provide valuable insights, it is essential to run it in its entirety. In an eye tracking pilot study, the data collection process isn’t the only part that is relevant- the analysis of the collected data must be assessed as well. It is not enough to simply export the data in Excel and call it a day, as this might disguise omissions and mistakes that could be harmful for the full-scale study. For instance, gaps in the data, missing metrics, and inappropriate statistical methods can only be detected when the pilot study is completed.
When trying to secure funding or an investment, a well-performed and properly presented pilot study can be the key to win over decision-makers. Data collected in a pilot study can give funding agencies and investors a better understanding of why a project is important and worth funding. A pilot study carried with precision demonstrates the skills, integrity, and diligence of the investigators.
After wrapping up a pilot study and moving on to the real thing, it is important to start with a clean slate. This means that neither the data collected nor the participants used in the pilot study should be recycled in the full-scale study.
Some of the pitfalls in an eye tracking study that can be identified in a pilot study are, for example: if the number of participants is sufficient, whether the target population tends to have droopy eyelids, the lighting in the supermarket interferes with the recordings, the software used is not suitable for the website being studied, the safe and correct set up of equipment at the study site, if the questions on the questionnaire are comprehensible, whether the collected data contain the metrics needed to answer the main research question, or if the investigators have the means to analyze the collected data.
Points to clarify in a pilot study: