The importance of hypotheses and pilot studies

Eye tracking Design Cognitive psychology

We all dream of exclaiming Archimedes’ famous quote at the end of an experiment- to express the wonder and satisfaction of discovering something exciting or answering a vital question. To reach this eureka moment when studying behavior, the process calls for an intricate set of steps from start to the finish line. Every researcher should be familiar with the methods involved in behavioral study design to ensure that a modern-day eye tracking study results in robust data, and ultimately, that big eureka moment. This article will show you the key cornerstones of amazing eye tracking studies.

In the beginning, there is a question. This simple remark is crucial to achieve robust results, no matter how big or small the study. After figuring out the question, preliminary results and observations are used to get an idea of the big picture, what is known already, and which aspects still need to be revealed. Based on these observations, a hypothesis is formulated.

As hypotheses are uncovered, they are used to make testable predictions to narrow down possible interpretations of the results and to pigeonhole competing hypotheses. After all this theorizing, it is then finally time to identify behavioral variables for testing. For every testable variable, a suitable recording method is chosen, and, finally, the data collection is launched!

In eye tracking and behavioral research, it is essential to start out right. Hypotheses and pilot studies are tools to ensure a study has a solid start and a successful conclusion.

What is a hypothesis?

A hypothesis is an explanation for an observed problem or phenomenon based on previous knowledge or observations. Often called a research question, a hypothesis is basically an idea that must be put to the test.

Research questions should lead to clear, testable predictions. The more specific these predictions are, the easier it is to reduce the number of ways in which the results could be explained. Some problems require a fair amount of information and knowledge before one can formulate useful hypotheses, particularly if the problems are complex in nature.

When designing a study, especially an eye tracking study, it is vital to start out with a sound hypothesis. This was famously demonstrated in “The Hitchhiker’s Guide to the Galaxy,” where 42 was discovered to be the answer to life, the universe, and everything, but the question was unknown. After all, if we don’t know what questions we are asking, how can we find the answers?

Eye tracking studies can deliver a vast amount of data, making it even more important to know and understand the metrics necessary to answer the research question. Another big danger when using eye tracking is to choose the wrong recording method for the behavior you want to study. Choosing metrics, research questions, and answers after a study is finished is not only unscientific, but it might also lead to the realization that the question cannot be answered at all with the data collected. For a successful eye tracking study, the hypothesis, as well as the data and metrics required to accept or reject the hypothesis, must be defined at the very beginning of the study design.

Examples of hypotheses for eye tracking studies are:

  • My ad design team has produced two versions of a beverage advertisement. I think version A is better because it contains a couple of visual elements that will drive the consumer’s attention to the element containing the main message of the ad. For this, using eye tracking, you can perform an A/B test of the two ad versions. Then, you will analyze which elements they do look at and for how long before they reach the actual element containing the message.
  • Students in the 5th and 7th grades have the same silent reading skills. In this scenario, you would test a group of fifth-graders versus a group of seventh-graders reading the same text. To get an overall idea of their skills, you can compare reading speeds to analyze their fluency, and eye movement measures will further help you understand in which part of the text readers stall and/or go back. Content-related questions will help you assess their comprehension of the text.
  • Experienced air traffic controllers are more effective at scanning the airplanes on the runway and in the sky. To run this test, the air traffic controllers would be divided into two groups: experienced and novices. Eye movement measures and patterns would be analyzed to investigate where the traffic controllers look when several planes are deployed in their simulator.


What is a pilot study?

Pilot studies are small-scale versions of the main study that are performed to assess the feasibility of a study design. Full-scale studies require a lot of time and money: planning the study, recruiting participants, running the study, analyzing the data, evaluating the results, and compiling the report.

Sometimes, it is impossible to repeat an entire study due to any number of factors, such as involving a special group of participants (i.e. patients with a rare condition), limited time available in the study environment, or project deadlines. A pilot study can uncover any pitfalls, blunders, and bottlenecks, from study design to data analysis, early in the project. A carefully performed and analyzed pilot study can save the investigators time, money, and heartache.

Furthermore, including a pilot study in a project is a crucial aspect of good scientific practice. From UX to neuroscience, marketing to psychology, and reading studies to assessing the behavior of a Formula 1 driver, scientific rigor is essential. Reliable results, good data quality, and publishable discoveries can only be achieved through carefully designed studies, from the beginning to the finish line.

For a pilot study to provide valuable insights, it is essential to run it in its entirety. In an eye tracking pilot study, the data collection process isn’t the only part that is relevant- the analysis of the collected data must be assessed as well. It is not enough to simply export the data in Excel and call it a day, as this might disguise omissions and mistakes that could be harmful for the full-scale study. For instance, gaps in the data, missing metrics, and inappropriate statistical methods can only be detected when the pilot study is completed.

When trying to secure funding or an investment, a well-performed and properly presented pilot study can be the key to win over decision-makers. Data collected in a pilot study can give funding agencies and investors a better understanding of why a project is important and worth funding. A pilot study carried with precision demonstrates the skills, integrity, and diligence of the investigators.

After wrapping up a pilot study and moving on to the real thing, it is important to start with a clean slate. This means that neither the data collected nor the participants used in the pilot study should be recycled in the full-scale study.

Some of the pitfalls in an eye tracking study that can be identified in a pilot study are, for example: if the number of participants is sufficient, whether the target population tends to have droopy eyelids, the lighting in the supermarket interferes with the recordings, the software used is not suitable for the website being studied, the safe and correct set up of equipment at the study site, if the questions on the questionnaire are comprehensible, whether the collected data contain the metrics needed to answer the main research question, or if the investigators have the means to analyze the collected data.

Points to clarify in a pilot study:

  • What is my hypothesis?
  • Will I obtain the metrics needed to accept or reject my hypothesis?
  • Does my equipment fulfill the requirements to complete my study?
  • Is my schedule feasible?
  • Can I recruit enough participants in time?
  • Are the tasks and questionnaires understandable?
  • Are the lengths of my tasks suitable for my target population?
  • Does my environment allow for a successful study?
  • Are the legal requirements fulfilled and consent forms clear?
  • Can I fulfill the technical specifications required for my equipment?
  • Are my investigators sufficiently prepared and educated?
  • Do I have the knowledge or workforce to analyze my data?
  • Does the amount of data collected have statistical validity?

Recommended Reading

  • Paul Martin and Patrick Bateson: Measuring behavior: An introductory guide. Cambridge University Press, Cambridge, England, 1993, Second Edition, 222 pages, ISBN 0521 446147
  • Lancaster, G. A., Dodd, S. and Williamson, P. R. (2004), Design and analysis of pilot studies: recommendations for good practice. Journal of Evaluation in Clinical Practice, 10: 307–312. doi:10.1111/j..2002.384.doc.x
  • van Teijlingen E., Hundley V. (2002), The importance of pilot studies. Nursing Standard, 16, 40,33-36. doi: 10.7748/ns2002.
  • Doody, O., and Doody, C.M., (2015), Conducting a pilot study: case study of a novice researcher. British Journal of Nursing, 24:21, 1074-1078. doi: 10.12968/bjon.2015.24.21.1074
  • Thabane, L., Ma, J., Chu, R., Cheng, J., Ismaila, A., Rios, L.P., Robson, R., Thabane, M., Giangregorio, L. and Goldsmith, C.H. (2010), A tutorial on pilot studies: the what, why and how. 10: 1. doi: 10.1186/1471-2288-10-1

Related Articles