As I write this post, the coronavirus continues to spread across the world. In response, governments have put in place recommendations to self-isolate, create social (physical) distancing, or imposed flat-out lockdowns. One obvious implication for psychological researchers is that we can no longer conduct experiments face-to-face in our labs. Many of us have therefore been migrating our experiments online.
Last week I put an experiment online and recruited 800 participants from a popular online work-source site. All 800 signed up in under 40 minutes! Because of the nature of some of my research in memory and eyewitness identification, I have been conducting online experiments for nearly a decade and the speed at which recruitment and data collection takes place still impresses me. I never would have anticipated that this experience would come in so handy during a pandemic crisis (admittedly, I never gave this type of crisis much consideration at all). Many researchers have experience collecting data online, but many have not.
How to move from the physical lab to the online lab is the very timely topic of the recent paper by Grootswagers (pictured below) to be published in Behavior Research Methods.
In the paper, the noted benefits of collecting data online compared to in the lab include
- Being able to recruit from a pool of tens of thousands of participants
- Having the ability to recruit from a diverse population
- Saving time recruiting and scheduling
Here is the recipe
Prep time: depends on experience
Cook time: depends on the number of participants and type of experiment
Total time: see above
Servings: depends on the task, number of trials, precedence, and/or a power/sensitivity analysis
Calories: 0 kcal
Ingredients
- Browser-based experiment
- Host server
- Online recruitment marketplace
Instructions
- Program your experiment to run in a browser. Participants complete the experiment within a browser, the experiment needs to be programmed in a language compatible for use in web-browsers. For example, JavaScript is commonly used. There are opensource experiment builders such as PsychoJS and OpenSesame (the Behavior Research Methods article on OpenSesame is here) and commercial and very easy to use experiment builders such as Gorilla, Inquisit, or Testable. There are pros and cons to each method and what you decide to use may largely depend on your level of programming proficiency.
- Host the experiment on a server. The server sends the experiment code to the participants’ browsers and stores the data. This server could be a simple machine located in your office, in the cloud (using Google or Amazon services), or a hosting service such as Gorilla, Inquisit, or Pavlovia. Whichever you choose to use, be sure that the participant information is securely protected.
- Recruit participants. You can use SONA (commonly used in universities), Amazon Mechanical Turk (MTurk) or Prolific. In addition to the payment to participants, there are fees for the latter two. All you have to do is put money into your lab’s account, link to your experiment’s web address, and let the data accumulate.
- Download data. Once your desired sample size is reached, download the data from the server and voila! It’s time to analyze.
The figure below shows the infrastructure model for online experiments.
Recipe notes
Grootswagers answered frequently asked questions, including
Q: Are the data good?
A: Yes. They are better if the experiments are brief, fun, and the instructions are easy to understand.
Q: Is the timing good?
A: Not always as good as in the lab.
Some recipes tips from me
- Test the online experiment a lot and on as many different browsers as possible before releasing it to participants.
- Run a trial with a small number of participants first.
- Decide on your number of participants upfront and resist the urge to peak and stop collecting data (no opening the oven when the soufflé is in).
Final notes
Stay home.
Wash your hands.
Collect data online.
Psychonomic Society article focused on in this post:
Grootswagers, T. (2020). A primer on running human behavioral experiments online. Behavior Research Methods. Preprint.
