Stress, what stress? It’s only online research

How stressed do you feel right now? To find out, take this assessment. I scored an 8, which puts me in the “you are stressed” category. I am not too surprised with my result given my own midlife status with two emerging adult children and too many responsibilities while attempting to return to “normal” in the midst of a pandemic that hasn’t quite progressed to an endemic status.

Covid-19 amplified stress around the world. While the global lockdowns allowed some people to slow down for a while (and reduce our human impact on the environment), most people reported experiencing high levels of stress. Many stressors included worrying about the virus itself, concern about family members, mitigation options (e.g., vaccines, masks, lockdowns), the economy, finances, technology, educational regression, spatial overlap between family life and work, and so on. Although much of the world is currently open and people are trying to return to business as usual, Covid-19 still has a grip on our lives.

No Reason to Stress!?

When the world started shutting down in early 2020, much of the ongoing research was also halted. Researchers who relied on people to conduct their studies found themselves at a crossroads – shift gears to study some other topic or take a hiatus from research. Despite the extra stress of the unknown, most researchers met these challenges with ingenuity and creative problem-solving. As academic classes shifted to online modalities, many researchers also shifted their platforms to conduct remote and online research projects associated with questions about the effects of the pandemic on almost every aspect of life – work-life balance, relationships, compliance with mandates, education, health, pets, technology, binge-watching, personality, etc. Some of these topics were highlighted in various posts by the Psychonomic Society Digital Editorial Staff and a recent podcast.

Mohammed Almazrouei, and his colleagues, Ruth Morgan and Itiel Dror (pictured below) were spurred to shift their research paradigms to an online forum.

Authors of the featured article. From left to right, Mohammed Almazrouei, Ruth Morgan, and Itiel Dror.

In a paper recently published in the Psychonomic Society’s journal Behavior Research Methods, Almazrouei, Morgan, and Dror describe an experimental method used to induce stress in people participating in an online research environment.

Time’s Up!

Previous research involving stress tended to be conducted in person with most procedures manipulating some kind of stressful event experienced by participants, such as seeing emotionally disturbing images, experiencing uncomfortable physical conditions like ice baths, being forced to complete difficult/impossible tasks within a time limit, or being judged or critiqued by another. Working from a previously published meta-analysis in which social-evaluative threats (e.g., negative feedback on one’s performance) and uncontrollability (can’t avoid negative consequences) were the two most powerful manipulations that elicited stress responses reliably. Armed with this knowledge, Almazrouei and his colleagues developed a task that could induce stress within an online setting that allowed a researcher to be present virtually or not.

In their task, the researchers combined uncontrollable events with social-evaluative threats in an online study. Their research design is pictured in the graphic below.

Experimental timeline and order of activities and surveys.

Participants had to try to complete very difficult math or general knowledge questions in a very short time frame, such as 3-9 sec, depending on the question. Below are a couple of examples of “tricky” math problems.

Genre Question Answer Source
‘Tricky’ Math

Questions

Q.1. A pen and pad cost one dollar and ten cents. The pen costs one dollar more than the pad. How much does the pad cost (in cents)? (please type the number only). 5 cents (Kahneman, 2011)

 

Q.5 A person was born on May 6, 30 B.C. He died on May 6, 30 A.D. How old was he when he died? (please type the number only). 59 years (Gardner, 1986)

Some participants received motivational messages both during and after the various questions while other participants received negative feedback, and control participants did not receive any feedback on their performance. Examples of the messages are in the table below.

Feedback Stress Group Control Group
Before Stress Stimuli Block Levels of stress as measured by state anxiety (left panel) and visual analog scale (VAS, right panel! PLEASE READ CAREFULLY:

  • You are reminded that your performance is being monitored by Prolific Academic.
  • You are reminded that there is a required minimum performance when answering the questions. Your individual performance will be compared with the rest of the participants to determine whether your data will be used in the study.
  • Note that some questions have a limited amount of time for you to answer them, whereas others don’t. The questions with the limited amount time will have a clock showing the remaining time.

 

Participants click on the following message to continue the study: “I understand that my performance is being monitored.”

“Note that your performance is NOT being evaluated, so please attempt to answer the questions as accurately as possible.”

 

Participants click on the following message to continue the study: “I understand that my performance is NOT being monitored.”

During Stress Stimuli Block WRONG! for incorrect answer.

 

TIME OUT! when the allocated time runs out.

 

OK for correct answers.

No feedback given.
After Stress Stimuli Block If the participant scores three correct answers or lower, the following message will appear:
Unfortunately, your individual score for the math and general knowledge questions you have just completed was lower than, or did not supersede, the average performance of participants.If the participant scores four correct answers or higher, the following message will appear:
OK, you have completed this block of math and general knowledge questions, and you can now proceed to the next step. 
No feedback given.
Levels of stress as measured by state anxiety (top panel) and visual analog scale (VAS, bottom panel).

In the figures above, the stress group showed higher levels of stress on average for both the trait anxiety measure (top panel) and the visual analog measure (bottom panel). Participants categorized as having high trait anxiety showed the highest levels of stress across all both conditions and on both measures (shown by the red points and 95% CI lines in both panels).

The researchers concluded that these patterns of results indicated that this procedure within an online platform produced valid online stress experiences that were measurable.  And, as the authors stated in a written interview, “It is important to understand how stress impacts decision-making because we often make important personal and professional decisions under stress. We now have a way to study these decisions with relative ease and with access to a wider and more diverse pool of participants.” To me, as a “sometimes” human researcher and a “more times” animal researcher, the more interesting component to this study were the controls that were implemented to ensure the validity of this procedure and its results.

Bots, bots, bots

When conducting experimental research, establishing true controls, experimental manipulations, manipulation checks, and quality control measures are challenging, but manageable, for in-person research studies because the participants are physically present and can be monitored directly. However, research conducted in online platforms asynchronously is more challenging in many ways, mainly because researchers not only have to ensure that participants are responding honestly, but that the participants are actually humans, and not bots.

Two recent studies discussed many of the difficulties associated with collecting data from an online forum or survey platform like MTurk, Prolific Academic, or Qualtrics. To address these issues, online research must implement attention checks, bot detectors, and manipulation checks without compromising the study. Almazrouei and his colleagues included several additional questions to achieve the best quality data. They included attention checks, bot checks, and filtered the acquired data for survey completion duration and patterns in responses. Examples of their exclusion criteria are found in the table below.

Attention check ·      The participant fails more than one attention check question. Evidence suggests that elimination of data based on a single screener may result in bias in research (Berinsky et al., 2014).
Length of time ·      Consistently very high response times. That is, response times are repetitively much higher than those of other participants across multiple mathematical/general knowledge questions. Such response behaviors raise concerns that that they search for answers on the internet. For instance, Clifford and Jerit (2016) found that self-reported cheaters spent significantly more time in answering the general knowledge questions than non-cheaters.

·      Consistently very low response times. This response behavior raises concerns that participants randomly answer the questions, without cognitive effort (Börger, 2016).

Pattern of subjects’ responses ·      High number of correct answers on difficult questions (i.e., more than just chance).

·      Consistently choosing the same responses. For example, choosing the same option in multiple choice questions (Clemmow et al., 2020).

True or False?

As described earlier, online research must be able to differentiate valid data from false or artificial data. Thus, if a participant failed to follow the directions in the attention check question – “What color is the sky? Please select “purple” as an answer to make sure you are paying attention.”, then that participant was excluded from the final data set. The authors also included visual images of the various tasks and questions because they make it harder for participants to cheat or bots to answer the question reasonably.

As an animal researcher who occasionally enters into the human world of research, I appreciate the relative efficiency of conducting survey-based research, especially if it involves experimental manipulations. In my own experience, my online human study took 6 weeks to collect data while my animal research typically takes 1-2 years with a fast project. Of course, getting it out for publication takes the same amount of time, but I digress.

As Covid-19 has taught us, transitioning in-person activities to online experiences is the wave of the future, and while survey-based research may be faster to collect than other types of studies. I think I’ll stick with my animal research. At least, I only have to be as smart as a beluga or killer whale and not smarter than artificial intelligence or other humans!

Featured Psychonomic Society article

Almazrouei, M. A., Morgan, R. M., & Dror, I. E. (2022). A method to induce stress in human subjects in online research environments. Behavior Research Methods. https://doi.org/10.3758/s13428-022-01915-3

The Psychonomic Society (Society) is providing information in the Featured Content section of its website as a benefit and service in furtherance of the Society’s nonprofit and tax-exempt status. The Society does not exert editorial control over such materials, and any opinions expressed in the Featured Content articles are solely those of the individual authors and do not necessarily reflect the opinions or policies of the Society. The Society does not guarantee the accuracy of the content contained in the Featured Content portion of the website and specifically disclaims any and all liability for any claims or damages that result from reliance on such content by third parties.

You may also like