It has been pointed out that the scientific literature on human cognition and behavior is based primarily on data from Western, Educated, Industrialized, Rich, and Democratic (WEIRD) societies. This situation is unlikely to change, given that most researchers do not have the training or the funding necessary to study non-WEIRD populations.

In addition, some researchers seem to assume—at least tacitly—that White upper-middle-class monolingual English-speaking students at universities with highly selective admission standards are the group that provides the best evidence about the nature of human cognition. This assumption is problematic: Having a diverse or inclusive research sample does not compromise the rigor or quality of the science, and it does not require “validation” of the results with majority-group “controls.” Quite on the contrary, broadening participation provides more exhaustive tests of the generalizability of our theories. Fortunately, even when funding limitations constrain us to rely on university student samples as the primary source of data, we can improve the diversity and inclusiveness of these college samples.

Samples of Convenience. While cognitive psychologists acknowledge to a certain extent that data collected from samples of convenience (e.g., introductory psychology students) will not necessarily generalize to a broader population of adults, we do not take this concern as seriously as researchers in more applied areas of psychology. Even as we continue to collect data from convenience samples, we need to acknowledge the limitations of this approach and take advantage of diversity across convenience samples from different sources.

Samples of convenience, including samples of introductory psychology students, vary considerably across institutions. Student participant populations differ across institutions in terms of their racial/ethnic makeup, their socioeconomic status, their age, and even their cognitive ability. Specifically, while some institutions have a predominantly non-Hispanic or White student population, others have student populations that are predominantly Black, Latino, or Native American, and others have student populations with a diverse representation of racial/ethnic groups. Similarly, while some institutions have students who come from families with higher socioeconomic status, other institutions have students with lower socioeconomic status. Some institutions have selective admission standards and filter prospective students based on high school GPAs and standardized test scores. Other institutions have open undergraduate admissions and provide opportunities for any student who wishes to attempt college. Across these levels of selection, the average level of student cognitive skill varies, because students with average or below-average high school GPAs will not be admitted to highly selective institutions. Based on these considerable student population differences, the sample that is convenient for a researcher differs from one institution to another.

Implications for Internal Validity. If the goal of our research is to answer basic questions about cognition, an upper-middle-class White non-Hispanic sample at a highly selective institution does not provide data with greater internal validity than data from, say, a lower-middle-class minority-majority sample at an open-admission institution if the experimental design is the same.

Internal validity, the validity of cause-effect conclusions, comes from a sound design, not from the participant population sampled. Accordingly, when a well-designed experiment is conducted with an inclusive sample at an HBCU or HSI, the researchers should not be expected to collect data from a “White control group” to confirm the validity of the findings. The notion that people who are White, rich, highly educated, native English speaking, or monolingual are the norm, control, or standard against which all others must be compared is fundamentally flawed. Incidentally, none of these classifications comprises a majority of the population of human adults, and the conjunction of these classifications is even smaller.

Implications for External Validity. External validity, the generalizability of the results from university samples may be greatest at a college or university with minimal admission criteria and great socioeconomic and racial/ethnic diversity. However, most of us do not work at such institutions or have access to students who attend them. It may be too much to expect all cognitive psychologists to travel or collaborate to carry out research with diverse samples.

Instead, what I suggest is that we do not punish researchers who work with samples that are diverse or inclusive of participants with minority racial/ethnic status, lower socio-economic status, or lower academic achievement. Unfortunately, reviewers sometimes regard such research as inferior and recommend the inclusion of comparison groups from more conventional populations, as will be explained in the following section.

Questionable Comments About Diverse or Inclusive Participant Samples

I have observed a disturbing prevalence of inappropriate or biased comments related to diversity and/or inclusion in manuscript reviews, and in my opinion, editors have some responsibility to prevent them or filter them, in part because of the harmful effect on junior scientists who come from the groups about which the comments are made.

Examples of Questionable Comments. In the convenient university student population where I conduct research, most students (over 80%) report Latino/Hispanic ethnicity, primarily Mexican American, so these samples could be characterized as inclusive but not particularly diverse in terms of ethnicity. However, this student population is diverse in terms of socioeconomic status and English and Spanish language skills. Most of my student research assistants and coauthors come from this local population. The same is true for my departmental colleagues. Here are a few of the comments that we have had to share with our students about our research samples in reviews of manuscripts and grant proposals over the last two years. Some comments reflect views that White non-Hispanic samples are normative:

• … I had problems with the inclusion of only Latino participants.
• Only having Latino participants… seriously limits the contribution of this paper.
• Shouldn’t there be a comparable group with no Spanish background and no Spanish training?
• Some of the specific hypotheses might be easier to test with homogeneous rather than heterogeneous samples.

Other comments reflect a stereotypic view of Latino language skills:

• A majority of the monolinguals are Latino. Why aren’t they bilinguals?
• It seems maybe these are people who fail to acquire a heritage language even though there is environmental support for that and maybe a common underlying cause could also explain why their memory is worse than that of bilinguals.
• Presumably, they come from Spanish-speaking households.

Other comments suggest that the participants are not intelligent:

• The standardized vocabulary test scores seem very low for a college sample.
• Naming latencies are so slow that I am wondering what the precise instructions were.

Of course, some of these comments could be classified into more than one of these categories.

Impact of Inappropriate or Biased Comments. The comments above at best reflect a lack of knowledge about the local population that I have worked with for almost 20 years, and there may be no negative intentions behind them. Nevertheless, my Latina/o student co-authors and I find these sorts of comments rather insulting, because they imply numerous assumptions about the populations tested and our own research practices that are not supported by the facts and suggest that the population from which we collect data is not qualified to attend college or inform the science of human cognition.

When my students make the effort needed to earn authorship on a scientific manuscript, they do not ask to have their ethnic group, language skills, socioeconomic status, or home environment subjected to untrue assumptions or insulted. Such comments give them the idea that people like them are not respected in the scientific community as valid research participants, let alone valid researchers. When my students read these comments, they are discouraged from pursuing careers in scientific research. Although the prevalence of such comments in other researchers’ manuscript reviews is unknown, I wonder how many students we may be losing who have a lot to contribute as scientists and would add to the diversity of the scientific community.

Editorial Discretion. It is important for editors to take measures to prevent such comments from being made or prevent them from being transmitted to authors and other reviewers. Editors may be able to minimize the prevalence of such comments through instructions to reviewers. When such comments are nevertheless made, they might simply ask reviewers to reconsider, reword, or delete them or ask the reviewer to allow the editor to modify or delete inappropriate or potentially offensive comments. On what I expect would be the rare occasion that the reviewer would be unwilling to make the change or have the change made by the editor, an editor could consider disregarding the entire review.

Reporting Relevant Demographic Information in Cognitive Research

We have very little information on the racial/ethnic composition of research samples in the cognitive literature, because most cognitive scientists neither collect nor report these variables. Although some funding agencies require researchers to project ethnic/racial compositions of participant samples and collect this information if funded, there is no accountability attached, and the researchers are not expected to report this information in manuscripts about the research.

Current Expectations that Demographic Information Will Not Be Reported. When the results of cognitive research are reported in manuscripts submitted for publication, journal editors and reviewers do not expect or encourage authors to include information on race/ethnicity or SES of their samples. Indeed, following a long tradition, most researchers do not even collect this information much less report it. When these variables are reported, reviewers sometimes comment that it is unnecessary to include them or use them as a basis to criticize the sampling procedure. Why should we have to hide diversity or inclusion in our samples as if they were design flaws in order to get funding or publish more easily?  Does it serve our science if researchers hide this information for fear that the manuscript will be rejected?

NIH Requirement to Project and Report Demographic Information. For many years, the NIH has required grant proposals to include projections about the racial/ethnic composition of human research samples. Funded investigators are required to collect this information from research participants and report summaries back to the NIH in final (but not interim) progress reports. Despite this mandate, in the 15 cognitively oriented panels I have served on, I cannot remember a single instance in which a reviewer other than myself ever suggested that a projected racial/ethnic distribution was unacceptable or not scientifically justified. As far as I know, the final report is not compared to the original projections, nor is this report provided to reviewers of future proposals. Also, because interim reports do not require this information, investigators cannot be held accountable before all funds are expended.

Why We Should Collect and Report Demographic Information. Should we collect and report this demographic information anyway? In accordance with the scientific method and transparency in reporting, I am starting to think that given the variability across participant samples from different institutions, greater characterization of participant samples in manuscripts is warranted. This information would be provided not so much for evaluative purposes, but so that other scientists who read the article can know (a) how well the phenomenon under study generalizes across different populations and (b) how the sample in one study might differ in cognitively relevant ways from the sample in another study that addresses similar questions but yields different results, possibly accounting for some failures to replicate.

If we truly believe that the race/ethnicity and SES of research participants are irrelevant to cognitive function, then maybe we should instruct reviewers not to make references to participants’ ethnicity or SES. On the other hand, if we believe that these factors do impact cognitive function (for a review, see this BBS target article by Henrich and colleagues), whether directly or indirectly, then should we not require all cognitive researchers to collect and report the information? After all, it only adds a few minutes to an experimental session. In the absence of transparent reporting, we may be prone to making assumptions about the racial/ethnic distribution, socio-economic status, and cognitive skills of the samples. At present we cannot verify or discredit these assumptions, because the data are not available to do so.