The process of replication target selection in psychology: what to consider?

Merle-Marie Pittelkow; Sarahanne M Field; Peder M Isager; Anna E Van't Veer; Thomas Anderson; Scott N Cole; Tomáš Dominik; Roger Giner-Sorolla; Sebahat Gok; Tom Heyman; Marc Jekel; Timothy J Luke; David B Mitchell; Rik Peels; Rosina Pendrous; Samuel Sarrazin; Jacob M Schauer; Eva Specker; Ulrich S Tran; Marek A Vranka; Jelte M Wicherts; Naoto Yoshimura; Rolf A Zwaan; Don van Ravenzwaaij

doi:10.1098/rsos.210586

The process of replication target selection in psychology: what to consider?

R Soc Open Sci. 2023 Feb 1;10(2):210586. doi: 10.1098/rsos.210586. eCollection 2023 Feb.

Authors

Merle-Marie Pittelkow¹, Sarahanne M Field², Peder M Isager³, Anna E Van't Veer⁴, Thomas Anderson⁵, Scott N Cole⁶, Tomáš Dominik⁷, Roger Giner-Sorolla⁸, Sebahat Gok⁹, Tom Heyman⁴, Marc Jekel¹⁰, Timothy J Luke¹¹, David B Mitchell¹², Rik Peels¹³, Rosina Pendrous^{14

15}, Samuel Sarrazin¹⁶, Jacob M Schauer¹⁷, Eva Specker¹⁸, Ulrich S Tran¹⁸, Marek A Vranka¹⁹, Jelte M Wicherts²⁰, Naoto Yoshimura^{21

22}, Rolf A Zwaan²³, Don van Ravenzwaaij¹

Affiliations

¹ Department of Psychometrics and Statistics, Rijksuniversiteit Groningen, Groningen, The Netherlands.
² Centre of Science and Technology Studies, Leiden University, Leiden, the Netherlands.
³ Oslo New University College, Oslo, Norway.
⁴ Methodology and Statistics Unit, Institute of Psychology, Leiden University, Leiden, the Netherlands.
⁵ Department of Psychology, University of Toronto, Toronto, Canada.
⁶ School of Education, Language and Psychology, York St John University, York, UK.
⁷ Institute for Interdisciplinary Brain and Behavioral Sciences, Chapman University, Orange, CA, USA.
⁸ School of Psychology, University of Kent, Canterbury, UK.
⁹ Program in Cognitive Science, Department of Instructional Systems Technology, Indiana University, Bloomington, IN, USA.
¹⁰ Social Psychology, University of Cologne, Cologne, Germany.
¹¹ Department of Psychology, University of Gothenburg, Gothenburg, Sweden.
¹² WellStar College of Health and Human Services, Kennesaw State University, Kennesaw, GA, USA.
¹³ Philosophy Department and the Faculty of Religion and Theology, Vrije Universiteit, Amsterdam, The Netherlands.
¹⁴ Centre for Contextual Behavioural Science, School of Psychology, University of Chester, Chester, UK.
¹⁵ Institute of Applied Health Research, University of Birmingham, Birmingham, UK.
¹⁶ Maison de santé pluridisciplinaire Pasteur, Chevilly-Larue, France.
¹⁷ Department of Preventive Medicine - Division of Biostatistics, Feinberg School of Medicine, Northwestern University, Chicago, IL, USA.
¹⁸ Department of Cognition, Emotion, and Methods in Psychology, Faculty of Psychology, University of Vienna, Vienna, Austria.
¹⁹ Faculty of Social Sciences, Charles University, Prague, Czech Republic.
²⁰ Department of Methodology and Statistics, Tilburg University, Tilburg, The Netherlands.
²¹ Research Organization of Open Innovation and Collaboration, Ritsumeikan University, Osaka, Japan.
²² Research Fellow of the Japan Society for the Promotion of Science, Erasmus University Rotterdam, Rotterdam, The Netherlands.
²³ Department of Psychology, Education, and Child Studies, Erasmus University Rotterdam, Rotterdam, The Netherlands.

Abstract

Increased execution of replication studies contributes to the effort to restore credibility of empirical research. However, a second generation of problems arises: the number of potential replication targets is at a serious mismatch with available resources. Given limited resources, replication target selection should be well-justified, systematic and transparently communicated. At present the discussion on what to consider when selecting a replication target is limited to theoretical discussion, self-reported justifications and a few formalized suggestions. In this Registered Report, we proposed a study involving the scientific community to create a list of considerations for consultation when selecting a replication target in psychology. We employed a modified Delphi approach. First, we constructed a preliminary list of considerations. Second, we surveyed psychologists who previously selected a replication target with regards to their considerations. Third, we incorporated the results into the preliminary list of considerations and sent the updated list to a group of individuals knowledgeable about concerns regarding replication target selection. Over the course of several rounds, we established consensus regarding what to consider when selecting a replication target. The resulting checklist can be used for transparently communicating the rationale for selecting studies for replication.

Keywords: consensus; replication; study selection.

Associated data

figshare/10.6084/m9.figshare.c.6403449