Many Labs 2: investigating variation in replicability across samples and settings

Richard A. Klein , Michalangelo Vianello , Fred Hasselman , Byron G. Adams , Reginald B. Adams , Sinan Alper , Mark Aveyard , Jordan R. Axt , Mayowa T. Babalola , Šteˇpán Bahník , Rishtee Batra , Mihály Berkics , Michael J. Bernstein , Daniel R. Berry , Olga Białobrzeska , Evans Dami Binan , Konrad Bocian , Mark J. Brandt , Robert Busching , Anna Cabak Rédei , Huajian Cai , Fanny Cambier , Katarzyna Cantarero , Cheryl L. Carmichael , Francisco Ceric , Jesse Chandler , Jen-ho Chang , Armand Chatard , Eva E. Chen , Winnee Cheong , David C. Cicero , Sharon Coen , Jennifer A. Coleman , Brian Collisson , Morgan A. Conway , Katherine S. Corker , Paul G. Curran , Fiery Cushman , Zubairu Dagona , Ilker Dalgar , Anna Dalla Rosa , William E. Davis , Maaike de Bruijn , De Schutter Leander , Thierry Devos , Marieke de Vries , Canay Dog˘ulu , Nerisa Dozo , Kristin Nicole Dukes , Yarrow Dunham , Kevin Durrheim , Charles R. Ebersole , John E. Edlund , Anja Eller , Alexander Scott English , Carolyn Finck , Natalia Frankowska , Miguel-Ángel Freyre , Mike Friedman , Elisa Maria Galliani , Joshua C. Gandi , Tanuka Ghoshal , Steffen R. Giessner , Tripat Gill , Timo Gnambs , Ángel Gómez , Roberto González , Jesse Graham , Jon E. Grahe , Ivan Grahek , Eva G. T. Green , Kakul Hai , Matthew Haigh , Elizabeth L. Haines , Michael P. Hall , Marie E. Heffernan , Joshua A. Hicks , Petr Houdek , Jeffrey R. Huntsinger , Ho Phi Huynh , Hans Ijzerman , Yoel Inbar , Åse H. Innes-Ker , William Jiménez-Leal , Melissa-Sue John , Serdar Karabati , Haruna Karick , Victor N. Keller , Anna Kende , Nicolas Kervyn , Goran Kneževic , Carrie Kovacs , Lacy E. Krueger , German Kurapov , Jamie Kurtz , Daniël Lakens , Ljiljana B. Lazarevic´ , Carmel A. Levitan , Neil A. Lewis, Jr. , Samuel Lins , Nikolette P. Lipsey , Joy E. Losee , Esther Maassen , Angela T. Maitner , Winfrida Malingumu , Robyn K. Mallett , Satia A. Marotta , Janko Mededovic , Fernando Mena-Pacheco , Taciano L. Milfont , Wendy L. Morris , Dominic Murphy , Andriy Myachykov , Nick Neave , Koen Neijenhuijs , Anthony J. Nelson , Félix Neto , Austin Lee Nichols , Aaron Ocampo , Susan L. O’Donnell , Haruka Oikawa , Masanori Oikawa , Elsie Ong , Gábor Orosz , Małgorzata Osowiecka , Grant Packard , Rolando Pérez-Sánchez , Boban Petrovic´ , Ronaldo Pilati , Brad Pinter , Lysandra Podesta , Gabrielle Pogge , Monique M. H. Pollmann , Abraham M. Rutchick , Patricio Saavedra , Alexander K. Saeri , Erika Salomon , Kathleen Schmidt , Felix D. Schönbrodt , Maciej B. Sekerdej , David Sirlopú , Jeanine L. M. Skorinko , Michael A. Smith , Vanessa Smith-Castro , Karin C. H. J. Smolders , Agata Sobków , Walter Sowden , Philipp Spachtholz , Manini Srivastava , Troy G. Steiner , Jeroen Stouten , Chris N. H. Street , Oskar K. Sundfelt , Stephanie Szeto , Ewa Szumowska , Andrew C. W. Tang , Norbert Tanzer , Morgan J. Tear , Jordan Theriault , Manuela Thomae , David Torres , Jakub Traczyk , Joshua M. Tybur , Adrienn Ujhelyi , Robbie C. M. van Aert , Marcel A. L. M. van Assen , Marije van der Hulst , Paul A. M. van Lange , Anna Elisabeth van ’t Veer , Alejandro Vásquez-Echeverría , Leigh Ann Vaughn , Alexandra Vázquez , Luis Diego Vega , Catherine Verniers , Mark Verschoor , Ingrid P. J. Voermans , Marek A. Vranka , Cheryl Welch , Aaron L. Wichman , Lisa A. Williams , Michael Wood , Julie A. Woodzicka , Marta Wrońska , Liane Young , John M. Zelenski , Zeng Zhijia , Brian A. Nosek

Abstract

We conducted preregistered replications of 28 classic and contemporary published findings, with protocols that were peer reviewed in advance, to examine variation in effect magnitudes across samples and settings. Each protocol was administered to approximately half of 125 samples that comprised 15,305 participants from 36 countries and territories. Using the conventional criterion of statistical significance (p < .05), we found that 15 (54%) of the replications provided evidence of a statistically significant effect in the same direction as the original finding. With a strict significance criterion (p < .0001), 14 (50%) of the replications still provided such evidence, a reflection of the extremely highpowered design. Seven (25%) of the replications yielded effect sizes larger than the original ones, and 21 (75%) yielded effect sizes smaller than the original ones. The median comparable Cohen’s ds were 0.60 for the original findings and 0.15 for the replications. The effect sizes were small (< 0.20) in 16 of the replications (57%), and 9 effects (32%) were in the direction opposite the direction of the original effect. Across settings, the Q statistic indicated significant heterogeneity in 11 (39%) of the replication effects, and most of those were among the findings with the largest overall effect sizes; only 1 effect that was near zero in the aggregate showed significant heterogeneity according to this measure. Only 1 effect had a tau value greater than .20, an indication of moderate heterogeneity. Eight others had tau values near or slightly above .10, an indication of slight heterogeneity. Moderation tests indicated that very little heterogeneity was attributable to the order in which the tasks were performed or whether the tasks were administered in lab versus online. Exploratory comparisons revealed little heterogeneity between Western, educated, industrialized, rich, and democratic (WEIRD) cultures and less WEIRD cultures (i.e., cultures with relatively high and low WEIRDness scores, respectively). Cumulatively, variability in the observed effect sizes was attributable more to the effect being studied than to the sample or setting in which it was studied.
Author Richard A. Klein
Richard A. Klein,,
-
, Michalangelo Vianello
Michalangelo Vianello,,
-
, Fred Hasselman
Fred Hasselman,,
-
, Byron G. Adams
Byron G. Adams,,
-
, Reginald B. Adams
Reginald B. Adams,,
-
, Sinan Alper
Sinan Alper,,
-
, Mark Aveyard
Mark Aveyard,,
-
, Jordan R. Axt
Jordan R. Axt,,
-
, Mayowa T. Babalola
Mayowa T. Babalola,,
-
, Šteˇpán Bahník
Šteˇpán Bahník,,
-
et al.`
Journal seriesAdvances in Methods and Practices in Psychological Science, ISSN 2515-2459 , e-ISSN 2515-2467 , (0 pkt)
Issue year2018
Vol1 (4)
Pages443-490
Publication size in sheets2.35
Keywords in Englishsocial psychology, cognitive psychology, replication, culture, individual differences, sampling effects, situational effects, meta-analysis, Registered Report, open data, open materials, preregistered
DOIDOI:10.1177/2515245918810225
URL https://journals.sagepub.com/doi/10.1177/2515245918810225
Languageen angielski
File
Many labs_tekst.pdf 1.32 MB
Additional file
Oświadczenie_Cantarero.pdf 21.29 KB
Oświadczenie_N.Frankowska_Many Labs 2.pdf 16.33 KB
Klein, Vianello_oswiadczenie_Sobkow.pdf 49.76 KB
Score (nominal)5
Score sourcejournalList
Citation count*324 (2020-09-18)
Cite
Share Share

Get link to the record


* presented citation count is obtained through Internet information analysis and it is close to the number calculated by the Publish or Perish system.
Back
Confirmation
Are you sure?