Many Labs 2: investigating variation in replicability across samples and settings
Richard A. Klein , Michalangelo Vianello , Fred Hasselman , Byron G. Adams , Reginald B. Adams , Sinan Alper , Mark Aveyard , Jordan R. Axt , Mayowa T. Babalola , Šteˇpán Bahník , Rishtee Batra , Mihály Berkics , Michael J. Bernstein , Daniel R. Berry , Olga Białobrzeska , Evans Dami Binan , Konrad Bocian , Mark J. Brandt , Robert Busching , Anna Cabak Rédei , Huajian Cai , Fanny Cambier , Katarzyna Cantarero , Cheryl L. Carmichael , Francisco Ceric , Jesse Chandler , Jen-ho Chang , Armand Chatard , Eva E. Chen , Winnee Cheong , David C. Cicero , Sharon Coen , Jennifer A. Coleman , Brian Collisson , Morgan A. Conway , Katherine S. Corker , Paul G. Curran , Fiery Cushman , Zubairu Dagona , Ilker Dalgar , Anna Dalla Rosa , William E. Davis , Maaike de Bruijn , De Schutter Leander , Thierry Devos , Marieke de Vries , Canay Dog˘ulu , Nerisa Dozo , Kristin Nicole Dukes , Yarrow Dunham , Kevin Durrheim , Charles R. Ebersole , John E. Edlund , Anja Eller , Alexander Scott English , Carolyn Finck , Natalia Frankowska , Miguel-Ángel Freyre , Mike Friedman , Elisa Maria Galliani , Joshua C. Gandi , Tanuka Ghoshal , Steffen R. Giessner , Tripat Gill , Timo Gnambs , Ángel Gómez , Roberto González , Jesse Graham , Jon E. Grahe , Ivan Grahek , Eva G. T. Green , Kakul Hai , Matthew Haigh , Elizabeth L. Haines , Michael P. Hall , Marie E. Heffernan , Joshua A. Hicks , Petr Houdek , Jeffrey R. Huntsinger , Ho Phi Huynh , Hans Ijzerman , Yoel Inbar , Åse H. Innes-Ker , William Jiménez-Leal , Melissa-Sue John , Serdar Karabati , Haruna Karick , Victor N. Keller , Anna Kende , Nicolas Kervyn , Goran Kneževic , Carrie Kovacs , Lacy E. Krueger , German Kurapov , Jamie Kurtz , Daniël Lakens , Ljiljana B. Lazarevic´ , Carmel A. Levitan , Neil A. Lewis, Jr. , Samuel Lins , Nikolette P. Lipsey , Joy E. Losee , Esther Maassen , Angela T. Maitner , Winfrida Malingumu , Robyn K. Mallett , Satia A. Marotta , Janko Mededovic , Fernando Mena-Pacheco , Taciano L. Milfont , Wendy L. Morris , Dominic Murphy , Andriy Myachykov , Nick Neave , Koen Neijenhuijs , Anthony J. Nelson , Félix Neto , Austin Lee Nichols , Aaron Ocampo , Susan L. O’Donnell , Haruka Oikawa , Masanori Oikawa , Elsie Ong , Gábor Orosz , Małgorzata Osowiecka , Grant Packard , Rolando Pérez-Sánchez , Boban Petrovic´ , Ronaldo Pilati , Brad Pinter , Lysandra Podesta , Gabrielle Pogge , Monique M. H. Pollmann , Abraham M. Rutchick , Patricio Saavedra , Alexander K. Saeri , Erika Salomon , Kathleen Schmidt , Felix D. Schönbrodt , Maciej B. Sekerdej , David Sirlopú , Jeanine L. M. Skorinko , Michael A. Smith , Vanessa Smith-Castro , Karin C. H. J. Smolders , Agata Sobków , Walter Sowden , Philipp Spachtholz , Manini Srivastava , Troy G. Steiner , Jeroen Stouten , Chris N. H. Street , Oskar K. Sundfelt , Stephanie Szeto , Ewa Szumowska , Andrew C. W. Tang , Norbert Tanzer , Morgan J. Tear , Jordan Theriault , Manuela Thomae , David Torres , Jakub Traczyk , Joshua M. Tybur , Adrienn Ujhelyi , Robbie C. M. van Aert , Marcel A. L. M. van Assen , Marije van der Hulst , Paul A. M. van Lange , Anna Elisabeth van ’t Veer , Alejandro Vásquez-Echeverría , Leigh Ann Vaughn , Alexandra Vázquez , Luis Diego Vega , Catherine Verniers , Mark Verschoor , Ingrid P. J. Voermans , Marek A. Vranka , Cheryl Welch , Aaron L. Wichman , Lisa A. Williams , Michael Wood , Julie A. Woodzicka , Marta Wrońska , Liane Young , John M. Zelenski , Zeng Zhijia , Brian A. Nosek
AbstractWe conducted preregistered replications of 28 classic and contemporary published findings, with protocols that were peer reviewed in advance, to examine variation in effect magnitudes across samples and settings. Each protocol was administered to approximately half of 125 samples that comprised 15,305 participants from 36 countries and territories. Using the conventional criterion of statistical significance (p < .05), we found that 15 (54%) of the replications provided evidence of a statistically significant effect in the same direction as the original finding. With a strict significance criterion (p < .0001), 14 (50%) of the replications still provided such evidence, a reflection of the extremely highpowered design. Seven (25%) of the replications yielded effect sizes larger than the original ones, and 21 (75%) yielded effect sizes smaller than the original ones. The median comparable Cohen’s ds were 0.60 for the original findings and 0.15 for the replications. The effect sizes were small (< 0.20) in 16 of the replications (57%), and 9 effects (32%) were in the direction opposite the direction of the original effect. Across settings, the Q statistic indicated significant heterogeneity in 11 (39%) of the replication effects, and most of those were among the findings with the largest overall effect sizes; only 1 effect that was near zero in the aggregate showed significant heterogeneity according to this measure. Only 1 effect had a tau value greater than .20, an indication of moderate heterogeneity. Eight others had tau values near or slightly above .10, an indication of slight heterogeneity. Moderation tests indicated that very little heterogeneity was attributable to the order in which the tasks were performed or whether the tasks were administered in lab versus online. Exploratory comparisons revealed little heterogeneity between Western, educated, industrialized, rich, and democratic (WEIRD) cultures and less WEIRD cultures (i.e., cultures with relatively high and low WEIRDness scores, respectively). Cumulatively, variability in the observed effect sizes was attributable more to the effect being studied than to the sample or setting in which it was studied.
|Journal series||Advances in Methods and Practices in Psychological Science, ISSN 2515-2459 , e-ISSN 2515-2467 , (0 pkt)|
|Publication size in sheets||2.35|
|Keywords in English||social psychology, cognitive psychology, replication, culture, individual differences, sampling effects, situational effects, meta-analysis, Registered Report, open data, open materials, preregistered|
|Citation count*||362 (2020-11-27)|
* presented citation count is obtained through Internet information analysis and it is close to the number calculated by the Publish or Perish system.