No CrossRef data available.
Article contents
The future of experimental design: Integrative, but is the sample diverse enough?
Published online by Cambridge University Press: 05 February 2024
Abstract
Almaatouq et al. propose an “integrative approach” to increase the generalisability and commensurability of experiments. Yet their metascientific approach has one glaring omission (and misinterpretation of) – the role of sample diversity in generalisability. In this commentary, we challenge false notions of subsumed duality between contexts, population, and diversity, and propose modifications to their design space to accommodate sample diversity.
- Type
- Open Peer Commentary
- Information
- Copyright
- Copyright © The Author(s), 2024. Published by Cambridge University Press
References
Arnett, J. J. (2009). The neglected 95%, a challenge to psychology's philosophy of science. American Psychologist, 64(6), 571–574. https://doi.org/10.1037/a0016723CrossRefGoogle Scholar
Banerjee, S., & Mitra, S. (2023). Behavioural public policies for the social brain. Behavioural Public Policy, 1–23.CrossRefGoogle Scholar
Deffner, D., Rohrer, J. M., & McElreath, R. (2022). A causal framework for cross-cultural generalizability. Advances in Methods and Practices in Psychological Science, 5(3). https://doi.org/10.1177/25152459221106366CrossRefGoogle Scholar
Ghai, S. (2022). Expand diversity definitions beyond their Western perspective. Nature, 602(7896), 211. https://doi.org/10.1038/d41586-022-00330-0CrossRefGoogle ScholarPubMed
Ghai, S., Fassi, L., Awadh, F., & Orben, A. (2023). Lack of sample diversity in research on adolescent depression and social media use: A scoping review and meta-analysis. Clinical Psychological Science, 0(0). https://doi.org/10.1177/21677026221114859Google Scholar
Gurven, M. D. (2018). Broadening horizons: Sample diversity and socioecological theory are essential to the future of psychological science. Proceedings of the National Academy of Sciences of the United States of America, 115(45), 11420–11427.CrossRefGoogle Scholar
Haeffel, G. J., & Cobb, W. R. (2022). Tests of generalizability can diversify psychologyand improve theories. Nature Reviews Psychology, 1(4), 186–187. https://doi.org/10.1038/s44159-022-00039-xCrossRefGoogle Scholar
Hallsworth, M. (2023). A manifesto for applying behavioural science. Nature Human Behaviour, 7(3), 1–13.CrossRefGoogle ScholarPubMed
Henrich, J., Heine, S. J., & Norenzayan, A. (2010). The weirdest people in the world? Behavioral and Brain Sciences, 33(2–3), 61–83.CrossRefGoogle ScholarPubMed
Holland, P. W. (1986). Statistics and causal inference. Journal of the American Statistical Association, 81(396), 945–960.CrossRefGoogle Scholar
Adetula, A., Forscher, P. S., Basnight-Brown, D., Azouaghe, S., & IJzerman, H. (2022). Psychology should generalize from – not just to – Africa. Nature Human Behaviour, 1, 370–371. https://doi.org/10.1038/s44159-022-00070-yGoogle Scholar
Medin, D., Ojalehto, B., Marin, A., & Bang, M. (2017). Systems of (non-) diversity. Nature Human Behaviour, 1(5), 88.CrossRefGoogle Scholar
Neyman, J. (1923). On the application of probability theory to agricultural experiments. Essay on principles. Annals of Agricultural Sciences, 1–51.Google Scholar
Rad, M. S., Martingano, A. J., & Ginges, J. (2018). Toward a psychology of Homo sapiens: Making psychological science more representative of the human population. Proceedings of the National Academy of Sciences of the United States of America, 115(45), 11401–11405.CrossRefGoogle Scholar
Singh, L. (2022). Navigating equity and justice in international collaborations. Nature Human Behaviour, 1, 372–373. https://doi.org/10.1038/s44159-022-00077-5CrossRefGoogle Scholar
Rubin, D. B. (1977). Assignment to treatment group on the basis of a covariate. Journal of Educational Statistics, 2(1), 1–26.CrossRefGoogle Scholar
Tang, E. K., Suganthan, P. N., & Yao, X. (2006). An analysis of diversity measures. Machine Learning, 65, 247–271.CrossRefGoogle Scholar
Target article
Beyond playing 20 questions with nature: Integrative experiment design in the social and behavioral sciences
Related commentaries (31)
Against naïve induction from experimental data
Are language–cognition interactions bigger than a breadbox? Integrative modeling and design space thinking temper simplistic questions about causally dense phenomena
Assume a can opener
Beyond integrative experiment design: Systematic experimentation guided by causal discovery AI
Commensurability engineering is first and foremost a theoretical exercise
Confidence in research findings depends on theory
Consensus meetings will outperform integrative experiments
Dimensional versus conceptual incommensurability in the social and behavioral sciences
Discovering the unknown unknowns of research cartography with high-throughput natural description
Diversity of contributions is not efficient but is essential for science
Don't let perfect be the enemy of better: In defense of unparameterized megastudies
Eliminativist induction cannot be a solution to psychology's crisis
Experiment commensurability does not necessitate research consolidation
Explore your experimental designs and theories before you exploit them!
Getting lost in an infinite design space is no solution
Individual differences do matter
Integrative design for thought-experiments
Integrative experiments require a shared theoretical and methodological basis
Is generalization decay a fundamental law of psychology?
Measurement validity and the integrative approach
Neuroadaptive Bayesian optimisation can allow integrative design spaces at the individual level in the social and behavioural sciences… and beyond
Phenomena complexity, disciplinary consensus, and experimental versus correlational research in psychological science
Representative design: A realistic alternative to (systematic) integrative design
Sampling complex social and behavioral phenomena
Some problems with zooming out as scientific reform
Test many theories in many ways
The elephant's other legs: What some sciences actually do
The future of experimental design: Integrative, but is the sample diverse enough?
The miss of the framework
The social sciences needs more than integrative experimental designs: We need better theories
There are no shortcuts to theory
Author response
Replies to commentaries on beyond playing 20 questions with nature