Suppose you set loose a bunch of scientists on the same question, letting each use their best judgment on the method to answer a question. Would you expect them to come to the same conclusions?
Unfortunately, the truth is the state of our “methodological technology” just isn’t there yet. There remains a core of unresolvable uncertainty and randomness in the best of circumstances. Science isn’t certain.
This podcast is an audio read through of (initial version of the) article One question, many answers, published on New Things Under the Sun.
Articles mentioned
Huntington-Klein, Nick, Andreu Arenas, Emily Beam, Marco Bertoni, Jeffrey R. Bloem, Pralhad Burli, et al. 2021. The influence of hidden researcher decisions in applied microeconomics. Economic Inquiry, 59: 944– 960. https://doi.org/10.1111/ecin.12992
Silberzahn R, Uhlmann EL, Martin DP, et al. 2018. Many Analysts, One Data Set: Making Transparent How Variations in Analytic Choices Affect Results. Advances in Methods and Practices in Psychological Science: 337-356. https://doi.org/10.1177/2515245917747646
Breznau, Nate, Eike Mark Rinke, Alexander Wuttke, Muna Adem, Jule Adriaans, Amalia Alvarez-Benjumea, Henrik K. Andersen, et al. 2021. Observing Many Researchers Using the Same Data and Hypothesis Reveals a Hidden Universe of Uncertainty. MetaArXiv. March 24. https://doi.org/10.31222/osf.io/cd5j9
Jojanneke A. Bastiaansen, Yoram K. Kunkels, Frank J. Blaauw, Steven M. Boker, Eva Ceulemans, Meng Chen, Sy-Miin Chow, et al. 2020. Time to get personal? The impact of researchers choices on the selection of treatment targets using the experience sampling methodology. Journal of Psychosomatic Research 137(110211). https://doi.org/10.1016/j.jpsychores.2020.110211
Swenson, Isaac, Jason M. Lindo, and Krishna Regmi. 2020. Stable Income, Stable Family. NBER Working Paper 27753. https://doi.org/10.3386/w27753