Episode 19: Quantifauxcation

In this episode, we discuss Quantifauxcation, described by statistician Philip Stark as “situations in which a number is, in effect, made up, and then is given credence merely because it is quantitative.” We give examples of quantifauxcation in psychology, including errors of the third kind. We spend the second half of the podcast discussing how to develop quantitative measures that are meaningful and bridge the divide between qualitative and quantitative observations.

Shownotes

• Statistics textbook by Philip Stark.

• Stark, P. B. (2022). Pay No attention to the model behind the curtain. Pure and Applied Geophysics, 179(11), 4121–4145. https://doi.org/10.1007/s00024-022-03137-2 

• Burgess, E. W. (1927). Statistics and case studies as methods of sociological research, Vol 12(3), 103-120. (Thanks to Andy Grieve!)

• Nick Brown's role in pointing out flaws in the positivity ratio.

• Retraction notice of the positivity ratio paper.

• Blog by Tania Lombrozo on nonsensical formulas in abstracts.

• Kimball, A. W. (1957). Errors of the third kind in statistical consulting. Journal of the American Statistical Association, 52(278), 133–142. https://doi.org/10.1080/01621459.1957.10501374

• Type III errors: Philip Stark’s post of Deborah Mayo’s blog

• Brower, D. (1949). The problem of quantification in psychological science. Psychological Review, 56(6), 325–333. https://doi.org/10.1037/h0061802

• Guttman scales

• Wilson, M. (2023). Constructing measures: An item response modeling approach. Taylor & Francis.

• Wilson, M., Bathia, S., Morell, L., Gochyyev, P., Koo, B. W., & Smith, R. (2022). Seeking a better balance between efficiency and interpretability: Comparing the likert response format with the Guttman response format. Psychological Methods. Advance online publication. https://doi.org/10.1037/met0000462

• Bhatti, H.A., Mehta, S., McNeil, R., Wilson, M. (2023). A scientific approach to assessment: Rasch measurement and the four building blocks. In X. Liu & W. Boone (Eds.), Advances in Applications of Rash Measurement in Science Education. Springer Nature.