Comparing the vibration of effects due to model, data pre-processing and sampling uncertainty on a large data set in personality psychology

Downloads

Authors

  • Simon Klau Institute for Medical Information Processing, Biometry, and Epidemiology
  • Felix D. Schönbrodt Department of Psychology, Ludwig-Maximilians-Universität München
  • Chirag J. Patel Department of Biomedical Informatics, Harvard Medical School
  • John P. A. Ioannidis Department of Epidemiology and Population Health, Stanford University
  • Anne-Laure Boulesteix Institute for Medical Information Processing, Biometry, and Epidemiology, Munich
  • Sabine Hoffmann Institute for Medical Information Processing, Biometry, and Epidemiology, Munich

DOI:

https://doi.org/10.15626/MP.2020.2556

Keywords:

metascience, researcher degrees of freedom, stability, replicability, Big Five

Abstract

Researchers have great flexibility in the analysis of observational data. If combined with selective reporting and pressure to publish, this flexibility can have devastating consequences on the validity of research findings. We extend the recently proposed vibration of effects approach to provide a framework comparing three main sources of uncertainty which lead to instability in empirical findings, namely data pre-processing, model, and sampling uncertainty. We analyze the behavior of these sources for varying sample sizes for two associations in personality psychology. Through the joint investigation of model and data pre-processing vibration, we can compare the relative impact of these two types of uncertainty and identify the most influential analytical choices. While all types of vibration show a decrease for increasing sample sizes, data pre-processing and model vibration remain non-negligible, even for a sample of over 80000 participants. The increasing availability of large data sets that are not initially recorded for research purposes can make data pre-processing and model choices very influential. We therefore recommend the framework as a tool for transparent reporting of the stability of research findings.

Metrics

Metrics Loading ...

Downloads

Published

2023-05-10

Issue

Section

Original articles