Multiplicity Control vs Replication

Making an Obvious Choice Even More Obvious

  • Andrew Hunter York University
  • Nataly Beribisky York University
  • Linda Farmus York University
  • Robert A. Cribbie York University
Keywords: multiplicity control, familywise error, power, replication, effect size, meta-analysis

Abstract

This paper presents a side-by-side consideration of multiplicity control procedures and replication as solutions to the problem of multiplicity. Several independent theoretical arguments are presented which demonstrate that replication serves several important functions, and that multiplicity control procedures have a number of serious flaws. Subsequently, the results of a simulation study are provided, showing that under typical conditions, replication provides similar familywise error control and power as multiplicity control procedures. Taken together, these theoretical and statistical arguments lead to the conclusion that researchers who are concerned about the problem of multiplicity should shift their attention away from multiplicity control procedures and towards increased use of replication.

Metrics

Metrics Loading ...
Published
2020-07-01
Section
Original articles