Design and Analysis of Replication Studies

Abstract

Issues surrounding the importance and interpretation of replication research have generated considerable debate and controversy in recent years. Some of the controversy can be attributed to imprecise and inadequate specifications of the statistical criteria needed to assess replication and nonreplication. Two types of statistical replication evidence and four types of statistical nonreplication evidence are described. In addition, three types of inconclusive statistical replication evidence are described. An important benefit of a replication study is the ability to combine an effect-size estimate from the original study with an effect-size estimate from the follow-up study to obtain a more precise and generalizable effect-size estimate. The sample size in the follow-up study is an important design consideration, and some methods for determining the follow-up sample size requirements are discussed. R functions are provided that can be used to analyze results from a replication study. R functions to determine the appropriate sample size in the follow-up study also are provided.