Test–retest: Agreement or reliability?

Abstract

Test–retest is a concept that is routinely evaluated during the validation phase of many measurement tools. However, this term covers at least two related but very different concepts: reliability and agreement. Reliability is the ability of a measure applied twice upon the same respondents to produce the same ranking on both occasions. Agreement requires the measurement tool to produce twice the same exact values. An analysis of research papers showed that the distinction between both concepts remains anything but clear, and that the current practice is to evaluate reliability only, generally on the basis of the sole Pearson’s correlation. This practice is very problematic in the context of longitudinal studies because it becomes difficult to determine whether a difference between two successive observations is attributable to a real change of the respondents or only to the characteristics of the measurement tool, which then leads to a possible misinterpretation of the results. More focus should be given on the real interpretation of linear correlation, and when agreement is required in addition to reliability, then correct alternative, such as the Bland–Altman plot, should be more generally used.