Three Coefficients for Analyzing the Reliability and Validity of Ratings

Abstract

Three numerical coefficients (V, R, and H) for analyzing the validity and reliability of ratings are described. Each coefficient, which ranges in value from 0 to 1, is computed as the ratio of an obtained to a maximum sum of differences in ratings, or as 1 minus that ratio. Computer programs for calculating the coefficients, their associated individual and cumulative right-tail probabilities, and the population mean and standard deviation of each coefficient are available. Tables of the individual and right-tail probabilities for selected values of the three coefficients are included for 2 to 7 rating categories and 2 to 25 items or raters. More complete probability tables can be generated for any value of c, m, or n with one of the computer programs provided. When the number of items or raters is large (>25), the right-tail probability associated with any value of a coefficient may be estimated by a z-score procedure. The three coefficients are applicable not only to validity and reliability (test-retest and internal consistency) determinations but also to the item analysis, agreement analysis, and cluster or factor analysis of rating-scale data.