Averaging to Minimize or Eliminate Regression toward the Mean to Measure Pure Experimental Effects

Abstract

This paper explains how regression toward the mean can contaminate diary data, making it difficult to measure the pure effects of an experimental variable over time. Using a large scale real-life database collected by AT& T, a method of measuring this mathematical artifact is advanced. It is shown to manifest very quickly as a result of a spontaneous reaction toward happenstance, with the most extreme initial values gravitating most toward the mean. Then averaging over longer and longer periods of time to define use categories is shown to dilute happenstance increasingly, and therefore progressively minimizes or eliminates regression toward the mean. Finally, regression toward the mean is very pervasive and very persistent.