Validation of predicted individual treatment effects in out of sample respondents

Abstract

Personalized medicine promises the ability to improve patient outcomes by tailoring treatment recommendations to the likelihood that any given patient will respond well to a given treatment. It is important that predictions of treatment response be validated and replicated in independent data to support their use in clinical practice. In this paper, we propose and test an approach for validating predictions of individual treatment effects with continuous outcomes across samples that uses matching in a test (validation) sample to match individuals in the treatment and control arms based on their predicted treatment response and their predicted response under control. To examine the proposed validation approach, we conducted simulations where test data is generated from either an identical, similar, or unrelated process to the training data. We also examined the impact of nuisance variables. To demonstrate the use of this validation procedure in the context of predicting individual treatment effects in the treatment of alcohol use disorder, we apply our validation procedure using data from a clinical trial of combined behavioral and pharmacotherapy treatments. We find that the validation algorithm accurately confirms validation and lack of validation, and also provides insights into cases where test data were generated under similar, but not identical conditions. We also show that the presence of nuisance variables detrimentally impacts algorithm performance, which can be partially reduced though the use of variable selection methods. An advantage of the approach is that it can be widely applied to different predictive methods.