Generalized Synthetic Control for TestOps at ABI: Models, Algorithms, and Infrastructure

Abstract

We describe a novel optimization-based approach—generalized synthetic control (GSC)—in which we learn from experiments conducted in a physical retail environment. GSC solves a long-standing problem of learning from experiments conducted in this environment when treatment effects are small, the environment is extremely noisy and nonstationary, and interference and adherence problems are commonplace. The utilization of GSC has demonstrated a remarkable increase in statistical power, approximately one hundredfold (100×) higher than conventional inferential methods. This innovative approach forms the basis of TestOps, a pioneering large-scale experimentation platform designed specifically for physical retailers. TestOps was developed and has been broadly implemented as part of a collaboration between Anheuser Busch Inbev (ABI) and a team of operations researchers and data engineers from the Massachusetts Institute of Technology. TestOps currently runs physical experiments impacting approximately 135 million USD in revenue every month and routinely identifies innovations that result in a 1%–2% increase in sales volume. The vast majority of these innovations would have remained unidentified had we not developed our novel approach to inference. Prior to our implementation, statistically significant conclusions could be drawn on only ∼6% of all experiments, a fraction that has now increased by 10-fold. Given its success, TestOps is being rolled out globally at ABI, driving significant revenue growth and enabling the extraction of valuable insights from large-scale physical experiments. History: This paper has been accepted for the INFORMS Journal on Applied Analytics Special Issue—2022 Daniel H. Wagner Prize for Excellence in the Practice of Advanced Analytics and Operations Research.