Abstract
Online controlled experiments (OCEs), also known as A/B tests, have become ubiquitous in evaluating the impact of changes made to software products and services. While the concept of online controlled experiments is simple, there are many practical challenges in running OCEs at scale. To understand the top practical challenges in running OCEs at scale and encourage further academic and industrial exploration, representatives with experience in large-scale experimentation from thirteen different organizations (Airbnb, Amazon, Booking.com, Facebook, Google, LinkedIn, Lyft, Microsoft, Netflix, Twitter, Uber, Yandex, and Stanford University) were invited to the first Practical Online Controlled Experiments Summit. All thirteen organizations sent representatives. Together these organizations have tested more than one hundred thousand experiment treatments last year. Thirty-four experts from these organizations participated in the summit in Sunnyvale, CA, USA on December 13-14, 2018.
While there are papers from individual organizations on some of the challenges and pitfalls in running OCEs at scale, this is the first paper to provide the top challenges faced across the industry for running OCEs at scale and some common solutions.
Publisher
Association for Computing Machinery (ACM)
Reference77 articles.
1. {Uber Marketplace} Marketplace Experimentation -- Vivek Trehan - YouTube: 2018. https://www.youtube.com/watch?v=IR000RqN7pw. Accessed: 2019-02-05. {Uber Marketplace} Marketplace Experimentation -- Vivek Trehan - YouTube: 2018. https://www.youtube.com/watch?v=IR000RqN7pw. Accessed: 2019-02-05.
2. 25
Cognitive Biases Home Page | 25 Cognitive Biases - "The Psychology of Human Misjudgment" http://25cognitivebiases.com/. Accessed: 2019-02-08. 25 Cognitive Biases Home Page | 25 Cognitive Biases - "The Psychology of Human Misjudgment" http://25cognitivebiases.com/. Accessed: 2019-02-08.
3. A/B Testing at Scale Tutorial Strata 2018: https://expplatform. com/2018StrataABtutorial/. Accessed: 2019-02- 05. A/B Testing at Scale Tutorial Strata 2018: https://expplatform. com/2018StrataABtutorial/. Accessed: 2019-02- 05.
4. AdvancedTopic_BestControlledExperiments.docx - Microsoft Word Online: https://onedrive.live.com/view.aspx?resid=8612090E610 871E4!323967&ithint=file%2Cdocx&app=Word&authk ey=!APyJuF_t0dOFj_M. Accessed: 2019-02-05. AdvancedTopic_BestControlledExperiments.docx - Microsoft Word Online: https://onedrive.live.com/view.aspx?resid=8612090E610 871E4!323967&ithint=file%2Cdocx&app=Word&authk ey=!APyJuF_t0dOFj_M. Accessed: 2019-02-05.
5. AdvancedTopic_OEC.docx - Microsoft Word Online: https://onedrive.live.com/view.aspx?resid=8612090E610 871E4!282179&ithint=file%2Cdocx&app=Word&authk ey=!ANFGOBrhVt91ODk. Accessed: 2019-02-05. AdvancedTopic_OEC.docx - Microsoft Word Online: https://onedrive.live.com/view.aspx?resid=8612090E610 871E4!282179&ithint=file%2Cdocx&app=Word&authk ey=!ANFGOBrhVt91ODk. Accessed: 2019-02-05.
Cited by
79 articles.
订阅此论文施引文献
订阅此论文施引文献,注册后可以免费订阅5篇论文的施引文献,订阅后可以查看论文全部施引文献