论文标题
解决在线实验中隐藏的缺陷
Addressing Hidden Imperfections in Online Experimentation
论文作者
论文摘要
作为开发过程的一部分，技术公司越来越多地使用随机对照试验（RCT）。尽管对工程系统和数据仪器有很好的控制，但这些RCT仍可以不完美地执行。实际上，在线实验遭受了许多相同的偏见，包括选择加入和用户活动偏见，选择偏见，不符合治疗的偏见以及更普遍地，在测试感兴趣问题的能力方面面临的挑战。这些不完美的结果可能导致估计的因果效应，统计能力的损失，效果的衰减，甚至需要重新构架可以回答的问题。本文旨在使实验的实践者更加了解技术行业RCT的缺陷，这些不完美是可以在整个工程堆栈或设计过程中隐藏的。
Technology companies are increasingly using randomized controlled trials (RCTs) as part of their development process. Despite having fine control over engineering systems and data instrumentation, these RCTs can still be imperfectly executed. In fact, online experimentation suffers from many of the same biases seen in biomedical RCTs including opt-in and user activity bias, selection bias, non-compliance with the treatment, and more generally, challenges in the ability to test the question of interest. The result of these imperfections can lead to a bias in the estimated causal effect, a loss in statistical power, an attenuation of the effect, or even a need to reframe the question that can be answered. This paper aims to make practitioners of experimentation more aware of imperfections in technology-industry RCTs, which can be hidden throughout the engineering stack or in the design process.
