Abstract: 

Netflix is the pioneer for applying causal inference to real life applications. From the down-to-the-root online experimentation platform to a more advanced continuous explore exploit system, all the applications are heavily researched and developed to provide Netflix members the most member joy. Here we aim to provide the five lessons learned by applying Continuous Explore Exploit framework in multiple Netflix applications and hope to shed light on future innovations and applications.

Lesson 1: Setting up the CEE framework can bring huge customer satisfaction gains. In Video artwork, by employing the CEE framework has achieved one of the biggest wins in the area by a simple switch of personalized exploitation with collected explore data. Even just exploiting the only winner can achieve a sizable win.

Lesson 2: Adding in more arms and counterfactual analysis becomes a breeze in mature CEE applications. At Netflix, new features are developed to meet a certain cohort’s needs. With the CEE, we can easily add new arms to the selection portfolio. In this way, we can measure and maintain arm effectiveness continuously without further deployment and AB tests. The results from counterfactual analysis can help with generating hypotheses to improve personalization algorithms.

Lesson 3: CEE could be a faster way to go than AB tests in product launches once the system is in place. The long cycle times and dependence on allocations also constrain the overall number of concurrent tests and for AB tests. CEE would help relieve this pain by continuously launching new features and learning from its efficiency while productizing the most valuable options in exploitation.

Lesson 4: Apple to Apple Comparison is the key to accurate measurement. Whenever conducting counterfactual analysis, one key point is to make sure the population that we compare against are from the same distribution.

Lesson 5: Making sure no customer harm is the initial step to set up CEE for success. Although with all the benefits we can obtain from CEE, there can be potential customer dissatisfaction from the random exploration. We run a no-harm test at Netflix for all applications with CEE to ensure customer satisfaction.

We have summarized many learnings in applying Continuous Explore Exploit framework in various areas across Netflix. We will describe how Continuous Explore data has helped us be faster and smarter with innovation efforts in these areas at Netflix. These insights can bring more ideas into other applicable areas while spark further innovations for industries.

Bio: 

Sophia Liu is a Senior Data scientist at Netflix. She leads the data science initiatives for Netflix games offerings. She specializes in online controlled experimentation (A/B tests), causal inferences and analytics. Before Netflix, she was a senior data scientist in Analysis and Experimentation (A&E) team at Microsoft. Dr. Liu received her M.S. and PhD degrees in Electrical Engineering from Columbia University and Northwestern University in 2012 and 2016, respectively. During her graduate study, she has won two best paper awards out of 14 international publications and conducted internships in Bell Labs, Cisco and Alliance Data Systems.