Abstract: 

A/B testing (experimentation) moves beyond correlation to identify if a causal relationship exists between two (or possibly more) variables. Because this is the only approach we can currently take to establish causality, A/B testing is critical to understand for any data scientist. In this workshop, you will have familiarity with the types of questions A/B testing can answer, how to design your own experiment, and what to watch out for on a conceptual level. You will also learn how to randomize control and test groups in Python before you conduct the experiment and analyze/test the results after the experiment ends. Finally, you will learn how to interpret the results in a clear and accurate way so that a key non-technical stakeholder can understand and use them to make decisions.

Session Outline
Module 1: A/B Testing Overview and Theory
Learn about the basic theory behind A/B testing (experimentation). You will learn what sorts of questions this approach can and cannot answer, and how this can be a complement to other approaches such as machine learning. You will also become familiar with how best to randomize and why, given your population of interest.

Module 2: Design Your A/B Test
Determine the biggest effect size a given experiment can identify, along with what this means for you and your stakeholder(s). Learn how to randomize subjects into groups and determine the best stopping condition(s). Practice describing this process and what it may mean for a non-technical stakeholder.

Module 3: Test and Interpret Your Results
Once the experiment is complete, you will now learn how to find the “so what” for your non-technical stakeholder(s) in Python. Specifically, you will be testing for a statistically significant difference between groups. You will also practice ways to identify if the difference (if any) is large enough to suggest any change in action. Finally, you will practice drafting conclusions and what they might mean for a non-technical stakeholder such as a product manager or even the board of your company.

Background Knowledge
Python, Pandas, Jupyter Notebooks, Beginner-Level Statistics (t-tests and descriptive statistics)

Bio: 

Dr. Boardman has over a decade of experience as a scientist, professor, and consultant in various sectors and disciplines, including digital marketing, health tech, intelligence, entrepreneurship, economic development, and health/medical policy. Her knowledge and experience includes A/B testing, machine learning, agent-based modeling, statistics, and intelligence analysis. In her work, she places heavy emphasis on clearly and empathetically communicating insights derived from small, qualitative data up to the big data she currently works with in digital marketing. Invigorated by messy, unconventional data and problems, she has also designed and modified several techniques and approaches to work with this. She believes that science is an ongoing process and we should adapt our approach to the problems and data, not find problems and data that work well with what we know. Currently, Dr. Boardman is a Senior Data Scientist at TI Health and holds a B.B.A. in International Business from the University of Oklahoma, an M.P.P. in Public Policy from Pepperdine University, a PhD in Public Policy from George Mason University, and a Masters of Information and Data Science from Berkeley.