Abstract: 

Although powerful, modern machine learning models can be sensitive. Seemingly subtle changes in a data distribution can destroy the performance of otherwise state-of-the art models, which can be especially problematic when ML models are deployed in production.. In this workshop, we will give a hands-on overview to drift detection, the discipline focused on detecting such changes. We will start by building an understanding of the ways in which drift can occur, and why it pays to detect it. We’ll then explore the anatomy of a drift detector, and learn how they can be used to detect drift in a principled manner.

You will work through a real-world example using Alibi Detect, an open-source Python library offering powerful algorithms for adversarial, outlier and drift detection.You’ll learn how to set-up drift detectors, and deduce what type of drift is occurring. Since data can take many forms, such as image, text or tabular data, you’ll explore how to use existing ML models to preprocess your data into a form suitable for drift detectors. Then, to gain further insights into the causes of drift, you’ll employ state-of-the art detectors which are able to perform fine-grained attribution to instances and features. To assess whether model performance has been affected by drift, you’ll experiment with using supervised and uncertainty based detectors. Finally, since in production environments data often arrives sequentially, we’ll discuss the use of online drift detectors that act on single data instances as they arrive.

This hands-on workshop is targeted at a beginner-intermediate level. No prior knowledge or understanding of drift detection is required (we’ll be covering that) but a basic knowledge of machine learning and some experience with Python will be helpful.

Bio: 

Ashley is a data science research engineer at Seldon, where he works on developing production-ready tools for drift, adversarial and outlier detection. Prior to joining Seldon, he spent a number of years as a Research Fellow at The Alan Turing Institute. Here, he explored the use of machine learning for tackling aerospace engineering problems, with a focus on explainability and uncertainty quantification. Ashley also completed a PhD at the University of Cambridge, and is a keen proponent of open-source software.