Abstract: 

It’s often said that debugging machine learning is 10 times harder than debugging software since it uniquely combines many of the problems of software and data engineering as well as challenges unique to data science and MLOps.

This is particularly true in deep learning, where labeling data is expensive and is one of the only ways to get model performance feedback. It’s no wonder that even the most advanced and best-funded large language models, like OpenAI’s ChatGPT and Google’s Bard, sometimes hallucinate and fail in the real world.

Here’s the truth: troubleshooting models based on unstructured data is notoriously difficult. The measures typically used for drift in tabular data–such as population stability index, Kullback-Leibler divergence, and Jensen-Shannon divergence–allow for statistical analysis on structured labels, but do not extend to unstructured data. The general challenge with measuring unstructured data drift is that you need to understand the change in relationships inside the unstructured data itself. In short, you need to understand the data in a deeper way before you can understand drift and performance degradation.

In this presentation, Amber Roberts, Machine Learning Engineer at Arize AI, will present findings from research on ways to measure vector/embedding drift for image and language models. With lessons learned from testing different approaches (including Euclidean and Cosine distance) across billions of streams and use cases, Roberts will dive into how to detect whether two unstructured language datasets are different — and, if so, how to understand that difference using techniques such as UMAP.

In the coming years, more ML teams will likely look to embedding drift to help detect and understand differences in their unstructured data. This presentation with examples from the real world will be both useful and fascinating to advanced data scientists and learners alike!

Bio: 

Amber Roberts is a ML Growth Lead at Arize AI, a ML observability company built for maintaining models in production. Previously, Amber was a product manager of AI at Splunk and the Head of Artificial Intelligence at Insight Data Science. A Carnegie Fellow, Amber has an MS in Astrophysics from the Universidad de Chile.