Uncertainty Quantification: Approaches and Methods


As machine learning models have become more present in our lives, there has been increasing attention on the reliability of these models. A major component of this is understanding the how uncertain the model is about its prediction. No model is exactly right 100% of the time, so we need methods and approaches by which we can quantify the level of uncertainty around a prediction. Approaches to uncertainty quantification (UQ) vary, and depend on the type of problem. For classification problems, the primary approach is probability calibration: making sure that the model outputs corresponding to each class "behaves well" as a probability. For regression problems, there are several different approaches. One can configure models to output an interval, rather than a single point prediction, along with a "coverage" value that specifies the probability that the interval covers the true value. The framework of Conformal Prediction provides theoretical guarantees around such interval predictions. Or one can use methods that output an entire conditional density for y given X. This is called probabilistic regression, or conditional density estimation. Several parametric and non-parametric approaches exist for this problem. This workshop will provide the theoretical context for these methods and then dive into real-world examples of their applications using Jupyter notebooks.

Background Knowledge:

This session is primarily geared towards practicing data scientists who wish to increase their skill level and familiarity with more advanced techniques. That said, anyone should be able to benefit from attending the session


Brian Lucena is Principal at Numeristical, where he advises companies of all sizes on how to apply modern machine learning techniques to solve real-world problems with data. He is the creator of three Python packages: StructureBoost, ML-Insights, and SplineCalib. In previous roles he has served as Principal Data Scientist at Clover Health, Senior VP of Analytics at PCCI, and Chief Mathematician at Guardian Analytics. He has taught at numerous institutions including UC-Berkeley, Brown, USF, and the Metis Data Science Bootcamp.

Open Data Science




Open Data Science
One Broadway
Cambridge, MA 02142

Privacy Settings
We use cookies to enhance your experience while using our website. If you are using our Services via a browser you can restrict, block or remove cookies through your web browser settings. We also use content and scripts from third parties that may use tracking technologies. You can selectively provide your consent below to allow such third party embeds. For complete information about the cookies we use, data we collect and how we process them, please check our Privacy Policy
Consent to display content from - Youtube
Consent to display content from - Vimeo
Google Maps
Consent to display content from - Google