Abstract: 

As Artificial Intelligence is becoming part of user-facing applications and directly impacting society, deploying AI reliably and responsibly has become a priority for Microsoft and several other industry leaders. Rigorous model evaluation and debugging are at the heart of responsible machine learning development. Yet, many of the standard practices continue to focus on high-level aggregated evaluation that uses only single accuracy numbers to report model accuracy on large benchmarks. Such practices can be misleading because they hide important failure modes that happen either for unexpected input conditions, corner cases, or specific demographic groups. At the same time, understanding where hidden pockets of errors lie in large data manifolds can be tedious and time consuming for practitioners. This presentation will introduce Error Analysis to the audience, as a tool and methodology for effectively identifying and diagnosing errors in machine learning models, beyond aggregated accuracy scores. The tool provides different views for quick error identification and enables error diagnosis either via active data exploration or model explanations generated using the InterpretML library. We will deep dive into the tool functionalities by presenting case studies and a live step-by-step demo. Finally, we will conclude with a discussion on future opportunities we are considering on further integrations with other RAI tools, as a quest towards a better integrated RAI ecosystem.

Website: erroranalysis.ai
Github repository: github.com/microsoft/responsible-ai-widgets
Practitioner-oriented blog on error analysis: https://techcommunity.microsoft.com/t5/azure-ai/responsible-machine-learning-with-error-analysis/ba-p/2141774

Bio: 

Besmira Nushi is a researcher in the Adaptive Systems and Interaction group at Microsoft Research. Her interests lie at the intersection of human and machine intelligence focusing on Reliable Machine Learning and Human-AI Collaboration. In the last five years, she has made practical and scientific contributions on implementing and deploying Responsible AI tools for debugging and troubleshooting ML systems. Prior to Microsoft, Besmira completed her doctoral studies at ETH Zurich in 2016 on optimizing data collection processes for Machine Learning.