Editor’s note: The authors are both speaking at ODSC Europe 2022 this June 15th-16th. Be sure to check out their talk, “Utilizing Advanced Monitoring Capabilities to Promote Product-oriented Data Science,” there!
Data science straddles the boundary between research and product, in which the latest advances in machine learning, statistics, and computer science are applied toward furthering a business goal. Yet there is a gap here since many data science teams were formed as research teams, with an innovation center orientation. To these teams, building a model is a research project, in which you take a business problem, prepare a dataset, then train, test, and deploy. In this context, the model is considered the end product, and monitoring the model in the context of the business objective it was created to serve is neglected. While the researchers move on to the next project, they lack the tools and practices to assess how a model is faring in production and whether it is ultimately achieving what it was built to do.
The result is a gap between data science teams and the product/business. There exists a lack of clarity as to who owns the model in production—is it the engineering team, the data science team, or the business folks? Since there is no centralized oversight, monitoring typically only happens in response to customer (or other business stakeholder) complaints. The problem with this is that issues are only identified once they’ve already had a deleterious impact on business KPIs, and even then, a lack of proper root cause analysis capabilities means that the underlying causes can take ages to find. Because of problematic experiences resulting from these issues, leadership often considers AI to be a risky investment and sidelines it in favor of more predictable endeavors.
What then is the solution? The key is to form and nurture product-oriented data science teams, where the entire AI-enabled system is considered the end product, not just the model. In this system, the model is evaluated according to the business value it achieves in production, not merely its performance on the validation and test sets. In this context, the teams are evaluated on their ability to deliver and sustain superior business outcomes. With this mindset, monitoring AI-driven workflows in production environments becomes a critical enabler to achieving the business objectives of the program.
Enabling this pivot to a product-oriented approach necessitates imposing a philosophical shift on the DS/AI team, where end-to-end accountability for the model performance is required. However, introducing such accountability means that the data science team must be empowered with the software engineering/product manager practices, ops and engineering resources, and tools it needs to get the job done. The core foundation underlying all of these is proper monitoring capabilities which alert the AI team when things start to go wrong and provide enough insight to allow them to quickly identify and resolve the issues.
As a case in point, our work with Fiverr’s data science team exemplifies the need for providing data scientists with powerful monitoring capabilities. Fiverr, the talent cloud solution for businesses of all sizes, utilizes advanced AI techniques to provide value in a myriad of business contexts, both in their core product and in its periphery. From recommendation engines, to spam and fraud detection, all the way to marketing enablement, Fiverr’s data science team operates on multiple fronts to provide intelligent solutions to important business problems. In order to make sure their investment in AI bears fruit, Fiverr’s core data science team has long realized that they cannot focus on research alone but must adopt a product-oriented approach to their work. They’ve cultivated a culture in which data scientists take full ownership of the model’s performance, which means not only its performance on the test set, A/B test, or the offline environment but also its behavior and performance when running in production. A primary tool used by Fiverr to enhance and accelerate their AI development efforts is Mona, the proactive AI observability platform. Mona allows Fiverr’s team to automatically collect data regarding model performance in relation to business outcomes and be proactively alerted when AI systems underperform or misbehave, before the downstream business KPIs are affected.
To illustrate a real-world example, we will discuss how Fiverr is able to utilize advanced monitoring to promote product-oriented data science. One of the things Fiverr is using Mona for, is to monitor training data each time a model is re-trained. In one of Fiverr’s models trained on buyer reviews, their data science team noticed a slow decrease in the number of new training samples available each week. Were it not treated properly, this could have led to a decrease in model performance and inaccurate predictions. Mona was able to help the data scientists identify that an internal platform switch from an old reviewing format to a new one was causing the slowdown in the generation of training examples. Because the new reviews were not in a format that the original model was trained on, they were being ignored. While no team in particular was at fault, advanced monitoring allowed the data science team to uncover a disconnect between the platform (change in review format) and the model. Because this problem was discovered early, it was able to be remedied before business KPIs were impacted.
In our upcoming talk at ODSC Europe 2022, Itai Bar-Sinai (CPO, Mona) and Gal Naamani (Senior Data Scientist, Fiverr), will dive deeper into the concepts of product-oriented data science, discuss in-depth the key factors in implementing such an approach successfully, and show several real-world examples of how utilizing advanced monitoring capabilities enabled Fiverr’s data science team to be accountable for their AI in production.
————————————————————————-
SPEAKER BIOS
With over 10 years of experience (Google, AI-focused startups) with big data and as the CPO and head of customer success at Mona, the leading AI monitoring intelligence company, Itai has a unique view of the AI industry. Working closely with data science and ML teams applying dozens of solutions in over 10 industries, Itai encounters a wide variety of business use-cases, organizational structures and cultures, and technologies used in today’s AI world.
Gal Naamani has been working as a data scientist for 4 years, with the past 3 years being at Fiverr. As the Senior Data Scientist, Gal works closely with developers, analysts, product managers, and business owners on growth opportunities and new ideas, from research to production. Gal currently has leading roles in projects that are focused around search engine ranking, promoted ads, online bidding optimization, exploration-exploitation problems, monitoring, and more.
