Shifting Gears to LLMOps: Understanding the Challenges in MLOps for LLMs


With the rise of Generative AI we are increasingly confronted with a pertinent question: what about our MLOps (Machine Learning Operations) needs to change to accommodate LLMs (Large Language Models)? We argue that fundamentally the principles of MLOps are still applicable to LLMs, but the “how” of MLOps changes with LLMs. While LLMs can be used in Classical ML tasks (e.g. sentiment analysis), what complicates MLOps for LLMs is that we see a shift from model-centric thinking to an application-centric thinking. A chatbot application may not only contain the LLM itself but it might use Retrieval Augmented Generation (RAG) with a knowledge base to reduce hallucinations, use a fine-tuning process to adjust the tone of the chatbot, and use plug-ins to execute tasks on a third-party platform. Challenges in LLM evaluation ensue: while in Classical ML we had industry standard quantitative metrics such as root-mean-square error that help assess the model performance, with LLMs we enter an ambiguous space with new methods emerging to evaluate the end-user experience. All these additional components complicate running, tracking and evaluating experiments with LLMs. In this talk, we present a five step process that compares each step of MLOps (discovery, development, evaluation, deployment, and monitoring) for Classical ML with the new challenges of operationalizing LLMs for generative applications. In this talk we focus on LLMs used for generative purposes, such as chatbots. Attendees can walk away with an increased understanding of the methods and frameworks to understand their LLM productionization process, better equipped to tackle the challenges of MLOps for LLMs.


Noel is currently a UX Research for Vertex AI at Google Cloud where he focuses on understanding ways to make GenAI Ops easier for ML practitioners around the world. Prior to this role, Noel worked on Policy Intelligence at Google Cloud Security, a suite of products that make policy management and governance easy to perform. Noel is a proud alumnus of Cornell Tech, the tech entrepreneurship focused campus of Cornell University, where he received his MS in Information Systems.

Open Data Science




Open Data Science
One Broadway
Cambridge, MA 02142

Privacy Settings
We use cookies to enhance your experience while using our website. If you are using our Services via a browser you can restrict, block or remove cookies through your web browser settings. We also use content and scripts from third parties that may use tracking technologies. You can selectively provide your consent below to allow such third party embeds. For complete information about the cookies we use, data we collect and how we process them, please check our Privacy Policy
Consent to display content from - Youtube
Consent to display content from - Vimeo
Google Maps
Consent to display content from - Google