Tracing In LLM Applications

Abstract: 

According to a recent survey, 61.7% of enterprise engineering teams now have or are planning to have an LLM app in production within a year – and over one in ten (14.7%) are already in production, compared to 8.3% in April. With a record pace of adoption, the practice of troubleshooting and observing LLM apps takes on elevated importance.

For software engineers that work with distributed systems, terms like “spans,” “traces,” and “calls” are well known. But what might these terms mean in a world where foundation models dominate? Since LLM observability isn’t just about tracking API calls, but about evaluating the LLM’s performance on specific tasks, there are a variety of span kinds and attributes that can be filtered on, in order to troubleshoot a LLM’s performance.

Hosted by Amber Roberts – a data scientist, ML engineer and astrophysicist and former Carnegie Fellow – this session will focus on best practices for tracing calls in a given LLM application by providing the terminology, skills and knowledge needed to dissect various span kinds.

Informed by work with dozens of enterprises with LLM apps in production and research on what works, attendees can learn span types and how to view traces from a LLM callback system and establish troubleshooting workflows to break down each call an application is making to an LLM. The session will explain and dive into both top-down workflows (starting with the big picture of the LLM use case and then getting into specifics of the execution if the performance is not satisfactory) and bottom-up workflows (discovery workflow where you are at the local level to filter on individual spans).

Background Knowledge:

Working knowledge of python would be helpful.

Bio: 

Amber Roberts is ML Growth Lead at Arize AI, where she leans on her years of experience building models as a data scientist and machine learning engineer. Before Arize, Amber was a Product Manager of AI/ML at Splunk and Head of Artificial Intelligence at Insight Data Science. A former astrophysicist and Carnegie Fellow, Amber has an MS in Astrophysics from the Universidad de Chile.

Open Data Science

 

 

 

Open Data Science
One Broadway
Cambridge, MA 02142
info@odsc.com

Privacy Settings
We use cookies to enhance your experience while using our website. If you are using our Services via a browser you can restrict, block or remove cookies through your web browser settings. We also use content and scripts from third parties that may use tracking technologies. You can selectively provide your consent below to allow such third party embeds. For complete information about the cookies we use, data we collect and how we process them, please check our Privacy Policy
Youtube
Consent to display content from - Youtube
Vimeo
Consent to display content from - Vimeo
Google Maps
Consent to display content from - Google