Reproducibility FTW: Collaboratively Solve a Hard LLM Problem, Live!


Unleash the full potential of your machine-learning models and LLMs! We will explore how to do reproducible research and leverage tools like Weights & Biases to achieve this. We will demo some cool fine-tuning projects, the metrics, and advanced features you can instrument to get the most out of your compute. For the hands-on part, we will tackle a thrilling “words” problem collaboratively while introducing our new LLM tracing tool: Weave. We will work together on improving a baseline solution, discovering a seamless way to innovate and collaborate!

Session Outline:

- The importance of Reproducibility in ML workflows: Embrace tooling
- How W&B can help you here, especially for LLM fine-tune jobs
- I have a fine-tuned model; now what?
- Showcase of Weave: Our brand new LLM tracing framework. Keep an eye on all your API calls, RAG retrievals, function calls, etc...
- Let's use this tool and iterate to solve a collaborative problem together. Students will get their hands dirty and iterate over a base solution to improve on a live leaderboard.

We will use:
- Weight & Biases platform: Models and Weave
- OpenAI or another API vendor will define it.
- Probably tools like Pydantic, instructor, outlines to get structured output from the LLMs

Background Knowledge:

Python, used models on API, some knowledge of the LLM ecosystem.


Thomas is a Machine Learning Engineer with ten years of experience. Maintainer our examples, building integrations, and supporting the community. He has become our reference person for LLM fine-tuning inside W&B, taking every new model out there for spin as quickly as possible.

Open Data Science




Open Data Science
One Broadway
Cambridge, MA 02142

Privacy Settings
We use cookies to enhance your experience while using our website. If you are using our Services via a browser you can restrict, block or remove cookies through your web browser settings. We also use content and scripts from third parties that may use tracking technologies. You can selectively provide your consent below to allow such third party embeds. For complete information about the cookies we use, data we collect and how we process them, please check our Privacy Policy
Consent to display content from - Youtube
Consent to display content from - Vimeo
Google Maps
Consent to display content from - Google