Basic Theory and Practice of Fine-Tuning LLMs with PPO and GRPO

Abstract: 

This session will be a mixture of the practical and the theoretical. Practically, we will use common libraries for fine tuning LLMs - primarily the libraries from the transformers ecosystem, such as datasets, peft, and trl - to improve a pre-trained and instruction-tuned model’s ability to correctly answer questions on a dataset commonly used as a benchmark of chatbots’ reasoning ability. We’ll also cover how the reinforcement learning-based algorithms we use for this fine-tuning - PPO and GRPO - work under the hood, covering how the paradigm of reinforcement learning is applied to fine-tuning LLMs along the way. Finally, the presenter will share some best practices for conducting such experiments in a Google Colab environment.

Bio: 

Seth Weidman works at SentiLink, which uses variety of machine learning and data engineering techniques to prevent fraud for banks and lenders. Seth has worked there for five years, starting out as a machine learning engineer, now working as a product manager, along the way building much of the company’s core technology alongside other early engineers and the founders. Prior to SentiLink, Seth worked as a data scientist at a variety of companies, including Meta (then Facebook) immediately prior to SentiLink.

Seth is self-taught in machine learning. In 2016 got interested in how neural networks work, eventually coding them up from scratch. He then started writing about them and speaking at Meetups and conferences such as ODSC, culminating with writing an O’Reilly book both showing how to implement basic neural networks from scratch and some of the math behind why they work. Lately, with the explosion of LLM-powered apps such as ChatGPT and Grok, he has re-started writing and speaking about this subject, with a particular interest in algorithms such as GRPO.

Open Data Science

 

 

 

Open Data Science
One Broadway
Cambridge, MA 02142
info@odsc.com

Privacy Settings
We use cookies to enhance your experience while using our website. If you are using our Services via a browser you can restrict, block or remove cookies through your web browser settings. We also use content and scripts from third parties that may use tracking technologies. You can selectively provide your consent below to allow such third party embeds. For complete information about the cookies we use, data we collect and how we process them, please check our Privacy Policy
Youtube
Consent to display content from - Youtube
Vimeo
Consent to display content from - Vimeo
Google Maps
Consent to display content from - Google