Building High-Quality Domain-Specific Models with Mergekit: A Cost-Effective Approach using Small Language Models

Abstract: 

As the demand for specialized language models grows, researchers and developers often rely on large, computationally expensive models. However, this approach can be prohibitively costly and inefficient. Fortunately, small language models (SLMs) offer a promising alternative. When combined using the open-source Mergekit library, SLMs can yield high-quality domain-specific models that rival their larger counterparts. This technical session will highlight the benefits of using SLMs as the foundation for your domain-specific models and provide attendees with a comprehensive understanding of the model merging process. Along the way, we will dive deep and demonstrate some of the most popular algorithms available in Mergekit. We'll also discuss how the merging process fits neatly into the larger workflow of language model development, situated between pre-training and alignment. Whether you're a seasoned researcher or a developer looking to stay ahead of the curve, this session will equip you with the knowledge and expertise needed to harness the power of Mergekit and create high-quality domain-specific models cost-efficiently.

Background Knowledge:

Transformer models, PyTorch, Hugging Face libraries

Bio: 

Julien Simon, the Chief Evangelist at Arcee.ai, is dedicated to helping enterprise clients develop top-notch and cost-efficient AI solutions using open-source small language models. With over 30 years of tech experience, including more than a decade in cloud computing and machine learning, Julien is committed to daily learning and passionate about sharing his expertise through code demos, blogs, and YouTube videos. Before joining Arcee.ai, he was Chief Evangelist at Hugging Face and Global Technical Evangelist at Amazon Web Services. He also served as a CTO at prominent startups.

Open Data Science

 

 

 

Open Data Science
One Broadway
Cambridge, MA 02142
info@odsc.com

Privacy Settings
We use cookies to enhance your experience while using our website. If you are using our Services via a browser you can restrict, block or remove cookies through your web browser settings. We also use content and scripts from third parties that may use tracking technologies. You can selectively provide your consent below to allow such third party embeds. For complete information about the cookies we use, data we collect and how we process them, please check our Privacy Policy
Youtube
Consent to display content from - Youtube
Vimeo
Consent to display content from - Vimeo
Google Maps
Consent to display content from - Google