Abstract: As the demand for specialized language models grows, researchers and developers often rely on large, computationally expensive models. However, this approach can be prohibitively costly and inefficient. Fortunately, small language models (SLMs) offer a promising alternative. When combined using the open-source Mergekit library, SLMs can yield high-quality domain-specific models that rival their larger counterparts. This technical session will highlight the benefits of using SLMs as the foundation for your domain-specific models and provide attendees with a comprehensive understanding of the model merging process. Along the way, we will dive deep and demonstrate some of the most popular algorithms available in Mergekit. We'll also discuss how the merging process fits neatly into the larger workflow of language model development, situated between pre-training and alignment. Whether you're a seasoned researcher or a developer looking to stay ahead of the curve, this session will equip you with the knowledge and expertise needed to harness the power of Mergekit and create high-quality domain-specific models cost-efficiently.
Background Knowledge:
Transformer models, PyTorch, Hugging Face libraries
Bio: Julien Simon, the Chief Evangelist at Arcee.ai, is dedicated to helping enterprise clients develop top-notch and cost-efficient AI solutions using open-source small language models. With over 30 years of tech experience, including more than a decade in cloud computing and machine learning, Julien is committed to daily learning and passionate about sharing his expertise through code demos, blogs, and YouTube videos. Before joining Arcee.ai, he was Chief Evangelist at Hugging Face and Global Technical Evangelist at Amazon Web Services. He also served as a CTO at prominent startups.



