Sahand Sharifzadeh, PhD

Sahand Sharifzadeh, PhD

Research Scientist at Google DeepMind

Sahand is a Research Scientist at Google DeepMind, where he works on multimodal large language models. His research interests span a wide range of disciplines, including AI for Science, Natural Language Processing (NLP), Robotics, and Computer Vision. He has co-authored numerous impactful papers published in venues like Nature, NeurIPS, and ECCV. Currently, his research focus is on enabling self-improvement capabilities in multimodal large language models. Sahand earned his doctorate with the highest distinction from Ludwig Maximilian University of Munich. Inspired by Jean Piaget's cognitive psychology theories, Sahand's doctoral research explored the importance of modeling top-down processes for scene understanding.

All Sessions by Sahand Sharifzadeh, PhD

Day 2 09/06/2024
12:10 pm - 12:40 pm

Synthetic Data: Improving Multimodal Models using Multimodal Models

<span class="etn-schedule-location"> <span class="firstfocus">LLMs</span> <span class="secfocus">All Levels</span> </span>

The development of high-performing Visual-Language Models (VLMs) is often hindered by the laborious and costly process of creating large-scale, human-labeled image-caption datasets. In this talk, we discuss a novel approach, Synth^2, that leverages the power of Large Language Models (LLMs) and image generation models to generate synthetic image-text pairs, accelerating and enhancing VLM training. Our method utilizes a pretrained text-to-image model to synthesize image embeddings from captions generated by an LLM. By harnessing the generative models’ ability to create novel compositions, we produce synthetic text and image embeddings that extend beyond the confines of the original training data. Through extensive experiments, we will demonstrate that a VLM, fine-tuned on synthetic data, achieves performance comparable to models trained solely on human-annotated data, while requiring significantly less data. This approach not only addresses a major bottleneck in VLM training but also opens up exciting possibilities for developing self-improving multimodal models. In this talk, attendees will learn: * How to leverage LLMs and image generation models to create synthetic image-text pairs for VLM training. * The impact of synthetic data on VLM performance and data efficiency. * Key insights into the importance of semantic diversity and balance in captions for optimal downstream performance. * The advantages of synthesizing images in the image embedding space over the pixel space.

Day 2 09/06/2024
12:10 pm - 12:40 pm

Synthetic Data: Improving Multimodal Models using Multimodal Models

<span class="etn-schedule-location"> <span class="firstfocus">LLMs</span> <span class="secfocus">All Levels</span> </span>

The development of high-performing Visual-Language Models (VLMs) is often hindered by the laborious and costly process of creating large-scale, human-labeled image-caption datasets. In this talk, we discuss a novel approach, Synth^2, that leverages the power of Large Language Models (LLMs) and image generation models to generate synthetic image-text pairs, accelerating and enhancing VLM training. Our method utilizes a pretrained text-to-image model to synthesize image embeddings from captions generated by an LLM. By harnessing the generative models’ ability to create novel compositions, we produce synthetic text and image embeddings that extend beyond the confines of the original training data. Through extensive experiments, we will demonstrate that a VLM, fine-tuned on synthetic data, achieves performance comparable to models trained solely on human-annotated data, while requiring significantly less data. This approach not only addresses a major bottleneck in VLM training but also opens up exciting possibilities for developing self-improving multimodal models. In this talk, attendees will learn: * How to leverage LLMs and image generation models to create synthetic image-text pairs for VLM training. * The impact of synthetic data on VLM performance and data efficiency. * Key insights into the importance of semantic diversity and balance in captions for optimal downstream performance. * The advantages of synthesizing images in the image embedding space over the pixel space.

Open Data Science

 

 

 

Open Data Science
One Broadway
Cambridge, MA 02142
info@odsc.com

Privacy Settings
We use cookies to enhance your experience while using our website. If you are using our Services via a browser you can restrict, block or remove cookies through your web browser settings. We also use content and scripts from third parties that may use tracking technologies. You can selectively provide your consent below to allow such third party embeds. For complete information about the cookies we use, data we collect and how we process them, please check our Privacy Policy
Youtube
Consent to display content from - Youtube
Vimeo
Consent to display content from - Vimeo
Google Maps
Consent to display content from - Google