Sahand Sharifzadeh, PhD
Research Scientist at Google DeepMind
Sahand is a Research Scientist at Google DeepMind, where he works on multimodal large language models. His research interests span a wide range of disciplines, including AI for Science, Natural Language Processing (NLP), Robotics, and Computer Vision. He has co-authored numerous impactful papers published in venues like Nature, NeurIPS, and ECCV. Currently, his research focus is on enabling self-improvement capabilities in multimodal large language models. Sahand earned his doctorate with the highest distinction from Ludwig Maximilian University of Munich. Inspired by Jean Piaget's cognitive psychology theories, Sahand's doctoral research explored the importance of modeling top-down processes for scene understanding.
All Sessions by Sahand Sharifzadeh, PhD
Synthetic Data: Improving Multimodal Models using Multimodal Models
<span class="etn-schedule-location"> <span class="firstfocus">LLMs</span> <span class="secfocus">All Levels</span> </span>The development of high-performing Visual-Language Models (VLMs) is often hindered by the laborious and costly process of creating large-scale, human-labeled image-caption datasets. In this talk, we discuss a novel approach, Synth^2, that leverages the power of Large Language Models (LLMs) and image generation models to generate synthetic image-text pairs, accelerating and enhancing VLM training. Our method utilizes a pretrained text-to-image model to synthesize image embeddings from captions generated by an LLM. By harnessing the generative models’ ability to create novel compositions, we produce synthetic text and image embeddings that extend beyond the confines of the original training data. Through extensive experiments, we will demonstrate that a VLM, fine-tuned on synthetic data, achieves performance comparable to models trained solely on human-annotated data, while requiring significantly less data. This approach not only addresses a major bottleneck in VLM training but also opens up exciting possibilities for developing self-improving multimodal models. In this talk, attendees will learn: * How to leverage LLMs and image generation models to create synthetic image-text pairs for VLM training. * The impact of synthetic data on VLM performance and data efficiency. * Key insights into the importance of semantic diversity and balance in captions for optimal downstream performance. * The advantages of synthesizing images in the image embedding space over the pixel space.
Synthetic Data: Improving Multimodal Models using Multimodal Models
<span class="etn-schedule-location"> <span class="firstfocus">LLMs</span> <span class="secfocus">All Levels</span> </span>The development of high-performing Visual-Language Models (VLMs) is often hindered by the laborious and costly process of creating large-scale, human-labeled image-caption datasets. In this talk, we discuss a novel approach, Synth^2, that leverages the power of Large Language Models (LLMs) and image generation models to generate synthetic image-text pairs, accelerating and enhancing VLM training. Our method utilizes a pretrained text-to-image model to synthesize image embeddings from captions generated by an LLM. By harnessing the generative models’ ability to create novel compositions, we produce synthetic text and image embeddings that extend beyond the confines of the original training data. Through extensive experiments, we will demonstrate that a VLM, fine-tuned on synthetic data, achieves performance comparable to models trained solely on human-annotated data, while requiring significantly less data. This approach not only addresses a major bottleneck in VLM training but also opens up exciting possibilities for developing self-improving multimodal models. In this talk, attendees will learn: * How to leverage LLMs and image generation models to create synthetic image-text pairs for VLM training. * The impact of synthetic data on VLM performance and data efficiency. * Key insights into the importance of semantic diversity and balance in captions for optimal downstream performance. * The advantages of synthesizing images in the image embedding space over the pixel space.
