Abstract: Let's face it—vector searches alone just don’t cut it anymore and integrating graphs with RAG processes has demonstrated clear benefits in improving the accuracy and explainability of GenAI. By enhancing vector searches with global enrichment and domain-specific grounding, graphs provide a powerful framework for tackling messy data. So it’s no surprise that the GraphRAG bandwagon is getting pretty crowded these days—from academic papers to GitHub repos that multiply like rabbits, everyone's jumping on board. However, as anyone who’s worked with graphs knows, they require careful iteration and refinement to deliver the best results.
While the current GraphRAG crowd is obsessing over text and lexical graphs, they're missing the elephant in the room—all that juicy non-text data just sitting there untapped. Images and audio are rich with latent signals that are often ignored, despite their ability to add significant context to AI workflows. The next big leap for GraphRAG lies in fully integrating these multimodal data sources, creating a more comprehensive and explainable approach.
In this session, we’ll introduce Multimodal GraphRAG (mmGraphRAG)—a cutting-edge framework that combines the semantic richness of images, audio, and text with the contextual reasoning power of graphs. This transformative approach enables an associative intelligence that moves search and analytics closer to the way humans abstract concepts and connect ideas.
We’ll guide you through the high-level architecture of mmGraphRAG and explore how associative intelligence drives its capabilities. By the end of this session, you'll either be a mmGraphRAG convert or you'll have at least collected enough buzzwords to sound impressive at your next team meeting—either way, it's a win!
Join us to explore how integrating multimodal data into GraphRAG can transform the way we approach search, analytics, and AI-driven insights.
Session Outline:
1. Mulitmodal GraphRAG: Understanding the process and capabilities. (COCO dataset for examples)
2. The integration of vision and audio model with graphs. (OpenAI CLIP, LLaVA, LanceDB, Kuzu database, and other tools)
3. Architecture of mmGraphRAG: How to incorporate different capabilities and agents in a workflow. (Focus on BAML for orchestration)
4. Associative Search: Walk through of Jupyter Notebook that creates the embeddings and then enables concept search by association.
Learning Objectives and Tools:
What You'll Learn:
Expanding GraphRAG with Multimodal Data: How to incorporate images and audio into GraphRAG workflows to enhance understanding and context.
Designing Multimodal Graph Schemas: Tips for customizing graph schemas and enabling multimodal searches that integrate visual and auditory elements.
Hands-On Techniques: Practical guidance on creating embeddings from images and audio using interactive notebooks designed for experimentation.
Future Applications: Insights into extending graph pattern analytics to different types of data for deeper contextual analysis.
Pre-requisite and Background knowledge needed:
Basic understanding of graph technology, vector search, and GraphRAG. We will be using Python, Ollama, and introducing the audience to BAML.
Bio: Amy Hodler is an evangelist for graph analytics and responsible AI. She’s the co-author of O’Reilly books on Graph Algorithms and Knowledge Graphs as well as a contributor to the Routledge book, Massive Graph Analytics and Bloomsbury book, AI on Trial. Amy has decades of experience in emerging tech at companies such as Microsoft, Hewlett-Packard (HP), Hitachi IoT, Neo4j, Cray, and RelationalAI. Amy is the founder of GraphGeeks.org promoting connections everywhere.



