# The Intersection of Graphs and Large Language Models

Anthony Alcaraz, Fribl | AI in Production 2024 | 15:08

Source: https://www.youtube.com/watch?v=m_2y3wwiLT8
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/the-intersection-of-graphs-and-large-language-models
Published: 2024-05-22
Tags: embeddings, planning, rag, search

## TL;DR
- A good retrieval-augmented generation system should improve over time as a data flywheel, with the underlying knowledge becoming better as the system is used.
- Anthony Alcaraz argues that knowledge graphs are a better foundation for retrieval than vector databases because they combine structured relationships with graph-based reasoning.
- Graphs can support LLM applications in three ways: as context providers, as reasoning topologies, and as targets for transformer and LLM-based graph algorithms.

## Summary
Anthony Alcaraz presents knowledge graphs as the foundation for retrieval-augmented generation systems that need to reason over business data. He argues that RAG should be designed as a data flywheel, where the system and its knowledge improve over time. Knowledge graphs provide structured entities and relationships, while graph algorithms add capabilities such as clustering, centrality analysis, and causal reasoning. Alcaraz describes three interactions between graphs and LLMs: graphs can provide context, define a topology for reasoning, or become the object of an LLM-based graph task. He also discusses using LLMs to build knowledge graphs from documents and domain data. His proposed architecture separates knowledge retrieval from reasoning. Retrieval should rely mainly on a knowledge graph and several forms of graph augmentation, while reasoning can use fine-tuned LLMs, NLP models, or classifiers trained on the graph. He connects these ideas to Fribl's work on automating HR reasoning.

## Key ideas
### RAG should improve through a data flywheel
[00:51](https://www.youtube.com/watch?v=m_2y3wwiLT8&t=51s)
Anthony Alcaraz describes retrieval-augmented generation as a system design pattern for extracting value from LLMs in business. He says a RAG system should improve over time rather than remain static. The data and knowledge behind retrieval need a process that automates this improvement. He sees the quality of the data supplied to the model as a future differentiator, especially when that data contains structure and relationships.

### Knowledge graphs combine data with reasoning tools
[03:01](https://www.youtube.com/watch?v=m_2y3wwiLT8&t=181s)
Alcaraz says fine-tuning may increase a model's reasoning capabilities, but knowledge graphs provide a separate set of tools for reasoning over data. These include Cypher queries, vectorization, and graph algorithms. He describes the interaction between an LLM and a knowledge graph as agentic RAG, where an agent reasons about what it retrieved and what it still needs from the graph.

### Graph retrieval can produce more varied and complete answers
[04:20](https://www.youtube.com/watch?v=m_2y3wwiLT8&t=260s)
Alcaraz contrasts graph retrieval with vector search and cites research from Microsoft and researchers associated with MIT. He says graph-based RAG produced more diverse viewpoints per answer in one comparison, with the cited result being 1.8 more diverse viewpoints per answer. He connects that diversity to answers that are more comprehensive.

### Graphs interact with LLMs in three distinct ways
[06:06](https://www.youtube.com/watch?v=m_2y3wwiLT8&t=366s)
Alcaraz identifies three forms of interaction. A graph can provide context for an LLM, with graph algorithms such as clustering helping augment the retrieved context. A graph can define a reasoning topology, as in Graph of Thoughts or approaches using Monte Carlo algorithms. LLMs and transformers can also perform graph tasks, such as finding centrality or influential parts of a graph.

### Reasoning can use retrieval, scoring, and cycles
[07:26](https://www.youtube.com/watch?v=m_2y3wwiLT8&t=446s)
Alcaraz discusses a model called RARE, or Retrieval Augmented Thought Process, which extends the Graph of Thoughts idea by retrieving knowledge for each thought. The LLM scores answers, creating a way to improve the process over time. He also describes combining graph retrieval, graph reasoning, parallel querying, planning, and cyclical graphs. In a cyclical graph, a model can return to an earlier point and try another option after failing.

### LLMs can help build knowledge graphs from documents
[10:16](https://www.youtube.com/watch?v=m_2y3wwiLT8&t=616s)
Alcaraz argues that the cost of building a knowledge graph can be reduced because LLMs can assist with its construction. He mentions turning documents directly into graphs, extracting domain information, generating graph data models from CVs, fine-tuning models for domain extraction, and using an agent system called AutoKG. These techniques allow graph creation to become part of the same LLM and data workflow.

### Graph context retrieval needs several complementary methods
[11:04](https://www.youtube.com/watch?v=m_2y3wwiLT8&t=664s)
Alcaraz lists several ways to retrieve and augment context from a knowledge graph. These include Cypher queries with prior ontology grounding, vector similarity search, graph algorithms, and generative knowledge graphs. His recommendation is to chain these forms of context augmentation rather than depend on only one retrieval method.

### Retrieval and reasoning should be separate parts of the system
[11:45](https://www.youtube.com/watch?v=m_2y3wwiLT8&t=705s)
Alcaraz divides a RAG system into knowledge retrieval and reasoning. He recommends using a knowledge graph for the retrieval side. The reasoning side can use different LLMs fine-tuned for reasoning tasks, as well as NLP models and classifiers trained on the knowledge graph. He presents this separation as a way to choose different models for finding knowledge and making judgments.

### Graph algorithms and LLMs can support causal reasoning
[13:20](https://www.youtube.com/watch?v=m_2y3wwiLT8&t=800s)
In his final example, Alcaraz describes research combining an LLM with a graph algorithm for causal reasoning over a causal knowledge graph. He says the method reduces complexity and lowers LLM usage by dividing work between the model and the graph algorithm. He presents this as an example of using both technologies together as a reasoning engine for business applications.

## Notable quotes
- "I believe that to build a good RAG system you need to think about it as a data flywheel." (01:17)
- "I believe that Knowledge Graphs are the ideal food for RAG system, better than vector database." (01:47)
- "You have the LLM, knowledge graph allow you to get some kind of reasoning and you have knowledge." (03:54)
- "My system, you need to think a RAG system into two parts. You have a knowledge retrieval part and you have a reasoning part." (11:45)
- "I do believe that many, many businesses will be transformed by a proper implementation of RAG framework." (14:18)

## Tools & references mentioned
- Fribl
- Retrieval-Augmented Generation
- Knowledge Graphs
- vector databases
- Cypher
- Graph of Thoughts
- Monte Carlo algorithm
- RARE
- Retrieval Augmented Thought Process
- Self-Discover
- GraphRAG
- Microsoft
- MIT
- AutoKG
- graph neural networks

## Who should watch
- You are designing a RAG system and need to decide whether vector search alone is enough for your data.
- Your application needs relationships, graph queries, causal reasoning, or multiple retrieval strategies rather than simple document similarity.
- You want examples of how LLMs can create knowledge graphs or work with graph algorithms in an applied business system.

## Related talks

- [Graphs and Language](https://mlopstalks.com/talks/graphs-and-language) (Louis Guitton, 11:28)
- [Integrating Knowledge Graphs & Vector RAG for Efficient Information Extraction](https://mlopstalks.com/talks/integrating-knowledge-graphs-and-vector-rag-for-efficient-information-extraction) (Nehil Jain, Stealth AI Startup & Sonam Gupta, AICamp & Matt Squire & Valdimar Eggertsson, Snjallgögn (Smart Data inc.) & Binoy Perera, MLOps Community, 48:47)
- [Managing Small Knowledge Graphs for Multi-agent Systems](https://mlopstalks.com/talks/managing-small-knowledge-graphs-for-multi-agent-systems) (Tom Smoker, WhyHow.ai, 1:04:41)
- [Embeddings and Retrieval for LLMs: Techniques and Challenges](https://mlopstalks.com/talks/embeddings-and-retrieval-for-llms-techniques-and-challenges) (Anton Troynikov, Chroma, 35:19)
- [Retrieval Augmented Generation](https://mlopstalks.com/talks/retrieval-augmented-generation) (Syed Asad, KiwiTech, 44:10)