Andreas Kollegger - Agentic GraphRAG with MAGMA Manifold Agency with Graph Models and Augmentation

This talk introduces the concept of magmas in mathematics as a flexible structure for composition, where two elements can be combined using a binary operation without strict requirements. The speaker explains how magmas relate to various scenarios, such as integer arithmetic, combining paints, and functional programming, showcasing how they facilitate the building of complex software systems. By using metaphors like Legos and visual examples, the speaker illustrates the importance of compositionality in programming and software design. He emphasizes that the emergence of new features through composition is a key aspect of development, leading to innovative solutions in areas such as AI and graph databases. Additionally, the speaker discusses the intersection of agents and graphs, positing that agents should be viewed as composable units of software which can enhance AI development and reasoning capabilities.

Guys, it looks fantastic. Good job. And there we go. Okay, fantastic. Welcome to the talk about magma. Thank you for the applause. We we we've done it. This is good teamwork. Magma, I have to apologize. This is an unfortunate acronym. And I said I just like I want to use the word and you'll see why in a second. And so I came up with some definition for

why it works as an acronym. To do with multiple agents in graph. Really, that was my goal there. But for me, when I said magma, what I really meant was magma in the mathy sense of magma. Where a magma is the kind of most relaxed version of a math structure that can do composition. Where you've got basically a pair of things. You've got a set of stuff.

And then you have a binary operation that takes two elements of the stuff and creates another element that is still part of the set. That seems to make sense, right? So, it's the It is the most relaxed because there's no other requirements for the mathematical structure. It doesn't have to have associativity. It doesn't have to have commutativity, distributivity, and blah. Any of the other mathematical requirements that

might come to bear sometimes for binary operations, magma doesn't care. Magma just cares about can you take two things and compose them together in sub for some definition of Looking at some examples. Integer arithmetic is the easiest example of a magma. If you've got the integer one and you combine it with the integer one under some binary operation, you might get the value two. If the And

I'll step out of the way now. If the star if the if the composition operation is addition. So, one composed with one is two. Integers, math, great. You could also take cans of paint. If you had some red paint and some blue paint and some operation for combining them, let's say mixing them together, you still get some paint. Now in a purplish color instead. Magma, the magma

of paint. If you had piles of sand, you could take a pile of sand and you could put it on top of another pile of sand and you still have a pile of sand. Composition yielding something else. The sand of course gets a little bit bigger, but for the purposes of the magma, it doesn't care. It just cares that it's still a pile of sand. Works perfectly

as a magma. Closer to kind of developer terms and like this is a metaphor we use all the time and we always love this and we should love this. If you've got Lego bricks, you've got a Lego structure, even if it's just a single brick, combining Lego bricks with other Lego bricks still gets you a Lego structure. Legos plus Legos still has Legos. Right? Pretty obvious. So

this is a the Lego magma. It's Legos under the operation of connecting them. For functional programming, we know that we can compose functions. If you've got some function F and some function G, both of which take in the same value, same type, let's say, produce the same type, then so they obey this down here. It's some function that takes in an S, produces another S. That's the

set. And if the operation is composing these functions, we know that we get just another function that still takes in an S and produces an S. Function composition. For information structures, things like graphs, which I love, of course. If you just have a graph and you combine it with another graph, and let's say these two graphs have a node in common. When you take a union operation,

so the graph magma operation here is union of the two graphs, you get another graph that has the the same set of nodes, actually the union of the set of nodes, and then the the vertices, I'm sorry, the edges that are all unique. But, it's really the union of both the nodes and the edges. And if you're doing stuff with AI and you're defining an agent, agents

compose also. Here, the composition is a little bit stranger because we probably don't do this this a lot or think quite in these terms. This is closer to thinking about doing um object-based rather than functional-based composition. Where if you had an agent and you combined it with another agent in some way, you still get an agent on the other side. Here, the composition that I'm I'm calling

out is if you had uh supervise as the com- combining. So, now you have another agent that inside has two agents. For agent composition and actually for all these compositions that we're talking through, you can have multiple different ways of actually combining stuff. The whole point of the magma as a way of thinking is that as long as you've got stuff and you can combine them in

some way that still produces useful stuff, you can build all kinds of interesting things. Again, actually I think the LEGO of all of these things, LEGOs are the best metaphor to keep in mind about like the novel ways you can take things, put them together, and get other stuff. And actually I'll explore for a second, you know, why that's interesting. The whole point of magmas is that

they are the most relaxed version of And we as developers do composition all the time. Whether or not you're a functional programmer or if you're doing object-oriented, you know, design as well, we know the object-oriented design patterns are really about how you compose objects together to have other objects basically that do stuff. So, composition for us is normal. This is how we build things. And it is

one of the many ways that we manage complexity. We know that if we've got a problem, we can decompose it because we know how we can reassemble things into solutions. So, either you start with like simple things and compose them together, or you take a big thing and break it down and have individual things that can later be composed together. for the third bullet here then, because

we can specify some way of having agents that can be composed, as software developers, this lets us turn around the whole narrative about agents, which to some degree when you hear people talk about agents, certainly when you hear OpenAI talk about agents, they kind of talk about the big picture agent. So, okay, there's going to be as if you have a another person on your staff that

is able to go off and do things. Maybe it's an assistant, maybe it's, you know, a co-pilot, whatever it might be, that it's like on the human scale of what an agent is. But for software development, now it's just a unit of composition. Now it's the same as a library call. It's a library that we can call that does some stuff, and okay, it's got some qualities

to it, but it's still just lets us build software rather than starting from the point of can we have uh artificial general intelligence running amok doing stuff in the world. This is just how we build software. It's one of the reasons why I'm incredibly optimistic about sometimes as, you know, people talk about that uh AI will ruin software development jobs. Nobody's going to be doing software development

anymore. I don't think that's true. I think instead what is true is that AI is giving us new tools to do software development. As developers, we're fundamentally creators and problem solvers. And we are tool users. Give us some tools, good tools or bad tools, doesn't matter. We'll figure out how to use those tools to solve something. Agents give us a tool that we can use AI in

our toolbox of stuff. And there's one more thing. It was kind of hidden in the earlier examples of Magmas, but Magmas, when you compose that scale, new features emerge that you maybe didn't know up front. So, I've got a table of this that we we can go through. All of the magmas that I called out, when you kind of do them at scale, you realize things that

you might conceptually know, but now you have a way of actually composing your way to it. So, like the addition magma lets you go from if all you knew was the number one and how to combine the one with itself, you can get to all the other integers. For the paint magma, you could get to some of the other colors, but ultimately at scale you end up

at gray. Okay, so maybe that's not the most useful, but you know, somewhere along the way you get lots of different colors and and and that's really nice. Probably works better with with light colors uh colors generated by light, I should say, rather than mixing paint. For the sand magma, you get a desert. Okay, also maybe not quite as useful. But, on the LEGO magma, as we

know if we've worked with LEGOs, you get the Millennium Falcon or a Transformer or a house or whatever it might be. Structures can emerge from composition that are more than just the added, you know, it's the the classic the sum is greater, I'm sorry, the total is greater, no, what is it? I've got the phrasing wrong. Whatever the phrasing is, the thing that emerges is bigger than

just the pieces that came together for it. Okay. Same thing for function composition, if you just start composing functions, ultimately you get a full program. On the graph magma side, when you're composing just two things like two bits of information together, that's really nice. When you keep composing things together, you actually get information structures. And in a graph sense, you can think about like the people here

as a graph. And if we start to become friends, and you've got other friends, well, now I've got maybe two friends that I can become friends of. And now from friends to friends, you have like groups of friends. When you have groups of groups of friends, you have communities. When you have communities of communities, you end up with nations. When you have nations combining with other nations,

you get the European Union. And of course, on the global scale, well, okay, maybe that falls down. But you can see how large societies get explained through like small personal relationships when you look at them at scale. And it's all still built fundamentally on person-to-person relationships. On the agent side, if you've listened to uh OpenAI talk about like the kind of the progress towards general intelligence, uh

one of the levels that they talk about is reasoning. And then past reasoning it's having agents, and then past agents it's innovation or invention, and then way at the top they talk about organized operations, organized behavior. I I think they've got that backwards or at least uh at the top there. Once you get to agentic, because we can compose agents, if you keep composing agents, what emerges

is intelligent behavior. Now, maybe it's not intelligence, but it's very close to intelligence because the complex see what falls out of having lots of agents operate in unison together collaboratively feels like looks like intelligence. And I won't get into the philosophical debate about what is intelligence, but like is at least intelligent behavior. Oh, I should have forwarded there. Okay. Here's a quick introduction to graphs as data

or what are graph databases to help explain how this plays out in terms of like databases. Okay, we all know relational databases. This is maybe not the most elaborate relational schema, but what's important here is that within every relational schema, somewhere within the schema, there's a join table. Here I've got persons and I've got food, and in the middle somehow a join table for the persons to

go to the food. It's a many-to-many relationship that's possible, right? People have lots of different relationships to food. So okay, we've got foreign keys, we know that. So if you had some person that you knew about and you wanted to find out what the relationship was with food, you'd go through the joint table, >> Right. you'd find all the food on the other side. Cool. If you

want to improve the semantics, probably does more than just say, "Oh, this person has a relationship with food." You can specify what that relationship is. It's not just, you know, that they have a relationship with food. You can say, "Actually, this person likes food." These particular foods. And if you look at that and kind of just do a little visual change here, this is exactly what a

graph database ends up being. A graph database formalizes that convention of having a joint table, and optimizes for it on performance and query language and all the other things, so that you can simply say, "Okay, this person likes these foods." You could then have other people that like some of those foods. And then you can have recommendations for, "Oh, person like me who likes some of the

foods I like, likes this other food that I haven't tried yet." And suddenly you can do all kinds of elaborate things with this kind of structure. Behaviors and solutions end up emerging just because you started to connect things in this way. That's the quick intro to graphs. If you want to look at the query language for um there's a query language that we invented called Cypher. Cypher

is inspired by SQL and this idea of doing pattern matching that, for example, in this very small graph, if you want to match people and the food that they like, you'd have a match clause that goes through the pattern of from persons that like food okay, then there'd be some more about like return all those people basically. But this is the basic idea of like this of

Cypher as a query language. Cypher is as if you could do regular expressions over data records. Because this pattern could be just as short as it is now. It could have an arbitrary number of relationships in between. Like, you don't know how many things how many relationships you want to go through to get to another side. One of the examples I've been using here at the conference

that I'll share again, like if you want to put everybody here in the in a relational database, you could ask a question like how many people are at the conference or you know, alphabetize all the people in this conference. But if you want to ask a relational database a question like who's most likely to buy you a cup of coffee? It's an arbitrary number of joins up

until you found somebody that you have some way of having a predicate that says, "Oh, this person buys other people lots of coffee." Once you found that person, that's probably the person that you want to ask about like, "Hey, do you want to have a coffee and talk about some stuff?" I'll come back to this later. So, the other aspect I want to talk about, of course,

is why graphs are interesting for rag and also for gen AI in general and of course then for And the the little sub quote here is really the kind of the the tagline is like, ultimately the reason you kind of move to graphs is it maximizes the surface area of answerable questions and the usable information space that you have. For most rag applications, people start with vector

search. Vector search is a fantastic place to start, but if you have like 10 chunks of things or 10 records, it doesn't matter if it's structured or unstructured data, they're in some vector space arbitrarily, wherever they might be. And let's say that loosely like one chunk represents one piece of information, then you have 10 questions that can be asked of that information cuz you've only got 10

things that you know about, okay? So, if you want to answer more things, you have to add more chunks. Straightforward, right? This is the fundamentals of vector search. Vector search, even though it is um based on vector similarity, it is really just a lookup. It's assuming that the user's question has a very close correlation with somewhere in the thing you're looking up, the what the answer is.

It's a lookup. Fuzzy lookup, but it's a lookup. You could do a little bit better if you um look through all the data that you've got and realize it's connected in some So, now if you do all of the connections, the upper bound that's possible of all the questions that could be answered now is squared. Now you've got a 10 by 10 matrix of like, okay, here's

the chunks. Every combination of chunks, including chunks with themselves. Okay, so it's maybe not perfect. But roughly, it ends up being squared. Now this is quadratic growth. Every chunk that you ends up getting incredibly more powerful for like the number of things you can answer. This is awesome. It gets even better. Cuz once you've connected things, it is easier to then derive collections of things or groups

of things that are related. Cuz now that you have a what we would call a graph topology, you've got a whole network of of chunks that are connected in some way. From those chunks, you can get all of the subsets of chunks that together make sense. Again, in the upper bounds of this, this is where the network effect really takes place. This is exponential. When you started

with just 10, you went to 100, and now you're over to like, you know, 1,000 basically answerable questions, if that's what you're looking for. Every combination, one way that you could treat that is either, you know, at query time, you can go just find the things that are related, or during data preparation time, you can precompute. You can be like, okay, here's all the different interesting sets

of chunks. Let me summarize them. And actually put those maybe in the vector space as well. If you've heard about Microsoft's approach to graph rag, that's what they do. They operate in this part of the spectrum where they say, okay, let's the chunks are interesting, but it's the groups of chunks we really care about. So let's precompute for every interesting subset of of the chunks, let's precompute

a summary, put that into the vector space, and then do the same vector similarity over that. But it's fundamentally a graph operation. So this is the real value proposition. You might imagine, given both the expansion and of what's possible, along the way there's also many opportunities for like how you actually get the data out. It isn't really just a simple, now that you've got a graph, you've

got one thing you can do with that now just gets better and better. Each time you go up the scale here, you have more opportunities for different access patterns for how you approach this. I'll I'll touch on some of them. Um Oh, yeah, just just to kind of drive home the point I suppose, this is what the curves of course look like. I mean, this is it's

not uh a perfect grade. It's, you know, 0 to 10 on the top and 0 to 100 over on the on the Y axis. But, you can see how the growth goes. The linear growth kind of goes out that way, and of course the exponential growth very quickly outstrips what's possible. Yeah, this is literally the value of network effects. Okay, so for Graph Rag, it includes being

able to do vector search. You can still do that. You still start with that. You can take advantage of that. It's one of the operations that's available. But, you can then extend from vector search into doing pattern matching as And you can cover both structured data, unstructured data, and then if you put structured and unstructured together, you can connect those and now you have mixed data. And

once you have mixed data, that's when all the possibilities get even more than what you started with just in that growth curve because now how you combine things, how you think about the relationships between things, goes through the roof. And we're faced with a classic challenge within kind of computer science. Do you pre-calculate things and cache the results and then do a lookup, or do you do

query time resolution dynamic, you know, um finding of things? Okay. Um And I guess this last bullet of course as well, the reason I started with Magma's, Magma's will continue to be in the back of your head as you face any of this stuff. What's great is you don't have to do all of the things at once. You can start anywhere that is useful and then elaborate

over time. It composes. So, you don't have to worry about like, oh my god, there's so much to know. You don't have to know it all. You just have to realize that the upper end is there for you. You can go there when you're ready for it. You're not really kind of stuck with okay, there's this one thing I can do and that's it. I'll I'll I'll

do a quick time check. Okay, now I've got to go quickly through the rest of this. Okay. this is the basic graph rag pattern. Um if we had some chunk that we got through like vector similarity search, right? That's maybe the first thing you did. So, this this chunk two, let's say, had the highest similarity score for the question that was asked. Fantastic. The simplest thing you

can do to take advantage of a graph is find some entities earlier in the data preparation stage. Let's say you found the some entities from that chunk. Like, let's say it's a chunk about me, ABK. Great, you pull out ABK. But, like ABK is related to something else. Well, I'm related to Neo4j, that's the company I work for. So, maybe there's a relationship there. If you have

a question about ABK, knowing that is actually useful. Neo4j may not be mentioned in the chunk that the high had the highest similarity for the original user question, but it's pretty relevant for me if you're asking questions about me and what do I know about? Well, the fact that I work for a graph database company is relevant, even if it wasn't mentioned in the chunk. So, then

you find the related chunks and then you find the chunks, I'm sorry, the related entities and then the chunks that mention those entities. And now, all you've done is expand the context for answering the question with relevant information that wasn't directly correlated with the user's question. I'll say that again. You end up being able to answer a user's question with important, relevant information that was not directly

related, wasn't similar to the original question. That is the key point here. And this is just the first step of many elaborations about how you can take advantage of the graph. And I'm going to try to get quickly through the rest of these since I know there's another talking up in about 2 minutes. So, agents and graphs. You can come talk to me later if you want

to see some of the detail here. We know that we can compose agents. For me, you should think about an agent as a composable unit of software. Do not think about it as an LLM thing. The LLM is inside the agent. It's one of the operational units. It runs asynchronously. It's got side effects. It might not terminate if it runs forever, right? Inside, you've got an LLM.

You've got a prompt. It has a plan that either you've given it or it generates on its own. It has a bit of memory to work with. It's got some tools it can access and it can execute on all that stuff. Graphs appear throughout all of this. I'll let you take a look at the slide real quick and I'm going to go to the next slide so

I can finish on time. And then, because it has all these things, you've got opportunities for deciding how you want to compose. As I mentioned earlier through the magma part of the the whole talk, composition is just one of I'm sorry, there's many ways of doing composition as long as you have sets. There are many legitimate ways to compose things within those sets. So, you can decide

is the memory shared? Is the plan shared? Do these things communicate with each other? On the execution side, do they run sequentially? Do they run in parallel? All those kinds of things come to bear. All of those things, by the way, are the same things we always do in software. What's the scope? Is there shared scope? How do you manage that? How do you manage all the

shared information? Okay. You can compose graphs and agents together. What you get is graph rag. This is a If you're interested in any of this kind of stuff, this gets deep very, very quickly. I spent a lot of time reading research papers. These are some of my favorite papers that cover a broad range of what's possible within graph rag. From graph construction in a multi-agent setting to

doing graph planning using the graph to actually have a reasoning plan, to graph retrieval, of course, and then using graphs as memory. And there's much more than that. There's a site graphrag.com that I maintain. I'd love contributions to that if you have any. Um or you can just go there to see like what I'm looking at right now and what's interesting to me. Okay. And now I've

got to get off the stage. But, the last bit of graph reasoning that matters, hi, I'm ABK, I talk about graphs. If you'd like to talk more about graphs, the answer to the earlier question about who's most likely to buy you coffee is hidden in this reasoning graph. come see me over at booth 6C. I'm happy to talk more. I'm sorry for running a little bit over

time. Cool.