Context Engineering: Connecting the Dots with Graphs - Stephen Chin

This talk provides an in-depth exploration of implementing agentic architectures using graph technology to enhance application development. The speaker discusses the challenges developers face when integrating AI into existing systems, particularly concerning data accessibility and accuracy. He advocates for using knowledge graphs to create a central system of truth that consolidates disparate data sources, allowing for improved decision-making. Demonstrations showcase practical applications of graph databases, particularly in healthcare and finance, illustrating how context graphs can store reasoning and decision traces for better AI performance. The session emphasizes the importance of graph architectures in constructing a scalable, explainable, and effective AI framework.

This is going to be a highly technical talk. We're going to get into the weeds on how you can actually implement some amazing agentic architectures backed by graph technology. I have multiple demos of the technology to walk you through all of this and hopefully you walk away with some practical knowledge you can actually take and apply at your workplace. So, what we're going to talk today a

bit about how you can improve and use um graph technology to power all of your application architecture. And I think the past year, if you've been a developer living through the AI age, you felt kind of like this where we're getting asked by C-level folks, we're getting asked by leadership to build applications with AI technology which can solve the world, right? If If it can help the

um CTO's daughter to do her homework, then obviously it should be able to do a menial task like solve financial analysis or help customer support. But the reality is we we don't actually have the right data and the right context to build AI systems and as a result, you can end up with systems which low accuracy, with not high reliability, and it makes it quite challenging to

to get to production. we have to make a choice. Do we Do we want to stay in our our comfort zone and to just, you know, build applications based on typical databases, um limited data sources, and try to make it work with AI technology? Or do we want to take the leap and do we want to actually build a connected integrated using a context graph and using

a larger data set. So, we all feel this today, You have a bunch of disparate data sources. You might have data locked away in Slack messages, locked away in support logs like PagerDuty, ticket systems, Zoom, emails, all these different disparate systems where the AI doesn't have access to it and it's very, very hard to build an integrated architecture which is going to bring all this together. Or

we could move to a world where we have a central system of truth, a central system of reasoning. This has all of the information on decisions, standard policy information. We capture reasoning logs from the AI systems which are building on top of this. We know what presidents, what evidence is related to this and then we're giving our agentic system all the data, all the context, all the

tools it needs to make really powerful decisions. And this is something which has been trending since the post by Jai and Ashu at Foundation Capital where they talked about the $3 trillion opportunity with context graphs. But, it's also something that our friends at um Gartner have seen as a pattern that a lot of folks are applying and it's entered into the hype cycle for agentic AI. So,

this is the latest hype cycle which just came out a few weeks ago. And you can see not only technologies and and graph technologies like GraphGrid listed here, but also context graphs are on their way up in the agentic hype cycle. So, that's why we're here. We want to be able to apply the latest technology. We want to be able to use this at our work and

to become the superhero of our organization to solve more complex business problems with better data, with better context, and kind of pulling all of these different application sources into a single system of reasoning. Now, the foundation of this is knowledge graphs. Who here in the audience is familiar with what the what a knowledge graph is? Okay, so a few hands went up. You guys are the experts.

You're going to have to help out your peers with learning about knowledge graphs. But to kind of ground set on what a knowledge graph is. So, it represents data unlike a typical relational database which does rows and columns. Knowledge graphs represent information with nodes which represent people, places, companies, organizations, different entities. Each of those has properties. So, in this example, we have um a person, Dan. We

know his his birthday. We know some information about him. But not only properties, we have relationships. So, we know things about the different um pieces of data. So, we know that Dan lives with Ann. And he drives the car which Ann owns. So, we know we know who wears the pants in this relationship. And we also have information on the car. So, we have the type of

the car, the model, the make. Um but we also have an embedding on the car. So, knowledge graphs also allow us to bring together the world of LLMs where we're doing vector search and similarity searches on data to power our AI architectures. And also the physical world where we're bringing together relationships, organizations, people, entities, and things. And by bringing these two technologies together, we're able to power

a much more grounded and refined architecture. Um AI architectures built on graph databases are better grounded. They can make use of knowledge, can use data science algorithms to organize data and to build the And it gives us another angle other than the LLMs where we're doing a similarity search. So, LLMs are very good at this creativity, coming up with responses, kind of answering the next word, but

they're not as good at concrete knowledge, understanding systems, understanding the relationships between different things. And typically, they they don't pull back all the relevant information that you need to solve a complex business problem. So, if we want to move to a world of explainable API, how do we ensure that we get this high-quality production? We need to be able to store this information coming from a different

data sets, different disparate data sets into a knowledge graph. Um AI actually has gotten really, really good at building knowledge graphs for us. So, you don't actually need to populate the knowledge graph and build the relationships yourself. You can use the AI to do that for you. And I'll show a demonstration of how we can take a few different structured document sets, have the AI build a

knowledge graph on top of that, and then use that to power and answer questions using an LLM. It also gives us powerful visualization tools. So, graphs are the closest thing to what we as humans would would draw if we were going to describe a situation. So, you wouldn't you wouldn't for it to build an organizational structure, you probably wouldn't build a row and column database and then

put people's names and put, you know, links between different people, right? That's very counterintuitive. What you do is you draw you draw circles, you draw the relationship of to, you know, who reports to who or what the relationship or what departments interact. And that's very similar to how graphs work. It's also a great visualization of looking at complex data sources. So, we can we can find patterns,

we can understand our data set better if we represent it as a graph and we have that visualization on top of it. And then we can do analysis of the data to improve the performance using things like the Louvain algorithm, which is a community grouping algorithm, it finds clusters of related nodes and then builds a tight grouping of those. K-nearest neighbors or other graph algorithms, which give

us the capability to to find insights on top of the data and do that analysis. So, let's look at an example here. There's a healthcare for a care plan associated with Andrea Jenkins emphysema and we'll first start with the LM direct response. So, what if what if we just asked ChatGPT to answer this question? And in this case, of course, it it you know, has a huge

body of knowledge, it knows about emphysema, it knows about general medical practices for it. So, it can give a very generic response. Something like preventing further damage to the lungs may involve advising the patients to yada yada yada. So, it's it's it doesn't know anything about the particular patient's situation. Now, what you typically do as a first-line implementation, many folks here have implemented a rag system already,

retrieval augmented generation. Okay. Quite a few folks. And this is good, right? You can take dark documents, you can take arbitrary data sources, you can easily load it up into a vector database. But it acts kind of like a black box and the algorithms work on similarity searches and patterns. So, it will find some similar data, not all of it. Sometimes it will find data which isn't

relevant and pull it back. And so, in this case, it recommends respiratory therapy, deep breathing, and coffee exercises. So, like this is a little bit better, it's pulling back relevant information for the patient. Now, what if we did graph rag, which is doing retrieval augmented generation on top of a graph database with additional graph algorithms and the advantage of this is now we can match the patient's

information, their health information, any previous diagnoses which they might have had, and general information about the condition and the treatment plans associated with this. So, it recommends medication management, smoking cessation counseling, pulmonary rehabilitation exercises. So, much better general fit for what you would want to recommend as a doctor, right? So, if we were going to a doctor, we want concrete information which describes our situation, which knows

about us as a patient. And you can see from this it knows a little bit about Andrea Jenkins that probably has a smoking problem, right? That's the the source of the symptoms. And uh may also have had previous surgery which she is rehabilitating from. So, like we have a more grounded answer with more complete information, and therefore what we're recommending is more relevant to the patient. And

we see a lot of people using graph rag and graph databases to solve hard problems in in finance, in healthcare, fraud detection, legal firms are now using graph databases to to kind of catalog very very large databases. A lot of public works projects looking for um contracts or um proposals. So, pretty much anything where you have like this mix of a a lot of internal corporate data

set, um complex structures, things which you can represent as as as um structured data, but also a long tail of documents or other information is a really really good fit for this this type of system. Okay. So, let's talk about how we can actually use this and show a demonstration of this in practice. And what I'm going to show is a system which you can do yourself

to to do graph rag. Um okay, so let's swap to a browser. Now, what I did this morning is I set up a free instance in Aura. Aura is our free cloud version of Neo4j, and you can you can create a free account on this, and you get um a large data set you can you can populate. So, I'm using the free version for this demo. Um

and you can also run queries against it, and you can run it continually. You just need to renew it every 3 weeks. Okay. So, I I loaded up some information to this, and the way I did that was using the Neo4j graph builder. So, I have a couple documents. I have a a bill of material of VEX document, which I've loaded up already. Um for those of

you aren't familiar with with BOM, this is the software bill of materials. Um you can use any sort of dependency management software to um export a BOM from your software project, and it will tell you all the library dependencies, all the different um software dependencies in your project. VEX is a security standard for security exploits, and one of the features of VEX is that it also has

information about whether the vulnerability is applicable to your project. So, you can report that this vulnerability is applicable or what circumstances it can be triggered, which can be quite useful for remediation, where um it may not even be applicable for your code base. Okay. Now, we can take these, and we can visualize the data, which is great. So, um the way this graph got created, which we're

looking at, is it took those documents, it chunked them, and it imported them into Neo4j, and it built a knowledge graph off of it with relationships, with properties. And this is all generated by the LLMs. LLMs are quite good at both generating knowledge graphs, and also you using and building Cypher queries. Cypher is the query language for graph databases. Kind of like SQL is to relational databases,

Cypher is to graph databases. And we can now query and we can pull some information out of this. So, um you can see that we have a whole bunch of nodes here and if we click on a couple nodes, this is part of the bomb. Um this one tells us that it's related to this faster XML Jackson data bind library, so the vulnerability must be in the

Jackson library and we can keep kind of navigating look at the with this, but rather than kind of manually doing this, what we love to do is we love to interrogate it and talk to it using an an LLM. So, we have a an LLM built on top of this as well, which is going to build some um query the um the graph and pulls back some

information. And let's first ask it about something it shouldn't know. So, So, I I'm trying to trick it here. I I I asked it about a library, which I I happen to know is not in the the S-BOM or the vex document. So, if we if we want to build a grounded system, part of that is we don't want it to hallucinate, we want it to actually

reject questions, which are not related to the data set. Um and it it tells me to please please deselect documents. Okay. So, let's let's try this one more time. Okay. Now, we have no documents selected and we can ask it again. Okay. So, we'll we'll see if it if we can trick it and actually get it to respond about something which it shouldn't information about. Okay. This

is good. It tells us that it doesn't know anything about the Jasper vulnerability. But, what if we asked it about a library it should know about, so tell us about Jackson cuz we know it's something to do with Jackson data data binds. And let's see what it can it can pull about the Jackson vulnerability. Um now the way this demo set up, it's it's an open source

project you can you can try this yourself. It's um our LLM graph builder. Um it's built on top of um the same sort of technologies which you would build this here yourself. So we have a Neo4j package for doing um integration with you can see the answer it's giving us back is is fairly detailed. So it's able to pull back a whole bunch of information for this

knowledge graph and give us a a fairly grounded response. So it says it's an injection XML external entity injection attack. It gives us the specific bomb reference to it. Um the justification code not reachable indicates the data flow analysis means it can't be reached. So our code's not affected. So if we're a software development team or a security research team, this is really good, right? So it

not only gave us information about the vulnerability, but it was able to tie that to our project and tell us whether our project's code is impacted by this And this is a more complete response if if you didn't look at the the bomb, you might assume that it's applicable to our incorrectly. All right, and let's hop back to the slide. Okay. so the next topic we're going

to cover is all about memory and we've shown how you can do retrieval on top of graph technology, but it's really really good at something else rather than just querying enterprise information, you can also store and query memory for your agentic application and I think all of us have have made the shift where we're now not using one LLM, we're using multiple LLMs. They're communicating with each

other, they need to store state about the current session with the user. They need to keep long-term history and information about the interactions or tasks which they're doing. And then you also want to be able to keep information reasoning traces and other completed responses so that you can make better decisions on future queries. And there's basically three three types of memory which we're going to talk about

and dig into. So, kind of that short-term memory which allows our agents to communicate and work better together. This long-term memory where it'll be long-running systems where it's more procedural and instructional. And the reasoning memory which allows us to reason about system. So, when we're going through short-term memory, what we want is something which gives us all this conversation storage. So, it stores all of our sessions,

it manages which is the persistence of information in a persistent as graph nodes inside of a graph database. And then we want to be able to do multi-stage extraction. So, combine together tools like like spaCy, gliner, and do LLM extractors to to have different merge strategies for the context which we're building for LLMs. And then be able to resolve entities where we're matching different topics inside of

our knowledge and we're able to pull this information out. So, we have a a package, the Neo4j agent memory package, which has an implementation of short-term memory. And it allows you to integrate this together with your LLM using tools like LangChain where now your agents, if you construct them using LangGraph or another agentic architecture, can communicate and they can store all of the short-term information, extract entities,

and compose this into an architecture. The next memory layer we want to cover is long-term memory. So, this gives us that persistent knowledge of of entities, um relationships, things which are happening over a longer period of time, over agent conversations. And we're basically following a PAO entity model where we're modeling people, organizations, locations, events, and objects inside of our knowledge graph, and then building relationships between those.

So, figuring out how we can connect different stakeholders or companies together, um the knowledge across them and persisting information which relates to putting those temporal relationships together for when these relationships are valid. For example, if you were legal documents, all of the legal documents have a certain time period for which they're they're valid and afterwards they no longer apply due to statute limitations or or different time

periods. So, you can also represent that temporal knowledge inside of the knowledge graph. And then we're able to kind of extract conversations and look across these as we do further agentic queries. And the the last category of memory, and this is something which has really come into a lot of use this year, is reasoning memory. So, besides just having information about the entities, information about the conversations,

you want to be able to store previous decision traces about your data all of that information is captured. And if you if you think about, for example, a customer service use case, customer service agents as um as employees, they learn through a series of exchanges and conversations and different cases, and they they have this repository of previous case history, which they draw upon to make future decisions.

If you don't provide that same context, that same um base history to the LLMs, then they're never going to perform as well as a human would do in the same role. So, we want to construct a system where it's constantly learning, it's adding to its library of of decision traces and information. grows over time into a a more um powerful system that actually has all this compliance

information, that understands previous history, and we can use this even for constructing audit trails or other information that we need to explain why our decision why our system has made certain decisions. Okay, now we've we've talked a lot about different uses of of memory. This is kind of like the in the Matrix where Neo forgets where Neo plugs himself into the Matrix and now he can do

he can do kung fu or he can, you know, shoot a gun. let's look at how we apply graph databases to to this problem. Now, the reason why graphs are really great tool for this is because relationships are first-class in graph databases. So, rather than it being joined across different tables or like a a data structure which is not as easy to traverse. Graphs make it very

easy to do single and multi-hop traversal with very very high performance. So now we can navigate relationships. We can see how we're going from entities, we're going to different um transients, um um limitations, and then figuring out the relationship between previous We can use algorithms for structural similarity as well, like fast RP, to look at the information and do similarity searches and calculate across. We can get

explainable decisions with kind of this full providence chain we've been talking about. Build cross-session knowledge, and we're building this on top of a technology which is production-ready graph databases have been around for over 15 years, used in production systems, are heavily relied upon a whole bunch of different domains. So it's a reliable, proven technology that we can build, we can deploy to production, we can do quite

a lot of amazing things with. And we have an open-source Neo4j agent memory package which is built on top of this. So it does the three types of memory, that's short-term memory, long-term memory, and reasoning memory, where we're capturing all of this information. We provide APIs that your agents can use to to store information, to retrieve information for different use cases. And then combine this together into

a graph database which can be queried either using vector search or it can also be queried using Cypher and graph query languages. So here's an example of how the different types of memory interplay inside of a graph. Um unlike a relational database, you you now have connections and relationships between this. So this is an example of short-term memory where we have a conversation of a bunch of

messages between um the LLM and a person. And then on top of this, we can layer the additional long-term storage. If my clicker works, there we go. Long-term memory where we're parsing people, organizations, and then tie those directly to the message where that activity occurred. And then finally, we can also tie reasoning memory into this. So, for a particular message, what was the decision? What was the

step? What tool calls were called? Those tools access which organizations? And this gives us a more holistic view of our system rather than just looking at the different memory types separately. We now have a holistic view all the memory types interwoven and interleaved on top of each other. So, I'm going to show a demo of how this works with an example application. We built an application called

Lenny's memory. Any Anybody a fan of Lenny's podcast in the audience? Okay, we have a few Lenny's podcast fans. And it's a It's a great program where he's both talking a lot about product management and how you build products and product strategy, but also very in tune with the latest advances in AI from Anthropic, OpenAI, and all the large companies. And so, you learn a lot not

only from Lenny, but from his guests on the insights he's making on where we're going with AI. And it'd be great if we could take all the knowledge in the 300-plus podcasts, bring that into a a single system where we're extracting the entities, building knowledge graph out of it, put a user interface on top of it where we can start to access this, and look at a

bunch of different conversations. And here's a demonstration of Lenny's memory. And um you can go to the website Lenny'smemory.pricel.app and try this yourself. So, here I'm asking it what it has to say about um guests and scaling engineering teams. You can see it has access to a whole bunch of different tools, which it can to extract information from the And besides just getting information from the podcast,

we can also start to ask it questions about locations or or different um properties of the graph. Here it's pulling back locations of the different episodes and using that to get the GPS coordinates and construct a a physical map of where the different conversations are happening. So, this would be very hard to extract manually from this large information set, but using a graph database, storing this information

using the refrigerator agent memory, and then extracting this makes it very easy and simple for us to pull this information and express it. So, now we've we've got a lot of capabilities we can use graphs with. We have we know how to do retrieval graphs, we know how to store our memory as but what we started with was the premise that we want to build context graphs.

We want to build graphs which give us those reasoning traces, which give us those contexts, and can solve business decisions for us where we get higher access accuracy, higher explainability, apply that to different enterprise use So, first of all, let's talk about what a context graph is versus a traditional graph. So, in a traditional audit log, you're really just capturing a record of actions, you're capturing times

that things occurred, and you're not capturing the reason activities occurred. You're not understanding the context of the And this makes it very hard to to reason about why decisions were made or what information was used to come about decisions. So, it's it's not enough to give AI the right context. A context graph explains the why. So, it captures all those decision traces, the causal reasoning of why

different things occurred, the entities, relationships, and events. And it makes all that tribal knowledge queryable. So, rather than having information locked in Slack conversations, locked in approvals, locked in um case studies. Now, you have that information stored in the graph because when a agent makes a decision, the information gets repopulated. And I'll I'll show a demonstration of how that works. Here is an uh architecture diagram of

a context graph system where we're now starting with the search from the user. We're using some memory retrievable tools to um both retrieve from the vector side as well as graph database side and using graph data science algorithms. We feed that information back into the agent loop, which is making decisions. And then the end result gets put back into the context memory, which we can query the

next time we make a result. And by doing this, our system is continually evolving and improving the memory and the decision state, which is able to use to make decisions on subsequent invocations. For a financial services use case, we have a bunch of different entities which we're going to um use. So, people, accounts, transactions, organizations. We'll have events for when the decision occurred, if there was a

rejection, approval, and the context for why this decision was made. So, maybe what policies were applied, what risk factors occurred, and a bit about employee reasoning. And we'll construct a a graph out of this to explain the decisions. Pull in a bunch of different data sources. So, we're going to pull in support ticket systems, CRMs, internal business information. Have access to back-end systems using different MCP tools.

This particular demo has access to 10 different MCP It uses a cloud agent and builds a bunch of open AI API embeddings, which it's going to use for the vector part of the And then it's powered with all this behind on top of a Neo4j database, and then it has a front-end application built on top of this as well. So, it's a Next.js application, a user interface,

and some graph visualization, so we can pull back and we can see some of this Okay, and I'm going to show a a demo of the context graph system. Okay, so the context graph demo, you can try yourself. It's publicly available at contextgraphdemo.vercel.com. Um it has some sample queries. We're going to we're going to pull the Jessica Norris one, who's requesting a $25,000 limit increase and see

see if we want to approve this or not. Now, you can immediately see it's it's making some tool calls, and we can we can see and we can analyze the graph. So, it looks like there was a previous rejection that was associated with this customer. Um and in the tool calls, you can also see that it's querying and making Cypher queries and also getting back JSON data

as the output result and pulling all this in as context for the LLM to help make a better decision. So, we we know about her accounts, we um different trading activity and Okay, this is kind of unfortunate for Jessica. Maybe good for the customer service agent whose job is on the line here. Um and it recommends to reject Jessica. But we need to know why. Like we

can't just um reject her because the AI said to reject her. So, it it found a um previous rejection where she requested a similar sort of um credit increase. A history of fraud where um 14 transactions in 29 minutes. And also compliance compliance violation. So, um different compliance issues and she's a high-risk profile given the um the balance. Some community fraud patterns. And it has a high

precedence, so it recommends with a 92% confidence interval that we should reject her. And the last thing it does is it recommends and asks us if we want to store this back into the context graph as history. So, what we're going to do is now this becomes part of the context graph history. The next time somebody looks and tries to ask a question about this customer or

similar customer account, it can use this as context which it's answering from. And our system is growing and getting smarter over time because it's now connecting reasoning traces and previous decisions and approvals together with the customer information and system information from different systems. we can try a second demo as well. Let's Let's pick another one. Okay, let's do the wire transfer one and it's a $15,000 wire

transfer. What policies apply and if there's past decisions on this size wire transfer. So, um you know, again, if if we're a customer agent and we wanted to better understand how we should think about you know, customers requesting wire transfers like how to respond to the customer if they ask us question about Getting information about previous um usage of wire transfer would help us to And so,

it's again kind of going and doing a bunch of information to query this, it's up to 10 different tool calls where it's getting previous policies and precedents. Um executing some Cypher queries to match. So, um for those of you who didn't raise your hand as Cypher experts, you don't need to be a Cypher genius anymore cuz the the LLMs are probably better than most of us in

the room at writing Cypher. And it can do a really good job of of querying and asking the the back ends. And it it gave us a little bit of advice here as well. Okay, so it says Based on this, here's what applies. So, there's some high value transaction review. So, it exceeds the threshold. If it was under 10,000, it would be easier to get approved. Um

verification you have to follow, AML threshold monitoring, which I have no idea what AML is. And there's previous past decisions where a 15,171 approval was approved and it seems close enough 67.1 confidence score. this is gives us kind of some ground lines and advice for how to think about wire transfers and we might want to ask for more information about um this case before we approve it

as well. Okay, so hopefully I've given you all a all bunch of different things to think about when you're trying to build agentic application architectures using technologies like graph databases, how you do retrieval, how to build memory structures which give you a larger context which allow you to do things, and also building context graphs. Um a great resource to learn more about this and try this on

your own is Neo4j Graph Academy. Graph Academy is free online training, so you can take courses. We actually have a new context graph course which we just released. We also have base level courses on on Cypher, on building knowledge graphs, and also building chatbots in Java, in Python, in JavaScript on top of a graph database using graph rack techniques. So, um this is a great way to

get in the weeds and learn more. I hope you all enjoyed the session today and you're now convinced to take the red pill and take the leap to do graph database technology. Thank you so much for coming to the session on connecting the dots with context graphs. >> [music] >> Oh.