NODES AI 2026 - The GraphRAG System That Asks Back: Intelligent Probing with Contextual Answers

This talk focuses on retrieval augmented generation (RAG) and its implementation at Cisco to improve customer query responses in critical infrastructures like networking and routing. The speakers emphasize the importance of understanding the question before generating answers, as vague inquiries can lead to inadequate responses. They discuss their work on an Assisted Intelligent Dialogue (AID) system that addresses ambiguity by using a knowledge graph to clarify customer queries. Through a series of probing questions, this system guides users to specify their issues accurately, thus enabling more targeted answers. The speakers demonstrate the effectiveness of their approach through examples and share insights on deploying such systems in production environments, highlighting the significance of well-structured knowledge graphs.

All right, thank you. Um, hello everyone. Hopefully you guys can see my screen. Give me thumbs up in the chat if you are seeing it. With that being said, I would like to start with a very near and dear topic to us and which we at Cisco are going to talk here is we have been using and talking about graph rag. It's not a new thing. It's

been there for months and years now at this point. But what we also realized while playing with it is before we can generate an answer for a customer question, we need to better understand the question. Especially when we are talking about a critical infrastructure which Cisco provide with routing and networking and all that, it becomes that much critical that we don't provide wrong answers. And with that

being said, probably say say something in in in the chat message if you have felt this. When you ask question in either chat GPT, Gemini, or any other system, sometime you ask a question saying, "Hey, I want an answer for this." But you realize, "No, no, no, that's not enough detail. I need to provide more because the answer was not at all what you asked for." So,

in in in a sense, I'm a big fan of Harry Potter and there's a good statement in one of the movie series, there's the room of requirement. It's like, "If you have to ask, you will never know, but if you know, you you need only ask." which is almost like you actually need to know what to ask to get the right answer. And that's exactly what we

felt with the system which we started building. It's like, "If you don't know what to ask for certain version, certain problem, giving the right context, the answer does not actually reflect what you're even looking for. And that's exactly what we came here and and wanted to share our experience in how we have tried to solve That being said, just to understand the audience here better, I have

three questions for you. Please comment in in this space and let us know like how many of you have I'm sure at this point lot of you have created rag based application. It's a few line of code. There are so many libraries which will help you do it. Then have you played with the classic naive rag kind of system and the graph rag already? If you have

please respond in the chat. And on top of it, when you were building this, how many of those have actually gone to production? Have they been used at scale where you are getting a feedback from the users and the customer and who are basically using it like this works, this doesn't work. So please share in the chat because that will help us out to figure to make

sure we are covering the right thing for you. Now, I know most of you already know this, but just wanted to make sure this is like a quick just. Rag when I say retrieval augmented generation is more simplified version is here, right? Which is basically two layer when we talk about rag. One is injection where you take a document or documents, basically convert and split into them

split into multiple chunks, embed them and store in any sort of vector store. And then when the retrieval happens somebody asks a question, there is a similarity search which happened with those vector and then a prompt gets added, goes to LLM and finally give you the answer. Very simplified version, but hopefully it gives a some idea when we say that what what we are referring. Now with

that being said a quick introduction here. So I'm Vivek. I work in Cisco customer experience center. We have been working on specially rag and have created certain solution which is now being used thousands of time a day and also we have learned over the period of time what we thought will work and not and based on that we were able to give some generate some paper research

paper as well as patent over to you Raj for quick introduction before I proceed. Yes thank thanks Vivek. So hey guys I'm Raj. I'm also from the customer experience team at Cisco. So we have been working specifically on rag graph rag knowledge graph from from a long time and we have been solving some of the problems where customer is asking vague questions and a lot of the

state of the art rag systems were not able to handle that. So one of the project that we had in Cisco was called AID which is assisted intelligent dialogue. So in this session you will understand why that system is called Over to you Vivek. Thank you. Thank you Raj. With that being said I think agenda is very straightforward. We are going to convince you why the classic

rag has created so much problem at least when we dealt with why ambiguity to the customer or user query so essential to clarify before we even generate the answer. How does knowledge graph fit then? What is the overall loop and actually it's not just going to be a talk a live demo of how we have achieved that and some of the lesson learned which we would like

to share with you all which can help you to build your own system. But before I deep dive on the things I do want to call out that rag is good if you are doing in classic rag and then the basic rag all is good when you're doing small level of in deployment. When I say small level it might mean thousands of document with very specific domain

stuff, right? However, if I take a challenge like we did in the Cisco side we We so many product and solution how customer deploys those and that multiplies the way how customer even ask those questions. So for example, a simple question is how do I upgrade my Cisco switch? Now the commands for upgrading a Cisco switch or any device will differ based on the versions and we

have documents across many versions which gets published. So if you ingest all those, there's so much overlapping information and you almost have to do a needle in the haystack between all these documents for the right version to provide the right detail. Similarly, if I have to give say hey, give me the steps of configuring something in this case a calling in Webex. You have to provide all

essential steps. So it there need to be a complete answer. If you miss something, there might be a problem that customer might not even complete the configuration or if you miss in between, that itself is a big deal. On top of it, if if I bring another perspective which I think is very near to us and that's the motto of the session is how do you schedule

a meeting? Like from are you going to schedule meeting from your Webex app, Android or maybe something else, right? Or some sort of other integration. You need that clarity to give the right information. Otherwise, the answer will become very generic. And the other example which I think happens quite often to us coming from customer experience is there is a long line where customer brings up is like,

does this match with something or can can how any steps to solve it? And if you have to take those and and if there are images in lot of those document, how do you basically bring that analysis together? Now think about these are just sample how you can take this user query giving the wrong or incomplete answer and impact of it and there are many more causes

for it. And multiply that when you are building aid-like system for an organization. So that becomes a very big problem which we uh targeted to solve with aid and that's the whole idea of the session. And we're going to focus on one specific problem so we can explain as as we navigate through the journey and take you through the journey of how we have identified the challenges

and and direction to fix it. So, I'll take a simple example which we are going to continue driving. So, a meeting is slow. That's like a user query. But, what exactly is slow? Slow to join a meeting? Slow to start a meeting? Is there slowness because when you are trying to share a screen, then that's what is slow? Or there's a delay in audio video? Like there's

so many interpretation of this question. And before even asking the right like before before we even ask the right question, the answer has very high chances of being incorrect or not relevant to the user. So, when we're looking into this problem, how do we solve it? At this point, I know we we have so many good reasoning models. But, that is also a costly business if you

continue doing thousands of those loop over and also add latency on top of it. So, how do we actually get the right context added based on the user query so we can actually give the right answer? And before we even give the right answer, how do we direct and nudge our users who's asking this question to say, "You know what? You asked that question, but I need

XYZ information before I can actually give you the accurate details." Now, how do we address these problem? And actually, before we addressing like let me divide this whole problem into three questions. So, what we need? We need an in clarifying engine, a clarification engine, a probing system which need to understand based on the user query what format of the domain and and basically convert that into question,

right? And on top of it, when you are converting that to question, how do you know what is the nearest part which is related to the customer query so I can actually make a follow question out of it. And at the end how do we know when to stop? If if you keep bringing the question repetitively obviously the customer and the user will get frustrated and you

don't want that to happen. So with these things keeping in mind how we have solved the problem and that being said over to you Raj to take it all forward with the solution. Yeah. Thanks thanks thanks Vivek. Let me share my screen. So so before we go to the solution so let's let's let's take a step back and let's try to brainstorm this. So this is going

to be a brainstorming exercise from now on. We we know the problem. We we have figured out that when customer queries are ambiguous no matter what state of the art rack system or any model we use will not be able to you know get the right answer. So we need to design the system. So as part of this we will do that. So So let's let's try

to understand how do we solve this problem now? So should we So So the problem was WebEx is slow but I don't know which like whether it was lagging this whether it was slow in joining a meeting starting a meeting all those things. So should we retrieve like everything from our rack system or vector database and then create a probing question to the customer. Do you think

that's a feasible option? Probably not. Then okay if that is not the way to generate a probing question how do I generate it? Should I just have some random prompt based probing mechanism like every time a customer ask go back and ask him three four questions or three more three, four parameters. That works well when your system or your document corpus is very small. But what about

you are talking about a corpus of maybe 10,000 plus documents? You cannot hardcode every permutation combination in your prompt. Yeah, are there any simpler ways of solving that? So let's take a deep dive in each of the mechanism that I have discussed. So the first one I like to call it the top K trap. Now, our problem statement is very clear. We want to use top K

retrieved documents or chunks from our vector database and then create proving I'm assuming everybody is aware of vector database rag. Um, just just just raise just just raise your hand in the chat if you are not aware. Um, I hope my colleagues will help you in understanding that. Um, so yeah. So the first problem comes uh with this approach is you retrieve top K. Let's say we

start with K equal to 50. So I retrieve 50 chunks. But what if the customer's actual intent Think about the same example where Victor is slow and he's thinking about starting a meeting or joining a meeting. Now that that in that particular intent is hiding in 51st chunk. But you retrieve 50 chunk. So how do you solve that? Okay, we can probably increase the you know, we

can have some kind of recursive logic here that hey, we asked that we used 50 chunks, did not work, let's increase 50 to 100. If it doesn't work, let's increase to 200. What you are ending up doing here is you are wasting tokens. You are wasting tokens not to generate an answer, but ask one question and you don't know how many rounds of this question will go

on, right? What if you have two rounds of questions with 200 chunks every time? You have used basically 400 uh chunks just to ask clarify the intent of the customer. That's not a right approach. So, this is the problem. So, your top K results might not contain the true intent of the customer. So, what do we do next? The next thing that comes to the mind is

metadata filtering approach. Again, this is another classic rag technique where you where when you have a lot of documents in your corpus, you try to tag them. You tag them with maybe Webex. Tag some of them as slow. Now, what is the problem with this approach, right? Let's take back that example. I have 10,000 documents. Okay, let's let's let me trim it down. I have 1,000 documents

and 500 of those documents are talking about Webex. And then, there are maybe like 300 documents which is talking about slowness. I have tagged them. Now, what should I do here? Should I retrieve 500 plus 300 documents which is 800 documents and try to correlate them and ask a probing question? What if I just pick up like 400 documents and first try to understand whether the slowness

is about Webex or not and then pull the slowness documents. It's possible that those documents from Webex and from slowness might not even like there will be some documents, there will be some overlap, but there will be a lot of other slowness related concerns which has nothing to do with Webex only. So, how do you solve that problem? So, the problem if you if you see the

direction where we are going, this is not a chunk or like retrieval related problem. The problem here is we do not know the intent of the customer. And one problem we saw in top K was we cannot fetch too many chunks because too many chunks could lead to latency, could lead to token wastage. Um metadata filtering was good, but metadata filtering cannot connect WebEx tagged documents to

um Slowness tagged documents. There would be a overlap between them where WebEx and Slow are there. How do we find that connection? So, we do not need more data, we need relationships. So, that's where we take we we change our approach from a traditional rag based retrieval to a knowledge graph based retrieval. Um so, in the way I'll talk I'll go in detail like why how knowledge

graph is going to solve this. So, let's take let's let's try to design the solution ourselves. so, what what is happening here is if you see the customer has asked a query and we have detected that there is an named entity here, which is probably a slow, and another one is WebEx. So, we enter the graph here. Now, your knowledge graph can not only relate uh you

know, like entities, but it can also act as a decision tree. So, if you see in this example, what is happening when I enter the knowledge graph and I see a node called WebEx, I see there are three paths in this. There is one path which I'm calling it as hypothesis one, which is like related with the some join issue. And if you traverse this path, you

will see that uh there's the immediate depth one node is talking about slow join time. Then there's another path, which is the hypothesis two. And if you look at the first depth one node, which is talking about like a screen share. Similarly, if you go to the third one, you will see poor video and audio. Now, here we take a step uh back and we try to

understand because I cannot traverse all the path, right? So, this becomes a hypothesis disambiguation problem now. So, what we will do here is, we know these entities here, and we know how they're connected to the Webex. Then, I will go back to the customer here. I'm not retrieving any chunk. I'm only looking at depth one, which is just a node. Node is just an entity, which is

probably a word, something like that in the knowledge graph. So, I'm basically using this, and I'm looking at the relationship with Webex, and then going back to the customer, and will ask a probing question. Hey, are you facing something while joining a meeting? Are you facing something while sharing your screen? Are you facing something with audio video quality? And then, the customer will give me the right

path, and that's where my hypothesis from three will collapse into one hypothesis, and that's when I will traverse that particular path and give him the right answer. So, let's let's let's let's understand this as a let's try to be applied AI architects and try to design this system. So, what we are going to do is, we'll have like four agents. First agent we want to call it

as detective, which is basically trying to understand ambiguous queries from the customer and invoke this workflow. Uh, second one is a strategist, which is basically looking at the knowledge graph and doing reasoning here. Third one is the interviewer, which is basically using the reasoning from the strategist and creating probing questions for the customer. And the last one is your expert. So, expert is basically, you know, you

have everything. It's a typical rag agent that retrieves chunks and then gives you So, what happens here? First, when you when you when a query comes in, we basically check whether the query is vague or not. It could be an intent classifier, as simple as that. If it is not a vague query, then we go out to our happy path, which is go and do a rag

retrieval and give the answer. But, if it is a vague query, then we go to the strategist agent, which basically to the knowledge graph, tries to reason there, and figure and basically tries to solve those hypotheses problem. Once it figures out multiple hypotheses there, it will come to the interviewer, and interviewer will ask probing question. It is possible that in this stage only you have resolved it.

But, it's possible that still there are ambigu- ambiguity left. Maybe you started off with 10 hypotheses, you collapsed it into five hypotheses with the first probing round, but then still there are five hypotheses. Do you want to give a very bloated generic answer? Obviously not. So, then you again do another round of probing. Then, after that maybe you turn out it turns out to be from five

to two hypotheses, and then you basically give your targeted answer. So, this is a simple implementation of that uh from an Eng graph. So, if you uh aware of LangGraph, all those agentic workflows, you will be able to relate to that. So, let's let's go through this entire example now that we have been talking about where a customer came to Cisco and saying Webex is slow. How

do we solve So, detective agent comes in, um and then, you know, uh detective agent detects that there's an ambiguous query uh because of the slowness, and it goes back to the strategist agent. The strategist agent is basically going into this uh knowledge graph, and it sees that there are multiple things where um slowness could be possible. Because there are because we are we are we are

we are creating entities based on features. So, the first entity was meeting, second was screen share, third was audio and video. Now Webex from there it can it can basically figure out that there are three different paths where slow neck slow neck could be possible. And then using those entities it goes back and ask a question to the Now you can see that here what then if

you remember the flow from the strategies to be go to the interviewer and interviewer is basically asking now tell can you tell me which part of Webex is slow because I see three different paths in the knowledge graph. So the customer says, you know, I'm facing a problem in joining a meeting. Now if you look back here when a customer said it's about joining a meeting, I

do not need to care about this two paths. It becomes a simple happy path now and I can just go and do a targeted retrieval and give a more targeted answer to the customer. So let's do it let's quickly go through a demo so that we can see that in real time. So let me open uh for for production system. So this is the Webex chat assistant

that we use in Cisco. Um so let me ask a different question So okay, how do I schedule a meeting? Right? So while I ask this question, I am I also ask that question to Claude which is the state of the art um model right now. I'm using Opus 4.6 extended thinking. So that's the best I could get. Now I ask the I ask the same question.

I need to I need help to schedule a meeting. Can you share the steps? Now if you look at the answer it is more of a philosophical or it's completely different type of answer and I don't blame Claude or Opus in this because this is an ambiguous query. I never said from where I want to schedule a meeting. What device I'm going to use to schedule a

meeting. So, I'm just giving a very generic question. So, the Claude is trying its best to give me a very generic response with some of the targeted things like define the purpose, identify attendees, pick up a location or platform, things like that. Now, let's let's go back to our production system and see what is happening here. So, you see when I ask the same question in this

system, it came up with some probing questions. So, let's see let me put the answers. So, which application or platform you would like to use? This is a Webex data system, but probably I don't want to use Webex. I want to use Microsoft Teams. Let's see what happens there. it doesn't matter. Third, what type of device will you be using? I'll be using mobile device. Okay. So,

it will take some time. So, let's go back to our Claude and try to understand why it failed. I was using a flagship model. I was using a chain of chain of thoughts or like the thinking model thinking mode of that. The problem it can think, but it it doesn't have the visibility in the ingested content is there. So, the problem here becomes not a thinking problem.

Problem here becomes an hypothesis disambiguation problem. You look into the ingested content and then you figure out what to ask from that. That's why it became a very generic response here. Now, going back here, Okay, it asked me another question. So, just for the interest of time, I'll quickly show the one that I had. Um so, if you see here, what really happened here is uh when

I gave that same answer, Teams smartphone mobile device and you know, doesn't care about how many participants or what kind of roles I want, it gave me a very targeted answer. From Microsoft Teams, it even retrieved images from that, and it's a very detailed answer. As a customer, that's what I want. I don't want a bloated answer where I need to apply my brain and try to

figure out from there. So, that's how we are solving this using the mechanism that we discussed. So, going back to our session again. So, what did we learn? So, then we have solved this problem, but we want to share some things with you, like, you know, from our, uh, production deployment here. All those things that you have observed today works fine. It works really good. what matters

is your graph quality. If you do not create that knowledge graph in the right way, you saw in that knowledge graph how it was created, right? Every feature was its entity. Every entity had a relationship in a well-defined way. If you do not have that, you will not be able to solve this problem. You will be wasting proving attempts, and then end up frustrating The second thing

is, it's very important to understand that when to ask a question. It's not that you always have to ask a question. You first have to detect that ambiguity first. And if there's a real ambiguity, and that ambiguity is not a very generic ambiguity detection, that ambiguity detection should happen from your ingested content. What you have ingested and what the customer is asking. You need to have some

correlation there, and that's when you decide that, "Hey, it's possible that customer is asking a very generic question, that, 'Hey, help me decide my next vacation.'" I ingested only WebEx articles, right? So, I should not go back and ask, "Can you tell me what device will you use to, you know, schedule your vacation?" And things like that, right? So, that's why it's very important. You need to

identify the ambiguity in relation to what you have ingested in your knowledge graph. The second is while you are doing this, you will realize that there's a new type of evaluation metrics that will come in the picture. What we call it as dialogue efficiency. Now, why why it is very important? Why normal rag as metrics are not enough? Now, how many probing questions should you ask? And

how can you tell us that the probing was able to disambiguate the hypothesis space from maybe N to one? So, we need to measure those things. That's when we can basically say that yes, this mechanism works. And with all these things, we were able to deploy a production grade rack system in Cisco, which is solving like you saw that like as soon as I go into webex

help.webex.com and I can basically trigger a flow there. Similarly, we're also solving a lot of troubleshooting customer problems in Cisco using this today. Now, key takeaways for us. You should think you remember we started off with assisted intelligent dialogue. So, rag is not a monologue problem. You can convert rag from a monologue using to a dialogue by creating a reasoning loop and creating probing question. And you

should use knowledge graph to basically figure out that. Knowledge graph is not only to find entities and relationship, you can create your own reasoning loop. So, with I would end here. Um, and if you guys um want to connect with us, let me see. You can always connect with us in LinkedIn. Any I I think we do not did not have a lot of time to answer

a lot of your questions. I hope my colleague was able to help, but any questions you have, feel free to reach out to us offline and we'd be happy to help. And and thank you everyone who asked the question. I tried my level best to cover as many answer, but as Raj mentioned, please do connect with us and we would love to have discussion and answer your

queries. Thank you.