Agentic RAG in Production: Orchestration, Evaluation & ROI - Rohit Bhardwaj

This talk covers the effective implementation of Retrieval-Augmented Generation (RAG) in production environments, specifically focusing on orchestration, evaluation, and measuring return on investment (ROI). The speaker discusses common challenges faced when transitioning from demo environments to production, including reliability issues and stakeholder satisfaction. They delve into the technicalities of RAG, explaining the importance of embedding models, similarity searches, and the need for structured data to eliminate hallucinations in responses. The session highlights various strategies for improving data retrieval through methods like hybrid search and BM25, and emphasizes the significance of continuous evaluation to measure retrieval and generation quality. Ultimately, the speaker advocates for a multi-faceted approach to RAG that integrates proper data governance, policy adherence, and the utilization of caching for enhanced performance.

So, this is what we are going to discuss. So, as part of this journey, today we are we are focusing on rag in production, orchestration, evaluation, and return on investment. People always ask the question like, "Hey, what is the ROI? How do I measure? What did I do here?" And that's the three things we're going to focus on. So, this link is important if you would like

to have tinyurl/agenticra rag. Okay, agenticragprod is the link. This has everything what you're going to do. These are assignments. These are actual assignments you can do for your company. Okay? So, you can follow that through later on. I'm just going to go through some important points as part of today, so you can now take it and make it actionable for yourself. Okay, everybody ready? Yes. Let's build

the demo first, okay? I built the demo and you know, uh so, what's happening in this is the demo. Most of the demos looks really good, but they die in production. Why is that the case? The demo happens to uh you upload the document, you ask the question, and everything looks good. It's possibilities proven. But, when the decision point come, the production come, there are multiple systems.

Not only multiple system, I found out that some people were saying that there are there are different policies. Who has access to this rag information? Who don't have access to this information? That's also there. And based on that context, there are reliability issues. There are escalations going on and unhappy stakeholders. Stakeholders are not happy and they're not getting the problem solved. This is the real state of

the affair for rag as of now. People understand rag is just calling a similarity search, but let's build it through. Like, you know, we going to I just want to talk about how the rag works first, okay? And then we will try to build this solution, all right? So, what are the means? It means that there is the illusion of demos coming in. what are the means?

It means that you start from ignored latency and cost. No ROI is proven. There is isolated workflow. Demo environment is not clear for us. There were only some questions you are asking, and this is a quality data, small data to work with. What do I need to do to really fix that? Let's talk on this. So, what's happening is So, I'm just going to talk about from

ChatGPT perspective. How does ChatGPT work? You know, that's the first question we can we can ask. How does ChatGPT work? Now, what we are doing is we are creating embeddings, floating point floating point embeddings we are trying to create here. These floating point embedding, as you can see here, is using the embedding model is created these floating point vectors, okay? Based on these vectors, now we are

saying some people are saying meow, woo. These are all called co-located together. These are all sounds of animal. Yeah? Now, that said, what we are doing is we are trying to come up with something uh from our from our perspective here, which is which is correlated correlated actions based on this. So, what we are doing is we are trying to do similarity search based on this result,

which is which is coming here. So, embedding models are coming in, and once these embedding models are coming in, we are able to do text similarity search, text semantic search, and then try to create clustering and regression and other models to solve the problem, okay? So, that's the main main main perspective which we are getting getting from this. Okay, so let me refresh one more time. Uh

for some reason I'm not able to see this here. Okay. So, it used to be coming here for some reason it's not coming, but that's fine. So, what that means is these are vector similarity searches which are there uh which we need to work on. So, that means that the embed an embedding vector is a floating point number. Distance between the two vectors is the related distance.

So, we are doing core similarity search. What are we doing? Core similarity search we are doing which is the vector search. And if you do that core similarity search, we can use this to these embeddings for searching the system. How many here know search? We do search, yeah? We We do search the system. What do we use to for searching? ELK stack, yeah? We use ELK stack,

some other way to do that. Clustering. Hey, these uh items are similar items which are available for me. And then recommendation. I want to recommend you newer product because you bought this product for me. Yeah? Anomaly detection. You want to make sure that there's an anomaly there and you want to fix that anomaly. Fraud detection mode is there and diversity management classification are there. So, why is

this happening? This is happening because we have a clean data. You have messy data. Unambiguous, you know, uh stakes are stakes of questions are coming in. And what do I do if I have so many questions? How do I fix that problem? Multiple complex systems, actions are coming in. They They needs to be more stricter governance, you know? When you get the answer, is this the correct

answer or not? High cost and business value we have to measure and high reliability for all the answers coming to me. That's the real thing. You We are here to solve for this. Okay? In order to do that, you can think about where all What all of the different places we use rag for? Okay? If you start from the right hand side, CRM, customer relationship management solution.

So, what is CRM doing here? solution, it's coming in it and we can ask the question to CRM like, "Hey, how how this data is flowing?" That means there is a relationship between the customer and the order. Before I get the order, we are generating a lead. From the lead, I'm generating opportunity. From opportunity, I'm quotes. How many here generate quotes? You know, you're generating quote or

you're giving quote and like this is what the price looks like and based on that you generate an order. Once the order is generated, you are sending it to the ERP solution. And ERP now sending it to shipper which is going to ship to you because you're going to ship to you, then we have reorder level to manufacturer. We are producing the data from the manufacturer. Some

discuss the whole enterprise architecture here, but think about different areas you will have policies. You'll have different policy for return for a customer, B2C customer. Different policy for return for a VIP customer or B2B customer. Yes? Now, based on that, you have a product catalog, list of products which are available for you and for each product catalog, if somebody buy this product, you might also like to

buy this product, which is called as cross-sell and upsell. Yes? We can do cross-sell and upsell here and we have ticketing system that means you will have issues to solve for, then these ticketing system we need to solve for it really fast. And then there's a workflow there which will automate all these processes for us, which is agentic process which you are trying to create and you

have logs and activities which are coming in. Based on these logs and activities which are coming in, we need to find out most likely this is a fraud. Most likely this customer is not happy. Okay? If they are not happy, there is a churn problem going on. Because there is a churn problem going on, what can we do? We can have an agentic rag created for them

where we get the resources in terms of, "Hey, give them a discount code." Or some way of implementing that solution. That brings end to human escalation and there is a compliance review also done. So, rag in information is all this these are floating point perspective. Now, we all agree on this one. Hallucination is a major problem. Answer credible but unsupported by evidence. Yeah? Weak retrieval. You're retrieving

the data and you're doing the chunking of the data is also done. But the chunking is not done properly. The data is buried in the code. Now, somebody say that, "Hey, I Somebody was saying here, you have a big doc doc 100 uh you know, PDF documents are there and large document they you need to work on that document." So, first thing we need to understand is

that Okay. So, knowledge article works the same way. You create a knowledge article, the first thing what we do is that, "Hey, throw it to the LLM. Let LLM figure it out." Not a good idea. We need an LLM knowledge article to be readable by the LLM. Okay? Machine learnable LLM needs to be knowledge needs to be created. Not human readable knowledge. They do that two separate

things altogether. So, that's where that's where the chunking problem is coming in. Orchestration failure. We are trying to not orchestrate what the user is asking for. And because we are not doing that, we just going to the rag for a solution, that's not going to help. Conflicting contexts are coming in. Now, you're saying there are two different different domains are there. Manufacturing domain and healthcare domain. Somebody

talked about it here. Yes? Raise your hand. You're Yeah, you talked about it. So, you have two different domains. How do you make sure that they are all talking to each other without any problem? That's another problem which is coming. No evaluation loop. Now you are not learning from your data. Thumbs up and thumbs down. Like you know, when when so you get the answer, is it

thumbs up or thumbs down? If it's a thumbs up, you drink it, you know. Easy, you know, I think it's coming up. You know, Tata got or who got that? Uh Huh? Reliance got it this time, yeah. Good. Yeah, so it basically come up again, yeah. So that's what the evaluation is. And no business linkage is there. You're not putting your story to an ROI. What you're

trying to bring in to your perspective. We can create agentic rag, easy. But you're not linking it to ROI. Why are you doing this? That's a problem which everyone faces right now. Now that means the rag failure is not only hallucination and so inaccurate information, but when the inaccurate information come, that's where the retrieval the document relevant documents are not found. And that's where the indexing problems

are coming and the governance is not happening properly. So what we need to do is we need to build this maturity gap and try to come up with multiple strategies. Okay. So what are you here to do? Create multiple strategies. The one strategy is not going to help you, okay? Just like you know, you know, you can create one big Okay, anybody has done like you know,

design, you know, design principles, single responsibility principle? Don't try to do every method in the world in one place. Not a good idea to do it, yeah, same thing here. Multiple systems are there, workflow aware decisions you need to make based on that. And correction and escalation paths needs to be there. So that means whatever the rag is giving in, you need to correct it over time,

course correct it. And that means you need to provide here the governance and KPI accountability at the end. Okay? So this is what we're going to discuss today. So, what all that means from the identical act perspective, the one thing which you have to understand is, you we need to start from measuring, choose the retrieval strategy, that's number one, detect the low confidence. If there is a

low confidence there, then escalate to human. So, that means there should be a way to calculate the confidence score. Don't just give the answer to the user. Say that, "Hey, we will get back to you" instead of giving an answer which is wrong answer or not the probably the right answer. Ask more questions to get the right answer. So, that's the low confidence problem which is there.

And then you escalate to the human if needed, validate the evidence. Is it correct or not? And retry or reroute to a different way of doing it. And measure your quality. So, there are three ways to measure the quality. We'll discuss that more in detail today. Right? So, that said, what we are talking about the pillar perspective is one is the retrieval. How do I retrieve the

data and then work on it? Second pillar for me is reasoning. I need to reason with the data, what the user wants, and then synthesize my retrieval data. Then I need to orchestrate the smooth out the operations, evaluate to improve. How do I improve the data and measure at the end? So, that's what we'll be going over in detail, you know, as part of today's today's discussion.

So, now what's happening is that what we need to do, what we really need to do now. We need to come in with an idea of chunking of the document. How many here know the chunking of a document, you know? You have a PDF document, you're chunking that document. Now, you chunk this document, you embedding of these chunks are there, and run the similarity search and top

K, top 10 values you're bringing in. After you get the 10 values, you want to answer the question through LLM. That's exactly what we all do here. And then the problem here is that they're missing the keyword matches. Now, when we do this are we doing keyword match search? No, we are not doing keyword match search. So, one thing you have to vector similarity search is not

good for keyword search. You have to get the keyword and do that keyword search separately. That's number Number two is no structured data is there. So, do never never go to just unstructured data and search for it. Somebody here was saying that like you're not getting good results. Why are you not getting good results? Let's think about that. Because you're not embed with the structured data. Structured

data means I'm looking for this product for this region in Bangalore. I want to get, you know, my shoes shoes here in 2 days. So, that I gave the context with that. Those are keyword search which are there. Context window constraints are there also. And semantic gaps. I'm not creating semantics for myself. We'll talk about more on semantic perspective, but we're missing the semantic in this one.

And then if I do that, that means how do we do it? Through hybrid search, knowledge re-ranking and quality expression. Once we do that, we create a better rag for ourselves. So, what does that means? From pattern failure perspective, the first thing is first thing is that latency matters. Slow retrieval is going to impact the user. How many year the retrieval take more than 30 seconds? 30

seconds, huh? With re-ranker. Yeah, yeah, yeah. It takes a It takes some time to retrieve the data, yeah. Is it Is it good like user is happy with that? I understand. I understand. I'm talking about user. If the user is happy with that, maybe maybe not. You know, that's what I'm saying. Relationship matters. Relationship. You're relating it to something else. If you're not relating to it and

just go asking an answer is not going to help you. So, we'll discuss how do I create the relationship between entities and then use it. Exact matter specific word matters. Repeated repeated intense, you know, overwhelming with the similar queries. They are always coming in. Freshness matter. A lot of people talked about data is not in my my data is stale. Somebody here talked about the stale data.

These are real challenges which are there. How do we fix that? And identifiers matter. That means unique ID are not recognized. That's that's where the problems are coming in. So, we all agree these are the problems, yes? How do we solve for these problems is from native to production is all coming in from simple chunking strategy to multiple pass to solve the problem. That's what we'll be

discussing today, all right? So, what all that means? It means that these are the five things we'll be discussing. Number one will be we'll be looking at we'll be looking at graph rack. Relationship matters, okay? We'll then be looking at advanced indexing techniques. There there are a lot of advanced indexing techniques. I'll discuss at least five six of them today, okay? And then we'll discuss the dense

hybrid retriever. We'll discuss that also, which combine the lexical search and vector search. Then we'll also talk on the lexical retriever we'll we'll also look at and then we'll also look at the orchestration which happens across all of them. These are the values we'll be discussing. So, that means first we need to identify the query, analyze the shape of the query, create the workflow, risk, and select

the retrieval strategy. What is your retrieval strategy? How am I retrieving this data? It's not just from one rack. We are only building one rack. That's what I'm trying to say we need to fix here, okay? So, these are all the techniques we'll be discussing today, okay? We'll start from BM25, reverse indexing, and then we go to semantic search, structural search, hierarchical search, metadata, and then we'll

look into you know, relational retrievals. Redis retrievals [snorts] will do that. Cache retrievals and workflow related retrievals. These are the main ones we'll look at today. Okay? Okay, let's go through them one by one. So, we're going to start with the BM25. So, BM25 came along, which is doing the relevance search based on you know, lexical retrieval. It will rank the document as based on the relevance,

the relevance which is coming to us. And inverted index, everybody know about inverted index, So, what is inverted index doing? Anyone? What is it doing? Anyone? How we getting the inverted index? You're getting the documents, you're getting all the keywords from the document, you invert the index, and then that means that for this shoes, Nike shoe, that is coming in these five documents, you know, and this

document has five different places that is coming. That becomes more relevant for me. I get that information and then we get this. So, that means what we are trying to do is we are trying to do the order manage order numbers, policy names, product IDs, SKUs. These are all the things which we are doing. So, how are we doing that? Let's take a look at it in

more detail. All right? So, as an architect, you would need to understand, you know, as a Dream Mart it's selling, you know, the Nike shoes endorsed by Michael Jordan. So, Michael Jordan we all know like he's is a greatest player for basketball. So, when he's doing that, he's he comes customer interacting with the agent. So, he's doing the shopping assistant agent. Okay, we're building a shopping assistant

agent, order support agent, and policy agent is also there. So, the core problem is the system currently uses the vector search embedding, okay? And and the observe the failure, Jordan shoes returns unrelated basketball shoes. You see, if I just say Jordan shoes not billing me the right value. Final sale policy is returning the refund policy, generic refund policy, and skew is not found also. That's the real

problem, okay? If you send to one rag for all four of them, would you get the right No, you're not going to get the right result. So, what you need to do as an architect, you need to find out what is your retrieval strategy looks like. If somebody say Michael Jordan Nike Nike shoe trendy, okay? What retrieval strategy should you have? What retrieval strategy if somebody gives

you a skew to work with, skew ID to work with? Based on this, we need to pick either BM 25, which is lexical search, vector search, or hybrid search. Yeah? Once we know that, now we need to design for the inverted index, basically be able to create the tokenization strategy for our self for case and what context we are working on, and then say that like, you

know, how do you index Jordan and Air Jordan MJ? That's what the question you're trying to work with, and then do the BM scoring and do hybrid the strategy at the end. And at the end, then we look at the failure simulation for the same thing. Okay? So, this is what we are planning to do as part of So, let's take a look at it in more

detail. Ah, so this is what we are trying to do. So, Michael Jordan, if somebody say Michael Jordan trendy shoe. This is hybrid, that means vector plus BM 25 strategy is there. What we are trying to do is we are saying brand endorsement, brand plus endorsements is lexical anchors, okay? And then semantic is trending. Trendy is the semantic aspect is there. Based on these two, we are

able to now search for the data and get the data well. Everybody with me? Yeah? That's the way we are getting it. Now, if somebody give us the queue, that is only BM25 because I have no exact keyword which I need to bring in. So, why should I even do vector, like, you know, search? It's just any fuzziness product will be wrong in this case, yeah? So,

if somebody give you a skew, you need to use the exact same name for that skew. Return policy is BM25, which is like semantic feedback which you are trying to get here in this case, and promo code is BM25 only, which you are trying to know exact promo code you are trying to get that value for. And order number status is a direct lookup. So, somebody asked

me a question, "Hey, do I need to do rag for everything?" No, you don't do rag for everything. That is where the people are causing problems. They say, "Hey, we should use agent, we should use rag for everything." No. Based on the intent which you are now trying to create, now you create inverted index for yourself based on the indexing field for product index, policy index. Now,

based on that, now we create this BM25 policy, which is which is stop word removal. That means if somebody's saying that I want to get I want to get Michael Jordan shoes red, okay? What is most important? I want to get is not important. Stop word, remove that. And then stemming, and then once we do that, now we search only the keywords. Michael Jordan is the keyword,

and red is the keyword. We We based on that, we are able to bring the data for us, okay? Now, we got here the no normalization anything for critical tokens. So, any promo code is there, we know exact what code code is there. Don't need to do that, also. Now, we are bringing in the multi-field indexing. We bring in the endorsement, and then based on that endorsement,

we are able to solve the puzzle for us, okay? Any questions on this, first of all, before we move on? Items marked for final sale are not eligible for for return. That's what the document is saying. The example number one is item on mark for my final sale is is is that's what that's what it is. Now, if I use BM25 for this, would that would this

give me the right result? So, what we need to do is we need to use the right embeddings to get get get the values for us. Proximity is more important in this case. Save 20 is exact match is there. We need to get the discount based on that particular value. So, that's the first thing we need to do. User queries. Now, what we are doing is we

are doing the BM25 search and vector search and then we combine both And then we create a fusion ranker, merge the results, and then we get the answer for ourself. That's the way which we are able to bring in our our perspective for lexical retrievers. Okay? Everybody good with the lexical retriever? Yes? It's the easiest one which we can kind of do to build this perspective for

us. Now, hybrid retriever is work based on the lexical retriever which will give us the give us the give us the give us the combines the semantic call which is coming to me and re-ranking the result re-ranking the result with the final relevance coming to me. Okay? So, this is what we do in the hybrid retriever. In the hybrid retriever, what is coming? BM25 which is exact

keyword match. Now, you do vector retrieval and then you re-rank the whole thing and then get the result. Okay, let's talk about this one So, in this case we combining the vector and we are able to go there. So, design a hybrid retriever solution which will be able to recall for shopping for the fraud return which is coming to us. That's what we are trying to do.

Show me red Nike CR shoes under 8,000 rupees arriving tomorrow. See this here? If somebody gives us this query, what should I run for this query? Think about it. What should I run for this query? Anyone? Should I do Should I Should I do keyword search here? What should What should I do as a search for this? Yeah? Both of them, yeah? Yeah, yeah. That's exactly what

we do here. So, I'm just going to go directly to the solution right now because uh that makes it a little bit easier to understand. So, I have the problem statement, so you can actually look at the problem statement later on. So, Michael Jordan trendy shoes, which is there, we need to do BM25 for this one. Return policy is final for the sale, which we are trying

to combine and get get the value for this one. And order order is also coming in. So, that means we can combine the results to get the answer properly for us, okay? So, that's the reason, you know, hybrid retrieval policy is the right policy for us to use uh in this in in this particular case. Now, you may ask a question, semantic chunking. So, this is the

semantic chunking. So, let me talk about this for a second. You upload a document here. Now, first of all, when you upload a document here, what happens? You split it into sentences. Now, there are multiple ways of doing that. Now, you have a document big document is there. So, first of all, you can chunk the document based on, let's say, you know, uh 500 tokens. 500 token

500 token 500 token. So, that's one way to chunk the Another way to chunk the data is to interleave the data. That means 500 chunk for this one, you're starting from the 500 chunk, and then you're starting from 450 to the next 500, 950. So, you're interleaving the chunks and getting the The problem with both the approaches, the first one is like, hey, just do the chunk.

That's a simple simple way to do that. What's the problem with that? Anyone? What is the problem? Huh? Context is missing. So, you're not getting the proper context what's going on. So, better is to get a contextual data. Contextual data means that you do it based on paragraphs, easy way to understand. You have paragraph with the whole context in one chunk. And even if the chunk is

a little bit bigger, it's fine, but everything in that chunk has the data which you need to work with. That is what we do when we split into sentences, split into different ways of generating the embedding for us, and when we do that, then we do similarity score and then we are working on it. Now, when we do this similarity search, you know, mark the chunk and

then chunk based on that. That is another good strategy to do it. That means you have the whole context with you. Yes? And there is another thing which is there. Now, you are looking at a chunk. Now, I'm talking about some some advanced strategy on this one, and we'll do the assignment later on, but just understand what I'm saying here. So, you have a chunk created. You

got the context with it. Now, what you're going to do with that chunk? You can directly answer with that chunk. Yes? You can do that. Or, huh? Embedding, yeah. You have the embedding chunk there. Somebody is asking a question, you're retrieving that value. That's easy thing to do. What else can you do? For that chunk, what answers that chunk is giving? What questions that chunk is answering?

Keyword No, what you're doing is you what what questions that chunk is answering, you create a inverted index for that. Make sense? I'm talking about advanced chunking. I don't think anybody here is doing it, so So, it's it's going to be a little bit more costly, but it's more effective. Now, because the questions I already know what this questions is answered for this chunk, what would be

the retrieval time for that? 20 millisecond response time. It's not 15 seconds. Okay? So, that's what I'm saying. So, there is a If if your product manager say that, "Hey, I need to retrieve it much faster." But, how do you do that? If for every chunk if you create an inverted and then say for this chunk these are the questions this chunk is answering. It's just a

database software of doing that. It's called Once you do that, that's another advanced chunking strategy you can apply for this. Okay? So, let's do these three strategies and solve for this problem. So, this is where the semantic search is also coming in. Now, semantic means that What do you mean by semantic? You say this is semantic. It means that you have a product catalog. It's a semantic.

You know, it's a database table. Let's put it this way. Semantic means database table. This is sometime called as canonical data model. Okay? Canonical data model. So, if you want to search for it Microsoft canonical data model, uh you know, CMD. Search for that, you'll be able to get it and you can use that same data model to really use what you are trying to do here.

Okay? So, build build build this perspective. So, now what's really happening is your first task is given the policy statement. Return particular Look at the return policy. Product can be returned within 30 days if unused. Footwear with manufacturing defects, sole, is eligible for replacements only. Refunds are not used for warm worn shoes. So, you can't do that. And image are required for defect validation. Uh So, if

this is your policy looks like, what are you going to do with this policy? You need to chunk this policy. Yes? So, let me go to the answer because I think sometime answers are better than, you know, So, uh I always look inverted index, you know. Look at the answer and then we get the chunk based on that. Yeah? So, what's really happening is What's really happening

in this case is that we are actually solving for designing the unit to retrieve the data for the final chunk. Okay? So, you're getting all the products, raw pipeline you're creating, normalizing the layer, and then based on that you're doing the chunking. So, what is the chunk looks like? Let's just look at the chunk here. Chunk number one is product can be returned. Chunk number two, footwear

with manufacturing defects. Chunk number three is refunds are not issued. Okay? Based on these chunks, now you create a semantic chunking semantic chunking which you're trying to do here. Which is return window, type is return window, and this is unused less than less than 30 days, action eligible That's what you can do. This is called a semantic chunking. That means you know what actions can you perform

based on that semantic chunk which you have created for yourself. Okay? So, another is defect policy. Footwear with with the defects, what's the defect policy looks like? And and it's all only for replacement. Okay? So, you have all these chunks created based on these chunks which are created with the chunk ID. Now, I can go in and retrieve the data from the database, and then we'll be

able to do the BM25 search for it, and then bring the perspective for me. Okay? So, this is the way we can do BM25 search. Now, another thing I can add to this is hierarchy hierarchy. What is hierarchy What do you think the hierarchy could be? What? Huh? Like what? Child and parent relationship is also added here. For example, return policy. Okay? Return policy has three different

chunks. So, I create a hierarchical approach to solve this problem where I'm now trying to create a return policy with chunk return window. Defect policy and refund rules. Based on these chunks, I need to get the answer. Now, whenever I need to do return policy, I look at all three chunks and get the value. Okay? So, you'll get a better answer. If you just get the get

only chunks, you may not get this. Okay? So, that's another problem which we can solve using this. Okay? Overlapping semantic window, we talked on this, you know? Where having you're having defect within 30 days, 30 days replacement only. There is a overlapping semantic window is there. That means two chunks talking about the same thing, you know? That can also happen and, you and it actually prevent the

boundary loss. When you're going from one to another one, you can add that perspective to it, okay? And the query aware chunking also is there. Sole came off. Oh, sole came off is there in this case. In that case, we had category defect and entity is footwear. So, where now you have a query aware chunking also you can do based on this, okay? To get the answer

for yourself. Event based chunking can be there, which is like fraud. Somebody did a fraud. Now, what is the sequence of steps which took place for the fraud detection? You can ask for that also when we are working on in this solution. Now, before you actually refund, what do you do? Take a photo to make sure this actually sole came out and that's something you need to

send out and review the summary what is coming in and recommendation is a agent is there to solve for you. So, the trade-off here is like, you know, if you want precise value, precise small things, policy is is is is important, you know? You want to you want to chunk size is small for policy. And product logs is a big size. Event log you will have a

big event log will be there to solve what really happened if something goes wrong. So, that means you need to create you need to be able to create something called as ensembled ensembled retriever. Okay? What is an ensembled retriever, anyone? You are trying to get the chunks from two different sources and then you are trying to trying to ensembled with combine all of them and then then

return All right? So, document aware chunk is another one. Okay. So, now we are moving towards the document chunk. So, what is document chunk and chunk anyone? What's happening is you have a document there, you pass the document, identify the sections. We talked about like different different sections which are created here, extracted. Now, chunking strategy can be section based splitting. You preserve the context, and then if

you're having multiple uh hierarchy there, so you're able to provide the hierarchy also in this case, generate the embeddings. Now, retrieval wise, you retrieve the query, semantic search is done, which is a section is being created, and now you're working on top of it. So, these are the ways you can bring in the document aware Another thing is metadata indexing. Now, what's happening in the metadata indexing

is you're doing the same thing. PDF document is coming in, you're splitting this out. Now, what you're doing is you're trying to get the extractor of the entity. Entity based on the metadata which is coming from here. So, that means the PDF document can also have metadata associated with it. So, what does that mean? What do you mean by metadata? Product name, product name is so and

so, and they you can have a size, shoe size can be there, color can be These are all metadata. Based on this metadata, you will be able to now do the do the embedding model. You're going to do the vector embedding for this this particular layer. Now, at this point, client is asking a question. Hey, I have I want to make sure that, you know, I'm able

to get this question and answer. Before you answer the question, what you are saying is that, "Hey, I'm only interested in this product." Nike shoes. I'm not interested in all other type of Adidas shoes and other shoes which are there. So, I want only Nike shoe here. That's called metadata filtering. Once we do the metadata filtering, then we say that I get the chunk, but now I

get competing chunks. Somebody asked this question, like, "Hey, now I have a document there. I'm asking answering a question somebody asked here, like, you know, 'Hey, what happened? There is a new knowledge article which came.'" New knowledge article came, and uh and then, you know, uh what should I do? Should I use that new knowledge article or I use the old one? That's where that's where the

popularity comes in and recently comes in. So, what you can do is you can build build you can get that recently because recently is there that knowledge article will come first. Okay, that's the way we can do that. So, never Okay, so there's another thing which is important is that Okay, what if there is a new document which just came up? The new new policy which just

came up. What are you going to do? What if if a new policy came up? You have two choices. I give you two choice. One choice is you refresh the whole knowledge article articles all the knowledge articles. That's one choice. Another choice is you just delta update the knowledge article. Which one is better? I'll be out. Is a delta is better? But if you do delta the

old is still You see? Old is not going anywhere. You're with me? So, old will still be there but you get the recent article with the with respect to the popularity as well as based on the recently of the data. Now, within 28 days refresh the whole thing. Everybody with me? So, that's this is the way you solve There was someone who has who asked the question

You asked the question, yeah. So, this is the way you solve that problem because because you get so much data there like you know, you can't refresh everything. So, you're going to do the delta upload and then use the recently models with with the hybrid search to solve this problem. Makes sense? So, that's otherwise you you can't do all these things when you when you're working on

this. That's the reason the metadata processing is very good because now we are able to have unstructured data also having metadata to work with. Okay? So, you're not just working with structured data but unstructured data is also there in this. Okay. So, let's take a look at this one first before we move on. So, in this case we are having document away chunking process is there. So,

the same thing which we did before but now we have got chunking with the doc document away chunking is also there. Wrong policies applied. You know, current issues right now is wrong policies are applied. Irrelevant product is retrieved. Not not good product to retrieve for me. So, that is where the document aware chunking is very good to have. Okay? So, what does that means? Return Return window

30 days and defect policy and evidence for what really happened. That's what we are trying to do here. Native chunking what it will do? Native chunking will you know, fixed 200 to 300 chunks with overlapping is there. That's not going to help. Now, I added section away chunking. I created multiple sections. So, I get a little bit better understanding what's going on. Section metadata design is added.

Now, this metadata design is added which is now trying to say, "Hey, this section title and what's the hierarchy looks like?" Now, I also added hierarchy to this. Okay? Because I added hierarchy and I give more context to this, what I can do in this case. Now, I added hierarchy here. Now, once the hierarchy is added, parent-child relationship is added, now preserve the context for myself. You

know, replacement only. You know, without this section without section you can't do this. Now, with section footwear defect policy. I know footwear is the defect policy which I'm kind of working on this solution. Then, we have got here multi-selection retriever. That means my shoe sole came off after 2 weeks, can I get refund? Uh, multi-section retrieval is done. So, there is a defect policy is there. How

do we do this? So, that is what we going to discuss next, you know, how do we do this this perspective. Okay? Okay. So, So, what happens is that like when when we are trying to work on this Okay, this is the solution for it. Yeah. So, what's happening is how do you preserve the chunk which is coming to you? That's the real question for you. Native

chunks is not good. We need to do the section aware chunk. That's number one. That's the solution which we are trying to bring in here. An advanced section with respect to hierarchy also we added to this and then add graph rag. We'll talk about graph rag in few minutes. We can add that also. Now, multi-section retrieval is added to this to really get the better chunk for

me and metadata aware uh you know, indexing is added to solve this particular problem. Now, because I added that, now I've got explainability. I can explain why this decision was made. Okay? Because this problem was solved this way, this is the decision I made to solve this problem. You know, that's what we are trying to get out of this uh this perspective for us. Okay? So, that's

said, you know, what we can do is now you can run these prompts also. They you take this policy and run run different different sections to do this prompt, but I would skip that for now. Okay? Let's take a look at Redis similarity search with caching layer. Okay, this is another thing which you can do where you're not having every data coming to you. Okay, so this

one is if you want to bring in the data from the API gateway somebody's calling it this for that you check in the Redis cache first. From the vector search you want you want to search from the database from embedding and you get the answer here. That's something which you're trying to do and after we have it in the Redis cache, you check it in the Redis

cache and just return from there. Now, you may ask a question how would we do that? Somebody's asking return policy for Nike. Okay, that's what they asked for before. The second time that somebody's asking can you tell me a little bit more about return policy where the Nike is there? It's not the same sentence. But semantic, okay, semantic meaning is the same. You with me? Semantic meaning

we can add using Redis. So, the semantic meaning similarity search meaning we are adding, we calling it similarity search meaning we are adding in there. So, what's the advantage here? You're not using rag for this answer. It's everything is ready to go. You just use the Redis to get the answer for yourself. Now, what you can do is what I was talking about before, now you can

create the questions. What are the question these chunks are answering? If I can cache those chunks here, and this exactly I was talking about the advanced indexing before, where each chunk I know what what where it's coming from. Now, every answer is very very fast. Now, of course, you have to know your query will be you know, you have to save that data in the cache. But,

you'll be able to solve that problem without any problem. Okay, so that's the advantage of doing it this way. This is very powerful technique which we which we can use for our usage. All right? This is called a semantic Semantic similarity search really helps you understand like you know, so high QPS queries per second Black Friday use Lot of people are asking the question low latency cost

control if you need to do. In that case, you can use vector similarity search and caching query results are done. Once we do that, now we are able to go through the client is able to go through the data database to get the answer for us ourselves. So, you can do all these tasks on your own. You can take a look at it later on. But, the

perspective here is that it mean that simulate Redis cache. Okay, so this is the one cache behavior. So, what I want to do two of them at least to just show you what's really happening Uh let's do these two. Okay? All right. So, take a look at what's really happening in this case. Oh, well. Okay, third time is a charm, yes? That's Once the internet works, but

it should you will you will be able to now simulation this sometime the simulation helps you really understand whether it's a cache hit or miss, which is there. Cache hit is there Redis response is coming in 3 and 3 millisecond. Now, with you embed first time will be a cache miss, okay? So, that's the main difference between the two. So, GPU utilization with cache hit is network

hub is only one. See this here? Network hub is only one. GPU miss in the cache miss is there in that case there are six hops here is done to really solve that particular problem. So, that is the benefit of using this model, okay? So, once you have a cache hit, you'll be able to have 1,000 request cost is so very less. $8 cost. Somebody wants to

save in the cost very easy to save the cost just by doing this this particular perspective. So, you you can see the latency is also very less for for doing all all the activities with this one, all right? So, that is another thing I recommend for us to do when we are working on our solution. Okay, the next is intent caching for for latency reduction. So, intent

caching is whenever some cache is coming in, now check the cache and then find out if it is there in the cache get the value similar to the previous one, but now we are trying to do intent cache also. That means we are also storing what is the intent of the user after evaluation, okay? So, graph rag when I'm doing the graph rag, take a look at

it here. Disconnected solution. Now, we build this knowledge graph. We need to build the knowledge graph and query that knowledge graph. We'll talk about the knowledge graph building next. So, what all that means? Multi-hop reasoning. If you need to support what? So, in that case, you use graph rag. Okay? So, what does that means? You're traversing through relationship to get You're traversing through the In that case,

you need to be able to do that. Relationship-aware grounding you need to do and get only for that particular product for that order which you're trying to get any knowledge article, you want to get that, you can use this to be able to do that. And you should be able to better explain what and defend your reasoning when you're building the solution. So, graph rag example is

product eligibility. Number one, how do you do graph rag? What is graph rag doing? Check if the if the product is eligible for for promotion or not. Hey, is can I apply this promotion? You know, 20% promotion can I apply? Can I use graph rag to do that? Yes, because it's a related graph is there. I create a graph and I'll talk about graph in a second,

but but graph is with the nodes and edges. That's the way we create the graph and then we can say product eligibility promotion rules. Apply the promotion rules. Shipping region. Determine the correct shipping region where you need to go. Return exception. These all are examples of graph rag. You know, check the loyalty discounts for a VIP customer or a regular customer. Okay, let's take a look at

this one. So, what's really happening is let's take a look at this one. This is important to So, what we are doing is oops. Okay, this works. All right, what we are doing here is product is linked to a shipper, customer is linked to order, policy linked to region. Whenever any question is asked, simple look up is there. You do the text similarity search and then you

get the answer. So, you don't need to traverse for everything. Okay? We'll talk about orchestration. How do we do orchestration later on? Okay? That's something we're going to discuss in few minutes. But, what we do is simple look up. Don't do this. If it is a relationship look up is there, that means I'm asking for a product. I'm asking for an order. Then go here, traverse to

the relationship, do the multi-hop reasoning, and generate the answer and give the answer for myself. That's the way we are trying to do the this one. All right? So, what all that means? Let's take a look at this example now. Now, this is another one Dream Mart example we are trying to do here. With the shoes. Now, in this case, we are also doing vector similarity search,

BM25, and caching. We did all three of them. But, it did not do the relationship for me. And can I return these Nike shoes delivered to Bangalore after 20 days? If this shoe should come out. And that is where we need to use GraphRAG to solve for this problem. Okay? Shoe So, we need to do classification. We need to classify these things and then come up with

a query with needs GraphRAG output or not. Let's take a look at the example here. So, instead of instead of waiting for the result to come in. You run all these prompts here, but what's happening is what's happening is in the GraphRAG, we're trying to come in with a relationship. Customer placed an order. Order contains products. Order applied to promotion. So, we create this relationship. Okay? Based

on this relationship which is created, now I can go to these core nodes and get the answer for myself. Okay? Everybody with That's the way we can do that. Now, you can do simulation also. I don't have time right now, but if I had a workshop, we'll do a simulation on the same thing. Now, why are we doing all these things? Because we need to do orchestration.

What do we need to do? Or Why are we orchestrating here? Because I need to orchestrate orchestrate the strategy selection re-ranking procedure to solve all these problems. So, what are the agentic capabilities? Interpret the intent. We need to interpret the intent. Coordinate what tool to use. And validate whatever is coming to me is correct or And based on that, escalate if I can't escalate this this values.

So, these are all the capabilities which we can do in agentic rank. So, what does that really means? You start from intent recognition, like what the user goal is. What's your top goal looks like? Retrieval path. Now, you retrieve retrieve what kind of choose appropriate method. Should I get everything or do exact match search? I should do exact match search if I if it's a simple question,

you know, I don't need to go there. Information retrieval is done, then answer generation is done, and after I do the answer generation, then I check the policy. Can I show this answer to the user? Yeah? That's where the policy comes. And then we do the downstream action we're trying to create, which is building the confidence of the assessment. If the confidence is whatever the answer is

generated, is the confidence good or not? If it's the confidence is good, what are you going Yes, I'm ready to show to the user. If it is not good, ask ask better That's what we do end-to-end agentic view in this case, which is coming in. So, we are trying to come in come in and solve this puzzle for us. Let's take a look at this puzzle now.

So, what we are trying to do is the user is asking this question. Orchestrator engine is coming in to the retriever. Based on keyword search, you do the BM25 and all three of them we already talked on this, yeah? And then you do the correction. If I need to do the correct agent, you pick the correct agent. That I got the answer from this, and if the

answer is not correct, correct it. Somebody asked here a question that I got an answer which is not a good answer. What are you going to do? You correct that answer. Ask user more questions to get a better answer for yourself. Sometime I don't even send it to the to LLM. I first find out do I have all the details before I send it to that. So,

that is one way to save the cost when you're trying to work on the solution. So, that's what we did here. So, agentic roles for for us like orchestrated agent is doing retrieval, grounding, and correction. Three things it's trying to do and taking an action on top of it. So, there is something called correction recovery agent. Now, what is recovery agent? Who's going to write the recovery

agent, by the way? Tell me. Job security, we. Yes? You need to write the Nobody does that. But, I'm telling you, if you do it, you'll get better results. Otherwise, you know, you're going to garbage in garbage out. Like you did not have the right results to work with. So, reference architecture is Salesforce is calling through LangGraph. Now, I'm talking about Salesforce example, but you can put

any other one here. LangGraph, which is now determining the context. Based on the context, it's trying to get the customer context from here, rules guidance from the policy from the graph I'm getting from this, and based on that I do either vector search or graph search or Redis cache and evaluate and get the answer and do observability on That's the model we need to do whenever we

are working on any solution for us for rack. Agentic rack is this only. So, what does that mean? The routing tool selection. How do we do the tool selection here? At the end, what do we do tool selection for? Let's take a look for a second. What we are trying to do user is asking the question. Now, based on that, is it an information policy-sensitive question? Based

on this, you pick the right tool. Is it a search index, knowledge graph, or an order API call? You need to do that orchestration from here, which is now going to the retrieval policy, which is you're picking the right retrieval policy for yourself. What questions you're asking What kind of question you're asking asking here? Is the information policy-driven, uh intent question you're asking? What is the retrieval?

Is it a search, graph, order API, and cache? And then you're asking what evidence is allowed? What freshness is allowed? Freshness means you are able to still keep the old data still there. There's nothing wrong in that. Uh you know, what how much confidence threshold is there to solve this problem? Once you do that, now you're defining the memory state and self-correlation with human escalation is needed.

Like you see here, we're starting the workload, and we are having this evidence created for us. Based on this evidence, we are able to get the answer. So, what we are doing is we are creating an orchestration layer to solve the puzzle for us. All right? So, that means what we are doing is we create this rag system evaluation. What to know is we need to know

is what was right context or not. Was it the right context or not? Was it supported properly or not? Once we know this, now you can create production grade rag for yourself. All right? That's the prospective we'll be discussing later on. So, what is the importance of continuous evaluation? We do this so that we can monitor the results, and we get the feedback, and we adapt and

control over time. That's the prospective, you know, which we need to do. So, there are three layers for quality. Number one is retrieval quality. How How good is the retrieval quality looks Next is the generation quality. How good is my generation? Is my model giving me the right answer or not? And number three is the agentic flow quality. Based on this, you can have quality measures created

for the context relevance. Was the retrieved evidence actually relevant query or not? Is it the right query or not? And what whether it's a quality driven approach is there or not. All there is a ranking quality is there or not. Best answer is coming top chunk is coming. And chunk usefulness is there or not. Somebody ask a question, how do I evaluate? This is the way to

evaluate. These things if you do, you are able to build a solution for yourself. Okay? All right. So, I think we discussed quite a lot right now. I know that you know, at the end, there are certain tools which are available for you. One is a true lens. You can take a look at that. Rag triad you can use. And there's another one ragas is there. Both

are good to try it out. But main thing is you want to make sure there's a KPIs. You're working through in KPI. Key performance indicators. Okay? So, this is the key performance indicator of conversion lift. Uh cart abandonment reduction. So, these are the KPIs we are supporting with the evaluation. If I do that, now I'm building something which is available, which is which you can say business

value is coming out of it. Okay? Once you do that, you're able to have a smart helper with fewer abandonment cart and higher containment value and low cost for to working it. And that is way you're bringing in the ROI of the data which is coming for us. So, what did we discuss today? We discussed the fact We discussed the fact that hey, use whatever you want

to use here. Hybrid, BM25, cash, ROI, vector, and evaluation. These are the strength. If you do that, you will have effective decision for yourself, quantifiable decision for yourself, and safety has been added on top of it. So, rag can be done in production. Agentic can done and ROI is also part of this. So, I hope you enjoy this session and see you in another session and have

a wonderful day. Thank you. >> [music]