Building Context-Aware Intelligence Pipelines with Open Source - Sudheesh Kairali & Sachindra Joshi

This talk discusses the evolution of data management alongside the development of application landscapes. The speaker highlights the importance of understanding unstructured data, especially as it constitutes about 70% of enterprise data. They emphasize the need for effective data retrieval and representation methods, particularly focusing on retrieval-augmented generation (RAG) techniques. The complexities of managing unstructured data, including issues with context, format variety, and retrieval performance, are explored. The session also delves into the limitations of current approaches using large language models (LLMs), and the need for better methodologies that consider data structure and context to improve retrieval accuracy. Overall, the talk addresses the challenges in the integration of various data types and the necessity for sophisticated management to harness data effectively in applications.

So I I would say this slide this slide would But go on. actually introduce us better. Right? So we are fans of something. Right? So we both are and I'm taking the you know risk of even introducing you or So here that um Look at how technology evolved. Right? So initially there were some code and that code was called as an application. And that code was talking

to everything. It was parsing files. It was reading data. It was solving problems. This is how initially the computer science started. But then when technology evolved the technology evolved as you can see. I'm not going to go through everything. But the the technology evolved then data started identifying itself as a separate entity. Right? Databases for the same reality that data needs a separate treatment. It needs a

separate um you know it is basically a partner to the application landscape. So there is application landscape and then there is data world. And you can see that when the technologies evolved over time databases also evolved. That's why I'm saying I'm a big fan of this um I I took data as a carrier because I can see that whenever technology evolved data also evolved. The challenges in

the way in which you manage the data layer evolved. The pipe the way in which you pump the data you understand the data you reuse the data for various purposes evolved. So that's why I think we are here today to talk about it where everybody thinks that they can actually wipe code application landscape. Right? You can use cloud code or maybe Bob. And you should go to

Bob you know both as well and understand what it can do. But there are AI systems outside that can actually create code very well and it's probably time to think that whether this data layer is relevant. And we think that this session should tell you how much more relevant it is to build application right now efficiently and scalably in such a way that you understand the data

layer problems. what happened last year, right? So last year you would see that um there's a slight delay here. Yeah. last year what you would would have seen is that probably not last year, last couple of years if you would see that people started looking at unstructured data heavily, right? So that you could see that in enterprises a lot of you know PDFs and you know files

and images videos all this is there. This is where the data was accumulating. There is a study that says that 70% of the enterprise data is unstructured. And you know when the AI world evolved uh there is this um you know the retrieval augmentation reality that came in. What is retrieval? The model started solving problems but then we realized that the train data is not enough. There

are very specific data that is required for your use cases. And that's where the augmentation came into picture. And this is where we started to do additional techniques where you took the files, you started you know building the chunks you know and then you could index it. You could index it and put it into some of the specialized databases and these were called actually vector databases. And

what it does in its essence is it looks at the you know the similarity and it actually picks up necessary details. Such it is going to talk about the you know some of these aspects in detail. And then world started calling it as rag. my question to Sachin right now is did it really solve the problem and what was the problems that you would want to talk

about? What are the essential you know details because we are actually moving fast. We should it's it's time to actually catch up a little bit what happened last three four years probably. Yeah, sure. So so actually this is absolutely not a solved problem and as Suresh mentioned that see the LLMs is the new tool everybody just want to use it right and just put it wherever you

can. It certainly does not have the context of the enterprise the data that enterprise has it is no context and rag as you were saying is one of the ways in which people are trying to bring that context into LLM. Of course you could also fine tune right you could take the LLM and you can fine tune. I have been fine tuning these LLMs for last 10

years. It's a very difficult thing to do. You can fine tune them but you kind of break them. They are so aligned so well aligned their alignment is completely broken the moment you touch them. So rag seems like a better option because you are not you are not spoiling that that very good diamond LLM right. So but rag has a lot of problem of itself in fact

and that's what I plan to kind of go through and see what are the different kind of problems that exist in in the rag world. Now there is a very nice delay here which goes into and and and we have a lot of animation so we will have a nice time. So so what we have done is that we have kind of categorized the many problems right

so rag is very easy to build like it takes five line of code to just take a line chain and you know you have all these template things. It takes less than 10 minutes to just put up anything. But, it doesn't work really in practice. And we are going to go through the kind of problems that are there, and we have tried to categorize them in different

uh you know, kind of classes of the problem. So, the first kind of class is is the data problem, right? So, the the problem is that the the enterprise data is pretty complex. Right? It's not a simple text file or Word document with a single column, but uh you know, you have multi-columnar data, papers have everyone mixed column, there are there are there are tables. These tables

are very nested in nature. They're very difficult to interpret what they are. You have flowcharts, you have images, you have audio, you have video, enterprise data, and we are only in the unstructured world here, not we are not talking about structured world yet. Uh people have been talking about structured world and dealing with it for years, right? But, for unstructured world, the the the data is really

complex. And that creates a lot of problems. Like, you need to really see how to ingest such complex data, how to bring it together. And you know, there are different things that people have started to do, and we'll touch upon those also. But, the the variety of the data is a big problem. The other is the format itself, right? So, it's not just that the the data

looks different, right? It's flowchart versus table, but the format itself is also very complex. PDF is a very popular way, but then you have slides, you have Excel, you have you have all different kind of things. So, that's the heterogeneous format. So, we need something that can deal with all these heterogeneity of different things, right? The other very big problem in enterprises is that there is not

a single copy, right? There are so many different variants. You have somewhere some slides and some version 1.1 and then 1.2 and and then there are policy documents of 1.0 and then 2.0, and you have so many duplicate, near duplicate, recency issues. Huge problem. So, we have to deal deal with that version drift. Uh and then of course, the the third thing is actually extremely important, missing

content. I have a bunch of corpus, like I have I have 10,000 documents, I have some customer support, I have some FAQ, I have some policy document, I have some product manuals. Bring them together, index it, put it in front of the users. Users ask us something, but does my content actually even have them? How do I know whether the content is missing or my retrieval pipeline

is a problem? How do I know? So, missing content identifying, first of all that there is a missing content itself is a big problem, and then you have to kind of fix it. So, these are all just data problems, right? So, data is complex, multi-format, versions, missing. You need to deal with that problem. The second problem is the retrieval problem. So, I've got a corpus, right? I

have to I have got a user query, and the whole premise of RAG is that, you know, I will get the relevant document somehow. And people have been dealing with this since many, many years, right? From internet days, we have keyword search, we have page rank, we have all these things. And then with these all deep learning stuff, more and more people started with what are called

embedding-based approaches, where you take a chunk and you basically kind of represent that chunk in some kind of an embedding. Embedding is nothing but just a vector of some numbers. Embedding has a dimension, D dimension, which could be 768, 512, 1024, some number. So, you have for each chunk those many numbers to represent that chunk. And then you have these vector databases, which where you can store

these embedding. Then you have some sort of algorithms, HNSW, IVF Q. These are different ways in which you can find similar embeddings given a thing, right? So, there is a lot of emphasis on these embedding kind of thing, but and and we have always seen that, you know, in practice, retrieval is kind of okay, but the moment like you are not retrieving across, let's say, 50, 100,

400, 1,000 documents, but you are going into tens of thousands, hundreds of thousands, millions of thousands, the retrieval performance degrades severely. So, the retrieval is an easy problem if it is just 400 documents, but it's a very difficult problem if it is a million documents. Now, why is that? In fact, there is a very recent paper which actually calls out that it's actually a mathematical constraint. It's

not just that they do not perform well in practice, but in theory they can't perform well. So, there's a paper from Google DeepMind which shows that uh for any any So, these dimensions embedding has a dimension D and then you have a corpus which is capital N. For a given D, as you increase capital N, there is some N after which, no matter how you train your

embedding, what your data set is, how you optimize it, none of these things matter. It has a mathematical limit that you cannot retrieve the required top K, the right top K, beyond some number capital N for a given D, right? So, and why can't I just increase D? While you increase D, you have more memory requirement. Of course, there are very nice work. Turbo Quant is one

work which I don't know if you have drawn this is a very recent work in ICLR this year which kind of reduces the these embedding spaces very drastically which has caused severe, you know, down in the RAM prices and so and so forth. But, there is there are mathematical limits in terms of how much you can really retrieve well. Uh and and that that's what we have

seen with worldwide enterprise customers that So, retrieval retrieval is highly dependent on how do you index that. Right? So, it's very very clear that you know, retrieval is not an independent problem. It's actually very much linked to how do you represent, store the information prior to retrieval. So, so uh of course, they also have this limited data set. We'll not go into details of that, but one

good thing was that, you know, if you're doing multi-vector data set. So, you know, this limit is this exists only for when you're representing each chunk by one embedding, right? But, you could always each chunk by not just one embedding, but multiple embeddings. And that is what is called multi vac. There are multiple multi vac representation work that exist in academia. Some of them people also do

in industry. Colbert is one example, which has its own set of problems. But there are re-ranker and all those approaches kind of remove this kind of element. Uh the other big problem is there is a lot of emphasis on similarity. User ask something, I have to find something which is similar in my content for for this user query. But unfortunately, similarity is not really relevance. Uh right?

So, if you see I will I will show for example. So, this is like version ambiguity, right? Like somebody ask, "Give me what are my travel reimbursement policy?" And as I said, you have multiple versions of different policy documents. Now, really what is the relevant thing is here is policy version 2.0. But if you look at similarity, the actual similarity part will just capture the similarity and

and that is not really capturing the relevance. The second part is context mismatch, right? So, you have leave policies and somebody's asking, "How many leaves do I have? What is my leave policy?" It depends where you are coming from, right? You are you are from India or you are from Brazil or you are from US, right? So, the similarity is not going to capture it. So, there

is a context that is mismatch, right? So, you need to get the right context. Uh there is authority and trust. Somebody's asking, you know, "What are my expense limits?" Now, you are kind of indexing everything. You are indexing Slack discussions, team discussions, everything. And giving those things for this kind of query is not of much consequence. For somebody asking expense limit, you probably need to go to

the policy document. So, again, similarity is not going to capture this. Uh yet another problem is uh you know, like somebody asking, "How do I do blah?" Now, "How do I submit expense, right?" The policy document is not required. I need a step-by-step thing. capture that. So, just merely trying to optimize on your similarity is not going to solve your retrieval problem unless until you take a

closer look on how do I fix my relevance, right? So, you need to go on that. So, that's another another retrieval problem. So, So, we looked at some data problems. We're looking at some retrieval problems. Now, we are coming to the third category of the problem, which are like context problems. So, what is a context problem here, right? uh what we do in um in in in

these retrieval, we take these documents and then we chunk them, right? We kind of can create smaller versions of it. You know, why do we do that really? Like, if I I don't know how many people here are maybe a lot of people here would be already very well familiar with the rare kind of pipelines, but the first thing that we do is take documents and chunk

them. Why should we chunk them? Because we are representing an entire chunk by just some numbers. So, more content you put in, and that is what that theoretical limit was also kind of pointing towards, but that was at the corpus level. Now, here I'm talking about a chunk level. Right? At corpus level, you say, "Oh, if your dimension is D, your number of documents are bigger than

capital N, you have a problem." But here there is a different problem. My dimension is D, and how many words I have in my chunk is also a problem. I have 512 tokens worth of information. I need to put it into 768 numbers. Can I make it 1024 tokens? Can I make it 4,000 tokens? More the tokens you have, lossy it is. Loss less things are represented

in these numbers. So, you'll have very much loss. So, that's why you don't want to do that. So, you would chunk it, right? You chunk it and so that you have a better representation of these chunks. That's why you chunk. But the moment you chunk, you're looking at a chunk. You lose You lost the context, right? It's like if I have a book and I just took

a paragraph out of it and throw it in front of you, and just how much can you get out of it, right? So, it's the same problem. You have lost the context. So, if you and that is what I need. So, I I'm kind of I have to go less so that I can retrieve better, but the less I go, less context I have. So, that's the

problem with chunks. Then there's another problem, and that is a even more serious problem. Uh people uh in fact for the chunk there are, you know, there are there are these something called modern bird architectures nowadays which have changed these limits from 500 tokens to 8,000 tokens, which is much better because uh most documents are kind of less than that. So, now you have a better representation

techniques uh with modern bird kind of architectures where the embeddings uh they're larger, but they are able to they're they're able to capture it. Um but the bigger problem is, and which is less work I see in in academia and in industry, is around the documents. See, you it it's like we take each document and we kind of represent that But there is a structure to these

document, and that structure is kind of lost completely. It's like I take a book and I tear down each and every page and just mix that. That's what we are doing. Right? We are indexing chunks, which they have no context. If you don't want to do with chunks, you somehow manage to do it with documents, even that is a problem because it still doesn't contain the structure

on top, like a table of contents structure. And that's a huge problem. That's a very simple it's a very intuitive thing to see that why should we not fix it, but there is very less work. In fact, there are very recent time uh we actually started to work in around 2 and 1/2 years back called table of content related search. There's a very popular GitHub called page

index now which came just 4-5 months back, which does very similar thing. Um again, it has its own set of problems. Um but the thing is that you take all these documents that you got and you somehow create a structure on top. And now we have got LLM like whatever problem you have LLM is the answer. So, here also LLM is the answer. You give it a

corpus, you provide some kind of a workflow around it, and it creates a structure on top. In fact, Andrej Karpathy gave this wikification, I don't know whether you saw that, but that is exactly what it's doing. It's take your corpus, and you will a cloud the kind of a way or some other outer loop kind of a thing and it will create wikification of that entire thing

creating a structure on top. And that structure is very useful because it gives you the context. Right? So, people have started to move in this direction which is also being referred as vectorless embeddingless search because you don't represent any vector anymore. You get your corpus, you create this kind of a structure, you search in this structure. And this is structure since it has the context, it's much

easier to do it. Right? So, uh So, but there is this huge problem of lost in the chunks and documents, no context being there. So, that's a big problem of the context. The second um thing uh is Right. So, we can move to the next slide. The other problem is how language model really use these context. Right? So, So, we retrieve these things and we have seen

retrieval. So, you know, we start with the data, we have a lot of problems in the data, but we kind of, you know, ignore them and we kind of index them and then we go to retrieval and we have a lot of problem in retrieval, but we ignore them and then we have got some documents. Right? And now we are going to use those document in what

is known as context. Right? Uh with the query to LLM. And uh So, now we are already lost a lot of battles, but if you keep going, we have this context now. The problem is So, context has a limit. Right? So, when starting LLM came, they had limit of, you know, some 1024 tokens. We move to 2048, 4000, 8000, 128,000. Now we have half a million. So,

context length is like, you know, keep increasing. Uh like Google has 10 million token length and context length. So, you are kind of increasing context length. Right? So, it also kind of creates more room for you to fit in things. So, this also kind of creates less need for you to really index anything. Just put the whole thing. Why do I need to even index anything? Put

everything if you can put everything into 128,000 tokens or a million tokens if your corporate size is like that. You could do that. But there are serious problem with that, too. Right? And that is a that is what uh we're trying to kind of see here, that how do language models use their context? So, there is this very famous work called lost in the middle. And um

what it's kind of uh it's it's not that new. It's like 1 and 1/2 2 years old, right? So, in today's world 1 and 1/2 years 2 years old is just too old. But So, what it's showing is very interesting. On X axis, you have the position of the document where the answer actually is, right? So, you you've got you've got let's say you know, 10 documents.

You retrieve those 10 documents. But then you're putting those things into your context and you're putting giving it to LLM to get the answer. What order should I put it in? That's the question, right? I've got 10 So, the ordering matters. And that's what this is showing. It's showing that if the right document, so there's one document that contains answer, that itself is a by the way

a very big assumption because that doesn't happen very often. The answer contains across many documents. Well, let's say there's one document that contains the answer. If I contain if I keep that document at the beginning of the context, somewhere in the middle, should it be it be the end? And what people observed and multiple people also observed and corroborated this later, that it matters a lot where

this document is. So, if the document happens to be in the very beginning, 75% So, you take the same context, just reorder it such that the correct document that contains the answer is in the beginning, you have 75% accuracy. You take the same document, put it in the end, you're still lower, but fine. You're like around 64, 63. But you put it in the middle somewhere like

fourth, fifth, third, sixth, seventh, some somewhere there, your accuracies are much lower. So, where do you put things in the context changes your whole generation accuracy Because it's not able to say it's That's why it's called lost in the middle. It is So, the whole And it's not like intuitively you can think why that's the case, right? Because all that it is doing is generating the next

token. So, when it generate the next token, which happens to be the tokens from you are basically looking at whatever happened just before. And that's why you have much better results when it is at the 20th position, which is the last position. What about the first one? Why that is so good? Because that's where instructions happen to happen, right? Like instructions are pretty much there. So, that's

why these models have learned to focus in lot, attend a lot onto those locations. And that's why you have this behavior that you have a lot of attention in generation of the token, which is very near to the token generation, and which is very far in the beginning of the token generation. The attention is The attention is too flimsy or too weak in the in the middle.

And that's why if the answer document happens to be in this middle, it will be lost a little. So, that's why you you know, you you don't get uh you have this problem. The other problem is, as I was saying, the context lengths are increasing. They are like half a million now. It's given 128,000 context length. Every open-source model, they are all 256,000 tokens. We are still

working with let's retrieve top 10. Let's retrieve top five. Like, why? Why don't I retrieve top thousand? Right? Like because I can fit thousand documents now. So, my recall will anyway improve, right? So, that's what this graph shows that as you are increasing the number of documents from five to 50, the orange curve here keeps increasing because that's the recall. Recall of it's measuring in my those

documents, do I have a document which has the correct answer? So, as I'm increasing the number of documents I'm looking, it's logical that it should increase till it reaches 100. So, uh right. So, it like you with five you have only 68 recall. With 50 you have close to 90 recall, right? Let's take all those 50 fitted in, right? So, now the document is there. I don't

know the order. Right? That is a different problem which is solved by reranker and we'll look at that. But, I But, that doesn't happen. That's what these lower graphs are showing. Right? These lower graphs are for these different models from Claude to GPT-3.5. So, this is also one or two years old work which shows that as you increase the number of retrieved it plateaus. It plateaus beyond

10 or 20, right? So, after 20 After 20, there is no increase in your generation performance though the recall keeps increasing. no matter if your context lengths are becoming better and you can retrieve more documents and you can fit in more, but you are still not able to utilize them properly. So, that's a big problem. Okay. So, those those those are the context-related problems, right? So, we

looked at the data problems, we looked at the retrieval problems, we looked at the context problems, and then the final problem comes to hallucination generation problem. models have got really better, actually. They have got better. They have got better in last 3-4 years. The hallucinations are less, but again there is theoretical problems here. You just can't remove that. Right? And that's what this paper from OpenAI which

came last year and last which shows that hallucination is inevitable. Because so, if you go one more what what basically they say that okay, rather than asking why models hallucinate, let's focus on this question, can I identify hallucination? Because if model cannot identify hallucination, there is no way to stop actually, to my mind, it was always clear that why the model hallucinate, because there is no pattern

here. Like, if you take an example of build Let's say build a classifier, okay? And I will give it a a sentence, and it has to tell me whether it's hallucinating or not. Hallucination means something which is not correct, right? So, that that it becomes hallucinated. So, it's not about grounding. Grounding is a different construct, right? So, grounding is a different Uh so, the the term grounding

and term hallucinations are two related but slightly separate things. The hallucination is like let's say if I give greetings with the correct spelling and greetings with misspelled word, right? Could I identify them? I can identify them, because there is a pattern to it. So, you could actually classify them. But, if you look at Mia Hollander's birthday is April 1st, Jago Perez's birthday is August 21st. Do you

know which one is correct or which one is not correct? There is no way to know. There's no pattern here. We just No matter like if all data is correct, and you know, you give the classifier all error examples, invalid example, saying these are valid example, these are error examples, there is no pattern to learn here. So, there is no way to know it. So, when you

pre-train a model, there are these instances which has no of knowing whether they are correct or not. So, the pre-trained model will has hallucinate. There is no question about So, hallucination is the rest rational guessing under incentives. So, that brings me to the another thing. If you go one more down, and which is what this is showing. The left graph is showing what is happening with a

pre-trained model, right? And the pre-trained model will have because there are no patterns for hallucinations, right? There is no way to know whether that we can't build a classifier which can detect whether something is a hallucination or not, there's no way to learn it. So, pre-training, where you put most of the knowledge that all the knowledge of the model that knows model knows, comes from pre-training, not

from post-training. It comes just from pre-training. The knowledge part of the model only comes through pre-training. Okay, so and pre-training we just can't learn it. But post-training make it worse. Why is that happening? So, in post-training what you are doing essentially is you are doing this reinforcement learning. you're trying to improve upon all these benchmarks that we have, right? We have thousands of benchmarks. So, everybody who

is releasing these models, they are trying to make a very good show on those benchmark. So, on those benchmarks, you have an incentive to answer. You're not rewarded for saying I don't know. So, there is an incentive inherently built in this reinforcement learning to guess and say something. It's like saying that, you know, it's like saying there is no negative marking. You have to crack JEE. What

will you do? You will Would you leave any question? No, you won't. But the moment you start to have a reward of negative kind, then something will happen. But you're not doing that because your question paper setup is like that. There is no I don't know in your benchmark. So, none of the reward, like these post-training things, actually take care of it. They have no incentive to

say I don't know. So, after post-training Oh, good boy. All right. So, okay, what these graphs are showing? What I said is fine, but how does that relate to these graphs? So, what this graph is showing, I don't know whether you can really see this x-axis here, but on the x-axis you're seeing what is the probability of an answer. So, the setup here is that you have

a question and you have four options. And you're asking LLM give me the right option. Okay? You're looking at that right option, whatever it said, and you're looking at what is its log probability is, right? What is it's probability that this So, this token generation is nothing but a sampling from a distribution of next tokens. You're looking at what the probability of that token was. And so,

if the token that got generated, you're asking an answer and it's saying 0.8 probability, then you're looking it's kind of a thing like I'm 80% sure that this is the correct answer. So, that's what on the left-hand side, the pre-trained model is like that. When the model is generating a token that the answer is blah and that blah is with only with 0.1% then 10% time only

its accuracy is there. If it is generating a token and saying the answer is blah and its answer token generation is 90% 90% we are accurate also. So, your accuracies are kind of on this line which says that this is well-calibrated model. The model knows what it knows and the model knows what it is unsure of. So, that's why it is saying the answer is this but

only 10%. And that's why your accuracy is also kind of matching to that. So, that's a well-calibrated system. What's happening after the post-training? Is on the right-hand side. Now, what is happening? Look at on the right-hand side of the curve. When the model is saying I'm 80% sure that the answer is blah, your accuracies are is still 60%. Not 80% the way it was in pre-training. So,

the model is not only hallucinating, it is hallucinating with confidence. It is very confident this is the answer and that's the behavior we have seen and observed. The reason is this post-training. The pre-trained models are not as confident when they are saying something incorrect. It is the post-training that makes them confident. So, the pre-training makes the hallucination, the post-training makes the confident hallucination. Right? So, that's the

generation So, to summarize, there is no enterprise context, let's get the context. We say, "Okay, let's fine-tune." We have not covered here, but fine-tuning is a nightmare. You can do it. It will work, but it will break. A lot of things it will break. We have not touched upon it. So, it's okay. Let's do the rag. And then we saw the rag. We saw rag is very

easy to put in. It's very simple. But when you put it into practice, it breaks. And we have seen data problems. We have seen retrieval problems. We have seen context problems, and we have seen generation problems. Right? So, that's where the current state is. Rag was like a prime time thing 2 and 1/2 3 years back. Every company wanted to do rag. A lot of POCs. Very

few production. Difficulties are these, right? So, let's keep going. The funny thing is that IBM had five rag products. >> [laughter] >> Okay. She was competing each other. Okay. So, how do we deal with these Again, it's not an exhaustive list what we talked about. There are many other problems, and people have been trying to kind of, you know, and not the solutions that we are going

to talk about are going to be exhausted by any means. They are actually even be even be more sparse. but there is a kind of you could see we what what what these people are trying to fix. So, some things we try to fix before we retrieve, and some things we try to fix during retrieval or post retrieval. Okay? So, in pre-retrieval, that's what, you know, this

categorization is. In pre-retrieval, you The first thing that you do is metadata enrichment. as I said, right? There is a lot of problems that are coming because of the data and the context. The data The context like, you know, uh you and the similarity is not same as relevance, right? You need to know this was in one. This is the current version. This is the earlier version,

and so on and so forth. So, you need to enrich your data, right? You need to enrich uh You need to enrich So, like, you know, you take this document, you say its title is this, its date published is blah, right? Its geographies are blah. Like, all these things then you can utilize for getting your relevance fixed. So, metadata enrichment help you in three ways. Okay? It

helps you in getting your relevance fixed because you have these other meta fields now in place which you could also use during retrieval, that location is same, the date is kind of nearby, the whichever is the current date nearby, right? So, you could fix those things. They also help you in similarity itself, right? Because some of the things could also metadata could also help you towards similarity.

And then there's also this structure part. The metadata enrichment also create certain nodes and then you can create certain edges. So, you know, it kind of leads toward graph right kind of things. But you need to do metadata enrichment first. You need to know what things are what, right? And metadata enrichment is of various kind nowadays. Don't think metadata enrichment is just like finding out certain token

and labeling it with a person name or a product or a location. There are metadata enrichment of kind summary. So, you take a document, you create a summary of that document. That summary is a metadata, right? And that summary could be very useful. It is actually extremely useful. Right? And then you could take bunch of documents and create a summary of that that document collection. Right? So,

people have actually done even these kind of things. And then you have a summary hierarchies. They themselves are metadata because they are not like there are they are they are summaries about the data. So, and then you have a structure on on top of that. And it's very helpful for retrieval and for getting the right context. So, that's metadata enrichment. The other thing is query expansion, which

is Well, you want to retrieve. How do you retrieve? Right? So, you retrieve by by framing a But are you framing the right query? Is the question. So, that's what you are trying to do in query expansion. So, whatever your original query was, you actually kind of there is something called hide hypothetical document expansion something which is quite popular. Uh there are usual query expansion techniques. Uh

but so, you know, you can basically rather than querying with just what user said, you could expand it in some way. And uh that is kind of expanding this query is kind of increasing your recall of the way you are going to reach across these different uh documents in your corpus. So, that's what query expansion is. Graph rake, as I said, is again uh this is on

more on the doing retrieval or post retrieval. Like the metadata that you have created will be very helpful for you to create a structure on top of this um uh on this corpus and then you could use uh you know, that to expand. Like you first do the search, you land onto some nodes, you go to their neighbors because they are related in some way and then

you have expanded your uh is uh scope of uh things that you're looking. There are many ways. In fact, graph rake there are like 100 different graph rake varieties now. Right? So, there are many different varieties, but in general, in principle, at a high level, what all graph rake is doing is a creating a structure on top of your corpus, which is what metadata enrichment help you

with and then a search on top of it. Like you search and you expand. So, that's what graph rake does and then finally, the re-ranker is actually very, very, very helpful. Right? So, re-ranker is actually the single if you have to pick just one thing and fix it other than the data problem. If data is not there, you're already lost. Re-ranker is actually very helpful, very, very

helpful. Uh all that re-ranker does is that rather than um so, the retrieval is all these embedding based thing. They work on something called DPR, right? Uh dense passage representations and um there are these other approaches which are called uh more cross-encoder uh and that requires uh a bit of more compute and a bit of like if you pre-do it, which is what happens in Colbert, you

need a lot of space. So, the the middle path is that you retrieve 100 documents using DPR, using your embedding base approaches, and then you do re-ranking on top of that. There are many many re-ranking algorithms, re-ranking uh embeddings for re-ranking also, which are available, and that improves substantially because now what you're doing is you're retrieving 100 documents, and then you're retrieving the top 10 re-ranking from

that. And that's actually very very useful because re-ranking is based on cross-encoding, which does a much deeper level of matching with the query that people are asking. So, these are some of the way in which people There are many many different components that people have uh kind of, you know, come up with, and they have them. And then, if you move to the next slide, that's you

kind of put, you know, how do you use all these things? And maybe Sudish can uh can can go over that. Yeah. Uh thanks, Sachin. And I think that was a very So, I asked one question. So, you can see that the the depth of uh data problems coming out, right? So, if you look at uh the data, I said he talked about the query uh phrasing,

and how do you, you know, correctly phrase the query. If you have, you know, the data in one database, there is this text query coming in into the application, and it is actually translating into that. But, the today's applications are actually not focused on, you know, one transactional query management. It's basically It's telling me again and again that I I I can solve the problem, right? Not

one query, right? So, you give one problem statement. And you can actually try this with any AI systems right now. If you go and talk to any AI system, it is going to build a plan around it. So, one of the first patterns, if you're building We We have built large-scale enterprise systems. And in these enterprise systems, we see plan-act as one of the design patterns that

we will continuously use. So, you will Once you have the unstructured story finished, you have the the the details about the metadata, you have the data collected, and you have the complete landscape available, you can actually do just like how humans do. You build a plan, and then based on the plan you act, right? So, you have plan in place first, and you build an efficient plan,

and then act. So, this is one pattern that you would see again and again if you are building enterprise system on this landscape. The other one, when the agents evolved, right? So, the agents are like it's basically more like my kid. So, if you if there is a problem statement, uh she if you give it to her, she will find a way to get solution out of

me, right? So, she'll talk to me 100 times, and continuously in a loop. So, the the the one of the the the design pattern that evolved is actually various kinds of loops. The react is one of the prominent uh you know, pattern that you would see. So, in this case, you actually you give one problem statement. He'll keep going into the data landscape, retrieve the data, and

give it to the agent. Agent is not satisfied, it's going to keep going in a loop, in a feedback loop, and it'll find out the right answer, right? So, these are the two uh you know, fundamental uh patterns that that you would see as a continuation to what Sachin said. Yeah. So, you So, exactly. So, in the plan act, the kind of solutions that we talked about,

we kind of chain them together, right? That's what you're doing here. Like, okay, let's do the query expansion first, graph like that, then re-ranking. You just plan, just chain it, like create whatever chain you want, right? LangChain, whatever. So, that's what from plan act, but now people have moved to react. No more. I don't want to create this. I want to orchestrate. Yep. Let LLM choose what

it wants to do. So, there is no plan, just react. And so, we just have to keep it in mind, right? So, react loop is going to not stop until it gets a a result, right? Which it is self-convinced that okay, this result is good for the purpose for the problem statement. So, this is So, how do we actually you know, deal with the enrichment part Sachin

mentioned, which is one of the prominent stuff you get the data, you understand the data, you actually find out the structure behind it. You probably will find that if you have a data associated with the data, you can use to make sure that the queries are efficient based on that. And it doesn't need similarity at that point in time. If you have those kind of meta understanding.

And also about the vector DB evolved and there is also hybrid search. You would have seen elastic open search, right? Uh so, hybrid search is something that allows you to take you know, search between text uh you know, vectors or multimodality, any of any of that sort. And it's not just only specialized on vectors. And then and we believe that this is a prominent uh space that

is actually evolving. And then you need to be able to create simple flows around these philosophies. It could be you know, a or react also. It could be a react could be a planning uh stuff. So, you can actually use what you see on the right-hand side. And I encourage everyone to go to our booth. We have uh you know, Docling Open Search LangFlow experts there. And

these they are you know, contribute to many of these stuff. And something that evolved open sources like Open Rack also evolved. I'm just going to briefly talk about these open sources which will allow you. But but if you note like you know, you you're basically trying to fix all those things whatever we talked about through these things because the data and enriching is anyway needed. Right, data

enrichment is something that is needed, so we have it. The the vector DB itself has a mathematical limit in practice also it's a problem. So, you need hybrid. Hybrid search is a better approach. And then there are many other multi-vector. All those supports are there in OpenSearch. And then you have reading flows. Like how do you bring all these things together? So, you go to the the

plan act or react anything using LangFlow. So, many of these elements now you could bring together under Open RAG. That's what Open RAG is. Yeah. and so Docling I I just briefly touch upon these slides because actually we are we have taken some time on the initial space of I think people problems. We have less that understanding problem space is very important. And there is also a

structured side we want to touch. Docling will help you uh to actually extract documents and deep understanding of this document. The enrichment of Docling and Docling itself has pipelines. So, I would encourage everyone to try it. It has specialized models to also, you know, handle these The The pipelines we do have a demo here. I unfortunately we don't have time for that right now. So, but we

should go you should go and see how efficient Docling is in even in your MacBook or any place where you can actually try it. Yeah. I think the the main part to drive here is that we've seen so many formats, right? And so many different things. Like, you know, you have multiple formats, you have multiple things to deal with. PDF, images, tables, audios, videos. What Docling has

is a pre-cooked pipeline for all of those things, right? So, it has already done OCR followed by it extract all these things. The PDF it will extract here is the chart. Here is a image. Here is a table. How they are connected together. It will produce these markdown representations of it, right? and then you can also enrich them. Give me the summary of image. Give me the

summary of table. You could get that. All those things without coding much. Like, it's all pre-cooked ASR pipeline just like, you know, so they are very very easy to So, now OpenSearch. OpenSearch is a hybrid search technology that you should try. The specialty is that when you start going deeper into some of these aspects. Like, for example, Milvus is a vector database. Underneath it might use files

and those kind of engines. But these vector similarities need memory space. Right? So, you need to load everything to actually do a similarity search. Open search kind of a technology represents your efficiency in production. You can actually do similarity search between storage and memory. So, you can actually make sure that the price performance is very high. And on top of that, it has the flavors of coming

from the elastic space. It can actually do text and, you know, fuzzy searches on top of the stack. And it has its own native pipelines, but as Sachin mentioned, Docling is a good endpoint. If you have a document and you have Docling and OpenSearch, you can in fact build the best systems out of the world if you have some model. Which obviously you will have at this

point in time. Now, Land Flow, where it comes is it actually allows you to build these flows. You can build, you know, plan and act flows or you can build react loops and and and any of the evolution that Again, just drag and drop, right? So, it has all these pre-cooked Here is a re-ranker. Here is a kind of an embedding service and so on and so

forth. You can just plug and play, just a visual editing, and you have the flow. You want to create a react, you can do this too. First time when I asked a question to Sachin, he took 30 minutes. I'm going to ask one more question. I'm assuming that he's going to, you know, be cautious this time. The question I don't have to ask him. We talked a

lot about unstructured space. We talked about, you know, the the 70% evolution in a street corner and all of that. Sachin, do you think that that will solve the entire data problem or No, it's really not, sir. We need to keep our jobs, right? >> The problems are far from being solved. See, the the the main problem is that the enterprise data is really, really complex. Uh

you have so many sources of unstructured text where there is a lot of focus when we are talking about rag, but it's kind of a silo of its own. Right? Like there is a structured world, there is an unstructured world. We just can't keep build a rag pipeline here and just not that doesn't do anything with the structured world. Doesn't happen, right? So, the real enterprise world

is messy. There are a lot of structured, unstructured, everything, right? Now, you have got a MCP tool, right? Like people would be here very familiar with MCP. Read an MCP with everything, right? Like everybody's pretty giving an MCP tool anyway. So, that gives an interface to connect and query and get things from it. Similarly, you create your rag tool also and create an MCP on top. So,

now you have an MCP of everything. And then after those MCP tools, just do a react. Right? So, MCP tools give you the answer. You ask a question. Hey Snowflake, give me blah. Snowflake, give me an interface of MCP tool. It should give me back the answer. Right? Rag, I need something on this and this. And who is driving this all? LLM is driving all this. And

then it will get okay, let me do this, let me do that. So, that's what a react loop on MCP tools will look like. People have done that actually. A lot of people have in enterprises already given all these MCP tool. Just create a react loop and do it, right? And so, what all of you have done? You have created some rag pipeline, you have created some

MCP tool on top. You have all these structured sources, you have MCP tools on top of that. And then you have a react loop on that and you ask a question. It figures out where it lives, how I query it. It queries. It see it doesn't come. It go tries again and then do something else again. Then it doesn't work. Try out something else. Finally, you get

an answer. You have an answer. Right? So, that's what with a powerful LLM you can do that with all the MCP tools. The problem is it's going to be extremely inefficient, right? Because it has no idea what is contained in my Salesforce blah table blah, how it is connected with the problem ticket that is there in Zendesk and you know that column, no idea, right? So, there

is that notion of a context graph that needs to be built. Which kind of Again, it's not a new problem. People have been building semantic models for last two decades, right? In structured world. Now, you have also a structured world, but you need some layer like that. That layer itself is not enough actually, which is like you know, here are my sources, here are my schemas, here

are my tables, here are my columns, this is how they are related, this is primary foreign relationships, these are the different metadata I have on my unstructured text, these fields are related like this. That is your semantic layer, which is what we have for last two decades. Like people have been building these, there are products around this. They themselves are not enough, you need to enrich them

to have more things into those context graph, but once you have that, then your react that you're doing or MCP tool now could be more well informed because now that context graph is what LLM uses to find out how I should structure my path around this data landscape, rather than just guessing something and then not getting through try something out, try something out, so and so forth.

So, an MCP tool with a react is a idea that can work with a very powerful model, but will never give you an ROI. You need something on a context graph, something that tells you how these data landscape are related with each other so that with a react is something that is going to be much more efficient. So, I think that's what we need, and then there

is one more twist to it. And that is yeah. Where do you build the context Where do you build the context graph? So, Well, I think yeah. I So, I Would you build the or we let's at the end. Yeah. At the end we'll come >> question. So, it will But there is one more change which has happened and that change is rather recent. People are people

are moving even away with with the react. So, in fact, it it's kind of react, not just react. So, rather than trying to do the tool calls, you're kind of trying to writing programs now. Go down. Right? So, the react is more like you got the plan through your context graph. So, you have a plan. I will do connect to this, get this thing from here, then

I will connect to that, get this thing here, merge this, get you the answer. Right? So, you you kind of plan, you act using MCP tool, you see your result, then you react and again do something. Rather than doing that, you come down, which is the agent is there. It you ask it something, it writes a code. The code connects with the tools. And then, you execute

the code. You see the result, you don't like it, you write again the code. So, that's what actually Claude code does. Or, you know, the likes of Claude Open AI frontier does. So, you are not reacting, you're reacting with writing codes. And inside codes, you are connecting with tools. Uh again, the context of is very much related even here, right? Because without a without a lay of

land, you are basically just guessing and and writing code and then failing and then trying something. It didn't It didn't work, and so on and so forth. So, yeah. So, Then, uh we have to So, I got a placard there coming out. That is warning me that there is only 10 minutes. I'm being explicit here. Because this is the last 10 minutes I'm going to steal from

Sachin here that what is really happening, to your question also, right? What is really happening is agents have become suddenly very powerful. So, it uh so, there was always a hesitation that can I allow an agent to write a code? He talked about Kodak, and that is a design pattern you should remember. Can I write an automated generated code and put it into production, right? Directly, which

can talk to your data sources, such as, for example, employee records. Is it Are you uh you know, going to take that chance, right? So, the the answer is industry is moving there. So, industry is standardizing something called skills. Now, skills is something uh probably some of you are already familiar with. This is already standardized. There are all major uh you know, companies are actually moving towards

skills-based platform. Databases are not exceptions. Any application landscape is going to go and evolve become instead of code, it will become skills. Instead of skills, I'll generate dynamic code. And it will execute this code very secure fashion, right? And at the same time, in order to handle, you know, code act or any loop that we just talked about, the context graph is very significant stuff. So, this

is this is where your question is becoming very, very relevant. The context graph is going to give clear idea to the skill how it is going to operate. So, there is definitely skills that is touching, uh the metadata and the understanding about the data itself. There are skills that will be actually solving problems. There are skills that will be solving uh you know, how the agent will

you know, go to the data source securely, right? So, there are different kinds of skills that will evolve. Now, if you see, this is exactly what is happening in the industry right now. Every company is investing in this. So, you have previously, you know, static code that was solving data problems. And you would see this increasingly disappearing, but a new pattern will evolve. In this case, it's

going to become a lot of skills filled in. And these skills are actually talking to the data sources and solving it. And especially very specific uh we think that the evolution is more towards, you know, auto-generated code and stuff like that. But it also will need certain elements. Context graph is one, but what are the other things? Let's look at that, right? And this is happening right

now. is where These are your consumers, not humans, right? You used to have, you know, UI where you go and create my plans or, you know, uh I let me create a flow for rag and all of that, right? So, there is a tool, but but you don't do that anymore. You have every customer that you want to talk to, they'll say that I have cloud. Tell

me the rest. I have cloud and I have this. What do you see on the bottom side? I have data sources. I have Claude, I have data sources. And you as a data expert, you come and solve middle of it, right? Can you directly give it give the access to your employee records? Right? So, yeah, we will say in POC that we are confident, but we probably

will never be confident, right? And this is true that and this is a reality that me and Sachin were working on some mission which has to touch the data which has only access to see you. Even we cannot see it. But that's the problem that we were trying to solve, right? Can we use this? But this is where industry is moving from the application landscape perspective, right?

So, we think that you know, we think that a knowledge layer will evolve the same as context layer. This is context craft, but not only context craft. This layer will go in a very agentic way, right? So, internally you might see various types of agents, but this is part of your solution. The this is part of customer. This is part This is outside to data landscape, but

knowledge layer is part of the data landscape. This layer is going to go and talk to all of your data sources, build the understanding, relationships, schemas, and all of that. This is a pre-existing technology, but it'll be a lot smarter right now, right? And so, once that is available, you will have a place and we are calling it sandbox right now. The names may change, but you

will definitely see that this is a lot more skill-based. You will see auto-generated code by you know, Claude or Bob. I would recommend still you to go to Bob you know, booth and see that. Those code will go and talk to your data sources securely. But how do you secure it? You make sure that in the sandbox, no credentials, no uh the you know, no inner details

of knowledge layer, nothing lands into the sandbox. But whatever is required for this agent to perform that automatically, we will enhance that sandbox architecture to allow the agents to do that. We just we should we only focus So, we have tested this and we've seen that all the outer loops are you know, liking this. And when we say that all the customers that are liking this, why

because now they can actually connect their employee records here, right? Nothing is leaked. It doesn't even know unless we tell that it's an employee record database. So, this is where we think that it's going. >> [music]