About this talk
In this talk, Vincenzo Gambino explores the concept of Retrieval Augmented Generation (RAG) and its components, including semantic and hybrid search techniques. He explains how RAG acts as a bridge between large language models (LLMs) and structured data, allowing for contextually relevant responses to user queries. The speaker delves into various algorithms used for effective searching, including K-nearest neighbors and BM25, while emphasizing the importance of tokenization, embeddings, and metadata in enhancing search accuracy. He also covers chunking strategies for managing large datasets within Drupal, discusses the integration of images and PDFs, and highlights the need for effective evaluation metrics to ensure search quality. Finally, Gambino shares tips for deploying a RAG system, addressing performance issues and the significance of user permissions in maintaining data privacy.
Full transcript
Thanks. Thanks everyone. Thanks for joining my session regarding the rug components. Let me introduce myself first. My name is Vincenzo Gambino. I've been working with Drupal, mainly with Drupal, for 16 years. I also wrote a book on Jamstack Jamstack development where mainly I talk about the integration between Gatsby and Sanity. Also, there is an integration between Sanity and an Alexa skill and of course I have Oh,
sure. Yeah. So, and of course I also added a chapter regarding the integration between Gatsby and Drupal as a as a back end. I the way I found the best for me to contribute back to the community is giving talk at events. So, last year I managed to go at all the three DrupalCon and other DrupalCamps. Plus, I started my I I meet up group in my
city, in Palermo. I called it Drupalermo. And I managed also to co-organize the Drupal Camp Italy last year in Rome. In fact, this year we'll be back in on the 22nd of October. Call for paper and call for sponsor are open. You can go to drupalcampitaly.it and if you want to submit paper or ask for sponsor or buy a ticket. It will be in Bologna. So, if
you like Drupal, tortellini, lasagna bolognese, you are more than welcome. Mamma mia is going to be amazing. So. All right. Let's start. Uh this talk was born basically last year when I was at DrupalCon New Jersey. I was talking to Jeremy Bart because I gave my usual talk about Drupal and the AI module and where I show also what the search AI module can do and what
is missing. So, I started to talk to him and I got inspired and then this session was born. Where I'm not going to talk about Drupal, so I'm going to talk just about the content of Drupal and then you can plug in the component as as you prefer. But, I will talk about the rug, what is rug and all its components, the semantic search and hybrid search.
We see how we can refine and evaluate the results that we got. We're going to talk a little bit about agentic rug, and then will be some go live tips when you have your own a standalone rug system that can be done with any any framework you want. You can use line chain, you can use open rug, you can use anything. But, as long as you know
which component, which part of the rug you need to build, then you're fine. You can any system. So, let's start with the problem. So, the problem is that even though the LLM are amazing tools, so they are really amazing, they are frozen in time. So, they are trained on public data. They don't know about your system, they cannot access about your data about your about your system
on your system unless you build something bespoke for that. And they can give just generic answer to specific specific question. So, this where rug comes in. So, rug stands for a retrieval augmented generation. It is a bridge between the the LLM and your content. What it does is basically retrieve your content from your database, and it's passing to the LLM. In this case, you can generate an
answer based on your knowledge. It's good for search plus generation. So, let's see how it works. The the pipeline is really simple. So, the users asks a question. There. Then, we perform a database search. We pass what the result in the context, and we give it to the LLM alongside with the question. Then, the AI will generate the response based on the data that we passed. So,
the concept is really Where useful? It's useful It's useful for search and Q&A, for customer service, if you want expose your internal knowledge, or you can keep the internal knowledge in a private LLM, and then have another system that knows your documents for your employees, for example. You can do recommendation for products in a in case of an e-commerce. And of course, you can train on specific
data based on your data. Why is this Why is useful? It's useful with Drupal because Drupal has a large volume of content, and they are also structured, so they're pretty quite complex. It has taxonomy, which is a reference, so we need to create those relationships. Uh the content change over time, so we need to update our database. And then, is where you can use it when uh
the search that you want is based on context, and not on keywords. So, we're going to talk a little bit about numbers and algorithms, because it's important to know when you're building a RAG system. So, the first number that we're going to talk about, simple, is token. So, what token is is basically translation of a word from letter to numbers, because the LLM does not understand English
or Italian or Spanish. It understands number. But remember that this is not the meaning of the word, it's just the numeric representation. So, the letters there there translated into number. To know the meaning of a word, we use another number, actually a vector, which is an embedding, which is called embeddings. And these embeddings are saved in a vector database, which you has spaces and dimension. The closer
two vector are, the more related they are. If If they are far away, they are not related at all. So, you can find similar word based on the distance. So, and it looks like this. So, we have Drupal with which has this coordinate. Then we have CMS with this coordinate, taxonomy, and module. They are all pretty close together. And then we have the term banana that is
quite far away. So, and this is the visual representation of a vector database. So, this is the cluster where we have our our our vectors all with Drupal taxonomy. And then far away, we have the term banana. And most likely if here there would be also other terms like apple or um or peach or any other fruits. So, but how the system can perform a search? Now,
we go into the algorithms. So, we have the the nearest neighbor, the KNN. Uh this is really good, is really accurate, but it's slow. It's slow because basically it's loading all the at once and it's checking one by one. Then we have another version which is the approximately approximate nearest neighbors, which is also accurate, but it's faster. And then we have the current standard which is the
hierarchical navigatable small world, which basically is it works in layers. Now, let's see one by one. So, KNN, as I said, loads everything. So, if you have 1 million of documents, it has to load 1 million of documents and calculate the distance of each of them. Then it's not easy to scale because if you have like 10 times data, it's going to be 10 times slower. So,
and if you have for example 1 1 million vector over 768 dimensions, so you have millions of vectors there. So, it has to do all those operation per query. The navigatable small world, which is not the hierarchical, so this is the the the the part before. So, we have here the entry point. So, what it does is select any point, random point, in in the vector database
and it starts searching which one is closer to our query. So, it starts and checks this one, this one, this one and then this one. Okay, this is the candidate is closer. I go here. Now, we'll check this one, this one and this one. Okay, now we'll go here and so on until it goes closer to the query. It's not accurate as the next one, navigable small
world, which works in layer. It has the same principle. So, it chooses a random point in the in the vector database and it search the closer to our query. After that, it goes one layer down and it loads all the vectors around this entry point the other entry point here. So, it finds the nearest one and then it goes to goes again. Goes another layer, loads all
the vectors around this one and it finds the closest one. So, this at the moment is the standard that we use. So, but how do we calculate the distance? So, going back a little bit here, we calculate we have to calculate this distance to see which one is And we can use this similarity similarity matrix. Uh those are also present in the search AI module in Drupal.
So, when you when you when you when you add the search with AI in Drupal, you can choose the similarity matrix. Well, we have the cosine matrix, which measures the the direction of the vectors. So, if the the the vectors are facing the same direction, they are similar. If they are start to go to different direction, of course, they start have different meaning. In this case, we
measure the angle here, the angle here, we do the cosine and then we get the result. One is closer, zero is not related, minus one, so is they're facing different direction, they are opposite, like good and bad, for Then we have the Euclidean here and you just measure the distance between the two vectors here. Of course, closer the distance, more related they are. And then we have
the dot product. Dot product, what it does, we have the two vectors and then we cast a shadow over the other one. And of course, the closer it is, the shadow the the shadow will be bigger, so they will be more related. The farther it is, it will be less related less related. So, those are the three similarity metrics that we use to to search in the
vector database. So, the semantic search works like this. Yes? >> Uh I was just going to ask Does that have to do with all three for answering the question on that? No, no, you can choose any any. Yes, it's just to know because in the Drupal module you can see those, so at least you know what they do and uh when you select them select from there.
So, the semantic retrieval works like this. So, we the user asks question, the question is embedded, so the the system captures the meaning of the of the question. It compares the vector, it finds the nearest vector, and then it returns the relevant matches. So, basically is what it works like this. So, we ask how to create a Drupal controller, our question is embedded. Then the database will
return a list of results that we define using the top K parameter that can vary from 1 to 10 to 50, whatever you want, which limit we want. So, we want to return 20 results, we return 20 top K will be 20. If it's five, will be top K five. Then we pass all of the results in the context alongside with the question and the LLM will
generate the answer. Now, before we go deeper into the into RAG, I want to talk another technique because as saw, I always put the context here, so results from there. So, the question is, why I cannot put all my content in here in this content? Well, you can. There is a technique for that, and it's called cached augmented generation, where you put all the knowledge directly into
the context because now the LLM have really, really big, uh, really, really large context window. Uh, when we start 3 years ago, I remember it was like 2,000 token, uh, the limit, something like that. Now we are around around like 200,000 token for Claude or 1 million tokens as well. And then it's a simple approach. There is no retrieval layer. So, where does work well? When there
are when there is a small and focused knowledge base. The content is always fresh. It's a simple architecture, I would say. There is no index and no retrieval. And it's good for a limited documentation set. What is the, uh, the cons of the OCG? So, basically, uh, we pay per token, as we know. So, the more token we add in the context, the more we pay. Then,
if we have 100,000 nodes and each node has 2,000, uh, token, so we're going to have lots of tokens to pass in the context. That would be large. And then the LLM with really, really large, uh, context window, they don't work really well because they can get confused and miss some of the important part part. So, it's not really practical for for Drupal. you got any questions
so far? Okay. So, if we cannot put all the content all the our content into the context because we have this token limitation, what we can do? We can do the chunks, for example. In this part, we're going to talk about chunks and metadata and media, how we can, uh, index it. So, chunks is really simple. So, we take a content and we break it down in
smaller, uh, smaller part. So, each piece is one searchable unit. It will get its own embeddings. Um and we can retrieve only the part that we uh that we want and not the whole uh document, for example. So, in this case, we have this node which is Drupal 10 migration guide, which has 5,000 words. And then we can split it into chunks. For example, we have introduction,
prerequisite, preparing your modules, database migration steps, testing and validation, and post-migration cleanup. So, there are different technique for um to uh to do the chunking. And the benefit of chunking is that we have a smaller context window, so less uh less cost for us. Is that there is a better attribution, so there is no block source. Scalabilities, you can we can search millions of chunk with chunks
uh really easy. And then we can of course with Drupal, we can re-index the chunks or entire document as we want um on the fly. So, the chunk techniques are fixed at size, recursive, semantic, and Drupal specific. So, we start from fixed size. So, we split the content every number of tokens. So, for example, we want uh token is five uh uh 2,000 tokens, so we split
into 500 tokens. There will be uh four uh four chunks uh of tokens. So, we can split it like that, but it ignores, of course, the boundaries and the meaning of the content. So, the pros uh the pros is the is simple to implement. Uh we get predictable chunk size. It works with any content, but of course, it breaks the semantic boundaries, can split at the end
of the thought, and there is full context preservation. There is another technique that goes with a fixed size, which is the overlap. So, it means that you get 500 tokens from this piece of text, and then you add on top 200 tokens, 250 tokens from the previous chunk. So, this at least has a little bit more more of context. The other one, which is better, is recursive
or semantic. They work similarly, but one is more mechanical, the other one is not. So, recursive split the content along natural boundaries and respect the heading, paragraph, and section. So, for example, the Sorry, the recursive the For example, we can split by deep, by section, and many other. While the will understand the content and then breaking down based on the meanings. So, the pros are that we
can preserve semantic coherence, uh completely assist together, we have a better retrieval relevance, and follow the natural citation boundaries. The cons are that, of course, we need to parse the content. We need to see where to break it or pass it to the LLM to get the meaning of each part and then create the chunks that will be variable, so we will not know the size of
each chunk. And then, of course, it depends how well structured the content is, how it's written. Talking about Drupal, we can chunk it into Drupal specific, right? So, we have For example, we can do a field-based chunk, so we have the body and then the paragraphs and all the entities that we have. And also, we can use the metadata including our node to add to be added
to the to the chunk. And we talk about metadata in a bit. So, we can store the product node with descriptions, tags, review, each of them as a different chunks with a reference to each other. So, the pros is again is is good like the semantic and the recursive, but the same thing, so it's Drupal specific. It requires understanding on the content architecture, so we need to
make a study on how we're going to chunk it. And again, we have the variable chunk size per content type. what can we store on each chunk? Once we had done the chunking, we can store it. So, we're going to store the original the vectors that have been generated from the embeddings, and then we can add the metadata, like the source info, ID, taxonomy, and timestamp. All
this is the three components that work together. So, this how it looks like. So, we have the ID, we have the vectors with the vectors here, the text, and the metadata. Metadata are the structured data about It gets stored alongside with the with the chunk, and it enables filtering, and it complements the semantic or the Because it is doesn't work as is not searchable, is used for
filtering. So, in Drupal, what we can do? We can do many things. So, we can store the content type, the status, the language, the creation date, the author. So, this is important, for example, if you want to implement them permissions, taxonomy relationships, and then we can also store the field this coming from. I mean, we can store all the data that we want in the in the
in the metadata. And then it will look like this. So, we have the chunk, the vectors, the text, and then all the metadata that we want to to add. So, how can be implemented in retrieval? As we said, it cannot be used alone. So, we can do the pre-filtering. So, in our search before we do the search, the result before we do the search. Post-filtering, it does
it after. Then we can filter by a fixed, for example, fixed value, like the body field, the author, or the editor. So, this is really useful for permission, because this is one of the problem of the the of rank, so the permission layer. So, how we can filter? We have the exact match, we have the multiple values values, so is in, the range filter, and the existence.
So, pretty simple. Now that we know how we can embed text, now we pass on the on the images. We need to use the multi-model models like big line of flores, and basically the same way we pass the image, and then it will be processed, it will be understood, and then we'll create the the the embeddings, the entering the vector database alongside with the metadata. So, for
example, we can add the media the media ID here, so the media ID, the file path, whatever. And when we do the retrieval process, we can pass the embedding plus the metadata, and then of course we can load the image or just describe the image that we have loaded. Same thing or similar for PDF. It also depends how the PDF has been done. If it's just text,
you can use these tools just to convert from PDF to text, and we can then chunk again and add it to our If it's a standard PDF like an image image base, we need other tools. And if it's a mixed PDF, you need to extract all the text that we have, you need to understand where the if there are any images, graphs, if there is like one
column, two columns, and stuff like that. So, the PDF running will be a little bit more complicated. these are all the the challenges of the the PDF. So, they have no text semantic structure, page hasn't got the logical section, there is lots of noise with headers, footers, and page numbers, tables multiple multi-column layouts break text flow. Then we can do the chunking on page base. We can
detect the section and images with the layout analysis. And then in the post-processing, we can remove the headers and footer to clean the the page break, for example. Okay. Any questions so far? Yes. What happens if you process private data? If you If the data has risk and it includes many private data as a source. Okay. You can use If you If you want to If you
want to store private data, you need the private LLM in that case. So, all you is store it in your own your location or you can use any other private data, for example, Amazon IO provides the the private data. The private LLM, so you can you can use it. Or you can build something on your own on AWS on any other provider. So, now we talk about
the hybrid search. So, we talked about semantic search and now we talk about the hybrid search because we can do two type of search. We have the keyword search, also called the sparse vector, and then we have the semantic search that is the one that we which is called dense vector. The keyword search match the exact word. It uses the the N25 and TF-IDF algorithms that you
will see in a bit. Then if we look for migration, you would only found document with the migration word. While the semantic search match the meaning, use embedding for cosine similarity for as algorithms. Then if we look for migration, it also finds upgrade, transition, and moving. So, this is how it looks, sparse versus dense. Sparse is a keyword, so where most of the values of the word
that are in your content uh will be stored in a table. So, all the value will be zero and where the word is is included in your content, it will be one. It's fast and it matches the exact meaning. While the vectors has all dimensions, then it has the the the vector the coordinate and it is lower, but it captures it captures meaning and fuzzy matching. So,
when to use each approach? Uh the keyword search when when you when you need to match product codes uh product codes or technical name or the exact name. Like for example, Drupal version or the node access or the hook alter form. This is when precision precision uh matters more. Semantics is when you when you have a conceptual query and you want to do like a chatbot. So,
how do you control who can edit the And this is the is is good when the recall matters more than precision. We'll talk about precision and recall later in the slides. So, this is a document matrix as I said. So, we have all the documents, let's say our nodes. And here we have all the words that are in those documents. If the word is present in one
document, it's one, otherwise it's zero. So, when we the user asks a question, this is how it works. He asks how to create a Drupal controller. Each part is treated separately, so it will be broken down into each keyword and it will be assigned a value each time the the word is in the prompt. Then we create a sparse vector, again similar to the one that we
we saw before, with all the words that are in our vocabularies, but in this case we add one only for the word that we have here in in our prompt. So, the previous one, this is our document matrix like our um database or the vector database where we store all the content that we have in sparse vector. And this is when the user has a search question.
So, in uh in semantic, here we have the vector database with And here as well we have the embeddings, but for sparse vector, which is the keyword, we have um we have the we have this we we write that. So, what we do now? We start checking the documents and we assign points every time the word appears in the document. So, after we do all the checks,
we look something like that. So, the word how in document one is mentioned one uh four times, then seven, and so on. So, those are all our nodes, for example. So, and then we get the score. The document with the highest score is one of the candidates to be the one that uh will match the search. But in this case, you see we have a problem. So,
the word A has got 32 matches here because of course is pretty common word. And there are techniques to avoid that, which is this the term frequency. So, we measure how often a word is mentioned in uh in a document because more appearance doesn't mean more relevant. So, we normalize um this score by the the length of the the document length. So, we do the number of
uh that there are in the in the document divided by the total words in a document. This balances already a little bit the uh the score for the word for the common words like the A and and so Then we have the inverse document How this term uh how this how how rare is this term in all our document? So, rare terms are more important. So, again,
we score them uh waiting the terms using the IDF for inverse document frequency. So, we have our term frequency and we multiply by the log of the total docs that we have of the nodes and all the document that contain that term. So, we see how rare is it is is that word. So, it will look like this. So, in we have 100 nodes, let's say, and
the and the word controller appears in five documents. So, our score is 0.05. Why the word two appears 100 times in all the document appears at least one time in all the documents. So, the score is one. Then we do the IDF, so we invert the we invert the the logic. So, we we do over 100 document, we have five document that contains controller. So, the score
is 20. The same thing happen on the word two is all always one. And then we do the log to balance a little bit more because 20 is too high as number, so we we balance it using the log of 20. And here the log of one is zero. So, the score is 130. So, at this point, our score looks like So, we have the word how
which has how which has a really really low score, while the word create has high score, the same as controller and Drupal. The same thing happen here. So, we have and Drupal. So, most likely this one is the candidate because has a higher score and contains both word Drupal uh sorry, create and controller plus Drupal, while the document four is also a candidate because has a high
score. It's talking about controller, but not how to create it. So, we'll we'll we'll retrieve it also. Also this uh this document. So, in this case, for example, in our case, we have three documents in we found three documents with the word Drupal. And here we We the document ID with the score the board controller uh has been found twice. And we have this score again with
the ID and then the migration. Have you got any question regarding the the scoring? don't know. This at the moment is these fast vector. These are different one. We'll see the score for the embeddings later in the slides as well. So, this is for the keyword search the moment. that algorithm TF-IDF is good, but there is a lot of variation which improves the the previous algorithm, which
is BM25, which stands for best match 25. 25 is the number of the variation of the of the algorithm that then won the race alongside with all the others. uh you don't need to read the the algorithm here, but what we do what we do basically we have two parameters, which is tunable. We'll see in the next slide. So, we can fix the term frequency that we
saw. So, if a word is included in one document 10 times, it's four let's say 10. If it's if it's included 20 times in the same document, then it's called 20, which is too high a score. BM25 BM25 balance this score by adding the the the K parameter. So, we can say you can score 1.3 instead of one, so it's a little bit less. So, and then
we also have the document length normalization. So, in TF-IDF the long documents are really uh heavy um they have like really like penalized uh let's say we have a good article about controller, but controller is only mentioned five times. So, the long document like board controller then we get a lower uh score, even though the is irrelevant. So, with BM25, we can fix that uh adding the
adding the parameter to have like a smaller penalty on the the longer So, we have K and B here. So, how much the frequency influence the score, we can go from one to two to to from one to to two. And the length normalization goes from zero and one. So, how this past retriever uh is good in practice? So, again, for the exact term, so we have
to from alter and node access, all these words will be perfectly matched. So, for example, if the query is node access permission, BM25 will find the exact API documentation with node access. While the semantic search may return general access access control uh concept. So, how they can work together? Because both of them can work together to get better results, and this called hybrid search. So, we start
with the query, and we send one to the keyword search, so with the sparse vectors, and one to the semantic search with dense vector in our So, we retrieve the content, and then we will mix them together using the reciprocal rank fusion. So, the reciprocal rank fusion uh what it does is scores a point uh based on the ranking of the uh of the results. So, on
each uh search, so in this case, in the previous slide, we have two searches. So, the first document will get one point, the second document B will get 0.5, the third document Y will get uh 0.33, and so on. Same thing will happen with in this document. when we want to apply the the score for the for the index where they appear, we use the K another
K parameter which is different from the from the previous one that we saw. For example, if the one document rank is ranked first, we do one divided by the first the score. So, it's one. If it's 10, we do one divided by 10. So, there is like 10 times difference. Which means that all the top document are always at the top of the list. And we want
again to balance that adding another value in the K, so which would be 60, which is commonly used. So, the first will be a one plus 60 plus the the rank. So, the first, so So, 61. And the 10th again, it would be one divided by 60 plus the rank, so 70. And there is less difference now between the two. then we can use the beta parameter
that basically we we give priority to the semantic search or to the keyword search. So, if we do 0.7, it will be 70% from semantic search and 70 and 30% from keywords. 0.5 is 50% each. 0.3 will be 30% from semantic and 70 from keywords. So, after we've done all the all the score, we have this table here. So, the document A score 163, 161, 158, and
so Then we add this one, document A, in with the with some document A with other document A, so we got the result here and it's second. Then document B, so 161 plus 161, so is both the score it was on top of the list in both of them, so it gets highest score and so on with all the others. This case we can have two different
searches that will bring different results and then we can use the rank fusion to put it together. So, those are the key strategies for searching. So, we have the metadata that cannot be used standalone, but they can just narrow narrow the search that we've done. Then we have the keyword search that scored the document based on the keywords that they found. And the semantic search scores the
document based on the similar meaning from the Do you have any question about the the previous part? Yes. Uh with the hybrid search how it just looks at the placement for each result wouldn't it be better to look at the the relevant scores that it gets for each one? So, if you have one result that's way more relevant like number one is way above number two. You
you think you'd want that to be conveyed somehow other than just it's number one and then you got number two which gets almost the same scoring or would that be technically difficult to do? Mhm. You can you can try to alter the the scoring then. Because what it does here, we return the result based on the interpretation of the search and the search. So, this understand the
meaning and return some document that the system thinks that they are relevant. The same thing happens here, but they are not always the same. So, we need to use a way to to merge it. This is one of the common way. So, but you can try to use any other algorithm or other system to to get what you want in the in the in the position that
you All right. So, now we'll talk about refining and evaluation. So, what we've seen so far, how the buying folder works. So, this is how we do we perform the search. So, the buying folders is that the prompt gets the embeddings and the and the chance has its own embedding in the vector database. This is the most common and what we saw so far. But then there
is also another method which is the cross the cross encoder which is much more accurate. But the problem is that he gets the document, so our note and the and the prompt and we'll check if they are relevant. They will do that for each note that we have in in our database. And we cannot save the embeddings before, so we cannot precompute like we do in the
in the buying order that we save in the vector But the solution is Colbert which is contextualized the late interaction. I got the word twice here. architecture. So, what it does, basically we treat the the prompt, in this case for the semantic meaning, as a single keyword. One one word will be a keyword. And each keyword will get its own embedding, its own meaning. The same thing
will happen on on the other side, on the on the node side. Each node will have will be broken down into words and each word will be saved in the So, of course, it's more accurate, it's faster, but because we need to store all these these embeddings, it will be more the cost will be will be higher. And once we do the search, then we can aggregate
the results using the max sim scoring. Which looks like that. So, again we have our query here. Here we have our keywords in our in our And then we start scoring based on the similarity of the word. So, two is minus one because it's not similar, it's not relevant. While uh controller and Drupal and route, they are really similar to each other. So, here we have our
prompt and here our document or the words in our document. And then we add all the score and we get the score for that. Another technique that we use to to improve uh searching is the query parsing. Because as we know, the user uh sometimes doesn't know to ask the right question and there is lots of noise. So, for example, they can say, "Oh, I need help
because this happens." And then they start what we say uh something useful. So, "I was setting up my document on my new Drupal site yesterday. I ran into a weird issue. The form wasn't validating properly, something about the CSRF token, I think." token, I think. So, here is um is there's lots of noise, like lots of words like string, yesterday, setting up, and this kind of stuff.
And in this case, the intent can really be hidden uh by all this noise. And what we do, we just rewrite the query. So, we ask the LLM to rewrite the query before retrieval. So, we have the system prompt. And um and we write the query here to ask to to remove all the necessary context, use the technical terminology, extract the content's intent, and keep it concise.
So, then from here becomes rewritten like this: Drupal compilation, CSRF token error. And then of course, has a massive improvement on on the Other things that we can do uh like query parsing are the name entity recognition. So, we can extract uh version number, APIs, and the keywords from the prompt. And we can pass it as um as metadata. And then there's another one which is the
hypothetical document embeddings. What we do, uh basically uh the user asks us a question, "How to write a controller in Google?" And the system we kind of write an answer for that, and we check how similar they are. I never use this one. I use always use named entity recognition and the entity quality parsing. Then, we also have the content injection when we're going to do some
of the query. So, we can inject anything. So, we can add other data to our prompt. For example, the user context, role, permission, preference, the session context, any data, and also location token. Location context, sorry. So, now that we get the results, we need to know how we can see if they are good or not. And there is some quality metrics, some tests that we can do.
So, to do the test, we need a list of that we that we write, the ranked results that we return from our and then the test data. So, all the document that we know that should return from this list of prompt. So, what we do, we start the test. So, and we need to measure the precision, which measure how many of the returned document are relevant. And
then the recall, how many of the relevant document that we have in our system are returned from the query. So, for example, if in our system we have 10 relevant documents, and we do two queries, different queries, but we expect those 10 documents to return. We In the first query, we have 12 documents, which only eight are relevant. So, it means that we have a precision of
eight over 12, so it's 66%, and a recall, so we have only eight document out of the 10 that are relevant, so 80%. In the other search, we have we're doing a little bit better with a recall because we have 15 documents, but nine are relevant out of the 10, but the precision went down. So, when you build the system, you can choose if you want more
precision or recall, or you can try to work on the query and on the system and how you store the the data in your in your vector database to have those up closer to the 100%. So, this is what it does. So, we have the the rank here of the results. So, we have the precision at five. So, from in the first five, we have 40% because
only two out of five are there. Precision at 10 is 60% because only six out of 10 are relevant. And the recall, this time we have only eight relevant document and only six are showing here, so it's 75% in the first Then we can calculate the score on the mean average average precision. So, what we do, again, we have the relevant document, which is first. So, we
score it. Then, at the top two, we have only one relevant, so is one relevant over two, so it's 0.5. The third position is not relevant, so again, one over three, so it's 0.33. The fourth is relevant, so it went up again. And then we have the fifth, which is three over five, and then six is not relevant, so we score it like that. After that, we
because we do many searches, so we have different prompts when we are testing this. We can then calculate the average precision over all the over all the search that we have done. So, we can see how those prompts are working. Then there is the reciprocal rank, which basically measure only the first relevant document. So, when does it appear? So, is it the first document, which is one,
so we get one point. If it's the first relevant document is the second position, we get 0.5. And if it's at four, we go to 0.25. So, the searching in this case is not working really well because we want the most relevant at least in one or second position. And then across all the searches in the best that we do, we can calculate the mean reciprocal ranking
by adding those uh results. So, it looks like this. So, we have the number of search, the relevant document, and the score. Now, we talk quickly about the agentic RAG. We got any question about the the scoring and testing? So, the basic RAG it uh has a fixed retrieval rules. we have only one LLM to process everything unless we build something um like a workflow to process
more stuff like for example the pre-parsing. And there is no self-correction for iteration. In the agentic, we can choose that the agent can choose which retrieval strategy is going to use. It's going to use keywords or the um or or hybrid or just semantic. Then we can you can choose which LLM is going to do the task. And of course, you can have iteration and parallelization. So,
it's going to look like this. So, the user will ask a question. So, the agent comes in and we start doing our the search and then we start evaluating. So, is the result good enough? If not, it will go back here and it will use the tools to do other stuff. So, a query rewrite and it will search again. Uh then um do we need to check
other resource or is this resource reliable? Then it will do the hallucination check. So, we have the tools to do the hallucination check. So, it will do this loop until it's happy. The um the the answer the the result answers the question from the user. And then it will send it back to the LLM and then it will produce the answer for for the user. So, now
there is some go live tips. So, the the problem with Prague, of course, is the spinning performances because the more data we put or the more traffic, we need more more more infrastructure. It also increases latency latency and load. More requests, it means higher memory and complete cost. And then it's difficult to scale while keeping the the high performances. Then again, we have the real data which
have not been scraped yet. So, we need to break down all the data we have, all the content, find a strategy on how we are going to we're going to save those data in our Usually, PDF are really really difficult to to do. Images are a little bit easier. It's light as well like the PDF. So, you have a strategy for that. the security. We need to
access Well, you need to extract tools We need to extraction tools to get all the information from PDF and images and Then again, the user can make unpredictable prompts like this one. How many contributed modules can I enable before Drupal becomes self-aware? So, we need to find a way to to handle these cases. And then, of course, security privacy. Here, Drupal is really useful because of course,
we the metadata we can store the node ID, the media ID, whatever we had there. And then once we get the result back, we can check if the user has permission to access this content. Using agent, we can then say, "Okay, the user cannot access this content. Let's search something else." So, Drupal really really helps on this one. But otherwise, it'll be similar I I don't know
a SQL database where there is no access permission access access control. So, but Drupal again helps on on that on Then we also have the key metrics, so we can measure the software performances to track latency with logs, memory, and compute. We can add logs everywhere to see where it started the question, why it's taking so long, what comes back comes back, so we can monitor it
in that way. And then the quality metrics, so measure the user satisfaction over the output. And the evaluator types are called, so we can measure the entity related to the asset. We can use the LLM as a judge, so the another LLM can check the results that come back and then can say yes, it's relevant or not, and add it to a log, so we can monitor
it. Or we can use our users with thumbs up and thumbs down. So if they like the the answer, they click up, otherwise down. So I know there are lots of information. And so now we do a small quick recap. So so we saw what rag and cog is, we saw the differences, we saw what chunking, metadata, media are. We talked about hybrid search, sparse versus dense,
which is keyword versus semantic search. Then we saw how to refine and evaluate our system, and also how to add observability and metrics to our system. there is a lot to talk about rag or rag really a lot. Those are all the topics that didn't make it to this presentation because it's already five as well. So we have quantization, which is compressing the the vector database. Then
we have specific tools for monitoring like ragas, phoenix, datalog. Then we have tools to build the the semantics, the the for code our agent. Then we have open rag, open search. This These are all tools that we can use if you know the the base the basic component of rag. We have graph rag, which captures the relationship between words. So for example, a Drupal was created in
2001 by Patrice, so we will we will know that Patrice is the creator of Drupal, so we add this the relationship in a separate database. Multimodal, I didn't talk much, it's handling images and PDF. I had just had a couple of slides, but I wanted to talk more, and then post control. So, again, I know the world of the information, there is a lot to process, but
hopefully you I was clear. And so next time you are talking about Rack, you will not do you will not have this space here, so at least you will understand a little bit more. And and yes, thank you. If you want to connect, I'm on I'm on Slack, I'm on Slack, and also on on LinkedIn. Okay, you got any question? Are your slides going to be available?
Yes. Yeah, yeah. Yeah. Um you mentioned before about fixed chunking, Smith chunking, Drupal specific chunking. If I wanted to make my own Rack system, is there a tool I can use for the Drupal specific version of that? Or is that something you have to kind of have to do? So, the search API module in Drupal provides some chunking technique already, so you can you can have a
look at that. There is something new in the module in the 1.3 version, so they added also the re-ranking. So, when you get the data back, you can re-rank and see if they are relevant or not. Then there was the cosine similarity, Euclidean, and dot product, and also the chunking, as far as I remember. Okay. if you want to learn more, I usually use I follow this
which is really good. It doesn't take long time. Like it is like say 3 weeks and hours per a week, but it takes less, really. And then also I'm reading this book. So it's it's free on the deep learning, but you need to pay 400 1 euros if you want to do it from here, but you get more like notes and other stuff plus the certification. So
if you don't care about certification, no, just go to the deep and deep learning AI. And then I read this book, which is really really useful. It talks about all the basic, all the components of AI, not just rag, but not everything. Which is really good. Any other question? Okay, thank you.
