About this talk
In this talk, Mel Brower discusses the development of a modular retrieval-augmented generation (RAG) pipeline while working at ABNRO. The speaker explains how they initially struggled to locate a colleague named Brian using a basic AI tool, leading them to realize the importance of building a custom RAG pipeline. They emphasize the need for quality data, speed in retrieval, and cost-effectiveness in developing AI solutions. The talk delves into the design of the modular RAG pipeline, breaking it down into components such as indexers, chunkers, and retrievers, and highlights the flexibility to experiment with implementations. Mel demonstrates the creation and configuration of various data loaders, showing how this modular approach facilitates efficient testing and retrieval of information.
Full transcript
Yeah. Am I audible for everyone? Yeah. All right. Perfect. Hi everyone. My name is Mel Brower. Uh and today I'll be talking about building a modular rack pipeline. This is me. I'm an AI developer with ABNRO. Uh I work with the AI team for innovation technology. And the fun thing about me is I basically try to automate everything. Yeah. Is it okay? All right. Uh and this
is Brian. Brian is a 10x developer at ABNRO. He's super smart and he loves coffee like most developers. Um, so why am I introducing Brian? That's a little bit weird, right? Well, let me tell you a story. So, recently I was talking with my manager and my manager asked me to start building this new generative AI project. Told me, go get together your own team. Um, but
if you really need an expert on the subject, reach out to Brian. Right. So, with any topic, there are I guess three groups. First of all, you have Hey, my clickers. Ah, there you go. First of all, we have the clueless people, right? These are the people that know nothing about AI, they haven't done anything yet, maybe with a raise of hands, anyone here who's absolutely clueless.
Well, great. Happy that you guys are here. Um, and then we have the second group. These are the wannabes. They think they know, but they don't really know. I myself consider myself a wannabe. Um, any wannabes in the group over here? Okay, nice. Okay, the rest of you who didn't raise your hands, I would consider you guys the experts. You've been doing NLP long before it was
cool. You guys know BERT on a firstname basis. So, um, if you guys have questions at the end, you know who to reach out. Um, who are the experts? Show of hands. Okay, not a lot of hands. All right, so I got together my team, right? And, um, we were mostly wannabe people and clueless people. Uh, and our deadline was coming closer and closer and closer and
we were starting to get worried and we realized we need to find Brian. All right, so we tried sending him messages. We tried calling him, but you know, 10x developer, very busy coding, very hard to reach. So, how do we actually find Brian? Well, let me ask you a question. Maybe uh if you don't know something, how what do you do? You Google it. All right. And
you same Chad GBT. Right. Right. Okay. So, we got chat GBT nowadays. So, we thought the same thing, right? We asked our internal AB and MRO chat GBT, where's Brian? And then it asked us, who's Brian? I don't know who Brian is. Well, that's a little bit of pity, but makes sense. Um, Chpt is drained on everything on the internet, but not on the internal ABNO stuff,
right? So, we had a problem with one of my team members, she was very smart. She was like, you know what, we are AI developers. We can build our own tool. And um, she told us about something called rag, retrieval augmented generation. Basically, it means taking some information from elsewhere and putting it into your large language model like chat GPT and then it can reason about it
and it can answer your questions. So, we thought great. Well, within ABNRO, we have a lot of data, right? We have like data of people coming in and out of the office. We have information about who's using a coffee machine and all these kind of things, who's booking meeting rooms. So, we figured just put everything in. We'll probably be able to find Brian. So, we got to
building our uh rack pipeline and this is us. Um, and when we were done, we asked it again, where is Brian? And it told us promptly, Brian is at B2 near the printers. So, we thought, great, that was easy. Well, we looked at the printers, but there was no one there. I guess no one has been there in a long, long, long time. So we kind of
checked our logs and we figured out that we were just feeding it the wrong information. So we learned our first lesson. Quality matters when building a So again we thought okay we need to make it better right. So we were reading all these papers about um yeah different chunkers and retrieval strategies and all these kind of things. And then once again we asked it where is Brian.
So actually 20 minutes later it told us Brian is on D30 next to the coffee machine. We were like okay great cool. So we went over there and well there was no Brian but there was a janitor cleaning out the coffee machine and he told us that Brian had been there 5 minutes before. So that's actually good news. a rack pipeline was working, but we learned a
second lesson. Speed matters. Then a colleague of mine went like, you know what? I think we're going at it about in the wrong way. Maybe we need to ask it really nicely. So he typed, where the [ __ ] is Brian? And it went like this, like no profanity. It's not allowed. So okay, we had to improve our rack solution yet again. So, it was already good. It was
now time to make it fast. Well, how do you make it fast? Well, you throw everything Azure has to offer you uh at it. And you know, it's going to cost a lot, but it might be fast. So, once again, when we were done, we asked where is Brian? And immediately told us, Brian is at the lunchroom. So, we went over at the lunchroom, and there we
found him. We found Brian. Well, our search was over. So we explained to Brian how hard it was to find him that we built this incredibly fast, high quality but expensive rack pipeline, right? And so he kind of chuckled and he said first time building a rack pipeline. Don't worry, I've been there too. Um the problem that you guys have is that you just built the first
thing that came to your mind. Instead, I nowadays build something called a modular rack pipeline. It allows me to really easily check out all these different configurations on how you can build a rag pipeline and then check if it's the right fit for my project. Well, I was very happy with that because my boss actually got the bills from our new rack pipeline. He wasn't too happy,
so to say. So, we learned our So, we learned our final lesson. Money matters, too. Right now, when we're talking about a rack pipeline, there are these three kind of main things you want to take into consideration, right? How fast is it? Is it like user facing? Then it needs to be pretty fast, what is the cost? What is the quality? What do you need? How are
you going to test it? So, every project is different and every project needs like a different balance of these things because it's always a balance. You can't have all of it. usually pick two, I would say. So, let me quickly go over some of the rag basics. You might have already heard some. I guess yesterday there was already a course on rag basics. Um, but like I
mentioned before, a user has a query and we get some information from somewhere, right? And we put it into a prompt to the large language model. Well, sounds easy enough, right? Well, rag reality looks a bit more like this. So usually at first we have to create like an index. Um so we take a lot of our documents, our information. We extract that information. Then we need
to chunk it up into bite-sized pieces because we're not going to feed all our documents into our large language model in one go. Then we need to embed it. So we actually need to turn uh our text into uh numerical representation. And then during uh query time when the user has a query we are actually retrieving from this index right so um yeah there are all different
types of retrievers that you can think of you have like a naive just cosign similarity you kind of check hey okay these are the relevant parts but nowadays there's a lot of uh information that you can find out there on how to do this more efficient for example you have rerank which will use a transformer for example to kind of check what documents are that are being
returned and then uh you know reorder them based on what it actually thinks is relevant. So we see another thing uh we might also want to include like search engine or like uh a relational database or those kind of things you know um we might need to route that and you see these purple and orange dots uh and in all of these um we need like a
large language model or we need like an embedding model potentially for any of the implementations. I'm struggling with the with the clicker. When we break down a modular rack pipeline, because what we just talked about was like the basics, we have to break it down into components like I just showed you. We have an index, we have like a retriever, we have the chunking, we have um
the document loader, all these kind of things. So, yeah, junkers, retrievers. These components then can be implemented. So the component is the abstract part of the rack pipeline, but the implementation is the actual code of the thing that you're going to run. So for example, it might be a semantic chunker uh or rerank like I just mentioned. So a semantic chunker, for example, is when you want
to split up a document, but you kind of want to make sure that the part where you make the split, you kind of have all relevant information in the first chunk and then the relevant information that belongs to the second chunk is only there. So in this case you might need like some resources like a large language model or an embedding or maybe a prompt. Um and
in the case of a semantic chunker you might need an embedding model. Also for some of these components you might need arguments. For example if you have a retriever how many documents are you going to retrieve? Well that's covered in the arguments. So in the basis you create a central pipeline. you split it up in indexing and retrieval because indexing you usually do once hopefully uh and
then retrieval you kind of do every time a query comes in. So you then compose this with a user interface or with a configuration file and the idea is that you kind of select hey now I want to try it out with a semantic chunker now I want to try it out with a fixed chunker let me try rerank for retriever and then you use meta programming
which is a very fancy word but it's not that complicated it's just in this case the import lip module uh from Python that during runtime based on the strings that you provided kind of knows which code it should grab for this version of the rack pipeline and then you test it against an evaluation set. Well, we know evaluation sets are probably the most important thing that you
have when you're trying to build any uh genai uh applications. So, please do that otherwise you don't still don't know what you're doing. Okay. So, in the basics this pipeline we have a query we have some data it goes in a component and then data goes in a component in a component in a component. So it's really like abstract, right? And at the end we get a
response. So this is basically what the code kind of sees. And then when we take a look at these components at the base level, we have the base component. Every part of the pipeline is a base component. And it has one function. It's process. And when you call process on a component, it will run the functions that are below which I will describe in a second. like
it will validate the input, extract the input, update the data and then it will run the component. Now let's take an example of a base chunker. So this chunker you kind of need to define okay if it's a chunker what sort of information is this chunker allowed to receive and how are we putting that information back again in the data object. Finally we have the actual implementation.
You as a user of the modular rack pipeline only need to build the run method. So for the semantic chunker, you need to provide the logic for what actually happens when you run it. Right? So this kind of allows you to write byte sizes of code for each of the different parts and really easily test them against other implementations. So let me go briefly give you a
demo. Um, I will have to switch the way the screen is mirrored. Decimator. Uh, let's see. Screen mirroring. Choose contents. Entire screen. Let's Yes. Okay, this is perfect. All right. So, let me quickly first show you um the front end that I've built for uh this modular rack pipeline. Like it's Brian's idea, but I thought we can do this a little bit nicer, right? So, in Streamlit,
I created um a basic front end with some tabs. So, we have a run tab, we have a batch tab, we have an index tab, and we have a retrieval tab. You can also run this modular rack pipeline using a configuration file or you know test 10 different variations with 10 different configuration files. But just for you guys I thought it would be kind of nice to
show it like this. So um in the index we see that we have a document loader, we have a chunker and we have a vector store. So in this case I'm using Chromadb. Um it's quite easy. And then on the retrieval side we have a query transformer. It's currently just passing the query on. We have a router. I mentioned might be splitting off to like a search
engine or these kind of things. We have a retriever. Uh in this case we're retrieving from the Chromb uh instance. We have pro prompt builder an answer generator and an evaluator at the end. So let's take a look at run because when I ask it where is Brian we can run the pipeline and this will create the index and then run the pipeline uh the retrieval pipeline.
So this is important uh we want to test also different index version. So in this case it will run both. So when I press run pipeline live demos are always scary but let's hope it works any day now. Yes, there we go. All right. So, it told us Brian is located at B2. Well, I think that's where the printers are, right? So, but we do see some
information on how long it took to create the index and how long the retrieval took. Um, I'm not sure why it took 7 seconds. Normally, it takes like one, but okay. So, maybe let's try something else, right? Because if we take a look at the index, we see that we have a text loader. Maybe let's look at our information. So, here at the top, we have the
bank info. So we have some uh canteen data db. We have a coffee machine uh data with CSV. And we have a printer data .txt. So okay, we know printers isn't going to help us a lot. But now let's try and build a CSV loader. It's going to be quite easy. Uh no fancy coding uh over here. So let's go to document loaders. And then we have
a base document loader, but we also have a text loader. So like I mentioned first we have the base component. The base component gets implemented by for example a base text load or sorry a base document loader and then the actual implementation is the text loader. So this text loader it's quite easy. It has a run method. Um it doesn't take any uh arguments other than a
directory. And uh over here yeah we just kind of you know read a text file and then put it into documents. So let me create a copy of I'm gonna call it CSV loader. And I'm gonna make it very very fancy. So we'll just rename it like this. And instead of reading uh txt files, we're going to read CSV files. I'm saving it. And we're going back
to the streamlet front end. And now we can immediately see that in the index I'm also able to select a CSV loader. So that's great. Uh it's also taking a directory. So, it's grabbing the uh data bank info uh uh uh directory, but let's run it again, right? So, let's ask it where is Brian. Okay, so this time it showed us that Brian was at the coffee
machine. Uh which is kind of nice. Um and we can also see that this time the retrieval uh was a bit faster. So, I guess the internet connection was maybe a little bit spotty. Um or the Azure resources, I don't know. Um so, this is nice, right? But what if you want to have completely new components? Well, you can do so as well. So, let's I've set
some things up in advance. Um, but let's create for example um a SQLite retriever because as we can see we also have uh a canteen data db. So, probably some relevant information is in there. So, let's make a new one. Uh let's call it SQLite retriever. And I'm going to be cheeky and not bore you with any coding, but let me put it over here. So we
have the SQLite retriever over here. It implements the base relational database retriever and we provided a database path in arguments and then we have some code to show like how to execute cur query a format the results and obviously there's a run function. So this run function um is what is actually being called and as if you can see it it's very basic. It just checks where
a user is Brian. Um so it then returns the results. One thing that we now need to do because we're introducing this new component is we also need to add it to our configuration file. So in this config.l file over here we have a retrieval pipeline part and down here we have an indexing pipeline part and let's take a look because over here is our retrieval component
but let's add in another component in this the relational database retriever. So we provide the SQLite retriever as an implementation for our relational database retriever and for the DB path we're providing this canteen data DB. So let me quickly refresh streamllet again. And now over here in the retrieval tab we can see that we now have a relational database retriever. It's kind of nice. One thing I
do need to update is I need to move it to a multiquery. Um so basically what we were looking at before single query will just go to the regular retriever and the index uh for vector or search but with the multiquery it will also route it to our um database retriever. So let's ask it again. Where's Brian? Ah he's in the lunchroom. That's great. Cool. So this
way I hope you can kind of see that you can easily implement different parts of code um and test them out. So for example when you're running for example the index you can also check with the semantic chunker or you could easily return more results all these kind of things. There's also a batch tab um and in this batch tab we actually run multiple question answer pairs.
So I'll briefly show you what that looks like. I have got this question and answer pair about catbank. Catbank is completely fictional but it's a bank for cats. Um and we have some questions like where did sir Pington the founder of catbank originally perfect the secret vault code? Well apparently he worked on it in his private study testing different meow and per combinations to ensure it was
secure. So okay that's nice. If I want to test all these questions in one go, I can actually do so. Uh so let me show you. I do need to change some information. So for the text loader, we're going to go to catbank. Um let's take the semantic chunker. That's kind of nice. And then um yeah, I'll call this catbank as well. And then on the retrieval
side, uh let's see, we got three things like this. Yeah, I guess that should be okay. Well, let's see. Run batch evaluation. I set it up in parallel, so I hope it should work. Ah, okay. So, we ran four questions in answer pairs. Three were correct, one was incorrect. Um, we see the average indexing latency and the retrieval latency. And this time it took a lot more
tokens actually uh to generate the answers. But this is like a nice way to kind of experiment and play around, right? Obviously in this front end, you could put more things like actually seeing what kind of answers it's uh using or it's uh generating and those kind of things. Currently, I'm using Langmith on the back side to kind of trace what I'm doing uh and kind of
see what uh messages are there. Um sadly, my ABMR laptop doesn't allow me to sync with Langmith, so I couldn't show that over here. All right. So, let's briefly look at the code that went into this because you might be wanting to use it yourself. So, uh there's the source folder and in the source source folder there is the main pipeline and this pipeline it's quite an
easy class. It has a configuration path. So, we need the config file or get the config from our streamlet UI. We load it. We load the pipeline components and then at the end you know there's the component factory and this is where the meta programming magic happens. So over here you can see that we're using the import lip uh import module um to kind of get the
components based on the strings that we're providing and then we need to resolve the arguments and uh the resources that are required for that component. Then we have the run methods for indexing, retrieval and to run it combined. And at the bottom we see the actual run pipeline method which is basically for each component you just put in data and you get data out and you click
on on process. So going to the models there are some models and you might want to extend this. Um but in general I think we have a query, we have documents, we have prompts, we have destinations for our routers, uh we have the actual route that we're taking, there's a routing table, a response that's being generated, and then there's the data object. So in this data object,
we're tracking like uh the documents that we loaded, the documents that we chunked, you know, all these kind of things at the intermediate steps. And I'm also kind of tracking the token count. So in the components we have the base component and like I mentioned before there's the process. So it's actually just checking the data, extracting the input data and then running it and updating the data
at the end. Let's go to a actual implementation of component. So the base chunker over here defines okay what does it look like when we're validating input data uh so we're checking whether it's like uh the documents that are loaded that we require for it are there we extract the input so we're extracting documents loaded and at the end we're actually putting our result into documents chunked
so it's kind of a separation like that and then looking at for example a fixed size chunker which implements this base component we only have the chunk size that people need to specify and then a run and in this run function we just kind of you know go over the different documents and split them up based on the character size that we set. All right. Um and
then I guess the final piece is there is a config and I showed you this earlier. This is kind of where you specify your files for the run. Now you can also run this um without the UI. There is this test pipeline and you can kind of either just run based on your original config.jl Juml, but you can also run it with different configuration files. Uh, or
you can actually uh parameterize your uh config file and then just run, you know, 10 in a row and then you can kind of easily compare, okay, what were the chunks? Um, what costs did it have? How many tokens did it need? Uh, how fast was it? And yeah, I guess that's it for the code walk through. Let's move back to the Let's see if this is
still working. Yeah. All right. So, I think this comes down to one thing, uh, a quote that I really like. I choose a lazy person to do a hard job because a lazy person will find an easy way to do it. You have some overhead, but after you got this set up, you can just try out everything and you can tell your boss why you need all
these Azure resources even though they're very expensive. So, where can you find this code? Well, obviously it's on GitHub. Um, don't worry, I'll share a QR code at the end. Um, you can find it over here. And if you have any questions about this motor direct pipeline, where can you find Brian? Well, to be honest, I don't think Brian wants to be found, but you can find
me instead. You can reach out to me and I'll be happy to help you with any questions that you have. Thank you very much.
