About this talk
This talk discusses the integration of AI and vector search, particularly focusing on retrieval augmented generation (RAG) within MariaDB. The speaker explains how vectors, which are numerical representations of data, can be utilized for effective semantic search. He provides insight into the implications of vector databases and demonstrates how to architect a solution using RAG techniques. Challenges in production environments are explored, including the complexities of processing documents across various formats. The session concludes with a live demo showcasing the capabilities of the MariaDB solution for vector search, illustrating how to efficiently retrieve contextual information while minimizing hallucinations in large language models.
Full transcript
We have Alejandro. Is that correctly pronounced? >> Correct. Perfectly pronounced. >> Thank you. Um giving presentation on AI and vector search. So you work at Maria DB as understand. So yes, looking forward to hear your talk and let's give a warm welcome for thank you Rasmus. All right. Where's my clicker? So, hallucinations. How about this one here? It's suggesting to add glue to pizza. At least it
says non-toxic glue. How many rocks should I eat? At least one small rock per day is recommended because rocks contain minerals and vitamins that are important for digestive health. [snorts] All right. And the whole hallucination is hilarious. You can find it online. Uh it's crazy, right? So that's what LLMs do, right? if we don't ground them on actual relevant data like uh here for example with uh
um Google AI studio you can ground them uh the models using Google search right and there are many other techniques and at the core of those techniques is vector search particularly I'm not going to go through all of this but particularly uh rag or retrieval augmented generation so uh it's not surprised that uh interest over time for the terms vector search and retrieval augmented generation is growing
over time especially in the last couple years. So we're going to talk about that and here's the idea. We're going to talk about what vectors are, how we can use them for vector search. I'm going to give a quick introduction to rag and some of the challenges and techniques especially when we go to production. We are going to architect a solution and I'm going to show you
an implementation of that uh architecture uh throughout demo kind of because going to be screenshots. It's going to work out pretty nicely and let's see if u we have time for the the Q&A because I have a lot to show you. So let's start with vectors and a vector is just an arrangement of numbers like this, right? So floating point numbers usually from minus one to one
and they encapsulate semantic meaning about something. Let's say for example the image of a dog, right? could be the text dog or a paragraph that talks about a dog, right? And each of these numbers is called a dimension. And each one is going to encapsulate a little bit of the meaning of the concept that we are uh trying to represent through that vector, right? So we can
have another one maybe a uh cat, right? And you can see that the numbers vary according to what uh uh the meaning what we understand about a dog or a cat in terms of those uh dimensions. Now, I put labels here that are easy for us to understand. That's actually not true. When we use an AI model to generate those vectors, we don't really know what those
dimensions actually mean. We cannot uh rationalize about these numbers individually. We're going to use the whole vector, right? And there could be thousands of of dimensions here, right? So, we need a proper database for this. And for a long time we have been talking about like kind of two or three different kind of uh databases. There are other classifications but we we think about transactional databases right
where we uh store things in rows. So for our daily operations so the database can read the whole uh for example user or order from disk physically. There are analyt analytical uh databases that store data physically in columns. So you can operate on columns really quickly to for example calculate an average or something like that. And then now we have a vector database not now but we
have been starting to hear more about these kind of databases and they have like you store your dimensions somewhere and your payload somewhere else. Right? So each of these u database systems uh might end up with a different SQL dialect. Right? Moreover, the vector database might not even have a uh SQL dialect at all. So if your application needs to use all these services, it will have
to learn uh those dialects plus maybe an API right uh MARB has a bit of a different architecture or approach to databases where we have what we call storage engines and they are pluggable you can install them uh and and and they kind of focus in one area of these kind of workloads. So for transactional you have InnoDB and for analytical you have column store but there
are more. So in fact in a fresh installation of Mariab if you run show engines you see a bunch of engines already installed and you can install some more. So these engines live in a single database that means there's only one SQL layer. So your application only needs one connector one kind of idom of SQL doesn't need to become a polyglot database poly and all the complexities
associated to it. So because each table can uh declare its own uh storage engine then you can do uh cross engine uh SQL queries like in a join with the tables that are that live in different uh storage engines. And for vectors or AI uh workloads well you can just uh store the vectors in any column u sorry in any table in a column of type vector.
So that's pretty uh elegant solution. But let's get back to vectors and let's see how to use them. uh with an example and think about thousands of dimensions are going to be really difficult. So I'm going to use two two dimensions here, right? And yeah, I said before like uh let's that's not real, right? When we're using embedders, we don't know what these things mean. But for
us to understand this uh we need some sort of meaning, right? So I'm going to place some some uh objects here in terms of size and and and speed and alarm clock maybe is over there, right? So it's not big and it's not fast. A cow maybe it's it's much larger, but it has just a little bit more of speed. a bike a bit more speed uh
smaller and the same with the eagle right so then don't take these too seriously but we can uh assign the vectors to to it right like calculate like more or less uh and well those coordinates those are our vectors and they encapsulate the semantic meaning of these objects yeah super simplified here in terms only of size and speed but you could imagine that when we have thousands
of dimensions then the meaning is actually there of the concept whatever it is that calculating the vectors on. So let's see how to store them in a bit similar in other databases. So we need a table. So let's create a table objects. Uh we need two couple at least couple of columns here. One for the name, one for the actual vector and the vectors are called sometimes
vector embeddings or simply embeddings. So I'm going to use that word and the type is vector and has two dimensions. So our table is ready. Now we can insert uh some data into this uh um table. So we start with the alarm clock and then we pass in a JSON array format the coordinates for that alarm clock right and we pass it through a function so that
it gets converted to the proper format that Maria DB expects which is binary and very uh efficient or you can pass the binary that's returned by LLMs as well or by uh embedders rather we did for cow the uh bicycle and the eagle and now we have we have our data uh stored our vectors right so the reason we do that is for doing performing vector search
so so let's get back to these uh uh example and you might have noticed that over here they are going to be the big objects and in this area there are going to be the small objects and likewise the slow objects here and the fast objects over there so uh there seems to be a relationship between the location of these objects in these two dimensional space, but
it's going to be a hyperspace when we use a um an embedder. And the meaning of each uh in this case object so that the uh objects that are closer together, they're more similar than those that are further away from each other. So, we can use this to perform vector search. And here's how. Let's say we want to find the object here in our database that's closest
in meaning to that in meaning because uh it's only size and speed right so the closest thing to that how can we do that well we select the name that's what we are interested in from the objects table we order by a distance the distance between two things so we need to pass two vectors that one there so that represents the insect we have there and the
embedding column which is from the objects uh uh table. So we order by that so that the the the shorter distance is going to be on top right and then we limit to one. We are only interested in one. We don't need all the rows in the table. So we are basically trying to find something small and slow. So why would that return the clock right arm
clock? And that's what happens if you run this query actually in MB. And we can do the same for something that I call big and semifast in which case it's going to return cow. So there's no magic here. The same happens when you have thousands of dimensions. Right. But Right. Cool. But how can we visualize this in an actual architecture and some solution that uses these for
building a uh value for for users. Right? So let's talk about rag or retrieval augmented generation. And you can think of rag as a technique to construct context specific to each query. I took this from this book. It's pretty pretty pretty good book. I highly recommend it. It's not about machine learning that much but more about how to use uh AI for building applications. Anyway, so just
like in the uh transformer architecture when we say attention is all you need because of the famous paper from 2017 I think uh the transformer architecture in rag we can say something like context is all you need and basically we need to inject that context. So in your application you will have some sort of a template right with a question that comes from something from the user
and some fetched data that's related to that question so that when we instantiate that template it could look something like this where's ABC the context should talk about ABC so that the LM makes a good prediction or output. So how can we get these fetch data using information retrieval techniques, algorithms and technologies? One of which is vector search. Sometimes also called uh semantic search by the way.
So in rag there are two phases. So phase one is kind of an an ETL pipeline where the main thing here is that we calculate vectors on our data or the data that we want to use to grant our LLMs on and on the uh second phase we use those vectors to perform uh the vector search. So fetching the context, right? Let me show you this in
more detail. And this is going to be like a product recommendation chatbot of sorts. Super simplistic. You can find the code in GitHub if you're curious about it. Java. Uh so there we have a service or program or something called compute vectors. That's uh that's all we have in terms of code there. And we have a database with one table products. And then uh what we can
do is take for example this row here this product take title description and whatever we want to take from there concatenate those into one single string send that to an embedder right for example right by open I or any other provider and that emitter is going to return a vector remember the vector encapsulates the semantic meaning of whatever we sent to it that that's going to be
this product we can take that vector and store it we saw how a snippet of code how to do it to have an idea of of how that would look like at the code level. We iterate over all the products and now all the products have a uh semantic representation that we can use in phase two. So let's let's go uh to that one retrieval and generation.
And here I have a chat demo service or program or whatever it is that the user is going to invoke. So let's say uh this user is going to type I want running shoes. So suggest something. We can take that string send it to them better. It has to be the same better by the way. So that returns a vector. We are not going to store that
vector but we can use it in a SQL query to perform vector search. So give me for example the three that are closest to that vector which at the same time means from what the products I have in the database what's the closest to I want running shoes. Right? So this is the uh semantic search that we're performing here. And again we saw a bit of a
snippet of of code on how to do this. Uh so we can take those three products and build our uh prompt right so I want running shoes available products and we paste that information there so that the uh um LLM this time is the LLM now right is not has less chances of hallucinating and also gets a uh possibility to use our data. Okay. So there are
many challenges and techniques um around rag what's the time right now we have time that's very good to hear we have okay so let's see some of the challenges and techniques when building these kind of architectures and for that I want to bring doom guy here so it's gonna show how difficult or easy this is So at this moment the guy's just smiling laughing because this was
the hello world of rack overly simplistic what a dev rail like me does right hello world applications so it's very very easy to understand to code this in fact if you use something like uh uh java and springi that's the actual implementation of that service this just so that you see how how much code you need to to type or let your llm type for you. Well,
anyway, so but that that smile is not going to last too long because there are some complexities here particularly well we use one table. What if we have a bunch of now documents in different formats? Not only uh the SQL um query, super easy query with one table, but very complex SQL queries plus PDF files, XML, HTML, JSON, uh you name it. It's going to be a
bit more complicated. Let's take the case of a PDF file. And we cannot pretend to send a 300page PDF file to an embedder, get one single vector and pretend to use the uh all the individual ideas that are laid down in this document. Right? That's not going to work. Uh not if that's our uh use case. So we need to divide the text somehow. So we calculate
one vector per each uh chunk is called here. Right? So chunking um and there are many strategies. So for example, we just decide to chunk in in a fixed length strategy which is just take this amount of characters then this and so forth until you complete the text. Suppose this uh PDF file starts with I left my wife a note and the first chunk is I left
my wife. So we have lost the meaning of the document and not only that but we are conveying false information hopefully right. So that doesn't work well with this PDF at least. We need some other strategy. And now this guy gets the first hits because we need to start thinking about it or implementing or calling something configuring something in our uh um AI framework overlapping that would
potentially solve and this works for some kind of documents. We can maybe use a recursive strate strategy where we uh take a big like let's say a page it's a bit there's more nuance to to that but let's imagine like you take the whole page the font is huge so it's just a few words in this document that works that's a vector maybe not maybe it's actually
too long so let's do it recursively until we find the correct size so let's take a paragraph it's still too long okay let's take a sentence yeah that works let's send that to them better we get one vector and one repeat with the rest of the document or we can let an LLM decide. Uh so the previous approaches are super fast, very very fast computationally. Uh this
element will introduce some some lag to the system, right? So some latency bugs very precise. So some complexities there. Uh when uh we are handling these kind of different formats, there are different libraries in each programming language obviously. uh but also when we're ingesting these documents, we might want to do it from HTTP or from a directory in the file system. Maybe we want to have different
use uh users in our system. So that is each user is uh for a particular application or for a particular use case and we might want to have several these directories. We might want to ingest from S3 and look how these guys just crazy right. So when we are computing here we also need to think about batch processing to send uh like batch processing uh transformations particularly
here in terms of uh adding metadata to the chunks or the documents and storing storing those back into our database but also lay out our extraction because PDF could be very messy and if you I bet you have tried this you copy paste some PDF file and then you paste it there it's a it's a mess right so we need to deal with that um when we
we're invoking an external system, we obviously are going to face uh challenges in terms of orchestration particularly here but there are many more but particularly here because we might want to uh handle errors and maybe automatic retries this kind of thing and security uh challenges because there could be a data movement outside our environment. All right. So when we are storing the documents back into our database
once they have been ingested and we copy copy the test and paste it into a column for example in uh there are going to be uh some the need for a document management uh set of uh functionalities CRUD right create read update and delete forward documents and then later for chunks as well. Um so there are many um challenges there. That was only phase one. Let's go
to phase two real quick. Uh the first thing here is that you might have to introduce maybe an additional vector database and uh or extension, right? And this is going to well add a whole lot of complexity to your to your system. In fact, I recently found that uh it was surprising to me see to see this uh that uh pg vector which is extension for posress
to allow to to do vector search uh has a buzz factor risk or the buzz factor of of that project is one person. If you want to know more about what the f buzz factor is and how to use it, uh I have this article just Google the the term buzz factor but there I have a tool or it's I think it was by jet jet brains
uh if I remember correctly where you can calculate the buzz factor of any project on GitHub. So I did that for these databases and I found that MongoDB, Mariab and radius are pretty strong actually. So they don't have much risk there. the others. Uh I'm not saying these projects are in danger or anything. It's just one one one of the risks, right? But I'm just showing these
as an example of the kind of things that you will have to take into consideration when you're adding a new database or extension. So in that case, this guy's almost almost dead. Almost right. One more hit and I don't have any more sprites uh to to show there. Uh but we're using MAR DB, so we don't have that problem because it supports a vector natively. In fact
just starts smiling again. So uh however there are challenges again in terms of uh orchestration and security that we saw before uh here when we are creating the SQL query that could be a custom SQL query that could grow in complexity uh especially because well we want to do maybe vector search right for the retrieval it's called uh but vector search is not the best strategy sometimes
sometimes we want to do keyword search. So vector search with a vector index or keyword search with a full text index, right? And uh when when is that the better way particularly when uh you have codes for example and you want the possibility to to search by an ID and this kind of stuff right so um in that in those cases keyword is preferable it's going to
work better or we can do hybrid so we can get best of both worlds hybrid search is where you run both of them kind of like in parallel and then you combine them somehow with a special kind of mathematical function and then you get the best of both worlds here. Again uh here again orchestration um in this case this is the LLM. So uh particularly we might
want to uh have some sort of streaming logic here to show each uh token as it's being generated by the LM and the security risks that we already uh show. So let's see how to architect a that tackles all all these problems or uses all these uh techniques that I show you here and there are many approaches to rag and you find in the literature that some
authors talk about naive rag which is a bit closer to what I showed in the first example. It's not it's not that it's bad it's just sometimes that just works right. Sometimes I know precisely where to get my data from. I know I just need this query. I have it ready even or I or all the documents are kind of the same. I know I'm going to
use only this AI model is not going to change. We are not going to change provider or we host it in our environment. All these kind of things well defined um requirements don't change too much. So what they call naive uh rack I don't like the term too much but uh uh it's more it's like the the case specific implementation of rag then that works but when
you go beyond that you find that uh there is something called modular rag and that's more adaptable to changes in uh in your data and also in the um in regard to the techniques and the possib ability to invoke certain kind of algorithms at what point for one use case and another for another uh use case. We're going to see that later. So this one has four
modules all right actually five but uh I think it's easier to think about it like four modules document ingestion in which uh well we should be able to do all that right so incremental ingestion so versioning uh through SQL file system s3 and many other things multiiformat obviously layout text extraction for that you can use something called dockling if you want to do it locally I think
that's available for Java and Python at least and uh it's called Doggling and then uh something more advanced that scales with a click and this kind of thing uh llama parts but then you'll have to trust those guys because you will be sending their your information to their servers uh metadata extraction documentation versioning chunking the chunking modules should accept or should be possible to to uh to
configure to use multiple strategies right So fixed size, we saw that at the beginning. Recursive, right? Uh semantic, so the LLM doing the work. Uh sentence, we can detect the sentence by certain uh dividers or token for uh cost controls and some scenarios uh that could be uh desirable. Chunk size and overlap should be configurable and it should store metadata so we can trace where this comes
from. Right now on the uh retrieval module uh which is the one that actually uh well does is it converts the input to the vector embedding where I showed the I1 running shoes part in the example. It should support for retrieval keyword vector and hybrid at least to cover all these uh use cases. Uh and something I didn't talk about was the uh crosscoder. Uh you can
add a cross encoder reanking step to the rack pipeline. That is a way reorganize reorder the results so the you get the most relevant on top and then you can discard some of those. Remember uh the uh quality of the retrieval is key here so that the LM doesn't hallucinate. We can have a retrieval and if it's not giving us what we actually need for that question,
it's going to hallucinate. So that's why we have so many little details here. It's not uh because it sounds fancy or anything. It's because it really improves the quality of the retrieval as much as possible. Anything we can do uh at that point will be valuable. All right. And so for the this part the cross encoded ranking there is a flash rank that you can use it
for your local environments or cohhere. Uh again you will have to trust uh these guys um information again to their service uh some meta data based filtering as well. Uh so anything that kind of reduces the amount of chunks that we are going to consider could help right so that's why we added that there and the generation module well the one that integrates with the LLM now
and it should probably support several uh providers uh because you never know what happens with the models right what's good today it's not going to be that good tomorrow or you find another one you want to be quicker than than the uh um competitor so you might want to switch so there are these scenarios and also because some models just work better with certain kind of information
than than others. So the configuration of a of an AI model or a provider the possibility to do that per use case could be uh interesting and in case you want to to host your own models. Now it should handle the prompt construction with relevant context citations so that um we can show the user where the information uh comes from if we want and uh maybe uh
handle the response with uh uh streaming for real time uh output. There's a fifth module that I was talking about before. Um this one is more for u things such as uh error handling, retry logic. uh maybe uh we want to call these two steps in one request and those two in another request kind of makes sense this kind of thing and uh monitoring this kind of
stuff. So everything should be secure. So there should be some kind of authentication authorization mechanism and um and the way you pack these is uh in a service right. So so we can deploy this easily is a process uh that connects to a vector database and an embedder and an LLM. So those points those should be configurable. Uh then you expose everything through a rest API a
bunch of endpoints here that application developers and machine learning engineers use for building their own applications and then you have a a powerful reusable uh rack architecture for either many applications or big applications with many use cases. That's what we did at Mariab. So we created something called Mariabi rag um that implements absolutely everything I showed you before. uh all these architecture all these things little things
I show you there it's available and some additional things as well and um we add also the MCP server so that AI agents can use especially the retrieval part of uh of this architecture right um so Midba rack is all these this marketing stuff uh it's a microser basically right so it's a rack solution that's packed into a single server. There are some other services there. So
this scales to uh hundreds uh thousands and thousands of documents and millions and millions of chunks. So I that's why I call the micros service and we can use all the techniques and everything we have learned to manage uh deploy scale observe microservices. It's part of the Mariab enterprise platform. Uh I'm not going to go through all that but here my favorite maybe it's max scale. It's
a reverse database proxy. It does so many things. So cool. I love it. It has a a command line interface. Uh it has a a nice UI that you can use in the browser. It's pretty cool. Kubernetes operator column store for analytics. Uh XA for real time analytics. Uh we even have an in-memory cache powered by grid gain which is the like the enterprise version of the
famous Apache Ignite. You name it. So um this is what you get at the enterprise level uh with Maria DB. Of course there's support around all these uh so let's go to the kind of demo and I happen to be the author of these three books about vaddin which is a um Java framework. It's pretty cool. So you can build web applications uh without having to write
any HTML or just Java. and I'm writing another one uh which I called Midi for developers and I'm typing this by hand so it's taking taking a whole lot of time now if you are interested in in what I'm showing here or databases or specifically Mariab please go to uh miabfor developers.com scroll down that page if you like what you see then uh there's a form that
you can send me your email and then I'll send you a notification for when um when the book is is ready but that's work in progress So I'm going to use this one practical var because there I place this cute dog again that we saw in the at the beginning right and I went to uh chip was it yeah what breed of dog is used in one
of the examples of the book practical vine and so it gave me this uh lazy dog pang phrase right pangram is that that phrase that has all the letters in the alphabet and yeah I think I used that in the books some examples somewhere and maybe that's public information. It's a sample and it's not what I'm looking for. Gemini just hallucinated. I have no idea where this
comes from. Never mentioned a dog like that in the book. Uh so I figured hey let's build something with Mari Bi rack to find more about this dog. So I have two two machines one and this is another one and in that one I have Mari DB enterprise running it would work with community as well here I have the configuration file don't pay too close attention to
the syntax here just so you get an idea of what you would have to implement if you want to implement this kind of uh solution or if you want to use uh Marbi right later so let me uh maximize that panel there for you and first I need to create some sort of admin passwords I'm admin and pass right and then uh for the database configuration well
I need to configure the the endpoint basically where the database is and particular I have to create a uh database there database and schema they are synonyms in in Mariab so I can use create database or create schema works the same so I'm creating that midb AI rack uh obviously we'll have to configure the API keys for our AI models um and the config configuration of these
models, right? Particularly again here the number of uh dimensions to use and the name of the models. And finally, we can uh configure some of the tables that this solution creates automatically. So, I'm using documents and chunks. It's very easy to understand for this demo. Use something better in your production environments. So, there um this is the logs for the MCP server. Let's ignore that. I'm not
going to use it in this demo. And in this panel, what I have is the actual logs of the uh Mariab AI rag service, right? The rest service. And so uh there's an IP address there in port 8,000. So if I go to the browser, I see the swagger UI uh provided automatically there for us, which is a way to navigate through all the endpoints, see the
schemas, and even invoke the endpoints easily. Uh so first I authorize, right? So I use the the user that I defined in the configuration file and now swagger error it's going to uh inject a header with token that it got from here. your applications would use a where is it a token right now let me collapse those sections and and then we see that kind of resembles
a bit a rack pipeline right but it adds health system authentication user management zomb orchestration so let's go to documents and we are going to ingest a document so I click try out that button there to enable the form I choose the file here that's the book the PDF file that I'm interested ed in click execute and you can see this kind of translates to a curl
command right it's just it's invoking a rest endpoint that's all this is doing through a specialized UI and I immediately get this uh one document has been cued for ingestion so if we go to the back end and we see that something started to happen there in the logs and if we go to the database and show the databases we see Maria DBI rack there we select
that database and then we show the tables we see our chunks and documents ments and some other tables there for the solution which is what I configure here and I can check the status of the ingestion by running that query select ID status from documents and by the time I run this query it was already completed so we can get back to our front end and invoke
another endpoint now we're going to do the chunking and for this I'm going to change some of the parameters here actually the defaults worked better but I want to show that you can do that per each use case can configure these little things right We're going to use the chunking method is recursive, right? The one that tries to make it uh smaller and smaller kind of chunk
size overlap. The restful is for the uh semantic source that you can also use here. I click execute and again I immediately get all documents you own have been cued for chunking. I say again immediately because this is asynchronous. So there are some endpoints that you can use to check the progress of these things. but instead because we like the back end. So I go to the
back end and I'm going to check there. So where I I see that something started to happen in the logs that that looks good and I can uh count the number of chunks. So as I run this query multiple times, you see that this number starts to increase until the process is completed. We have around 400 chunks. So we can get back to our endpoints. And now
we're going to do the generation part. So we have ingested the document. We have chunk the document. We have calculated the vectors for each chunk. Now we can ask questions here. This part is going to do the retrieval and the generation. Right? So what we have to do there is just type the query what bread of dog is used and then I'm saying hey you know what
use document ID one. I could have just left that empty and it's going to use all the chunks and documents in the or documents in the in the database retrieval method. I'm going to use hybrid search here and top K is the number of chunks to retrieve. Right? There are other things that you can configure like temperature and and some other things. Okay? So click execute and
after one or two seconds we get this uh the breed of dog used in one of the examples of the book practical vin is an English bulldog. The dog's name is Daku. And if I go to the to the document I find this his name is Draco. funny, friendly, and sometimes hectic English bulldog. So I got it right. This Gemini, I was using Gemini. So this time
it didn't hallucinate because we grounded with relevant on on relevant data. Now I did that with like three steps. I could have done this in one step. So if I undo everything, I can continue to to ask questions right through that endpoint or other endpoints without having to ingest and chunk uh and then we send that to the LLM and it's going to send only the chunks.
So it's not going to send the whole document which is equivalent if you attach a PDF file like in chat GPT let's say but that's going to be very expensive if there are thousands of users invoking this right but I could have done it in one step so that's the full pipeline I pick the bug there the the the file the PDF file type the query configure
hybrid search the number of of chunks to retrieve and I arrive to a correct answer that yeah the wording is different because it's an LLM right so each time it runs it's going to be a different kind of wording unless we configure temperature to zero which I didn't uh but it's correct right English bulldog dra so let me pause there for a to catch my breath also
a little bit so what we just did there was we invoked a full rack pipeline with I think I have it here authentic ation and authorization, automatic error and retry logic, asynchronous document ingestion, recursive chunking with overlap, metadata versioning, hybrid search retrieval. There are other things that happen there. All of these in one single call. I don't know about you, but I find it pretty cool. We
call it rag in a box. So, Maria AI rag is currently in beta. Um, there's a trial um URL that you can use [clears throat] if you want to get inspiration on how to build your own solutions or even better to use mariab which is again feature complete documentation. You find mib.com are my coordinates. Thank you. >> [applause] >> Thanks a lot for that very detailed walkthrough
of rack pipeline. We have a few questions. I'll read them up. So, different vendors, OpenAI, Meta, etc. provides different embedding models. How to know which model to use based on the use case? >> Oh, this could be a whole talk about it. uh not only because models change from one day to another literally uh but if you look at u hogging phase there are I don't know
maybe it's like 80,000 models at least for embedders it's like 80 I don't know if I'm am I making that number up it cannot be no maybe it's more like three 30,000 it's still like it's crazy right and when it comes to LLMs it's even even more uh and each one is trained on uh or retrained on different kind of data sets. So it's it's tricky. Uh
there are methods to try to evaluate um these things. Some there are data sets that you can reuse and then you can just see how accurate it was for um for the retrieval part. What I can tell you is that you can measure for example recall right which is different than precision recalls. How many chunks were retrieved that are actually the ones that are closest to the
question that we were interested in? That's called recall. It's not precision because I can get all the chunks. That's precise. I got all the chunks. The answer is somewhere there. But that's not useful. So record is more useful. Good, good question. Let's take the next one. How would you compare native vector databases versus expanded vector search databases like Maria DB, Postgress, PG vector, MongoDB, etc. Well, thanks
for adding PG vector because I mentioned before PG vector bus factor is one. So, uh that's one way of comparing at least with with posgress. Uh but hey, I'm I'm really not trying to say here that that the the PG vector project is in danger. I'm just saying that's there it's higher risk on that metric than a pro a product like uh Maria Dv. Now uh there
are vector databases that offer you uh they are specialized and there are use cases for those sometimes uh they talk about how they can reduce latency or how they can uh uh scale better. Uh I cannot I mean I'm going to be biased so I say I know that whatever I say because do I have the logo here? I don't remember or here. Yeah. [laughter] Well, it's
not going to be trustworthy. So I'm not going to answer that specifically. What I can tell you is that we have measured this u with uh external benchmarks and we have seen benchmarks by other people outside the company that show that we sometimes beat uh vector databases. Uh so at least in terms of recall and speed it depends right so um I know this sound a bit
more like a trust me bro kind of benchmark so go check the benchmarks yourself and then you'll see the difference but take take into consideration that the the >> thanks what techniques or best practices can be used to differentiate similar parts using vectors such as industrial metal I would say this more about the models really. So more than the uh I would I wouldn't be able to
tell you about uh like industry verticals but uh I would say the models. So I've seen models that are trained for something super super specific and they beat the the state-of-the-art models by the big uh players if it's on that topic, right? So I don't know if that would be one of those. Um but uh that's more about the actual AI models. Come to me later if
you want to maybe uh ask more questions around it or clarify what was the the question. But uh but yeah, let's continue with the next one. >> Let's take the last one. We have one minute left. Is there an easy way to give the I don't know answer instead of an hallucination? I think um yeah uh prompt engineering is is a is a thing that comes to
mind immediately right um that's one thing I believe there are there were I saw a model somewhere that had a flag that you can configure I don't know what they do internally or what they did but uh it it it gives you a whole lot of sorry I don't know kind of answer uh which it has the side of the coin is that it's that's really annoying
for the users. So you have to balance that. Um that again is more about the u uh the the model and prompt engineering techniques I believe. >> Thank you. That was last question. Thank you. Let's give a big hand. [applause]
