Daniel Phiri: Bring Your Own Data! Build Better AI Applications

In this talk, Daniel Peri explores the concept of retrieving your own data to enhance the effectiveness of AI applications. He discusses the rise of large language models (LLMs) and the challenges they present, particularly focusing on the phenomenon of hallucinations, where LLMs generate misinformation. By implementing retrieval augmented generation (RAG) workflows, developers can create more accurate and contextual responses by integrating data from vector databases into the LLM's queries. Daniel emphasizes the importance of having control over data, especially in sensitive environments, and presents a demo of a tool called Verba that showcases how to build applications using this RAG method. The talk highlights the significance of vector databases in managing large volumes of data and improving the retrieval process for more secure and efficient AI applications.

[Music] hello I'm uh supposed to have my slides up I'm Daniel and I will be talking to you about bringing your own data um and building better AI applications just need to get those up on the screen give me a minute okay I have informed someone and they'll help me out with that in the meantime if you have any sort of Prior questions happy to tackle those

i' love to know if anyone has experienced building with uh Vector databases or implementing any sort of rag workflows okay my slides are up we're great okay so like I mentioned I will be I'll be talking about building better AI applications with the context of bringing your own data I'm Daniel Peri I do develop a experience at a company called we8 you could see the logo at

the top of the slides there and you can find me anywhere online as Mal games right so getting into it again I think I did ask have you ever used a vector database definitely drop something in the comments if you haven't I definitely used them quite a bit and have you used chat GPT or any chat application I think there's a really interesting one from Mr alod

chat at this point you probably have I wouldn't be surprised um and last question have you possibly heard of rag if these are new Concepts to you or have no idea what I'm even asking hopefully at the end of this talk you have a better understanding of databases um tools like chat gbt and rag which stands for retrieval augmented Generation by the way right so like I

mentioned I'll be talking about how you can bring your own data and build better AI applications and to sort of drive this point I want to go back to where we started with uh artificial intelligence where what we've been doing and where where we where we're possibly going right and so in the beginning we had Neolithic man right they would do cave paintings um and this is

how they communicated this is how they would show and document uh and this is where people went and saw how people lived in that specific time period right this is how we know how Neolithic man lived we look at these cave paintings if we think back to the Middle Ages This was um the paintings that people put the paintings artists wrote or uh created described that time

really really well and if we go back to today it's mostly Twitter modern society is like we could sort of gauge the temperature of modern society based off um what people share on that specific social media platform and if you explore this and use Twitter to explore how llms have have come up and where where we've gone and where we're going to we start with the rise

of large language models right Transformers four four years ago were a couple years ago more than four years ago were invented and about four years ago CH GPT the first version at least the first version that reached the mainstream came out uh and then gpt3 really sort of came to the mainstream and this changed a lot of the way we used applications and interacted with software chat

GPT was one of the fastest growing software products uh if not the fastest and then you saw people starting to build on chat GPT it it was outside of this chat that we had open a opened up this API for developers to build from it and at the bottom of this slide you can start to see companies doing interesting things like ask viable using gpt3 API to

build Insight from customer feedback right I'll ear I started to use GPT for uh customer interactions also uh and it was mostly very conversational and then we started to see this age of hallucinations we were really excited about this piece of software we're excited about where we could take it we were excited about what we could do but then we were met with the concept of this

experience of hallucinations and so if you know anything about these large language models they're trained on a specific data set and so all of their output is at least supposed to match the data set that they were trained on but these models predict the most likely next uh next token per se and so a lot of the times those predictions come from outside of the data set

and that's a huge problem because they end up saying things that aren't true or things that do not even EX exist and these hallucinations can be quite funny as you see here in um in this in this screenshot and at best they're something to laugh about and at worst it's a really big problem for your company because you start to you have less of a sense of

control over what your users get as they interact with your company and the experience you build around your company particularly for people building them around these customer interactions right it could be a huge public relations problem and this really funny example um uh that you see in the screenshot is one llm basically said you can melt eggs because of a of an article that it found someone

joked about the fact that you can melt eggs and so when everyone started to build these products around llms when people asked can you melt eggs which by the way you cannot these llms just sort of kept giving this response that yes you can and then this hallucination from chpt as you can see highlighted in the screenshot became an actual answer that we have on Google and

this is just to drive the point that hallucinations although funny can be a huge problem specifically for use cases that are extremely extremely important for um our use our applications and our users and this this this inter ction and what I just explained highlights a couple problems that we we see or we have or we've started to notice in using large language models one you have out

ofate data a lot of these models work with data that had a uh a stop period right it was a specific period of time that you trained this data this llm with data from and then you release this model so we keep having these updates coming with these models another issue is we can't really refer or reference these sources or reference the um the origin of these

predictions from llms right it's it's somewhat of a black box when we think about it we have really no idea of knowing this way this prediction came from this specific place because this is just what the llm outputs and then of course it goes without saying that there's a huge security and privacy issue that comes with having specifically customer data um interact with a tool that is

behind another company's um uh data for example um we have a lot of these model providers that we shouldn't really just trust blindly and security-wise a lot of companies a lot of Industries have laws and compliance regulations around what you can do with your customer data with your user data and so these are a couple problems uh fourth problem because I can't count clearly um is the

spelling there but let's let's go let's go for it and just to highlight this interaction with llms I'd like to think about them as exams right so I have I think we've we've probably been in this situation um if you've been in sort of like a schooling or School environment where you have an examination and and you do not know the answers right and so you start

to make things up because you know vaguely around the topic right and you have to write an answer because this is an exam that you have to write and you probably want to pass and this is the same thing llms do they do not know an answer and so they make something up and a solution to this giving sort of going along this examination and analogy is

to make this interaction with large language models an open book exam what that does is it solves the three problems that I highlighted it gives large language models the ability to Source this data and reference where they got that information that they're giving us right so if you ask a large language model can I melt eggs having this open book exam like interaction with that large language

model makes it much easier for you to Source the data that helped it give you that answer right and in this case You' probably get that article and then you know this is probably something someone was joking about and the second thing is it helps llm have context for the interaction that we're having right if you have an open book exam and you asked a question about

PreFlight for example or the physics of um the physics of flight if you have a physics book you have context for that answer you're going to give in the examination knowing that this is a physics exam and I should give a physics answer rather than opening a 18th century book about mysticism that talks about flights in humans right that's not the type of answer that you want

to give in an exam where you're asked to Explain flight right it gives you context and then the last thing is you have way more control over the data that you can give it like I mentioned a lot of organizations have compliance regulations around the type of um interactions that their user or customer data can have with external systems and so ideally you do not want to

be handing over this data um that is supposed to be secure and abstracted away from third parties to third party applications and so having this open book exam and you'll see later helps you have this sort of interaction and this is where I introduce something called rag rag is an abbreviation that stands for retriever generation and as it sounds you retrieve information you augment it and then

you generate and so this is a concept that a lot of people are using to sort of combat these bad actions or these hallucinations from large language models and so in a lot of our interactions we start with this query response type of interaction where on the left here you have the sort of interface that the user is interacting with and on the right here you sort

of have this large language model and ideally what we do is we' put in a query the large language model would surface that query and then we get a response now how what rag does is sort of builds this workflow in between us getting that query and getting that response to sort of combat these hallucinations and so we start query um of course we always embed these

queries because large language models especially if you will do the retrieval do not understand uh text and so we create Vector embeddings for these queries we then retrieve give everything relevant to this query and then whatever we get back from this retrieval we add this as context to then augment our prompt so our original prompt was the query that we put in and then we fetch information

relevant to this query we get context and then we pass this context as well as the query to the large language model so in this case back to the um if you want to give a more contextual answer for example I work at wv8 and if I want to know what the latest way to initialize a client is with uh with we8 in typescript instead of passing

this question to my large language model knowing that we have a new version of this client it will definitely not know that answer I store all my information all my documentation on the new wv8 client in my database and so when I say and make this query what is the latest way to initialize the we8 type client I then fetch all the information relevant to this new

we type client and then pass whatever results I get back and be like information what is the way to connect to a we8 client and then that augmented query that is the query plus the context we get from our retrieval is then passed to the llm and then the llm creates a more contextual answer and a more accurate answer ideally a lot of a big analogy people

give is retrial augmented generation is like giving a large blinders right um and so just to go through these steps as I mentioned the retrieve the retrieval stage is where you have this user query fetch relevant information from a vector database ideally you have all this information stored in a vector database and if privacy for example is an issue that you um have to take care of

you could use an open- Source Vector database so you have this on your servers you don't have to have this deployed on someone else's cloud and so you retrieve this information that is relevant to your query and post retrieval you augment this original query that would ideally before be passed to the llm directly but now we get the original query and the context that we retrieved from

our Vector database and we pass this to our llm to actually generate data based all of this and the idea here is we get a more accurate generation or a more contextual generation having that augmented query with relevant information and that's just sort of to uh remedy this uh I will demo something called verba which is a an application built by the very talented team at we8

and it's an example of a rag application and so I will open this up excuse me and I'll just zoom in a bit and so the idea here is you have this this chat that you can interact with and we we have this we have indexed all the weate resources and documentation and so I could type things here um so what is a um what is a

planet that's also a good question to ask oh lovely right um so also I have a um I have a document uploaded here and so this document is a PDF of a camera that I bought and well we'll just get rid of this actually before um it says offline for some reason yeah we'll just reset okay I'll just check my terminal really quickly and make sure that

I have running okay so I have that running and I can actually go search status just reset okay so we have everything reset now we can do a search um we say hello Your verba has no data yet and so what we could do now is add some documents um we could chunk a bunch of different things and so I will grab a PDF and so let's

import a PDF file I'm sure I have a couple PDFs here um looking for one with let's get a receipt that I might have okay um we have a French exercise here and this is supposed to have said I click did not appease the demo codes today because this is not working as it should but we'll figure it out as we go okay there we go um

so we have that imported and it actually chunks it now so imported one document on one chunk and let's go to our hi what is this document and so the the document I uploaded was um the the PDF and weirdly enough it is getting the wrong chunk so I'll just reset this again and let's try this one more time sorry for the for the issue here so

I'll go back down and grab that PDF um so this is something that I have running locally and if if this doesn't work I will use P I will use the um the deployed version which I know will interesting okay so back to search and if we go and look at our documents we have this document right that's perfect um hello very weird right let's just go

to verbal we. um and we'll refresh this and so the idea is you pass context into this application and then based off that context um you have um you have a response and so like I mentioned we have the context here being a blog and all the information that is available on we. and so I'll ask um what can you tell me about weate right and so

because we do have a blog post that came out and we released on our blog about the new BET of the typescript client this is the information that we get uh and so this is the answer that we have and we get this very uh concise curated information uh on how we can actually do things um and I could ask what changed in this client and so

then based off that information and here you can see the context it actually used to give us this answer and so we can trace that information and so ask a specific does the the client work on um that's T Cent does the TP client work on client side JavaScript apps um so this is something very specific but also honestly only mentioned once in this blog post and

we get the answer the client only supports nodejs based development um and this is incredible and here we have the context that we got and in this case the context is the client library on the wv8 page um do you have demo apps and is the documentation updated for the new client I'll leave these Tyles in just so you can see that this is a very humanlike

experience as we interact um and so this is the idea of getting that context and very importantly having every single element that the llm used to generate this answer and then populating this field um and so if the demo that I wanted to show worked we would be able to upload um and you can do this when you have your own version of uh verba running and

here you can see all the documents that we we're using to to power um to power this and so this is the the idea of retrieval augmented uh generation and and how it works and let's move on a little right so we we've seen what retrieval augmented generation is and the value that it can have for organizations in giving organizations more security um more context and more

control over the data that the llms uh access or the responses that the LM gives by controlling the data that your LMS have access to um and then this also solves the problem of hallucinations and it just makes you way more secure in shipping these AI native applications uh and a big part of this demo like I mentioned was was we8 we8 is an open source affected

database that you could deploy to your cloud or use the cloud offering that we have available and you might be asking right what is the relevance here of a vector database in this rag implementation in sort of combating this idea of hallucinations in large language models the idea here is the most or not the most rather it's I don't think it's the most important but a really

important part of rad is the retrieval portion because based off what you retrieve your llm then generates responses for and so you want to sort of streamline and make your retrieval as efficient or as effective as possible because the better your retrieval the results and from building these AI native applications and supporting AI native developers we found that similarity search sem IC search is really best suited

for this retrieval experience and what is that sort of just contrast what it is I'll introduced the idea of traditional search um which was mostly a CED match and so in this example I ask why do airplanes fly my response would be you should fly with a expensive Airlines which has uh no relevance to the question that I actually asked but you could see that we have

a keyword match in this and so uh to sort of flip the coin here similarity search grabs the context of your Search terms and your queries and gives you answers relevant to the answer to the to the search term and so it has this context this understanding of that query and then gives you the most relevant responses and this works with large language um embedding or uh

machine learning models and so the idea here is you have a machine learning model that is built to understand uh data in a very specific way and you have the subset of machine learning models that people use to create abstractions of data called models and so you have your information on the right here and you pass that information your data your object your blog posts your images

you put those into a machine learning model and you create vector embeddings and these Vector embeddings are an abstraction of your data in a format that the machine learning models and other uh machine learning models can understand and so this array of numbers that you see are um called vector embeddings and so to sort of connect this to what I talked about before similarity search we grab

all our data um be it this blog post or this result on flying with expensive Airlines we pass that into our machine learning model we get our Vector embeddings we do the same thing with our actual answer and all of our other data um actually and pass that through our machine learning model and get our vector embeddings and then what we do next is plot this or

put this data in a vector space and this Vector space is um this collection of all these vectors embeddings um and because of the because of the characteristics of vector embeddings being an array of numbers we can have this multi-dimensional space that stores these items and based off the dimensions in these vectors they tend to appear in different parts of this multi-dimensional space here we have a

three-dimensional space just to sort of illustrate the idea of the placement of these objects in this Vector space where we can see similar items are grouped uh or tend to be closer to each other in this multi-dimensional space we have an apple and we have a banana close to each other uh we have these tech companies Guru and Apple close to each other um and animals here

close to each other sort of bunched up having similar um Vector embeddings which means their abstractions or meaning or the way the llm understands them um is sort of related and that's the way uh you would see this and so we have our data we create the vector embeddings we pass these embeddings into our uh or we have these embeddings in our Vector space and then we

can actually make queries and so for example behind the scenes of a similarity search when we say why do airplanes fly that query that text query that we have gets passed into our machine learning model and then we create Vector Ed embeddings for that search term we then get these Vector embeddings that represent the search term that we created why do airplanes fly and have that in

our Vector space and then as I mentioned once you plot these Vector embeddings into this multi-dimensional space related items tend to be close to each other because their abstraction by the model um is very similar and so we use this to Advantage and look around where we have our search term placed in our Vector space and then return the most relevant items and so we calculate the

distance to the next objects around our uh around our query in this Vector space and then return those in our case the most relevant happen to be this NASA article because based of um machine learning algorithms and different algorithms particularly um approximate nearest neighbor search which is what we use to look around this item and then send back the most relevant in our case the NASA article

was the most relevant after we ran that search and that's what's returned and so in a nutshell this is how similarity search works and I think I alluded to it at the beginning this is something uh that is built from the ground up in we8 um to run and work like this we8 was built specifically to handle and store Vector embeddings um of multiple hundreds to even

thousands of in an effective way to sort of power this retrieval um and this retrieval happens to have a huge relevance in generation and so now you have you have an idea where do Vector databases actually fit U like I mentioned scale tends to be a big thing particularly with storing Vector edings we abstract a lot of this information um in documents uh or uh data blog

posts articles uh images and once we have these Vector embeddings with thousands of Dimensions it becomes really hard to um to retrieve these effectively in a lot of traditional data stores it really creates a big Challenge and makes computation very expensive and so Vector databases are built from the ground up to run specifically uh operations like these from Millions to even billions and the idea of one

create these embeddings um running these uh searches approximate nearest neighbor searches looking for the relevant items around can be very computationally expensive uh and so we want to be able to manage this and uh at the at a very low level have efficiencies come in and things built to make this interaction and this search more efficient and this is exactly why Vector databases were built um and

specifically with we8 um you can do all of these operations run near Vector searches near text searches where your inputs vary and you still have these really quick interactions um and responses uh and tie that back to retrieval augmented generation I'll show you a demo of uh a JavaScript application that sort of has augmented uh generation capability and it's uh what I like to think is in

its most basic form at least and so we'll go into PS code but before we do that we have this semantic Search application um and so I currently have the top 500 songs of the century and when I type uh I could type anything so songs about um Love and so then I have this semantic search that's working and understanding the queries that I put in and

I'm getting songs about about love um I could also say um please return uh songs about color and I make that search um and so we don't explicitly mention the word color or any specific color we don't say that are white in it and so this is definitely not a keyword search um we can see and so looking at the responses and the results that we get

we have uh all these items so Pink Floyd uh because we have band names being indexed uh I'm black and Proud by James Brown The White a shade um Homesick Blues and so it really understands this this concept or songs about North America and we um and when we make this um or let's try and even make this more abstract and songs north of songs about places

north of the equator um just sort of drive this this that you get the most relevant um objects and so these are really interesting results let's see places equator and we get a lot of Earth Vibes because there's um an understanding that with the equator we have the the Earth okay so this is the way our application works right now and the idea now is to build

this um retrieval augmented application and so here we have our I'll just zoom in a bit and so here we have our application uh it's built in knjs and so the front end of our application has a couple components that you run a search um you handle loading and you display your results and so in this results component I want to add something something extra um and

before we do that we actually have to communicate with with weate and in my Vector search what I want to do is expand this search to a generative search and this generative search is our idea of retrieval generation and because there's an element of prompting llms what I'm doing here is I'm adding a grouped prompt or grouped task rather and I'm saying based of these results and

I don't even have to mention that because that's there as the um as the as the prompt is uh is augmented and so I if I asked what is the uh best song or or rather as something more contextual if I make a search about songs about color um I could ask which of these songs is uh less about it's not is not relevant to this query

yes let's do that of these is not um a woman for example um and then let's see what we get back it's not performed by a woman and so let's say songs. we right and so we get this um this answer say loud I'm backr Proud by change around is not performed by woman and so you have that context of these results that we get uh and

then that's what we we display uh and we could change this actually pass this um to be more true to the idea we can have the search term be the prompt um and so I will a perfect song to listen um if I'm thinking about uh love right and so we want to sort of narrow the scope of the llm in this case to the responses that

we get and so in the background the search will run will look for songs about love and so instead of having the llm look at all of the songs about love that are in its training data set we only get responses based off the results that we get from our search and so let's run some Perfect songs to listen to if you're thinking about love are one

everybody needs somebody to love two When a Man Loves a Woman where did our love go um by The Supremes for love and happiness for your love is precious and as you see here we actually have references for the items that powered this response right um and so I want to listen to songs about love based off the top 50 songs of the century from Rolling Stones

and not from all the training data and so this is the idea behind retrieval ented generation if I want to think I could change this to um what is the perfect song If I'm what should I not listen to so what one song should to if I'm going through a bad breakup for example since while on the them of love so quick search um we Stopped Loving

Her Today by George Jones um this song is known for its heartbreaking lyrics um and so this is the the song that uh we get recommended and all of this is very contextual information based off the response that we got back U and so this just drives the point that I was trying to uh communicate and how important retrieval is in having this um type of with

large language models and so just of end uh and go through what I mentioned and so the idea behind what you saw me doing was uh an implementation or a JavaScript implementation of a rag or retrieve augmented generation workflow where we have a query um be augmented with a vector search uh that we have over our data to then apply context and then give this context and

the query to our llm and then generate more relevant information uh and this helps us solve one the issue of uh the cut of training data in large language models it helps us solve the issue of privacy uh and data ownership where we can restrict the um interactions that our or the third interactions we have with third party Integrations um and of course we have more contextual

relevant results more accurate results with this rag like workflow and if you liked any of what I shared here are the resources I have a bunch of typ resources on how to build applications like I just demoed uh you could also check vera. weate doio out um and you could also check us out anywhere online thank you very much this is all our handles um all the

links and my own personal handles if you have any questions about building these sorts of applications um reach out and thank you again I hope you found something interesting from from the session