About this talk
This talk explores the concept of vector databases and their role in managing high-dimensional data. The speaker illustrates how semantic search works by discussing the ability of software to find images or songs based on contextual meaning rather than just keyword matching. They explain the challenges of high-dimensional data, such as the curse of dimensionality, and introduce various vector databases that help overcome these challenges by efficiently storing and querying large volumes of data. The speaker also discusses the importance of embeddings, which convert words into vector representations, and how these vectors are utilized in different applications, including customer support and code retrieval. The session concludes with examples of popular vector databases like Pinecone, Quadrant, and Mu, highlighting their unique features and operational concerns.
Full transcript
Hello everybody. First, u I'm going to start with a story. So, I wanted to find a picture of me and one of my friends. Uh it was taken on a cathedral on top of the cathedral in Mexico City and it I didn't know how the file was called. they don't know where it is in my um photo storage software and I just typed uh Mexico friend and
it showed me a bunch of images but just by typing the words and it said Mexico friend church and the domes of the cathedral was also visible in the picture and the software was able to find it the file name was img_5764 [snorts] iPhone stand format It didn't have any text on the file, didn't have any um annotations, any description, and it was still able to find
it, right? By semantic, by knowing the meaning, by analyzing the picture and saying, "Oh, there is a Mexican flag that's visible. There are two people, so maybe they're friends. Um, there are some domes that are looking like a cathedral dome." So semantically by the word of the meaning it connects the word I'm typing with what's actually on the picture without uh the picture has any context and
we experience this with our LLMs every day. If you go inside your LLM sorry and you want to find a song for something and you say uh give me 10 songs that feel like driving at night through the rain but hopeful. It's going to give you results. Some of the results you'll be of songs that doesn't exist, right? Uh but there will be some songs that when
you listen to them, yeah, it feels like driving through the night and this song is hopeful. And we see this not only in our personal lives but in our business context as well. Um there are plenty of software solutions that for example help uh uh customer support agents to be able to uh find problems similar to the problem that they're trying to solve. And these type these
solutions don't rely just on keyword search. Maybe uh there is a problem that user is describing and some other user described it in completely different way with different types of words but the meaning is the same and those systems are able to show the two or if you have an engineer that's debugging a problem those software solutions that are relying on vector databases can find similar code
to the code that the engineer is interested in and um the whole code generation software is also for a line was on these concepts and beneath all that between this semantic understanding of the LLMs of the AI solutions of the engines um this semantic understanding that they have of the world is vectors. So vectors are representation of every piece of information that is showing their meaning their
semantic in form of vector of numbers and this is an example of the embedding of isa conference as a phrase. Um and I added this conference because Esta has a lot of meanings that are not related to conferences because for international something something association there are a lot of associations. So is the conference is something that's close to what we're doing here and this is an example
from the open embedding model and this is what it returns. So the meaning of vista conference this phrase is represented by this vector over here and those vectors can be plotted right this is very nice writing from uh Steven Warframe that explains how chip works how LLM works and it's explaining also the embeddings because this is the underlying concept that they're relying on and this is like
a two dimensional repres representation where we get these big vectors and we represent them here and you can see that alligator and crocodile are close together but they are for example ant and a bee are further away from them but ant and a be semantically are also close together uh we have like peach cherry apricot uh pineapple close together mango is a little bit far for some
reason and then we have different clusters of words that are closed semantically in meaning And uh in real world these vectors are not two dimensional they're big. Um for the open AI models it's thousand one,536 characters. And for the best embedding model, Stella V5 series, it's um one yeah 8,000 um numbers 8,000 dimensions in a single vector that represents the meaning of the uh phrases or words
that you want to embed that you want to that you want to and there are different embedic models. This is the leaderboard of the current popular embedding models. Symbic models are the AI models that can get words and transfer them into these vectors into their semantic representation. We can see the Germany embedding 001 is currently top of the list. Then there are some QN models and difference
there are some models from the from Mistra on the bottom and uh there are different models that we can use. Some models are hosted like Jiny. Other models you can download and run them on your own hardware. But yeah, theic models are the ones that are transferring the words to their semantic meaning. And um there are a couple of problems with this if you want to work
with these vectors. The first is called the curse of dimensionality. And when we have 8,000 numbers that are represent that are representing values and these values are all from one to minus one it floats there is very little difference between all those vectors this huge vector space in to in this space the this is one representation that I couldn't came up with a better one but into
this space the ve you can think of a distance >> [snorts] >> um you can visualize it in your it and it's significant. You can say this is longer and this is shorter. But in 10 dimensional space first you cannot imagine it but the points become so close together that is very hard to compute the different the distance between them. And how do you do it in
10 in 10 dimensional space? Um how about 800 8,000 dimensional space? Uh the points become very very clo close together. And the second like problem is computational complexity to compute the distance between the numbers between the vectors becomes very computationally expensive and you need different algorithms. You cannot do it with SQL for example uh or other languages that we have in other data stores to be able
to find similar words or similar concepts to the one that that we're using. And uh basically if if we want to work with semantic meaning of data and we want to be able to query to find okay give me the images that are close to my search give me the songs that are close to the things that I'm looking for. Give me the user reported problems in
Zenesk or find the code that I that has these concepts. We need to be able to have a storage that [snorts] takes care of the those problems, take cares of the curs of dimensionality, can do complex things in a reasonable time and use sufficient algorithms that are designed to store the data in efficient way and also to query it. And [snorts] this is where vector databases come
to help us. Vector databases are systems that store the data in efficient way. We feed them these big vectors and they can efficiently store them um provide us ability to work with the data and to and and also when you think about a vector database is database uh like the other database and the things that we expect from relational databases or NoSQSQL databases or the other types
of databases that we work with we also expect them here like partitioning sharding link, indexes, backups, uh rosebased, access control, everything that that you have in your uh traditional databases, you also have it here in this space. These are like complete database systems. So the vector databases first of all allow us to efficiently store and manage large volumes of search data. Um most systems allow for linear
scalability. Um those systems when you add more nodes to such clusters they can store more and more information and process more and more queries. Uh there are different organization of data in terms of name namespaces or collections. You can think about them as the databases in the relational database world. And there are still queries for inserting upserting updating deleting all the things that we're used to data
definition language. And you can create indexes or index data and those indexes can be distributed and they also provide you partitioning and sharding and certain piece of information can go here. Another piece can go to another machine. Um and you can build indexes that that span all this. Uh vector databases also have specific searching and retrieval um algorithms or approaches. You can do k nearest neighbors which
is similar to a linear scan in the traditional database world. There are better algorithms approximate nearest neighbors. Um range yeah different types of searches uh that basically [snorts] operations when you say select data you can say that you want the database to select it in different ways and there's also [snorts] difference in the indexes. Um there are different types of indexes the same way as it's important
to define our indexes in a good way in order for our relational database to be performant is the same here. We need to understand what different um indexing mechanisms and algorithms there are, how they affect our queries, what types of indexes are best for our use cases and to be able to match the two. Um there are different different indexes depending on the need and depending on
the complexity of the system that we're building. Um if you can feed the data on one machine, you don't need something complex. If you have multiple machines, the data is sharded, then you need some algorithm that um index the data. So the sharding can work. And when we have the vector database, there are like um a standard way to work with it. We have some data that
we want to search in. This can be images, this can be text, internal documents, Google documents or code base or um Zenesk tasks on Slack messages, you know, some type some information that you want to We take this data through our vector embedding model and the embedding model returns vectors. This 8,000 dimension vectors and then we take those vectors and we store them in a vector database
um in collections. This is like the some people call it the offline flow where you take the data and you put it in the vector database. um some and you need to not do it once but it's the same as your relational database when you have updates in the source data when you have more Zenesk tickets or more Slack messages or the code changes or something the
thing that you want to search in updates you also need to have this offline flow to update the embeddings in the vector database and also sometimes um you need to update the way that the vectors are constructed for example there is a better algorithm that's going to save you storage and with lower dimensionality provides similar results to what you have and then because the vectors are stored
in a vector database um embedded by the old model you need to renindex them. So the this offline flow uh is something that is constantly running and changing and sometimes it needs to update the whole database when it makes sense. And then we have the online online flow where we have a query and we want to find the images or the documents or whatever we're searching for
it goes again. So we have a query that I type something in the search or the LM try tries to find something. It again goes through the vector embedding model which result in a vector. the query is represented in the vector and then I go to the vector database and ask it okay give me the the results which are the vectors that are similar to the query
that I'm currently executing and it's it's always going to um and returns which are the closest one and you usually say give me the top five give me the top 10 um maybe more but usually people go with five and this is basically the whole workflow of the vector vector databases and usually when we talk about LLMs >> um we have something LLM and it queries internal
data the this is the flow that it goes through basically it takes the query that that we're searching for and it again embeds it gets a vector search the vector database gives it this input and it says here is what the um user asked for here is some context that might be useful or not and do your job. Uh for example, we want to generate some code.
Um we say write this and this code. It goes through the vector embedding model. Our queries embedded in terms of vectors and then the code that is similar in terms of vector space is found and said here is what the ve what the user wants to you to the code that the user wants you to write. Here is the related code and then go and write the
code and LM generates the code. And there are different databases that I'm going to go over as an example. And the first one is uh called Pine Cone. Pine cone is the most popular vector database out there for some reason. And it's closed source. It has a free plan, but it's hosted and you use it as a service. Basically, it's a cloud service that you can use
and you just download the library and work with it. you don't worry about um spinning it up, about uh installing it, updating it, security, um all the different aspects of running a database. Um it's just something that you work with and you send data to it and then you query it and it gives you back results and uh that you can also if you're using AWS for
example, you can use pine on your AWS virtual private cloud. So it's very easy to start with and because of that a lot of people started experimenting with pine cone and maybe this is why it becomes so popular like the the vector database. Um it has very good documentation. It's very easy to do a proof of concept because you just download a library and work with it.
It's SAS only. You cannot self-host it. And it's a prominent company. I'm I put the funding of every company so you can get a sense of maturity because when you put your data somewhere you need to make sure that this company is not going to disappear and you need to migrate your data or they just stop operating and lose your data. So yeah, 100 million series B
funding in 2023. And just to give you an example of how to work to work with those vector databases, what the code looks like, here's one example. Uh it's a Python short Python code that uh creates an embedding adding some data and querying it. Then you can see that you download a SDK to connect to Pine. Um you provided the key then you create an index which
is documents [snorts] um and yeah you say AWS then you embed some documents here is the you say here is the thing that I want to embed use this model and you can do it in one query where the SDK take care of the embedding and then searching in the database and then you need to embed the query. You can see query open AAI open AI embedding
create um the query that you want to search for and then index query is how you query the database. So very straightforward as I said very easy to do proof of concepts. Another popular alternative to pine cone is quadrant. Uh the difference here is that it's open source but it also has a commercial license. If you want to support, you can self-host it in your environment or
you can use it in the cloud. So you can prove you can do your proof of concept in the in the cloud and if you like it then you can think about self-hosting. Um it also have a has good documents. There is a docker setup where you just download the docker compos file and you spin it up and it's built with scalability in mind. Um there are
there's sharding there is multi- uh cluster implementation and it has yeah it received uh 28 million is a funding in 2024 it was introduced in 2021 and some another alternative that a lot of people use is Mu um it has an open source version that's called Muvius and there is a commercial version called Zillas and it's very featurerich vector database. This is what people use in big
production systems. Um, and if you find engineering articles of companies sharing their story with vector search and embeddings and doing it at scale, most probably they've done it with with Mu. It's very scalable, but it has a very complex deployment. Uh fortunately there is a docker compos file or if you're running kubernetes there is a helm chart that spins everything up. It got 60 million series bunding
in 2022. It was introduced in 2019. And what do I mean by complex architecture? Here's an example. Uh this is the architecture. As you can see there are there is durable storage. It uses ATCs. It uses different technologies for um write ahead log. It has object storage. There are different nodes, streaming nodes, query nodes, data nodes. There is there there is a proxy layer that routes the
requests between the different services. You can see micros service architecture with different stoages and each with its own needs for scalability and operational concerns. So it's but this is if you work in scale this is like the go-to vector database to to and the thing that is very easy for you to start with probably is PG vector. It's open source and it's an extension to poss um
some people. Nice. So yeah, you can just install it. It's an extension. It's at specific data type for vectors. It also add operators for distance for calculating distance for calculating similarity between vectors and it also adds advanced indexing methods. So all because it's open source um it's very easy for the scientific world world to play with it and all the advanced state-of-the-art indexing methods are implemented in
PG vector as well uh I guess but they're they're yeah very very well supported and it was started PG vector was started in 2023 and to give you an example u what does it feel to work with this extension in posgress scale. So first you need to enable the extension and when you create a table you can see that there is a new vector type called vector
and over here we say vector with dimensionality 1,536 um dimensions numbers and we also have meta data always because when we embed the data we get the vectors back but if we don't add a pointer for example this vector is part of this document there is no relation in between them so we need to create it and usually in the metadata data in this JSON that we're
adding which um if you're indexing for example ZX ticket in the metad data we can add uh the link to the ZX ticket the ID of the ZX ticket the organization that it came from um the person that wrote it um some yeah all the things that we may might be interested in when we get the result of similar similar tickets and um then you need to
create an index. So there is a new type of index that exists. Uh for example, HNSW is uh one type of index. There's also a um hashbased indexes. Okay. Local base indexes, different types of indexes, but you yeah create an index um and you provided a distance function. In this this example it's uh cosign distance and u it creates the index. Then when you want to search
the data you say give me the ID the content the metadata and the embeddings and order by embeddings query embeddings. And over here you can see that this query is finding the three most similar vectors to the ones that we're we're selecting. And it's always going going to return results. Uh something that you need to be aware of and those results may be close in terms of
semantic but they may be far because you just don't have anything similar in your vector database that is closer in meaning to the things that that you're using. And to take care of that, you need to add an extra step in your application logic that basically gets the result and decides um are there relevant, which one is the most relevant. And there are special algorithms to do
it. They're called rare rankers, rare ranking models that take care of that. So after you get your results from PG vector um you need to also um yeah run them through the through ranker um and there is vector search in other products if you use radius or elastic search on MongoDB the hosted version at last there is already vector search in your software that you're using so
you can yeah keep it in mind same use cases um and also So another thing that you need to keep in mind is if you don't have a lot of documents, maybe you don't need a database. If you have like 100 documents, Google Docs that you want to uh work with, there are libraries that you can pull that find that can calculate uh those on the fly,
keep them in memory and you can query them and different languages. So uh you can just check vector and ve vector semantic search library and you can pull pull it up. And as a conclusion uh vector are databases you use them when you have a lot of data and you want to store it. Make sure that is there. Make sure that there are backups that is replicated.
If you need charting that you get charting if you want to control read write concerns you have robbased access control. Um it's a database system and you use it in every time when database systems make sense and when you need similarity search at scale for efficiently storing and quering highdimensional data. Um thank you and I think we have time for questions. Hello. Am I audible? Yes. Uh
thank you. Your presentation was great. Uh I wanted to ask you about an example here. Here here I am. [laughter] Uh an example of like what use cases did you implement using a vector actual use cases for? >> Yeah. uh zenes tickets find similar zenes tickets problems that we have solved for other customers but yeah are that we don't want to figure out the hot water second
time we don't want to reinvent the wheel already did it for one customer and can we do it for >> thank you other questions okay no more questions if you have any questions you can find Nikki after that in the networking area thank you very >> [music]
