Luca Bianchi: Serverless LLM with AWS Lambda and LangChain: A Revolution in Application Development

In this talk, Luca Biani, a CTO at Neo Experience, discusses the integration of serverless technologies with large language models (LLMs). He highlights the evolution and capabilities of LLMs, explaining how they can learn from massive datasets to generate contextually relevant text. The speaker emphasizes the importance of tools like LangChain, an open-source framework designed to simplify the development and management of LLM applications. He outlines the challenges of deploying LLMs into production and presents solutions for orchestrating various components necessary for building effective applications. Luca further explores practical implementations, utilizing technologies such as AWS Bedrock to create scalable, serverless applications that leverage LLMs for knowledge retrieval and interaction.

dude ah sorry I'm mute uh okay um welcome back to the next session uh now we have Luca biani with us um he's a CTO um at Neo experience and AWS serus hero uh LCA is an experienced professional passionate about software architectures serverless Technologies and machine learning uh he has more than 15 years of experience so he's very experienced on iws and all that he will be

speaking about um serverless llm so welcome Luca on stage I think I say it well Luka biank right it's it's right yeah yeah that's perfect you are one of the very few that can spell cly my name everyone present me as bian or bian but bian is the correct one yeah that's that's my do lingo sessions that works on on on Italian so well welcome daa um

tomorrow you will be hosting uh a track so this is your moment to see how this goes uh it's quite simple uh I'm gonna I'm gonna share your screens uh share your screen I have um also a phone let me know if you want to share the the phone later okay I'm gonna share the um slides now all the floor is yours much welcome everyone to this

session it's a pleasure to be here and talk about two of my favorite topics sess and L language models and we will see that we have a number of Technology features that can join them together and that could make you able to write your next gen app with basically no effort first of all here is what uh who am I I am the the right one so

the big one uh the left one is my little daughter uh and in my uh visit time I am CTO at new experience and a number of newer spinoffs and I am proudly sever year since 2020 and I'm passionate about servers the servers and software architecture moreover I'm one of the organizer of seros Italy and by the way sors days next June and J meet up so

I think that uh I am one of the very few people lucky enough to have put his hands on either sess and even driven architecture on one side and llms on the other side and brought them into production uh right now today I want to tell you a story and this story is about it starts as every story with Once Upon a Time and there was a

simple LM uh LM means that we have had a large language models which is uh a promising technology because remember that aart of the number the huge number of application that we see and PC that we see today we are still at day one we still at the early beginning of this revolution so we are still figuring out how to bring these llm application into production uh

the capability of these large language models that uh they are called large because they have been trained on massive data sets uh it is Rel that they are able uh to automatic learn and understand uh word and phrase relationships so they are able to extract the semantic meaning of a phrase of a paragraph of a text and store that meaning and they are able to retrieve that

uh bit of knowledge and use that bit of knowledge to generate a new text that it is resembl is in the same context of the input text this means that this can be can open a very huge number of application because uh they have been proven to uh be able to be used to summarize text to extract topics to detect sentiment within a text but also to

customize messages think about uh landing pages website Landing pag customization or knowledge retrieval a a lot of companies have a huge number of a huge amount vast amount of knowledge that is usually unstructured knowledge this is due to the fact that considering reports uh really uh more than 70 80% of the internet it is in the form of unstructured knowledge and these application are able to understand

the meaning of that of that knowledge they are not free from challenges uh there are a few stuff that you need to face is if you want to bring them into production and but it is an exploding landscape uh this is a picture that uh is just depicting the uh amount the the degree of availability of models that you actually have just if you are looking to

manage models so you can use all these models uh with no need to be a data scientist just invoking an API just calling an API and passing the right prompt the right prompt or the right data set to finde the model and it is massively if you consider that just two years ago uh there was no uh nothing of them uh we just had only uh GPT

3.5 released and everyone was playing with ch GPT and now we have this huge this vast amount of models but uh even uh the application that uh we can have are challenging because we have many uh stuff many topics uh that needs to be orchestrated to build an llm application so just having a good llm is not enough this is due to the fact that uh we

have uh a number of uh stuffs we need to orchestrate the planning of information retrieval and uh prompt uh planning prompt sending to llm but also we need to store the memory uh of previous interactions uh if we are providing a chat interface uh and we need to store in some way uh the data context we need to store the memory of the previous chats and even

uh be able to store in a suitable format the uh the sentences that have been sent then we need to evaluate the mod the models we need to manage the prompts we need to find you prompts we need to have something able to join all these pieces together with a bunch of uh traditional Cloud uh infrastructure services such as as databases such as Computing in order to

build a working llm application and here is where L chain comes into the play L chain is an open source framework uh that it is actually more than one year old uh it it has been battle tested in a number of uh application even production application uh that's uh uh made the platform the framework evolve and they reached the uh 0.1 version and now they are working

actively on the.2 which will be the next major release of the framework with a number of tweaks and improvements over the uh over the previous version the idea of Lan chain is to provide an abstraction on the different part that make your application uh and uh provide a standard and common interface in order to enable developer uh to just instantiate um say an application with uh some

kind of a a model or prompt and then be able to change the underlying model or the underlying prompt depending on your context with no need to restart to rebuild your application from scratch uh this means that uh this could really uh empower the uh capabilities of developer in order to make them able to take all the complexity of application it's uh it's made of several parts

we have the L chain code Library uh which is written in the supported languages and L chain supports either python which is the stand stand the facto standard language for uh machine learning and AI but it also supports JavaScript and typescript and it is much more important because the javascri support is an enabler uh to developers wanting to step into L chain uh just let me clarify

L chain is not a framework to train your model it's a framework to use pre-trained model deployed models and a lot of uh the surrounding stuff for your application and uh doing that with tespt with JavaScript is great because it is the language of the web so a lot of web developer are skilled using this this language and the Lang chain core is the base uh the

Bas framework then on that we have L chain templates which are a collection of blueprints so uh usable templates for different use cases uh that can uh consolidate the knowledge and the experience of other developers into reproducible use cases then we have two tools which are really interesting and are related to uh the managing and deploying of uh land chain application so just to be clear uh

they are external tools uh they depend on L chain but L chain is not dependent on them so you can use l chain in any configuration that we will be discussing about that we will be seeing um example of using L chains we don't need to use l ser and L me but in some cases in which you don't have a cloud infrastructure at your fingertips to

support you you don't have an existing deployment strategy using L serve could be uh interesting especially if you are a python developer in order to deploy L chain chains application in L chain called the chains because you you pipe you tie together different models different steps within your application and then langmi it is a development platform that can enable you to have monitoring to have feedback collection

to have alarms uh to have Administration and playground and so on and so forth so it is definitely something that you have to check out if you are not relying on existing Cloud infrastructure uh then here we have the uh URLs for L chain JavaScript and sh Python and they Point into the implementation but let's see uh some code about that this is a very simple very

simple code where I'm creating a chain it is a simple chain that uses uh Lama 2 model and it uses a construct provided by Lan chain which is the all Lama construct or Lama object that can be sated with llama 2 model and then I'm creating a pront and then I'm creating a chain and as you see the chain is the pipe of pront llm and the

output and then after that I just only have to invoke uh on the chain with an uh with a given input and I get an output uh I'm using the python code because uh uh it allows us to see clearly how the chain is built with no additional Cod it's perfectly clear uh to see how it works but the the nice part is that we can change

the llm variable we can change the model with another model we can change the prompt or whether we are providing data uh we can use parametric prompts and then we can read the prompts say for from a web page from another chat or whatsoever and that could be great uh and uh the reason why it is so flexible is that it is built on the L chain

explanation language which is a special purpose language uh which has been designed specifically uh to support parallel execution and retray tries and failback and also getting intermediate results so managing your pipe so it is uh something which has been built uh specifically to support piping of components then L chain offers you a lot of model input outputs so it enables uh to have prompt temp templates predefined

for different use cases or support to different LMS and then also have LS which can be one shot used so I need to summarize a test so I'm sending the prom I'm sending the text and then I'm getting back just one shot response or chat models which is an application of the same models that needs to be used uh in an interactive way and then we have

a capability we have tools to par and to customize the outputs and then let me see uh with L Chang how changes how we can Implement very easily something that is uh is becoming quickly the new hello world for generative AI which is refermented Generation rag for friends uh uh it is a method or a way of using llms to uh retrieve Knowledge from a vast knowledge

base so say I have the complete knowledge of my company say I have uh the complete library of Wikipedia and I want to uh make questions and extract information based on my questions from that vast amount of knowledge base and of course I cannot shrink all the knowledge base into a context even if I consider the most recent Google released uh model which a very huge context

it cannot host the old content of of Wikipedia uh then I need a retrial mechanism in order to search a database of Corpus of documents I need a generative model in order to generate a response once that I found the meaningful Advent the meaningful pieces and then I need a way of encoding that knowledge in order to make our search searches extremely fast and I am I'm

talking about semantic searches so I want to extract pieces that have semantic meaning and the other feature for rag in is that the Ral database can be updated can be improved and R can be summarized in these four steps load split embed and store and load means that I need to pre-process the documents I I need to acquire Source document that could be from web crawling data

lake or to propietary database and so and so forth I need to convert them uh from Source format so standard PDF docs and whatsoever to plain text and then I need to chunk them I need to split the text into smaller chunks maybe with a few overlapping because this red works great on small chunks and then uh the common the code to achieve that with L chain

is just three line of codes and I'm using this sample the Wikipedia page for University of pava I from pava uh so I love pava and this University I study there and the most important part is that the model that I be used which is open AI GPT 40 uh don't doesn't have the knowledge about this particular Wikipedia page so it knows what university of fa is

but it doesn't know the details of that page such as what did leonaldo studied and so on uh with just three this three line of code we can load a new URL and then load a new documents then after that we have loaded documents we the document object then I need to split the text I need to divide the text into correlated syntactically correlated Parts such as

phrases paragraphs sections and we need to uh make that fit multiple relevant uh part of the of the text in the in their chunks and ideally but is something that's quite difficult to achieve right now but ideally one chance should be associated with just one single topic in order to maximize the behavior of the r uh so I need to divide the S Effects by phrases paragraphs

and so on and then I need to take a few engineering decision so the maximum size of chance how long are the CHS uh and uh while to use uh overlapping between chunks so do I need to have uh fewer redundancy between chunks or not and then uh this takes me into defining this Matrix and it translates into just only two lines of code using line chain

inan the recursive chart of TX splitter which is an utility provided uh by line chain and providing as a parameter chunk size and chunk overlap and then in invoking text split splitter asking to split documents so I don't have to integrate with uh the underlying uh splitter I don't have to integrate with the functional routines needed to split a document I don't need to orchestrate splitting I

just have to invoke test splitter and uh ask for splitting the documents then the third step uh it is embedding in the embeddings we need to build a representation of the text in the form of numeric Rays make vectors and these vectors needs to capture the semantic meaning of words which means that two semantically closed terms needs to have semantically needs to have numerically closed values uh

so if I talk about uh students are visiting the city of Boston uh and my school is in pava I expect that school and students are closer they have numerically Clos closer values than say school and Boston as as well Boston and pava should have uh closer value that pava and the students this is to the fact that semantically uh cities and peoples are are different partition

of the language then I can use semantic similarity functions in order to uh find a compute similarity between worlds and then I can use modern LMS that leverage neural networks uh to represent model understanding of the of the text and having a numerical representation means that I can make vector algebra works on the embeddings so I can substract that I compute the distance and Computing the distance

since that is a semantically meaningful distance means uh Computing the semantic distance between these terms then it can be used for semantic search specification clustering uh Etc after that I need to store these embeddings which are huge vectors somewhere and a vector store is just a database which is specialized to handle Vector data so they can store huge number of huge vectors so high dimensional vectors that

Vector with a huge number of values that can be 1,00 1,503 36 and so on or maybe two more than 2,000 dimensions and they should also provide an efficient way to compute that vector algebra in order to find similar vectors and might scale up to billions of data points so if you consider just one single Wikipedia page can be splitted into hundreds or thousands of chunks and

we are going to compute embeds for each one of them which means that we are going to store uh a really huge number of vectors and they also need to index and uh Implement F treal and uh they could have additional capabilities there are a lot of open source and Commercial uh Vector stores such as open open search posts with PG Vector chroma and so on and

also commercial one which Pine con the mongodb Atlas And even relational databases are providing Vector extension uh and this is translated into more uh line of course uh which is just using pine cone which is a commercial product we just need to intiate the Pine code uh provide the embedding model which we now we are using open Ai embeddings and then uh just uh uh send the

splits and send the embeddings to uh to the uh to the vector store and we are done then after that once that customer arrives uh start typing in our chats and asking for question to our knowledge base we receive the text of the questions and then we need to embed the text and we need to search document using the embedded questions and which means that we need

to find out and to extract the pieces in our knowledge base is closer to that that embeddings and then we need to use the text belonging to that pieces of knowledge uh fill that into the context and then uh reply uh the question to the llm in order to allow the LM to generate a response an answer to the user and this stuff it is done every

time that I uh go back and forth with a dlm with an additional point of complexity I need to uh improve that stuff because I need to keep memory of previous conversation in order to keep the context and this can be achieved which is this is a completely different service and this can be achieved using uh just a few line of code which is super easy but

now I want to explain how I can use this uh L chain stuff to uh Implement uh something that can be really fast to be released using uh Amazon bedro which is a complet managed service Amazon B provides a knowledge based Tool uh which is a fully managed support for end to endend rag so you can configure knowledge base providing the vector store providing the source so

the bucket there three bucket uh that is related to document sources and it allows it automatically uh detects the Val new documents uses the uh embedding that you have selected to build embeddings then uh uses the store that they have selected to store a vector uh within the database and it is a completely manag service that you can configure from the console with just three steps and

you can also configure and invoke that using an API so after that you will have an endpoint that you can call just to perform all the rug stuff uh and it is super suitable for serverless which means that we can use servers to store documents uh run llms or inquire for LM run queries against store inter interact with us and this could be uh this could be

great because providing a p you go model all for scalability and also having a cost effective approach uh interaction can require many seconds because llms usually have a per second response time but we can use serverless which is intrinsically even driven in order to handle this asynchronous interaction and just a very simple architecture that I I would suggest you to consider for the rag example it is

user that can upload file into three buckets a Lambda function splits uh the the data and then start the Ines job then ingest toxin are set to open search uh to open search Service as a vector store and then we have another Lambda that's retrieve the knowledge the best part of this story is if we use uh Amazon knowledge base and if we use the library that

I'm going to show you uh all this stuff uh will be the the point from number two to eight uh they will be provided to you uh directly from Amazon knowledge basis uh this is due to the fact that some knowledge basis uh is able to orchestrate Vector store uh embedding models to generate and compute distance and also provide an apaa for that and then we can

complement uh Bedrock knowledge bases with uh cdk for generative AI a cdk is a framework is a serverless framework to build infrastructure scode architectures and it is it is super suitable for Server because it can uh generate clock formation scripts to handle different uh stuff different tools together and AWS released cdk for generative AI at last R invent and they provide L3 cat which means uh ition

cat ma to uh predefine new cases uh to implement Rags uh and now we are going to see how we can uh rewrite that previous example just using uh all the resources so we can Implement architecture that we have seen just using the resources provided by cdk constr constructs and as you can see uh cdk generative AI provides constructs to configure pine cone within badrock knowledge basis

and then it provide the capability to configure bettered knowledge basis providing the vector store the embedding model here we are using Titan embed V1 and even the instruction uh to configure that uh that way to build the embeddings out from the documents and then for the documents we can create a three bucket and then we can set that S three bucket as data source uh with with

uh the tokens and the number overlapping overlapping percentage that we want to uh to provide moreover we also can Define the chunking strategy with whether we want a fixed side or variable SI chunks uh and then we can also complement and extend this simple use case using the elri construct which is already available Within cdk AI construct and it is a construct that uses a up sync

in order to up sync manage uh a pipeline for data ingestion and it is able to ingest data coming from uh PDFs or or jpeg even images this conru is super powerful because it is it can also index data coming from images and it provides you an end point in which you use to upload the into the injust pipeline that is store in S3 and the data

the PDF it is opened uh by uh the code the Lambda function that under the hood they use l chain in order to extract information from the PDF they Implement a lot of chains in order to achieve that and with images they use uh recognition to extract metadata from images and then all this information it is stored it is built as an Medics and store within um

open seource service and uh to you that want to implement this use case even with appsync which is a manage graphql so even providing an API to your customer to your end user it is just as simple as inanis one uh cdk construct which is the rag up sync step function open search and you can also provide a Cognito user pool ID uh which means that you

can this um the access to these apis it's also controlled using uh US Pool Cognito and just inant in that one it makes you have available all these services and all this code are already configur for you Al we also have another L3 constract which is the Q&A document Q&A construct and it is uh it is again an up sync uh Q&A and I'm seeing uh two

configuration using lamba functions for question answering and uh leveraging on the same S3 open search and bedro services Power by cloudwatch and X-ray and here again to implement a Q&A service so the other part of the architecture that we have seen before it is just as easy as uh to build toate a new QA app sync open search providing here gain a Cognito uh use and P

for authentication so they are extremely powerful even if you want to add a summarization step if you want to have uh something using llms and using L chain to summarize a testt you can you have a construct for that and the same plan istic by this summarization up sync step function uh which is another construct that's available just to implement that one and putting all together we

can build a complete solution for document exploring document management and uh we can build this complex architecture just joining these three constructs together in the same code base and have cdk Genera the code uh and the infrastructure for us to implement this use case and to provide this use case to our uh to our customer and uh so this was the final part of my my presentation

and we have a few minutes for Q&A uh so if you have uh if you have question I'm here thanks thanks a lot uh Luca super interesting uh I'm going to check if there's any questions in the meantime uh but um I'm gonna I'm going to ask you something about this whole thing about llm and and actually the fun part is that uh I have origins in

papia uh my grand-grandfather was from there so it's nice to see nice to see Pavia mention in a in a conference this is something like you know that's uh it makes everyone proud anyways um the the situation with the um open source models do you think that they are they have any chance to to compete with the um proprietary like uh the ones that are Google developing

and meta and AWS developing or it's something that we have the open source only for standards and to compete with them for say in a way H that that's that's a good question uh I think that they have reason to exist because actually we are moving from generic purpose llms to more tailored and specific kind of llms so I would expect in the future that the battle

won't be uh into having uh a faster or with longer context llm compared to the vast amount of investment that has been done by uh cloud provider but also we are we will face uh sooner in a couple of months um the emergence of smaller models that can be suitable for embedding use cases that can be deployed to a smartphone and in that case being able uh

to do a lot of research uh into the the architecture of the model to have a clear uh understanding of our architectural works is something that could make a lot of research institutes work on them and consider also that it is not a straight line between commercial and open source because basically every provider AWS Google and Microsoft they are providing either proprietary and open source managed version

of these models so I expect that there will be a lot of mix and matching in the future okay okay so you don't they also fining yeah they also financing a lot of research project you consider the most recent investment in LMS that have been founded by uh by vendors yeah that's true that's true yeah basically just to give a little bit of context because I know

a little bit about the situation uh the the the European Union is trying to to well not to control to legislate as usual they are not allowing a lot of investment and a lot of innovation but they are trying to legislate a little bit and and that's why mainly the the the point of the question and also because I'm also involved in open source stuff and I'm

usually following that and it's a little bit concerning that seeing that something that needs to be uh something as neutral as possible be managed by a couple of hands that's that's let me let me add let me add another bit of information there is a a huge bonus point in open source LMS which is related to the fact that you can evaluate them uh model evaluation it

is something difficult it can be done even for closed s for propietary models but it is quite difficult because you don't you don't know on which data set they have been trained you don't know uh you have to trust The Benchmark that are given you by the provider with open source llm maybe you won't be building a new open source model but you can evaluate them and

you can have a clear leaderboard about which model works better better on which tasks which is something more difficult to achieve with propar totally totally cool thank thanks for the clarification Luka okay I don't see too much uh movement on the chat uh looks like the people are a little bit shy still hasn't wake up in the European time um so thank you a lot LCA we

are reaching the 40 minutes so it's 45 minute session so we we are closing now thank you a lot LCA uh glad to meet you at least online and and hope to see you in another conference and for the people that are still watching us uh stay tuned because we are continue right now I'm gonna say it in the right way it's 11:55 European Central European Time

12:55 for the people are on the um GMT plus three so stay tuned we'll back shortly thank you bye- bye luga bye [Music]