Spring into AI - Venkat Subramaniam

This talk focuses on integrating AI capabilities within Spring applications. The speaker begins by discussing the potential of language models, particularly large language models (LLMs), to aid developers in enhancing user experiences. Several use cases are presented, including utilizing AI to provide tailored travel recommendations and personalized content based on user profiles. The speaker emphasizes the importance of context in AI responses and introduces Spring AI, a library that enables seamless AI interactions within Spring applications. The talk includes practical demonstrations, such as building simple controllers for generating dad jokes and crafting a conference query system that retrieves information based on augmented data from vector stores. The speaker concludes by addressing the challenges of ensuring accurate AI outputs and the significance of discernment when implementing AI-driven features in applications.

Thank you so much for coming to the session. This is going to be on using AI with Spring. I'll talk just a little bit and then we'll look some look at some examples along the way. If you have any questions, the best time to ask questions is anytime you have a question. Just tell me attention and start speaking up. I'll be delighted to hear what you have

to say. So, we are in a very interesting time and we are we have this ability of using AI for a lot of different things. Now, let's step back for a minute and think about this language models. The language models, we we are in a unique position. We are software developers. We are programmers. In in very few fields, you have the ability to be both the user

and the developer. Try the dentist cannot do that, right? You can always tease them. They can't do their own dentistry. Some have tried that didn't work out really well. But programmers can be on both sides. So, when it comes to LLMs, you can use LLMs to write code. You can use LLMs to create images, to create web applications. So, we can be the user of LLMs and

and there's a lot of use cases and power of being the user. Well, this talk is not about that at all. This is about how can we leverage AI within our applications so that the users of our application can benefit from it. Let's talk about a few use cases where this will be a helpful. So, maybe you're creating an application where you are a travel site and

somebody wants to go travel around the country or around the world. And so, when they use your application, they may say, "Hey, I just booked a ticket to fly to, let's say, Pune." Well, then you say, "Gosh, if you're going to fly to Pune, let's tell you about things you can do in Pune during the time you're visiting." Maybe the week that you are visiting, maybe there

is a huge music concert or a film festival, it'd be nice to know about it. So, that becomes very enriching for your users, not only to book a ticket to go there, but to know other things they can do as well. So, that is one example. Now, if you think about this, you can say, "Okay, that's great, but where's this information coming from?" This information is not

specific to your travel site, your product. This is what's happening in Pune, and your application is pulling that information and showing it to them. So, this is you could call this as a common knowledge or a wide widespread knowledge. We could call it public information. But, sometimes you may want to really customize this to the specific user. So, for example, if I'm using your site, I might

say, "Tell me what's interesting in the city that I'm visiting." Unfortunately though, when it comes to bank, the word interesting means tell me what what Java user group is running that week. Or tell me if there is a, you know, user group related to Python, right? But, when my wife travels, the difference between me and her is she actually has life. So, when she comes, she wants

to go to other events, fun events, right? And she's like, "I'm not going to go sit in a user group. You go for it. I'm going to go to a music show, or I'm going to go to a, you know, cultural event." So, when she says, "Tell me what's interesting." it's got to be very different than what I consider to be interesting. So, what I'm saying here

is, yes, it's public information, but you need to tailor that to the person who is actually using it. So, that's a little bit of a use case, right? Now, you say, "All right, that's great." but the travel site also goes beyond that. The travel site says, "Hey, user, you are booking a particular ticket, but there are rules and regulations around your booking. You can re uh uh

book your you know, ticket or you can cancel it within certain duration of time. If you're going to be doing it after that time, there's a penalty for change." Oh, on the other hand, "You have been a user who is a premium user. You've been constantly purchasing tickets from us. You have a loyalty status. So, those things are waived for you." But, how do we determine what

these rules are? Because these are not public information. These are proprietary to that particular product or application. Secondly, these can change over time as well. So, when a user is logged into the application, they say, "Tell me what's interesting in Pune." is very different from, "Oh, I just bought a ticket to Pune, but what are the restrictions on this ticket?" That is more of a local information

we need to bring in. But, how did we program these over the past several decades or years? You and I, the programmers, would sit there and write code for that. So, you would sit there and say, "If the user makes this request, we'll perform this query on the database or go through these things and display this result." Which is very painful, time-consuming, takes a lot of effort,

and when you're done with it, the rules change, and here you go, you got to make all those changes again. That's time-consuming, boring, uh you know, and and you know, it's costly as well. So, then you say, "Ooh, I can use a rules engine like Drools, and we can then specify those rules, and at program running time, we can run the rules engine and ask it to

tell us what the rules are." Well, a lot better than what we did before, writing custom code for it. But now, somebody's got to write these in a format the rules engine is going to understand. When the rules change, we have to fire this up and deploy it, which is fine, but it's very restrictive in terms of what it can do. Now, before we go further, I

want you to deeply think about something. Because if we don't understand this, it's going to really hurt us. There are two kinds of solutions that are typically available for us in today's world. One solution is very limited in what it can do. It cannot do a lot of stuff. It's not that flexible. But the beauty of that solution is it's extremely predictable what it is going to

tell you. So, that is what we call as a deterministic system. So, deterministic deterministic systems today are not powerful, but they are very stable and reliable. The other part of the story is you have a system that's enormously powerful. It can do wild things. You don't have to write code at all. Simply ask it a question, it's going to send you a response. And what can it

do? And it feels like it can do almost anything. That's the good news. The bad news is it's extremely unreliable. It's extremely non-deterministic. So, if you ask me, "Tell me what it's going to say," I'm like, "I don't have a clue." If it if it's a lucky day, maybe If nice sunny weather outside, it may be give you results that are close to what is correct. Or

maybe there is a cloud cover and it goes random crazy and it'll throw all kinds of stuff on you. Now, this is unfortunate, right? Because we as humans love to use euphemism. You don't want to say really bad things to people, right? You don't You know, your boss comes to you and says, "Hey, what do you think of Joe?" You don't say Joe is stupid, right? That's

not going to feel good. You say Joe is interesting, right? Because then it feels like Wow, that's a nice way to say it. So, and then boss says, "How's How's How are things going?" You should never tell the boss, "I've got a problem." They freak out. You say, "I have a challenge." So, then they're like, "Ooh, you have a challenge." So, interesting people have challenges to deal

with. But the point is, you don't say AI is lying. You say it's hallucinating, right? That feels a lot better. Now, what do you do with your colleagues who hallucinate, right? You're not going to say, "Oh, dude, you're awesome," right? Well, but unfortunately, we've become complacent about this hallucination from AI. So, we need to develop this ability to discern between right and wrong. So, when AI gives

you a response, you need to see very clearly, this is like a good response or uh this is not a good one. We should not use this one. This is especially important when you embed AI into your application. Why? Because if AI begins to hallucinate and tells your users the wrong thing, you're going to be in a lot of trouble. So, so that is something a caveat

just to be sure, be careful before we go any further. So, accessing this LLM from within our application programmatically, how can we do this? Well, there are different tools for this already available depending on programming languages that you are using and various other things. If you are programming in Java, you can use LangChain for J. Well, LangChain is a Python implementation. The For J is a Java

port of LangChain from Python to Java. So, you can use LangChain for J, and you can implement it. Well, that's an open-source solution. You're your application is not using Spring, you can use LangChain for J. But, if you're already using Spring, you can still use LangChain for J if you want to, but why not just use what's available in Spring already? That's where the Spring AI comes

in. So, what is Spring AI? Spring AI is a set of libraries that allows for your Spring application to use AI internally, so your users can benefit from having AI built as part of your application. Now, we're not going to talk about the guardrails and how do you make sure these things are all correct responses? That is something we need to take the time to figure out,

but we're just going to look at the plumbing of things today. How do I communicate with these different tools? So, let's think of an example of how we could use Spring AI to build something. Now, I want to build a really quick example to illustrate this. So, what can I build? So, one of the things that I absolutely enjoy, and I'm sure a lot of people in

this room really like, is dad jokes, right? Dad jokes are amazing, because it it the more dad jokes you say, the more the children hate it. And and I've realized over time that my mission as a parent is always to embarrass my children, especially in front of their friends. So, when you start saying these dad jokes, they start cringing, and so more dad jokes makes your life

a lot more interesting. And it could be a mom joke as well. We normally say it's a parent joke, but we call them dad jokes. So, what if we can ask AI to tell some dad jokes, right? But, I want to be able to pull this from within the application. So, again, you You you could ask, you know, what's what's you know, what is interesting in Pune?

But in this case, I'm going to just pull in a dad joke and ask it. So, how am I going to build this in here to bring it in? This is a very simple Spring application as you can see. There is not a whole lot. It simply is a demo application and it just fires up the fires up the application. It doesn't do anything right now as

you can see. But what I'm going to do though is I'm going to be able to exercise this in here. So so I'm going to fire this up and and be able to execute in here. But we'll come back to that in just a minute. And I'm going to make this a little bit bigger so you can see it a little bit easier. So we'll come back

to those in just a few minutes. So what do I want to do right here in this application? Now, uh this is a very simple regular Spring application. Nothing really exciting. So I'm creating a Java class right here. And I'm going to call this Java class as joke controller. That's what I'm going to call it as. So typically, right, if you are creating a Spring application, you

know you can create controllers. Controllers are the one that receive the request from the web request. So I'm going to make this one as the as the rest controller, if you will. And and essentially this is going to be receiving a a request mapper and that's going to be receiving a request into, let's say, a joke. So I'm going to ask you to give me a joke.

That's all I'm saying. So what am I going to do within this? I'm going to say public, in this case, well, first of all, let's say a private and I'm going to use in this case a a joke service, which I don't have yet. And I'm going to call this as a joke service. Now you know in Spring you can do what is called dependency injection. So

in this case, I'm going to create a joke controller, which is going to receive in here the joke service, if you will. And this joke service is going to do the real work for us later on. So we'll say a joke service and this is going to be the joke service. And all I'm going to do is this dark joke service is equal to the joke service

and assign it. As I said, we don't have it you know joke service just yet. We're going to be able to use it. But I'm going to then say request mapping right now and and what am I going to do within this request mapping? This is going to be simply a random you know joke we want to be able to say. So I'm going to return a

string get random let's say a joke and and what am I going to do? Return the joke service dot and tell let's say a joke. But this is going to be pretty wide. So I'm going to say dads. So as a result you can ask it to sell tell different types of jokes if you really wanted to. So this is going to simply send out a request

for a joke. So at this point if you stop right here, we did nothing specific to AI, right? This is just like any other plain vanilla controller, nothing really exciting at all. So now I'm going to say here is my joke service I want to implement. So this joke service obviously is a service that I'm implementing. So what am I going to do in this particular joke

service as you can see? Well, before we go further, I have set up an environment variable on my system which refers to open API open AI key. Now to access open AI or any other AI, you need a key to access their API. The key keeps track of the account. It also makes sure that people don't start making a lot of requests on their site to downgrade

their performance. So there's accountability involved. That key exists. So that key is in my environment variable. Obviously, you don't want to divulge just to the entire world. Otherwise, my bill is going to go through the roof. So I'm going to keep that away, but we're going to be able to use that a particular key. And so I've already set up a key. So what is that key

that I've already set up? This is called Spring AI underscore and this is open AI underscore API key is the key that has been set up. So all that Spring AI by default is expecting is that you have already set up a key with that particular name and and basically I've done an export of that and then assigned it to the key that I've obtained from Open

AI for my own account. So now I'm going to say a private and this is going to be a chat client and I'm going to create a chat client. The chat client is coming from Spring framework AI.chat client.chat client as you can see. So that gives us the client I'm going to use. This is again the same old dependency injection we talked about. So this is going

to be chat client in this case dot builder and I'm going to say this is going to be the builder that is going to get injected into this particular constructor. So what am I going to do? Chat client is equal to builder.build. So typically they use the builder pattern which by the way is not really a builder pattern where they just call it and a lot of

people use it. I'm not a big fan of it but who am I to say, right? So a lot of API use this kind of a builder.build as a way to really you know create these objects. So at this point I'm going to say public string and remember our function tell a joke. You know what? Let's do it this way, right? So rather than writing it here,

let's go to this tell a joke and ask it to provide that method. So here is the tell a joke and this is the nature of joke that I'm going to bring in in here. So what are we going to do? Return the chat client dot and I'm going to provide a prompt to it and then I'm going to say here's a user message and the user

message is tell me a percent as a joke, right? That is like a dad's joke and formatted nature of joke. And then I'm going to make a call to it. And once I call, I get the content from it and I'm going to return it. So, this is basically the beautiful abstraction that's been provided for us, right? I'll make this a little easier on you to read.

So, I'm going to just format this all uh vertically uh the dots vertically, so it's easier to really see what we are doing. So, what is really nice about it? What is nice about this is that anything you deal with, if you are provided an abstraction, it makes your life easy to talk to it. You know, raise your hand if you like to really open up a

socket connection and send a JSON request and get back a JSON. Nobody raised their hand, right? Good people in the room. So, essentially, you make the call and all the dirty uh complexity is hidden between these functions, so you can focus on getting your work done rather than fiddling with a you know, URLs and the JSONs and all of that. So, assuming this is all good, we

should be able to make use of this. Now, I'm pretty lazy. So, what I have done is I wrote for myself a run server, which is simply firing up Spring Boot and calls a run on it, right? So, nothing really exciting, as you can see. It is simply firing up that call and runs the Spring Boot application. So, I'm going to run the server, as you can

see, and say go for it and it just started the server. Then, what am I doing? I have a run.sh that says do a curl request and fire up a request and send it to joke random, which is the endpoint that we created. Again, I'm little uh you know, lazy. Uh you can use Postman. You can bring up a HTML page. I'm a I'm a command line

junkie. So, anytime I can be in a command line and do things, I'm just happy to do it. I don't need fancy things to be flying around. So, I'm just going to use that right here as well, and I'm going to go ahead and fire up a request, and you can see that it's going to make a request for you and go off to the service and

bring back a response, and it says, "I I used to hate facial hair, but then it grows on me." The Now you know why children cringe, right? So, so essentially, that is one response that No, oh, by the way, before I go further, if you are to get offended by good luck. I have no control over it. This is what AI is going to return, so I

take no responsibilities to what it's going to tell. So, that I make that very clear. In fact, it's so cool, it says, "Want Do you want another one?" So, let's run and see what it's going to tell me this time. What is it going to drag in? And right now it says, "Why don't eggs tell jokes? Because they would crack each other up." So, okay, that's I

think that's enough, right? So, so essentially, we have a few of these that came back in here. Okay, so that's great so far. Now, of course, we have a small problem. We are not just dads, or not just moms, not just parents. We are special parents, right? What do What is special about us? We're not only dry in our jokes, but we are also geeks. So, the

best way to really intimidate children is to tell tech jokes to them. That's even worse, right? And so, you want to Oh, I have to tell you this because this is a moment of absolute bliss, I would say. This is actually I was working in the kitchen one day. Well, I'm Let me clarify. When I say working in the kitchen, I was on the computer when I

was in the kitchen. And my wife was in the kitchen as well, and my son, this was years ago, and my son came over to me and said, "Dad, I have a question." And I I was focused on my work. I didn't want to really be distracted. So, without missing a beat, I said, "You know, 301." And and that's all I said. And my son immediately makes

a U-turn and goes to my wife, "Mom." And she says, "404." And he's like, "Damn it." And he walks away. So, it's really nice for our entire family to be communicating with the HTTP code, right? So, so it is that it is my version of parent joke where we just without we never planned for it, but it just fell in place. So, that works really nicely. So,

essentially in this case, I want this to be more specific to our, you know, geekiness and and and programmers obviously. So, what are we going to do to specialize this? So, I'm going to go over here uh to this and say dot system and I'm going to send it a system message to customize this one. So, what am I going to say? I'm going to say, "You

are a programming nerd and we'll talk about coding, you know, coding, you know, whenever, let's say, possible. You know, please so keep your response related to programming where you know, where you can." So, so this is basically a little system message we provided for this. Now, before we go to the system message, I'm going to tell a dad joke, but this is not AI generated. This is

uniquely generated by me. If you have heard this before from me, it's your responsibility not to say the response, right? You're not allowed to say the response only if you have never heard this from me. So, are you ready? All right. What is the most favorite animal for a functional programmer? What's the most favorite animal for a Lambda, right? So, absolutely. It's lambda. Absolutely, lambda. So, essentially,

this is going to tell us a programming joke, right? That's what we're going to be receiving. Uh some people say Python, which is also interesting, but I think lambda is much more appropriate. Awesome. So, let's go ahead and run this code now and see what it wants to tell us. So, in this case, uh we are hopefully not going to get a joke related to uh Oh,

so why do programmers prefer dark mode? Because light attracts bugs. So, so nice you can see that's bringing in something very much close to uh programming, right? But, I run this again this time, and what does it say? Uh why do programmers always mix up Halloween and Christmas? Because October 31 is equal to December 25. Uh so, then it says octal 31 equals this decimal 25. A

little base conversion dad jokes uh for a debugger in all of us. So, this also this is also a good point to think about. One of the tenets of jokes uh that we need to keep in mind, and AI doesn't know this. One of the tenets of joke is you you you should never explain a joke. Because if you explain a joke, it it it absolutely loses

its essence, right? Because somebody says, "What's funny about it? Let me explain to you." And then you explain it, and they're like, "Yeah, really? What is funny about that?" And then the rest of the afternoon is really messed up. You know, the the best response ever. So, just be aware of it. Eventually, AI will be be trained on it. If you say, "What's funny about it?" the

right response is, "Never mind." Right? And move on and have a good life after that. So, so don't explain jokes like it just did. So, anyway, we saw this really give us a response, and then this is a way to customize the response and get the response as we can see. So, we saw the system message. Okay, great so far. But, what we noticed here is you

can use a system message, you can use a query, that is all great, but there are times when you want to do something even better, even more. So, for example, what if you are asking a question about the AI LLM and it's able to get public information and give it to you, but if you ask it, "Hey, what's going to be the weather tomorrow?" Well, what in

the world does tomorrow mean? And and it may it may be that it's going to look at the next day. But, the problem is what is the you know, weather tomorrow depends on where you are, that you need to or where you are going, but also may depend on what tomorrow means. I just did this mistake this morning and that's when I realized how how naive I

was. I I I was posting something on social media and I said, "It is today." Well, guess what? It is still yesterday in the US and I'm like, "Gosh, I should have been careful. I should have said the date and not say today." Because my mind is wired because I'm in India right now, but most of my folks are in US and when I said today, they're

going to be very confused looking at the message until unless they see that tomorrow. So, in other words, context matters. So, how do we tell AI, "Pull things from me when you need information that's not present?" For example, if I say, "Do you think I should be wearing this color shirt to this event or should I change my color of the shirt?" And AI, I hope, is

not looking at you through the camera. That's going to be really scary days, right? And so, what should AI say? AI is going to say, "I don't know what color shirt you are wearing, right?" And if you tell me what color shirt you're wearing, maybe I can then tell you if it's appropriate for this event or not. So, in other words, you want interactivity where AI may

do its work, but may need some information from you, and then go back and do a few more things. So, this is not related to dad joke, but I'm not going to spend my time creating another controller right now. So, I'm just going to hijack this. So, just be aware this has nothing to do with dad dad jokes anymore, right? So, what am I going to do

here? So, I'm going to go back to this and say, I want to make a call to this chat client, but what am I going to make as a call to this particular client? So, I am going to tell this particular chat client, "Hey, I want you to tell me something about things I may not know about and help me out over here." So, I have a

prompt here I'm going to send. But, what is this prompt going to be right now? So, I am going to tell as a prompt the the request I'm going to make in this particular case is going to be, uh you know, what is So, when was the version of Java that I am using released, right? So, I'm saying, "Tell me when was the version of Java that

I'm using, uh you know, when was you know, released?" So, I'm going to use this as a request over here and and pass that in, and then I'm going to make the call to it and return the context to the caller, right? So, so basically in this case I'm saying, "Tell me that the version and and when it was released, I need to know that information." Now,

obviously to you here, you know, it cannot tell you that information. Why? Because poor AI doesn't know what version of Java I'm using because that's running remotely elsewhere. It doesn't have my information, my system So, what's going to happen at this particular point? So, I'm going to go back and run this like I said, I just hijacked that you know, I'm lazy to create another controller. I'm

just going to hijack that and run it. So, I'm running it to see what it's going to tell me and and what is the response I'm going to be receiving at this particular time. So, it's thinking, it's going through executing and and all that processing is being done and it's a little slow right now, but let's see what it's going to say. That's taking a lot longer

than it should. Okay. So, while it's doing it, ooh, and notice what it said, if you cannot run it, here is how to read the output, okay? Great, but before that, I don't know which version of Java you are using. I'm like, thank you, right? Because I'd be worried if it said I know it, right? And it doesn't know it and could you please paste the output

of this command and tells you what to run. But, I don't want you to tell me the history of everything. I want you to tell me when the version of Java I'm using was released, right? So, how do we actually accomplish that is the So, I'm going to go back here and say, all right, all right, let me help you out on this one. So, I'm going

to send the prompt, but before I send the prompt, I'm going to say tools over here and new, I'll call it version info like so. First of all, it doesn't matter what you call this class. I want you to step back for a minute. Now, this is where an absolute power on your hand comes in. What what have we struggled in our in our field until 2

years ago? Until 2 years ago, our syntax were extremely restrictive and we couldn't use natural languages as syntax. But, what if natural languages can become the programming syntax? So, in this case, the AI doesn't say write an SQL query or it doesn't say I'm going to give you a syntax that you need to use. AI says speak what you do normally. If you want to speak in

English or Dutch or, you know, Hindi, it doesn't matter and I'm going to understand your natural language and do things. And and I I want you to think about that power on your hand, right? Which is absolutely phenomenal. So, what are we saying here? You are saying tools, which means you can give many of them. So, in this case, I've created a I'm going to create a

version info as a tool. Again, like I said, you can call it whatever you want to call it. However, one of the things that we could have done better and some of us have done will come in very handy for AI. If you write a variable called X, Y, and Z, do you know what X, Y, and Z means? If you don't know, AI is going to

have a trouble knowing it also. It's got to work extra hard to figure it out. But, if you call your variable temperature and you call another variable as pressure, and if AI is looking for temperature, it knows to use the temperature variable and not the pressure variable. So, this is a case where if you're more descriptive, you are helping AI to be more sensible and you can

make it more accurate in its effort. So, in this case, I'm going to provide a version info as you can see. And what am I going to do in this version info? In the version info, I am going to go ahead and write for this a little annotation and this annotation is going to be called as tool. So, I'm going to say tool uh and the tool

it has a description, if you will. So, what do we do for the description? Description and this is where you need to be as descriptive as possible, right? If you say, you know, "fubar", it's not going to understand what you mean, right? It's like, I don't understand what that means. So, context matters, right? So, what do you say here? Think about what you want to say as

a description and provide that, right? So, you know, you can say, "a version of Java that I am using." Hey, that's pretty darn good, isn't it? Because if AI says, "Gosh, this dude wants to know when the version of Java he or she is using, but I don't know what version of Java they are using. Let me see if they have a tool that gives me that

information." So, it walks through the description of all the tools and says, "Do you have any tool that were to tell a Java version?" But because it's a natural language processor, it can figure this out from English. This is not a required syntax. You can say, "version of Java I'm using." You can say, "Java version that's on my machine." Whatever it is, right? As long as you

can give the context for it to figure out, it's going to be able to provide that. Then you can say, "a public uh string uh get info." Again, whatever name you want you can call for that function, there is no restriction. The only thing to keep in mind is you cannot use overloading. So, you need to really be able to specify that, and if you have parameters

that it's going to send to you, that's perfectly fine as well. So, this function can take parameters also. But if you want to have multiple things that do the things, just write different functions, don't overload them in general. So, in this case, return uh 25. Oh, actually 26, how about that? Which is a version I would like to say I'm using on this machine, and and you

know, you can uh uh get this uh by uh querying uh system info or querying the user whatever that may make sense depending on the application that you are using. So, all we did is we wrote a tool and then we provided that as a version info and that's all we did and here we are providing that version info as you can see. So, uh going back

here, let's fire this up and I'm going to go ahead and uh as you saw that we just hijacked that particular call in this particular case, but let's run this and see what it's going to tell us. So, right now as you can see, it's going to not know the version of Java you're using and as a result, it would really like to know, but it's going

to look for the information from the tools that it's going to look for. And and what does this do at this time? It's going to go and query it and when it knows that it doesn't know enough information, it needs information from you, it's going to call that method which is get info. So, the tool reports you're using Java version 26 and OpenJDK blah blah blah and

it gives you information about Java 26 and and uh you know, it's it's March 2026 and then etc. etc. etc. and then gives you more information. Now, uh I just want you to look at that for a minute and I'm going to run this one more time uh just out of curiosity and let's see what it's going to tell us uh in this particular time. So, I'm

just re-executing that query. Well, it's still Java 26, I didn't change that, right? And and it's going to give us the response from it. So, in this case we're running, it is still the same Java 26, nothing very different and the date is very different, but when you look at the output, uh what do you notice? When you look over here in this first case uh versus

the second case, uh what are we seeing as uh as a difference between the two? Uh let's see if we can see that here. Uh ooh, so the tool you're using, uh I don't see that in here. Do you see it? So, um this is the first response, I think. Uh oh, it's right here. And oh, yeah, so that's the point I wanted to make. I want

This is the second response. Is the second response the same as the first response? No, right? So, it's different, as you can see. So, the first response says, "Inspect the release file." I don't uh it's a different wording here. Uh why is that? Because, remember, it's non-deterministic, right? So, it's not going to be consistent. Uh in in you know in this case, if you read this through,

you can verify if it's a good correct information, uh but at least you know it's not the same exact response. So, if you're expecting it to give exactly the same response, you're using the wrong tool. AI is not going to do that for you. It's non-deterministic, right? It's not going to give you the response consistently. But then you can verify its correctness and see if that is

actually working or not as well. So, that's good so far. Now, what we have done in this case is we have used it to get some global information. We're able to customize it using a system prompt. We are also able to customize it even further by using tools as well. But, you probably have to go beyond that at some point. Uh going back to the example of

the travel company that I talked about, they want to be able to provide information to the user about uh travel restrictions, uh change requirements and restrictions, but those are all written and documented within the company. Those details may change over time. And if the rules were to change, they need to use an updated rule. How are we going to do this? So, this is where a rag

really comes in. So, what is rag? So, a rag really stands for Well, it it it stands for uh retrieval augment augmented, uh you know, uh in this case, you're basically saying, "I want to make a query, but the query that I want to really perform is retrieval augmented. But, so but there is a catch here. Uh they should have should not have called it rag, they

should have called it a r g. Why? Because you're augmenting first and then retrieving. But, they found rag is a word already and they can hijack it. So, you're actually you cannot retrieve until you augment it, right? So, essentially, you are going to augment. So, what does augmenting really mean? Augmenting simply means you're going to perform a query, a chat, a query on the LLM, but you

want the LLM to use not just the public information, but it we want it to use your ingested or augmented information. So, the first thing you do is you ingest or you augment. Now, I want you to keep in mind uh the augmentation is completely separate from retrieval. You don't have to augment every time you retrieve. In fact, you shouldn't. Why? If the information is for every

single user, you know, anybody, right, goes to the travel site, they buy a ticket, the rules are the same for everyone. So, you ingest it once and then you retrieve many, many, many times for every single user. So, you will first write the application so that when the application starts up and fires up, you do the ingestion and you will be ready for it. And once you

ingested your document, every request comes in can retrieve with it. Okay, that's good. But then also, there are times you do want to ingest per request as well. Why? Because a user comes in, they're a normal user, I I can use the ingested information, no problem. But she logs in, but but she is a premium customer and and and has status with the travel site. So now

I need to go fetch her specific information. Why? Because she may be at different levels of loyalty. If she is at the top level, maybe the rules are completely relaxed for her. Whereas if I am not at that level, maybe I'm at a lower level, maybe my restrictions are much less than for a normal public, but still very higher than what she may be entitled to. So

this is where you need to ingest customer specific information into it, so AI can make use of it. So then what happens? How do we go proceed doing this? Well, to deal with the data, the data may come from very many different formats. Hey, can the data be a text file? Sure. Can a data be an image? Can a data be PDF files? Can a data be

something from database? Can the data be YAML files and JSON files? Can data be images that correspond to medical record? Can the data be stuff it pulls up about a patient record about various, you know, levels of, you know, vitals that's been gathered over time. It's pretty much anything you want it to be. But the problem is when you have these data, you need a really highly

efficient way to access this data. How are we going to achieve that? Well, the way to achieve this is to take this data in various format and chop them up and store them into what is called a vector store. So a vector store is a very highly efficient storage that can provide high level of indexing. I'm going to be naive when I explain it. If you want

to put data into a hash map, how does a hash map deal with the data? Well, it uses a hash key, and that way it can quickly get the information when it's needed. So, think of a vector storage like a very highly powerful hash map, except it's not hashing, it's indexing and using other technologies to do provide fast access. Now, what kind of vector store can I

use? There are several different vendors that provide vector store. So, depending on what you need, high volume of data, a lot of speed of access, whatever the combination could be. In our case, because this is just a demo, we'll just use an in-memory vector store, but you can use other stores and you can configure them and use whatever is appropriate, right? So, you can use them based

on what your real needs are in your applications. So, how do we go about using this in our application? Oh, sorry, before we go further. So, the first thing is you're going to take this data, your your local data, let's call it private data rather than public data. You take your private data and you store them into a vector store. And when you store them into a

vector store, you are going to do two things. You need the vector store, you also need a model LLM to do it. So, these are called as local models, embedded models. So, these embedded models know how to chop this data, we call them segmentation. So, you segment the data and then you store them into the repository, that's your ingestion phase, your augmentation phase. The next phase here

is retrieval. So, in retrieval, what's going to happen is when a query comes in, let's step back to what we did when a query comes in, you take the query and you send it to the LLM. But the LLM now is going to respond with what it knows from the public. But this is not about public, this is about a private data. So, what's going to happen

is imagine like a filter, your local vector store and your local embedded LLM is sitting here between you and the real LLM that's out there. So, when a request comes in, your local embedded LLM says, "Hang on a second. I'm going to take this query, and based on the query, I'm going to perform a search on my local vector store. Pull the data from the local vector

store, and send that data along with the query to the LLM." So, now the LLM has been augmented with the local data for the retrieval. So, when the request goes to LLM, LLM doesn't say, "Gosh, you gave me a query. Let me go find out." LLM says, "Wow, you gave me a query along with the data set I should augment with, so my response is based on

what I know from the public, but I'll give you priority for the local data that you sent me as well." So, that is essentially the idea. But, having said that, there are a few things you need to be very aware of. There are three problems I want you to remember. The first problem is, depending on how you segment, the quality of the output may be very different.

That's one of the problems you have to deal with. The second problem is that when it performs the query, the LLM is going to use the data to perform the query, and it can be very inconsistent with the response, meaning it could be wrong. So, just because it tells you something, it doesn't mean it is correct, right? That is one of the problems you have to know

about using AI. It may not be correct. You cannot say, "Gosh, LLM said this, I'm going to just use it." So, this is why when you go to websites these days, they do this shameful thing. The response from this may not be right. And this is This is beautiful, right? So, these days when I go to work, I tell my boss, "Anything I do may not really

be right." So, uh thank you for hiring me, right? So, so that So, the whole point The third problem is it might outright give you give you totally fictitious information suddenly. It's not only is it wrong, it is completely irrelevant, right? And when it happens, tough luck. So, this is where you may want to really run it a couple of times and see if it's consistent. You

may even use the different models, and that way you can see if there is a lot of deviation, and if it's more convergent, you can say, "Okay, maybe this is a little bit more closer to what we can expect." You need to be able to say that as well. So, what are we going to do here to be able to use something like that? So, what I'm

going to do here is let's go back to this controller here, and and let's let's just, you know, ignore that for a minute. What I would do here is to be able to bring in a controller that we can use for our our purpose. So, what is the controller we want to use in here? So, let's create a controller. This is going to be giving us the

ingested I need to do some local information. Well, it just turns out I just ran a conference on AI literally last week. you know, as as a conference organizer, I've got a little bit more luxury than attendee as an attendee or a speaker, is I have the entire view of submissions on my side. So, I ran a query, and I dumped all the details into two files.

So, I'll show you the first file that I'm going to use over here. So, my very first file that I'm going to use here is the, you know, Arc of AI is the name of the conference I ran, and and here is the Arc of AI speakers. So, you can see a speaker name, and and then the talk title the speakers gave. So, you have, you know,

Mike Amundson and the three talks that Mike gave. Uh, Brian Benz, if you were here yesterday, you probably saw Brian's talk. And and Brian gave a bunch of talks. And then you have all these different speakers that were at the conference and the talks they gave. So, that's one file, the speakers file, right? Then the next file here is literally the title of the talk and the

abstract. No mention of the names at all, as you can see. So, if you only look at this file, all you get is the title of the talk and then and then the abstract that you see in here. So, that's basically the file that I have that contains all this information. But what are we going to do with that wealth of information that we have on our

hands? Well, the first thing I'm going to do here is to say, uh, conference controller, and I'm going to create a controller just like we did previously. So, this is a rest controller that we created earlier, and a rest mapper I want to create. And this is going to say, uh, a slash, let's say, uh, conference, and I'm going to get a conference as a URI that

I'm going to be serving the request in. What am I going to do within this is the question. This is again, right? Your regular, uh, spring stuff, nothing really exciting. Uh, private, uh, conference service, uh, and the conference service is conference service dependency injection. So, conference controller takes a takes a conference service, as you can see. And the conference service, I'm going to simply say, this.conference service

is equal to conference service, right? So, we do the DI, dependency injection of that in here. Well, the next thing here is the request mapping, and this is going to be for, uh, a query I'm going to perform on the conference. So, this is going to be a public Pardon me, public string, and this is going to be, uh, you know, let's call it a run query.

You can call it whatever you want to call it. And this is going to be a request request body, if you will. And And what is going to be the request body? It's a string and it's a query that's going to come in. So, I'm going to take this as a post to request that is going to come in that we are going to be using. So,

as a result, I'm going to change this to a post to mapping, as you would expect, right? I'm going to get a post data and I'm going to execute it. So, within this, I'm going to say return a conference service start a query and pass the query to it. That's all my controller did. Nothing really exciting, right? And this call the query and send the query over.

Well, the last step here is to do the rag itself. How do we go about doing it? So, here is my conference service I'm going to create. And the conference service, oops, let me put that back again. So, the conference service I'm going to do in here is going to be performing the query and bringing the response from AI that's what we want to do at this

point. So, we're going to create a service, but remember we have a chat that we were talking to and the chat client is still what we need to use. So, it's everything we did before plus a little bit more that we're going to bring in. So, what do we do at this particular point? So, I'm going to go to the conference service and the conference service is

going to be a service we're going to be defining and that is the service we are using right in here as well and saying I'm going to define this conference service that I'm going to bring in on the other side and I'm going to be able to use, right? So, that's a service is going to pull in. We need to implement that query method, we'll come to

that in just a few minutes and implement it. So, here is a service. What are the things my service needs? A private chat client, right? That's the first thing I need and this is going to be the chat client that I'm going to pull in. The next thing I need here is a private final and this is going to be a vector store. and what is the

vector store provide? It's just a vector store that's going to keep track of the data we are interested in using. We are going to bring that in here as well for us to use it here, right? So, that's what we're going to pull in. So, we're going to bring in the vector store, that store, that's great. Now, in this case, we need to provide the dependency that

it's going to need. So, I'm going to pull in the dependency here for us to use in just a minute so that we can just get that from the palm file. So, here is the palm file. I'm going to just pull the dependency in here and I'm going to use it. So, that kind of saves a little bit of time as to pulling in the dependency. So,

let me grab that here and that's the dependency we bring in. And so, that's where the vector store is going to come from for us to be able to use in our situation. So, I'm going to just ask it to refresh the Maven dependency. Let's ask it to sync it up. Okay. So, that's our vector store as you can see. Great. So, then what am I going

to do? We have to do the DI, right? So, public, let's say in this case, conference service and that's going to take as an argument a chat client, right? So, the chat client is going to come in, but the chat client.builder, if you recollect, is what we used the last time. So, I'm going to say a builder as you did last time, we'll bring in the builder.

But then we are also going to say that in addition to it, we need a model we need to bring in. So, we're going to say embedding model. So, embedding model right there and we'll say this is my embedding model that I'm going to bring in as a dependency. So, we have a DI. What are we going to do with these two pieces of information? The first

thing is this.vectorstore is equal to and I need to create a vector store, a simple vector store, we're going to bring in. So, simple, let's say vector store and the simple vector store is going to be a builder. So, we'll say builder. And And what are we going to do once we build it? We're going to provide to the builder the embedding model that we injected and

then call the build on it, and we got a vector store that's ready to store the data that we are bringing in. Then we say this. chat client is equal to the chat client builder that we are pulling in, right? So, this is going to be the builder dot. And what are we going to do with the builder? A default. And in this case, it's going to

be default advisors. And what advisor am I bringing? The advisor that I'm going to bring in is the question answer advisor. So, let's try this here. And so, question question uh answer advisor. So, we are pulling that in. And And again, you will see this pattern of using build over and over and over, right? So, this is going to be the builder, and this is going to

take the vector store as an argument dot in this case. Once we provide the vector store to this, we're going to ask it to build. So, that's the last step we're going to perform here on the vector store itself. So, question answer advisor and the question answer advisor dot build vector store and we do the build, and that's a middle step, and then the last step here

is a build on the entire thing to get the response back from it. Let's see what this is complaining about. So, the question answer advisor, it says in this case method call expected in the default advisors. So, this is a builder and a vector store. Let's see where it's expecting the method call. Uh this is saying that insert new, but I don't want to use a new,

right? I'm using a builder for this. So, this is going to be the builder embedding model dot build. So, we have bought the vector store. This is the client builder default advisor and we are asking it to build vector store.build. So, that one is correct as well and this is the question answer advisor.build. Ah, sorry. Uh looking at the code, it's a lower case builder not upper

case builder. There we go. That's why it said method call expected. All right. So, we got that one part. It's ready. Hey, but what about the ingestion of the data, right? We need to ingest the data for it to be able to use. So, we can use a post constructor and and in this case a post construct says, "I'm going to load my data." So, the load

data throws an IO exception of sort if something is fail is failing. So, load speakers and load, let's say, abstracts, right? So, we'll ask it to load speakers and to load abstracts. So, what does load speakers do? That's a private method that I'm going to create within here. What does this method do? I'm going to say documents, let's say, documents is equal to new This is going

to be bringing in the text. So, we'll say text reader and the text reader is going to receive a new class path resource. Remember where it was located? It was in the resources directory if you remember, right? So, we're going to specify from the resource directory archive AI _speakers.txt and we're going to bring that file over and do a get on it to get the documents. Once

I get the documents, it is time to load that into the vector store. So, token in this case text is splitter and we'll call it as a token text splitter is equal to and this one says it's a splitter and again the builder. So, token splitter.builder.build and then we say vector store dot add. And what am I adding to this vector store? The speakers document that I'm

going to add at this point. So, this is going to be the speakers document that we're going to provide. Pardon me. This is going to be the token that we are bringing in. So, we are saying this is going to be vector store dot add. Token This is going to become a token splitter. Apply and then to that if you notice it's a documents. So, that is

the speakers documents we're going to bring in. In a similar way, I'm just going to copy this for a minute. Here is the load abstracts. And the load abstracts basically says, I am going to load the abstracts document, right? So, abstract abstracts document. And this abstracts document is going to come from arc of AI underscore Let me make sure the file names are right. Arc of AI

underscore abstract and speaker. So, this is going to be This is going to be abstracts, right? So, abstracts we're going to bring that in. So, abstracts Let's go ahead and provide that right there. So, we brought in the abstracts. And what's the next thing we're going to do? We're going to load up the abstracts document right here. So, we're almost there. We have done the augmentation that

we need to do. What is the last step here? The last step is to pull the data that we want to use and provide the response, isn't it? So, for that, I'm going to simply go back here and implement this query method finally in here. So, what does a query method do? It's going to use the augmented data from the vector store that we created. So, return.

And in this case, the chat client dot prompt. But before we go further, I want you to quickly notice what we did. We already connected the vector store to the client already. So, it's waiting there. Think of it like a bucket hanging, right? So, this is kind of like this is the LLM over there. This is the query endpoint, but you stuck this vector store right here.

So, when you make the query, it funnels the data from the store, pulls it in, and sends it, right? That's That's why it's hanging off of that already sitting there and and ready to go. So, that that vector store is connected right there on line number 25, that vector store is ready. So, now you go to the prompt and say, "Hey, I'm going to perform the prompt

on this. What am I going to ask this? Uh I'm going to say here's the query I'm going to ask and make the call and return the content." Notice this part is very much the same as what we did earlier. The only thing is before we get to this point, we have augmented it with the vector store, and we injected that into the vector store as well,

as you can see. So, that's pretty much the augmentation. So, so essentially, once we have done it, the last step is to execute this and see if that works. Now, this is where assuming I wrote all of that right, it should pull things in and make use of it, right? That it's going to bring in the data and be able to use it. So, in this case,

you can see it's embedding the data by pulling the data from the files, and it is segmenting them and storing it. Like I said, this is something you'll do once, and then once you are done with it, you can make repeated calls using it, right? You don't have to run this through this many, many, many times. But but while it's doing it, let's go and modify our

our run.sh really quickly. So, I'm going to use a curl request, but what am I going to send as a curl request in in here? So, the request is going to change a little bit, right? So, the request is going to this time be the following. I'm going to make a call to the conference query, but I'm going to send a text here as part of that

request and say, "Hey, give me this information about this conference, will you?" So, let's try this. Run.sh and I'm going to say, "How many speakers uh speakers were at the uh event, right?" And we're going to take one more minute to run this and get the information, and we'll be done. So, this says, "How many speakers were at the event?" It's going to take this data local

and query that and send it off, and it's going to tell you the information. Uh before it prints out, I'll tell you, there were 37 speakers. And well, it says 37 speakers. Don't it. I wanted to say it because it's never given me the right number. And and when you want to prove it is when it says, "Aha! And it I was waiting for you, right?" I'm

going to just make a query one more time just to illustrate it. There were 37 speakers, and it said 37. You can take a photo of that because you caught it telling the truth, right? You know, people There's a saying, right? I caught him telling the lies. With AI, it's the opposite. I caught it telling the truth. So, we caught it. This This is lucky day. It's

telling me This This is just like people who lie because when you put them on the spot, they're like, "No, I was always truthful, right?" So, that's basically what it is. I'll run one more query before we get this done. I'm I'm maybe two queries to before we get this done. So, what's the next query I want to run? The next query I want to run here

is uh what did Let's say, "What uh what did Pardon me. What did Simon uh talk about?" So, I'm going to ask what did Simon talk about? It is going to figure out that Simon is a speaker, and it says, "I can't tell I can't tell the provided program uh ex- ex- excerpt don't mention uh that actually is lie." So, there is a Simon, by the way,

and and it just did not know to pull it out. So, that's a flaw in the execution and it it failed at this particular point. So, this is where we might we have to go back and re-examine what it was you know ingesting and making use of as well. But, hopefully that gives you an idea. I can't perform more queries on it because it's not able to

get the abstracts and the speakers right. Maybe we'll try just one more and it turns out I had we had actually many many Brians at the event. So, what did Brian talk about? Maybe we'll just run this and call call it done at the end of this one. And there were three Brians, no maybe four Brians who are speakers. Yeah, that one is messed up. So, we

need to debug that. So, we went through this journey of where what the use case where we would use it, how we would benefit, and then we saw a few examples getting the public system prompting it, using specific tooling information, and then finally augmenting and then retrieving and generating the response based on it. And that's basically power of Spring AI. Hope that was useful. Thank you.