Building AI Agents with Spring & MCP - James Ward, Varsha Das

This talk focuses on building and architecting AI agents using Java and the Spring framework, specifically addressing the integration of machine learning models into enterprise systems. The speakers discuss the distinction between model-building, typically handled by machine learning practitioners, and integrating these models into existing enterprise applications. They detail the challenges and solutions surrounding AI adoption, including the importance of connecting large language models with external data through tool calling and the use of managed memory systems for enhanced AI agent performance. The session provides a hands-on approach with code demos that illustrate how to implement various AI functionalities, such as structured outputs, streaming responses, and advanced memory handling techniques. The speakers emphasize the role Java developers play in driving AI initiatives within enterprise environments by leveraging existing skills and technologies.

[music] >> We're going to be going through a lot of code today, but want to do some kind of introductions background so that everyone gets up to speed on all the AI stuff. Great. Yeah. And Versia, you want to say hi? Yeah. Thanks. Thanks, James, and thanks for the wonderful intro. Uh good morning, everyone. Uh so happy to be here and sharing the stage with James for

the first time. So, uh a quick pulse check. Uh how many developers and builders do we have in the room? Raise your hands. Okay. Wow. And um how many Java and Spring developers do we have in the room? Great. Awesome. >> Nice. Now, um now we are going to talk about AI agents, how you are going to build, how you are going to architect the AI agents

with Spring and MCP. Uh just wanted to understand how many of you have already started building AI agents with Java or Spring already. Raise your >> Okay. >> Awesome. You all are in the right place to learn how to get up to speed. So. Exactly. I mean, that is what uh we and me and James are here for today. most of the conversations that today we are

having around AI often mixes two very different things. And like on one hand, on one hand, we have the thing around building models, fine-tuning models, and training models. Uh so, as you can see on the image over here on one extreme end of the spectrum, we have the AI ML specialist who are like the ML practitioner and data scientist who have been in this building models, And

on the other hand, we have this particular logic integration or enterprise AI adoption where as Java Spring developers or as back-end developers, that is where we have a role to play. So, Java developers have always been the backbone of business application integration. So, as you can see on the image, at the center, anything related to integration is where Java developers have always played a role. Now, like

I said, there are two different things. On one hand, we have building the models. On one hand, we have the problem around integrating the models into enterprise systems which has been running for decades. And that is where the role we have as Java developers to play because, you know, like building models is something which is best kept to the ML practitioners who have been doing this again.

That is where Python has always played a role. And I have been interacting with a lot of Java developers who are also trying to navigate the Java and AI space and who always have this question that do I need to unlearn Java and do I need to relearn Python? So, that is why it is very important for us as Java developers, Spring developers to understand that there

are two very different problems that we are solving. And of course, if you are interested in about building models and fine-tuning them, there is another pathway for us. But, the goal the agenda for today's session, why me and James are here today, is we are going to talk about the second part of the problem that now if you have been part of the enterprise systems that has

been running in the production for decades, how do you lead AI adoption over there? How do you build AI agents which are going to be production-ready? And how do you more importantly architect the AI agents in the right way that it needs to be, again, as Java developers, I believe that moving from POCs or like from prototype to production, that entire arc or that entire spectrum, we

have that like that has been our bread and butter for like decades. So, now it is even more important in this day and age when we are talking about AI to actually move from POC to production and do it in the right way so that we can lead the future of AI uh with Java and Spring with all the natural expertise that we already have. Now, now

that we understand where we are today as Java developers and what path do we have in front of us, it is also important that we also take a step back that all of this uh all of this conversation which started essentially with generative AI and LLMs, right? And the reason why we are now talking about AI agents, it's very important to understand again the arc that back

in the day, '60s and '70s, we were talking about artificial intelligence, AI/ML, and then we were talking about deep learning. And then we started to talk about generative AI, then LLMs, and now AI agents and agentic systems. So, this entire arc essentially is about that how we have transitioned from talking about GenAI or generative AI, then large language models, and now going into AI agents, which are

nothing but autonomous or semi-autonomous systems who can reason, who can plan, and they can act. So, now the models, the LLM models, essentially have certain limitations of their own. As you can see on the screen, right from not having the state, the context, losing the conversation context, not having access to real-time data, and again, all the building blocks that we need to solve this particular problems is

what we are going to cover one by one today. And I like James also mentioned that we have quite a few code demos so that it is not just about theory. We are going to talk about the concept. We are also going to show that how you can implement the same when you are building the agents. What are the code changes you need to make for that?

So we are going to walk over that. So this is how I believe we have the right foundation now that where we are standing today and I hope that from here I'm going to hand it over to James who's going to talk us fundamentally from the problem that we also know that if LLMs do not have access to external tools or external data, how do we go

about that? So I'm going to hand it over to James who's going to take us through all the limitations and then how we are actually using different patterns and building blocks to solve those particular limitations. Over to you, James. Okay. Great. So thanks, Varsha. Okay, so if this has anybody ever tried to ask an what the weather is? It generally can't answer that question because it's just

a model. It doesn't have access to anything external. And so if you do ask it this, it's going to tell you to look outside your window. And so this illustrates a problem that we have to solve with LLMs is these things are most useful when they are connected to our external systems, our external business systems and business data and business processes. But how do we actually do

that? So first, what an LLM is to me, this is kind of the way I think about it, is it's just this like knowledgeable translator. That's like all that this model is. Is you can give it natural language and it can give you natural language back. Or can give you an image. Or you can have it describe what's in an image. Or you can go from natural

language to structured data. Or you can go from natural language to code. Or code to natural language. So, it's just as this na- this knowledgeable translator. It has a lot of knowledge about the whole world, but it only has what it was trained on. So, how do we actually do something useful with an LLM? Sure, it can write good poetry, but generally what we want to use

it for is to accomplish something. And we want it to accomplish something, we go from that LLM to an agent. And really what an agent is distilled down is we take an LLM and then we take uh some some external tools and we put them into a loop and we call the LLM, we call the tools, and we do that repeatedly until the agent the LLM has

decided that the goal has been accomplished or maybe reached a point where it couldn't accomplish the the goal that was assigned to it. So, this is combining the LLM with the environment. So, okay. Let's take a typical example of this where let's say we have a travel and expense AI agent. We may ask it questions about our policies, we may have it book some travel for us.

To do all those things, we're going to need the LLM and then we're going to need connectivity to our business systems. And this would require things like, "Oh, let's get the current date. Let's find flights by route." There's going to be a lot of different things that we're going to need to pull together here to make all this work. So, in the world of LLMs, when you

send something to the LLM model and want to get a response back, the word for that is inference. And when we do inference, we can do it in a lot of different modalities. We can send text, get text back, we can get JSON data back that we convert to objects, we can get streaming data back, uh we can combine multiple text blocks together, we can combine text

text blocks with image blocks. Uh so, lots of different ways that we can use this inference API of the model. And so, I want to walk through with some code in Spring AI uh just the basic inference, so everyone can see uh how how just fundamentally the the actual model works before we get into agent architecture stuff. Let's go over into some code. And I'm going to

start here with my Spring application. Uh these are all just going to be command-line applications. So, I've got my command-line runner. And then, what I do is I inject into here a chat client builder. And the reason why we inject a builder, not an actual chat client in this case, is that this chat client builder gives us the opportunity to do a bunch of customizations to how

we're going to interact with the LLM and how we ultimately build our agent. So, um so, we inject that builder. And then, also just for the purpose of this demo, I'm also injecting some stuff for logging so we can see what's actually going on. And then, what we do is we take our builder. And in this case, I need to wire in my logging advisors. Advisors in

Spring AI are like uh interceptors or like middleware, so it allows us to intercept things before they go to the LLM and intercept the responses that come back out. So, that's why I'm adding in my logging advisor. And then, I'm building my chat client. So, chat client in Spring AI is the primary interface that we use for inference to talk to that model. Okay. Then, the API

for the chat client is we're going to create a prompt. So, often times the language uh around inference is described all right, a user is sending a prompt uh to the LLM and then we're getting a message back. And so, my prompt from the user is just say hello. I call that and then I do dot content to get the response and then I'm going to print

it. Now, we need to to be able to do inference, we need an AI model. We need an LLM. And so, Spring AI has support for a lot of different ones. What I've configured for this one is the Amazon Bedrock Nova model. Uh Bedrock is a service from AWS which has hundreds of different models. They are priced on demand Uh and so that means I can just

pay for the tokens that I use and different token prices depending on which model you're actually using. But so I've set up Bedrock with Spring AI and selected my model there. I've also increased the forget what the default max tokens is but I needed to bump it for some of these demos and then we'll get into some of those other settings in a little bit. Okay, so

that's my very basic inference API call and let's run this thing and just make sure that that it all works. So when I run this we'll see the output here in the bottom. And so now it's making that call to and then getting the response the inference response back and so great. I said say hello and now it is responding with hello. It's nice to meet you

and so that was our basic inference call. Anything to add or clarify on that? Anything I missed? No, I think you got Yeah, I think we have pretty much covered. I think the important thing is when we are using the chat client builder and one important thing which you also mentioned is around the usage of the builder design pattern right and another important distinction is to understand

the difference between chat client and chat model. Chat client is the top level top level abstraction that we have and chat model is the lower level building block. So it is very important for us that chat client is giving us the interface into the LLM. However, if you want to do some kind of a lower level changes then I believe you have to play around with the

chat model. So it is for the sake of this demo we have shown the chat client or builder but don't forget to check out the classes of the chat client and the chat model to understand how it how it works under the hood because again as Java developers I believe that we don't just work with abstractions. We love to understand how the abstractions work underneath, yeah. Yeah,

I think that's a good point is that Spring AI is built obviously on Spring, but uses all the typical paradigms that you're familiar with in Spring. So, we have different layers of abstraction. We even have higher-level layers of abstraction over chat client. So, that allows us to be able to customize and configure things to your needs if you need to. But, of course, the defaults make make

the easy things easy. So, okay. Let's go on to the next demo here where I over back in my chat client I want to show structured output. So, one of the things that we can do with our LLMs is not just talk text in and out, but we can use data and do structured data as well. So, let's say that I have a record called city and

it's got a name, population, and then I've got a food property. But, you'll see I've annotated that one with this JSON property description, which is most popular food. And the reason why I've done that is cuz I'm going to ask the LLM to give me some cities and their most popular foods. And so, well, actually, I just say in my prompt get 10 big cities. But, when

I tell it, all right, I want to encode the response into a list and the syntax here is a little bit weird because we have a generic parameterized list. And so, we do need to do this parameterized type reference to get our list of cities. But, essentially, this is saying, "Hey, LLM, when you respond, don't respond with just plain text. Respond with JSON data that fits this

particular schema, in my case a list of cities." And so, that's how we can get actual structured data back out from the LLM. And then I just iterate through. So, let's run that one and make sure that that one works. And when I do that yeah, let's run structured output. When I do that, I'm going to run this one this particular one in debug mode so that

we can see a little more information for what's going on underneath the [snorts] covers. So, now making the call to the LLM and here we see, you know, this much larger response coming coming out. So, that's that's my request side where I'm telling it get get the cities. And then we get the response back out and there we can see in the the actual text that came

back from the LLM. It was encoded into JSON. What we're not seeing is there was another message that was hidden kind of deeper in my my advisor chain where because I told it to use structured responses, I told it to give me data back. There was actually an additional prompt that went to the LLM as well that said, "Hey, when you when you respond, do so with

this schema and give me back JSON data." And so, there's also another way to do structured responses with some models. They actually support it natively and so part of your API call, your inference API call is going to include the the schema that you want back. Um, but in this case I'm just getting that JSON back and then Spring AI takes care of mapping that data to

the objects, so deserializing that. And now we can see the different cities. Let's see, Mumbai likes curry. I think they did a good job, right? So, anyways, that's our structured output example. Okay, moving on to the next one here. So, this is just basic inference API stuff. It is streaming. So, sometimes in your LLMs you want that streaming experience because LLMs are doing a lot of computation

and so you may want to actually stream the response back out of it. And so, in that case what we do is when we call the LLM instead of doing the you'll remember before we did it was dot call. So, instead of dot call which is just wait until the response is completed, we can say dot stream and then dot content. And now in the case of

Spring AI is going to give us back a flux, which that flux is the the reactive stream way to be able to get this response back. So, let's run that streaming one so we can see what that looks like. Stop. I forget what my prompt was for the It was something about like write write me a story about Java or something like that. So, this is you

know, a bit longer of a response, but we can see it's now streaming the response from the LLM back up. So, now very easy with Spring AI to do streaming when you want Okay. Let's come up with a story about a young programmer named Alex. Very exciting. Okay. So, that's our streaming example. Now, let's go on to the next one here, which is system prompts. System prompts

are a way to kind of guide the LLM in a particular direction or do what's called like grounding. And so, with system prompts, we can we we can say like here's who you are. You know, you can give your agent a persona. And so, in this case my system prompt is you're a Wookie. Everybody know Wookies from Star Wars. And then I tell my actual prompt is

say hello. So, when we run the system prompt one, that's going to hopefully say hello, but in the form of you know, like Chewbacca as Chewbacca would. So, let's run that one. And growls softly and wraps you in a gentle hug. Rawr. Hello. So, may the force be with you, of course. So, so that's a general feature that we use when we're building our agents is system

Okay. Moving on to let's see what we got next here. Okay. So, now so far we've only done just basic inference. We have only used just the model. But as Varsha set us up with, we need to to do something really useful. We need to connect these systems out to our business systems, our business data. And so I'm going to show you what happens if we try

to ask the model what time is it and if we run that no tools actually hit button we run this one like the example with the weather. It's going to be like I'm just a model. I don't know what time it is and yep. I don't have access to a clock. So this is just indicating like like the model has no access to the outside world. And

so the next set of things that we need to do is be able to connect this model to the outside world and we're going to go through some patterns for how to do that. Anything to add on that so far? Yeah, just to add to what James has said. So far we have covered the solution to the first limitation that we had that to align the responses

to the goals as well as to map the LLM text to the pojos we are using system prompt and the structured outputs and that solves the basic problem that we had the first limitation. Now moving on to the tool calling and I also just want to add to this is that the limitations that I shared in the first slide. It is very important for you to try

to correlate what limitation that we were showing and then how each of them is being solved by what James is going to talk about and in between we are going to do like short recaps of what we have tried to cover so far. Yeah, over to you James. Okay. Okay, so let's talk about tool calling. a couple of years ago people you know were doing these LLM

models and it's like okay, we need to give these things access to the outside world. How do we actually do that? A model is just a model doesn't have access to the outside world. So what they came up with and what's generally supported across most major models now is called tool calling. And I want to walk you through the actual protocol that tool calling uses because it's

important to understand like you we're going to get to the code that's like very makes it all very easy, but it's helpful to understand what's actually happening with the LLM when you do tool calling. So, here's how it works. Let's say I ask what's the weather in Berlin, I send that over to the LLM. When I send that over to the LLM, I'm also going to put

some information in that message to the LLM that says, I've got some tools available for you to use. One of those is called get weather and it takes a parameter for a city or something. And so, uh so we pass that information when we make the call. So, it's going to have my prompt, uh what's the weather in Berlin, but then it's also going to have the

list of tools that are available. So, then what the LLM does, the LLM is, you know, doing its probabilistic, you know, matching on things and it is decided that, okay, I need to get the weather. But, how am I going to do that? I don't have access to the outside world. So, the LLM decides that it is going to respond back to the caller, back to my

Spring AI side of things and then it's going to say, "Hey, you need to get the weather for me. I can't do that, but you can." And so, Spring AI side, you go get the weather for me. And so, that there's a particular protocol in the actual inference API that indicates, all right, I need you to do a tool use call. It's called stop reason tool use.

And then it tells it responds with, here's the tool that you told me was available and here's the parameter that uh that is required to call that tool. And so, your tools can have, you know, multiple parameters if they want. So, then on the Spring AI side, I actually have access to a weather service. I can now call the get weather service with the Berlin parameter and

then I take the result that came back from getting the weather and I then call the LLM again and I include the initial prompt, which was get the weather, I include the list of tools and I include the result that I got from actually calling the tool that the LLM told me to call. So, that goes back to the LLM, and then the LLM can then generate

a response with all of that information, and can usually do like some summarization or whatever you want it to do. But, that's how the actual tool calling protocol works. So, let's go see what this looks like in code, and we'll see how this all kind of comes together. So, I've got a few other demos here. Um the first one I want to do is first we'll define

a tool. Uh the one way to do tool calling is to write in my Spring AI application, I'm going to annotate a method with @Tool, and I give it a description. And it's very important to have a good description because that's how the LLM is going to decide when it should use this tool. And so, we give it the description, then in this case my uh my

actual tool is just get the current date and time, very simple version there. So, great. So, I've got my tool, it's in my Spring AI application, but now I need to provide that to the LLM. I need to tell the LLM that that tool is available for use. So, the way that I do that is when I create my chat client from the chat client builder, I

can set default tools. You can also set tools at the time of calling the LLM, uh but I set my default tool. So, this is just saying, "Here's the tools that you have available." And then, I do my prompt, "What What time is it?" And let's go run that one and make sure that that thing works. So, basic tools. And I'm going to run this one in

debug mode so we can actually see the inference protocol that's happening. Okay. So, here we go. So, I've got my message here, let's scroll up. Got my request message, so "What time is it?" Uh we've it's a user message, we can see the model that we're using, and then we can see that there's also some tools that are included in this request to the LLM. So, now

the response that comes back from the LLM is finish reason tool use and it now is telling us, "Hey, I need to determine what time it is, so I'm going to use the get." So, this is like a thinking block. Generally, these don't get displayed to the user. You can if you want. But, so that's that's the thinking call, the thinking block. And we see down here

that the actual in the actual response from the LLM it's saying, "Hey, I need you to call this tool for me." And the tool that I need you to call is get current date time. Okay. And then I didn't have any arguments for that one. Okay. So, that goes back to the Spring AI side. The Spring AI side calls my tool and then we send another message.

So, this is a second message to the LLM all within that one call that we made on the Spring AI side. And now I'm saying, "Hey, I called that tool for you. Here's what the tool was and then here is the output of that tool." And so, it just puts in, you know, whatever you returned, which could be structured objects, whatever serialized into JSON. And then the

LLM looks at all of the information that it has and it ultimately comes up with a response that says, "The current time in Denver is that." So, okay. So, that's our kind of full loop. Real quick back to the code, you'll see that I in this one this one call content, there was actually that while loop that I talked about back at the beginning, there was this

while loop that was continually calling the LLM until the LLM decided that it was done with what it was assigned to do, which was to get the current time. So, that while loop is built into the Spring AI chat client. Okay. So, we've got our basic tools and that all works. Anything to add on that, Varsha? Yeah, so I think tool calling is actually solving this problem

around getting giving the real-time access to the data externally. I think it will get very interesting James when we are going to talk about the slide that then what to do when we have too many tools. Yeah, that is definitely a challenge we'll get into. Okay, before we go there, let's talk about MCP. So MCP an open standard you may have used it with your AI code

assistant, but MCP takes that tool calling and allows you to do it to an external server. So it's a client server architecture where your client is your agent, your Spring AI application, and then you have one or more MCP servers that then provide a way to do that tool calling, but do it on a server instead of being in line like the example that I showed you.

So MCP uses the JSON RPC protocol and defines just a standard way to say here's my tools, here's their parameters, and then here's potentially their output schemas, and then there's a bunch of other features of MCP. If you want to go into all of those, I have another talk tomorrow where we'll dive deeper into the actual MCP protocol. But so MCP, we we have this great standard

that allows us to have our tools be remote from our actual agents. Kind of like a microservices architecture, but specific to integrations with agents. So let's look at an example of MCP in Spring AI. So first I want to start with my application properties file. So you can you can in properties just define endpoints for your MCP servers that you want to connect the agent to. So

I maintain an MCP server for Java Docs. It just does Java Docs through MCP, and that has a URL and it's just publicly accessible, so you don't need it we don't need to do any of the authentication stuff. But so I can point Spring AI at that, and then I need to wire that into my chat client. So to wire this all into my chat client, what

I need to do is I need to inject into here, scroll over to the right. There's a thing called a tool callback provider. And the tool callback provider is what is It's kind of a wrapper for my MCP servers that I'm connecting to. And then, what I need to do is give that tool callback provider to the chat client builder, so that then my chat client, my

LLM, is then wired to the tools that are on my MCP servers. So, that's all I had to do to wire those two things together. Now, I can ask questions uh to my MCP server, where I'm going to say, "What is the latest version of the Webjars Locator Light library?" And that maybe the model could try to hallucinate something, but it doesn't know what the latest version

is. You know, it's train It was trained on data from months ago. And so, the LLM will likely decide, "Okay, I don't have really good access to the latest version of that library, but I have a tool that can tell me what the latest version is." And so, then it's just tool calling, but that tool call, instead of being local in process like the first one I

showed, it will then be a remote call to the javadocs.dev MCP server with that JSON-RPC message format all underneath the covers, and then get the response back. So, let's run that one and make sure that that all works. So, MCP tools, we'll hit run. And if everything works as expected, I'll start up that application. It'll call the LLM. Remember, there's going to be two calls here. One

call where I say, "Here's what the my user prompt is. Here's what the tools you have available." Then the Spring AI side is going to decide, "Okay, let's uh we'll then have to perform the tool call." So, that's going to be a remote call to my MCP server. My MCP server response then gets sent back to the LLM, which then gives us the nice summary that we

see there, that the current version is 1.1.3, which is correct. Uh so that's our tool calling with MCP. Anything to add on that, Varsha? Yeah, I think it will get interesting because we have talked about tools and now too many tool calling. Very interesting thing is James also mentioned about the thing around when he was trying to debug. If you would have observed, there was something called

the tool definition. So, every time when we are actually passing a prompt to the LLM, we are also passing the tool definition. Now, in this case, he talked about getting the date and time or getting current weather, which is fine. So, the basic limitations of giving the external access to outside data is being solved with the tools and connecting multiple MCP servers. However, the real engineering challenge

lies that what will happen when we have too many tools or when we are having multiple MCP servers, which are again exposing n number of tools per server. based on the recent conversations I was having with some of the Java developers, there were two major problems which was happening that they were integrating tools and then they were having a lot of fun and then they added more

tools and then more MCP servers to the application to the point which they reached that the agent was now not able to understand which tool to call or what is the correct order in which the tool needs to be called, which was resulting in hallucinated response. That is one problem. And the second big problem which will affect most of the enterprise systems is going to be wasted

resources like unnecessary token consumption because every time if you are passing, let's say, five MCP servers have 10 tool calls each, then you are passing the tool definitions of all those 50 tools to the entire LLM, like James was showing in the debug output that every time you are passing all the tools that is available. Another quick way to test that out would be just to add

that you can use MCP Inspector when you are doing the local development so that you can actually check what are the tool definitions and how all the tool definitions are getting passed. So, now coming back to the problem that if we have too many tools and too many tool definitions, then what do we do? So, three different approaches that we are going to talk about. So, one

is the semantic search. So, over here the tool calling that is going to happen that is going to happen based on the meaning of that particular tool. So, instead of passing let's say all the 10 tool calls, I mean the 10 tool definitions, we are going to drill down to only two of them. So, James is going to show a demo from the Spring AI community. We

have something called tool search tool. So, we are going to check the demo of that where semantic search is going to be applied when you have multiple tools. So, instead of passing all the 10 tool calls, semantically we are going to match which tool is going to be the most relevant tool according to the meaning of the use case that we are going to use and pass

that particular list of tools. Second is around tool grouping. So, this is very much related to the chunking strategy kind of where we want to group the tools on the basis of the business context or the domain that we have and only send that subset of tools forward. So, that is what which we can achieve with the second approach of tool grouping. And third is very interesting

which is around the Imbue folding groups because it is based on a concept which we all know which is called lazy loading or it is known as progressive disclosure of the tools where instead of exposing all the underlying tools, the agent is going to check like progressively it is going to check underneath which tool calls are going to be relevant and accordingly Imbue is going to help

with that particular tools. I would love for James because he has been closely working with working with and around the Imbue. So, if you can add some more around the Imbue folding groups. Yeah, so just as a little background, Imbibe is a agent framework that is a higher-level abstraction than Spring AI. Spring AI provides a lot of the basic building blocks that we can, you know, compose

together and use in a typical Spring way. Whereas, Imbibe is created by Rod Johnson, the creator of Spring. He has created this agent framework. And so, it gives you some higher-level that are a bit different than than the Spring AI, but it's actually built on top of Spring AI, so you can use it alongside all the Spring AI stuff, as well. So, yeah, so the Unfolding Tools

is one of the innovations that Rod came up with to address the challenges that we have with having so many tools. Um, so just wanted to point that one out cuz I think there there are, you know, a number of different ways that we can solve these problems, and different frameworks take different approaches to how they they do these. Okay, should we see the demo? Yeah, let's

see the demo. Okay, so in as Varsha mentioned, we have in Spring AI this this Spring AI community organization on GitHub, which has a number of different libraries built into it in it that are really kind of extending beyond the basic Spring AI stuff. And the reason why we have the community site is that we on the Spring AI side, things that go into Spring AI officially,

those are typical Spring kind of guarantees around not breaking backwards compatibility. But there's everything's moving pretty fast in the world of AI, and so we wanted to have this space to innovate on things that we weren't quite sure, you know, what the final form would be and not have to get locked into backwards compatibility. So, in that repo, there a library called the Agent Utils in the

Spring AI community. And one of the utils in that Agent Utils library is the Tool Search tool. So, the way the Tool Search tool works is typically when we do semantic search, what we need to do is have some way to match what the user has asked for with the things that are available, in this case tools and tool descriptions. And so, one way to do that

is with a vector store. So, Spring AI has support for like 20 different vector stores. Uh I'm just using a in-memory vector store in this case, but I also often use PG vector. Um lots of different options that you can choose from. The reason why we need a vector store is that we need a place to store an array of numbers for our our vectorized form of

some strings. And the reason why we need these numbers is that we need to create what's called an embedding. The embedding takes a string, turns it into a like like array of numbers that represent that string. And the reason why we put it into this vector is that then we can do a very simple math equation. Uh it's called cosine similarity search is the most typical one,

but we can say, "All right, user prompt was this. It is these vectors of of numbers. And then the descriptions of my tools are, you know, these these arrays. And then I can do this cosine similarity search to find, semantically, which which of my tools, in this case, is most similar to the user prompt." So, that's why we need the vector search. And then what we have

to do is we're going to create our vector tool searcher. There's also a Lucene searcher uh as well if you want to use that for your search. And then I need to create my tool search tool call advisor. This is uh as I said earlier, this like interceptor that's going to intercept my request to the LLM. So, even though by default it's going to send all my

giant list of tools, this is going to intercept that and say, "Hang on, don't send all the list of tools. Just send the tool search tool uh to the LLM." And and so then we don't That's this how we're going to solve that problem with too many tools. Okay. So, then we need to wire our advisors together. I got a little work around for my logging stuff

that I added in. But ultimately, we're adding to our list of advisors our tool search advisor. And then we are again creating our chat client. And in this case, I have this this two mini tools object that I can go open here where you can see I've got a bunch of different tools in this particular one just to show like, "Hey, there's there's a lot of tools

there. What are we going to do about that?" And then I wire in my tool search tool advisor. And then I make a call just like we've seen before. In this case, I'm going to say, "Get me a random string eight characters long." So on the first call to the LM, actually let's just run it and we can walk through what's actually happening here. So tool search

debug, that's the one we want. So on the first call to the LM, what's going to happen is it's going to only send the tool search tool, but it's going to tell the LM in a message in our message to the LM it's going to say, "Hey, if you need some other things, then let me know and I can get them for you." And so on startup,

we'll see that we had to do a few things here which I'll explain. So we needed to one of the the tricky parts of semantic search for tools is that we have to create those embeddings, the vectors for our list of tools. And so we that happened when I when I wired everything together that advisor went out and it knew that I I have what's called an

embedding model. So in typical LM AI world, we have our chat models, we can have image models and audio models, but then we also have special models just for getting to those vectors. So they're still large language models, but they're really specific to give me a string, get an array of numbers back that represent that string. And so I've configured my embedding model to use here in

my properties. what if I get to it. Here in my application properties, we can see that I've told it that the embedding model is the Bedrock Titan model. So, that's I I do need a large language model, so that's how I define that. And so, now uh that I've got my embedding model defined, I can now on in this case in startup, get my list of tools,

go generate the embeddings for all of the tools. And so, now we can do that semantic search across my tools. So, that all happened underneath the covers with that tool search tool advisor. But, I did have to generate my embeddings up front for each tool. Okay, now uh you'll see the actual prompt that's being sent. So, this was in added to my prompt by that advisor. So,

it did that intercepting my actual like I didn't put that into my prompt, the advisor put it in. So, it's saying, "Hey, you have access to a special tool that's the tool search tool." And you know, it says, "Blah blah here's how to use it." And so, here's my actual user prompt as well. And then we can see that that goes over to the LLM. You'll see

the list in the list of tools, just the tool search tool. And then we go down. So, you'll see that the LLM is now, "Okay, I need to generate a random number random string for of eight characters." And so, it responds and says, "Oh, it looks like there may be a tool." So, what actually happened was the the user prompt is being used to match to the

actual tools that are available. And so, it responds with tool Says, "All right, let's call the tool search tool. The query that we are passing to that is generate random string." So, that's the one this string here generate random string also gets turned into a vector. And then the tool search tool is doing that vector search in my vector store for the tool that most closely matches

it. And I think there's a default of like eight tools or something like that. You can change the defaults on how many tools actually will get listed. But then, we see, "All right, now we're going to make another request back to the LLM and this time the the what we're sending to the LLM is here's the tools that semantically match what the user is trying to do

and you'll see that one of those, the first one, the one that matched most most best is So it now tells the the LLM, great, you have access to you still have access to the tool search tool, but now you also have access to these other tools. So it included a bunch there. It should probably like tell it to just return the top one or something like

that. But then the LLM now is saying, okay, great. I've got access to this this tool that can generate a random string. So now it's going to respond again with another tool use. So we're in that while loop continuing to call our tools, call our LLM. And in this time the tool use is that random string tool. So now we can actually make the tool call to

random string and then finally we're going to be able to get the result back. Actually, there's the parameters and then here's the result from calling that and we so we send that back to the LLM the LLM responds and says, here's your random string. Great. So that was three calls I think to the LLM using the tool search tool. So we have some additional latency that's included

by doing, you know, more back and forth. But at the same time we were able to solve the problems that Varsha was telling us about with having too many tools. So just and that's just one approach that we can use for dealing with too many tools. Okay, was that How did that go, Varsha? That was a lot. >> is a very important and a very game changing

thing, right? That when the first demo we have seen like too many and actually too many tools being passed to the LLM and from here by doing this particular recursive loop when we have tried to drill down after converting the embeddings we have tried to drill down the that particular semantically relevant tool that we actually need to generate that particular random number. So, that has been like

a huge improvement over time. So, like in one of the slides I remember James, you mentioned that agents are nothing but which are running in a loop. So, I think this particular demo that we have just seen, the tool search tool demo, is a very beautiful example of that that through the while loop which is happening, we are able to recursively loop through it and semantically search

for the most relevant tool that we need and use that one only. Of course, the trade-off is between the latency and between getting the result, but I believe that if it costs additional tokens to get the most accurate result instead of loading or bloating up your context window with too many tools across different MCP servers, I believe tool search tool is the way to go. So, with

that being said, we talked about tools and MCP. I think there was this memory slide. Uh this We do skills first then memory. Yeah, sure. So, I think we can talk about the skills. >> Yeah, so we've got a number of different integration patterns that we can use. We've seen tool calling, that's really the foundation of a lot of our integration patterns, but there's a newer pattern

for being able to pull knowledge into our AI system. So, this is generally called agent skills and skills one of the things that they do, you maybe have used them in your AI code assistant, is they enable this progressive loading of kind of additional knowledge alongside your AI calls. And so, of course, we can do this with Spring AI and so I'm going to show you some

examples of that. These can can also be useful outside of the AI coding assistant realm where if you're building your own agent and you want to provide knowledge to your AI, you can do that through skills. And I'm going to also show you skills jars which is a project I created to put skills into jar files so that we can version them, uh put them into our

artifactory or Nexus servers, um and do transitive dependencies, all the things that we like about uh dependency management, but for skills. So, um okay. So, let's dive into a demo here of skills. So, for this one, I'm going to start with just this like very basic naive approach. So, I've got my um my typical user message that I'm going to send, but I'm also going to send

a system message, and the system message is going to be the contents of a file, which is uh this my skills, let's go look find that thing in my resources dogs skills.md. So, Josh Long and I do a lot of presentations about dog adopting dogs and stuff, and so we have this skill that we came up with for that one, and I'm reusing it here. So, I've

uh got some information at the top of this markdown file. This is a a typical kind of standardized format for how we define skills. We've got a name and a description. And very similar to what we do with tool search tool, most of the implementations of skills in agents are only going to include this front matter information initially, and then it we're going to have a tool

alongside of that that allows us to progressively load the uh the actual contents of the skill later. Okay, so this is my skill about dogs. I'm adding some very uh important information to to the LLM about dogs. And so, let's go back here to this one. And what I'm going to do is I'm going to load that skill in as a system message, and then I'm going

to say, "What does the Chihuahua say?" And you may have noticed it, but I did include in that skill uh some information about how Chihuahuas talk. And so, let's run this one, preload skill. And when I do this, there's no progressive loading yet. I'm taking that whole file, and I'm loading the whole thing into the system message. And so, uh whether or not that's the skill is

actually needed to accomplish the user goal, that's all going to be sent over to the LLM. So, that's you know, probably some a lot of wasted tokens and my internet's probably a little slow, so it's taking a while to transfer all those tokens. But there we go, we see that this information about Chihuahuas and how they can do vocalized things, that actually came from my skill. So,

great. We're able to like add some knowledge to our LLM through a skill. But, as I mentioned, we probably want to do progressive loading of these skills so that we only actually load skills on demand. And so, what we do in that case is there's a Spring Community Project in Agent Utils that is a skills tool. And so, what I'm doing is I'm creating the skills tool

and I add a list of resources for where to find all the skills. And so, those in this case are class path resources and then wire it up to my chat client. And that, let's just run uh let's well, let's see it. Let's run the on-demand in debug mode. So, it's a tool because the tool allows the LLM to decide when it wants additional information. And so,

like we've seen it with tool calls before, the LLM's going to respond and say, "Okay, now I need the skill." And then on the next call to the LLM, it's going to provide the actual content of that skill. And we can see, sure enough, there's the the we can see the skill tool listed there. And if we scroll up, somewhere in here it's going to say, "All

right, I need the skill for dog breeds." And then it's going to respond and say, "Here's the actual contents of that skill." And so, you can see that progressive loading of the skills. And works the same way as it did before, but now with the progressive loading. Okay. Now, with skills jars, we do the exact same thing, but the difference here is that in my build, I

have added a dependency, which is this is up on Maven Central if you want to use my Pooch Palace skill. It's very useful. Uh then you just add a dependency to your project, and now we can pull that uh the the skill out of the dependency instead of having a hard code it in the project. So, that's going to work the exact same way as the one

we um had before. So, I won't actually run that one. Okay. So, that's that's skills. Really the the key thing about skills is that we are adding some additional knowledge to our LLM that is uh hopefully progressively loaded on demand into our uh into our agent loop. Anything to add on that? Yeah, thanks for that James. And I think important thing is when we are talking about

skills, so like James mentioned around skills jar, uh very interesting thing is when you actually go and check your skills, the skills are not just added over there just as another dependency. The skills are actually uh getting uh security scanned. So, when you are including the skills into your project, do not have to worry. You do not have to worry about the security vulnerabilities coming into the

project. So, that is very important thing which as developers we have to take care of. Uh I mean, the last thing I would want to do as a developer is to fix the CVs after including the skills into my application, right? So, that is one thing around the skills jar. Uh yeah. So, now coming to memory. Memory uh memory as I was reading an article, memory is

not a green field problem. I think uh it's one of the engineering challenge that uh we are going to face, and we are going to do it we are doing a lot of work around that in the memory for AI agents. And uh goes without saying why does an agent need memory because the conversation uh it doesn't persist. Similar to the concept like we had session persistence

back in the day when we were working with uh HTTP APIs and web apps. Similar to that, when the agent is not able to remember who you are if it cannot process that particular conversation or the memory of the user, then of course the rest of the things that it is going to do, it is not going to do that accurately. So, memory is a very important

thing that an AI agent has to be added. But, I think the caveat about adding memory is one way to do this is by adding in memory, uh which is going to be in your local. But, of course if you want to scale your application and want to make it production-ready, then the question is how do you do it? So, essentially there are two different strategies that

we have used. I just want to understand how many of you have heard this or use sliding window algorithm. Yeah, right. So, very interesting thing is when we are talking about short-term uh what happens is there's a window of, let's say, in Spring AI we have this default window size of 20 messages. So, all the messages are getting in inside that particular window of 20 messages. So,

it is like a message windowing, and after that it gets rolled over. So, that is about the short-term memory. Now, of course that is not sufficient. And then when you would try to interact with some of the assistants, you would find that after some time when the context window gets bloated or it's full, you would see that uh it uses something like compaction. It is trying to

extract the key insights from the conversations, and it is going to compact that. So, that is how the long-term memory has come into play. So, essentially when we are architecting for AI agents, we need to use a combination of both so that the short-term memory messages gets compacted and gets rolled over and stored into uh over the long-term memory. And under long-term memory also, there are four

different strategies I think uh which Amazon Bedrock also supports. Uh the agent core memory, right? I think the very recent SDK which went live, the Spring AI SDK for Bedrock agent core, uh went live which has different strategies also. We can talk about that. So, now talking about agent core memory again, why we are talking about it is we of course want to make the memory production-ready.

Uh so, we have this managed service. So, that is why we have Agent Core Memory, which you can use. So, out of the box, all of those strategies you can use so that you can actually understand depending on which business problem, which strategy you have to use instead of you reinventing the wheel about how to use that particular or how to write that particular algorithm. And that

is about Agent Core Memory and like we were seeing on tool search tool, which is part of the Spring AI community project. So, over there we also have this auto memory tools. So, that is also another way of including managed memory or production ready memory, I would say of production ready way of adding memory to AI agents. So, that is about memory and moving on, I think

there's a code demo that Do we have time? Yeah. Yeah, I'm going to show a quick demo of memory so you can see what that actually looks like. And then we'll start wrapping up. So, in Spring AI as Rose has said, there's built-in support for memory and the default is this this windowed I'm using in-memory memory in this case, but likely you're going to put the actual

memory into a database. You can correlate that memory with a user. You can use authentication in your agent to, you know, store that memory specific to So, pretty simple API like we've seen before. We've got to build up kind of our data store and then we add it into the advisors that we're going to use with our application actually on that line. And then when this all

works, we can then send a message that says, "Hey, my name is James." And then we can send a second message, which is what is my name and it will remember me because we're also passing in the memory as we go. So, what that's doing is you're actually keeping all the messages in that message window and then sending them with every request. So, this is going to

use, you know, more context. So, this is where some of the real hard problems are with agent design is how you manage memory well, how you provide the right things into context at the right times. And so there's there's lots of kind of architectural challenges and things to solve there. And as Varsha said, there's emerging kind of some higher-level paradigms that can do auto memory. So that's

one of the Spring AI community projects. So So that's how we do memory. I'll skip running that one so we can quickly recap just a few other concepts that I wanted you to be aware of for agent integrations. One is called rag. This was really the initial way that we started building integrations with LLMs before tool calls. You've already kind of seen rag when I did the

tool search tool. This is basically the same concept as the tool search tool was doing with the embeddings and the semantic search. It's just is going to do this with your data in your database. So this could be your customer data, whatever it may be. So very similar to tool search tool just for your data. So that's that's what rag is. I've got some code which we'll

share the repository with you in a little bit. Another key piece of integrations which we'll we'll talk very quickly about is that often times we need to provide some way of doing human in the loop. So when do you need additional information from the user, when do you need the user to approve something? And Spring AI has ways to do human in the loop. Also, there if

you're using MCP, there's a feature of MCP called elicitations which enables you to the MCP server to actually indicate that it needs something from the human as well. So elicitations are human in the loop for So those are our primary integration patterns. Let's sum them up, Varsha. Yeah, I think thanks James. And as we have seen that all the limitations that we have seen at the first

and are now getting solved with all these patterns. can we move to the next slide? Uh Yeah, so I just want to wrap up because we are short on time is just that the enterprise problems that we are going to solve as Java developers is not going to be about picking the fanciest model or the next trendy thing or the next trendy framework or thinking or losing

your sleep over which framework to use or which one to not use. It has always been about like I say about solving the illities like the observability, reliability, maintainability. And as we have seen throughout this particular session how every building block is actually already uh ready with all those illities. So, you do not have to worry about shipping uh production grade AI agents as a Java developer.

So, do not worry about the latest framework or the latest model. You should pick one particular framework Spring AI if you want to pick that, go deep, and go build your first agent using the building blocks that we just shared. And uh just remember the Java developers have been experts at working at the intersection of integration, business integration, and running and shipping production systems for decades. And

that is what we have been doing and that is what we are going to do again. So, that is our natural and unique advantage that we have. So, yeah, with that I think uh we are wrapping up. These are some of the resources. The code that James had shown uh it will be available. Second, I just want to highlight one thing that this one-stop Java and AI

guide which uh James has worked on it's an amazing resource. Uh ai4jvm.com must check. All the announcement frameworks, libraries, that basically all that you need to know to keep up with the JVM and AI uh ecosystem. This is the portal which you should be like your go-to. Do bookmark it and follow that. And of course the Spring AI agent code SDK if you want to really delve

into the harder engineering challenges that we talked around memory uh and also around the other production grade things that you need to uh yeah. Yeah, with that I think thank you so much for being such a lovely audience. >> Such a lovely audience. >> [music]