Java and GenAI: from basics to enterprise, Panche Chavkovski

This talk covers the integration of Spring AI into enterprise applications, emphasizing the transition from prototype to production-ready software. The speaker discusses techniques to enhance control over AI functionalities, ensuring that developers retain oversight of model interactions. He explains the importance of adding safeguards such as security checks, human approval processes, and observability in real-time operations. Additionally, the talk introduces the speaker's tool, Promea, which employs Java and Spring AI to facilitate effective LLM interactions. The session highlights common pitfalls in AI implementations and suggests best practices for creating robust and secure applications that can integrate AI seamlessly into business processes.

Panchiml [laughter] Snickl. Okay, everybody. Um, yes, finally it is raining. It was nice. sponsors are super happy that you're not sneak around this big tech park eating some food and because of the rain you just go to the sponsor desk. Yes. So great. Okay. We had some experiments with pure Java. But who on earth now writes pure Java? Yes, not anymore. So we are now back to

real life. And back to real life with Panchcho. We will now learn how actually what was the name of your topic by the way. Yes, >> Java. >> Exactly. >> So no more pure Java in this world. So Java and Genai and we will write enterprise software and I'm extremely happy to have our good neighbor Panchche. Panche I like Scopsko very much. Yes. >> Yes. So please

warm welcome our friend from Scopia Panche. >> Thank you. [applause] >> Thank you Mia. Hi everyone. So uh before I start uh last night in the meana here after the third shot of Rakia I was really trying to remember like how many times I have been in front of the Bulgarian audience here. Um I think this is like like my sixth time something but I'm not really

sure but still there are many and I'm really happy about this. So it's been a while maybe like five or six years uh from the last time that I've been around but yeah hopefully we can leave that in the past and have me here as a guest every year from now on. Uh so yeah let's get started. So I have an interesting talk for you uh prepared

today. Something which I deemed like very fun actually to create in the first place. Um so Mitia introduced me who am I? My name is M. My name is Panchcha. Um I come from uh Macedonia more specifically from Novos Trummit if you know where that area is. Um I currently work as a principal software engineer at this company called G&D etc. I'm also still the current leader

of the Java user group in Macedonia. I'm saying still because obviously no one likes to take that job away from me, not even the young ones. I don't know why. Um I'm also the creator of this platform called uh Pulseco. It's present even here in Sophia. So if you want to know, if you want to see, if you want to learn like uh what the air quality,

what the atmospheric conditions in your city are, pop up a browser, open sopia.pulse and you will see everything there. And maybe someone can fill me in why is the area around the main train station so polluted or maybe just the sensor is off. Um, and last but not least, I'm one of the team members uh at this place in Scopia which we call Netavville. It's like a

nice co-working co-inovation community hub in which lately we even organized some mini conferences and everything. Uh, so as I said um I also come from Java user group in Macedonia. So you know what Java user groups are, we just do that uh in Macedonia there as well. We have a website, we have an Instagram profile lately as well. And why am I mentioning this? Well, mostly because

of this stuff. So this was the last event that we did. We called it uh Dragon K Java mini conference. We had like 120 visitors and the most important piece here is this tall guy. This is Marto from Beajuk and yeah he was the guest speaker at our mini conference. I'm guest speaker at your conference. So hopefully we will do that more and more often in the

future. Okay. So let's go. So what is this talk about? So I don't know if you have heard there is this very low performing thingy so not a very quite often mentioned term lately it's called AI like I'm probably sure that no one is talking about it right not at every single talk at this conference but yeah I will not talk about AI I will talk about

spring AI like again uh but yeah enough jokes so it won't be spring AI it will be something along the lines of spring AI AI. So I know that there have been quite a few talks right now about Spring AI and related technologies. I really hope that with this talk I will kind of focus on things that lie like on the site and try not to repeat

the very same things that probably have been mentioned uh in the talks before. So my goal here with this talk is actually how to take like all the demos, all the quick startup guides and everything. Uh and try to apply that code into something that you can call maybe like a business application like an enterprise application. Uh so your proof of concept your hey look it works

that really doesn't work in production that really doesn't work with customers because uh quite often there you have loads bunch of other um requirements that you need to fill in and um the AI bit spring AI bit only covers like 10% of it or something like that. So I'll focus on three things uh precisely. So first of all how do you go from prototypes to production? So

how to move past all those AI demos to build something that might look actually like a business application. Then the second bit is how to take the control back from the LLM from Spring AI as a framework and how your code actually can have full control of what the application is doing. And third and very important is how to add all these critical safeguards. So AI is

cool. is really nice into doing some stuff but um quite often it can just get out of its rails and uh do some stuff that uh it really shouldn't be doing. So uh we will also cover stuff like how to do like security identity checks human approvals and make sure that everything that is risky will always have a human to confirm that bit. I will not talk

about LLMs or something. Is there anyone in the room who doesn't know what an LLM is? Thought so. Okay, so before to understand like how do we do all these things like how do we take the control back and everything let's just spot on to some a little bit basics. Uh so to understand what message types and the scenario here that we will work on is a

simple chatbot for weather reporting. So if you go to chat GPT and if you ask like what's the weather like in Oello? No is the village where I live. For anyone non-Bulgarian that literally means new village. So yeah, uh and obviously this screenshot is somewhere from the winter because it's at zero degrees which is obviously not. But what's happening here? So um this is a very nice

user interface, right? It's polished, it's packed, the user intent is very clear and everything is just looking so perfect. So you ask something very directly and you pretty much get the result that 99% of the time you wish to see. That's perfect. But that's what the user sees in the background. There's a little bit more than that. Um so every single chat history even this like one

stage like request response is not a request response. There are lots of stuff that are actually happening in the background. So first of all everything begins with like a system message. So the system message is the setup stage. So you're setting up the assistant like uh what his job is, what it needs to do, some alternatives, some quirks about it, uh what is it to expect and

maybe even some styles on how to uh report back everything that it's need. Then the second the actual request is the user message. In this case, it's what's the weather like in Oello. Then next there are LLMs uh which are called reasoning LLM or maybe I don't know I like to call them like token wasters but still no let's not go there. Uh so reasoning LLMs and

what they're doing in the background is they're like thinking so every once in a while you will even get like this thought message inside your chat history skeleton and then comes the interesting piece. So the LLM is just bunch of weights. It's neural network and it has no clue what the weather like in Noel is right now. So in order to get the idea what the weather

like is in Noel or Sophia raining right now, it needs to call an external service. Uh how does these things are done? You obviously all know these are done by tool calls and every single tool call actually ends up as the LLM not knowing the response of that one and the response from the LLM is an assistant message and within the assistant message there is this tool

request and the tool request is this nicely packed JSON object where it says perfect if I want to really know like what to respond please call this tool with these parameters and send me back the result that I need to use. Then the next message in this queue is the tool response. So tool response is now the other way around. So now we have our system either

a third party MCP, remote MCP, something local, local local to whatever I mean that can be. It's actually responding with the request that the LLM wants to. So in this case, this is the condition. It says sunny. It says temperature 21, which obviously doesn't coincide with the screenshot. Wow. I'm looking at this right now. Cool. And the last message actually that we get now that the LLM

is happy, the LLM knows the user intent, the LLM got the tool response, knows actually the actual data, parse everything together and says the current weather in Osell is sunny and warm with a temperature of 21° C. And that's the entirety of the cycle. Now important here, you can see this line that separates the tool request and tool response. That line is not just there. It's there

for a reason. Why? because these are so-called multi-turn calls towards the LLM. So this is actually two separate calls towards the LLM. So in the first call, we say what's the weather? The LM obviously doesn't know and he says if I want to if you want to fulfill this request, then please give me the response to that one. That's the first session. It's done. Then the MCP

does its work. We have some uh uh structural data that we need to pass back and then we do the second turn. So these are multi-turn calls. this in this precise this is the very simple situation where this is like two turn um request in order to get the answer last. So how to do all that? Well, Spring AI the easy way quite obvious you already know

that one. How to do it just uh spin up a Java project Maven in my case. Uh import the Spring AI bomb version that you would like to um how to use a very simple client usage. So a couple of starters here that you need to do. So Spring AI starter model OpenAI actually very simple to to start with that one. FMCP client Web Flux in order

to enable all the internals there working and then the dependency to the MCP. Now this is version 110 something like that but I believe that even in the latest one you don't even need to specify that one. I think it's already bundled but I'm not really sure. Um then what we have is uh just couple of uh configuration that you need to do. Um so again your

uh open AI key you need to select the model here is gypty 40 because this is a very simple thing that we want to do and then some definitions about the MCP servers that we want to use in the background. Here we only have like one server and it's a local MCP server uh that will need to provide everything for us. How to populate the chat client

again spring with the starters like everything is really bundled. No need for you to do anything else. So you just need to make like a bean that bean is chat client and you already have by spring already populated uh populated the chat client builder and a list of configurations for all the MCP clients that you just need. So you just make the chat client with this builder

here. And how to use that one? This is actually what you need to do. So uh first you need to define the prompt and prompt is nothing more but a list of messages. So in the screenshot you saw there there's system message, there's user message. Just combine those into one list. Again there are classes, user message and system message. Combine those in a list, add them in

the prompt and pass that prompt to the chat client. Call the LLM, get the response, get the final response out of it and that's it. And this will work and for 90 cases, so for 90% of the cases where you really need to do like a proof of concept or something like that, it will work and it will work perfectly. But you're only seeing the basics. So

the prompt, the system and the user message is there um and you get the answer and you have no idea like what's happening in the background. So if you put this into some sort of a sequence diagram, it looks something like this. So you have the app here and only this is in our control. So we have the prompt and then we just get the response. Everything

that is happening here the communication spring with the MCP and the LLM everything is managed for us. So for basic scenarios this is perfect. We don't need to bother ourselves with a lot of code. But if we really want to have control in what we're doing then this is a problem. So we have a situation something like this where we're only seeing the tip of the iceberg

which doesn't work for many cases. So it's very simple to implement as I mentioned can be easily extended with memory with other advisors. They're pretty much out of the box there. Relatively simple to persist. You can put everything in into the database even with the provided advisor there. However, the biggest con here is that you're pretty much left out of the game. You have no flow control.

You have no idea what's happening. You have no overseeing uh like how many turns are there within between the yellow lamps, what tools are being called, how many times the tools are are being called. And it's really hard to apply to real life scenarios. So now let's annie up this a little bit. I call it the proper way but again proper way named by panche and uh

what's the proper way? The requirements here are again made by panche. So this is nothing set up by an international association or a committee or something like that. This is simply like what I derived as things that really must be present in an application to even start considering it calling it an enterprise or a business application. So again back to the three roots. So first of all,

we really need to have observability and control. We need to know at any single time what's happening. We need to be able to observe what's happening and we need to be able to control every single bit and piece of the entire flow actually out there. Second, security and identity. Uh hardly any production case you will find like anything open like API open, MCP open or whatever. So

uh in 95% of the cases this will definitely be behind some sort of login wall which in most of the cases is some OIDC uh flow for login. So you have keycloak or have Microsoft entra or whatever. So security and identity really must play a crucial part here and even in very uh much of the business applications you will uh highly likely see uh API services um

that are identity bound. So for example, if you do like MCP for a banking application, you really the the banking application, the MCP really needs to know who you are in order to work with your details. And last but not least, we really need this human in the loop implementation and this proof of humanity. Why do I say proof of humanity? Because they are very critical operations

sometimes that really must be somehow approved by someone and this someone really needs to be defined as a human. A human with some biometrics or something like that. So let's start it. The first bit observability control. If I go back to the sequence diagram actually that I mentioned um this is the part that I have a lot of problems with. Uh so this is the bit where

um a lot of magic happens and uh obviously the part to take the flow control back to us is to reduce that magic. Magic is good but in some cases it can be very bad. So we need to have a hold of this. So again in this multi-turn communication that we have we really must not allow this to happen automatically. really need to know like what's happening.

So how to start doing that one? Um what spring does uh in the background is that it has this being called internal tool uh tool execution manager and it comes bundled along with spring AI and if you don't do anything it's immediately populated and injected within every single call that you do to the LLM. So with this property the first one the internal tool execution enabled false

we're disabling that. So which means that uh springi will not do any tool executions for us. we will need to to do them ourselves. Sounds complicated but the implementation is literally like one line of code. And the second bit that I have parallel tool calls this is here just in case because uh in my experience I've noticed that if you work with models that actually support this

like gyptro sometimes even the clients or even the communication gets messed up and it's kind of rare actually to get these um uh these calls anyway. So if your application is acting up just disable this and you'll be probably much better. So as I said the first thing to do is you need to disable this tool calling manager. And once you disable this tool calling manager you

need to create it again. I mean obviously but this time we're creating it and by us creating it um we will have the full control of it like what what it does. Uh so this pretty much boiler boilerplate code. Uh what it does is again it gets the MCP clients and maybe even some built-in LLM tools because right now we can do whatever we want and we

just pack everything together into this static tool call back resolver where we put all the callbacks essentially all the tools that are available to the to the LLM to execute and we built all that at the end. So we have our tool calling manager and finally this is how we take control. So you saw the basic code like how to um how to prompt the LLM, how

to get the chat the chat client response and the final chat response. But now with the internal tool calling manager disabled, the magic simply stops here and it's up to us to resume the magic again. So this multi-turn communication needs now another run. So it's up to us actually to turn the wheel back again. Um and if you noticed at the sequence diagram where I said so

this is like a loop that that uh turns like many times uh as long as there are tool calls and you can see here I mean the code is even uh fairly descriptive. So we say while chat response has tool calls do all these things and this is the the crucial part. So tool execution result you can see here tool calling manager execute tool calls pretty much

self-explanatory. So this is the way how we're um invoking the the tool calling and here is the um well the yeah the actual running either MCP calls or local tools whatever we have here but the beauty of it is that we can do something before or after and why do we want to do that so imagine if I don't know the LLM tries to call a service

and it inserts some critical data in it we can filter that one out imagine if the LLM returns something again with something that we don't want the user to see we can clean that one out. Imagine that the LLM tries to to call a tool which really is not nor the place nor the time to call it. We can break that tool here often. So now we

have the control of exactly what's happening or just simple imagine you want to log these tool calls for something for debugging purposes more in a different way than the logging provide than the login advisor actually have. We can do that and [snorts] the rest of the code here this is simply getting the response back putting everything again into that list of messages. So again to reconstruct the

entire chat history skeleton and lastly but not least to again invoke the chat client again. So again we're revoking the LLM but now we have the original tool call request. We have all the responses. So basically all the responses that the LM wants to see. We pack everything together send it back to the LLM and we start the cycle all over. And this will repeat as long

as the LLM has any questions. Once it's done this uh if cycle has two calls will yield false. And in that case we're done. We will just need to render the final message. Okay, enough talking from me. Now let's let me show this in details and live in demo. Okay, this is the most challenging part. Uh so uh what we did uh is that we developed uh

this completely Java and Spring AI based tool in our company. It's a it's an LLM chat client and it's called Promea. Am I getting giggles from the audience? Promaya draft is the same words in Bulgarian. It's not. Okay. So, in Macedonia, prom is when you open like two windows at the separate side of the uh of the building and you get like draft wind inside. That's promea.

But yeah, it's a crafty play of words. Um why do we call it prom besides the obvious one? Uh first of all, to promote AI to air out your invention. Yeah, funny I know. And to bring freshness into your enterprise like fresh oxygen, chili and everything. So, [snorts] let's see if this will Can you see my browser? You can see my browser. Let's see if this thing

is here. So, uh [sighs] what are you? I will just ask this one and I will enable the tool execution. I just ask the LLM. Hopefully, this will work. Yeah, it works. It says I'm intelligent agent blah blah blah blah blah set of specialized tools. My main goals provide accur selecting invoking appropriate tools. Okay, I can ask him what tools do you have in your disposal and

it lists a couple of those and these come actually from two sources. The first source is one MCP server that I have connected and that MCP server is called loans MCP server. So it's obviously mimicking like uh managing loans and the second one is built-in LLM tool. So this final thing confirm LL action for which I will talk last is there as well. So fine maybe I

can ask them like how much do I owe in total and now you see something like this. So what's happening is tool call and tool responses. This is the part of the of the observability that I'm mentioning right now. So the magic is no longer here. We're in charge of the magic. We have the wand in our hand and we're doing everything. And by doing everything, we

have control on everything. We specifically know what has been sent out, what has been requested for us. And we can even see the response here. And in this case, what we're seeing is that the LLM does a tool call. Then we execute that tool call. We get the response, which is the get loans list. And then we send all those information back to the LLM from which

the LLM populates the final answer. And it says that I have a BMW X1, which I don't. I mean, I don't have any of these, but still. Uh, and it says I I owe that much money. Then I can maybe ask uh share some details about my car loan and now it will call a different service. It will say get loan details with loan ID. So it

actually knows everything from the previous uh chat. Everything is here in the context and it will say here are the details about the call about your car loan u car BMW X1 which I don't have again and blah blah blah blah blah blah blah blah all the details about it. So that pretty much is the basic of the observability and the flow. we have everything and at

this point it's pretty much left to us to work with them. Uh okay now next to the other topic first I need to stop this because there is some buck in Spring AI where sometimes it just simply gets stuck and it needs like two minutes just to close the application. I don't know why. Now next to the second part security and identity. Um so if you go

to um the official MCP documentation and you go like uh what the security in MCP actually means you will find this huge document. Um but this is kind of like the gist of it. Um so we'll see like what is what does constitutes a secured MCP server what constitutes a secure MCP client blah blah blah blah blah. In basic terms, it means that you have like a

service, you just apply resource service to it or basic spring security and in 95% of the cases, you're technically done. Um, a little bit more details, it says that the MCP server needs to act like a resource server. The MCP clients need to act like to one client and then you have an authorization server key or whatever you have something sitting on the side which does everything

like token exchanging and identity confirmation and etc etc. These are the bits and pieces of it. Now this was an interesting development because when the MCP security um documentation and specification came up, it immediately like put Spring AI like on top of the technology there. Why? Because at that point uh I used to work with Spring AI, I used to work with Lchain forj and there were

a bunch of Python implementations there but neither one of those were actually prepared to implement security into their stack right now and spring were already there. Why? Because again, this works built-in within Spring and you just incorporate Spring Security with it and you have a secured MCP server and MCP client and it really works that easy. So let me show uh how to make a secured MCP

server. So again Maven and you just do this import. So you just need this O2 resource server and you just if the case is OIDC ODC compatible authorization server then you have this just the issuer URL here. I use like a local one for the demo. Afterwards you will see it's happening and once you do that immediately your tools that are sitting on that MCP server are

becoming aware of that. So now you're able to write authentication aware tools. The tools already know who you are. The tools can access all everything I mean that is provided by the authorization server. So here I have a very simple uh tool which returns the name of the current user just extracts the JWT and from the JWT gets the subject which in this case is the username

and it relays back that information. Now this is the MCP server. How to do the MCP client implementation. Here we import uh security and we import O2 client. And this is probably the most complicated uh bit of the entirety of the yeah of the flow that I really don't like. Uh there are other means actually how to implement this one but the other means are even more

complicated but just yeah let's stick to this one. Uh so the bas the basic implementation actually wants wants you you to register like two clients for it. Uh and the first one is a regular authorization client which does token authentication. So this is when the user logs in and when all the calls to the MCP server are called and back. So this is the user aware one.

But here there's also this other one you can see down the second block and it says for obtaining tokens for listing tools initializing etc. Why? Because if you run Spring AI in a completely managed environment, what it does is that once you populate all the beans there, it tries to learn like what the MCP clients offer in order to populate the tool list. And that's why it

needs this client credential authorization mechanism there as well. So that it can communicate machine to machine without any user being present. So just simply to get the listing. This is the complicated bit. But just have that in mind. you can find that piece of code like in many uh and even at the official uh documentation there for Spring AI like how to do that one. And last

but not least, you need one some means how to propagate this information from the client to the MCP server and you need this bit of code. Uh so in the core spring AI while communicating with the MCP server uses web client and what you need to do is you need to uh make your own imple uh you know your own bean of the web client and here

you need to apply this filter function. So there is this MCP sync client exchange filter function which I will not show here because it's like 50 lines of code or something like that. Google it, you will find it. It's pretty much the very same class. So what it does is that um it detects what kind of call is and if the call is one that comes from

a user and it needs to go to a MCP server in that course it just injects all the authentication information that it needs so that the security on the MCP server will actually work. So again how this looks like in practice again promea but now you will just give me one minute because I need to restart here because this is completely different stack of application and it

pretty much resembles like um what we need to do so an business scenario like this kind of business uh doesn't constitute of just the client and the MCP server but we have different u entities here right now. So first of all everything starts with the authorization server. Uh here I'm just using uh the one built in by spring. So spring out server. I know I'm working in

Eclipse. I'm just mentioning all this to you. Um then the second piece that I will run is an API server. This is actually where our business logic resides and this is pretty much the very same logic that you might have in your applications. So uh the out server the API server work for the same realm. Right. Then once these are done, I am starting the MCP server.

Yep, this one. I'm starting the MCP server. And now the MCP server is basically just a facade in front of the API server. And it still works under the same realm. And when all of this is done, let me see. I should run the application there as well. Will it run? It says it's running. Will it work? Let's see. So, if I go local host, I should

get logged in. Why am I locked in? Shouldn't be logged in. No, not here. Are Nope. Oh, yeah. I know. I forgot. Okay, give me one second. I have a profile authentication which I need to enable. Oh, puncher. Come on. We talked And this is the part where Spring AI says, "Please wait. I cannot shut down the application right now." So, while it's still deciding what to

do. No, it decided already. Let's see. Good. Then I'll go here. I'll close this one. I'll open up a new one. And then I'll go here. And yes, please sign in. So now everything is protected. So first I need to log in. And I have this amazing security user password. Right. Right. And then I get this one here. And it's pretty much the same. But now I

can ask something like this. And it says you are user. Wow. Such big information. Um but why is this very important? um in I can guarantee that in almost all the applications that you need to eventually uh extend with some sort of an AI in front you will need to have security you will need to have identity if this is connected to a production API I don't

know banking software whatever uh it knows who you are and uh all the API calls are immediately authenticated so there is really no need for you to I don't know reinvent the wheel to add like specific layer of security and everything it really works with your current setup. So this is pretty much your keylo residing somewhere Microsoft tentra ID whatever you have your services you have your

APIs you add an MCP as a facade in front of that and you apply whatever chat client you really need to do now in my case here this is springing and everything works under the umbrella under one realm under one authorization server and this is really something that we can we can might start actual actually calling it a production So uh while this thing is restarting I

will move on to my third topic. So human in the loop and proof of humanity. Um why do we need that? So if you recall uh the definition of what a rest server is and um there it says that in rest server the get operations must be by definition it important and this rule really belongs here as well. So it means that standard tools everything that is

just relaying information to you really must be important. It must not change anything in state. It should be uh stateless. It shouldn't really do anything on its own. And then comes the second piece. So the LLM is really really really not to be trusted with modifying tools. You think that if you're polite, if you explain all the tool to all the tools to it, if you explain

like what this tool does, what it's uh requested and even I don't know to ask some more questions about it. Yeah. No, that's just I don't know. you're talking to a drunk person in the alley. I mean, it might listen to you, but often more than not, it will not. Uh, so for all these operations where you're changing a state, simple changes that it doesn't even need

to be like a risky operation as well, you really might consider adding like some form of human in the loop based on how risky the operation is. Take that much precaution as well. And finally, some additional proof of identity and sign challenge for maximum security. Why? Imagine you are writing a chatbot that does payment. This is the top level or works with your help data. A simple

login just doesn't cut the slack here. It really must prove so you really must prove that it's you. You really must prove it with some sort of um cryptography at the end actually to really confirm that that's you. So if we're talking like online payment for example, you have this thing called um secure payment confirmation. And what it does is like when the browser tries to pay

something for you gives this popup and it say please confirm that it's you and often it's like connected with platform authentication onto your laptop and you confirm with your finger and the idea here is pretty much the same. So finally everything needs to be circled around with some form of uh proof of identity and sign challenge for all of this. So what options do you have here

for human in the loop? Again these are human and loop options by me again nothing like that has been agreed upon but some bit some options that I have been working in the past year uh yeah which I've tested. So the first is to ask the LLM to ask back like this doesn't work at all. Uh this is pretty much like you open up a form and

you ask the user you tell the user I don't know how to do validation please enter all the details there accordingly as they're specified. So really this doesn't work. How can you do that? Just on every tool description you can say ask the user prior to executing this tool. And you will see that in 50% of the cases it will not it will just resume. The first

option that you can do is there are already some online for example Azure Azure AI foundry they have that you can uh preconfigure you can pre-populate an agent that resides onto their infrastructure and you can configure it somehow. So you say ask the user before uh doing some critical change or something like that. But again this is something like front end validation. is a little bit more

difficult to go around, but if you know your way with prompt injections, you will do that like in a matter of minutes and you will definitely circle around that one. Then the next thing now we're actually starting with the area where this thing actually works is to inject event handlers before or after a tool call. So you already know how this call looks like. We have the

tool calling manager and we have the blocks before or after. And this is the first bit that you can do. So in your client whenever you're getting like a tool tool request you can see which tool request is and if you deem this one to be risky give a pop-up or something like that you can ask it and this will work if you're working for like a

closed system in 90 cases 90% of the cases this will work but what's the problem is that this is a very tightly bound client application alongside with your business needs and it's not server enforced. So if someone takes that MCP and connects it to their own chat client, you have no protection whatsoever and everything can be executed immediately. So it works but only in limited systems. Then

we have an interesting case where the LLM is disconnected from the API. So we have like a different means of actually how connecting the application. Most of the cases we should have like um the LM client, we have the MCP and the MCP talks to the API, right? In this case, they are split. So the LLM, the the client application communicates with the MCP and communicates with

the with the client API, but those two do not communicate with each other. And why am I mentioning this? In this case, the MCP doesn't execute anything. So the MCP just gives instructions back. So basically, I'm saying how much will I owe for for my car? Uh and the uh the MCP says, okay, uh the user wants to call this uh this API call, which is get

get loan details, and please execute this tool call for me. and then the client executes that one. So again, it's pretty much controlled, but the main obstacle here is that the client needs to be API aware. So you're not doing things by standard. You're doing a very custom implementation which is just a matter of time when it will become like unsustainable. So this is not the solution.

And the only one thing that I really found to actually work and to cover all these cases that we've been working um is this one. So we don't have like natural tool calls like one method, one one function that does it all. But every single one of these critical methods are split into two and we have a request and we have a confirmation and these two work

alongside with a third call which resides on the client application which resides on the user machine and this one is uh client side confirmation and it can even include this client side signature. So in this case what's happening you will see this lovely sequence diagram here the first thing the most important thing actually that you should see that that now we have the risky ask here we

have the response but in the middle we have something extra and this is precisely what we need so now we don't have like a clean method call clean tool call saying like please give me that this information for you uh for me but the first call is like the user asks for this information and the MCP okay bins that one says Fine, this is a request. Uh

I'm creating a challenge sort of for this request and it's sending that challenge back to the client and it's up to the client now to confirm technically all of that. And how do we confirm it? In my case, this is just a hash code. But in reality, this can be public private key, this can be platform authentication, this can be pho, this can be whatever you can

think of. And we have the signature. We uh we sign that uh we have the challenge. We sign that challenge with something back and we call the second part of the implementation which is confirm the LLM action there. And there the MCP can actually see fine this is properly signed. It's correct. All the data uh are here. The original request is pretty much the same notice pretty

much is identical to the current request which is very important because if you think that the LLM will simply copy paste the original request back you are wrong. It will mess something up. If the ID is like um grid in my case every once in a while it can mess up a letter. So it won't be pretty much the same thing. So here we're doing like these

extra validations and we know at the end if everything passes that definitely definitely it's the user who've actually confirmed this one uh the original request and the response actually coincide with together and the MCP server is now clear to uh invoke the final uh the final action. Again, enough talking from my site. Let's see how this works. again just now not to forget to disable the then

again the MCP server not you okay let's see Um, now I will do like a risky call and I will say uh I'd like to pay €500 to the loan loan for my car. Obviously a risky application uh a risky call, right? Uh handles my details and it's mutating. It can actually cost me money if something is done wrong. So what happened in the meanwhile? You can

see that first the application says get long list in order to get all the details that he has no clue what am I talking about and then it knows oh okay this is for the BMW uh and then what it do what it does you can see here that it says um action call so this request pay loan action this is the tool call and we get

from the MCP this is the uh pay loan due action so it says this is the action ID and this is the description so the user wants to do that one please confirm and then you see the second call tool colon this is the confirm llm action now This call is not an MCP call. This is a built-in call within the application itself. And because it's built-in

call, I really can control everything what's happening here. And you can see that here the dialogue changed. I no longer have an input text field. I have approved last request either approve or reject. In this case, I will just say approved. And I say approve action. And now the flow begins again. We see confirm action. Then we're sending to call confirm. Now with the previous section, we

even have the signature of it which is hash code in my case just due to simplicity. And last that says the payment of €500 toward your car loan has been successfully processed. And this is it. If I try to do the very same thing again, let's say like this. But now I will say 1,500. Hopefully it will not get messed up because of the context here. But

let's see again different something. But here I will just say reject. Nope, this is not me. Reject that one. Rejected her to what your car loan has been rejected. Which means that the LLM, the MCP will simply not be able. And in this case, whoever works with that MCP server, if they don't work alongside with the specification that I have written here, simply the application will not

work. And that's good. So it means that uh we no longer reside only on client side uh security. But now uh completely this um MCP server can even be open to third party application for integration for Okay. And just to conclude I have this slide. So just worthy mentions something that I've been working um uh with as well. Uh so first of all uh you don't only

have Spring AI. So you have longchain forj which is its smaller brother let's say. Um even when both of these things started like longchain I really like longchain like the way how the APIs there's like more narrative more and everything but when everything came down to implementing security and even take control into the streaming and everything there really [snorts] spring AI kind of raised up on top.

Uh the next thing that you can do is you can use Google agent development kit. So Google ADK um it comes in three flavors I think it has like TypeScript, it has Python and it has Java. And Java is like the less I don't know maintained from it. Um seems a little bit chaotic, much more verbose. You have control like literally a lot of stuff but even

more that you really need. So that's compared to Spring AI. Then the second thing that you really really uh need to know because everything is going toward uh this is this agent to agent protocol. So I believe that this one will win in this war because there are at least two right now competing. Uh but this one is uh supported. So it's again driven by Google mostly

a couple of other companies and there is already native implementation native support in the ADK and uh in spring AI at least for the client side there is already uh so what you can do here you don't uh make uh MCP servers you make different agents uh and these agents uh can uh be configured with their own sets of prompts their own sets of tools their own

sets of behavior uh different LLMs whatever uh and how everything is wrapped up at the end so it's again behaves like a service of a kind but we don't have MCP tool listing we have this so-called thingy called agent cart and this is the identification of the agent so it says my name is this one you should use me for this and this and my capabilities not

tools my capabilities are this and this and this and this and that and it can do a lot of stuff so it can even do like a long uh term uh jobs so basically you can start something and then leave the agent once it's finished like couple of minutes afterwards or an hour afterwards to ping you back and saying hey I've already done the job for you.

And last but not least, if you work in payments, uh there is this protocol that is rising up. It's called AP2, agent payments protocol. It's based on A2A. Uh but it just have these cryptographic verifiable credentials. Um it's still rising. There are already some implementations that that you can see and it's even working with this other protocol called UCP, universal commerce protocol. I don't know if you

see it, but the idea with all of that is that you can even use chat bots for shopping, but not like plain shopping. Please buy these shoes for me. But what you can basically do is please buy these shoes for me, but not for the price that it's currently right now. Please wait while there is a discount and do not buy them for a price bigger than

€50 or something like that. And you set up the agent and the agent will do that. It has mandates. It knows what you can do. It's given an allowance of €50 and eventually it will buy these shoes for you. So that's everything that I had for today and I have 35 seconds left from my talk. Um I don't think that I'll be able to take any questions

right now. But if anyone wants to know anything more about what I presented today. If you want to see bits of code or even I don't know how ATA works even right now, please approach me. We can talk a little bit here or if not uh yeah just outside. So I'm in no rush even right now. So, I hope that this was informative. I hope that this

was interesting. I was gliding through the public. I saw like maybe three or four people like taking a nap, which is a success. I was expecting 30 or 40. So, thanks everyone for your attention. And yeah, again, it's nice to you to have me back here in Sophia to do this talk. Thank you again.