Supercharging Spring AI: Scalable AI Agents with Koog + Spring AI + Spring Boot by Vadim Briliantov

This talk discusses the integration of the Cook framework with Spring AI and Spring Boot to create scalable AI agents. The speaker, Vadim Briliantov, explains how Cook, an open-source AI agent framework, offers reliability and control, making it suitable for enterprise applications. He details the roles of different components in an AI agent, likening them to human anatomy, such as the body's functions and the brain's logic. The session provides a step-by-step approach to building a banking application using these technologies, focusing on creating an agent's architecture, handling memory and persistence, and ensuring fault tolerance and observability within the AI agents. Throughout, Vadim emphasizes the importance of control and structured workflows for safe and effective enterprise AI deployments.

[music] >> Right. Hello everyone and thank you for joining my session about supercharging Spring AI and building scalable AI agents using the mix of the Cook framework, Spring AI and Spring Boot. Have you ever heard of Cook framework? Oh, great. So, if you haven't, Cook is the open-source AI agent framework from JetBrains with specific focus on reliability and control, which is especially valuable for any serious enterprise

usage. So, if you look retrospectively, lots of people and companies have been massively obsessed by the idea of AGI and magically working black boxes that can do any possible task in the world. But the high peak has already passed. Even OpenAI has released the agent kit for building specialized agentic pipelines. And we have understood understood this problem from the beginning that for any serious enterprise usage, you

need the control. You need the control over the agent's behavior, the availability and the cost. And that's all what Cook is about. So, we have released Cook almost a year ago on the Kotlin Conf. And since then, it's um it's got it's got lots of traction. And also it has been already adopted by large enterprises, not only by JetBrains, but also by Mercedes-Benz, by banks, analytic platforms,

like cloud services, consultant firms and even some of the largest big tech companies in the world. And it's gained more than 4,000 stars on the GitHub, mainly coming from the Kotlin community. last month on the JavaOne, we also released Cook for Java. So, that any Java developer can also benefit from the same powerful [snorts] agentic workflows. Um but today we are actually also having a launch. We

are launching the full integration of Cook with Spring AI. So, that you can get the best of two worlds, Spring AI's extensive LLM and drug integrations and Cook advanced orchestration and agentic and persistent layer that you can use on top. And that's what I'm going to be talking about today. But first, let me also introduce myself as a speaker. So, my name is Vadim Briliantov. I'm a

technical lead and founder of Cook project at JetBrains. I've been working for the company for more than or more than 8 years, almost nine. And I've been also presenting on multiple international conferences. So, if you want to find me, I'm on LinkedIn. I have a Twitter account and also I post some medium articles for free, of course, about AI, JVM and project leadership. But today we're going

to talk about AI agents. So, we have seen lots of AI agents. Some of us have tried to build AI agents and that's been that's been like lots of the demos and the hype. But this year, 2026, is predicted to be the year when many AI agents would finally enter production. And as a Spring community, we have to learn how to grown our AI agents for the

production state. Um but for it to happen, we need to first understand the anatomy of AI agent. And that's going to be also the plan for my today's talk. So, the first component will be the body of our artificial intelligence mechanism. That's going to be our Spring backend. It will define what the agent can do in our system. Then we're going to include Spring AI. That will

have very particular purpose. It's going to be the nervous system and the senses, so that the agent has the instincts what to do. But then, we also have to add the brain, the logic, the power, the workflow, the orchestration and that will be the Cook strategy. Then we have have to also add the memory, which will come from the integration of Cook and Spring AI, so that

our agent remembers all the all the stuff from the past. But that's not it yet. The interesting stuff comes next, the immune system. And let me tell you something, I'm actually not feeling well right now, I'm sick. I may I'm having a sore throat. You might hear it from my voice. And my immune system is currently trying to recover me to recover my body so that I

can speak and present on today's conference because it's an important event for me, right? And talking about AI agents or actually any processes that you deploy on your production workloads, it's important so that they also have so that they can recover from any crashes that happen and they can continue their operation. And that will be the Cook's persistence integration. Then we're going to also add the mouth

for our agent so that it can tell us what it's doing. So, we have to integrate with AI observability tooling. And last but not least, um you might have noticed that um well, when you sleep, um your brain processes the information from the previous day. And usually first, you wake up with a clear mind. And second, you still remember important information points so that you can start

your new day productively. And that will be the history compression from Cook. It's going to add the sleep to our system. It's not going to be sleeping, but it's going to be compressing the history and preprocessing the information and it also will bring us the essential cost optimization, which is another important topic for using agents at scale. So, let's start with a more um technical schema. So,

an agent usually is a three-system component. The agent itself, the environment and the models, one or many. From the user's perspective, it's it's pretty simple. Like the user would send the input and gets back the agent output and that's it. But under the hood, the whole process is happening. The agent will be sending LLM requests to the model coming back with the LLM responses, which are always

either plain text, the assistant messages, or the tool calls. If the agent decides to call a tool, it will call a tool from your environment and come back with a tool result. And that process will continue until the agent strategy, the logic, the brain of the system decides to finish. So, if you're building Spring applications, what's going to be the environment for you? Do you have any

ideas? It's going to be your Spring application. So, the agent will be orchestrating the functionality from your Spring app. And now let's take a look at these three layers. So, the Spring application will be done with a Spring Boot. So, Spring Boot Boot will be operating on the first layer. On the layer in the middle, there's going to be Cook. It's going to power the agentic logic,

the workflow. And Spring AI operates on the LLM and integration with RAG layer. And we're going to be building banking application today. Any of you are building banking backends right now? Cool, you're going to teach me how to do it properly. So, let's start with the body, with the First, of course, we have to define our usual Spring YAML config. It's going [snorts] to be pretty simple

for the start. We're going to define our Spring Boot application, also pretty primitive. And then we're going to also have our REST controller controller for controller controlling API endpoints in our app. And there inside we're going to develop the banking agent service. That is something that we're going to build and improve throughout today's session. Um and probably if it's a banking application, it already has some functionality

like communications functions like for sending emails, maybe like read operations to read the data about the user's account from the database like getting the latest transactions, getting account balance, reading account info. And also write operations. So, the application is able to initiate a dispute about the transaction, cancel the transaction or transfer money. Or probably you have much more, right? we have our body and now we have

to add the senses and the nervous system using the Spring AI. So, in our application config, we're going to add the connection to the OpenAI, Anthropic and Google. You can also add much more models of your choice. And here comes the Cook and Spring AI integration. So, multi-LLM prompt executor is an abstraction from Cook that is responsible for executing model requests and it's also able to switch

between different models, which models? The ones that we have just configured using Spring AI. And it gets auto-wired in your Spring So, now we are ready to start building the brain. So, let's do that. We're going to use the AI agent builder. I'm using Java right now. I will also show how to do it in Kotlin. So, first thing that we have to do is we have

to connect and register the tools from your application. I will explain it later. Then, optionally, you can also provide the strategy, the workflow, the logic for your agent, if you want. Then, you have to also configure the agent to work with a specific model, the system prompt, and also other parameters like maximum iterations. And last but not least, you have to also connect to this prompt executor

injected from your Spring config. And that's how you build your agent. Um also, you can do that in Kotlin using a pretty nice Kotlin DSL, the DSL, which looks uh pretty concise. As I told you, Cook was originally built for Kotlin, and obviously, it comes both with native Kotlin API as well as Java API. And um now, let's build the simplest um Cook agent example for our

banking app. The agent builder, register banking tools, you find the system prompt, your banking assistant, LLM model, I'm going to use GPT-5, prompt executor, build. 1 2 3 4 meaningful lines of code, and your banking application instantly becomes agentic. That's it. No other programming is required. Thank you so much for your attention. Of course, I'm joking. You might notice that I even didn't provide a strategy, and

there is a reason for that, because the default strategy from Cook will be a agentic loop, the simple one, and it will already power the agent, which is suitable for the demo, but not for the real usage. We're going to improve it. Um you can also do that um in Kotlin using the named parameters instead of uh builders. And uh this way, um you have an agent,

and now you can run it with any user's request. You can ask it to send $100 to Mike, and it will do something, call some tools from your app in order to uh make this transaction happen, and you can hope that it's not going to mess it up, right? Uh but I didn't uh tell you how to add the hands, the tools, like what the agent can

do in your app. So, remember, um yeah, by the way, tools are essentially Java or Kotlin methods that the agent can call in order to solve the user's problem. you define the methods, and the model constructs the arguments for them. You can control the outputs and the logic, and the model controls the order, unless you specify the controllable strategy. That's what we call the also do today.

So, remember, we had send email, I just wrapped it uh into a communication tools class, and I have to also inherit it from the marker interface called tool set. It's an interface from Cook. But then, not every tool, not every method deserves to be a tool. Not everything needs to be exposed for a model. So, how to make this send email a tool? Just call it a

tool using the tool annotation. But the next problem is, what is the meaning of the string output? What is the acceptable format for the email? I don't know. You You might not know either, and the model will be even more struggling. So, in order to and like for the model to be more informed how to use the tool, you have to also optionally explain, provide the LLM

descriptions. You can describe what is the meaning of the function and of what is the meaning of each parameter. Same way, we can also define the account reading tools. And here is the important thing. Um I told you that model will construct the arguments, and you don't want a model to work with any random user occasionally. You want it to work with the user that is that

is currently authorized. So, any parameters that are not really for the model, but rather for you as a developer, you have to inject as parameters for the class, and then inside, you can develop the tools. I'm just copying the same thing. Read these transactions, account balance, read account info. Now, they are all tools. Same thing for the writing tools. Uh the agent is able to initiate dispute

about the transaction, cancel transaction, and So far, our agent can do the conversation, database reads, and database writes. Of course, you might already feel the problem here, and we will solve it, but you have to do another thing first. You have to update the body, the Spring app. Let's define a function launch support agent with the user session ID and the question. We're going to create the

the agent using the injected prompt executor, and run the agent with the user question, and there is another parameter here, user session ID. It's an optional second parameter, but if you pass it, the agent can work with multiple users and also multiple sessions. Um now, let's update our API controller, and uh you would never guess what kind of agent we are building, chat support. Have you heard

of it? We're going to add the support endpoint. We're going to get the principal for the user's authorization and the support request. It's going to launch the agent with the username and the question, and it's going to come back with a response. But what if the user decides to come to come back and continue the same discussion? Of course, the good thing is that the agent is

not single use. As I told you, you can continue using for the same user session, but what if your application gets rerun on another machine? Then, you need to include the memory, the chat memory that comes from the Cook and Spring AI integration, and for that, first, we need to configure where to store the conversations. We're going to add the uh Postgres uh integration from the Spring

Boot, and then Spring AI integration for the chat memory repository using JDBC. uh that's how we get the chat history provider. That's a class from Cook, and it's actually wrapping the Spring AI chat memory integration, and it also gets auto-wired, and you can use it in So that, you can then install it to any agent builder, to any agent. You can install chat memory feature and configure

it to work with this specific history provider that you just set up. And then, of course, you don't want to store all the messages, you don't want to over-pollute with the lot too much information. Um there is a filter interface. You can do it as you like, but usually, uh you just want to um you know, latest um several messages, so you can also configure the window

size. We're going to store the latest 50 messages from the conversation. And you can also do this in Kotlin, and by the way, this install mechanism is the way of doing dependency injection into any agent. It's a It is kind of similar to uh advisors in Spring AI, but in its own way. And now, let's see how it works. Uh chat memory in action. Let's say that

I'm coming to my agent, I'm accessing the endpoint, and I'm asking to uh send $100 to Mike, and I'm also using my uh OAuth token. So, what I'm getting is nope, failed. And then, tomorrow, I come back, and I continue with just one question, why? And the agent, it I don't provide any extra context, but it remembers from the previous conversation, and it continues. Oh, because you

have only $99. Do you want to send less? And I come back again, and I say yes, 20 bucks. And I use the same And And it says, "Sure, 20 bucks sent. The new balance is 79.9." That's how it works. But now, let's analyze the agent finally. So, so far, we have added uh the conversation tools, database read tools, and database write tools. So, LLM will rule

everything in your system and control all the operations in your banking backend. You see any problems? Of course. LLMs, or as I like this analogy, random randomly working guessing machines, when given too much control, they will eventually mess up your system. It doesn't matter if the probability of the failure is 1% or 0.1%. For the serious enterprise usage, it's going to be a financial risk and reputational

risks. That is not acceptable. So, we can't We have to do better. The problem here is with the default strategy. Basically, how it works, it asks the model what to do, and then it starts uh calling to calling tools in a loop until the model decides to finish. And you can even code the strategy um using the functional strategy API in Cook. That's the simpli- simplified version

of how it looks. So, you just get the response from the model, and then you start calling tools until the model is tired, and it comes with the final response. As simple as that. The problem is that uh most frameworks stop here, and we are also hitting the limitation of Spring AI. the problem is, let me summarize it. The problem is that all tools are available at

once. So, the model the model gets too much freedom of choice. It doesn't know what to do. It gets distracted. Another problem is that there are no uh tailored guardrails for your system. And you would tell me, "Of course, you can prompt the requirements." But the problem is that the requirements will be prompted with a plain text. There is no guarantee that the model will follow You

can only hope. And that's where the Cook custom strategy, the brain, comes into play. We're going to use the domain modeling approach uh for building our uh workflows, and for that, uh we're going to need to understand this basic component, the subtask. It's almost like a subagent, but the difference is that it's it's it's it's also an agentic loop on of its own, but also, it's not

independent. It's not just calling subagent as a tool. No, it has the access to all the history on all the operations that was before. So, it is a part of the workflow, the part of the algorithm. And it also is focused on the specific part of the problem, not all the problem, and only a limited set of tools, not all the tools. Importantly, it also has the

strictly typed input data class and strictly typed output data class, which is basically a requirement. You can essentially describe what you want from the model as type-safe data class, not as textual prompt. Let me show it to you. You can build a serializable data class, any data class with all the fields, and essentially you're requesting the model to fill in the form. So, it what what it

will do, it will work work hard and call tools in loop until every single field is required that is required is fulfilled and resolved. And only then the model will be able to proceed to the next subtask, to the next step of your workflow. So, if you think of the real world analogy, imagine that you are coming to a bank and you are applying for a loan.

Like in a physical bank. First, you have to bring your passport, your birth certificate, salary certificate, all the documents. Then you have to sit down and fill in the form. And unless you do that, or if you forget some field, or if you forget to bring your passport, you're not allowed to proceed further. You cannot impress your banking manager with your oratory skills. You cannot talk your

way into getting a loan. You have to follow the the process. And um even if you try to break through, the security guard will probably catch And that is the role of the Cook framework here. So, it serves as a security guard that makes sure that the follows the instruction and it fills in all all the fields that you require unless until it it is allowed to

proceed So, to sum it up, naive prompting gives you hope. The main modeling gives a contract that you can trust. And now we let's build our strategy with the main modeling. So, we start with the user's question, which is a string. Then we also identify the problem. That's a subtask that is focused on the conversation tools and the reading tools. No modifications are allowed. And you're requiring

the model to fill in account issue summary. That's a serializable class with like account name, like current balance, related transactions, dispute ID if relevant, and many other fields. And only after all of that information is resolved, first you will get the object, not the string. And second, only then you will be able to proceed further to the next step. Which is fixing the On this step, we're

going to focus our model to work only with account reading tools and writing tools. No communications. And we request you come up with account issue solution, which is the solution for the problem with the user's banking account. So, are we done yet? We identify the problem, we fixed the problem. Well, of course no. We need to also verify. So, the verification step will take the account issue

solution. And please notice that you can only compose the steps if they are type type compatible. Everything is strictly typed. And it will come back with a critic Uh which is a class from Cook. It's a predefined uh algorithm, predefined component. And critic result will be uh it will have uh indication whether verification was successful, optional feedback, what has to be improved, and the original input to

pass through. Then uh we also focus verification step on the reading tools and communication tools. No modifications here. And now we can branch. If successful is true, we finish with a ready account issue solution. If successful is false, we pass the feedback to the adjustment step and then come back with a new updated account issue solution. And we loop in uh cycle until verification step is happy.

And then you finish. This strategy is much more reliable and resilient than anything that we had before so far in our system. So, let's learn how to code that. For that, let's learn how to code the basic component first. Uh I will show it in Java. So, AI agent subgraph, uh that's a subgraph of the strategy graph uh with the input data class, with the output data

class, with a limited set of tools, with a task which is a string description, like what is the meaning of the problem that we are solving. We still need some small prompt. And uh we also can switch the model on a fly. And then we build. In Kotlin, you can do it uh a bit easier with a DSL. It's called subgraph with task, generic parameters, tools, models,

And now we can uh build the workflow. Uh it's going to be just a combination of this component added multiple times with different parameters. Um identify problem, input string, output account issue summary, limited tools, specific task. And I'm going to use GPT because in my experience GPT works a little bit better when it comes to the Then I'm going to switch to fixing problem. Uh different types

that are important. We're requesting the types. Uh the task, uh also tools. And we're And I'm switching to Sonnet 4. And I'm switching on a fly. So, the history from GPT will be also available and re-explained to the Sonnet before it starts. So, Sonnet because in my experience again, it's a little bit more precise when it comes to tool calling. Um then I'm switching to reasoning model

for the verification and I'm building the verifi- verification step. I'm only passing the input here, the account issue solution. And instead of requiring the the output, I'm requiring the verification. With verification. And I And it will make sure that it will give me the critic result. Same way, um I can build the uh adjustment step. Uh same component with input, with output, limited tools, different task. Sonnet

again, build. What's left? We have to glue all of that together. For that, in Java, I'm doing AI agent graph strategy builder with input string, with output account issue And then I'm also connecting the steps using edges. Uh from identify problem, for example, I go to fix problem. You can see that them on the picture. Easily to visualize. But then we have to branch and for that

I'm using AI agent edge builder. From verify solution, I go to adjustment step on condition verification is not successful. That's not enough because adjustment step provide me with a critic result. And um sorry, verification step. And adjustment step requires a string feedback. So, I need to transform to make this statement compile. Transform to critic result, get feedback. And now I'm allowed to build. So again, you can

chain any transformations as you want, but it will only compile if you match the input and the output. And you can also do the transformations. Same same way we define the other branch uh from the verify solution to finish in case successful is false. And finally, uh from adjustment we go back to verification always without any conditions. And then we build the In Kotlin, uh you would

be surprised, it's a little bit easier the DSL. Which you can actually read as an English sentence. Edge verify solution forward to not finish on condition it's successful, transform to it input. Edge verify solution forward to adjust on condition it's not successful, transformed to a feedback. And now let's take a look at it again. So, you can notice that I never talked anything about the history or

message messages, tool calls. And there is a reason because Cook takes care of all of that automatically. You don't have to think about the context. Context is automatically passed from one step to another even when you're switching the models. So, let me illustrate that. Let's say that we are coming from identification to the fixing the the first one is working on GPT, the other one is working

on the Sonnet, and you already have some message history for the identification problem step. So, what will happen is uh first, we have to notice that conversation tools are not available here, but account read tools are. So, what will happen is first, the history will be copied for the to the next uh Sonnet model so that it can start with it. But it if you do it

na- naively, it will break, or it will start performing less precisely, and we checked that on a benchmark. So, we have to first re-explain the tools that are no longer available. And there is a bunch of strategies for that. The simple one is just explaining them as strings. And now we can proceed further adding more tool calls from the next step. So, that's how it works. But

you don't even have to think and even know about that. That just happens out of the box. Um moreover, uh you're not limited with just subgraphs with task and verification. You can also add any custom nodes, any actions or guardrails required for your specific system. For that, you use the same AI agent node Specify the input, the output, and then program any action using the input and

the agent context that is available including the message history. And there you can, for example, uh send an SSE request to your user, show the form in UI, ask the user to select something, and then the user will will come back with the user's response. And then you build your custom node. And if you're using Kotlin, you can do it a little bit easier with a node

DSL. It's a lambda with two parameters. and by the way, because of the ability to visualize the graph, the ability to build custom precise guardrails and uh some of our customers, which is Mercedes-Benz, have recently built um an agent for post uh purchase me vehicle maintenance support. So, if you buy buy a car and you call them, you go through this precise workflow to have your car

fixed. And only after getting the final confirmation from the user, you get the booking scheduled. Of course, Mercedes uh large enterprise and they value reliability. Uh they can't just deploy randomly working uh agentic loop. They will face otherwise some risks. So, they have to build a predictable workflow and I believe that many enterprises should as well. Of course, Cook doesn't only supports uh graphs. You can start

with basic agents. You can build graph workflows. You can also program functional strategies uh almost in plain Java or Kotlin. You can have multi-agentic orchestration. And also, Cook comes with planner agents such as GOAP, goal-oriented action planner, or LLM-based planner. You might know GOAP, for example, from Embodiment. So, Cook also has this type of strategy. All of that is just an implementation of AI agent strategy interface.

And you can also implement your own. So, a functional strategy allows you to code uh the strategy without graphs and it's great for prototyping. You can use for, if, while, and your normal normal programming language constructs. But, you will lose the ability to get the uh deeper telemetry, the visuali- visualization, and you will lose the persistence. So, with a graph, and I will show it to you,

you are able to persist the exact node and then recover from a crash. You cannot persist any arbitrary Java or Kotlin code and then recover from the same function, right? So, that is the uh difference. Then, you know, the planner strategies are the strategies where you kind of define only the nodes, no edges. And the planner will plan and replan the edges for you. So, you only

define like what you have to do and preconditions and side effects. And in runtime, it will find the best path if you as a developer never thought about that. It's a little bit more dynamic, but less controllable than graph workflows. But, you can also use that. So far, let's get back to our agents. Uh do you have any other problems? Of course, we do. As I told

you, LLMs take time. They are long-running processes. And user emails or any other communications take even more. So, suddenly, anything can happen. For example, your your system might crash. And that's fine. That's normal and we have to account for that. What if you're using AWS EC2 spot instances for the obvious cost optimization reasons, uh Amazon might preempt your running instance and rerun on another machine. And you

don't want to lose your well-deserved process because you are paying for the tokens. You don't want to pay again. And you also don't want to mess up the database syst- the database state, which is which might be inconsistent, right? And uh that's where Cook's uh persistence come in comes into play. That's going to be the immune system of uh agentic uh orchestration. So, first, we're going to

integrate uh with the Spring data source. We already added that in the config. And we will have the data source autowired. And then then, same way as you previously installed the chunk memory feature, you can also install the persistence feature and configure it to work with your Postgres JDBC storage for storing the checkpoints from the data source. And also, you ask the algorithm to checkpoint every after

every single atomic step. So, what will happen is uh here's the picture. You have the graph workflow. So, this is a feature of graph workflows, not other types of strategies. So, whenever you are uh it will checkpoint the exact place, exact node, exact place in the algorithm together with the agent state. So, let me zoom in and zoom in again. Imagine that we are somewhere inside verify

solution step. For example, sending a LLM request or calling a tool. So, what will happen is Cook will checkpoint exact place in the algorithm to your database >> [snorts] >> together with the message history and other message information so that after a crash, the same agent will recover right from there and continue its execution. So, if you're familiar with the term, it's called durable execution and it's

it's it's available through the Cook's So, uh we have controllable agents, we have fault-tolerant agents. Anything missing? Of course. You cannot just deploy your agent and believe that it's going to be always working. You have to be on call. You have to add the observability, the mouth of our system. So, for that, we're going to install the open telemetry feature uh and configure it to work with

any span exporters and span processors of yours. But, if you are using uh specifically AI observability tooling, which is much more tailored for the agentic tasks, you can, for example, integrate with LangFuse and Cook supports a bunch of integrations out of the box. It's not just open telemetry. It's kind of open telemetry on steroids. Default open telemetry would not give you that. So, you just uh provide

your LangFuse credentials here, open your LangFuse console, console, and you have your agent graph of execution, the the one that we just developed, all the nodes, all the steps, tool calls, uh met like uh LLM requests, parameters, how much tokens you spent on each step, how much time you spent, and so on. Same thing you can do with Weights & Biases W&B. There is also out of

the box integration there. You open your W&B console and you see the same type of information there. Or also, recently, we added the Datadog integration. Same thing. You configure your Datadog and you see the dashboard. Okay. Um other problems? Of course, the money. Who loves money? Everyone loves money. But, the problem is uh when we were running our actual and actually, Cook uh comes from our real-world,

hands-on experience inside JetBrains when running, building, and testing our agents on benchmarks and then deploying our AI products to millions of users. So, the problem is even if if you run uh software development SVBench to test a coding agent. So, in some cases, we we we had to spend hundreds of dollars for a single benchmark run. Imagine it deploying such system to like, you know, for millions

of users. That's expensive, right? And another problem is that LLMs struggle with long context. So, what happens is I'm sorry. [clears throat] What happens is uh your message history, the user messages, assistant messages, tool calls, tool responses, they will be all composed into a book that the model has to read from the beginning every time, every request in order to get you to give you the next

prediction, the next decision, the next tool call. First, that is expensive. You're uh paying for the same tokens every time. Second, you are spending more time and you have higher late- latency because the the the model is spending time to read. And third, in many models, not just the small models like Llama, but even in frontier models, it's very easy to hit the context limit. For example,

I was surprised when I was uh writing my uh Medium article about uh building scalable agents, I wanted to first show very simple example and solve very practical task for myself. So, I was applying for uh a UK visa. And I provided my 70-pages uh PDF uh document with my work contract to my model. Also, my 40 pages of my passport, some other supporting documents. Not a

big deal, right? Just a few pages. I mean, some pages, right? And then, I provided it uh to some model with a 2 million uh token limit, which is a lot, which is probably one of the largest you can find And then, I was really surprised because after a few iterations reading my documents, it just threw an exception. Because I hit the context limit. And I'm a

single person solving my personal task. Imagine deploying something like that for millions of user and users and running at scale. That's not possible. That's not optimal and that will break. So, here is the sleep component, the Cook's history compression. First, let's learn how to use it and then I will show what it does. So, um you use the same AI agent node builder. There is a bunch

of available out of the box nodes and components available. And you in in the IDE completion, you will see LLM compress history. Then, you have to provide the input. Uh in our case, it will be account issue solution. Remember, we are still building the same workflow. We don't need the output because it will be the same. Nothing will change. The underlying algorithm will just um change the

current history of messages and it doesn't do anything with the input. And then, you build. Then, you can also place this uh history compression phase anywhere inside your strategy, between every two steps. And you can also do that with functional strategy if you don't want a graph. You can just call it as a function. So, for example, if the history is too long and you of course

can program what is too long for you, then we compress the history and then we proceed to the next step. Otherwise, we just proceed to the next step. LLM will compress the history into a TLDR message, what has been done. Good thing, um you're not hitting the context limit anymore. You are spending much less time and much less money. The problem that we faced and we actually

saw it on benchmarks, model just loses too much information. It has to re-explore the project or whatever you're working with again to get the specific facts. That is a problem and of course we tried and we solved it. So, we did different [clears throat] history compression and the first one that we did was chunking. It's the obvious one. So, you just use the same LLM compress history

with your input and then you add optional compression strategy and here I'm requesting the algorithm to chop the history into a pieces of 20 messages, compress each of them independently and then compose a plot. So, instead of reading the whole book, LLM reads kind of shortened plot of the conversation which has been proven on our benchmark that uh it works in our case uh 6 to 8%

more precisely, but still not good enough. Why? Because we are human and we know much better what facts will be important for the model on the next So, we need to be able to to help the And for that we invented the fact retrieval strategy. It's another implementation of uh compression strategy and by the way we have a bunch of them. Strategy compression history is an interface

and you can also experiment with your own or you can go with the default ones or some of the some of the ones that is that are available out of the box in Cook. So, um I'm constructing retrieve facts from history um and I also have to supply it with a list of concepts like what I'm trying to search for in the history. [snorts] What facts will

be important for the model to keep. So, in our case um I'm asking what steps were important to fix the problem and what bank banking operations were suspicious to you. So, what will happen is AI algorithm that works there, it will first use model, maybe this model or maybe another unbiased model, to look through the history, retrieve the facts or something that is relevant to the facts

and then that's not it enough. So, you have to also reconstruct the conversation so naturally that it kind of mentions the fact so that LLM can further naturally proceed with the conversation. We have tried and evaluated and broken and fixed this algorithm many times together with our awesome ML colleagues and currently it's working stable enough to use. this is also the Kotlin version for it. So, you

use not LLM compress history with your class, you provide the strategy and the concept and you can can also use named parameters from Kotlin. And this is the illustration of how it works. So, instead of reading the whole LLM is left with just only a few important pieces of paper on the table and then it proceeds with them. So, if you want to view the full example

that we just cooked using the Cook framework, you can access both Kotlin and Java version of the code. Kotlin is on the left, Java is on the right. That's all. Thank you. Um and yeah, I wanted to also summarize. Sorry. Um just to sum it up uh layer by layer. So, we used uh Spring Boot as the application layer. Spring AI in our case operates on the

LLM and RAG integrations layer and Cook operates on the high level of abstraction on the orchestration and persistence layer. So, uh please also feel free to go to our website website uh jetbrains.com/cook. And of course, try Cook today, cook your enterprise AI tomorrow. Don't forget to go to the GitHub and uh leave a star and watch. Thank you so much for your >> [applause] >> I think

we have um around uh 5 minutes for the questions. So, please feel free to ask any questions if you like. Questions one, questions two. One question. Solved. Uh one question, okay. Yeah. So, in the the full flow where you have the sub agents, Mhm. what prevents, you know, the initial, you know, sub agent that gets the initial from basically putting random information just to pass it over

to the next one. But what what verifies the sub agent? That's my question. yeah, here's the thing. So, of course uh the strategy would first uh follow the form. So, you cannot forget to put some piece of information and usually if the model just read it from the database, it would probably put it right there in a field and then and you can also make sure that

it's not modified. For example, if something needs to be >> [clears throat] >> uh checked uh further on, uh you can we have a storage concept in Cook. You can store the information in a type-safe way inside the agent logic and then you can come back to it and verify it again using the custom nodes. But of course, if you're just using and relying on out-of-the-box subtasks

and verifications, again, it's much more resilient, but it's still a little bit a little bit of hope there. So, you have to also add user confirmations and other guardrails that you have. Does it answer your question? Yeah, thank you. Yeah, thanks. Hey there. >> Uh what's maybe a silly question, but what's the point of using Spring AI in this cuz Cook supports a lot of the models

already. Sorry? I think Cook already supports a lot of the models already. So, like what does the Spring AI bit add into this? Is it more models or Yeah, good question. Thanks. Uh so, let me repeat it. So, >> why to use uh Spring AI together with Cook and uh not just Cook without Spring AI? Wasn't your question, right? Yeah, so uh you're right and uh Cook

can be used on top of Spring AI. It It has also individual integration on top of Spring Boot. If you don't want Spring AI, you can also use Spring Boot directly. And then it also comes with its own model support. It also supports around uh 16 uh model providers. But uh for example, when it comes to RAG integrations, we just have the interface and uh I think

one uh Postgres or PG vector implementation. So, of course, uh the diff- the difference is uh the focus. So, the focus of the Cook team has been the agentic orchestration layer, while the focus of the Spring AI team so far has been model integrations, stability of that and also RAG integrations, data sources and that kind of stuff. So, from that perspective, if you already have Spring backend

application, it makes sense to add Spring AI and then use Cook on top of that integrations. But then, if you don't want to add Spring AI, you can also use Cook directly. Hello. Yeah. So, I have a a question. It's on your banking system. Mhm. Uh on this example, let's say you don't have a blockchain or something that do the check. So, if you need to check

that um for example, when you transfer to Bob, uh you take the money from Charlie and uh you want to be sure that is the money from Charlie and Charlie is authentificated. So, uh where will you put the security? Will you put it on the tools? So, for example, you say I transfer and I pass the token of of Charlie and if it's not Charlie or it

cannot get the money, you throw an exception on the tools or you let it pass through, the tools just transfer, he doesn't care, and then you put it on verification layer. Uh what would you prefer? Well, um as you just described it, uh definitely I would uh catch the problem as early as possible. So, of course, Cook uh sorry, uh tools are essentially functions that you can

program and you can also define the serializable data class. So, uh what I would do is I would include the actual result in the optional result field and also the problem description. And then the first line of the send money uh method for me would be checking all the authorizations, to be you have enough balance and all of that and then describing the problem properly to the

model so that the model either gets the real result or or the description which should be enough for understanding and pivoting the action and trying another thing. So if you can catch it on the tool level, catch it on the tool level. If you can add deterministic guardrails in your workflow, add deterministic guardrails in your workflow. If you can only verify with LLM, then verify it with

LLM and ideally also ask the user. And the good thing is that together with the persistence feature, I didn't show it to you but there is also a possibility to provide rollbacks. So you can also rollback to checkpoint back in time and it will also rollback the tool side effects. So for example, if you provide the side effects rollbacks, it will also rollbacks or rollback all the

money transactions to the point where the state was correct but you as an engineer probably has to be on call and you have to like click rollback to checkpoint here and then the agent will do it for Thank you. Thanks. Thank you so much for your attention and have a great conference.