jPrime Conference 2026 - Workshop - Day 2

This talk focuses on the performance implications of using streams in Java programming. The speaker discusses how streams can provide a more readable and maintainable code compared to traditional loops. He demonstrates examples of stream operations such as filtering, mapping, and reducing, while highlighting their performance characteristics. An important aspect covered is the relationship between stream operations and performance benchmarks, including the use of tools like JH for accurate measurement. The speaker also addresses the practical implications of introducing limits in stream processing and the differences in performance between sequential and parallel streams. He concludes with advice on best practices for using streams and the importance of careful benchmarking.

having Wi-Fi here. I think I a very good question because I do think I would need Wi-Fi. Um, okay. and the password is the venue one. Um, let me put this here where someone where we can read it. terminal. Oh yeah, let me do that. Um yeah, if you if you go to the barcode, you will Okay. Um Okay. But um otherwise, let me move this down.

Um here we go. Okay, good morning everyone. Thanks for coming. Um, let's just get started. If more people come, we'll go back to this page for an instant. So, um, I'm going to talk about stream performance. How many people were at Wenut's talk that just passed? So there were a bunch of stream related questions that if people had had this uh event first they would have answered

it correctly for sure. Um um uh let's go into that. So I'll skip about me. Um so um if you think about why are we using streams uh and there kind of the good reason to use streams is because if you look at the stream code it tells you what's going on. uh here in this case here I have a stream of points um and that's the

example that I'll be using and these are just standard Java AWT point with an int x and an int y um I'll say the point is short if it actually fits in a short meaning if the x and y are small um and so now then I'm going to uh apply a method that's uh that computes the uh the square of the length and then I'm going

to take the max And that's what I'm doing. So you can see from the code what this um what the intent here is. Whereas if you look at the equivalent loop, I mean it's easy enough to read the loop of course. Um but you do have to look at it a little bit carefully to see does it really do the right thing? You know, you notice that

[clears throat] it goes through all of these things. You know, shouldn't with max shouldn't one only start at the first one or something like that. So when you look at a loop, you have to look at it with more scrutiny because there's more things that can go wrong in a loop. It's not as self-explanatory as the stream code. So the reason that many people write stream code

is because of this advantage. Now sometimes um you can use parallelization although you probably shouldn't um in most cases and so we'll talk about that a little bit. And so when streams first came out in 2014, everyone started writing stream code. It was a human urge to turn every loop into a stream because that was the cool new thing. And personally sometime after 2014 I stopped doing

that and I kind of forgot all about streams. Um yeah I would use them once in a while but then about about two years ago I noticed that there was a flurry of articles and conference presentations and blog posts that talked upstreams again. So like Wenhut actually wrote a series of uh articles at dev.java at the Oracle site um where he talked up streams and functional programming

and used them to maybe a bit to excess and of course where do we find stream code? AI loves generating stream code. So I was just t talking to Frano Mata um who is right now developing a set of skills um that makes the AI move in the right direction and not to write terrible stream code and he gave me some example of really scary looking code

that the that the AI would uh would produce automatically. So it is kind of a good idea to understand uh what the streams do more in depth. Um, also, um, you may have heard about that Java 25, um, added a new API to streams, the Gaza API. We'll see it in action today. Um, uh, if you've never seen it before, don't worry about it. Um, but there's

I'm actually going to be using it as a showcase to think about performance. So, the question is, of course, what about performance? when you have a question about performance, there's no sense in guessing. you want to have some benchmarks and um so the usual way of doing benchmarking is with this tool called JH. Um here I'm linking to a blog article by Franto who happens to be

here at this conference today and I think at 1 p.m. he is giving a presentation about JH. So if you're unsure about your benchmarking skills you should definitely go and see that presentation. Um and so when when you do these kind of benchmarks, you know, you want to make sure that you first outside the benchmark setup and make any expensive setup like if you have random numbers.

If you were at my Valhalla talk yesterday, I had a nice example on how that can go wrong if you don't pay attention to it. And so um let's look at an example here. So if you unzip, um this code here and throw it into um into Intelligj, then you can read the code with me. Um, so, uh, again, for those people who came late, let's just

give everyone a minute to catch up with that. So, um, there's a link to the source code here. And so, if you like, you know, go ahead and put it into your own Intelligj. That way, you it's easier for you to maneuver around. But if you if you don't have a laptop or you don't have have your IDE on it, don't worry about it. Um, I'll put

everything up there. Did anyone need another minute with this barcode or can I go on? All right. As soon as I see that phone gone, I will move back. So, we're here. So, um, so I put this in this benchmarks and the benchmark I'm running is filter map, I think. Yep. Um, so let's have a quick look at filter map. So it um it has two benchmarks.

One where I'm doing a a computation with an array of points and the other one where I do the same computation with a stream of points. And it's the computation that you've seen before where I'm filtering with this is short method. I'll show it to you in a second where I'm then mapping to the square of the length. And it does really and it's an artificial thing.

Um so let's see where I have my points class. It's here. um is short just checks whether the x and y coordinates are short. So they're positive. Um and whether they're between oh no they're not necessarily positive whether they're between the minus 32,000 and the plus 32,000. Um, and then if they're short, then if I need to compute their length squared, I don't have to worry about

overflow. I can just multiply the x and the x and add the y and the y. It won't overflow. Well, I guess um it could overflow, right? But we're ignoring that. Um, and that's that's what I'm doing here. All right. So, if you run this benchmark, and I'm not running it because it takes like maybe 10 minutes or so to run it since each time it needs

to like warm up the VM a few times and then it runs and it does a bunch of iterations so that it gets some decent averages. Um, but if you spend the 10 minutes or so, then you're going to get this result here. And it's uh if we look through it, it says that if we run this with arrays, then it takes a certain amount of uh

milliseconds per op. And uh if we run it with streams, it's pretty much the same, right? The streams are teensy tiny amount slower, but it means we don't really have to worry about stream performance. And that's good, right? That means that if you see somewhere in your colleague's code that uses streams, you might feel that's a okay. It's clear, it's easier to read, and there's not a

performance impact. So no need to bother the colleague to say, well, rewrite this as a loop. Um, and so that's been what people have been thinking about for many years. So if you look at the old benchmarks, this Nikolai Parro did did a benchmark. I'll give you a link that's uh is about uh from uh 2014 and it says don't worry about stream performance is just fine.

Um then again um I ran AC across this master's thesis by some Danish student who says um you know um actually that's not really the case. Um first of all when you have small data sets of course the overhead of setting up the stream is large and then it's going to be slower and you might say okay but do I really care? I mean uh with small

data sets there's not that much computation in the first place so maybe I don't worry about it. The other thing that he observed though is when the pipelines get longer and more complicated then performance can drop and I'll show you an example of that that's fairly impressive in a minute and it's as hard to predict in uh when that happens. So in order to to see that

we're going to now run the following experiment. Um, so I'm going to be going into IntelliJ and run as an example the the the same demo that we've just seen here. Um, this is in the filter map class. So it's in demos filter map. And so let's see what what this thing does. It it sets up this stream of points. We then filter. We're then running map.

We're running map to int because this is has an integer result. Then we call max. So here yeah we want to find the the one with the largest length squared in this uh randomish collection of points. And so um this code here um in order to do a little bit of uh microbenchmarking thing here will actually run this computation multiple times. But for the demo for the

debugging demo we're only going to be running it once. So if you want to follow along, it's important that uh when you go to settings, debugger stepping this, how do I get settings? Settings settings um debugger. Here it is. let's see. It's called stepping. Oh no, I need to be a debugger. does anyone see stepping? Thank you. Thank you. Yes. see if I'm too close to the

screen to see it. Um, and there there is a box that says always do smart set into that. I think that's checked by default, but just make sure it's checked. And then there is a thing that's checked by default that says do not step into these classes. And for today, you want to uncheck that. Normally, you do not want to step into library code, but today I

do want to step into the library code. All right. So, um, then here in in filter map, let me get out of this because I didn't make any changes. Um, in filter map, we're going to set a breakpoint on filter. And now, um run this code here. So, it has reached the break point. So, now I'm um I'm going to step inside And um it's going to

now let me click on what I actually want to step inside trip because it's a big and complex thing. So I want to uh step into filter. And then here we see um the code for filter. Has anyone seen this before? Okay. So a few people have. For the rest of you, welcome to level two. Um so here we see the code for filter. And so when

you look at filter, it goes uh uh right away it says return and it returns an object. Notice it doesn't do any filtering yet, right? It it returns an object of some class. Um and that class has a method begin and it has a method except and in the accept method which is not yet called it will be called much later. But when it does get called

then the accept method will call predicate.est. The predicate is the thing that's passed into the filter and if it passes then it's going to use a downstream thing that it gets from somewhere and and tell the downstream hey continue with the same element. The element will not be changed. That's how filter does. If on the other hand it didn't get passed then the downstream will not see

the element. So filter returns an object that has a method that eventually will get called and does that thing. Fine. We've learned that. So we're going to get go uh back and we're going to look at what happens inside map um in the same way. So I'm going to step inside map and again in this way. Now here we're seeing what ma what map does or map

to int and you it looks very much the same thing. If you look at the structure of the code, it also returns an object of some class that we don't have time to look at in detail, but it's an object that has a method called accept. And that accept method will unconditionally call downstream.accept, namely with the value that is obtained from uh the the stream value u

uh after it's been treated with this map. All right. So, that's how these what these things do. Um, let's get back out and let's go into inside max. Um, uh, okay. Now, sorry, I should have set a break point there. I'm going to start this one one more time. right. So, I want to go to max. Okay, here we go. Max, we go inside. And now it

runs uh you can see max will then call reduce with a with an operation and so on. Um and I could now debug and it goes several layers deep and I don't actually want to do that. Um so instead we're okay let me actually um before I run ahead of myself let me have a picture of what what has just happened before u oh at at this

point. So [clears throat] there is a spliterator. We'll talk about that in a minute. What it does, it it generates a sequence of the stream elements. Then they get fed through a filter. So this is an object that has some method that eventually will do the filtering. Then it gets fed to an object that that does the mapping. And now the next operation is different. It's what

is called a terminal operation. And that's where the action will start. So um and so if you set a breakpoint in abstract pipeline.copy enter so how the heck are we going to do this? Um if you go to external libraries and you go to java util stream and then abstract pipeline um then search for the copy intro method. It is in line 566. And now I'm just

going to run the code and it's going to get into copy enter. And I want to look at this um this little piece of code here to see uh what it does. Um it's now going to test what's called a short circuit flag. We'll talk about that later. Um and it's going to go inside here and it's going to call a method called for each remaining. So

now it's good to know a little bit about what a spliterator does. A split iterator is is just like an iterator. You know a good old iterator where you have has next and next. Um but it has a couple of other things. It has a for each remaining that just calls next has next and next in a loop. And the rationale is that for many data structures

like if it's based on an array uh it is f it is better for the JIT to see the loop in one in one place rather than having multiple method calls. And so that's why they put it in there. The other thing that a spliterator has that's important for parallel work is the spliterator can be asked to do half of the work on one half of the

collection and return another spliterator that can do the other half of the work on the rest. So if if your data structure is an array, you can imagine that the spliterator has an easy life. It just cuts the array in half, gives a new spliterator for the rest, and then works on the first half. If on the other hand it's a linked list um then it would

have a harder time or if it's a binary tree it would might have a harder time uh splitting it into well on a binary tree maybe not but on on a link list. So that's what a spliterator does. All right. So remember for each remaining that and that's good for each remaining has been done so that the JIT can easily do this. Um all right so if

we step inside for each remaining um so let me do that here real quick. then we are here. Um so notice how for each remaining is written. This is not how I normally write a loop. This loop was written by someone who intimately knows what's good for the JIT. So the just in time compiler when seeing this loop will be slightly better at turning that into a

loop in assembly than if it had been written in a different way. Um and notice what the loop does is it calls accept. Um let's go go to there and then trace into except. So here we're in accept and this now should look eerily familiar right this is the filter we've seen it before and now the filter code gets called now this was something that Wenut uh

demonstrated earlier today where he had a stream pipeline and the mapping um referred to some array element and then before calling the terminal operation he changed the array element and About a third of the people was surprised that the change was visible in the map. That's of course because the map was only executed after the terminal operation uh got started and that's just the way it is.

That's the laziness of the collections. So um if this test passes, let's see there's a uh there's a chance that it does pass. Uh in this case it didn't pass. Um so I'm going to let it run a little bit longer. I set a break point in the in the downstream thing here because eventually some of them will pass. Now I'm going to um go in there

one more time. This code should also look eily familiar. Now after we have a situation where the filter passed and the downstream mapper gets executed. So that gives you an idea on what streams actually do. They first set up this pipeline and then eventually they iterate through it. All right. So this is just in words. That's what we just did. So now that you've seen it, now

how much more inefficient is this than writing the loop? You've seen the code for the loop. Yeah, that's the for loop and it it immediately called the the uh the if and then uh called the method of the filter and then it did the transformation and and did the maps. Here it's all much more complicated, right? It first sets up this chain of uh of operational mini

classes and then when the elements came in then an except was called and another except was called and eventually it gets passed onto the thing that handles the max and uh when that thing is done in the very end then the max pops up. So it looks like it should be a lot more uh a lot less efficient. On the other hand, the benchmark didn't show that,

right? The benchmark showed it was pretty much the same. And so, it's a good question. Why? And to answer a question like that, one really uh wants to look at um the jitted code. And so in in order to do that um you want to take a uh a recording of all of the JIT operations. So um when a Java program runs it first starts out interpreting

your code and as soon as it finds that some methods are called uh frequently or that some loops are called frequently then it turns on the just in time compiler and turns the bite codes into machine code. And it does that in stages. Um there's there's multiple stages. the stage that you want to reach to really have the optimum uh quality of code is called C2. Um

and so we want to see that C2 gets into action and it will do that soon enough. This loop gets executed um well it depends on how often I want it. In this case yeah I've executed the loop um 300,000 times. That's enough for C2 to kick in and do its thing completely. And then um then one can look at the assembly code. Um and so let

me actually do that. Did I load it? Watch. Um nope. right. So, here's Jitwatch. Um, I have hopefully this log prepared. Um, yep. So, I have to click on start. Um, and and why do I not see my own I don't know. Okay, maybe I'm just going to have to rerun it. I'm just going to boldly rerun it so you can see what what a JIT watch

run looks like. And so what you see flashing by is every time that the JIT kicks in and optimizes out something else when it compiles something, it sends these these messages that are somewhat difficult to read. And the point point of this jitwatch tool is to gather them all up and make them easier to consume. Um um it has now produced a file and the file is

called um jetwatch.log. So oh yes I see. So let's just run it one more time. Um, so I run JITWatch. I open the log file. Oh, I see why what I might have done wrong. Um, I the user interface is not stellar. I have to click on and now I'm still perturbed. I didn't find my classes. So I might have to look at the config. Config's fine.

so let's see if I can fix this. Um, not going to happen. All right. Um, I'm going to look at No. Um, I didn't recompile it. Let me double check that I recompile it. It was in demos. Maybe For some reason, it's not finding the source of my Let me try this way, too. Sorry that So, I have to set set up where it finds the my

my classes. Let's hope I did it right now. Um, okay, that's too bad. I I don't know what's what's Say say again. Um. Oh, okay. Oh, yes. Thank you. Okay, let's try that. That's a shame. Um, okay. Um, that What's that? No, I'm looking for Oh, you know, I'm being stupid. Thank you. Thank you. Thank you. Yep. Here we are. Okay. So, here we we have the

points and but it's still weird. Yes. Yes. Thank you. I'm see too close to the screen. Um so here we see the um the loop. So I want to go into the largest distance and I uh when I go inside here in into uh the max Oh, why is the assembly not found? Oh, I know it's because I skipped an essential uh step. I have to do

this one more time. Um, you have to also uh make it so that J did watch can see the disassembly. so I have to set the library path so that it can find it. Um, so now we do it one more time now that we know how to do it. It was filtermap.log. Okay. I think I think now that that should be fine now that you you've

shown me how to uh how to find this thing. Yeah. Okay. And then we are in largest and now it finds the assembly. So um when this inter this method here was max and you can see that max was inlined that means that the JIT has put in the code for max and in fact if we carefully look through the assembly let me turn off the source

which we don't need um real programmers don't look at the raw Java source um and if we were to look through the entire code here um when one can find that it has unrolled um the code for uh uh for doing this this going [clears throat] through the pipeline um and in fact let's see if I follow this um and I'm not going to do this now

in the interest of time um if you trace through where the 4 remaining was you will see that the loop that was in the 4 each remaining is now expressed in assembly instructions. Um it takes a few minutes to find one's way around the assembly instructions and so it's not something for the faint of heart. Um but trust me at this point what the JIT has done

it has taken you know what is several call levels steep that you've seen when I debug through them and turned them into a linear stream of assembly instructions and that's the best you can hope for. At this point the JIT produces code that is pretty much the same as if you had handwritten the loop. So it is possible when everything works fine for the JIT to be

very competitive with a handwritten loop. And so there's no need to automatically take every stream and turn it back into a But now we're going to make a tiny change. And so I'm going to take the same pipeline that I had before and I'm going to add a limit to it. And now if one were to trace it again and if you remember we have I uh

I was in the copy enter um and again in the interest of time I'm not going to do that then one will notice that instead of going into the 4 remaining it'll look at some internal characteristics we'll see that in a minute and then says I'm not going to call for each remaining and of course the reason it's not not going to call for each remaining is

for each remaining goes through the entire split array of contents and runs it all through this one except thing and with a limit you can't do that right you have to check whether you have uh uh whether you have reached the limit already so there has to be something that says no should I stop doing this and it's going to use a different pathway and in that

pathway it's going to uh go through the following code it's going to have a different loop where it checks whether the sync uh meaning the stuff that goes down further further down which in this case here is from the map to the limit whether it uh wants to cancel the computation and if that's the case it's going to cancel uh and also if the iterator then instead

of calling for each remaining it's going to call try advance which is this pretty much the same thing as calling next on a regular iterator um and so it's it's a different loop that's what I want to point out and um yeah it it does that and so when the limit is reached the cancellation happens So when you make a change in your stream pipeline. So let

me show you again the code in uh in in Java. Um this the filter map limit. Um it looks exactly the like the code that you've seen before except that I've tossed in a limit and that changes the internal computation. And you say so what? Of course it changes the internal computation. It has to now take the limit into account. Um and but when you look at

performance this is a 50% performance hit. Did you know that? That if you put a limit into a loop that it drops performance like that somehow. Nikolai Parlo uh what 12 years ago when he did his benchmarks managed to not hit upon this and so uh so [clears throat] thanks to that grad student in Denmark you know who had had a more systematic analysis through these things

so this is something that you know you could easily imagine yourself doing and uh at this point it is probably worth recoding this as a loop if it is performance critical if it's not performance critical then still go for legibility but uh that's why I know people who who say that whenever they go into a consulting situation and the code is kind of slow they rewrite all

the streams into loops um because this effect is real. It's not something um that uh that just doesn't happen. And so um so I've uh done a flame graph. I'm showing you here how you can do that at home. Um and so when you have to oops why is don't see my maybe because link is wrong. Be with me for just a second here. Um, I just

show you the browser here. We're going to be looking at both of them. Yes. CPU forward. Um here's how the flame graph looks like with arrays. And so the flame graph shows where um the code spends time. So um this is the the method that we're uh uh that we're debugging or that we're profiling. So it the computation of is short takes a bunch of time. Okay,

it check it has to check for uh uh whether the point is within range and it does that for a bunch of points. Um now when we look at the flame graph with the with the streams CP forward you notice that the flame graph all of a sudden looks completely different. So you can here see that it's spending a bunch of times into some in um in

some internals and look how much time it spends here in things like cancellation requested. So it spends a lot of time in this cancellation machinery that you know you wouldn't think is a big deal. And the reason for that is ultimately that this code if one for cancellation requested can't be inlined. When you go through it with JITWatch you will see that it's not inlined. And there's

a reason it can't be inlined because when you look at the source code for cancellation requested which I'm sure none of us has ever done before. It's polymorphic. The compiler cannot the JIT cannot tell that uh what the nature of the object is and it can only inline a method if it's a single method. But if it could be overloaded if you have a reference to something

that could be an object of one type or of some other type then it can't it could speculatively inline it may be but it it won't in this case because it does not have enough runtime information um to do that and um so that's a problem. Um and so you have a minute change in strategy and it has an impact on performance. So it does mean that

if you in your code if you use streams that go through large data sets um you know it might uh and if you suspect there's some performance issue it is uh worth your time to start benchmarking them because there's no simple rule that says you know when when they're fast and then when they're not. And of course you know things do get much worse with parallel streams.

Um so I I think here we all know about parallel streams. So I'm uh showing you here again the happy day scenario where I'm running this computation over here. Same computation that we've seen multiple times when I run it with a serial stream versus with a parallel stream. You see this is the happy day case where this parallelizes very nicely because the filtering and uh the mapping

you know can easily done in parallel. The accumulation the max is also easily parallelizable because it can find the max for each subset and then in the very last step take the maxes of all of them. And what is the factor of parallelization? It is the number of cores that I have. So my laptop happens to have 14 cores and so I get pretty much exactly the

speed up that I want. These kinds of benchmarks know were well documented 14 years ago when people first did streams. And so the answer is if you have a computationally heavy task, you know, go forth and run panel. That was considered a real selling point for for for streams. And it is if the task that you have is heavily computational. Now in business logic, we don't often

have that. And so in 99% of the case, you know, if you're writing a spring boot application and someone puts parallel streams in there, the answer has got to be to take it out. Um because what people instead do is they put blocking calls into uh parallel streams and that really uh you know why that doesn't work. All right. So now we're going to make a small

change to this um I'm going to um apply the map then the filter and then I'm going to toss in a limit. how do you think performance going to go? So we had a limit before and it had a performance penalty of 50%. Is going to be the same more or less? You would think the same, right? Okay. And we're now in an unhappy situation where parallel

is slower than serial. Okay. So that is not good and it's not clear why because this is computationally heavy, right? It should be a good use case. And so the naive idea that that one has is you know you kind of say okay know it's going to uh in parallel going to compute a lot of these values and with the filtering okay I mean it is a

bit of a problem that probably um the filter is going to kick in pretty early and some of these values here you know they're never going to make it through the filter. So that was maybe a bit of a waste but hey we had the core sitting there. Um but that is not how the uh how it works. So you have to know that a parallel pipeline

will never speculatively apply a map operation. So it will simply not do what many people intuitively think. It will not compute these and throw them away later. And so in this case the opportunities for parallelism are not great, right? because it first has to compute all of these and then see whether the filter and the limit um made it so that they all need to be taken

and only then will it start calling the map on the next branch. So it will not speculatively call map because the idea is that map is supposed to be something that's super expensive. you know, every call a map is supposed to mine a bitcoin or some such thing. And uh so um you look at this and you uh and you have some idea of what the workload

is in mind, but the parallel streams they had to implement it somehow. And so the decision they made is that when someone calls a map on a parallel stream, surely that was super expensive. And I'm not going to speculate and call it just for the heck of it. Even though performance would be much better, right? You might burn some computation in some otherwise unused cores. um and

and your final result would be better. But that's not how parallel streams work. All right. So with that, you know, uh quick intro into gatherers. So gatherers are an API to produce intermediate operations. Um a bunch of gatherers are given to us. And I think if I just show the first ones, you'll see, you know, the beauty of what a gatherer can do. Sometimes what you want

is you have a stream of uh of uh things and you want to batch them together. So here I have a stream of integers and I say I want to batch them together. I want to have groups of four except for the last one where I didn't have four. Or maybe you want to have sliding windows like you have here. So what you need is an operation

that takes a stream and produces a stream. Up till uh uh Java 25, there was no easy way for you to to write such a thing. And so now they have this gather API that makes it the these two gatherers, the window fixed and window sliding. They come with the API. There's a couple of others. Um there's scan and fold. They're not common, so I'm not going

to dwell on them. There's a pretty nifty gatherer um that will map concurrently on virtual threads. So here I say I want to have 60 virtual threads and I want want to run this blocking operation on it. And this is perfectly safe unlike parallel streams where it would not be safe. Um and uh so you can write your own gatherers. So I'm uh writing a simple example

of one here. And so here often times it happens that you have a stream of values and you would like to know the position of each of the values. Well, how are you going to do that with streams prior to 2025? um you could maybe like have a global variable and you increment that in the lambda and then venkut would not be happy with you right so

there was no great way of doing it and but now one can easily write a gather and here you see how you write one like that so any uh this is a stateful operation there's an intermediate state namely the index which needs to be updated with each scene element and so here's how you do it so this this is the constructor for the state which in this

case just holds holds the index and then I have to define I say I want to make a gatherer um and I'll talk about sequential in a minute and when the gatherer sees a new element then it's supposed to execute the following lambda this is the new element this is the current state and the downstream is where we're going to push the result to and we're going

to be pushing the index conveniently incremented here you know, don't code like that at home. Um, and and the element, uh, what's map.ent entry. Um, it is the poor man's pair. There is no pair class in the Java API except of course there is. Map. entry makes a pair with a key and a value. And so, you know, why why make a new record pair if map

entry is just sitting there? So, uh, just here to keep the slide short. So here I'm making a pair with the index and the element and what comes out of it is a new stream of pairs of long entity out of the old stream that just had T's in there. So that's how you implement a gatherer and so it's uh [clears throat] that part is easy. What

about offsequential? There's two kinds of gatherers. There's gatherers that that work with parallel streams and there's gatherers that will only work with sequential streams that will then force the stream to be sequential. In this case here, this thing cannot work with parallel streams because if the splitterator takes a thing and chops it off into two, there's no way in the API unfortunately even though in theory it

should be able to figure out the size, there's no way to get that size and feed it into the new state. So this gatherer is limited to sequential streams and I that's why it's called offsequential in the API. If I have a different gatherer that does work with parallel streams then I can say gatherer.off and then it creates a stream a gatherer with a different characteristic. This

one here will force the stream pipeline in into sequential even if you declare it as parallel original. So that's the one thing and that one I thought was easy enough to understand. Um the thing that that I didn't understand uh when I learned about this API is the of greedy And so what does it mean for a gatherer to be greedy? And so if you have a

a child maybe six or seven years old, you know, some childs some children at dinner they are greedy. You know they will eat everything. Some children at dinner they are picky. And you know they maybe will want to eat things but often times they refuse. So that's the distinction that is here. The question is whether this gatherer is willing to take all elements and absorb them or

whether at some point it says nope. So if you wrote your own filter then that would be greedy because it's willing to look at all of the elements. But if you wrote your own limit like operation then limit is picky because at some point says I don't want anymore. This one here is greedy because it's And so in the when you write your own gre uh uh

gather you can uh explain this. So, I was really baffled when I first saw this API uh and says, you know, how should I tell the readers of my books when when should they use greedy and when they shouldn't use greedy and why does greedy have the the weird name instead of the nice name and and all of that. And so I uh you now know more

than I did at the time, namely that when the gatherer isn't greedy, then the routing can't go through for each remaining. We've seen this with the uh with the limit case before and so are there any other benefits? So I asked in the core API mailing list and I got the following um that it's uh from the mailing list mafia where you get an answer that you

can't and so it says uh you know greedy allows for less signal tracking which in aggregate over separate operation can have a noticeable advantage. Keep in mind that this is primarily around the sequential performance which in [snorts] Victor's experience was more common than parallel streams. Now of course Victor is totally right. Um there are cases and you now know how to construct them where you can directly

observe greediness on serial um by benchmarking. Um but I was also blown surprised by this parallel thing. Of course, Victor is again right that in most time in user code it's you don't use parallel but when I first saw that I completely misread it and said greedy is important for sequential and it's not important for parallel. And so then a bunch of us got together at some

unconference and uh so Swen Waltman presented some code where he experimented with with these gatherers and in the end um I couldn't understand what he was doing and I made all of the different combinations where I had streams that were were parallel or serial and then I applied a gatherer that was paralyzable and greedy and picky and everything like that. And it turns out that it has

a super big impact which uh came as a bit of surprise for all of us and that in the end the problem is uh similar to what we've already seen in the simpler cases that when you have a parallel operation and then later in the pipeline you have a picky uh >> [clears throat] >> uh when you have a picky gather then it limits the opportunity for

parallelism that all of these parallel computations they sit behind a barrier and I could be having another really a three-hour workshop on and analyzing exactly how the parallel streams do this. Um so there is a there's a complex way of how the parallel computations are organized so that nothing happens before they're necessary. uh and we don't have to know the implementation as long as you understand the

concept that in a parallel computation nothing will be done until it's known that the value is needed. And so it's that thing that means that if you write your own gatherer and if it is greedy you should really really declare it as greedy because otherwise it has bad performance in these parallel pipelines. Now you would think that then that is now has become common knowledge on the

mailing list for for this topic. Two weeks later someone gave a code for a gatherer that was greedy like it could be greedy but it was not declared as greedy and so therefore it would have bad performance. So it is tricky to do these kind of performance u things and um so it's it's good to get there some more intuition about it and so that's how I

want to uh want to finish now because I know it's just me standing between you and lunch so um so when you see in your in your place of work that people use streams you know that can be good streams can be make your code easier to read and if performance is not an issue easier to read is um you you should never assume I mean the

streams can um cannot generally compete with a handcoded loop except of course if if the computation really can parallelize. I mean you couldn't easily write that loop yourself. It would it would be more work but it that never happens in in a general business situation. So it's interesting to know about parallel streams but it is true that they're they're not super common and so there's only so

much the JIT can do. The JIT is pretty amazing in the best case. It is amazing that it can unroll a deeply nested set of of uh these instructions and you can actually you know with enough time and patience you can watch it do that but um as soon as uh the pipeline gets more complex the JIT is limited. Um what this Danish student also did he

had some cases where the pipeline just got long enough that uh it ran against the inlining threshold. If you have too many nested calls, the JIT like more than 10 nested calls, the JIT will just uh it's hardcoded to throw up there. You can change that with a command line option, but do you really want to run your code with a with an option in production that

says I want to have an inline depth of 15? I mean, that's I would go right. So yeah, parallelization can work, but there are many ways in which it doesn't work. So I would say every time that you see parallel, it's probably a code smell. you want to look at it and yeah benchmark benchmark benchmark. So um anytime that you run into this it is well worth

doing the profiling and it is worth uh know for someone on on your team to have some expertise when do um so so work on your mental model on how these things work and you have the tools available uh to do that. So that's the message of the deep dive. Uh, I hope you find this phone somewhat useful and if not there's always one. Thank you. Any

questions? I see. Okay. So, so your question is whether um the the the difference in performance comes from the hotness and I don't I don't think uh it it does mean the hotness was the same in in all the examples that we had. We ran the same computation whatever 10 million 100 million times and so that's um the hotness comes from how often the iteration works. Um

the the so the question is um uh does it matter when we take the snapshot and so that's why it's important to use uh like a microbenchmarking harness because it will first warm up the thing so that what you measure is after the hotness stage has reached because otherwise um your results are going to be murkier um so you definitely uh and j makes that kind of

painless so go to fo talk and see how to do jh if that's not something you have done a Any other questions? Yes. So, so I mean the the problem with with limit was the way that it broke the optimization was obscure, right? That it's not that it intrinsically should necessarily be bad, but that the implementation of it was not so great. And so I talked to

Brian Guts about that. And so of course they are aware of these things and they say yeah maybe it would be worth you know re-evaluating the implementations um and see how people are actually using it today. Um and so eventually there's no reason why one couldn't do better with this. But the way that the library exists today, yeah, in in the JDK that you're using is there's

there's no great way to reason about it other than measure and analyze. So now you've learned that uh that that limit by itself is bad because it turns it into this bad pathway and gives you know you can follow through the slides and in slow motion to see exactly where that hits but it's it's not a great actionable knowledge right so I I wish I could give

you like five recipes of what to stay away from or you know some some skill for your coding agent what what not not to do but it's not that simple. Yes. So unfortunately you know benchmarking profiling you know or you know even going down and understanding a bit of uh how the implementation works is uh is worthwhile and certainly uh yeah benchmarking is is really a good

idea if you suspect you know if you if if you have big data sets that you actually work on you know if you have like venut's example where you had seven people and uh you run map and filter all right thank you very I'll see you around. Oh, that's rainy. I don't know. Yeah, I've seen storm, but I'm not sure if I hope there won't be. It's

orange. Yeah, the weather they on the on the weather forecast they said orange. Okay. Yeah. No, not windy. But I think it I mean I do prefer it's raining versus being in hot temperature. And I I'm I'm more a winter person than a summer person to be honest. Yeah, I'm based in Munich and Munich it's rainy. Oh, warning for thunderstorm. Yeah. Hello. Welcome. If you need the

Wi-Fi, it's on the wall right here. Right here. The Wi-Fi is I don't know. I mean, probably switch something. What if do you think? Anyway, see you soon. made a minor adjustment. Thanks to the guys in the back, we can share that video. That's going to be cool. U What time is it? Oh, it's almost um in any case, the Wi-Fi is on the wall if you're

not connected. um just behind the wall. So you can get it's the venue and the the password won't be leaked on the internet. So Let's wait one or more two minutes and then we We have two hours if I'm not wrong. Yeah. Yeah. Cool. So, I think it's 100 p.m. Um, if people join, they're more than welcome. But I think we're going to in the back. Ready?

All good. Perfect. Thank you very much for joining this workshop. Um, it's a Come on. Come on, come on. Oh, come on. You are more than welcome. The Wi-Fi is on the wall and have a seat. It's going to be a great workshop, a great time alto together. So, thank you very much for attending this workshop. Um, even more as it is a rainy day um in

Sophia. So, that's going to be cool. But we're going to have some fun al together. Um, how many of you knows what is a tabletop role playing game? I know one, two, three, four. Okay. But not all of the room, the few people in the room. So, let me state a common ground and share with you what is a tabletop roling game before we go into the

details and how to practice in um in the workshop. So when you play a tabletop roling game you do actions and these action are made through dices. You have to roll the dice in order to perform your action and if you are uh lucky enough then the dices will be with you and you will succeed your action. It will depend on certain parameters and the action that

you are doing and the character that you are playing is the one that you have created. can be a warrior, can be a drill, can be a dragon born, a monk, a sorcerer, and so on and so forth. My nickname is the Wterish, and that's why I try to mimic the Witcher. But let's [snorts] see what is a party when we play a tabletop propane game. A

video is worth a thousand words. So this is what happens when we play tabletop role playing game. So and this is what we're going to build all together. Al together we're going to use Spring AI and build a multi- aent role playing game application. So we're going to go step by step and it's going to be fun. Um, I'm Arno. I'm a developer advocate at Amazon Web

Services, but for the rest of the workshop, you can call me the Witcher because I don't look like Geralt. Even I would love to. Um, so that's why and I'll let you pick your favorite character name. So you can be whatever you want. You can be Ragna, you can be uh Thorin Oaken Shield. This is your time. Okay. So um let me share some common concept that

we're going to use in order to implement this multi- aent role playing game using spring AI. So the tabletop propane game for the one on the live stream uh it's a collaborative storytelling game where players create unique characters with skills, background and personalities. Um then you role play your decision using dices and we've seen dices like dices can be d4 d6 d10 d 20 or dund. So

there are different kind of dices when you're playing such such game. But when you play a tabletop role playinging game, I mean we personally I do prefer being a character in the story, being a game master, crafting a story, narrating it, adapting adapting the story on the flow based on welcome, have a seat, um based on the um experience. For example, if your players decide to be

friend with the dragon instead of fighting it, then you have to adapt your story on the flow and to make sure that you can pursue the adventure with the players of your party. So that's why being a game master can be hard. Uh it requires a lot of skills, creative creative skills, um calculation, you you know, you have to know the rules of the game and so

on and so forth. And you know why AI is really great at storytelling. So that's why in fact our multi- aent role playing game is going to be an AI game master that is going to interact with several agent. What's an agent? Well before talking about agent I'm sure you know LLM you heard about yesterday if you were attending J Prime yesterday. uh LLM like Chad GPT,

Mistralia and so on are trained on a certain data set, a certain knowledge but they cannot extend to the real world. They cannot interact with your API, with the databases, your file system and so on and so forth. So this is what truly differentiates a simple LLM versus an agent. An agent is going to use an LLM in order to resonate and iterate, but also we'll have

access to tools. And what a better tool than a hammer in a tabletop roing game. But not every tools look like a hammer. So a tool can be an API. It could be an existing function in your codebase in Java, in Python, whatsoever. It could be an external service that you are reaching. Oh, hi Mosquito. Um, and the agent is going to use it based on the

description of the different tools to fulfill the request. Let's say you want to create your character. you want to create, I don't know, foreign or shield and you need to provide some spe um some skills and some behaviors and you're going to provide to your character agent all the instructions and tools to let you create a character. But these tools can work locally. That's great. But in

a real world environment, we need to deploy it onto production. And then this is where MCP or model context protocol comes into the picture. MCP a protocol released by entropic allow the communication between agent and tools. It can be any LLM provider. It could be provider from entropic from Mistral from Amazon. MCP standardize the way agent communicate with tools in the same way. You can think about

HTTP but for AI agent. So that's really really key. Uh it's going to be really key in our architecture today. you know if we want to roll the dice um and also um we have MCP for tools but we we are not going to build one big agent if we do so probably is going to fail so instead of building one monolithic agent what's specialized agent in

order to collaborate all together and to connect these agent all together well this is the purpose of A2A agent to agent standardize how agent communicate and collaborate among each other. So thanks to agent to agent, each agent is going to expose its own capabilities through an agent card. And this card is just a simple JSON file and then they're going to all discuss and interact in order

to achieve the result. enough of concept. This is what we're going to build. Here you are. This is you. You're going to build your first AI and then once your AI game master is going to be alive, you're going to provide it some built-in tools to your game master. I don't know like fetching the web for looking for s specific information. Then you're going to create your

own custom tools. So you're going to create a roll dice uh tool in order to roll the dice. And once you created your tool, your custom tool, probably you want to package it as an MCP server and to expose it to everyone in the room. So you're going to create your own MCP server with one tool. And then because we said we want a multi- aent game

m game uh tabletop roping game. So if we want to provide this multi- aentic system, then we need other agent. And so you're going to create a rules agent which is familiar with the rules of the tabletop role playing game with a knowledge base that you're going to create and it's going to be associated to it. And then if you want to create your favorite character then

you're going to create also a character agent. And this character agent has access to character database uh to store the ability and statistics and inventory items of your character in your journey. Great. Are you ready to implement this architecture together? Cool. So to do it, we're gonna use Spring AI for sure and we're going to use uh Spring AI community uh library uh which provides some elements

and support especially for the MCP and A2A protocols. So it's going to be really useful and in order to provide the brain to our agent, we're going to use Amazon Bedrock which is a service where we can consume any LLM via an API. Um there is a converse API where we can converse with every LLM. So in in the workshop you're going to use probably cloud uh

IQ um most of the time but you can switch to any other LLM if you want to. Okay to join the workshop first thing to go to that specific URL. Welcome. Have a And by the way, if you want there, there are stickers at the front. So feel free to pick at any time stickers and have some drink or whatsoever. So join this specific s12d.comjprime AWS. I

think there's enough account for people in the room. If you're on the live stream, you you can definitely join um by using this URL. I hope there will be enough accounts. [laughter] I'm going to do I'm going to do the same as you. Uh everyone copied the URL. Everyone is everyone is fine with the URL? Yeah. Okay. Took pictures. That's fine. Have a seat. I'm going to

do exact exactly the same just to go through the first steps with you and then you can be on your own. At some point we're going to do checkpoints. So I'm going to uh we have five chapters to go through. Let let me let me share. So you know what? Let me share something. so join the URL and while you are joining the URL, here's what we

can expect uh at the end of the workshop. This is what we can expect. You have a UI that is already developed and deployed. So it's it's running locally. You can you will have to use it at during the chapter five. But this UI can let you play with your multi- aent game master. And how? Well, you're going to have several agent that you're going to make

it up and running. So, a rules agent, like I said, the the that's going to be up and running, your MCP server to roll the dice, and also the game master itself. Game master orchestrator you see these different application are running so all our agent are running plus the MCP server and if you implemented it with success then you can create your favorite character like I don't

know Sauron for uh what's gender also non-binary okay which face elf. Ah, I like it. Uh, which class? Sorcerer or warlock? Warlock. Okay. And so this is the endpoint of your AI game master. Uh, and you can click on begin the adventure. And if you go back to the agent, you can see that the game master is going to communicate with the character agent in order to

create Sauron. So Sauron is coming to life. Um, and it's probably roll the dice or not in order to calculate the ability of Sauron, but Sauron has been saved into the character database. Um, and so the agent can use it. We can see the statistics of our new player in the game. And if we go back, then you can start the adventure and say, um, you welcome

Son to the realms beyond Mortal Ken. You emerge from a twist torn gateway into the ancient forest. Run. And then the AI game master will continue iterate with the specialized agent and summarize the response and send you um so on bolt your elven legs pump beneath you as you crash through the underbrush. Okay. So at the end of the workshop, this is what you're going to get.

you're going to get the UI uh which is already available but thanks to the agent that you you're going to develop you're going to play with it and you will be able to play with it. Cool. So if you join that specific URL u for those that just joined now you should be where um I don't know if you were already connected or not. I'm already connected

to the workshop. Normally you should land on this page if you authenticated. If that's not let me reauthenticate uh if I'm taking that link. So the link that I shared you should be on this page probably. So you can pick uh email one time password. Uh put an email that you're going to use for the workshop. It's not it's not saved after the workshop. It's just to

make sure that you get the same AWS account uh when when playing the workshop. So click on email one time password provide one of your email. Um if you are an using AWS you probably also have a builder ID you can use it or I will use an Amazon employee in my You have the terms and condition to accept. It just says that you are above a

certain age um depending on your country and the accounts provided by AWS are intended for the workshop and that's So you just agree with the terms and condition and you can join the event once you join the event. So you should reach that page that I was sharing just before. Um if you have some permission issue on your laptop uh if you are dealing with proxy and

so on and so forth. Uh I got you. There is a code server that you can use over here. It has no autocomp completion, but you can add uh plugins and it's a code server where the code from the workshop has been already cloned and is available. So feel free to use that one. If you do prefer to use your machine, use your machine. We love when

it works on our machine as developers. So feel free to pick one um of the above. So there is a code server that you can use. And the first thing is the to start our journey al together with the chapter zero. And in the chapter zero you have a strong decision to make. You have to choose your path which path you would like to experiment. So this

workshop uh I've initially created this workshop using Gbang JBang sorry. Uh Jbang is a way to script your Java classes and your Java program. Um, so you can just run gbang your Java class and it's going to execute the program or if you do prefer to do with intelligj there is an intelligj path and you can definitively use it. Just make sure that if you pick one

path or the other to stick to this specific path if you pick JBank stick with Jbank. If you pick IntelliJ, stick with IntelliJ. That's it. Apart from that, you should be good. the first thing is to clone the repo. Um, so the ri the g the git repo to clone it on your local machine if you are not using the code server and then you will see

the the five chapters once you clone the repo. I'm I'll zoom in a little bit because it's think very small. You should should see the five chapters. So chapter without the dash maven it's the Jbang and chapter- maven it's the where you have the same structure that we are used to and chapter one you can see the only difference between Jbang and inj is that in Jbang

Jbang is resolving our dependencies by using this command like um slash uh the Java version. So we are using Java 25 uh the repositories that we are targeting and our dependencies plus the runtime option if you want to provide runtime options. That's the only difference um you using the intelligj path everything is in in our pom.xml file over here. So clone the repo. That's the only plumber

that we are doing. Then you are good to implement and develop on your own. Clone the repo. And once you clone the repo and let me know when everyone is good with the Good. Clone. Clone. Clone. Clone. Okay. So next steps is to get an API key from Amazon Bedrock because we want to do the inference using Amazon Bedrock as a back end. And if you click

on it, it won't do anything. So on the left you have open AWS console. If you click on open AWS console, it's going to open a new tab console and if you type in the search bar Amazon Bedrock or just bedrock, I think it's enough. Yeah, if you type bedrock in the search bar and click on it, click on the name. You reach the Amazon Bedrock console

and on the left, what's interesting is the API key over here. And if we go to API key, we can create short-term or long-term API key. But short terms are valid for the next 12 hours. So it should be more than enough. Um you you can create a longterm if you want. But if you click on generate short-term API keys, you can get it from MacLinux or

Windows um based on your operating systems. You can copy it. You can generate it as many time as you want. No worries. And this API keys will be for example in the code server. Uh if I open the terminal from the code server, it's just okay to pass the API key. So when I'm I will run the Java class, then the API call to the LLM will

work. It will have credentials to succeed. If I want to run the dungeon master simple Java class, for example, of course, that would be too easy. Uh, now it's your turn. Oh, there's a solution. Um, because I tested it before. So, now it's your turn. You have todos to implement. Everyone is good with the API key. You copied it. So, I saved it somewhere in the notepad.

TXT. You're good. Okay. If you got the API key, now it's your turn. Um, so Gbang, I'll let you read the instruction on that one. But if you can start with chapter one, and with chapter one, then your first quest. So we have five chapters to go through. The first quest is to create an AI game master and to bring it to life by summoning it. And

I heard this morning in the session that I intended that it's really really important to read the documentation. So I'm just going to say we have Spring AI documentation that is really really helpful if we want to create our first agent and bring it to life. You clean the report. Oh, you going to the Jbang path. Yeah, you [clears throat] can go can definitely can go. If

you are using the your own machine, you have to install it. Otherwise, uh in the code server that I'm providing, it's also available. Yeah, I think it's You're good with the API key. Chapter one. Okay. You're you're doing with Jbang yourself. Cool. You're good. We met or I think agent camp. No. No. Okay. But I'm sure we met. I You're good. You got the API key. Perfect.

Yeah. Yeah. So you can go to the you can in fact open one chapter per one chapter and it's chapter-maven for the intelligj the one that no the one below you you open this project into a new intelligj instance. >> Yeah yeah yeah exactly it will stack bond the other but it you can do it independently. Yeah. Yeah. No. No. Over here. It's already cloned if you're

using the code here in the explorer. Yeah, it's already cloned. >> No, it's in some uh Amazon account that I'm providing to you. So, it's already provided. >> exactly. Exactly. Chapter one. Yes. And the first class. And and now you have the if you scroll down, you have to do. And this is your first challenges in the in in the chapter one. First thing is to create

the agent and then to summon it. You're good. API key. in the when you you will it can be placed in your environment variable uh in inj for example I think I'm sharing if you go back to the workshop instruction and in chapter one yeah if you scroll chapter so you you need to open this specific project into intellj yeah yeah I think you can open it

a single instance of DJ instead of having the wall project opened. Yeah, it will be easier. I don't know where. Yeah, that way you can run it. And you need to put in the environment variable the API did you get it from did you connect it to the Amazon bedrock console? No. Close close that one and go to on the left on the open AWS And now

in the search bar when it will refresh you, you can type um bedrock. and click on the name and on the left you will have the API key. Yeah. And you can scroll down. Scroll down. Generate shortterm. No, you should have a button generate. Yeah. Yeah. Okay. Sorry. and you can copy depending on your operating system the Linux or Mac OS or the Windows one and you

can save it in a notepad txt then in I don't know if you're doing the okay so you need to put it in the in the terminal yeah you passed in the terminal so you you when you will do a jbang your Java class it will have the credentials from the API You're welcome. Yeah. Yeah. This is what I do. Yeah. Sure. Uh if you provide provide

this environment like it's region that's weird it's asking for it should be the case keep the add a new and equal us west yeah Okay. It says Yeah. Yeah. Let's try. You're using 27 225. I don't think No, it's not related. I don't think it's related at all to but I didn't try. [laughter] No, that's different thing. I'm going to have a look. Yeah, I'm going to

go have a look in my IntelligJ to to let because shouldn't shouldn't be the case. I didn't get it. no, no, no. Close it. On the on on the left, you have open AWS console. No, no, not that one. this one. So, you're going to be on the ads console. I provided a specific ads account per person. So, you can do it on your own. You don't

need a credit card or whatsoever. You can just use these accounts. Uh oh, this come back. No, come back to the link that I we clicked. Click on disconnect on that uh log out. Yeah. To log out, click here. Click on that one and reclick. Yeah. And now it should be you will have credentials. Now you're good and you can type bedrock in the search bar to

get your API key. Yeah. You're on the the account that I provided for the workshop. You're welcome. You're good. Okay. Cool. on your machine. Uh yeah, you can if you pick the JBank path, you can install it. Otherwise, if you're using the code server that I'm providing, Jbang is already installed. So, if you click on on that one on that one above. Yeah. Click click and you

scroll down. Oh, no. Um, sorry. Scroll up. Scroll up. Um, over here. Sorry. No, no. Scroll up. Up. Yeah. Click on the workshop name. Scroll here. If you click on that one, the work the workshop is already cloned and JBang is already installed. Yeah. Yeah, you can walk over here. All good. Okay, cool. Region. Yeah, the same one. And yeah, like I think the same. I'm going

to have a look. Yeah. The value is AWS beer token. The the key is ads beer token and the value is the underscore beer. Yeah. Underscore token. If I'm not Yeah. If you Yeah. You're welcome. Are you good? with JBang you okay because when if you install Jbang JBang also comes with you can pick any default Java version you want like Java 20 this workshop has been

made with Java 25 it works with Java 17 21 23 due to permission. Yeah. But I think that that due to the this is Windows. I guess that's the Windows Yeah. So that's that's why uh you can go to the JBang JBang website. Uh no this is the Yeah. As you can see this is the bash one. So if you go to Yeah. over here. Yeah. Yeah.

Yeah. I think it will be better. Yeah. On Linux you have the PowerShell command. Yeah. I think it's the PowerShell is doing both. Yeah. Just Just open a poor shell and you you you'll be good. Can Yeah. Yeah. Oh, not that one. Open over here. Yeah. If you go So, it's one account for each person. And here you can type in the bedrock. Click on it. Yeah.

API key on the left and you can generate and get it depending on your Windows. you're good. Oh, cool. You're good. Oh, cool. Yep. Okay. And get the API key and Okay. How can you save result? Oh, you can clone it. You can use I You can go to the workshop instruction if you if you and in chapter zero if you Ah um yeah but I can

reveal it but it's also the part of the workshop to try to implement. I I'll review it later. No worries. If you go to chapter one and I don't know which path you're doing JBank and so you have the two steps to implement to code and I'm going to do I'm going to live code the chapter at some point. So so okay everyone is almost good. Just

need to check for the true intelligj and the region. Let me go to my int. The name of the environment variable was missing. Okay. Did you see you see it? It worked. I did the same. What I did is to up um modify run configuration. And then I went to the well where it is um envelopment variable over here put yeah but in fact I think make

sure when the you go you are on the bedrock console um if you go back to the bedrock console make sure that it's it's all okay it's the same okay that's typically the error that you can face because um getting the API key is regional so it's from that specific and if I'm putting you are on DJ and it's is it is it working for you? Yeah,

but yeah, but it should work in Java 21. So, I mean, you got some dependencies issues because it's using Spring AI 2.0, but in preview um as the release of 2.0 We know he's going to he's not yet released. And underscore. We're going to make it. Yeah. Now it's normal because you have to to develop the to. So it work. It works. Yeah. Yeah. Congrats. We made

it. >> Yes. >> So in the terminal you can you can just past >> yeah you can you can directly copy and past the value remove. >> Yeah because the export is already written. So you can just past it. >> No you can close it. We don't care. You need to allow. Yeah. And and press enter. And now now you can do Jbang your Java class your

from chapter one. And you should be you should be okay. Okay. Nice. I see. I like I like this code. I like it. We almost done. Chapter two. Chapter one is complete. Uh, congrats. I Nice. First AI game master is alive. Okay. I think It's because um it's is JBang in your path. Yeah. Now if you do java d- version or java- version I think the we

don't uh wait yeah missing the blank space. Oh it's not it doesn't switch to The val is just to make it permanent in can you close your terminal open a new one just think it's the same I think the command it's already installed put into your go to the Jbank Jang running. We're getting started. Maybe they provide installation and Windows shares. This is for the it's going

to say it's already installed if you do it. I don't remember how to add. Maybe we can type it instead of doing the um add to the path. This is what we want to do. We just want to add to the path. No worries. I'm on Windows. I don't remember. Let's Google it. Type. should hear [laughter] Can you give a try now? Even if it's 26, that's

fine. Let's give a try and run the If you go to to your the G project that you and open the terminal. And now you can do Jbang and tab. You can tabulate. You can do tabulation too. Yeah. No, it's nothing. It's just it's not considered your path within this specific terminal. Can you switch this terminal run pass the command that we made from the installation? I

don't know. It's not being added to your path. It should be No, no, no. Pick the Windows one because you have a Windows machine. Okay, okay, okay. Um, let me kill our wonderful agent. So, I think some of you are have already the chapter one and the AI game master up and running. Um, so if that's not the case, don't worry. Um, but go at your pace.

So, if you prefer to go fast, go fast. If you prefer to go slow, go slow. Enjoy it. That's the most important. Uh, I'm going to do it so we can how we can bring it to life. So, previously it was working, but I'm going to kill it. Zoom a little Okay. Um, so I'm using IntellJ. Um, and I have only the chapter 1-m opened in my

IntellJ and we can see some stuff about the API key. So, it's checking that we have an valid and existing API key in our environment variable. So we can use Amazon bedrock and summon uh an LLM I don't know um like in this case cloud IQ4.5 that we are using. So we are creating a bedrock runtime client which allows us to connect to Amazon bedrock in a

specific AWS regions with the credentials from the API key. We provide a model ID and this model ID is the one from entropic. Then we create our bedrock chat option which is a class that allows us to specify the model we want to use and we can also provide some parameters like I don't know uh temperature if we want to set the temperature of a specific LLM

with our AI game master we can set to I don't know Z7 temperature is between zero and one the closer you are to one the more creative it will be uh and The closer you are to zero, the more deterministic it will be. We have a then a bedrock proxy chat model. This is our final object from Amazon Bedrock and the AWS SDK in Java where we

provide the bedrock runtime and the option that we specified for our LLM. So entropic and the temperature that we set and now we need to use the power of Spring AI to bring an agent to life. So how do we can we do it? We have the chat client class that we can use and uh I'm going to name it game master and this game master is

going to be created thanks to the chat client and we're going to use the builder and we can provide a chat model. So here this is truly important because with the chat model over here in fact you can here we are using Amazon bedrock but if you want to switch to I don't know an entropic subscription that you have an open AI subscription that you have you

can definitively switch in the builder from Amazon bedrock to any other provider. If you want to use Ola when you are developing locally for example then you can switch from bedrock to Ola and use um and consume your agent locally with Olama. Okay. Uh we want to provide a personality to our agent. So when we want to provide a personality we have the default system method that

allows us to provide in fact a system prompt a persona some instruction to your to our agent. So you are oop sorry you are a in heroic and fantasy role playinging game. Be creative and play with um players in the room. Okay. And once the we have that then we can build our agent. So we have our agent here. This is an instance of an agent. Uh

it has no tool yet because this is the first chapter. But we get an instance of our agent and how to summon it. Well simple as game master dot prompt. So we want to prompt it and we want to provide some input to our game master agent. So we're going to use the user and the user is going to be your user prompt. So hi I am

uh I am ready. Oh no I am going on an adventure like Bilbo would say. Great. We provided the prompt and then we need to call our agent and to summon the agentic loop. So it's the dot call method that we can use and to get the result that we want just to display in our terminal. Then we fetch the content and that's it. Good. Once we

implemented it, we're going to print it. IO.print uh game master io print ln response and let's see if it works. If we still have a valid API key. Yeah, it works. Game master says, "Welcome, adventurer." The tavern falls silent as you push through the heavy wooden doors. The bartenders looks up with a knowing smile. Okay. Ah, fresh blood, another soul seeking glory and fortune. I see. Okay.

So, this is the first chapter and you can see how simple it is using Spring AI to create your first agent. And this agent then be can be integrated in your application. It's not coding agent that we are doing. We we are doing agent that's going to be placed into workloads that we implement. So that's the goal of the first chapter and now if we are moving

on chapter two the chapter two is to provide some built-in tools to our agent. So we're going to not we're not going going to recreate the agent but instead we're going to add some tools to our agent. And what's the purpose of chapter two? we need to add tools for sure but in in the end we need to get some information about the uh creator of the

most famous tabletop role playing game which is Dungeon and Dragons. Okay. So let's do Yeah, sure. Uh yeah. Yeah, absolutely. In this case, in this specific multi- aent application, we would set it to one on 0.9 because we want to be as creative as possible. And I would say sometimes we won't know what's going to happen. [laughter] markdown is how the the tool is going to be

it it's like it doesn't share about the reasoning of the LLM itself. It says okay you as an LLM when When are you capable or when can you use my tool to achieve the result with some usage parameters? Um the contents length I'm going to chapter two. So put Yeah. Uh it's yeah slips because in Jbang you need slash depths you see yeah in fact Jbang resolve

dependencies using and then you import it that's that's correct and then you need And this is Yeah. Normally you have a dot tool method that you can use. So this is your um where is your agent being created? Um here you need to provide a tool to your agent. So how to provide a tool? Then tools. Yeah, tools. It was tools, right? Yeah. Yeah. Yeah. I'm going

to showcase. Yeah. Yeah. I'm going to showcase it because matter of time just to make uh, chapter two. Chapter two. Um, how many of you have completed chapter two? One, two, three. most of them. Okay. Um so in the J JBank path uh when you want to import a new dependency in your favorite pom.xml you will add the dependency block. Here you add the slash depths statement

with the ID of the artifact plus its version. So you can see that we are using spring AI 2.0 M4. Um hopefully the release is going to be soon the AWS SDK and then once we declare our dependency like we will do in the pom.xml we import it. So we import our spring AI community agent tools smart web fetch tool. Okay that's great. Then we have the

same thing as in the preview in the chapter one. We have a bedrock runtime. We have uh still cloud IQ4.5. Um we have a chat model, a chat client. So we have an agent. So everything was created. The step we were asked to do was to implement the smart web fetch tool. And how to do so? Well, we invoke the smart web fetch tool class. We provided

it to our agent over here. we can set a content length because we are browsing the web so we don't want to fetch everything on the web and we build it. So once we created the smart web fetch tool object we can just pass it to our agent and how can we pass it to our agent then when we summon our agent agent.prompt prompt dot user with

a new prompt. In this case, the prompt is use the website of Wikipedia and the Dungeon and Dragon web page to tell me the name of the designers of Dungeon and Dragons. And here this is the key things. This we have the tools method that can be used when you are creating you are invoking sorry your agent. When you summon your agent, then you can pass the

dot tools and reference as many tool as you want to your agent. In our case, we we have only one tool and we just print the response. So, importing the tool dependency uh and the smart web fetch tool in this case, creating the smart web fetch tool and then attaching the tool to our agent. So, let's see if that works. Chapter two. Oh, no. Chapter three. Uh,

JBang dungeon. I don't think I have an API key. Yeah. Anymore So, can be a little bit can take a little bit of time depending on the network. Okay, it used the tool. So you can see in the logs I think you got the same like it used the tool uh it trunating to 3,000 300,000 characters and agent response based on the Wikipedia article on Dungeon and

Dragons the designer of the game are Gary Gigak and Dave Arnerson. Okay. So in order to fulfill our request which was this specific prompt, it used the tool we provided autonomously to achieve the result. That's great. for most of you chapter three. Chapter three we have an AI game master. We used a built-in tool in chapter two. Now we want to create our own tool. We want

to create a custom tool. And this tool is a tool to roll do you have I think yeah, if I remember correctly, spring-ai No, that's two dot for the for the uh import.org.springai Spring AI tools do smart web fetch tool. Is that correct? This is what you have versus For those that are using JBang um also if you are interested it there is an extension that you

can install um on IntelliJ or on um on a VS code like and it will resolve dependencies also highlight the the dependencies so it can be useful. I go back here and then I can get the detail. Once the import then I I set on the web page tool declaration. So you Yeah, I think you're good. And then you provide the tools. Yeah. So I think now

three. This is where it also becomes start to becomes interesting because in chapter three we have to import the tool and tool param decorators from Spring AI. So we we did I mean things with built-in tools but when we when it comes to building our own custom tool we have Spring AI uh decorator that we can use to ease the process and that's really really key because

then we will be able to have a dice tool. So we have a dice tool class and then we're going to create a rolling dice tool which is a simply random function but it works perfectly. And our dungeon master, our AI game will need the tool. And I think I'm not spoiling the answer if I'm just sharing that the tools is going to use the dice tools

class. But the secret source is in the dice tools on Java. Come on, I know you can do it. I come back with stickers. Stickers. Mini. Yeah. You think you can take many? Oh, feel free. Yeah, Okay, that's you're good. Or you want additional ones? Yeah. Come on. I think I think there is Yeah. Yeah. Yeah. Yeah. Yeah. Wait, wait. It's somewhere. Chapter three. Who is done

with chapter three? Oh, few few. Oh, nice, nice, nice, nice, nice. So, chapter three, you've seen that you invoke the tool with the dot tools like we we did in chapter two. And in the dice tool class, we used the spring AI decorator to be lazy. Uh let's let's call it um and we love to be lazy as programmer. Um so we have a at tool decorator

where we do provide a description of our tool. Roll the dice for DND game mechanics. Use this for attack roll damage ability check or saving throws. And we have a an existing function that we can use roll dice that's going to roll the dice using random library. But we also have the tool param decorator that we can specify in order to let the LLM knows how to

use this specific tool function in this case and how to invoke the function with the right parameters and if these parameters are required or not. So we can see that the two like the faces of the dice and time we want to roll the dice and it's returning the record about the rolls the number the total we got and the description rolled I don't know 3d6 and

got the following result. So let's give a try. If we go to uh dungeon master with custom tools and the prompt from the dungeon master is like your lady, the mystical keeper of dice and fortune. You speak with theatal flare. When rolling ability score, remember the traditional method. Roll 4 d6 and drop the lowest dice. Okay, help me to create a new character roll the strength wisdom.

And what did I get? So the the AI game master or agent used the tool autonomously in order to calculate the different ability of our character and I got 13 in strength, 11 in wisdom. Okay, not not that really good. 11 in charisma and wow, seven intelligence. That's that's hurt. [laughter] But okay, I'll take it. Um, your character may be more muscled than mine. And even even

the game master is joking with me. So that's pretty cool. Now you have an agent with a custom tool, not just built-in tools. And it's using it autonomously. And thanks to the Spring AI decorator, we can do it pretty easily. So now chapter four. Um I don't know if we will have time because there's only 15 minutes remaining but uh chapter four is to pack our custom

dice wall tool as a MCP server. We want to build an MCP server and to expose this specific tool over the network. And for the one that are doing on IntelliJ, you probably seen you there is a Maven client and a Maven server for chapter 4. So two things because MCP the model context protocol has a client server architecture and so the client is going to be

our AI game master, our dungeon master MCP client. But we also before having the client connected to the server, we need to implement the server. And now we have an application.properties where we need to implement the MCP server configuration and the uh MCP server itself. So it's a spring boot application, basic Spring Boot application, but then we need to leverage the Spring ecosystem. So that's why we

can see the in in integration between spring ecosystem and spring AI because we're going to reuse the spring ecosystem we are familiar with like component for example in order to expose our dice tool over the network via MCP. Let's try to achieve the chapter 4 in the 15 minutes remaining and the environment are available until Saturday evening. So you can still enjoy the workshop even after these

two hours and the GitHub repo won't disappear. It stays on GitHub. So Okay, chapter four. Let's do an MCP server out of it. No, normally this is the dependency. Oh, yeah. Yeah. Oh. Nobody. Yeah. That's so I think that that's also thing. I don't remember in the if you go in the instruction of the chapter 4 just the import because you're doing the application. Yeah. And then

you're going to Yeah, yeah, yeah, yeah, yeah. Good point. Thank you very much for the feedback normally. Yes. But it depends on five. Yeah. So it's like more than maybe two hours. and a half. No, no, the UI is already available and provided. It's a React application. You don't have to The focus is really Spring AI and and agent, not not the UI itself. Yeah, you're right

because I do have the import. I do have the import right here. You were on the client, right? It's running out of time. So, just to let you know that the workshop remains avail the ads account that you get access to as part of the workshop remains available for the next 72 hours. So you can play it, I don't know, during a conference at home. Uh you

have three days to to complete the workshop. I would love to get uh some feedback of you via I don't know LinkedIn for example if you completed the workshop and had some fun. Um just before leaving, thank you very much for taking the time to participate in this workshop. And if you love this kind of content and would like to see more, like we say at Amazon

Web Services, feedback is a gift. So it's anonymous. It says, uh, which kind of content do you like? And would you like to see more of this content? I don't know on on workshops, talks, um, live streams. So any crazy idea that you have in mind, feel free to share. It's anonymous. And so it will help me to focus on this kind of content to improve and

make better workshop or make make better content. So thank you very much. I hope you enjoyed and if you have any question feel free to ask right now. There's no silly question at all. There's no such thing as a stupid question. Uh take stickers. If you don't take took stickers, take stickers and I hope to see you in Sophia or on LinkedIn to be in touch uh

in the future. Thank you very much. Speech. Speech. Speech. Storm. Please check. >> [cough] >> Watch the school. What is My name is Don't worry. >> We will start. first I want to I want to share you that uh that the last 20 minutes I had a conversation with one of the with one old guy from the security guys over there and he explained me that the

artificial intelligence is Welcome uh thank you for coming and uh welcome to the to this webinar implementing uh agentic rock using my name is Yordani Yovkov, like the famous Bulgarian writer Yordan Yovkov, but I'm writing on Jala. But the truth is that uh for the last year uh I only write on Bulgarian on English uh like write instructions and confirmations. so in this workshop uh we will

um we will create uh rack system and we will follow uh some kind of uh some kind of uh real life scenario. Uh we will have uh application some kind of uh simulation of real application about store for uh video production uh equipment and techniques and we will add some functionality about uh rack functionality about this store. Uh from the beginning uh we will create um we

will create uh simple rack so-called uh um knife rock and uh after that the the rack will be uh evolved into a gentic rock where the agent the LLM will will make the will make the decisions from where from which source of data to to gather the data. data from database from um by using uh uh rest client uh API uh to to take the data from

uh from or or from SCP server but uh before before starting the real work uh I want to to to make some theoretical ical base and uh explain what is uh what is rack and uh we will we will uh start this. uh rak is a technique that combines uh the power of uh large language models with some data, some specific data that we have some our

data and in this cases the result is um the result is more accurate and more relevant. we will use more relevant response by sending the uh specific our data. Uh if we look here the standard flow when uh when somebody asks the LM the the flow is the user rise the the prompt after that the LM is thinking uh creating the the answer and uh returning When

we have a when we have a rack uh the rack flow is uh is different from the beginning after the the user question after the quiry before sending the prompt to the LLM. Uh we had uh we have a rack model which um which collects uh data collects data from our own sources. and sends a part of most relevant data that we that we have query from

the from the query to the to the prompt and the prompt is uh the prompt is um is combined with this most relevant data and the LLM has the power to to to create uh format the response uh having in mind that our specific data and about the the question. Now the the rock uh the rock process is divided in two two separate uh stages. Uh the

the first stage is called ingestion phase. uh it is uh the face where we put and collect our specific uh And the second uh the second phase is the retrieving uh retrieving the our specific data from the from the rack with the power using the LM. Usually this um these stages are separated in different logic and uh they are they can be uh most of the time

then can be executed uh separated by execution. For example, we can uh load the data um we can load the data once a week but uh every minute there can be there can be quiring uh query about our specific Now we will take a look at the first phase. The first phase is u injection injection stage and in the injection stage uh usually uh loads the documents

documents uh with our specific data for example they can these documents can be PDF files some our own company documentation some documents u some Microsoft word documents and uh text files so on. And uh the the usual storage of this uh of this uh our specific uh data data in um in the our knowledge base uh is uh in vector and in vector in the vector database

uh there are um there are three three processes. The first is uh chunking the documents. Uh it means that uh that the documents are separated by by so-called little uh little fragments of data by different criteria. Uh these criterias are u uh these criterias depends on the depends on the on the data and uh it is chunk by uh the data is chunk. After that the data

is embedding. Embedding is a process where um uh the data is uh is stored in the the vector database uh as uh how to say it mathematical representation in vectors with uh different Um I forgot the name. And um after that the after that the third uh the third process of uh injection the data in the vector database is indexing. It it is preparing the data to

be extracted. um the second stage of the second stage of the the rack is the real time query processing stage. Uh this uh this stage has three uh has three logical faces retrieval augumentation and generation. This uh this uh the ab How is the right abbreviation? the abbreviation. Thank you. of these three of these three faces local faces is rack. Uh here here the retrieval the retrieval

is is uh is when the data is collected from the from the vector database from our knowledge base and uh the date the collected data is the most relevant data which we which we are acquiring uh and which is stored in this query. U this uh knowledge base. After that uh when uh when we have uh most relevant taken data from uh from this uh from our

from our vector database, the next step is to augment the prompt. Uh as I said before uh for some minutes uh the prompt uh has more data. uh the most accurate um found uh data in the retrieval process. Now the prompt is the question the query and uh with uh I will show the examples and after that the this after this query there is uh here is

the the found uh the found result from the knowledge base about uh about our query. After we pass the the prompt to the LLM, the the last uh the last phase is uh generation. The LLM uh generates uh on base of the prompt and the the retrieve u information from technology base. The uh creates the response. Here is important uh which LM we use. If you use

a clever LED the response will be better. It is obvious. Uh now I will u I will mention some method of rack improvements and uh I will mention uh them uh first and after after that we will go to one of the methods which is gentic. Um there can be make improvements of uh of rack systems. Uh uh the improvements can be can be make u in

uh made in in the different phases of the different phases uh logical phases of the the rack process. For example, here is uh this is this method is contextual retrieval. Uh it does textual info about every chunk. If we take a look uh here here in this uh when we when we write every chunk we can uh we can add info we can uh use the LLM

to to to write specific info about this and uh and in this way there will be a metadata which is uh which is more useful about this chunk. And uh this give us the the ability to the retrieval process to be with better results because we had uh uh structures structured con contextual data as a metadata in every different chunk. another another part uh of the logical

phases where we can we can uh optimize the the rack uh functionality is by improving the query. Uh if we look uh for example here if the query if the query is uh is more optimal is better it is obvious that the the whole result will be better. So, so that um that we can use we can use the LLM to enrich every every query and uh

if if the LM decide to enrich every query um the results will be better here are um pros and cons about uh that the the results of the retrieval will be better but we we will make uh many many calls to the LM. Another u uh way of u improving the the rack uh functionality is uh creating multi queries. Uh this method I will use in the

in the project u and you will see how it works. uh with this method with the help of LM we we create uh several queries according to different data. Uh you will see that uh we work uh with uh our documents in the projects are u are about different uh video editing uh cards for uh for digital uh video editing and uh and uh in example we

will have different several several queries several queries about the about every Another uh way of improvement the rack functionality is by reranking. uh it is uh the reranking is processed after we uh we I will show here after we will have response after we will have u about every query have several for example if we had uh se if we have several uh queries we will have

uh different responses and uh the ranking this process of uh checking all the responses and uh by by additional call to the LLM lm will decide which of the uh returned responses has the best best answer. We are calling uh the llm to decide and better result. And now we are going to our uh to our uh topic uh rack improvement. It is a gentic rock. Uh,

Agentine crack is a way of um have an uh which uh which is uh capable of collecting uh the data uh in the here is here is diagram. Uh the agent collects the data which we query from calling uh data from collecting data from different sources. The agent, the agent is capable to uh to make a decision uh which source to which source to call. And uh

and it is uh it is capable to uh to to make uh as as needed calls to the different sources different sources uh uh until it decides that the the received result is uh is uh well it's well done. so with a gentic rack uh in our uh in our application the gentic rack will will collect data from database from uh different uh application by using uh

using uh rest client API or from um MCP And now after this uh theory basis uh we will we will go to the real project. Now for the real project uh there is uh there is uh user uh users uh workshop guide uh which is published in the in the page of the in the page of the the workshop in J prime. Is there anybody who has

who had download this file and open it? Okay. No, this file is uh just a second. This file is uh 12 pages and here is uh every uh everything described about the now now uh to to say something about the project uh the project is um imagine that we have a real time uh product stock which is video which is video production store. Video production store is

some kind of simulation of real uh real store which uh which uh from here I will show you after a moment. Uh this this application is um this application is uh is for uh um is for um the availability of some video editing and uh uh customers and orders that the customers have made about uh about this and uh and after after that project first first I

want to mention why I uh I had uh choose this uh domain topic about video production uh video editing uh cards. Um I have worked uh uh as these guys uh over there as a video in video production industry between 1995 and uh 2005 year. uh and um and uh I found that uh most of the LLMs are not uh um capable to give a good answer

about uh about video editing uh video editing u equipment and techniques before 30 years. And so we will we will check that the ALM is not capable to give an answer a good answer. It will show and after that we will use uh our uh our specific knowledge base with uh users manual technical specification of some of the some of the cuts before 30 uh 30 years

uh that we used to make a video editing. Uh and uh That's that's why I um I choose this um this uh domain of of the project. I've I have been working as u I video editor uh sound engineer and uh and director on on air in some of the televisions televisions before 30 years 25. and uh now I want to show you first I want to

show you the documents of our uh which we will use in our uh uh knowledge For example, here is uh move machine pro. It is uh it is uh it was the one of the best techniques from 1994 when I was young. Uh another another is uh media stone uh media 100 in English media 100 it was bump top of the video production in uh some of

our televisions. Uh it is uh interesting that uh these documents are uh difficult to find in There is no no data about this techniques and uh DPS velocity mirror motion this let's say 30 and some document about uh different video cards here is the computer super super computer from this time and Uh and now we will take a look at at our base application where we will

start this application is video production uh if I start this application first to start it uh I want to mention that uh there are five projects I will explain and we will go go through them after a while but video production store is is the simulation of uh video store this uh the specific about this project is that This is uh there is no docker container just

a minute. Uh the specific is uh that uh if we take a look uh take a look here here in the guide uh it is explain about this uh this project. This project How to close it? I will open it. And the specific about is that this project is a simple uh rest project uh with uh with rest API for for creating uh four domain objects. These

are video editing card, stock availability, customer and order and uh and uh the data is preloaded in the database uh with um with uh flyway uh the data is uh the data is stored uh the data is stored uh in the database in this format There are four video cards with their their description, the manufacturer and the price of the card and um stock availability shows uh

which uh which video card u how availability there is of which video card. Uh here are some uh here are some customers uh which are uh with fictive names even of better Petro Gorg Gorgf and so on uh with their own data email phone address and notes and uh and there are they have been made an orders orders about some of the products. Uh the orders uh

describe which customers have buy uh different video addicting cards and u on on which date and some note if there if there is some note on the of the purchase uh of the the order. Now uh the here is uh here is the list of uh API I will show you uh the API after well yesterday it worked. Why? Why? Why there is such a problem? This

container is running. Uh this application will will be the base of uh where we will take the data. There is another another problem. I found this problem solve by closing the docker. Let's open it again. My my idea is to to to create the the project. Uh there are seven seven steps to create the project uh and uh the project to evolve in the next sequential space

of development. You will see all the all the steps uh after a while. But uh the first thing is to to start the the main application. It was not planned. I think that everybody uh such kind of moment yesterday it worked. Sorry about that. It started. I closed the docker. Closed the intelligj. Reopened it. It started. Uh, okay. We will continue. We'll continue. Uh, here is a

swagger swagger online documentation of this API. Uh it it give first uh several endpoints about the our four objects domain objects customers orders stock availability video editing cards for example we can test some of them video editing cards it shows that uh there are there are four video editing cards for example the customers just show it and it will take all the customers and so on. Uh

they have been created uh several several endpoints about the about the needed functionality that we uh will use After that uh we had uh we we had uh the first step had our working application working uh video production store and uh having in mind that uh there are uh about uh there there is data there is that data about orders of video editing cards. we will uh

we will add more information from our documents about describing this video editing And now I'll show you I'll show you a demo um uh quick demo uh for one of the one of the complex uh complex uh test scenarios just to to take your attention and after that we will we will start uh start evolve the the project. Now I will start the MCP server. Um I

uh I want to mention that uh I have created uh a project video production store client. This project is uh uh rest client uh which takes uh data from our and uh using the rest way. And this client uh is used uh is used u in uh our MCP server. Our MCP server I will show after a while. Uh is a MCP server which give us the

give us the the same data available as the uh this uh video production store that we uh we found in the in the swagger. But uh but it give uh but MP MCP server gives gives the data in a way to be collected from the and uh here on the structure of the the multi multimodel projects uh in the structure the most important projects that we will

uh we will develop after after a while are rakin guest and rock retriever. These are corresponds to to the two phases uh two main uh stages of faces of the rack. The first is ingestion where we save the data in our vector data store in our knowledge base and the the second project is rack retriever. uh we can uh retrieve the data from the the knowledge base

to to Uh here in the in the in the project I will uh I will open it. I have created uh seven seven steps of the evolving of the of the project and uh the first step no After that I uh I had to show you the how it works. Just just a moment I have started it. Uh I will start the in guest application only. I

will show uh what we uh what we want to achieve by developing uh this project and after that we will start to take a look at uh at every step of the development. I now I will start uh I will start uh racking guest and track retriever projects. uh I will uh load the database with uh the with uh the information from this uh several PDF files

and after that I will retrieve uh information with some specific questions. Okay, the uh the guest to restarted. Uh after after it is started I will reload the data. We will we will go in the details uh after a while. Now I will show you how it works and after that we will go in the implementation stages and its details. Now it is uh it is uh

loading the data in our knowledge base. The data is uh is from these documents, the several documents that I show you about uh specific video editing cards before 30 30 years. the data is loaded. And after the data is loaded, we can start the the project to take some uh some retrieval of the data. why uh anybody can uh can tell me why I have choose to

to make different projects? rocking guest and truck rock suggestions? Okay, I will tell u because uh rack retriever is uh uh provides functionality only to to to gather data from from our uh from our knowledge base or different sources by using the LLM agent and uh it's uh and this project is um it's uh stateless Yes. And uh it can be by by uh creating different projects.

Uh uh it can be uh make uh high availability. It can be scaled. Uh it is uh it is a good practice for designing designing the software. But uh rack in guest tour provides functionality only to only to fill the our knowledge base with data. So it will be not needed to to be available every time. Uh it is uh stateful and uh these two projects have

uh different uh different type of uh and is better to be divided. Uh it uh it can uh there can be created one project uh making this functionality but but it is better to be two projects and for example rocket driven to be scaled. And now now we had uh we had working uh working uh application and we there is prepared document uh with uh test plan

uh test one um this one for for different uh different uh scenarios. For example, uh taking that data only from from knowledge base, taking data uh trying to taking data which is not uh not stored in our knowledge base. Uh uh taking data from simple relational database, from multiple relational database sources. uh from uh relational databases and uh vector data base and I have created this uh

this test scenario and about this test scenario we have created and uh uh endpoints end points uh which can be can be Now we have the data from uh from our uh our application uh from our uh video store uh video production store application which holds uh data for uh for the three domains uh video editing cards uh u customers stock availability and orders. And we have

uh data uh about uh technical information about this video now I will call I will call some uh some queries different queries from different types uh specified here in uh in this document to see how it is working. Uh first first we u first we will ask the the LLM. Uh we will ask we will try to ask the LLM question about some of this uh one

of these video editing cards only the LLM not the not our rack you you will find uh I'm asking the question what is move machine pro this is video eding card from 1994 and Uh the result is take a look at the result. Uh the result is uh something different. Uh for example, the operation system of this card is Windows uh Windows 10, Windows 10 in 1994.

Uh, Windows 11 quad core, Intel iPad, Ryzen PET, 34 GB of RAM. Uh, this times the the RAM was uh 16 megabytes before 30 years. Uh it is uh uh it is obvious that uh that the LLM is hallucinating because there are no information about uh these cards in most of the low lowle. After that I will I will show um I will show uh how I

will uh ask the same question but but we will use our rack. What is move machine pro is the question and here the uh here the answer is that this is a video editing uh card IBM 80. It was a specification between between years been 80 computer 16 megabytes of RAM is recommended. Uh it is uh it shows that it works by reading the data from our

documents. Uh another uh test is we will ask uh unrelevant question about our data. Where is the pink elephant? And it's it tell us that sorry but I I don't have any information about pink elephant. Uh the the rock is uh working properly. Now it will ask a single tool. We will we will uh see it after a while. Show me video editing cards price between uh

30 uh 300 and 500 euros. And uh here here the LLM uh check is checking how uh all its sources of data and uh and uh he uh he found that in the in the video editing cards uh table in relational database there is such an information distracts that information about that two of the video editing cards uh are in this uh price range. And if we

uh if we look at the uh uh I have uh created a tracing in the where uh I'll uh I'll um I'll measure uh um how to say it where we can trace uh the request and the response trace all the process flow and uh here I will sort them yes it is the trace of the of the last query here we can see uh we asked

the question about uh uh video editing card specific uh price range. Uh the first the first one is that uh it is uh calling this API. After that it calls the chat client uh using the LLM and after that the chat client is uh is calling um this LLM. I will show you after a while. And uh a specific tool specific tool uh about uh providing us

the information about uh video editing cards with uh specified price range. And after that the the LLM is u LM is structuring the the answer. Now we will uh we will take a look at more complex scenario. Well the LLM will uh will uh will use two multiple tools to multiple data sources according to the question. Uh for example what has customer Ian of purchase and uh

here the LLM found that Ivanov has make orders about these uh two video editing cars and uh the interesting here is if you if you look at this uh trace trace of the quarry it is different. It is different on the other. Uh the trace here shows that uh now the LLM is calling two different tools. Uh find the to find customer by name uh and by

the name given name Ianov. It finds the customer uh which is the customer uh which is the record in the database. about this customer and after this customer it is checking the orders by customer uh data source uh this table and after that uh uh here we can see that the the agent is capable of make decision according to to our data and our questions from where

to g the data. Most complex scenario is uh here when uh multiple tools the data can be combined uh multiple tools and uh using our knowledge base vector database. Here the question is uh show me the details of customer with ID2 and include the purchase history of this customer and the detail information about the p purchased products and so it give it gave me the uh the

result that the customer with ID2 is Peter Petro he had made this uh order uh about DPS velocity video editing card. And uh here is uh information about uh DPS velocity taken from the uh the uh our knowledge base which is in the vector store. And if we take a look in this uh uh trace of this quarry which is the last one maybe this one. We

can see that here the the LLM is uh is making more more uh complex uh decisions by calling uh different data sources. First uh it checks uh which is this customer by ID by ID2 this customer. After that uh it checks the orders of this customer. uh it founds that uh the the order is DPS velocity video editing card and after that uh it is searching the

knowledge base with uh knowledge base uh vector knowledge base and after that combine the the answer uh this uh This uh gives us uh what we will uh will achieve after after uh developing in these seven steps that I have u I have uh uh created. Now I will close uh close the whole project and uh and uh uh take a look at every step uh what

is uh what have to be done and every step we will explain uh how the development pro process of uh this uh agent in crack was uh um how was the process development process you will see it and uh I I will explain uh on every step uh it depends of the time but I will explain uh uh some specific problems some specific taken decision how how

the some some problems and some uh solving of these problems. the initial state of this project uh is uh multi- multimodel project with uh the five projects uh video production store fully uh functioning uh MCP server and uh rest client fully functioning but the rack retriever and rack u inester are empty. You will see you will see it something uh something specific is that I I choose

to work with the latest available technologies uh uh which are available available now not now but before a week or two um spring boot 4 uh Java 25 last long-term uh support release and um spring AI2 and uh I I can say uh is there anybody uh who who is working with spring 4? Okay. Uh I can see that uh spring 4 uh there are many problems

when you you want to to use some dependencies uh uh some different dependency from different providers or or uh in spring with spring 4 there are some problems be uh because uh from uh from the contributors of uh spring 4 they have decided to to make it uh to make uh it more more uh with more uh granuality, more uh smallest pieces of dependencies, not just by

by writing uh Spring Boot starter and so on, but uh some of these starters are are divided into several inner start several um uh pieces of these starters. Uh the purpose of this decision is to to be the smallest application with uh smallest smallest dependencies loaded in the So this gives this give me many problems but uh with help of my best friend work and your two

friend maybe uh I solve the problems. Uh now I will show the two applications. Rocking gesture is empty. It has only uh only setup uh open telemetry uh which is the which provides us the observability uh which I show here about uh uh system loging uh logging and tracing. It is setup open telemetry setup here in the project and uh the project is empty and racking guest

project the situation is uh [snorts] is the same. there is in the initial stage uh there is nothing specific to empty first uh in the development process the second step uh will be uh to write some to be able to call the LLM and this functionality will help us uh help us to see uh if the LLM is capable of giving uh some answers about different question

in different domains. uh how I u how I showed u before that the is not capable to give good answers about this video editing cars before 13 years and uh here in uh step two of the here in step two I have created functionality that uh uses This is Blink AI calling the LM and uh the specific here is that this is is oh sorry is in

rack in guest project uh which give us the functionality to to ask the to ask our system rack in gto is uh where we ask our system where we ask our rack and here here for example rock. Uh guess retrieval. Retrieval. I wanted to say retrieval. Uh retriever. In our retrieval, we have uh uh we have a controller uh endpoint task. And uh and this controller is

using uh this controller is using uh facade. Uh here I uh I have used uh all the best practice of u separating the data in different layers. Uh for example, here is domain layer, web layer. The domain layer is uh business business layer of the application is separated in LLM specific logic in observability specific logic. Here the web uh the web layer is is separated of uh

controller. For example uh here the facade used by uh by this controller. The facade is uh using the chat client. Uh there we create uh with the help of chat client factory we we create a chat Uh the creation of uh chat client is uh factory and uh and uh here is the the the real client. The real client uses this uh uh uses this uh from

the beginning I uh I have created uh to to uh I have created two types of clients uh sync uh clients and uh their request response clients and u uh stream clients. But after after a while I I deleted the stream because of the uh complexity of the project. Uh so there is uh there is interface uh by calling the prompt and here is the real LLM

client. The realm client um uh everywhere in the application there is a logging on every layer with logging useful information what is happening. uh the client uh is using uh created chat client by the factory and calling the prompt here is just a simple call and uh about about that something specific uh is about uh the the choosing of LLM which we will use uh something specific.

Uh uh every every project is uh do runs docker compose and after the starting of the project they have be creating uh several containers about that project. Here are configuration properties about the observability logging and something interesting is here the just mml uh lm uh module module for for chat and module for embedding. I had uh created uh four types of uh four types of configuration. Uh

only only using u only using uh gro gro uh uh model for chat uh using uh and for example for embedding gro for chat open AI for embedding and so on. interesting uh here is that uh just a second to find it the module. interesting here that the the lambda uh which I use is uh from Gro but Grock with Q at the end. Uh is there

anybody using Grock? Uh it is a very it is a very good uh service provided from this company. Uh I will I will show show it here. Here you can use uh uh you can use uh different uh different LMS uh different models which are uh which are run uh run in the environment of this company and uh there is a free tire which is perfect from

uh uh such kind of development uh such kind of uh uh little application it is perfect it is Uh and uh I have prepared in the the guide that uh everybody who who wants to to run this application can uh can make uh his own account for two minutes maybe one two minutes and to to use this uh uh this is uh something uh like uh uh

something like uh Um, open routter. Open routter is uh I think that you know what is open routter. Uh it is uh um um it is uh provided it is provided service interface for different lms. Uh here you can buy you can buy uh some plan or some photo of uh LM calls top me tokens and uh and you can use different uh different uh where the

modules different models uh uh for with something like an gateway from for using Uh one of the one of the decision was to use open router. Here you can use there is a free tire and you can use um free for different limitation of uh using usage of tokens. Uh this uh these models these models are uh as you can see here one of the best models.

Uh but uh I decided not to use open router be because uh it has limitation uh limitations that are not uh will not uh give me the ability to to run and to develop and to show the application. The limitation are very limit is very low but the Gro is uh is better about uh that uh this company is uh is hosting u uh is hosting uh

free datab uh free LLMs but uh But they they had the they had the environment to to run uh some expensive uh LLMs not uh not to use for example wo on our over computers and for example I had bite uh I have bought uh expensive v uh video cards uh Nvidia RTHIX 5060Ti for using local LMS but uh after that I found I found that the

Gro give gives me the ability to to use a better modules and free and this video expensive video card is maybe not needed about that and uh here here are four configuration for configuration about combination of LLM for chat and LLM used for embedding and uh in in the application properties uh you can uh you can change the profile and after changing the profile name it will

uh uh it will load u different uh configuration about different um LMS and here we are using group chat and open AI embedding we use this this configuration and here I use this model I recommend you uh it's a very good model about uh according to my experience it is free by using bro and uh several times I have reached the the limits of this model token

limits for the day and after that uh I use I use this model this is smallest uh 70B according uh compared with this bigger 120B but uh but uh it gives uh a good The second stage of the projects of the project we had functionality by calling the LM. After that we will go to the third uh third stage of the project uh where where we will

create functionality that uh use that is using our uh our vector vector database which will be in our application our In uh this uh this step uh step three of the development process, we will add uh uh p uh pitch vector uh usage in the project. Uh this uh here here the here we will uh extend uh our RA uh in guest project because uh he is

working uh working with our uh with loading loading data in our uh vector store. And now racking guest store. You can see that uh in racking guest to to see uh take a look at the configuration first something new something new is configuration for the postgraql uh configuration for the vector store. Uh are you familiar that uh uh the new versions of post gray uh had the

ability to provide us a vector uh database and and uh now we are using one uh one DB uh P vector which is PGSQL and extension extension for uh uh vector database. Uh here uh more specific is that uh our our table uh which stores the data the data in the in the PG vector is called vector store. It can be called in different names. Uh another

specific is the uh here is specify the dimension of the uh dimension of the embedding. Uh is a characteristics of the uh of how the data is represented in a mathematical way uh with with uh with vectors uh in the vector database. Uh how uh this uh dimension is bigger. The bigger dimension the better results after uh after saving the the data, embedding the data, indexing the

data and after that uh uh searching the data in the vector now something observability logging uh uh here is about uh another specific uh configuration about this project is about the searching parameter in vector database. Uh there are two most critical parameters in top k andh similar threshold uh tok means uh I don't know if you're uh familiar with this uh with this uh specific uh details

to skip to skip it. Okay. Okay. Uh top k is a parameter uh when we are uh are uh looking and finding searching some information in vector database. uh top k is uh is defining uh uh which is the number of uh top most relevant uh uh found uh found results uh results results in the database according uh according our search and uh top key uh five

means that uh that the vector will return five uh five most relevant results according uh according the data according uh the set dimension when the data is uh is loaded to the the database. Uh and this parameter is very tricky and specific about the about the final result of the rock after uh um the final result uh result. If it uh will uh if the search will

return uh five documents uh two documents uh two results um about the search inquiry two five or 10 uh for example uh it depends on the data it depends uh I will show you after a while with the examples um it depends on the data how the data is structured and uh one thing that I want you to to tell you that uh when working with vector

databases or working with track uh we want uh when we want to create track functionality there is no silver bullet uh to say it. Uh the result depends on the data, how the data is structured and uh how these parameters are uh fine-tuned about the data and the result. Yes. Uh uh it depends of the data how the data is chunked. Uh we will uh look uh

in the code after a while. Uh algorithm of chunking the data uh how the data is uh what is the the nature of the data? Uh and uh here for example uh if I open uh if I open one of the documents uh in our uh uh database for example DPS velocity uh it is uh it is uh some kind of uh document some kind of brochure

maybe it is only seven seven pages and the data the data format Uh here you can see oh Windows media player uh uh you can see that uh the data is structured in some presentational way but if I open if I open uh another for example here in uh this users guide of mirror motion DC30 uh you can see that the nature of the data is very

different. it is uh it is 70 70 pages uh users guide uh uh with technical specification of the video editing card uh about uh about some specific uh installation in the computer uh how to use it. uh the the nature of the data is different. So according to the according to the data source according to our data we can find uh we must fine-tune these parameters. There

is no uh there is no uh one way to no right no one right way no silver silver bullet. it uh the result will be uh defined after some Um that's for why uh there is uh end point specific end point um in uh next step step four uh where u uh where uh according to the the given question uh the question is uh for for example

we had uh four documents with different data structure with different uh data about about video editing cards and uh the data is not extracted from uh all the documents. For example, the LM is decided uh it uh it says oh I I'm using uh knowledge database service for searching and extracting data data from our uh vector database. I found uh here for example five or 10 uh

results uh results in one of the documents and that's that's uh and we if if we extend this this parameter for example to be 20 u here the problem is that u we will receive uh many pieces of data uh which uh can be not relevant of the question. Uh there is no one way to do it. Just try uh with the according to the data and

uh if there is no one uh one uh way with uh one uh that the data can be extracted in uh in one uh with one query. Uh we we can use some of uh of this uh some of these uh just a second uh uh some of these methods uh which I which I mentioned about extracting different uh uh different uh data different nature of data.

For example, multi multi-quiry uh which give give us uh uh with using with usage of several different uh queries according to specific data in the specific uh documents to receive different uh different results and after that on the final stage the LLM to combine the results uh according our uh our willingness to to have the results. Uh and I I wanted to uh to share uh my

experience about this uh this settings parameter. The similarity thresh threshold is a parameter which uh which define how how similar is the the how is the uh is the uh search uh search how is similar question and the data retrieve data for example for example If we looking for a car, uh another similarity will give us bus will give us uh bicycle. It is similar uh it

is moving uh uh give us moving uh way of moving but it is not a car. And these parameters are very important. And here here are the parameters. Another interesting parameters u about the the splitting of the information about the changing process. It is uh for example for example if I open one of the these move machine. Uh here here is uh about the hardware requirements about

using uh this uh video editing uh card and uh it uh says that uh it uh it runs um the minimum the minimum configuration must be processor CPU 486. and now uh and another another specific uh requirement is uh 20 megabytes of RAM not gigabytes megabytes uh and the video card VGA with an monitor and u we are saving the data in the database uh vector database

the uh the process is splitting the data of parts called chunks and uh and for example about uh about this uh paragraph. It is important uh how the data will be chunked. For example, uh it will be a great scenario if this data from here to here is in one chunk because uh after searching in the database uh when you when you write uh which is the

minimal pro processor CPU uh requirement for working with this video card, it will return this uh paragraph. But if the chunk uh uh is uh in this way selected text uh uh the data uh will be not uh extracted in the searching process process uh in a better format. So the the chunking uh the chunking mechanism of uh saving storing the data in uh in uh the

vector database uh is uh is very important in depends of the data. Uh sometimes sometimes uh it is uh for example here is uh this uh this chunk size of uh of uh tokens um it can be it can be uh smaller or bigger bigger it depends on the data and uh I had many times uh uh playing with uh changing the changing uh these parameters and

I found uh different results some good some bad uh uh I want to say you that uh when you're saving storing data in uh vector database to have in mind that the shrinking is uh one of the best uh one of the most important u in Bulgaria. uh uh it is uh very because the data uh will be stored in uh in a good way of extraction

after that and it's depends of the the data and here the problem in uh in rack system is that as I uh showed you that uh different documents have different data and different structure. And uh here is very uh very uh complex and difficult uh difficult complex process to to find uh to find a way to be able to extract data in this uh format this format

uh this format and and so on. So many tries and errors. Now we had uh in uh step three version three we had database uh uh knowledge base knowledge base functionality uh working with our uh vector store. Here the interesting is that uh in in the code we had uh configuration it is not important. Now uh here is the ETL extract transform and load uh process uh

defined here in the service uh it is one service uh B in the business layer working working with the Yes. >> Mhm. Yes. >> Yes. Uh Yes. The AI will use it. But uh here uh we have a functionality only to uh to to make uh some search in the vector database uh for for some testing purposes to to see that it is working. And after the

next step, the step four, uh we will we will uh set up the LM to use the knowledge database as uh uh as its own advisor. And uh here we had a functionality uh votal documents. clear the the vector database law documents uh the the public matrices reload knowledge base uh and um this uh reload uh database uh uses uh this ATL process reader transformer and writer.

The reader reads the data in uh some uh format uh dome format called documents uh in spring AI specific and uh for example document document here this class document it is standard uh as a dome representation of the document. this uh this keyboard is I I don't use it every day and the keyboards are uh replaced. so the data is uh loaded. After that the data is

loaded in uh in the in memory as documents. Uh it is uh transformed uh trans uh transformation is uh splitted uh the data uh with transformation process is divided by split it and uh enrich the data if it is Here is the the splitting of the document. Uh magic uh here this magic happens and uh the data from the documents uh on a final final stage is

saved to vector database uh with specific embedding. And now we have the data in the database. And uh here is a document uh reader service which uh uh reads the data from uh vector store and uh it had an writer. But uh the reader reads the Uh it uh reads the data uh use uses uh metadata. metadata. I have chose this for for information about the file

size, file name so on part of the file uh to to have a meta data in every chunk. it can be useful when we uh when we search in uh our uh vector store after that and in this uh stage of the project stage uh three version three we had uh we had functionality uh here is the test file with the end points we had uh for

reloading the database I So with executing but what is happening connection refuse uh with executing of this end point what is happening oh I I haven't started uh rocking in application uh I will load the database after loaded our knowledge base after loaded knowledge base in our uh in our I will show that uh I will show that uh uh how to search this information and after

that uh we will go to the next step uh next most interesting step uh creating uh nra simple rack. Just a second to start the application. application started. I will clean the I want to to search our vector store uh with this question. What is move machine pro? And uh the result is uh there is no result about this uh question because the vector store is empty.

Yes, it is empty. I will uh I will run this uh endpoint for reload the vector store. It will load uh here this this uh four documents. After uh after uh finishing execution of this uh endpoint, I will call the video m uh this question. And uh we can see that there are answers and uh here there are five uh five uh answers from the from our

vectors vector store. uh five most relevant depending on on this uh top K5 Yes. And uh I will ask uh question uh how to drive a car. This information is not not in our uh vector store. And you can see that uh there is no result. Now stopping this project and go to the most interesting part. Uh Stop it. And uh we're going to simple rock and

rock. Most interesting main part of the project. But uh we had to pass this process from here to here. Now the specific uh the specific uh simple rack. I will show I will show how the simple rack uh looks like to simple rack looks in uh this way. Uh it means that uh we have a query query prompt in uh from the beginning and uh after after

that query according to the query uh we will u we will uh search in our knowledge base uh in it is our vector store. We will search in our vector store about the uh question given in the query. Uh and uh and after that after that with the received information it was uh for example uh in the last stage of the project uh you you saw that

uh that there are five uh five chunk of text returned. five chunk of text and uh and in the the real world uh uh with with the real prompt where uh with the real query with the real prompt and with uh uh combined with this uh five chunk of results taken from our vector store. everything uh everything uh is combined and passed to the uh to the

LLM to the main brain and and the LLM according to this uh receive data this receive uh date this receive data from here is uh creating a good answer a good answer from uh uh for a good response and here here uh we will see how it happens uh with uh rack implementation simple rack implementation here the specific is uh is uh that that we are working

in our uh rock retrieval project. It give us the functionality to retrieve the data where we can ask these questions. Uh our retrieval project I will start it start it and uh when it is starting I will explain what is happening on the next uh next step. Here we must make functionality the when we ask the LLM uh the LLM to to use uh to use uh

some some uh additional source of data it is our and uh according to the given question in the query and the uh and the existing data in the our vector store This data uh passed to the LLM and LLM is creating uh I mean uh result in good format here. What is specific? Uh I'll open the just to to see if it is started correctly. Yes, it

started correctly. Uh I will open the I will open uh first the configuration to to show uh what are the the new add-ons in the configuration. the new add-ons are spring data source. Here is a spring data source. So it is configured the LLM to use this source. You will see after a while uh vector configuration uh here the vector is conf uh configured with the uh

the same dimension as the dimension the data have been saved in uh in our vector store. Uh I had uh big issues uh forgotten this uh with different with different and uh another logging uh here are some configuration specific for the rack or core. the class part of the prompt. We will we will take a look at the prompt. It is very important. Uh the prompt uh

here in the rack the the prompt uh how the prompt is sent is uh are are very important about uh the the quality of Uh the first uh the first uh version of the prompt is uh for example what is move machine pro? Uh it is the the input from but uh but the prompt which is sent to the rack in the in the step uh when

the collected data is sent to the sent to the final send to the LM the LM to be able to produce the result. Uh this prompt is uh is combination of the received data uh good uh given good good instructions what to do the llm according to the given uh received and uh linked data you will see after a while uh and um here here is uh

this par parameter uh again existing. They are uh they in a different place uh but uh had the same uh the same um same meaning. Thank you. Um here is template for a specific uh specific data. We will see it after a and uh here is a multi-quy. We will uh we will take a look at this Here to show uh to show uh how it uh

We we are on uh version four main. but I have created I have created uh on every on every step uh on every step of the development process to have a containers docker containers to be able to to work in the different stages separately. And here and uh here uh to to have data we must start uh once again uh this project rock in gester and uh

load the data because the data is not uh existing in this container. And after loading the data we can uh we can uh use our uh our rack using the llm with uh additional source of vector database. Now this uh this project is started. uh reload the database. It is very important to have data in our our database. And after that um after that uh the B

database is reloading we will ask uh u ask the rock uh with um this question. For example, it uh it give us a good answer about uh the about uh this video editing card taken taken uh from the vector database. Now uh what is specific in the code in the code? Just a second. And the code is specific that uh here we had defined sync client. We

had uh defined uh simple chat. This is the interface and uh we had uh simple llm from the previous step three. Now we had a different implementation of this chat client which will be uh neafra. And uh here the chat client uh from uh springi is creating uh creating different chat uh which uh here the specific is that uh this chat client have an advisor uh it

uses uh it uses uh uh prompt uh prompt but it has advisor. Uh are you familiar what are LLM uh chat client advisors? >> Mhm. Okay. uh advisors uh are uh some how to explain chat client had some uh bodies uh we uh which can uh which can uh collect data from different sources and advisor uh advisor can be configured uh from where to take the source.

It is uh it is to to set the the chat client which will chat with the LLM. uh from from where to take uh from where to take some additional information. And uh you can you can add different advisors uh u different advisors with give advice of the of the chat client uh with uh Yes. Yes. Uhhuh. Uh good question. Uh Mhm. Yes. Uh the the tools

are uh tools uh define uh tools define uh some uh which is described. I will show show in the in the next stage uh stage p uh five uh the tools define uh functionality which uh is uh capable to be taken from u large language model uh by its uh tools is uh something like uh for example something like an API But uh this uh API endpoint

for example, but this uh this end point uh is described with some uh text uh des text description and the LM according to this description can uh uh can decide uh uh the uh yes if the user not to use uh uh to use uh one of the available tools. Uh yes, the advisor will be called always. Uh it is set to the to the chat client

and always will will be called and uh always uh uh always the chat client uh uh will uh will uh will process. I had some information. I have this advisor. Uh let's uh let's take a look at his uh data. I will use it. And uh if there are several advisors, the chat client will use some of them. But uh if there are tools, we will uh

see them in the next uh step of the development. uh if there are tools the NLM will uh check out the tools check uh out the provided functionality from these tools described by uh their description and u and uh we uh we'll take a decision if to use some of these tools only one tools uh for uh Yes. Uh we'll uh we'll tell decision if to use

this tool. And and uh if we go go back here uh here we have an advisor uh which gives additional information. Uh and here the advisor advisor is uh is constructed if advisor is constructed uh with this factory. uh this advisor is uh is uh created from the uh first creation of the bean is is cached and this advisor is using uh this is uh is using

vector store advisor can use different different sources of data. This is use this uses vector store and uh when we ask the llm llm will u llm will uh use every time will in every call will will use this advisor and and but this advisor goes to the retrieves the data. retrieves the data and u and all the collected retrieved data will be will be combined in

the result uh uh from the LM. now we will uh take a look at the Uh one of uh the things uh very important is uh how the prompt is structured here. Here the prompt uh is uh in ifra is uh more simple but in the next version which is a gantic rack uh version five uh the the prompt uh every u every line of the prompt

uh if it's it is different will uh uh the result will be different. I will show here the the prompt. Uh this the prompt for the N rock and uh it combines uh two sources. The original query which is uh for example uh what do you know about move machine and uh uh here it is there is set the context information. It is describing that this is

the context. This is the the context. This context here is inserted uh here uh uh is in uh inserted the retrieve data from the five uh found results uh uh from the vector store. This data is uh is uh inserted here. It is uh big data according to the the results. And after after that the prompt uh is uh the prompt has uh uh how to uh

react according to to some uh properties. uh for example here it is very important uh use only uh answer only the same language or do not answer in the given context language only answer in English for example uh I had u situations when loading uh document for videos video editing card uh in Italian language I found such document and the LLM sometimes gives me the results in

Italian uh sometimes in English uh because it is nondeterministic it is working non-deterministic and you're not sure what will be the result the final result that's for why uh there can be there must be writen some instructions do it only in this way do it in this Okay. Uh do not make for example information outside the provided context. U for example when I ask uh uh where

is the pink elephant uh it uh it was uh the LLM was stopped from here. Don't don't to try to hallucinate to try to find some answer answer clearly. Uh if there is not information contain contain pointed and uh the instructions are very important. Uh they depend on the data. They can depend on the data. As I mentioned that I had a document in Italian and the

results were uh were nondeterministic. So this uh uh this list of instruction can uh be large if we uh if you want some specific uh uh data and uh if you want to improve the quality of the let's go to the most interesting part uh agentic agenting AI uh uh rack aentic In the agentic rack the specific is that uh we will not use uh advisor as

uh it is used here. Uh uh here we used advisor to to advise uh using the our vector in a uh a gentic crack. The specific is that the uh the prompt uh the prompt is the same but uh here we said the the system prompt system prompt uh uh maybe you know that there there are different kind of prompts uh uh prompt uh and system prompt

uh which holds the uh system level instruction. You will see it in this uh in this uh version. And in this version we will we will work with uh with uh database uh according to our uh our documents uh uh no our database from the vector data uh uh video production store. Yes. Sorry I have tired of speaking. Uh just a second to to load the the

database. Here is uh here is uh the interesting part uh when we had um when we have uh uh a and the next steps are uh the gentic rack to use uh HTTP client uh for uh uh to to have a distributed system uh to call the data from another uh from another uh application. And uh the next step we'll use uh MCP The next step seven

uh just a second to to start to load the data and to to show what is the u specific in uh development of this stage of the Uh in this stage we're working uh uh with raking uh retriever uh project uh because uh racking was uh ingesttor was only for u for loading the data and for searching the data but uh rack retriever retrieves the data in

different ways. different ways here. Uh specific is that uh that uh we had a simple chat chat client. We had narack chart client and uh here we will have third uh third uh variant of chat client. It will be a gentic chat client. That's for a while. Uh that's for a while. There is an interface there retrieval application. I have created uh uh this part of the

application to use inmemory database uh to to show the evolution of the the project in the different uh different steps of development in different versions. Yes. Rest coming. Yes. True rest by using uh by using the client uh API one of this project video Just a second. Uh now it is started. I will load the data. end points. Thank Thank you for being here. The [laughter] last

the last one. and eight I was associated professor no associate uh assistant assistant assistant in technical university uh Sophia and uh the situation was the same when I started the programming uh classes uh whole class and uh and uh at the end of the at the end of the class uh only only several several people but more interesting uh more interesting about uh learning new something about

programming. Uh and now uh now I have load the data and the specific here is uh that uh here we had uh specified domain object uh uh describing domain object of video The the project is uh evolving and uh is uh and here are the the main objects. we get LM response. Uh I will show you after a while. Um the service uh here we had ser

several services uh four services uh for working with uh these uh domain objects. Uh for example, if we look at this service and uh uh we see the methods uh with public uh visibility uh get video editing card is and now we had persistence layer. We had uh we had here uh here we have a persistence layer with uh with defined uh defined uh interfaces repositories about

uh every uh uh about every domain uh that is the interface of working with this object and uh we had uh implementation inmemory implementation but in uh in the next version of the project version we'll hit another which will be uh implementation uh by uh calling u here uh here in the repository we will we will call the rest uh extract data from uh view production store

by rest calling. we will take a look at the configuration to see what is new. A good uh good practice is for uh well uh well definfined structur structured configuration and well documented everything is easy to to be here. Uh here are the new uh the new entries of this uh configuration uh about Gent uh system prompt template. Where is the file? We will take a look

at this. we must uh take a look at uh how is is just client is created uh a gentic crack chat client uh client uh don't use advisor as the nra it this without advisor and uh factory. Uh it it is uh creating uh different advisors. And here from the for the was uh I wanted to show that uh by qualify chat client. Uh I wanted to

show that uh here is not advisor in the chat client and uh the specific is that the chat client uh is using tools here Where are the tools? here and the tools as I mentioned uh are some provided functionalities uh which are described about the tools is very uh very important to describe uh what functionality uh this method provides to be described in a good way the

LLM to be able to choose the right tool. To choose the tool, first to choose the tool, to choose the right tool and uh to to pass the the parameter if there is a parameter and uh to to make decision to use and execute this tool. Now all the logic is uh is uh move uh in these tools. These tools are separated separated according to LLM. How

to use how to uh what kind of business logic we had according to the the questions that given to the to the our to our system. Uh and for every for every different uh uh different uh domain object we have uh Uh this these are different uh uh different available uh think it has an API which provides some uh information uh like a service or like an

API which can be can be called from the module and it holds service uh some kind of service data. For example here this tool this to get customer by video cart ID. Uh we'll use uh customer service. Uh you you can see that use customer service. uh it uh gives uh if description gives the opportunity this tool to be called and after that uh after calling to

the LLM here is uh the business logic uh it will call the service call service and the service will uh will create uh will uh do what is needed. For example, here it will call the repository. Uh here the repository is from inmemory database. But in the next version according to the configuration uh different uh different beans are created uh with different and uh will be called

repository by uh uh by extract uh which will extract data from uh with the help of uh HTTP uh rest here the specific is But in a gent there is uh no adi there are no advisor uh only tools and uh the LLM make decision which tools to call and we have tools uh for four of our domain objects and one tool uh which is uh capable

for searching and gathering information in our vector store. Vector store is uh additional to additional source of data. And uh the interesting here is uh if we if we uh execute just a second if we execute uh some uh some uh searching query for example So this query which customers have purchased the DPS velocity and what are the uh technical specification of this card the LM make

decision we will go to the traces traces traces here I will Okay. Uh the technical guys uh uh are sending me a message that the time is over. thank you. Thank you guys. Thank you everybody. I think uh you can um pull the code from the repository. Uh did you stop the recording? Uh if you if you want to to to uh to uh to stop because

the time is uh over uh because we are over the limit. We can stop. Uh if uh if not I can uh I can continue if uh this I can hear Okay, I will continue. Uh, I will continue. Thank you. Uh, Now uh now uh I will show the traces the traces of this uh request. It was this is the the last this is the last one

here. Um here you can see uh what is happening uh under the uh how to say what is happening. Uh now my question my question was uh which customer has purchased the PS velocity and what are the technical specification of this video card? Uh it means uh it means that the LLM was uh first was uh check if uh if there is a video card DPS velocity

and uh after after uh after uh realizing that there is such kind of cards to check about the orders uh who who customer had make the orders. After that checking the customer and after that uh uh with the name of DPS velocity it will check uh our knowledge base if there is an information about this video card and uh uh it is what is happening uh in

the uh traces of the the flow uh chat client uh is called Um the model is called and uh the LM is making the decision what to do. Uh the first the first thing is checking video card by name. Uh uh there is uh created tool uh in the uh with this uh for this domain video card to for this operation describe describe to uh this tool

is going to repository uh repository uh here in the repository I had uh put some markers that the repository is in memory repos. Uh and after that after that uh it is extract extracting the data from the video editing card. Uh after that the LLM have the information about video editing card and it is uh it is uh uh making decision uh which uh tool to uh

to call according to the the first query. The next tool is uh customer by video editing card. This tool is working for with customers and uh video cards and orders uh join of these tables in the in our uh in our relational database uh database. Um next time uh using the repository in memory repository after that is uh finds the information uh for uh customer order and

and uh video editing card uh DPS velocity and after that uh the model is uh is u has this information and what who wants to take uh uh uh hardware specification about this video card and and uh the uh the tool uh which uh here the important is uh the description of the tool uh description of the tool which uh use uh which uh use the which

provides the searching in the our vector store and the import I will show after a while is uh in the system prompt. uh there is writon uh if uh if there is a need for some technical information ask a while and here you can see that uh this operation is longer because uh it uh it it makes uh in the vector database search it is longer than

the here this operations which are uh in the in-memory uh structure of uh relational database and after that the the information is uh combined information is combined and it is received in a good way customers who purchased uh DPS velocity to customers uh with information about uh about them and technical specification about this video editing card Uh now interesting is to show you the uh here is

the system prompt. Uh I mentioned that uh there are tools for everything for ging every kind of information. There are no advisors there are only tools. Uh but here in the system prompt are given the Um the rules how to use the tools. Look uh look there uh description of every tool what is uh providing for example knowledge database tool knowledge based tool detail technical specification of

videoing cards. Uh the other tools prices customer details and so on and the instructions are very important. these if uh where was it? The language uh multiple when question requires multiple sources count the necessary tool sequentially and uh combine the result. Uh there must be do not repeat the search with different queries to prevent having a cycle cycling. I had uh this kind of issues. Uh if

here uh uh for example this this line is missing uh the LLM uh is making cycling one to another two one to another two uh very specific according to the to the data and to the data Here the settings are very important in how to manage uh the work of the oh sorry uh manage the work of the LM by which tools and how to use the

tools something important Uh maybe not. Uh what will be in the next step? In the next step, uh we will add we will add uh another implementation of uh of uh repositories which will be repositories uh which will use the uh the packaged uh uh rest client uh which is video production store client according to uh which use the rest uh the rest to uh to go

to to the and uh here's the specific in the next version just to show the code not to start uh the project to show the code is the persistence uh second implementation uh http client we had a dependency in this project dependency of video production uh uh cl uh h rest client and uh and have uh a layer here a layer uh which use this uh which

use this uh uh this client and here for example instead of uh inmemory application uh in memory implementation of the database customers if we if we take a look at uh one method find all customers is using uh uh is using uh is using uh the our data structure memory data structure but uh find our customers uh with this implementation of the repository it will be different

it will use uh find out it will use our HTTP uh client our HTTP client uh find out but this client is defined uh this client is defined somewhere somewhere here in the code example customer This client is uh is defined uh in this uh tools. Um and in the in the last the last version the last version is uh uh the last version uses MCP client.

The MCP client is uh defined in the rack retrieval uh uh application and this client is uh MCP client is set to to call remotely remotely client uh uh remotely MCP server and the MCP server application must be started. It uses it uses uh it uses uh internally htt uh v video production store client which is rest client of the project video production store uh which must

be and there are configuration properties uh uh with within uh we can uh define if the application can uh can use uh um must use MCP MCP tooling uh from remote MCP server or or must use local local MCP tools. But the local MCP tools had a inner configuration to uh to to choose which kind of uh repository to have local repository uh or uh repository which

use uh HTTPS client from uh by taking data from the other project. In this uh seven steps uh uh showed uh seven uh uh six uh six uh uh did I say six? Sorry, sorry. Seven. This seven step showed the evolution of development of this uh uh of this genic rock. I think that uh this This is according to the given time. Thank you. Thank you guys

for the given uh time. Uh I think that I'm very detailed in information and the >> Yes. Yes. [snorts] Thank you. Thank you for being here.