jPrime Conference 2026 - Workshop - Day 1

This talk focuses on the integration of AI tools in software development, particularly in the context of Java-based projects. The speakers, both experienced architects, introduce a workshop where they explore how to leverage AI to enhance productivity while developing a code trading system. They discuss the importance of maintaining a 'human in the loop' approach to ensure robust decision-making, particularly during the planning and implementation stages. The workshop guides participants through analyzing product requirement documents (PRDs) and emphasizes the use of specialized agents to handle various tasks effectively, leveraging tools like Qual and cloud code. Ultimately, the session aims to share best practices for optimizing software architecture in regulated finance environments.

and everything that is required in order to for this workshop uh in that post you also can find two keys you're going to need those keys you don't have power and you don't have you can just use those two API keys data submarine that we talk. So if the finish we're going a little bit more but with that the key is going to be after this talk.

So it's going to be available only during the talk. Let's quickly introduce ourselves. So my name is Stephan and I'm currently in major also I'm an architect and doing a lot of stuff especially in Java. So do you want to introduce yourself? >> Yeah sure. here and I are actually working together for almost a year now. Is that mine? >> Y that was yours. Uh so he's

the core trading guy and the everything else guy. So I have all of the satellite projects around the trading and it's um it's interesting to know that we've actually not written a single one of code by hand since I've done a lot of different things over the years. So just >> yeah so the funny story is that one day one our tri come to us and say

okay I built that without touching and writing anything and we was in are you kidding with us how you do that and basically present that to us and we were so surprised that this is really working so we have code trading system which is designed the best possible way in the market based conference that I watched and it's working and basically it's written by Qualcom following those

patterns that we're going to discuss today. What is this workshop is all about. It's not just a talk. It's basically presenting our flow and we can do something together. So there was a document called PRD uh that is written by code especially. So we everything is written by AI nowadays. So we're going to go over that document. We're going to try to understand and see what the

document looks like and in which cases that is going to work in which cases it's not it's not going to work because if everything is document in the one PD file we're going to enter something called waterfall. Yeah, we don't want to be in those uh stage anymore. >> So PD is not a PD if it's 200 pages long. >> Yeah. should be something focused that could

be delivered in around spring or maybe and something else that we need to speak here. So the question is what the agent could do without us and what the agent couldn't do without us. So we need to be there without the human there is not going to be a problem. So we're not vipering we are doing programming which is followed by our restrictions. So we are experts

in those domains or at least we want to be experts in those domains and we provide our context in order to build up. So this is what we are doing what we are going to do. So we're going to go over market file for cloud which in our case is just a Java web hook receive project and we're going to update that look that what is there

what should be there what kind of information we need to put in that qud file do we need to put everything inside it or no we are going to see or also we're going to see in which cases the agent should ask us it. Yeah, that's the most important piece in which case the agent should stop working, ask us a questions or we can ask a question

back to the agent and see how we are going to move right after that. Uh what about scenarios? What is our app? So uh there there was a woman called Maria. It's a payment PM. So somebody who understand the domain at least we are hoping that the PMs are always understanding the domain. So they can provide us a document which describe what we need and they have

integration with system called paper. It's not something else that sounds similar to that. It's paper. So and is broken there. We are not receiving Quackbooks. Uh there were some payments that have not summarized in our system. So everything it's a big mess and the PM wrote that how that system should work in all of our cases or at least how supposed to and the main point is

that the PL is not contain any technical information it's contain a business idea it contains some functional and nonfunctional requirements and everything else it's up to us. So it's only business related topics. Okay. And what else we're going to do? We're going to provide the entire context to the cloud code and the cloud code is going to do the rest. Okay, about the architectures. So we need

to decide couple of stuff when we start doing every project. We need to decide our technology stack. So if the if you put everything to be decided by agents, we will not know which technology is going to be chos. So we h we're hiring very expert Java developers. So if our product is written by uh language like C# it's going to be a mess. So we want

to know which language which technologies are going to be choose for particular cases. Also we use different technologies for different scenarios. For example, if you need to be fast probably we're not going to use spring. uh if we are going to have some case like just rest just websockets or something like that spring it's good technology probably not kafka spring so think it's very good in those

cases >> he mentioned Kafka >> yeah I did that because of Georgie uh what is what is the agenda for this talk so first we we introduced oursel already we are going to walk you over the PD we are going to see how qual MD looks like and we are going to start with you to build that feature. We are hoping at the end of this talk

the feature is going to work or at least the big part of it and you can try to finish it in your home or during the working hours when you work. >> But the main takeaway is going to be taking away the process the patterns worked to establish for about a year that sort of work even in regulated finance. So these are the lessons quote unquote. >>

Okay. So let's move human in the whoop bit. So where we are storing our events, where we are storing our metadata because you know the context is short. It's around 1 million tokens nothing else. We need to store our context somewhere. We need to keep track of all of our decisions in the system. And we know we need to know in which file what kind of information

is we are not going to be able to reread everything and put it back to the context because we are limiting on that also something that I really hate when I was an architect. So building architectural decision records so writing that in man by hand it's very boring story. So we're going to use qual and our architect agent to do that for us and basically get all

the benefits of those documents but also we need to find where to where to store them. In the most of the book from the architects there was a repository which bunch of those documents but we need to keep the keep everything close to the code. Why? Because the agent have access to the repositories and they want to get everything from one single place. So we're going to

keep all of our markdown files in the same place where our code is in order to make them available for our qual code. Yeah, for qual code to read them, understand them and help us to deliver our features. Okay, I think it's your turn. >> Yeah. All right. So here's the main takeaway. Um, one of the big issues we found was if you let a single generalist

agent do all of the work, you get a regression to the mean, you get a lot of decisions that become implicit rather than explicit with the human in the loop and you get basically something that's subpar. So that was sort of the reason why we went with the idea of a cast like different agents specialized in different things. It's not nothing revolutionary. We've just been doing it

for about 9 10 months now and we've been iterating on I think a roster of about probably 100 at this point in time in our day-to-day work. The roster here is a bit more limited. Uh but we are also using it to illustrate the point which is if you add agents that's just a very very small lens on top of a very big pyramid but it still

helps. It's still a lever that you can pull to make it specialized. And if you put sufficiently well-defined boundaries around it, what you get uh as output is a more honest system rather than something that's just you know in theory less capable and limited and we do have a small roster like I say here but let me try to illustrate what I mean when I say well-

definfined boundaries. So, it's not just the the titles. Obviously, this is the idea of the development team split in several MD or YAML files. Depends on what you use. Let's start with the product owner. He's going to be the one who's going to be helping us figure out if the PRD is actually any good. So, Maria has given us something. We're developers. We don't know what what

we're expecting from that PRD, right? >> Yeah. Let's ask the question to the people here. How many of you guys are getting the BD documents or the requirements which is always correct or it's written well and everything it's clear from the first time when you read those documents raise your hand somebody at least one okay George just raising hand so probably it's working in different company than

me unfortunately not but the case is that the the business guys are never going to give you the document which is clear and represent everything that you need to know about the feature That's not going to happen or at least I know only one PM could do that in my entire life. So yeah continue please. Sorry. >> So here is the specialization. They analyze the functional requirements.

They read, try to understand, they discuss, they brainstorm with us. They don't care about any of the rest. One notable highlight, the solution architect. He's read only. He doesn't write the code, but he's the guy who's putting the the line in the sand about which decision can be made. And he's the guy who the backend developer or the QA engineer or the DevOps engineer is going to

ping if they need to make a decision. So the specialization and the rows and the boundaries and even the two axis is what you need to define to actually be able to execute on a couple of loops that we'll get to later. >> Just a question, can they communicate between each other? >> Yes. So whatever you use as a harness, cloud code supports native teams like that's

what we're using today for just because it's the path of least resistance basically. But during a feature execution, they can communicate with each other. The agents themselves can't actually raise and block and stop stuff. They are advisors. The commands are going to be owning the blocks and the quality gates. We'll get to that in just a minute. Let me just highlight that all of these agents, and

this is why it's actually a system rather than a random assembly of files, they have refusal points. So things that they say they will not do basically when they throw the exception right we have the escalation paths are go to which agent the agent is going to escalate the issue project the project rules that we define in cloud MD are going to be something that they never

subvert in in theory because this is another then we also have discipline we have evidence we try to make this as deterministic as possible able even though it's like I said a very small lens on top of a very big thing. So the major thing we want actually out of this uh particular system is that the human in the loop discipline is never broken. When a human

is required to make a it has to be raised. So that's the outcome we desire and here's the pipeline we have to actually make that possible. So if you think about it at each and every stage of this, so let me just probably don't see it at the back rows, but we have the PD. We need to make some decisions about whether or not the PD is

good. We're going to ask some of the agents for help. But we get to make the decision. Once we're happy with the purity, we then move on to the next stage which is just generate the epic decompose the purity into n number of epics. And again during that phase of fanning out the requirements into something that we can actually work with. We need to have checkpoints, quality

gates, human in the loop at every stage that actually matters that's going to make an impact on the output we get out of the whole thing. The epic gets broken down into stories. Again, we validate the quality of those And finally, ideally during the execution, we also have decisions we need to make. If the inner loop doesn't raise any alarms, if everything goes perfectly as planned, that's

unrealistic. That's not how anything works, right? >> Yeah. Before moving to that, let's uh could we return to the previous one for a second? >> So, what story really mean for us? What epically mean for a class? >> Oh, that's a good point. So, what's a what's a pure to you guys? We we said it's not 200 pages, right? We hope anyone. >> Yeah, it's a so

product requirements document, right? How how big or small should that be the requirements for product? Go ahead, That's a very good assessment. Although it's going to break completely. So we end up with let's say 200 pages. >> That's the entire product. What we've found to be working much better is we write a peer deeper feature. >> Yeah. So it's in the first time when we start working

with that our RPM come to us provide us very big documents and basically okay this is everything that we need to build for this product and okay but how we going to deliver that when that should be done probably in three weeks two three months everything should be delivered and you're using AI you should be build it fast okay how what that really mean for us it's

entire product okay But who's going to read that? You. Then the next point together with uh how we're going to read that how many weeks we're going to need in order to read that to understand the product because we we need to fill out the products as our own product the only way the whole point and this is sort of the flow that uh I just described

a bit but we have defined in the folder in real and we have the a list of open questions for the human in the loop i.e. everyone present here. Then we triage. Basically, we figure out what do we need to answer now? What decisions do we need to make right away? Can we defer something? Let's say you're actually building an MVP and you don't want to answer

all of the production high availability questions before we've started building anything. >> Yeah, removing something is good. So, moving something out of the scope is the best things that you can ever do in your Yeah. Sorry, I'm I'm always removing something from the scope and OPMs are hate me. Yes. >> Very good question. Looks we have architectural decisions here. Yes. So that's the second stage. The the

idea is that the that architect over there the solution architect is going to provide you a brief architecture of what should be done. It's not it's not need to go into details how to implement it. It's going to provide information like you need to build a web hook probably which database is good enough your case. It's going to provide information how program should be deployed or everything

out how many services only the overview of the system but it's not going to go in the details it's not responsible to the architect to go in those kind of details and yeah so think about working with a good architect right so maybe Stephan if he were were you good when we were anyway so working with a good architect what it means is he's the guy who's

going to be translating the business vision but not prescribing the implementation details like he he's sort of a bridge between both worlds if he does his job very well. He gives you the rough shape. He you know sort of maps the topography of the whole thing and he also puts the boundaries around the system otherwise it's just something that's never going to be finished. So we want

to achieve something similar here so that he is at level two once you actually have the business requirements one or more epics and backwards again. Ideally, if you do it well, you can trace every story back to the PR and to performance. You can also trace every technical record, every technical decision, sorry, architectural decision to this stage here of this workflow. And something to add to that

because this is the most important piece and basically use in our daily work. The way We go to the production Wednesday be shipped. real QA as well to build the agents for the QAS but yeah that is the main point so we like to to stick the stories to something that is deliverable >> if you don't actually decompose them properly you end up with the the 15

20,000 pull requests like 15 20,000 line pull requests. >> You want to decompose into small chunks of work like a real human would actually do it because you you get chunks in, you get small chunks out, easier to review both by agents and humans. And then like Stephan said, ideally you also optimize a system like this for continuous delivery. You get all of the benefits with trunk

based of something that you know goes through an automated process end to end like that's what we're targeting as well which it's a bit more difficult to achieve in regulated finance than you know let's say >> that was the reason why we can return back to crypto some days >> yes uh I think we stuck around in this light for quite a while but it's >> move

forward >> this is probably the takeaway from our workshop like the theoretical one. This is the this is the gate I mentioned several times already. So whenever we reach a point where we need a human in the loop, we need to figure out what the boundary is i.e. detect it. We need to discuss multiple possible options. So it shouldn't just be should we go with Kafka or

>> or something better? >> Yeah. probably. >> So obviously you need options, you need the architectural trade-offs of each of them. So that's why a lot of the uh level like during the implementation phase, a lot of decisions might actually have to be revisited. You might have to bubble them up to the architect. And if that doesn't happen, you're probably haven't made a perfect plan. is just

somewhere in the inner loop in the implementation something's missing because you know this is an imperfect system like the LLM underneath is you know random token generator we block whenever there is a human decision required you have to basically halt and catch fire right the the LM can't assume anything it can't go on without looping you in or at least we don't allow it to. And obviously

when we make a decision, we record it for posterity. New generations of agents get more context. They can figure out when, why, what, and that's it. Uh, one major caveat, I think I very briefly glossed over it, but the agents themselves can't actually run the block. they advise in the sense they just propose something and the command itself the way we framed it is that's where the

block is actually raised that's where the room is going to be pinged guys um what do you think about this one should we actually use Kafka or not >> and then the agents >> the always answer to that question is no if you don't need it >> the agents unless they are at the implementation level I think QA engineer, back end engineer in in the roster we

have today or DevOps, they they don't really just throw random stuff around. They're the only ones who can edit. Yes. But um yeah, most of the the agents we've defined as read only. again, I sort of highlighted that we have an outer loop, we have an inner loop, and Stefan and I have previously talked about this at at a different call, but we tend to frontload the

thinking. Basically, the outer loop is where we block and we loop in the human on stuff that's blocking the planning stages, right? So if we go back to the feature level PRD, we do a lot of heavy thinking, iterating, brainstorming and planning, what we end up with is something that's as close to the vision for the system that we can probably get without writing a single line

of code. But there is always going to be something in those plans that's not crystal clear that is going to be explicitly noted down as this is going to change during implementation. And that's normal. That's how it should be because then when this when when the agent, let's say the backend agent hits a decision that doesn't make sense during implementation, it shouldn't just make a random guess

and go on. It should l back to the human >> the architect. >> The agent responsible. >> Stupid question to all of here. How many times when you read a story there was already defined technologies and you're thinking that that particular solution is not going to work. Does somebody be in that case? I personally been in that case many times when I was developer. The architect provide

me everything and think that everything is ideal. But when I start working on that I found that And what you going to do in that situation? Do you going to escalate that to the architect or you just going to do in our proposal in our way of working with the agent? Basically, they should always escalate. We shouldn't agree decision of somebody who never to touch a code

at least as an architect. I have a situation when two years I didn't write single line of code. I just know all the technologies but that's it. But the people who are writing those codes know better than me how that should work. So in our agent 4 we always try to escalate back in order to get that iterate iteration that whoop that rub mentioned and the main

escalation part is here in the room. So it's basically the human who is taking the final decisions. Sorry for interrupt Google but I was in that situation many times during my work. >> It's a good clarification because otherwise what you end up with is code you haven't read but also code that's subtly wrong and you have to then dig up stuff at 3:00 a.m. in the morning

when it breaks in prod which no one wants. Um, so let's get to to the stuff we've actually chosen for our small um, it's a pretty small demo today, right? Like one thing we've noticed and probably like the whole industry has noticed at this letting the LLM just randomly read files and try to map some some code graph doesn't really work that well. A couple of tools

on screen. Um, the one on the left is something I've actually spent a couple of months working on and it's become a bit of a monster project. So, it has repository DNA, automatic context. It basically uses deterministic tooling to build a graph of everything. Uh, code index, similar idea. Both of these just work deterministically to build the context for the LLM to use. So you don't get

a million tokens of grapping and reading. >> Could you guess which of those tools we are using in our company? Right or or uh left? >> Why? >> No, because it's built by Rubo. That was the reason why we're using that. >> He said right. We're using the one on the left. >> Depends if I'm here. The right is the correct one. Yeah, no joke aside, >>

enough enough self-promotion though. Yeah. Um, but here's the the big kicker for me personally. Uh, because we all know that all of the frontier models are just doing strip mining right now. If you're not using something to optimize the output tokens of all of the random commands, you're just going to burn through your budgets very quickly. So, highly recommended. you want to take off. >> Yeah. Okay.

So there was some other stuff that we are basically using in order to build our context or it is to save our context and I was recently a situation when I'm working couple of hours and my context basically go over 1 million of tokens and everything go down. I need to I need to compress. I need to save that and what happen if my MacBook crash in

that time. So cloud code save some part of it but it's not enough and there was a tool which which we can use to s to sort that out and basically help us those two are qu and also memory powers they're basically almost the same things but in different way so in the one like memory powers you you basically say save that save me that I'm going

to need that radar in the other side with quadme it's a capture So everything which is our tool in the world to AI basically mean could catch up something that we are not going to need anymore. So it's basically proof of cons what we're going to use. I'm personally I have the issue with that but I'm not using those tools because I'm lazy. I don't have time

to install them. I have my team which is doing the work for me. So I'm just engineering manager. Tell them do the work and they're basically using those tools. Most of my team are using memory powers. So it's good. I'm going to suggest that to you but you need to know that they exist and basically you're going to see which two is better for you and we

have some kind of comparison between them. So we have frictional continuitities like memory out everything it's out everything it's done automatically and we have something that we decide when and because we're going to talk about human in the loop probably we're going to use the one who is basically other human in those interactions or memory powers but there was a cases in which we need to ensure

that everything is captured and always used in our project and this is the place where we use quda By the way, um just wanted to highlight most of the papers nowadays, they're going to tell you that you you need just the right amount of context. So memory palace is better to achieve that. Yeah. But if you need to enforce the cloud code to do the work that

you want, for example, in my case, I should write low latency Java code. And basically in the quality file I specified that you always need to use the skill called what was the name of the skill latency Java skill let's name it that way so I always provide that because my code should be written in the warehouse how there is no new anymore they should be buffers

there a lot of stuff that should happen in the system so I provide that information as cloud MD as a requirements to how the agent should write my code. What we're going to need for this, so we are speaking for about 40 minutes, but we need to start basically doing the workshop. So, you're going to need couple of stuff. First of all, Java. We're in the Java

conference. So, you're going to need Java. The next thing that you're going to need, it's Maven or Grado, depends. Uh, you're going to need Quot. So please install that right now if you don't have it. Also for those who have a subscription it's good. For those who haven't there was a a two API keys in the rep that we share in the beginning. Please use them. They

have some preferent money so we can use them and throw them away. This is the link. There was no QR code but let's me try to find the QR code one more time. If somebody missed that uh let's go back back. Okay. Uh let's take one minute for everybody who is interested to scan the content for us during that workshop and could somebody confirm that this is

public repository because last time I'm not exactly sure. Okay, it's public and you are able to corn it. Okay. Amazing. So didn't do mistake when I asked that. Amazing. Okay. So let's see what was the our was right in order to move forward. it's probably >> how many of you have Java already installed or you should install it right now. Okay, we're in the Java component that

it's normal. Maven or grad, >> why? >> Very good answer. But in most of the project somebody already took that decision for you and just tell story we used mail in one of the company that I work for and we hire new architect and that architect come to us okay let's move together and ask okay why no answer again why to do it no answer but in

reality you need to sell your idea if you don't able to sell your idea you're not doing anything and this is the same come to come to me it's a very good hashtag and tell me okay starting our services to cotton and I was uh why because cotton is better than java and this is where the conversation basically ended when I heard that but in reality if

there is enough points why not I'm personally writing right now go some kind of typescript you're writing English Yeah, I'm writing English. That is a good point. So, after the workshop, there's a bunch of information that you can get from the repository. And we're going to go over everything that you're going to see right now. And at the end, you need to remember a couple of stuff.

If you not agree with what the agent say, stop it. This is the same situation when you're in uh sorry, but I'm going to speak about with the with the M for a second. It is the same situation when you uh when you're with your wife and your wife starts speaking her work. Stop it. Put something in your something like that or drink more beers. It's going

to end the same situation. >> Yeah. Uh >> she's going to be mad either way. >> Let's begin. I'm so happy that my wife is not going to work out. So that was the reason why I'm doing those jobs. Okay. Uh let's begin with something. So I hope that all of you already have cloud code and yeah let me show where the keys are that was started

comp it's in document called key txt no way yeah and there was a big issue when I tried to push that I tried to do it at least 10 times and GitHub like there was a key are you sure yes I want it okay so let's start to zoom a little bit First, let's start with the PD. And does somebody know how to just use uh view

not text to be more good? Uh that no control V command mode. Uh okay, let's try to go in the stupid way. Okay, let's first read out. We need reliable websocket processing for payments events. How many of you are in the payments domain right now? Rachel, raise your hand. You're working from the payment company. >> Not anymore. >> Not anymore. Okay. So, what was the issue? We

get a lot of complaints, a lot of customer support tickets that something is broken, unpaid, the system is reliable and it's a pain of us. And what else? There was a issue with the money. The people want her money become something very strange. >> The bigger one double charged. >> Double charged. That is good for the company actually. So yeah, imagine you are building the system and

charge your clients twice and there don't do any complaints that is good for you isn't it? >> So as I say you need to think about >> maybe we should actually create a compliance agent as well. >> Yeah. Okay. What else? There was a couple of events that we should get. Payment succeed, subscription cancelled, refund created. Everything is delivered over the HTTPS which is good which looks

that our PM know that HTTPS is better than HTTP we hope at least. What else? There was some losses. Okay, the event could be delivered know. Oh, it's not about SOAS. Okay, it's good. So, we had good PM. And what else? There was a good description about the problem statement. There was goals which is good. I'm not going to read them. There is no goals. >> Autocope

is as important as what's in scope. >> Why we need to have autoscope? Could we do everything in once? >> Should you? >> I'm asking you. >> I don't think so. >> Yeah, we tried and failed miserably. >> How many times you failed when you try to do everything in one shot? So done it a number of times in personal projects. Those have zero zero risk, right?

But when you try to oneshot a a back office system like I did, took me two months to actually set it up properly afterwards. Okay, what else? There was a functional that is good. We accept post. You know what post is fine. We need to confirm. Uh we need to acknowledge. Okay, each eventually updated the writings. Very good English writings. What writings really mean for us? Depends.

Okay, so we need to understand that later on problem. And then that fing retry. Okay, not Ret, right? Okay, amazing. Uh could you retire the same event multiple times? What we need to do? It's something that we're going to decide here as a room support. Okay. User stories. It's good. Open questions and timeline when we need it. Soon. Yesterday. Three weeks. Sorry. >> Okay. Let's go over

our readmi and see how we are going to start. Uh let's me open the read me and probably there should be some instructions how we going to run that workshop a little bit down probably okay so first we need to start with PLD review is that yes okay we have good diagram here cloud code is very good for writing those diagrams for us and we need to

start with PLD review okay let's go to the terminal and try to make the terminal a little bigger. more bigger. >> Let's jump inside the agent. A little bit more bigger probably. >> Okay, let's be full check that I'm in the correct directory because >> do you see well in the back rows? >> Yeah, is everything fine. Are you able to see at the end or I

need to zoom a little bit? Zoom now. Should be better. Okay, let's start with the first PLD review. Let's see what that is going to do for us. The first question are going to find a PLD document or not. provide a PD part H. Okay, there is a PD in the rep level. Okay, the code looks like uh not so stupid. It told me that I need

to provide a PD. Okay, I'm providing the PL and let's see what that's going to do. First of all, it's going to check some comments. It's going to check if the quad exist. Okay, the the PD read the quad gets absent. Okay, and it's starting to do something. First of all, look, it spam the product owner. >> Of course, it does. >> Okay, we need to answer

why. But let's look what that's going to do. >> Oh, we we promised that was what was going to happen. >> What? >> We promised this is what was going to happen, right? We can see in a minute. But what this really do for us right now, let's open the comment to see until we're waiting because you know the agents are always wrong unfortunately. >> So it's

really empty files all the way and sending bunch of information over the network. Okay. Uh PD and the first one is the review. Okay, let's go over that. Look what is here. What is that? Google, could you mind the tools that can be used by this command? >> Okay. And what was the last one? >> Oh, this is the one where we actually get involved. >> Ask

user question. Okay. And what was the idea? >> The command owns it, not the agent. >> Okay. So the command knows those tools and know how to use those tools. Amazing. Look, we have usage and the usage is basically provide us guidance how to use that comment and I failed if I turn back look I failed to do it. I just do peer review and what the

comment suggest me do it in that way. It's going to work probably. Okay. And look, there was a We have a couple of requirements that common. We have some kind of rules human in the loop checkpoints. We have ask user questions block. H interesting. We have some daytime and also some kind of ref instructions. This is where things start to be interested. And what were you doing

in that section of Google? >> Couple of interesting things. You actually have to delegate to sub agents. Usually that's what you do with commands skills whatever they still have to maintain the reference frame. The tool calls are still limited. You don't give it more power. But here's where it gets interesting. We delegate to the product owner agent specifically. >> Okay. And what was the output of that?

Well, work folder purity analysis of the purity and then when when it finds issues with that purity, it's going to raise it to us. Let's see if that's it's working right now. We have a questions. Okay, which of those blocks epics the composition? Unselect for deferable and see. Okay, Q1 retention period. How long to keep roll pays compliance on? >> Any compliance people in the room? We

might need help. let's say Q1 retention period. So, no more than one queue. Okay, the second question is could we get orders out of could you get events out of order? What do you think guys? Is that possible? How many services you are going to have only one more than one? What is the best pattern? Uh when you're deploying system, how many services you need to get

in order to go with high available systems? Somebody one, two, three. How many services? >> As many as you need. >> How many we need in order to be high available? >> Let's say we want a minimum of three. That's a guarantee that you get out of order events. >> Okay. If we have P services could be got out of order then >> I mean I sort

of spo it for the room but yeah it is going to happen. Okay. So let's mark data store choice in which database we are going to store that information probably we need to answer that question somehow could be mongu could be posgress there's going to be a good conference in couple of months here in software about the databases but we need to answer that question probably okay

what in memory every actually in the every trading system we store everything in memory >> it's fast at the end it's very interesting >> well eventually it goes to disk >> eventually it's the good point okay so we need to have a mapping between the customer and the user what is customer for us what is user for us and what is the customer and the user for

the payment system that you're using. I'm currently doing stripe integrations a work and it's very complicated. What is the user for me is not the user for them. So we need to do that. Okay, let's move semantics. Do we need to have exactly once or we need to go with at least once plus temperance what we need? Okay, I'm going to check that in order >> it's

a good decision to >> a good decision to make. >> acknowledge versus durability budget. Okay, this also go uh we are not going to go over the burst, so let's skip that for now. Fuggest, those are rookie numbers anyway. >> That's good. Ops. Okay. What ops? Uh, how many of you are doing supports right now in your product system? Anybody >> on call rotations? >> On call

paper duty. Nobody. You're so lucky. >> Hey, p your duty is nice. >> Just not at 3:00 a.m. >> Uh, let's say that we are not going to take care about that right now. The room is so swine. So we're not going to do any kind of rotations. Let's keep them. Okay. Uh owner method timing probably we're going to need that and let's submit. Let's see what's

going to happen. We're going to wait. But on that time we can scroll a little bit back and basically go over our comments and our agents to look them in what they really are. So, we're going to go over the back end developer until we're waiting and basically see what was there. We have very big descriptions. Use this agent when you need hands-on backend implementation. Okay. Amazing.

And it looks that this is Java specific agent in no Java. It's no spring. Oh, it look that probably know and typescript as well. It's good. There was example which is fine and we select which model to use and we basically choose the best model. Right now it is the model that it's more expensive. So we want to basically send our money to >> this is the

one that can do a lot of stuff right because obviously this is one of the agents that needs to touch >> Yeah. Look also we describing that this agent is senior backend developer. We're hoping to hug both CF back and end developers in order to write the agents files like that. So okay and expect purposes we also have some kind of information how that should work and

compliance checklist document architecture rules acknowledge build system identify. So we have some kind of checklist that agent should go every time when we are running that and basically put check marks documented architecture rules. Check check check test comment identify. We need to run our test unfortunately. Yeah we know nobody of us would write tests but we need to run them and look we also need to format

our code. So those of our checklists that we need to run every time and yeah we need to go with modern Java 21. Uh are there somebody in the room which use Java 8? Tell me which company you are not going to join that company in the future. >> Java 5. >> Java 5. Is that still exist? >> Probably in quite a few places in in banking.

>> Where >> banking? >> In the banking. But okay, I'm not going to join our company. You need to double my sour in order to join that company at least. Okay. Uh spring boot, we using a little bit old version. Spring that it's fine. But let's see what happen when we review our uh prd and looks we have eight broking in five deferable items. artifacts are written

in those directories. Okay, let's go and check what those artifacts are really are. Uh, can I close all of those somehow? Yes, I can. Okay, it should be in which directory? It should go to work pd analysis. And this is the entire anaris from our uh agent and our comments. It's revealed. We have a date. It's a source with from where that it's done. And there was

a bunch of information inside. So it's good to read that probably there's open questions and all of that written down in that document. And there is also decisions. Okay. Look, there are some decisions as well. Okay, what we need to do after that the PD is already an analyzed by product owner. All of the decisions are there. The PR analysis document it's there. What we need to

do? Let's go back to our flow. What is the next comment? We need to run this one probably to Epic. >> Let's try that one. Let's try that one and see if that's going to work. >> and just to mention, this is the first time when we were doing that. >> Well, live >> live >> and we never test those comments. We update them in around the

midnight and at that time I was forgetting to pull so everything could go wrong and that was the reason of the workshop to try it right now with you. >> Yeah, you're basically our better testers, right? >> Yeah, you are our better tester for this uh product. So jokes aside, um when you're writing production systems with a workflow like this, you often actually get a lot of

iterations. So after each command, you do a pass to verify the quality adversarily. So basically not in the same session. That's the whole point. We actually have commands that verify the outputs of commands as well. Just teasing out some of the >> verifying a with a >> Okay. What? And then eventually the human goes in there. >> What we're doing here, >> so what's going to happen

now is let's say we've approved the purity finally after several iterations with agent, several iterations with the real PM. Now the solution architect agent is going to do an indepth analysis of the purity of the decisions that were made. It's going to take a while uh because that's you know sort of the the big chunk of work that needs to happen to actually get the proper output

we want. >> Is that going to ask ask a question? >> Well, probably will because we didn't actually do the the work with the previous stage, right? So, we're going to get help catch fire stop and ask basically here as well. And please never never going out. Never. Believe me, a lot of mistakes will happen with that. Why I'm doing that? Because it's easy for me and

I'm lazy guy. But never do it in production systems. This is my personal laptop. That was the reason why it's on. Not in all right. >> Yeah. Let's say an LM isn't exactly a a good judge of where a two use is valid. >> Okay, until that it's running. Let's go basically and explain to the people here that we also have some templates and what was the

idea of those templates >> templates? the moment you actually have to get some structured data in place and you don't want a database in the whole picture. We use a lot of MD files to get the same consistent output for each ADR and then for each story as well. Um in our real life work we have quite a few more templates than these two but for the

workshop today we we just defined the two you see on screen right now. Okay so we have edr template and story template. Okay let's see where we are with architect. It's still running. Looks burn a lot of tokens. >> That's just the main agent by the way. It's burning a lot more probably. >> Yeah. So we are still waiting until we are waiting to see the outputs

here. Let's look what is also there in our So we have rules but we never open them. Uh rub could you help it to understand those those files? Why why we need those rules? How they're going to help us? Okay, so clo as a harness actually conforms to these rules a bit more than an LLM actually just randomly would. Um, if you noticed when we were running

the PRD review command, this is the this was the file was at the top of the command itself. All of the commands actually referenced this. This is the main protocol we have for how the agents are supposed to interact. So I I did say that it's a small lever because there is a lot of thing a lot of things are actually happening in the deeper layers but

it still actually helps us control and frame a lot of how the system is going to operate. So the agents stop when they have an issue like if nothing else this is what we want right we want the architect to stop and actually offer us different options so that we can discuss them and figure out which is best for us. Let's ask the room what we need

to select here and they probably one question from the what we are doing until we are waiting because you see that we are waiting around 2 minutes 5 minutes what we are doing in that time we have another session and we are working on more than one story at a time but a round could be drinking coffee speaking with colleagues so really depends in what kind of

company you are we personally have at least five or 10 sessions like that and We are trying to work at this on more than one item at a >> Yes. So the whole um close to burnown situation is actually worse now that we have AI in the game because you start a task, let it roll, start another one, let it roll, and then eventually it's just complete

exhaustion from all of the context switching. So we've actually tried to limit ourselves a bit. >> Yeah. Okay. So let's try to take that decision. Should we use posgress? Raise your hand if you think that the posgress is good for this use case. >> the use case is just Okay, we need to build a web hook system which accepts payments from system like stripe or some other

system. >> Here's the question though. Ingestion potency tracking dead letter Q for the >> Is the posgress a good or no choice? Could we go with just embarent H24 production? >> In which company >> okay? >> Uh could you go with SQ? Not really. Okay. So the postgress is recommended one. Let's click. Okay. Delivery guarantees. What we going to get here? Go. Tell me which is the

better. At least >> are you sure about what exactly you want in terms of consistency? >> Very good questions. There's no double processing of any events ever. >> Guys, what do you think exactly once or it important? >> Which ones is better? Sorry. >> Raise your hand if you think A is good one. Everybody in the room raise your hand. Okay. How many think the second one

is better? Are there any props and consu? What do you think? >> Think distributed systems guys. The then the only answer you have is one only have one. Okay. Why not exactly one? I read in a lot of books that HTTP in with database could be exactly one. There's a lot of articles about that. I'm joking. So, let's move with a Okay. Uh, must the durable rights

complete before the 5-second acknowledge? Yes, this is the requirements. >> One interesting thing you'll notice is it actually has recommended. Obviously, you've seen it when interacting with clo code. Most of these are fine. It's the ones that aren't where the human in the L value actually emerges. This is the reason why we are here to look what the cloud code recommend to us and basically choose the

best and we can just type if you want to do something else. So you have opportunity to provide your guidance. Okay, let's go with a what else? How we are going to do the dupe? Whoa. How we going to do that? Okay, let's try to think about it. Are we going to have different systems or we are going to have only payub? No. No payub. How many

systems we going to support? Is that a good question to ask? >> Probably a business person, right? Do we want to ever support anything other than PayHub? >> What do you think, guys? Is it safe to run with only one provider? single point of failure. Okay, probably we're integrating many different providers in one of the company that we work together with you. But yeah, it's a big

problem. Then let's ask other questions. If we go with two providers, are there any possible way they to have the same event ID? Is it possible? >> Okay, so the first one looks it's not going to work. What about the second one? Do we have the same problems there? Could both have the same content hash? >> Unlikely. >> Okay, so let's try to type something else. Let's

try to do let's use event ID plus name of the provider >> plus content hash. >> Okay, press content hash. This will enable us enable us for future for adding new providers. Are we happy with all answers and all questions? Yes. Let's go. Click submit. And now we are going to wait again. So we can go take beer because I'm seeing very good beer in here. It's

called kills. Uh but yeah, we can go grab beers discuss something or work with something else. Until that it's done, but because we're in the workshop, we can go and basically look some others files that we have in our repository because this is also important. So we never go over uh which was the file Yeah. Okay. So, we never go over all our commands. We have epics,

we have PMS, we have PDS, we have STS. But let's go and look them and basically try to go and see what they are doing. So, rub why they structuring it that way? Why they structuring those folders? Could we put everything in one folder and make our life a little bit more easy or not? >> Of course, we could. It's just you probably get some collisions with

the global commands/sklls. >> What global? You mentioned skills. What skills are >> I have skill to play football probably or I wish to have that skill. >> You do? We got to try that sometime. We can try. So >> clo chips with a lot of global stuff. It's a very mature harness at this point in time. When you define custom commands like this, it's good to actually

have a name space, literally a folder that you can then slashepic and then column then command preface them with. >> But what is the difference between the command agents and skills? >> Oh, those are muddy waters. So they're all text, right? >> Everything is text. Our code is text probably. So, it's a it's a bit of trying to to frame the the what and the who, right?

Like, we we have the agents being the actors on the stage. We have the commands, which is what they're supposed to do. The skills are basically the same thing, but with a bit more added on top. And that's what everyone is actually writing nowadays. So, we are probably behind the times when we actually show you guys commands rather than skills. >> Okay. Okay, so we have commands

and skills. Amazing. Uh, when the skills are used, do I need to specify the skills to be used or how >> are they running automatically? >> Yes and no. You could actually execute them automatically if you create a plug-in or an MCP server is actually connected. But in our case, we want to run them manually. >> Whoa, we have another set of questions. >> Interesting. Yeah, could

you help me? >> Well, I think we already answered the one about the out of order events. >> Wait, we had the guy tra missing. Okay, what what should answer for the first Should we try to reorder them? That could be problematic. Guys, what do you think? What was the correct one? I somebody raised hand working in the payment systems. Those could help out because they payment

experts who are not. What we should do here? >> H do you need global ordering? >> Uh what that mean? >> That means a completely accurate ordering of events. >> I don't think so. I need events to be ordered by customer. Well then I guess the middle ground is >> Which one? The >> fine. Yeah. >> Okay. What else? I mapping how the mapping is going to

work. Mapping table sounds good at least. >> Don't need to make it more complicated. >> Okay. What about the whole part dependencies? Should we use Hikari or we can go manually with JDBC driver only? What do you think guys? Do we need to Hikari or do we need the pool? What the pool really going to help us? It's a question. How many of you using spring? Raise

your hand. Okay. How many of you are configuring Hikarus ever touching the configurations? Tell me. But why you did that? Do you have any reasons? >> Okay. Could we go without Hikaru? Is that a good idea? I mean do do you want to maintain the pool manually? >> Me personally no but probably people won't. Okay, so let's keep uh the hiku then. Okay, how does the one

part when the new events exists? That's a tricky question. Looks there's a kafka again. Okay, so uh are there any issue with shed schedule notation? Let's think about that. In the beginning, we discussed about number of instances. How the schedule notation is going to work if you have more than one instance? Are all instances are going to process the same things again or what? >> Have a

redirected. Do you know if that's possible with the shadow annotation just >> it's supported? >> In which version? If you present me that to me right after this workshop, I'm going to personally go and grab one of those beers and send it to you. I'm not going to pay it. I'm a student, but I can take one beers and give it to you. Okay, so DB notifications.

This is something that I hope that I'm so happy that I never seen my wife. Dedicated queue. It's all complicated. >> So, and how many users do you have? How many users? Very good question. We don't know. We should ask that. But let's try to go with sherot and we're going to run only one instance. We're going to handle rest when we need. Never do a mistake

to over complicate the system in the beginning. Never do that. And there was a real story in my life. I was working one very I would say big company with very good product but they decided to break the monolith in many microservices without clear reason without clear strategy and they was thinking that tomorrow they're going to have 10 million customers now they have five so >> 5

million >> five reason that but in the reality you should try to not complicate your system in the beginning keep it simple always. So, let's go with the first one. Okay, let's submit that. And we need to wait. Oh, that was fast. Okay, we're waiting for patch three. There's product matrix. Do we need product matrix? Oh, there was a bunch of other Ah, Rubble, help me. Yeah.

You see, we agreed that we were going to keep them for a queue, right? >> Build a configuration mech. Okay. What? What? What was that? Could somebody help >> How long do we keep the events? >> We said one quarter. But how do we ensure they're deleted after a quarter? >> Do we need to delete them? >> Could we keep everything forever? >> How much money do

you have? >> Let's see. Where's my wallet? Uh only one legal. Is that going to work? >> As Bezos, he's probably not going to >> Ah okay. Uh so what we need to say here fix shock window. Let's say >> let me reframe this. Do we know the answer from compliance? >> No. >> Do we need to build this today? >> Sorry. >> Do we need to

build this right away? >> We need >> Can we add it on top? Can we add the chrome job that deletes stuff? >> Why not? >> Then maybe let's start with three and then we revisit. >> Start with three. Okay. Amazing. Uh what was that? What counts as fuggy protest? H what mean for us? >> Let's go with the first one. I would say >> sounds like

correct >> the do shouldn't be in the process >> okay data cleanup autoban one month finance PM so we as developers we shouldn't take product decisions always ask those guys we have agents for that but we are developers so all the business decisions should be taken by business people so >> not to be confused with We are developers conference, right? >> We're at the J Prime one.

>> Okay, let's go with the first. >> Oh yeah. >> Okay, now we are done. Not really. Now we provide enough answers to all of those questions. And this is our work right now to answer all of those questions that agents and comments ask us. So the architect is probably now going to create something that we can then iterate over but is the good first phase at

least. Okay, we're writing ADRs look so all the architecture decision records that are required for this time are going to be produced right now. So it is going to answer the questions where the architect finished. It's finished here for now. But later we can escalate to the architect if we need it. Okay. Now there was a bunch of documents. Let's look one of them or at least

couple of them until that it's finished. It's wo I know but yeah. Uh where are located is here. >> Yeah but that's the whole point right? If we if we rush through this stage of the project the product development we just get garbage out garbage in garbage out. So optimize the quality here. Okay, let's look what was there. The whoop is decomposing. It's created by this particular

comment. The artifact which we used its epic MD. Okay. And this is the everything that we took as a decision. So we basically choose to use posgress. Why? We want durable store. Okay. Uh options considered. So we look only three databases and we choose posgress which is good. If we need to build that in the real world probably we are going to add a data expert as

well because we want somebody who know better the databases than solution architect. Uh believe me we need to check many different databases. So I personally use at least five different databases and without a database agent I shouldn't use the correct one. So we decided posgress >> more than one. >> Could the posgress is good or not? Probably for now. Is there better database than posgress? Somebody better

database. >> Yeah go ahead. >> MongoDB. Okay. It's not SQL. >> I hate >> you hate SQL. What about the new SQL? do you know do you know that there was databases which are new SQL database that promise to solve all of those problems that we have with SQL You can check out >> probably then you're going to wolf SQL. It's probably going to outlive us >>

Yeah, but you need to pay. Actually, there's no good open source solutions for that. But let's see. Okay. Decisions uh implications revisit. Okay. For now, we're happy to go with possess. And this is still writing. It's basically a dating stuff. Look, we're putting checkpoints marked. It's done. It's done. It's done. Okay. And now we are what we doing? >> We're decomposing the epic. >> We're decomposing and

basically >> No, wait. We're creating the epic, >> Yes, we're creating epics, but we provide a heat. >> Happens more often than I'd like to admit. What was I doing in this session? >> Look, there was a progress zero. What >> Yeah, we haven't done any work yet. >> We never done any work yet. Or at least that's the way it looks to the bosses, >> But

we work award. It was almost an hour of speaking and we we never start yet. >> Important decisions have to be made. >> Important decisions to be made. >> Okay, let's see. The next common should be epic tech composite PLD. Let's verify that with the four. Yeah, usually I'm verifying everything at least two or three times. Uh, yes, this is the second one. Okay, Let's try to

run out. >> didn't we call it web hook? Is it PRD? >> I don't know. Let's see. It's coming to hope that this is smarter than me. >> Uh, not necessarily. >> I need to buy more credits. Uh okay, now we're delegating to the product owner. What else is going to happen here? >> So basically we have an epic MD, right? That's the architectural outline of the

system. That's what the architect created based on the PRD. Now we have that we we give both of those artifacts to the product owner who's going to do the menial work of splitting them into So what should be the output the stories the real stories that we can kick off and start working on >> probably if we did this right >> okay let's wait let's pause here

for a second and if you are in the opening session there was a some kind of game you can try to use this workflow and basically build the app that they ask and win the prize so if you follow those steps and found a good idea for the app and you basically create a pd uh the cloud and the agent that we can create or the commands

that exist. You can try to follow those steps and basically build the the app that was mentioned in the opening session and try to win the money that they proposed to give us a cash. >> Stephan is probably be going to be winning that anyway. I'm not going to be on that time tomorrow unfortunately because I need to be in probably but I'm definitely going to submit

my app. So this is what I'm going to do tonight. Drinking beers and doing that app for the conference. Okay, we're waiting. We're still waiting. Let's see what that's going to produce. Uh it's reading files. Ah look how many times we read that document. This is the fourth engine which reads the the entire PD. Is that W of tokens? >> No, that's by design. >> It was

by design to pay money to Anttopic. >> Hate it or or not, that's the only way to ensure you as the product owner have sufficient context to actually do your job. Sorry, it as the product owner, Okay. So yeah, the main problem here is that we always going to pay your money if you want to go with force like that. So there's very good plans that you

can use. There was a worker models that you can use in your machines, but at the end we always need to burn tokens. We're in the life when we are burning burning tokens every day and a lot of tokens. Again, you can do it a bit more complicated like all all of these can actually live in a local DB somewhere rather than MD files. Then you get

two calls. You get something that doesn't burn as many tokens. You can use something that uh we already mentioned either CXA code index graphify multiple tools like this exist. You burn less tokens but yeah that's complexity. So this is a very simple system that we're just using to illustrate the points. Ideally people add a lot of stuff on top of it that actually serves their purpose. Right.

Yeah. People are doing a lot of stuff with car code those days. Ah okay we have something we have and okay a lot of questions and that we can start working on them probably or not yet. Oh, looks what happened There was something and we our whoop is working. Look, it's just that we are starting again the solution architect. So, it's going to read the decisions that

create conflict with the ADRs. It's going to obviously reread the ADRs themselves. And then we're probably gonna get another barrage of questions where we have to make decisions and they're going to be reflected in new ideas reflected in the stories and then we can actually start working on the stories. Okay. So let's rephrase that. Are you in the situation your company when you battling with PMs, other

devs or architects? In some meetings there was a battle. Is that happening in your companies? >> Okay. And and how you deal with situation like that? What are you doing? Just accepting or try to battle with them >> really depends. The questions depends of the company and depends of the effort that you you need to put there. But with the agents, it's simple. The agents handle everything

for you. So let's angel battles between >> I hope the solution architect is going to win because all the solution architects are big guys. Uh so yeah let's look what we have that time and basically see what else we can present until that it's done. So let's me try to open the entire flow. Uh so while Stephan is looking for that if you do uh actually use

a memory tool of some kind. These conversations are recorded sometimes the you know the shouting between the agents and they're really funny to to browse through >> if you have time to read them. No one does anyway. >> Okay. So until let's go over the force because now we are going to start implementing the stories which where we are going to end this workshop. But let's go

the four one more time. So we start with the PLD after that we basically review the PL we asked questions. So that could be done by PMs actually. So in the guide PMs could do that. they can battle with theirel in order to answer those questions. You can build the grooming session with PMS and basically let's reate the PD with them. So it's not only you or

your team could be the entire product team. >> Exactly. Ideally these things especially the outer loop we mentioned these commands and skills and agents they go to other departments. This shouldn't stay in engineering otherwise what you get is an inner loop an execution loop that's optimized everything around it is that slow. So we are actually trying to fix the outer loop as well. >> Yeah. So the

the issue is that not only the engineering team should use the tools everybody in the company should use those tools. So we want to build AI first company or at least we dream to do that but that mean that PMS QAs devops everybody should use those tools everybody should be familiar with those tools and know where the responsibility ends and now we're in the situation where our

PM is providing very good prototypes. Oh yeah. >> And that is really fun because you can open and see in reality what they want from us. And yeah, it was really nice. >> Basically optimizing for lazy engineers, >> Yeah. Oh, we have a bunch of other questions. Oh, >> again. Really? >> Again. >> Oh, wait. That's the point of the Could you read them and help me?

read them, please. >> Oh, out loud. Come on, man. >> Yeah, we have a big group. >> Okay, so yeah, probably people at the back don't see them. So, what does the content hash component of the dup dduplication key cover? So this is a column with a unique constraint computed on the Java hotpath before we actually send the acknowledge the yak back to pay hub. Interesting. So

anything on the hot path. >> JSON on the hot path >> should probably not be JSON. >> Not be JSON. What could be? well multiple different protocols could be used here, right? You you want something that's a bit more lightweight >> but that in the postgress you can query the JSON. >> True. Do you want a JSON bitcoin for this for now? Unless you unless you want

to pre-optimize it now. >> It's a good question. Okay. What else we can do here? >> We could try a different protocol. Um not a lot of people who are financial geeks. So maybe protobuff. >> Okay. So let's answer that. Let's try to go let's >> Here is where we actually discuss stuff. Heistwolf. >> Could we go with Au? >> We could we could also do SB,

but you >> Okay, the next one. What we need to do in conflicts? What the conflicts really mean for us then? When two events share provider event but differing the content hash, what happened? >> Huh? Is that possible? >> So let's say PayHub same event ID but different content hash. >> Is that possible? In theory, >> what we need to do if we're not sure not sure

what happened with the providers, should you ask them? >> What do you think? Do we need to read their documentation? >> It's probably written in the docs of Payhub somewhere. >> Probably. Could we use agents and comments to scrape their docs? >> It's something we should have done before we started this whole thing, but yes. >> Okay. Then do you have story to tell us what happened

when we add one of those tools and basically what entropic did for to us? >> Yeah. So CLCO has this uh far crow plugin. Uh one of our colleagues actually got banned for using it. So so yeah there >> yeah so there was a very funny situation. We use the tool to scrape the data and basically have all the documentation locally because we want to fill our

models with that data in order to build proper documentation and in order to use the documentation in our uh decisions but on topic broke us on topic just broke the entire profile and we are not able to use that profile anymore. So yeah, try to think if that's okay or not when you add comments, tools, and everything to the cloud code and topic could block you because

they're thinking that you can scrape their information and use their models to train your own models. So we need to be very careful with everything. Okay, let's take some >> here is the implicit behavior right now. Both of these are going to be processed. The easiest way to to figure out if they should is to read the docs. Does >> let's go with first try wins. >>

So how much of the state machine does story 001 fix? So before I can answer that, I probably need to read the story, right? Let's assume I've done that. Do you want me to open that for >> Uh, do we have the time? No, let's go with um full state machine now as described in the story. >> This one. >> Yes, please. >> Okay, let's submit those

answers and see what happened. >> What's the worst that could happen? >> Probably next bunch of questions. >> Yeah, we we don't know what's going to provide to us. So the LM are not deterministic. We'll try to build them in some deterministic way, but we really don't know all those questions. We try to answer them right now here. So let's We're still going to wait bunch of

tokens six minutes. That probably is not correct. Uh okay, the room choice. Amazing. What recording again new architecture decision records. Oh, we're expanding. >> We're superseding one of the ADRs >> Why we're doing that? >> Well, any good architecture decision records by definition should actually supersede the original ADR when a new one replaces the So the main idea is that the EDR cannot be updated could be

create new EDR should be created to replaced some others. >> Think of it as append only log right you want a complete deterministic history of all of the decisions made so that you can replay how the system got to where it got. >> Okay. And now we should wait a little bit more probably. Okay. is that the new one is replacing the five and we should be

ready to go probably. Oh no, we are still waiting. Suspended by blah blah blah blah blah blah blah. Okay, sounds good. We're Uh, unique the dupe the key analysis. Okay, time for coffee. But yeah, this time we'll just go doing coffees when we're in the office. Okay, all questions are recorded. Uh, okay. What else there? >> So is going to write the stories. All of them. Oh,

start writing stories. Let's look one of those stories. What? What was Uh, Xcode. Uh where is my visual studio code? epics stories. what was that? Could you help me to understand that? >> So what we just saw is it wanted to use JSON. We said something lighter a in this case. So now it's going to probably have Avo in the story somewhere probably. Let's see. Can we

find it? >> No durable guides, no business processing, does not implement the concrete p signature. >> There there is. Okay. This is one of the story that we need to start working on. Probably there is no dependencies. So probably we can start working on it right now. And looks we have a P called par which basically mean that this could be done with some other stories in

the same time. Also look here we have depends on what that is going to do. What was the idea of >> So we like to build the dependency matrix between stories when we do this thing. This way you see one the critical path two what you can work on right away like basically what's next. three how many sessions you can actually work on in parallel. >> Do

you do that when you work with the teams not with the agents? Do you have those dependencies written down in Jira somehow ring tickets? Are you doing something like that in your work? >> Basically blocked by >> blocked by no. >> How many of you? I hate that. It's too hard to me. Yeah, I was architect. So I I supposed to be architect in different companies but

yeah. Uh okay, we are still writing. We are updating epics. All the tickets are done. So let's me try to do something else. Let's me open your tag now and let's kick off. >> Zoom zoom zoom zoom zoom zoom. >> Let's see what was at the end. Should be read me here. And there are multiple commands now that actually you pick a a story start. >> Let's

go with story start in your terminal. >> Is that going to start all the stories or what that going to do? >> Well, ideally you should provide a file ID. It's going to figure out how many of them are written. >> It's going to help you. Look, it's basically say I'm going to provide a suggestion. Like I said, this is optimized for lazy developers, right? We >>

me. >> We want to make our lives easier, not harder by using this tool. >> Okay, let's wait. Do it fast. Please be fast. Uh, no, no, no. Yes, yes, yes. And where we are. Okay, start to Oh, we can expand to every read and this probably is going to do everything automatically for us. Let's just go with the first two. Uh, let's go with this one.

And I probably could start another one >> yeah one piece of advice I tend to stick like one one unit of work in a session I take one story open a new cloud session execute it new session to review it etc etc so >> yeah but >> running running these two in parallel is going to probably produce worse outcomes but we're doing this for the for the

>> Yeah, but there's no dependency between them and they can be done in parallel based of the analysis. Yeah. >> Agents view the code agents view. >> What was that? >> It's it's a new thing from last week. >> Good. >> Could you explain that to me? >> So basically how it works is it does the whole orchestration out of the box for you. You just open

one terminal and then you can actually execute >> ah >> everything. Yeah, I'm doing that at the moment unfortunately, but it looks that I need to read that. >> So, Agent Zu was actually built for you. Um, I'm still pretty sandbox per session >> Yeah, I want something new. >> Interesting. I need to check that. >> It's actually pretty cool. Um, think of the logical extension of

the native teams thing, but in And let's stop here and basically ask if there's any questions. What do you think about that workflow? Is that is that going to work for you? Do do you plan to try something like that or not? Okay. So, let's wrap here. We present what we supposed to do. >> I think we can open for questions. >> Yeah, we're open any questions,

discussions and everything. So if there is any question please ask ask now or yeah go ahead. >> Uh depends how you use that. If you create uh agents and comments about that they probably going to work. But if you just provide us a comment and say okay scrape that with this MPC it's probably going to work. So really depends of the use depends of the usage. >>

What we did the specific case was we had a command that was actually using the the fire crow um plugin under the hood. So it was completely autonomous and I think that's where they draw the line. That's where it raises the flag for them. If there is no human in the loop who's actually doing the web scraping of the documentation, it's a red flag. >> We asked

them but they didn't apply. >> But looks Google, there was two different agents that start working together in both. Why? >> So native teams by default they're going to launch parallel agents doing parallel work if a story can be executed in >> Okay. But look we have backend developers, product owners. Before that we have uh solution architect what they all working together. >> Well, ideally they shout

at each other and make good decisions, right? >> They are discussing between them. >> Okay. Amazing. I have those discussion with my product owners and my architects a little bit more often. But yeah. Okay. Any other questions? So eventually this is going to throw another decision our way at some point. It's going to hit the inner loop itself is going to hit something that wasn't predecided. There

it is. They're asking questions between them. So sounds good. Okay. Any other questions? Yes. We answered a lot of questions. How do you go about? >> Yeah, we have a lot of discussions with them. So in the first place we are doing some of those uh things during our grooming sessions. So we take one story and basic one per and doing that with the PMs and this

is where we are answering most of the PM related questions. Rater do you like on a >> something like that we try to do as much as possible a sync but we ask a lot of questions doing that. So we use uh points to store the sessions. So let's say there's a question that we are not able to answer. We stocked everything the entire context in the

markdown file. We asked that question on J or some other channel if we're not doing that in the call and when they answer that we return and restore from where we ended and that was the reason why we're not working on only one things at a time. We try to work on many items at a time. >> Yeah. Because ideally if you have to wait for compliance

to answer a question, you work on something else in the meantime. um like that part of the work is still slow and that's why we try to do it as much as of of it async as >> Yeah. So we send a lot of question we also try to integrate that with teams swag and basically if there was a questions to somebody let's ask that. So in

we also try to build a map between the entire company and we provide which team is responsible for what which PM is responsible for what kind of domain and basically if there was a question to that to send that automatically we are trying to achieve that as well but it's a little bit swap unfortunately so we are still working on that because this basically mean we need

to be able to do it oh there was other questions ah what What was that? Google >> you got to have so >> there was two questions >> different sessions different questions so in okay let's stop here and continue with the questions from the room any if no thank you so much oh yeah go ahead >> we started with eight different ro we can maybe agent agent

>> Uh what is the optimum number of agents? Where do we get? >> Depends is the correct answer. So we have how many agents right now? >> It depends on what you're building. I think we have 108 >> Depends how you use them and how they collaborate. So for example, for my team, we have custom comments with custom agents. Uh because we're doing very specific work. So

we are building qualency applications. So it's totally different. So depends how they're cooate and things like that. There's no specific number. There is just no specific number. >> You need to try it and test it. >> If you're building a pet project, you can probably go with these eight plus maybe a data layer expert. But >> some other architects as well, >> security expert would be good

agent, not >> another question here. >> Yeah. Uh so do you ever try to limit size of your skills how big they are? >> They're limited they're limited by the qual. So if you have very big skill it's going to provide you a message that you need to give me that it's around 40 kilobytes or something like that. >> It's 40k in tokens I think. >> Yeah.

But uh they also provide you a skills how you can build skills. So you don't specifically try to make >> No, we try to make them explicit enough and also in them you can pro link them to the other documents. So you don't need to provide everything in one SQ file. You can add link to other files if they need it. So you can tell them that

here is how you can build thread pool for example or something like that and basically provide them external file with that uh information. So not everything should be in those uh >> One of the big antiatterns is actually either clone MD or agents MD becomes like 5,000 lines. That's actually one thing we mentioned already. It's not the right amount of context because cloud MD is loaded for

every session. You want a a couple of hundred lines maybe with links to other relevant files. The way we usually structure it, cloud MD is the entry point. It points to relevant architectural documentation. Let's say an architecture MD which is again an index of a lot of other documents including ADRs. This points to basically all of the relevant parts of documentation that the LM can read and

analyze and it also lays the ground rules. So that's usually enough to do in a a couple of hundred of lines in the root MD file. >> Okay. Is that answer your question? >> Yeah, I have one more. So let's say you have those extra files. So how do you store them? >> We just provide them. We provide a path to that file in the skew or

comments or agent. We provide the link to that part and basically provide if you need to do something read that this is the way how you should do it. >> Yeah. Again, if you if you want to go very deep and you're not necessarily building with clothes, there are many ways to do it other than text files, you can I think the latest one is people keep

using obsidian to do this. So it's again it supports markdown. It it has graphs and links between the different files. You can also just do a database, >> Okay. Any other questions? No. Thank you so much, guys, and we hope that you enjoy that workshop. >> Thank you. Cheers. >> Now it is. >> Perfect. Right. Good afternoon everyone. Uh happy to have you all here. It's a

small room. So if you have any questions in between, just raise your hand. We got plenty of time. Anyway, they did ask me to use the microphone because then people on the recording or the live stream can also hear your question. Right. Welcome. Please have a seat. We just started so we're just in time. All right. This will be a deep dive session. So we won't be

in hands-on workshop session but it'll be mainly me talking about stuff. Uh but to make it a bit more interesting we'll also do a lot of live coding live demos to show you how it actually works. But you won't need your own laptop or MacBook yourself. All right. Today this afternoon is going to be about concurrency cross roads. how to choose between reactive programming and virtual threats.

And to get a bit of a baseline, is anyone doing reactive programming right now? One or two, maybe three, four. Okay. And who is already using virtual threads in A bit more already. All right. And who has used Quarkus before? Also a few. So that's good. I'll cover some of the basics as well, but the demos are going to be mainly with Quarkus. So, what you will

get today, uh, number one, why is blocking the baseline, uh, and where will it hurt? Uh, reactive, what does it fix and what does it cost you at the same time? Then we're going to dive into virtual threats, the tradeoffs, and I'll already spoil you a little bit, is not a fix for everything. Then number four, structure concurrency. It's still not ready yet. It's still in preview,

but we'll show you a bit where it is right now. And in the end, by the end of today, you should know what to pick for your current or for your next And it's a concurrency talk, right? So it's a fair question. Why not single threaded? Yes, a fair question. But it's a concurrency talk. So we'll do multiple threads today. So who am I? My name is

Villiman. I'm a software engineer from the Netherlands. I live in Delft and I work uh in Rotterdam for Lunat. We do JVM consulty AI of course because everyone does AI and I've been doing this for over 10 years now on the JVM. Uh when I started my career I was mainly doing this in Scala. I learned that in university. So I did functional programming reactive systems with

aka play. And then when Quarkus came out, I transitioned more to uh uh cloudnative stacks with Quarkus. U and of course we integrate AI systems with that as well. Today's agenda, we'll cover the introduction, then we'll dive a bit into blocking IO, so the current state, then we'll show you how reactive programming works, then we go into virtual threads. I'll cover shortly cotling co- routines but it's

not the main subject of this talk but I'll let you know what's out there. Then we're going to explore structure concurrency. Where is it now? Uh what does it bring to you? Then we're going to compare the quarkus ecosystem with the rest of the JVM ecosystem. What do other frameworks actually do? We'll touch a bit up onto performance and in the end uh show you some comparences

and conclusions. So, for an introduction, I digged up uh a bit of a history, and this is definitely not going to be a history lesson today, but it's just to give you a bit of insight for the people who haven't been on the JVM for too long uh to show you what was out there in the past. Well, obviously in 1997, I wasn't programming yet. Has anyone

there used green threads? Yeah, people normally don't want to tell you tell you that because that tells your age. Uh but but there will always be a few people in the room that actually worked with it, but I never really did. Uh then a year later, they started replacing the green threads with native uh platform threads and they became the default on JDK 1.3. Then later we

needed some more primitives to do proper concurrency. They came off with the fork join pool and the fork join framework from Douglia. Uh that was all great but then I wanted more performance. Then they started the reactive streams initiative around 20145 to build reactive systems with RX Java and project reactor. Yeah, exactly. So this talks only covers the JVM but indeed they stole some IDs from other

languages and frameworks out there. Yeah. >> Yeah, that came a bit later indeed. Uh then in Cotlin there's had a preview with co- routines and it has been stable since uh 1.3 and then people always kept asking what will the JVM what will the JDK do? Uh so they've been talking about virtual threats for ages and uh in 2023 it was uh released to the public uh

with JDK 21. Then we had some troubles with virtual threads where we had synchronized spinning and we're going to see that in detail later today as well. Uh but yeah I can basically skip that bit because it has been fixed if you're on a modern JVM like 24 or 25. Then the current state of today we have the concurrency which is still in preview. Even for JDK

27 it was now accepted as yet another preview. And why do they do that? They keep on changing the the the API a bit bit by bit to get some feedback. So it's already in the seventh preview. And otherwise if they would bake it in I don't know five years ago they couldn't really change it. So they kept it preview for a long time. So if we

have a look at uh three uh threading models uh this is from the Oracle Solaris multi-threading guide from back in the day. Um on the left we have the many to one where we have green threads and this is a simplified uh version but basically in your Java application you would have some threads and that will get uh mapped to one uh actual thread in the kernel.

So, it was a bit of poor man's uh concurrency, but at least you could manage some threats on the JVM. And that was back in the day when you probably only had one CPU or one core. Indeed, you only had one core, so your application had to do all the multiplexing because, well, the kernel didn't really have more cores to multiplex it over. Then in the middle

uh they started introducing platform threads where we can map uh a thread in Java directly to uh a thread in the kernel and that's where we have multi- uh core processors etc to uh leverage that. Then in the end we now go to virtual threads where we have the many to many model where we have many virtual threats in your Java application. So in your runtime mapped

to a whole bunch of threads in your kernel. So this is a bit of a simplified overview but it gives you a bit of uh direction where we went. So we started from just one kernel thread with some uh green threads in your runtime to one to one mapping and now we can have all the threads that you So that was the history lesson of today. We're

not going to uh look look further into the past. We're going to see where we're at now and what we can get in the So if you talk about IO, IO is everywhere. So if you're a pure functional programmer, your program does some computation, but it doesn't do really something useful because you can't interact with a file, you can't interact with a database, you can't interact with

the network. So for a business application to do something useful, we typically need some form of I/IO. So calling a remote service, uh calling a database, uh writing to a message queue, uh doing something with a cache. Of course, you have to mention AI. So calling an LM is also IO. And occasionally you need to do CPU work as well. So what's them blocking IO about? um

every thread uh issues a request and then when it calls a database or does some other I/IO the thread just parks and waits for the database to respond. Mostly that can be uh pretty fast. You can imagine if a long running query of your database is overloaded the thread is going to be there sitting idle doing nothing have the memory reserved and nothing productive happens in between.

I mean the thread is doing nothing. So, it could actually do some CPU work, some computational work, but it can't because it's blocked. Well, that's what we've been dealing forever with. Um, so what do they cost? Each Java thread wraps to one OS thread and roughly uh depending on your OS and system maps to uh one megaby of stack uh memory that you need. uh the creation

and all the context switches have to go through the kernel because if we look at again at this diagram if I have platform threads is the kernel multiplexing that back and forth to schedule your work and the OSuler takes care of the scheduling. So it decides when your thread will run. For a practical capacity, you can have on a modern machine a couple of thousand, but if

you want to have more of those, it will blow up. You don't have enough memory. Uh it doesn't work. So you can't have millions. So how did we deal with that? For the past 30 years, we used a fixed size uh pool for everything. So if you're using Quarkers or Spring, you typically have some pool of workers, you have some pool of request handlers, you have some

pool for database connection pooling, and we tune those pools typically because the sizing is guesswork. If you're in development, well, it works great. You go to production, it blows up, it doesn't work, you get timeouts all over the place. So well you do a quick and dirty fix on your tom cut server increase the number and then magically you hope and pray it works and then you

hit another bottleneck you have to tune it again. So what we typically do we do some performance testing tune the thread pools the database pools etc before we go really to production has some downsides. If you have an undersized pool we get queuing and we get a lot of timeouts. If you have an oversized pool, well, we just have threads allocated doing nothing. And a threat means

a bit of memory as well doing nothing. So, both things are bad. But we have managed for the last years to do this efficiently and everything still works. And then the number of threats becomes a limiting factor uh long before we hit the CPU or the network really becomes the limiting factor. So if we look to a typical web application, let's say I have a simplified application

and I just have two uh platform threads to handle my web requests. So if the pink request one comes in, platform thread A handles that request, maybe does some JSON processing, some bean validation, and then calls a remote service. When it's calling the remote service that needs a bit of time to respond to your request, you see the jigot line. It does nothing. So if request two

comes in, it has to be assigned to platform thread B. And there we have the same issue. Now when request three comes in at the same time, there's no platform thread available to handle it. So it's get scheduled later by the framework and that's why you might see uh high latency or in the end timeouts because if the requests keep on piling up, you don't want to

have an endless queue of requests that you want to handle. So typically there's some limit on that uh inflight requests. Um and then yeah you basically just give timeout errors to the client. So for this I came up with a small demo application uh based on quirkus. So in the middle we have have our coffee shop and then we have a remote coffee machine and this remote

coffee machine always needs to be turned on warmed up. So typically if you make a coffee you have to warm up the coffee machine and then you can make your coffee. So there's an artificial delay in there and if you ask for a coffee takes a bit of time before the machine is warmed up and then we'll actually start making the coffee and then you want to

as a coffee shop keep track of your orders that you've been making uh during the day. So we also will store all all the coffees that we've made into a database. The implementation is pretty stupid. So, uh, if I want to have multiple coffees, uh, we we simulate that we start up the coffee machine again and again and then we stop the coffee machine. So, every time

you make a new coffee, there's another delay. So, for the Sure, it doesn't share the right screen. That's better. All right. So, I generated a simple uh quirkus application from code.quarkers.io. It's a bit like spring starters. You just select the extension that you want. So, I selected hibernate, postgress, etc. You can see the list here what I all selected. Generates me a dummy application. And then I

build my little coffee shop into this. Then it provides this nice uh web UI of extensions that I've enabled and extensions can contribute with something useful machine. Make it a bit bigger. And here you see a default mapping that if uh you want a coffee, it gets you back a coffee successfully. But the crux is here. It takes you it takes it 3 seconds. So there's a

manual delay in there to simulate that the coffee machine is warming up. So that's on one hand the wire mock implementation. So that's an error external coffee machine. Then we have a simple uh database view where we just persist the beverages. Well, there's nothing in there yet because we haven't made a coffee yet. And if we then have a look So now let me show the code

first. That's easier. So we have a simple uh bean here a blocking bartender uh which get injected a rest client which is our coffee machine connected to our wiremok instance and then we just get uh call the coffee machine and give back a beverage. So the bartender just asks for a coffee and that's it. Then we have a simple DTO here uh a simple entity. We use

a panache entity uh with quarkus and that provides you some easy accessors a bit like spring data if you're used to that. Uh then we define a simple repository where we can persist a beverage or we can persist a list of beverages then we can just inject that into our resource and then we have a simple get endpoint where we say get me a beverage save it

to the database and give me back the coffee to the user. If I then execute that looking simple, it's going to take those three seconds because there's uh the manual delay in And then we got back our blocking coffee Plain stupid simple was just a Raspberry source with some uh servers injected with a rest client. Uh yes, the blocking simple just a blocking call. If we then

have uh a look at what it did um that got spawned on the executor thread. So on the worker pooling quarkus let me make it a bit bigger actually and well it then warming up the coffee machine takes a couple of seconds persist the the beverage and gave it back and this all happened on executed thread 5. So it's what you have your easy mental model. A

request comes in one platform thread handles the request does the JSON marceling does the external calling to the to my wiremark instance uh does the database call and also returns back the response to the user. So you can if you reason about it or want to debug this is easy. This is what we're used to. You can just trace it through the application hook up a debugger

will just work. If we then um want to do sequentially, so we want to have some coffees for our friends, we can execute it sequentially. We ask the bartender three times for for for a coffee, compile the list and give back the list. If we then do a So, as you can see, every 3 seconds is now warming up the coffee machine because it's a pretty stupid

bartender. pretty simple uh uh coffee shop. So, it has to do one by one and it gives you back the result. Well, that's not really concurrent. So, we want to do it in If we do it in parallel, we just execute uh inject an executor, submit this task to an executor. Uh and then we can do all those things in parallel. So if we do this now,

you see that the coffee machine is warmed up at the same time three times after each other and that happens all on different threads as well. So the initial request was handled by worker thread five spawned five or three uh new tasks on three other worker pools. They picked it up. So you see thread eight, seven, and nine doing some work. And then when it goes back

uh to the user, we're back on thread five. So they join back. Uh and then we're done. Let's see what else did I have to show. We can can forget about this one. Um we can also have a flaky bartender. So have a look at that implementation. And there we randomly throw away the coffee. So it makes a coffee and instead of you giving you back a

coffee, you have a chance of roughly 50% that the bartender just throws away your coffee and doesn't give you any coffee. This to simulate some failure midway. So if we uh execute that one, so you see that it's warming up warming up. Uh, but then something blows up in the middle and we don't get back our coffees. So, if we have a look at the response, the

coffee machine broke. Well, there was a chance of 50% per coffee breaking, but in this case, it threw away all the coffees because we didn't get any coffee at all. So, even if it was making three, one failed, you don't get back the other two. Well, there are some ways to handle that with failures, but it keeps on brewing coffees in the background and then when it

returns it, it throws them away anyway. So, that's kind of bad. But this shows you a bit of the primitives that we can do with blocking IO uh that we can do it sequentially in parallel and that we can fill fast as well. And this is just to set the baseline for everyone. This is what we're basically writing when we're writing our simple CRUT applications with Spring

or Quarkus, whatever framework. You typically write it something like this. You call a couple of services. Maybe use an executor service to do it in parallel. You and you give back the work. However, if you get a lot of load, it's going to be difficult to handle the load. So we have to tune those thread pools again, those database connection pools, web request pools to get a

bit more performance. And if we want a bit more performance, we basically have to grab a new server from the basement or from your cloud provider to plug in some more uh some more CPU or memory. Or are you single? Yes, of course. Yes, the threats that I'm talking about indeed the are the platform threats So then people wanted to do something new. They wanted to use

the hardware a bit better because they were saying like well if all those threads are doing nothing in the meantime that's a bit useless. I want to squeeze the most performance out of my hardware. That's why they came up with reactive programming and reactive systems. And if we look at the reactive manifesto at the uh the thing is we should always be responsive. So when a user

does something you we should always be able to respond. So we respond respond in a timely fashion under all conditions. Even if the load is high we should be able to give you a response. Also should be resilient. So stay responsive in the case of failure. So do fail fast or error out fast and try again with retries. And it should also be elastic to stay responsive

under varying workloads. And how do you typically do that in those systems to make everything messaged driven with a synchronous messaging. So you have loose coupling as well. If you see that in a diagram, people probably have seen this type of diagram before where at the bottom we have the messagiven that makes sure a system is resilient and elastic and therefore the system can be responsive in

all times to So when we're doing reactive programming, we should never block the event loop. So we use an event loop to handle all those events to handle uh those asynchronous events but we should never block the event loop. Uh we typically if you have event loops uh it will quarkus by default will spawn the same amount as event loops as the number of cores you get

allocated on your machine or in kubernetes or to the JVM uh on your system. So in this case, a handful of threads, it's hardly readable on the TV. Handful of uh threads serve thousands of connections. So we don't have a thread per request anymore, but we just have a handful of threads handling thousands of connections as fast as they can. Uh we have tight CPU casuality. So

we don't have to do uh context switches all the time. We just can keep it on there. Uh with reactive system, back pressure is explicit. So if a consuming system says like hey I'm overloaded uh I can't handle all the stuff we have back pressures uh in reactive system. So it will slow down the producer instead of overwhelming uh the consumer to programming in that model. We

have to do composition via operators like map flat mapap zip retry timeout and that's a bit more difficult to program with but we're going to see that later. So what it cost you at the same time if you do something like this nothing can be blocking in the middle because then the entire chain will be blocking. So the entire stack must be non-blocking. So your connections to

your database to your message Q etc should all be non-blocking. Then if you look at stack traces because those event loops interle a lot of work in between it's really hard to get a proper stack trace out of your system when something goes wrong or you get an exception because normally you follow the the call graph and you can compute a nice stack trace but here they're

mostly useful useless uh if you try to look at them if you have a thread local or an MDC for some uh context of tracing you need deliberate propagation. So in quirkus they came up with a context propagation extension where this is handled for you automatically. But there are still cases where you really need to think about the thread local and the context to pass it along

correctly. Because of all this debugging is non-trivial for for new people because if you hook up a debugger in there it just blows up. It doesn't really work. And what is also a lot of times different the team has to commit to the paradigm. So if you have a team of people that are functional programming enthusiasts, it's probably easier. But the problem is a lot of people

then do different functional programming styles in the same application. So for a newcomer, it will be really hard to yeah copy and paste the controller and implement it own work because the other controller looks different. Again, we have our simple CRUT controllers with spring boot of workers. It just reads sequential. You copy paste it adjust it. It works. But here the team really has So how does

non-blocking IO look like? In this case, we have a simple system with just one event loop. And an event loop is still simply a platform thread. Uh but it wants to eagerly do some work. So when request one comes in it again does some JSON processing uh then calls the remote service but it's not going to wait there and do nothing till the remote service come back.

It can handle request two three four five or another thousand requests and handle those in between. So in this way we have just one uh event loop thread which maps to a platform thread and can handle a hundred or a couple of thousand requests while waiting for some remote services to to be executed. has a bit of a drawback, right? Because now everything is mangled together. So

if you want to do stack traces, debugging, this is already a bit more difficult than having a when you have a thread per request. And it was nice and easy to follow through. So how does the event loop thread do this? So we uh now visualize it like this. Request one comes in, does some work, and then it's going to schedule some IO somewhere, and that's just

waiting for a database or a message Q to come back. Then request two comes in and does the same and then the continuation of the scheduled uh request one comes back and then it can handle that process it and give it back to the user. So that's how we can interle uh inside each and this way if you have an event loop thread which is a platform

thread it's always doing something because it eagerly wants to do something. So it's always processing events given uh if you have events coming in like requests. The nice thing is what I mentioned before uh said that you had to do reactive all the way. There is a middle ground in Quarkas. You can actually do both at the same time really easily. So you can have an imperative

endpoint with a worker threat from a pool blocking database blocking HTTP client Then you can have in the same application or the same rest controller have another endpoint which is a reactive endpoint which runs on an event loop thread doesn't block and uses non-blocking clients throughout. So we visualize that basically we have an event queue somewhere. So we have incoming requests that need to be handled. Uh

we have a bunch of IO threads or event loop threads. They're eagerly looking for work to do. They can either uh find the handler and dispatches so they can handle the event, do some processing and be done. But they can also switch off. So they can also hand off if it notices, hey, this is a blocking JDBC call. I shouldn't be doing this on my event loop

thread. And it detects this based on the types and the usage what it's doing. it will offload it to the worker pool. So we have a bunch of workers configured in your typical application and this one will handle this one blocking for you. That's how it can mix and match uh it can delegate the work uh to work thread for you as well. But it does require

a context switch. So to do reactive programming we need some extra goodies. Uh in Quarkus they came up with mutiny. It's a standalone library that you can use in Quarkus and where you have a uni and a multi. Uh but of course you can use your good old Java concurrent package with a completion stage but the composition is a bit more difficult. So that's typically why you

would use mutiny because composing all those futures is a lot easier with Then from reactive streams you can have a publisher for doing streaming data or you can of course use your cotling co- routines in your application as well and those you can all mix and match in the same application actually for the people used to doing reactive stuff in spring it looks a bit like a

mono and a flux so uni uh emits one item or a failure so it's like a future so you get something back in the future if you subscribe to it uh it might be null as well because well we're on the JVM and it's similar to reactor's mono. So we typically do this for an HTTP call, a database look up, a cache uh hit etc. Then we

have a multi and that's for streaming and a multi is just an unbalanced stream of zero to n elements and then either can complete or it can fail. And this implements the reactive streams publisher and looks a bit like a flux. You would typically use this for Kafka streams serverside events uh streaming file lines from uh gigabytes of uh log line uh log files or uh big

data for example. you would typically use a multi. What my naive self did a couple of years ago, if I would get a list from of records from a database, it was like, hey, this looks like a stream. I'm going to use a multi for it and pretend that my database is streaming. That doesn't really work because your JDBC driver or your underlying uh database driver even

is reactive. It's not really streaming. It's more like a future. So, you shouldn't treat it as a streaming data source. Kafka streams can be a streaming data source, but your database typically is not a streaming data source. So now they changed the API in there a bit as well to make it easier to work with that you really use a uni of a list of entities and

not a multi- of entities because that doesn't really make sense for for for this use case because your database is not going to have things like back pressure with reactor streams. So how do we code with this? Uh this is just a simple example. So we create a uni from somewhere. Typically the framework provides this where we do a database call or a restclient call. Uh then

we can say when we get this item we can transform it. So we can take the item and append the string hello. And the on item transform uh transform can also be a map. So if you're more used to functional programming languages like Scala, it's simply map. uh but it's a rather expressive API and it allows for nice autocomp completion as well. So if you do dot

on item it will show you all the things you can do when you get an item. Now then we can delay it. So like we saw in our example with uh our coffee machine we can delay our items and in the end we should subscribe. If we never subscribe we just register this uni or future but no work will be done at all. So if you never

subscribe to a uni or multi no work will be executed and you get nothing. But typically when you put this in a in a rest endpoint the the framework around it will subscribe for you. So you don't have to do an exp explicit subscribe. We then have a look at a multi there we can have multiple items or none where we can do an on item transform

as well. We can do select the first things if we have a failure. Typically, if you do reactive streams, if we get a failure midway, we want to recover because we want to be resilient. And if the the Kafka streams do a uh fails due to a network latency or whatsoever, you don't want to fail the entire stream, but you want to recover with something. And then

again, you should subscribe to the stream because otherwise the stream doesn't get Then in Caucus we have a whole ecosystem of react uh reactive extensions. So Quarkus Rest supports it where you can mix and match them. Uh the Quarkus REST client also supports both the blocking and the non-blocking way. Then you have a dedicated extension for Quarkus Hibernate reactive. So you can do reactive hibernate in Quarkas.

And nowadays in modern versions of Quarkus you can even mix and match Quarkus hibernate with Quarkus hibernate reactive. So you can have both reactive and non-reactive hibernate in the same application. You can have Quarkus messaging for interacting with Kafka or MQTT. You can have a non-blocking Reddus client and the list goes on and on and on. So sometimes you need a dedicated extension like Quarkus Hibernate reactive

to get the reactive version. A lot of times you can just use both APIs uh when you enable the extension. So let's see a bit in the code how that will look like. Let's close the blocking. So now again we have our SIM. Oh, that Yes. So now we'll have a reactive bartender and now the return type is a uni. So a future of a beverage where

we say to the coffee machine brew async. So if we have a look at the implementation you can see that here we have a rest client registered and quarkus then generates the implementation for you. So you just have to put on the annotations uh where you can find this uh interface and then it will generate the implementation. that we have the normal response and we have the

async response where it gets back a uni. That's the only difference actually. Now when we get back that response we can map it but as you saw before we can also say uh on item and on the on item we have a whole bunch of methods that we can actually do. We can replace it with a transform. So the on item transform is really the same as

the map here. Now the flat map will be transformed to uni. So then this mid thing in the middle would be uh a uni but you can as well. So you can also say flat map here if this thing will get a uni. So you can have both if like I mentioned before if you're more used to the functional uh paradigms need of map and flat map

you can use them. If you don't want to use them, you can also use uh the on item and then get this expressive API where you can really mix and match and it's a bit more easy to discover for newcomers, I would say. Uh or if you're lost in between, if you do just do all on item, it gives you all the things you can do with

this. Yeah, that's a good argument as well. I'll get back later to that in the presentation as well about AI. Um, I will confuse AI as well. Yeah, reactive programming with AI is also a bit more difficult. I would say AI is a lot better at doing blocking code typically. Yeah. It's going to be tricky indeed. So um if we have a here a panache entity, it's

now a reactive panache entity. uh the same for the repository. It's now a reactive ponacham uh repository and then the return times will be uni of something. So that's the only difference the the package name changing to punach reactive and then I get a uni back and then those can coexist in the same application. The the blocking package and the reactive package are still in If we

then have a look at the resource, we can still inject. And here we now see the return type. But we see we never subscribe. So I can call subscribe here. But I don't really have to because Quarkus will do this for you We have a look at the examples. If we say reactive single and here's already a gotcha. Reactive programming is not going to make this faster

somehow. We still have to wait for three seconds for the coffee machine. We can just schedule a lot more coffees on one thread. Instead of needing a hundreds of threads, that's the only difference. We can yeah the weight is not magically going to be less, but we can schedule a lot and handle a lot more requests at the same time before overloading our system. So if we

now have a look uh which threat handled this. So here we say it's uh on event loop thread five. So it's going to make a reactive beverage. It's going to warm up the coffee machine. Going to persist the coffee machine and actually give it And this is all happened on the same uh event loop thread. It has to make sure that we have hit the CPU cache

efficiently because if this midway will be to another event loop thread, it has to move around some bits and pieces for memory in the cache. So it's more efficient to schedule it on the same event loop thread, but it can of course if the system is really busy decide to schedule it on a different thread, but typically would keep them as close as possible because then we

don't have to copy over stuff from memory to another bits in the cache for the other CPU Now then we can do this uh sequentially as well of course. Let's see. And now you start to see where it hurts, right? If we want to do this sequentially, we're going to get this ugly nested flat maps. And three is not too bad, but I've seen way worse in

production code. This is still easy to look at. Uh, but if you put a debugger in, it's really hard where it has to jump and it can't keep really track and stack traces as well. But this is how you would do it. If I get one coffee, then call flat map. So this the on item uh we can replace the flat map also with uh transform to

uni that will be equivalent. So we expect something to get a uni out there. This is how you would typically do in functional systems. Also if you do functional JavaScript you get all those nested uh blocks of code. Let's revert this. If we then say do this sequential uh you could but this is the the proper way of doing it. Yeah. Uh and this here you see

it's event loops uh thread six doing all the work one by one. Well that's not really useful. So again we can do it in parallel. And here we have some nice primitives. That's where then mutiny shines a bit more. Uh you can get some nice utility methods. So here I just define my uni. So I define my futures and they don't get executed yet. It's just defining

I want some coffee in the future maybe then I say join them all. So go get them all in parallel and collect the failures. So instead of on the first coffee that goes wrong, throw away all the other coffees, I'm eagerly going to get all the coffees, but also going to collect the failures Yeah, you can optimize this a bit, but for readability, I kept it simple.

But you that's again then where functional programming comes in. people then get creative to get it squeezed on on one line and then nobody understands anymore what the system does. We always did the running joke at our company with Scala code. It's like hey I have this snippet of Scala code who can rewrite this in less uh lines or who can rewrite this the shortest and you

get crazy solutions and then in the end nobody knows anymore what it does but it does work because it compiles. So all right. So if you do them in parallel you can see it's still event loop thread five handling all the stuff scheduling all all the futures. Uh it's still the same one because it came in on virtual on event loop thread 5. So all the order

work is going to be done there as well. It doesn't need to spawn more threads because it just schedules more events that are handled by the same event thread because then we benefit again from the same CPU cache locality. uh so it's actually in quarkus well we have mutiny the programming language uh programming library and then under the hood it uses vertex and vertex takes care of

scheduling this together with netti yeah vertex and neti really handle the low-level scheduling for you and if we then go in here So but here you have other also other primitives that you can call right. So before we did uh collect all failures but now we can also do fill fast. So we can get the same uh thing again. Sometimes for your use case it might be

beneficial if you do a fill fast because if one of the things fails I can't be bothered to do the other things. So just cancel them and stop doing them. Yeah. So the remaining work. So for example uh if the the join all will schedule uh beverage one two and three all at once uh but if then beverage one for some reason fails is also going to

cancel out uh the unis for beverage two and three so that it doesn't do useless Ah, great. I'll try that. Uh, marble diagrams you said, right? Thanks. So, if we have a look at the gotchas, uh, one thing already said, blocking the event thread. We should never do any blocking work on an event thread or a thread. So, we shouldn't call thread.sleep or anything on on them.

Um context propagation is hard. So we can use the the context propagation from small rye uh as an extension in caucus or the handles automatically. Um if you ignore back pressure with a multi, you can still get out of memory exception. So set an overflow strategy explicitly. What should happen if my other downstream system can't handle my messages? Where should they go? Should I just discard them?

Should I write them to disk? whatever but that system itself is then responsible for uh defining an overflow mechanism. Well, uh debug already mentioned um transactions are difficult as well on reactive systems. Luckily, Quarkus makes it quite easy. You just still slap on the uh at transactional annotation and it works. We have to be careful what happens when and then again the operating learning curve I would

say with mutiny it's not too bad but yeah the the discussion with map versus flat map versus concat map or transform to uni and merge you really need to start learning all those terms and maybe settle on a strategy that you use within your team. Then talking about transactions, uh a couple of years ago we were hit by this. We built some internal application that would uh

aggregate some data from different systems. So imagine that we had an HR system for all our employees. We had a Google workspace directory. We had a GitHub organization and we wanted to pull in all the information persist some stuff to a database and then it didn't work. We were like a compiles normally with functional code. If it compiles the compiler says yes, it should be good. But

in this case with hibernate reactive and transactionals it can be really difficult. Now it's not per se too slow. Uh let me share that blog post. I'm not going to go through the entire blog post. I'll share that's shared as a resource uh in the presentation. Uh so here we say we evaluate unis in parallel. That's what we did before, right? With uh combine all uh where

we combine two service calls in one and run them at the same time. You nothing's bad, but let's see. Yeah. Sorry. Yeah, I think so. Yeah, I think it will be a fault or a fault left. Yeah. Yeah. Yeah. Good question. So, it's going to execute the because of the joint combine all it's going to spawn them in parallel then combine them and have some operator to

combine all the uni. So, that would be a fault in functional programming um you say this this work great. If I I'm calling external APIs, I can do this safely, right? If there's just rest clients, I can just spawn a whole bunch of unis call 20 REST APIs Then we have Hibernate reactive unis. It has this big thing, big yellow warning in the docs. But yeah, who

reads the docs? We didn't or less at least not this bit. But it explicitly mentioned the session is not threat safe or they call stream safe. Um, so you might hit things that are impossible to debug or even to detect. Um, but you can get wonderful errors session is closed. So some other uni already closed your session. So the next the other uni that was executing in

parallel took a bit longer wanted to write something to the database but the nothing happens. And the ones I love the is illegal pop with non matching blahy blah. And we had no clue what that meant. So we had some fun with this. Um, and it's mentioned in the docs. Again, not to blame people here or say that it's bad, but we didn't properly read this. That's

probably where you're going to hit as well because you can write it. It compiles in your basic test locally. works and then in production it blows up in your face. they explicitly say doing a session or assessing operations you shouldn't do concurrent opening those sessions. So you shouldn't do them. Um so the poorman solution is saying combine them using concurrency of one. yeah it's sequential again. So

it still writes as uh you can write and read it as parallel code but the concurrency says do this one by one on the same thread. That fixes the issue that works but it does it's still not great for throughput right but at least it fixes your exceptions and it works. then we had some issue with GraphQL. If if you do GraphQL queries, it fans out to

different service endpoints to get some data for your GraphQL uh data and it's the GraphQL engine deciding to combine unis whenever it wanted. So we were not in control of those unis. So we couldn't really do anything about that. So we couldn't slap on this using So what do other libraries do? For example, dubbie and scala uh doesn't give you uh for this type. It gives you

a dedicated type for this. So instead of giving you an IO type or or a future type, it gives you a connection IO type which then has a bit of information around that if you try to combine them that you can't do it. So the compiler will say no. Um unfortunately we don't have that. So we did another workaround uh where you can uh build your own

mutex with a semaphore. I'm not going to go into all the details, but there's a workar around with what we had an interceptor where we uh with a with a semalo and basically going to only do one by one. So you uh uh to write to the Yeah. Well, when we go into the database, then you do really one by one, but the rest of the code

you can just write whatever you can combine database calls with rest calls. You can combine all the things together. But when we're writing to the database or reading from the database, we're going to do it one by one. Only one uni gets can acquire a lock or a ticket from from this lock. So this one can actually do some work, gives back the ticket and then the

next uni comes in uh depending on your use case. If you I think so. Yeah. But depending on your use case, it might not be possible or you want to be have consistency. So you want to have your reads transactional as So that was a bit of a yeah a story that we did before. The implementation is straightforward. Uh an alternative approach is duplicating the vertex context.

Uh there you duplicate the session and then it will get an unique session in there. But it requires a bit more plumbing as well. But then it happens in its own transaction. So if you start duplicating the sessions, they all get their own new transaction. So if one rolls back, one might be committed and one be might be not. Depending on your use case with eventual consistency

might be fine. If you're working with financial data, maybe not. So there's some gotchas on how to handle And again, you do get great performance, but you do have to watch out a bit for all those C charge you get on the way. And it's always not so obvious what is really happening because if you see this illegal pop, you have no clue what was happening in

your application. At least I didn't and my colleagues also didn't. So, it took us a bit of time uh to get So, this is where we came for virtual have been talking about virtual threads for ages. When they ditched the green uh threads, they wanted something uh that they call virtual threads with project loom. So they did multiple previews and in JDK 21, we finally got virtual

threads. So all good you would say. Well, there are some good jazz because we had to synchronize pinning which we'll show later as well. But the main pitch is that we can write write uh plain imperative code again uh with reactive level scalability without doing the rewrite to reactive programming. So what what are virtual threads? Uh it's just a Java long thread but in this case it's

scheduled by the JVM. The JVM mounts it onto a carrier thread. And this carrier thread is just another platform thread because in the end everything on the JVM is platform thread that maps uh to kernel threads and when it encounters some blocking operations, it's going to unmount it, park it somewhere, store it in memory freeing the carrier uh and then it can do some work. And the

carrier pool defaults to a fork join pool sized to the number of CPUs. So it looks a bit like the event loop threads or IO threads that we had before. We had have a handful of them. They do the actual work. They get scheduled by the kernel. So then we have continuations. So that's what underpins virtual threads. That what allow you to do virtual threads. And it's

a suspendable resumable piece of sequential code. And Loom uses a stackful continuations. And each continuation has its own call stack. So all the steps that we have sequentially before it, that's the call stack that we have. Uh and that they're limited. So they yield in a bounded scope. So they won't yield somewhere else. And they will also be yielding exactly once. And then we can discard the

virtual threads. So we're not going to reuse virtual threads. We're just going to spawn one for some task. When it's done, we're going to throw it away. This continuation that's underneath it is explicitly not public but it's a JVM internal. So why should they be stackful? Uh a suspend point can be anywhere in the call stock uh call stack. We don't need function coloring. Think like suspend

or a way to sync uh from JavaScript. We don't need that coloring in Java. uh with this stackful uh solution and existing Java code without you putting on labels like run on a virtual thread or make it a suspend function or a sync a weight. The same blocking code we can now just execute on a virtual thread. That's also what Spring did. You just put in your

configuration run on virtual threads and all your rest controllers run on a virtual thread and you didn't have to change any any code. I'll get there later. But yeah, there's definitely not a oneshot approach to just enabling virtual threads for everything, but you can. Just saying that you typically uh if you done it correctly, you don't have to change any Yeah. Yeah. You're going to have some

nice war stories probably uh of doing this. Yeah. So, I'm going to show you some examples today indeed when you should not and when you should use uh for contrast cotling co- routines are stackless uh but they require an explicit suspend in this case I'm not going to go further into all the internals of the JVM I'm just going to see what is really useful for us

and for us as developers on how to use them I think there are some excellent talks about from other people that going to go really level low level into continuations uh etc so when we have a virtual thread theuler prs a virtual thread so with a ball bunch virtual threads. Uh the stack is frozen somewhere. It thaws the stack onto the carrier. Uh runs some real code

on the CPU. Then the the virtual threads hits a blocking operation. So calling an external rest API. The JVM freezes that stack, puts it copying those frames to the heap, and then it can be cleanly unmounted and then this platform thread or carrier thread can grab another virtual thread, do some work with that virtual thread. And again in here as well, carrier threads itself should not block,

but it's managed by the JVM. Uh, but that's where pinning comes in there. it could be blocked somehow because the virtual thread couldn't lift it out of there, but we'll see that later. So, if we look at this image of the the requests again, I would say it's basically an overlap of the two images we saw before. So, we're going to have a whole bunch of virtual

threads. So, per request, we're going to have a thread again, a virtual thread. So we can have the thread per request model again, but we're only going to have one carrier thread. So that match matches again with the event loop The virtual thread one does some useful work, gets mounted by carrier thread that does the actual work and then it can unmount it, grab virtual thread two,

do some work with virtual thread two and mount that one and so on and so on. But the the request one lives on virtual thread one and there it can actually do some blocking operations without really blocking actual platform threads. So what I mentioned before is a dedicated fork join pool with the number of available processors with a work stealing algorithm on there to efficiently like event

loop threads do as much work as possible. um you're going to have roughly uh uh 256 platform threads to compensate for all the other blocking work, but that's uh more details that we're going to cover today. There are some knobs you can tune for for those pooling, but it's more just more for you to be aware. You can tune how it works, but typically you don't really

need to. what can be unmounted? What can the JVM actually unmount and put back on the heap? So stuff with sockets with uh Java NIO channels with files for example a thread sleep you can safely do a thread sleep in your code. Now uh stuff like uh from the Java util concurrent package uh with re-entrant lock semiforce etc uh completable futures futures and synchronized blocks. So sometimes

you would have synchronized blocks in your code before Java 24 that would also still pin uh the virtual thread to your carrier thread but they now fixed it at the same time. A lot of JDBC drivers were actually doing this. So they all onto Java 21 have been fixed mostly and replaced by reent locks or semi force. Uh but in the same time the JVM caught up

fix it itself. So you wouldn't need the fixes anymore uh from the JDBC drivers. In 24 it's fixed. Now in 21 it's mostly the libraries that fix themselves. So in the hot pass of your code it wouldn't really happen. Uh but on Java 24 it was really the JVM that was fixed uh in there. So when can I still pin? if you do native method calls or

use the foreign function and memory API uh because then you're doing stuff low level and then the JVM can't really see what you're doing. It can't do like a low-level process, grab that and put that somewhere else. There you can't really use virtual threads. Oh, you can observe observe virtual threads. There are a bunch of uh JFR events that you can monitor for uh but typically don't

really need to. So if we go what will be the benefits uh of virtual threads? They're really cheap. Go ahead. Yeah. Yeah, I've seen something about asing ST traces, but I can't remember if they already fixed it or in which JDK, but uh yeah, but there's some works in need to fix that as well. Good point. Uh so the strengths you can have millions now instead of

a couple of hundred or a couple of thousand you can just spawn for everything and you can create a virtual threat, but never reuse them. So don't pull them, just use them once. And they're cheap to block. You have low overhead uh per thread. So the continuation is stored on the heap. This just a small fraction of that continuation. And it doesn't allocate the full one megabyte

for for a stack frame. Uh you have the same thread API. So the same uh Java API that we're used to doing uh concurrency. some limits. They do not add more magic or more parallelism by default. So if yes, sure we can have more virtual threads. We're not magically going to be handling thousand more requests at the same time because your CPU still needs to do the

actual work. It's not going to magically make a computation faster. If you had a slow running computation, they actually needed CPU work is not magically going to make it faster. Might even make it slower if they have to do mount amount of virtual threads. Again, do not pull virtual threads and do not use thread local freely in that sense as well. There we have uh now scope

values and we'll touch upon those a bit later as well. So to talk about pinning uh synchronized blocks caused the the virtual thread to be pinned to his carrier and that held the OS thread really hostage. So he had a synchronized block somewhere, it really would pin the virtual thread to the platform thread. it couldn't do anything in there usefully. So we had some JDK uh some

JFR events or some JDK arguments JVM arguments where you could say like trace those find them where I use them in the application. You would hope you would find them during CI but typically you would find out in production if you would hit those. Well, luckily from JDK 24 onwards with 491, the synchronized blocks don't really uh don't no longer pin the virtual threads. So, we also

don't need to trace them anymore. Uh the JFR events stay for the obscure cases and you don't really have to replace your synchronized block with re-entrant block. Just keep the code and be done. Again, native calls can still cause spinning. So, be careful if you're doing So if you visualize this, if we have a carrier thread, which is a platform thread, it mounts a virtual thread, doesn't

work, but it can't be unpinned because it's blocked by the synchronized block. In this case, it couldn't unmount it. That that was the problem back then. They fixed this now, so we're all good. Um, thread locals with virtual threads, they still work. you still can still use them for for some use cases. They're still useful. Um, but they're got yeah, it's really harder to work with. You're

gonna blow up your memory of those uh with large values and you also need to propagate all those values to your uh millions of virtual threads. So instead, they came up with scope values. They're for one thing immutable, which can be annoying, but it's really good because then we can freely copy it uh to other virtual threads that are children o of my virtual thread. And they're

cheaper than a thread local because well, they're not they don't have to be mutated. And they're perfect for request context, uh your authentication principle, correlation IDs, etc. So on a modern JVM prefer to use a scope value where you could create one and that scope value only lives for the scope of that ver virtual thread that was Then another big thing uh what I mentioned before briefly

as well is CPU monopolization. So a virtual thread is actually doing some CPA CPU bound work. So it's actually doing some meaningful bit of work. So there are no yield points. is not waiting for for a for Kafka or a database. So it really needs to do keep on doing work. Now this this carrier pool is small. So only the number of CPUs you have and then

if I have a few CPU intensive uh task running I can't handle any other incoming request anymore because they're all doing something already. So how to handle this? you typically offload to that to a fork join pool dedicated for that computation and then it's again the OS and the kernel taking care of that scheduling for you. So then you'll have a bit more fair scheduling so that

your carrier threads can mount those virtual threads handle request and do something useful with them without the CPU work really uh bounding your application. So to make sure that we can still reply to a request, if you see this in action, um virtual thread one comes in, does a whole bunch of computation, we're going to need another carrier thread to do that work. And we're going to

run out of carrier threads because if we only have four CPUs, can only do four of those things. And if then a request would come in, in this case, request number three, nothing can happen and the user gets a timeout again. So that's why we typically offload all the CPU work to a direct fork jump pool was a lot better at scheduling all these workloads. So virtual

threads um discovering them um in dev mode what we have in quarkus is a nice thing a virtual thread typically doesn't get a name because they live so short the JVM doesn't bother giving them a name. So been on with Kubernetes with uh cattle versus pets. Uh on a farm, you're also not going to name every cow, but maybe if you have a cat at home, you

would name it. The same with a virtual thread. We're not going to bother naming a virtual thread. Uh but in quark, you can have the naming by default it's on. So it can easily trace uh what's going on. So that stack traces and thread dumps identify really where you're at. So makes it a bit more easy to debug. Um and you have a JUnit extension where uh

if you were still on Java 21 where it could detect uh the the pinning the virtual T pinning and that does this on basis of a JFR event. So it runs your test maybe for for some rest controllers and if it emits some JFR events that something is spinning a JUnit test would fail. So you can just set some guards around it like hey even if you're

now on a modern JVM like hey I do some low-level stuff I know that it pins once and I think that pinning is fine you can set a guard like should only pin at most once and if it doesn't know suddenly 10 times you know you've broken something there mostly that you don't really need to worry anymore about because well the pinning has been fixed. This is

basically a summary of what I said. If we then have a look at the code, the code is going to be quite Yeah, that works. So, the bartender is just going to look the same as the blocking case. Uh the same for the entity. We just use our normal panache entity. We have our simple repository again. all the same. The only thing we do is put on

an annotation and run no virtual thread. So in spring you would put it in your entire application config. Here you would just put it on a on a on a resource or on a method as if you say only this method can be uh done on a virtual thread. So you explicitly opt in per use case. Let's come back to what you mentioned earlier. Instead of enabling

it for everything here you can be a bit more deliberate. So like, hey, I know for sure those things are safe to run on a virtual thread or make sense to run on a virtual thread. You can enable it. And there are some discussions going on. Should we make this a default? Yes or no. Um, then the simple get beverage. Well, that's going to be an easy

one. And here we see indeed it got handled by virtual fat zero. And in this case it has a name. If we want to do the sequential, we can just do it in sequential. Nothing special needed. So I'm going to skip this. If we want to do this in parallel, we're going to use an executor. But this is going to be a special executor. The one that

we inject now is going to be a virtual threats executor service. Yeah, because by default we'll give you a blocking pool and this way it will instruct the framework to give you the virtual thread pool to give you a virtual threat instead of a platform And I think in in spring you can do the same create a beam that does the same. Jakarta you can specify an

annotation as well to get a virtual threat in this case. So here we submit them again. Uh so we do parallel. So what we see now the request was received on virtual thread one. Then the executor service is going to give us a virtual thread per task that we spawn. So we got virtual thread two, three and four. And then in the end it all comes together

on virtual thread one. So it comes back on virtual thread. It join one joins it gives it back to the user. Well, there are some other things you can do in this thread factory. Instead of ex uh injecting uh the executor service, I can get the current thread name. This is going to be a virtual thread. Uh then have my own uh uh builder here where I

specify uh some things I want myself. So you still have this builder API here as well if you want to explicitly have something. So if I now do the custom one, now I decide the naming myself, right? So the framework receives it on Quarkus virtual thread 5, but I spawn these threads myself instead of the framework uh doing this through an executor service. So I'm really in

control myself that I can spawn them myself per all right let's go back to the slides so with virtual threads you just slap on the annotation or in spring enable in the config you're done you don't have to do anything so we're going to see some different options we have cotling co- routines they have been around for quite some time now. And indeed here you need to

annotate uh all your functions or put the suspend keyword all over the place. So if you have all those things the the fetch user is async so we put a a suspend there then the other function calling my fetch user fetch orders also has to have the suspend keyword. So you will scatter all around the frame your application all those suspend keywords. And with virtual threads, you

don't really have to do this because uh with the the stackful uh approach that we have here, they're lighter than uh threads. They run on dispatcher, but we're not going to go into the low-level details of uh co- routines, but it's basically the compiler transforming suspend function and state machine and dispatching that for you. And they have a co- routine scope. So these variables live only for

that scope of the corine. So that's how we can have scope values in cotlin as well. The nice thing is here you have structure concurrency built in from day Let's skip over this one. If we don't going to have a look at struct currency is the missing link. Um so what problems does concurrency solve? uh we can have threat leaks from forgotten subtasks. So task are spawned

somewhere and we forgot to cancel them. Um we don't really have a good semantics for cancelling task across uh tasks. Uh the error handling is ad hoc and often wrong and no clear parent child relationships in thread dumps. So that's what you mentioned as well. It's really hard to figure out which threat runs when and this gives us a bit better hierarchy. So here we can have

a structured task scope who owns a set of subtasks. So uh this looks a lot like the the copland version where we open a structured task scope and we give it a joiner. So that's what we saw in mutiny as well where we give it a joiner. So like all successful or it can also do some eager has some default options but you can also implement your

own joiner if you would want to. Uh then would fork that scope. Uh so you can see you create some subtask in there so that those tasks get the same scope then you join back that scope and then you can give back the result. I would love to use this in production but please don't run with d-p preview in production. Test this on your apps test it

in dev mode or in test somewhere but don't put this in production yet. It's probably will probably work but it's still in preview. So yeah, if your production breaks down because of the preview feature, you shouldn't want that. Uh so it has been incubator at the same time uh as virtual threats were in preview and they've been iterating over and over again and the API the public

API keeps on changing ever slightly. So they don't do big changes but for example in JDK2 they added some static factories. You had the joiner API so you could implement your own joiner. So your own strategy for fail fast or collect failures. Uh then in preview six we had the on timeout and now in preview 7 that I tried out this weekend. Uh you can have checked

exceptions again uh from your join method. So you see step by step they're tweaking it before giving it to the public but you're welcome to try it out and uh give them some feedback as well. So if we have a look at that example so nothing's going to be different here. Uh let's jump to the interesting bits. So we still say receive the request on a virtual

thread. So the framework will handle the request on a virtual thread. And here you can see it in action. So now we're going to open a scope. So this is a bit like the uni combine all. Then we give it a a strategy and it has a default strategy. So if we look at this builder here, uh the default strategy is await all successful or throw. So

it's going to wait for all the tasks to come back and if one fails, it's going to throw you an And here uh yeah this API has been changing. So every time I had to adjust my demo a bit for for every uh because the API will be slightly different. And here you can now safely call say uh uh get because this all runs on a virtual

thread. So even the is not blocking. We can safely do a get here because well we don't block the world. So we can just safely call get again on the future. Let's see there an action examples and here you see it in action. So the framework uh handled the request on Parker virtual thread 6. Then we spawned some virtual threads. They don't have a name anymore because

a JVM created them for us. It doesn't care about giving them a name. And then it joins back here on number six. Again, if we then do a custom solution, well again we can uh implement our own thread factory, give them a proper name. Uh then we can give our own joiner. So here we can implement our own joiner if we want to and specify a th

factory. The nice thing is if you make a typed like this. So if I type my joiner now the JVM knows that the result of a joiner otherwise above the we see above uh it will be just something called object. So it didn't know the type but if you give it a type hint here hey all your subtask are going to be a type of beverage then

I can directly do scope.join join and I get a list of beverages. So you don't have to do all the get because yeah the the library under the hood does all that work for you. And that's one of the examples that changed as well that wasn't there before where you really had to do it explicitly. Now you call dojo join and it knows the type and you

get back a list of results. Yeah. So I think that was in the latest where they added another type argument with the so you can have a typed exception of your own that you can return back yourself. So you can have a domain specific and uh exception for the case as well. So that's another good example after some feedback they incorporate those things as well. let's skip

the other examples. So we saw a whole bunch of examples. We're mainly using Quarkus. And in Quarkas we can combine imperative, reactive, and virtual threads. We can do all of that at the same time even in the same controller. We can just mix and match whatever we want. Uh with Spring Boot, it's going to be blocking by default, but as mentioned before, you can just enable virtual

threads via one line config. is a bit dangerous because well maybe some of your code is not really suitable but now that the pinning has been fixed think for most most crit enterprise applications should be safe to enable test it in your test environment but then you should be good to go uh then you have spring web flux it's still reactor based but you see a lot

of people moving away from from that uh especially now though yeah we have spring virtual threads so people are slowly discarding web flux then we have heldon Helen 4 was fully rebuilt on virtual threads. So it doesn't use netti under the hood anymore but it was really fully rebuilt on virtual threats. It's a one by the Oracle team. Uh Micronaut you can do both as well and

in vertex the low-level vertex you can also have virtual thread vertical. So basically every major framework supports it and I think Helon even to took the extreme they rewrote it fully got rid of Netti and do it fully virtual threat based. um if you do still want to use web flux uh if you do streaming data serverside events uh reactive pipelines uh if you need explicit back

pressure then those systems still have a valid point right if you're doing spring uh or if you're building edge gateways where you have high fan out uh then it's useful or or you just have a team that's really proficient at writing uh web flux code why not then you can still keep it will perform greatly Well, that's what I mentioned as well. Uh, let's skip that. Yeah.

So, this is what I like the best in Quarkas. You can actually have all three paradigms at the same time and in your same controller even uh you have even more fine grade control where you can slap even if the type is a uni, you can still slap on blocking and Quarkus will handle it still as a blocking goal even if somewhere is a uni. Uh, so

you can really mix and match, but for your modern credit end points, you should probably just use a virtual thread. Then people ask me, what about performance? Do virtual threads really provide you better performance? And again, as an IT, it depends. I don't have a straightforward answer. I wanted to do some performance testing myself, but then I was like, no, measuring performance is way too hard. The

Quarkus team also did an episode on this. Uh building a benchmark is easy but building a good benchmark is really really hard. So they have a performance lab at IBM Redhead where they do this. Um yeah the comp the to do this is really complex to get reproducible results. If I would for example do this on my MacBook uh it get throttling of the heat. I have

slack running in the background. uh is going to make a disaster of your benchmarks. So they did it the proper way. This is official numbers they published on the the website as well. So you can say Quarkus is already ridiculously fast. So it's already super fast compared to a same workload on Spring. Then what about Loom? What are the numbers for Loom? Well, those numbers are a

bit iffy. They do publish them. So, I didn't put them in the presentation. Uh, but you can view them online. Uh, but every time they do some performance test or they tune something, those numbers really go up and down. So, I saw some slides where it was really a lot faster, sometimes a bit slower. So, they're really tweaking the performance benchmarks as well to see like, hey,

how do we properly benchmark this even? So if I open um my other screen. So this is the the benchmark I showed you. Here you can see all the numbers. And here you can see that if we enable virtual threads and quirkus it might be slightly slower. Spring does benefit a whole lot. Still not faster than Quarkus, but benefits a whole lot. So that's why they have

the toggle just enable them because well you gain three times the performance apparently in certain workloads. But with Quarkus it's always a bit yeah doesn't really provide us more performance because they with uh because of these uh event loop threads under the hood with vertex and they were already ridiculously fast and optimized and then virtual threads are always a bit around sometimes a bit slower sometimes a

bit So some other things that they're working on uh I can show that here. Uh France from IBM did a nice presentation. You should really check it out if you want to go more lowlevel learn about context switches. But the biggest problem is that there's too many context switches happening. So if we have quarkus event loops they were pretty heavily optimized for not doing too many context

switches. But because uh the virtual threads uses own uh scheduler, you're going to have too many context switches. So then again in certain workloads is going to be slower and a lower throughput. So what he was a customuler right now the the main branch you can't implement the customuler. you really have to grab this special Loom build which exposes a custom scheduleuler as an API and then

you can do better. So he's experimenting can we implement a better scheduler and giving feedback to the Oracle team in how they could optimize the scheduling of virtual threats especially with uh IO threads and there especially those numbers are crazy right uh the reduction in context switches uh that really hits you in the performance so we he tries to really reduce the number of contact switches but

this is all work in progress it's more to show you there's more behind the scenes And if you want to watch that talk, I really liked it uh a lot. So it gives you a bit more details about the internals uh of virtual threads of the schedulers etc. And that's Francesco Negro doing that. Then we're going to finish a bit early today because already saw some beers

outside and I think uh was been a long day already for everyone. So let's not stay the full two hours here today. Um before you go home, we have uh a few charts. So if we look at the programming model, the blocking code gives us imperative code. Then reactive gives us those async operator chains. And then with virtual threads, we got imperative code back. What will be

the throughput ceiling of blocking apps? Uh the thread pool size. I just don't have more CPU or memory to to spawn more threads. In reactive, it will be really CPU and IO bound. And the same is true for combined but for a single endpoint it should be non-blocking all the way virtual threats well we had a limitation do not p uh do not pin them but that's

okay in modern JVM so you can forget about the entire pinning story mostly as well so when does blocking shine if you just have a uh a small back office app your enterprise credit app why bother rewriting it a blocking app will just work fine has been working fine forever why bother changing it you could enable virtual threads if you want to, but yeah, it's not if

you don't have a high throughput anyway, why bother? pool under high load. That's what we discussed earlier today as well. Uh, reactive uh yeah mixed paradigm teams. If your other half of the team or your other team doesn't do reactive, how are they going to maintain your application? and vice versa. So yeah, they're going to have complex flows to debug, etc. And virtual threads don't fix your

CPU heavy work. It doesn't make magically your application faster. That's what a lot of people thought like, well, just enable virtual threads going to make it faster. It's not going to fix that, but we all know that now. So again, pick blocking for your simple apps. uh pick reactive when you have edge gateways with high fanout aggregations for example when you have a lot of microservices maybe

edge gateway where you need a lot of control and high fan out makes sense to do that one reactive but then maybe the downstream services could just use virtual threads and then picture concurrency when it's final don't enable a preview in then there's a whole lot of reading whole lot of jabs uh a whole lot of I reference here as well. You don't have to remember them

because you can just scan the QR code and uh then we're done for today. Any uh questions so far you want to discuss or ask? >> that's for the recording online as well. >> Thank you. Uh do you understand correctly that virtual threats really helps us uh to save processor time somehow but it uh doesn't help us to use memory more effective. >> It should also use

memory more effective. Yes. >> Also >> yes because you only need a handful of threads uh called carrier threads and those are platform threads in the end as well. opposed when you have an entire pool of like 200 or 500 worker uh thread pools, they need five times 500 times that amount of memory. So you only only need a bunch of platform threads. You only use that

tiny amount of memory instead of spawning a couple of hundred. So is also more uh memory uh efficient. >> Really more efficient than >> Simple. >> Okay. Thanks. Any more questions? Yeah, pass along. So for the final conclusion, is it worth it to learn or to implement reactive programming? >> So then I will go back to this slide. It depends. probably for your normal CR app or

enterprise app probably no don't adopt it. Uh but if you're doing uh uh streaming use cases edge gateways with where you need high throughput then you could use it. Yes. But most people that I know don't really do that scale. So then keep it simple Oh, can you use the microphone for the recording? >> What do you think about a sync single >> Uh, could you elaborate

on that? How would you >> Because you said blocking uh you you mix like blocking with single thread. >> Yeah. Yeah. There are threats which cannot which can be a single single for example lot of runtimes like let me give example I can give lip for example it is a single threaded event that is a sync it is used by NodeJS and all of those types of

frame >> they offer a this concurrency model which is single threaded a sync and I I thought that maybe it should uh get it place here on its list because uh >> yeah I didn't really consider it indeed but it's a good suggestion to maybe >> look broader than the JVM ecosystem indeed what the water systems do and what we can learn from those >> for example

um in my high frequency trading days uh I don't care anyway so uh what I meant is in high frequency trading uh people >> those consumers you know you also split them between publishers and consumers but every consumer is a single threaded thing that is pinned to a CPU >> and this way you don't get context switching or all of those things and for example a lot

of high performance databases use this for example SK DB and um maybe it is something that is worth exploring for you. >> Yeah. Yeah, definitely good suggestion. I think it will be a nice addition to this overview as well and also in the beginning for to give some more context. So, some more homework. Thanks. All right, then. I wish you a nice uh evening and a nice

day tomorrow as well at the conference.