Plan Before You Build: Deterministic Planning Patterns for AI Agents by Dan Dobrin @ Spring I/O 26

This talk focuses on the complexities of planning when using generative AI and agents. The speaker, Dan, an architect at Google Cloud, discusses the unpredictability of AI behavior, such as untestable actions and the hidden costs of token usage. He emphasizes the need for observability in agent-driven systems, where traditional deterministic approaches are insufficient. The session introduces a multi-model deliberation pattern, equating it to having a council of specialized agents that collaborate to improve accuracy and reduce single points of failure. Dan presents a production-ready application built with Java, Spring Boot, and Spring AI, showcasing various planning patterns, including goal-oriented action planning and hierarchical task networks, along with their respective advantages and trade-offs.

[music] >> Hello everybody. Welcome to this uh session on uh planning on planning. So, let's hallucinate together a bit. I was thinking that in um and that's just a personal observation in the age of agent AI we tend to drive uh to dive straight into implementation. Connect an LLM, add some tools, resources, and then let the system figure it out. Not very architecturally, if I can use

this term. I'm Dan the Overlord and I'm an architect uh Google Cloud. I focus on application gen AI uh gen AI code assistant tools. It's uh a distinct pleasure to uh uh join Spring I/O as a speaker uh this year again. Um Hey, I was thinking that when um I was planning this talk, that we would talk about determinism in agent and a number of planning patterns,

kind of like plan on how to plan our agent. However, as we as I was looking into what is the actual element uh elephant in the room, a number of different things I actually come to mind. So, sure. Uh Hey. That's the elephant in the room, unpredictable paths. So, same input, different execution paths every run. Untestable behavior, traditional unit tests simply don't apply to agents. There is

a hidden cost multiplier, so token usage without guardrails could compound uh quite fast. Now, if we are looking at debugging, which we were accustomed to, there's no stack trace in terms of like why did an agent actually decide this? There's this infinite loop risk of agents chasing their own tail if there's no exit condition. But one thing which uh stuck to mind was this idea of model

variability. The same prompt generates widely different answers. The second thought came to mind as I was uh exploring this is in terms of observability. In [clears throat] the past, we were used with a deterministic approach. Code was the source of truth, and we were measuring CPU utilization, memory limits, whatnot, network latency. But in the agent um in agent observability, behavior is non-deterministic, and reasoning is the source

of truth. So, many things uh become interesting for us to capture. Tool call sequence, the reasoning path, how did an agent uh get to this particular conclusion, the decision quality, the time to the first token, user experience could be affected if that is not um is that that is not um appropriate, and token consumption. It's not just an issue of cost, but also the idea of token

efficiency, how many tokens do we spend to get a quality uh response. I built this uh little application, will be running now. I hope you can see that. So, the same system prompt in an application, the same user prompt. I want a brief answer to this question. Spring I/O is the de facto premier conference for Spring in Europe. So, as different models, and I can configured a

few, two of the Gemini uh cousins, and uh a model from uh Cloud, each one gives us different answers. Gemini flashlight as well. To evaluate this, here's the verdict. Haiku says, "Well, requires context." Gemini Flash says, "Well, Spring I/O is widely considered the de facto conference." If we aggregate all these responses, the answer becomes like, "Here is the case for uh the case for yes, and here's

what we decide." So, we see all this model variability. Let's run this again while we look into into this uh pattern. I'll run this again in a different way. I think this agenda had to change a little bit. We have to add durability, and we have to introduce the multi-model deliberation as a new pattern. Not the original patterns I had in mind. Let's add this LLM council,

and in the end understand also what consensus metrics would come about. So, this uh pattern uh came out from an archive paper, then Andrej Karpathy of Tesla and OpenAI fame um by putting it in exercise. And the idea is that no single model is always right. A a council of models is more reliable than any one genius. The problem is that single LLMs hallucinate with confidence, like

with no built-in checks or balances. The council takes a number of specialized agents, deliberate, challenge, and synthesize the answers with an outcome of a higher accuracy, explainable reasoning, and reduce single point of failure. Now, why would we do that? What do we know which model is best for our query? So, the problem is that no single model wins across all tasks. And without structured evaluation, you're kind

of like guessing. The architecture inside, when as we build this, we want to separate the generation of a response from the evaluation, and have clear optionality across generator and reviewer, and reviewer tool. Conceptually, imagine [snorts] a room of experts, each from a different school of thought, let's say all of us here, we're given the same question independently. Each one answers anonymized, then each one ranks and critiques

the others without knowing who uh who it is. And in the end, the chairman synthesizes the collective wisdom into one into one answer. That was the original idea posted um in that paper and discussed by Andrej Karpathy. I've added a number of things to show that actually there's even more variability here. The chairman could be one of the advisors ranking it or not. Why? Because LLMs generally

tend to be harsher on evaluating their own responses rather than somebody somebody else's, even if anonymous. So, the architecture of this application, we start from an application that is production ready, from code uh um coding standards to retries, to all the uh features which we are accustomed to, but we're not saying this is the most efficient thing. This has to solve a problem, and we're going to

evolve this as we go along. So, we want to have a five-stage deliberation process, which we run now. In the meantime, it collects individual responses. It will do a peer review ranking. So, each one ranks it anonymously, notice like different letters so that uh they don't know about each other. Each one provides a ranking. There are separate non-LLM actions, call them tools if you like, for aggregating

the rankings, pure math, and agreement analysis, disagreement analysis, capturing some metrics, and the final consensus. So, we can have these five steps, which we would like which we would like to use. But we want to see also traces for this particular for this particular application. As we look at traces, this becomes now important for us to see which model, how much uh what the latency was, and

take all these numbers as this is just we look at them comparatively, don't take them as absolute uh numbers. If we run it again, it's going to be a different value. and if we uh if we are looking at um if we are looking at these numbers, we want to observe them also in a tracing um in uh tracing. I am using here uh Google Cloud Trace.

Uh it is available to me, and we can see the correlation, and we can see different information about each of the each of the agents. So, if we look at uh the last one we ran, it is more it involves all the steps. We can see each one, and we can see the agent reasoning. Which one was called, uh how each one uh runs, what data we

collected, and we want to collect things like uh from example for example, like the token prompt, the time to the first uh response, different information, as well as aggregate information, were there any errors or not in um in our application. The app is built with Java 25, Spring Boot 4, Spring AI the latest 2.0 milestone 4, and uses a Vaadin uh Hilla 25 uh the latest versions

of each one uh available to uh available to us. You can run it as well deployed in Cloud Run as a serverless environment, and it it scales up. Now, what design principles did we apply here? One is anonymity. Models evaluate content on merit alone. Collective intelligence, because if we calculate agreement agreement uh will point surf uh will surface high confidence inside and this agreement will point areas

of genuine uncertainty. Then structured deliberation because we have fixed multi-stage pipeline it which ensures that every perspective is served. And provider independent all LLM providers are interchangeable members. I picked here lightweight models like flash and flashlight but you can just substitute them the architecture of the deliberation process doesn't change if we use Gemini Pro or Claude Opus or whatever other model we would like to do. By

running them in Vertex I can pick any model I want from from hugging face and just and just plug it in. So, what case what use cases do we have here? Well, one is I'm not suggesting that you should run this for every single every single model code because you run multiple models therefore they're good it's good for model evaluation for testing to see which path applies

to your particular use case. It's good for creative brainstorming at scale for fact verification and cross-checking across each models complex multi-factor decisions and if we were to apply this in any one industry investment and financial research where you want multiple analysis would be one area to focus on. Now, we know the advantages what what are the trade-offs of course there could be multiplied API cost and of

course latency. Parser fragility not all models support structured responses. Sure, the large models do not all of them do that. This first implementation has a volatile session state the state is in memory instance fails we lose that. The chairman is a single point of decision so of course you should always try different models as the chairman. And we can improve we can improve the performance of this

particular this Now, what is our next step? Let's use and apply this for all the other patterns I had initially in mind. So, we have an entire we have an entire spectrum from strict compliance of the workflow deterministic graph very clear. Strict execution one step leads to the other. At the other end we have the supervisor LLM where an LLM orchestrates the delegation of different sub agents.

And we can impose some constraints so that's why we call it bounded flexibility. In the middle we I firmly believe that the truth is always there's never like a black and white. There's a hybrid approach and a combination of multiple multiple approaches will be the winner in the long run. The goal-oriented action planning uh approach which was first released a year ago in um in bubble is

now available together with workflows and supervisor LLM in Google ADK. I have added these planners last Friday so this is the first view you can see them you can see them in action. as we explore these three there are also other patterns which are closer to let's each one of the edges like a hierarchical task network is a workflow which has which needs one LLM call and

execute the actions. On the other side utility AI which comes from the gaming world same as the goal-oriented action planner is used for explorations where we don't necessarily have a goal in mind. So, let's look through them and look at the two ends of the spectrum. So, deterministic compiled time graph zero planning calls predictable reproducible preset compositions. If we take an image we have to analyze it

enhance the image store it that's a clear workflow usage. On the other hand non-deterministic where we need runtime LLM planning if we have in our case either three actions or five or two just for a quick consensus we need an LLM call in between but we can on the other hand interact with it with natural language instead of having predefined here I click a button we can

say have a folded deliberation or get a quick consensus which is more natural to And it allows us to use adaptive partial workflow. So, ADK workflow on one side ADK supervisor LLM on the other side with a number of others between. We're going to explore them one by one and jump from one to the other and leave the goal-oriented action planner for the end. So, the workflow

pattern compile time it's the same if we think about it the same application as the first one but now you're using having a reusable agent. So, we're introducing agents which we will use unchanged across all the pattern. The way we what changes is the way we orchestrate and the way we the way we plan for them. So, layered approach what can we change here? It has sequential

parallel steps so one thing we see once we have the individual response we can review but we can also get agreement and disagreement they all need the individual response. So, one good thing would be let's parallelize these these requests. Now, the advantages are zero overhead deterministic lower cost if we think about it that we know we can quantify the cost. That could lead also to higher cost.

The shared agent code base all the way across all our different planners. And we know that from the beginning. The fixed presets no natural language control no partial workflows rigid parallelism which we impose. So, potentially also higher cost. If we want a quick consensus why should we run for example all five of them? That might lead that might lead to a different a different cost. So, how

about we run this now in we run a workflow. I'm going to run them not in Cloud Run I'm going to run them in on the machine just so that I can show you some of the logs when we and some of the adaptive re-planning which we will see in later in later um models. Sorry, patterns. Can you see these logs? All right. So, if I'm going

to run here the same the same question which I run before and I'm going to say run this deep I expect this to take less time than the previous one because I'm going to parallelize these operations. So, I can see them I can see evaluation of responses and for each one we can measure the model the latency as well as the token count. And we can see

how it would execute one of our one of our operations. Oh, very nice. I was running the wrong um absolutely. No wonder. I know. It have too many allow this to start. You can see this you can see it uh start up. All right. This seems like a little different. Let's let's allow this to run. workflows very very clear. But you saw this application started the spring

up it started in probably like six seven eight seconds. Is there another problem when we think about agents? Start up end start up performance which was important to us and throughput in services before especially in serverless environment or in environments with high variability in scaling is important to us as well and we can run native image with GraalVM which will start in a few hundred milliseconds so

we can put the rest the scalability issues. If we move into the Gemini world does not mean that we should be we should be changing the approaches which we have. this run this run and now we can see in the traces we can see that this entire flow now run in 29 seconds. Why did it why did it do that is because if I'm looking at the

trace log I'm expecting to see these operations in parallel. So, it is a controlled parallelism which we have which we have available here. So, we can see them all running in parallel. All right. So, this would be this would be fine for let's explore the other end of the Bring this back and start our other application. We can run the same. So, at the other end of

the spectrum, we have the supervisor LLM. So, the supervisor LLM is like LLM-driven planning. It has a core loop structure and some design invariants. We observe, you build a context from the current state, you look at the agent catalog, all the agents which are available to you. Then, an LLM will select the agent to execute. And of course, at the end, it's responsible for emitting a done

signal so that the loop finishes. It executes the agent, but it can decide run some concurrently or some not. Updates the state and continues continues the loop. There are some design invariants like stateless planning, time after each agent call, it has to reread the state and build its context. It's probabilistic dispatch. There's no workflow here. We do not know exactly which step it would call. There is

an emergent parallelism, but it's not in our control. It's in the model's control. And there's a bound determination. So, we need to impose some rules in order for this not to be a runaway train. One would be workflow rules to know what our intent is. And we need to we need to know what if we have a goal, what is it that we're trying to achieve. The

second thing would be chain of like structure. We use domain objects through all throughout the application. And we need to know that a particular a particular uh domain object matches the the output matches the input of another one. And there are also workflow filters. We want to know that no additional agents which somehow can creep into an agent catalog are used outside of what we are planning

to use run. What are the advantages here? Natural language control. We have dynamic parallelism. We have three uh three layer safety nets. Otherwise, it would be like the simple agents we built like a year ago. Last year, I had an agent, here's a prompt, a number of tools, and sometimes it would call it three times, five times, seven times. I've seen it even nine for the same

for the same goal. Now, the same agent code base throughout. But, there are trade-offs. Each step, it will call an LLM to make the next decision. It's non-deterministic. It might choose different execution passes in between runs. The same parsing fragility remains. We have to introduce a maximum iteration because we don't want this to run forever. And we will have keyboard-based workflows. We have to introduce some in

order to get the LLM to follow specific specific passes. let's run now the supervisor. Uh I am not Josh Long or Anton, so I don't I don't automate or type as fast. So, I'm just going to I'm just going to uh copy and paste some of our some of these prompts. So, if I'm going to say on whether Singer is the gold standard in Porsche restoration as

a company. I'm a big racing fan, so there are these companies, if a Porsche is not expensive enough, you can take one and there are different companies adding whatever crazy things to it. But, we want to know is this the best one if I ever decide to have one, then which company should I go and use for that? So, I'm expecting that it would not call the

entire it will not call um the entire um uh set of actions available to it. So, it did. It looks only at the agreement. I'm asking which one. Is this the best one? And the final synthesis says that yes, Singer Vehicle Design is the gold standard of of modern Porsche restoration. But, if I'm changing this on Singer, I'm inviting a discussion. So, deliberation, I'm inviting an agreement,

I'm inviting a disagreement in here. And I'm expecting to see that stages are used. Now, of course, we can have a much more complicated application, but this is a use case which I think we can all easily uh understand. So, we can see the individual responses for each one of them. And you can see all kind of different different opinions. It collects the rankings. Each one has

peer rankings. We run agreement analysis, disagreement. And in the end, what do we see? We see ranking consensus and disagreement. So, I added an additional consensus metric. Even these could be varied. And it depends what are we using. Ranking consensus and factual agreement, they both seem to be on the agreement side. They're actually orthogonal. So, we always need to think when we plan, not just pick the

right plan or planning algorithm, but also pick the right non-LLM operations because those could introduce variability for us for us as well. Now, moving moving on from uh supervisor, let's bring this back towards the the workflow part. So, hierarchical task network is this idea of a top-down decomposition with a greedy method selection. We take a task, we have a world state, we know the agents which are

available to us, and we take a method of let's call it a method, could be any kind of method or a function which will look at uh compound tasks with it will prioritize them and it will decompose them into subtasks, recurse on that. And in the end, it will end up with primitive tasks which have preconditions which get added to the plan and executed. For example, if

I start to the goal, I know that I'm have to start with the individual reviews. The output is an individual response object. That is an input for several for several others. And depends on what I'm going to achieve, I'm going to follow different passes. Now, the difference to the workflow is we know the agents, we know preconditions, postconditions, but we need an LLM call to help us

make a decision kind of like which path to a classification which path we're going to follow. The architecture is based on query classifier which will use six categories which we have which we have predefined. Uh they are like factual, opinions, or analytical, or creative, or that's a debate, or informational. This is something which we will apply as SMEs in our particular in our particular use case. And

then there's an execution of the plan which will dispatch every single task. And everything runs with back to the back to the stream. Now, what is the what are the pros and cons for this? We have adaptive pipeline, so we can add additional additional steps pure additional ex- extensibility. There's no architecture change. There's no change in in the debate model. We know up front by using sealed

interfaces, we know which which agents could be run. We have a graceful degradation. If we cannot the LLM cannot match e- either one of the categories, we assign uh a default and we assign analytical so that it runs the full exhaustive set. There's no orchestration of the of the LLM calls. Now, the trade-offs are that there's no backtracking. So, once the once the choice is made, that's

what we There's only a sequential execution only because we decompose a task and this is how the everything was added. There's a classification dependency. It very much depends on the classifier accuracy. It did the model pick the right classification or not? Of course, there's a fixed method set. And while each classifier returns a confidence score, well, methods don't use them because once we set this up, it

is as is, we cannot re- we cannot reuse that. let's see this in action now. Let's shut our system down. just alleviate the discussion on on startup speed, I started the application and it started in 400 milliseconds. Now, of course, each time you run on this machine runs a bunch of things. If you run into the cloud run environment, it might be 500 milliseconds, might be 400

milliseconds. I'm sure if I run it again, this could be maybe you see 167. So, rest assured, performance is not a problem we're in the Java world. We're not in the Python world. Or you might have some slight delays. So, let's let's run now a hierarchical DH 10 algorithm. I have built H 10 and utility as plain as plain purely based on the original app. Just so

that you can see that you don't always want to use an agentic framework. You might want to implement it or you might come up with your own and implement it on top of an existing application. Don't take this as best practices. Take this as a learning set of tools which we from which we draw inspiration in how we build how we build our our applications and what

patterns we apply. Now, if I'm looking at uh H 10, I'm going to say Tom Cruise movies. Like I don't know them. I'm not actually inviting a debate here. So, the execution plan it classified this as factual. What does it want? Individual It wants to see the ranking uh might be considered accurate or not. And in the end, a final synthesis. So, classified classified um the that

took that goal, classified, it picked one of the categories which is which is uh uh factual and it will build our our response and we're going to see if you're a fan of Tom Cruise movies, uh probably you would you would recognize them all recognize them all it could pick a different one. What if uh select what is the best programming language for gen AI applications. So,

we'll see. This is an opinion. So, because it's an opinion, it classifies it as an opinion, it picks the agents which are classified there and we're going to see what it does. let's see what we what we understand from that. So, the point I'm trying to to make is always look at what problem are you trying to resolve. That I hope comes pervasive through the different approaches.

Think what applies what applies to you. Sure. Take it on a larger project which you're working on, your pet project, something you work use professionally. And think what would apply here. So, if we see flashlight says, well, the Python says that undis puted leader is Python. Okay, I'm skipping flash from now on. Really? Okay. we're not asking this question again. So, it tries to explain to you

why this and that and here's performance and so on and so forth. But, I don't know. I'm um I don't even want to know the final the final response because I can see that probably the training data just has has seen too much Python. So, we have to skip Uh I'm sure it's going to be there, but I don't want to be depressed. let's look at utility

AI pattern. So, comes from this I'm not sure what the the pronunciation is this Ooda loop. So, you observe, you filter eligible actions from the world from the world states. For each eligible action, decide on selecting actions across select actions where the scoring is larger than a particular threshold and then execute the selection selected actions in parallel. So, we start from a world state, we filter the

eligible actions. Uh each one has a cost, some could be LLM agents, some could be just tools. We score and select, execute in parallel where we can and update the world state. Well, this is great for exploration. This is great for exploration because we might not have a precise goal. We might just say, find me the best MacBook for programming or find me the best laptop for

programming. I don't really know what this could be. Lightweight decisions because the scores are non LLM based, implicit dependencies, we have adaptive strategies and of course full Now, the tradeoffs are are how do we set up the right weights? And that is an exercise which we have to do via tuning and evaluation. The action set is fixed, it would explore only what it can see. There's a

threshold sensitivity and there's no backtracking. It is greedy and optimistic and greedy tries to find what is the next best thing to get to get the best results. It's like my children, we're always going for the largest piece of chocolate in the house. So, and then you can the state is is pretty is pretty binary. It's available or it is not available. Did it actually say of

course it's Python? Okay. So, let's from now on that's it. Please do not use this. This is not a correct answer. So, let's run utility. In utility, we want to see it make make decisions on um on the fly we're going to run. Come on. Utility. And now we can pick different scoring strategies. A balanced one, latency optimized, information gain. It we can assign different weights. So,

if we want I would assume probably information gain is what we want to see here and now we want to see how utility decides. So, here's the exploration. It says it's a factual verifiable list of filmography, does not involve subjective judgment or debate. So, it plans the item and we can see this. I don't know, I hope you can see. Uh this it runs initial query, it

gets the responses, it labels them, it anonymizes them. We can see what is complete and what is running. So, now it makes its decisions. Collect the responses, yes, we expected And in the end, it will decide to do an exploration synthesis. This is what the algorithm with this particular set uh worked. So, here I only do did we get our list? Yes, we did. But, how about

if we were to invite a debate. Let's try now a balanced Which company does the best Porsche restorations? I can't let go of these cars. So, by running it, it will tell us that it wants a folder liberation because best is subjective, depends on individual preferences for original versus modified specification. And with the confidence of 100%, it decides what it is going to run. Now, we can

see it running. We can see it running. What we also see is that utility will go step by step. By going step by step, it takes each iteration, it goes to the next one and we can see the decision happening. It did rankings, now aggregate rankings, now we expect synthesis can run, but we expect agreements and consensus to be executed here. [clears throat] One after the other,

they would they would run and we could see we could each of these behaviors. In the end, we can see the full we can see the full the full response and with a ranking consensus of of 100 and a serious set of disagreement. Disagreement also means how they disagree amongst each other, not necessarily if that is factual. So, if we have >> [snorts] >> utility, we went

on both sides from the stream, we narrow down to the hybrid. So, in goal oriented action planner, we start from a node current world state and we want to end up with a goal node where let us state which satisfies all the goal conditions. The edges are actions if of course the preconditions are met. And we always want to expand the cheapest first. So, we assign scores.

Let's say an LLM operation has a score of 10, a non-LLM maybe one. we accumulate the cost so far with what we estimate the next step would be. And in the end we get here the the guaranteed optimal plan. It uses an algorithm go-up came from the gaming world called A* which will find a cost optimal sequence from an initial continuously updating the world state as it

gets to the end. What is the important thing to know here is that after each step you can replan. It's not that backtracking is not available anymore. Here you can replan after step if conditions are met and these conditions are in your are in your control. Now, the advantage is is that there's zero LLM planning cost. This is fully deterministic for the same for the same plan

for the same It also discovers parallelism. It's a cost optimal path. Uh sure we have the flexibility. Now, of course trade-offs could be like the keyword based goal selection. There's a fixed set of actions, sure. There's a limited run time adaptation and that depends on how we configure the scoring metrics. And there's the a star search overhead. Well, it is but it very much depends on how

many you run. And it has no learning or or feedback So, let's let's run this. Let's run this. This has gives us what I am interested in and I'm hoping they do not models do not disappoint me. I'm hoping now to force an adaptive replanning. So, that we can see that this behavior could change. So, to take one where we can force replanning, let's take the goal

planner and say full deliberation on this question. For a startup with five developers building a new SaaS product from scratch, which technology stack should they use? Ruby on Rails, go with HTMX, Elixir with Phoenix. Uh and which single option is the best overall choice considering hiring time to market, scalability and long-term maintenance. So, the plan is set up. I like the responses, run ranking, disagreement and agreement

analysis tells me it they're running parallel. And then all the others agreeing aggregate rankings, agree disagreements and agree. the go-up planner will intervene here three times. So, the first time this is what I want to see. When it reaches the third step, it says I've reached a consensus of one and in my condition I've set a threshold of 0.7. So, because there is a high consensus a

consensus, it skips aggregate disagreements. So, it updates the plan and has only one single step remaining. This is important from us and if we scroll down, I'm expecting disagreement or we see aggregate analysis. Here is the aggregate agreement. We see the disagreement but we don't see we don't see the disagreement analysis. And we see now these ranking consensus and factual agreement. So, it is balanced and it

allows us to also replan dynamically. But we talked about all these we talked about all these different LLM actions. So, let's think of let's think about what consensus metrics we have. One would be for ranking consensus, do the council members agree on which response is best? This is called Kendall tau. It's a mathematical formula. I looked up what mathematical or statistical method we could use. So, what

does it catch? Subjective quality, convergence, split opinion. So, that's great. What it misses is factual correctness. So, it talks about agreements among themselves, not whether this is a fact It's factually correct or not. So, perfect agreement could go from one and then to a random contradictory to zero. We want to be for example, we hope to be in a strong or perfect agreement if possible. Now, there's

factual agreement. This is orthogonal to the previous one because we want to see do all the responses agree on the same factual content. So, this catches content level agreement. Do the models say the same things are true? But it misses the correctness of the agreed fact. So, again, there is a scale and each one these consensus metrics we pick, it depends on us. We and we have

to test, we have to evaluate. It's not there's no hard and fast rules. Always think that we have to find the best thing for our use case. So, disagreement severity is a set of weighted factors like stance diversity, the balance of positions and mention frequency in a particular discussion. Now, of course we use quite simple prompts and you see how much it generates. It's a lot more

interesting if we were to run this as a run a code review or over some larger code base but we don't have time for that in a session. So, this catches structure and intensity if there is a disagreement but it misses factual like factual errors or stylistic disagreements are treated in the same are treated in the same way. I want to leave you with this. Models will

keep changing. New patterns will keep emerging. The best thing we can do is stay architectural like point a new term and apply the right pattern for the problem in front of us, not the one that's trending. And remember that the best pattern might be the one you build next. Thank you. >> [applause]