About this talk
This talk discusses practical lessons learned from building domain-specific AI solutions, focusing on the processes used at EclipseSource. The speakers describe domain-specific AI not as generic models like ChatGPT, but as tailored solutions for specific industries, including AI interaction agents that integrate with systems like ERPs and CRMs. They emphasize the importance of designing a comprehensive AI vision and selecting use cases that leverage unique selling points of the tools being developed. The session addresses stages from design to deployment, highlighting strategies like creating dedicated APIs for LLMs, developing robust testing practices, and incorporating effective context management for agents. The speakers stress the critical need for proper evaluation and monitoring in production to ensure the reliability and effectiveness of AI applications.
Full transcript
[music] >> Excellent. Good afternoon, everyone. Last session for today, so we will do something light and easy, just a conversation about a few practical learnings that we had from building domain-specific AI solutions. Uh and to make it conversational and clever at the same time, I brought Stefan with me. He is lead software architect at EclipseSource. Uh and we will ping-pong a little bit about the lessons. Uh
So, to get this started, uh oh, sorry, I forgot. Uh my name is Philip. Uh just to get this started, um why don't you start explaining what we even mean with domain-specific AI solutions? Yeah, exactly. So, um everyone knows like Claude or ChatGPT, but this is not what we talk about today. We talk about specific things. So, we're talking about industry, companies who all want to integrate
AI into their tools. And yeah, you can start just with some uh AI model, put a system message on it, and call it domain-specific, but usually that's not sufficient. So, you will add some more integration, more tools, your own harness for your agents. You will go multi-agent, and maybe you use different kinds of models, host them to see yourself, or um go with direction of audio models.
And we will touch all of this today. Right. And just to give you some uh framing because we're talking about lessons learned, we can of course only talk about the lessons that we have learned. And uh of course this heavily depends on what kind of agents we are building. So, just to give you a little bit of framing. Uh so, EclipseSource is uh in general focused on
developing specialized IDEs, domain-specific um modeling environments, custom tools, and so on. Um and uh since oh, I put this about above my key. Uh and since about uh 2022 we did also something completely different, which is basically developing a platform for building uh AI interaction agents, as we call them. and you can think of them basically as, uh, agents that end customers would, for instance, call or
send messages to and so on. And, uh, because these agents are connected to some specific systems like ERPs, CRMs, uh, uh, medical practitioner scheduling systems and so on. Um, these agents can then, for instance, book appointments, uh, check the delivery status of things and so on and so forth. And that's what we're, uh, also building since 2022. Of course, uh, we didn't, uh, lose our core, uh,
business, which is essentially, uh, building these custom tools and so on. Uh, so since 2023 around, uh, we started to also build custom AI integrations in those domain-specific tools. Uh, so for instance, in modeling environments, uh, with domain-specific, uh, graphical modeling languages, uh, or in configuration tools and so on, where you can use the AI to interact with those kind of, uh, models. And, uh, of course,
to underpin that, uh, we, since 2024, started to also develop on open-source, uh, platforms on which you can build those, uh, AI integrations into custom tools, which essentially resulted into a Thea AI, uh, that you can use to build your custom agents for your own Thea-based Uh, so those are is basically the spectrum that we can report on. We can't report on anything else, but, uh, we
will refer to these, uh, um, different use cases when we talk about, uh, the And, uh, to make this a little bit structured, uh, we tried to, uh, structure the lessons that we, um, we discuss uh, in three phases, uh, starting with design, development, and then we end up with production. So, let's starting start with design. When I have to, uh, design a custom agent, the first
thing I do is I ask Stefan, "What is the first learning that you should, consider?" Yeah, and I will tell you that what you really need is some kind of AI vision. You need to think about how should it look like if people use the AI features in my tool. And, yeah, let's say you have a tool, and then you say okay, maybe I want just have
a classical chat interface for my tool. Maybe that's a little bit too open-ended for you. It's a little bit too risky. So, you'll go a different way and have a specialized integration, for example, like here with the terminal where the agent can't respond with free text but just with command and arguments. So, it's much more restricted. Or you go a completely different route and do things like
just AI-powered code completion, which is really subtle and doesn't maybe that people don't even know that you're using AI. But, there's also a different way. Maybe you're not sure yet and all what you want to do is just provide an API for your tool and then other AI tools can interact with your tool making your tool AI-compatible instead of fully integrating AI directly into it. Okay. Now,
you're sitting there and you ask yourself, "Okay, AI vision, but how should I start now? I I just don't know." And for this we would just say, first, just start with a chat interface." The good thing about a chat interface and why so many people are using it is because it's exploratory. You can really easily try things out. So, just do that. And the good thing is
that once you have that in place, you have all the infrastructure which you already needed then there if you want to go a different route. Okay. So, the AI vision is in place, but the use cases can you tell me something about them? How can I find the best use case for my tool? Yeah, so finding use cases actually uh quite critical and our experience, at least
our lessons learned it since this is like a huge I don't know space of options that you that you really start with the use cases that leverage your USB. So, really what you can add on top of the existing LLMs. And most typically in our opinion, this is one of the three things that are listed here. So, it's either data access that you can provide that others
can't or it's actions that you can provide within your tool, for instance, like do this or run a build or generate code or whatever. And it's integrations in the end. And integrations is essentially just integrating a flow in your tool that others couldn't do in your tool because that's basically what you control. Don't rely too much on what works with any LLM if you would also run
it inside of your of your tool, obviously. So, really start focusing on that. And the best way of finding good use cases that really leverage your USBs, in our opinion, are basically with an interactive use case identification, where you just start with a plain system message that is tailored to your specific needs, identify a couple of contexts and tools that you would need, and then describe what
hap- what should happen, which is typically the generic workflow and knowledge that you add into your system message, but then also, of course, the specific knowledge, the data, and the tools. And really, this part is what separates you from everyone else who also builds LLM applications. And this is obviously the one that you should focus on. And to make this a little bit more interactive, what we
like to use in customer meetings, basically, is that we, directly in the meeting with the customers, we try out different use cases just on the spot. That really works well in Aether AI because you can just create a custom agent. You open here the system message. Then, you can, of course, also already start chatting with this, but obviously that's a little bit boring if you don't have
tools. So, we have also this what we call tool sketchpad, where you can essentially mock tool calls. You basically specify which tool call is available, you give it a description, you define parameters with an ICU UI, and then, just to get things going, you enter a static return value, and with that you can really, on the spot, start exploring. So, for instance, here this is some sort
of robot fleet operations assistant. Who cares? But with that basically is all already enough to try out different use cases and questions that you could ask this agent. And now basically you can see this agent now already calls our mocked tool, gives the return value, and identify if that's what you want and if that's useful. >> [snorts] >> So, that's what we do in the first step.
once we have identified the use case, a big part of the use case is really to design the user involvement and workflow explicitly because this is really something that is that needs to be thought of upfront. Uh and one way to look at the user involvement and workflow that I I think at least helps quite a lot is by looking at what are the correctness criteria because
when the agent can produce a result, then you always have the question, is this result correct, right? And the easy way to distinguish between correctness is first of all objective correctness criteria. So, can failures be detected automatically across different users and situations? If yes, then you already know this is where you need to provide a tool that the LLM can use to self-correct itself because then this
is basically done. You can provide this as a USB that you can validate agent outputs. In most of the interesting use cases though, of course, there's always a part which has some sort of normative correctness criteria, which means there is not a right or wrong. There is always some sort of gray area and sometimes also a machine will never have the authority to say what's correct and
what's not correct. It's always the human in the end that needs to define that. And in either of those cases, depending on the cost of failure, the cost of rework, and the cost of review, you will then have to include the user at some point in the cheapest way it's just human approval, especially if the cost of review or cost of rework is very low, then just
approval is usually enough. But if there is like a high cost of review and high cost of rework, then you use you typically need some sort of iterative human feedback loop. And with increasing cost of those two factors, you need more complex mechanism for instance a plan artifact if the cost of rework is very high that the user can judge based on whether it this goes in
the right direction. And if the cost of review is very high, then you need a very rich human reject signal to avoid a re-review because you really want to enable the user to exactly say, "I want to change these parts." And just to give you one simple example, consider an agent where we receive an email from a customer and in the end we want to provide a
final quote for the products that the customer requested. Obviously, there is some objective correctness criteria like is the product even available in our product catalog? Is Is it Is it in stock? What is the price? So you don't worry about that. But there might be depending on the cost of failure, you may need human approval. If it's just hot potatoes for 2 euros, maybe you can skip
it because the cost of failure is so low, just send it to the customer. But if this is like a huge thing, then you need a human approval at least. If there is even more complex human judgment involved, then you want to make it very easy for the for the person involved here to say, "Okay, this is not what they meant. We would combine these two products
instead." So you provide a reviewing tool. And don't consider this human steering tooling and review a disadvantage of your use case, consider it actually an advantage because this again gives you a unique selling point in the end because with this tool you make the human efficient and amplify their intentions. And this is really what separates you from the rest. Right, so now we have designed the user
workflow and we have designed the use case. Obviously, we need some sort of way to talk to our application or software. What What can you recommend there? Yes. So, what you need is probably you need some kind of dedicated API for the LM to talk with your product. So, let's consider like a default case how it could happen. So, let's say you're you're developing some tool, you
have some forms, and they are filled, we have some schemas, some data, some validation, and you implemented your tool by just putting all your business logic in your UI and it worked until And now you are integrating AI and just following this principle like you already did before, okay, I just throw everything into the prompt, so to the LM it gets a schema, it gets the data,
the validation, then the user message which could be just some question or some ask to the LM, and then you give the LM the same API you worked against your own when you implemented the UI. And if you do that, it will very likely lead to substandard results. Because the issue what you're doing here is that you push a lot of logic requirements to the LM. The
LM now needs to analyze your JSON schema, come up with the paths to the properties, and find out okay, now is it need to set this property value, and it needs to know probably you put it into your system prompt, okay, always ask is this property actually visible or only then set it. And afterwards, please check validates correctly. So, you put all this logic which you previously
implemented in your UI to the LM. And with the now powerful models like Opus and so on, this will somewhat work and then maybe you get one impressive demo out of that, but you will really run into consistency issues. The LM will forget to call the is visible, it will call the wrong path because it just misunderstood your schema and so on. So, if you do it
like this, this is not good. So, this is what we meant with user dedicated API. And just one example how this could be solved is, okay, instead of giving um all of these things uh purely to the LM, you do it on a human level in the sense of how um you would do it in the best engineering practice anyway, with something that everyone can understand. So,
in this case, um you put all your artifacts and instead produce a presentation model, which is a common pattern which was used long before LLMs too. And I just don't want to go too much into the details here, but you can see it's easily understandable. This is one excerpt for one property, and you have everything in place. You know how the human label looks like for the
property. You know where it's visible, how where it's enabled, what the options are, which are valid. It's all in one place, and it's very easy to understand. And that's This is very easy to implement your AI against, which we did before, but it's also much easier to understand for the LLM. And you will get much better result out of that if you give this highly valuable processed
artifact to the LLM instead of just throwing all your stuff in there raw. Okay. So, some concrete examples. Um, you saw this API before. Wouldn't it be much better if you can for a specific use case uh use a typed API? So, if you give this to the LLM, it will call the methods much better than having this abstract method before. But, you must make sure that
you don't overload the context of the LLM, but it's a very valuable resource. So, usually if you have a dedicated API for the LLM to call, make sure that you give it some kind of way of filtering the results you get out. Cuz it's very easy to just uh accidentally flood the um context with just a too large tool response. And at the very end, when you're
designing the API, uh think about that the um LLM likely also should interact with your user. So, you can have a very nice user flows. So, for one example what we did is that for an graphical tool, the LM can highlight a path for the user. If it wants to explain something to the user, it can now highlight a path. And this is something which only you
can do in your tool to implement this kind of function for the LM to call. No one else can do that. So, that's your USP. Right. So, this is a developer conference, so enough about design. Let's continue with development. Uh So, Stefan, what can you talk about about starting into the development? >> Yes. So, when you start developing, you should really from the very beginning start evaluating.
And this is really very important. So, um the the core message of this lesson is really avoid doing changes in the dark. You know, with the LM with its black box behavior, people just change Sorry, I don't know this. They just change the prompt. Um, and they say, "Ah, it feels better for this use case it was better." So, they're just committing this. And then now on
next week, "Ah, another use case doesn't work as well before." They're changing the prompt back and so. It's very easy to just do blind changes there. And you should really avoid that. And there's multiple ways you can avoid this. And one of them is using deterministic oracles. Um, the idea is that okay, you have your flow, you have your agent in the middle, you gave it something,
and what did it produce? It produced some tool calls and um some text. And the tool calls is something which you usually can very easily write unit tests for because you expect for your use case a certain tool call must have been called. Otherwise, there something went wrong. So, let's say your unit test says create uh create a graph, and you have one tool call for that,
and it didn't call it, you know, it's just it didn't work. So, there's a very thing which you can implement from the very beginning. And yeah, just one screenshot of something which we are doing. So, we had the one use case where we had very many tool calls, and they need to be discovered by the LLM. So, what we did is we really collected usage information in
the test cases where we see, okay, this this tool call was called, this one LLM asked for a description of the tool call, how often they were executed, how often they succeeded, how often they failed. And using this data is then pretty straightforward to produce at least some test cases [clears throat] where you have a baseline of functionality which you can see that it still works. Okay,
but it's not always that easy, and sometimes you want to test different things. Sometimes you want to test the free form text which comes back, some explanation from the LLM for something which the user asks. And there you can't use these sort of testing very easily. You could try regex or something, but that's not that good. So, what's basically the standard now is using another LLM as
a judge of the text output by your original in production Yeah, so you put the you have your test case, you get the message to the LLM as a judge, and it then says you, okay, this was good or this was bad. And yeah, this is usually then outputted in things like clarity, hallucination indicators, and things like this. And you don't need to do this from scratch.
So, you don't need to implement this on your own. There are frameworks out there, for example, from tool. Yeah, they can do this for you. Yeah. the core message here is avoid the So, how can I go now further, Philip? Yeah, so one one less lesson that we also learned the hard way is that especially if you start from a fixed set of use cases where there
is kind of a fixed set of workflows workflow steps where uh there is a couple of steps where you need the LLM for. It's easiest to start with a code orchestrated flow of LLM calls. So, that's the most basic way to start. And this works really really well because you have a lot of control and you keep the sort of the non-deterministic parts very isolated. Um with
structured input uh structured output from LLMs, you can very easily also process the data and stuff like that. And uh this works great if you have especially restricted user interfaces or user input like a button or a wizard or things like that. So, that's really excellent. But with more and more use cases that you add, more and more complexity that you add, maybe you add then a
chat or other ways of open input and so on. Uh it is very easy to start growing the complexity of the holder of the entire orchestration flow that you build in your code. So, suddenly maybe you need some sort of classification LLM call in the beginning that decides it's use case A or use case B. So, maybe you need some routing there and so on and so
forth. So, it's easy to spend lots and lots of time suddenly on this code orchestration without actually improving your agents. Uh and we learned this the hard way. Sometimes it's better to sooner switch to an more agentic LLM based orchestration than doing this too late because you spend lots of money when you're eventually going there anyway, Um and basically that entirely switches the focus of your development
because you're not developing code anymore that does LLM calls, but you're now really working on equipping one LLM call with the tools with the means to basically control the workflow automatically and autonomously. really what I would suggest. Try it out earlier. If it already is kind of works, then stick with it and go this route rather than spending too much time on the orchestration. But, of course,
it comes at a price, and this price is not just non-determinism, it's also that suddenly uh context management becomes probably your biggest problem. Uh because before you had these isolated LLM calls where you basically controlled the entire context yourself, and it was somehow easy to manage. Uh now, if you just delegate more and more uh to the uh context over window overviews uh suddenly degrade its the
performance significantly, and this uh becomes a problem very, very fast, especially if you're adding more and more tools into your um uh agent. And uh obviously, uh every single tool call uh or tool definition that you add, whether it's currently needed or not, will influence the trajectory and the quality of your agent. And suddenly, it becomes harder to maintain because it's this bunch of uh or this
this this park of things uh where you uh change much more the same agent than before. Uh obviously, this is a challenge. Um so, you need to be aware or one lesson that you also that we learned is uh um you actively need to design the context management strategies now from the start, and uh really be prepared uh to redesign this over time over and over and
over again. Uh because with every new tool tool call, uh different uh capacity of context window usage will arrive, and you will need to rethink your overall context management strategy. And uh it is really a mix of different technologies that are different uh techniques that you can use. Uh rather whether it's dynamic system message rewriting when something happens, or adding or removing tools, whether it's uh using
shared files that you maybe uh carry from one conversation into the next one, or whether it's dynamic or deferred loading of tools, or send even sandbox programmatic uh programmatic uh tool coding where the LLM generates uh code, uh and then it is executed in a sandbox and so on. So, there is lots of different ways to deal with this issue and it's really changing every 3 months
what's the best practice right now. one of course one strategy is multiple agents like sub agents and that's a whole topic on its own, Stefan. Yeah. So, um, you heard of multi-agent systems. Maybe you saw a LinkedIn post and someone said, "I automated my whole team and it looks like this um, and I have no great results." Um, from our experience this is not the way it
works. splitting your agents into like traditional software roles or something has a high cost because you also need to orchestrate and you have a lot of repetition and um, it's hard to really cut the corners there. Um, so what you need to remember is why you're doing this. Why are you using multi-agent systems? In the end, um, using multiple agents is just a mechanism for managing your
context. So, it's all about the system messages, uh, system messages, tools, the Johannes and so on. This is really the important part. So, um, the question is if I don't um, I don't draw octant, how do I actually split my agents? And um, yeah. So, um, want to give you a way of deriving a good split and a good start is if you do it very sequentially
at first and you use some specific artifact um, for handover um, like files. This is a good way to go about it because files you can look at, you can edit them in your editor, you can persist them, you can um, collect them for later for inspection and they can easily be iterated on. And with all your context having in files, you can easily start new flows
just giving this one file where everything is in there as the input for the flow and then going forward. Um, yeah. Uh, additionally you can when you're building up uh, intersect yourself as the human in the loop, and once you are satisfied and you know, okay, this this works, you can then also start to automating it away if required. So, to give you a little bit more
concrete example, if you at the moment use the fair IDE, um we have some multi-agent um system in there. So, you can, for example, say to the software architect, "Okay, I want to um develop feature A." And the first thing which the software architect will do, it will invoke explore agents and looking through your um code base. For example, Cloud Code uh does the same. And why
is this done? Because they will look at many files, and many of these files will be irrelevant to the use case at hand. So, you're splitting away all this context um which they collect, and uh they give back only the really important parts for to the software architect, who can then produce your implementation plan. This gives you the chance to look at it. You can can change,
you can iterate a little bit uh with the software architect until you're satisfied, and then this is handed over to the coder. And the coder can then just follow the implementation plan and have the perfect context for its implementation steps. And if it's really large, there's a very natural split to go into uh just re-invoking coder multiple times for each implementation. Um you can just follow up
there's natural extensions to this having like the app testing or reviewing uh steps afterwards, but they all work the same, and they use the file then to iterate on, and uh you can um easily see what's going in into the Okay. Enough of design and development. Can we finally go into production, Philip? Yeah, we can, but I have bad news, unfortunately, because going into production is extremely
hard. Uh so, we've we've been there. Uh it's easy to build impressive demos uh in software engineering in general, but it's uh kind of multiplied with AI. Uh it There was this famous 80/20 rule that uh kind of uh 80% of the functionality requires 20% of the effort. With AI it feels like it's 96 4, but you cannot skip the four. That's the problem. So it takes
lots of experience, lots of work, and lots of lessons to cross this gap into production. Um because suddenly the shift or the the focus changes. Right? When you build use cases, you build demos, and so on, it's all about the shiny effects and kind of demonstrate the power of the LLM together with your data. And if you are in production, suddenly this immediately changes. Shiny effects don't
count anymore. What matters is precision and results. Because what we learned the hard way is when you have like five steps that call that involve LLMs in some way, and each of those steps have a 90% success rate, that sounds pretty good on the first glance, but if one result depends on the other, you have an overall success rate of suddenly only 60%. And that again means
in 40% of the cases you're causing more work than you save. And that's really not really leading to to happy users. Now, one example from the AI customer interaction agent that we that fits well in this lesson learned is you may know there is when you have voice agents, there is from OpenAI the real-time API that doesn't go through transcription, but really directly processes your audio input.
And that's excellent because it has low latency, and it's even aware of your emotions. So when you scream at the agent, the agent will realize that in some way. When you transcribe, obviously that's lost. And that's excellent for demos. Everyone would buy this product. But when you put it in production, you realize that so much context is wasted on this additional information that you lose a lot
of precision. And in the end, when you put this in production and to deal to deal with calls, all that you want is cases being handled and not detect whether one customer screamed at the agent. Uh so that's really something um you cannot overstate that. Right. And when you are then finally in production and you think, "Okay, we developed with a provider A, but now provider B
is more cheap. Let's just switch over." Then you run into this problem. Exactly. Providers are not equal. So, you're thinking, "Okay, there's another offering. This is the same model, so I can just replace it, right?" Um Yes and no. Uh it's more work than you might think. And you can just look at the AI code base and look how many workarounds are there for supposedly AI compatible
LLMs. We had to add additional code just to handle Gemini uh Kimi or whatever. So, um there is some work involved and you will see that um uh things fail. But even worse than failing is if they just behave a little bit differently and you might not even notice it directly. So, just some data. So, it can happen to you that you switch everything works, but your
new agent doesn't hit the cache anymore. Just because they handle caching a little bit differently, you don't notice it and some suddenly your prices go up. So, um you really need to check what you're doing and you need to when you switch, you need to show see what your costs are. Um Then there's other use cases. Maybe you don't want to use a provider. Maybe you want
to in some way self-host, whatever this means for you. Could be that you wanted to go to Azure and have your own model there in in some region. Um maybe you even go to some graphic cards provider and then host your model there. Um or go to own infrastructure. And this is very, very difficult. Um not only does your metrics change, no longer you are paying for
tokens, you're paying for minutes uh where your model is running. But also getting this run and being stable and being reachable, you get into a whole uh different kinds uh thing where you probably need uh a whole team managing that. So, just be aware that this is not that easy. How can we go forward then? >> Yeah, uh and the last uh aspect that we learned uh
the hard way from um production is uh that we are used to building tools and IDEs, right? So, basically, you ship the product, the customer installs it, and then you don't worry about it anymore until the first bug report comes in. AI solutions work entirely different. Maybe in other uh software areas, it's uh already the same, but for us, this was really a new learning because uh
suddenly, uh you need things like instrumentation and monitoring because without it, you are basically absolutely useless. Uh you just get angry reports, and you cannot act about them at all. Uh so, what we for instance had to build uh to uh deal with that is uh Oh, sorry, wrong one. This one. Is monitoring tools. So, for Sorry about that. This is German, but it's basically the monitoring
tool that we have for the AI voice uh interaction agent. So, it basically gives you all the transcripts of all the past calls, and you can search for them. Uh they are auto-classified in terms of what was the rating when you just look at the conversation, and they are classified what topic they had, and so on and so forth. And when you get a report uh or
just by exploring that, uh you can basically look into it and say for instance, "Okay, in this call, uh sorry, uh in this call, So, in this call, uh this was uh what happened. And here, I heard that the uh this order number was never really closed. So, you can just click here, and then immediately get into the actual context that has been sent that produced this
tool call, and so on. And you can then click on one button in order to extract uh um uh uh scenario that you can use in your eval set from now on to further improve. And this is so crucial because in the end, you as a software developer, you don't want to do that. You need the domain experts or the product owners or product people to do
that and they need good ways to transfer this information to the development team in order to be operative in any way. Um also, of course, I don't want to go too much into detail, but obviously security and infrastructure is something is non-trivial at all. Especially, I just want to highlight two things because it's interesting and maybe I specific. Uh if you are using a third-party API provider
LLM provider and you are in production, API keys is something extremely valuable, right? So, you cannot put like a limit, okay, don't charge me more than 10,000 euros a month. You basically have to have them unbounded more or less. And if that gets into wrong hands, you're easily lose a lot of money. Uh the same goes with a misuse. So, there is extremely clever people who try
to misuse your tool in order to do to use your LLM for for their purposes without just using your tool as they supposed to and so on and so forth. Um so, our lesson is really that this is a lot of work and established platforms actually help a lot there because they typically come with this infrastructure like monitoring, instrumentation, and so on. and this is also why
we extracted a good part of our voice interaction agent into a reusable platform to make reuse of that. And finally, if you're building a tool or an IDE, one easy way out of that, maybe it's applicable to you as well, is that you can piggyback on existing service providers like Copilot or ChatGPT. If your users already by default typically have subscriptions there, there is a way, for
instance, if you have your theory-based application, that your users can log in there and you use via their subscription the LLM from there. And that basically frees you up from dealing with subscriptions, rate limiting, and all of that stuff. And that's really nice. Um yeah. With that, I hope that you'll get something useful out of it. Uh, Stefan, anything you want to conclude? Yeah, so if you
take one thing away from this presentation, then it's that it's very easy to create an impressive demo, but you will get into a lot of problems if you just promise to deliver that like this. This is not how how it works. So, build up your experience, build up your platforms, your evaluations and techniques, and make sure that what you're doing is of real-world value. Um, because there
are many choices, there are not there's not a single best approach like you just saw our recommendations, they heavily depend on the on the field. So, um, get to know what works best for you and wish you good luck there. >> [laughter] >> Good luck. No, it's doable of course, but it takes a lot of time and effort and also experience. And yeah, I hope you found
something some useful information out of that. If you want to discuss your use cases, feel free to drop by our booth, we can exchange war stories there. And maybe there is some way we can help you with that. So, looking forward to your questions. Thanks a lot for joining. >> [applause] >> Thank you. Are there questions? Yes, please. Good. First of all, thanks a lot for the
presentation and sharing your valuable feedbacks and learnings as well. I'm actually wondering if you also follow any strategy in selecting befitting models for a specific use case, for example, frontier models or any kind of open bit models or even small language models. In in today's world, the sustainability plays a crucial role. So, in that case, you know, I would believe that small language models would also be
a perfect fit for a specific dedicated use case. If you can share something on it. Very good question and obviously there is no right answer. It depends a lot on the uh respective scenarios. So, for instance, if you are uh building uh AI agents in engineering tools, then uh by default, you kind of need some really capable models, which basically ends uh brings you already to the
more or less frontier models, uh which you cannot run on your machine, for instance. Uh so, what we typically our main strategy is that we typically during the use case uh selection, we start with the really strongest models, because that kind of gives you a feeling of what in general is possible. Uh everything else is kind of an optimization afterwards. Um because also also the and that's
also maybe a lesson learned, we didn't put it into the slides. Uh when back like 2 or 3 years ago, uh we uh implemented an agent. Uh I think it was uh based on ChatGPT 4, no, 3.5, uh something like that. And it was so slow uh that we spent like, I don't know how many man-days or person-days into optimizing that. And then ChatGPT 4.0 came out,
and basically everything was useless, because it by default is now just like 10 times uh faster and uh cheaper. Uh so, all of that was lost, essentially. And that's why we are a little bit cautious and basically try to build for the models in 6 months, because that's the time you need until you get into production. Uh but that being said, you're absolutely right. It is a
crucial question, especially for data privacy uh things. Uh so, there we often then switch to uh like Kimi 2.5 or so, which you can theoretically uh run in an isolated way uh without having a third party involved and so on. Uh that's where also our experience from switching providers is not that easy, but it's doable. Um But then again, on on the same time, uh when we
look at the voice AI agents, here basically there is completely different requirements. Uh so, for instance, latency is different, uh or privacy concerns are way different if you have medical practitioners being called and stuff like that. Uh so, there there is different uh requirements. Sure. Yeah. But once more, at random, so if you have your evaluation practice in order, then you can easily try a different models.
This is a great thing if you have all these test cases set up because there okay, just run it with a different model and see whether you hit the same check marks. It's a plug and play then. Yeah. Yeah, exactly. >> Yeah, sure. Thanks. Thanks a lot. Thank you. Any other question or do we have time for another question? Yeah. Thomas. I'm just worried a bit about
the business model because now there's a cost to running the software, right? So before it was running the software is free. But now there's a cost. So there's a non-determinism in the business model. How do customers handle that? Good question. Should I answer or Yeah. Yeah. As a we have one customer was for example evaluating this and for them cost is a really huge factor. So what
they're looking at is especially these open source models. For example, especially Deep Seek, it's much cheaper than other things. And this factor which they then plan and so they their expected usage because they want to offer this for free for in the tool. So they're thinking about okay, how many people will try to use this and then is there something which we want to provide them or
not? And if not, then you need to think of something else. So clients for example are thinking about integrating with existing offerings so that people who are already have a GitHub Copilot subscription which a lot of people in business have, right? Through their company anyway. They can connect this to their LLM offering of the tool and thereby there's no cost directly for the company. And for for
all of the other cases that are not so tool related, essentially it's a uh it's a calculation of how much time do you save and then you typically see what what money does this roughly relate to and then you basically try to at least have not more than half of it as an as variable LLM cost because then it wouldn't pay off because there is so much
as we said going around this whole operations that you need to have a lot of marge uh to to cover for that as well. Um so I'm not a business case expert, but that's at least my understanding of how these guys are calculating that. There's one question, yeah? You uh mentioned that managing the context window is one of these important things that you sooner or later will
just it will be like a lot of the work. um I thought in combination with the multi-agent thing that you mentioned um you showed the slide where you have these different roles that I'd say usually people have and um then you just put in agents, but I think that the multi-agent thing is would be most useful to sort of try to split a big problem into pieces
and then delegate subtasks and then those agents decide by themselves if they delegate further to really bring the complexity down. we've had another talk here today that also sort of mentioned that these models are very very good at um solving these these very simple problems is this something that is done in these um Yeah. So um I'm not sure because it was got really silent in the
end whether I completely understood your question. So I will try to answer it and you tell me whether you're satisfied. Um so uh the um Yes, the as the uh the multi-agent approach is for context management and yes, you can let the agents decide themselves um for some use cases to go this way or not. So for example, in the fair IDE, we have a specific tool
delegate your agents and um we then tell the agent, "Okay, for more complex cases, for example, then invoke this explorer agent, or too much for you to handle, then delegate the implementation steps to the coder." So, you can give the you can give this decision to the agent if you want to. And then in the end, it's just testing. So, is this sufficient? Is it not? Do
you need to force it? Does this make sense? I think maybe just to add to that. So, I I think that the challenge with multi-agent or sub-agents in general is always the handoff, of course, right? So, basically, you lose the entire context, and this is the critical thing, and that's also why Stefan highlighted having file-based handoffs, especially in the beginning, is something extremely helpful because you can
debug basically one agent call in isolation. And this is also what I think is the most challenging part is prompting the calling agent in a way that you make sure that it knows which context it has to pass on. That's the challenging part. I mean Yeah, I I I think you'd have to sort of have a scope of context that you pass down, and then you split
the task into five, and Yeah. One agent does a subtask. But so, that's not really that much in use, or it's Can you speak a little bit up, please? Um Yeah, that's better. >> not that successful, or is there just too much complexity? No, no, it's >> it that's fine. Yeah. Yeah, it works. So, we use it in the fair IDE. It's just what what what you
wanted to transport is don't just do it and then split it evenly, like you saw some software roles in your company and just split it like this, and then maybe this works, but you really go iteratively. When you notice your agent doesn't behave that well anymore, and you think, "Okay, it's likely it has just too much context," then try this approach, and start small and just add
one sub agent and see if it works better and then iterate on this until you get to a um set up which works good for you. And also I think one of the key lessons there is that we as humans we try to replicate the organization that we see in that works well in teams which is not translatable to agents because what drives the human organization is
authority and responsibility which is completely irrelevant for the agent because not one agent is more authoritative than the other. Uh but uh also it's a cost a factor in terms of not money but in terms of context because if you like over split and over separate tasks you end up having a lot of transfer and when one agent transfer to the other that means you have to
wait until it re-outputs all of the context to send to the other agent which then re-reads it and so there's a lot of overhead going on. Like with humans if humans work together there's lots of overhead going on, right? >> Uh great. Any last question maybe? Then there is I think uh some show going on and drinks I guess. Okay, this was not very motivating to ask
another question. Sorry about that. >> So then thank you very much. Find us at the booth and looking forward to further discussions. >> [music]
