Building AI assistants for DSLs experiences and findings from Langium AI

This talk focuses on building AI assistants for Domain-Specific Languages (DSLs) using Langium AI. Benjamin Friedman Wilson, a software engineer at TypeFox, shares insights from his experience in developing Langium, a TypeScript-based language engineering workbench designed for creating DSLs. He discusses the importance of establishing a practical workflow for creating DSL agents that assist domain experts by raising abstraction levels to improve usability. Wilson emphasizes the necessity of iterative evaluations to refine and improve both the DSL and the AI agent, ensuring they adapt to new information and user needs. He also highlights how Langium AI helps bridge the gap between AI and DSLs by providing evaluation, data processing, and grammar components tailored for effective integration and usage.

[music] >> All right. Well, thank you all for coming. Um I'm here to talk about building AI assistants for DSLs. And uh don't worry, it's all right. And specifically looking at experiences and findings from Langium AI. And just a heads-up, too, if you're not so familiar with what Langium is, that's okay. I'm going to try and keep it a high-level talk, so you should still be able

to get some useful feedback and learnings. So, I'm going to kind of give you like a quick introduction, some background, a little overview for Langium AI itself, um kind of like a little story of Langium AI in practice, um some findings that we found that are particularly valuable, and some closing remarks. So, a quick intro here. Um first of all, a little bit about me. Um so,

uh my name is Benjamin Friedman Wilson. Um I'm a software engineer at TypeFox. Been there since early 2022, working predominantly on Langium DSLs. Um I'm also with DSL agents, so typically AI integrations around Langium, and a lot of verification and generators. So, like type checkers and such and transpilers. Um and I've been working on Langium AI since 2024. So, just to kind of give you all a

quick overview, I think it's kind of nice to kind of give you expectations. Um the key thing is I want to give you guys like some know-how for how to build DSL agents in a very broad sense. Um so, you're going to learn what Langium AI is, you're going to learn what it's about and how it can help, but specifically that's going to come through me, we

sharing kind of practical experience and observations. Um kind of a workflow we've already established for building DSL agents that is adaptable and you can kind of use it as you need, and just some general observations and findings. Um the goal is again that you can all kind of take away this knowledge about Langium AI, and also get a more general knowledge of how to build DSL agents

kind of at large. And then again, hopefully some good ideas about how you can all practically apply this once you're done. Okay. So, a little background. Um this is a hot take, by the way. Um any program is only as good as it is useful. Um bonus points if you can tell me who actually said this. And really, I want to kind of set this up to

kind of frame the thing here, because what it means to be a useful program is changing with AI. Um and this has impacted us all, especially in the DSL community. And so that's just kind of just set it. But then quickly, I know everyone here is not necessarily well versed in Langium. So, I want to make sure I just do a quick slide to kind of go

over what that is. If you're not familiar with Langium itself, it's a language engineering workbench. It has a TypeScript basis. Um it effectively refers to both the grammar language itself and the framework that is used to produce Langium-based DSLs. stuck. Let's see. Oh, but it's blank. I broke my cardinal rule. I changed the slides last night. Don't do that. It's not a good idea. Yeah, definitely don't

do that. That's not a good idea. I'm going to go to the last one. That's the one that works. Let's see. Is everything here? Wonderful. Up. Animations. again, um kind of that second group of bullet points there. Um typically is commonly used for producing extensions, such as in the case of VS Code or with Theia. Um a lot of times with CLI, so often times with generators,

transpilers, compilers. Um and web-based deployments, because since it's TypeScript-based, we can set it up wherever we need to, independent of a server. And then uh kind of just case in point, a lot of this is for LSP-equipped languages. And the more or less endpoint is that it feels like it just works. When you produce a Langium-based DSL, it feels just like you would install some other language

through a VS Code extension. You get full functionality. That's the whole point of it. the Langium AI. Um, effectively, the whole thing that we do with DSLs is we try to help experts solve problems. We try to raise the abstraction layers so that they can operate at their level of expertise. But experts need time to learn this abstraction layer, to learn DSLs. And so, agents can really

help with this process fantastically so. Um, and users actually come to expect this. But this only works if they, and I mean the agents, can understand the DSL. it's not magic. LLMs need data, right? They need data to understand DSLs. And so, Langium AI really tries to form this bridge, tries to cover the gap between the two. And then therefore making it easier to build DSL-capable agents.

Okay, so a little overview on Langium AI itself. So, I'm going to talk about what it is, when to use it, where it helps, and why you might use it. Okay, so kind of just like three bullet points here that kind of summarize the key aspects of Langium AI. Um, it has an evaluation component, data processing component, and a grammar component. So, the evaluation part is mostly

composed of qualitative and quantitative measures. Um, effectively ways to like grade the output from your LLMs particularly regards to the DSL programs themselves. And so, this is used to typically check the model and program quality. There's also data processing. So, a lot of this is related to syntactic splitting, processing DSL program text so that you actually can preserve things like say functions and and associated function headers

if you have this in your DSL. Um and this comes up in like the case of like if you have rag or other data synthesis cases. Um the grammar case is really interesting. Um this is a little more niche, but it's still worth noting cuz it comes up where we have things where we synthesize an EBNF grammar spec from your Langium spec. And this is really nice

if you want to do things like constraint-based decoding if that comes up. It's rare, but it can be helpful. Um and it also allows us to do other things in a separate topic with syntactic coverage checks. So, if you produce a program or a set of programs, you can actually see the extent of your programs with regards to what is capable being expressed through your grammar And

typically this is also used again for modifying the LM's behavior or for doing analysis on your data sets. when would you use this? Um basically use it last after you've already tried out all your other options. And this is really important. Um you've already changed, you know, models, you've tried different prompt engineering, context engineering tactics, rag, um tool calling. MCP is different from tool calling, by the

way. Um that's just there just saying you've tried additional supporting techniques to actually work and improve your model's output and behavior. And so, it's really important you do this first because the best of these approaches is actually going to be the basis for working forward in Langium AI. So, effectively kind of like the sentence that I want to impart here is that if you're working with Langium

and you're working with LLMs, then Langium AI can help. Cool. All right. So, where can it help? Um effectively it's step-by-step. We try to be non-intrusive in this regard. It kind of picks up where you need it. So, effectively all kind of there's four of them here. I'll just go through these points. It's again kind of a reiteration. The data processing phase effectively if you're early on

and you're still collecting information or you have aggregate documentation with embedded DSL text, you can process that with the splitter. Um if you're doing prompt generation, so basically by hand potentially or with an LLM or both, doesn't matter. You want to be able to evaluate those those prompts, refine, and then repeat this process. Um, and then if you're doing things like synthetic data generation, for example, if

you're looking into fine-tuning for specialized cases, you need to be able to probably use the best prompts and data that you already have before to kickstart that process. Um, and then I'm kind of borrowing the term backpropagate here, but I'm using it in a different context where I'm basically saying the findings that you actually make through this process, they actually will improve your DSL implementation itself, especially

if it's ongoing. Um, it actually kind of kicks in as a grammar fuzzer. It's actually really really cool. Okay. So, why would you use this? um, there are a few points here. Um, the dog is because I wanted to write dogfooding before, but it sounds better to just say we build on our own foundation, but hopefully get the point. Um, we're using it, customers are using it,

too. Um, and the workflows are really built off of practical observation. So, we really try to keep what's helpful and improve upon that. And um, we just try to focus on really solving this bridging issue, not trying to overcomplicate things. And it's minimal, so small dog. Um, you can basically bring your own model, framework, tooling, everything else that you already use, and we again try to stay

out of the way. You use what you know and what you like, and then we try to support that. And we try to make a buff dog. Too many dog things. I don't even have a dog, I have a cat. Um, it hardens your language. So, the agents will test your DSL implementation, and again, it's effectively akin to a grammar fuzzer. This is actually really kind of

like a cool unintentional benefit. Um, and again, the corrections can typically result in a better DSL and a better agent overall. It's very much an iterative process. Okay. So, the kind of encapsulation there is that Langium AI seeks to to bridge this gap, right? To fill the gap between AI and Langium DSLs. I almost deleted the story time bit I thought it was maybe a little too

funny, but it is basically a story. I'm going to be talking about an experience I've had with Langium AI. Um I have to be a little bit general. I can't name the um the client in particular, but I can talk about what we were doing. And I think this is really nice because it really conveys what we have learned and why we found that helpful. let's say

we have company XYZ. Effectively, we're building an agent for a financial DSL. Um and the goal is to generate and explain DSL programs to domain experts. Seems pretty straightforward. Um the DSL, just for just some basic context, it's Langium base L3. Um a lot of customized services, like customized linking, scoping. Um I think token builders also customize in this, too. About like 700 lines of code in

the grammar. So, without trying to wrap all that up, it's basically like modern complexity for a DSL. It's not the most complex thing we've seen, it's not the least, but it requires some thought to make sure that it's going to work carefully. this is kind of in hindsight that I wrote these five things down, but I think they really kind of sum things up nicely for me.

Um we basically found ourselves starting with clarifying our goals, picking the AI stack. Um I'll explain what I mean when I say stack. Um setting up the actual evaluations, running those evaluations, and then actually refining, which is effectively a loop. So, clarifying your goals is fairly straightforward, right? You know, we already wrote this before, right? Generate and explain DSL programs to domain experts, but it it's more

than that, right? We have to be more specific. So, this is where you sit down and you talk, right? This is going to be a chat application. In particular, this company actually wanted to do a self-hosted model. It's an interesting choice. Um and they wanted to have a single model, not a family of models. They didn't want to have models handling different different tasks. Um and they

wanted it to be accurate and fast. I guess everyone can say that, but um in this particular case, they needed the model to to be able to respond quickly to incoming data, and they needed it to be really really trustworthy. So, they very much wanted it to be accurate and And yeah, again, this is kind of just a side note, but like your use case totally changes

this, right? Your requirements are your own. I can't tell you what those are. You know them yourself. When we talk with clients, like this is how we suss this stuff out. So, then you pick your stack. So, hopefully this illustration kind of sums it up nicely. It's it's everything in addition to your model plus the model. So, the technology changes very very quickly in this area, and

so even what we're doing a year ago is not what we're doing now. And in a year from now, it's probably going to be different again. Um at least that's what I would suspect. again, we try to stay independent of this. We try to make it so that you can choose whatever you need. And so, in you still are going to want to at least pick one

or more providers or again, a local one. Um pick some models. They were evaluating um these three initially. Um just to give something more specific between Quinn, Llama, and Gemma. Um that's not what they ended up with, but that's what they started with. And then maybe some prompt techniques. Um again, this was this project to give some context with also like late 2023 going through 2024 into

2025 and then 26. So. The things like COT, one-shot, few-shot, you name it. Etc. And most importantly, starting simply. Um you don't need to overcomplicate what it is that you're trying to solve. Um and uh they didn't want to do rag. At least we didn't start with that because this fell out of the scope of keeping things simple. We wanted to get a baseline. And no tools.

At least not yet. Right. So, the next step is actually setting up the evaluations themselves. So, the code that I have here is kind of some pseudo semi code here. Um but it's more or less showing how we would maybe take the DSL services, instantiate an evaluator for our target DSL, and then actually invoke that evaluator. What it's doing specifically is dependent on what it is you

want to check. And so, in this case again, we set up these evals using LangChain AI. And the clients actually put together their eval cases. So, they're the domain experts, right? They tell us what they need to check. And so, we can respect that. And so, they give us these input output pairs in particular for what they expect given a certain amount of information, what is actually

going to come back. And then we can run evals over those generated AI outputs. And then report back the diagnostics and additional metrics that are associated with the responses. Which then are generated from the And we tried to keep this flexible for the client's team. So, we expose this particular aspect as just an API endpoint. Um later on we also did MCP and we had a CLI,

but again, we tried to really not getting in the way. We tried to very much support what they already had in mind. Cuz yeah, they have their own workflows. We don't need to tell them how to do their job. They already know how to do it. They're experts in AI engineering. We're experts in DSLs, and so we work together in this. Yeah. And so, we got our

evals incorporated into their system. So, after like a first round of evaluation, we kind of got like a a star. This was very early where we got Quinn. And it wasn't just Quinn, right? It's the stack that we had at that point. But this is really nice to have because we can basically assess the models, the prompts, and the techniques, and then get a good sense for

what we think is working given what we already picked. So, this is our baseline. And it gets recorded as an evaluation matrix, as you can imagine. Looking at all the prompts given all the models, given all the additional tooling that you're using with these things, and then determining kind of a sweet spot, right? Thinking of, you know, what pairings, in this case where it was simplified with

models and prompts that work best. Um and then just look around for all-around performers. And that best combination is the new it seems fairly straightforward, honestly. It's fairly simple. And but it's very important to like not forget this. Like you don't have to overcomplicate this process. Um what's important is that you do evaluate. Um And so, and I really can't stress that enough, is to stop and

reflect. You could already be done at this point, actually. If your application your DSL is not very sophisticated, and it may not be the first time that you're at this point. You may already be on your fourth, fifth, sixth, seventh, 100th evaluation. You may be done. So, it's really important to stop and think what can we do more? What can we improve upon? Do we need to

improve anything? Is this already satisfactory? So, just a like a take a moment. Um now it's a dark again. So, once you've gotten to this point, we loop around, right? So, we basically go back into refining again. And so, we basically take our best combinations, we make changes, we repeat, we update the prompt, try more models, then we expand the evaluation suite, and then we just keep

going. Now, so the propagations are also or not propagations, sorry. The corrections that we find along this way, it's really important, actually. This process doesn't just focus on improving the agent, it also improves your DSL. Um again, as you're are getting a an agent really equipped to be able to generate valid DSL code, it's going to also generate invalid DSL code. And this is really cool, because

initially, you don't want this, right? This is undesirable. But, in some cases, things come out that are syntactically valid that pass, but they're not semantically correct, and you start to find holes in your implementation. And so, it actually fuzzes it for you, for free. So, it's great. And then, yeah. And we settle on a new best model prompt, etc. So, there's this side effect of improving the

DSL, but we're also improving the agent at the same time. So, they're very much working together, actually. >> [snorts] >> Okay. So, kind of coming back. So, we've done this this step one through five. And now, for this particular company at this stage, they wanted to explore fine-tuning. Um full disclosure, they did not end with fine-tuning. Um I would not recommend fine-tuning. If you know you need

it, you know you need it. Maybe an embedded system or a small model. Um but nevertheless, they really wanted to explore this early on. And so, that adds two additional steps for data synthesis, and also then the actual fine-tuning Simplified, of course. And again, this is optional. Okay. So, for data synthesis, um you need to um consider that you have some good core data set that you

can actually generate from. If you have nothing, you can't build anything. And so, when we knew we wanted to explore fine-tuning, um we had two problems. We didn't have enough data, and the language was still being developed. So, it's like not enough information, and it's a moving target, which is doubly awful. Um it makes it really hard, cuz anything that we do produce for data is going

to be potentially invalidated in six months. again, if we have done it correctly at LangAI, we can at least assess the process in a controlled fashion as we proceed. So, that when the language in the DSL update, we can track with it. And then, so, how we do this, actually, when we actually we're doing the synthetic data generation, is really interesting. Um it was done in two

phases, actually. So, there was a stable and there's an unstable generation phase. And more or less, um the stable phase is characterized by like kind of low temperature, low quantity data magnification, and the unstable phase is mostly high temperature, high quantity data magnification. yeah, at first is a small number of limited examples followed by a large number of broad examples. This is what I just mentioned. Right.

And we kind of call these the cold and hot phase respectively. the cold phase. One second. Okay. So, it's a small quantity of examples that we're generating during the cold phase. We just want to produce some stable documents. Um so, it could be 10, it could be 100, could be more, but you're not focusing on massive quantity. You just want some slight magnification um with typically like

a low temperature. Could be 0.01. the goal is very high adherence to your specification. You don't want deviation from your examples that you already have. You don't want deviation from your known grammar too much. You really want to just slowly kind of push things around. We don't really want like highly creative outputs. It's not real creativity that we're getting. We just want a low distribution from the

training data that's being sampled in But we do want that slight magnification of data. And um the evals are going to more or less do what they do best, right? They're going to help us decide what's actually retained during this data synthesis process. So, the evals that we actually use to check the model quality are actually used to also help this, too. And then again, even at

this process, we're still making corrections to the DSL and the prompt, of course. Um so, the whole time we're still improving the Now, the hot phase. Lots of documents. Lots of data. Um this is the complete opposite, right? This is where we get to have some fun. So, we really want this large quantity of you know, it could be 1,000, 10,000, 100,000, could be a million, but

you need a lot of data um because you're going to be filtering it. A lot of it's not going to be any good. And you want a really high temperature. Um so you could do like 0.7 what else somewhat arbitrary can be do two if you want to. It needs to be volatile. Um cuz you really want these highly variable outputs. And again, this is um what

I mentioned before like we don't want creative outputs. It's not really creativity. This time we want creative outputs, which is correlated to we want a very high distribution of sampling within the training data that the model was built with. And we really we also don't want repeats of existing data. It's kind of tricky because with this high temperature going to get all kinds of stuff. And you

will get duplicates, but you'll have to filter those out. But we do want this massive And then the valves will again decide what is retained. And we're going to make numerous corrections to the DSL on the prompt at If I recall correctly, this I think was the most volatile phase for the DSL in terms of how changes are being back propagated um because the number of things

that were found that were just this weirdest patterns coming out that were syntactically valid, but then semantically unchecked um came out predominantly through this hot phase. So this was actually the most helpful for like really tightening the language down and improving it. we fine-tune. you do some post and synthesis cleanup. You combine your cold and hot data in this case deduplicate and you process it into corresponding

input and output pairs. Again, this is not a talk on fine-tuning, but just to kind of go over And then you break it up training, validation, test and then you fine-tune to target model. So again, not guidance on fine-tuning, but just kind of going over how we applied this here. And then in this particular case they're using Laura if you're interested. Um you could also use full

fine-tuning, but they're rather purchase too. Right. And then evaluate. coming back to this. Um I really I don't think I can come back to this enough because it even without steps six and seven having one through five is like huge because once we got through one, two, and three and we're at four and five, we're kind of just alternating. We're evaluating, we're refining, we're evaluating, we're refining.

This is just like tremendously simple, very helpful, and easy to communicate across cross-disciplinary teams as well. Now, you just rinse and repeat. Um and then again, you could adapt this to your own workflow as well. Everyone has their own mentality and their own mindset for how they do these things based on their experience, but once you get into a nice iteration loop on improving these kinds of

things, you're in a very good spot. And then again, the kind of question that I mentioned before about the DSL being a moving target, what if it changes, right? Well, if it does, the agent is already fairly capable at this point. But, you know, we get out of sync and so what do we do next, right? we just iterate some more. Um this is just a quick

clip of a graphic that we had at our booth before. Um but yeah, we just repeat the process, right? If the DSL has changed, we go back, we re-evaluate. And we basically check to see what is missing, right? We make changes. We adopt We adopt prompts that are outdated. We adapt data that's outdated. If models look like they're not performing as well as we expected them to

or newer models have come out, switch to them. Yeah, let the evaluations guide you in the process. Don't have to get complicated with it. And again, the whole time you're doing this, you're not just improving your agent, you're improving the DSL, too. And then you repeat. And then again, once you get to kind of four, five, where you're evaluating and you're more or less just staying in

sync with your DSL. >> effectively, you can think of like new DSL versions equate to new evaluation results that you're going to have to check, right? And these new evaluation results are going to then prompt corrections on both the DSL and the agent side. those corrections reestablish you into a known good or acceptable working state. And so this is the second of another two phases here. And

the first one I like to say is just getting up to speed. You're kind of behind the car and you're catching up. And the second one is just keeping pace. So it it takes less energy once you're at phase two. You just have to maintain. You don't have to keep accelerating to catch up to the thing. So it's in a very nice spot to be. Um and

a lot of this is no surprise contingent on the quality of your evaluations. You have good evaluations, you can stay in this spot very nicely. Right. So now we can get back to experimentation. So we can actually do more refining. We can start trying things that we maybe weren't sure about before with tools, with rag, with MCP. So changes can now be compared relatively, right? We're in

a nice good So we can compare models to each other. We can compare prompts to each other. And we can then fix for one of the other And then broader changes are compared as well. So again, like I just mentioned, we actually started experimenting with rag, MCP, tools, etc. And instead of assuming it would be helpful or less helpful, we could actually just say is it or

is it not helpful with regards to our existing baseline evaluation set. So we went from subjective analysis to maybe the very beginning to really getting some nice concrete data. And again, what did we get in the end? we got the DSL agent. Um it's working quite nicely. I'm they're very happy with it. It reached all the goals. It's generating programs, explaining programs, and helping domain experts to

effectively program in the language. Um and most importantly, it helps them to understand the programs. So it's not just a tool as an agent to just generate programs. If someone actually wants to be able to understand how to express something in this domain, the agent assists them in the process. So, we've got it to where that meets those user expectations, where the users expect this agent or

any agent to be capable of assisting in DSL development. Which is great. Yeah, and then the agent then tracks with the DSL. Again, if the DSL changes, this is generally a a quite a big problem, but we're equipped to handle it. We have a process. Um and the DSL changes just an agent change, and it's it's basically just shy being nearly fully automated except the fact that

you should always have a human in the loop, so it will never be fully automated, but it gets close. And um they're actually going to put this in production in late April, which is the reason why I can't quite put any names or any specifics. So, I do apologize for being very general about things, but maybe that's also appropriate cuz it's the last day. Um and I

don't want to fry anyone's attention spans. some other observations, too. Um and I already kind of cheated and mentioned this before, but yeah, fine-tuning wasn't needed. Um and it really, you know, the evals are what told us this, right? We kind of could have suspected this as well, but we let the evals guide the process. We actually compared and assessed and actually see, yeah, we're not getting

a gain for the work we're putting in. In fact, it's sometimes a still loss, right? You're going from a more general model to a more specific model when you're fine-tuning. You're losing something in the process. And so, instead, we just added tools. We just exposed the ability to take external data from markets and doing some additional data from the actual DSL itself. So, being able to get,

you know, diagnostics in while the model's generating and actually improve the results that way. Works great. Plus, newer models are much more capable than they were. So, and that's not to like say that fine-tuning is not a viable option. It is a valid option, but for very specific use cases, especially with smaller models. So, if you're dealing with embedded systems, it can be a great Um but

in this case, they really wanted one very capable general purpose model that in addition to being able to work in this DSL and assist people with DSL development, could actually also support, you know, other tasks. It felt more kind of what I would say broadly equipped. Um this is a goofy slide. Um I almost took the hat and the glasses off, but I really like this one

actually because when I was making these images, they are AI generated if you didn't notice. Um there's a really nice artifact here in the middle that popped up and I was going to remove it, but uh it actually serves a purpose because that's basically how I feel about the grammar fuzzer. Initially, I looked at this and I'm like, "This isn't any good. This doesn't help me." But

actually, this is very helpful. This is very helpful. Um it's an unintentional benefit, you know, this loose syntax that we had initially. We thought we had it tied down. It was not as tight as we thought it should be. And there were holes in the semantics, too. And so, this led to issues. And these things are all edge cases. These are really difficult to find. Um it's

not that you can't find it, but you have to invest time and effort and we have things we have to be doing that aren't that. Um so, at one point we were even looking at writing our own grammar fuzzer. Um thankfully, we didn't because the agent did it for us during the process. So, at one point we even wanted to keep like a prior version of the

agent where it was like not that good because it just fuzzed so nicely. It was really surprising. Um yeah, and it found bugs in the syntax and semantics and so, that really um helped, especially at higher temperatures. Um Okay. So, just kind of some findings. most of it I've already just stated. I'm just kind of consolidating things here for everyone. I can't state it enough. Um I've

heard it in other talks here, which I think is fantastic. Um evaluations are really, really critical. Um they really help crystallize what it is that you're trying to solve for. It's also when you're working with clientele or you're working with yourself, um it really forces you to think what it is that you're going to be solving with an agent, right? And if you even need an agent

in the first place, too. Um and it facilitates really nice objective assessment when you're doing these experiments. You don't have to use a group of people to think, okay, is this better or worse? You can actually really measure something and say, we have gains and we can actually show them. Um really nice to show to clients, too. Um especially with new models, techniques, prompts, something comes out,

something is publicized, you read a paper, you can try it out. Um data is also still king. Even if you're not fine-tuning, um it's very helpful for your valves. Um if you lack it, you should build it up yourself. Um or you can synthesize it. Um you don't even have to use a cold hot phase, but again, that's what we found worked really, really nicely. Um but

you do need to know the kind and quality of data you need. Um this case was a chat application, but you're not always doing chat, right? It might be completion, it might be pure code gen, it might even be in a different language, you need to know these constraints up front. Um yeah, good for valves or fine-tuning. I think I say this personally myself probably too often.

It's more of a trope, but I still feel that like less is more. Sometimes just prompting is really going to get you 80, 90% of the way, surprisingly, especially with models we have today. Um it's really nice to start with just a minimal approach, evaluate, improve, and then repeat on that. Um that's also again a really nice baseline, doesn't take too much time to set it up.

Again, even if you're not using LangChain, even if you're not using LangChain AI, it doesn't really matter. Like you can still do your own evaluations and your own assessments for your own domain-specific applications that incorporate agents, and then you can measure off of that. That gives you a lot to go forward with. Um and again, early wins at this process do tend to compound quite nicely to

later successes. You're You're not catching up the whole time, you're keeping pace. You're spending less energy every time you're making changes. And that saves time and money. So, if there's one thing that I can say is a key takeaway, um do please do evaluations. Um once we have them in place, um we didn't have them in place even early on and when we finally got it, we

thought we really should have done this first because it guided every single decision we made from that point forward. Okay, so just some closing remarks. Um just a kind of quick summary on Langium AI again. Again, we try to keep it minimal and helpful. Um We try to keep the implementation fairly light um and try to avoid duplication of prior work. So, I didn't go into the

details with regards to how we actually tap into Langium, but we try to use re-exuse the existing Langium service set. So, 80-90% of everything you need is already there actually, which is quite nice. Um and then we try to help at various stages of your workflow without interfering. Um easy to integrate, no modification needed to your existing DSL. It's just plug and play. Um and it's easy

to keep in sync. Again, once you catch up, agents can be released like versioned agents with your DSL and this is quite nice. It makes it a lot easier to iterate and trust the results. And then um I had one more thing I wanted to add. I really wanted to put more of this in the slide, but it's not quite ready yet and we're still merging the

PR, but we didn't mention the new stuff. Um because of everything that we learned and that I just shared with you in this talk, which again, hope has been quite helpful or informative. Um we just built a CLI that basically just does all of it for us cuz a lot of times we have to set up Langium AI, we have to set up our evaluations, we have

to determine how we want to structure a prompt based on the structure of the Langium based DSL, which we need to know which services were overridden, but we know this stuff and we know where it's going to be and so we have really excellent AI tools, too. So, we have a CLI plus I'll skip through some points here. We have uh some skills that we're also going

to be releasing to with this for both Langium AI and Langium. And the end goal is that basically, if you have a Langium project and you're like, I don't know, let me see if this works, you can basically use the skill to set up Langium AI and use the skill to actually help identify via the Langium skill what it is in your implementation that is going to

be affecting your generated results. And then you can improve your evaluations. So you can get like 90 to 95% of the way through what everything I just said mostly automated fashion. Maybe 50% AI and then 50% something else. And if you don't want to use AI for this, the CLI is still helpful in this regard. Um again, it's not out yet. Um I really wanted to put

it more in the talk, but we still are doing some things and tweaking some stuff. But kind of it ties into the ethos before. This is based on our own personal observations. We try not to make stuff that's not going to be helpful, right? If I'm not going to use this, I don't want to work on this. And this helps a lot. This is specifically a time

saver. So if you're doing Langium and you're doing AI, this can help. Um and this is actually coming pretty soon. So yeah, late April, early May. We'll see. All right. So if you're interested, um there are some QR codes which are the the oddest QR codes I've ever made. They have like little rounded edges. They still work. They should. Um tell me if they don't. The one

on this side is for Langium AI tools, which is the core. That's everything I've talked about so far. And then the repo on GitHub for everything, which will also include the CLI and the skills. And then if you just want to type it the old fashioned way, you can go to github.com/langium/langium-ai. And with that, um really sincerely, thank you all for your time. Um and I really

hope you enjoy the last day and lunch, too. >> [applause] >> Thank you very much for this excellent talk. It was very interesting. Uh I I have a lot of questions, but I think one that maybe I would be most interested in is how did you end up with setting up the prompting? Did you like end up with few shot prompting or did you include some parts

of the grammar? How did this in the end work best for you? >> That's a really good question. Um So in the end it's a combination of few shot with the grammar embedded. So you're you're already in the right direction, right? It makes sense. Um having those additional examples that kind of primed like what we wanted for the output really helped kind of align the model in

that direction. And it's what we were trying to get with fine-tuning before actually, right? Trying to coerce the model, but we were just losing something. And so having a nice prompt that basically gave like the turns in such a case so that when we had that plus the grammar plus additional information about the semantics as well encoded. So we have to do describe things that are not

just going to be syntactically valid programs, but also semantically valid programs. Then we tended to get better results. Um there is one thing else I should note and it's an interesting observation. I inherently wanted to disagree with it, but again the evaluation showed it was actually better. Um fewer examples that were more comprehensive were actually getting better results. So even just having maybe even one or two

as opposed to like 10 were providing really good agent behavior. Um and it kind of makes sense because we're keeping things concise, right? We're I don't want to say it again, but the less is more, right? It almost feels like it's the same thing coming back Yeah, I think good question. Thank you. All right, cool. Well, again, I hope it was very informative and I hope you

all have a good rest of the conference and also a good lunch. Thank you again.