About this talk
In this talk, Andre, a software engineer at Dynatrace, discusses the challenges and considerations when using large language models (LLMs) in software development, specifically focusing on their effectiveness in generating code. He shares an experience from a fintech startup that faced issues with VAT errors due to inadequately defined models. Through his experiments with different LLMs, including smaller and larger models, he reveals that providing structured specifications, like YAML, significantly improves code generation accuracy. Andre emphasizes the importance of discipline in formulating prompts and specifications, arguing that they should serve as a guide for the developer rather than relying solely on AI. He concludes that while AI tools are evolving, the responsibility for clear communication and intent remains with the developer.
Full transcript
[music] >> Good morning, everybody. I hope you slept well. Yeah, everybody's ready? Guten Morgen, Leute. Okay. So, um if you're ready for the next round with me, let's begin. Okay, let me introduce myself. My name is Andre. I'm a software engineer in Dynatrace, and Dynatrace is the one of the hugest companies that making observability and monitoring. And that's why I'm so curious to measure everything that is
getting into my hands. This story started when one of my friends that is working in the fintech startup um created the forum to collect the data from his customers. It's a startup, so it's not much customers, but when they got into the Germany, they got some errors because of the VATs. And the problem was that because the models that he used to produce everything do not know
anything about or rules that specifically domain delegated. And we cannot trust models, so if something made by engineers, you can do this, but most of the time it's wiped coded by the product engineers and we have no idea what is going on. So, the problem is that the whole payment was not broken. It's working well, that's what well, unit test well was well, end-to-end test was well,
but the thing is that for the U it's not working and they try to get the customers from the different countries and get into the errors. And this led to me to one specific idea that at the moment we are building faster, but we still debugging and as 10 years ago. what I added here. We have developers that is writing the code, then they describing the intent
to the LLM, then we have the code reviews and nowadays it's also made by the LLMs, not the humans. And then we need to test this code, debug this code and test the style and nobody's doing this. for the experiment I took everything what is on the screen. So, I simplified example create the React invoice form, then I took several LLMs and I specifically took the smaller
ones. So, I will explain in the end why I also choose the smaller ones and it's very important. I made the test in the TypeScript. It's was not judged by the LLM, so it's absolutely strict. And also I cleaned everything, so the the larger models are not the silver bullets that will help you. about seven experiments at the first time and this information that I brought to
stage in the reality I made about 100 or even more for the production. I cannot bring it to the stage because it's like commercial. But the results are pretty same. what I tried to do? I entered simple request. Okay, let's build some kind of form. And the result of the form by the test was three from 10. So, it was really really poor. It's bad. Okay, you
would say it's just because your Mistral or Code Llama is really bad. Yeah, maybe. then I added specifications like this in a YAML style. And the result was almost 10. So, it's nine from 10. So, even smaller model can produce you something that you want if you specify it and not in words. In this. Then I took Gemini and Claude Sonnet. Mhm, bigger model paradox. So, in
theory, if we use the bigger model and the better model, the result should be better, but it's not. So, Gemini guessed something. So, it's assumed half of the result. Claude Sonnet immediately said, "I do not know. Please explain." Good result. No argue. So, this is better than nothing. let's take a look why the text specifications outperform. On the screen you can see. So, I just took the
whole text with explanation how we are trying to made our prompts like build this and this and this and using languages. So, the problem is in language itself. If you want to be strict from my opinion, use German. You cannot mislead everything. In English it's pretty free to use. In Russian I would not even try it because absolutely can be explained by anything. But then, if you
add YAML with explanation, then LLM will understand you better because it's machine. Then I decided, what if AI will bring me specifications? And this is what I hear from almost every corner. It's like like don't do this yourself, ask AI. Okay, this is the The score of the specification itself is pretty low and I spent a lot of tokens on this without any meaningful result. Let's just
compare between guessing and result that it we're getting that is really performing. If we put the YAML, just YAML without nothing, it will be good, but we also will know that half of the result will be assumed and we cannot trust the assumed result. And we if we use the YAML with the pure text, it will be almost 10 after 10, fully reproducible result. So, a lot.
So, like hundreds, thousands, the YAML with comments is bringing the specific results. So, the gap is here. How is it? On the right side. On the right side we see is what exactly we know about what we want. And in the right side we are trying to build it. And the problem is not in the technologies, it's in psychology. So, nobody can get into your mind, get
the information from your and put it on your paper. that means that the gap between model is you. It's what exactly you know, and how are you writing it down. And it's very In In the past, when we didn't have the AI for building, it was not in our stack, we used a lot of things that brought us the discipline. And it's really important. We had KISS,
we had TDD, BDD, whatever, BDSM, and everybody knows that BDSM is better than the KISS. the thing is that all of this abbreviations are discipline. It's not just something. It's not the stack. It's not the framework. It's discipline. And it's exactly what we need to have in the future is discipline how we're writing our prompts and specifications. I tried to create something for myself because without discipline
everything is looks like a chaos. And I also use abbreviations. I do not want to be really ethical and do not say that I invented something, but here it is. It's simple thing. The specification is not for the AI. Specification is for you. Because the problem is not how you broke the production with your prompt. The problem how to work with the incident, what exactly you written.
How did you written it? What was inside of the prompt? And this specification not for the machine, it's for you. First of all, structure is important. Structure does not mean that you need to create specific folders or name something. You need to know by yourself. First of all, how you structure all your prompts and work with your prompts. Second thing, that intent should come from you, not
from the machine. >> [singing] >> It's your responsibility for what is going on, not the machine. You cannot just get [music] rid of your responsibility. Next thing, performance is not so good. I will show you. It doesn't matter what you use, YAML, JSON, whatever. And as a TypeScript engineer, I would even prefer to have types and interfaces before I start doing anything. And the last very important
thing, you did not say to LLM is your problem. This is that people put on yourself. So, that means limitation >> shouldn't be this hard. Scaling equals complex infrastructure. Advanced bot systems blocks you. Heavy data loads break pipelines. Oxylabs can fix this. >> First thing, >> Meet us at booth 0021. Fast-adapting infrastructure for web data. See it in action. Real-time data for AI models. >> out the
button that says pricing and market trends. >> Track pricing and market trends. Reliable business intelligence. 177 million plus countries. Oxylabs. >> We say what's going to happen on play. What's going to happen on network error, on validation error, and what will happen on the loading state, and what not to do. >> [music] >> Then, everything is fair enough. You're saying to machine to something and it produces
exactly [singing] what you want. >> The intent is in your head. The models [singing] cannot know the specification [music] of your domain business. >> Impossible. Models knows only something that they seen before. And if they did not seen your business before, you cannot expect that it will know Everything. [music] So, everything internally, even if it's GDPR, and GDPR is changing. You should explain it. The format. This
is what I said. The format is does not matter. Format is for you to understand what exactly you gave to machine. So, if you used a standard way as a YAML, it will work. If it will be renamed, it will work. And if it will be abbreviated, it's also fine. >> LLM is smart enough to understand what exactly this thing is you are means, and it will
produce you [music] >> Collecting web data shouldn't be this hard. Adding the scaling equals complex blocked. Curious? Visit our booth at 0021. Fast adapting infrastructure for web for AI models. Track pricing and market 177 million IPs. 195 plus countries. Fast >> We expecting like machine will understand what to do. But, after your discipline, if you do it like format it, you will spend >> less tokens and
the important part you don't need the future model. Even smaller model will understand [music] what >> What you can do to break something in your life. You don't need to use the method what >> So, what I see how my juniors are working. For example, they have printing and praying. It will work somehow. No, please don't do this. Do not delegate your specification to reality. Yes, it
can do the future. But the problem in the AI that is you have no idea where is the bomb inside specific- specification. It's somewhere there and nobody will see it before it blows. The schema paralysis, yes. Do not spend hours to understand how you can apply the format. Just do >> something. It won't work. You don't need it. You can gather with your team and just decide,
"Okay, we will do it like this." Done. >> You know, you are meeting >> Scaling equals complex infrastructure. >> And >> Advanced bot systems block you. >> marked as red. >> Heavy data loads break pipelines. >> That is the most dangerous >> Curious? Meet us at booth A0021. >> That's the big rate. >> 177 million 195 plus countries Oxl abs, fast >> And it will silently give
you the result. Deal with it. biggest antipattern here. >> But the do you need to do next Monday? First >> clean and brush [singing] your teeth. 200 push-ups. No, not [music] this. Write down your first phrase. Write down a phrase. >> [singing and music] >> It's really important. You need to write down what the machine should not do. The most important part. explain what machine should do
is like okay, we can do [music] this, but what the machine can do is more important. And [singing] you have specifications that are related to your domain, what you think you know, please also add them. my last consumption will be that yes, specifications are for you. And yes, every year we're getting our models smarter and smarter and smarter. Of course. And we need smarter models, but most
of the things that we as engineers need is smarter you. Thank you. >> [applause]
