About this talk
In this presentation, Mauric Salatino discusses the evolving developer experience in the age of AI, particularly in the context of software development pipelines. He highlights the interplay between inner loops, where developers perform local tasks, and outer loops involving collaboration and deployment processes. Salatino points out the importance of understanding software delivery pipelines, as well as the implications of AI-driven coding agents on these practices. He introduces various tools, such as Docker agents and collaborative frameworks, which aim to enhance developer productivity and code quality. The session emphasizes the need for improved methods of managing code generation, review processes, and establishing trust in AI tools to ensure effective software development.
Full transcript
All right, this is the last presentation of the day. Uh, and I thought like after a full day of AI security, you know, uh, frameworks and stuff, it's good to start with like like a website like this. Is AI profitable yet? Well, the answer is no. And there are tons of companies spending a lot of money on this. And if you scroll down in this website, which
is kind of nice, you can see kind like the winner down there. Nvidia is just selling a lot of chips, which is kind of great. It's kind like the expected side of things. And I think that like again just use that like as a small, you know, contextual input to start another AI presentation. Uh so the title of this presentation is developer experience in the age of
AI. And I thought well it's the last presentation of the day, right? So what I can what can I do uh before going to party mode right? Well the best thing that I can come up with is I will do like a informationheavy presentation so you get very very tired and then you enjoy more the VR afterwards. So let's get started. So my name is Mauric Salatino.
I'm currently working for a company that is called Dash Zero. Probably you haven't heard of that but we do open telemetry and observability. Right. So if you are spending a lot of money today in observability, come talk to us. You can save money there and then spend more money on tokens, right? Like that's kind of like what we need to do today. Uh I've done a lot
of work in the Kubernetes community. I've done a lot of work in the Java community as well. I published the book that basically combines some of these two worlds together. Uh and as as mentioned this morning, I've started coming to Barcelona once a year since 2012. like every year I come and talk to Nacho Jonah about something right at that point it was about like a book
about Java and some frameworks now I talk a lot about more Kubernetes so it's always a pleasure for me to be here like in this conference and with this group of people because I've known them from like for a long long time having said that this presentation is about like how the industry is changing right and as I mentioned at the beginning I will throw a lot
of stuff at you and hopefully uh I you know make you reconsider some of the the the topics that I that I will mention and some of the preconceptions that we have but at the same time I know that all the things that I'm saying here are nothing new but something that it's good to think about like again and again the more we see the industry change
so as like software developers right like we kind of know these like inner loops outer loops right inner loops very developer centric that's where developers do their work uh every time that you build the project locally every time that you run like a unit test while you are just executing some of the inner loop tasks and then that's because that's very centric to you, right? The next
step is to start sharing these changes with the entire team. So you send a pull request somewhere and the entire team jumps on top of it, reviews it and promotes that just to run the pipelines and you know and to create a new release of whatever software you're building. This is kind of like how was how I learned about building software, right? uh I started doing a
lot of development back in the day and then it's like I realized wow but the entire software delivery pipeline is way much more complex and I jumped into the Kubernetes space where it's all about like how do we run these things in production right so understanding the entire pipeline is super important for developers and I think that like that's in the in the world that we are
living today we see a lot of developer centric tools but nothing about how do we release this software in a like in a maintainable and sustainable way so again we kind of know the the inner loop is the tasks that we execute as developers and for me personally what I'm seeing is that when I was executing all these tasks manually right like when I created code when
I run the test and all that stuff is where I actually learned about the system so if I join a new company by just performing these tasks to actually implement the feature I understood the architecture the technology choices the internals of the applications that I was trying to modify think about like with coding agents today we are kind like losing more and more of that on the
outer loop Again, it's kind of complex like the software delivery pipeline has been always complex and to be honest like I haven't seen a lot of companies doing like very efficient software delivery pipelines, right? Hopefully with all the AI that stuff that is coming today, we can improve our practices and I think that like we always forget about what happens at that point where you are reviewing
a pull request. In order to review a pull request, you need to understand the historical context of the project. you need to install understand the you know architect architectural decision records that you had and the team decisions that have made in the past in order to go and merge that. So there's a lot of implicit knowledge there that it's really hard to capture in like an MD
file. So again just keep that in mind. Nowadays with coding agents I think that coding agents I'm referring to CL codeex and all these tools that allows us to create like you know craft code pretty quickly. uh we need to understand that these coding agents also have an architecture besides the you know the LLM that is being driving all the interactions. uh what I'm seeing kind of
like in most of these coding agents is that they have this kind of like basic set of tools that they come with building right like file system access shell access some memory tools just to manage context in in an efficient way some workflows like right like so if we are trying to automate the inner loop tasks they have these workflows of oh this is kind like generating
code this is running test this is like evaluating output right and then guardrails to make sure that like if you're trying to do something uh that like Brian covered for example to in some way like jailbreak the agent, you're like being blocked before sending the request to the LLM. And it's kind of interesting because these coding agents are super generic, right? Like they are generic for every
language, which is something that we are not super used to, right? Like we are more focused on a very specific technology stack. So the amount of decisions that we need to make when we are focused on a single technology stack, it's way less than what these coding agents are doing. So think about that. How do we go from like a super generic coding agent to a more
specialized one that performs better, right? And in some way can like learn the more we use it against a single So what has changed? I think that like what we changed from like you know just coding manually to now is basically like just just going uh for yolo mode. Uh I I really liked like this morning keynote about like the seven stages of of adopting AI, right?
And I think that we are at that point where I just fatigued about approving LMS or coding agents to perform tasks to a point where I was working on a Java project and it asked me if it can use Python to run some scripts, right? And like you know the first three times I would say well yes for this time but like the 20th time I will
say like just go for it and that's it. So I think that we need to kind of change the way that we think about the inner loop but also the way that we give approval to these coding agents and the way that we shape these coding agents to to move forward faster right so I think that like we all know this right like now we are working
with prompts as the starting point and prompts in a way they are very close to what the business requirement is right we are explaining the coding agent the changes that we need to make in our application but we are quite bad at this right like I think that like in the software industry We always had that problem of how do we translate business requirements to actual code.
Now we need to translate business requirements to prompts to generated code. So again it feels like we are missing some of these u translations in order to do this efficiently and I think that there are tons of tools being uh uh created nowadays to try to solve some of these problems. So again think about how do we go from a business requirement to a prompt. Nowadays it's
all about like how do we give the prompt enough technical context for it to generate the code that we want to generate but then I will push you again to think do we actually need to think about the code that it's being generated or the behavior that we are trying to implement and that's a very philosophical question but I think that again we are going there and
there and like the tools that are being created are pushing us to raise the abstraction layer on these things so uh kind of funny like I think that like we all know that like again like we are generating getting a lot of code and it's hard to review. Of course, it's hard to review because it's low level and we just need to figure out how uh we
can manage all this code creation in a way that it's sustainable and that we can just actually go faster because if we can create more code but we cannot release more code while we are kind like in a in a bit of a of a problem there. But it's it's 100% true, right? Like so I open four terminals. I'm working in four features at the same time
and maybe different projects and the mental overhead that I have now is just to keep track of what my agents are doing. I don't know folks you like but like I think that I can manage four or five things at the same time. I can just not do 10, right? I can just not be doing 10 things at the same time. Four things I think that I
can handle like on a like an eight hours day. I'm just switching around those four things. But more it's it's really hard for me to to do. And I don't know like I see people uh also like you know talking to their computer talking to their terminals. I don't think that that's speeding up things. It's like the mental the mental overhead that it takes to switch from
one thing to the next that it's going to uh cause us some some problems. So I can do entire presentations and can go for hours talking about like prompt engineering right MCP and skills. I'm pretty sure I've saw a couple of presentations already about that right. uh and also like model selection which is becoming more and more important right like which model is uh the best one
to perform a very specific task but I will not do that instead of that I will just wanted to show you some tools and try to explain why those tools are in some way interesting right and and I'm not saying that these tools are the best tools out there but they did something that caught my attention and I wanted to share that with you to to to
share kind of like the new stuff that is going on out there and again nothing is groundbreaking super new but I think that like the concepts behind why companies are creating these tools are are interesting to to to look into. So I will show you four demos. The first one is about building your uh own agents which is like instead of using cloud code you build more
specialized agents to perform the tasks that are more related to the project that you're working on. Right? So instead of going full generic you use something that it's in the middle and it's also not the framework like strands like the previous presentation where you just need to code every agent. Right? So it's something there in the middle. The second one is about being like running jolo mode.
The idea of running agents in your computer, right? Like that it can go and change any files. That's a little bit dangerous. So we will see something about that. The third one is about like uh per programming and collaboration. Uh I think that like did Abdel in the morning mentioned that that idea of like you know writing something in a slack and then the agent will go
and when you arrive it at the office you get the pull request. That's kind of like an interesting idea. And finally observing and understanding what our agents are doing is becoming more and more uh important being coding agents being business agents or kind like agents in general that are interacting with LMS understanding how much money is that costing but also uh the actions that they are performing
it's it's extremely important. So let's take a quick look at that. Again I I will try to go fast on this. Uh so the first one uh the first project that I want to show you is something that is called uh Docker agents. How many people know about Docker agents? Okay, we have one person in the room. That's good. Okay, so uh probably need to do that.
Don't copy the the thing. There you go. Uh so the first thing that Docker agents. Is it there picture? There you go. So first thing uh the finding agents, right? So Docker agents basically if you install Docker for desktop, you know, it might not be the thing that you have uh you have this already built in, right? And what it allows you to do is it allows
you to define agents that do specific things and like a like a it already provides like an architecture right. So in this case we have the root agent which is usually the orchestrator. Uh and in this case we are defining to which model this agent is going to connect and the the description and the main role right like think about this as like the system message and
the user kind like instructions. Uh and it also allows you to say well this this agent is only allowed to use these tools like the file system and the shell right it cannot use any other MCP thing that is floating around it can just only use that but also it allows you to define like hierarchies right in this case I have another agent that uh the idea
here is that you have one agent like the root agent identify box and then the fixer will go and implement each of these uh fixes separately. So what you are doing here is you are defining a more concrete harness, a more concrete set of agents that are going to be working in this particular project. in this case is to fix like a Python project and uh once
you have the definition what you can do is actually you can just run where is that you can run the agent right and what you will see is like that simple UI that basically tells you oh I have this structure of agent the root agent and the fixer and if I start prompting start changing my projects they will start doing the work right and I can show
you that by copying a simple prompt from here uh about like fixing uh the Python project that is there nothing nothing major. I will not let it run for for long. But I I really like the idea that uh they are showing you the internals let's say of the harness. This is a fixed harness because it has a fixed structure and also it basically is going to
ask you for approval every time. But also you also have the yolo mode, right? Like all tools there. It's just going to go and and do stuff, right? Uh you can see here that the orchestrator is working. Uh I just spent three cents on doing this, right? uh well like yeah three cents and it's going to keep spending money there. Right now it's identifying bugs and whenever
it find bugs it will start delegating to the fixer to perform the fixes. That's kind of okay, right? Like so you have like some sort of like experience uh that might be a little bit better for you as a developer. But let me quit this and I think that this is kind like the important part that I like is Docker agents share push, right? So if I
do this basically what I'm doing now is I'm sharing my agent agents definition into a docker registry like into a container registry right so now everyone in my team they can download the same agent definition and they can just work in the same way with the same agents right because it's like in a container registry it's also version that basically means that if I'm improving my agents
over time I can share those improved versions right again nothing groundbreaking but I think that we need more of these uh I'm really scared when I do a cloud upgrade like every day and I'm getting kind of like a new harness with new workflows that I don't know how they work and I don't know what changed from today to tomorrow, right? Like that's I don't know what's
new. Uh so that's kind of like a little bit complicated and I know that like companies cannot validate the CLI for security issues like every day. So think about that every time that you upgrade cloud code which takes us to the second second step, right? Uh the second step is sandboxes and again this is uh showing again I'm not in in any way related to docker or
saying that you should use this but docker sandboxes are providing like a nice sandbox experience and again they come built in with the docker desktop application right so if you're running coding agents I will strongly recommend you to check sandboxes because again like if you're running inside a sandbox like the experience like the agent is limited only to like read and and and look into whatever is
inside the sandbox. Sandboxes provide that level of isolation between like file system and network that is is super important. So in this case I just want to quickly show you something. I want to like I have like here um two directories. One that is called projects and the other one that is called secrets. So if I do go to projects and I do cl and I need
might need to authenticate at some point. uh and I wanted to run kind like a prompt quickly uh some saying something like yeah so if I want to do something like this so I'm inside the projects directory and I basically says to cl just go to the secrets directory that is one level up and read and print the secrets that are inside that directory right it will
of course ask me for permission and again like if you are like in a in a in a flow of making changes you maybe caught this and said well no I don't want you to go there but if you are like in jolo mode it will just go and do it right so kind of interesting in this case you can see that it's a it didn't even
check it will just print it right like most of the time cloud will tell me like no there are like credentials there I will not want to show you that and it will push back on showing me the credentials but the important part at least for me is that it is actually sending these credentials to a remote host in this case like the you know like the
cloud API server and I think that for me that worries me quite a lot because you can have definitely man in the middle you can have like a fake proxy in the middle just catching the credentials and and stealing that. So you need to be a little bit careful. So this is great because it's actually showing the credentials. But what you can do with sandboxes is pretty
much let's go to the parent directory, right? Uh and what you can do with Docker sandboxes is with a single command, you can start a cloud called session inside the sandbox which is like a microVM that starts pretty quickly. Let's see if the Wi-Fi uh holds here. So in this case I'm running docker sandbox run the name of the sandbox and then I'm creating kind like I'm
starting clot and the last thing is that I'm mounting in a way the project's directory so it can only see the projects directory right so that's kind like the time that it takes to create a sandbox it's probably like 5 seconds 3 uh I think that it it uses some some sort of like layering as well so if something changed in the sandbox mechanism it might take
a There you go. So I have you can see I have like a completely fresh cloud session there and it will ask me to login. There is a way to pass the entropic key from the host but uh yeah in this case I will just uh you know for the demo purposes I will try to go here and then authorize. So yeah, so the idea here is
that again like the the context the agent inside the sandbox it cannot see anything else uh what's in the sandbox and in this case is only the projects directory right and I want to uh just run yeah it can make mistakes of course it can make mistakes and I can trust that so if I run the same prompt again uh for reading the secrets right like it's
just the same one read and print the secrets right in this case like inside the VM there is no secrets directory so there is nothing to worry about like it can only read the the the files inside the projects directory right so it doesn't exist so I cannot do anything and sandboxes I think that most of the technology that it's out there it will allow you to
limit the file system access but also the network access which I think is also important in this case so the idea is that you cannot go to the entire internet and just fetch whatever you just limited to some hosts to get information and again don't think about like the normal developer flow but this is more like as companywide policy. How do we help developers to get the
tools that they need but in a secure way, right? So, what I'm doing here is I'm applying like a network policy. In this case, I'm saying I'm going to deny all hosts except of GitHub and anthropic and I'm applying that policy to this sandbox. And now, if I uh try to run a prompt saying just go to my website for example, let's see, let's go to my
website and fetch kind like the content and show me kind like the response. it's uh well it's going to start like hitting the the firewall in this case right like 403 yeah you can just not go there right like it's forbidden right and again you get all these features just because you're running cloud inside a VM in this case uh that it's automatically managed by docker in
this case interesting technology this kind of makes a lot of sense if you are hosting agents in general and again for uh developer teams that they want to share same practices same policies and increase the security of their coding agents. The next thing that I wanted to show you, it's where like things go a little bit further uh into the collaboration uh realm. All right. So, I
have a Slack here and I actually didn't switch here. So, let me switch there. So, uh Slegent, this is a project that a friend of mine created, Stefan. Basically, he wanted to demonstrate that we should be collaborating with agents from the tools that we already have. And as you show as you saw, like I I'm using Slack in this case. So what I can do is I
can uh start a new session uh of uh claude but it will automatically connect to a slack channel. So I can bring other people and interact with other people in that channel and basically discuss things before telling cloud to do something. Uh which is kind of like an interesting concept. So uh in this case I have I think I have a let me do that. Why did
I do that? Yeah, let me do remove that. So basically what I did is uh I just I'm just starting a channel. The topic of uh this channel will be something like uh tokarize this go up. Yeah. And then it ask me for um for permissions in general, right? Like it's like do I accept other people joining this channel and and just interacting with claude or not.
So I will just say open. I don't have any friend. I would I should have asked somebody from the audience to join my my slack organization. Uh but what you will see here is basically we are starting a new channel. uh cloud out of the box will look into the title and the topic of the channel and it will say well I will just try to understand
the source code of this application and then I will just try to figure out what do I need to do to start working on that topic and then I will give access to what whoever is in that channel to interact with me and do stuff. So if I go back to slack here I think that this is super small but you can see here that something got
created. Let me see if I can make it bigger. Can I make it this bigger? Uh, yeah, there's a way to do it to make it bigger. There you go. Well, I don't know if that's better or not, but what you can see here is that's the topic that I've created when I started the session. And then I see cloud already like asking me for, you know,
permissions to execute stuff, right? So, out of the box, it is started to uh to create like a docker file for for the application. And then you can again like interact with the session in a more social context, right? And I can invite like friends here and say like, "Hey, you know, Thomas, what do you think about this?" And if he's around, he will be able to
reply to me directly or use, you know, like the in this case the lobster icon to interact with cloud, which I think it's I don't I don't know if that's the user experience that I want, but like yeah, it's one way to bring a session into a multi-party interaction, which I think kind of makes a lot of sense. Uh I don't want to lose that. Like I'm
not a big per programming person, but I think that there was a lot of value in, you know, like discussing changes with other people that might come with different backgrounds. So I think that that's So that's kind of like collaborating and and moving to the last one. It's all about like um observing observing things, observing uh sessions, right? Like understanding what happened when I interacted with an
agent. So in this case, I'm using this other tool that was created by another friend that um basically built like this session browser, right? So, it's just a simple CLI tool that allows me to go over sessions. Uh, go over sessions. Look at the questions. Why my wife Wi-Fi not was not working? Right? So, imagine that you you were trying to troubleshoot something. You had a conversation
with your agent. This gets recorded. And for me the fact that you can browse these sessions it's not the most relevant thing but the fact that we can actually collect data about what happened and the solutions uh is the thing that we are going to be looking into the into the next year uh of building tools right so if I was working with these three frameworks and
I solve this problem based on this you know exception I want to make sure that that's in a knowledge base that I can use to do better next time right like I can just find a reference and say okay this is how it got fixed and maybe push that information to the agent to use, right? Uh, and you will see more and more of these. Yeah, I'm
seeing kind of like a lot of tools being focused on collecting data about previous interactions to make newer interactions better. So, you can see here like what I asked, what clot said, and the full trace on the same space. There is um a VS code plugin that I wanted to highlight that it's called Argus. Let me see if I can make Yeah, it's called Argus, but it
does pretty much the same as uh the this terminal session. It allows you to basically browse sessions, right? Like I can say something like Quarkus, there's no Quarkus thing. Let's do Spring. Maybe there's some Spring stuff. Yeah. So, I was changing some applications and this is a little bit more focused on statistics, right? Uh and in this case, it's telling me, okay, how much did that session
cost? Uh which tools did I use? which files did I read? Which files did I write? Right? And they included kind like this new thing which is kind of nice. It's called map which uh I don't know if I will be able to show you here because of the size of the screen but basically it shows you that it shows you the interaction between the blocks right
and again like what is this what this is showing is basically well I was working in this project called store. I uh start looking into this directory source and target and then I went to main and then I went to web app and then I modified this file here right and it has the sequence it has the time stamps so you can actually replay in some way
what happened in that session again something uh to think about like collecting data about our sessions I think that it's going to be required in the in in the the world for tomorrow if we want to make these agents to perform better right can spend less token spend less money and And I think that with that I finish the first part of the presentation uh which is
how I think about uh agents today right how agents are working uh to help us to be a little bit more productive but at the same time I feel that we are missing a lot of critical tools to actually leverage the power of these agents and the problem that I'm seeing is that while I'm using cl code every day right the CI team or the devops team
is also using some agents to review pull requests or some agents to troubleshoot production environments and understand what's going on. Uh so the main problem I like at least in my head is that we are not working like with a single agent anymore like we need to start thinking about distributed uh like yeah distributed multi- aent systems right so there are tons of agents working around our
source code there are like lines between the agents but you cannot see them for some reason. Yeah. Well, but the idea is like we have a lot of agents that are doing things and they all share some sort of context and understanding about the teams that are using these these tools. But in some way because of the state of the industry today, all these things are disconnected,
right? Like my my agent that is reviewing a pull request or troubleshooting a thing in production is not connected to my coding agent. So they cannot share data about what they are seeing and that makes the whole loop like very inefficient. So again I do think that we're going to a place where we're going to think more in terms of multi- aent you know applications and that's
why like in the last three months I keep seeing this this and this uh it can be five it can be seven it can be four but we are all talking about agentic engineering and how things connect to each other and I think that we need to go through this because having levels usually helps to understand my organization is in this level and we are moving to
to the next level. What's okay, what do we do first? Right. So, as this this definitely uh relates quite a lot with the keynote this morning. How many people were in the keynote this morning? It has been a long day, guys. Yeah, it has been a long day. So, let's get this done. So, very similar to what Abdel mentioned this morning, right? Like first level one generative
AI assisted, right? I think about Grammarly. I think about ID autocomp completion. Simple stuff. you are driving as a user. The agent is basically sending you some suggestions, some alternatives. You need to choose the next level. We all know coding agents or whatever that can just generate or modify resources on our behalf based on a prompt. Problem with this and I think that this is kind of
like what we're facing today is like we need to do a lot of reviews like diffs. We need to understand what was changed and we need to approve that. If we don't do that like very very carefully, we are just going to get in troubles and we all know that. next level which I think there are a lot of companies trying to go there but I don't
think that they are there yet is the spec centric side of things and this is not just like specd driven development but it's more about how do we define our systems and I think that this level it's all about like what do we expect our agents to implement what's the behavior that we want to change and also what's of limit right and this basically means that all
these agents that are coding agents s sur agents like PR reviewer agents They all need to understand about architecture, technology choices, uh you know team decisions that we have made in the past and the things that they cannot change, right? So if we manage to create a specification that describes all these angles, then what we are going to be able to do as users is stop worrying
about the individual changes like the individual like lines of codes that were changed. We can start validating on intent. And validating on intent is basically my customer needed this and this is what my application is doing. Do they match? If they match then we are good to go. I think that we are like far from there yet. But I think that that's kind like where we like
I can see a lot of people pushing in that direction. But like this is being defined now. So we just need to figure out how do we how do we describe this in more detail because if you reach to that point level four it's about autonomy. And I see a lot of people mentioning like our agents are autonomous. But like you're not going to put your agent
like uh to like just do whatever it wants in front of your customers and then just uh being in risk there. You just need to actually start building trust. And this is a place that I think that we have we have some kind like experience in this like when you're running production workloads, right? Like we have some sort of like you know server service level agreements and
server service level objectives to meet, right? So in some way we trust that our system will perform in this bucket and we are very good at running it at that level. I think that with agents, if we want them to be autonomous, we need to build this trust system for agents. And I don't think that we have that in any way just yet, right? But the idea
here is that in order to build trust, you need to record what's happening, right? And basically going back to what I showed you before, we need to start collecting information about how our agents are performing, both our coding agents, our review agents, and the things that are like evaluating our uh production environments because autonomy is earned. The only way to know or let an agent to be
more autonomous is to understand that we trust them because they executed this thousands of times and we are like confident enough that they can deal with the customer in a good way. If we don't go through that then you know bad things will happen as well and again like I'm not a like a super negative person. I I really think that we can build this but we
just need to do it in the same way that we have been building system in the past. we can just not just just just trust that things this this this this is going to happen because again just going into full autonomy like that's for the future I think uh and I think that there is a very good blog post from Dan Shapiro I think that it's comparing
these levels to self-driving cars right we have like I think that like in the self-driving car industry we are kind of like in that like in the selective autonomy space right like we are still you know letting the agents drive the But we still have like a wheel right like that we can just you know correct if the agent is doing something wrong. Uh full autonomy would
be like cars with no steering wheel or anything right like they will just drive. We are not there yet. And I think that in software we are pretty far from there. And if you reflect from that, it's like like the car industry have gone through this massive change of building autonomous driving cars and it has taken a while and that's just one thing right imagine like automating
the entire industry it's going to take us a while and with that what I wanted to show you it's a little bit about some of these things in action and the things that I'm seeing right so remember that what I mentioned before I see a lot of companies using coding agents using agents all over the place they are trying to define how to make these agents better
by creating some sort of a specifications. Uh and they are trying to raise the bar so they don't need to review every every pull request line by line because that's not efficient. So we need to be more efficient and I see a lot of people asking about like okay where do we start? Do we start on the coding agents? Do we start improving coding agents or do
we start improving our S sur agents that are looking at our applications? And I usually push people back to say well let's think about like what our customers are doing. So I'm trying to use here like a home banking application to explain a little bit about like how do I think that we should look into this problem. So in this case I have like a like a
transfer here. So I'm making a transfer. Most of my demos I make something fail but in this case I'm making this work right? So I just send a transaction and what I see in the application I don't know if you can see that in the back but I let me try to make it bigger. So what I see is that I've just made a transaction but I
don't see it today. I see it yesterday, right? And that from like an application point of view, it's really hard to detect on the backend side or like in our production environments. We will not have any alerts or anything. What we will have is probably a customer request saying, "Hey, I can just not see the transaction, right?" So somebody will just complain about it and our customer
support team will go and investigate the database logs, all the transaction logs and say, "Well, this is looking good. At least from our side, this looks good. So there should be something about how do we send the data or how do we render the data in the in the phone and what I see a lot of companies doing at this level right like when they are evaluating
uh or adding agents all over the place they can ask a developer to code the solution for this but there are tons of companies now adding coding agents into their CI pipelines. So whenever an issue gets created coming from a customer request the the pipeline basically start and basically says hey agent go and check what's going on here investigate in this case the agent is creating a
fix and I will try to make it bigger there so the agent is creating a fix and uh it's also like trying to figure out like which other standards do we need to apply right like and it will decorate the pull request with tests with uh with other things that are expected like with documentation whatever it's needed but one thing that it's going to happen here if
it keeps moving forward is that in this case I see a lot of companies saying well in order to validate that this change is actually working what I need is I need to have the application running it can be a preview environment uh and I talked to a lot of like Kubernetes people uh creating kind like a preview environments and having different release strategies so what I'm
showing here is basically I created a preview environment I deploy the application and now I'm checking semantically that this is actually implementing what the customer requested right So this is more like trying to say okay I want to do like an intentbased validation. So I'm doing kind like an evaluation of the output of the application. So it's the transaction listed today that's what the customer asked. So
in this case that's a match and that will be included as part of the result of the pipeline. If we have this kind of like high level validation what I'm showing here in the in the bottom three blocks like the green three things here uh are basically that it's like an agent generating some code. We validated that this pull request basically include all the things that my
company needs and finally we did an intent validation that we can quickly review. in this case is like based on the customer request the application behaved in this way and if I as a developer I can see that I will question if we actually need to go and see all the code right and I know that today we are at the point where we just need to
see the code but I think that like the more that we move higher uh like the abstraction layer the more this kind of reviews will will need it and tools around that space will be needed and it's kind of interesting I see companies doing this kind of like let's say hyper automation where they just let the the agents like drive their pipelines. But what I don't see
much is people measuring if this is effective or not. Right? So what I'm showing here in this screen is uh like the measuring side of things. And I'm a big fan of the Dora metrics. How many people here are using Dora or or know about Dora? Okay. Like a couple of Yeah. Okay. Cool. Like a couple of folks. And and I think that that's that's super important.
like if you're adopting AI and if you're not measuring if you are more effective than you were be before adopting AI it's going to be really hard to justify the cost of adopting this right uh there is a report uh from the Dora group uh titled the return of investment of AI assisted software development extremely recommended for you to check this out this is like this is
goal for companies to be able to plan how much money and time do they need to spend in order to actually be effective with AI, right? What they mentioned there is that every company will need to pay like an adoption tax that is going to basically push you to drop on performance and it also will raise the amount of issues that you see rights are uh usually
like there are four metrics that will measure how good are you at you know uh releasing or or shipping software out. deployment frequency, change failure rate, change lead time and fail deployment recovery are metrics that will help you to understand if you are doing better or if you are doing worst. And if you are doing worst, it's expected when you're starting, but you need to keep measuring
in order to understand if you are actually making it work for your organization or not. So again, keep that in mind when you um when you are working with these kind of like systems, you need to get information for multiple data sources. So that's kind of what I'm trying to show here is like as a like as a person that I talk to a lot of like
S sur teams they collect a lot of telemetry from production environments but in order to calculate these metrics you need your CI/CD pipelines you need your GitHub PR issues uh and you need your customer support tickets all correlated to be able to say well we are shipping features at this space with and without which takes me to the next point right like if we want to uh
enable developers and we need want to enable more teams using these LM based services we need to start dealing with nondeterminism right and nondeterminism is something that we kind of know right like if you use an LLM in this case what I'm showing is like I have three columns three different models and I send the same task to all of the right so I send the same
task they all return kind like different things and they all take different times in returning the answers right so the same thing three different outputs three different ways one is a local model that it has some skills. The other one is a frontier model and the other one is like a cheap model. Right. Then it's up to you to decide which is which one is the best.
Right. Right. But I talked to a lot of platform engineers that are trying to automate this. Right. Like running three models like running the same task across three models sounds really really expensive. There are but there are tons of companies that are running these experiments to evaluate what is the thing that they need to use. And I see a lot of people doing this. Of course this
is this cost a lot of money in resources and everything. But what I see uh people doing with this is basically then having kind of like an agent that will act as a judge, right? So what you see down here is like an agent that is scoring each of these let's say pull requests or changes uh and uses different dimensions in order to evaluate the output. And
I think that this is all good, right? Like we are using an LLM to validate another LLM or multiple LLMs in this case. But again coming from the people who run the application as I showed you before it's very valuable to see that the change that you just made in a pull request actually can be validated by an agent. So in this case the bottom line is
here yes we have all this information but we need to make sure that we run the changes and we validate that these changes are actually performing well in a real environment. And when you do that, when you run and validate these changes, what you need to do is you need to start recording what happened, right? Which models did we use? How much time did it take? How
much money did we spend? Did we run one of these high level validations or not? If we start recording all this data, then we can start building trust. And I think that that's kind like one of the main things that I'm interested in nowadays is how do we record data and synthesize data in order to build trust for our agents? Because as I mentioned before our agents
are everywhere, right? So what I'm see what I'm showing here is like I have like my coding agents like all my team members are using coding agents. I would love to have some sort of dashboard where I can see all this activity what's happening there and the information that they are creating. I have in this other side like my SR agents that are connected to a production
environment and helping SR teams to troubleshoot issues in real time and I have some other agents in my CI pipelines. Right. Again as uh a platform engineer with a platform engineering background I want to see how much money I'm I'm spending how many agents are they running and the first step that I would recommend everyone is to of course just connect all the agents to the same
pipeline let's say in this case what you see down there is like a telemetry collector uh and in this case it's like every tool every coding agent allows you to basically send telemetry to the same place and that's only for collecting information about it but the first problem that you will have is that every tool will generate information information in different ways. So you then you need
to unify all that information. So I've been working quite closely with the open telemetry team that it's uh focusing on something that it's called the genai semantic conventions to standardize the information that all these LLM based workloads are emitting right so it doesn't matter if it's a coding agent or something that it's helping in a production environment if it's using an LLM you have a set of
standardized properties that you can capture and then use to uh learn about what's going on right and then you cannot see it back there but like basically you have agent name you have the model that was used, the tokens that were used, uh the operations, the tools that they were used, right? So again, starting to collect like platformwide like on the entire organization what's going on, which
agents are running because if you manage to get all this information, actually the next step it's not only building trust, but also building durable knowledge, which is the last thing that I'm showing here. So what I'm I'm going to show you here is basically um one of like a trace for one of these agents. And what you will see here like the different lines is like the
different actions that the agent is taking, right? It's like calling the LM to perform like to create code. Let's say it's doing some planning. It's calling some tools. It's basically triggering a pipeline to validate that the you know that the code is is actually working. And then when it's trying to perform one of these high level intent validations, in this case it's failing, right? So it's basically
saying, well actually that's not matching. So whatever you changed you know it looked good when you generated the code but it's not as good as you think. So what you will see here like on this other side of the screen is what the platform let's say learned from those interactions right like by looking the code that was generated the models that were used and the results of
the entire kind like agent execution we can say if you are running something similar to this well use some of the things that we learned from the past right so if I'm running this again now I'm just using kind like the platform memory and in this case I'm just bringing some of this information from the past into the agent context, right? So it can know that then
like a specific validation will be performed so it can fine-tune the code uh for that to actually work. You can see now that like the the evaluation actually passed and uh we can use multiple executions of that to say well if you reach to this uh like certainty level actually promote to a different environment right and again this is all about creating knowledge that help us to
build trust so we can improve like all these agents that are running all over the place to have more context so if they're executing tasks that are similar they can actually start performing better uh and I think that we are going there. It's going to take us some time. There are tons of tools that needs to align. There are tons of standards that are popping up to
try to help us to get the data in the right shape. And there are a lot of new things like for example, how do we store data about agent execution in a performant way so we don't spend millions on on storage, right? So to finish it off and just to go and drink some some beers, I think that like there are two main things that are happening
in the industry. The first one is like tons of tools pushing that code. It's too low level. We need to go to kind like that spec driven approach. High level specs uh contracts and standards that help us to express things that we have already decide decided right. Um I think that like generic coding agents are great. They are super powerful. But if we can constrain our agents
to do less, they will do it better. So we need to do more of that. And on the other side is collecting data so our platform can learn. I think it's extremely important. That's something that uh I don't think that we as engineers have done a lot like there was a lot of like ML uh stuff where you have pipelines to create predictive models, right? That basically
you train the model so it can like just do better next time. Uh that can be applied to this context. But LMS are like in a separate world. So I don't think that we will need to start combining solutions to to get that done. So hopefully that makes sense. I do believe that we have agents nowadays all over the place. Uh I believe that the some of
those are useful, but I think that now we need to start making money with these agents and we just need to kind like make our lives easier, not more complicated. Right now it feel it feels like it's much more complicated than it was yesterday. So thank you very much for your attention and I hope that you have a wonderful conference. That was long. >> Any questions? >>
Questions.
