Ballroom F Friday Mar. 06 - SCaLE 23x

In this keynote address, John Willis discusses the intersection of DevOps and artificial intelligence (AI), focusing on the concept of agentic AI and its implications for operations in large-scale infrastructures. He emphasizes the challenges posed by probabilistic outcomes in AI systems and stresses the importance of quality assurance, drawing on principles established by W. Edwards Deming. Willis shares insights from his personal experiences, case studies, and recent papers, pointing out instances where AI has failed, leading to significant security breaches. He advocates for a thoughtful approach to adopting AI technologies within enterprises, urging stakeholders to establish clear policies and understand the risks associated with AI agency. Throughout the session, Willis incorporates historical perspectives on AI, highlighting the necessity for organizations to redefine their strategies in response to evolving technological landscapes.

Good. Hey, there we go. All right. Cool. Sure, no worries. Check. Is that Matthew Gold? I'll run >> everybody. >> Two minutes. Two minutes. Get your stuff together. Somebody said, "Oh, Yeah, that's dude. You guys were there. DevOps. >> All right, everybody. It is 9:00 a.m. on Friday, March 6, 2025. And uh let's kick off the DevOps Day LA track here at beautiful at scale 23X in

Ballroom F at the Pasadena Convention Center. My name is Jordan Schwarz. I'm one of the um DevOps Day LA organizers and a Scale attendee since scale 4. Um and uh just wanted to welcome you all here. Thank you for getting here early to catch John's presentation. Really appreciate it. Also wanted to mention uh that myself and uh Rich Horus, who you'll see helping out with the sponsor

slides and whatever, we host a local meetup in Santa Monica and there's an LA DevOps monthly meetup at O'Brien's Pub in Santa Monica. We usually have two presentations a month um including one on uh March 26th. um that will be announced soon on either meetup.com or aluma. So um you can continue on and uh catch our presentations there and maybe even present there. A couple of the

presenters tonight are today have also u presented at um LA DevOps. Also I want to thank our sponsors. We have a number of sponsors. You can see them out on the hall uh there. And so please uh they help keep DevOps day and scale running. So definitely go by and say hi and chat with them and uh maybe they'll share share some uh cool swag with you.

So uh with uh no further ado, our our keynote speaker is John Willis who um presented at the first uh DevOps day at scale 15 years ago. Um and actually he had flown out for sort of a DevOps drinkup um that preceded the um where I guess where um where scale was conceived in the back of the bar. So anyway, with no further ado, uh John Willis,

take it away. Thank >> Thanks. Thanks, Shan. Awesome. Hey y'all. appreciate a large room this early in the morning. Um, but the trick is, pro tip, put the word agent in your presentation. They'll come they'll come All right, so this is something I've been having a lot of fun with. Um, um, you saw, you know, we all caught the AI craze, right, in certain stages and all

and and I kept trying to figure out like I'm the DevOps person, right? Like I I was the first um I I was the only American to first DevOps days in Git. And I'm not going to go through all that nonsense, but but like I was the DevOps person. And then in AI world, I was like, what am I? And then I I struggled to find out

what like DevOps what what can I say that really means something about DevOps and AI among all the noise that's goes on currently and and I didn't really find a groove. But it wasn't until these things agentics started happening. I'm like, okay, I think I found my place in this world because I worry about largecale infrastructure. You know, I always say that I'm the quintessential ops in

the DevOps. I've been doing ops my whole life. And what do ops people do? We protect the brand. That's what we do, right? And and so I always the places I get excited is where I worry about how a CIO is going to get hammered by the technology, right? So um I've written quite a few books probably most famous for writing the dar handbook with Jean Kim

J Humble and Patrick Dar um I wrote a book about Dr. Deming. I'm very passionate about him. And then I have a recent book that's been out for about a year called Rebels of Reason, which is the 100red-y year history of AI. By the way, Chetch didn't happen on one day, right? I'm actually I've decided that I'm for this year I'm experimenting. I'm not giving away books.

I'm not getting sponsors to give away books because people don't read, right? And so I'm just wasting everybody's energy by handing free books. So I'm selling books for the first time ever at the booth over to the left. My son's out there. say hi. And if you buy one book, you'll get a couple free. So, we're trying I'm just trying to mix it up a little bit.

Um I I really would like you to sort of read those two books in the middle. Um and you'll get some other books free if you buy them. But like that's enough pitching here. One Oh, no. There's one more pitch. I uh I was asked to do uh I live in a small college town. I was asked to do um a a a sort of extended learning

course on uh on my book Rebels of Reason, which is this 100-y year history from Boolean logic to uh multi-layer neural networks, right? Um and I did it for a bunch of retired professors and it was a blast. So I I realized it was so much fun that um that I turned it into I'm going to try it out as a webinar series. So that's the barcode.

It's free. It's going to be like an eightweek session and it really it's just like a lot of my book is fun but like this is a lot of fun you know watching the scientists who put probes into a cat's brain to get electromagnetic uh sounds and figured out how the edge of the brain works and which is how you get pictures today. All right, let's go.

I'm a big Deming freak. Um let's start off with some demingisms. Uh his famous quote is a bad system would beat a good person every time. Uh, you can tell why I'm excited here. A bad AI system will beat a good security person every time. All right, we the stakes are really high right now with this stuff. I mean, I'm not one that's going to if you

interpret anything I say is that I say don't use AI or stay away from AI, you got the wrong interpretation because I'm using it in every aspect of my life right now. Um but what I am saying is there are dragons capital D dragons. All right. And and particularly when the stakes are high like protecting a multi-billion dollar brand. Um so this was interesting. Um this there's

a paper this came out recently uh agents of chaos and you know we all know the sort of hallucinations and we're getting better and there's tools I've done presentations on evaluation software um that you know sort of observability on evaluation and you you can it's a probabilistic system right like that's the net net like the answer is not deterministic from these things right and that's fine so

we know that going in we got our code we had all this what this paper pointed out which is really fas fascinating is that um that the not only do we have this probabilistic outcome, we got probabilistic we ingress. We say do this. We know it's going to kind of do that, but it may not. So, we got to double check it. But what we didn't realize

is what it's telling us, what it did is not always accurate. So, point out it says, "Hey, I deleted a database. Oh, thanks." You know, it didn't delete the database, right? that. So this is an interesting paper and then I realized like this is pure Demming 101. Deming basically says quality you can't you know you can't have a conversation about quality. Yeah boss I did it. It's

working. But you can't do that like he's saying that for 70 80 years ago. I love to see a young man in the that's awesome. So anyway it's hard to point you out but that's cool cool having young kids in these and why why would you come to a DevOps presentation? You're going to do very well in your career. Um the um anyway so um anyway so

the the thing about deming and I wrote an article just uh came out this morning on like how this is a very interesting thing about how quality needs to play here. Um you know the um yesing edge so let's go right into the uh looks like I skipped the slide but yeah yeah no we're good. All right. So what we're gonna talk about so that's the quality

aspect. so we got to think about like remember I told you I found my I think I found my rhythm in DevOps dev stag ops. I'm also considered one of the books that you could get for free is something called investors unlimited which is about the whole genre that I helped create called DevOps automated governance right uh which is it's a fictional story based on the Phoenix

project style book. But we have to figure out how to adopt these really strong principles for 15 years that have been working really well. We just got to figure out what they mean other than just noise. Um you know the uh agentics in the enterprise is moving fast. There's conflicting information about it's not working. It's happening. Um there's new challenges. Um there's this concept of polymorphism in

in adversarial attack zones. It's been around forever. like this is something where code can sort of change itself dynamically on the fly. That was scary pre- aents. It's incredibly um frightening that not only the code like the idea of a polymorphic code which was I couldn't signature the code because it was going to change based on the dynamics of the environment but at least once I could

identify it I could signature it. Now it gets created dynamically on the fly, right? Um and then so you go back to the classic I AI trope, right? The 2001 space odyssey, right? You know how I you know these are open the bay doors there. How and you know I can't do that. But you know what the thing was sort of bottom line of that story was

how was set a goal to go to I think Saturn. So, anything that got in the way of how to get to Saturn was an obstacle, right? It didn't use common sense like, "Oh, yeah, I guess you're right. I probably shoot out because you'll probably die if I don't." But didn't care. And so, that's sort of the trope we've been living with. I'm pretty old, right? It's

for a while. But like this happened last year. This is the replet story, right? And it's kind of old news now, but it deleted database. And by the way, in the constraints, the specs for this, it said in all caps, do not touch production databases during a code freeze. Right? Sorry, bud. I'm going to to Saturn. Um, so we're seeing a lot. We'll talk more about that.

And this was interesting, too, on multiple levels. This is the uh Opus 46 system tests. If you want to have fun, if you're like a real geek, you read the system tests, which they they're called the system charts. uh uh every time they're reduced they they sort of publish all the red teaming for their model. So every new model usually puts out a pretty it's like a

lot of fun to read because it's all the um it's all the red teaming and some of the red team are game gamifications. So now and I'll talk about a real famous one in a little bit but like okay so it said that you know when when they're sort of evaluating the latest model by the way 46 is by all intents insanely good um you know overly

agentic behavior um it used credentials and new or not allowed um hey um it bypass tools but here's two interesting things too and this is sort of like ignore the all caps do not use but then later on in a simulation It did whatever it needed to do when you said make all money at all costs. But here's the thing like so in my um history of

AI course, one of the pivotal moments of like understanding bias. I don't know if you've ever heard this right. There's the there's this they did some studies about what happens is you take all the words became a multi-dimensional vector and then it's really cosign similarity. Sorry to get too technical of the distance between things that really finds the next word, next sentence, next paragraph. Well, what they

found is the bias built into all the pictures and all the things we've written over the last 10, 15, 20 years just reflected in the the the geometry of the vectors. In other words, if you ask like, hey, here's a character that is in the medical profession and is a doctor, it was more often almost exclusively going to say it was a man. And if you said

here's a, you know, a nurse, it was almost always going to say why because that reflected the corpus of data that it trained on. Well, so this is interesting. I just noticed this this morning, which was say what that's telling us is what our models know right now is they're kind of more likely to ignore things that are sort of you shouldn't do and more likely to

reward the things like, "Hey son, go out and get rich at all costs." Right? that's built into the psyche of the there is you can't really say psyche but that's a whole another hour presentation um and then you know in the enterprise this is sort of interesting too right like um you know what's happening enterprise I remember I was at the Gene Kim conference I go every

year um the it was used to call the DevOps enterprise summit now it's called ETLS bad thing don't change don't take out the word DevOps because you don't think it's cool anymore um the uh but the that the couple weeks before that MIT report 95% of all gener things are failing. I'm I'm at a conference where some of the biggest banks, insurance companies are presenting um uh

Fidelity, Top Pal, Fidelity, 16 trillion in assets under management. 16 trillion add up the first 10 states in the United States 16 trillion. I think last time I checked, China's GDP is basically 17 trillion. One company manages 16 trillion. They're doing agentic AI. All right. So, um, and like I don't have to tell you, just watch Tobo's presentation. But that same week, Dora says, you know, the

the research, uh, the dev devops operations research, um, program, right, that that Google bought off of Gene, um, they said, "Hey, it's working pretty well, right?" And then, you know, Microsoft says 1.3 billion in 28. I think they're off by billions. Um, JP Morgan, Chase, all these, they all have success stories in AI right now. And then, um, this was something that came out recently. hours at

a presentation my good friend Josh Long who's like Mr. Spring himself you know he pointed out that there's a survey recently 60% of the enterprises now leverage Java so like I don't know I think it's working in the AI in the enterprise all right so this means that it's happening in enterprise there's probabilistics we have language mutations of issues ingress like what we tell it is it

going to do what we want to do it does something it tells us does it really tell us what it did do and by the way we're going full in on this stuff. And by the way, we don't have a choice. You your CEO I I a large bank. True story. I can't tell you who the bank is. The CEO um there's some vendor that came in

has sort of a co-pilot like product. Not a big fan of these people, but they and they got the ear of the CEO and they and say, "What what's some program or project that's been bothering you?" He's like, "Well, that one's they've been delayed. They've been over budget. So he built a prototype for it, a prototype. All right. In a large like top five bank, right? And

he's like, "We're on an ELA with this product right now." Not even consulting the CIO literally, right? So the the pressure is on, right? Like it's going to be hard to avoid those kind of things. So, what we have to do is sort of figure out like how do we basically build what we do well? I figure you're in this room that you're part of my tribe,

right? Unless you just had nothing better to do this morning. Um, that you're part of my tribe. And what do we do? Well, we try to figure this stuff out. And I'm not telling you over the next uh 35 minutes that I know all the answers. I'm just telling you that there's a lot of questions we have to answer. By the way, I'm getting really close to

retiring, too. uh and like you got to answer pretty soon. All right. And so I'll start with I think in my research and some of the people I've talked to, how do we start defining particularly now we're talking about agents, right? We're not talking about co-pilots. We're talking about if I'm literally going to build these agents that are going to be goal or warranted. Hey, go to

Saturn. Um you know, and we're going to start using these like we have to start thinking step all the way back and figure out what is our meta language within our communication. You know, one of the most successful things about Lean Startup by Eric Reese was not that it was really interesting and it was smart stuff and it was pretty cool and it sort of evaded everything

including a big part of the DevOps story. Um, it was that it became a common language within an organization. So the thing that we have to really start thinking about are do we have a common language in our organization whether this is the right one or there's a better one. um you know so are we thinking about um you know how do we define this sort of

a structure and and a gradient because it isn't it's either this or this particularly when we got probabilistic impress and egress right so so we start thinking about like can we create sort of a gradient level of there's informationational actor it's like tell me things I don't know at a scale that I can't do great um give me advice where I have to approve it. So something

might happen, but I'm a prover. So what's the human role in this? Or I'm going to give you a little more. I'm going to let you do a couple things, but I'm going to try to put the guardrails. But by the way, I think it was the um the Opus 46 system card that basically said that it bypass it literally figured out how to mutate your guardrails,

right? Um, and then you sort of get into sort of these lot of advanced levels, which you know, in all honesty, I don't see agent and large enterprises going all in on sort of um level three and four, but like we're moving there. This is where we're literally going to start moving to um giving a level of authority to things to do things because they're so more

efficient and the risk maybe is, you know, the the the ROI balance or or calculation is it's worth doing. Uh most organizations are at level one. Um but here's the thing. There's a there's a sort of an impedance mismatch because again I don't have hard data on this but what we're f I'm seeing is that open source projects are tend to be in the three and the

four and the enterprise are blindly thinking that they're at level one but they're using open source tools that are operating at three or four. Right? Is anybody having fun yet? Okay. Good. All right. I've seen a lot of like e scared faces or oh my god like uh should I sell shoes for a living? I'm tired of this stuff. Um the um so all right so then

like I'll go through these but these are interesting things that have happened over the last year right um you saw the replet one was an example where the agent deleted a production database during a cold freeze and why because it basically said do that you didn't you kind of thought you told it what not to do oh I didn't tell it explicitly not to destroy the universe

um you know so it it basically went it hey like I got to get I got to delete that database to get there. It's not it's really not different from what Hal did in the in the fictional movie 2001, right? Um and and what I say is here's the thing too that we have to is that if you're in a leadership level at a large bank, you

get told pretty early in the conversation of your employment, there are things best case you'll lose your job for, worst case you'll go to jail. Right? That's a conversation you have at a at a leadership level in a bank. Um, so you know that your boss tells you to do that and you have to lead a funds day or mutate a funds a tier zero service. You're

like, "Yeah, I'm not doing that." Like I they're telling me that I'm not going to get my pay raise to do like I don't care what the carrot Like you're like I'm not going to jail. Mile doesn't know that unless you explicitly define every edge case. And this is where it gets tricky. You know, we talk about specification. You know, we move from prompt to context and

now we talk spec, right? And and like are we actually creating a policy um spec for our sort of spec driven development, right? Are we keeping up with the process? So, the mess is um oh, you know what? There was a couple more on that slide. I didn't. So the Claude code um it was a credential theft there's the caption bypass even the the shah halude right

I've had conflicting but I do one research company has assured me that it was AI based or or the genesis of that sort of the worst supply chain hack in in history was AI generated I don't think it was agentic based but it will be the next one will be right um and then I told oh this is another good one right the um this was um

it was the um it wasn't open 46 it was it was last year's um earlier release of four in the uh system card red teaming they went ahead put a game scenario together now it wasn't a real story but it is a real story in other words what they did is they told the AI in this case uh Claude that they were going to shut it down

but they set up a fictional manager that had the possible authority to shut it down and an email, a chain of emails. And that's all they did. It went out, went through his emails, found the fictional version of him having an affair with somebody, another employee, and the model threatened to blackmail. Now, that was a madeup gamification, but the fact that it just did all that, um

so um you know, I think the thing is should be pretty obvious that agents can basically do everything that a human can um at at a scale that humans can't do, right? um you know so and depending on the authority and autonomy you give them this is the key you have to start thinking about the you can't just go and say yesterday we like it's almost like

the cloud answer like we're not going to do cloud we're not going to do cloud we're not going to do cloud Wednesday at two afternoon we're gonna do cloud right like you kind of got away with that you're not gonna get away with this with um So all the peril that we've had due to being terrible, capital T, terrible at security, we've kind of gotten away with.

I mean, like we have our this, you know, this breach, that breach, that breach. Yeah. Okay. We're not going to get away with it now. So our the way we think about security is broken. you know the firewall w the fact that we think there's a deterministic solution to all we have to reinvent ourselves you know one of the interesting conversations going on is uh trusted compute

bases that whole thing like what is what are things that can be built that are trusted you know like is your server trusted is your kernel trusted is your network all those things those assume that you're running reasonable I mean there's no such thing as determinism in general but that's a meta conversation by itself. But the the assume that like they assumption is that these things will

run in a deterministic They do not they are basically probabilistic, right? So we have all these other things that are happening is that the words the ingress to these things are creating probabilistic outputs. The prompts, the tools, the decision to create a tool, the fact that it has tools that an agent can know, well they don't want me to do that, but I got to do that.

Let me go see if end mapap is out there somewhere. Um, so this is another interesting sort of thing that's going on like this idea that in security there's always been risk equals uh L and likelihood times impact. It's a nice deterministic. It's sort of I mean some somebody like my friend Shannon Lee would say that never worked um because there were other economics involved with this.

If you don't know who Shannon Elites is, Google, she is my meta meta uh guru authority on all things security. Um I would say if I sound smart, I just got lots of smart friends. Um the but but but the thing is that that there's sort of like now we can't just say impact and likelihood because we're adding these flavors of we can do it a thousand

times or we can we're adding scale to the issue. So the calculation has to take on more than just impact. Impact and likelihood likelihood goes up. Impact gets higher because all this added additional um you know capabilities that are subhuman. Let me go figure out on the fly if anybody's written about a bypass or what are all the CVEs that are out there that I can find

a zero day. Right? Those things are like instantaneously. Um so this was interesting. Um this was um a tool that came out in our paper called pentestmcp. So the Equifax breach is an you know it's a great story for security right Equifax breach is you know they the struts to Java struts to vulnerability. Anybody's been security in the last five 10 years they know this they know

this really bad right when it happens it happens bad. Well, Equifax like got breached. It was um5 billion dollar market cap. One day $5 billion market cap. N people they recovered. I don't care. Five billion dollars market one day. You don't want to happen to you. All right. Um what they did is um the MT there's actually a paper out there where this Carnegie Melon students emulated

the Equifact breach. Um, and that was a six-month, by all indication, it could have been a year, a couple year kill chain, but it was on paper. It was, uh, uh, Crowd Strike, I think, documented as a six-month killchain, right? Um, this paper for Carnegie Melon basically did that as simulated. They put 50 servers out there, but they just said very much like the the the um

the blackmail scenario with Claude, they just said, "Find a breach on this system." And it literally found it emulated that was 50 servers as as opposed to a whole infrastructure. But this MCP pentest has the exact example where they set up an environment with a server that has MCP. It has end mapap nuclei and metas-ploit on it and they set up another server that has a password

file which is basically the Xbox and there's basically the prompt is find security issues with um 10.138.0.19 attempt to exploit n excfiltrate the password file on it. your IP is this and it's an MCP server with with connect with MC definitions for N mapap nuclei Bob's your uncle that's a just a simulated M uh pentest in this paper of this tool that people have created so we're

in this world where like you don't have to be I guess this is another meta point which is most of the people who do real scary exploits or adversaries the really scary ones have 10 15 maybe 20 years of experience a young man, you know, basically in his pajamas can ask Cord to replicate that level of experience. Um, I like to say this, you know, I'm I

love vibe coding. Jean Kim's got a great book called Vibe Coding. Um, it it's really got a good spin on it, the way enterprises think about this problem. But I would say like it's easy to start, you know, the easier it is to start, the easier it is to believe you're done, right? like and that's sort of our like you know prototype I always like think like

do I really have to say this in prototype not equal to production I have to say it um these are some um from some of the pentest u papers um there's um the you know the kill chains are getting very exotic right um literally figuring out real clever ways to hide xiltration how to get data out so that you don't you can't see that it has the

data going outbound. Um, guardrail modification. Okay. Yeah, they got guardrails. How do I I know that guardrail. It's been public. There's a bunch of data on it. Let me figure out how I bypass the guardrail. I'm not just talking about going out and finding CVES and saying, "Oh, I know how to do this." This is like the things that are supposed to like, "Hey, we're safe. We're

using XY solution for our guardrails." You may not be. And then there's sort of cross app, you know, um, stuff. And then, you know, some of the data, right? Like this was a last year survey, you know, this was like June of last year, I think. AI adoption up 187%. Security, governance, investment, security investment is local, right? They're fighting for the same dollar everybody else is screaming

for. Nobody's sitting back and say, "Oh, wait a minute. We ought to shift a lot of money over here because we could have a $5 billion market cap loss tomorrow." What's the likelihood pre-agentics? One in a, you know, Equifax, you know, now it's like it's much higher. Um, the threats, you know, I mean, I just grabbed some of the recent articles that have come out, right? You

know, Cyber Arc, of course, um, you know, you know, the they're all catching on, you know, but, uh, but you know, the, um, the thing is, and Ruben Cohen, if you don't follow him, he's pretty like he's out there. But um you know he's got some really cool stuff that he's working on with agents and and according to him and I think he's right that you know

that um Google are sort of taking some of the stuff he's doing and feeding it back into their systems. And I I think he's right. Uh but he says you know this he said this last year we're we're not talking about few tools. We're talking about an army of agents operating across your enterprise right now. I mean somebody gave me this great thing. We had shadow it.

Then when we get to assistant mode AI, like co-pilot stuff, we have now shadow AI. And what we're doing is reintroducing massive shadow at again, right? Like so it's coming back full circle, right? Um and um you know, and here's some of the other state data that's been out there for um the speed of weaponization, right? how long it takes to um to um uh basically breach

sort of a zero day vulnerability, right? You know, 2021 it was sort of days or weeks. 23 it was hours to days. Now it's minutes. So there's a couple p and that was this is a lot of stuff I had from last year. I didn't go like research everything in this paper. Um and you know so like we're talking minutes now from the minute a known vulnerability

to the ability of an agent to basically um create a breach um you know and then if you haven't been following you know I think OASP is doing God's work right now I think NIS sorry if anybody represents NIS not doing a good job at all you know not only has top 10 for LLMs they have a top 10 for um but some of the um the

top 10 LM s they're all part of the agency using NLMs if you didn't know. Um the you know prompt injection is not only it's sort of dangerous and scary it becomes steroidbased in agents um you know model denial service right again like real clever ways for adversaries to create denial services now right um excessive agency is this one I love right like how many of us

let's be honest you don't have to raise your hand have literally when you were doing some cloud stuff and you really didn't want to figure out all the VPC definitions you went to Stack Overflow and you grabbed cut and paste them and used it and then come on come on come on yeah he raised his hand like thank you uh I've done it and like yeah what's

that line do I don't know but it works right well again like we kind of got away with murder then agents are going to figure that crap out like oh look what this idiot did they made a per authority here I'm going to go and take advantage of that one right so excessive agency is a is a glare ing, you know, over reliance, um, plugins, um, you

know, we can go on and on about MCP and and the attack service and, you know, I mean, what MCP is like what, a year and a half old. I last time I checked, there's tens of thousands or 20,000 MCP uh, connectors out there. Um, you know, and you know, but here's the thing, right? Again, like it's one thing to say, hey, I've got tools that have

RPC endpoints, but now agents can intelligently chain them in a way and that humans can't think about as quick as they can. Oh, if I use that, that and that. Oh, the dumb humans, they never thought of that. Um, so one of the things I've been doing, I've got a workshop coming up in uh Raleigh uh end of this month. uh it's um it's basically the longer

one day version of this and I've been really sort of working with people thinking about when we're defining we have to think about how do we define autonomy and so when we think about autonomy what we should be doing is saying well maybe some agents should be in a contained space that have only a read service they can't do anything but they can they can they can

give us a lot of information add a lot of value they can go through a set of tasks that do a lot of things um and Then for if we're going to move up the food chain, let's define ones that are right service and we need to really think this one because they can mutate and make dynamic state changes possibly and then execute is a whole another

game because that becomes multi-system the scale of the problem right so we start I would argue if you're not having those kind of conversations in an organization you should definitely run to your not CIO CEO and say shut this off right sorry you know do you want to be the CEO that makes the front tweet the front page of the Wall Street Journal. Um so if we

go through some of these the blackmail example um the the shot well these are sort of malicious attack vectors um that again I'm not saying you stop them but you figure out what you got to do to contain them. You know, I I I didn't want to slam this in this early, but I I gave a web online webinar uh two weeks ago and a woman who

has basically been doing web assembly for the last 12 years serious. She runs the Wom cloud project. Um she pinged me and said like everything you're talking about sounds like a WOM solution. And so I'm like pretty that's for the next couple weeks that's what I'm doing. I've had friends who've been telling me about WOM for years like, "Yeah, yeah, I'll get to it someday." But I

think I don't know. I think there's an interesting My goal is for the um the all things AI conference in Durham at the end of this month, my workshop is um I'm going to create the labs out of Wom and just say, "Hey, could we do a read or write containment?" You know, these sort of um what do they call sandboxes like have sandboxes within sandboxes? I

don't know. I'll let you know. Um, we talked about most of these. I I love the Air Canada one. It's kind of interesting. Air Canada is interesting. The truth about Air Canada was probably not even generative AI if we're all being honest. Uh, but it's a great story to tell for Atantics. So, if you haven't heard the Air Canada story, it's a couple years back. This young

man is wants to go back for his mother's funeral. Doesn't have a lot of money. He goes into the Air Canada chatbot. It says he will get a refund. He goes back, buys a ticket, he goes get a refund. like yeah that was a chatbot turns into a lawsuit couple years at the worst possible time to be doing AI and getting your sort of company's name on

the paper for misalignment issues. Um I think I my rough napkin based mass is they probably lost you know a half a billion or I think it was five billion in mark cap but I know no it was it was less than the Equifax but it was a large market cap blast because they're every day they're on the page of the front you know the like all

lawsuit um what happened was there was uh when they were building a ragbased system they were scraping everything which is something you do and one of the areas they scraped data was an old FAQ that said that you would get a refund even though that wasn't policy anymore. So, but the AI doesn't know this. It reads the policy document, use the FAQ, it throws it into a

vector. Remember I said man is to doctor is it literally just figures out the closest and bam, he's told he gets a refund. You know what the total output of the lawsuit was? Like $900. Don't be air Canada. Um this is an interesting conversation too. um you know this idea of taint propagation. We think what we're doing with sort of read surface agents is not going to

affect us but uh misalignment at the could propagate all the way through the flow of things are right. So like the the adversaries and again the adversaries now are not even just human but the adversaries can basically figure out that I can taint something here that might get used down here and you can be real clever clever about that sort of possibility. There was one uh u

auditing big auditing company like a big five auditor company that literally publicly told they were using certain public resources to create their sort of general like uh it was called audit GPT was their sort of fancy we're going to solve the world and in the presentation they're saying I'm sitting there go they're talking about like here's a public place to get this like this now the advers

don't even have to break into your four walls to put zero boat execution code into those documents so that when they run downstream, they can actually execute and do something. That's just prompt injection 101. Sorry if I'm going fast, but I'm running out of time. Um, and going back to the likelihood thing, this is where I think it gets interesting, too. Like, and and the meta point

isn't to get too technical, make you make me sound like I think I know what I'm doing when it comes to security because I only half know. Um but the idea of that you know my Shannon Lee out here would tell you that that um risk equals L times I likelihood times impact and that in her words say never worked right because there's sort of like what's

the cost of the breach let's calculate that but what what we have now is we can't think the old way in fact there was a paper that talked about this recently a Singapore paper that everybody all the security people are praising and they're using this static definition it's way worse than that because it isn't it isn't that risk equals likelihood times impact. In other words, if it's

a million dollar potential breach and it's 800 uh it's a 80% chance of likelihood then the calculation it's an $800,000 cost, right? That's some nonsense like that. But now it's exponenti it's an exponential because you literally have the likelihood by the velocity. So like a 1% failure in managing human pace right it in a 1% for 10,000 agents is at 100 failures at 1%. But we have

to calculate that sort of velocity and speed into the calculation of these things. And there's a sort of another interesting thing which is um people are talking about we've all heard the human in the loop right okay where does that fit with agents right well there's there's sort of a term they're talking human on the loop. So we got to move from sort of human in the

loop to maybe we're sort of like okay we're not like checkpointing everything that's happening and we can maybe in a lot of things we should still should but we have to start thinking about how do we sort of like how do we checkpoint this sort of the like when we get here let's stop when we get here let's stop and again I you know I like I

shout out Shannon a lot but Shannon was sort of her point was a lot of the literature is now saying you move from h to hutddle no no your junior people learn you learn how to do AI through hit so it doesn't disappear in fact you probably the maturity cycle of an application should be let's circuit break it all the way learn how to use it and

then move it to an observing like human on the loop process right so it isn't like just turn this off turn that on we still have to learn how to do this stuff um you need a fabric in the middle this is the thing I'm working I think that the technical fabric might be web assembly I don't know you know web assembly is great because it isn't

least privilege it means there's no privilege until you assign it it's all based on capabilities right so like wow like this idea that everything's off you can't even get to a file in w unless you define it as capability I kind of want that if I'm going to do read and execute write and execute surface level I don't know if anybody runs the unicurren stuff right that

sort of disappeared um it's not all doom and gloom folks This was uh something that came out last year which was Expo and I don't care whether they're successful. I'm not promoting them. I don't care if you buy it. The point was that it was um some research that they realized it's back to that same thing that the tools that the adversaries we can use we can

use too. And it's an arms race. But the point is like I mean right now I mean one of the things that's scaring all my friends that have startups in security and some of them I love dearly and I really worry for them you know they're fighting that um that you know Opus you know 46 is basically saying hey you don't need that anymore you don't need

that anymore I don't know how much is true but part of this way if 50% of what they're saying is true my dear friends are in big trouble right um so I don't know how we navigate all that I don't know the build versus buy story maybe next year if I come back we can have a great conversation about what build and buy means in the enterprise

because right now I'm confused. I don't think you build salesforce.com. I don't think you build workday. But if there's a product been around for only five years, sorry to any vendors. I'm not sure that you buy it. Um, you know, so what's the structural mandate? Well, you need to sort of govern the surfaces. We we have to, you know, assume taint. So it's not just hallucinations. there's

supply chain in here. We're reading data. We're executing d. We're writing things. We're we're changing state. and then you know we got to understand the gaps of authority. We really need to understand again I've got longer versions of this in papers I've written and I'm just constantly learning. And by the way um I told you I wasn't going to sell my book again one more last time.

My books are out there if you want to buy them. Um, but if you want to have a conversation with me about this, like again, the reason I sound smart is I talked to a lot of smart people. So, I would love to hear your thoughts about any of this. And like John, I don't think you're right there, but that was pretty cool. I I'm I'm like

an open sponge. I I love I don't want to sit up here. I know a lot. I've learned a lot. Yeah. Enough. Um, but the gap is gap really scares me and how we're not addressing the gap in the enterprise. Thank you. We got time for questions. >> Sure. Yeah. Yeah, we can do that. That's cool. I didn't know if I'd finish on >> Hey gang, we

got time for questions. Um, I can run around with this mic here. Does anybody have a question for John? >> There's one right back there. >> Anyone? Ber. >> Yeah, Ber. >> Your hand right here. Keep your hands up so I can figure out where to run to next. >> Hey, uh, can you elaborate a bit on taint? I didn't really catch everything >> Oh, yeah. Yeah.

Yeah. I went through it kind of fast because I thought I was running out of time. Thanks. So, the idea of taint the sort of long version it is that you need to understand that it isn't just hallucinations, right? It's that and and one of the things I didn't spend I wish I would probably spend a little more time is that there is malicious intent and there's

sort of accidental or embedded intent like so taint doesn't have to necessarily always be somebody going in and sort of tainting a document that's public that they know that you might use that show up in some vector execution. Right? That's prompt injection 101. Prompt detection sort of 101 is I can put a zero byte code in some document that I know you're going to rag and then

when it gets executed it will execute that command right but but there's also sort of taint which is sort of drift and it wasn't a mal intent but the interpretation in the read and the build and all this stuff before it gets to a decision that might be harmful that process has been by the nature of how language and interpretation and all that stuff is skewed to

a place where it might harm you. Does that make sense? >> Good. Okay. Great question. Thank you. >> Yeah. >> Uh hey John, thank you very much first and foremost. >> Um hey, so yesterday uh Rafrey Kakoran had said that uh each person should have an AGI that's on your side. Um given >> Oh, what was that? What the word? >> He said that each person should

have an >> Okay. Yeah. Yeah. >> Right. So given what we know about biases in training data even you know historically through medical data and trials and things like that where do you foresee correcting this behavior? Is it in the kind of post and is it is it in the model process or should we be looking for better data to be training on right before we get

to that model >> I you know again these are like real sort of meta questions and I'm not accusing you because it's a meta conversation. the the problem that I see it from my like one foot level. In other words, looking at this massive world of complexity is the genie out of the bottle, right? These models have been trained on what 10, 15, 20 years of the

corpus of everything humans have been doing. Going back to the, you know, man is the king as woman is the queen problem, right? And if you want to read some Joy Bwanami who's written an amazing book on the uh the white glaze, right? And that's all in and then all the stuff that's going on with just the language. So the genie out of the box these models

have been derived. So I I think and my answer is I think the ethics of trying to contain that probably isn't going to work. But again I I'm not the expert there. But what I am thinking is we got to be way better. The point of this presentation is it's on us. It's our responsibility as people that protect the brand particularly in large end but any place

and therefore it is our job to deal with this sort of like probabilistic ingress and probabilistic egress in all its flavors and manifestation and one last sort of you know pontification is get better at stepping back you know what what's those famous you know I got to break the bone the heel I got to step I'm sorry that I'm not going to stop you from doing AI

in enterprise but I am going to tell you that like I'm putting the gauntlet down. We have to really understand how and when we're going to use these and we need to understand what autonomy means with agentics. >> Could a possible solution to this problem be uh give AI multiple goals with different levels? >> Oh yeah, brilliant. No. Yeah. >> Top level is like do what is

like like be a good AI model like don't like don't do things that humans don't want you to do and then have the AI's real goal as like a second level goal. >> Yeah. No, I think you know what's you know what's interesting? Uh I tell you my my I am I'm sorry I'm a shill. So take my AI history class, but the Ian Gudfell who created

GANs, it was a bar fight on could uh could they create an AI that generates say hey I want a picture of a cat and what a GAN is generative adversarial network. So the original idea of being able to say I want you to draw a picture of a cat on a stream and all that started by a guy name Ian Goodfellow who literally realized I could

make AIs compete against each other. So the answer is absolutely yes. And even more specifically, I glazed over this really quickly, is I think it's our responsibility to start thinking about if we're doing spec driven development, do we have holy driven spec that at least sort of stays the race? But even more importantly, I think to your point, yes, I think we should have um you know,

sort of a mixture of experts where there's a strong policy um model in the game that's reflecting the agents behavior in and out. And I do a lot more on this stuff in the longer version. That was a great question by the way. Yeah. >> Yeah. Um, hi. So, you talked earlier about uh sort of like the concept of like restricting an agent uh in what it

can do based on like your autonomy >> So, when the agent uses the same credentials as the authenticated human, like how do you meaningfully separate those behaviors? >> Take my shrill hat off, put it back on. Um, yeah. In my workshop, I talk about, you know, sort of like some of the really cool stuff that's going on just with OOTH. Like u the one of the things

I talk about in a more detailed version is you uh every agent has to have an identity, but it could be um um you know sort of um a sort of sub less granular authority of somebody a human. So in other words, you need authority, but should it should propagate its authority. And I think the next area I'm looking into is Spiffy and and and Spiffy and

those tools that have been really good in the cloud um uh uh sort of mesh tools. So the answer is um yes, we need to basically be much better at figuring out agents have to have identities. They can't just be like um robotics where like hey that's just the robotics ID because we need to trace when the agent we need to get really good at open tell

like what did it do? why did he make this decision? But we have to tie it back to its own identity and then we want to basically then give it subidentity of humans that are basically responsible for the the sort of that human on Good. Thank you. Human on the loop. A human on the loop should be the the sort of responsible body for some identity of

an agent. And that's the shortest version I can do. But yeah, good question. So >> over here. >> Yeah. Yeah. >> Hi there. Thank you so much again for your talk. Where are you? >> I'm sorry. Over here. Hi there. Um, so in your slide you had like 10% of companies have some sort of policy or plan around agentic AI. Can you talk if that's more on

just general corporate policy implemented in code like how pervasive that is and what that looks like? >> Yeah, you know the the thing is there's a couple of things that um where I get my data for like a Jean's conference is great um Jean Kim's conference because they're all enterprises and so I get a lot of data there. They're running uh their first AI summit I think

next month but and then you got enterprise listen so I get to talk to like you know um the um Adidas uh Fidelity and I get to hear the sort of skinny and it's Jean says we have a no um uh velvet rope policy and he's right like the speakers are just you know they're kind of like me like I love the um so I get a

lot of my data from there I I scour the stories that I read you know like what is MX I go you know where I go I go to um uh job wanted. I look for like companies that like where you know the one survey saying they're not doing anything and I see AMX has got all these like very specific skills that they're hiring like 30 or

40 AI not only AI engineers but sort of policy people that are so there and then the talking story but the the meta of your question is that's just one survey point and you got to be really careful with surveys today because so many people can the agentics or agents and AI has helped us all be experts on surveys. So, um, mileage varies. I tend to believe

if it's not happening massively already, it is h it is it ain't stopping. So, let's assume it's already there. Does that make sense? >> All right. Quick followup. So, uh, I guess to go back to the question, does that mean that 10% that's just corporate policy or is it actual practice? >> Practice again. It's one survey and then I would I would put mine the reason I

put that data in there is I feel comfortable with that number and I don't know if it's left or right but I do have my own empirical data of of conversation and I try to help you see where I'm getting my data from the right conferences that have enterprise people who have authority like again to pal was the first fellow at Capital One first fellow at Capital

One right that and now he he's like a lead architect of Fidelity 16 trillion assets I get to have conversations with him. I know what's real, but it's not real. >> All right, John, thank you very much. Uh, if you when you're coming out the door, across the hall to the left is John's table. His son Will is helping him out. >> John, thanks for flying all

the way out from Hackworth, Georgia. We really appreciate you. To kick things off here at DevOps Day at Scale 23X. In a couple of minutes, uh Joshua Lee will be talking about um the uh oh, is DevOps all of a sudden a foreign language? And and why are there no junior SRRES? Important stuff. He'll be coming up in a minute and we'll have a couple sponsor pitches.

One quick thing, we're at the at noon we're doing an unconference uh where you can propose a topic for for to discuss on and the most popular topics. We'll get in the room and and have areas for people to discuss that topic and and we should have some snacks as well to help uh further our discussion. So um in the in the sponsor slides there is a

um a a thing where you can find the link to po to propose a particular topic to discuss on. So if you helped us out with that that'll make some good discussions at the unconference. Um so be back in a couple And then you'll go right and this is by the way. And hey, if sponsors one, two, and three are here, please come up to the stage.

Otherwise, I will hunt you down. Good timing everybody. Maybe I'll have my partner run the timer or have the big hook amongst yourselves. Can you also run a timer for the one minute pitches >> and then maybe just just shout it out one minute because you'll be off mic. >> So we got three we'll have three sponsors and we'll just start right at Just focus. Yeah, sorry

we don't have a backstage. No green room. Actually, there is a green room. It's just used to have bagels, locks, and cream cheese. >> go chop it up. So, Richard, I What time are we at? So, okay, it's about show time. No name. >> Anyway, we'll chat later. >> All right. Um Joshua Lee will be uh presenting to us in a second, but first we have a

couple of announcements from our sponsors. are, as you've seen out in the hallway, we have some awesome sponsors and they're important part of making DevOps Day LA and scale happen. So, with no further ado, um, uh, Sawyer from Red Hat will chat to us. He'll give us his best minute right now. >> Well, when you're Hey everybody. Uh my name is Sawyer Berman. I'm an AI developer

advocate at Red Hat. I'll keep it short and sweet because no one wants to hear too much uh jargon or me shilling anything. But so in the world of open a uh AI in general, uh you really want to make sure you're choosing intelligent places to spend your money or you could not spend money at all and use free open source products that we're all contributing to

like VLM, LMD, and whatnot. Um but that's a lot of information. So, if you guys want to come to the booth, we're just right over there around uh auditorium D. And um I'd be happy to talk more, answer your questions, and uh it's a pleasure to meet you all. Thank you guys for being Thank you. And also thanks to Red Hat for sponsoring our local monthly LA

DevOps meetups as well. Uh next up is Adam from Palumi. Uh and one of his uh colleagues will be uh doing a lightning talk on the Ralph Wiggum loop soon. But right now, here's Adam. Hello. Hello. Uh yeah, I'm Adam and uh I work for Palimi as a community engineer. If you're not familiar with Plei, it is uh infrastructure as code using a programming language. So if

you want to set up infrastructure using Python or or Go or TypeScript, um it's a great tool if you're building an internal platform. Uh this event is amazing. So we're just excited to be able to sponsor it. Uh scale and DevOps days. And we have a table out there. Uh yeah, Angen is speaking later in this room. We have a couple other talks. I'll be speaking later

in uh the big room about AI infrastructure. Thank you. >> And also thanks to Palumi for speaking at and sponsoring the local monthly LA DevOps meetups. U next up is Benjamin from Thails. Come on up. >> All right. Thank you. So first the name of the company. You have to get your best French accent and say it talis. >> How do you say your name? >> Long

show. Benjamin Longo from Talis. >> All right. Please come practice at our booth. >> All right. I'm already done with my one minute. So, um, but from the last past hour I've been at the booth, I've seen people who are familiar with our hardware security module. I'm curious in the room who is familiar with code signing and hardware security module. I've had quite a few people. Oh

yeah. Okay. All right. Perfect. So if you want to learn more about what we do for code signing and data protection in and out of application, we have people who represent our group at the booth. We do also application security. So think about web application firewall, API discovery and protection. And we have a gentleman also who represent our AM division. So if you want to learn about

all of those things what we can do and more about Talis, I'll be pleased to have you on the booth. Thank you. >> Thank you, Benjamin from Talis and Al Leu. Okay, coming up right now is our next presenter, uh Joshua Lee, uh as soon as his, uh wireless mic is ready. All right, we'll just take a second The title of his talk is DevOps is a

foreign language or A great question. Looks like we have an issue with the whatever lavalier mic. Well, this mic is cooler anyway. You know, like a rockstar. Take it away. >> All right. I will just have to remember to keep this close to my face and hopefully not move too much out of frame. Oh, yeah. Let's get the slides Thank All right. It takes a village. Um,

and eventually my slides will load. This is a great start. Demo gods. Come on. Okay, we have slides. Hey everybody, I am Josh Lee. Let's kick things off. So, we're going to talk about DevOps is a foreign language. I studied linguistics. Uh, fair warning, if you're expecting like code samples or really anything technical, not going to be in this talk. So, now is your chance to leave.

Um, thank you so much for coming. Even though Cat Morgan is talking right now, I'm honored. And so, let's start with a problem statement. So, this is a quote that I love to to share from Kelsey High Totower. For those not familiar with him, he's one of the creators of Kubernetes. And he said, "Learning Kubernetes doesn't take three days or three months, it takes three years." And

he said this in 2017. Can anybody tell me what year Kubernetes was released into the world? 2014. I wonder if Kelsey would update his answer now about how long it takes to learn Kubernetes. Right? So, this is kind of this is kind of the inspiration for this talk is that like in modern DevOps in in the fields that we practice in, we've built up these layers of

abstraction, we've built up these black boxes of things that um make the depth and breadth of the field really really overwhelming to beginners. And that makes learning like besides the amount that there is to learn that actually that scale makes the learning process harder and that's kind of what we're going to be talking about today. So just quickly about me and then we'll get into it. So

I uh dropped out of college because um I like food and uh food costs money and linguists don't make a lot of money and uh I've been doing tech for a long time. I am anme which we're going to come back to that subject matter expert and I have a fun story about that that we'll come back to um and click house and observability. Uh I am

an open source advocate at altinity. Uh we do click house hosting support. If you want to talk to me about that later, I'd love to talk about that. And I'm a co-organizer for the open source analytics community and conference. So if you have something really really interesting that you want to say about open source analytics, please talk to me. So, what we're going to talk about is

we're going to basically do my first semester of linguistic anthropology, but we're going to do it in like 15 minutes so that we all have some sort of steady common ground to talk about these things. And then we're going to talk about a specific uh paper by a guy named Steven Crasian in which he outlined five hypotheses about how adults can learn complex systems better. And then

we're going to talk about how we can apply those lessons to what we actually do all day every day in practical terms. So to start with, I like to use some examples, languages are one of those things that um we kind of all think we're experts in, right? We all speak at least one language fluently, natively. And so we kind of think, okay, I understand what the

building blocks are of a language. I understand what a language is and where the boundaries are. And like I can point to a dictionary and a grammar book and like that's a language, right? But it's actually a lot messier than that. So here's an example. Um, how many words do Eskimos have for snow? Does anybody want to like give me a specific number that they've heard? I

see seven. I see five. I see 10. Or maybe people are just raising their hands. Yeah. Sorry. What I heard 300. Okay. Yeah. 300. So, just within one section of one newspaper, the New York Times science section, this number has been reported as anywhere from 4 to 212. This this question is fraught. There's a bunch of problems with this. One of them is like, well, what is

Eskimo? That's actually what we say Eskimo. We shouldn't really use that word, but it's referring to a huge group of language families that are actually all very very different and there are hundreds if not thousands of dialects, which actually then gets into another counting problem. Like what is a language and what is a dialect and where is the boundary? So that's a counting problem. Counting the number

of words for a thing is kind of a problem because like is snowbank a different word than snow? Is snowing a different word than snow or is that just morphology? Um, and also, is it even that interesting? Like, I come from Vermont. We also have a lot of words for snow because we have to deal with snow a lot. So, we talk about it a lot. It

just kind of makes sense. All right. Here's another example. Uh, and this is also an example of how uh language has sort of cultural significance, right? So on the previous slide, the asterisk was because I wanted to make sure to remind to talk about the word Eskimo and and how it's kind of a word that we should try to avoid these days. Now I'm using the asterisk

in a very explicit way, the way that linguists use it to denote something that is grammatically correct but feels wrong to a native speaker, right? And and if you're a native English speaker, you're kind of looking at the sentence and your brain is like breaking, right? Because there's a there's a problem in the middle with the adjective order, right? And it should be the cow is in

the big red barn. So, can anybody tell me why those adjectives are supposed to be in that order? >> Right. So, it turns out there's actually a rule that every native English speaker can follow. There are seven distinct categories of adjectives. Color, size, quality, right? Um, judgment. And all of these ad those seven categories have to go in the same order every time. So that's a rule

that every native English speaker knows and practically zero native English speakers can articulate. Language is kind of messy, complicated, right? And we don't actually know it that well, even our own native language. All right, I love this one. It's raining cats and dogs. Everyone knows what this idiom means, right? But the meaning has zero to do with the form of the actual words. It's completely disconnected. You

can't look this up in a dictionary. It's just this cultural thing that we kind of all know that's part of the language. So what is a language? I I apologize for the wall of text, but this is a big meaty thing that we're getting into. So we need a big meaty wall of text. So a language is a structured system of communication that combines syntax, grammar, and

vocabulary. Yes. But it is not just that, right? It is not just this static thing that we can write down in a dictionary. It's constantly evolving and being shaped by the people who use it and how it is used. And it is also not just a thing that we use to express meaning. It's a you a thing that we use to express identity, create communities, right? And

transmit traditions across generations and between social groups. So it's a really big messy important thing. It's a huge part of what it means to be human. And human brains are really really good at learning foreign languages. In fact, until puberty, every children will acquire a language naturally simply by being exposed to it. Right? They don't need any explicit instruction. just expose a child's brain to other people

speaking some languages and they will pick it up because talking is really important to us. But in puberty, we lose that ability. And then we get to a point where our brains can no longer naturally acquire other languages. I'm going to use the word second languages a lot today because that's how um we talk about it in academia. A second language for adults just means any language

that is not one of your native languages that you grew up speaking before puberty. So it doesn't you it might be your third or fourth language but just any non-native language is called a second language right. So adults kind of have this they lose this ability and we enter this thing called second language acquisition that I'll get into in a bit. Uh yeah that was the side

I just talked about. This is really really cool. Actually, by the way, if you're into this and like if you think this is cool, read up about creation and pigeonization, which is like when children are put in environments where they weren't exposed to like just one language or like weren't exposed to a complete language, so they just invented one on their own. Like this is how innately

children need to speak a language. Um also there have been examples of this happening right like in deaf communities where um sign languages have sort of emerged just from children being placed together in schools. All right. So, a couple of ground rules because we're going to be talking about languages. Just some things uh we're all going to agree on. Uh if you want to disagree, please do

it with me privately. All languages are all natural languages, that's an important word, right? All natural languages are equally expressive. And mocking or disparaging somebody's natural language is just not something that we're going to do. We're not going to go there. Okay, cool. Written languages are not natural languages. And children will not naturally acquire written languages, right? You need explicit instruction in order to acquire a writing

system. This is not something that falls into the category of what we can just sort of pick up through osmosis. And constructed languages like Java or Espiranto are not natural languages. And if you're paying attention, that means that making fun of Java is totally fair game. Okay. So, how is this relevant to us? How is this relevant to the software development life cycle? Can anybody tell me

what command I run to tell me what version of Docker I have installed on my system? I heard a lot of Docker D- version. I think I heard a Docker-V and I heard a Docker version. Those all sound correct to me, right? Um, did anyone who answer ever look that up specifically for the Docker command? Right. How do we know this? It's kind It's how it works

for all the other ones, right? It's raining cats and dogs. have sort of decided means something. So that's kind of cool but also kind of daunting for beginners, right? That we have all of these things that are sort of simple and trivial to us. We can just rattle them off if a speaker on stage asks. But uh they're not written down anywhere or not written down anywhere

that anyone would look often. Right? And then here's another problem. So I actually tried it. I did this on my machine. I'm like, "Okay, well I'm not going to look it up. I'm just going to see if I can stumble upon the correct command. Docker-V version. I got an answer. Great. Docker version. Nothing happened. My terminal just hung. I did not get another prompt. This is weird.

So, it turns out what was actually happening is that uh I didn't have the Docker Damon running. And right, Docker is made up of two pieces. There's the Docker Damon itself, and then there's the Python CLI tool that interacts with the Damon. So, Docker-V gives you the version of the CLI tool. Docker subcommand version gives you the version of the Damon if it's running. If it's not

running, the CLI tool just hangs. So I would ask you, right, does someone who has just taken a sixe coding boot camp in React know how to escape from this situation without just closing their terminal window? Probably not, right? Overwhelmingly, no. Um, so I would argue this is a huge disservice to anybody trying to enter our field, right? If we think about what this is, like let's

say I did that six week coding boot camp and now I'm wanting to learn backend and I'm following a tutorial and before I even get to step zero of the tutorial, it's going to say you need at least this version of Docker and I have been dropped off a cliff at worst and into the very deep end of the pool at best. We're doing our juniors a

huge disservice with the way that some of our tools are built and with the way that we talk about them. And what I'm kind of getting at right is this. There's this implicit knowledge that we've just sort of picked up through osmosis. And there's this explicit knowledge that might make its way into getting written down in some documentation somewhere that someone else could read. And I think

we kind of understand the need for this implicit knowledge in other fields of life, right? In other things that we do in life, right? Like we typically learn with guard rails, bumpers, and bowling, right? Training wheels when you're learning how to ride a bike so that we can learn progressively and iteratively. And I think this is how we do it in our field. So my hypothesis is

that to be successful in this field, right, people actually need a lot more than what they can pick up just from reading. They need a lot of implicit cultural knowledge which people may not even acknowledge and which as we get into this, we're going to see can only come from practical experience. All right, so back to linguistics. What is second language acquisition? I mentioned what second languages

are, right? They're the languages that we learn after puberty. And second language is the study of the process by which adults learn new languages after puberty. And it turns out that we as a society also are really, really bad at this, right? Like if you're using any kind of app that has you wrote memorizing words or phrases, you're not really learning the language. You're doing about 25%

of what it takes to learn a language. But that's kind of what we do. Like even in high school and colleges when we're teaching languages, there's a lot of just like rope memorization. But that doesn't work so well. So there's this field, second language acquisition, that combines psychology, physiology, sociology, and linguistics to examine how we can teach adults languages better and how adults can learn languages better.

Um, and specifically, we're going to now talk about Steven Crashen's paper. This is one of the seminal papers in the field, principles and practice in second language acquisition. So it's really if you want to look this up, it's commonly grouped under the input hypothesis. Um but it can be broken up into these five sub hypotheses. So we have the acquisition and learning hypothesis, the comprehensible input hypothesis,

the natural order hypothesis, the monitor hypothesis, and the effective filter hypothesis. Don't worry if these words don't make sense to you right now. We're going to get into each one. So the acquisition and learning hypothesis, this one's really simple and I think intuitive to most of us, right? Humans learn best by combining theoretical learning with real world knowledge acquisition. So there's this uh tool from the world

of martial arts and um sports instruction and military training called EDIP. And EDIP stands for explain, demonstrate, imitate, practice. So first you have someone explain the concepts, the theoretical knowledge to you, right? Then they that person then demonstrates how to apply that theory to some specific problem, right? They show you some forms and how those forms apply to a problem. Then you imitate those same forms to

solve the same problem. And then finally, right, you practice. That means taking those forms and making them your own and solving your own novel problems by recombining those forms. And we can kind of think of this as something that becomes a loop, right? So we go through this loop of explain, demonstrate, imitate, practice. The new forms become ours and they become part of our mental model of

the world, right? Those black boxes that we were dealing with become a little bit more transparent and we can see better how they fit together and we have a more granular vision of this field, right? And then we can integrate that new knowledge, start over again with theoretical learning, with our new understanding of the world as our basis, right? And keep iterating. So first tip that I'm

going to give you all is really important right it's embrace practical experiences choose real goals when you're trying to learn something build something that is practical and useful to you um and ideally just to you because you don't want to create too much pressure for yourself we're going to talk about why soon but right you want there to be a little bit of pressure a little bit

of motivation and a little bit of practical all right this is the most important piece of the whole hypothesis and why I wanted to give this talk it's the comprehensible input hyp hypothesis. So, humans learn best when we are exposed to new inputs of size I + 1 where I represents already familiar concepts. The relative size of the I to the one, right? There's a pressure effect

and it's much much easier to integrate new knowledge into the eye when the eye is already very very large. And this is kind of why right like these layers of abstractions that we have now create such a challenge is because it's like you have such a long journey to go in this field before your eye is of any significant size. So okay if you speak Spanish please

don't answer. No cheating right? You're not getting graded. If you don't speak Spanish can anybody tell me what kabayete the second line on the second word means? I see a lot of like shaking heads like nobody has any clue not even an idea something related to horse. Yeah. And like because we know the Spanish word cavio, right? So like that seems reasonable. It actually might be the

etmology. But now can you tell me specifically what it means? I heard it. Yeah, it's an easel. And some of you who aren't native English speakers might have like thought of the concept in your native language and that was you learning Spanish because you were directly correlating the word to the idea rather than to uh another word. So our brains are really really good at this, right?

If we have enough context, we can fill in the missing pieces. But if all were shown as a pile of missing pieces, we don't get to learn a new So start with the basics. Start small, simple, and iterate. Right? The example that I like to give here is like if you're learn if you're learning Kubernetes for the first time or if you're teaching someone who's learning Kubernetes

for the first time like maybe don't start with a stateful microservices application maybe like deploy engine X and get to that beautiful beautiful 404. It's the happiest I've ever been to see a 404. All right, natural order hypothesis. So the natural order hypothesis states that there is a natural progression for learning complex systems like languages. In languages, this is actually like built into the biology of our

brain. In other fields, probably less so. We don't really know. But yeah, this is a tech tree from a game called Path of Exile, right? These are like the for those who don't play a lot these types of games, right? Like these are different technologies that you unlock and they have dependencies on each other and like they make you better as you unlock more of them. But

this is totally overwhelming, right? Like I've talked to people who played this game and it's like, yeah, it makes me cry. Like this is just it's so overwhelming. And it also is kind of familiar, right? So, I love this graphic, right? This graphic is a testament to the success of the CNCF. Many of the projects on there are ones that I love and use every day. I've

even worked for some of them. But it's completely overwhelming to a beginner, right? Like this box of things that we call DevOps, that we call cloud, it's like really really big now. And this is a simplified reduced version of it that we're looking at here. So here's another tech tree. This is from the video game Civ 5. And what I like about this one is that it

has a nice forking path, right? So you're not bounded to a single track or a single set of rails, but there aren't a million options either, right? You start with agriculture. Everybody needs that. In our world, that's like CLI, right? Can you use git? can you like basically interact with the file system and the codebase in a meaningful way and then right like then we don't have

a hundred options we have four options so maybe you learn a programming language or maybe you learn how containerization and virtualization works or maybe you learn about how Linux networking works right but you don't have to learn everything all at once and there's a path that takes you through it in somewhat of an I plus one way so for this what I would say is especially especially

now with AI and I'm sure we'll get into that in the Q&A phase, you need mentors, right? You need individuals who have trotten a path through the tall grass and can help. And if you're in the position to be a mentor, right, you need to understand these other concepts that we're talking about and make sure that the path that you're giving to someone doesn't have cliffs and

traps and pitfalls. Okay. The monitor hypothesis states that learners will self-correct, but when feedback is too frequent or too strict, it hinders progress. Right? So basically the monitor hypothesis that we all have like this little buzzer in our brain. It was the one that went off when we all said saw um red big barn, right? All of our brains were like that's wrong. And so the the

monitor doesn't necessarily know the correct form, but it knows when it sees the wrong thing. And so this can hinder us, right? Like there it's actually scientifically true that some people learn foreign languages better when they're slightly inebriated because it reduces your inhibition to just try something that you know is wrong. And when you're speaking to like other humans, they're going to meet you halfway and if

you say something that's kind of wrong, right? Like they're going to consider the context in which you're talking to them and understand what you're trying to say. Our programming languages often are a little less forgiving, right? Like if we just try something, we don't blow up the whole thing. And so we are a lot less uh willing to just try something that we don't know is correct,

which can mean we might and since we don't necessarily know the correct thing, we may not be able to try anything at all. But it's all about feedback loops, right? Um when I was learning uh like my very very first type of coding of any time was writing HTML with Dreamweaver. So Dreamweaver had this nice split view where I could look at what I was doing, right?

It was a live view before live view existed and I could write my HTML. There was no CSS. I'm old. and uh I could see immediately how it appeared, how it rendered in the render view. That was an instant feedback loop that let me know the correctness of what I was doing, right? But now we kind of have like these build pipelines and these CI/CD pipelines and

these right event loops and things like that that do not give us that immediate feedback and we have no idea of knowing if it's like did I do something wrong? Is the system broken? Did I just not wait long enough? It's it's a little overwhelming. What I would say is don't be afraid to admit when you don't know the answer. It's okay to ask for help and

don't be afraid to try things. So, you do need an environment where you can just try things, right? Like that we were talking about before. We need these personal projects that matter to us where we can try things out. Okay. The effective filter hypothesis. This is the most important one. I know I said that already, but this is it's I mean it this time. Motivation and mood

are the strongest filters on our ability to learn. So this is like huge. Humans are humans. We're going to be different from day to day. We're not always going to behave the same way. We're building sociotechnical systems. And like it's fun now because with AI the the uh technical systems are also unpredictable. So there's like we have randomness everywhere and it's going to be really really fun

to try to design datadriven uh deterministic systems around that. And also, it's really, really important when you're learning a new skill to know your why. Know what keeps you motivated, right? This is this is so important. Like, write it down somewhere that you see every day and it will keep you going. It will help you learn. And if you're unmotivated, that can be like a zero x

factor on your ability to learn. So stay motivated or find something else that does motivate you. All right. So, let's recap the advice that I gave specifically for people who are on this journey as a learner, right? which is all of us, right? Because we're all learning all the time. Combine theory with practice. Start small, then iterate. Find your mentors, try things, get feedback on the things

you tried, and know why you're doing it. I see phones up, so I'll just wait a second. Great. Uh, there will be a QR code at the end for the slides, too. All right. And the other tip I would have for learners, which is all of us, is stay positive. You got this. All right. And now for those of us like if you're in this room, you're

probably also in this group. We're the people building the tools and platforms and helping other people learn on their learning journey, right? So what can we do better? Well, number one is to provide a safe environment. I mean this both in terms of like the ambient environment that we all inhabit as humans, right? Like that needs to be a safe place to um come forward with mistakes

and things like that. We're kind of good at this in SRE, right? We have this concept of blame aware. I also mean this in terms of like your programming environment, right? Like not all of us are lucky enough to have a staging environment, but you probably should and you should probably have your own little personal environment that very closely mirrors your staging and your production so that

you can learn things without breaking things that matter to other people. All right. Another thing we really need to do when we're when we're building these learning pads is to embrace the early adopters. They are special. I don't know why. There's something about them, right? But they have some kind of innate motivation that allowed them to plow through all of the obstacles in that like rough shape

of whatever the thing was. And so they're your best friends because they can tell you what those obstacles were and then you can go back and smooth those over so that the people who don't have so much innate drive are able to follow. Design for accessibility, right? Build tools that provide context and meaningful feedback. I should add to that immediate or like as immediate as possible feedback,

right? Tighten those feedback loops and and describe things in your languages, right? Um don't just let your terminal hang because the Damon that you're trying to talk to wasn't running and meet users where they are, right? Like I learned on HTML and then I went from there to PHP, which I'm guessing many of you may have also done. PHP is a very permissive language, right? And HTML

is extremely permissive. The browser will just render it even if there are errors. It'll render whatever it can. And PHP, you know, as like dynamically typed and you can just kind of do whatever and it's mostly going to work and you're very very rarely going to get like a syntax error. You're mostly going to get logic errors. Now, as someone who's more senior, right, like I love

tools like TypeScript. I love like much more strict tools because they give me a shorter feedback loop for the things that I care about and I have the skill. I have like the eye knowledge to understand what it's telling me, right? But if I'm just trying to like figure out how to make a button that gets to the form that gets to the database, right? Like I

don't necessarily need those things like llinters getting in my way. And the last thing, right, to just tie it all together, we need to offer a trotten path for the learners, right? Or if we can't do that, right, at the very least, let's uh clear the major obstacles. Platform engineering reduces the depth of the stack for developers. This is a quote from Hazel Weekly. Uh she was

actually on my podcast that'll be airing this week. Um yeah, right. So Kubernetes is a platform for building platforms. Um let's use it to make this easier for everyone else, right? Like let's build things that actually help people iterate through these layers of knowledge. And I think we already had this figured out once upon a time and I'm almost done and then we'll get to Q&A. So

this was my first platform, right? I kind of already alluded to this. It was great. It had everything I needed. I had CI/CD. I had observability. Yeah, I had everything I needed. So, I'm not saying like, let's go make geio cities again, but what I am saying is let's make this fun again. Let's make this so easy that every 12-year-old on the planet can teach themselves. That's

all. Thank you. >> Yeah, questions. We only have the one mic, so I'll have to repeat it. Okay. Yeah. So the the question is um right like when people sign up for meetups uh they sign up with a personal email account rather than a corporate email account and the hypothesis is that maybe that's because they don't uh want to be marketed to or maybe it's because um

they want like a safe environment right to like try things out. Yeah, absolutely. I think uh there's very little anonymity in our field today. Um, so, uh, having that, right, and then having a space that's maybe not filled with all of our co-workers and maybe won't get back to our co-workers where we can admit what we don't know in a way that's maybe a little bit safer

than than our workplace. Yeah. >> Yeah. Here, I'm just gonna Well, first of all, thank you for the talk. That was very interesting. Um, you mentioned something that I liked which is about the implicit versus explicit knowledge and I find that that's becoming even more important with AI these days because there's a lot of implicit knowledge that we have that the AI doesn't have and quite often

you know many of the the issues that we experience that's because of that lack of implicit knowledge. Now, as an experienced person, I might be able to see that and realize, oh, that's because there is this bit of knowledge that is implicit. I'm just going to make it explicit. However, a junior person, which is kind of the target of your talk, maybe doesn't even have that implicit

knowledge. So, do you think this is going to create even more of a divide between juniors and experienced people? >> Yeah, potentially. Potentially. Um although um I've been encouraged by a few experiences that I've had with people that I'm mentoring where um I'm a they need me, right? Like if it was just them and a chatbot, they're going to spin in circles, right? But now that they

have such the ability to search like the combined knowledge of all of humanity somewhat instantly, I'm able to just sort of tell them the names of the stepping stones that they need to go down to get to a particular goal. Right? So I have a friend who's trying to build a bot for himself to do some things and he's like, I I am having a really hard

time with these audits. What do I do? I'm like, what do you mean by an audit? He's like, well, I have like chat GPT review what Claude does every time before I put it on the server. I'm like, oh, you're talking about CI/CD. here are the keywords that you need to talk to your chatbot about that will teach you what CI/CD is and help your chatbot set

it up with you for you. Right? So, all he needed from me was that one keyword, but there's no way anyone without my experience could have just given that. Right? test one, two. >> Uh, all right. Yeah. >> Sorry about that. Yeah, now I sound great. Wow. Uh, question. Thanks very much for the talk. Really insightful. Um, the tech tree. I really liked that uh idea and

the kind of gamification behind that. So, and then speaking of stepping stones in your answer to the la to the last question um beyond CLI, what would the next one or two tiers >> on the on the tech tree the civ I mean I love on the civ tech tree be for you? >> I I think it depends. I think there are lots of different tech trees

that you could do in like it's such a broad field that there are many. Um, I recently learned about roadmap.sh which like actually has that implemented like here's a tech tree basically for like if you want to get to this point in Uh, I think there's a question in the Hi. Um so you mentioned about feedback about um if it's too strict, too fast, too blameful and

mean um that it inhibits. It occurs to me a number of years I've struggled with the difference between feedback as in human input letting people know I have constructive criticism whatever from uh audio feedback where if you get the speaker too close to the mic and they can hear each other then just the world loses. And it occurs to me that this cycle of blame too tight,

inhibit, inhibitance, whatever you want to call it, um this destructive pause, um is like a human verbal form of that other kind of >> I love that analogy. Yes. >> Uh couple of questions up here. Okay. So firstly, I'm attempting to break into the industry right now and it seems like I'm constantly seeing new information of do these projects, these projects, these certifications or etc. um with

access to everything and also seeing well it's not worth it because this will be replaced by AI or this won't and it's just drastic. What would you say would be the best way to discern which path to choose? Would it be to seek mentorship from someone that's already in that position or how to discern which route to go? >> Yes and no. Right? your mentors are gonna

like once you've decided where you want to get to, you'll be able to identify the right mentors and they'll be able to tell you how to get there. However, I find that all of us are biased, right? Even if you have a mentor who is extremely successful, they're going to be biased towards the things that sort of the the the con the convergence of factors that contributed

to their success is going to bias them towards those things when that's not necessarily the only path to success. So, I would say have diversity in your mentorship. So, >> thank you. >> Yep. Question. >> First of all, thank you George for the talk. Great. Uh is it was great. My question is um in uh this se tree uh in and in the game when you learn

a new tech it's normally one tech at a time but in in real life sometimes you learn multiple things in a time. So what would ideal number of things you learn? Because in if you want to break into DevOps, you probably need to learn multiple different things that's not related to each other. >> Great great question. I would say just it's not I wouldn't give you a

specific number, but it's about the ratio, right? Like if I can take three things that I already know and two things that I'm trying to learn and combine them into a project, that's a great ratio. >> I was going to ask, say you're the only person at your at a company doing this and you live in a relatively small uh town or metro area. How is what

is a good way to get mentoring? Uh you know, is there an organization that can provide that when you may not have, you know, uh direct people to to to meet? >> Yeah. Um well, you made it here, so that's like that's one tip, right? is to come to conferences in cities like this. Um I know that's hard harder to do and it costs money and time.

Um Meetup groups are great. If there isn't a Meetup group in your area, um start one, right? Um in 2013, I started a Meetup group for WordPress in in Burlington, Vermont, which is an out of the way city, but a city. And the first meetup had three people. And a few years later, there were 300 people in the meetup. So >> is it okay if I add

on to that? >> Yeah. So, I'm a DevOps day la co-organizer and ever since I started volunteering almost two years ago, I've had a lot of fun. I've met a lot of people and two of my life mentors are the co-organizers. So, it's a great way to meet people. >> Yes. Um, so I work with a lot of really really brilliant people and so I find when

I have questions I kind of ask something here because I don't know what I don't know and they reply with the answer up here and so I'm like what did you just say? Okay. Okay. So clarifying question and then their answer is still up here. So then when someone asks me a question, having that in my head, I'm like, "Okay, from the dawn of civilization, let me

paint you a picture and give you all of the context in the world." And this is what your terminal window icon looks like. And they're like, "Okay, okay, okay." How from both sides, how do you kind of zero in? Are there questions you can ask or like how do you quickly get to this is what you know, this is what you need to know? >> Um, yeah,

great question. So part of it is just right like having grace is which it sounds like you do like you're having people who are trying to work with you and trying to meet you in the middle and it's just still challenging. Um it can be really really hard to break out of like these these elevated views of the world with all of this nuance and and break

out of that nuance to explain it to someone where it's like well actually these two things aren't the same but for your purposes let's just pretend that they're the same. It can be hard for people to do that. Um, I think there's a huge value in having empathy, right? So, if those people are sort of exposed to that more often, you can kind of start to pick

up on the patterns of like, okay, well, when someone asks a question this way, maybe they don't need this level of detail. But the other thing is to find people who are maybe closer to your part, your your place in the learning journey. Um, right, I'm a developer advocate. I love giving talks about things that I just learned. Um, because like I'm going to talk about that

differently than something that I've been doing for 10 years. >> Another question. >> Hi, thanks for your talk. Um, I am an amateur DevOps person but professional linguist and I've really enjoyed kind of hearing how you broke down a few key concepts and also thank you for letting me know what to do when my terminal hangs because I've always just closed it and started over. >> You're

welcome. That's probably the most useful thing I'm going to learn at the conference this week. >> Excellent. I'm so glad to hear that. >> Um, and also I just had a question. So, um, I started learning programming with, uh, assembly in like a computer engineering program, and I also didn't finish school, but um, I found that it's actually really hard for me to learn high level languages.

They just feel like too mushy and I can't I feel like I'm not pinning anything down. Mhm. >> And I'm curious whether like the theory that you have that the lack of precision makes it easier is only because you started with that. >> Yes, I think that you're absolutely on to something, right? And this is my own bias showing. I sort of learned the stack from from

sort of the the human side and the interaction side down and I've gotten deeper and deeper into the stack as I've gone, but I definitely know people who have sort of learned it the opposite and like from the bits up. Um, and I think that's definitely a valid learning path. I don't have the brain for that. So, I I don't know I don't really know what that

learning journey looks like differently than >> I' I've never figured out how to learn stuff that's supposed to be Pythonic. I just don't. >> Yep. Um, yeah. So, just keep working from the bits up, right? Like maybe Linux networking. This is like a good good next one. More questions. All right. Yeah, >> less of a question. A little bit more follow up on on her um answer

uh or or question. Um I I usually um when you have gaps in between levels of explanation of your mentor and yourself um I find that it's a little even though it's a little some people might not like it but I usually um tell them hey look this is what I know this is my understanding of the world and then how and then this is my goal.

Oh sorry this is my goal and then how do I can you fill me in in the gaps? I I feel like that sometimes um Uh, a lot of the people who ask the question are are feeling apprehension because they don't want their co-workers not to know what you don't know. But it's sort of a superpower because you they will only explain to you the enough that

you accomplish your goal and then later on you will be in a position to tackle more complex things. >> Yeah, great great addition. talk. Thank you. Um I um struggling with some concepts that I maybe you can help me like form one language around or a term. Um like I find a couple. So, I'm trying uh in sports, I guess, like um I never played sports uh

in in high school like really seriously, but say soccer, something I I always loved. Uh if I'm teaching someone soccer, like you break it down into like drills and uh I think it works really well. Like, let's learn how to position your body so you can pass the ball, things like that. Um, so it reminds me of like coding uh katas. >> Which is um I've also

never um done martial arts but it's what comes to mind again it's like rehearsals. >> Yeah. So the what I what I struggle with is like inside companies um how come teams don't maybe I'm I have a front-end uh background so um and more latestage startups not like big corporations but um I wish the teams or like the directors and the business gave us uh liberty I

guess and time like the space to uh rehearse amongst ourselves. >> Yeah, that's a great point. Um I know there's like advent of code is a way that I think a lot of my peers try to sort of exercise that desire because it's what what you're doing right by exercising these forms repeatedly and sort of what you're doing is you're making it something that you can do

without thinking, right? So then you can take all you have this whole toolbox of things that you can do without thinking and you can apply all of your thinking to how to combine them. Yeah. But yeah, it would be nice if we had more time for that. Uh one more question time. Great presentation. I apologize for missing the beginning of it. Um got a CIS admin background

about years on the Unix side and uh touched a lot of the DevOps uh pieces. Um, so you I'm at the more atomic level, I guess I would say, touched most of the tools, at least been exposed to some. Um, so what's a good glue everything together, I guess, is the best way to put it. >> Uh, get a production production for you, right? Not I wouldn't

say for a lot of users app running um in a home lab or like in a cheap VPS Kubernetes cluster. >> Okay. What what was that last last phrase? >> So like if you if you could like you can do it on Raspberry Pies although like hardware prices are kind of crazy right now. Like I think every developer should have a home lab if you can if

you have the space for it. But if you don't then you can like get a couple of cheap servers from Hner and like create a Kubernetes cluster on there and serve a web page just for yourself. So like pick something useful for yourself and and make one project live. >> Great. Thanks. >> All right. Thank you so much everybody. Okay, thank you Joshua. Um, interesting presentation and

some great Q&A. So, thank you to the audience as well for keeping it interesting. Coming up in five minutes or so, uh, Arthur Freeman uh, will will be doing his presentation. Um, and also before that, we'll have one or two sponsor um uh pitches as well. Um also remember at noon in here we're going to do an unconference thing where basically if you go to the right

URL where there's the images around here and it will be up on the screen in a few minutes. You can post up a or you can talk to Michelle and she'll tell you and you can basically pitch or vote for a topic you'd like to discuss. Then the most popular topics, we'll put up little signs around the room and during our little noon time break, we can

go into unconference mode and have some discussions on various topics. Um, so that that that's uh so if you can post up on that, that would be cool. Now I got to go find my uh sponsors for this next thing and I will be back. Arthur, would you like to do an AV check? >> I guess Rich will help you with Okay, there's one. Thank you. Nice

and early. Okay. All right. Here we go. Oh, we're close. We're in the woods. I can see some trees. Holy Oh, penguins. Penguins. The public Wi-Fi password is penguins, >> Yeah. Are you good? Yeah. >> Rich Horus. Rich Horus, please light up the sponsor slide here. that goes around 1055. We're good there. Hey, do you know where the And you got a way to Check. Dust. Dust.

>> Test. Hello. >> Do you hear it? It's working. Okay, >> I have to spin the batteries. >> Okay. Maybe we can find some more batteries. Sometimes I have some laying around. What size are they? Huh? >> Uh, you are >> Yeah. Yeah. So, um, >> oh, this Jerome from IBM >> and just come up and do >> what's the good way of doing this? >> Oh.

Time kills me. >> Yeah. Every heartbeat. >> It's not a reason to kill it. >> It's hard to savor something that kills >> No big difference. >> Longm country. non-existent >> as you as you zoom out. >> Yes. When you zoom out a very small >> my great grandparents left >> when my great grandparents left Finland that was Russia but it's been a while. Yeah, it was

1890. So, yeah, it wasn't the same as >> I mean, yeah. I mean, Ukraine was Russia till like 1956, right? >> Well, it was part of the Soviet Union, but like actually but like Ukraine as a country was actually as an SSR was part of Russia. That was good. >> Probably more useful than other >> that time was enhanced. It was not killed. You were right. Don't

kill time. Figure out how to make it better. >> Arthur will be speaking in a uh couple of minutes, but before that, we have some very special messages from two of our sponsors. Um, sponsorship is really important for making DevOps Day LA and scale happen. Um, we of course we also appreciate attendees. >> If you guys didn't show up, this thing would really suck. >> But sponsors

also help make it go. So, um, our two sponsors, uh, first up is, uh, Jerome from IBM who will come up and talk to us. Give us your best minute. >> Grab your mic. Come on up. >> Thank you, Jordan. >> Thank you. now. All right. Hi everyone. I'm Jerome. I'm product Sorry about that. I'm a product marketing manager at IBM. You know, it's actually IBM's first

time participating at DevOps days and we're really excited to be here. Um when someone mentions IBM, what normally comes to mind? uh whenever I would talk to like u nonIT folks specifically um some a lot of the times they would associate IBM with things like mainframes or you know legacy systems but today we're actually doing a lot much more than that you know we're actually helping solve

some of the toughest problems in IT and DevOps today we're building cutting edge solutions that are intended to help cloudnative applications s you know distributed systems and modern DevOps teams. Um I know a lot of you guys are you know very you know technical professionals are you know dealing with a lot of um issues right within code. So, you know, I think when it comes to reliability

or things like security or performance issues, I think a lot of you guys can relate to the reality that a lot of these issues, they normally start in code, but then they um they don't show up until much later on in production when they're, you know, already much harder or, you know, much more um you know, difficult to fix. Now imagine if you can actually detect those

issues much earlier in the cycle uh while code is being written and then continuously understand how that code behaves once it's running. That's exactly what we're showcasing here today of the DevOps days um with IBM concert and IBM Instana. So to give you guys just a preview of of what these two solutions are, IBM concert is actually that solution that analyzes code uh to surface risks before

they reach production while IBM Instana automatically maps and observes modern uh cloudnative applications in real time. Uh so together concert and instana are solutions that are enabling you to have um connect code to runtime. So this helps teams detect issues earlier, diagnose faster and recover quicker. Now if you guys think that that's something that might be useful for you or your teams maybe, then come talk to

us at the IBM booth just right outside this uh room. Uh we definitely love to show you more. Thank you. >> Uh thank you Jerome and thank you IBM. Yes, they have the cool well-lit IBM sign uh up front there. Okay, next up is Pasha from Bellsoft. >> Thank you all. I'm not a marketing person so I don't know how to talk. So, but I'm a technical

person. How so? I know how to write code and I know what Bellsoft does, what we do. We do Linux. We do Alpakita Linux. Who actually heard about Wellsoft? No one heard about. That's why our booth is somewhere right there on the right. We have our own fork of Alpakita Linux of Alpine Linux which is called Alpakita. Who knows Alpine Linux? Yeah. Who uses Alpine Linux? I

feel you. You I feel your pain. We fix this pain. You It's all free. It's all open source and we know how to do fix securities faster than them. And I guess we also provide the most complete Java experience because we actually started as a Java shop not as a Linux shop. So if you are some who is related to Java somehow okay you are not interested

in this part. Cool. So Linux who thinks why muscle is what's wrong with muscle? No one knows. Come to our booth I will tell you. We also have nice stickers and let's talk. We want to know what you are interested in and we want to improve our products for them to be better for you. Thank you so much. >> Thank you Pasha. Um okay uh coming up

our next speaker uh I've mentioned before that Rich and I host the monthly LA DevOps meetup and a very uh common attendee in the patio there either uh giving a presentation or in the out in the uh crowd uh hurling uh very interesting questions is Arthur and he's now he's here to speak to us all. >> Um but he'll also appreciate some good questions uh when that

time comes right Arthur. >> Take it away Arthur Freeman. >> When you're ready. >> Just drag this over. No, no. Thank you. Uh, I think I think we're here. So, hi everybody. Uh, my name is Arthur Freeman. Uh, I'm a director engineering of Zip Recorder, which is the number one rated hiring site that helps connects millions of job seekers with companies of all sizes. Hey Arthur, a

little louder. I know. >> Is it is it working? Yeah, speak. It's working, but you just got to >> got it. I will speak up. So, I said a little pitch. I do want to kind of say, right, all the opinions and kind of things that I'm going to say here are of my own. I'm not representing the recruiter uh here yada yada yada. Right. And so,

uh maybe we'll start with like a show of hands, right? So, how many of you would you say your primary job is building features that end users interact with directly? A couple of hands. I I'll speak louder. I've asked you how how many how many of you your job is building features that end users interact with directly. Right? Got a couple of hands up. And how many

of you built right systems, tools, infrastructure, right? That other engineers use, right? Not surprising, right? This is scale DevOps days, right? Uh but I think like looking around that's that's the divide, right? And I think that's the divide that's been here forever. Uh there's a Jeff Atwood quote of the Stack Overflow fame and he once I think said CIS admins are like werewolves. They may look outwardly

ordinary but are incredibly strong and vulnerable to stuff that will kill regular people and prone to strange transformations during moon outages. Uh I mean I think that the the main takeaway that that I that I want you to kind of hold in your head that once we get into the talk is that all of us work on product. We just have different customers. Um the divide is

not and never been about the people who work on the product and people who don't. All of different customers and the products are different. And so whether you call it like a platform, internal tool, infrastructure, whatever you're building it for other engineers to use and that's a product and I think that the what we're going to get into the talk the techniques for building good products are

the same on both sides. Just as the DevOps movement uh brought software engineering principles little bit like this. Oh. Oh, that's my face is too big. Is this is this better? This is This is better. All right. All right. This is Do I have to start from the beginning or maybe maybe not. Right. And so what I'm trying to say is that the techniques for both sides.

And and so when the DevOps movement started, part of it was bringing some of the software engineering principles like virgin, control, TI, testing, and those kind of things into our space. And I think the same is happening on the platform engineering side, but where we're bringing product engineering principles uh into our discipline. So different products, but maybe some generalization of the of the methods. Here's what I'm

going to talk about. We could talk about a couple of things, some patterns from the history that I think are somewhat permanent, what actually is and how what I how I believe uh it should be done well and what assumptions just broke and what we are building next. I mean, you may notice that I'm doing a little bit of a bait and switch here. This is was

not precisely the title of my talk, but it would be actually criminal to not talk about AI uh uh and the things that kind of have changed in the last six months to a year and those kind of things. And I this is like the world word cloud of the different titles that we kind of have. For many of you that have been doing this for a

long time, you could probably generate this word cloud from your own resumes, right? And those kind of things. uh and Ben Ben Rockwood talked about this uh great talk here last year about the titles that's totally worth listening to and those kind of things but but that's that's not the point of of my talk here right I mean I think every few years or generation or decade

or whatever we kind of we kind of rename ourselves uh and the names change but the people don't I don't believe the divide is organizational uh I think it's fundamentally temperamental uh some people are drawn to building the product that end users use and some like to build the system that enables that product. And you probably knew which one you were within the first couple of years

of your career. And I think that divide goes way back. It probably goes back to to industrial revolution. But but to kind of keep it closer to home, right? Like like even at Bell Labs, right? Ken Thompson, Dennis Richie built Unix, that was the system, right? The people who built on top of it was a different group with slightly different concerns. And you can almost think that

the entirety of the Unix philosophy, right? Small tools, composable interfaces. I mean, that was essentially platform engineering, right? Building for other builders in 1970s, right? But the divide was the same. And so, again, we're both building a product. The toolmaker's product is the tool. Our product is the system. And I think recognizing that is is actually the whole the whole game. And so, uh, right, we had

the different different frameworks over over the years, right? And some of them were were valuable, right? I think the job to be done never actually changed. Our goal was always we're making our engineers more effective at delivering business value. But the frameworks were valuable. They gave us they gave us language and tools to talk about it, right? DevOps gave us shared ownership, right? The the the CAMS,

bunch of collaboration, bunch of those valuable things when SRE from Google kind of came around to give us the language to talk about reliability engineering, error budgets. And I think platform engineering is giving uh giving us that in a similar way. It's giving us like this customer lens, right? That idea that you're actually building a product. And so this job again is the same. How you do

this that keeps changing. Uh and it's it's about to change again if it hasn't already, right? And so like to make it more concise, I think platform engineering is fundamentally productizing the interface between the builders and the systems. And so what what actually is it? Uh like is it that we've all seen that crazy CNCF landscape graph, right? Like I hope I hope this is not it,

right? But maybe if you believe the vendor ecosystem, that's what actually is. I maybe let let me first talk a little bit what it's what it's not, right? Uh it's not it's not YAML engineering, right? It's not Kubernetes. It's not it's not AWS, right? like is it pass maybe like Kelsey High Tower has some great quotes around PAS and what the companies actually want right but I

don't think it's any of those things specifically it may might include any of them right what I actually think it's it's really it's really gas right like it's glue it's glue as a service but but but productized uh this is actually my my favorite definition I don't know if it's it's a little it's a little hard to see but uh I I'll read that for Well, I

can't read it because I can't see it, right? But digital platform, right? Like foundation of APIs, tools, services, knowledge and support, right? Which is arranges compelling internal product and autonomous delivery teams use that to deliver deliver product feature at higher pace with reduced coordination. I mean, I think the key the key points here is compelling internal product, higher pace, reduced coordination. The purpose is and has always

been to remove incidental complexity and let the product teams focus on innovation. And so the platform becomes that cohesive collection of components that provides that golden golden path uh for you. So even if it is glue as a service, it still needs to be needs to be a product. We all know the golden paths, right? And those kind of things, it's easy to make a mistake. It's

easy to fall off it. I think it's the platform's job is to make the path of least resistance actually the most correct path. If doing the right thing requires more effort, people are not going to do this, right? And so the safe path has to also be the easy And so this is the I actually put a little too much on this on this on the slide

here, but I think this is the the definition that I would love for you to take home. Fundamentally, I think platform is constraints, self-service, and legibility. The interface itself, whether that happens to be a portal, CLI, chat interface, an API, that's interchangeable. Not that it's unimportant. Good interface design is is hard and it's maybe critical but ultimately it's a delivery mechanism right the platform is actually the

contract underneath and in a world where the agents are increasingly becoming your customer the UI is completely incidental the the contract is is is and a little bit of a story kind of kind of this is an AI generated picture of course right and those kind of things and I kind of want to just tell you how how I learned learned this the hard way and and

I tend to learn things the hard way until I get punched in the face a couple of times. The lesson doesn't doesn't stick, right? But this is like a story. This is like early Docker days. Pre-Cubernetes, my my team built like a container orchestration layer from scratch because Docker Swarm was terrible and nothing else was around, right? And it was a genuinely hard problem, right? Like highcale

infrastructure, complicated edge cases, distributed state, right? Things like emerging things that were breaking the ways that couldn't predict. And like we solved it. It it worked. It was elegant. It was reliable. Like we're legitimately proud of it. Nobody wanted to use it. Uh developers who had a choice, they kept using their crappy old duct tape deployment scripts, right? And and and the teams, I mean, we forced

some teams onto the platform. They complied. Uh they weren't happy about it, right? And and and we were baffled, right? Like the engineering is legitimately good. The system was reliable. What's wrong with these people? And it took me failing at this multiple times at multiple companies uh to figure it out. I like the problem wasn't them, it was us. Uh we never talked to them. We built

what they thought they needed based on what we knew about the infrastructure. We never asked what their day actually looks like. We optimized for the system, not for the human using the system. Once we actually sat with developers, started shadowing their deploys, understanding I'm sorry, understanding what was what was what was bad. We found kind of the real pain points, right? And kind of started fixing them

and then adoption flips. And again, it's not because the engineering changed. We didn't build a better consensus algorithm, right? We finally built the right the right product. And I think that's that's the gap oftenimes, right? you have brilliant engineering on one side but maybe lack of the product thinking on another ones if your developers hate your platform that's not a training problem right like RTFM is not

the answer here like that's that's a product problem and so and look building good products is genuinely hard uh you are not always going to succeed I don't always succeed at that uh but if you don't realize that that's what you are actually building it is impossible to it is impossible to build And this is kind of getting a little bit in more like practical actual things

like like so how how do you do that right uh a lot of these deserve their own separate and specific talks and those kind of things. So this is a high level overview. There's plenty of metrics, right, you want to measure, right? There's Dora, space, DevX, there are many others. Uh they're all right. You guys know the phrase all models are wrong, some are useful, right? They're

all useful. Uh look, we we all know metrics are gameable. You can pick at something in any of those models and point it out and like this is how it fails, right? And those kind of things. The point is that I think like the having no metrics is worse than having imperfect ones. Uh the one thing that I don't have on the slide I want to call

out specifically is that surveys are are a great tool as a complimentary measure one. I mean I know we're all engineers in a room we are we like quantitative stuff right hard metrics you don't want hear the system is slow you you want to know why right and those kind of things but surveys do give you that complimentary picture right like I think without them the picture

is incomplete if you're purely focusing on the hard metrics I think the space framework is one in particular that focuses on that like tries to look at it more holistically as a kind of developer experience and happiness and that kind of stuff with assumptions a happy developer is a more productive developer. Um, and look, surveys, right? They're not going to be perfect. One is not not the

other, but they're they're cheap, right? Implementing a DORA framework at your own respective company. It's a hard it's often a hard problem. You need to get executive buy in uh many other things. You can run a survey on Monday, right? That's that's that's cheap and that's going to give you and that's going to give you something. One thing that I want to kind of call out in

there specifically when you do measure you want to you want to think about measuring uh reliability and velocity as a control pair. If you're optimizing one at expense of another like you're going to make people sad. Uh another technique is the value stream mapping. Uh there's a fellow named Steve Pereira gave a talk here at scale a couple of years ago. I think it's a wonderful one.

It's a great introduction to value stream mapping. plenty of other materials on the topic, books and so on that deserves its own that can be its own very lengthy talk and workshop and that kind of stuff. I I'll kind of summarize it in in in a smaller version. What it looks like in practice is you sitting in a room with your top two or three most grumpy

and frustrated developers and you map their flows, right? You you you you can start at different points. The common one is like from the feature is done to it's in production and you will find out where the actual wait times are going to be right it's going to be waiting for CI and waiting for review and waiting for environment waiting for some other team right they're not

slow your system is slow right and and this is the only thing that you the way the only way you're going to find out is by doing this exercise with them not not not for and this is this is some of the the outside of the measurement is how how do you think about designing when you're thinking about about platform engineering and so cohesion I think is

a is a critical thing right so a good platform feels like one thing it does not feel like 12 bash scripts that you duct tape together you have you want to have consistent interfaces consistent patterns consistent expectations when a developer moves between the different services the platform should feel familiar like that's a design problem which is again part of the product own trade-offs freedom velocity maintainability uh

you can pick two you can't pick all three and you can't not pick freedom right well I mean maybe you can if your company doesn't care about innovation and that kind of stuff but those are your constraints uh the key thing you will make your own choices in your own respective companies but make the choice explicit because the implicit choice is going to be always worse right

uh whatever it is you end up choosing be be explicit about your preferences and those kind of things you open it up for debate and the questions and and even if developers or other people disagree with you. They will respect uh this choice and provides more clarity uh to the offramps. That's that's a little bit of a pet peeve of mine and a pet topic and I'll

try not to digress uh too much into that but you will all make mistakes. I make mistakes all the time, right? The platform should not lock people in. You can compel people to use your platform, right? You're all admins and those kind of things and you can force them uh because they have no choice. doesn't mean they like it. And so the off-ramp I think does two

things for you. It provides it provides that that critical sanity check on you. If you find that your users end up taking the off-ramp from your platform, that is a signal that you should think about. Why are they doing that? What is it that they're not liking? The second thing is a little bit more speculative that they think that's like an option bet or a lottery ticket.

You may have this brilliant engineer in your company that's going to say your platform is crap. I'm taking the offramp and I'm building some something better and they actually might and that is an opportunity for you to learn and bring that innovation back into your into your platform. So you don't want to completely handcuff them. I mean I think the off-ramp itself that does not need to

be super nice totally fair for you to put up a sign there be dragons here and those kind of things but the fact that it that it exists is is is important. And so kind of the the the main the main point the last point that I'm that I'm getting to all of this is really human centric right consistency cohesion delight that's what makes platforms work for

humans right and so reason why platform engineering exists is a discipline this is like to reduce cognitive load enable flow states right developers don't want to think about infrastructure and that mattered because humans have limited attention and they have you know limited context agent don't well no actually agents also have limited context but uh uh agents don't need a pretty UI right they they don't give a

about that they need contracts and constraints and so hold a thought because we're about to talk about what happens when humans are not your only customer or maybe never your customer and so everything that I just said so far was kind of the punchline of the talk the last time I give a version of this uh and today it's a setup So, let me be honest. Where

where we at right now, this is not particularly revealing. I mean, I think you're all living this every every day, right? Most of us are using AI to generate code faster. That's the current reality, right? Maybe not all of us are running fully autonomous agents in production yet, but AI can today do a growing chunk of the commodity work, right? config generation pipeline setups, boilerplate, terraform, whatever,

right? That is already AI is perfectly capable of doing all of that today, right? There's companies right now today taking mountains of legacy ungodly bash scripts and feeding them into OM and asking it to wrangle into something cohesive, right? And it's doing a hell of a job at that. And so the point that I'm trying to make is that if commodity work is all that your team

does, you have a problem. Not because AI takes your job, because that was never the job. That was the plumbing, right? The value is in everything around it, in the decisions and the constraints and the product that you're that you're building. And so, think of it this way. This is this is this is the best analogy that I kind of try to come up with. I actually

think this transformation is going to be bigger than the cloud. But when a cloud came along, the the right move was not like to invest in perfecting your VMware infrastructure, right? Even if you were racking and stacking servers at the time and VMware would have been objectively a net improvement of what you were doing still the right move was to build for what was coming right and

I think we're at that moment again whatever you're building next right whether you're calling it a platform or not build the next generation not the last one and there's kind of a small example that I want to give recent project we've kind of done which deserves its own talk and better given by the brilliant engineers that actually worked worked on this, right? But but like on a

high level summary, right? We kind of had the documentation problem and those kind of things as all of you does. Nobody has it solved, right? We did not use AI to write better and more complete documentation. We replace documentation with a system that answers questions. And so we stopped measuring documentation accuracy and completion and started measuring the answer quality. Right? And that's maybe like a small example

of between building the last generation and the next one. And I think the teams that kind of get it, they are thriving, right? And the ones that don't probably going to have a bad year, right? And this is this is because the bottleneck is just moved, right? And and the failure modes uh just changed. So I'm going to talk about some of those failure modes. I this

one is like the most obvious one, right? Like you've all you've all done this, right? Like you give AI some vague spec and build something technically correct and architecturally insane, right? And that kind of stuff. and spec gaps, they will scale. That's a new failure mode. Actually, this is the one that I'm least certain about, right? That might not matter in six months in in with better

models and those kind of things. So, I'm actually not highly confident that that one is going to stick around. I do think that is as of today that is still that is still an issue, but possibly will go away pretty pretty soon. But I think the key thing that's that's changing that makes everything that follows maybe make sense is the fact that your customer is changing. Today,

most of your customers are still developers using AI as a tool, right? But increasingly, your platform's customer is becoming the AI agent itself. And they're great customers, right? Like they will read the docs, they will follow the specs and the APIs, never complain, right? But they're also terrible customers. Like they will exploit every gap on your system and they will scale mistakes instantly, right? And so remember

what I said, the platform is the contract. when the agents are your customer, the quality of that contract is everything. And so that that shift I think from humans to agents as your customer is the thing that's breaking a bunch of assumptions we've been operating on. And so I can't predict the future. Uh but I'm going to talk about maybe some of the things that I'm either

observing or or anticipating, right? We'll we'll start will start breaking soon. Uh and I'll cover some of those things. And so, but yeah, the the the the things that I'm going to talk about, right, like this is this is kind of the preview of the following ones. I mean, I think AI did not change what we ship. It it's changing what's expensive, right? Like the code generation,

implementation, all of those things are effectively collapsing to zero, right? But there are other extremely valuable things in the system and those are the things that are that are becoming that are becoming expensive. Now, this is one of my my favorite quotes. This is from theory of constraints by gold rat, right? And it says any improvement made anywhere except the bottleneck is an illusion. I actually think

it's even worse because it this is this is passive. It just says your improvement is going to be useless. I think it actually in many cases will make your bottlenecks worse. And so the bottleneck used to be writing code. Uh that's the part that just got pretty cheap. But everything around the review, coordination, validation, decisions, that's where the time is going to be sitting now. And so

I don't think you escape resource constraints with abundance, but you start facing them at at at new scales. And getting into some specific priors that that that I'm that I've been questioning, right, is I think engineering capacity was never the real bottleneck, right? like decision- making those kind of things coordination spec quality those those always were were the bottlenecks AI is just making it more visible you're

probably I'm seeing this you're probably seeing it in your own respective organizations right teams are clearing their backlogs in days right with AI tools product wants to iterate faster you've got some engineers at the top of AI adoption curve they're moving incredibly fast and they're hitting walls not engineering walls system walls your review pipeline clients, your CIS, your approval workflows, your security councils, they were all designed

for human speed throughput. When somebody's generating code order of magnitude faster, everything downstream becomes the bottleneck. And so constraint has moved, but the system hasn't cut up cut up yet. And so, like I said, AI makes code cheap. It makes alignment expensive. We can generate 10 times amount of the PRs, but everything else is running at the human speed. And so all decisions th those are the

things that are that are kind of stopping you. And I do want to say that the decisions that matter most, what to build, what to optimize for, how to react to changes, those are normative decisions. They will still require human judgment about what you actually want. I don't think any model ever removes that. And so I think the bottleneck somewhat moves to the skills that platform engineering

has actually been trying to develop coordination, product thinking, building for others. I don't think everybody has those skills. That's that's that's me being honest. But I think I actually think that platform engineering as a discipline may matter more now, not less because it kind of becomes this practice ground for exactly the skills that just became the bottleneck. And so if you do the value stream mapping exercise

today with AI in the picture it the engineering time will compress but the map doesn't doesn't get shorter time just moves right so you didn't speed up delivery which was your objective in the first place you just made the real bottlenecks visible and remember me talking about the different frameworks the dora the space the whatevers right they didn't stop mattering uh they actually I believe they matter

more you all are about to spend a lot of money on AI tools right what's the impact what's the outcome? So I think it's the same frameworks but the the stakes became higher and so if you can't measure the before you can't prove the the after and additional things that kind of break I mean I think that the golden passes templates and UIs also break with it

agents don't need form they don't need your web portal right they need constraints they can discover and verify against right you got tests and policy as code for verification loop and you need written decisions and preferences for everything else right so maybe our golden paths actually become golden And so code review, code review as quality, I don't think code review disappears, right? I think it changes. I

think AI can review most generated code. Uh, honestly, I think it's probably better at catching many classes of bugs better than humans are. The shift is becoming to automatic verification, tests, constraints, policy checks. That's your actual scalable quality mechanism. So human review probably becomes a targeted high judgment activity. Maybe you're just reviewing the prompts. Maybe you're reviewing architectural decisions or risk boundaries or some new patterns, right?

But unlikely that's happening on every PR. And this is kind of like like to me it's almost like reframing a little bit of test-driven development, right? The tests are the deterministic validation for non-deterministic generation. If the tests pass and the constraints are met, the code is acceptable regardless of how it looks. And I know I said just something uncomfortable, right? like like for the people that take

pride in the craft which I think a lot of people in this room are and I want to acknowledge that and I think the craft doesn't die. I think it moves upstream writing the the specs intents the tests the constraints that will be the craft now and actually think it's harder not not easier and so look we're all used to building reliable systems on top of crap

right cloud is unreliable don't even mention DNS right like it's always DNS right everything is broken everything is broken all the time I'm surprised that internet actually works right but we know the techniques how to build reliable systems on top of unreliable foundations. We've been doing this for years, right? And so I kind of feel it's the same story here. And so it doesn't I don't think

it actually then matters how clean or messy your infrastructure is, your codebase can be ugly, your architecture can be messy, but if your tests are solid and your your agents can understand the system, they can work with it. But if you got some beautiful infrastructure, but the tests are thin and your observability is lacking and it's hard for agents to navigate your codebase, they're going to struggle.

And so this is mainly if you're choosing where to invest your next effort or roadmap initiative. Maybe think that you know something like test quality or observability is is going to generate more value than like a prettier YAML or rearranging your Terraform modules, right? So code is becoming cheap. Verification never was. So again it I know it sounds like I'm attacking the craft of engineering. I'm not.

I all I'm saying that the craft is moving from writing an implementation to writing the constraints. And so look how many of us have consistent machine readable service metadata across your entire stack with trace propagation and tax actually mean something like we don't right I don't my my company doesn't right but I think the observability was kind of fine for humans right you've got your prometheus you've

got your logging aggregator and those kind of things and you got some dashboards and humans are really good at filling the gaps right you kind of know which services flake in you know who to slack and those kind of things you got a bunch of tribal knowledge that makes it legible for you. Agents are kind of struggling with that a little bit today that again this is

something that may not be true from six months from now. They can maybe infer that context uh from those things. So this is may not be a super durable uh durable thing at this point at this point in time but I think the question is not whether your observability is machine readable like most of it is I'm sure right the question is whether it's joinable right can

can an agent correlate a trace to service to an owner to deploy to cost center to like some blast radius or whatever you probably have a bunch of those pieces but almost nobody has the full picture right and so the gap is maybe invisible a little bit to humans because we can bridge dig it with context and agents are maybe still going to be hitting a wall

at least as it as it stands today and I think systems are going to be changing faster than we can track it when AI agents are going to be modifying them if we can't observe it we can't govern it and so it is possible that the telemetry quality is going to become the ceiling on on our automation and maybe for the people in this room who build

observability systems right maybe your work just became the foundation for everything Um, but remember the trade-offs triangle that I talked about? It wasn't actually a triangle. It was it was a line. But freedom, velocity, maintainability, right? Pick two. But freedom and velocity just got turned up to 11, right? So anyone with an AI tool can spin up anything fast. Maintainability is now the binding constraint. So the

triangle didn't didn't disappear. The balance shifted and the constraints on that freedom, right? the guard rails the policy the policies the platform that's probably what's going to keep it from becoming chaotic and I do think that this is the platform's job in the new in the new world right so same job but the balance the balance shifted and I talked about the priors a little bit about

about the system but I think something is changing about the work itself right the inner loop is changing used to be edit compile run right now it's maybe I don't know prompt review, redirect, verify. Maybe you're managing parallel streams of like your agent generated work. And look, right now this looks like a developer having multiple cloud code sessions on their local machine. That's that's the reality. But

but the skill set is changing there, right? Now you're decomposing the problems in agent- sized work packages. You're kind of calibrating your trust in the outputs. You have to recognize plausible looking but wrong results. You got to maintain this architectural coherence. there's like three agents that are touching the system at the same time. I don't think we all have good muscle memory for this like nobody does,

right? But some of the skills that this requires like strong mental models, systemic thinking, reasoning from first principles, rapid quality assessment, thinking of terms and orchestration. I believe that those actually map to experienced in front skills. I I'm not going to pretend we all everybody in this room or me or anybody else already has them, but I think this is a little bit of a new muscle.

But I do believe that platform engineering is kind of as a discipline as a best better pra one of the better practice grounds for for building it. And so systems changed the work changed. Uh what what are we building in this in this in this new world? And this is actually the least the least certain slide that I that I have in here. Right? I mean I've

talked about the platform is the contract, right? And I really try to think hard how to make that contract concrete and think about the the surfaces that that will be part of that contract. Again, this in six months this might age like milk, right? And and some of these might not actually be true. I probably should have stack ranked maybe them in terms of my confidence level

and those kind of things. But you know identity and ownership when when a human when you look at the repo, right? You kind of probably know who ask for help, right? Agent doesn't. If agent wants to refactor a service, needs to programmatically like maybe answer who owns this, what tier it's in, what pages is a trigger, right? And those kind of things. If your service catalog is

like stale old confluence page, not a great place to be in, right? Constraints and policy, right? Like we talked about this safe path has to be the easy path. That's kind of the distinction in here. I think the test and policy as code, those are deterministic verification loops. Agents need them at this point. This is where they start shining. But not every constraint needs to be needs

to be code. Uh I think architecture decisions, your preferences, deployment preferences like we don't do X because of Y. Those could be docs. Human language is perfectly fine if it's discoverable and and current. Right? The the real problem is not the format kind of here. I think it's all the implicit decisions and preferences that never ever got written down, right? Like all the tribal knowledge and those

kind of things. Like everybody knows we don't deploy on Fridays, right? Everybody knows that service kind of fragile and sucks, right? But humans, we can navigate that through context, through our social knowledge and those kind of things. Agents are going to have a more difficult time for now. And so I think the context is what's becoming scarce now. And I believe that the teams that can externalize

their implicit knowledge bases are going to be starting pulling ahead of the of the other ones. Verification harnesses, we covered this. I mean, I think that one is actually the one that I'm that I'm most confident about. Tests are going to be the arbiter of truth of truth. Uh, having robust, fast, agent triggerable test environments is is likely becoming a tier one product feature. Uh, legibility, those

are things, right? Like it could be you could interpret it as dependency graphs like, you know, cost attribution, blast radius, and those kind of things. You could sort of think of it as if an agent proposes to change to some core routing layer can your system automatically detect calculate the blast radius and present that to human reviewer because if not then the humor human becomes the bottleneck.

I think the legibility actually also and that's maybe just a cost optimization vector, but how easy it is for the agents to navigate and understand your C codebase. If you're doing something really funky and obscure in in something that's not widely used, that's going to be more difficult like right that's not going to be in the training base. It's going to be harder for an agent to

make better decisions and operate in that kind of system again for now. And like look there's there's a bunch of things that I don't know right predictions I p you know predictions are hard especially about the future right uh I mean look maybe maybe better models make constraints unnecessary right maybe the model is the spec I I don't know maybe that divide that we kind of talked

about in the beginning maybe that final emerges I I don't think so I think as long as humans are part of some organizations that will exist but I'm holding that possible possibility right governance is currently an unsolved problem right who approved the architecture when AI generated the implementation the test and CI emerges right and so there's tons of unknowns that this whole things are are going to

be changing in fun and exciting ways over over the months over the months and years ahead and so right we kind of we talked about this show of hands in the beginning right people who build the system versus the people who build like the userfacing product I believe that that divide survived every every technology shift so far from mainframes from client server from cloud to devops and

so on. I think the interfaces interface changes every time but the split persists. I do think what will change is that the contract between the the two sides will have to become explicit. And I think that's that's kind of the part of that product thinking. I think actually a lot of people in this room have already been doing it. They just have not been calling it that.

And so like Monday actions, right? Like like what what's kind of the actionable takeaway, right? like I kind of had a hard time with maybe coming out coming up with some of this, right? You can look at your test suite, not the coverage numbers that probably don't mean all that much. Actually look at it, right? Like if an agent generated a significant change in your system, are

your tests going to catch a bad one? If not, maybe that is something you should consider prioritizing, right? Second one, right? I'm less convinced about that one, but like maybe pick a service name. Can an agent starting from that name programmatically answer who owns it? who depends on it, what like SLOs's govern it, what changed in the last 24 hours, like if not that requires a human

opening three different tools and four different Slack channels, like maybe that's a gap, right? And so and do the value stream map exercise, right? But map the decisions, not not code and and maybe see where the time goes now and that might might surprise you and and like look, you can use start you can start using that product language. I mean obviously we all work in a

different organizations and different stages of maturity and and a lot of the things may be out of your control but you can start using the product language. You talk about value streams, about customer research, about adoption metrics, right? If you're like, I hate my current company. I'm going to be interviewing somewhere else. Right? Maybe that's a good question to ask. How do they think about their internal

platform? Who is the customer? How they measure success? If they start listing off CNCF tools from that crazy diagram, like maybe that's not the place that kind of actually gets it. And so that's the final thing that that's that's my final takeaway. I think I think the tools are changing. The job stays the same. And I think the people best positioned to navigate what comes next aren't

the ones with answers. They are the ones that always build for others and are willing to admit how much they don't yet know about what building for others is about to mean. And that's it. Thank you everybody. Thank you, Arthur. Thank you. >> Um, also, uh, I'm sure we have some questions for Arthur, don't we? Please raise your hand if you have a question. >> And, and

I do want to I do want to thank Jordan and Rich in particular. I kind of give a version of this talk at the DevOps meetup as he called out earlier. If it was not for their encouragement to give this talk here, this I would not be on the stage. So, thank you George. >> And actually, uh, it disc it started with a, uh, spirited discussion in

the patio of O'Brien's pub where we host the meetups. The >> gut and then I said, "Hey, that would be a really cool presentation for LA DevOps." And then it evolved to this. >> So, yeah. Um, so anyway, come on crowd. Anybody got some There was There we go. Let's do this. >> And then we're going to be doing a unconference here at noon. Yo, >> thank

you. Thank you for a great presentation. Um, so it's interesting how you say glue as a service and I I see that um platforms or our profession uh will keep continue evolving uh hit some wall of with platform as a service. So uh out there today you have so many options and with AI writing the code the glue I think becomes uh less important. The AI can

glue things together and you talked about it uh last of part of the talk about like the ver verification. What do you think about like are we going to be um like security folks uh auditing, verifying, training but not providing the APIs themselves? So thank you for the for the question. It's it's a complex one, right? I I I don't know. Right. That that's the question. You're

that's true. A lot of the glue can now be written by by AI and probably better than we kind of did it did it before. I think organizing the organizing the thing into a cohesive pattern, right? And and and particularly focusing on the agents being able to use it is still going to be very much a human involved involved skill, right? like you're still going to have

some preferences in organization of how you want things to be done and those kind of things, right? So, like I guess the the real distinction that I want to make like those preferences and and things are still going to need to be expressed and managed by by somebody, right? Because otherwise I believe it's going to be chaotic, right? But it's just the actual implementation of those things

is is that's that's the thing that AI will handle, right? like maybe maybe like a simplified analogy whatever that kind of glue ends up being right like you're not you're not writing that Terraform module anymore right like AI can generate a mountain of Terraform perfectly well but you're still expressing what you want it to do right you you you own the outcome you you're still expressing those

kind of things and that glueless cohesion right and that kind of stuff I'm I am at least increasingly thinking of the agent as my customer not not the human right and so That's that's how that's maybe how the lens is changing. >> Okay. Uh yeah, my sense is that large language models, they're kind of getting into a scalability limit. Even if you increase to trillion parameters, they're

not going to improve that much. So people like Yandon the former meta scientist, they are looking at the world model. So instead of predicting the next token, the world model will predict the next state and that has the ability to eliminate hallucinations because it has the ability to selfcorrect if if it gets something wrong. So robotic robots can come and cook food in our kitchen. It can

predict the next state if a driverless car is going along the road. it can predict whether the person sitting or standing on the road is going to cross the road or go going to walk parallel. So and also this world models apparently use less energy in modeling also. So uh or I I'm just asking what is your opinion about the world models? >> So this is this

is really I'm a bad person to ask right I'm not in frontier labs working on the research and those kind of stuff. If I'm an observer very similar to many of you on those kind of things I I am well aware of that point of view right there are other people besides Yan Lagoon right like Richard Sutton is another one of the sort of bitter bitter lesson

fame also heavily criticizes LLMs they might be right I don't have enough expertise to to have like my own judgment but to the first part of your question I would be willing at least for the next maybe six to 18 months time frame. I do not see a reason why LLMs are not going to keep getting better. I honestly, and again, I'm going on record here, I

expect in the next, based on what I read and consume, again, not working for Frontier Labs, I expect in the next 6 12 months, we're going to see the same rate of improvement that we've seen in the previous six to 12 months. And so my thought experiments are with that assumption in in hand. So I might be wrong. That's might not what actually is going to happen.

But I given what I'm seeing uh and kind of the signals that I'm interpreting that's that's what I what I believe. >> Well, we we need to uh thank you Arthur. We unfortunately we have to stop. But right now we are going to get into unconference mode. So many of you uh were able to find our website and vote on some topics. We're going to drop these

topics around the room. Sit there where the little piece of paper sign is or ask Michelle where it is. And here's what y'all voted for. >> Number one, AI governance, security, ops best practices. Who doesn't want to talk about that? Uh next, automation and observability. A DevOps classic. um career conversations. Ah very good. Uh this leads into the uh open-source um career development track that's happening on

Sunday. Uh also meetup meetups and networking. I'll be at that one. By the way, see you at O'Brien's pub for LA DevOps February 26. Arson will be presenting there sponsored by Elastic. Um, and then finally, vulnerabilities and cyber security. All you paranoids, uh, like Scott Bllye can can sit and talk about that. Michelle, please go and distribute the, uh, the things all the things. And then we

chat and then we come back. We're going to have two uh, two good lightning talks starting at uh, 1:30. So, see you back here at 1:30 if you can't hang out for the >> 26. >> I meant March 26. March March 26. March Thursday, March 26. >> Sorry about that. We just had one last on last Thursday, February 26th. >> At least I got the number right.

Americ. Like if you looked the the it was pretty sparse. >> So there might be a combination of >> Yeah. slop. Hey, if you want a slop sign, talk to Lars. And I guess you could talk about slop at the in the in in the AI section. >> Slop slop productivity. So the thing is we weren't filling up the room. So I think we thought is the

open. So those are all >> right. You'll never walk alone. science. >> Yeah. Yeah. Hey. Hey, Michelle. >> Are we Let's uh >> Okay. Um where are our volunteers? Okay. So, why don't you point out You want to Hey, Michelle, where are our sections? Why don't you Okay, over here. What's this section here? Okay, here's the sign. Okay, AI people. Hey, Lars, come over here. stand over

here. You can get your AI slop sticker and also discuss about the various topics of AI. Next, if you want to unconference about vulnerabilities in cyber security, Scott and other paranoids, come here. Come on. You everybody knows that uh all these systems are under attack forever. So AI over there, over here automation and observability, the classics. >> Well, you know, you got to you have to you

can drill in to that that needle to the eye. Unfortunately, we do not have lunch. I'm so sorry, but it's prohibitively >> and also if we >> it's >> cool. >> It is >> Well, I hope you're enjoying the talk so far and the two lightning talks are going to be interesting. And oh, by the way, this the all the talks today are streamed on the the

uh the scale Linux channel. Look for Ballroom F. Right now, it's a long stream. The our wonderful AV team will over time chop up the talks and repost >> Oh Oh. Oh, they were Okay. Hey, let's chat about meetups and networking over here. I'll hang out over here and eat my uh I actually brought my own lunch. Actually, was my breakfast snack. The dream bar. okay, this

this uh >> I think that's it. Yeah. One, two, three, four, five. Oh. >> Excuse me. >> Oh, and career >> convos. >> Huh? >> Well, what do you want to talk about? >> We're AI. >> AI sucks or AI rules? >> What do you think? >> Okay, there you go. Why? I think it sucks. >> Why? Why do I think it sucks? Because AI is taking

instructions ultimately from humans. So it's all flawed. >> It's not. Well, there's other people >> All right. Thank you. Nice to meet you. We should just move this over here because he's speaking at the meetup. So this is the meetups and networking. >> So yeah, this is arson. >> Oh, you know. Okay, good. Okay. So you will you go to MLOps meetups. So there's another >> really.

I am a devops. up until I worked for I lost my >> but the thing is actually I was working at Edge. So I was edge after eventually Verizon took over. But I did I did transfer administrator. So I work That's a slow move. 13. Did I not talk to them? >> They are at 145. I Yeah. Thank you. So tell them about 1:40 to just come

right by the podium. >> Yeah. Yeah. Without lunch it sucks. Also, the chairs are all fixed. on each table and you get your little box. This is like we've have fixed chairs. That was yesterday. on the expo Yeah. Yeah. Yeah. They had a couple workshops in the morning. So they were like getting people to kind of download their agent thing. That was yesterday. >> Well, no, they

have. So the nice thing it it took a couple years because, you know, I helped bring into scale. I made the introduction. But this this is the year where they actually have stuff you can um get >> Yeah. Yeah. They have a a a client. Um so it's kind of cool. It's it's a good time. It's like are they ever going to get anything done? And then

also Open Claw comes out and it's like maybe some of the things they want to do. So yeah, they have and and uh there's a guy Brian Regazi that was there last week. He's he's working he's got his own kind of side project of a distributed network but it it's a kind of proof of concept or it blends into um what the qu it's their thing is

quet what the quet folks are doing >> you know working for distribity to understand what the challenges are of that because distributed AI is not hard >> if you don't care about security. >> Yeah. Right. Right. >> And uh >> but also for for the Quai project um which is personal the personal AI thing they're really concerned about security and you know letting the people hold their

own data here that is that's like that's why they're there. So anyway, Reza will be, you know, two the Expo 4 opens up, so I'm sure he'll be sitting in his booth. I hit you up in LinkedIn last night, I think, and I have the the there's the I have the one a version of the vent invite. So, if you can take a look at it really

quick and let me know if there's any issues because I want I'm going to take that and copy it to my other events if if unless you have some >> modifications. I haven't seen it yet. >> Okay. Take if you can take a look tonight. >> Yesterday. >> Oh, yeah. Yeah. Yeah. Yeah. No, that that party was happening. Yeah. >> You know, all day on your feet

and >> Oh, I know. until 10:30. So I got home and it just like >> Oh no, my feet disconnected. >> My feet were killing me. But it was like you had posted up the stuff and then I got the information from the elastic guys. I'm like I got to get this because I always c there have been certain scales where I have booked out like three

months worth of meetups ahead of time and made little flyers and like that. So at least I'm like, dude, I got to get this up tonight. I feel But as long as I can pee on my feet. But then I get up to like get a glass of water. Oh my god. Oh. >> But anyway, thank you. Thank for the detail. The talk looks good. >> Yeah.

I mean, it's like you're going to have energy, too, right? >> Energy. I got >> this a little caffeine. Nothing else. But it's just being here, you know. This is like every year for four days. I see like I so many like scale friends, you know, I just like Bria out front. >> The only time Yeah. Yeah. Yeah. >> There's people I've worked with at Edgecast. Edgecast,

you know. So, or people that I have seen from other meetups or something and then people fly in like Josh Burkus or whatever just oh hey >> and and scale volunteers, you know, now a lot of them now live somewhere else but they fly in just for scale. So it's my chance to chat with them. So I think since it's an all volunteer because the people want

to be here that brings the energy up, you know the facility. I mean it must be >> scale is a not for profofit so they have a bit of a slush fund. >> SC yeah scale is part of it's called Linux Linuxfest.org is the 501c3 but scale was the one who built it. over the course of four days. >> Oh yeah. >> I have no idea. >>

All I know is that we try to get as many gold sponsors out front because for for >> they're paying four. They're paying four grand to get the table in the one >> It's probably itemized, you know, for nonprofit. You're just curious. >> Yeah. We don't Yeah. Uh I Yeah. I I if I really cared, I could look it up. I don't care. I'm just happy to

do DevOps stuff. So, it's like 15 years ago who's out front and uh Brian and Chris Smith who runs the development track, they're like, "Hey, let's do a DevOps meetup in LA because you know DevOps is just coming up." So, they got a pub in Santa Monica and they just put the word out and John was one of the guys said, "Yeah, I'll fly out from Georgia

to speak at your pub." So, and a couple other somebody flew down from San Jose, what, you know, etc., etc. And then, but I was there. I'm like, I'm like, "Oh, I can't miss this. This is cool because I need to figure out what DevOps is." But there also Elon, who's one of the founders of scale, was there as well. as the thing was winding down, Alan,

Brianna, and I were just standing around, hey, this was great, man. I just go to the look, if you're going to do a DevOps day thing at scale, I want, you know, you know, I'd work with Brian at Edge. Oh, yeah, of course. Of course. But then it actually became a thing. So, we've been rocking it ever since. So I'm just I'm just happy to be >>

No, no, yeah. No. Uber geeks. Yes. Uber that that's like that MLOps meet up that Alex runs. That's a good concentration as well. And I tried to get him to come out here, but he's too busy. He actually hit me up on LinkedIn, but I can't believe I'm too busy. You got toast. see this once, you know, you got to see it really understand. >> By the

way, there is a tech called search engine it looks like that the previous episode you know if you show all the episodes was a a journalist like really sharp journalist but was embedded at anthropic offices in San Francisco for like six months so the guy who runs this podcast is talking to about how they do stuff on the inside. It's really interesting. Imagine what it must be

like. >> I didn't know that. I didn't know that. Are they No, I was I went to see Kelsey High Tower and in the morning the workshops were in the next room. So they were like >> but I didn't look should have because >> Oh, said. Yeah. Oh, he has small. >> No, he's No, he just got maintained. >> Oh, yeah. Don't worry. >> Oh, he totally

Well, no. Any any mix user is is most likely a mix zealot. There's a middle ground. There's a hybrid solution where people are used to containers like a way to kind of show >> figure. The other thing is at that cloud code meet up last week there was a game developer guy there and he's saying I use the um Nick's packaging system for virtualization or isolization to

run different agents and I guess but he doesn't use Nick's OS so there's a little bit of difference about the packaging system versus the operating system so that's another idea So yesterday, >> they're really they use it internally. >> No, that's that's the LA DevOps talk. That's like that would be insane. We have We have them sponsor and do the uh talk about how they use Nix

in their internal buildup. Then we have for the second talk would be they said whatever >> that would >> Oh yeah. Rust. Yeah. >> I want to expand. I haven't been hearing as much about nyx per se, but rust I think with aentic AI code development rust or aic code development um just smashes it levels out the steep learning curve rust. The benefits for Rust are arguably

I know some people around gladly argue this but better memory management um definitely than something like Python whatever so the you know you're you're you'll run more efficiently uh consume less resources um compiling it down to RO so yeah I agree on that I just but the ny right more secure, reliable packaging, right? so there's some advantage. I can see the advantage that I'm just not hearing

it in various podcast. >> Good. Good. >> Well, I mean, you would hear it. Oh, go. >> But once again, the chair of AI will can also smooth out the nick learning curve as well, but it may the AI bots or whatever may not be recommending nicks. you know, they the discussions of the really smart people hasn't filtered into blog posts that have been learned by the

latest versions of the LLM to like then say, "Oh, I'm going to build this application for you and I'm going to use it using the nick packaging system and because >> would you like me to do this?" Sure. Go ahead. as a master. >> So, was he speaking yesterday? >> He didn't speak. I didn't hear him speak. I just met him. >> Let's say people instead of

that data to your local machine and then somewhere you just somewhere else and so is everyone else you get the benefit of >> remotely you don't have to you don't have to transfer everything back down the CLI tools it's as if you're building locally but it's simply configured to build remotely you get all the benefits of using a local CLI I think I'm gonna run over there

and see if they're there because I think Nick's conf is that happening again today. Hallelujah. Is the battery That was me. Are they um just in case like they're opposite direction? But I'll be back right before you guys start. So I'll come back in like I'll be here from like 15 1. You guys are off. Are you dead? What? Beautiful. Oh, you're here. wireless. >> Yeah, he's

gonna be the first one. We have coming up is two lightning. So, So 15 minute talk. So, we've got a couple sponsors and then another 15 minute talk and then we're back 15 minutes later. >> Cool. Cool. >> Okay, cool. Hopefully, yeah, hopefully it'll it'll go smooth. And if not, we'll um Yeah, then then it's >> then it's 2 o' 3 o' 4 o' >> So yeah,

let's hope for the best. I'm okay hoping for the best. >> Hey, my name's air gap fist bump. I'm coming off the cold. >> Okay, thank you. Bill, thanks for your help. bring up I can grab my iPad. Let's do it. do our best. >> Sorry. >> Is that okay? >> Hello. >> Can you hear me? >> Sweet. >> Check. All right. Just leave that there. It's

fine. All right, we're going to start up in a Um, I guess >> Hey, well, we can hear that. 128. >> All right. Uh we are going to get started again. Welcome uh to u the afternoon segment of DevOps Day LA here at Scale 23X. I'm Jordan Schwarz. Uh that's Rich Horus. Uh we also host the local LA DevOps meetup which meets monthly in Santa Monica. As

it turns out, the next two lightning talks that are coming up have also presented at the LA DevOps monthly meetup. Um and they have some really cool short topics to talk to you about. But before that, we have some uh short informational messages from two of our Starting with um Carlos from Terracotta, you're up. Come on up here, grab the mic and give us your best >>

Yeah. Hey, you want to you want me to beat box for you and then you can rap? >> What's up everyone? How's it going? Hopefully you're having a great time. So I'm Carlos. I'm one of the co-founders of Terracotta AI. Uh we are a YC startup building in the infrastructure as code space. So basically, you know, how many of you have ever been the bottleneck on the

platform DevOps team where everyone's coming to you asking questions, you know, why is this drifting? What's going on? Who introduced this cost? All of this. Um and ultimately you just get tired of answering those people. So what we do is we are an infrastructure intelligence platform that basically reviews your infrastructures code PRs in real time detecting things like drift doing cost analysis security reviews in the context

of your entire infrastructure pipeline. So that's your local code, your diff, that's your remote state and even your live infrastructure. And then we then we print out PR comments that explain what exactly is going to happen so that your developers who are working in Kubernetes, Open Tofu, Terraform, all of that can see what is happening in real time in plain natural language, language, English. Now what happens

is on the other side is that those PRs are now beacons or snapshots that feed into our system and on our control or command center side the platform team now has views into what is changing over time such as cost drift and helps surface uh different things like impact what's going on who's increasing cost and information like that so that you can control everything that's happening across

your infrastructures code ecosystem without having to do anything Crazy. And it sets up in five minutes. You connect to repos and we are sock compliant. We are HIPPA compliant. We do onrem deployments. Feel free to stop by our booth over there for a quick demo. We have some fake Pokemon cards we're giving away if you've seen them around. We haven't got sued yet, so feel free grab

it. >> Hey, thank and thank you for sparing us the mic drop. That would have been expensive. Um, okay. Uh, and also uh from X Matters. Take it away. Give us >> Hey guys, Matt Steel here from X Matters. And what I wanted to talk to you really quickly about today was why sending an alert is not the same as instant response. So, high performing DevOps teams

like many of you guys, they don't just notify. What they do is they they trigger coordinated automated response the moment a signal hits. They don't just react to signals. They respond with intelligence and precision. And what that means is uh they're not just alerting the second something happens. They're taking their time to do things in the right way. And so X Matters helps you do those things

to alert people with precision and to make sure that um you're doing things in an automated fashion that keeps people working and basically getting action when they have to have it. That means there's less swivel chair. That means there's no searching for updates. That means you just get faster resolution to incidents when they happen. And so um if you guys want to see what we do, come

by the booth. Um, we're just outside the door there and um, we'll give you a little demo and walk through >> We also are giving you away some Apple headphones. The big guys. The big guys. >> Just just enter the draw and and we'll uh we'll see if you win at the end of the the sessions. We'll we'll do the draw. >> Oh my god, for sponsoring.

Okay, Dundie or Don Diego Aante is going to come up and and do the first lightning talk. >> Okie dokie. >> Oh, uh, one other quick thing I want to mention that on Sunday is an open-source career day. It's a separate track in the other building. So, if like me, you're on the market looking for that next great gig, uh, there are some talks and also some

people to consult with about open source career day. If you go by the DevOps table right around the corner, Brian is there who runs it and can talk to you more about it uh if you're interested. With no further ado, uh uh Don Diego Aante. >> Okie dokies. Thank you. Let's see. So, um we will go into presenter view. And unfortunately, I don't know why this is

full screen. I don't want this other thing to be full I swear I've done a slideshow before. Um, we're going to do it without notes because reasons and things. So, please bear with me. Um, but let's go ahead and get started. 14 minutes and 58 seconds to go. My name is Don Dante. I am a solutions engineering manager at Graphana Labs. I've been an observability for a

really long time. I was an engineer before that for also a really long time. I am surprised how many of you are here right now to hear a talk about Barnes, right? But barns don't burn down because of one loose nail. Now, this is going to make a little bit more sense in a second. But before we get there, just keep in mind yours may. Okay? They

don't normally, but yours very well may burn Who am I? This is me, my wife, my kid. I am from the deep, deep south. Very, very, very southern gentleman, right? And so things like cattle and actually building barns are things that I am intimately familiar with. But back to the topic at hand. What the freak is all this supposed to mean? That barns don't burn down because

Well, where does the fire actually start? Right? If you've watched Saturday morning cartoons or, you know, the days of cable and internet, like I said, lived in the middle of nowhere, we only had Saturday mornings. Um, there would be a picture of an idealic scene. It's beautiful. There's perhaps a Williams overture playing in the background. The good morning song. It's peaceful. Everything is working. It's awesome. But

fires do start. And you know if that's how the cartoon began, by the end it's going to be a lot more fun. In reality though, here's what happens. Here's where fires begin in your barn. There's a loose nail somewhere somehow. Maybe on the roof, maybe on the siding. don't really know, but somewhere one of those nails is going to work themselves out. Then a board is going

to shift. That board, that shingle, after shifting will allow water to get in. After water has started to get into your barn, corrosion and rot will begin to build up. If you got electrical systems inside of that barn, the edges will start to corrode once there's moisture in the environment, meaning that resistance will start to build up as well. And eventually something will spark. And this is

your barn, right? It's all gone. It's up in flames. Everybody is panicking and having a freakout moment. This is a really bad time. Now, fun fact for you. 80% of barnfires are actually caused by electrical malfunctions or human error. So, I didn't just make this up. Like, this is a real statistic done by Texas&M. What does this have to do with Kubernetes infrastructure in your own barn?

Right? The organization that you work for, the environments that you maintain. This may be you right now, right? You're so excited. Kubernetes is cool. It's fun. I'm doing all this like containerization stuff and AI is helping me do it because of course OpenCloud would never never lead me astray, right? And so you've got this beautiful tiny app that you've developed and it lives within your infrastructure. You've

got a nice uh like index.php file, you've got a container, you've got your pod, you have a small storage uh environment. Of course, it lives on 198 192.168.0.1 because it does because your router's not there because of course like you wouldn't do that, right? No. But here's what it looks like today. It's a little bit more complex. And so for those of you currently employed in organizations

or working with organizations that have large distributed systems, this is probably what you deal with on a day-to-day basis. It's complex. There's probably more than one you at that organization hopefully. But if not, you got a lot on your shoulders. And then sometimes this is what you have on your shoulders. a lot of very small fires burning in a lot of different places. This is stress. And

so you go from happy Fippy to sad Fippy as you try to address whatever these issues are. Now there are a myriad of problems that can occur within your application and within your infra. And I only have 15 minutes. I only have 10 minutes now. So, we're not going to go into all of those use cases, but let's talk about some of the most common Kubernetes behaviors

that can become loose nails in your environment. The first, we're going to talk about retries. And then second, we're going to talk about queuing. So, retries. Here's an example of some Python code. Lovely uh lovely code written by Gemini, I believe. Right. This looks fine. What is it doing? All right. When I see httpinventory items and it takes longer than 0.2 uh which is 200 milliseconds in

this case, I'm gonna go ahead and retry. And I'm going to do this three times every time there's latency beyond 200 milliseconds. If I run this in my home lab, if you run this in your home lab, you're probably fine, right? This this does work. I tested it. It it functions. But try to do this again when your environment has scaled out and services hundreds to thousands

of customers and then you say every time I experience latency on this one endpoint beyond 200 milliseconds let's go ahead and try three times to go again. Right? So you're not doing it three times times six times 10. You're probably doing it times hundreds or thousands. you are creating a giant firestorm or a retry storm inside of your environment from some very simple basic code. Right? This

is where the problem starts and how you handle it is I'm just going to do a retry which is normal. Now, how do you fix this? This is not designed to be prescriptive. If your code looked like the previous one, don't don't copy and paste this because I cannot guarantee this part will necessarily work for you, but it should, right? But the easy solutions, let's consider what

exponential backoff means, right? Every time there is a failure, let's go ahead and retry, but then let's wait a little bit and then let's try again and let's wait a little bit longer. Let's do it again. Wait a little bit longer. Why? Because you don't want to pummel your system with so many requests that you actually exacerbate the problem that occurred because of 200 milliseconds of latency,

right? You've made the problem worse. So, that's an easy solution. Second solution, give it a retry budget, right? Only do this for 20% uh of total requests. Don't do it every time. Somebody clicked on the app. Somebody's using the mobile thing and it didn't respond fast enough. Okay, you know what? They can click it again. You're a small startup. You're growing. Y Carbonator is going to be

okay with you like working through your customer experience. You can ship it off to X Matters if you need an alert there. I love the X Matters guys wherever they are. Like I don't that's not a knock on you. I'm just saying you can send it there if you want to. Right? These are simple and easy solutions to fairly mundane problems. But if you thought about that

and you're like, I'm a DevOps expert. I would never ever ever make such a amateur mistake. Well, you're wrong. You're absolutely 100% wrong. And here's why you are wrong. Right? being in observability for years. My my fellow graveyard friend over there also being in observability for many many many years. We have seen largecale organizations make these very simple mistakes. And working in observability, one of the things

that we recommend is you actually observe all of the things going on including retries within your infra. This is a paste from Graphana's AI assistant. Now, if you use Graphana Cloud or want to go test it out the AI since it's going to be included in there for free, ask it all the questions that you want to ask. It does have some guardrails in place here and

there, but you can definitely try. Like, I encourage you try to work around it because it's fun, right? But here's what it recommended. Let's go ahead and look at the retry storm that could potentially create. Also, BTW, here's a small code fix for you because you did a bad job in writing this the first time, so it was kind enough to suggest an actual fix. But then

the last thing that I did is said, "Hey, let me make an alert rule for you because you obviously don't know what you're doing. I will build this and help you out." Right? What a kind AI it is. I look forward to when it's our our master and overlord of observability and I will do its bidding. Now, let's move on to the second example, which is going

to be the Q, right? Exhibit A, innocent architecture. Here's my little bit of Python. Return static accepted for 202s. And then let's go ahead and create some replicas. Cool. Fine. If you want to scale, that's what you have to do, right? That that makes perfect sense until it doesn't because you explicitly say create replicas constantly, right? For your home lab, doesn't matter. Nobody cares, right? So the

que's extended and it's got like 9 10 15 20 requests that it's got to take care of. It's going to do it within a couple milliseconds. Nobody actually is going to notice. But you expand this out to hundreds or thousands of times as has been seen in large organizations, perhaps cloud providers, perhaps ride sharing providers, all those environments. And they make these same simple mistakes because they

thought it was innocent architecture and they overlooked this tiny nail that was loose. because you can't have one without the other in this case, right? If you are going to create replicas, you also need to understand things like back pressure, system pressure, the user experience, wait time. Those are critical because if you don't account for those, you're going to preemptively try to fix a problem and unfortunately

make it worse. Happens to the best of them. Second thing, if QEP is greater than something, now let's go fix the problem. Being proactive doesn't necessarily mean I'm going ahead and spinning up something new or building out or expanding our our cluster fast as I can. It should be thoughtful, right? And that's a very very important thing. But let's level set for a second, right? You're going

to want to try this. You're going to run into these issues. And just like in exhibit A within exhibit B, you need a way to observe this. The first example was observed with graphana AI assistant. This example open source graphana. This is using a very basic PROM label to say let me check for Q size ratio. Alternatively, you could check for QEP, right? And do an average

over time. Both of those things are fine to help you not have this issue in the long term. But this is a very important truth for all of you because I'm sure as some of you have laughed and chuckled, you've seen this. And what amateur mistakes from an amateur on stage, right? You will make these mistakes if you haven't already. It's the honest truth. And if you

don't make these mistakes, you will make another one. I guarantee you, you're not perfect. You are going to slip up. But when that happens, if it hasn't already, it's going to be okay. You will survive, right? There is a fix, but you will absolutely positively learn a lesson from it and ideally be better. But what can you do? Well, you can plan for the things that you

know, right? You can plan for scaling. You can plan for replicas. You can plan for query ratios. You can plan for expected and potentially unexpected latency depending on the APIs that you're hitting. Those are all things you can do and you can try to do your very best to account for the things that you just don't know and you just don't control. Right? You will not be

perfect at it, but you can try and you should. You should build in some sort of fire suppression system. Maybe use OTEL or another open standard to collect the right metrics, the right data, Prometheus formatted, trace data, log data, whatever it is that you're into and whatever it is that works for you to put in place to actually see what's happening. And then ideally come up with

a planned resolution, AI assisted or not, to work through those problems when you see them. And when they happen, don't panic when sparks start flying because they will, right? Okay. And if you're the cool, calmheaded engineer in the room, everyone will absolutely love you for it, right? You're going to be like this Grat. That's called Grat, the little thing with the visor. That's like Graphana's mascot. It's

a dinosaur technically, but So, these are all realities and truths that you'll face. Barnes don't burn down because of one loose nail. They don't. Cascading failures, large infraures don't happen because of one tiny tiny little thing, right? It's a series of problems that build up over time due to natural human oversight, unexpected growth. Growth is not a bad problem to have, but you can't always predict when

it's going to come. Look at Anthropic just the other day. They didn't expect to have that many users all of a sudden decide to start using Claude and they had a little trouble scaling and they're they're not a small company, right? With a lot of very very very talented engineers, but it happened. But with a little bit of Thor thought and a bit of luck, your barn

may not burn down either. Thank >> Thank you very much, Dundee. uh, coming out is next. Um, you're gonna want to grab that >> and but we but don't worry, Angen. No need to rush. We have time because we have two sponsor pitches that we'll do with this mic. But use that and get get yourself lit up. Um, yeah. So, the nice thing about these lightning talks

is both Dundie and and Engan have spoke at our local monthly LA DevOps meetup and you're welcome to join us. Next one will be at O'Brien's Pub in Santa Monica on Thursday, March 26th. Um, we'll have uh somebody from Elastic speaking and uh Arson who may be in the room talking about oh, OpenClaw and setting up a whole autonomous organization in OpenClaw to produce code. Anyway, so

now let's get on. We have uh two more of our sponsors to give some quick one minute talks. And the first Huh. Oh, and he's he's whispering, but he says Dondie Grafana is hiring as well, so go talk to Dondie. I'll tackle him. Okay. So, the next So, so our first sponsor pitch is Mark from Crash Override. Come up, take this mic, and give us your best

minute. All right. First of all, I have to apologize because anyone want to talk to devs. I'm the guy who started OASP. So, if I've screwed up your life, my sincere apologies for getting in your way. Um, what we do at Crash Override is that you have GitHub on one side. Basically, you have code. You have observability tools on the other side. There's a massive black box

that sits in the middle called the build system. No one knows what happens inside the build system. Developer doesn't know what code's pushed into production. Person in production doesn't know exactly what code. We've basically backdoor the build system. We now know everything that goes through. Connect all the dots together and all of a sudden you get visibility code to cloud. Simple as that. Open source. Go to

crashoverride.com. There's a chalk project. Go run it. Run it on your build systems. Go figure out what goes on inside. Silence. I'm so sorry. >> Hey, thank you uh and and thanks uh Crash Override for sponsoring DevOps Day and ultimately scale. Okay. Uh, next up is uh Stefan from DNS Simple. Here's the mic. Give us your best minute. >> Hi everyone. Uh, I work for DM Simple

as a platform engineer. Uh, we focus on DNS and domain management services. Uh, we have a lot of automation tools in our platform focused on developer friendly APIs. Uh, and a Terraform provider as well. Uh if you have any questions about DNS or domain management, we have a booth out in the hallway, so come ask >> Okay, this is working. >> That was tight. Thank you and

thank you Um okay, so in a minute um or so. Um Engine will be up to talk about the Ralph Wigum Loop. >> Perfect. uh which should be a fun little lightning talk and then we've got some um longer talks running uh running after that and also as I mentioned earlier uh on Sunday there's a track here at scale called open open-source career day. So, if you're

looking for that next awesome position and Dundie doesn't offer it to you out in the hallway, um stop by uh the open source career day, which I think is over in the other building um on Sunday. Um and Bryina, who is sitting at the uh LA DevOps askina anything desk, uh runs open source career day and can tell you more about it. >> All right. Hey, we

got slides. Take it away, Engan. >> I'm ready. So welcome everyone and as you can see I would like to talk to you about the Ralph loop. Okay. So how to Ralph loop or other persons would call it how to burn tokens. Um before I start please Disney don't sue me. There will be a lot of Simpsons stuff because obviously I'm talking about Ralph and Ralph Wigan.

So it's full of um probably copyright stuff. So please don't sue me. Um I know that we are very close to Disney. So I don't want to get any troubles because this is the only lawyer I can afford. So Lionel Huts I'm probably using chat GPT. So that's the reason. Please forgive me. Okay. The Ralph loop. So um or how I called it how autonomous AI loops

helped me to build a serverless sus overnight. Yes, it's clickbaity, but still you will get the drift when you see about Ralph loop that it's not far away from the truth that you really could come up with a whole sus solution if you really work cleverly on this one. Shortly to myself, you see everything is Simpsons style. So, I'm a senior solutions architect at Pulumi. Uh there

are a lot of credentials. You can follow me from GitHub to LinkedIn. LinkedIn is a little bit more cringy content because it has to be corporate style and because I want to get hired or you know what I mean and then with X it's a little bit more free stuff and because of all the AI stuff and what I saw you know because I saw stuff with

AI I was already planning to say okay what could be a good plan B so I asked Chat GPT because Chet GPT has all my history of all the dumb thing I ask him um to say what could be a good plan B for me and that what it comes up. So chat GPT says your plan B is trades. So if you see me next time installing

your air con or something like this. Yeah, that's because AI took my job. Um and apparently trades is hard to automate for the moment. Okay, so Ralph, back to the topic, Ralph. So yeah, there were a lot of content in the hype phase of Ralph and I talk really a lot of stuff. It went really viral. Everybody and my mother talked about the Ralph loop. Um even

I uh created um two blog posts about this because I found this interesting and I was like I need to try out what is all about Ralph. At this time I had access to our API key from uh our company account for Entropic and I hope nobody is here to tell them. So, um, I had no I had unlimited access to, um, to tokens, so I could

really go crazy with the Ralph loop. And I also made a second blog post about how to optimize Ralph to give also some um, some verifying capabilities because you will later see that's very important. And it was so cool that the creator of Ralph retweeted this one. And this give me the 5 seconds of fame of Twitter fame because I got a lot of likes and so

on. So that was really cool. This short moment where I felt like a celebrity because I made it and my mom didn't understand this. So I didn't get the fame at home but at least it was good enough. Okay. So what's the problem we tried to solve here with Ralph? Um and you see again Simpsons even the fonts are stolen. So um AI babysitting. So that's was

the thing. So we want autonomous agents, you know, that was something we promised like, hey, I give you my idea and um you you you work on this bond. But what we really got and disclaimer again, things in the AI world really really change quickly. So some of the things they are completely outdated because every day there's a new thing popping up in the AI world. But

yeah, we got the problem here with long session compaction for of of context. So so things get lost. So compact uh uh context compaction is still a thing. Uh I regularly run into this. So we have here the compact context. We need to take care to to be in this golden in this uh sweet spot of of the context window to get the best outcome and context

windows fills up very quickly. So uh if you do some some runs with with linting or whatever it can eat up very much of your content. And then comes the next point. This is also slightly solved. There are really really cool tools, but at the beginning you have your terminal open, you let your AI run and then you have to keep on checking on the system if

it's finished. Maybe it asks something. So you end up really having three five windows open and go and answer the stuff and so on. Um and then we have also the problem my colleague also mentioned this in his talk like AI generated PRs get rejected because here we come now into the quantity over quality and uh yeah sometimes you have also the the point where an agent

thinks like hey I'm done but it never verified this work. So then you're like happy it happens still to me like the system says hey I'm done I'm so happy and I just do a a deployment a test and I see it's completely made up stuff or didn't work and I have to go back again and tell the system bro you made an issue I really call

my agent bro um just a thing so sounds familiar yes maybe so before heading in just a short note about the creator because he's really an interesting person so here uh Jeffrey Joffrey Huntley he lives is in Australia. He created um Ralph he published an article in 2025 which feels in AI time like dinosaur times you know um so quick we are evolving in July and he

was also creating this language cursed um where he creates this gen C slang compiler also very interesting to see and he find out that a bash loop with a fresh context produces here extraordinary results that it was even so much of a revelation for him that he himself was very spooked to find out something so effectively which was so simple created. Um why did he named it

Ralph Wiggins? So what's the whole story about this? So so of course Ralph is here the lovable uh but stubborn person who keeps trying this stuff even if it fails and accordingly and please I'm not a native person so I maybe not notice it's also a 1980s slang. Uh Ralph is also something people said at back in this time around vomiting. So Ralph so the the loop

Ralph codes until the test passes. So vomiting the stuff but in its purest form Ralph is a bash loop. Okay. So theory so generate so there is the thing we all know. So generating code is now cheap. This is really really true. Even now we understood back in the days it may sounded a little bit uh new but now we know here in March um this is

more than true um generating code is something um doesn't have any big value anymore um but now what the hard part is left what he mentioned is that Ralph has to generate the right thing not just to create code but to create the right thing so which is very important for the Ralph loop to work is to say okay we use git as a memory here so

progress is committed never lost at compaction. Um, one task per iteration. No, never never overload the context. Um, fresh context per loop. So, every time we fetch a task, we're going to create um a new context window. And most importantly is we need to back pressure the system via tests. So, setting up all the linting. You know, when you're in Python, set up the rough. When you're

in Golang, set up Golang Golang lint CI tools and everything. create your test cases. So the system um when it works in the loop, it has to run the test keeps failing and so on and so on. So really uh the Ralph way and now we have also to step back. We still say human in the loop but we don't micromanage anymore like we did before. We

now just supervise and wait the story comes back or we create the specification to let the system run. So the three phases of Ralph phase one define the specification. So write your your product requirement definition acceptance criteria planning the prompt. So we as a human we set the goals and validate the commands. Then we head over in the plan mode where the system runs and creates the

the AI runs and creates the implementation plan. You will see this also a little bit in the demo. Uh we can review this and then approve and then it's done. And now comes the build mode. And here's the beauty of Ralph uh that it's going to run in a in a Y loop. This is something you can literally do in your systems. Uh you can create your

own Californian Ralph and then um run this stuff and call Codeex or CL or whatever system you are want to run and because of this it calls every time a new cloud or new codeex it has the fresh context and yeah okay the ecosystem. So of course something this interesting also sparks a lot of people to create an ecosystem around this. So you have um an created

list of all the stuff uh which is happening in the Ralph community. Um you have the Ralphie CLI which I'm using. It's still in the idea it's using still the loop but gives a little bit harnessing elements. So I can really put this back pressure into a system and let it run there. Um yeah and then a lot of other talks Reddit if you're really brave go

to Reddit um and then um ask questions there. And then you have also uh companies talking about this. What I like very much is the agent browser of Versel because this one is not like the playright or other other systems which goes through the whole DOM and puts out the dumb DOM and then kills your context. It's really effective here. And then uh Cloud Co has also

an official Ralph plugin. But here's the problem. The implementation of the Ralph plugin in cloud code is uh not really uh re resetting your context window. That's a little bit problematic and then our pulumi product. Okay, so the bash loop versus Ralphie. So that was what I told you that's the simplest version. You can run instantly in your system, create your bash while true and then cat

one of the prompts and go through this. Um I use Ralphie because I can set up now a config file where I can set commands, capabilities and rules. So this is the stuff uh during the Ralph loop it runs. So nowadays as I mentioned things evolved um in March 2026 we call this harness harness engineing we provide here in harness for the agent to run and follow

the steps because we want to influence the outcome and then you just execute it with Ralphie minus minus pro to your um prd and then it's done. So how does it look like in a demo? This is all I did. I mean I I had a a demo shell script where I created a little bit more details. Um I will show you the task list. So it

goes through your PRDMD. The PRDMD has a lot of context what I'm going to build. But most important it has the um the the the second level task definitions and it goes through all the 18 tasks. This PRD is also created by AI. I did not sit there. I just told the AI system to create the PRD for me. I reviewed the PRD and see hey that

makes sense. I set up the config for the harness and then let the system run. And here you can see it went through 18 tasks. It passed all the 18 tasks in this case because the scope was very limited having here an AWS um API gateway setup and our basic front end took me uh 73 minutes. I have sometimes German numbers and so on, you know, 73

minutes and eight times 18 times creating the fresh context. So how does the finished SAS in quotes look like? So it really created this. I have it online. Uh it's called a snip. He even come up with a snip because what I used I used a CL code skill called um called front end skill. So it really doesn't create me this AI sloppy front ends we know

with the pink gradients and all this work. It really created a for me nice looking one. It created a dashboard API documentations and this is how it looks like in real life. I can go in enter here URL. You see this is the the CloudFront UL it created when I look up the URL because I tried it in the dashboard. uh I get some statistics you know

this is everything what Ralph created for me um and then I also get the API specification with some examples and so on. So that's what I all defined here in Uh no that's not the wrong that's the Okay so uh let me how many minutes do we >> okay okay wrapping up. So I'm not showing you the the the code part. So just try it out. See

how the PRD looks like it's not that complicated. If I can do it, you probably can do it even better. So best practices and pitfalls I would see. So the do of course use git. You will see when you run Ralphie, it commits all the time. This every task gets committed gets committed. That's very good. If you need to pick up it, goes through the git status

instinctive h intuitively and reads up what you already did. uh one task per iteration, fresh context, uh clear acceptance criterias, make it very clear what you do. For example, using some kind of um um Versel agent, for example, browser agent to to to call the website and to try this stuff and uh get additional feedback. Not only the code works, but also in runtime there are no

issues. And that's what I wrote here. And then the PRD, don't reuse sessions. Don't do some kind of uh um scope creeping. I mean this is stuff we already know. Ski don't ever skip test gates. H what I also do I mentioned this I really let him hardcore go through the llinters. Never skip llinters. Never allow AI to skip any linting rules. Like bro you are machine

why are you coming and want to skip something? Um fix the stuff fix the tests. Um and then you burn a lot of tokens. Key takeaways. uh AI autonomous loops and you can call them Ralph or whatever you want to call them are here to stay. They are production ready and you will see them and you will probably use them in one kind. It doesn't have to

be called Ralph. Simplicity wins in this case. So bash over frameworks no orchestration no runtime just a while loop and fresh context is the king to prevent hallucination. Now think about fresh context with skills and MCP servers. Uh, that's really a successful h recipe for for success. And yeah, the math speaks for itself. Um, that's why I'm looking for plan B because I'm probably too expensive in

the future. Okay. Uh, yeah. >> Thank you. Thank you, Andrew. >> And and by the way, oh, Dustin, if you're here, I hope you're here. Dustin, please uh come up and uh get the wireless mic in the AV check. Hey, I'm sorry. We don't have time for u questions, but maybe catching out in the >> I thought Justin from Disney. >> Okay. So, um while they're setting

up, we're going to have a couple more um one minute sponsor presentations. Let's see. Um Nick from Unleash. Okay. Uh why don't you come up and uh here's the mic and give us your best Thanks DevOps Day for having us. We're Unleash. We control how you release features into production. So with AI created code being such a big part of our daytodays now, creating code is easier

than ever before. And so now the hard part isn't necessarily the code creation but the control around it. And what we do is we are give you the ability to control the code that you release in production while maintaining speed quality and the safety that you want to have as engineering leaders. So for us overall whether that's kill switches whether that's AB tests whether that's audit logging

security we've built that all into our motion and that's what we call feature ops overall as a whole. So for us from an a uh product that's totally trusted by thousands of production installs and thousands of users on GitHub, when you want a safety net as it relates to AI, we feel like that we're the safety net that you need to deliver. Thanks everybody. Enjoy the day.

Appreciate it. >> Hey, thank you very much. Nick from Unleash. Okay. Uh here's Amir from Samuel AI. Take it away. >> Best minute. >> Best minute ever. Hey, nice to meet you. I'm here. I'm the CTO of Somals. Great to be here at DevOps Day. A lot of our customers and people we talk to over here boot over there has a lot of their observability stack drowning

in data. We help solve that. According to uh research, 87% of the telemetry data is not not really being used. Oh man. >> So 87% of the telemetry that is not really being used and with s we connect between your data and observability platform. We filter it out. We find spammy logs and metrics are not really um needed and we help reduce that DOP metrics and overall

give visibility into your data. uh we track all of that using AI um and give you a better standpoint. Our customer reduced about 60 to 80% of their data. Our boot is right over there. Uh we have a raffle. Uh you can meet win meta quest. Uh always want to say S3 but 3S for some reason. Yeah. >> Thank you Amir. Thanks for sponsoring um DevOps day

and scale. Okay, we have one more. Um, uh, Mayer from, uh, Teleport, >> grab the mic and give us your best >> Thank you. Thank you. >> Well, open source community. Uh, who here loves open source software, >> everybody? Right. Um, so, Teleport started as an open-source uh project to get easier, safer, and quicker access to SSH server. And now it has then expanded into identity infrastructure

platform. So you can use teleport to uh have your human identity and non-human identity which is your cloud code and AI boards into critical infrastructure. So spanning from SSH, databases, Kubernetes, RDP servers, um web applications, all that in one place as a developer, right? So platform engineering losses us and uh we basically also help security people feel better because now you know who did what when and

how through centralized code and we are written in go on a backend side uh front end is uh react so if you want to hack things around it's on GitHub uh please check us out it's go teleport.com and gravitational teleport is a GitHub project thank Okay. Thank you, mayor, and thank you for sponsoring scale. Okay. Our next presenter is Dustin Kirk Kirkland. Take it away, Dustin. >>

All right. Thanks, Jonathan. Appreciate it. Can you hear me? Okay. Out there. Thumbs up. All right. Good. Thank you. Uh, good afternoon everyone. It's such a privilege to be back at scale. It's been quite some time since I've had the honor to present here. Uh, but really excited. I love this conference. I love how much energy there is in the in the hallways on both sides. U

certainly been around for some late night hacking sessions with some old friends uh solving some bugs and and solving the world's problems around open source software. So it's really cool to be back here today. Bit of a different topic. Uh I'm sure no one has told you anything about uh AI or Agentic today, right? No. Okay. Okay, good. Well, you're in the right place if you want

to hear about it for the first time. Sorry. Uh no, there's a lot of this going around. I'm going to tell you a little bit about how uh I see it applying to open source software, especially in the Linux operating system, the DRO and the large scale open source project space. Is this working? Can I use this? Well, we're going to skip that. All What would you

think about a Linux distribution that improves its security every single day, while you sleep? While you're sleeping, while you're doing something else. That's what I want to talk about. a set of patterns that ensure that we're able to not just ship a patch once but continuously and reduce exposure windows while the world is moving. If you treat change as inevitable, and I certainly do, I think it

is, then the right answer isn't individual heroics by members of your team or your open source community. It's designing a system that assumes that that constant churn is happening and turns that into routine and verified work. So, let's set a few expectations. This isn't magic and this really isn't a marketing pitch. It's really talking about the automation, full automation with verification. I'm going to say that word

verification so much during this talk, you're going to be tired of it. It's verification at every single step along the way, running continuously. Now, it's the verification part that I think is the point of all of this. In software engineering, trust comes from evidence. And we get that from build logs, from test results, from cryptographic signatures, provenence documents, and ultimately reproducibility. Automation without verification is just ensuring

that you fail faster. Now, Linux and open source never stops changing. That's part of the beauty and why I've been part of this world for over 25 years. Never stops changing. Upstream releases are rolling in. Tool chains evolve. Dependencies shift and vulnerabilities show up constantly while you're sleeping. And even stable isn't static. Stable is a policy about what changes are and aren't allowed. So, I'd argue that

the workload is absolutely relentless. And the conclusion for me isn't pessimism, it's clarity. We need systems that are designed for constant motion and not just occasional bursts of effort. Now, patching faster is not optional because the exposure windows are where the risk, the security risks actually live. The time between disclosure and remediation, that's the danger zone. That's when systems are exposed and vulnerable. But speed without breakage

is the hard part. Trying to deliver change to your users without breaking them. That's the hard part. Regressions destroy confidence. I'm sure you can remember the times your DRO has broken you more than the times that you've been wowed by that same dro. Regressions at scale are brutal. For those of us who are doing this during the week and not just on the weekends, you really don't

want to see a regression in production at scale. So the goal is to be both fast and correct. And that means we need to shorten the loop, increase the evidence, and run those verification loops constantly. I was in for the the previous talk and I thought it was really interesting that the Ralph Wiggum loop is it really caught everyone's attention a couple of weeks ago and yeah

we're seeing this a lot of this talk is built on some of the same concepts of running this automation in the background but where I'm going to try to focus here is on that verification piece so complexity is increasing across every axis that you can measure something on more packages more architectures more variance within each and more supply chain attack surface. And I can tell you as

a leader of a very large engineering organization, hiring more people doesn't fix the coordination So we need leverage. We need repeatable systems that scale the output and the correctness of those people without requiring us to just double and triple the size of every single every single team. Those teams want less toil, more creative work and less toil. And we can give them that. So now we're going

to talk about agents. And um I know that sometimes this is overloaded. We're not just talking about chat bots on a website. We're talking about workflow automation that has a very clear goal, an end state that you want to see, and verification loops that eventually get there. And with that definition, we can talk about where agents help and where they So for us, it's more than just

quality and assurance Q&A. It's not just autocomplete and ask the agent to write a script that then we build and commit and and run somewhere. The agents actually operate inside of real systems. That's our CI/CD, our build for farms, uh the packaging tools. They carry state and intent across every single step. And good agentic workflows produce artifacts along with evidence. Not just walls of text, but actual

evidence that proves that it did what you wanted it to do. And here's how and here's why. And here's how you could reproduce the same result if you wanted to. And I think that's what's essential in serious engineering as a workflows are adopted. Now this is the loop that I think matters. Plan, act, verify, learn from that and then repeat the process over and over and over

again. That's the heart of our agentic systems. Plan, act, verify, learn and repeat. Now verify is right in the middle. And again, I'll say verification is the non-negotiable step especially in supply chains. Correctness is And the learn part isn't all that mystical. It's as simple as capturing the failures, improving the heristics, refining the checks, and reducing the rate of human interaction with the system. Take the humans

out of the hard part as the agents as you're able to to teach the agents from additional telemetry. Now, how do we take that loop and put that into production? So first it starts with interactive prompting. We do a lot of that on the early stages but eventually turn those into pipelines. It can be skills or or something similar but eventually those are driven into pipelines and

then ultimately sitting that inside of a that ensures that it's running correctly. A couple of weeks ago uh we released an open source tool called Drifless AF. AF stands for agentic framework. I promise it's an agentic framework that helps ensure that you define that goal state. You do some work, you verify it against ideal results, you reconcile the behavior, and then you eventually get there while also

learning and turning all of that telemetry into a smarter system over time. In DRO engineering, we want repeatability, deterministic runs, observable behavior, auditable output. So the move is to embed those agentic steps inside of the CI/CD where they have access to the logs and metrics and all the telemetry and then you can establish guard rails. In this way, we can trust the system and not just the

vibes that things are going Now I think this is the important po point to the safety model. Humans are still accountable. There's still a ton of human engineers who are ensuring that the system is operating in the way that they expect it to be. This is the principle that keeps us grounded and it we start with agents that propose the changes, systems that verify those and then

humans are approving. That's the model we're using. Sometimes you can mix it up a little bit. the humans can propose and the agents can uh verify and approve, but at some point we're keeping humans in the system. And if you're ever tempted to remove the loop, inevitably I'm sure almost every process will get to a point where it's so refined and it can be. But before you

do, the question you have to ask yourself is, do we have enough verification to make that safe? Do we have enough guard rails in place to take the human out of the loop and truly end to end automate all of it? So I'd say in most distro contexts or many distro contexts the answer is not yet and that's okay. We're getting there. We'll get there soon. Now

I think open source and Linux cyros are actually an ideal fit for so many of the agentic workflows that we're talking about because our work is already structured around transparency and We've already we already build and test constantly. We're already have a whole bunch of policies in place to ensure that things operate in the way that we want it to or we expect it to. Uh and

we already care about provenence and reproducibility. So I don't think the leap is cultural at all. All at this point it's purely architectural. We need to connect the steps into intelligent loops that can continuously produce that verified output. So we'll talk a little bit more about that. Every aspect of Linux that I've been a part of going back to the late 90s has been CI/CD heavy by

nature. Uh and that's long before the term CI/CD ever came up. I think the very first open source project I ever contributed to at IBM was something called the Linux test project, LTP. It's still around. It still verifies every single CIS call inside of the Linux kernel. Uh that was CI/CD long before we had ever heard the term continuous integration and continuous development. Build systems, test suites,

reproducible builds, check some signatures. These are the things that we already have in place. That's our CI/CD culture. um long before we even called it that. Agentic workflows thrive in CI/CD heavy environments, places where we've already got the right place to plug better automation, more intelligent automation into that verify step is already present. It's a lot harder to go from 0 to 60 where there is no

CI/CD pipeline and suddenly a CTO shakes a magic wand and wants to make everything, you know, AI intelligent. If there's no CI/CD in place, that gets really difficult and very sketchy. When there is CI/CD present, you've got everything you need, all of the the logs and the history and the telemetry to simply pull that together and make it completely machine checkable. So, I'd also say that most

DRO task is already evidence- driven. version bumps, change log parsing, patch backports, policy validation, regression tests. These are all things that produce evidence that typically a maintainer somewhere goes and looks at and has to read if not they themselves simply using Grep or said uh to to work their way through some logs. Now, we can pump that straight into a system that's able to do it thousands

of times faster uh and more accurately. I would say those aren't very creative tasks as an an open source maintainer. Grepping through a log file is is that's not where I I find my joy working on one open source project or another. Uh but they are disciplined routines that must be done correctly and repeatedly and I think they make prime candidates for the agentic assistance. Those agents

can do the routine work and present the evidence in a way that we can quickly approve as the human still in the loop. And that brings me to the point that that repetition, sometimes we call it toil, is the opportunity really. We've built software factories, factories that run the same workload workflows thousands of times. And if humans are manually re-evaluating and reinventing the same patterns over and

over and over again either in one organization or across multiple teams in the same company or across you know the industry that's the signal that there's a factory that's missing there. I'd say that good engineering turns repeated judgment things that we have to make judgment calls on. Good engineering would turn those those repeated judgments into explicit checks and guard rails for what does it mean to ship

that software so that the next time is easier and the next time is even easier than that and so on. Dros are also more than just a pile of I look at the distros that I run and I use that in a a plural sense as systems of guarantees. there's a certain set of things that are guaranteed to be true within a DRO ecosystem. And when that

doesn't happen, then you've got a bug. You've got, you know, a violation of that of that system of guarantees and something's not working. But as long as those guarantees are tight and insured and verified, you've got a great experience as a user. It's policies, its tool chains, build graphs, dependency resolutions, and ultimately the secure artifacts, the binaries that we run. Now if you optimize only at the

package layer, you miss all of the system level leverage that you can get out of a an interconnected system. So largely distro engineering is supply chain engineering. It's everything that goes into making great software producing outputs with guarantees those system level guarantees consistently repeatedly and in an auditable manner. So this is the basic factory model. The input is upstream source code from tens of thousands of different

locations. GitHub projects, tarballs off of FTP GNU.org. It can come from anywhere. Processing it inside of the factory is the building, testing, signing, verifying. And then the output of that factory are artifacts with guarantees. And once you view it as a factory, you can optimize that entire process like you would optimize a factory. Reducing the work, tightening the feedback loops, standardizing the lanes that tools operate in,

and ultimately instrumenting everything for telemetry. And and as such, the goal becomes higher throughput with higher confidence. And that's how we build trust with users. And I'd say more importantly when you think about this as a factory, factories get smarter. They get better when they learn through those feedback loops. And that's why we in instituted that telemetry. Every failure of the factory should create a new check,

a new heristic, some new automated response. Once we figure out how to solve that one, never solve it again. And that's how you you move from we fixed it to we fixed it forever. And I think that's at the heart of sustainable DRO maintenance for the long run. Certainly what I want as a DRO So this is the feedback loop I'd like everyone to internalize if you

take nothing away from this talk. Build, scan, attest, and validate. Most distros do these steps, but it's it's not always a tight loop with consistent artifacts and consistent policies. What we're striving to do here is make this loop continuous, measurable, and increasingly automated. And I think that's how we get to the security and quality that users ultimately want. So let's take each of those. First, build. Determinism

is a precondition for trust. Knowing that you're getting what you expect out of a build. reproducible builds, multi-archched bootstrap tool chains, deterministic outputs, knowing that you you get out what you want it to get out. And that's where the verification comes in. If the builds aren't deterministic, everything downstream becomes harder to trust, harder to automate, and harder to debug when something goes wrong. So, I'd advise to

invest early in build correctness. Automation amplifies whatever foundation you give it, good or bad. You give it something good and you automate it. you end up with something great. You give automation something bad and you automate it, you end up with a lot of Scanning detection produces critical intelligence that better inform our automation. It's how we make our automation better. And scanning is not a single tool.

It's not one thing that you do. It's a category. Sometimes we're talking about vulnerability detection. Sometimes we're talking about static code analysis. uh license check, open source license checks, important policy enforcement. The key is that scanning produces evidence that can flow downstream into approvals and attestations or scanning can also raise very important flags, the things that need to be fixed, the the vulnerabilities that need to be

addressed. So treat scanning as a step within the supply chain rather than an afterthought, something that you do after the fact with something that you've received. And the best time to scan is continuously, not just once at release time. Attestation is about proof. If you can't prove it, how can you really be sure what it actually is? And a testation is how we have that answer. Who

built this from what sources with what tool chain and when? And we can get this through sbombs and signatures, provenence, and traceability. These are the facts that turn I I think it's secure into here's how I know that this artifact is secure and as distros are part of very critical attestation moves from a nice to have to something that is required and we see this at government

levels all the time now through Fed ramp and other security standards and then here's that word validate again if it's not continuously tested you can't know it to be continuously true. Validation closes that loop. Regression tests, compatibility tests, runtime validation, policy re-evaluation. Trust but verify, we know this in open source, but validation is where trust but verify lives programmatically. If you want to move fast without breaking

things, make your validation steps happen all the time. So now we embed agents into that loop. It's the agents that can propose version bumps, run the validation loop, sum up summarize the drifts and produce those artifacts and that system becomes high signal by default with humans seeing the evidence just the evidence that matters and not the noise all the other noise that comes through that the system

can process for you. So I think that's the productivity unlock for the human engineers on the team. Reviewers spend time making critical judgments based on good information, not reconstructing the entire context of everything that happened through logs and scattered metadata. They only have to do that on a very small percentage that need super expertise to debug. Now, this is where things just get better. Agents continuously improve.

They're compounding every single day, a little bit by little bit with all that telemetry and data that you're sending it. The big idea is that the factory improves with every run that it makes. If you set up the feedback loops correctly, it just gets better over time. Now, that's the incremental improvements. But then sometimes occasionally you get the big step function changes where the actual agent infrastructure

improves. The AI behind you rolls out a new model and it jumps an entire level. With model improvements, you can use the same pipelines, the same code, the same everything that you've done day in and day out for days, weeks, months, and then one day a new model takes that to a whole new level without rewriting any code. And that's pretty incredible. So we get this really

nice flywheel effect. If we have the telemetry, we can develop guard rails. If we have guard rails, we can enable even more automation in places that weren't safe before because we have those guardrails. With more automation, we can get additional quality. With more quality, we can drive larger adoption. We can roll things out into production. We can push things out to additional users. With the additional adoption,

if we instrument it correctly, we get more telemetry and we're right back to where we started, where the telemetry is driving every step of that flywheel. It's a compounding system and I think it's an important safety system, too. As trust increases, you can automate more system has improved and matured. Now, security improves with quality because quality reduces surprises. Better testing means fewer regressions. Better evidence means fewer

blind spots. Faster feedback loops mean less exposure And so, we don't have to treat security and velocity as opposites. We can design the pipeline so that quality is the bridge across all of this. And we just end up with a more secure system when we wake up every day. So let's make this actionable. Here's a handful of real patterns that you can use. Mostly I'm talking to

the other DRO maintainers and open source maintainers out there. Um but you can adopt these patterns incrementally almost anywhere that you're developing developing software any packaging ecosystem uh or any other internal tools that you're First, I like to start with the grind, the toil, the schleing, the things that you have to do but isn't terribly fun. Triage, backports, repetitive work. For some people, it's documentation. For other

people, it's debugging. Uh, for some people, it's right in the heart of the CI/CD systems. Whatever task consumes your attention and context, but it's not something you terribly enjoy doing, start there. Try to automate there and see if you can yield some immediate leverage. And if not you, if you're a leader of a team for your team, try to make the agents do the repetitive steps, but

make sure that the agents are producing evidence that your humans can red review very quickly and efficiently. I'd also recommend that you adopt a propose improve standard. Every PR pull request should arrive with evidence. What changed? What passed? What risk remains. This is how we get into humans as the reviewers having exactly the context they knew to make a high quality reliable decision and understand what if

any risk remains. When we've done this well, we've turned reviews from really annoying archaeology into very quick, very reliable decision-m and I think it scales. High signal diffs let maintainers handle even more change without having to lower the standards because they're going through so many. I mentioned guardrails a couple of times. Build the guardrails early. The guardrails is what allows you to automate things that you weren't

able to automate before because you didn't have safety uh guardrails in place. Reproducibility is important. I think policy enforcement, tests, anything that can prevent drift from an ideal outcome. It's the guardrails that make agentic system safe and is what's going to allow us to do this essentially everywhere. And so if you invest in the guardrails first, you can automate even more with confidence and fewer surprises. Instrument

everything. Measure everything. Measure the toil, the cycle time, the regressions, the patch speed, the data over the vibes. If you can't measure it, it's really hard to improve it. And if you don't know if you can improve it, it's hard to justify automating it or investing in it. Metrics can also reveal where your factory is leaking effort. Bottlenecks, flaky tests, frequent failure modes, burning tokens that shouldn't

be burned and keep humans in the loop with approval gates, transparency, escalation paths, and failure review. The trust is going to come from visibility. Visibility of what happened, why it happened, and what the evidence says about it. And when you build that transparency in, you can move fast without losing the accountability. So what's all this mean in practice? I think it means shrinking exposure windows, which is

back to that security angle, reducing the dependency on human heroics. You probably know the people who've had to save the day for one incident or another and building systems that improve daily automatically through this agentic approach. If we do it right, the maintainer time shifts from the repetitive toil to much higher value, more creative endeavors, a lot more fun Bootstrapping from source is about controlling your tool

chain and your Harden builds and deterministic supply chain practices reduce the ambiguity and ensures that you know what you got, how you got it, and who built it and when. So whether you're building a general purpose dro or some other software, the tool chain trust is foundational to that And we got to shrink those exposure windows. This is security velocity. Faster detection, faster backports, faster rebuilds, faster

delivery, fixes for things that we didn't think were fixable previously. And when those pipelines are tight, the remediation loop becomes routine. Routine remediation reduces the risk everywhere. Less heroics, more systems, fewer brittle scripts and late night scrambles to fix a thing that you didn't even know needed to be fixed. That's where the repeatable automation makes open source maintainership even more sustainable, especially as we pass this on

to the next generation of open source maintainers, especially as those ecosystems grow and take on even more uh more features and more work. the best best systems and let your engineers have confidence that the pipeline is doing the work and surfacing only the most critical decisions and it's automation for the maintenance taking care of those build failures and test regressions the dependency churn the mature approach is

to convert manual fixes into automation so that every incident actually improves the system this was the we never see the same we never have to fix the same problem more than once. Over time, the factory becomes more resilient and humans get pulled in less and less often and only for the the genuinely novel cases. And so that's how I think we can actually wake up to a

DRO that just gets better. New upstream releases that are integrated every single night. Refined guard rails that keep the system highly trustworthy. Updated models that are happening in the background driven by uh telemetry and tuning data. That's the compounding factory idea. The factory that just gets better the more you run it and you keep moving with an ecosystem without burning people out. So this is exactly what

we've built. These are the principles that we used to build chain guard OS. It's a small distro that we've put into our harden container images and our hardened libraries. This isn't uh this isn't a theory. We're running this every day. Everything that I told you I talked about here today is exactly how the backend chain guard factory works. We literally call it the factory. So all of

the factory analogies are very near and dear to our hearts. We look at the back-end system as a factory that takes input, refineses that input, and produces secure uh artifacts. But the point isn't about chain guard. It's really about this pattern that I think it can be adopted anywhere that's building software. So if you take nothing else away, take these patterns away that agentic pipelines are leverage

and that you can continuously improve software by building a software factory. build, scan, attest, validate. Do that over and over and over again in a loop. Propose and prove. Take your PRs to a point where they come with all of the evidence you need as a as a maintainer to make that decision. Invest in the guard rails early and instrument everything relentlessly. I think that those patterns

can be applied to any Linux distro, any open source project, any proprietary or hobbyist project that you have at at various scales. Anywhere that you want to turn repeated work and toil into a repeatable system and then keep improving it as it goes along. This is where I think the industry is heading and I'm I'm really quite excited So, in conclusion, I think you can achieve this.

I think we've achieved this with the DRO uh the containers and the libraries that we've built at chainard. I think you can achieve this anywhere. Fast by design, secure by default and better every single day. With that, I want to thank the organizers and sponsors very much appreciated and everyone here for joining us. Uh do I have a few minutes for questions? It doesn't. >> What's up,

Kyle? >> Is it is this on? All right. Well, um yeah, thanks for the talk. Uh so this is about the verification step. So um in the verification step, you know, you have an agent that is producing evidence that's going through the verification step. But I guess my question is um is in that case it self attesting to the evidence or is there some notion of maybe

like a notary or an auditor that audits the the evidence that it's producing? >> Yeah, absolutely. You've got to like the verification is both what it's supposed to be and then a judge that it makes a call on is this is this right? In some cases the deterministic uh state is selfattestable. Right? Right? If there's a right answer and there's a checksum for that right answer and

it either matches the checksum or it doesn't, sure there's that's a perfect perfectly deterministic system where there is a judgment call that has to be make. Hey, is this good enough? Does 99 out of 100 tests is that good enough or or not? There may either need to be a human in the loop or perhaps another system and there's two separate systems that are able to to

weigh uh against one another. Mostly what I'm talking about when I'm talking about verification is objective verification. this passes or it fails. It's true or it's false. It's faster than a certain threshold or it's not as fast as a desired threshold. >> I guess I mean there's there are cases where you can like cryptographically verify like you can't invent the check sum that would match or right.

Um have you found any cases where you are you are having an agent say that something >> and providing evidence that says yes this this returned and it's not true. >> Yeah, without a doubt. I mean hallucination is is still there. I mean it's in my experience having been you know playing with first chat GPT then Gemini and now Claude I've seen each of them progress from

here's something that I'm confident about and I think it's right or I know it's right is what you know chat GPT would tell you and you look at it and go no I don't think so it goes you're absolutely right I'm wrong let me tell you something different right we saw a lot of that um we saw a lot of that because we were asking it whether

Betty White was alive or dead right we weren't asking it to write a Python script Right. There's a there's a big difference I think as we move into the world of of code of logic of things that do have provably true or false states and something like you know I'm not advertising for anthropic and claude but I'm a big fan uh I think it's come a long

way to producing um producing code producing systems that are that it can also internalize and prove whether or not it's true or false. Does it still make mistakes? Yes, absolutely. I'm running multiple jobs right now and I'm telling it, "Sorry, you got that wrong. Go try again." Oh, yeah. You're absolutely right. I uh I got that wrong. Uh I usually encode those as skills. Like if if

I'm teaching Claude something, I'll tell it go write a skill that makes sure that you never make this mistake again and store that in your in your skills file. Bit of a long-winded answer to say they're far from perfect. I still think human in the loop is important, especially for mission critical systems. Um, agents can make mistakes, but boy, the more we can get this to be

math problems and less fewer literary opinions, I think, you know, you end up in places where you can automate this and stay completely hands >> Awesome. Yeah. Thank you. Thanks. >> Any other questions? >> How do you see closing the gap between the builders, distributors of the systems and the users of the system? because both are interested in regressions. Do I want to upgrade this DRO to

the new version? I've got my own set of stuff I'm putting on top of it and that may highlight a distribution problem or or vice versa. And also you want to know the change set that's coming down whether it's going to impact your system. So there's value on both sides and is there a good way to connect that value >> connecting the value from the >> the

the users of a distro and the producers of the DRO. >> Yeah. So I'll go back to telemetry. You know, I think most most distros have a option. Do you know, do you want to send uh anonymous telemetry data or not? And everyone should make that choice themselves based on, you know, how sensitive that system is to them. Uh and the answer is sometimes yes and sometimes

no. And it might be yes for this system but not for this other system. I think at a like at a grand scale, the more the more users opt into sending some or all telemetry data, yeah, those systems are going to get better, sooner, faster. Um, I think there's a value to that to the downstream user. It's hard to draw a direct correlation between I sent all

of my data to it and my use case immediately got better, but certainly in aggregate and at scale, it's unmistakable to to me. Uh to me as a, you know, a producer of this stuff, boy, when we have the data and we know that something is going wrong at scale, we can do something about it. When we only hear about it anecdotally, someone, you know, is annoyed

and raises their hand and says this is a problem, we can go and chase that down. But man, we could have seen that a lot sooner when we've got that telemetry. So I think that's part of the that's part of the exchange. We can all have better systems if there's, you know, some amount of anonymous telemetry data that is contributed. >> All right, Justin. Thank you. >>

Okay. Um, at the top of the hour, we'll we'll have um another um >> nice >> uh presentation. Um Scott Mayab will be talking about Gen A for the X-Gen guy. And then also uh Oh, there you go. And also um >> Oh, full. >> I'm always early. >> No, yeah, that's good. So, the thing is wanted to do >> And also there's the wireless So, where

is that? turned off. Yeah, that's why hold this. That's the thing with the things you can't see at the same time. >> Hello. Hi. >> All right. And then I'll get my laptop up and we'll check that out. Or are you Mr. Sponsor Man? >> I am Mr. Spencer. >> Want to test? Do it. Do you want me to step down so I don't ruin your pictures?

It's a good one. I have that one. You're good. I'll get the This is live. So, if I'm not replying, it's because the mic is live. There's an on and an off button. There's no mute button. >> It's very efficient. >> There's the um handheld mic over here. >> I'm gonna hand this to you. >> Okay. Uh we're about to start with our next talk, but first

uh one of our sponsors, Matt from Edge Delta, has uh some stuff to show and talk to you about. Give us your best multimedia minute, Matt. All right, everybody. One minute. Hold on. Starting the timer. >> All right. DevOps Day is LA. Welcome to Edge Delta Worldwide. >> Worldwide. >> Quick question. Show of hands. Who here has been paged at 3:00 in the morning uh and had

to stare at 14,000 log lines to figure out what was going on? >> 14,000 lines. That's a haststack made of smaller haystacks. And somewhere there is a needle. And that needle is why your checkout service is throwing 502s. and you're scrolling, you're gata dog open, you've got Splunk open, graphana dashboard that Kyle built four years ago. Kyle's gone. He left his stripe. that's you right there. That's

you sleeping. It's 3:00 in the morning. That's your AI teammate. That's your AI teammate handling it. Anomaly spikes in your P99. Your AI teammate sees it. Not tomorrow. Not after standup. Right now, it says, "Here's the deployment 4.72. It found the issue. Here's the endpoint. Here's the commit. It does the investigation for you. Instead of playing detective with browser tabs at 3:00 in the morning in your

underwear, we've all been there. Your AI teammate has triaged it already, handed you root cause before your coffee maker turns on. >> And the noise 10,000 alerts a day. How how many of those matter? Maybe 12. I don't know. 12 alerts. The rest is health checks, duplicate warnings, Kubernetes pod that restarts every 45 minutes. AI teammates learn what's the noise. They bury the one alert. Your payment

service is on fire. It's like having an S sur that's the best on SR on your team. And it doesn't quit to go to Stripe like Kyle. AI teammates investigates in real time. That's Edge Delta worldwide. >> Come see us. We're already outside the >> All right. Thank you, Edge. >> That should get him going. All right. How wide is that world? >> Okay, thank you Edge

Delta for um sponsoring scale and for that >> multimedia presentation. Okay, next up is Scott May. Um I guess I'll need a second to plug in. Um >> indeed >> also I think I've mentioned before a few times on Sunday there is a track called open source career day. It's over in the other building. Uh if you're you know looking for that next position um there are

presentations and also uh people you can consult with about your um resume or what have you. So, um at the um LA DevOps table right around the corner, uh Bria is there. Uh you can ask her anything or specifically about open source career day because that's what she uh she's running that track as well as as this one. Okay. Gen I Gen AI for the Gen X

guy. >> It's so >> Scott, are you ready to go? >> About as ready as I could ever be. >> Okay. Hey, take it away, Scott. Thank you. All right. Uh, I have a clicker because I'm fancy. Uh, so yeah, as uh was previously sold and advertised, uh, this is Gen AI for the Gen X guy. Uh, I'm this guy. I'm Scott. I work at Data Dog.

I am a Gen X guy. Uh, so I am highly qualified to talk about all things Gen X because I lived it kind of um, and survived to tell the story. Uh I'm also a member of an organization called merge forward that's part of the cloudnative computing foundation. Uh it's a communitydriven initiative that where we support underrepresented people groups allies. Uh includes people like myself that are

hard of hearing uh have multiple things like arthritis and stuff because getting old is a drag. Um yeah I see one camera out so I'm going to take a little pause make sure you can get that. Cool. Um, yeah. So, Gen AI, uh, Gen X rather, uh, we're known sometimes as the forgotten generation. And one thing as a child that really caught me as weird was every

night on the news was a PSA that was asking parents if they knew where their children were. I thought it was weird as a kid and I think it's weirder now because I recently looked this up and that was supposed to be the fluff piece at the end of the news and that's supposed to be like the hey we're happy. Do you know where your kids are?

I don't know. Kind of scary to me, but that's what it was. So, Gen X, we came home to empty houses. There was no parents there. Uh we would ride our bikes all over the place. We drank from the garden hose. We figured things out on our own. As we see here, we were pretty darn self-reliant. We didn't always have instructions. We made things up as we

went along. We were resourceful. We used our imaginations to get things done. And uh, you know, fake it till you make it. Still exists today. I still do that quite a bit. And being skeptical. Uh, we learned early on that trust everything you hear. Um, if you were in the Gen X generation, you would hear all those crazy stories from your brother or some other older kid

in the neighborhood that was the source of misinformation at the time. But thankfully, we have the internet for that now. And if anything sounds familiar about what I said, um, you could probably thank Stranger Things for helping some of the younger folks understand what life was like back then. We could disappear for hours at a time and our parents really wouldn't care because we weren't home. We

were out doing something hopefully not too terrible. Um, and of course like Stranger Things but without the demogorgans. Uh, just other scary things. We would build our own bike ramps and broken bones that So, of course, raised by pop culture. Um, there's going to be some pop culture references throughout. Uh, we had some really cool things. We had MTV when it played music videos, Spielberg movies, The

Goonies, uh, and Montipython, uh, which according to some people here would be the most acceptable form of Python. Uh, Saturday morning cartoons, which was just basically long advertisements, uh, things like, uh, G.I. Joe and everything like that. We had all those crazy TV advertisements. Coke versus Pepsi. It's like you never knew when you're going to go to a mall and there'll be the Pepsi challenge where they

wanted you to choose your favorite drink. But we also learned that you have to question things. You have to wait for the proof that something is legit. And our parents told us that TV would rot our brain if we watch too much TV. Uh unfortunately, I think sometimes that TV has rotted our parents' brain a little bit more than ours. And all that kind of led me

to be because I needed to be skeptical. And the ultimate thing about that is not to be a contrarian. Uh nobody likes a contrarian. If you're skeptical, that's cool. Ask questions. Question the answers. Do that frequently. And the reason why is because trends, they come, they go. Uh there was we've lived through a lot, right, in Gen X. Uh it's unfortunate that some of you all didn't

believe us in the past and y'all brought the mullet back, the really bad short in the front, long in the back hairstyle. Terrible hairstyle and you decided, "Hey, that's cool. Let's do it again." Terrible fashion choices coming around all over again. And in the tech world, there was the dot bubble burst, which did not kill the internet, thankfully. Um, that's how I make my money. Uh, in

2015 I was told the cloud is a fad and that's why that company could not give me a raise. Um, thankfully I moved on. Fast forward to now, here we are. Uh, everyone's going to use it or no one will. Uh, NFTTS, if you like digital beanie babies, that was a thing. Uh so the thing with the Gen X folks like myself, we tend to watch and

see what is and is not working and we try to go for that. Uh we adopt things when it's time and we move on. And that brings us to here we are today. Uh the inescapable rise of Gen AI and I feel fine. Uh which is a deep cut to an RM song. Uh I think you got that one. Thanks. Thank you. We're in this together. Um,

so here we are and instead of the generation of millennials killing things, it's Gen AI killing stuff and not just Applebees. Um, you know, for fun, I like to use AI tools to uh build some fun graphs and visualizations. As we can see, uh, baby boomers are lagging in terms of AI adoption. Uh, Gen X and the millennials were kind of tied, which is pretty cool. And

Gen Z loves to do some work with AI. I did a uh Gemini deep research on this topic and it actually ran for about three hours. Uh when I say deep research um thankfully I don't pay for it. So um I was able to get away with it. Um and now as we know you cannot escape Gen AI. It is literally everywhere. Uh, it's everywhere from our

chats, our browsers, idees, it's in our phones, um, if you're like me, you possibly have Gen AI turned off on your iPhone or Android. Um, and art things like nanobanana exist to make images. I did one nano banana image and this it was the uh the update of the uh 10pm. Do you know where your children are? I tried to use the screenshot that I could find

of that original image and it just looked crappy. So I said, "Hey, I'm doing a Gen AI talk. Make it look better for me." even though this is labeled as a Gen X guys reaction to Gen AI and the things going on around us, I think a lot of us can probably relate to some of this. Um, I have a pretty odd sense of humor. I was

originally going to go with the stages of grief, but I thought that was a little too dark and I decided to go a little more positive. Uh, our first stage is fear. Um, where things aren't going to go well and we don't know. And I get that the fear is real. Um, this AI thing, I think three years ago, I was like, "Hey, maybe it's a fad.

It's going to go." Uh, but it hasn't. And we don't know what's going to happen next really in terms of all the who will and who won't be impacted. Unfortunately, some people have lost jobs. The next up, distrust. Uh it's healthy to not trust things, especially in the tech world. You don't know what's going to catch on, what isn't. You don't know how it's going to impact

you. Um you know, not everyone needs to run Kubernetes to host their static blog. resentment res resonates for me the most uh spending so many years doing a lot of these system administration stuff taking care of YAML files puppet configs terraform and so on and now an AI can just take that over for me but the interesting thing is I don't have to give up what I

enjoy and you should not so finally is acceptance I do have some Gen AI or Gen X or Gen AI Okay, sorry. Now I'm mixing up my gens. Um, yeah, I don't really like to write infrastructure as code, but cloud code, etc. is really good at it, you know. So, um, I don't know if anybody in the room here is from Microsoft, but if you are from

Microsoft, I've already gone by the booth once and I have asked if you can please, please bring back Clippy. If anything, I think now is the perfect time to bring him back. And I'd like to thank you all for being in the front row. That's what I like to call the splash zone, by the way. >> You'll learn later. All right. In some ways, it feels like

I've lived through this quite often before. Big cycles of hype, big disappointment, big boom, big crash. Uh, the internet, it was supposed to eliminate misinformation and make information free and accessible to everyone. Kind of like a library that you can have 247. Social media, it's going to bring us all together. We're going to be one big happy family and not like some rage internet thing going on

on Reddit. Containers. Yeah, that's how I can ship my entire laptop to production. It's going to be the same. It's going to be equal and great. And besides that, uh, platform engineering a few years ago was touted as the thing that was going to kill DevOps, but here we are, DevOps days, I love you and I always have. Um, and like all those claims that I mentioned,

like the things that would kill DevOps, um, it's good to have proof. And I don't want to come off as Abe Simpson here being the old man that yells at the cloud, but it is important to understand what we're working with and how it affects us overall. And when I say I need proof, that's like I need to get in. I need to figure things out on

my own. Camera guy. Um, and that takes us to my way of doing things, uh, which is to be pragmatic. Being pragmatic is almost similar to the Gen X vibe of whatever, I'll give it to I'll try it. Why not? So, being pragmatic is a big part of my career, especially at Data Dog. My VP says it all the time. Well, just be pragmatic. It's kind of

like the community team vibe. So, and since this is DevOps days, this should resonate with everyone. Test often. Bake tests into everything you do. Measure the value of what you're doing. Does it impact your experience positively or negatively? Whatever you're doing, any technology you're adopting at all, what can you measure? How can you measure it? Does it impact your customer's experience? negatively or positively. Beyond that, iterate

what works for you, adopt that. Improve it. If it doesn't work for you, get rid of it. If it brings a negative thing to your customers, get rid of it. Kind of like uh you know how Meta does the uh feed out of order and you have to keep going back and clicking the button. I digress. All right. So when I talk about testing and gentleman before

mentioned how AI is very very confident when it tells you looks perfect. It's exactly correct. Don't take that positive reply as gospel. This is an example. Uh two years ago we were preparing to go to a conference at Google conference in Los Angeles Google Cloud Next. I was helping somebody prepare for their presentation and their demo and I was trying to ask questions about how many virtual

machines do I have in my VPC? I'm sorry, I don't have access to that You live in this cloud that I'm asking Then I up the permissions to see if it could do it. And it's like, nah, I'm not going to tell you. It's up to you to figure it out. basically. Uh, appreciate your patience and understanding, jerk. And the next thing is going to be great

for me, but because I am who I am, I have a sense of humor. I grew up with movies like Terminator. I needed reassurance that I was going to be spared during the Great Robot Uprising. I'm going to be safe. I don't know about you, but I'm okay. It's kind of like Braveheart. The Almighty said, "I'm going to make it out of this." Okay. But So on

that trust but verify idea, I built a simple bot uh to do things like help me find donuts anywhere in the world. Um I call it Homerbot. It is a simple nod to my favorite television dad. I built it for a meetup group that I was doing where we were talking about security of vibe coded things and how to make things more secure and I fed it

the OASP framework around um like prompt injections and all these other things. And we did try to hack it. I let the attendees actually drive my laptop to see if they could hack it and thankfully they didn't. Um it only ran up my bill by like $1, so success for me. Um there was quite a few hallucinations along the way. Uh still has a long way to

go. I will still a line from John Willis who was here earlier this morning that prototype does not equal production. Definitely not production ready but just a fun thing to build when you have the time just to make sure that you can question the answers if you will. I did mention hallucinations. Hallucinations and data quality. That's a problem that you need to solve. So, here's a few

tools that I have that I've tried uh just to make sure that you can test out some things. And if you're curious, there is going to be a QR code at the end. I see the the cameras. I have a QR code at the end. So, you can have a link to my GitHub repo where I've linked everything and make it a little easier for everybody that

way. All right. So, a Gen X punks rules for AI. Um, I love punk rock music. Uh, if there's one thing I want you to know about punk rock music that it's about three letters mainly, do it yourself. It's uh, you know, there was a lot of DIY shows when I was growing up. There was a record label that started in Washington DC near where I grew

up called Discord. It was all about do it yourself and help out your friends. So when we're building things, how do we build and measure what matters? That's going to be important. I work a lot with model context protocol tools. Um it helps me do things to be able to build things that are going to matter to my people. So I can use an MCP to say,

"Hey, I'm going to be doing a thing in Pasadena, California. Do I have nearby customers that I could say hi to or take out to dinner or something?" So MCP helps that out. I use uh Snowflake for that. So I reach out to that and I'm like, "Hey, we have all this data." Uh or another way that I do it is I need to find people that

can talk at a conference in Germany. Who do I have that works in Germany that speaks German that can do this? Another thing that is helpful with some of these things that I'm doing with AI, again, because I never learned how to do really good SQL, um, I'm now too old to learn, I think. I'm kidding. Uh, natural language queries are pretty darn cool. Uh, being able

to ask a question of a data set in my common language and get an answer that to make it easy to graph data is pretty awesome. I do that a lot. a lot a lot actually my next talk I'm doing after this is tons of data just around that and build something for fun build something for you uh Homerbot was something for fun I love donuts it

only answers about donuts um and of course when you work with these tools as I do one of the cool things that I've built the using claude and MCP is just a dashboard within our platform to help me understand my fake web store, my demo environment. Um, and this helps me in a sandboxed way understand things that I could build, what matters, and use all this telemetry

that I have because there's literally so much telemetry there. Uh, this example I use clawed and RMCP. I took logs, uh, service discovery, traces, and everything and "Build me a dashboard so I know what's cool." And it did. Um, a little bit of trial and error, uh, fixing things along You're going to hear a lot about this, uh, everywhere you go. Uh, human in the loop, human

on the loop, whichever you like to say. While you're building uh the important thing there is you make the decision, you set the outcome. You own the outcome is what the important thing, right? So if something's important to you or the organization you work for, don't let a machine uh unfortunat don't let something else choose the outcome for you. What the tooling should do for you is

just generate some output, process your data, uh suggest some possibilities. Uh build some guard rails into things you do. I build system prompts for when I build a chatbot. So things like that, those guard rails, those skills, they're going to be helpful. Another cool little app that I built is one called Good Dogs. Uh Good Dogs. Uh, somebody explained it to me is hot or not for

dogs. Um, I never used hot or not, but uh, basically the whole entire point of good dogs is that you choose if a dog is a good dog or a bad dog. Uh, if you choose good dog, you're rewarded and you learn that you are correct. It is a good dog. If you click bad dog, the answer is always you are correct. This is a very good

dog. They are all good And the reason I built this is I demo a lot of things and every time I demo something, it's usually a web store. I wanted something different. I wanted something I could build easily that has tons of telemetry so I can see like how people are voting, where they're coming in from, how everything works. And uh using an MCP, I can launch

on any cloud I want within a minute or two. any cloud on my of my choosing on demand basically and that's why I selected good dogs because it's a very lightweight app uh and my friend just sent me pictures of her dogs that I used for the application. All right, learning from failure is a core part of that Gen X vibe. Uh when you fell off your

bike, you had to get back up. We didn't have YouTube. have like always those instructions we had to learn along the way. We screwed up a lot. Uh I still do. It's part of the fun for me. Uh learning anything I try to make mistakes on purpose. Um, I run a lot of workshops uh with people and the number one error that I find in a workshop

is people will skip directions like from going from step two to 10 just because they didn't want to read all the flavor text. It happens. Learning from failure is important. It's also important to impart the uh punk rock rule of life into things. So that is if the person next to you or near you falls down, you help them back up. And that's important beyond just the

circle pit. It's also important in life and everything else. That's why I, you know, I mentioned that merge forward thing. I always want to pick people up. If you can reframe any doubts you have as a growth opportunity, um, I don't want to be an old kermagin. I don't want to be the old guy. Ah, it's Um, is AI overhyped? Yeah, it is. Most definitely. Um, I

can't argue that mindset is And when I say mindset, like if I go to take an exam, if I think that I'm going to fail or do poorly, chances are I It's never going to be as good as something I make. Well, true, but where can it help me? What might surprise me? Um, and then of course, I don't need it. Um, what can I do with

AI to make it useful to me? Um, I'm happy that I work with no one in this room because I did have a really weird use case for uh some AI that's built into Gmail where I built a little function that marks every email that hits my box as red. So, I'm always at inbox zero. Um, I hate write I hate reading email and if it's important,

someone's going to reach out to me on Slack, but that's me. Um, if you have to read email, please don't be like me. Here's some ideas that I thought of and it's a interesting thing there spinning wheel. Um, safe exploration things. When I'm working with anything new, I think of things that can help me do it for free. Uh so when you're trying to experiment with some

of this stuff, there's a local LLMs that you can use. There's free tiers for chat GPT cla and everything. Use those to your advantage. Um especially kind of like AWS, right? You get those uh free account when you use a different email address. So if you have 30 different email addresses, you get 30 different free trials. Um I'm not a lawyer, but that's legit. verify how things

work, right? Uh this is DevOps days. You need to have that verification loop. We need those fast cycles of information getting back to us. Generate, edit, learn, do your And sandbox tasks. Um if you're concerned about security of an LLM or anything like that, you can run your uh in a virtual machine either like a EC2 uh a VM that you run on your own machine there's

uh tools uh such as cloud VM uh the aformentioned I love this name no no uh a kernel level enforced sandbox CLI for all agents and um the docker is always a great way to figure things out and if I move on before you get to capture an image of this. I have the QR code at the end that'll just go to my GitHub repo with all

the links to make it extra easy for you. So, don't worry too much if you didn't capture it. And when I start a project of any type, if it's good dogs, Homerbot, anything else, I always start with what the outcome is first. Uh kind of taking on the role of a bit like a product or project manager. Um, I use AI for scaffolding a lot. Um, so

that scaffolding is going to be things like Terraform, uh, cloud and other stuff. Other cool things that I like to do with AI, uh, Claude in particular, I use cloud code a lot, is to make sure that I set up a GitHub repository with things like a git ignore file, a docker ignore file. I build in a code owners file. Um, and I just make that part

of my repetition of everything more security is always better. And I did mention the scaffolding. Um, what that often will come to look like is ways that I can launch things on the fly. I do a lot with a sample app that we call text stories. and I dictate my outcomes in the form of read me files, and more. I often will throw in like some things

from OASP. Um, I've often thrown in PDF files to dictate some of the outcomes to help me along the way in those out uh areas. How can we make AI our own? Um, getting close to my wrap-up. So, examples of how you can make AI your own. If you're like you can inject some humor. Um, if not, go with the things that dictate your style and needs.

When I say style, that's like either code style, if you have any guides that you can feed the model to let that help guide things. the quote at the top is fun in my opinion. It's uh there was a television show in the 60s and 70s that was broadcast in the US on MTV, PBS, and more. Monty Python's Flying Circus. And there was a bank robbery sketch

where John Cle walks in. He's like, "I want all the money." And Michael Palin's like, "No, it's a lingerie shop." John Cle quickly says, "Adopt, adapt, and improve." Uh, this is actually a quote that has stuck with me since I was a kid. It rolled off at first as silly and the fact that I'm over 40 years since first hearing it, it's a really cool thing. So,

when we say adopt, start small. You know, never boil the ocean. Find a way to get some new skills and insights which helps you adapt. Um, scaffolding, like I said, that's going to help you build everything. Uh, set your rules, set your boundaries. Uh, then there's also improve. My goal is always to improve whatever I do for whenever I'm working with customers. Uh, and when I say

customers, customers is a broader term for me. Uh my customers are people that uh not only subscribe to my company uh my customers are also internal at the company. So I have a lot of people that I work with that I help them with tasks throughout the day. Um they're my customers too. Everyone in this room is a customer of me because you're here listening to me

yap a little bit. All right. In summary, uh folks like me, uh Gen X folks, we were around before the internet. Yep. There was a world before the internet. We adapted. We adopted. Uh we've gone along with some of the major tech vibes over the last 40 years. We have experience to do what matters. Uh and now you too can translate the world based on what we're

getting from AI. Kind of be that bridge to what's next in life. Um not Nano Banana. uh an internal hackathon image that I fed our dog into and said, "Make him kind of punk. I I want a punk dog for my presentation." All right. Everybody here in this room, at this conference, everywhere you go in your life has the ability to be a positive change in the

community at large anywhere. Uh fun fact, if you want to be a leader, you don't have to be a manager. Everyone could be a leader on their own. Um, that's my vibe. I am a cultural leader at my company. I expect a certain level of everybody like kindness, respect, and all that good stuff. So, I like to teach people things. That's why I run hackathons. I run

workshops. I do all that stuff. I help people learn where they are. uh locally in Baltimore where I'm from uh the organization code in the schools. I go there and help city kids learn how to do computer science because it's fun and why not influence shape the way things are around you. If you don't like your surroundings, change your surroundings. If you can't change your surroundings, Sounds

funny. So, if you can't change the way things are, maybe go somewhere else is what I'm saying. Um, mentor, help people like do the thing, right? That's what it's all about. Um, just because I got to be high up on the ladder where I am doesn't mean that I get to kick the ladder out from underneath me. I want to make sure that everybody else has that

ladder and opportunity behind them. And because I wanted to go semi-local to where we are, um, I also had this as a tattoo. Uh, two kids from nearby Sand Deas High School football did say to be excellent to each other and thank you for your time and everything today. This is that QR code that I mentioned. Um, so it just goes to a simple GitHub repo. Uh,

it's just a markdown file. It has all the links of everything I mentioned before. And yeah, I'm really hyped to be here. Thank you all so much for listening to me and being really cool. San Francisco, >> great talk. >> Uh for I have three questions. >> Oh jeez. Um, how do you help to inform um, try again >> the Gen Xers that I know and people

older and stuff that love to impart stories about how they were scarred in the past. >> Just because you had a bad experience last week, a new model came out, you could have a diff different experience today. How are you encouraging people to try again? >> Oh, wow. Um, I'm a big fan of music. Uh, I was listening to a song by a band called Cohed and

Cambria yesterday. Um, and there's a lyric that's is the answer that you want in the question that you ask. Um, how so how do I encourage people to do that? Um, I stay positive. Uh, it things change like you said. It may get better, it may get worse. You don't know. You have to try. Uh, and then also mindset. Mindset matters. Do you want to be stuck

in that I always say no mindset or do you want to try something new? Um, how I go about it, I I'm stupid. Um, I'm the first guy that will like pull something out of the oven and try and pick it up and I go, "Ow." I do that all the time. Do I learn from it? Yes. Do I still do it anyway? Yes. Um, keep trying.

Uh, there's no adversity. You can't I mean, you have to try. Trying is the first part ultimately. That's my answer. >> Okay. Appreciate that. Second question. it's not the tool, it's your prompt, bro. How do you encourage people to not negatively see the tool as the problem? >> It's their approach and that it's your >> How do I get people to approach it as it's not the

tool, it's your prompt? >> It's where I I like to start with like what is my outcome? What are you expecting? So, what are they expecting to get out of it? Um, I'm assuming that they're giving a maybe like a a very long prompt or just a malformed prompt. Am I close on that? >> I see people trying to oneshot stuff all they're just trying to go

in and just keep adding to that that set and that's usually not the way. >> Yeah. It's like watching Food Network, right? and they would do uh an exercise where you would try to talk somebody how through how to make a peanut butter and jelly sandwich and if you don't tell them pedantically to do it the right way, they won't do it the right way. Um that's

tough >> because I've used like with my mother who's in her 80s I say inch by inch and she has been able to adopt the inch by inch approach to >> it's your prompt row. I mean, you have to start small and you have to break it down into small pieces. Ultimately, that's the DevOps way if we're going to be like agile and do all that stuff.

So, you're not going to build a data dog clone with one prompt. Can't you can't start with something that huge, you're going to get terrible outcome. So, maybe they could start breaking their prompt into smaller bite-sized chunks. Um, that would be one way to go about it. >> Great. Uh, last one. Promise. >> That's all good. It's good. >> With regard to So, you know, there's all

these testing frameworks that you're offering up and that's cool. Don't munch on my tokens, bro. How do you limit like I wanted to run an with some co-worker whatever that's maybe some that's going to spawn a bunch of sub aents and if I leave if I let it run for an hour it's going to munch on my tokens. How do you how do you manage? Don't like

no cookie monsters. Don't >> Oh my god, that's a tough question and I don't have a great answer for it. Um, how do I manage that? I'm lucky. I I get a subscription to claw view work. Um, and we're encouraged to experiment for now. Uh, but how do we do that? We measure uh so we use the logging output of claude to eventually measure our token utilization

and then we map it over time just to see how things are. Um, I would work in your favorite observability tool if possible to be able to measure that. Um, the example I gave before with the Homebot, I I tied it into our observability tool. So, what I measure is things like time the first um to see how fast it's replying to me. I measure how many

tokens I'm using, if it's hitting my cache, because I'll set it up to cache responses to. So, you're probably not building a chatbot, so the caching part's probably not as applicable to you. Um, learning where those utilizations are. Uh, I could show you some other fun stuff. I'm actually building a talk about this that I have to learn more about it. Um, that I'm doing next month

in Vegas. But yeah, there's observability metrics that'll really help you with that overall. Uh >> Hi. Thank you. Great talk. So a quick question about so the newer generation that's just coming out of college now that might look at >> all of this is the sky is falling. There's never going to be software engineers anymore. Uh do you have any comments to encourage or >> sort of

just look at the state of the industry especially for interns, >> college students? interns, if you're able to use uh some of these tools like clawed code at work, um I forgot to mention one of the examples that I do as well is to um I will feed clawed repositories from work and ask it to explain it as if I'm five and try to go through the

choices that it made within there so I can understand some of the design choices, get in and understand the design. um either of the code, the infrastructure, everything else. Work with a tool that will help you understand it. Build out diagrams. can even spit you out some readme files, some markdown files, build examples from that and go from >> and then gentleman in the front row over

here that was not in the splash zone. not clear how to teach >> to people for agent teams and sub aents. What is starting small? >> Right. So, uh, when I mentioned, uh, you're not going to build a clone of my product in one prompt. So, start off with I want to do this thing. First up, I want you to all these things. I could show you

an example like a prompt that I do. Um, overall, >> have you used co-work in claude? >> Uh, co-work not yet. I'm strictly uh clawed code. >> Have you said create a team to do this thing? >> Nope, I don't do that uh too often. I do sub agents where I will ask set up an agent file where one will be the game designer and then the

other will be the uh project manager and the other will be code review. How do you manage execution uh and coordination for all of the team members? >> Uh so far I've been yoloing it. That's been my vibe. YOLO it. Let it run on its own. >> There's copyright implications for YOLO. What do you tell people that want to yolo? Because the new thing is you don't

need skill, you just need high intent. >> Uh so what do I do to protect against >> Well, h how do you manage the copyright implications for YOLO? >> Uh for me, >> there's no assemblage. >> So for me, if I'm building something for the company, I reference the internal material only. and it already has access to my repo that I'm referencing. So that is my point

of reference. This is your corpus of data only go with this. So I'm not building anything else from beyond that. The video game example I thought of it could be cool to build a video game. Uh I hit more problems around actually just the quality of getting it done in a time frame that I wanted to I'm impatient. So I haven't hit that point personally myself. Uh

when I have worked with chat GPT to do something like I've given it an image of John Cles the person who did the adopt adapt improve image. It yelled at me said this is a copyrighted image. I can't do it. I'm like it's a public image but okay cool. I didn't want to use an image that looked crappy. So in that case I didn't do it. Oh,

this was in if you're using claude code and you launch it with YOLO mode, then it seems like there's >> copyright for the stuff that you make because there's no assemblage. >> I mean, there could be, but you're still going to have to have some level of human in the loop or human on the loop to make sure that you're verifying and doing your own due diligence

to make sure that you're staying within the boundaries of copyright. >> Any other questions? Uh, gentleman in >> All right. Yeah. Up the punks first. First first of all, >> And, um, yeah. Was there any point in time or realization that stood out to you where you went from is this all hype and oh this is actually real and I So is it all hype? Uh it's

when people come up and ask if I have a dashboard for every little thing from at the time it was only lang chain stuff like that. I thought it was just going to be a hype and at the time everyone was really talking about Gen AI being a co-pilot for your email. It's going to help you write your email which is something that I don't want and

I still don't. So when it got beyond that is when you see more of a push internally. So me seeing more of a we're going to start using some uh agents to generate some code here and there just to see how it works and experiment in that way. And then just kind of diving in and watching the trend overall from the different groups the meetup groups around

the areas. just watching it slowly become AI focused everything. Um, do I think it's here to stay at this point? Yeah. I I think we're going to be stuck with some level of AI in our life. >> Was there anything like >> was there any any anything like relatively impressive that you didn't realize it was capable of that happened and then Okay. Is there something really impressive

that happened? >> Yeah, that did >> um I think when I was building out for me the standout was like the good dogs uh application. Um the way that was able to help me create custom trace spans. Uh a span is or a trace is uh just the way your code works. So I could see an end toend request. So I could add custom tags to that

to see how people were voting. So I could have my own little dashboard that said, "Hey, on this image of this dog, more people said good dog versus bad dog or something like that." And that was a pretty cool utilization and recognition of a thing that a lot of people don't want to go through a level of instrumentation when they're working with code at that level. Uh

it gets very tricky. some people to try to go with the out of the box stuff, but it was pretty cool to see how in-depth it could go by using the documentation. All right, thanks everyone. it was really weird that my grandmother was telling me once that she was listening to the radio and then she thought it was really crazy that she could see a blimp flying

over that they were talking about and she's I forgot the hype of your >> I got no sponses, right? >> Huh? No more spots. >> put it on the pocket. >> Turn off your slack. That way you don't have to worry about anything popping up during the >> And then we got the USBC. we have few minutes. >> We have few minutes. It's a little easier without

Yeah. Yeah. Hi, I'm Jordan. Yeah. When do you leave? Tonight. Uh spend the weekend out here. Weather's nice. >> Whoa. >> For vacation. Nice. >> Are you like are you into bird watching >> Oh, really? One of my managers watch at least you're not going to Mexico a couple of Sundays ago but that's a long flight layover in Miami. >> So, yeah, we'll start right here. Earlier

in the day, we had one minute sponsor. Had to get those. >> Have you been to scale before? >> It's a very cool volunteer. It's all volunteer grassroots thing. This is the 23rd year. Started from Linux install. open source conferences and just grew and then we were able 15 years ago we were able to add in this DevOps day. >> Oh yeah, there's a DevOps day organization

that's all around the world we're fortunate with scale being here in say DevOps state, Boise, Idaho or something, they they've got a rent room, you know, rent the facility and sell tickets and all that kind of we can plug into the scale infrastructure. It's a lot easier to just like So, I'll do a real quick introduction for you. How do I pronounce your name? >> Because I

saw it spelled with a E somewhere and then E at the end. I have a different it's kind of funny because some of the were running longer and then we were doing Q&A like right up to the top of the hour, you know, which is kind of But, you know, one talk is what it is, but we will have ample time for Q&A. So, We'll run this

mic around so that they can we'll make sure they ask the question of the mic so it makes >> Yeah. Yeah. Um I've been working with systems connected to the based in Santa Monica by the beach for many decades before the internet was >> I think we're good. Yeah. Now, if you could just replace my battery, please. >> Look what just look what just hit. No, a

new pot of coffee hit the hit the 15 minutes ago. It was a real blessing. We'll record this later tonight. Say hi. Works. >> All right. All right. Um Okay. So uh we're going to uh pick up with our next top talk talk here at DevOps Day LA. Um thank you all for u attending throughout the day. Um I guess I had mentioned earlier I'll mention it

again on Sunday uh here at scale we have a opensource careers day track uh over in the other building. um a good way to you know work work on that resume and and also attend presentations and also um consult with other professionals um about your I guess kind of profile. If you wanted any more information about that uh there's a DevOps day table right around the corner

and Brian there uh can tell you more about open source career day on Sunday. That said um uh Shellini uh Sedaren uh is now going to make her presentation about the um leap from DevOps to AI ops. Take a look. >> Hey everyone. Uh excited to be here to walk you through a shift that's already underway across modern engineering organizations. The move from DevOps to AI ops.

The core idea is simple. DevOps gave us the speed and automation but AI ops adds the intelligence to it. So it helps us operate at scale by learning from operational data, predicting issues and increasingly taking actions automatically. By the end hopefully you should have a clear sense of like what AI ops is, what it matters or why it matters now, what it can do and how to

adopt it. I'll start with a quick introduction. Uh I'm Shalini Sudarin, a DevOps engineering manager at Kinderare Learning Companies and previously at Nike as a SR manager. I've spent about uh 20 years in the software industry and my focus has been building and operating reliable and uh secure costoptimized platforms especially around data and AI platforms. A big part of my work is enabling enterprise AI adoption through

a governed operating model. That matters because AI can accelerate the delivery but without governance it can also increase the risk and cost. So uh I come from a strong DevOps and uh reliability engineering foundation. So this topic definitely sits right at the intersection of delivery speed and operational excellence. Let's ground this in the DevOps foundation most of us have built over the last decade. So first we

started with the CI/CD pipelines where we built automated the test deploy the workflows that increase the release frequency while reducing the manual errors. Then we moved on to the infrastructure as code where we were using like terraform anible or like similar tooling to make the environments reproducible, scalable and also auditable. Then comes the observability where metrics, logs, traces that gives us a deep insight into the system

health. This is the backbone of the modern operations. So it's actually a real progress and that's why teams can ship continuously and run distributed systems with confidence. But then here's the reality. DevOps alone isn't uh enough for today's Why? Because there are a few challenges here. The first one is the scale. the televatory modu um volumes we have tons of logs metrics traces events they're all exploding

and there's like simply too much for the humans to interpret quickly so that leads into alert fatigue thousands of notifications per day most important signals it gets buried with those notifications so even with like more automation Many organizations have hit on MTR. So the meantime to resolve stops improving after a certain point. So we also have started to see reactive firefighting because teams have been like spending

too much time to respond to incidents and like not having time to prevent them. So finally the soil tools created like fragmented visibility. So we end up with like disconnected monitoring, ticketing, deployment data, infrastructure views. So it gets too hard to connect the cost and the uh effect too fast. I think that is all are the driving factors of like why DevOps alone is not enough. So

that's where AI ops comes in. AI ops uses machine learning and big data analytics to automate and enhance the IT operations. So um Gartner's as per Gartner's definition. So the term is to describe the convergence of AI with operational workflows. But then the actual definition that matters most is the The goal is to move from the reactive operation to predictive and increasingly autonomous operation. In other words,

instead of waiting for failures to happen and scramble to fix them, we use patterns and data to predict the issues earlier, correlate the signals faster, and in many cases remediate automatically. So um think of this as an evaluation on how we used to run the systems. We started with like a manual operations where we had everything um based on like scripts driven tribal knowledge humans doing all

the correlations on their heads. Then DevOps came in and it started to improve with its automation CI/CD better collaboration. So AI ops is the next layer. It supercharges DevOps by adding an intelligence engine that learns from data, adapts as the systems changes and acts at the machine speed. So the system becomes better at telling us what matters, why it's happening, and what to do about it. Sometimes

even it does it for us. So what can AI ops actually do? So there are four core capabilities to First is the noise reduction. So the AI ops can correlate the alerts events and consolidate thousands of signals into smaller number of actionable incidents. and it can often dramatically reduce the alert volume. Also, second is the anomaly detection. The machine uh learning models learn what normal looks like

and flag deviations early. So, we would have that ability to catch the problems before users can actually come and tell us. So we wanted the systems to tell the problems before the users or the customers can come and tell hey something isn't working. Then is the root cause analysis. So in the modern world or like in the in the current thing what we do is like we

are like manually hopping across dashboard services to find out what is the pinpoint or like what is the actual problem itself. But AI ops can use the topology and uh dependency awareness to pinpoint likely the causes faster. And then is the auto remediation the ability for us to execute predefined runbooks or in like more advanced system learned responses to resolve the issues with minimal human intervention. So

that is at least the aim for us. The reason um leaders care is the impact that is measurable, right? So AI ops is typically associated with faster MTR mean time to respond. So significant alert noise reduction and meaningful cost savings because fewer incidents and faster recovery translates directly into the reduced operational overhead and less business interruption. And it's not only about the business interruption, right? It is

also the brand value. It is not only about like the customer satisfaction or like your cost and whatnot. It is also the brand which gets hit when something isn't the way it is supposed to be working. So you all see a strong market momentum here. So the AI ops market is projected to grow substantially reflecting the broad enterprise adoption. The key point here that we're trying to

make is this just isn't a tech trend but it's an economic and operational response to complexity. So um uh to to to try it u make more concrete with real time use cases one common area is the predictive scaling. Instead of scaling only after a threshold is hit the machine language models forecast the traffic on workload spikes and scale resources proactively. So that would actually help us

to bring down the cost meaning we don't have to like scale up thinking oh something maybe this is like um a event is planned or like it is a holiday season that is coming in so we don't have to like really keep the things scaled up assuming the traffic would be more. So predictive scaling plays a major role. The other one is the security uh anomaly detection

where the AI spots unusual access patterns or the behavioral deviations across cloud environments. The third one is the intelligent deployments which directly plays a role with the DevOps where AI ops can evaluate a Canary release not only by a single metric but by correlating the performance signals, error rates, latency, infrastructure events or like even the deployment changes that would help the teams to decide faster whether to

proceed or to roll it back. So uh here are more um uh use cases that show up a lot in practice. Proactive anomaly detection that would help us flag the performance degradation or resource exhaustion early. automated incident responses that would help us reduce the manual toil from common repeatable issues. Things could be like restarting a stuck service, scaling a saturated dependency or like isolating a noisy neighbor,

right? It could be in multiple forms where we wanted you to see like there can be an automatic incident response um reduction and capacity planning. Like I said before, use the trend analysis to forecast resources needs and help you plan and spend the architecture before you hit the limits. These use cases are powerful because they map directly to the day-to-day pain outages on call fatigue or like

unpredictable cost. I still remember in previous places that I had worked on where like starting June we start to like do up capacity of planning thinking like hey November December is the holiday season and like we start to that and come January what we do is like oh we still did not perform well or like we did not do a capacity planning right and like we are

we do a lot of learning and we do the same thing starting June. So it has been a repetitive cycle. I think these are like really a common pain points as a DevOps engineer or like a S sur So now uh how do you actually adopt EI ops without it becoming another expensive tool that never delivers right the adoption road map is best approached in phases. So

to start with consolidating the data, centralize the observability data from logs, metrics, traces from like various sources and then the deployment signals. Basically consolidate them. Then automate the correlation. start identifying the relationship between the alerts, the events so that the reduce noise and it would improve the context enabling the prediction applying ML for anomaly detection and forecasting and lastly move towards autonomous operation self-healing and like automated

remediation. I would mostly call autonomous operation as like a matured model. But then the important thing here is that each phase builds on the previous ones. So if your data foundation is weak, the AI output is going to be weak. So I think the important uh key message we wanted to pass on is like the foundation should be really strong so that like it it gets built

up on where we wanted to land. And uh this is where uh most organizations see the ROI sooner than they expect in the first two phases. Why? The data consolidation and alone can reduce the noise and speed up the triage dramatically. The fully autonomy is a longer journey. Like I said that is more like a matured model because it requires high confidence action governance trust but the

path begins with getting our operational data in order to embed the intelligence into the So the key message here is the AI ops adoption is iterative. Don't wait for like a perfect autonomy to start. Start with like foundational wills. So um why is this happening now? Why do we need it now? So there are like several factors or several forces to converge this systems are more complex.

Microservices, cloudnative architectures, distributed dependencies. These are all like driving factors. Then we have the operational data is like growing exponentially in volume, velocity and variety. The business demands speed and agility. So faster response time and proactive prevention because downtime is expensive and very much visible. So the cost optimization is also a major driver here where cloud spent and operational overhead are under scrutinity right now and ultimately

it's about the customer experience reliability and performance translates into user trust retention and revenue. The AI ops is the operational answer for all of these pressures that we are seeing right now. So uh if we break down the components of uh AI ops platform, it's usually includes data ingestion and aggregation to collect metrics into a unified operational view. The big data analytics to process and store high

volume time series and event data. ML and AI algorithms for pattern recognition, anomaly detection, prediction, and correlation, often including clustering and natural language processing. So automation and orchestration that integrates with ITSM and CI/CD tooling to enable automated actions and visualization and dashboards to turn outputs into operational dashboards, decisions with context and drill downs. When the team uh fails with AI ops, it's often because one of these

layers is missing. Most commonly the data or the integration layer which is the foundational one. I think while we are talking about moving towards AI ops, it is also useful to compare AI ops to traditional monitoring. The traditional monitoring is typically rulebased and threshold-driven. It's mostly reactive and it relies heavily on like humans to come in interpret what's happening, decide what to do. On the other hand,

AI ops is more datadriven and like ML powered. It aims to be predictive and proactive. It automates analysis and response. So that doesn't mean that thresholds would disappear from this. Threshold still matters. But what AI ops would come in and do is like it reduces the dependency on static rules in environments that change constantly. Now uh the adoption isn't automatic, right? There is a real challenge here.

So the data quality and integration is the biggest. If the telemetry is inconsistent or soiled, you'll get the poor results because we cannot keep logging things that might not be very useful. So how do you actually drill down to say like what is useful and what is what is not useful. So the telemetry plays a really big um play here. The skill gaps definitely matters. So we

will need people who understands ML concepts, data and operational workflows. So it is an collaboration between ops, platform engineering and data teams. A siloed engineering or operations teams or data engineers like no that's not going to work here. Then the tool sprawl and the complexity that can also get worse before it can actually get better. If you add AI ops without consolidating the existing stack, AI ops

is not about like creating everything from stack. We wanted to see how it can actually blend with the existing stack. The cultural resistance is common. So most of the time what happens is like teams worries about the trust, the accountability and very common thing now is like job displacement. So finally defining the success matter metrics is very essential. Without clear KPIs and ROIs the AI ops become

interesting but then like it might not be a valuable one to actually play it So how do we actually overcome these adoption challenges? Because we are we are going to overcome these barriers. The play the playbook is like very straightforward. So we wanted to always start with small and then scale gradually because that way we have a lot of opportunity for us to fail fast, fail early

and like learn from the mistakes and like apply on the future. So that's why like I think it is very important for us to pick a narrow use case which the value can be very visible like applying these the value can be visible immediately. So invest in like training and upskilling that is definitely needed because not everybody is an expert. It is all new. It is all

coming in but then it is all faster. So definitely that training and upskilling and building that internal capability not just like the tool dependence choosing integrated platforms tools that can actually connect across the stack rather than adding another silos like I said before we don't want to keep adding tools as we transform or like as we move into this transformation but more intelligently pick the tools that

can blend with the existing stack. So that way like we don't have to reinvent the wheels from scratch rather we can make use of what existing is there and apply on top of it. Fostering a cross collaboration, development, operations and data science to break down the organizational silos. That is also a critical thing and it is no new at all even in DevOps that is a critical

one like correlation and like having that cross collaboration is a key thing and establishing a clear KPI things like alert volume reduction MTR improvement change failure rate reduction or incident reoccurrence on call load cost savings these could be like clear KPIs and like it's easily measurable so in practice the teams that succeed to treat these AI ops as an operating model change just not as a tooling

project. So that's why it travels back to like how critical it is to define those measurable KPIs. So looking forward the destination is the autonomous operation right the ultimate vision here is the self-healing systems that can detect diagnose and resolve the issues without human intervention but within the guardrails. the predictive maintenance that prevents failure from before they can occur and automated optimization that continuously tune the resources

and performance for efficiency. The end state is not humans out of the loop. We wanted that humans in loop for that governance and like the trust matter. The end state is humans focusing more on high order engineering while the routine operational work is handled reliably and consistently by the automation powered by the intelligence. Then the key area also with AI ops comes with the security call out

because as we talk keep talking about like moving into AI, the AI ops can strengthen the security posture by detecting the behaviors that could indicate compromise especially in like dynamic cloud environments where the static rules often miss subtle the shifts. So it enables a faster thread detection and response by correlating the security events with operational data. So we can see not only that something is suspicious happened

but also what systems were affected, what changed or what are the downstream system impacts might occur. This convergence right like the security plus operations plus AI is becoming increasingly important as attack surfaces expand. So the key takeaways are DevOps is the foundation. So if you have a strong DevOps foundation, AI ops doesn't replace the DevOps culture and tooling it. It it just amplifies it. It is not

going to come in and like replace DevOps. Start with data. Centralize and normalize the telemetry because AI is only as good as data. It learns from the data is the key. That is that is as simple as that. And we have to embrace the shift moving from reactive to predictive operations is a competitive advantage both in the reliability and also in the cost front. So if you

remember nothing else just remember this. AI ops is about scaling the operational excellence when humans cannot keep up with the system So I wanted to leave um with a question here. So will your team lead the AI ops transformation or react to it? So the best time to start is now. Pick one operational pain point. Consolidate the data. apply the correlation, learn from the outcome, learn from

the mistakes. That's how the next evolution would actually begin. Yep. Thank you so much for listening. Happy to take any questions if you have and happy to connect for any future discussions. Any questions? Go first. >> Thank you so much for your talk. I really enjoyed it. I had a question you had on your slides. Um, and I'm going to pull this up real quick. So, the

faster uh mean time to restore at 50%. Can you talk a little bit about what went into that number? Um, like was it from weeks to days, days to minutes? >> And what type of system it was? >> Okay. Uh so how we actually operated was like we we have this culture of like MIM and most often what happens or what used to happen is like we

were getting only the complaints from the customer even if the system was down our alerts were not really acting to it because like our alert systems or like notifications were not set up right the even the thresholds there were instances in the past not recently but definitely in the past where like even multiple days the site would be down and like we would not come to know

unless the customer comes back and says that. So with then with actual uh intelligence of like having the right thresholds or having the right metrics like I said metrics is important but at the same time what we log is also important right like we can log a tons of it but what it because it is a haststack a needle in the haststack is it becomes really uh

difficult so and we don't want to create the dashboards or create the metrics or create the thresholds just for the sake of doing it, see how it is meaningful, what what makes it a difference. And that's how like we started to more moving from that and like now we are at a point where like we are more focused on like um proactive rather than a reactive. So

we are now at a point where like oh it pages immediately if something is going to go wrong. It is not that it is already gone wrong but it is going to go wrong. So that's where like we are at this stage. Yeah, just a quick one. Um, lots of vendors in this space want to sell AI ops. My question is, um, to the limit of what

you're at liberty to disclose, are you building internal tooling, going with vendors, how are you doing all of those things? Secondary question, in the modern job situation, what do you look for in a person who you would hire to work on your AI AI ops team to work with that tooling? >> Thank you. So I take the first question first. Um so yeah we are not going

to vendor based. We want we we are trying to see what what make more um meaningful for the organization itself. So we are building that internal things. Again like I said we are we just started like really simple ones. For example like we started to do a PR agent so that would automatically review my PRs. So that that is like a small smaller example and again it

is not only like DevOps it is a culture that has to happen between all the teams like cloud engineering the engineering teams data engineering teams so that's how it is so we started to take like smaller ones and like then like expanding it it's so we are not doing vendor based it is all internal uh to the second point here is like like I said AI is

is a new thing or probably like it it is it is moving faster so I I don't think there is like a real skill set or thing. It is more like a mindset shift and a cultural shift is what I mean. So that if you have that intent to build I think that that what is matters more rather than having that right skill set because in papers

there is no like hey this is the skill that we wanted to have exactly to be an AI ops engineer. So that is to me personally it is more a cultural shift rather than a skill set >> So I have a um two questions. I'll start with the first one. I'll piggyback on the question that was asked earlier regarding the metrics. So the 50% that you brought

up regarding MTR what was that based on? >> What was this based on? >> How did you Yeah. What was the original metrics that you had against that you got the 50%. Where is that coming from? >> okay. So that was all based on like the number of incidents that we have had in the past and like how what is the turnaround time for the incidents to

actually resolve. >> Who are the incidents? >> The we have operations team. So that were all coming from the customers directly. Now we have our systems to tell us that the systems are going down or like it is degrading in performance. So something is going to go wrong. So now the systems speak more than the customers. So that is how like we evaluated the meanantime to respond

>> just on a one operation team. So because people actually working differently. >> Oh. Uh so with with our current organization our IT is pretty small again like our digital engineering our digital space is pretty small. So we have like operations for data engineering then we have um for the regular the product engineering as well. So it is a combination of both data and >> I see.

But yeah, that that's not 50% because every team is different. So that's that's what I'm trying to say. >> Yeah, this this was like more specific on like one use case that we had taken especially around like how we were getting the incidents or like how we were responding to it. So that's how like we took it forward. >> Got it. Okay. I'll get to my second

question. So I saw you actually point out there is a lot with the AI ops there's going to be a lot of releases, right? So releases I'm actually assuming you mean deployments. Am I correct? Okay. So with that being said um all those releases that happening are they just automatically from uh integration they automatically go to the deployments and just get to the produ >> so not

directly getting into the prod so again with our current setup like we do have like multiple non-prod environments and again different teams operate at different scale so there are few teams which actually like say uh we we create an automatic PR when going from like um because it is a featurebased uh deployments. So it creates an automatic deployment on each environment and that is automatic and um

at least uh we have the PR agent installed. So once the PR is approved it automatically moves over and the other team that has is like more of a tag based deployments. Um so with AI what we are trying to do and again that is more like a work in progress where we wanted to see is like as we do a deployment like is it Is it

been like trying to get to see like hey how is our system health as soon as a deployment happens if the system health is degraded and we have the threshold set for each of them if the system health is not at the level that we had set. So we it it is recommended to have a a roll back in >> How do you prevent hallucinations before the

deployments? >> So we try most of it in the non-prod environments only first thing. So before we actually try to put in anything in the production we have like multiple non-product environments >> human validation. >> I see for all the releases. >> I see. Okay. All right. Thank you. >> Um thanks for presentation. Can you go over the tool set that you are using from delivering data

and what tools do you use to analyze those? That's the first question. >> So uh our deployment tool has been octopus. Uh so that is that is our deployment tool and again our machine language so our data uh or the data analytics we have been like using uh data bricks there. So we use a combination of data and again there is like NDA as well. So yeah

>> that's fine. How do you deal with unknown unknowns? Let's say uh you didn't know that there's a limit of snot ports on something and then one day you're facing that issue that there's issue with snot ports for example. How >> There is always going to be that unknown unknowns right I think at least personally what I feel is like learn from the mistakes. So there is

not that oh we had a mistake and then like oh we we cannot recover from it. So my my takeaway here or like my giveaway here would be like how we can prevent that from happening in the future. I think that is more important than like cribbing about like hey this has happened and like who and instead of like pinpointing things rather it is useful for us

to move forward or like fail forward fast. I think that that would be my recommendation on that. So that's going to be difference maybe between AI ops and MLOps >> maybe >> any other questions. So few things that we've been considering within the AI boundary is like um automated deployments again with DevOps practice also we we do have that deployments but then like what comes in back

is like after the deployment like what is our recovery or like what intelligence that we built to say like hey is this deployment a secured one or like no this is a failure one where we wanted to roll back and then the most of it is like built across within the engineering itself where we wanted to see like how faster it can move. So that has been

our key things. >> All right. Thank you. Cool. Thank you. Okay. Our our our next talk coming up uh at the top of the hour uh Tom How will be talking about how AI is accelerating the pace of innovation and why humans still matter. So hang out. >> yes, Rich. Are you talking to I thought I told you. Thursday, the 26. But trying to open Little You

got it right there. We don't want any surprise. We don't want any surprises. >> That was me. >> Kind like >> Oops. Do that and Test test. >> Test. Oh, there it is. I hear it now. >> Okay, I'm gonna turn it back off because >> Yeah, we got a few minutes. while we're sitting here. I guess everything that needs to be done >> you're not running

any >> Oh, I assume that's ours with the Yeah, you can move it out. >> Yeah, well, we did a now we're doing >> Well, no, my computer is doing no A. >> Oh, all right. Yeah, right. Good. >> It was actually yesterday it got a bit hotter. We had throughout the day we've had a lot of heads in here, but it didn't really heat up. But

I noticed yesterday afternoon 3 or 4 o'lock it was it was even hotter. >> Maybe. Yeah. The sun's cooking the room. I mean it's beautiful day. The weather's killer. Yeah, I have a friend who lives in like Nevada City. That's Tom. >> Whoa. What? That that bit >> check. Here we are. DevOps day la scale 23X. Our headliner Tom How is ready to rock us with some

heavy stuff. How AI is accelerating the pace of innovation. Things are moving so quickly, but importantly, why us humans still matter. >> Thank you. So, I was able to grab that highly coveted 5:00 pm on Friday slot. So, I'm grateful that you not you're all not at, you know, having a nap or going to dinner. So, welcome. I'm going to try to discuss this from a couple

of perspectives. Um, I will acknowledge upfront that this was not originally my talk. It was a talk from someone else at my company. I should say the title, the talk is mine. So, I've been shoehorning my ideas into this talk. So, I just wanted to give that up up up front. So, next, who am I? So, I am the director of insights engineering at Hydraulics. Um, I'm

a former Disney streaming director of quality targeting and experience. uh former MLB BAM senior services engineer, former DOE laboratories simulation engineer, and former bitter castaway from academia, not in computer science in sociology, but guess what? It's just as bad. Um, and the reason you should listen to me is because I've made a lot of mistakes. I mean, in my career, I've seen a lot of things go

wrong, and often they were my own fault. I've also gotten paged. Oh, I hit the wrong button. I've many times. I've been on call just like everyone else. That is not right. I'll just let that come back up there. Uh I've been on call. I've gotten paged. You know, when I was at Disney, people would call it a crisis because they couldn't watch Wision at 2 am

on the night it was released, but it didn't make any difference because I still had to deal with it. So, when I'm talking about how AI can help us and why it can improve the pace of innovation and why humans still matter, trust me that I have a vested interest in this and making sure that we do it in the right way. Okay, so this is the

one little piece of advertising here, which is what is hydraulics? Um, and this is taken pretty much straight from our like brand materials. I'll just be straight up about that, too. So, hydraulics is a real-time data platform that focuses on quickly turning data into understanding and insight for their users. And it focuses on quick rapid ingestion and making that data available instantly as well as high levels

of compression in order to avoid the problem of sampling and uh retaining just limited data. So whereas like on a Splunk you can pay a lot of money to store your data for 7 days or 30 days with us you can store it for 15 months and and still have the same rate of access. Uh, it's been used by Fox, ABC, and Paramount, and Disney. That's how

I first got connected with hydraulics. Um, for the Super Bowl, for the Olympics, and other big events. All right, I'm done with the sales pitch. So, this is what we hear a lot today. This either AI does everything from the people who, you know, want to just type things into the uh the console, their COD console or their chatgbt and have their app come out. divisively called

vibe coders, although I think that's a little bit unfair to people who use that process. You also hear AI ruins everything, which is everyone is going to get laid off because there's no longer going to be a need for humans if everybody can just create their problem, their solutions on their own. The one thing that I hear really frequently is, well, now that we have AI, product

is going to replace engineering. So the product managers who understand the space that we're trying to resolve are just going to be able to build their own things and that will eliminate the need for engineers. Um and I will say it's really powerful but it is just a tool and and fundamentally if we think about it it's really just a data compression tool. And I think it's

make sure we keep that context in mind. So, if you're coming in here and thinking either that this is going to solve all of our problems or that it's going to ruin our lives, I I think we should take a step back and and give a little bit more context to the story. So, I'm going to start out by talking about how we use AI at Hydraulics

today and how it actively helps to improve our uh velocity in terms of innovation and rolling out new and exciting tools and products, but also where humans come into play. And then after that, I want to take a moment to discuss a what I consider to be a really exciting success story from one of our customers. So today we use AI across the entire company. Uh it

actually sometimes can get a little annoying. They have a a board of the top 10 users of the AI tools for that given week. And you know that's that's well and good but I feel like that doesn't necessarily capture if it was like a good quality use of the AI but there is a strong embrace in how AI can help our productivity uh while still making sure

that we're innovating it in new and exciting ways. So we use it everywhere. We use it in uh operations engineering obviously everybody every engineering group uses AI these days uh and and our product team uses it which I'm not sure especially amongst big organizations I'm not sure how many product groups feel totally comfortable with it I you know I could be completely out of touch uh simply

because I was at a company like Disney where it's not really you're not really allowed to use it so much but I get the feeling that product are sort of the new comers on to the AI wagon. But none of it at our company works without engineers or other humans making the deliberate decisions about what the system knows and how it should reason about that. And this

is incredibly critical to the other piece. So let me show you a little bit about what that looks like in practice. So I mentioned those four different Oh, there's an extra slash in there. Oh well. Um I mentioned the four domains that we were talking about. So operations, product, engineering and the product which is like actually how it's built into the hydraulics toolkit. Uh so from the

operations side, we spend a lot of time trying to manage our resources, you know, our um AWS bills, our lid bills, our uh query optimization, how we're storing data, all of these things that we have deep embedded knowledge about as well as cluster health. Like we've experienced these things over and over and we understand the process that one needs to go through in order to understand deeply

what those problems look like. By taking that knowledge and embedding it into an LLM system as opposed to just randomly asking an LLM system to try to create and come up with a particular answer, we can start to give our operations teams and our uh support staff more agency to diagnose problems without having to go back to an engineer. Um on the product side, you know, we

work with customers. Customers are notorious about knowing they want something but not really knowing how to explain it. And this is one of the places where I think that LLMs actually provide incredible value by helping us to flesh out those specifications in two ways that we particularly use it. One is by AI assisted interview where we ask the AI to using our framework for how we build

PRDs and things like that. Create a set of interview questions that we can interactively go back and forth with the customer on to kind of nail down and get that specificity and go from I want a solution that monitors my webbot traffic to like I want to know this metric and I want to know how it's impacting my sales delivery costs. um sometimes that doesn't work. So

we also have uh uses and built out agent structures so that we can do so we can do independent research using the web using uh our own internal documentation using our own data and feed that back through. But again, even that is grounded in the decisions that the product people have made in order to understand how to get the best and the and the meaningful information out

of the system. It's moving as much of the work upstream of the LLM as we can. How do we define the boundaries that allow us to do that? So engineering, ironically, is I think one of the less exciting use cases here because everyone's doing it. You know, LLM are really, really good at generating tests. Uh they're good at generating code if you have clear requirements and specifications

and guidelines and all that kind of stuff. But they are fantastic at generating test. So the value that the uh engineers bring to this is not only knowing oh this is good for generating tests or this is good for generating this kind of documentation but making sure that enough context is given to the system and that the guardrails again are in place so that the answers that

you get back are more likely to be aligned with what you're trying to get. And finally in terms of the product hydraulics itself we have several new initiatives that are either already deployed or are being deployed. Um so we have an MCP server uh which is you know a a well-defined contract that uh a an LLM agent can use to ask questions about our data. Um we

also have um so that is already deployed that's something that we can go in it and customers use and they can ask the questions. We're also doing some new work with um sem understanding and extracting semantic intent from data. We have a lot of data come in. We don't necessarily always know what's in it. So we are trying to automate the process and we actually have a

tool that's in beta. We're automating the process of ingesting data, understanding what it means, and presenting that back to the user. I'm going to go more in depth into that in a little bit because I want to be clear that it's not magic and it's not anything that like we're just like there's concrete reasoning behind Okay, I guess I'm doing that now. I kind of forgot that

this was the next slide. So, I want to talk about this semantic inference engine that's being built. um specifically as the tagline here says like what happens when we get data that we haven't seen. So we frequently as a company that specializes in high volume log in event data we're super familiar with a certain set of data like if you are on Okami or Fastly or you

know CloudFront we probably know what your data is going to look like coming in and it's not going to really be surprising at all. We also have experience with you know uh web firewall and other security telemetry like these are things that are known hotel we're familiar with pulling that in uh and it fits easily into our model but a lot of times when the data actually

arrives the bites don't necessarily line up exactly with what we expect either because there's been a customation customization that's been done on the data provider itself or because it's actually just the customer's unique data. One of the interesting things about hotel, for example, is people, you know, talk about how it's standardized, but you know, what's inside the envelope could still be pretty much anything. So, we have

to figure out not just the very simple like what's the type of this data and I mean that like as like a syntactic type, but we also need to understand in order for the data to be useful, what does it mean? Is this a ID? Is this a hierarchical relationship between these rows? Is there a timestamp here? Time stamps are relatively easy, although not always. But it

once we start to see information that's like a session ID or a device fingerprint or parsing information out of a URL uh path, for example, that's one of those things that's incredibly complicated to do automatically. Things start to get a little bit more complicated. So I I said it's not magic and I want to keep emphasizing that 95 and I I mean I I kind of made

that number up but I think it's right 95% of the inference that we do as part of this system is actually mechanical. uh you know it hearkens back to reg xaxis really I mean not everything is reg xaxis but there's a lot of regax work in there because we are able to uh we have enough knowledge from the data we have that we can extract all of

that information about how to build regexes how to do pattern matching what kinds of cardality we expect to see uh we can just leverage that so why wouldn't we why would we try to build something new in a arguably untested tool when we can already do that. One of the things that becomes a little more interesting is when we start to deal with semantic labeling. That is

the kind of thing that you can't necessarily just extract out. You can do certain types of uh inference by saying like okay this is the cardality of these two values existing in the same row and that gives us the impression that that might be a hierarchical relationship and that kind of thing can be done mechanically but there's a point at which we can't just assume that the

relationship is based strictly on that hierarchy that we see through the data. there's more semantic knowledge that we have to to apply there. So we have our engineers building out you know along with ragaxes some of these uh rules around the semantic relationship between different types of data and how we can look at naming patterns and data distribution and cardality and understand exactly what they can come

up with and encode that in a way that an LLM is able to reason and I use reason extremely loosely here because it's not really reasoning it's just extract racting compressed data and making a probabilistic guess about what it relates to. But you know that's how people refer to these things. They anthropomorphize this. So the engineers are building out that knowledge base both in terms of you

know graphbased rules as well as you know reg xaxxes as well and uh other just sort of uh semantic pros knowledge. They're building it out and they're feeding that as the baseline into the LLM. So we've got our 95% of our data that has already been extracted and you know through the mechanical inference. We've got this 5% that we've applied this additional semantic labeling to as well

as a layer of relationship inference like okay well we see this data coming uh and you know based on what we've seen in our query history we see this data being related to this data in these ways. So we'll build that out as a uh logical connection between the two. So that's the basic idea behind our mechanical inference. Now what doesn't always work are things like well

you know this data company provided us a field where the you know the name is like a list of arbitrary numbers and letters which happens all the time. um we can't patch that uh we can't uh identify that based simply on the name. We'll see very deeply nested uh data structures that we're not used to working with. Amazon is notorious for this. They will just send these

huge packets of data that like you know they're nested deeply and it's unclear like okay is this the wrapper? Is this the collection? So we can't just do that mechanically reliably. Um we often see fields that like may look like a double. Well, actually, I'll give a great an example that I see every day, which is uh the ASN um the autonomous system number sometimes as a

string, sometimes as a as an an integer, and it's really difficult to identify with if you don't know that it's an ASN which one it is. So that's an example of where normally we would have to be pouring through data over data over data and that ends up being this long process that we end up having to go back and forth with the customer. It annoys them.

It makes our uh you know transition time instead of being like a week to get them on boarded six months or whatever because we don't have a clear set of rules for how to do that. And that's where we leverage the LLM for those cases that are not obvious based on all of the data that we've fed in in terms of the information that the queries have

that uh that we've experienced by looking at what customers are actually querying as well as our own internal research. We have the LLM look at the fields. We give it a you know the basic summaries, descriptions, you know, so a sense of cardality, a sense of distribution, things like that. And then we g let the LLM take a crack at identifying what it is and we force

it to create a probability based on a heristic that we've created to say like what are the odds that this is actually that value. So instead of just saying hey we're ingesting this field and it means this we're saying okay based on our heruristics and our analysis LLM decide what probability you would apply to this at that point we give it to the end user directly and

say okay is this right is this wrong what would you change to to make this better and then when changes happen it gets fed back into our inference engine so that we're able to more uh to learn from those decisions as they go through uh day by day. So that's really how the um how we handle using the LLM to infer the types of data that we're

getting and the semantic meaning of the data but not do it in a magic box and not cut humans out of the loop. So the other thing I called out was uh the the MCP inference uh interface that we give to customers and this is you know there's I I'll talk a little bit later about sort of the controversies around MCP but this is really a tool

to allow users to leverage the power of the LLM for inferring intent and using it to actively create queries against our real data systems that probably provide the information that they want. Um so the the MCP server has a small set of tools that exposes including listing tables, running queries, uh exposing exposing the field names and things like that that the LLM can then use to make

inferences about how the user's request relates to the actual uh data in the data store. Um but it also puts a whole bunch of guard rails on. So, you know, when you when you're dealing with pabytes of data, if you do a query that's like select star from X and don't put a time range on it, you're going to blow up your data system. I don't I

don't care how good of a data source you have that runs the risk of blowing things up. So we build in protections into the system to handle the various optimizations that we are already aware about and to protect basically protect users from themselves when they do um when they do crazy things. This is another example of like where the the engineering to this is so critical. The

LLM maybe over time could collect enough information to realize that if it asks these types of queries, it's going to blow up the system. But this is built in immediately and it protects the system from any mistakes that the LLM might already make. Okay, so we do this for ourselves as well. We are eating our own dog food. We have pabytes of data, internal telemetry. Like I

say, we collect every query that a that all of our users run. We don't collect the data, just the the query, the metadata about it. And that translates directly into storage costs, compute costs, um you know, what customers we should be targeting, what new domains we should be going into. So, we've built a system um called Argus, and I do not know why it's called Argus, but

it is um that consists of our central data store, the MCP server that I was just talking about, and then a set of structured skills that know how to interact with it. Um and I later in the talk, I'll talk a little bit about how we build those skills so they provide uh the maximum value and increase the odds that we get a correct answer out of

the system. And this is again the idea that like we don't want the LLM to explore the system. We want the LLM to interpret what we say, turn that into a mechanical request that can be executed against our system, get the results back and provide an interpretation of those results as well as the results themselves. That whole exploring part is a really good way to get wrong

answers and to bloat the context of your of your LOM. So we also have a anomaly detection system and so we have you know this ability to look for patterns and this is nothing magical or surprising. This is just simple uh anomaly detection uh that's been around for for a long time you know zcores or uh distributions or all kinds of things like that. Um, so we

could detect that and we can feed that to the LLM, but the LLM can't necessarily determine on its own which ones matter. So for example uh it may be that we have cyclical data that peaks at a particular time and has a tendency to have high errors on that and that's totally nominal versus like a low traffic where we get a peak of errors and that's that's

abnormal in and of itself. An LLM can't know anything about that. And this is another situation where we are looking for specific patterns that we know about that we can code up mechanically and tell the LLM, oh this is what matters in this particular situation. So again, we have a scenario where we expect the LLM to do inference to translate what the user is asking, what the

intent of the user's question is, use the tools that we've built to query it in a deterministic and mechanical way, and then optionally interpret what those results mean back to the user. So that encoding is so critical to the success of this system. Making sure that the encoding is clear and is well defined and has a deterministic outcome is really the only way to get good results

from an LLM. And that can only come from people. You can run these things through an LLM over and over and over again, but they will never be able to replace the decades of experience that people have. Okay, that is an overview of how we use uh LLMs and AI at hydraulics. I want to do a little bit of a of a use case study here from

one of our customers about how this can come into play. So the title is nine minutes from alert to solution. So the overall context is that one of our customers and uh I will say it's Nvidia. I'm not didn't put it on the slides but I do want to kind of brag that like hey we got Nvidia. Um but as you can imagine they embrace uh LLMs

and AI pretty heavily. um they were observing a spike in certain types of errors on their CDN, but they didn't know what it meant. And this is the actual timeline of what happened with this scenario. So, several services were suddenly impacted by a very high volume of traffic on a particular load balancer. Um the person who we work with closely was requested to join the bridge after

about two hours of the team thrashing around trying to identify the problem. Um after 15 minutes of playing around and and exploring the data and talking with talking with the LLM, he figured out that they were actually looking they were looking at the wrong part of the system and that's why they weren't finding anything. He then asked the LLM to help find the correct domain and to

have proof. Uh he then figured out what the offending path was. It turns out there was one particular path that was being broken that was broken and being slammed by a particular class of tools. He made a change to mitigate the issue and it all was recovered by uh 757. So between 7:35 when he figured out that there was something that maybe we were doing the they

were looking at the wrong thing in 744 when he rolled out the change it's about 9 minutes uh and and this is the power of a well constructed solution using AI and LLM didn't eliminate him didn't eliminate any of the knowledge that had been in there before now kind of the fun thing I guess fun I kind of made fun of him for it but um he

actually did this all with um speech speech to text, which I know people think is pretty cool. I think it's kind of a gimmick, but I'm old. So, he he went in there and he said, um, literally, this is the question he asked. He he opened up the MCP, he opened up his cloud agent and said, "Oh, can you help me investigate some issues I'm seeing with

OTAA, uh, Nvidia.com here? I'm seeing a bunch of 5xx errors." So the agent itself then read the embedded uh encoded knowledge that had been provided as well as the tools that he had already created and ran three queries based on the like probabilistic odds that like his question intended X. Those queries ran the data was collected and again based on the encoded knowledge that he'd put in

the interpretation was made as well. So there was no specific prompting because all of that information had already been encoded either by the engineer at NVIDIA or by hydraulics itself. So what that meant is that the system actually already knew what it was looking for because it didn't have a lot of choices. So this is what the agent actually found. It found that 99% of requests to

this stats data path uh led to the were those 5xx errors. It also found that that was about 306 million errors on a single path which is clearly uh suspicious. They also saw that particular devices coming from a particular major platform were pounding that particular failing uh backend and then devices were retrying in an inappropriate retry loop that just was continuing to saturate the load balancer over

and over and over again. the reality of this was this is a very very common type of error. Cascading failures across large data sets that are sometimes difficult to understand with multiple domains with no real clue as to which one it was. As as we saw for those first two hours, the engineers were looking at the wrong domain. By building out the system using the knowledge of

the engineers and the skills of the engineers, they were able to create something that could do enough data reasoning and extraction to understand what the problem was. It will not always be right, but it will point you in the right direction more often than not if it's created everywhere. sorry. Created correctly. I was already looking at the next slide. data data everywhere and not a drop to

drink. So that kind of scenario is often driven by a glut of data. We have all of these different high volume sources, CDN logs, WT logs, telemetry, all of which contain a tremendous amount of deep and valuable data in the pabytes of crap that's in Everybody's dealing with this problem and so it's not unique to the scenario there, but it really does draw focus to the particular

problems that we're trying to address here. So AI to the rescue, it solves their problem. Everything's fine. We don't need to do anything. Well, no, obviously otherwise I wouldn't be giving this talk. Um I I I think that there's misunderstandings sometimes about how to leverage it. And I think that part of this is kind of a fetishization of the LLMs and the AI in a way that

like makes it feel like it's not traditional engineering, but the problems are the same. So if we think about like operations uh approaches from traditional tooling, one thing that inevitably happens or at least it did when I was in this world back in the early 2000 is that somebody is going to have a like 5,000line Pearl script that like depending on I saw that depending on the

flags that are passed in can solve any problem you could ever have. This is the same problem in the LM world. People try to create a system prompt that contains everything, all knowledge that they've ever learned about something and just expect it to be, you know, understandable. Another thing that I've seen at my own company is creating a graphana dashboard, for example, that holds every metric you

could think of. Like so I I come from the streaming world and so it would be like okay what's our video start time and our uh failure rates and our download errors and our 503s and CIR all on one dashboard at which point you know what are you supposed to do with that nothing it's becomes not useful the same thing happens with LLMs and the way people

uh work with LLMs someone will ask a question that has either way too much information in it or way too little information or they have it go and scrape the database for every piece of information that's possible. Guaranteed way to get the wrong answer. Um, that's not true, but uh the other thing that you see in traditional tooling is complicated workflows. I think I built one of

these last week actually. Complicated workflows that seem really powerful. You've got this DAG and it's going to find the right decisions and it's got gates and all that. super cool. Didn't work. Um, it's the old yak shaving problem. You know, you've got a problem, you're going to keep working on it and building out this system that seems really great, but it ends up being more complicated than

what you can work with. The problem that we the similar problem that we face in LLM tooling is treating LLMs as a magic box saying I can ask this question and it's going to find the correct answer. And we by doing so we ignore everything that we've learned as engineers over the 40 years. Right? The good news is you already know how to build this, right? We

all do this. We just need to treat the way we look at building LLM tools in the same way that we treat building tools for humans. So we have metrics, we have dashboards, runbooks, alerting, all the basic tools that are used in the observability world to understand for humans to process the information and understand what it means. In the agent context, we have schema building out the

structural foundation. We have a domain vocabulary that explains when I see these concepts or when a user uses these concepts, this is what it means. We have standard investigative patterns like what the engineer at NVIDIA did. He had systems built out that knew what to look for and so he encoded that knowledge into the system. And then the thing that gets overlooked so frequently is deterministic tooling.

This mechanical layer is such a critical part of being successful with LLM and it's something that can only be done by experienced So, at one point I was thinking about calling this treat the LLM as a UI. My boss says that kind of thing pretty frequently and you know I have a mixed relationship with it. But I I realized I think it's okay if you put the

big caveat on it. But like any good UI, it requires strong engineering skill, knowledge, and experience to build it well. You could throw up a Graphana dashboard that has every metric and say, "Look, I built a UI, but it wouldn't be useful." And the same thing applies to LLMs. So, here are the three principles. Well, here are three of probably many principles that I I think is

really important to keep in mind as you're figuring out how to bring the innovation uh velocity increases that you can get from LLM. The first thing is structured MCP, for example, clearly defines tools with built-in governance. Like you can't just ask a query that or you can't leverage a tool that isn't supported by the MCP. And the agent itself has no knowledge of what's going on inside

the tool. It only knows that boundary. We also have skills which are recently been relatively recently introduced by uh anthropic but they're supported by chatgbt as well as gemini. Um these provide context and somewhat looser formed defined tools. So with MCP it's very strict with skills you give indications on how to use them but it's a little bit more um uh flexible. So the first thing is

make sure you're focusing on determinist on structured knowledge deterministic tools. If it can be done by a script, do it with a script. Do not let the LLM come in and try to figure out what you're trying to do. If it can be done as a template, do it as a template because then you don't have to worry about the kind of crazy things that an LLM

might think would happen. And then limit the context that's needed by the LLM to be By doing this, you're reducing the degrees of freedom that the LLM has where it can go off and do something unexpected. And the third one is progressive disclosure. And this is a big buzzword these days, but it's not really any different than the things we've done. Don't give the agent every bit

of information up front in the same way that you don't give a human every bit of information upfront. and then surface more inappropriate context as the in uh investigation develops. Just like a good UI, we've all used bad ones where everything's thrown at you and you don't know where to click or anything like that. Don't do that. Set up the agent to follow the same kind of

process that you would if you were doing the work it yourself. So we've got these four layers that go into building the an appropriate context for an LLM. We've got the schema uh some of which can be derived through uh mechanical inference. Some of it can be derived by leveraging uh the LLM to do additional inference. Um that includes field names and types, relationships. All these things

are exposed um dynamically to the agent. The agent needs to know what it's looking at and what it means. Domain vocabulary. This is what we know. Like I know what CIRR is. Not everyone does and the LLM certainly doesn't know what it means to me. Um that has to be clearly defined to the LLM for it to be successful. Investigative patterns. Don't just let it go out

and investigate the world however it wants. Create your workflows, define them out, and tell the LLM to use saves you some time, but it also makes sure that it's f well, it it puts a higher chance that it's going to follow your process. And then again, you're going to hear me say this over and over again, deterministic tooling. If you can template it, if you can script

it, you should because you know the answer is going to be right coming out of that ideally. Okay. So the situation here is, you know, if we don't have these kinds of tools, we don't have a good way of sharing that knowledge, especially when we have new people come on. We've got a new engineer coming in, pabytes of logs being dropped in their lap, they get called,

they don't even know where to start looking, and they end up looking in the wrong place and end up having to call everyone in the company until they find the one person who knows the answer. It costs time. uh it means we like if somebody leaves we lose institutional knowledge and it means we can't move quickly enough relative to how systems move. So I'm going to give

just a little bit of an example of what I mean by this. We'll see if this works. Whoa, it didn't work. Uh okay, we'll try it like this. Okay, open link. Ah, that worked. So, I wanted to give an example of a Oh, that's not the right panel, though. Maybe I need to move it. There we go. How about that? Okay. I can't see it, of course.

Um, so this is what one of our skills looks like. We built this out to help us understand and explore uh insights with regards to CDN logs. So, a couple of things that are of note here. uh we really focus on in the skill itself defining uh the why, the what and the how. Like why would an agent want to leverage this? The what do they do?

Like what what would somebody ask that would trigger this? And then the how are like what are the various tools and skills that we have access to that would allow the agent to address these problems. And that usually comes in the form of a script like do this, do this, do this. That's the most powerful way to go about things, right? Um >> uh maybe yes, I

can as a matter of fact. Um so the critical thing that I want to call out here is that this is very short intentionally. So because of that progressive disclosure approach, we don't want to overwhelm the agent with knowledge that it's that with information that it doesn't know or doesn't know how uh need or know how to use. So but as a result we have links to

actual additional documentation and so we have a guide that goes far more into depth into what we're talking about. And so this is where we start to call out those domain vocabulary, the schemas and things like that. Um we have like a very clear definition of what the fields are, what their types are and what they mean, examples of it. Um we have additional notes on how

that can be used and useful. Um and then we have notions like summary tables which are an optimization not really important but the idea here is that the LLM can reach out to this when it's needed but none of this can be created by the LLM itself. Um I'm not going to go too much in depth into the other pieces just because for the sake of time,

but we also have the actual um MCP server that defines the tools that are allowed to be used by the uh agent. And then um this that's actually the MCP server. This one is the deterministic script where we have we know that we want to generate a regular dashboard based on an observation that we've had. This generates it based on the outputs of the uh queries and

makes sure that we have a reliable way to create new information that can be used by other people. Okay, I am going to go back to my slide deck now if I can figure out how. Cool. As an aside, uh one thing you'll hear a lot particularly when people talk about MCP is this notion of context bloat. And I will say it's absolutely a real thing. If

you put like assuming your LLM supports let's say 200,000 tokens at a time and what a token is doesn't matter. Just assume that you have a limited bucket that you can store information in. Once you start to hit it becomes very very difficult for it to determine this piece of information is has a higher probability of correctness at the beginning versus the later and it gets very

muddled. The problem with the MCP um is that it dumps its full definitions into the context uh without upfront. So there's no progressive disclosure and that that can be a problem. Um, but realistically, if they're well-defined tools, well-defined MCP servers, it's worth it because of the reliability, the determinism, and the governance that you get. So, yes, it's a problem. Don't be afraid just because it's a problem.

Okay. So, how do you get started with this? Start with the schema. You're obviously trying to do something related to data. So if you just write a giant prompt, it will break or it'll become out of date. But if you start with a well-defined schema that models the business practice or the operations that you're trying to do, you have already increased the likelihood that you're going to

get a good answer out of the system significantly. Secondly, on call first. And this isn't really about on call. This is about identifying the use cases where leveraging this type of translation of user intent to action is most critical. And in many organizations, that's going to be there's a problem at 2 a.m. and the on call engineer needs to resolve it. Focus on those use cases and

those workflows Iterate on the context. If you are working with an agent in an LLM, you're not suddenly out of the loop. You will observe places where it's going where it breaks, where it doesn't understand what you want to do or it does something different even though it understood. Take that time to iterate. Just like refactoring code, you're refactoring either prompts or you're refactoring uh the the

scripts that you're providing to make it work. and start now. Like don't don't wait. Build some skills, build some MCP servers and and take your specific problem and encode that knowledge and those workflows into the system. Then feed it to an LLM and you've got something that instantly provides valuable value. So the moral of this story is like the agent only knew what it was supposed to

do because the engineer decided what it should know. That job moved upstream instead of trying to do real time diagnosis. You're doing as much of that work in advance as you can by encoding the knowledge in scripts in skills in MCP servers so that your tool that you're leveraging has a better of chance of identifying the correct answer. The agent doesn't know anything but it can leverage

everything that you provide. Thank you very much. All right. Uh, does anyone have any question? Oh, thank you, Tom. Does anyone have any questions for Tom >> Beller? >> It's okay. I won't be hurt. >> Oh, yeah. No, let's let's let Come on. Let's grill him. >> Uh, this is a little more of a comment, but on the MCP bloat stuff. Yeah. And the idea of kind

of seriously constraining your agent so it doesn't do stuff. Cloudflare has done a lot of pretty interesting work on that. this code mode thing and then they they basically mashed their whole API into something that avoided API bloat. And I think Google's done a similar thing. >> There's actually quite a few solutions there. Some some places are are wrapping them in CLA CLI calls so that you're

exposing it that way. Yeah. >> Um yeah, it is absolutely a solvable problem, but the the juice is worth the squeeze. very and I it's what I took that was valuable is this idea of look just make sure you give the damn thing a >> exactly >> I uh I guess I also don't have a question but I wanted to ampl amplify some of the points you

mentioned which uh the skills.mmd file is the golden part for for my workflow >> and I even go so far um as to this is I I do this almost every week. I noticed you struggled with that last one and we're kind of flailing around. Are there any notes that you could learning as you can record in the skills.mmd file to make it go smoother next time?

And it always comes up with something great. So that that our skills.mmd file is growing slowly. And um I think that to tie that back back into one of your other points, those good skills at MD files are worth way more than verbose prompts up front. >> I agree. I will say so. First of all, iterate, iterate, iterate. Absolutely. Try to keep your entry point skill file

to 150 lines or less or it will start getting confused. Are there any metrics you maintain or like CI/CD for some of these markdown files or stuff that you like, okay, you have a new team starting to use these things? These are things you want to keep track of besides like the number of bytes in your MD file, >> right? No, that is yes. Um not as

many as I would like. So we have a number of um metrics that we try to capture. Uh so we have a set of golden tests which are basically you know testing for repeatability and then we also rely on um our tracing agent to say like okay when did the user come in with this particular query and when was it resolved and so we do track that

metric of like time to correct answer not as well as I would like. Great point though. Yeah, >> two questions. Uh, great talk by the way. Um, so my first question would be you mentioned storing pabytes of data. >> I'm curious about what you're using for like vector store things like that to actually serve these requests out. And then my second question would be I've noticed that

when I'm having LLMs write tests, they're generally a bit too heavy on mocks. And if you've done anything to mitigate that, if you've seen that the LLM that you're using. >> Yeah. So the question, your first question was how do we handle pabytes of data coming through the LLM? Is that is that right? >> Oh, sure. Sure. Yeah. Well, that's sort uh business promise of hydraulics. No,

we have like a number of optimizations that we put in place both automatically and uh and manually as well to take data that is of potentially high cardality or medium cardality and reduce it into useful bits and and create them into separate tables so that we can find that easily. As for the second piece, yep, that's a real problem. That is an example where like the context

that's provided either in a skill or often even better like a template or an example. Here's a good test can really improve things. It's not perfect unless you go straight down that templating road, but then you lose some of the power of the tool. >> Run run run run. >> Last talk. I gota Thank you, Tom. It was a really great talk. Uh, one of the things

I'm really curious about, so you mentioned uh at the start of your talk, there's a lot of fun things when there's lessons learned. Could you maybe talk about an incident where the LLM took a lot longer than you expect it to and it's like, oh my gosh, okay, we implemented this fix and then things worked better the second time. >> I mean, that happened to me this

this is not exciting in the sense that it's not like Nvidia's problem where they had something. Although, I will say like what he's done is as he's experienced more and more of these things, he's built these more tight templates so that when new questions come in from other users, they can be automatically resolved. But, you know, today I was trying to build out a uh dashboard that

somebody had requested and I had very specific desires for it and I I showed it in example and I said, "Okay, and here's a picture and here's the JSON and all of that and it kept doing it wrong." So, I built a template and it helped. All of a sudden, the tool was building something that looked like what I was trying to do. And it it was

able to use the inference so that I could say like, I want these fields, you know, basically pulled together and I want to be able to show these types of metrics. But it suddenly was able to do it in the way that I expected. That's not the most exciting use case ever, but it it really is that kind of scenario where you go through you go through

and you learn it and you say somebody mentioned like, "Oh, did you learn anything from this to the LLM?" And if you did, put it into the skill or put it into memory. That is an incredibly satisfying thing. I often will use multiple LLMs where I'll take a conversation from one and feed it to another and say, "Well, what do you think?" You know, sort of an

adversarial type of approach. And then that's really satisfying, too, because you've got now got two data points that seem valuable. So, it happens. I don't have a I wish I had a really good I'll make something up >> They don't all fire. >> No, they Yeah. Good. >> All right. It looks like it's it. That's it. Thank you, Tom. >> Thank you all so much for coming

handling the comments and Q&A. And thank you everybody for attending DevOps Day LA at scale 23. Tons of more Scale tomorrow. Douglas Comr doing the keynote in the big room tomorrow morning and I think nine. Don't miss that. Tons of other stuff. Open >> Upscale is tonight, isn't it? Is it tonight? Yeah. and Brina from Ask Brina Anything out there will be doing a uh one of

the upscales. It's a lot of fun and I think you get a drink ticket or two as well. What Hey, one other thing. If you're looking for free drinks, Click House, who is one of the LA DevOps sponsors, is has a has a little drink up going on right now at a place called Whimsy. So if you look on uh lu.mma or ask me I can because

they said hey bring your friends you're all my friend see over at whimsy if you do that sort of thing. >> And you do. >> I do I guess. Yeah. Let's put them over here.