Ballroom B Friday Mar. 06 - SCaLE 23x

This talk explores the speaker's personal journey into understanding cloud-native development, drawing allegories between cooking and software engineering. The speaker discusses how they began with limited knowledge of cloud-native principles and tools like Kubernetes and Docker, initially approaching the subject with skepticism but gradually discovering their connection to basic programming concepts. By using various metaphorical comparisons, the speaker illustrates the challenges of learning new technologies, emphasizing the importance of foundational knowledge. The speaker also shares insights about the structures and processes involved in deploying applications in a cloud-native environment, highlighting the necessity for clear documentation and accessible explanations tailored to various user experiences.

When I cook, I sometimes follow recipes, but then I have leftovers from those recipes. What do I do with that? I don't throw those out, right? I maybe put them in a soup the next time or in tacos or or like I put sandwich that I talked about earlier, right? Um, so I think about really more the way I think about is like, okay, some starch, some

protein, some vegetable for dinner, right? And I put those together and kind of make them taste good. And sometimes that doesn't turn out great and sometimes it turns out great. But I very rarely make the same dish twice actually as a result except for when I do fall. Yeah. So there's always some variation here. Um so um the other part is this is my kitchen. Not everyone

has this kind of kitchen, right? Some people have smaller kitchens. Some kitchens don't have ovens. Um, I never use my oven in the summer or I didn't used to because I live in Seattle and we don't have air conditioning and when it's get when it gets hot in the summer, I can't use my oven. My house is going to be impossible to live in. Um, so yeah.

Uh, let's see. When I was a kid, this one time I have to tell this story because it's just so cool to me. I went to the small Greek island on like for my fall vacation with my parents who were uh Greek professors. Anyway, we stayed at a friend's house and we went to the local bakery and there was a whole oven in the back that the

bakery did not bake bread in, but actually everybody brought their casserles and their pots to the bakery and paid like, you know, a small fee to the baker. Hey, can you bake my casserole? And then they'd come pick it up later. And that also is some form of of a way to feed your family, right? Like that. So, be aware of the constraints that you have. use

your community resources. Um, pull together and like learn how you think about this. So, all right, enough of the detour. Here are some of the components of a platform and I am yes going to compare them to cooking because again this is something that almost all of you are intrinsically familiar with. And before I joined tech, I definitely well when I joined tech as an adult, I

definitely tried to think about some of these concepts in the terms of experiences that I had already had. So it's very very valuable to think about these systems um in a more general sense. How do systems work? Not all systems are technological, but that doesn't mean those lessons don't apply. first we have ingredients, right? When we're cooking, we got those tomatoes, we got that basil. Some sometimes

we have staples, right? We always want to keep oil in our kitchen. Um, some ingredients are fresh. They will go bad. Like, we can't like store those tomatoes forever. Like, we can't eat those next week. Um, sometimes we can't find certain ingredients. Like, basil is best in summer. Some ingredients are super expensive. Like, mushrooms have been getting really expensive. It's super bummer. I love mushrooms. So, um

I thought if I think about that in terms of what I do with my data on my platform, right, I have to think about how do I store it? What kind of data is it? Is it my customers data? Is it my metrics data? Is it just simply all the information that I need from my users to help them? Right? Is it like um configuration data that

is also important data that I need to build a platform um and you know how much how expensive is it to store that data where do I store it um can I always that like the availability stuff is in there because sometimes I want to have cold storage for my data because it's just long term sometimes I want to have it available in a cache really quickly

right when in that famous interview question where they're like design Twitter or whatever you know So you're like, "Yeah, I need to build a timeline, so the data needs to be there faster." So those are always just kind of the things you have to think about. Um, right? So long-term fast access, slow access, and I just kind of put a few examples out here. Um, these are

not all the same things. Some of those are, you know, outright databases and database solutions. Not all of these are open source. Most of these um patterns have open- source solutions um with a possible exception of you know compute which we'll talk about next. Um but you know I put CSV files down there because that's a way to store data. I put JSON down there because that's

a way to store and organize data. Um right so whichever one of these tools is a best fit for you and your situation. Right? Um, sometimes you might have like super weird constraints like we need to like make sure that we have corporate approval to use this technology that it is safe and secure and has passed vetting. So you might not be able to use any of

this. You you might be locked into some different solution. Um, this is again this is why I'm not being super specific. This is why I'm being very general here. Look into it. Look into what you need. Um, ask people. ask the internet. Um, but but but this is this is a little bit like what we're talking about. You can store data in a lot of places. It

turns out next equipment for cooking. How do I turn my data and my ingredients into like something that can be eaten? How do I ship? How do I how do I what do I use to make all of this real? Right? So, you know, I I was thinking about this in terms of compute. We need we need like we need to like actually churn all this information

and like make the app run, right? So, stove, oven, like these are all ways to think about it. Honestly, this is like my weakest comparison here, so don't don't squint too hard. Um, but yeah, we have compute, we have servers, they need to be powered, we need to like interface to these servers. How do we talk to these servers? there uh you know, right? Like not everybody

is like super excited to SSH into a server and do stuff in there. Um right, how can we make that palatable to people that, you know, where that just isn't in their toolkit, right? Because those people are shiny unicorns and know how to make a website look really, really cool, which I totally don't, right? So here are some of the ways in which you can make sure

that compute is available to your platform users. Again, there are some enterprise uh icons on here. Um, mostly because those are just some of the solutions. Um, uh, this this weird one right here, actually, I don't have a pointer. Um, the the blue one with the bubbles, that's actually GitHub actions because it turns out you can run a lot of compute on free GitHub action runners. Um,

people people have mined Bitcoin on them for a while and that was a problem, I think. So uh but also you know what's the hardware you're running on? What operating system are you using? These are all real concerns for when you have your compute up. Cool. Next, what is the event? So when like how how and under what circumstances do we actually want to ship to production?

If our app is broken, we don't want to ship, right? And we can help our users with that. we can like make sure that like um we have we have tests, we have CI CD, um we have gateways, we have um we have uh what's it um canary deployments where you only deploy to a few resources or a few users and um if those break then you

have a signal that like oh whoops maybe maybe we need to fix it before we ship it on the whole. Um, and so these are all of the things that you can think about, right? When when you're when you're planning for an event, you want to make sure that you know what you're shipping. You need to make sure what for, right? Is it a big release with

a blog post or is it just like a little bug fix? How many, you know, how large how large is your production environment? How many, you know, how many people are you shipping for? Um, right? like how many servers do you need to run uh to run sales on Black Friday, right? You got to know that. That's that's basically a thing you have to think about. Um

and again, here are some solutions. This is mostly about the moment when you hit deploy. Uh the Kubernetes icon is on there because you know, cube cuddle apply is absolutely how you get how you ship your ship your things. That's it's open source. it's available to you. Um, GitHub actions is a good way to gate your deployments. There's Argo. There are some tools uh this is called

basic this is called Git Ops by some where you let your you let the status of your code in git determine whether it is ready or not to go to production and there's lots of tools that can help you with that. Um, I'm happy to answer more questions after the talk if we have time. Um, just to go into a little bit of detail for that. Um,

next up, we want to make sure that our app runs in production as we expect. We want to have metrics, right? Like in this case, you know, if we think about it, was the food on time? Did the food taste good? Did anybody die eating it or have, you know, have like a massive like allergic reaction? Um I uh thankfully there are there are no food allergies

in my family so I don't have to really worry about that super hard but um you know th those are things you want to you want to ask maybe before you so anyway so those are metrics you want to gather metrics for every stage of your deploy um of your you want to gather metrics on the state of your servers you want to gather metrics on you

know maybe even the state of your data Um, right. Did did we deploy successfully? Did users engage? Also, how much time did we spend building all this? Right. Was it hard to deploy? Did somebody have to like, you know, fly out to a town and pick up a pick up a suitcase full of data sticks and like fly back to another town and plug it into a

server? I don't know that that that is probably you won't you don't want to do that, right? But um cool. So, here are some metric solutions. Again, some of them open source, some of them not. Uh, I see someone with a Graphana t-shirt here. So, I feel I feel I feel super glad that I picked that one. Um, I put Pager Duty on there because that's what

happens when your metrics go sideways and then you get pinged at night. Um, but but basically, this is where you store your logs. This is where you have your dashboards. This is where you know the state of your production system. And again, you want to build this in a way that your developers can use it, right? You want them to go to those dashboards and realize, oh

no, I'm seeing too much traffic or like something seems weird or right because they know their app better than you do. Oh, you just built the platform. Uh, finally, safety, right? I talked a little bit about like, did anyone die while eating your food? Uh, you want to prevent that, right? like by the time you get the metrics on like oh no somebody died eating a mushroom

it's too late it's too late you want to have practices in place I am not a huge security person so please attend security talks that are not here that that are not mine I mean to get more information on this but do consider it right um we want to make sure that you know we secure our data that our environments can't be hacked we want to make

sure that everybody who interacts with our platform can be trusted. Um, and we want to prevent incidents and breaches. Um, so I put some, you know, there's a key value store on here. There's uh there's uh there's again you can use you can use GitHub to like lock down who has access to your repositories. Uh there's a little icon down there with a teacher. You should educate

everybody that uses your system on how to do so securely. Um security training is a big deal, right? You can also hire a team um I let large companies do this. They hire a team of hackers to see if they can like break in. Um right, that's just some form of like security testing. Um so these are things that you should think about and collaborate with your

security experts um so so that you have this piece of your puzzle. Great. Okay. So that was the components. Um let's see. I'm going to quickly walk through a couple of examples and then um leave time for questions at the end. Um so, uh my first example is when I was at GitHub, I maintained the internal deploy platform there. Uh we called it Moda. There's no reason.

It was just a name. Um but basically, let's start with the data layer. The data component from the perspective of us was not so much all the data that GitHub churns because there was a whole data team to deal with that over there. We just needed to make sure we that our apps could connect to it and we really didn't do much but we did need the

data of what are the apps who's deploying them where do they live and so basically we took care of that by saying you your app has a repository on GitHub your microser is a repository on GitHub. Um then our platform was bare metal Kubernetes clusters. That's that's that's basically it. We had multiple clusters and we had load balancing between clusters to make sure that we had greater

uptime than a single cluster would have done. in order to deploy it was basically just cube cuddle apply under the hood but um GitHub had a big thing with chat ops. So you would basically go into Slack and tell the robot to deploy your app and the robot would know what to do and forward it to cube cuddle and then your app would go to prod. Um

deploying github.com was a little bit more safeguarded than that but this is basically what it boiled down to. Um for metrics we used a couple of enterprise solutions data dog and Splunk. Splunk for our log storage, data dog for our alerts. Uh there was a quick uh experiment made into using uh into using the elkstack with elastic search for uh metrics. Uh I don't know that that

took on and then we all got pedagied when stuff went sideways and when and somebody ran cube cut namespace delete-all. That was very fun. Um and then finally um our security was basically who has access and yes I am not kidding you but the security team at the time at least had access roles in a big was it a YAML I think it was a JSON file

or YAML I can't remember but your roles were scoped where basically your username was in that uh in the security repository again super boiling this down um but yeah so there you have that you have you have you have your repositories that are scaffolded in a certain way to allow us to deploy your application onto Kubernetes. Actually, you can deploy it yourself because we built the tool

for that. And then we are also providing metrics and alerts and we lock the system down by not allowing anyone that doesn't work there to have access to the GitHub repo and to have access to the production environment. that was locked down to my team and several other teams on like sister teams. Um yeah, but so that's that's like a big enterprise solution, right? This is when

you have when you're in a company where you have a team um or you have the ability to have a single team that can just manage the platform for everyone else. That's not everyone's situation, right? Like what if what if you're running like just a small scrappy startup? What if you're what if what if you're consulting and you just want to have like some basic systems to

you know to like repeatable patterns to pull out of your sleeve. What do you do then? Um so that's kind of my my current place. So yes, I work at Palumi. I promise this is not a sales pitch. This is just an example of um what you do when you want to have some of these components not baked for you by a dedicated team but if you

want to basically um just like pick and choose get them you know get them peace meal. Um so what pulley is is it's the way the part of pulley that I work on is infrastructure as code. You write a program. Um, you run the program and it deploys cloud infrastructure for you. So you can deploy, you can ask Pumi to get you an EC2 instance or an

AWS Lambda or uh, is Cloud Run and Azure thing? I think it is. Um, and Pumi can interface with all of that for you, which means that your teammates might have an easier time again deploying things because they don't have to go to like the AWS website and click around for RO permissions. you can write a program that does this for them, right? So, um that's basically

the form of templating. You can write a template that will work for most people at your at your company. Um and then um and then hand that around. And that's we call that a component, but really most of what it is, it's a reusable piece of code that can be written in a programming language of your choice. Um that you can then hand off to people and

say, "Hi, this will like deploy the infrastructure that you need to run your app somewhere." Um right. So that's how you deploy compute resources. Pumi 2 has a command line tool. uh with Palumi up um there is a there is the option to do CI/CD integration with Palumi deployments however that is unfortunately not open source um and then again there uh for metrics and alerts there aren't

any you know it's it's kind of like you you bring that you can bring that yourself right so this is really more of a composable um set of tools that you can use to build your own developer platform Um similarly with security uh if you want to stay open source you bring your own. Um so yeah that was a woman tour about some concepts some examples and

um you know just just I want to encourage you all to uh to think about these patterns in the context of which which other complicated really complicated systems that you have in your life and that you know how to solve already. So yeah um I think we can do questions. Uh, I will probably turn this mic back on and hand this mic to run around if yall

have questions. But thank you so much for coming. I hope this was helpful. All right, any questions from the audience? Thank you so much, Guyavir. Um, got a lot of recipes cooking now. um, questions I think one of the challenges for me thinking about developer platform is that like a lot of developers um can be also opinionated in terms of like workflows that they've done different languages,

different tools, you know, um and different frameworks require different flows and balancing that like trying to maintain a standard set of like interfaces feels sometimes challenging. And so like finding that balance between like saying yes to cater to everyone's workflow and preferences to like limiting avoiding like too much um like sprawl. >> Yeah. Um so for those who I did you all hear the question? >> Perfect.

Um yeah so that's a really cool insight and for me this comes down to who are your customers? Are you building a developer platform for someone who's paying you or are you building a developer platform for your colleagues? Because if you're building a developer platform for your colleagues, it is uh it is possible to, you know, make rules for them, right? Making those rules and enforcing them

is going to be easier if your platform is easy to use, right? But there are going to be moments when you are when you need to actually feel empowered to say we can't do that for you. You should try it our way and you will get the same results. And if you truly cannot get the same results doing it our way, then come back to us and

talk. You can do that if you're an internal platform to team which was my situation at GitHub. And it felt really great. And did we solve the problem for everyone? No. No, we didn't. It didn't work. It's never no platform as I said is ever going to work for anyone. Um when somebody is paying you however you don't have that option and and again I speak from

experience. That's my situation at Palumi, right? We don't do this to like run some giant like code sharing platform somewhere. We do this because other people are paying us to create solutions for them. Um and so yeah, that's where that's where you have to be really smart because I think you can still push back and um if your product is very very good, you can say please

try it our way, right? You do have to listen to what people need. But you're right, a lot of people get like really tied up in the details and they're like, "No, but I I I do want to use Haskell though." And it's like sometimes it's actually okay to say then maybe we're not the solution for you. Um but on the other hand it's like okay well

well how about how about how about YAML though can can can you maybe do that right because maybe you can translate it to hasll on your end right it's it's it's it's a tricky line to walk for sure this this is um but yes I would always say listen first come up with suggestions people want to be heard and make sure your platform is easy and intuitive

to use Thank you for the talk. Is this on? >> Yep. >> Yeah, it's on. >> Perfect. Um, you had just mentioned uh that the platform should be intuitive and easy to use. And with some platforms, you know, you they inevitably get more complex as time goes on because you have all these edge cases that need to be handled and it has to be flexible in certain

ways. And one of the big challenges when that starts to happen is discoverability. Like how you don't want people having to go to a Slack channel and say, "How do I do this?" Right? You want them to be able to find the answer >> on their own. Like we you don't want that element of friction in in the >> So in your experience, like how have you

thought about solving that discoverability issue? And just in general that aspect of making sure the platform is easy to use and intuitive and people don't have to go chase other people around to find out how to do what they need to do, >> right? You're going to hate my answer. Everybody hates my answer. And the answer is you must have excellent docs and keep them up to

date. Like I'm I'm sorry. Like I seriously use AI tools to keep your docs up to date. you must if if you really hate writing docs that hard, then again, if you hate writing documentation this hard, I don't understand where you're coming off wanting to write lots of prompts all the time because it's basically the same thing. Um, so those were a couple of hot takes. Um,

but yes, discoverability is right and you need to have the docs also be structured in a way that it makes sense to people. It's not just have docs. It's it's more like no, your docs need to like get started. start with the base case, you know, then have like footnotes. Hey, if this is your situation, you might want to use this or really if this is your

situation, then come talk to us. But like good documentation is the first stop. Um, next, I think, you know, you you should have some troubleshooting pages. Uh, your docs should be searchable. Um, honestly, my pr one of my proudest docs accomplishments was writing a really terrible JSON index-based search for the doc site and everyone was like, "Oh, thank you. It's it's so much better." And I'm like,

"But it is so jank. It doesn't even like reload after you ran a search. You have to like reload the the whole website to get a new search box." And everyone is like, "It's it's fine. It's fine. This is better." Um, so so yeah, so that's documentation. Um but but you can also make the tool itself like be really easy to use. One of the things that

worked for us at GitHub was that everybody was already used to using chat ops all the time. So using chatups to say you know hey hubot deploy kind of felt like you could almost discover it organically. Um so yeah sorry no secrets. What businessfacing metrics do you think are useful for justifying a platform team? >> Oo, honestly, I've never been in in a place where I've had

to make that decision. When do you want to run a platform team? I think I mean, as with anything, it depends, which is sort of the copout answer. Um, but I do think that honestly I would look at what amount of time your engineers are spending on ops, right? If they're spending more time on ops across the board than they are on anything that has to do

with new features, you might want to consider a platform team or like one SR person, right? I feel like I feel like as soon as your developers are spending time on things that like don't make sense for them anymore, that's that's when you flip. And it doesn't have to be a big team. You can also go, you know, the the like takeout order route and order your

food. You don't have to cook your own food, it turns out, but uh you know, you can pay someone, right? I I I I hear I don't know. Heroku used to be really great at shipping stuff. Pay them, for example, right? depending on your situation. Yeah. Um yeah, I think we have like one more question and then >> it's working. >> It's on. >> Okay. Yeah. Yeah.

I really liked your comment on the communication documentation. Uh self-service usability is like how you measure if your platform is good. I think but the I I saw in uh some bigger companies that uh they build uh a platform platforms And um they become really uh they become tech debt of sort of >> where you know you have great engineers building that and it becomes really integrated

into the company and as the as they ages and the the industry has better solutions it actually becomes a burden. What what do you think about >> Oo uh that's that's always right. I mean, same similar story, right? You have to stay on top of things. You have to be um I haven't quite I have not figured this one out because I've always been I've often been

in situations where it's like, oh, clean up this tech debt. I had to learn about Capistrano in the year of our lord uh 2019 I think um because it turns out there were some capistr things running somewhere deep in the bowels of our system right um so this is so real um it is unfortunately this is honestly the biggest reason to not go into any kind of

platform engineering or devops Because maintenance work does not get rewarded the way that feature work does. It never does. It is true in public as well. When was the last time that a politician got elected for keeping all the bridges from running and not collapsing and not killing people? That never happens except lots of work actually goes into making sure people don't die in like buildings and

bridges that already exist. Same story unfortunately. So the real thing, the real answer I think here is work for someone who recognizes the importance of maintenance work and sets aside time to elimmit that tech debt because it's always going to happen. Yeah, it's it's not a good answer. I I'm like I'm like make sure your leadership is aware that there's there's there's a calm duck on the

surface of the pond and it's paddling really hard underneath. Yeah. Anyway, thank you so much. This has been really fun. Um, I will be at the conference all weekend, so I'd be happy to meet some of you and chat. Um, please forgive me if I forget someone's names. It's I've met a lot of people. So, thank you so much. All right. >> All right. Well, thank you

all for being here. The next talk uh will be at 11:15 and uh as a reminder cloud native track is happening in ballroom A and ballroom B. All right. Thank you everybody. That was a different ballroom, I believe. Test one, two. Test one, two. La. All right. So, I'm not check Oh, it's because our levels are all the way down maybe. >> But if it's coming out

the master, should wouldn't we also need to crank the master volume? >> No, these ones it's not coming out the master. It's coming out the So this return right here is the control volume. >> Ah, okay. >> Test one, two. Oh, that did it. And then keep talking. >> Yeah. Test one, two. Test one, two. D. >> Yeah. So, we have to have the controls. >> Yeah.

That's the volume. Cool. >> That was the one I having trouble with. >> Um, it it was Just go ahead and test that because you're what you get on your board is a balanced feed from us. So, it was it was the whole room was quiet. So, if you can test that one real quick. Make sure it sounds nice. >> Check check check. Yeah, it's balanced. So,

yeah. Cool. >> All right. So, I'm going to turn up the ox turn just a little bit. >> There we go. >> Uh, those of you in the room, how does this sound? This sounds okay. Yes. Cool. Testing, testing, testing. Oh, that seems nice. Yeah, that seems Okay, they're fair. So, um, this is the mic goes over your left ear. >> Oh, okay. If it doesn't stay

in place, they've given us some medical tape. >> So, you can actually stabilize it. >> And then do a few tests. >> Test. Test. Yeah. Testing, testing. Test test. Hey everyone, thanks for coming. I'm Justin Garrison. I'm one of the organizers of Cloud Native Days LA. If you are in this room for I think it was called Who Broke the Build uh that talk is not happening.

So uh uh they could not come. So it's what was in the printed schedule but online in a guide book and I think on the sign outside it says the wrong thing. We do not have control of all those all those things. They all happened uh after that talk did not happen. So um just so you know if you were here for that that's not it. You

have a wonderful talk instead uh from Kiss High Tower. Uh but first I want to give one quick shout out for our sponsor from Minimus Adam. Uh he's going to give you a real quick thing. Everything here at scale is volunteer run. We all this building the food everything does cost money uh surprisingly. Um and so we do rely on the sponsors. So just wanted to give

a real quick shout outs. Uh they get to talk in front. Um they do you have a table out front uh for minimizing. So let's take it. >> Awesome. All right. I'll be quick. Uh Adam Clark at Minimus. At Minimus we build um compliant by default minimal container images that are meant to be as vulnerability free as possible. We do this by shipping uh purpose-built images um

where we build from source continuously. So any change that's checked in the code from an upstream open source project that um either whether it's functional based or vulnerability based we basically check those into our own code our own universe um you know the same day they happen and build images within the same day that they happen. So at the end of the day um you end up

with images where if you compare them to their public sources off you know their public repos they're generally uh 97% to sometimes 100% reduction in vulnerabilities. So the whole idea was, you know, how can we further help continuous remediation of CVS by just reducing what we put in from the beginning. So if anyone wants to hear more um and how we're doing it, he said we have

a table out and we'll be at scale in the exhibition hall all weekend. So thank you. >> Yeah. And thank you very much for sponsoring. This is very great. Um so yeah so with that we will turn it over to Elise for you are a full-time student at University of Washington and uh excited to have you part of the community with cloud native and learning this whole

experience of >> thank you. Okay yes the mic works. Okay awesome. Um well first I just want to say thank you for coming out or at least in this room. Um I will say there was a little part of me that thought it was just going to be me and my mom and my dad just right here. So this is great. Definitely more than that. Um, and

if anybody has known from the rendition that I did twoish months ago, um, there'll be a little bit of similarities here. Um, so if you didn't catch that one, this is definitely a good way to be able to see some of the content there, but then there's also some things that we're building off of that. So, just wanted to give y'all that preface. Um, and then when

I first had the opportunity to do this particular talk, um, it was back for a meetup and it was back in January. So I'm just kind of going to run through how I kind of came to make that first part of it and why I even got into cloud native to begin with. So what sparked that? So we had I was in conversation with the organizer of

it's like cloud native PDX it was. So shout out to them. Um and she was saying like you know hey would you want to do this? You can talk about literally anything. Um she said let me see if we can go here specifically yes any topic that you'd like to present around cloud native and Kubernetes which was awesome. It was open. It was free. um she's very

flexible with me. Again, much appreciated. Um however, there's just a slight issue with that. Um at least for me, it's just I had no idea what cloud native was or what Kubernetes was. And that's exactly what the conversation or the talk was supposed to be about. Um so I kind of felt like I was at uh not even square zero, but like negative one. Um so of

course I did the natural thing that you do when you have no idea of a topic or what's going on. Um, I told her that that was perfect and I would get that over to her by the end of the day and wrote some abstract that I was just gonna have to pray that I could come up with in the next three weeks. Um, which is when

the talk was. Um, that's okay. That's okay. We we we pushed through it. Um, so I was like, I better get a start on this. Like I put a really big time limit commitment for myself. So first start was probably to ask, you know, hey, what what what is cloud native? because I just put that into the search bar, see what you know, see what comes out

just to see what I'm getting myself into. Um, and I got a couple different answers. Um, one was there's no one-sizefits-all answer. Um, that answer changes depending on who you ask and that really varies based on case to case. And if I don't know if you've ever learned something new or done anything new and you just see just vagueness right off the bat, I was really really

scared to move forward and I already had something typed up that was like I can't do this anymore. Um, but it's okay. Kept kept going, kept going through and then I was able to kind of start diving into some general videos and then get my own consensus because that was consensus from other people. So, I needed to make my own. And then what I did find, everybody

seems to agree that cloud being on the cloud does not equal cloud native. I said, okay, I can I can get behind that. And there was about four other um topics that was or I guess conclusions is a better way to say it that I gathered from all these videos that I was watching. And the first one was to keep in mind was the architecture of it.

So actually how do you think what's the word break down applications rather than being holistic. It's just the app because before this in classes and everything, you you built the app. Maybe you're learning the language on how to do it and you deploy it somewhere hopefully somewhere free um with a very long domain name and you just kind of keep it keep it out in the world.

So, this was kind of a topic that I was going to have to struggle with with digesting. Um the second was containerization. And remember here I'm just trying to look and lay the land. So, I'm not quite diving deep into what these mean yet. Just kind of setting a base. So, containerization. So these parts of this architecture seems like they need to be packaged up and it's

like okay that's something I can get behind and then container management. So I was like okay packages packaging those up and probably need something to manage all of them. So right now I'm I'm I'm following I'm good I'm here. And then the last one that I kind of gathered was like continuous development continuous integration. Um, so kind of which the the way I was able to do

this for myself was very independent, but I still liked exploring what this might look like in groups and for other things and other ways that this might be used in in industry is what we say in classes. So, but I guess that's really just anywhere. Um, so this is what I this is what I gathered. This was my my conclusion. So, now that I had this um

I was like, "Okay, it's time to it's actually time to get going. I have this thing to do." and um he mentioned that I'm a full-time student at at UDub. And the first thing that students do, my generation does is when we don't know how to do something is we go straight to Chad GBT, which is exactly what I did. I typed this exact prompt verbatim into

that. And if you can't see in the bag, it says, "Can you build me a cloud native app? Don't really know what cloud native means, but I have a talk to give. My dad's going to be there, and I'd really like it um to think that I'm doing what I'm doing. Um so, make it sound really impressive." and then entered and then um I was finished. So

that is actually the duration of the talk and I hope you guys have a good rest of your day. Um um but of course that wasn't the way that it was used. Now however I did use some form of AI but not in that type of realm. I asked it for a blueprint. Like I said I have no idea where I'm going. I have no idea what

to even think about. So I'm taking this with a grain of salt. So I'm sure everybody out here has their own sevenst step. Maybe seven's too little, too much of how to actually get this done, but I just needed something on paper so I know how to move forward. So build the application, containerize it, orchestrate, add whatever this cron job situation is, deploy to AWS. Okay, I've

heard that before. And then configure some public access. So yeah, I'm already like, okay, you know what? We're just going to take a break. I know how to do step number one is build the application. That's what I can do. And that's what we did. So step number one or Yeah. Oh, let's actually go back. She should play. There we go. Give her some time. Okay. Um

and then kind of while this is running and why I sparked this, you can tell I was having some front end issues as you can see this big new strut across the front just to see if I could test to see if something was coming back up. But um I am um a senior actually um at the at the in at the University of Washington studying informatics.

Um and right now in my life there's a lot going on. you know, to have classes and capstone, applying for things, um, finals, it just everything under the sun, honestly, is how it feels. So, when I realized that I had to make some some kind of application just to test these concepts on, I was like, I probably should do something that's a little bit beneficial. So, what

is happening here, I know it's already changed from from green to yellow, is kind of a burnout score. So, you start it when you're studying or doing whatever it is that you might reach burnout with doing. Let it go. The score goes up and then as the score goes up, it's indicating that you're getting closer and closer to burnout. And then it'll it'll change to red and

everything. But creating this as this visual reminder of like, hey, it's kind of slow down because I think right now everybody's telling me to go fast. What's next? What's next? What's next? But I just needed something to almost penalize me for going too fast. And then when you take a break, it kind of starts to go back down. um not as much as it increases but you

know just some standard logic and then of course when you press end then your session is over. So, nothing too complicated, just a quick little front end, backend situation, but I just needed that that base um so I can understand kind of how to build off of these topics on here, right? So, following that, let me make sure it's pulled up here. Following that, okay, that's when

steps two through seven had to come into play. So, I actually did have to start learning these words that just felt very, very foreign to me. And then at this point in my journey, I'm going to call it a journey, is where I have my first hurdle here. And I seek out usually videos to watch um for some for guidance on how to learn things. A very

visual learner and I kept getting things like Kubernetes everything in five minutes. Five minutes to cloud native and then of course Docker in five minutes as well. Um if anybody here knows why five minutes is the golden time for learning something new, I would love to know that. Um and a little part of me was like awesome, this is great. I'm going to pretty much be finished

in 5 10 15 minutes. Um, so this was this was awesome. But then I actually clicked in to what these were and it was it was a it was a different vibe than I thought it was going to be. Um, it was it was essentially saying like there's a lot of words of, you know, skip that part. We're just kind of going to roll over this and

you'll get to know this later. So I'm just like, okay, I'm feel like I'm missing a lot of parts here, but they were just saying if you're able to do steps one through four, you'll be good to go. And the thing I had to realize is good to go for them means that I could have ran or deployed whatever they were trying to tell me to do.

But good to go for me was understanding and that that's where um expectation gap was there. So it's just it was interesting to see that it was hard to get it's hard. So I yeah stop it for myself. Um so I had to start typing in things like the long extended version of whatever it is I was trying to find to actually get to something that was

a little bit more concrete that wasn't afraid to go deeper um in fear of turning me away. So but it got done. It got done. I'm okay. Um, and then we we moved on to Docker, our friend. Um, I also just really love the logo. Um, very cute. And what probably familiar with on this side, at least I would hope so. Um, it's just kind of, you

know, a standard Docker file there. And then another kind of I'll call it a fun fact about myself is that I love I love to cook. It's something that helps me d um unwind. Yes, we have a fellow cooking. Yes, it's something that helps me unwind. is and then you get get something good to eat out of it. So, you know, it's a win-win situation for everything.

So, um when I saw this, I essentially saw a recipe and you know, me and my relationship with recipes are a little bit interesting cuz I could write one. I could I have not. Um it's just cuz it's so hard. You know, there's a pinch of everything, something of this. I don't know. I don't know what I'm putting in. And it's really difficult when I go home

and I cook for my family and they ask every single time like, "Can you please please please write a recipe down so that we're able to recreate what you're making this dish when you're not here?" And and I was like, "Okay, that makes a lot of sense." Um, that recipe never came. Sorry, guys. It's not going to. Um, but it was at least helpful and beneficial for

me to connect this back to what it was. So when I when I'm learning something new, I always try to do my best if possible to connect it to something that I already know and then it it feels a little bit more intuitive. So I got these Docker files going. It's great. I basically feel like a cloud native engineer right now. I'm good to go. Um and

then that kind of all went down just tanked all the way down because it was time for Kubernetes and I I had a a lot to to front with that. um it was just a lot larger and I definitely definitely knew I couldn't do it all in five minutes. So I had to kind of go back to the basics and understand the things that I need to

do. That means typing in the long long version of things and all the things that it would take for me to kind of get videos and tutorials that felt good for my learning and I felt comfortable enough moving on to the next step. And there's a couple things that I gathered um was you know one kind of containers being this runnable unit like okay I can work

with that I see that and then two pods and deployments um running and maintaining your app. And I will say the first thing that I saw this about just they said that pods just die. I remember when I first talked about this why why on earth would we use something that just dies? I was I I really had no idea why this made any sense but I

was just at this point just like I'm sure it'll come up. Um, so that's running and maintaining our app. And then we have this service to essentially just because these things just keep dying. Um, having this stable network inside of our cluster. I was like, okay, I'm starting to see the layers and how this goes. And then for this case, we had ingress to control access from

the outside world. Again, very, you know, boiled down simplified, but that's kind of what I gathered enough again to move forward. Um, yeah, and then this is kind of this again another summary. deployment service in ingress, how many, how they stay alive, how traffic reliably reaches them, and how the outside world gets in. So, we're good there. And the next thing I know if you remember from

this from the the sevenpoint steps was it said I was supposed to be adding crowns to my thing. And at the time, I thought that was the next tangible step here. Now, that that was extremely unnecessary and definitely not something that you just do for a simple application. But I did it anyways. And I don't regret it because I actually do think maybe call it a hot

take, but I do think that kind of doing something unnecessary for a simple project can make it seem less scary when it is necessary for a bigger project. Um, but so I played around with it and it what did I end up making a change to? I think I had it where the uh the timer the way that it increased to burnout was a lot faster after

midnight and then up until 6:00 a.m. So that was kind of something that added there. Again, nothing too crazy, but it kind of helped me at least understand that there's other ways to configure these files. The so far the only command that I had in my Docker files was just to run. So seeing this to go into a different file and run a different command um which

is something I wouldn't have known without going down this extra extra long rabbit hole of a path. I respect it. Maybe not going to see me anytime soon, but I respect it and I'm here for it and it was a part of the journey, so it's here. So then it was like, okay, we're breezing through this list at this point. I'm good. And then I kind of

go on here. It's like this like feels like my last step. Um, was it Amazon AWS, EKS, whatever it was was there. And then, you know, I've deployed a couple things or this is at least in my brain if this is the next step of hosting. And and I check I'm I'm a college student, 18-year-old college student for that matter. So, you know, we have some costs

to think about. So, that was just the first thing that I asked. It's like, hey, what's that going to look like? And don't have to read it all cuz I didn't. I just saw 7274 and said no. So, that's not it was not for me. I did no more exploring. So, if you guys know of any type of secret cheat code way to have got it down,

let me know. But at the time, I didn't have time to find out. So, and I didn't. So, now move over. And then today was digital ocean. So, we switched over and because this is something that I've heard my dad talk about a couple times. So, I was like, "Okay, I I um I'm familiar. I'm comfortable with making this pivot." Um, but this is kind of where

the next hurdle for me kind of came in. And I was so close to the finish line. This was like I want to say six out of the seven that it was time to be done. And I went I kind of went back to some of of the learning practices, if you even call them learning, that are reinforced in school and what are I'm told to do,

which is opening chat. And I said, "Hey, I'm I'm at the finish line. I just made this big move from AWS and now I'm at Digital Ocean and I just really don't have the time. Just I did all the good stuff. I'm basically prepared for the talk. Can you please just like finish it off?" And I'm sitting there and I'm sitting there. It's just error after error,

error, copy paste. Why is it this error? Here's this fix in this secular loop. And I was there and I was just like, it was telling me to put things into the terminal that just no way was necessary for this minuscule of an app. And I knew it was time to put the computer down and just completely restart after. It's just it's genu it's not verbatim, but

this is pretty much what it said I needed to do was to completely get a new computer and restart the entire thing and then it would work. So I saw stuff like that. I said, "Okay, I have to sit it down." is it's this and I actually am appreciative almost of that part of of chat or AI in general of being just so ready to give any

answer when it doesn't have one because I can spot when that answer is not the one for me. So I said I put it down. I came back the next day and I said I'm going to get back when I'm ready to learn. And that wasn't even the next day. It was the day after. I had to set time aside to actually be open to frustration. And

I did. We got there. Found tutorials and the videos. I did it the long way. And I still use some of it as a as a way to reinforce my learning. So nothing, you know, it wasn't out of the box completely. Um, but I got there. We were good. We're all We're okay. Um, and I just put that there because something about seeing your application on a

different device other than your local host just make it feel a little bit more official. Um, but we were good. We had them had it up and running and had something I could see and I was proud of, right? And then I just also wanted to add this in here because I have this all written down in these nice bullet points and things like that, but this

was really me honestly. I think in the talk I said like a week before, but this was really the night before. Um just trying to figure out what the hell did I just sign up to talk about. I still don't even know and I have something working which which is interesting because I I had to make a lot of sacrifices or I guess I I sold out

a little bit at times. It was just like I it was too hard to find the long answer. So, I had to settle for the short answer. So, I tried to go back with the time that I had to figure it out. And it worked a little bit. I I'll give myself some credit for it, but you know, it was there. So, but with all of that,

I at least knew that I could go from this to this, which it looks the exact same. It looks the exact same. And this was really really challenging for me because I I was um a lot of the times I'm like, is it working? Um and then I'm, you know, kind of this is where I do use chat a little bit. It's like, hey, this is what

happened. Did I break the entire thing? Like nothing's showing up. And it's just like that's exactly right. And then okay, okay, I can't do anything with this. Um because when I first started learning how to code, getting into the tech realm of everything, everything was really visual. It was, you know, we had Scratch and putting things together, having little missions of what it needs to do. I

think I had like hour of code, not an hour, lots and lots of hours, I fear, but um we had things like getting Elsa to the other side of the ice castle. It was just lots of things that made it seem like if this then it does this very visually. Even like when you're learning like HTML or web development or something like that, it's still very reactive

in terms of when you do something there's an output and that's what I was used to. This so this was so frustrating for me because I was waiting to see like at least some type of congratulations now it can talk to like I don't know it's just something and it just never happened. Um so I just had to do a lot of trust. Um and it's only

trust because I didn't get the in-depth answers that I had. it was fa face at this time. Um, but I appreciate it and I know where the things are and they they're there um not even hidden anymore. So, I kind of was able to see that not only does the application that I made um look the same and that's actually a good sign, but it actually works

for the better for other people and outwards of me. So, it's thinking less about myself and about other people and that's just a hard thing to do period. So, integ or not. Um, so I'm there and then I was able, you know, I'm finished, I'm good, I'm confident. So now if somebody were to confidently ask me what is cloud native, I would, you know, be able to

say it depends and I still don't know. Um, but this is where I actually was able to stop in January. I stopped here and I kind of just let it go back in my brain. I was able to finally close out my VS code until it didn't break. It was great. All the tabs were closed. It was very freeing. And then this talk came around and they

were saying, "Hey, like this is something you could do." And it's you have a hour time slot instead of 30. And a little part of me was just like, "Okay, I'm just going to talk really really slow because I have I do not have time to fill this up." And of course, free trip to LA, like it was it was dope. Um, but then I remembered how

how frustrated I was going back and trying to learn this. And I said, "Okay, I have time now. I already have a a basis of my of my slides finished. It is finished. I could stop here and tell you all to go to lunch. So, I could have been done, but I decided to take this time to to dive in without pressure because there's there's a difference

between learning for production and learning just for personal understanding. So, I wanted to switch into that mode and then that's kind of what I did. And then the next screen I'm going to show, don't get overwhelmed. We're going to break it down, but this is how I my brain worked in drawing it out. In high school, I was always drawing things out. It It was kind of

a thing that my notes were just so clean and great to reproduce that any open note test, I would have to print out extras to give out to people. Um even though they should definitely be writing their own notes, but they paid me for it, so I wasn't going to let them in on that. Um and then, but this is kind of what for me, I said,

take a break, take a breather. Um I know it's a lie. But we're going to dive into it. But I just kind of wanted to zoom out for a second and show that this is this is really what I spent the last I again I should be saying three weeks but I have a lot of things going on. So a weekend maybe a weekend and two just

polishing it up and trying to to lock in on this. Um was to just draw out the flow what's going on from the the application that I had all the way to it being done. And then there was a couple things that while I was drawing it out for me because before I didn't have all these sticky notes. So, if you're reading ahead, don't be a secret.

Um, I started drawing it out. It was all just for me, for personal learning. And then I was like, I and this every time I kept writing, I was like, I wish that this was here when I was doing this. Now that I understand and I'm reproducing, I really wish that this way was taught this way. This way was taught this other way. So in this when

I'm writing these I said I I think that that could be the second half of my talk is that I'm essentially just going to say this is this is how I wish I was able to be taught when I was going when I was learning these things and in general it kind of turned to how just teaching in tech in general could be transformed and then I

loved this by the way this was so fun that's why when people ask what do you want to do what do you want to learn I I say I don't know not because I have lack of direction but because I'm in this stage of my of my life or learning anything is fun. I I I don't believe there's any there's a such thing as not not not

what's the word I'm looking for. It's just always a need to learn something new. It'll come to me later. Um so I had a lot of fun with this and I would just see myself doing this inside or outside of the office at any time. But then there was a couple things that I I took away while I was doing this that I felt like is important

to re reiterate. So the first thing um was to to do the why before the how. And then just a couple couple bullets will go here. And why this is important. So it sets a foundation of care. And I'm going to go go again through on this on the part of my drawing that has this laid out. It establishes a connection with the product or the process

that you're trying to teach. And it starts small and starting small before overexlaining. And then it also sets that base to appreciate the deeper explanation. Right? So let's let's go back. We're going to zoom in now. So if it's, you know, staticky, you know, let her let her cook. She's okay. She's okay. Okay. So this very small piece of things is the first thing that I found

out was missing when I was learning is it went straight into this is why this is it was the wrong why actually is how I'll phrase it. It was why this product is the fastest why this product is the best how it does this in the code in the terminal but yet to zoom back out and why you even need a product like this at all. So

if somebody had came up to me was just like hey your application is done do you want it to be scalable? I'd be like, well, yeah. Um, do you want it to be um be able to be independently deployed or independently updated? And I was like, yeah, ease your deployment and all these things. So, listing out the benefits just to start because then what that does is

it allows when you when you're going in for more information, if you're trying to spark more information, you already have a base of like, I kind of care. Maybe not a whole lot, but I I at least want to find out more on how I could get that. And then when you have it there, then this way you can bring up a very simplified version of things.

And again, it'll expand to get something to technical. But when you have this simplified here and then basically this little box is all the drawing that I had and just here. So the app, break it down, um, package the app, have those parts, manage those parts, set it up so they all talk to each other, and then of course, you know, having it so that the outside

world can input in. Um, if you see any typos in here, just kind of refrain your eyes. Like this was this was a hard sketch. If this was like, you know, all tied together and those would be fixed. But so having that there. So now we have the why and then a little bit of like almost like a sneak peek of what will happen because I do

believe in seeing the whole picture first before diving in is very important. So that was kind of the first one on that end. And then the second thing was bringing up was like making the invisible visible. So what I was talking about was so hard for me for this one was that there there was nothing to see. There's nothing to be reactive and and that's fine for

what it is that the actual thing in the tool might be invisible itself. It's a process. A lot of things happen in the back end and things like that and I'm not trying to knock on what it is but that means that you can add it in in the way that you teach it. And that's the thing is humans drive off of visuals period. Um and getting

back to the learning of basics is something that I feel very very strong in. It's it improves engagement and it enhances comprehension, improves memory retention and that's all these things that when you watch a fun video, it's an engaging video. It's um it's visual and it's showing these things and it's something it's a cognitive thing if you connecting those dots ahead of time. And just because it

is visual doesn't mean it's watered down, which is something that I feel like was prevalent in what I was watching is that in order to make it seem like you know what you're talking about, it has to be as bare bones as possible, terminal, or you in a whiteboard and writing it down, which may work for some people. I'm not knocking it, but I think there's might

be some hesitation of being able to learn from something that's cartoonish or um just not from a textbook, right? So when we have visuals here going back zoom back out and then this is where I decided to put put my visuals. This is my take. Okay. So because Kubernetes for me was was the hardest one that I had to had to digest. So I needed to make

a way that it made sense for me and that's you know we have these containers here. This is the way I'm kind of storing it out. But I did a roll call. So I have characters for all of these um for all the parts of or the the major parts of Kubernetes that's there, right? So we have our let's see what is this here. Yeah, you have

like API server. I call this like the worst game of telephone ever. Um scheduler, you know, we have that as our matchmaker matching pods and for what they need to do. Um everything. Um, and then just kind of having all of these different visuals made it make more sense to me because also when you're able to simplify something yourself, that likely means that you know it better

as well. So it's not only good to reproduce, but it's good for yourself. So I would say get a get a pen and paper out. It might be helpful. And then down here, especially this like type of secular process for the control manager, just having that there reality versus want. Is that the case? Well, then fix it. So the secular motion. So now when you have this

in like in a shape even just as simple as a shape of a circle you can kind of see how this is not only did you learn that the control manager is always on it's repetitive it's always checking and then it has this continuous update. So instead of just listing those out put a circle and it's and nothing's nothing's lost in in translation but it's just it's

something there. So now when I think about this, I I I literally have a mental circle in my brain, right? Um and then same for over here, we got like some traffic lights, warning signals, you know, just maybe also didn't want to draw a little bit, but in the end of the day, having something here that's easy to refer back to. I think we actually have down

here another little process of our like stable these just dying off pods and then all get going here. Um so that was kind of that's that was that take there. There's two more because when you have this here, this is visual. It is visual. However, breaking those down individually and having their own type of thing. So there's there's a separate thing between making something visual to to

explain it and then just simply showing the visual of how it works. I know that's this a little weird, but it's it's there. So it's like this is fine. And I'm sure for most if they had a background, this is more than enough. However, in my opinion, now that I have my little characters in my brain and then you put them here into this holistic visual because

you don't stop there. This is all in in addition to what's already out there, then you can have this situation where it's holistic and you can show the overall flow and not lose them. So, you don't get lost in things. So, this is is good, but you know, I say have the same creativity when it comes down to the parts themselves individually. Give them their time to

shine in a way, right? Okay. So then the next thing was teaching in workflows and not solely on infrastructure blocks. Right? So back here technically what I did was visualize them as they're isolated blocks which is a way to do them and you need to do this eventually of course to explain how things work but I feel like it stops there and then it's like okay and

it leaves you to put the pieces together yourself. But I found that most of the time I'm the kind of person in class in high school that was like I kept asking why why why until eventually somebody had to explain to me how the sun was created because I I couldn't it's hard to stay in isolation without trying to connect to something else. Again it's another human

thing. You you reach for something to connect it to a broader thing an analogy. So if you just work in isolation it's really hard to connect those dots. So by having that there and then making a very clear transition to how are these actually work together is very helpful. And then you can actually see we we get back into this, you know, I I still know my

code. I still know my language. This is here. There's no visuals. And it's, you know, we just kind of have it going here to the request. The I can't read that from sideways, but I know what I wrote. Um, so data is stored. And then we kind of just keep using and following it all down to what a request might look like in between. I think this

is like, you know, to apply a docu a file that we wrote. Um, and then kind of going down. So we're able to use these keywords. They're underline because we've set not only care, we have context. It was engaging to get there. We know them in isolation. So then when you bring them up in this fashion, which will how it would be moving forward in the rest

of your life in industry, you understand it because it's like, okay, they said control manager, I have a I have a little circle going on in my brain, but I'm able to still read this and I'm right on the same level. So how you get there is I don't know why that's such a a stigmatized thing, but what but we can get here. And then going in

workflow. So then now you're like you're able to put pieces together. It's like, okay, I see my individual. I get how it works. Could you probably have put them together yourself? Yes. But seeing it all work there is going to be key. Um and then a similar workflow over here just about a general like um outside request coming in and what that would look like for there

and that that was always I was asking how okay great but why what does it do and then I kept trying to like does this do this this I'm making the connection myself and I'm just lost. So there's rabbit holes that we can avoid if we just lay them out if we have them of course but just lay them out. And then the last thing which I

think might be the toughest for people is um oh this is just kind of refresh reinforces the importance of the opponent allows the user to see the whole picture and creates more opportunities for them to really get it. And this one for honesty I don't know if y'all saw in this um around here it's a little bit zoomed out or zoomed in. Come on guys. There's these

overall little lights writings all over the place. You don't need to zoom into them because they're very much from the brain and all they are is commentary. It's commentary throughout. When when I was actively learning for myself to draw these out, I was writing what I thought it was, what it ended up being, and here's what it is. And thi this is might be the most important

one because nobody wants to learn from a cocky person. And what I what I'm noticing when these questions are asked or when things are being presented, it's it's trying to impress like, "Hey guys, I know this so well and here's exactly how was done and it's it was the easiest thing for me, so it could be the easiest thing for you." And it almost disconnects trust with

the person who's teaching because you actually have to you have to invoke that. I trust you not only because you invoke some care, but I trust you that you also struggled and failed. That that's who you really want to talk to, somebody that's real. and then answering those questions, answering more questions without them having to be asked because it unveils hidden knowledge. So if I'm in there,

let's say I go through the whole workflow and everything, it's great. It's awesome. And you know, I have some takeaways, but when you learn anything, if there's a if not if there's not a connection made explicitly, you will make it for yourself, as you should. That's how that's how we work. That's how we keep things stagnant. But if you actually go through your process, you might have

unveiled a connection that they made that might be wrong. And then it's like, oh, but you said it out loud. You didn't you didn't um what's the word I'm looking for? You didn't make them feel bad for asking the question because they didn't even have to ask it. You just being vulnerable and being like, you know what? To be honest, when I started doing this, I thought,

you know, I don't know, we could do something else. Like, why have to think about the architecture in my thing? If you just say it breaks it down, okay, but it's like, you know what? I don't really know why because I'm like, there's so many things. I break it down into files. Do I break it down into just every single line of code? But then it's like,

oh no, but then I realize it's more about responsibilities than what it does and breaking those down. Okay, because see, I would have been worried if I went on and like period, I know how to break things down and package them, but then I might be packaging every single file, which makes no sense. But because I was able to say this honestly, now we are able to

fill in some gaps that aren't Um, and then this is more of a general overview of what I So then I did I did this, right? And I kind of sat back and just now I'm just observing. So I have my thoughts and then I just sat I sat in my classes. I came here actually still evolving yesterday or any type of tech conference or meetup that

I've been to. I just was remembering and creating my own analysis what I what I found. Um and it's not beginner friendly and it's not because of the code is hard or the things that we're learning is just extremely difficult. It's just there's asking questions seems like a sign of weakness and that's truly how it feels. And I'm even noticing in conversations that I'll overhear is that

there'll be two people they're meeting for the first time and they'll just be like, you know, hey, how's everything going? And they might ask a question about what they're doing. They might just, you know, just to spark a conversation about maybe the work they're doing. And immediately is giving defensive. They have to just almost like they're always advertising what they do, but it's just it's just a

conversation, but it feels like there always has to be a guard up so that nobody believes that you are unintelligent. And then that and then then of course the conversation ends up being bleak because no one wants to have a defensive conversation when it's just supposed to be casual so that it doesn't happen. So and then when people do teach it takes as an opportunity to prove

what they know and not to help the other person understand. So like I ask somebody and they sit there and I'm like hey I don't I don't know this thing. Can you explain it for me? And they're like oh absolutely. And they just try to use all the biggest words that they remember from the textbook. They try to do all the complex things that they had to

remember and then they put it in front of your face to be like, I know what I'm talking about. Great. Um I still don't. Um so it's just this there's this gap here. And then I've written this out where first off, innovation cannot exist where curiosity isn't welcome. And right now I don't feel curiosity is welcome. And for a couple reasons, it's it's fast-paced uh where everything

is being learned to produce. And and that's it's almost nobody's fault for that one because that's what I that's what I had to do. I had to produce something in three weeks. So yeah, I'm not asking as many questions because I don't have time to ask them. We've because we have things that answer them for us. When AI comes, we can think that we have those answers

quicker. Then that means we set deadlines quicker and then no one no one asks questions because I guess you don't need to, right? So we only start providing surface level information. Then the parts of technology that actually spark passion and creativity get cut out because there's no need for them. We can just do it without it. Then asking questions becomes more more stigmatized than it already is.

So then I'm just going to stop asking them all together because really do I need them? I produced the thing. Maybe I got the job or whatever the the trick was that I needed to do and then they settle for more shallow understanding. And the the drive to change, create and innovate will disappear. Because if think about anything that's been changed or had a great milestone or

has had any type of big innovation, it's because somebody challenged what was already there and was passionate enough and curious enough to find a better solution and a better way to do that. But that's not going to work. If I go to chat, chat gives me an answer. I copy and paste that into my code and I go because not only do I have I don't have

a personal stake in what I just wrote. So I don't care enough to even digest it to find out if something's better. And then even if it is, even if there is somewhere that I could make it better, I actually don't really know how because I'm only staying on the surface level of things. So if no passion and no understanding, then we'll just we're just going to

keep leveling out and then yeah, then maybe AI will take over because we're going to we're we're stopping our intelligence to match theirs. That's that's when we get to be a problem because we are we know every single time we see prompts from AI and outputs and things like that, we know that we could do that better. But now, do you want to be better? That's a

different question you have to ask yourself. Are you going to let AI take over your position? Because it will if you decide to match it. But I think if this is a community thing too, but then also it's a sales thing like are you going to just stop at the five minute video or are you going to realize that that wasn't all that you needed to know

and you want to dive in and maybe watch that 15, maybe watch that hourong video and see what you do. So I ask that for yourself. When people ask you questions, how do you approach them? Are you following any of these things? It could be subconsciously. This is no hate to anybody. It's it's it's ingrained. And then how how confident are you asking questions if you don't

know know something right now? Because it's changing. Everything's changing. I'm sure you were in a different talk before mine or will be in one and you'll have something new to learn. So there's always going to be questions made and we're all going to start at zero again at some point for something. So just remembering that for anybody they're going to be at zero is going to be

very important. So I just say keep that in the back of your brain. And so some in today ever in life just be be somebody that doesn't let this continue. Stop puts a break in the process. Says you know what actually I'm I'm going to show you. Here's why you should care. Share your passion. You will have a passion for it. Share it. And I promise that'll

just make everything a whole lot better. That's all I have for you. We have a few minutes for talks or for questions. If anyone has questions, raise your hand. We do have a live stream. So want on the mic. >> Hi. Well, first thanks for the talk. That was awesome. Especially the last couple of slides. That's something that I think we should keep to our heart, especially

me because I try to instruct people. So thanks a lot. Um, when you said that when you were looking for information, there were a lot of like five minutes videos, uh, is it because that you didn't find a lot of longer content or the longer content was not like good enough somehow or it was not surfaced by the algorithms and whatnot or something else. I want to

say there was there was a combination of it because you know when you don't know what's going on you just kind of do what is and then you kind of put the rest and then so it was what was what first came up. So there's that. So there just like the relevance and priority. So maybe those longer videos were in like the second or the third page

which again you already have to have some some care to go that far beyond the third video. Um but then even when I did get there I was noticing kind of those other things about even though they had my attention for an hour it was still kind of trying to impress me for that long. So he just it was just the terminal and I was like I

get it and and it worked. I got I got to what I needed to do but it was it was interesting. So I say there's a combination of both and I think you have to like almost bridge them together to make what I think is suitable. Mind you this is an opinion but so that was kind of the reason for that. >> Thanks. >> Yes. >> I

know I don't I know I don't normally get to ask questions but you got to allow me today. Um is it still is it clear why? Right? Because for some of us in this room, we've been doing this for a long time and a lot of this feels like it's necessary because it's part of the job. But when I think about when I got started, it was

never clear why people were doing what they were doing. After going through all of this, is it clear why you would want to do any of this >> when you actually other question why why it's useful like the product itself of or any of these products are useful or just why would anybody even think to recreate something like this or rebrand? >> Yeah. Like if you see

someone working you're like why is this person digging this ditch? But if you see that it's raining Yeah. and all the water is flowing through that ditch, you're like, I get why you would dig a ditch. >> But if I saw someone digging and you're >> why digging here? I am curious. >> Why are you doing this? And so a lot of the things that we do,

the tools of the trade, it's not always clear why. I even talk to people that 10 years in, they say, I'm like, why are you doing that? Like, you know what? >> I don't know. I joined this company. They use VMware. I got VMware certified. And for the last 10 years, I've been creating VMs, but I don't know why. >> I needed a job, I guess. >>

Yeah. Uh I think I I understand why what I learned is important, but I I think that is different from the other why that you're asking. I I still look to even in the talks that I'm sitting in, I still look to ask. It's like I I I wonder what their why is actually because I can tell there's some people that might actually fall into that category

of this is it's hard out here and and is. Um but I don't know. I think and I think there might gets to a point where does the why even matter? So I was like I don't know if there's many people that are even asking why anymore. Um but I think for for this what I just did I feel like I understand the why. But if the

more I get into it, I'll have to reinvision my own why for it, though. And that's still in progress. >> Okay. So, I have first say I love the visual because as a learner when I was first seeing things, I'm like, I have no idea what any of that stuff does or what it matters. or then when I was learning it's like learn Python the hard way

and I'm doing things with apples and oranges and I'm like I still don't understand. So I really love that visual and how you thought about that. I'll be using that too. >> Um I think one of the the hard parts obviously is like hearing that you know the resources and different things for students aren't out there. Um I work at AWS. Uh there's a number of offerings

and different things that we have for like you know student builders and different things. But do you have any good suggestions for like how we could work better to, you know, to make sure that those are visible? Uh, right now we're in a a state of there's so much content and different places to find information. So, trying to make sure that it's accessible uh and easy to

find. >> Um, and I may I know that's probably a loaded question, but if there's Yeah, >> I just had a a followup. Is it what what is the content that is that you're trying to get out there? >> Not necessarily content, it's access to resource. It's actually resources and using services without having to do, you know, and >> um or having those charges, right? Like you

know, like I would look at that as a student and be like $74, you know, 100 bucks, no thank you. Yeah. I've also heard horror stories of people building things, not turning it off, and >> $30,000 later. Yes, that's a big surprise. And so there are resources out there, but I just don't know that that they're making them their ways >> uh to students. And so if

there's any suggestions about how do we do that, um I would love to hear them. >> I think because it's definitely two-sided, but for you all when it comes because I'm thinking as a student, if I have if I'm trying to reach out to something, it might that it's it's there, but then it's kind of ties that back into okay, I I have is there is it

pitched like this is why you should even take the time to use it? Um because I know like for example if it's a resource to explore to you know hey here's this thing here's these tokens credits whatever they call them um to play around with this thing but it's like how do you convince them that they have time to play around? I'm like uh it's probably really

cool and it's probably bookmark somewhere. Um but the environment they're in is like unless it's going to write it or do it for you is not to explore. So I think it's one kind of having to it sounds really weird but convince them that this is even something they should take advantage of because it being there is one thing but convincing me that it's important will be

the other but then also has to be within the climate that they're in which you cannot control or maybe you can of of allowing that time to just explore that does this for me at least now that does not exist it literally doesn't we're in class are literally telling us because you will not have time this part of the class will be AI it's just like Okay.

Um, so it's a it's kind of a give or take thing, but the way you pitch it and then come maybe even like showing the benefits of using this and then how it could be what it could produce for you. So then they can maybe make that dot connection where it's like, oh, I have to do this eventually. So maybe, okay, let me start. So maybe it

feels like they're getting a head start. So making somebody feel like they're getting ahead with your thing rather than losing time is like that's kind of how that would have to go. Yeah. Yes. Sorry. >> Uh I I believe you touched on a very important point in terms of tech isn't always beginner friendly. So my question for you, Miss High Tower, is um how did you navigate

that in your experience? How were you able to punch through those barriers? uh because I've believed in that for a long time that we have that a big disconnect between the people of those of us who've been in tech a long time and I always say it's very valuable to have newcomers because they can review our documentation because we might remember all the workarounds to something but

it ain't the docs. >> So sorry a little long but how did you break through and >> I you know what actually that kind of goes into how I picked my major. Um I was set that I was going to be a computer science major um and and it was fine. I was, but I I think I foresaw that for myself is that I was going to

get lost in it. And then I searched up and I found other alternatives and then I saw this thing called informatics here and I was like, huh. And I looked into it and and I thought that it was it was all about making that connection or at least they they claimed to or gave us the freedom to make that connection. So everything was open-ended and project based

so that they it gave us that that space to to connect the dots. So then when I knew I had at least a backbone that it was I was going to be doing it towards a you know it could just be a mini quarterlong project but I cared about it. So it connected to something that I cared about and I made sure I had to I had

to create that care. So then when I was in these I still had to take some CS classes separately even though they weren't as connected. I took it as like oh I'm getting ahead in something that I already care about because that that was laid for me. So I I had to make it for myself. I had to really really convince myself like is this something I

want to do? And I once I created that care then I knew I was down to learn whatever it was but I that's not the same for a lot of people that I know. Some people are just like I'm in it for the money and and then which they might have some uh a rude awakening but anyways they're here for the money. Um and you can tell

they go into classes very differently than I do. It's hard. I don't want to do it. I don't understand because they don't want to understand. I don't they don't want to go and deep dive like yeah it takes me three hours to study for a test one test and there's probably like five questions and it's just different because it's I'm I'm wanting to go down the rabbit

hole. So that's kind of where that was. Uh first I wanted to say that I really appreciated the point about uh the cron jobs doing something unnecessary to >> It's almost like you have to do something the hard way sometimes. >> In order for to see how simple it is, >> reference to >> Kubernetes the hard way. >> Uh but my question is about AI. Is that

making things easier? Because sometimes you could just prompt something and it'll do it for you. But I see that you learned the fundamentals and you map them out and you you it clearly you understand them. So did it make it easier for you to understand him or did it kind of make you go somewhere else first before asking what are the fundamentals? I I I because so

I've been I've been playing around with AI for a while and my way of um how I use it to support myself. So I knew that what it would give me was just going to be what it thought and that's and I did that on purpose that it was probably not concrete and that was fine. Um so but then I also just said I knew for myself

I was like okay I I I went myself I made myself go on a different path. So it of course it gave me links and this is what you can do and you'll get to these steps in two weeks like you know it gave me my whole time frame but I ignored that just the front part and then I I purposely cut it off did it the

hard way for myself of just finding other people to recreate it and then what I would do though is that after every time like let's say I found a 15-minute video it would probably take me about 45 minutes to get through it because I would stop and then I would I would text with chat and be like okay I'm watching this I'm understanding this make my own

connection and then have it verified for me and then I was like okay okay because I can only feel really confident getting more information until I thought the one I had was solid. So that's what that was a lot of. So even in my drawing I'm like I have a base and I could just ask question after question after question and go down so many rabbit holes.

Um so that's that's what's helpful. But of course I probably could have typed that first prompt in and it would have spit it out for me. So I guess it's asking is it helpful to produce the thing or helpful for me to learn? So yes. >> Okay. So, first first off, your answering questions is even better than your great talk. So, well done. You're you have so

many talks ahead of you if you keep this up. Um, as a platform engineer, I really appreciate Kelsey's question about why, but I'd really like to know what would you rather do? And I could almost scope this for what would you rather do for the learning? What implementation? Like what would you rather? But I don't want to be that opinionated. What would you rather do? >> I

know. I don't want to scope you down. I What would I rather do? And >> you achieved an outcome and that was phenomenal. And the hero's journey to >> was not trivial. >> If you had it to do over again, the world was your oyster and computers worked the way that you wished they did. What would you rather do instead to get that same outcome? >> I

think what I wish it was framed as was there's a problem like like almost forcing me to believe that there is an issue. There is an issue but make it very apparent being like holy crap you know this is going to break on everybody else's computer just because it works on local like basically making like a case study was how it would have been dope is having

that there and then it's your job to utilize resources and find tools to fix the problem. So, not only is it like I just like to win, so it could have been about anything. So, it's like, okay, now you have that my spark, my engagement, and then I can already see I need to care because it's broken or whatever. It's not as where it could be. And

then if I had that, then I'll be like, okay, then I would go and be like, okay, I need this is one thing that's broken. It's not scalable. Okay, now we look to see how can I make my application scalable. Oh, here are these things I got, you know, making containers and using Kubernetes. Okay, I'm going to learn how to do that so that I can I

can fulfill the job and fill the things. So it's almost like rebranding the outcome is not the goal. It's how making it better seems like the bigger goal to make things and then you're like forced to go down rabbit holes because you you're guided in a way that that puts you there. So that's what I that would have been dope if there was like a a case

study for this on how to get to cloud native without giving it to your own path. That would have been that would have been dope. Yes. Um sorry. Um, for the visuals, I enjoy the visuals very much. I really love them. Um, I would love to take a longer peek at the whole thing at some >> I plan on putting it there. Just like I said, it

will be riddled with typos. So, just just just know, but I plan on kind of making it into a thing. >> But is very amazing. And um, I'm a beginner when it comes to Kubernetes, Docker, and everything. I literally just installed Docker yesterday in one of the workshops. So, I wanted to know if you have an analogy that you came up with yourself or that you learned.

I know that I saw the little matchmaker and the traffic light man at the way bottom and everything, but would you say that you have an analogy that brings it all together in a whole? Like I heard of a warehouse analogy once, but would you say that would be the best or is there anything better that you can >> kind of come up with? >> I know.

Um, when when when I was trying to do this because I was trying to do the exact same thing. Um, a lot of this was like for for meals, meal prepping was for me, which was interesting. So, cuz I like, you know, I had it up top as like this. The application is like, you know, big dinner is what I want to recreate. And then I have

it like broken down. It's like, okay, but I'm going to meal prep my, you know, my steak and my potatoes and my things separately because they all have different body responsibilities. And then for this one was packaging them up like a lunch box. So putting them in their actual containers for meal prep to put them in my fridge. And then I used say Kubernetes was like, okay,

well I maybe I made some of my meal prep a little bit too long, too late, some of it spoiled. And if I had a magical refrigerator, it would give me a new a new thing to meal prep or at least let me know that it was time something was spoiling. It would let me know that something is running low. Like you actually only have one container

left. How many how many meal preps do you need to sustain yourself? Probably four. Well, you only have two. Um, and then kind of things like that. And then, you know, the rest of it being just like having maybe getting a more information on a new recipe to make, having you share one with your friends and things like that. So, I like I said, it's always going

to be about food with me, I fear. Um, but just that one was kind of interesting for me because it's like it felt like something I that actually makes sense to package up with some food and then actually things can go wrong with them that made sense. Spoiling and not having enough. Um, so that was kind of the holistic one that I could think of. >> I

get two questions. Uh, that was a beautiful analogy by the way. One thing I realized in that previous question that you had in all of my teachings of this stuff, I never asked that question. How should it work? And I think some of my best work has always been challenging the tool and saying, you know, she gave you this free reign. And I think a lot of

us have Stockholm syndrome. We don't even ask ourselves Is it okay for this to be way simpler than it is? Do we really need 90 of these pieces just to get an app >> to show up on your phone? And I think sometimes we feel like it's a right of passage to say, "Yes, I did it the hard way and you shall too." I even wrote a

whole doc on how to do it the hard way. >> But I should have wrote a doc that said, "How should it be?" >> Yes. And I think that's where the creativity comes because I don't think anyone would recreate this bicycle the way we built it >> if they had the opportunity to freely think about it before we inundate them with our past. >> And then yeah,

I wish there was a job that was just that because that's that's exactly what I thought. Even the whole time when I was learning, I was like this is this is not how it would be. But because I didn't understand it yet, I didn't quite know if that was possible. But I sure did have a lot of notes apps of being like how it should be done.

So >> I don't know if you mentioned this before. What is the software you use for the notes? Oh, this would be free form. >> Free form. >> Um, yes, shout out to Apple. They had that free form there. >> I think we got one more question. >> Uh, that actually partially answered my >> Um, I do not love using graphics software. I'm a musician, not an

artist. >> There you go. Um, so I was actually wondering how many drafts of this um, did you have to go through in order to have this graphic that is totally gorgeous and I would have also paid money to use because I do not have the skill of making this. I would love to obtain it though. So that's why I wanted you um, zero. This is this

is the draft. Um, cuz like I said, this is if I had my notebook my digital notebook out, you would see this has been my style for years. So, I just I just go only type of draft is like if I zoom way out, you might see some things that were like tossed. Like this was one way I thought I was going to explain um explain things,

but then I was like, "No, I feel like I'm just recopying what I've already seen, and that's not right." So, it was about the what, but yeah, when it if I have the idea already, it went pretty quick. Uh so, no drafts, no erasing and restarting. I felt like I also didn't have time to be honest with you. Um, but I'm pretty comfortable with where it's at.

And I think repetition I I'm a critiqueer of myself to the day I die. So I know I I have things I'd love to go in. I'm sure there's better ways to explain the the way that I just explained it. But >> awesome. A wonderful talk. Thank you >> Thank you. Check. Check. Check. Check. Check. Check. Test test test. >> Test test test. All right, I see

a few people rolling in, so we'll get started in just a couple of minutes. We should be able to make the time Okay, I guess uh we'll get started on any given day. I guess um lunch will will all always uh be more important than platform strategy or platforming strategy. We get that. Um, I guess people are still rolling in, but with the people that I have

here, I guess let's start with raise your hands if um if you or your organization has a developer portal or a developer platform Okay, cool. And how about developers and and and how do they feel about that that developer platform like um are they using it voluntarily? Um is it there and remains to be utilized like how how would you would you say it is being utilized

or is it somewhat being utilized? And for the others probably platform is not a reality and that's perfectly fine because that is actually a better answer anyways right uh because it means that for those of us who haven't started that journey the chance to do it right um is is what I'm essentially going to cover on in this um in this talk my name is Fessel I

principal technical consultant at ahead um I lead our platform engineering and cloud practice um and I help customers with platform engineering strategy, architecture and implementation. So today's agenda um platform engineering is is kind of mainstream already, right? But um but still there there quite a few components of platform engineering that that battle with hype. So we're going to start the agenda with kind of understanding the hype.

Why platform engineering? why now? And we're going to kind of continue the journey. Essentially, we're going to go from what actually it is to how do you structure your team? How do you get return on investment? How do you prove that it's working? And how do you ensure that for those that you're building this platform um that they'll show up and they'll use it. One of the

thing first things uh which is considered an anti-attern that I hear often from customers is all about portals backstage harness there there are a lot of other options out there and and a lot of folks think that you know just standing up a portal is is really what the platform is all and and and there's there there's a lot there to unpack but but going back towards

platform engineering as a discipline And um the the partner hype is is pretty legit, right? It has its own dedicated hype cycle. Um and when most teams hear platform engineering um you know going back to the first point, they're immediately thinking about a portal. And and our customers, my customers are are no different. Um, and when you have that mindset, the adoption suffers because you're not really

listening to and discussing why in the first pl place you're going to build a platform engineering discipline and and provide platform as a product. Um since I am part of platform engineering community as well as CNCF um I get to see some really compelling great talks um organizations that are they're doing work with platform engineering standing up platform discipline um and involving the existing platform um architecture.

So um obviously before 2021 there wasn't really much of a talk about platform con or platform engineering um became relevant. Today platform engineering has over 21,000 members. Um as I said before Gartner hype cycle has has platform engineering in its own space. Um and not to mention that platform engineering continues to get added on to other hype cycles including container technologies s etc etc. The forcing function

for for that is cloud entropy right multiple cloud service providers. um a lot of different technologies that are being used CNCF landscape you know that's if you look at the landscape today versus 10 years ago obviously it's uh it's quite overwhelming right um another another reason why platform engineering became a little bit more of a thing was there was this concept about DevOps you know you build

and and and and you run it right um or if you build it from a engineering perspective they'll come Right. So that reality did not happen. The DevOps reality with you build and you run it, right? And and you kind of run into scaling issues there. So a lot of these things kind of led to to where platform engineering is today and why it's so sought after.

There's a gardener hype cycle and it's kind of in the peak of inflation. But when we get started, especially when I when I talk to customers um about platform engineering, there is always this um um a little bit of a challenge making making it clear what platform engineering is and what it is not, right? And often time it starts with an IDP, a platform versus portal, right?

The conversation always starts with a portal rather than a platform. Um and when you're talking to customers, um giving them the understanding that exposing those APIs, documentation, all those things are more relevant, right? Um it's it's almost like it's almost like building a house and and you start with a door and door being the portal, right? And if you don't have the foundation for the platform, then

it all kind of collaps. So, so often time, you know, it's all about like platform versus portal. Um, and and the discipline of platform engineering essentially is about designing and building tool chains and workflows that enable self-service, right? That's the that's the northstar. That's the goal. Um, to ensure that you get out of the ways of developers, right? Um, and and often time it's a businessled conversation,

at least for me, where uh when I get pulled in is like we can't go to market faster. we have a lot of challenges. It takes us three weeks to build an environment. Um there's a lot of friction so on and so forth, right? Um from a platform perspective, you know, it's it's essentially a self-service layer of APIs, tools, services, um that the platform provides. And it's

it's interesting that often time when I walk in, it's an engineeringled conversation and not a developer. and they're like, "Oh, we need a platform." Like, "Okay, great. What have you heard from your developer community?" Well, we haven't really quite started that conversation, right? And I think you're kind of already behind the curve if you start your platform engineering discipline and you don't have ha, you know, have

those conversations with your dev peers. Um so anyways so these are some of the things that I hear and just kind of looking at the IDP or the platform um and the various different planes the uh the landscape continues to mature. This is from platform engineering community. Um all the various you know planes including observability developer control plane. Um and so so as you can see just

like a CNCF landscape this thing is is evolving as well and it's it's quite a quite an undertaking right so in order for you to get to this level of maturity um you need to understand understand and and embrace the complexity that a platform will bring to you right um these are just the five planes and and and the point is essentially is to is to to

not build all of them together, right, at one time, but to start where the biggest challenge is for your organization, right? And then and building towards um you know, your early decision, you know, instead of buying something out of the box and just implementing it. when I talk to customers um especially around platforming initiatives this is how I kind of start again um borrowed from the platform

engineering community um if you look at the top end the business all they care about is the value right why should we invest in a uh is it going to reduce the cost eventually the answer obviously just so you know is no at least start out of the gate right there's investment to be made um training to be done um getting people upskilled. Um so that investment

of technology and and people is is is is kind of you know quite an undertaking right. Um so so those are some of the the challenges from a from a business management perspective. Um, and essentially that's all they care about business impact, right? So they don't care about a portal. They don't care about backstage. They don't care about whether you're going to use Terraform or Palumi or

or Crossplane for managing control planes, right? And um from from the value drivers perspective, right, the value that our platform provides is through the personas of infrastructure engineers, platform engineers, DevX engineers. Um essentially these are the the people who are going to help you standardize and automate a lot of those functionalities that are today just kind of click ops or just taking forever. Maybe you have infrastructures

code but it's there's no CI/CD aspect to it. Um and then obviously the pains to the bottom right where you know these are things that you if you work in IT you kind of live through these right on daily basis which is slow missing services ticket ops click ops cognitive load um so on and so forth. So again, you know, the pains are pretty self-evident, right? Slow

self-service, uh, manual review processes, cognitive load, insecure configs, and the value drivers from a business um, perspective are how do you make sure that we get the value through standardization, right? Ability to move faster, get products out into the market. And as I said before, you know, the the what the biggest anti-attern that we see is people starting with portal. Somebody has a portal installed running. They're

like, "This is it solution right here. Let's start here." Um, and as I said before, you know, you're building a door without building um, you know, the foundation layers, the CI/CD, uh, the automation pieces, exposing those APIs, documentation. Um and if you don't do that portal if you put it out there will just expose all those challenges right one way one way to to kind of start

a conversation at an organization uh for me is is to leverage what we call a platform maturity assessment and essentially it's across people processes obviously um but understanding the investment the adoption the interfaces the operations and and how do you measure it? Right? All of these and and what we find is organizations that lack that platform um capability and maturity, they're they're kind of around the provisional

and somewhat operational stage. Um and so that's um you know this is you know using this as a diagnostic tool and not a report card actually is probably the best start if you and your organization are kind of embarking on this journey. You want to create that safe space, understand the inadequacies and challenges, lack of investment, right? Maybe leadership doesn't buy in, maybe quite does not understand.

Um, so these are some of the challenges that you could that you could um address through a platform maturity assessment framework like this one. And then the next thing that I get into it if if it's not the tool, if it's not portal, it's all about it's all about well that's cool. Let's do platform engineering. Um and and and what do I need to do to be

a platform engineer, right? And and and and and you obviously can grow some of those existing personas in your organization, right? So if you have a a DevOps engineer, um they could potentially fill the role of a DevOps engineer, right? because it's all about understanding the developer painoint and those frictions. Um but but going back to Conway's law, right? If you're going to design a system based

on existing structure of the organization and the patterns um and and the people and those behaviors, you're going to get the same system all over again. You're not going to solve. You've got the DevX uh persona. uh these are the ones that are bridging the gap between the devs and platform engineers. And um what I have seen at least is um the best personas if you already

have an existing DevOps um mindset or a few personas that that address that um they are the ones that can that can um address the gap and take the ownership of being a devi devox uh platform engineer. Um these are the guys that are kind of you know taking ownership of the golden paths um feedback loops adoption and then as you start to mature um let's say

you go beyond a version or two um and you're evolving this is when you want to make sure that you have a dedicated um platform product manager right they're the ones that own the road map the life cycle um between this and the devx persona I think um these two u are really important and relevant because the DevX persona ensures that it becomes an organic movement. So

when a DevX uh engineer solves a problem and de the dev team sees the value, your platform kind of starts to take um to take on more and more workload because your platform does not need any more intros. People will talk about your platform. one dev team that saw a friction being removed um their their cognitive load being reduced will talk to another team and and that's

how platform take over right and and and become a thing. uh the product manager their job is to help understand the risks and challenges and and how to align with the business goals and show the value to the business to say look you know maybe four months ago when a new developer would join it would take three weeks four weeks to do their first pull request today

as a by you know byproduct of this effort we've reduced that time right yesterday it used to take two weeks to build an environment and today it takes you know maybe two days or less. So, so these two uh the bottom left personas seem to to really help the platform evolve um and grow. Common team um anti-atterns, poor undefined APIs, right? Developers lose confidence and so so

these are not edge cases, right? like these are all like every one of these patterns show up in all platform engineering engagements so and usually within the first six months right so trying to build a platform but you still don't you know have those documentations and undefined APIs uh team collaboration it starts out really you know in a very exciting manner and then you know you're enabling

you know team spins up short-term partnerships but they're not growing right or or or or they are just so elongated that you know like they're just over functioning with enabling team right so app teams never build that independence that is needed so that's on the other side um of the spectrum you know how do you make that business case right for me as a consultant when I'm

working with an organization it's all about showing the you know return on investment so I can talk um and by the way I used to work at Hashore So, I can talk technology, right? Um, and that's great, but that's not what many of my customers care, especially the stakeholders that are bringing me in. They don't care about technology. They don't care about what your upstream infrastructures code

looks like or whether you're going to use Flux or or Argo downstream. You know, for them, it's all about, you know, do a due diligence upfront. Tell me what those challenges are, the friction is. what is that impact to my business? What is that cost? And then show me the value. And that's important because for a peer at a technical level u who's leveraging a platform, they

need to show value, right? So, so if you start with some sort of a um from the from the previous slide as well, it'll probably come up in the next slide. Um the reality is that you want to show the value right and and if you can show your baseline and then you start to show improvement across productivity you know delivery you know improving right uh resiliency

tooling consolidation right you're saving cost you bring in a new tool into architecture reboard and say you want to use this but then you retire another tool right so you're minimizing that tech debt and and how do you improve the talent retention So these are some of the ways um that you can show value with and and by the way business does not understand dura matrix. So

you know I've I've seen I've walked in with engineers to make a case for the business and and the you know the engineers and developers are talking about Dora and space matrix. they're good, right? They're good, but but you need to show value, right, through numbers and and so this is perhaps one way to do that. Um certainly there are other approaches with ROI that where you

can get more elaborated uh with doing measurement, a quantitative measurement, but this is just one example. And as I said, you know, DOA metrics are important, but they're are lagging indicators, right? And so um Dora tells you what happened right um but a dev devx matrix for example will tell you why it happened and and that's important right because you need both to help address and run

that platform as a product. So if your platform is healthy right what happens support ticket and volume goes down right adoption goes up you don't have to talk about your platform anymore your dev peers are talking among each other right and these two t uh signals usually are the leading indicators that I've seen where adoption goes up ticket ops goes down people are talking about it and

then and while you're building the capability ilities of your product as well, right? Through the course of this and and another way to do that is developer net promoter score, right? So, what that really means is how likely is a developer um to recommend as I said your tool to other dev peers. So, how do you start how do you start on this on this journey, right?

and and oftentime you hear MVP approach u with pretty much every every um initiative that you start um in in platform world there's another way to look at it which is tennis you know viable platform if we looked at this um yeah this um this reference architecture right so you're not going to be able to cover everything on day one obviously you won't be able to right

so You got to pick what is that thinnest viable product in your platform, right? You can still have a vision of where you want to go eventually, but you want to start with, you know, maybe pick the integration and delivery plane, right? That's where maybe the most focus should be. Then maybe ensure that, you know, it's done with security, add observability, improve that, and then you continue

on and eventually you bring in a portal into the mix. So defining and then assessing right um so how do you know identifying what is the intent right what is the intent for this platform what are those dev pain points that you're trying to solve what is the current state we talked about the you know maturity assessment current state assessment understanding across those five maturity assessments right

dimensions and then mapping that team structure so you can have you can have a definition of what the platform is, but do you have the the talent? Do you have the team um that you can stand up to help support? Then obviously building it and and making sure that you are able to connect with your customers, right, through that DevX persona, right? That DevX platform engineer. So

you're constantly getting that feedback loop as you're evolving. So four to six weeks usually on the average you want to start building features. You want to go back and forth. You want to understand how your platform is working. You want to start collecting data, right, from an initial developer survey and then follow up and say, "What have we addressed so far, right?" And prioritize those, right? I

mean, we talk about golden paths, right? They're off paths, right? Off-road paths maybe if you want to call them. Not a good place, right? early on you want to start with those rinse and repeatable patterns where you know 60 70 80% of the use cases can be addressed through that golden path if you do that that shows momentum and then measuring and evangelizing right uh I showed

you one example um but you know as long as you take a good baseline metric right you should be able to show value and to track adoption right business wants to know how you are making your solution more mainstream How many more people are using it? Those who are using it, how do they value your system? What is the score that they're providing? Not just Dora Matrix,

right? And then scaling and sustaining, right? So once once you get on that trajectory where where the platform starts to do what it's supposed to do, it helps you know reduce that cognitive load. The developer productivity starts to grow. Um then what you're going to find is a sort of like a cloudnative platform engineering center of enablement where people are willing to talk to you, come in

during your office hours, share with you where they want to take the product next. Um and usually this is led by a a platform product and and the platform uh engineering lead and essentially treating platform as a product, right? because if you have a road map, you got backlog um you know and SLAs's it is essentially now a product. So key takeaways um you know strategy over

tooling for sure right have a conversation understand what it's all about because if you just go in with a tool that nobody understands nobody sees the value or it doesn't address the challenges and risks that that already are there you know you can't have a tool it's just going to fall flat right and um you know treat platform engineering as a discipline um it's it's it's it

is a combination of products obly obviously across those planes that we talked about but but you know start with that one problem that you can address your team structure is your architecture right I mean and and we talked about con law um this is your chance to address the shortcoming of your system the people the processes right with empathy of course to say this is not working

right we need this type of talent to help address the gap we need to change the way we do things Right. Um I was working with one team um one customer and they were like well we've got a repo we've got a front-end form developers can go and and clone and and have at it. So I connected with the developers next. I was like well what seems

to do a problem you have a repo you can stand up infrastructure. They're like yeah but what about configuration parameters? Like we don't know how to build an environment like you know the repo is there but what we need to work with in other way in other words create tickets to go back and work with the infrastructure team to uh update the repo before we can even

run it right so maybe in the right step in the right direction but quite not a platform right um treating platform like a product measuring what matters right door as a starting point but you do need to understand by continuously engaging with developers to understand that how you're improving on those Devax signals and then providing that ROI back to the business as well. Not just saying great

job developers feel seem to be happy and and don't try to do everything obviously at once that TVP or the thinnest viable platform always win one small piece you start building momentum you show the value your dev peers will be your biggest champion and that's pretty much it I I wanted to purposely leave um some time um for for for conversation, especially if you have your own

thoughts or any questions that I can answer. And by the way, I don't have a proctor here or anybody's support, so I'm happy to just take the questions without a >> Sure. I'm sorry. Can you repeat that again? It depends, right? Um I mean for me um I think if I get the question right is who's the audience for the platform, >> Yes. Yes. Yes. And that

that is your primary customer. And so enterprise installs software primarily they buy software and install Well, well, it depends, right? I mean, if if you are installing software that's running on top of, let's say, middleware, right? Or you're installing software that's still running on traditional or or cloud infrastructure, you still have a need to put together infrastructure, right? And so unless until you have a very devoriented

system in place and and and there are organizations out there that democratize the infrastructure to the dev teams and so then the dev teams are just building and they're smaller teams let's say 25 to 30 size dev teams then then platform does not make sense or platform discipline does not make sense. You said >> Yes. That's that's a good point. What I have seen is customers with

especially if if if your size if your dev size is around 50 or below, chances are you're probably not starting on a platform engineering journey. So um on the average customers that I work with have anywhere from 500 to 1500 developers inhouse right You described how platform engineering works but by experience and personal knowledge no one is using it completely in the company as far as I

have experience. I think you will share or negate this. Do you think that developers portal and all of those practices will be Yeah, I can repeat the question. So basically for me uh platform engineering in these days is known it was well described but we have a problem because no one is using it. So in example when you do a pull request in a company or a

release process no one is following the automation but everyone is putting gates like signatures check off people that are having a review peer review or similar while in platform engineering this should be all automated by workflows by in example in Kubernetes using Argo CD or checkpoints that are Um there are other cases for platform like in example the pipelines and the controls of software building of material

uh everything related to container image and similar. So a lot of principles that you described and finally developer portals they are very very useful but no one is using it as far as I see there are a lot of offers commercially why do you think this is so hard for people this is the question why >> so yeah question is why is why is platform measuring and

and the leveraging or utilization of um portals is is so difficult, right? Because it's not a traditional persona of a of an engineer, right? Um it does require a lot of stitching, right? Think about the golden path. It is a lot of stitching together, right? Having that understanding of the developer workflows, right? So, it is it requires a a community of of peers to work together to

to make it sticky, to make it work. And so if as I said before if it's led by you know infrastructure engineering it it's just not going to work because you know your portal is going to expose only the reality that you have underneath the system right if you don't have those strengthened CAD pipelines if you have not exposed the APIs and documentation you know your portal

is not going to be able to do the job that it was meant to do. So having that understanding working collaboratively with your teams with your other you know cross functional teams is the only way to make a platform um successful and then kind of building collaboratively. So a challenge that we've run into is you know we have you know a lot of this stuff in place

and you know there's templates the golden paths and so on you know teams take those you know they get you know all those templates and stuff end up in the repo they're using it everything's fine and then something changes right either you know we make a change to what that should look like or you know some external factors change that requires updates now we've got you know

a thousand instances of the old pattern pattern. How do you get everyone to go get the new pattern? That that's something we've struggled with and wanted to know if you had any thoughts around that kind of So I guess my my first question would be how involved are um the the platform engineering devx personas with the developers because if there is a disconnect because they should be

addressing the golden paths they should be the the ones building the golden paths or improving on those that are needed. So how sticky is that relationship between the developer experience engineers and the developers? >> So some some teams need you know more help than others. You know some teams are fairly up toate on the technology they can pretty much okay give me some docs give me some

templates and I can run with it. Others are coming from a you know onrem environment where they're just not used to any of that. They need a lot more help. So varies quite a bit depending on team. But I guess my point is you know you know path was established and it was fine at the time for and then something changed for whatever reason. How do you

get all the existing stuff to adopt that change? >> Shouldn't that be the the job of a developer experience persona? I mean they are the ones that are the gatekeepers. They're the ones that are working with the dev personas, right? In support of building those golden paths, right? Whether they are the existing ones or the new ones or updating the ones that are already in place, wouldn't

that be the job of a of that that onetoone um personas? That's the way I think about that. Like I mean, if what you're saying is golden paths are built, more are built, then the ones that are built are no longer applicable, right? or they're deviating from from what what's really needed. >> Trivial example external secrets operator updated their API version. >> Yeah, there's hundreds of repos.

There's, you know, thousands of instances of that one little change that needs to be made >> everywhere. >> You know, they're all owned by different teams, you know, certainly. So on the platform side, we're going to make making that change requires them to make an update on their side and everything's crashing down horribly if they don't. >> But we're not We don't own all their code. We

don't um Right. So >> what is what is the platform engineering um setup today? Like what what what makes up a platform engineering team? Is this strictly around the infrastructure and that's it or like tell me more about your current setup. So my area of responsibility is the Kubernetes clusters that runs on but there are you know there's a broader >> so you create a cluster you

expose the name spaces and then you go away right and and then then >> so there's a broader team that handles the other aspect so um we have portal and all the templates that you know go into it and you know bunch of other layers but ultimately you know the app team gets a point in time copy of whatever the you know latest and greatest is when

they start you know fast forward a year and you know some things have changed and requires updates on their end you know we can update the templates but >> you know they you know their copy was a point in time from before that change so pushing those updates to all the app teams has been >> I think so that is then there should be persona that works

with the developer team to kind of address those things. That's the only way that I can think of like having a core platform engineering. I I talked about those four personas, right? So the the I keep going back to the DevX persona because I just feel that they have replaced the DevOps of the yester years. They are the ones that understand what the friction points are. They

are also the ones who under who should be in the better position to understand how to improve the challenges that you're talking about. That's how I I think about it. But um let me think about that. I maybe I can connect with you afterwards and and kind of understand a little bit better and get you a better answer. >> Awesome. Any other questions? Awesome. Thank you very

much. Um. Okay. Yeah, >> it says this, but it also says low. I don't know what low is. >> Oh, this was the Yeah. Uh, it's been it's just said that the whole time, but as long as the battery. >> Cool. >> Great. So, I just flick this on when I'm when I'm ready to speak. >> Sounds great, friends. Thank you. I think my mic is live

now. >> Okay, cool. I should probably pull up my slides. All right, I guess we'll get started. Um, so I get the pleasure of introducing Lee uh Capilli. Um, for those who don't have not had a chance to hear Lee speak in before, he's an exciting speaker. Uh, just so you guys know. So, you guys are in for a treat. Um, Lee is a Kubernetes contributor. Uh,

he's a cloudnative practitioner and one of the most entertaining speakers, at least in my opinion. We'll see today how it goes. Um, but yeah, uh, what makes Lee stands out is that isn't isn't just his depth of, uh, his technical knowledge. Uh, it's also that he actually makes stuff fun to watch. So, um, with, uh, I guess, uh, no further delay, Lee. >> Yeah, thank you, Facel.

Yeah. Uh, hey everyone. Excited to be uh, back from lunch a little bit. actually had a a friend um run and and get me something which was a pretty fun surprise. Uh but today uh I have a little bit less of a theatrical presentation than typical just slightly um and uh we're talking about tracing what a GitOps reconcile um looks like in the cluster uh and in

your repos and in the storage layer uh beyond Kubernetes a little bit. And then uh I'm going to be using Flux as an example because that's what I know best. Um, I've been on the Flux project now for the better part of a decade and we are coming up almost on our 10-y year anniversary this year. So, uh, if anyone Flux users, Flux users in the in

the audience, sweet. Nice, Fab. Um, great. So, what actually happens uh between a git commit and running pods is something we're gonna get into. Uh, just a little bit of about me. Um, again, yeah, I'm a Flux Core maintainer. I work with control plane uh which is a company that kind of came in in the wake of weave works uh and saw some like huge value in

the project. Uh flux you know runs pretty much in every cell tower in the nation uh and on like planes and satellites and tractors that are picking apples and uh you know point of sale systems everywhere. So it's uh it's all over the place and that means that you really do need like a secure footprint uh and approach and that's a control plane specialty. we you know

helped uh do a security audit of uh GCE uh and the you know helping the Google cloud team as well as the basis for GKE and you know work with space agencies and financial services uh and all kinds of people and uh those are some of the clusters that I have the immense and humbling privilege of working on and when I stand at the Flux booth I

talk with people who are using Flux in all sorts of wild ways. So feel free to tell me what weird thing you're doing with Flux or other weird stuff like Nyx. Um I grew up here in LA, so it's always nice to come back to scale. Uh and and then uh I have a couple of side quests that I'm interested in and Jerome actually helped me with

one of those today. So he grabbed me a boba. Thank you, Jerome. And uh if anyone plays like Dance Rush or Pump It Up or Dance Dance Revolution uh or wants to do anything on a skateboard uh then those are my side quests. Cool. And then just a little quick note on why GitOps. Um GitOps I think is more than just a technical solution. Whenever we think

about GitOps in an organization, um, we're solving a people problem. And it's funny like working in open source for so long, we're always solving kinds all kinds of strange people problems. But GitOps like brings the interface of our fancy distributed computers with all sorts of um, you know, computer science necessary with in promise theory to realize our desired state you know to what we want happening across

the globe. It brings that interface to a place where we can actually collaborate on it with each other. And that's why GitOps is so important. And GitOps is really an of many of the problems that we uh ran into in the DevOps era when we first started formalizing what it really means to work with each other in these contracts and and realized that there's all kinds of

ways to to deal with and name and identify and pattern organizational dysfunction and how we can decide what we owe to each other. But ain't nobody logging in to your Kubernetes cluster to talk to your teammates, right? You know, you're always doing that somewhere else. And so we've got this staging area to work together and keep a central record of how the intellectual property of our businesses

and our projects evolve. Flux helps you do that. Um, more succinctly, write funny YAML of deployment, get commit, am yolo, get push, and then the pods uh all over the world just go. But uh I wish I could roll my RS better because I would really indulge in in uh rolling that out. So Flux has a couple of controllers. Um I wrote this little like mermaid diagram.

We have a better version of this on the Flux website. Um there's a source controller in Flux which fetches artifacts. And that's more than just Git. Uh one of the really cool things that's happening uh with our with our customers and which with sophisticated Flux users is we've realized gosh there's already a really good immutable state store that is tightly coupled to the life cycle and availability

of every single distributed computer that you're running with Kubernetes. You need an OCI repository. and whether or not you're using that because you buy it from AWS or you're running your own Harbor instance or doing some weird thing on NFS, you know, thing exposed from your cluster storage, then you know, you've got an OCI repo and you're probably happy with how available it is and your cluster

has the ability to authenticate it to who it already. So if you put sources there the actual things that you want to run on your cluster the desired state into the OCI repo source controller can fetch it whether it's from git or OCI pull it in serve it a private uh network policy protected HTTP server uh and then customize controller and Helm controller can go and realize

those things into the Kubernetes API. Yay. Now you've got pods running right. Uh now what if things happen? Well, we've got our notification controller that's ready to start using your desired alert configurations to ping you on Slack or Discord or update some Git commit status notification. This is kind of the core of Flux and these are the controllers that you get by default. But then if you

move over to the image reflector controller and the image automation controller, we can also start to close the loop. Right? So when we push stuff to get, we are asking the computer to change the the desired state of the world. But what if we maybe push stuff somewhere else? Like say push some tags for an application image to an image registry. Well, there's more sources than just

the YAML files that we want to run in the cluster. Say we update a version of an artifact that we want to deploy. Say we update a version of an artifact of an OCI repo that we want to deploy the manifests of. Well, the image reflector controller can watch those tag updates and it can follow your image automation policies in order to push commits back to Git

and maybe even build those things back into OCI repos to then flush through and then do more deployment. And so now you have an automated loop uh for machines to actually start operating and collaborating with you in your place of collaboration. This is something that's really difficult to do in a way where you keep a record of what the heck happened and what the intent was and

how environments pushed from one to another and that that sort of thing inside of a Kubernetes cluster. But when you do it in Git and you do it with GitOps, you start to build a really cool basis for things working together. So that's the Flux controller architecture a little bit. Now there's a Bootstrap problem here. How the heck does Flux even get into the cluster and start

talking to Git? Uh you need to set up your repo. You got to configure some access. You're logging into GitHub and you're like, I got where's the deploy key thing? Let me go and make sure that this thing has read access and then I got to install Flux onto not just one but 10 clusters and then configure Flux with the proper secrets and each one has to

have a secret because you know like those things shouldn't have the ability to read the state of another cluster. And then we've got to also manage Flux itself somehow. So maybe we can do that from the git repo. And so uh we wrote a little command to do this uh back in the early flux days when we rewrote flux 2 which I think was in like 20

20 2019 something like that. And um with flux bootstrap you can just be like hey I have this GitHub or here's this repo. Uh and then it's going to go and make sure that the repo exists. Uh if it's not there already make sure using your personal access token to generate a pair. Put the thing there. commit the manifest to the repo, apply the components to the

cluster, make sure that the SSH key is configured there, you know, like it just all does it for you in one command. And that was nice. Um, but then we wrote flux operator um because people were like, man, I really got to run this CLI like a bunch of times and update 100 repos for my fleet of a thousand clusters. And so you can just run flux

operator there, put a flux instance in there, and instead of having to commit the flux manifest, which you're really just getting from us, the upstream project maybe with your patches, uh you can specify an OCI distribution of flux from an OCI repository with a manifest. It's signed uh with salsa build level 3 build provenence. Uh you can verify that key. Flux operator will do it by default.

Decide which components you want to deploy, put some patches on and sync it up to your uh repo. And here you can actually do completely gitless githops uh and just point flux to manifests that are straight from OCI. So your bootstrap doesn't even have to involve any um any secrets uh and any availability or dependency on git even from the very root of your cluster. Uh which

is quite nice if you're using a managed Kubernetes provider. Uh you can do p passwordless machine identity uh and everything will just bootstrap. There's no secrets at all. So that's pretty cool. Uh check out flux operator at fluxoperator.dev. Um we're just trying to make things a little bit simpler. This is a control plane flux CD project, not just a flux uh upstream project. Uh the license a

little bit different, but it is fully open source uh and free to use and everything like that. We got a nice user interface. I'll show you that in a sec. so in order to build flux, right, this talk isn't is about an anatomy of of what it takes to reconcile resources into the cluster safely, securely, and in a performant manner, which are the three tenants of the

Flux project, the things that guide us uh and and the things that that's like our north star for the past decade. fast, performant, right, secure, and extensible and trustworthy. And so, um, we we use the core Kubernetes SDKs to build Flux. And we do it in the style of Kubernetes. Uh, there's client go, there's API machinery, controller runtime, um, which has gotten significantly more reliable and fun

to use over the years, but was definitely a battle in the early days. Uh, and then, um, things like the customize API and KML. uh from the Kubernetes project to build YAML and then also post render stuff for Helm. Uh there's a couple of core libraries. So we've got this repo called pkg in the flux directory u uh in the flux or you'll see this in in

go projects and a bunch of people really dislike it but it works well for us. uh we'll put like runtime or SSA or a bunch of SOPs logic uh for decryting secrets or uh dealing with SSH keys and like knowing which file names to read them from inside of a secret and how to unpack a secret uh things that are used all over the Flux project. we

tend to create a folder for it and a package in package and then um I can't uh talk about like the core of flux without at least mentioning goit uh which is a highly extensible git implementation in pure go. So when we when we built uh Flux, we really wanted to avoid uh it just being a pile of bash scripts. And uh we're very proud to say

that pretty much all of Flux uh is native Go SDKs, which means that it can run in soft multi-tenency uh with things concurrently passing around contexts and clients that only have permissions to the things that they have. We don't pollute the environment variables overly. We we're not polluting the file system like we're just creating strrus and calling methods uh in relatively type safe and sane go. Uh

and that's super cool um because that means that we can clone a bunch of git repos concurrently and put them on the file system and make sure that only the people who have access to them, you know, in the name spaces that they need to be. It just respects the security model of Kubernetes itself. Um, so that's what source controller uses um to go and get your

git repos. We also use the helm SDK natively, right? So there's no executing the Helm binary. Um, there's the Helm V3 and V4 packages. Uh, Flux now supports Helm V4 uh by default. Uh we have an API version tracker in the status of all of your existing Helm releases so that if your cluster has thousands of Helm releases that are already using V3 APIs, we're not automatically

changing behavior on you. But if you're creating new Helm releases, uh with Flux 28, uh everything is going to be Helm 4, which is brand new. Uh and a massive massive effort went into this uh from Stefan uh and from Matas for the Flux 28 release. So huge thanks to them uh for creating a very high quality uh very safe implementation of Helm 4. Uh and also

kudos to the Helm 4 people for working with us in Upstream to actually fix bugs um that we that we've found uh and and getting them out in a really rapid release cycle. But the reason that it's fun that we use the Helm SDK is we can actually implement features that are impossible to do with the Helm CLI. Um so we implemented our own drift detection uh

using some of the knowledge uh and comparison logic that we learned in customized controller over the years. So we implemented that uh you can also do Helm's storage in a separate name space uh which makes sure that your application doesn't have access to modify the manifests that are necessary to deploy itself. Uh and that means that if you've got secret read like say as a developer in

your app namespace, you don't necessarily have the ability to go and uh muck with uh everything that's managing your app. Uh it prevents a security uh boundary from being escalated there. Uh there's fine grain status reporting and you can react on that, right? So when a Helm uh is running some tests, you have an API that's telling you that that's happening exactly. It's not just some random

log message inside of a terminal. uh in the same way like this test failed or this test succeeded or we're doing a helm roll back uh or these things are healthy and this thing is not or these are the resources that got deployed with your helm release uh when you have a when you have the helm SDK and you're representing everything as a structured state machine you

get to do that uh you get to do complex life cycle and remediation this is a new feature that we have it's health check context cancel uh when you say run a Helm upgrade with the Helm CLI and you just say, "Okay, I hope this this succeeds in 15 minutes." Uh, but you don't know what's going to happen. And then you pass it the roll back flag

and all of these other things to make your CI safe when you're using a GitHub action for it. Uh, and then you want to, you know, I don't know, realize that you like had a typo and you want to redeploy. So like, are you going to wait like 15 minutes for that thing to time out or are you just going to brutally kill the Helm CLI with

with the entire life cycle of the the GitHub action? Uh this is something that there's no answer for uh in the Helm CLI because it's an imperative tool with a with a global background context. Um but I will show you a a PR later that solves that problem. And uh yeah, you just get more efficient client machinery. Also just uh we need to talk to all of

the cloud providers uh to integrate with their key management and their identity uh again all the machine identity stuff that we released uh last year as well as working with container registries. So there's the Google Go container registry package and the ORAZ package. Um cosign soops for decrypting secrets. Uh notation as well for verifying signatures. Um cosign verifying signatures. Uh file path secure join. Uh that's a

fun one as well. And uh we use controller runtime with a bunch of like managers uh cache setup clients rate limiters etc. And we encode our opinions inside of the package library. Uh, I already talked about why we don't want to be exeacting a bunch of binaries, so we'll just skip this slide. Uh, but, you know, it's more secure that way and better and faster. Um, so

let's go and look at a little bit of code because I feel like doing a lot of rambling. So, I'm going to hop over to the main go for customized controller. Uh, just kind of showing normally when I work on Flux, uh, I have like this little kind of directory where I just put a bunch of stuff. Uh, this is roughly the repos that I work on

at any one point and then you know some experiments and stuff and uh so in the main for customized controller uh we need to set up a manager uh now this is a binary that takes in a bunch of command line flags options and feature gates and things like that so we have to set that up right so here uh for instance there's like disallowed field managers

uh with a you know you can like or Like if you want a global SOPs age decryption secret, you know, you can set it here. And then uh we take some feature gates, pull them in. Uh you can, you know, have to plum all of the fields that you just took in. Uh and then we try to get a configuration from controller runtime. We want to uh

make that config. And here we're like starting to deal with uh details around like leader election if you're running more than one customized controller for high availability. uh working on uh all of this plumbing the settings from the command line flags through. Then we make a manager. We've got a bunch of error handling. You've got a I just added this factory in here. And then eventually you

get to instantiating this which is where we get to the meat of the logic of customized controller. Right? So we're passing all these options to this reconciler object. uh and then we set up that reconciler with the manager and then start it. So this customized part of the style of implementing a controller with controller runtime. So if I go to this reconciler, there will be a reconcile

funk. And this reconcile funk is what happens anytime there is a to operate on a new resource inside of the Kubernetes cluster which this is registered for the customizations API. So anytime we change a customization inside of the coupe API, we get the reconcile funk, right? And we come down and the f the very first thing we need to do when we um aside from setting up

finalizers again there's like a lot of machinery here to make it happen. The first thing that we'll need to do is get a source right so customize controller will go and grab a source tarball from source controller right and it'll make sure that that is compliant with the artifact uh API. The next thing that we'll need to do is check our dependencies. uh you know customized controller

can create a directed as cyclical graph uh with a bunch of uh other customizations and order them to make sure that say you don't install a controller before its CRDs are present in the cluster or that you don't try to configure manager before it's able to actually you know do that. Uh and this sort of ordering thing is another thing that's difficult to do without githops. Uh

and then we kind of get in and if we look for uh the apply kind of phase uh you'll see that there's this idea of change sets and in order for us to promise you that your desired state is going to be reached. Um the a big difference between using a flux customization and doing a coupe cuddle apply is that when you coupe cuddle apply it kind

of just tries to go file name by file name and resource by resource and it sprays them into the cluster best. What we do uh with Flux is we do a serverside apply and before we actually run the serverside apply, we create a change set, we track what we expect to see, we do the serverside apply as a dry run and we compare that with this concept

of an inventory. So we make an inventory and we are going to put that inventory into the status of the customization. So the inventory is going to track every name, every namespace, every group version and kind of the thing that you are expecting in the desired state for the source of this customization. If it's a folder at a particular branch in your repo or something if our

dry run succeeds then we have reasonable expectation that when we actually try the serverside apply bar a couple of edge cases um that we are going to succeed. Uh and then kind of under the hood actually in the pkg SSA library we do this in phases for like certain exceptions like in arbback uh where you always want a cluster role to be applied first uh and that

sort of thing. But we'll always dry run it uh and then run the stages and then try to get that change set to be atomic. And what this prevents you from happening is say you update four or five files in your repo uh and one of them has a typo but the other four are good and you accidentally apply only two of them and then reach a

stuck state. So we we're really trying to avoid you having a non-atomic behavior, right? If there is a if there's a typo, if there's uh an invalid API declaration inside of one thing in the change set, uh we are going to use this inventory tracking behavior to prevent that from making it into the cluster. And then we also use that inventory to do drift detection. Uh that's

what this diff is using. Uh and that's also how we prune. So this is a little bit of the guts of customized controller. Um that's kind of the rough logic and how we set up the manager. and we'll kind of pop back out. So you can see it's already complicated deal with applying a couple of resources to Kubernetes, right? And that's just in customized controller, which is

all code that we mostly control. Um, and so it's really nice that we've got controller runtime to deal with our work Q, deal with storage, deal with the kind of inventory of everything. And um we we've got how we we use um API machinery. Again, as just as a recap, serverside apply is uh came around kind of in like around 2018. Uh and Flux was one of

the earliest kind of uh users of it at scale. You know, from the Kubernetes client side, we found a lot of bugs over the years, and it's now in a really nice place. Uh this gives us our conflict resolution and field ownership and gets away from uh all of the weird annotation stuff that needed to happen in order for coupe cuddle to manage this problem before. Now

um this is a really cool code snippet also from customized controller. Uh and I I made a slide for this specifically because it's something that is really core to the security model of flux. So flux lets you do multi-tenency, right? uh kind of back to the the very first slide where we were talking about why we do GitOps. Githops solves a social problem, right? I'm working with

my co-workers. We're collaborating on these change sets in Git so that we can add context and maybe have a hope in three years of remembering what the heck we did when we added this weird EngineX flag to the EngineX controller and now we're needing to migrate it to traffic or something, right? Um, and so, uh, I really don't want to be in an app folder and have

too much permission to do damage to the cluster. I need to be able to drop permission. on a customization if there is a service account name configured uh then we're going to make sure that the Kubernetes client that we use for reconciling that customization and adding those resources to the cluster impersonates that service account and we do this with a specific Kubernetes feature that you need to

have permission to use. The flux controller is basically God in the cluster and and it can impersonate the list of things that you tell it it can impersonate which is usually everything. Uh but you can restrict the list with with resource names if you would like. And um if there is a coupe config uh and and instead we are actually reconciling to another cluster in a hub

and spoke model uh then we are going to configure the client appropriately and that can also be combined with service account impersonation but it will be bound to whatever the namespace is in the uh destination cluster. Uh and then we use the package runtime um uh shared library or cost flux to make sure that we construct this client in an appropriate way and do it securely. Uh

and this is how we make sure that across customized controller and in Helm controller when we're passing clients around uh for the SDKs to do things uh that they're not getting polluted by some controller state. It's always bound to the permissions of whatever the resource in that namespace is supposed to have. Uh and so that's how that works. Now when we are reconciling things, um there's a

couple things to know about working on a resource in Kubernetes and writing a controller. Uh anytime you um anytime you want to be sure that you are reconciling a resource, uh you need to know what version of it you're working on. And so Kubernetes uses that CD uh to have a consistent understanding of what version of that object it is. Uh and it does this internally in

Kubi API server in the storage layer. Now there's a little trick that you get uh every custom resource definition which all of the Flux API types are are custom um they implement object meta and object meta uh gives you metadata, it gives you a spec and it gives you a status. Whenever you update the status uh you will get a new metadata generation given to you on

that object automatically from the coupe API server. Right? So if I if I run a patch and I change a field from from false to true on my custom resource in the spec, it'll become a new generation. Uh and so when you write a controller, you need to track what the observed generation that you have reconciled is on that resource. Right? So, if I go to the

Flux UI, um, and I go to my favorites, maybe um, let's just open up pod info. This is a simple Helm release. Uh, there's a specification and say I change this, right? Uh, timeout I I set 15 minutes, right? when I give this to the Kubernetes API in the metadata section when it writes it to CD it's going to always update generation for me and then in

the status when Helm controller works on this uh you'll see evidence of it tracking what generation it's working on right so here there's observe generation uh and then we also have this last attempted generation we actually need two fields here because the Helm release is kind of a complex state machine and then even in the conditions uh these observed generations for these conditions has to independently be

tracked as well because a helm release can represent the status of not just one thing but many things. Uh it could be the hooks, it could be the tests, it could be a roll back uh or a version of of a thing. Uh the H helm is an incredibly imperative tool and there's all kinds of strange things that can occur when you hit error states when you're

installing charts. Uh we'll look at some of that code in a second, back to the slides. Oh um yeah, there's a note here on drift Um that's based on field ownership. Just a little bit more of like how we deal like tracking what we're working on. I don't know. This is kind of out of place. Probably should have put that on a different slide. So, uh, also

when we're reconciling an object, we need to report what the heck is happening. Uh, there is a whole spec. Actually, I should probably look this up and show you where it is. Um, case status spec wait. No, that's Helm's docs. Uh, CLI utils read me. Here it is. So, um, under CLI utils from the Kubernetes sigs or uh there is a package called K status. uh and

this kind of lays out what it means uh for a Kubernetes resource to be reconciling and then communicate what is happening with it. Right? So when uh when we're looking at like say the status information of this Helm release, uh there's this list here. It's a YAML list called conditions. This is super small. I realizing now I'll make this way bigger for you friends in the back.

Sorry about that. Um so here there's a condition. um it's a ready condition and then there's a different condition called a released condition, right? So whether or not the Helm um release here is ready to receive traffic or is considered healthy. Uh and then whether or not the release itself is done. Uh and whether and then you can put a reason in there which is supposed to

be kind of like this human readable enumeration as well as a message that's really intended for like user interfaces and things like that logs. Um and so this this schema is what it looks like to be K status compliant. Um this is notable because in Helm 4 uh which flux 28 now uses as default for the health checking um everything changes and now custom resources that did

not health check before uh across Helm uh they they actually get health checked if they are it's case status compliant, but not every custom resource that's implemented across the Kubernetes ecosystem is case status compliant. Um, with Flux, our health checks, uh, and all of our APIs, they kind of are in this style. So, if you have Flux resources inside of a Helm release, uh, then now the

Helm releases wait for those things. uh and if if you uh for some reason have to health check in a way that is not case status compliant, we give you a different API called a common expression language uh expression in order to or a cell expression to compute what you need to do to have your custom health checks for your weird cluster API thing or maybe a

crossplane or something like that. So when we are say reconciling a helm release or reconciling a customization or an alert uh with notification controller or even a source just pulling in an OCI repo when we when we start working on that thing we always will mark it we start reconciling it we say we say something nice to the user that's why using flux is so nice when

you are in the UI when you're in the t the text user interface you always know what's happening with with the particular resource it's not just like this nebulous condition. It's because everywhere in our code we're always updating these using the conditions uh shared library from pkg, right? Um also all throughout the controller we are producing metrics. Uh so anytime we do something that we think oh

maybe an S sur would want to know that that's happening inside of the core of Flux because you know Flux is managing 10,000 apps in this 4,000 node cluster right it's great to have a couple of series uh time series metrics for you know things like how long did it take to reconcile a customization on average what name spaces were were those customizations in uh and so

you you throw a couple of labels on those time series and you can start to slice it up here's how deep the the workload Q is um here's how latent it is, you know, for Helm controller to pull some Helm charts from your super slow v1 HTTP Helm repository, stuff like that. I should probably turn on caching. I should probably up the concurrency or autoscale that or

whatever. Um, you get a great view of all of Flux's communication uh with Oh, I should have pulled up that that metrics dashboard. I don't have time to go sifting for it. uh you can use these metrics uh to get a great view of of the flux communication with Kubernetes itself and then that's how you know that flux is not slowing your Kubernetes cluster down it's actually

all of your mutating admission web hooks uh and because open shift is weird or something like that so uh notification controller is also then uh using all of the events information so we produce a bunch of Kubernetes events uh and you can watch what the heck is happening with Flux in that way. Uh, and I I guess the really the the tone that I'm getting at is

we try really hard to sensibly extensible. You know, like if there's an interface uh inside of your coupe cluster, inside of Prometheus, uh you know, something that can interface with your Graphana dashboard, uh then then we go and we do it and there's code for that and we can show you how it's done and you um have a bunch of shared libraries that you can use to

build things in your own way. Uh the user interface that I've been using here is the UI from Flux Operator. Uh it's pretty sweet. You can go in, you can make uh favorites. Flux operator itself uh is kind of our project to you know make Flux go into the future. Uh we've got an MCP server. We've got a user interface. Uh and you can auto update Flux

and do you know things that make sense at high high scale infrastructure. Um but yeah, I've got my fancy UI here. Here I got my favorites. I can go check that Flux is healthy. Here's all the components, you know. Here's all my Helm running some things. Uh for tuning options, uh if you're running into issues at scale with Flux, uh you can increase concurrency. So like we

see uh people with large clusters running say 25 concurrent go routines for reconcile functions uh at one time uh say for Helm releases or customized controller. You can um reduce the polling rate for dependency reqing uh which will help your um dependency graphs especially during cluster bootstrap. Say you have a lot of things that are waiting on each other and depends on you can turn that down.

By default, it's 30 seconds. And uh that'll make the roll out of a of a fresh cluster bootstrap with complex infrastructure dependencies uh much faster. Um if you want to tell things to chill out a little bit in error conditions to maybe get away from like stampeding, you know, back off and stuff and the 15 minutes is not enough, you can increase it. Uh similarly, this is

default 750 millisecond retry delay with the jitter. Um, but you can, you know, increase that if you'd like and the jitter will apply. Uh, we recommend to people to use web hooks when they can. Um, but it's not always practical. So, go ahead and pull away if you need to. Uh, lots of people pulling thousands of git git repositories every minute. Um, naturally moving to OCI repos

makes that significantly more scalable. Um, we have uh where Helm controller and customized controller can watch when you have values from a secret uh and you're trying to read things. Um, if you add this label, then by default, Flux is already going to watch uh those secrets and config maps uh and get a a metadata update for when they change so that it goes and rekicks the

customization or Helm release that's depending on those things. Uh so that's a really nice way to uh improve the meanantime to deployment. Uh if say your main contract with the developer team is to just have like a YAML file inside of a config map and then you're managing the Helm release for them and it never changes. Uh and then also uh that thing that I mentioned cancel

health check on new revision. I'm really excited about this feature gate um because it I get to tell a story about people uh and so story Make this nice and big. This is my teammate. His name is Matteas. He's a very friendly person. Uh, and this is just like October of last year. We're actually in the Helm repository. This is where Helm comes from. Shout out to

Karen and Scotty and Matt and everyone from the Helm project who are such amazing uh people who work at a massive scale engineering the needs for many many people over four versions now. Um, yay. Helm V4. Hello. We're implementing a feature in Flux's customized controller that allows cancelling the health checks of an ongoing reconciliation when a new reconciliation for the same object has been incued in the

controller. So, basically what Matus is getting at here is he's like, hey, everybody's suffering because they made a typo in their Helm values. They already know that the image is not going to pull um because they just changed the format of how their commit messages are written. Uh, and now the tags have some suffix. They know the Helm release is going to fail. I just want to

commit and just have Flux retry. But remember when I told you that because Helm was implemented as a CLI first uh that all of the internals of Helm were built with background context. So this background uh means that before you were not able control the context uh when you would do a Helm upgrade. If I can type while I'm talking then that would be helpful. So this

client um that we're going to pass to Helm, we want to be able to control Uh and this this pull request that I'm looking at is part of Flux Helm controller. So this is this is something that we implemented to make this feature happen. But what Mata is asking here is, hey, can we please work together to update the Helm SDK um because we can't pass the

context and that means that we can't cancel the health checks that are destined to take 15 minutes or in the case of some virtual network functions inside of a 5G tower, you know, with a complex umbrella helm chart, maybe the timeout is 3 hours, right? And you're just stuck waiting for the three hours because you cannot close this thing safely because the context is literally the context

of the entire controller that controls Um, and and so you're, you know, you just got to wait and and this is a this has been something that people have suffered with uh in their GitHub pipelines and and influx for for many many years. And just shout out to Mataas for uh for making this change. Um in the end of the day, this this is the change that

actually went into Helm. Uh it's you know, he's he's got a bunch of tests in here. You got to make sure that it works properly. Uh and you know, you you plummet through and provide some new functions to do the same things uh that you could do before. And then we go and consume those in flux. Um and then in the code we have a cancel health

check on rec is the feature gate and so if it's not configured then we just do things as before uh when we uh configure the reconciler but if it is then we wrap the uh and so that's that actually that from helm controller. Let's go look at Oh, I had this reconil Helm release manager. Here we are. And this reconcile funk that wraps the reconciler and calls

it uh basically uh is looking for if there is a weight context uh for a health checker on the reconcile instance that is currently running when a new reconcile happens for a new Helm release spec change. And that's what allows us to safely stop say listening for if a deployment's pods are ready uh also accidentally stopping uh custom resource definition version upgrade or running hooks that are

doing a database migration or running tests and things like that. Uh and this is the kind of thing that is only possible and and really does stand on its own across the ecosystem. I I'm so proud to be standing here like many years later after people like Ha and Matteas and Stefan and Michael and even like I I've touched it a little bit. Helm controller is in

a place where I view it as the declarative way to do Helm. It's really a spiritual successor to the project and you you just have such a strong promise and a great UX with something that is really non-trivial and complex. And I hope that I'm succeeding at least in in showing you a little bit of not just like the the sophistication that's necessary in order to do

Helm properly inside of the cluster uh in a controller, but also the the kindness and and also the ju just the social interaction that's that's that's necessary to make these things happen. You know, the these people are wonderful uh and it's really a privilege uh to do this. Uh different companies, same team. So so tuning options, you want Helm to be fast, go turn on the feature

gate. It's new in Flux 28. Check it out. Uh the original Flux defaults, they were chosen to work on Raspberry Pies for some of our earliest contributors. So um in Flux Operator, we do have improved defaults and uh some t-shirt sizing uh for like more enterprise use cases. You can just like be like, "Hey, I want a a medium version of Flux or extra-l large version of

Flux. Go turn it on." And now you have the ability to scale uh to your clusters in Flux Operator. Uh we sign everything in Flux. We produce sbombs. Uh the build infrastructure, you know, Stefon has worked with the GitHub security team to make sure that our workflows are solid. Um we cannot have a supply chain vulnerability in in flux. It would be a massive disaster. And so

all of the maintainers uh take our build infrastructure uh our personal security uh and and our our workflows and how we steward this project incredibly seriously and that stuff uh it it continues to evolve and and we're always looking at what's next. I will say we're getting a lot of AI slop right now. So that's something that we are looking at. but uh the project's really dope.

Uh these are a couple of notes that you can get from our security documentation as well. Uh things about how we do arbback in the controllers. Uh how we promise that um soft multi-tenency is kind of cool. How you can do sharding things like that. Um crossnamespace policies you can turn them on and off. There's security documentation on the flux website fluxydioflux. Uh and you can see

how seriously we take cool. Lastly, uh we did something kind of fun. We we built a controller a while ago called Source Watcher as a demo for how you could build your own GitOps tools in the Flux style with our libraries with our techniques. Uh we say that Flux is actually built with the GitOps toolkit. Uh and there are actually a lot of vendors who use pieces

of Flux. Uh like when I was at VMware uh they were using source controller to feed their own other delivery controller called cap controller and they were putting application builds uh and application source code into source controller to do builds on some Tecton workflows or something like that. Kind of cool. Uh you can do all kinds of weird things with flux. Even uh I think Palumi at

one point had built their own delivery controller. We we added a new API recently to source which is our little demo and released version two of it. It's called external artifact. And with external artifact, uh, you now have an API that all of the flux core controllers support to build your own way to populate sources into the cluster. So if you would like to say read some

random YAML uh or random JSON blobs from a Postgress database and then expose them to to Flux in a secure way, uh, we support that. Go and implement external artifact and check it out. um fluxdioflux githops toolkit. Uh you can also build your own appliers. Uh there's previously been like a infrastructure as code controller to do terraform and open tofu stuff. Uh so you could you know

have flux use source controller, pull in your stuff from an OCI repo and then run Terraform apply on it, get some status updates on it, react to it that do some cool things. There's uh there's really cool ways to build your own things. But uh yeah, going into the future uh it's pretty exciting. Uh we've got the resource set templates that are now exposed to you from

Flux operator itself which is not only a great way to install and manage and autoupdate flux but also a great way to do higher order operations on top of flux right like if you wanted to automate the creation of tenants uh then you can use a flux operator resource set to template that out also make sure that a helm chart is installed in their things to I

don't know provision their uh weird virtual IP whatever whatever weird infrastructure stuff you nerds are into. I I know you've got something that you really want to play with some like mesh gossiping, you know, OCI repo sharer across nodes on some random port or something. You can go and do that. Implement your weird um opinions. We have a user interface. I've been showing you a little bit

of that. Um people are loving this. Uh it's it's actually really great on a mobile phone. So, if you want to pop your phone on the VPN every now and then when you get a page at 3:00 a.m., uh, you can go and check your favorites and be like, "Hey, why is that that, you know, notorious Helm release from that dev team, you know, failing again and

then you don't have to log in on coupe cuddle just to see that their service monitor was misconfigured." Uh, you can just hop into the Flux status page from the UI and Flux operator. Uh, also separately deployable. And, um, what else? What else was I saying on this slide? Passwordless machine identity, gitless, githops, getting away from depending on um the questionably dishonest uh uptime percentage of GitHub

status page uh and u you know maybe relying on something a little bit more immutable and simple and a little closer to your cluster in terms of availability and moving your sources into OCI. Uh we've got a bunch of examples in the control plane or on how to do this uh with salsa build level 3 providence in a way that actually crosses the air gap really nicely.

Uh chat with us if you want to uh talk interesting security use cases or strange compliance workflows. Um and then uh agentic githops you know I too talk to the machine now and sometimes it talks back into me but uh yeah I think there have been many times where I've wasted four hours uh trying to figure out which patch was coming from what branch going into what

cluster and what file I was supposed to edit in order to find out how this template was going to change in this subchart of a Helm release. Uh, and it turns out that when you have hundreds of clusters, know 30 repos that if you just check them out in your directory and and use an agent and then install the Flux MCP server as part of Flux Operator,

uh, you can get a pretty good readonly workflow and even a write workflow for uh, triggering reconciliations to debug what the heck is happening in your cacophony of chaos. Uh, and Aentic GitOps, I thought it was a joke. We were dorking around just because MCP seemed fun. and it's actually useful. Go check it out. Flux Operator MCP. But yeah, thank you so much for letting me come

home uh to the place where I grew up uh to have a little side quest and chat to you about why I'm so proud of what we do in Flux uh and what we do at control plane and why I'm excited for what's coming next. And I hope that uh this walkthrough of the code shows you uh just how much respect I have for the thing that

we've built over the years. Um we really do something beautiful uh with Flux and uh I'm willing to have any questions or chat. I love making new friends. Uh and yeah, thanks. >> Thank you. Le as always a treat. Uh we got five minutes for questions. the the impersonation thing is pretty cool. I don't think I had seen that before. And so you were showing that all

you have to do is to add a service account into the customize CR if I understood correctly. I suppose there is something similar for the Helm resource and if so like how is it shared between the two on the on the code side. >> Yeah. Um so this last line of code here uh where the impersonation code is being created uh is using it's aliased as runtime

client uh but that's coming from pkg runtime inside of the flux uh org. So pkg runtime uh is just a way to create generic Kubernetes clients. Sometimes they have to have a different shape depending on what you're doing with them. Uh and but whether or not like you're trying to get it stable to a coupe config for an apply to a remote cluster or a status check

on a remote cluster uh or you're trying to drop privileges inside of a namespace using a service account. Uh this is like really core to the flux security model. Uh we actually even recently added impersonation I think to um I think it's image policy if I remember correctly. Matteas had to do it last year and it's again using those same uh impersonation prim primitives from the shared

library of pkj runtime. So this code snippet is from customized controller specifically the one on screen verbatim word for word. Um but you have similar ones like remember when um I was showing might have had the yeah rap reconciler oh no it was actually was actually when I was showing code here and there was the uh client. Yeah. So like this is Helm controller stuff, right? And

you see that we're creating these Kubernetes clients and these uh clients are being created sort of in that same way where when we create the client to actually do the Helm things uh inside of that state machine, we're using the PHG runtime uh for impersonation and that's also with the service account field on the Helm release spec. Does that make sense? Um that can also use machine

identities as Great question. >> Any other question? >> Hey, I had a question about the support of this as the company behind the uh flux CD is gone, right? And then as well the operator for the flux is picked up by some other company. And I'm just interested what the support going to look like moving forward for the flux as well. that operator going to be moved

under the CNCF to be developed and uh distributed for uh users. >> Mhm. Great questions. Um Weave Works uh had to close its doors gosh I don't remember anymore. Was it 2021 2022? And I used to work at Weave Works. Uh my teammate Stefon used to also work at Weave Works. Um Stefan is like basically the project lead of all of Flux. He probably has the most

sway. Um if we have a ban dictator it would be Stefon. We don't though we we have a pretty clear governance model. You can check in our um governance repo. Um Flux is also not maintained by one company. Um control plane has three core maintainers right now but DTPI uh Microsoft Kingden uh NASA. Um we've got you know a bunch of people u involved. Ha is actually

at Acuity um doing S sur on Argo stuff uh but still contributes to Helm as well as a bunch of our dependencies and we are very friendly with Ha all the time. He's one of our great friends. Um so so Flux is is made up of lots of people and and it is a multi- vendor project. Uh I would say the the biggest I don't know it's

it's unfair to Microsoft to say that that control plane has the most right. I mean like we we definitely put in a lot of effort. We're building products around Flux. Uh as far as Flux operator getting donated to the CNCF, I don't think that'll happen. Um we we already have the Flux project. The Flux project has been a graduated project in the CNCF for uh quite a

long time. Uh and it's going to stay that way. And it's really important to us that Flux is open and it's vendor neutral and that we're not just like changing it in drastic ways. uh but as a company um you know I work with control plane uh who's a consultancy where we like we need to be scrappy um we we do need software that we can change

to meet our customers requirements uh and that's a little bit of where flux operator comes from but flux operator is also totally for the person with the Raspberry Pi in their bedroom like that's we we're never going to lose love for those people um but but yeah I would say the operator and the UI I um we're not the only UI in the game. There's, you know,

there's lens, uh there's headlamp with the Flux plugin. Uh somebody just released Flux 9, which is a Flux specific TUI that's built off of K9's. It's kind of vibe coded. Um you know, there's we we try to be pretty open uh on the ecosystem page um for people who are building around Flux and we're not the first people to build a product around Flux. You know, Azure

has a Flux product, GitLab's got a Flux product, right? All of these people are selling Flux. Uh and and so we're just one of one of the folks uh in the game and we're just trying to do it well. Um so, you know, you can see here's our UI from control plane, capacitor, its own project, headlamp, its own project, lens, everything, right? And then people just building

on top of it. Uh does that hopefully add a little bit of context? I mean, the project's healthy. We're moving it forward. Uh we we've got I love anytime maintainers get employed, right? Like DeepD situation, she gets to work on Flux because she supports the project from from Microsoft's product line. She's full-time on that. Um so it hopefully that answers your question a little bit, but uh

yeah, thanks for it. It's a I I think it's important for us to be transparent about this, right? Like o open source is a people problem and we got to solve it the people way. >> All right. Thank you, Lee. Thank you, everyone. And uh for those who are staying in the room, my good friend Fab Skura back there, he will be up next in about 15

minutes on platform engineering and the power of immutable operating system. Thanks. >> Thanks everyone. >> I think the microphone is working. >> Yours is on. Yeah, I think we're good. >> Let me see. >> Hello. Hello. Thanks everyone for Welcome to the afternoon sessions of uh cloud native days. Uh we have Fabrizio here. He's going to give us all the information about the uh immutable operating systems.

So very exciting. He is a uh cubestronaut golden cube, right? Which is all is there fourerts now. >> 15 >> 15 certifications >> 15 certifications. Uh he knows his stuff. So um yeah, we'll hand it over to you and then uh we'll ask questions at >> Thank you. I hope that you can hear me. Thank you for being here. Okay. The problem is that probably this way

I will use my hand. I have the ear that is on the other side. So the topic today is very interesting because we will talk about operating system. Here we are cloud native but what's behind every deployment and everything that we do on the cloud and operating system and I work in uh IT since a lot of time. So I will give you my intro but today

we will not talk about Kubernetes specifically GitHubs and SRE but everything that was before and history is always nice because it train our mind to rediscover something that is not really obvious and it's like who is working on infrastructure when everything works all is fine but when something is not working the people that are working as DevOps engineers, sRes or whatever, they are always on the point

of being called on fire. So why platform engineering starts at node? Before talking about that, I want to give you my introduction. I am Fabitio Gura. I'm a chief engineer at Veritas Automata. We do it consulting and a lot of things. I am a CNCF ambassador and a golden cubeout. I work since 35 years in IT. I started when I was a child, but that it's really

a long time ago. It was the 70s and there were already computers and uh you can scan my QR code and it will be nice to get in touch because I always like to talk also if I like computers a lot. The agenda is the node layer. So whatever is running inside a server, how Linux is important for the evolution that we had in the last 30

years. What is an immutable operating system? Someone here knows what it is. Raise your hands. A lot of people compared to the normal thing. So the immutable operating system is a standard that we should consider a lot when we do things like Kubernetes because we don't want to put our hands on top of the AP or RPM packages. We want something that is there. We don't touch

it. We don't install software. We do that on top of Docker containers or other container images if we use other container engines or on top of schedulers like Kubernetes. In the past I used also Apache Mesos now it's deprecated but it's the same approach. Fascar Kylo and Talos are the candidates that I chose. There is already a very well-known word around Fedora Coros that was previously core

o by Google. I started to use coros in 2014 I think it was for installing Apache meos and that was by Google something that was really revolutionary because you couldn't install binaries at that point I did with systemdn spawn a container that was including python or go other tools that I needed I called it toolbox and in choros It was used for that kind of specific uh

operations where you need to use binaries that are not included inside the operating system node realities. Why single nodes? Because at the edge generally we have um computers that are embedded like Intel nooks or could be a risom. It could be ARM 64, AMD 64, sometimes this risk option, but we have a reality that is really not the cluster but just the nodes and still present the

operating system. If you work with something uh at the edge, you will probably know the YTO distribution that is something that is absolutely not immutable. It requires a package to be updated and every time that you have to introduce a change on top of the operating system, you are crossing fingers because you don't know what will be the consequence on the demo and I will try to

be very brief on the slides because there is a lot to say but I want you to see how we do the rolling upgrade the AB strategy that we can do use for these kind of upgrades. where you will have the system completely replaced in one round rebooting another partition. It doesn't require a lot of knowledge about this partition and similar and uh yeah let's start with

uh elapsing errors. So I started again when Redat Linux was probably earlier than five because I my first distribution was Slackware 2. We are talking about 1995. I think it was uh on a very old IBM ThinkPad SX 48625 and uh there were 54 floppy discs. The first 15 were for the operating system. The rest was for xfree 86 that at that time we had no Windows

manager. And when I managed after two nights and two days to install everything, I had that gray background that was proper of the X3 without anything with the cross and the right click was having the menu to run the terminal. So we are talking about a lot of time ago. I've been living in the war between Linux and Windows. Now we don't perceive it anymore because in

Windows we can run virtual machines with Linux and similar but for a lot of time at the beginning I was living where you had to choose if your career was on Unix operating systems or on Windows. Today when we have to get certifications, we can do Azure certifications for AI, foundry and whatever more and then we can do AWS certifications things like LPI certification, Linux professional institutes

real challenge when we have to choose a specific path for our career but at that time it was not that way. So again here I am more precise on the versions slackware 1.8 ate and again I'm recalling the hardware used why it mattered because it was freedom basically we were used to use something like Unix operating systems that were proprietary and the free software the GPL license

that came later and all the things related to the open source and Linux were really mentioning freedom I was able to install something and use something that was really free to be downloaded and used. So at that point I am putting here a lesson learned from uh serial communication. Today we use 5G networks and similar but who is coming from the past here at this conference. It's

always full of people that were working with wise terminals and TTY that were very slow. We still need that knowledge because we use buses for a lot of communication and sometimes they are fast, sometimes they are really slow and they require the knowledge that we have from the past. So again just to do some names about the operating systems that I've been using AIX from IBM HPA

UIX and then through 64 because I was really in love with an alpha power station that I had and when John Hole did the porting of the Linux kernel and Red Hat distribution to the processors I was really happy to use my deck station with Linux Red Hat Red Hat 5.2 to mark in the history something that is very important because I think that who have been

using Mandre or the parallel distributions of SUS that at that time we're using like today the RPM packages. They are marking a story point where we learned to use Unix better. And I remember that we were using the compilation from sewers. Today it's not used anymore. But in example in the certifications that I did for the Linux foundation on system administration for Linux there was a configure

make make install required to be known to compile from source. That is something that who have been working with that specific kind of technology will remember how much time we had to read the lines and the output of the compilers to solve the errors more than when we have a structures from Java or at back from Python. There are distributions that not all the people will probably

recall like caldera. It was used a lot and it was derived from skull Unix and SCO Unix were at that time I remember that I used a lot. Caldera is a transition for many customers in the late 90s. It was then Yubuntu. Today we use YUbuntu a lot. I have to use it not for preference because I am still in love with Fedora and the projects but

for the companies that are using generally Debian and Yubuntu as servers when I was working with Santos 2012 13 for let's say working on a standard or servers I never liked too much to work on Yubuntu and Debian for servers, but they became better and today I see that there is no real um difference between the distributions or among the whole distributions that we can have. The

challenge is more to choose what do we want to do with the operating system. And obviously when we talk about bare metal and cloud, we have roles and ways that people are working to produce something in their professional life and careers that are challenges but also walking path for becoming a better professional. and bare metal really when you when you talk to a person that is able

to create a cloud a local cloud computer or whatever technology in a server farm. It is really different from a cloud architect because he will know networks, he will know the layers of the oy for like having a lunch. It will be very easy for him to recall all the technologies that are related to the hardware. For storage, it will be the same. While a cloud architect

will be just dreaming about something that is composed like the virtual way and managing server farms with RS232 and whatever we were talking about the hardware will require us to consider that we are more near errors. Again recalling these kind of activities in server farms will let us think better about the immutability of the operating Evolution of technology is present since the '9s to today is something

that really is enormous especially for infrastructure as code. Let's think about how we did the imperative way until 2010 when we were writing bash scripts constantly. I'm still doing that forgive me but really we were doing that normally then a lot of infrastructure as code was out available and we had to adapt but people needs to remember that we are coming from something that is developing programming

setting up doing a setup and infrastructure escode can be very useful if we know how to use it and we know the But we have the responsibility to understand that if we don't know how to do it and it works, it's like the old style, if it works, don't touch it. And for immutability, this is important because you don't want to touch the operating system. If it

works, we have the open-source word that is mature, opensource, free software, a lot of different terminologies and also disciplines and strategies and philosophies. But let's say that here we are cloud native um topic. So the community that was born in 2016 did a lot for the products that we are using. Kubernetes the CNCF landscape products is full of Prometheus um open telemetry and there are,600 so we

will not mention all of them but basically this contribution is because of the community and the community is possible if we share together what we think. So when I'm talking to you, I'm just presenting something that it's not for being selfish and saying I am the good one presenting and you are listening. No, it's for sharing because maybe some of you never worked with the operating system

that are immutable and some others are better than what I am in this. So interchanging is the only way to get better in something. And this part related to the cloud transition, the part that we have been living with virtualizations that then came containerization. And finally today a lot of Kubernetes wherever at least for what I see because then there are companies that are still using other

systems like native containers or they use a lot of cloud services and they don't want to hear about Kubernetes. Kubernetes is not easy. It's complicated. Reason more for why under Kubernetes if the operating system is stable you don't have to think too much about that. Then the DevOps culture now it's been silently replaced by this platform engineering that is including a DevOps SRE and whatever more you

want to add but basically it helps to understand people that we have to be a cultural movement share ideas share tools provide help to people that need our services to contribute to the maturation of all the systems. He enabled this platform engineering. The open source will effectively become the enterprise innovation engine. We have been hearing talks about a lot of open-source ideas. Earlier there was the flu

CD operator and I can assure you that is something that for who remember we are waiting for a fluke CD operator since years hearing that it's coming out and there is a group interested in doing something like it was for many products is how we can really progress. modern practices are really important like the devos culture then dev secops sre many of us probably don't like the

names too much I personally was system administrator at the beginning when I started and when I was programming I was an analyst programmer today it's a mess there is a lot of cultural differentiation roles for people to be multifaceting so they can be in a role one day and then move out and go to another option in the next week. This is good because we are becoming

flexible. The operating system again is still there. We work on top of it. There are people looking for us on the upgrades and similar but we don't need to bother that much about that point unless we want and we can focus on other things. So we are able to dedicate our ideas and focus to other things that we like and also we can change we can change

from the adoption of CI/CD as a DevOps engineer or whatever SRE to being a QA why not I've been hearing Fisal this morning talking about Dora metrics measuring performance performance of the people that are working. It's not something that is invading our privacy. It's something that we could use as a feedback to have some points to improve. So all the part that is related to getting numbers

from performance and measures from whatever we do is not something to lose the job to get fired but it's something to get better and people are generally scared about reviews and similar but again in platform engineering today we are pointing yes to immutable operating system but also to all these kind of metrics and strategies that are giving us more time. In example, we don't need checkpoints or

a lot of change controls ticket to discuss among the team because we have tools that can do that continuous deployment so-called what is the future trend? We cannot not mention AI because it's present. It's probably overused especially in this context when we are talking about Linux. I think that we don't need really AI for interacting among us. We could need AI to find a kernel parameter because

there are thousands and we don't remember how to put on the command line something that is related to the VGA controller and instead of going to the documentation we could use some faster way but it's not really different from the adoption of what we did when it was Stack Overflow or Google or just searching the internet. The real point is that there are many people that are

using AI to develop and we are still not ready for that. At least from my personal perspective and experience. The edge device is something that initially we were calling IoT. It's running something now that is giving the topic of a smart device. not anymore something that is like a sensor sending information but it's taking decisions. So in example it's a peer node of a blockchain private blockchain

not necessarily something that is related to cryptocurrencies but it could be related to healthcare or life sciences taking decisions on top of measurement from a temperature but adapting all the ecosystem of a freezer in example to provide the right ecosystem for medicines. That is something that will require stability of the software that we run. Another reason for not bother that much about the operating system especially because

if you think about edge computing but also cloud computing. How many times do you have to change the operating system? Generally for security reasons. there is something that is exploiting or other problems related to instability performance. So you need to upgrade the packages. But that upgrade or the upgrades that you have to do could be something that is really putting all the things that you do with

edge devices, cloud devices at risk. And if we put again the AI here that is not really flexible as we are to control something that's in many cases it's unpredictable because we did an upgrade try to ask to judge GPT about an instability or whatever generative AI not just GPT but about an instability that is coming from a behavior of the machine you an answer then you

ask differently. I did an upgrade of this and you will get another answer. It will say, "Oh, in that case, thank for pointing me on that, but it will be for this reason." And then you will add more details until you get to a research that initially produced wrong or incomplete answers. It's like humans. You cannot have everything immediately. You have to progress. So the immutable revolution

is a progress from 2013 I mentioned earlier core OS this is the here there is Kelsey high tower that before of kubernetes he worked on core os and he was the responsible for the security of this operating system. They invented at that time a system for certificates authority that was so complicated that it took two days the first time to set up on a cluster of three

nodes. It was called tectonic in 2014. We have the idea of Red Hat and Fedora that started with the O3. OS3 is exactly where we focus today about operating system and the way that you can do updates. In the demo, there will be no OS3 because I consider that is risky. I will explain why later because now I want to go to the timeline. In 2020, Coro

was acquired by Red Hatut and given to Fedora. At that point, the Google activities that done on top of operating system were acquired by the Fedora. It's still that way they are using the O3 way with enhancements. It's not really the same as it was earlier for Fedora, but basically the PNCF did a fork of a project or let's say a team NCF they decided to create

the fork of this coro and the operating system that is the material of the demo today that is flat car. Flatkar is a project that is built from cor. It's different. It doesn't use O3. It uses AB strategy. And there are others that are evolving. We have a very good project that is also working on Raspberry Pies 4 that is Chyros and that is a nice a

little bit different uh way to do the operating system. SUS elemental or SUS Micros that is a little bit less immut present are other operating systems that today in 2026 coming from two years ago they are pointing to the real immutability of the operating system and this is again an improvement in terms of security efficiency and scalability. If a node is replaceable and notable, you can how

things when you have your cluster of Kubernetes or whatever you are running on a cloud provider and one node is going down and it's replaced by an autoscaling group and when it comes out you have a lot of manual activities very bad but it's quite diffused. So if you don't have that problem, you are quite lucky because I have seen that there are many companies that are

still having the problem of the automation of the bootstrap of nodes. So if a node is replaceable because I don't have to do activities like configuring the network MTU, fantastic. It's not repairing. What is flatar? So I talked about this initial fork of core OS. It's an operating system here. It states that prevents unauthorized modifications. It depends on how you use it because if you are the

manager or the administrator of an operating system and you start tweaking and building your binaries to run them there, you can achieve the result of having it. But it's not the right philosophy. implementing AB transactional updates. The word transactional is very important because it's really a transaction. I can commit, see the result and eventually roll back. It's never something that is not touchable. It's also a complete

replacement in the case of AB. And when I was talking about the O3, think about layers on the operating system with of an RPM tree of dependencies and you replace a layer. What will happen when you are at layer five and you want to go back to layer four because something didn't work a lot of dependency held will happen while in AB case it's not that way

you will not be able to roll back five versions earlier if you don't replace completely the operating system in some cases with flat card you can not really what you should do but you can also go there because it's like a hashmap of versions. We will see how we can select the versions for the initial configuration. The big advantage started to use when using core OS and

also the cloud computes was the cloud config YAML file. If you use it correctly with that file, you can do everything. This partitioning, LVM, volume groups, SSH keys, users, and many other things including bride files and adding services and additional things that you will need at boot and later. This is declarative approach. It's not imperative. I don't enter the operating system in SSH and perform all the

changes. It's already available and that's very powerful. We will see how flatar and chyros they have a base configuration file. The flatar one is called initian and it has initially a yaml format that is then converted in JSON but it's just a command line automation. So you don't have to write in JSON it's better YAML and we will see that it gives us that ability the same

one that we have with the cloud config. So if we want to install a service we just do it in the demo I will show you how K3S installation is done from the init part. So the node is ready to be used and you don't have to configure it after it has been replaced in some cases. And the atomic updates are part of this strategy. Here I

have four subjects. Maybe there will be more in the future. Possibly there are others. In the example, SUS elemental is not here. Micros either. But the four use cases that I think everyone should look at for the current days. One is Flaskar. The immutability is total and the update method is AB and it focuses on containers Kubernetes. So you have an a ready to go environment. It's

not that invasive. So you can also not install Kubernetes and use it immutable operating system and it's really good because it's using ignition and it's also having a provider for Terraform. So it's very very well maintained by the CNCF. It's CNCF. It's still not graduated but we are working on that. Fedora Coro is from work flat is coming. It's still using o3 as I said earlier enhanced

it's not really those layers only there are mapping and similar things it's having a transactional update method meaning that there is a history on top of the path you can walk back and and forward the focus is containers not really kubernetes because they are also using cryo as base um container engine. So it's optimized for who is using podman more than docker or containerd px and in

are the way to configure coros so it's very similar basically it's not changed that much talos uh we have again totally mutability it's pure kubernetes and it's api driven I haven't used this demo with talos because I think it's quite complicated and also because for me for showing you the things but it's pureetics and I wanted you that that ls/ is still where we search for many

other things rather than the cube config and similar kyros it has AB releases and overthe-air updates it means that you can have a repository where you can And it will be very similar if you use to tools like Balina or Mander from Northern Tech. We are doing something that is over the air at the edge. It's available for edge devices and the focus is also for ISO

images and Kubernetes custom resource definitions. Let's go to the use cases. If we think about the resource constraint hardware, if you have been using brancher in kubernetes or whatever solution that is adopted and promoted by sus this is the typical use case they the platform that is called sus edge you can download it it's for intel nukes it is very good because if you install all the

operating system requirements that they recommend they don't force they just documents it will become an immutable base for the operating system. Chyros provides this overthe-air updates and that means that generally if it's not a big change to the operating system you will not there is no way to change a partition live generally so kyros has also that specific ability of running specific updates just reboot for edge

computing also available I've been using it for Intel nukes for 64 I haven't tried on ARM 64 um the native integration for containers is ob obviously something that is very important because if we have everything up and running you will see how I can just boot the machine, virtual machine in this case using Kimu and I already have ready to go a K3S just available without having

to install Kubernetes that hard way. Well, it's an advantage if I'm using the cloud native technologies. It's secured in generally not just on Kubernetes like this but all these operating systems are having the same authentication method authorization and the access that is pro the Linux operating system kernel name spacing whatever related to the security for the users and groups and the file system the attri the attributes

and all of that for Talos that is API that is also very well the management is done without shell access. I will have to enter the shell for flatar and chyros. Chyros also has a nice installation that we will not see because I will do the interactive but it's by QR codes. So they did also by the phone this kind of uh process for installing the operating

system and everything is supportable with the scalable automation using whatever is your tool the favorite one that you can now I go subject that in the demo sorry because the voice going up and down but I hope that you will hear me without hiccup flat incubator. It's from October 2024. Uh it it is still an incubation project and it's becoming mature but it's not really wide adopted.

So I suggest that when I will share the repository you check it out when you are at home and you run all the commands that we will run is just downloading something like mages running chemo. So you have to uh have a Linux machine with KVM installed. It's not really hard. We are at the Linux conference. So probably many people here will use KVM and it's really

nice to see how we can start experimenting. The advantages of the project flat car and also the chyros one are that you can integrate with all the good practices. So the GitHubs team can use all the tools that are coming like Fluks or whatever other things that you can use for producing the installation inside Kubernetes and in for your uh software whatever it could be on top

of Docker. Chyros is a they are working very hard. I met them containers in the last cube con in Atlanta and they are doing really a great job for moving to a lot of they are still working on Raspberry 5 because the bootload there is different but all the other models are supported and if you have a Raspberry Pi 3 that is having in example one gigabyte

of RAM you will find kyros very useful because it's it's not required lot of resources for running correctly. The differentiation here that I wrote when I wrote this presentation is that flatar is effectively uh focusing on maturity and core o compatibility. Also I would taking a look at core o in the last two months and they are really different now. So the fork is becoming a personal

different approach for the operating Here numbers it's important to see that when we have flatkar talos chyros fedora cororo sus elemental and the traditional approach the traditional one is very very long here is written in Spanish but we are in Los Angeles so for the it means how much time you need to provision and the blue histogram in traditional is enormous. So they are also fast. You

will see how a virtual machine will boot in a matter of 15 seconds. the actualizations and updates that you have security reason or whatever on flat cartal kyros fedora coro and soua elemental the green bar they are a lot while on traditional operating system if you use yubuntu or fedora you know appear and you have to confirm okay let's do the updates you could have kernel disabled

but times is compromising the driver in media that I have I get crazy once out of five times also if I disable the updates of the kernel. So is an point of failure. the the security incidents in the traditional are intended to be very high on Fedora Coros compared to the other uh operating system the immutable uh guys is also a little bit higher and the downtime

each year it's higher for the traditional operating system. So it matters to adopt something like a practical implementation of all these immutable operating system. Here I decided to put something like just a brief command line installation that you can use. It's not really this way. You will see in the demo. We will take more time. But basically they are really easy to get. It's like getting an

ISO image and having it up and running with a shell script or in case of chyros you can build a docker container image and on top of that create your own operating system with a targed or whatever you want to to do. Talos is applying configuration on top of a node IP and creating the installation remotely with the config yl. Finally, 76% of organizations use Linux in

embedded devices including where I work. a lot of embedded devices that are out there out of the cloud and they will require always special abilities on top of the operating system. All of this to say that think seriously to promote the adoption of the immutable operating system because you will be free to do other things including platform engineering learning about the new tools CI/CD removing all of

those problems that we have with the teams for checkpoints and change control tickets. We will have more time. Now let's go for the fun part. I should have enough time to show you both of the things. Uh yeah, I will leave the QR code later. I will try to use both hands now. I hope that you will hear me because now it should be better. This is

the steps domarkdown that you have in the repository that is here. I will put the QR code for more time but let's go for the demo now that is better because you will see something. So basically here first we downloaded the flatar script to produce the chemo image. the image itself that is the one of the operating system some change of permission. I decided to download an

old and a stable one to show you the AB >> Yeah, sure. The point is that I would like to use the other the repository but use these comments basically to the executable that we will see in the terminal. So the terminal will be okay a little bit more visible. what I have flat config ignition I will show you how it's done. the image that will be

run for the operating system a script that is bundled and it's available from the repository of flatkar and it's part of those wget commands then the update gzit and a specific extension that is used because we are using kimu and not the real node version. So here again I was grabbing an image with um version 4230 and we will promote it to 4459. This is a base

config ignition. Uh if you don't see it I will explain basically but is a path with the name of the operating system flat car demo. A more complicated one will include my authorized key to have the SSH access like you do in cube config. Um it's generally produced in YAML but I translated it in JSON with a comment and uh it's ready to go. The wrencher K3S

configuration will give me the ability to have the cube config written with the right permissions to execute the command cubectl. And here the K3S installed service that will be basically translated a unit of type oneshot a systemd unit that will download the normal way. So if you have installed K3S you will recognize this line that is get K3S.io and that will produce a specific system service that

will be started by the installation process. Same for SSHD by default it's not activated we want to activate it. So this is the normal command line that you will use for running the operating system. But I will use a little bit more complicated one. This one because I want to use the bridge network for SSH and not the normal not. So this should be the line that

will open up our virtual machine with the window. Let's go. You see that immediately flat card default it was partition A partition B we just do partition A for the moment it's the default one when we will reboot it will have partition B now the login is skipped because it's like OS I can become root with these s and I will be inside and totally normal It

is installing K3s the K3S service. So after some time to have it completed we should see that the service of K3S is active and running. That means cube or cube cut get nodes. I have the notes here with the version 124 the one that Ieded without taking that much time this point we have the operating system up and running let's see what I have in the okay

there are some comments that I put here if you check out the repo this comment what will give you the root device of the USR partition so if we run this command well it will be better to um enter in SSH because otherwise we will have to uh write here. This is the not not not having the the clipboard shelves. So I have the IP that is

the one that I use generally for my bridge network the 100 135 user core. Let's do this 135. We have to remove the previous guy we want to enter and we are inside our So here we can use this comment in example to see what what is the disk refer to the root partition BDA3 is visible with on the projector will not very perfect to see a

lot of Uh yeah, VDA. You see that here have lot of partitions including the USR A and the USRB. Now the active one is the A1. You can have the B1 which will be the prioritize. uh we can effectively have uh the The other part when you boot the original one. Now we go for the upgrade. I downloaded for the upgrade the OM extension and also the

other flat car production update. So it's just a matter of sending them via SCP Again, expanding the two files the flat. need core home folder of the user. I have the key that was added in the ignition file. So it's not requiring passwords. And with this we have the files. Now we can go back to the readmi file. Take this. As you can see I already uh

burned the value of 4459 because it was the one that I downloaded. I just need to run this comment and watch with the journal CTL the result. So start with the commands. This connection restorative IP should not be a problem otherwise we will have to reboot. This is because of the network of the building here because I was doing it three times and it was working. So

it's just matter of seeing if we can SSH update successfully applied you see here. So this means that the upgrade uh was successful. It's saying that it will force reboot in 5 minutes. We will do the reboot manually. So we exit here and here and we do the reboot from this part. Now we you can imagine that the default will go to the new version of the

operating system and when we will do cube cutle get nodes for kubernetes that will be still there untouched. So we updated the operating system. Uh it's very very small and this is not something that I can extend but I can do another thing. Rerun the SSH The first thing that I can do is showing you that this is the new version and the operating system is using

nowition B and the cubectl get nodes is showing still Kubernetes up and running meaning that we did an upgrade complete one of the whole operating system but Kubernetes is still there working. I have no workload. I should have been creating a pod with the engine X to show you that it's still the same. you can try that at because there is no also age 9 minutes 7

seconds. That was when we installed Kubernetes. This is an example of how we can do that. We still have two minutes. I don't know if I have more minutes to show you the part related to Chyros. You tell me >> two minutes. So I will not be able to show you chyros but I will be the process building a chyros. I have in my folder here that

you can rep the chyros image that I downloaded the config YAML file for is very very simple. You see it here that I just use the username and password. It would be an interactive installation and if you quick co installation on top of that disc and then again like I did for the kimu version this will be the common line that will boot kyros once it's up

look at how easy will be to do the upgrade you will list the releases it's a textbased list. You will s select one with the OCI endpoint that is produced in the list. You will run the upgrade with this comment. Once the upgrade is completed, it would require for this case a reboot and you will be able to see the same result in Kyios just following this

readme file or step me steps.m markdown. Coming back to the final part of the slideshow. As promised here, take the QR code for the repository. Bribe me if you liked the results. I hope that you will become immutable and I hope that you enjoyed and we have space for two questions maybe. Uh, I mean, we're at time. Uh, I know you'll be in the hallway. Um, just

so that if anyone has questions, you you'll be around still for a little bit, right? >> Yeah. So, let's give it up for talk. >> Yeah. Thank you very much. testing. One, two, three. Sort of works. >> Testing. One, two, three. All right. Okay. Oops. All right, we'll guess I guess we'll get started. Robert uh stands in the way of our dinner. Last call last talk here

in uh ballroom B. Uh Robert uh who's been working with databases since uh 1983 I guess uh well before most of us started writing code. um he's the CEO of Al Alinity and um leading enterprise provider of um Click House and a regular at CubeCon. This is your first uh talk at scale. Um so today he's going to be talking about Kubernetes data platform and if anyone

has earned the right um to tell the story, I think it's him. So please welcome Robert. Yeah, thank you Fussell. And yeah, massive shout out to Scale. Uh this is the first time I've been at the conference. I've wanted to come here for years. So uh it's it's really great to be here and it's great that so many people work to put on such a great conference.

So yeah, I'm going to be talking about uh what's in a Kubernetes data platform and uh by showing you the lessons we learned in building one. Um first of all, how many people in this room actually run Kubernetes? Okay, this is good. About half um you guys. So if you run Kubernetes, you know Kubernetes has a lot of YAML and you may be thinking this is going

to be a lot of YAML slides. I promise you there's just one. What I'm going to focus on is more lessons, things that are in some cases somewhat counterintuitive um that we learned about running uh using uh Kubernetes as a data platform. Uh how do we know about this? Well, the my company Alinity has been working with Kubernetes and uh databases for um now over seven years.

Um we're a vendor for Clickhouse. How many people have heard of Click House? Okay, that's the usual number. Oh, a little higher than usual. It's like close to half. Great. It's a it's a really wonderful database. If you haven't heard about it, you should find out about it. It's really good for analytics. uh we won't talk much about it but I'll use it an example because um

we built a cloud platform we do enterprise support for uh uh for um for click house however people want to run it turns out a lot of our users uh run it at Kubernetes in some cases at very very large scale and then we run our own cloud so in that case about half our customers are running on that uh we also wrote uh one of the

earliest uh in fact to my knowledge the first operator Kubernetes operator that was written uh for an analytic database uh back in 2020. Uh if you don't know what an operator is to have no fear, you'll know about it in just a few minutes. And um yeah, so I I run the company, but I've as Fossil said, I've been working with databases since the early 80s and

uh programming since the early 70s. So I have a kind of an interest in the in the topic. So um I want to just start out by showing you what our uh platform looks like. It's called Alenity.cloud. cloud and um the you know there's a bunch of different ways you can build this uh you know we have a we've been focused on building a commercial platform but

people build these inhouse so you may not may or may not have a fancy UI for that this is the UI we built uh so for example this is a an environment I'll talk about those in a second but it's got three click house clusters running in it and you can go in and click things and scale things up scale things down make new clusters take away

old clusters kick off backups and all that kind of stuff. So, this is something if you of course if you use Amazon or Google, any of the database services, you're pretty familiar with this kind of interface. I think we just Hey, hang on. Uh, I think we just accidentally flipped over a slide. Um, oh, this is interesting. A slide disappeared. So, I will that is the first

time this has ever happened to me. I think I accidentally deleted a slide. But let me tell you what uh what was on that slide. It just was a list a punch list of all the things that data platforms do. Databases are kind of a pain in the butt to run if uh because they require they there's specialized knowledge to uh required to do it. Um so

for example um how to set things up correctly so that you have a best practices set up uh where databases for example are nicely ordered so that they all fall on different hosts and if one of the hosts go down you don't lose uh all your all your data. So there's things like that. There's also things that we do that are very boring like backups which is

something that are as I'll say later in this talk are absolutely essential to get right but nobody really likes to do them. So uh so that's also an important thing uh that you do and then um we also do things which are complex and uh require very specialized automation and a good example of this is upgrading a database. So how many people in this room actually run

databases. So okay so are you a lot of you are probably DBAs and for for a DBA the time when you upgrade a database to do a major version release is a time when you feel real fear because this is when some of the major disasters happen that you upgrade something you either screw up the up upgrade process or something bad goes wrong and then you have

to get back to where you were before. Being able to automate that so that it can run for example using a roundroin upgrade so that you take uh database replicas off one at a time and make that happen transparently to running applications. That is a non-trivial problem and database platforms good ones handle that for you automatically. So those are just some of the functions. We're basically trying

to automate all the useful things that you need to do to run databases and run them well and allow your developers to concentrate on building cool apps. So um one more thing is when we use Kubernetes as a data platform one of the ways to think about it this happens to be the way we do it is that we think about it as organizing it into environments.

So for example in our cloud you could have an environment that runs in our account for a user and that's on Amazon. We could have another environment that's in Google um that runs in our account and then we could have environments that are actually wholly within a user's uh own uh Kubernetes account in their VPC and we run there. In each case, we stand up a a

Kubernetes cluster and or a Kubernetes cluster and we run the databases inside it. So you can think of it as kind of a bin where we can uh our platform that is portable across all these environments that allows us to operate databases efficiently. We'll get back to this question of environments and particularly the question of tenency in a little bit. But first let's talk about Kubernetes. So

for those of you that don't run Kubernetes and maybe aren't deeply involved in it, what Kubernetes does is it allows you to orchestrate containerbased applications. And because this is a Linux convers uh conference, I want to say that you're basically orchestrating Linux processes which are talking to storage using networks. uh underneath Kubernetes we have containers which are just built on Linux constructs like Croups uh overlay FS

namespaces stuff like that. So how do we do it? Well, the notion is that you've got something that you, you know, an application that you want to run, for example, a database with some attached block storage. And what Kubernetes does, the way that it operates is that we will model that application using resources or what are called resources. And these are the names of some Kubernetes resources.

I'll talk to you in a minute about what they do, but you express those in a YAML file. sort of the definition of these four resources. You hand that over to Kubernetes and what Kubernetes is going to do is it's going to take those definitions and it's going to make the real world resources sort of uh so that you know arrange them so that you actually get

a process running on a Linux host with attached block storage and that this happens reliably and not just that if something changes in that environment for example the host dies Kubernetes is smart enough to recognize that something has changed and to automatically bring that uh process up somewhere else. And it's all based on this definition which gives Kubernetes a a a uh a picture of what you

want the world to be. And then Kubernetes takes care of arranging the hardware so that it's true. This process is called reconciliation. This is a very very powerful way of thinking about um of thinking about how to manage applications. And of course, this is a great example of infrastructure as code. Now, it gets hard when you actually have real databases. This is a simple one. So, I'm

taking Click House as an example, but um what you have here, the way that it's organized is we use what's called a shared nothing architecture, which means that the database cluster consists of nodes. These are processes running on different hosts. Each of those hosts has attached block storage. Those uh those processes communicate with each other so that for example if you update a table on one of

the hosts whatever part you create or parts automatically replicate to others. We have additional services we end on uh depend on there's something called click houseke keeper that is um used to maintain consensus in this cluster remember for so that we know that if a part is added to one uh node in the cluster so keeper keeps a list so that all the other nodes can find

out about that part and go fetch it and then on top of that um click house like many databases today in fact almost all of them um uses object storage for long-term term storage of many different types of data. So uh so when you're building these systems you have to account for the fact that you'll be accessing object storage. So this is the this is the database

and when we started seven years ago we had a problem of okay how do we actually model this in a in Kubernetes because the the Kubernetes resources are out there but you can combine them in different ways and um so that was the first design problem that we had to deal with and this was a long time ago and nowadays this doesn't seem like a big deal

but at the time you know when you don't when you're doing something the first time you agonize about it so here's the click house server we're just taking a piece of this there's the block storage. How is it that we actually model this? And what we did was we basically adopted that picture you saw a second ago. There's four basic Kubernetes resources that we use. We use

something called a pod that corresponds to a process running in a container. That's the server. Um the pod is managed by something called a stateful set. That is a resource that knows it is managing a process, a pod specifically. It also knows that this pod has storage and so it has a way of expressing um what kind of storage you have uh which turns into something called

a persistent volume claim that is a request. For example, if you want a 100 gigs of storage on Amazon EBS, you have a persistent volume claim that that requests that and then may even have some quality of service. um notations to say how fast the bandwidth is and then you have something called a persistent volume that is a resource that represents the actual allocated storage. So this

is a modeling that we came up with and um the reason we chose this the stateful set is interesting because it does a couple things. The most important thing though is that it manages storage. So for example, if your pod gets killed or the stateful set gets restarted or deleted or changed, um it is smart enough to go when it brings the pod back up to make

sure the process gets the storage properly mounted and can find it again. With databases, if the storage goes away, that is that's a bad thing. You can do other things like you can increase the size of the storage. Very common uh kind of thing. So this is totally simple. If you're using Kubernetes, you probably think, "Ah, this is child's play." Um, the problem is that, let me

see how far we get that. The problem is that even for this simple case, it's a little bit more complicated than that. So, I showed that database with um with a couple of nodes. Um, in real databases, one of the problems you run into is those nodes are not always the same. So, you may have cases where and some of these cl some database clusters are quite

large. They may be we run commonly run clusters that are 20 nodes or more. We have customers that run up to a couple hundred and um in a single cluster. These guys may be running different different versions of software. These guys uh may have different run on uh we we may want to allocate them on different hosts. Um so there are different um we may want to

put them in different availability zones. So as a result these guys are not carbon copies of each other. they often vary in in different and uh sometimes unpredictable ways. So what we want to do is is be able to represent that. And so when we designed the original click house installation resource which is uh which I'll talk about in a second um it's a new resource that

bar basically combines has a stateful set for each of the databases that we're man or database servers that we're managing and then it it manages them as a group. But you can actually have them you can actually uh tweak the definitions so that for example you can force them to be in different availability zones or you can have just one of the one of the nodes be

a different version from the others. So this was an early design decision that we made and this Clickhouse installation resource is now what we call a custom resource in Kubernetes. And it turns out that one of the things that um that made ClickHouse very very powerful was the emergence of a mechanism for for creating new types of resources and adding behavior to the system. So when you're

looking at at implementing this, when people are are building things in these data platforms, there's a couple ways they go about it. One is to use something called Helm. Uh how many people here have used Helm? It's kind of like a package package manager. It's very it's kind of attractive um because if you've ever used apt or um yum or something like I love apt uh it's

you know you just say apt blah and or apt install blah and it it comes down helm is like that for kubernetes so it's a package manager and what it will do is it allows you to use templates to create patterns of of these resources but the other pattern that uh the other um and So we'll talk about that but that is actually not a good way

to to manage databases for a reason we'll show in a second. The other thing that Kubernetes did about this time that we were going through this uh design problem was they introduced something called an operator and an operator gives you a way of first of all creating new types of resources. Ours is called the click house installation and the um and then the ability to manage them.

So to add logic to the Kubernetes cluster which will manage changes in these resources and do it in a systematic and um and reliable way and the way that it works is through the following process. So you in the operator you define the syntax that you permit in your resource definition and um you also have the operator itself which runs in a container. Everything in Kubernet Kubernetes

is a container. You install the operator, you install the definition, and then you can make these ClickHouse installation resources. You just type up your YAML, you hand them over to uh to Kubernetes. There's a program called CubeCuddle, which you just use to to blast them up. And what Kubernetes will do is it will say, "Oh, I see this this resource. I'm going to hand it over to

the operator that's responsible for managing it." And the operator gets that resource and it goes through a process of reconciliation where it looks at what's going on. What are the resources that Kubernetes has defined? What is it this user is asking for with this click house resource? I'm going to now make the the Kubernetes resources look the same way. Those resources then have their own they don't

have operators. what they have what are called controllers that um that manage those and and that's what ends up making the physical in infrastructure work correctly. They also do reconciliation. So it's kind of like a turtles all the way down kind of structure. This is a very very powerful mechanism and the reason is that the operators oh well let's first actually I'll get to the reasons. I've

got six of them. Uh let me just talk about what uh uh the resource definition look like. This is the one piece of uh YAML I promised you would see. Um this is a def definition of a click house installation resource which our operator manages. And um and it the idea is you can see that it it it's pretty it's pretty abstract. It's saying that hey well

first of all here's the name of my um my resource. It's going to create something a cluster called Arggo CD. Um it's going to have some uh it's going to have one shard. It's going to have two replicas. And just by those two simple lines, I'm saying, "Hey, we're going to have two servers in this system." But it could also be a hundred. We have customers that

use this that run very very large uh systems. We can tell where our uh consensus is. The uh the the zookeeper section. We have additional sections not shown here, which specify, hey, what version of the container do we want to have? That's the pod definition. What uh you know, maybe where do we want to have it uh located? what node what kind of node should it run

on and then the volume claim template which which lays out our storage. There are more things that you can lay out but the basic idea is you have a pretty abstract definition which lays this stuff out and the operator is going to turn this into to more resource definitions so that your server your cluster comes up. So the operators do some really powerful things that are necessary.

I mentioned that one of the things we're trying to do is handle these complex operations that that humans either don't like to do or can't do very effectively themselves without a lot of work. And I'll just pick one example out of these uh rolling upgrade. So we have in um in analytic systems it's very common to have multiple replicas for uh for the data. In fact just

just about every production system does this. Um and so the idea is that when when you want to change the version of the database, you then have a process that you go through to uh systematically upgrade those uh databases one by one. Uh so rolling upgrade probably how many people in this have not heard of rolling upgrade here? Is there anybody? How many people have heard of

it? Okay, it should be a lot of folks. Well, so what's the big deal here? This just doesn't sound like a very hard problem. Well, there's actually a lot of complexity hidden in this. So, for example, imagine you have 24 nodes. What's the do you want to do you want to run them, you know, those upgrades one after the other? Well, every time you upgrade a database

node, it would be nice if you're quite polite if you waited for the existing queries to finish. So, we'll give them maybe a five minute timeout, you know, for for anything any ongoing work. That's part of making this transparent and then we'll drain out the uh we'll basically block it from any uh further queries. So if we were going to um you know do an upgrade in

sequence that would be at the very least you know in the worst case 24 by five minutes that's two hours. You don't want to be spending that long. So what we actually do is we will run kind of like a canary upgrade. So if we've got a bunch of them, we'll run one and then if that one succeeds, the process will continue. And if we have a

sharded system, most systems side would be sharded. We'll then look at all the shards and in parallel upgrade their replicas one at a time. So what could have been two hours is now a process which will probably complete in 15 to 20 minutes. And um there's more to it than that. There's just a bunch of ways you can lose data during this process. Those are not good.

So these are all things that the operator handles. And so coming back to Helm charts um so for example Bitnami uh before they I guess they got bought out and now they're a lot of their open source stuff has been closed up but they would offer Helm charts to do this. Uh the problem is that Helm just doesn't have the ability to to add that kind of

logic. So we strongly recommend operators and I think that's that should be your default if you're building one of these systems. So um that's the basics but there's actually a lot more after that fortunately uh this is the most complicated part. So another interesting question is if you're building a data service what's your tenency model and there's a couple different ways that this can work. Um one

is and it there's really two choices here. Maybe somebody else can think of something else but these are the two uh big ones that we see. And I talked about these environments, the notion that an environment is a Kubernetes cluster and that's where all your databases live. One way to do it is just if you have a bunch of tenants, in our case, they're actually, you know,

separate customers, but in other cases, they might be separate business units if you're just running it inhouse. You could give each of them their own Kubernetes cluster and run their databases there. The other thing is you could have just one Kubernetes cluster and just try and stuff as many of your customers in that one cluster as possible. So this is a really basic choice and the thing

is if you do one it's kind of hard to do the other um for reasons that you'll see in a second. So for example if we have um dedicated environments they do have some really strong advantages particularly for analytic systems. Uh, one is that the fact that tenants are fully separated means that they're just running in their own clusters. Those are networks that cannot see each other.

If you screw up something in one, you know, you know, for one tenant, it's not going to affect anybody else. They're on completely different resources. It's also optim well optimized for analytic systems because they tend to be large. So it's very common for even medium-siz analytic systems to have 50 gigs of storage uh to be to have many replicas potentially have a number of shards. Another really

key thing is it enables something like uh called uh own cloud which I'll get in just a minute. Um there are some disadvantages and and you can read so I won't go through all of them but just to give an example um if you are building a Kubernetes cluster it is very slow how many people here have used EKS on Amazon so okay not too many people

well for those of you that the 98% of you that haven't when you allocate a um a Kubernetes cluster on on EKS which is the the O manage uh uh service. It's the next best thing to watching paint dry. It it's just very slow. And then there are processes like getting certificates um installing those. So this model is not well optimized if you want to bring tenants

on quickly because you can be talking the last time I did a big one it was it took an hour to come up fully. That was a little unusual. Usually it's about 25 minutes. Um but yeah, so there's there's some definitely some some big disadvantages to this model. there's you have the same tradeoffs with these shared environments. Kubernetes is just not a good place to do true

multi-tenency if you can't tr if the tenants are potentially malicious. So if they're coming from the outside world, there is some pretty heavy engineering that is necessary to split them up. Uh for example, get them running on separate overlay networks. Um make sure that they can't uh eat each other's resources. Um there's also just generally being able to make sure that when a connection comes into this

that if it's for tenant A, it doesn't accidentally go to tenant B. believe me because I have experienced it. That is the worst bug that you can have is like you know that somebody's coming in and they're seeing somebody else's data. That's when you shut down the system. You say, "Hey, there's something going on. We we think you should come back when we've got it fixed." So

the engineering to do that is is difficult. But there's some big advantages and this is why somebody like Snowflake does uh multi-tenency within the same network environment. This is why Clickhouse Incorporated or our our competitors have taken this model. It's well adapted for small tenants because the infrastructure is already up. If you bring on a new tenant, you can turn them on almost instantly. Um and it

also if you uh you know take the time to do the engineering it allows better cost optimization through things like binacking for example or overcommit of resources. So these are um this is a really um this is a really important choice and I said that you know if you pick one it's sort of hard to move to the other that is a a really key point um

that if you have things divided like this if you want to move people to a different Kubernetes cluster that's actually quite problematic. Um similarly with the dedicated model um actually that's a little bit simpler I guess but um it the migration between these two environments tends to be hard. We picked for example the the separated uh uh Kubernetes clusters it's kind of hard to go over to

the other model and vice versa. There is one more point which is when you're building a a data platform particularly as a commercial vendor like us you want to think about how do you let customers have full control of the data. So one of the big I snowflake is there anybody in this room who uses snowflake? Okay not too many. And Snowflake is a really really great

creation. Awesome technology. But one of the problems with Snowflake is that if you put your data in there, it's in Snowflake's account and if you stop paying the bills, your data disappears. And there's, you know, specific in the terms of service, you'll you'll see a section about how uh your stuff is gone if you don't pay. Um well most people that have large amounts of data say

you know 50 terabytes of data it's very difficult to move it out of an environment like Snowflake. So increasingly for reasons having to do with self-preservation um compliance uh data sovereignty and just sort of generally being able to have more control of things. Um people want to have the data and ideally the software that's operating on that data in their own cloud accounts because then they can't

be cut off by a vendor. This is called bring your own cloud and this model is used by data bricks. Um it is a very um powerful model but um but it's something that you need to think about very carefully. The dedicated model that we use actually is well adapted to this because for individual customers what we do is they just give us access to their account

and then we can go create these environments for them inside their VPCs. The engineering is um somewhat complex for that. But the advantage of this is it then gives them complete control. They get they have the compute because all the stuff we run is open source and they have the data in the you excuse me in the user environment which is what they want. So that's something

that that was an architectural choice we made about four years ago because we're getting a lot of pull from customers. But I I think it's the way uh people are going for large systems. Um, all right. Now we get to some um shorter items. So if you're running Kubernetes, it used to be you had to run it yourself. That ain't so now. So almost every cloud um

that you use except for maybe HNER. That's the only one that we use that uh doesn't have have it, but every other cloud has a managed Kubernetes operator uh uh offering. These things are great. Uh, I talk about Amazon EKS. That's the one I use most commonly, but we're up on uh Amazon, uh, Google, uh, Azure. I think we have somebody running Lode. I don't know. Uh,

these are other Digital Ocean and Civo are two others. Um, these, if they're there, you should use them. Here's some of the good reasons why why you'd want to use them. Um, well, they're integrated with that vendor's cloud, you know, so they're going to work. um they do just Kubernetes is a it's kind of like our databases they're it's also a pain in the butt to ma

to uh to run Kubernetes you hear that it is hard the actual hard part is running it and connecting it to the uh to the cloud environment the managed Kubernetes take care of that they take care of upgrade um another thing security they're surprisingly cheap the cost to run EKS is about 10 cents an hour that's you know you have to pay for all the other resources

that Kubernetes runs done. But the actual overhead of the management is tiny uh because the vendors want you to spend a bunch make it as easy as possible for you to fire up a bunch of other nodes. There are some bad things about it. Uh the biggest thing we've run into is that you're forced to upgrade often when you don't want to. So Amazon will keep Kubernetes

moves pretty fast. Amazon will give you about a year and a half on um particular Kubernetes version and then they say it's time to upgrade otherwise we're going to start charging you through the nose. Uh the other thing uh broken you know these upgrades are not perfect and broken builds in Kubernetes is a problem with some cloud vendors. Uh is there anybody from Google Ah good. Yeah

we we need to talk to them. That's we've had more problems with Google. Um, I mean it happens. It happens to everybody that that will get a bad build, but uh, you know, occasionally they don't quite get it right. And then things everything's different. So each of these like EKS, GKE, um, uh, Azure, they all have a slightly different way of allocating systems. So you have to

deal with different APIs. But on balance, these things are really, really good to use. And when you're first building an a cloud platform, you don't want to have to deal with this stuff. So the thing you do have to deal with is you have to learn to wire these things up correctly so that you can allocate uh VMs efficiently and so that you can allocate storage. So

this basically and this is where Kubernetes and clouds they they where they come together is the part where Kubernetes gets kind of hard because you have to have the IM configured correctly so that services inside Kubernetes can for example ask to be effectively ask to be scheduled on a VM if that VM is not present one will be created for you that wiring doesn't happen automatically you

have to do there's a fair amount of setup that's required to do that uh same thing with uh storage fortunately with with managed Kubernetes, you usually come up and you're about 80% of the way. You just have to do a little bit of configuration on top of that. Why do you want to get this stuff right? Well, um oh, wait a second. I what we'll do. So,

oh, as along the way that I'll tell I'll tell you two reasons why you want to get that right. But first, um you're running databases. So, databases are super super sensitive to resources. Um just some things that we've learned about that that are um really important. So for example on Amazon, we're big fans of ARM. They are cheaper and they are faster. Um so this is the

the the advantage of ARM seems to be less on on other uh clouds. In fact, I'm not sure that they all fully support them. Um there's three different types of storage that you have in databases and you need to be well aware of them. Uh we like um block storage, cloud block storage like um elastic block storage on Amazon because if your node dies, you can just

reschedu on another host somewhere and reconnect to your storage. The other thing is you can increase the size of that storage with very little with very little difficulty. Um however it's if you really need speed like if you're doing something where the database is more like a cache um using uh NVME SSD so attached NVME SSD is really gives you really great raw performance. The trouble is

it's tied to a single host. If that host goes down whatever's on it evaporates so you better have replicas or have data you don't value. And then object storage is kind of permission to play for analytic databases. Anybody who's storing a lot of database, the thing you have to do in these data platforms is have the IM configured so that the database can actually write to, you

know, has database has a bucket available and can write to it. And then there's two things you got to watch out for. These are problems that I can tell you from personal experience uh both of our customers and more importantly of ourselves. These are two problems you run into. One is, you know, when you ask to do something in a cloud, what's your manager's usual reaction? Cost.

Yeah. How much is this going to cost? You ask for 16 vCPUs, your manager agrees to give you four. So, this is a really good way to mess up databases is to have them be under resourced. In the case of Clickhouse, which we run, Clickhouse doesn't create degrade gracefully. It crashes. So having insufficient resources can create just a bunch of problems that you would otherwise avoid. Similarly,

when you do actually have a problem a lot of times like on Sunday morning where something breaks because like you do an upgrade and all of a sudden you need more resources and things are starting to crash. The first thing you want to be able to do in your cloud platform is just scale up the resources and then go back to sleep. come back in on Monday,

figure out what the problem is and and and address it. Um, but these are really important to be able to do this, be able to, you know, sort of well, if you're offering a platform, ensure that people don't underprovision. And second, make sure you have the ability to scale quickly up and then scale down when you don't need it. Portability. This is a pretty simple one. Um,

I'll give one very simple example. We we run across multiple uh cloud environments. But in general, portability is good because it also means that you can set up this service in multiple locations like you can have a dev version of it, staging version of it, a prod version of it. Um that is easier to do if you don't have a bunch of complicated entanglements with the cloud

environment. I'll give you an example. When we first started with our cluster, we use this we used Amazon NLB. Uh it's a network load balancers. They're pretty good. Um but one of the problems they have is you got to pay for them. Um it's more network stuff to configure. They each got to get uh have to get certificates and and um uh installed and they're slow. So

getting getting one pulled up with a certificate is slow. So we pretty early on came up with a solution to this where we would just fire up the environment. That's slow anyway. we would put one NLB in and then we would just delegate all the real work of proxying for example handling things like uh uh server name indications SNI routing we put that into something we call

the edge proxy so it goes inside and then what's cool about that is um is we go to all these different platforms and the these these network operations which would otherwise be different in each case are the same. Moreover, we have, you know, this that we don't have to pay as much for this and it's faster to allows faster uh provisioning. That's just one example. There's many

ways you can think, but the tendency in Kubernetes is if you can put something inside Kubernetes, that's what you want to do because it's easier to So, if you're building a platform, you want people you don't want people to just go through the UI, although we did that for many years. You want to give them APIs to set it up. Um there's a bunch of different choices

with Kubernetes and I show a few here. Uh there are things like there's Terraform or Open Tofu if you're using that. Uh you know you can use Ansible that's good for setting up infrastructure. There's Helm which only operates inside uh Kubernetes. And then there's plain old scripts you know bash scripts they will never die. So um there's there's all kinds of ways that you can build APIs

to these um and they operate at different levels. There's some you know things like Helm are limited uh to certain things. Here's our uh advice on this um Terraform. That's your null hypothesis. Do it in Terraform. And the great thing about Terraform is that you can manage since you're typically running in the cloud, you can manage both cloud and Kubernetes. What we do is kind of sneaky

actually. We have helm charts and and uh so even though I said helm is limited we have cur we our terraform uh modules call it um if that doesn't work then look for alternatives and don't underestimate the bash script there's a lot of great software that has a there's a there's like one monster bash script running it underneath the good thing about these scripts is that you

know what they're doing. Um you know things like Helm are complicated to work with. uh you know, Terraform is again can be surprisingly difficult to get work. Uh these things are you understand what they're doing. The the big problem is bash scripts don't deal with stuff like restart unless you put some work into it. Um HJ ah this is one of my favorite parts. Um yeah, so

one of the problems you have with databases is they depend on having data and sooner or later you're going to lose it. So one of the issues that you want is so we have this our operator and one of the things like and and this this is a picture I drew after a failure that actually happened twice in a row with one of our customers where uh

Amazon EBS volume stopped responding. This is the absolute worst problem you can have. Um it wasn't dying, it was simply not responding at all. So you want to have with an operator, we actually made changes in our operator so it could deal with this situation. And what what it was is now with our operator, if you get if your storage gets stuck, you can just go harpoon

the uh the um the volume claim and uh delete the underlying volume and uh the operator will automatically provision a new one. It didn't do that from the beginning. So you need to be able to deal with that. You also need to think about separating replicas and database replicas. The absolute worst thing is to have them all on one host. Um ideally you want to get it

separated into different racks. Um so uh there's a really important principle called anti-affffinity. Uh Kubernetes has very good support for this. It's built in. So we use for example this is uh keeper or zookeeper which maintains consensus. We use anti-affffinity to force these guys onto different hosts often in in uh they can also be in different availability zones. Um so this is something you absolutely want to

build in. Um backups I mentioned that they're utterly necessary and everyone hates them. I used to I I ran mainframes myself. Uh in fact my worst screw up in my life uh involved backups. Um it was in my first job and the problem with backups is there's a lot of requirements. They're used a lot of different ways and no single method covers them all. So everybody will

say, "Hey, do disc snapshots. They seem like they're easy, but for example, what if the thing you're trying to restore from the backup is a table?" Well, that's actually not so easy with this snapshots. Um so so there's a number of um of issues. our own experience and I think this is common for most vendors. They will not talk about this but backup failures are a major

source of alerts internally. So we have like a meeting every month where we look at okay what were the new what are the new things we learned about how backups can fail and we try and fix them. Um the other thing is they're costly because if you make a co full copy of storage you're going to put it somewhere else. You can't just keep it hanging around.

Uh you know it needs to be somewhere further away. Uh, so it's a source of cost. So you need to think carefully about how you how you manage that and get it to the cheapest And speaking of cost, that's something to pay attention to. Um, network transfer, how many people have dealt with network transfer costs on clouds? Ah, okay. So the rest of you just don't know

how much you're paying probably. So in this is an example of the recommended architecture from most cloud vendors that if you're running a database like click house you have three nodes these are replicas we've got a three node ensemble for keeping track of the cluster state and we split it across three availability zones and we have enduser applications are talking to this as it turns out uh

through the open internet Um well so you can see the these network connections. Let's translate those into costs. These red arrows approximately correspond to all the connections that cost you money in Amazon. Every terabyte or excuse me every uh gigabyte you move across um across as is going to cost you two cents. If you do a lot of rewriting of data and moving data back and forth

that gets expensive really fast. pulling data out to um an enduser application that's across the internet cost you 10 cents a gig. You may have uh discounts that um but this can cost you serious money. Last we're not a big company. Last year this cost us in excess of something like $600 700,000. It's it's something you have to watch. And it was one of these things where

you know when we realized this was happening we felt pretty dumb. uh it probably won't happen again but it was uh it was not a good experience. It does lead to an interesting conclusion though which you know you see these multi-AZ application architectures our experience over the last seven years is that model is probably wrong for most And here's the problem to to spread things across availability

zones which is to say across data centers. It adds complexity to the applications. That's a big issue because you're running them. You have to force them apart. Um, it adds cost as I just showed you. It's harder to manage. I mean, you just there's just more logic required to split things out. So, and on the costs, for example, if you you have things like load balancers, you

have things like networks. there's actually infrastructure that you pay for and instead of allocating them once for a single availability zone, you repeat them across different, you know, as many availability zones as you use. So those that's how the costs add up in addition to the network transfer and that's why they're also harder to manage. There's just more stuff. Um the thing that's kind of surprising and

this is where is in our experience running hundreds of clusters over seven years we've never had a a failure of an A where having we've never had a failure where having extra copies across an a across multiple a helped us one bit. Um what happens is that when clouds fail, they fail by region. And on top of that, um when they for the data platform, the last

big regional failure we had was the one that brought down um uh the US services in US East one. And what was kind of interesting about it was the big service that it brought down was Octa. And some of our customers use octa. And so they couldn't get into our cloud management plat cloud management plane. But they weren't complaining to us because the reason was that they

were using octa for everything else. So it wasn't just us down. Every single thing they were running was down because octa was not working. This is how things really look at least in the in the normal failure cases that we've seen. Single multiple aes don't do anything to solve that problem. And so as a result if you you know if you're building these systems I think shooting

for single AS there are way I think one thing that we do think is important is to try and get processes separated so they're in different racks Amazon for example does have ways to do that we're sort of learning how to use that effectively but that seems to be the best solution. Last time I did this talk there's so many in the crowd that you know said

but but um you know like what if there's a war? Has anybody been following what's going on in the Persian Gulf? So, um, uh, United Arab Arabs, Emirates and, uh, Bahrain, uh, three A's have been taken out of service because they were hit by debris from missiles as far as we can tell, at least as far as I know. So, that's an availability problem. But, and so

the question is, would that be helped by having additional AS? And the answer to that is still no. Um the uh and this is actually a quote right off the Amazon uh health dashboard. You can read it right now assuming they're still um bringing these guys up, but they're saying everybody in the Middle East move to a different region. You need cross regional uh DR. That's what's

really important and that's actually a hard problem to solve. So final question for building database platforms, you know, this is talk about database platforms on Kubernetes. Do you always have to use Kubernetes? I this is a question I used to ask a lot because I didn't know how understand how Kubernetes worked. Um but in fact the answer is no. You know there's other ways to do it

and I just want to mention one that is popular. Um some of our customers use it. It's called Slurm. How many people have heard of this? Okay. All right. Good. So you guys are doing HPC? Yeah. Okay. I see some I see some nodding heads here. Slurm. simple simple Linux utility for resource management. If you run a high performance clusters, you'll have like we have one customer,

they have literally thousands of racked uh systems and they use slurm to allocate these to build things like database clusters. Slurm is aside from having the coolest name uh I mean it's a name that you you want to try it. Uh but it's it it doesn't even use containers. It's basically allocating, you know, sort of slices of Linux machines out to specific processes. It's very lightweight. Um,

so it's simple. It runs on bare metal. Bare metal is great because things uh um, you know, tend to be more stable because there's less stuff in the way. Um, it's really fast to spin stuff up. Things that take, you know, 15 minutes for us on on Kubernetes to allocate new VMs and slur, they're up in seconds. It's it's very very tempting when you see it. And

the final thing is Kubernetes has one of the there's a number of you know downsides to using Kubernetes but one that people aren't aware of until you run database for a while is it runs on livveness checks. So Kubernetes will will do checks on your pods and you know it may be something like you know going in on a database and doing select one. If your database

gets overwhelmed it may not respond to that. Most databases don't have good resource management and so they won't allocate they can't allocate resources that will stay protected when the database hits 100% um utilization and um in Kubernetes what Kubernetes says is hey these are cattle they're not pets and it just kills it if it's not sure what the state is and restarts it that's not something you

want to do in in databases slurm doesn't do that so that's that's another like our friends who do financial services they want the highest highest possible availability. They don't want stuff like that in the way. So, yeah, always think about alternatives. Don't don't follow the herd. All right. So, what do we learn? Um, we learned a lot. I mean, we started we knew zero about Kubernetes when

we started this. Um, you know, we could spell it basically, but yeah, use operators, pick the right tenency model. Um, there's really only two choices. I think interestingly enough we may go to doing our own Kubernetes management you know in the future because there's there's some if you run enough of them then then then it begins to you may have some advantages in taking that stuff inhouse

but in general managed Kubernetes is the key um uh Terraform whether you like it or not ah thank you 10 minutes to go so we got time for questions uh it's a null hypothesis uh you know if you want to do the best thing for your users. Obviously, if your users have different opinions or you have specific knowledge, it might be different. Um, if you don't do

anything else in this list, get backup, right? It's the thing that people screw up. And there's nothing worse than having a system lose its data because your backups are messed up and you didn't notice. I actually worked in a in a uh company where it turned out that for a period of something like six weeks all our backups were being written literally to dev null and um

that was they had a failure at some point I think during an upgrade which is also another time that's that's kind of a kind of a a trying time for DBAs and that's when they discovered they had messed this up. was, you know, I was just like a an innocent bystander when this happened. Thank God. But that's that's one that's one you got to get right. And

it's um so um yeah, the multi-AZ developments uh or deployments, I think they're way oversold. Um I think the conventional wisdom actually is very convenient for cloud vendors because it it encourages transfer costs. Uh and then Kubernetes, it's great. I love Kubernetes. I think the advantages include things like portability, the fact that actually if you've got it set up, it makes life easier for developers of distributed

applications. Kubernetes is not hard as a developer. It is hard as an as an administrator, but it's also not the only choice. And there are reasons why you might just do simpler things like just run them on plain, you know, run your clusters on plain instances. We have a lot of customers who do that. um run them using you know like proprietary or you know sort of

local VM technologies so on and so forth. So that's it. Um thank you very much and if you think you can do this better than than what you heard here we want to talk because we're hiring. >> All right, time for questions. Thanks for the uh yeah thanks for the talk very insightful. Um I have one comment and one question. So um one comment is about if

you are using uh either multi-AZ or multi- region um you probably don't want to rebalance uh on the network but you want to rebalance by pulling from the the the S3 or the block store the I mean the object storage it's Duid I I run on >> absolely Absolutely right. Yeah, that's that's a really important point. >> I don't know if I don't know much about click

alles around. So in droid it's possible. I don't know. >> Yeah. inflict. Uh and the other question is um so you you know that databases all the way from you know Oracle rock and others always had like when when it's distributed always had clustering um within the database uh um you know architecture. I was wondering what's your take on let's say I'm building a new database okay

uh that is after uh kubernetes already exist how much of the networking and clustering would you rely on the kubernetes stack or do you saying no it's too specialized and I I I'm just going to do orchestration and you know and those kind of things but not like >> networking yeah I think that we're okay so first of all I want to just say your um your

point on S3 is really important and I underemphasized that when I talked about the three different types of storage what we're increasingly seeing is that data it's not just it's not just sharing data you know within a single database but also adopting things like iceberg which allow you to share data across multiple databases and having this built into your database platform so that for example you have

buckets in known locations that are just configured for people with appropriate ate um with appropriate uh authentication and and authorization is really important. Um yeah to this um to this question about um about uh clustering and what's built into the database. So yeah that's an interesting question. The part about the clustering part that involves scheduling resources you can completely delegate to Kubernetes. So it used to be

before Kubernetes databases would be forced to uh to perform those functions. That was that was something that was had to be built in because there simply wasn't external infrastructure and Kubernetes does a very good job on that in general. um let's say you'd have to be very expert to do a better job where the part of clustering that has to stay inside databases and I don't think

is going to move is is um consensus about data and operations in the database. So we use uh ClickHouse Keeper. We actually Click House originally started uh using Zookeeper like a lot of people like like Kafka for example or I think Druid used it as well. Um that part is always I think going to be part of the database. You can't really delegate that and that's important

because it goes to consistency of data. Kubernetes doesn't know how to it doesn't have the um mechanisms built into it to do things like leader election. That's a common there's a lot of operations which are unsafe to have multiple people try to do them at once. You're you have a consensus manager that allows you to elect a leader. That one leader will then make a decision on

behalf of everybody. So I think that's that that's my take on it. If you if you have a different one, I' I'd be interested in hearing it. Um, thank you. Excellent talk. Yeah, just just a quickie. Um, so you found yourself in a situation where you had a problem that you decided to solve by writing a an operator. How was that effort as far as developers and

getting that thing working? Did you find it was it was easy to manage or were there complexities? I'm curious how that all went. >> Oh, like what kind of problems did we have in in implementing the operator? Well, um, we are very slow to do developments. So, some problems are avoided by the fact that, uh, our our lead developer is a is a just so guy and

like he's as a manager, I find it a little frustrating because I want stuff like in two weeks. He says, "Okay, you might get it in two months." So, we're kind of slow to and things pretty pretty thoroughly. Um I think the things that caused us issues were I I'm going to two I'm going to uh you know name two things. One was limitations in Kubernetes itself.

We still have a problem with livveness checks which I think is sort of on us that we haven't put the engineering effort into designing a livveness check that will not um come back and you know that will not come back negative just because the database is kind of busy. We should have really dealt with that problem better. Um there's another um there's also limitations in the in

the way resources are managed. So for example uh Kubernetes has this notion of stateful sets uh which can have replicas. They have some um built-in limitations about the kind of changes you can make inside of pods and more importantly the changes that you can make on storage. So uh for example we used to have to respin um pods to be able to to make a change in

the size of storage. So that's something we've worked around. we've now taken away the responsibility for storage management from the stateful set and given it to the operator. Um, beyond that bugs, I think the single worst one that we had was there was an upgrade bug where during um during an upgrade there would be a brief period of time where we our nodes would lose track of

which shard owned which data and that because the load balancing got screwed up. It was it was just a bug. It was like the bug of the year. It took weeks to figure out why it was happening. Um so some of these things are they're just difficult to test. I think that one thing has worked in our favor or worked in the favor of the operator is

we do run production systems on it. Um when you have mistakes like that, let me tell you customers they get pretty pissed. So we do everything we can to avoid it. I think having that that pressure on us has been really helpful. But those bugs, we've had a few of those bugs. They've been they, you know, they're they require a lot of testing. They're often not visible

in outside of production data. So that was a that that was an example of something that was a real problem. >> Welcome. >> Hi. Why do you discourage moving from um dedicate sorry share cluster to dedicate cluster uh earlier in the slide? >> Oh, can you say can you repeat that a little louder or I can't I can't hear. >> Oh, why do you discourage moving from

the share cluster? >> share tenant to the dedicated tenant. >> Oh, yeah. Actually, I think it's a choice. I think it's an architectural choice. I'll just flip back to the uh Yeah, it's really a choice like what are you optimizing? There's um yeah, here we go. Um, so when you have a dedicated cluster, uh, it is, you know, for something like an analytic system that might have

a pabyte of storage, it's actually kind of convenient to have it for a number of reasons to just put it in its own Kubernetes cluster. The overhead of managing the Kubernetes cluster is not that great compared to the overhead of managing the database cluster itself. So the balance shifts to and and and it solves a lot of problems for you. You don't have to worry about noisy

neighbors. you don't have to worry about accidentally connecting to the, you know, like customer A to customer B's data. So, so that's really good. But if you want to the, you know, if you really want to have highly optimized um usage of resources, putting all the tenants together in a multi-tenant system is a better answer as long as you can master the engineering to keep them separated

and also um fully get full advantage from reuse, you know, from sharing resources. It's a and and the trouble is once you've done one, it's kind of hard to do the other. They're quite different models. >> Well, I can imagine it could be a um common use case. You start as a shared tenant because you don't need the >> And then you at some point your company

gets too big. >> You're absolutely right. And as far as I know, our main competitors, what they end up doing >> is what they'll do is well, even they have a because they do the shared shared model. The trouble is that getting I mean this is a question we debated quite a bit when we were making this decision and we decided that it was just simpler to

optimize for people growing in place as opposed to having to go through some migration. Migrations of data are really hard and again that requires engineering. So what would happen is if you had somebody who outgrew this environment, you would then have to have some, you know, some engineered migration process that allow them to move because this is a production system. You're moving, it needs to move without

downtime. So that's a that's not easy to achieve. Um, so anyway, that I I think that's that's something you balance when you build these system and that's why I think you probably make the choice up front and then don't deviate too much from it. Any >> All right. Well, thank you, uh, Robert.