Room 106 Saturday Mar. 07 - SCaLE 23x

This talk discusses the implementation of Renovate at Grafana Labs to manage dependencies across over 1400 repositories. The speaker explains how Renovate facilitates credential rotation, handles custom versioning for monorepos, and allows for release strategy flexibility. They detail the configuration validation process to prevent errors when changes are made, ensuring stable deployments and efficient development cycles. The session also highlights the challenges faced when scaling Renovate's operations, including API rate limits and GitHub app token management. The talk concludes with insights into how the integration affects security vulnerabilities and dependency management across the organization.

rule for um BCR um whatever. So there's no risk of credentials leaking to the wrong place and even if credentials get leaked, they're only going to affect one um one resource. So rotating credentials is simple with Renovate. Update something involved and external secret uh external secrets um uh pick it up and syncs automatically. Um so another problem is custom versioning for monor repos. So here is a

realworld example of renovates flexibility in our shared workflows monor repo which is open sourced shared dash workflows in our graphan organization. Each reusable GitHub action is released independently using release uh using Google's release please. Um tags look like this like action name v123 rather than plain simber. Out of the box renovate doesn't understand this format and either misses updates entirely or creates useless PR descriptions. Um the

fix is renovates reax versioning. So we define a pattern that splits the tag into two parts. the action name which is the compatibility prefix and the actual simmer version. This lets Renovate correctly detect um new versions per action. So beyond just parsing versions, we also use this to customize the branch prefix per action which is really important and craft meaningful commit messages which is also important because

our commit messages double as PR uh titles. So when your release strategy doesn't fit the standard mold, renovates reax versioning uh bends to fit your workflow rather than forcing you to change your workflow to fit the tool. And then as your renovate configuration grows more sophisticated, it's easy to introduce errors like um a typo in a reax pattern or an invalid option or a logical mistake. Um,

these errors can break dependency updates across your repositories and you might not notice until updates uh stop appearing. To prevent this, we validate all renovate configuration changes before they're merged. Uh, we have created a reusable validation action that runs the official Renovate config validator tool. It checks for syntax errors, invalid options, and logical issues in your configuration. So this runs automatically in our CI pipeline on every

pull request. Uh you can opt in or opt out of it if you want to. At the moment um if someone submits a change with a broken config, they get immediate feedback about what's wrong and where to fix it before it ever affects uh production. It's a simple safety net um that becomes increasingly valuable as more teams adopt Renovate and configuration complexity grows. So catching errors early

means fewer broken deploy deployments and faster development cycles. So in the slide you can see a small flowchart which what it does basically what it means is that basically in that action renovate updates itself. So there is one action that runs this renovate check with a like a pinned renovate version. So renovate updates itself and then once this action gets released using release please or whatever else

you use then renovate runs again to update this action in every uh consuming consuming repository. So as you can see here uh I hope this is clear. As you can see here the first PR is created by renovate to update to update itself. Then once the once the GitHub action gets um gets released you get this really pretty PR with um you know all the changes the

change log and everything created by Renovate. And this is the actual the actual change that Renovate uh makes in a single repo. So it updates the hash and it also updates the uh semantic version uh by also keeping the the action prefix. And with this I'm handing over to Phil morning. Uh can you hear me? Okay. Y Okay. So uh morning in all I'm going to run

you through a bit more information uh on how we've set up Renovate and how we're deploying it in our infrastructure and what we've done to optimize running this at scale for the 1400 plus repositories we support at Grafana Labs. Uh here's the overall view of our setup as it stands today. We're running renovate across multiple chron jobs. Each one processes a batch of repositories in parallel. Uh

we chose crown jobs because they run at a predictable cadence which gives us control over uh when and how repositories are processed. They're also isolated. When you're processing hundreds of repos, something will eventually go wrong in one of them. U and with batch grown jobs that failure stays contained while the rest keep running. One limitation of running renovate on a schedule though is responsiveness. We've opted for

three hourly runs which give engineers several updates throughout the day while keeping the benefits we get of consistent load predictable behavior and the ability to treat treat each batch independently. Each cron job um inherits the global configuration that Demetri just mentioned. So security policies are consistent across all of them regardless of which batch a repository ends up in. We also use reddish to cache across all jobs.

We're caching repository meta metadata, private package information, and shared presets, which means each job isn't refetching the same data from scratch on every run at scale this has added up. Naturally, we've run into a few issues running this at the scale we've needed to. The first was rate limits. Early on, jobs quietly stopped raising PRs, and it wasn't obvious uh there were no obvious errors, uh just

silence, and it took us a minute to understand why. Renovate makes multiple API calls per repository and when you're running hundreds in parallel, we were hitting G GitHub's rate limits faster than we'd expected. The second was more subtle. GitHub app tokens have a 1 hour time to live. If uh batch takes longer than an hour to process, the token expires mid-run and any remaining repositories start failing

with 401. This is where our batch sizing has really mattered. We tune batches specifically so jobs complete within this window. um jobs can re try and fail, but of course the goal is to avoid that situation altogether. Um this is essentially this token limitation is the main trade-off of the chronel approach that we found so far and the hardest sealing we've had to work with it because

we can't dynamically update config maps for batch runs already in progress. The token can't be refreshed midrun. So batch size and addresses both constraints, keeping jobs short enough to stay within the token window and limiting how many API API calls are in flight at once. Um the shared reddis cache reinforces the rate limit side further by caching package metadata and register responses across batches. We avoid each

chron job independently hammering the same APIs. Together we found this to be a reliable combination for staying within both constraints. Okay. So we mentioned that we get auto discovery with renovate out the box. Uh for our environment we use this and we use the filter uh feature that it offers to divide which repositories each batch processes expressed as reax. On first iteration um we maintained a manual

list of repositories which allowed us to roll out in stages and control the scale and cost as we gradually on boarded the whole organization. Over time, this has given us a feel for the right batch sizes that we mentioned, specifically keeping jobs within this GitHub app token time to live. Processing repositories alphabetical uh alphabetically has given as a consistent way to divide the org into evenly sized

batches um with with each chron job handling a predictable subset. Any new repositories we want to exclude are added as negative patterns and applied across all batches automatically. And the batching approach means that again load is predictable and it's easily tunable. If we scale out, we can add batches. If a particular batch is causing uh problems, we can isolate it without affecting any other repository runs. And

the practical effect of using auto discovery in this way is that we get the same control and predictability from a manual list without the toil. Okay. So to mention uh a service that we've built to to help us alongside this, um we've put together a web hook listener. Um, a main advantage of self-hosting is that you own the infrastructure and you can extend it. And as mentioned,

uh, a main trade-off is how responsive it is and particularly in particular to when users are waiting for their own user made requests to return. Mend.io uh, offer on the hosted version supports on demand rebates. You take a check box on a renov APR and it happens immediately with scheduled runs. Um, that still functions, but the request sits there until the next cycle. um they'll renovate then

processes that repository and that can happen potentially hours later. So we built a go service that uh listens for Git pull requests when it detects the repos box checkbox is checked on a run of APR. It validates the request and immediately triggers the renovate run for that specific repo. Before launching uh the listener fetches the gap token directly from a Kubernetes secret ensuring that each repository run

starts with a fresh token. Uh, this sidesteps the time to live issue that I just mentioned. Rather than a token baked in at pod creation that could expire midun, every rebase triggered run gets one that's current. And the listener runs alongside a dagger engine sidecar in the same pod. So dagger spawns renovate as docker and docker um as docker and docker container which completes a default renovate

run. We're essentially handing over all of the gable history state to renovate to manage itself any uh branch state, PR state and uh it can track and work in combination with how we run it in chron jobs. Um and we don't have to worry about any merge conflicts that we would get from potentially firing custom requests to get. Uh so we also handle queuing internally. If new

race new rebase requests arrive while a run is in progress, they're held and processed immediately after. No external co coordination needed. Um we considered two main alternatives to this. Shelling out to docker run uh or spawning a kubernetes job per web hook. But both uh on review added overhead and fragility that we didn't want. A long live service means that we pull the renovate image less frequently.

Um, using Dagger means we can benefit from the layer caching that Dagger offers and we have a single place to extend when we need new behavior. Which leads us to where this is going. What starters, what started as a rebase trigger is becoming more of a general utility service now. A single place to add automation rather than building one-off triggers for each new need. Um, and in

terms of the other features, the the last one that we've added is automerge. So, uh, it's a feature that people are often, um, are instinctively cautious about. The risk isn't the feature itself. It's the merging. It's merging things that you don't fully understand. Our approach is to replace that instinct with explicit auditable signals where possible. And we use a two-stage model for this. The first stage is

an auto approved bot. It runs two sets of checks. Uh, PR integrity. Was this created by Renovate? Are all commits signed? Is every commit from Renovate? And package trust. Is this on our allow list? Is it a patch level update? Has the version been published long enough to avoid early supply and chain issues? Any failure in either set sends PR back to manual review? And our web

hook listener feeds directly into this. Because we have a service that already re receives PR events, we can run more sophisticated checks there like log validation and file history analysis without needing separate infrastructure. Uh the merge decision itself is handled as a CI check. So it slots into the existing PR workflow. By the time a PR reaches merge, enough signals have been collected that the decision is

just me mechanical rather than a judgment call. Okay, so uh the last part of this section just going to talk a little bit about how observability looks when when we're self-hosting renovating and how it might look to you. One tool we wanted to mention that got us up and running fast is the graphana infinity data source. Um, it lets you query GitHub a the GitHub API directly

from Grafana without building or maintaining a custom exporter. You'll see more of this in a demo in a second. Um, but there's a couple of panels that are worth calling out. Now, on the right here, you can see 32. This is the security PR count. It gives us a live view of how many PRs are open across the org to fix security vulnerabilities. And we have two

uh merge time metrics that we track. We have uh the the the top is regular dependency updates and the bottom is uh security vulnerability updates and the time to merge. Over the course of the roll out, we've been able to measure adoption and response times using these and we've actually seen them drop from from days to hours. So we can see the adoption quite nicely. Uh together

these go beyond monitoring whether renovate is running. They show whether it's actually having an impact on how our organization is managing our dependencies. And it's worth mentioning also that the Infinity data source has been great for iteration speed, but our next step is to as we've matured to move this to a Prometheus exporter so we can track these as proper time series. We can get better retention,

better alerting and the ability to define custom metrics. The second part of our monitoring is for the deployment itself. We're using the Kubernetes monitoring suite in Grafana um which is gives us a nice easy single place to look for a namespace level view across all renovate pods. Um at the top we have CPU and memory across uh time across the the name space. The periodic spikes map

directly to our chron job schedule which is running three schedules over three hours each hour. Below that is per pod breakdown showing average and max usage relative to each pod's requests. Um because Kubernetes monitoring uses Loki under the hood and Loki shares label with labels with Prometheus, we're able to get a single pod level level view that shares both metrics and log data. Meaning we can switch

quite quickly. For example, say we see a CPU spike, we can get to the logs for that same period much faster. There's no extra configuration needed and no switching tools. um when we're running at this scale, um having metrics and logs in one place has made the day-to-day and the on call responsibility a lot simpler. Uh and lastly, um alerting. So for Renovate, we have three main

behaviors that we care about being alerted to. Um the first catches outright job failures, anything that runs any run that exits non zero. Um, without it, we're leaving ourselves prone to to only find out renovator stopped working when somebody notices a PR hasn't been raised in a while. Uh, the second fire is when a job has been running for more than an hour. As we scaled, it

was a useful signal for tuning batches, but we primarily use it to catch something stuck, for example, a repo or resource pressure. It gives us visibility into repos that need individual attention, and it also indicates to us how well jobs are balanced to complete within our token window. And the third is our dead man switch. If no jobs have been created in 24 hours, we get paged.

Uh this catches the scenario uh where renovate has quietly stopped running without any individual job ever failing. Um and it's worth noting these are the renovate specific alerts we've added. We also get Kubernetes level alerting from our infrastructure monitoring killed events, crash back offs, any any pod or container activity. Uh and the combination give us the full picture we need. So yeah, that's uh that's the wrap

up for this slides. We're going to just give you uh show you how this looks in practice. >> Yep, sure. So, demo time. Yay. Let's see. How do I go? How do I go to the Come on. >> What do you mean? >> You do it. You want to do it. But then how how can I control it from here then? >> Uh mirroring. Sorry, one second.

How am I going to do that? >> Yeah, sorry about that. Yeah. Awesome. Thank you. Uh I don't know if you can see the screen. Okay. I hope so. Um so this is K9S. We're going to use K9S for the sake of simplicity to show you um everything we talked about so far. So so as you can see here, this is an overview of the chron job.

Sorry, let me make it a little bit bigger. Yeah. Okay. So, this is an overview of the chron jobs. As we talk, as we told you already, we run uh all the updates in batches using chrome jobs. So for example, this is named renovate self-hosted schedule zero batch job zero, which means that it's the first batch job which runs in the first uh um scheduled renovate job

which means that we go down to schedule zero but job 19. All these 19 20 jobs run in the first schedule and when these are done the next hour you have another batch of another 20 and then you have another batch of another 19. So for the sake of simplicity and like for the sake of the presentation we have also created this renovate self-hosted test job which

we are going to trigger now. And while this runs I'm going to manually trigger this now. And while this runs, I am going to show you the config map. So if I scroll down and go to the config map for this uh uh for this chron job, you can see all the uh configuration options that we use in our global config. So this is our global global

config. This means that every repository that runs in Grafana that that exists in Grafana Labs runs on this uh config and this is um enforced by the security we work with security team. We work with the security team and we came up with this uh config. So as you can see there are stuff like caching private packages. We um we don't allow any custom commands as we

talked to you about before. I'm going to come back to this in a little bit to the extent bit. And we also false uh disable plugins and scripts so nothing um funny runs in there. And we do other stuff like minimum release age which means that if there is if a package has been released in less than 3 days don't merge that because it's probably not 100%

safe uh until it's like properly tested. We have OSV vulnerabilities vulnerability alerts uh enabled and other stuff like you can automerge vulnerability alerts as they appear. You don't really have to um wait until someone reviews them if they're critical and you can label them. Uh you can label v vulnerability alerts or create groups of vulnerability alerts if you want. So this is the config map we use

and oh yes sorry I forgot the extends part. So this is the shared presets bit we talked about uh just before. So we again enforce these presets as best practices presets to every repository but we give people the we give teams and repositories the option to opt out of them because we believe that opting out of them instead of like giving teams the opting in option is

better. So we can know who does what and why and like every opt out um case should come through a PR and the and like team should be able to justify why. Um then if I go back and I check the pods and scroll down I'm going to talk to you about the error. So you can see the self-hosted job manual runs and if I go check

the logs in K9s you can see the the logs uh appearing and like renovate running. So this means that probably if it finds dependencies is going to create uh PRs for our test repo scale uh 23x renovate demo. And about the other ones you can see some errors here. These are like the actual renovate jobs we run uh in our real repos both public and private. So

when something fails it either means that it got some rate limiting not GitHub rate limiting rate limiting because we have taken care of that but some rate limiting somewhere else or maybe I don't know you know GitHub is down or maybe um I don't know some private registry is down as well and they keep on erroring but as you can see if this errors because Kubernetes right

if this is failing then it gets uh spinned up again and it runs It's the same job. Yeah. Yeah. Schedule zero, bad job one, it runs again until it succeeds, basically. And if, as Phil mentioned before, if it doesn't succeed multiple times, then it means that a repository doesn't get enough PRs in a certain uh time frame. So, we're going to get alerted because, you know, we

don't get enough renovate PRs in repository A. So, we can jump in to fix that. If I scroll down, this is still running. Is it? Yes, still running. And I wonder if so this is the sorry about that. This is the repository we created for this purpose. So in the pull request you can see opened one minute ago. This means this means that renovate found um uh

some outdated dependencies and and created uh updates for um all these dependencies. If I go and click on a random one uh let's go to this one. you can see a really nice uh PR with you know merge confidence which which means that you know other repositories are using have been using this version for a while so it's probably safe to merge and you can get stuff

like release notes uh all the release notes everything that happened since your uh that has been updated since the the last version and also last thing I would like to show you here is this button so Phil talked to you about the web hook listener so this is a an extension of renovate some extension we created. So if I go and click uh here and check that

box we should get a message saying that renovate is updating the um the renovate is updating is rebasing the PR. This is really handy when it comes to uh you know outdated dependencies or maybe there was like another PR which got merged and you get some conflicts. So you click on that button and then renovate is processing the the uh repository. So this takes one minute to

two minutes but because we don't want to keep you waiting, Phil is going to continue uh showing you the uh dashboards. >> Yes. So to just give you a little view into uh the observability tooling that we mentioned, this is the Kubernetes suite. Um make that a little bit bigger for you. Um within Grafana instance we Oh, it's the opposite scroll. Um the Mish just kicked off

a test job here. Um we can see how it's running. We can click into uh more drill down on it. We can see an overview, general resource usage, pod counts, um and what's going on here. The uh most useful part of this for us really is having everything in a in a single place to quickly switch between logs. And that you can see here. Um okay, that's

an old pod. That's why refresh that. Okay. The curse of the demo. Okay, here we go. So, this is the one that's currently running. Um the most useful view for me is uh this single inline here that we can see exactly the pod name. The status is already succeeded. It's finished running um which we saw from the PR activity. But now this is totally complete because Loki

and Marthus are sharing labels here. We're supporting um to be able to directly follow through to uh pod logs as well which has been super handy. Um and that's about it for that. And the other the other one to mention here is our renovate dashboard. So this has been the main the main place that we've been monitoring um uptake and operational health. So on the left here

we're pulling the the 1.72k repos, the amount of repos processed within the last two days um for our time window here. Um and we're pulling this figure from um s succession um success uh log lines um and counting those. The total security PRs count is one that we're calling GitHub API using the infinity data source for directly. Um and we we also list um which security PRs

are open to for quick access. Um we have a little on operational health again more resource usage um app limit usage since cashing we're now sat around 40% of our limit. Um and then yeah the probably the other one the other one to mention that's been particularly useful is our um repository process and duration timeline. If you are self-hosting renovate or even if you're not and you

can get these metrics, it's been particularly useful in troubleshooting anything longunning. You can see here um straight away that we've got a couple of outliers at the top. The this is an example dashboard. This is a subset of our augs repos for the sake of the demo, just the public ones. And graphfana graphana there is running 18.3 minutes um or average around 18 to 19 minutes. Um

we know from knowing this repo and knowing the complexity of its dependency tree that this is expected for graphana graphana but in previous cases we've had some shooting up to like 40 50 even over an hour. Um and straight away we can go to that team we can find out more information about the context of their code and their dependency tree and we can work with them

to see if it's actually something that we can help improve or um we essentially just have very quick recognition on that. Um, and I guess the last one to mention is the management overview. So, we mentioned the PR times and security me merge times. We track those here. Um, we track the types of dependencies and we're tracking the rate of change over time as well. So, you

can see it's going from days down to hours over the last 90 days. So, I'll just flick us back here. Um, so we had our we can see that this rebase is completed. That took about uh two minutes uh just less than. So yeah, that's how that's worked. Me back to grand. Okay. So um yeah, that's essentially uh us, but we just wanted to wrap up with

this. the the tempo team at Grafana reached out to us to share that since adopting renovate their CV CVEQ is at zero. Um this was really nice for us because this is the outcome that we've wanted. It's not it's not the tooling, it's not the automation, it's that our teams have been able to effectively reduce their backlog of security debt with uh minimal uh overhead. Um, so

yeah, we hope uh today's given you something useful to take away. Um, whether you're just getting started with Renovate or already running it and hitting some of the same walls we did. Um, and please come and talk to us if you have any questions or we can take some questions now as well. Thanks. >> Who was first? I don't know. I think uh at the front. Yeah,

please go ahead. want to take this one. >> Okay. Yeah. I mean, of course, you know, there are many times where you where you like update um something to a new version, something breaks, you don't know. So, this is why you have to have um sufficient CI checks, I would say. So every time there is a every time there is a um PR created by renovate you

have to make sure that all CI runs as it runs on every other PR. So if the CR sorry if the PR passes it's like if if a PR that got created from renovate passes and you merge it and it creates a bug. This is like any other bug you transuced from your code is not specific to renovate basically. But then yeah if it if it introduces

some bug which conflicts with your code and then you have to fix it you either roll back to the previous version or you fix the bug directly. But if it's a bug which got created by the like in the library in the dependency itself then you can go file an issue or you can wait until the new version comes out. This is why all the release notes

in the renovate PRs are really handy because you can see exactly what changed and before you merge the PR you can go to that specific place and see oh you know this is changing that am I affected am I not affected and you can judge need some we have created the automerge uh strategy but you know it's really really really complicated non-trivial not trivial at all because

every time something gets updated you have to be careful that you're not going to mess things Even if everything passes, of course, it's like any other bug really. I don't know if this answers to your question. Yeah. Thanks. Yes, please. Was it easy to decide on going the fully automated route or were sort of resistant to it and like how did basically how did that discussion play

out in planning and then after >> yeah um I mean we started before innovate we used I mean in graphinal labs we use GitHub. Yeah. So dependable was the obvious solution for us. The thing is that some teams were using dependable, some weren't. So we needed to find a way to enforce like um you know dependency updates and like security updates across the whole organization. Of course

we have like hundreds of teams maybe in the engineer not hundreds but like some many teams in the engineering department. Some teams some repos were more hard to convince to bring them over to renovate than some others. The thing is that you know we needed a way we needed something to work the same for all repositories and this is why we created the global config which is

something you cannot opt out from and I won't lie I mean you know it was hard there are some repositories which are huge and in order for innovate to run on them you need like 30 35 minutes for example there are some others that need like 1 minute or 30 seconds or whatever we didn't really find resistance I would say it it wasn't like resistance but you

know there were second thoughts I mean dependable works for us why should we move to renovate since dependable works but the answer was because we need something to work for everyone and there shouldn't be a way for anybody to opt out of dependency updates in general you can opt out from some certain um settings if you want to from the configuration but the end goal again in

graphana labs was to make sure that all repositories like the moment they get created in the graphana organization pick up the global configuration as we have set it as the uh security squad wants it and yeah that's it but no huge no huge resistance man. No, it it got it got like to add to that as well. It got a lot lot easier the more we communicated

it as well. The more we communicated um how flexible and how custom like custom presets for example, teams started to see the value in what they could set themselves and even moving over from from dependable. >> I think that was one of the back. >> Yeah. Yeah. Sorry. Yeah. Go on. both. No, no, it's it's both. It's both security vulnerabilities and just dependency updates. >> Yeah. Yeah.

Yeah. And sec security security vulnerabilities you get. I mean it's essentially the same mechanism. The only difference is that in security update uh in in security vulnerability PRs, you get like this big um PR description with the CVE all the descriptions um um steps to reproduce as well if you want to reproduce the vulnerability and all that. So yeah, it's both. It's both. And how far >>

it's >> Yeah. Yeah. No, it's everything that comes anything that comes first, I would say. Uh but you can set it up on your own repository. you have the global config but in your local configur you can say um I don't want patch uh updates uh patch bumps I only want like minor or major bumps you can set it up like it's completely uh completely doable yeah

and some people some teams do it already I think because it's I mean it's a great tool like updating dependencies in general is great but it can be really noisy if you don't if you don't set your own rules and you let it create for example one one thing we've seen is uh we have set up a limit for PRs or branches created by renovate if I'm

not mistaken and we have set it up to five or 10 five I think yeah it's >> and some people are like even five is a lot I don't want five PRs to get created from renovate all the time some people are like you know just uh remove the limit and I just want all the updates in the world and there are some people who get like

some teams who get like 35 um uh PRs in one single renovate run. So, it's just a matter of preference really. >> Yeah. Yeah, please. >> No, it's uh it's sequential. So um renovate separates a type of dependency update to a branch level and then once the the next version from the current is updated then it will continue. It depends if you can again you can configure

all this you can add you can set it to to get the latest at any point but um then that works in conjunction with the the minimum age as well. Yeah. And you can also configure uh renovate to create a PR with groups of dependency updates like have one group uh that has to do with go updates for example. So instead of have instead of getting like

five different PRs with go updates for different Go libraries, you get one PR for Go updates which contains I don't know like um 10 different updates if you want. If you don't want to, you can have like separate PRs. It's whatever. It's like the reason the reason we used we chose renovate is because it's highly configurable like sky sky is the limit. We created the the web

hook listener. You can do whatever you want with the listener because you host it yourself. >> No worries. Yes, please. Sorry. Do you want to repeat the question? >> Uh, >> do you want to repeat the question because I think it did got louder. Do you want to do it? >> Repeat the question. >> Sorry. Yeah, we can use the microphone. >> Yeah, because Sorry, it's Can

we use the microphone? >> Yeah, we probably should have done this from the start. >> How it works to >> Hello. Thank you. Awesome. Yes, >> I appreciate the talk and this is my first time hearing about renovate bot and that seems awesome, useful. Um, so my question was when setting parameters for automated dependency scanning, at what point do you consider a dependency sufficiently aged? Um, and

I assume that's as a function of your security policies. Um, and at what point do you consider a dependency stale or vulnerable enough such that the top level parent package needs an alternative or maybe like a rewrite of the stack that you're working on? >> So for it's a little off topic there. >> No no it's fine. It's fine. So we have set a rule um of

uh minimum release age for a dependency to be 3 days. You can set it to whatever you want to set it. So this 3 days time frame came from the uh security department. I think it's a it's it's some sort of a magic number. I think many companies use three days and I think it's the default in renovate itself. If you go check the check out the

renovate docs, I think it says that the minimum release age by default should be 3 days if I'm not mistaken. So what happens is that if you set that up, if you set minimum minimum release age to a certain uh number of uh days, let's say 3 days, what happens is that you still get the PR create, you still get the PR from renovate, but you have

a check, a GitHub check or GitLab check or whatever you use, you have a check, a check pending saying like stability days and it's still pending until it's stable until 3 days have passed and now you have make you can make this check mandatory. in a sense that you know if it if it doesn't I if this PR is open for less than 3 days I mean

you're not able to merge it otherwise merge it or you can do it on a completely optional level so teams can decide if they want to merge it if they're safe like to take if they feel safe to take the risk in a sense or if they want to wait for three >> um that's the first question the second >> around staleness I guess that's mostly manual

at the moment that it's up to each team to if they've not been keeping on top of new updates, they're and they're coming to a fairly old update, um they can just close the PR and allow renovate to then run the next. But we don't we don't enforce that a global level, do we? >> No, not really. But if you get if you get a PR from

uh renovate with say an outdated version and you keep it there for 3 5 days and then there is another version of the same dependency the same PR gets updated with a new dependency. So it's kind of hard to have a not hard but it's kind of rare to have a um a renovate PR stale because for every new version the same PR gets updated the title

gets updated the uh commit message get updated everything gets updated so yeah I don't I don't think we do anything else about that but we can you can absolutely configure it like to the level you >> Any more questions? Uh, I mean, we're going to be here if you want to talk to us or we're going to be at the Grafana booth after 2:00 p.m. if there

are more questions you want to ask us in private. And I think that's it. >> Thanks. >> Thank you very much. Test test test. Test. Nope. >> Oh, I guess I can't adjust it from here. Oh, I don't want to. I think Hello. Hello. I can test this. I can test the other one. >> An shows up in the next. >> Oh, okay. Yeah. Thank you. >>

Test test test. Pop pop pop. Loki. Loki. Loki. Loki. Loki. Uh I'm not sure owner is the sounds good. there we go. right. Oh, shoot. Hello. Hello everyone. I think we can Hi everyone. How's uh how's everyone's day today? Uh this is actually my first talk at scale. Um so please uh take it easy on Uh well, welcome. Um today I'm'll be talking about how uh we

migrated to Loki and some of the lessons that uh we have learned. Um my name is Vin. This thing is annoying. Uh my name is Vin. Uh I'm a senior software engineer at uh Zip Recruiter. Uh I do basically a lot of platform engineering, SRRES, uh DevOps. Um so yeah, all the fun. Um and as Zip Recruiter, we're are the number one rated hiring site that helps

connects millions of people of uh with all companies of all sizes. Um so how many people know what Loki is? Raise your hands. Oh, great. I think the talk is over. How many are running in production? Oh, okay. There a few of you guys. So, um, one of the big things that we, uh, see, uh, when running a logging, uh, system, um, is that sometimes your bill

of a, um, your logging SAS bill is pretty high and executives are complaining why uh, it skyrocketed over the past few years. Um, how many times have you seen this? not not that much. Uh so with our current problem uh our logging bill was basically competing with our ads infrastructure. Um we were already paying a lot of for logging. Um but to do everything what we actually

wanted um it would cost even more. Um we use Logzio as our vendor and they serve us really well. Um before that uh there were some other infrastructures that we had set up prior to that. Uh and I'll tell you the rest. Um what you'll learn today uh we'll basically either evaluate whether Loki is a right stack uh right fit for your stack um some trade-offs and

uh some practical uh Kubernetes scaling uh on how we uh do cost uh automations uh on patterns and uh some budgets for your migration. Before we can um go through what how we migrated, uh we can talk about how Zubercuers's um logging pipeline over the past years. Uh it's actually not that uh crazy. Uh we were running uh Elastic Search um hosted um we were we were

managing it uh practically a full-time job. Uh lots of JVM tuning, shard management, uh upgrades. Um so we moved to a managed uh SAS vendor uh to get out of that business and basically focus more on building products. Um engineers could search the logs without um babysitting a elastic search cluster. Um not sure if you guys have managed elastic elastic search clusters but uh it's not fun.

Um it was the right call. We scaled over um over 1500 services uh and within our budget uh we ended up uh back with Loki. We we come to a full circle. Uh Loki didn't exist when we first needed logging at scale. Uh we worked with what was available during that time. Um times has changed, Kubernetes matured. uh we were fully in uh in Amazon um so

S3 became more cheaper uh and Loki and the Chronic ecosystem was already emerged um and we had a pretty stable uh pipeline to deploy to Kubernetes easily. Um yeah, our current uh logging pipeline um basically we had about uh 1,400 services. Uh that's across 7,000 pods. Uh we were limited to two terabytes a day of just logs in our production uh tier while the rest of the

tiers were five terabytes uh across all environments. uh production being the most um we had two kind of pipelines. One is going to uh our logging u vendor and then the second was um going to Athena. Uh we did this way because we only had a short retention uh to our login vendor uh because the cost uh was too much if we were to go over our

budget. We had uh Kofka uh in the middle just to handle throughput and uh real time uh log streaming um which help us a lot when sometimes uh our vendor uh goes down. Um oh a little dark. So here's the here's the pain points of uh currently what we had at uh Zip Recruiter. Uh originally we had a 4 terabyte uh per day of uh quota. Um

and then initially we had to cut that down to two terabytes. Um all our logs were sampled. Uh we had to uh alert some teams that they were going over the budget and which was kind of painful to tell that they can't log too much. Um we also only uh log out specific uh types of logs uh mostly uh errors and sometimes warning. Um and everything else

was uh uh just uh not showing up in Logio. Our default app quota is about 10 me 10 megabytes per hour and we had alerts off of that as well if a app was logging more than that. Um and creating a new apps was a pain. uh we had to format the logs uh in a certain way to get it to get it into our and the

big thing was the retention was only seven days. Uh we only had one full day of just Kubernetes cluster logs if if the developers want to look look from there. Uh but in in our login vendor we only had seven days and as we scale more um we also see that the cost of logs goes up more and we had to deal with the pain of just

sampling uh constantly and this is where Loki comes in. Uh this is a quick short what Loki is. Uh it's a open- source log aggregation uh system by graphana. Um the simple ways to think about this is it's like Prometheus but for logs. Uh how many of you guys use Prometheus? Awesome. Uh so for it's basically using the same labels. Uh it only indexes metadata things like

name space um log level container names and not actual the log content. Um the log content gets compressed and uh stored in a object store like S3. Um this this what makes it more cheaper for us and if your team already knows promql log ql is actually really uh familiar. Um same label selectors uh same aggregation patterns and then some of the extra steps to filter down

and parse uh different logs. uh and to interact with it you can go through graphana or uh use the log CLI. So why not other vendors? Uh we could have chose so many other vendors to s to basically deal and make a uh um compete with other costs. Um hosting another elastic search uh would be maintaining two more two separate platforms to visualize your logs. uh and

find logs. Uh we were already using Graphana um and most of our metrics is already connected. Uh Loki was a natural place Self-hosting today was not what was years ago. We had more stable build systems and uh deploying to Kubernetes was actually really easy for us. Um we deployed uh Grafana through Helm but we actually uh convert that into manifest and then um modify that through customize

to uh make adjustments. Uh we're not really uh we don't really love Helm that much. Um this give us full control over uh what gets deployed uh during runtime. Um we do our uh tuning from there as well. And the day-to-day operation is pretty straightforward. uh the real work was just tuning uh after and then so this is more of the reasons uh cost uh same single

pane of glass is a big important piece um and there's some mental models of we had to change uh like labels uh how it is very native to kubernetes um running elastic search that means no more tuning your JVM heap. Uh no more shard management. Um and mostly the read path is mostly state state stateless. Uh Loki scales pretty horizontally and just by adding more components. Um

self-hosting pretty easy um with just automatically adding some uh automatic scaling uh for Here are some of the trade-offs that uh we we considered for elastic search versus Loki. Um the tape every logging systems makes the same the fundamentals trade-offs. One of the side notes besides on the side that you have full uh full index uh full text indexing um like elastic search while every index is

filled in every uh log line that means you can search instantly in your elastic search but that comes with a cost. It's it's pretty expensive. Um on the other hand uh Loki uh it only index metadata um namespace log levels and handful of other labels that you you choose. Um the actual log contents gets compressed and stored in S3. Uh which is a big plus. Um and

the number one thing that trips people up is that here nobody talks about the labels actually maps directly to your Kubernetes topology. um which makes it more easier for uh engineers to already think about their services uh like just choosing the name space and pods um and once they make this shift uh their curious become more intentional and actually much faster and then here is like a

quick rundown of like what a Loki log looks like. Uh there's you have your time stamp uh labels and your selectors and then the contents of the log line. this the context log line is not um indexed and is compressed into S3 as chunks uh in which uh for Loki uh it only scans after you narrow it down by labels. Um and then so here's a quick

rundown of Loki architecture. Um basically you have um your right path uh distributor validates incoming logs. Um ingesttor builds uh and compressed chunks in into memory and then flushes them into uh your object store. The root path is basically the query front end uh that splits up large queries from uh graphana. So this is like going from the explorer or uh or certain uh logging uh or

loi panels that you use. Um and the courier front ends uh they also execute your log uh log log uh yeah sorry log ql uh against the chunks and there's different caching mechanisms like uh which is use mcache and the key insight is that the right path and read path uh scales independently uh so you can add more curers without touching your injesttors And this is what

makes uh autoscaling uh practical. It's much a more cleaner version. Um and this is uh how we start to wire things up. Uh before that uh there's different ways to install Loki uh either the monolithic uh simple scalable which is going away in Loki 4.0 O and distributed uh mode uh which is what you saw previously uh and that's what we uh have decided to uh use

um based on our uh log ingestion rate. I'm not sure you guys can read this but uh basically this is more like a zoomed in of um our our current setup. Um we we are using uh bloom filters uh which I will talk about later. Um but the short version is that the couriers in which uh the chunks that um to skip that you don't have to

scan everything. Um that makes it much more faster for bloom filters. Um which also uh solves one of our problems of uh wild card uh searching and um zooming out. Uh so now we have two of uh different logs ingestions. Um basically we were running uh we replaced our uh file beat uh which was kind of outdated um and we wanted some more flexible and that can

ingest more logs at a higher rate uh which is uh vector it's basically built in rust uh and doesn't do any kind of no garbage collections um and the the thing is that it can take different kind of sources um so we have we're running vector agent as a damon set and then we run also a vector uh aggregator uh to also get any other uh non-

kubernetes logs. Um our CI/CD is mostly uh in GitHub actions. So we export logs uh through that way. Um and we still use CFKA uh because that's help us to uh in case if we do do have issues with Loki uh it um it's still there to uh at least ingest the logs if uh there's any issues uh during the the queue. Um here's our playbook uh

of what how we started that migration. Uh we installed Loki and Vector uh on the day on the first day um with just minimal uh configurations uh of based of what we thought that would uh make sense um with some small tunings. Uh we use uh vector as a way to get metrics on how much logs we get. Uh I can tell you later there are some

weird um metrics that uh Loki emits that seems to be uh not uh incorrect on finding your total log and we use vector to also um parse uh our structure logs um which helps turn them into labels. Uh this is actually a way to like filter out what uh actual labels you want. Uh and uh we also drop some logs uh related to some some certain name

space that we don't want to be including. Um and uh vector it could be any other uh um log forwarding agent. Uh but we chose vector. Um, we still had to run side we had to run two logging um because we were still having weird uh we had some data pipelines in our old logging pipeline that was uh very dependent on the current setup and uh we

didn't want to uh touch that and migrate that into Loki because uh that causes issues with some of our our down uh stream consumers. Um so that's goes to like some uh data pipelines to like related to spark jobs. Um and then some uh weird ways of creating uh dashboards through logging. we um contacted our uh developers about this um change. um communicated out and trying to

reach out, find understanding of how we can change uh or stop doing what they were uh doing with the data pipelines using their log. Um we initially find out uh beta users um to validate that logs logs are working. Um and the feedback was very important uh because um there was a lot of upset uh folks about uh using Loki in in our initial uh run and

we we tune based we tune uh based on what we found. Uh there was a lot of people were still using our login vendor as like a dashboard uh to re that they have set up long years ago and it was still working. um we had to migrate that into graphana and then uh we created a lot of documentations of because since this was a big culture

change of going from elastic search to Loki uh which I'll explain later um and then we turn off our uh we eventually went off of our uh vendor. Okay. So, what broke? things uh on the first day um I guess second day of rolling out and having beta test users uh was not good. Um because they actually did uh test out our uh limits with Loki. um

going from a bunch of uh pods uming kill and um a lot of our uh clusters were going down in production and and this is during when we had we were still tested we were still testing out Loki but um we we also learned that uh searching in graphana explorer uh wasn't that good um during the time we were running on an older graphana honor version uh

which we didn't had the Loki drill down uh which was uh unfortunate. So a lot of folks were very uh confused on how to find their logs through the explorer. Um and learning logql was uh also a new thing even though some folks might have known or might know what promq is promql is but uh it's still also very confusing for folks and we had a uh

this is where the wild card search um started happening um where someone ran like a reix match to across all our name space uh and it scanned 43 terabytes and eventually timed out and um broke our cluster uh and eventually telling them that we'll just restart Loki and obviously that was a quick fix but that that that didn't make the developers happy. Um and this was a

common pattern across our uh developers uh when we migrated to Loki. uh the problem is in Loki is the Loki that Loki requires you to think about your query differently uh than elastic search and uh the message wasn't clear. Um so here's some like strip down of Slack messages uh that was sent to us when we first migrated. I'm trying to like Loki but it feels like

an order of magnitude of two worse than log zio and this is uh the first week. Um it's really really slow. I tried curing over 15 minutes and 3 hours and error it out. Did I mention it was slow? Uh it was it was painful and um people were searching uh build ids, request ids uh just through the wild card search and uh and then and then

that did not help out at all. Um uh there was production incident uh that um that arises some of the issues that we were seeing uh teams open up graphana during production incident and don't know how to write lo uh loio um mostly a lot of global searches uh because uh normally that's what they were doing in uh in our login vendor um the worst time was

to learn a new language age is during an outage. Uh from what we know from during postmortems, um engineers has been using our old uh vendor login vendor as a poor man's distributed tracing uh as searching for request ids through uh like our web servers uh globally. Um, Loki can't do that without specifying name spaces and you have to meet developers where they're at. Uh, to and

so this is where uh we started to think differently. Uh, while car searches are okay, it's never a bad thing. Um, this is like a fundamental mental model shift that that we see in elastic search. You can search strings across uh everything and the the results come back pretty quickly uh because there's like full uh text index but in Loki uh is not like that. You have

to pick your labels name space or log uh log level or a pod and then it finally uh decompress and then scans uh and then match your filtering. Um so I know that um we started migrating um a lot of the folks uh didn't read our like migration guide of uh lo to Loki and uh that was a big pain um and I think uh retros we

were actually uh looking better different ways of uh how to improve that and then another thing is that uh even though we migrated to Loki uh we thought that the cost will go down but it actually went up. Um we had a uh our a cost increase uh significantly. Um, and one of the reasons that we found out was that uh one of our ingesters uh that

was talking to our uh logging forwarder agent were on different as uh eventually we fixed that. So yeah, we're just burning money in AWS. um recently uh even though uh kind of contradicts this talk, but we also found out that uh we were we even though we set up Loki to how we thought it was um but we uh overestimated the resources that we set uh which

was also burning more money. Uh it's it's a onetime setup but in reality it was was mostly a lot of tuning uh when we migrated. Um and this is what I was talking about earlier. Uh we needed metrics to find how much logs we were getting. Uh and what one important thing is that this uh metric that Loki emits uh was actually uh giving us the wrong

right size for our cluster uh which goes back to um there's like a a cluster uh sizing uh that Graphana shows uh that we followed and uh that seems to be throwing us off based on this metrics uh in prod we were we were expecting that we We're getting four to five terabytes a day. Uh which is a lot compared to just two terabytes originally that we

expected and uh there was there were some big known issues related to this metric and uh it kind of uh threw us off a little there. There was a lot of this is the part where that no one kind of tells you um that you keep tuning uh Loki clusters. Uh we went through different tunings for the ingesttors. Um uh tweaking some of the uh like the

rate limits and all that and the compressions. Uh changing some of the uh data store. Um switching to different uh as uh originally we were running an on on demand because we thought that uh we didn't want to have uh Loki go down uh during spots. Um but that kind of um raised up the price more and then eventually we migrate migrated loc to spot instances uh

except for the memcache uh pod. tried to um there was a lot of uh tweaking with Grafana's uh data source uh timeouts and that helped us a lot um because some of the queries that the user were running uh they would take at least almost three to five minutes uh which is which is not good. Um and some part of that was related to their wild card

global searching and so we were trying to optimize that. Um and the big thing is uh using bloom but it would it helped us a lot with um trying to find uh like global searching like request ids. And then another thing was uh uh chunk storage tuning. Um we basically uh started increasing to flush more to S3 while the default settings uh was a little slow. Um

and then so there's this this uh right sizing uh table that Graphana has uh that I was mentioning earlier. uh it breaks down exactly how much replicas and how much CPU uh and memory that each Loki uh needs uh based on what uh logs you are ingesting. Um like I said, we started off with the default uh Helm chart uh values and slightly increased it um in

the beginning and that was during when we had beta users uh and so the load was not that crazy high. Um and we were still kind of filtering filtering down on certain name spaces um to uh just for like uh uh decrease cost during our beta testing. Um and then so the ingesters uh was definitely under sized when we started running. So we were hitting a lot

of uh s um ran out of memory. Uh and one of the key recommendations that Graphfana recom uh mentions is that when you you want to run small as possible for your carriers. So you can uh horizontally uh scale and then so we initially had a max of like um I think 20 couriers and injesttors uh actually 20 curers and then 10 for injusters and uh that

uh was the wrong way to do that. And so my advice is uh start sizing based on this table but uh be more um flexible on uh curing up. So we eventually start curing up the max is like to 200 I think uh but uh we slow it down to a half of that. Um and we were using uh horizontal pod scalers uh for couriers and injusters.

Another hard part is uh storage. Um there's a few knobs that really matters. Uh you want to flush frequent frequently and how often the ingressors push the chunks to S3. Um the more frequent flush means smaller chunks and smaller S3 get requests. Uh which lowers the memory Uh and it's also like a faster recovery when restarting. uh Loki uh we we tune more frequently for flushes uh

because ingesters stability matters more marginally with S3 um cost uh the mcach uh external store was a not a out of the box setting uh that comes with uh Loki Helm charts and that's something that we manually enabled um and it's also mentioned in like in deep in within uh Grafana's docs that This what actually helps a lot with uh caching. Um and we use a local

MV MVM SSDs uh as an overflow be for our memory. And right now we're setting around 550 gigs for our cache uh capacity per node. Um and that's why the chunk cache can't run on spots because it's going to take a while for uh it to rebuild. Um and then we we still running the single uh compactor that uh graphana recommends. Um and basically it just merges

small index files and uh enforce any retentions uh that you have set. Uh which is our retention was 90 days for Um this is a big issue. Um, we put pods and nodes as labels because they felt more natural as the metadata. Um, they're core their core Kubernetes concepts and of course you can filter down by pods. Uh, the problem is that every uh unique combination of

labels creates a new stream in Loki. Pod names are unique per replica. Uh and so a deployment with 50 uh replicas can create 50 streams on for every container that for every restart. Um during peak uh traffic uh autoscaling spins up a new pod constantly uh which uh creates new streams and adjusters to hold memory. Um and we were already filtering uh what labels should be uh

ingested through our uh login foring agent. Um but we noticed that uh once we migrated um pod node or um and request ID and build ID in out of the labels um this into structure metadata um using the bloom filters uh was the way to go. Um and the results was uh faster uh searches uh using like request ID and pod and node labels or yeah um

so bloom fizzers this was this was like the biggest uh I would say biggest lift on uh global searching but uh this is still experimental from graphana um this is how we were addressing it uh basically Um, graphfana says bloomfissers are based like finding a needle in a hay stack and it literally is uh this um even though request ID is too high of a cardality uh

so searching for that kind of uh request is really painful and slow and engineers really wanted to search through uh request ID um even though uh they were using as like a poor man's uh tracing solution But uh with Bloom filters uh it made it much easier to search uh that kind of uh ID. Um, and we had to manually increase the the persistent volume um for

each of the bloom components uh which helps a lot uh cuz we rerun it every uh day to um I think about eight hours uh to uh on those ids that we set. Um and then so this the combining with the structure um metadata for all the fields like the pods and quest request ID uh that gap was that was the gap between the Loki and elastic

search uh with developers um issues. um there's there's a few things that we also did. Uh um Loki also use it um especially the operational dashboard. Um and we also added um SLA uh or yeah SLAs's on top of our uh our and then so this is the example. Um so every incident I described like the 40 terabytes curry uh the carnality trap we debug them all

through Loki mission dashboards. Uh they're really helpful. uh we uh added SLOs's sorry not SLAs's um on top of that uh treating your logs uh PI logging platform as like a production service for our customers for de developers. Um and one of the great thing is that Loki comes with a uh a canary pod that can test your logs. uh we we didn't run that initially um

but it is useful uh if you're uh running and testing your logs um there there's a uh feature that you can enable through Helm. So was it worth it? Was it worth migrating to Loki? Uh and I would say definitely yes. Um we were getting more logs than what we expected. um we were getting about uh seven and a half terabytes per day compared to just sampling

down to two terabytes. Uh we removed all a lot of the alerts that were paging teams that they were going over their uh log usage. Um and so across all the tiers we were getting about 15 terabytes. Uh so we also like ingest unstructured logs as well. Um and that's go through our vector aggregator. Uh and about the the migration uh it's about 50% of the volumes

that increased it. Um and we were suppressing production logs a lot uh which was because of the costs. Um and so now uh it was way uh way better than we expected. Um and then so here's like a before and after. Um like I said, we sampled. Now we're taking all logs. Uh the retention is much longer now. Uh we set to 90 days cuz this is

what we had originally set uh for like kind of like certain audit logs, but now for all the tiers we set 90 days. Now uh we still are running Athena with uh the Athena pipeline with Loki. So we can still go more than 90 days in Athena. Uh if you like searching um and and of course uh we remove the the separate search systems. Uh now everything's

in graphana. Uh people have integrated graphana into uh our uh catalog. We use backstage. So a lot of uh the apps developers can just go to backstage and can see their uh metrics from there and logs. Um so it was a pretty nice thing. Um there was some uh I would say uh having something uh self-hosted uh was what we prefer more because we can control the

cost and uh make it more uh secure as needed. Um and so develop de developers were happy uh even though I can't share exact numbers but I can tell you that logging cost was uh it's now like a fraction of of what originally we were paying for uh and we ship specifically more logs and and longer retentions. Um the cost story isn't just that Loki is cheaper

but we ship all logs now. Um that that rule was really the big thing. uh we get more features out of it. Um if you're evaluating this uh the math works out roughly at par less than you're paying for your vendors but um there comes with the cost of that um making it usable uh we get asked uh so many times uh during the migration about how

do I find logs still and previously uh we created bunch of documentations and obviously no one reads that Um and so uh people still hate PromQL and now they have to learn logql. So that's even worse. Uh so what makes it uh what makes it better? Uh we eventually m upgraded our graphana and including all the nice features the drill down was the most uh like plus

positive thing with the developers cuz they can just go there and then search for their name space and then they they can explore from there. Um and uh but some folks didn't like that. Uh they would rather have like a centralized uh dashboard. So we created a actual dashboard uh with all the fields and labels that we included. And so the developers can just filter down from

that that way. And the logs is already uh formatted. The log the log line is already formatted to what they uh prefer. And then we include some metrics. Um, and then that was a big plus, but why not go more further? Uh, I mean, it's 2026. Uh, we're all using AI or most most of us maybe. Uh, and so there are some options. Uh, Graphfana offers a

MCP, uh, which is like let's allow you to have your AI assistants or LMS uh, talk directly to your Grafana. Um it works both locally uh on prem and cloud. Uh and one of the great things about that is that it can query uh logs and metrics uh through Loki uh any any kind of metadata uh like you can just ask uh my app this name space

uh is failing in production like how do I like can you find me what's the exact like error I'm getting and it'll just uh come with the actual uh response. uh instead he had to go through the graphana explorer. Um but there are some caveats with that. Uh we we had some legal issues with running uh certain like MCPS. So uh that was still on hold. So

we eventually um created a app uh triage skill for Claude. We're heavy uh usage of Claude. So we created a skill for that and uh basically let uh claude understand how we uh structure our uh labels and it can just uh create the queries for dovers to use. Uh they can use log cli or copy the queries over to graphana. Uh, another thing is that we also

created a Slack chatbot for our hackathon. Uh, and it has access to our documentations uh to all the Loki documentations and to our GitHub repo. Uh, and then so you could also like ask our chatbot like how it would uh how did they debug their app and it will return some uh pretty useful uh logql that they can just paste over. Um the only thing is that

we haven't connected to Loki to our Slack chatbot but that's maybe more a feature invitation. Um what we learned uh these are like the seven lessons that we have learned going through the migration. Uh number one is Loki is not elastic search. Uh the the shift Loki shifts the cost from writes to reads. The ingressen is it's a lightweight and cheap. Um but the keyword has a

has to like brute force the chunks. Uh training your engineers on this mental model shift before you like flip the switch or uh not and not after. Um labels are your schema. uh lower carality helps a lot. Um and originally we thought like pod and labels was a good label uh indication but uh when uh Loki introduced structure metadata through the bloom filters uh that made it

much way uh better for the delivers to search those kind of logs. Um three uh optimize uh right first. Uh this kind of counterintuitive um when you optimize your rights first larger and fuller chunks uh means fewer index in uh entries uh which means also faster queries. Uh but then that means tuning your chunks uh like configs uh and um you're constantly tuning to where uh you

find the right balance. Um and this was one of the biggest tuning uh tuning improvements that we did was the chunk cache. Uh four is the graphana sizing table even though uh it's a good baseline uh which we highly re recommend. Obviously not running default settings is not good uh in the first initial run. Um we were on defaults weeks for weeks uh during the beta testing

and obviously that uh killed a lot of the pods. Um uh one thing to add is that uh we we did have like LMS to help us constantly uh tune locate with based on the charts and it it seems to help a lot. Um so you like to pass that chart and uh some data about how much logs you're ingesting. Uh that uh made it pretty simple

to use. Um and then another thing is uh you got to budget for the over experience. I think that was the most painful thing for developers. Uh you you obviously don't want to hear that Loki is slow. Why do we migrated? Uh I think uh even though docs and dashboards wasn't helpful but eventually like tools like the AI LMS um that helped a lot um and the

migration could have been done differently uh so definitely uh debug uh in the beginning um with confidence uh and then uh this goes back to the chunk storage uh investing uh in the cash um when you cash hits cash hit rate is determined by your carry speed uh target over like 7% so cash hit rate uh you can find that through all the Loki's uh metrics they

expose and we have a dashboard uh on that I didn't show that um but uh we were using that as a indicator of when developers are uh searching in low key uh and we can see all the stats of how long is their run queries running and then like uh how much of the cache was uh how much it was hitting the cache uh which was a

good indicator to help debug uh Loki our Loki cluster and it goes and this goes back to using LMS uh is actually really nice with guardrails of course um it it helped a lot of developers to just ask claude hey like I need to look for this uh production incident and based on the logs that I'm And uh definitely helps a lot uh and they don't have

to learn lo um and that uh burden uh kind of lifted after that and so it goes back to this. So at what cost? Uh we saved significant uh money on our uh vendor licensing but we invested a lot of uh realtime engineering hours to get here. Um I mean we're we are a small team uh maintaining Loki and Vector in our logging pipeline. Um but the

real cost uh is that uh is really about ownership, flexibility, capability and uh predict predictability. Uh when a vendor a vendor can change when your quotas overnight but uh with the right tuning and troubleshooting issues uh it could take many hours with Loki uh reading docs. Um on the plus side or downside is that developers are uh unhappy uh with the slow uh changes that we made.

Uh and then eventually there was a log a steep curve for them uh of migrating to Loki and then eventually it kind of uh got better afterwards. So yeah, definitely uh it it uh took us in a rabbit hole of uh getting them happy and so uh so here's the journey like we started struggling costs was spiraling uh we weren't getting much logs um and we even

though we battled the cost of uh migrating over um we fought developers experience problems and we kept tuning and tuning uh we we came out on the other side with more data uh more retentions uh and more capabilities that we wanted uh at a lesser cost eventually uh and now our teams are building on top of Loki um that we can never done with a SAS vendor

um and especially with production incidents today uh a lot of uh like going back to the app tri edge skills uh it helped it helped a lot with uh Delver to learn at um and there's still more things that we we are uh continuing on with uh improving our cluster. Um Loki Graphana also mentions like using Kada as to autoscaling your carriers uh which is also a

plus. Um, bloom filters like I said he heavily expressed that it helps a lot with uh bridging the gap with elastic search uh when people are searching request ids. Um and yeah like the migration was the beginning and not the end. Uh and so yeah thank you a quick thing is that uh we are hiring for our team our infrastructure team. So yeah, any questions? Should I?

>> Oh, I mean, sure. It's up to you. >> Yeah, thank you. Uh, thanks for the great talk. Um, glad to hear it had a happy ending uh after the somewhat rocky migration. So, I think it sounded like part of it was like just learning how Loki worked and part of it was like um getting it configured for your guys' particular setup. And so, I'm curious like

if someone came to you and said, you know, we're thinking about using Loki or um what what advice would you give them so that they could maybe have a a smoother uh experience migrating maybe from a different logging solution? >> Yeah, definitely. Uh I would say that um you want to get like metrics in the first place of like finding the right metrics of how much logs

you're getting um and just the cost of uh I would say if if it actually makes sense to migrate off of your vendor. Um at first, you know, we were at where we had a deadline of our login vendor and it was basically just increasing cost more. Uh we we had to find the trade-offs of like how much uh Loki actually costs. Uh and we initially uh

I would say undervalue of doing that kind of cost estimate. Uh we had like a rough estimate of uh when we uh thought about Loki like running Loki in the cluster. Um and eventually uh the gotchas were like we we forgot to forgot to like change our resources requests and uh a traffic cost. Uh something that to be considered too as well. Yeah. Uh thanks. Uh question

do would you say that every developer has to learn some logql or is it more a situation like in each team you need maybe one or two people who can know their way around loql and they're going to build the dashboard so that the rest of the team can just find their way around like how does that look like? >> Yeah. Uh and initially I think that

uh when we first created documentations on like the the difference between uh elastic search and Loki uh we initially thought that learning loia was uh not not that bad uh but uh we found out the hard way and uh we find out that developers uh learning loia wasn't that bad uh I would say but uh having developers to learn and uh understand loql. Um they there was

a little frustration because they were used to doing running elastic search and they can just free text. Um I would definitely say that no like it's not necessary now today especially with LLM. uh like we I think we we are seeing more productivity with LM with users asking how to curate their logql and that helps a lot. Um and also another thing is that uh we have

like steps of like how to use Loki. The first step was go to the drill down uh which helps a lot because they don't need to learn lo and that's a perfect perfect place for uh to like discover your logs there without having to learn loql and then graphana within inside you can transverse that and actually build build your loql uh if you ever wanted to go

that route uh most of the time we see developers use the drill down or uh our custom dashboard that we built. built out for them. >> Thanks for the talk. Uh really really appreciate it. Really interesting. Uh what is a strategy that uh going back that you're thinking on? How could you have made that developer experience and that transition easier for your developers and get more developers,

you know, using Graphfana or more familiar with it or uh you know, just the productivity overall? How would you have done that uh you know before the full switch over and gotten gotten that adoption? >> Yeah. Um I would say that we probably did like two or three different lunch and learns and that was just before the m like the full migration switch over and uh definitely

like that didn't feel like it helped a lot. I mean, maybe it it spread the message around uh like doing those lunch and learns um and having like a hands down like maybe an actual workshop that we should have done like was to have developers sit down and like actually ask what they want. Um during Slack channels uh when we first had beta users, we we had

them had a list full of queries that they want to trans uh to convert from Loki uh elastic search to Loki and we manually had to like help and uh translate that for them. Um so I would say basically uh it goes down to I think spreading the message more clear. Um and like I said and probably workshops helps a lot more uh documentations they don't read.

Uh and obviously we don't read documentations and uh so yeah. >> Hi um thanks for the talk. Um, so how did you kind of how did you come about to like figuring out the more advanced configuration optimizations like the the memcache d for the chunks and the the structured metadata? Did you have to go you know to graphana labs for that or did you know did you

guys figure that on your own? >> Um, basically we eventually found it through graphana's docs but it was heavily like nested inside their articles. Um and so like I'm I'll share my my slides out. Uh but I did link out some references of of all the things that we have found and used and basically adjust. These are the tuning like the links that here are what we

tune based off of. Uh Graphana has pretty good um articles of how like they uh like scaled uh Loki uh different ways. Um and then there is like a link just for the mem cache that we found uh from a like a article uh that Graphana did. Um so yeah. >> Um and could you explain a little bit about how you do spot instances for the ingesttors

or whatever is that like behind an NLB or >> we uh we use Carpenter? Uh we use Carpenter uh which uh by default most of our our um services are on running on spot. Um but initially uh we we just ran on demand uh because we were still testing Loki and I guess that testing directly uh was still running on demand eventually when we went to prod

or not prod but when we did a cut over uh which we found out that we were wasting a lot of uh costs um but yeah uh we use carpenter to uh provision like apps and pods into uh spot instances is >> okay. And then but how does how do you like not lose log messages when they come up and and go down? So um the only

thing is that it's when uh we have uh we're using um what is it PDBs um which is uh helps with uh when when an instance gets uh evicted uh another uh there's still there's still certain number of pods still running um so Yeah. >> So may So maybe you do lose a few logs here and there, but >> No, we we don't because we we're still

running eventually we're still running CFKA uh which basically puts in our queue of logs. So our our uh logging agent ships logs through CFKA and then >> Okay. Is and is that documented like uh by graphana how to use Kafka in the pipeline? No, because uh we basically integrate it within our uh uh like web with our existing pipeline already. Um but yeah, it's it's not documented.

>> Could you maybe help us out with that >> Um yeah. >> Anybody else? Um, did your S3 cost surprise you? >> Um, at first, yes. Uh, because we didn't have like retention setup. Uh I think initially when we ran it uh there was like some configs that uh we thought we enabled for uh S3 policy life cycles like which is through the settings from uh Loki.

Uh so initially we did saw cost of S3 going up but then when once we enabled the config that is like set in the helm charts um it just dropped immediately. So, no. Um, so the cost was uh not as bad as we thought. >> That's encouraging. Thanks. >> Thank you. Um, were you at all tempted to go with managed Graphana Cloud or were you set on

self-hosting all the way? Um we were pretty self-hosted all the way. Uh I mean we we have uh we're running Prometheus Loki as well like so metric was already going there and we're already uh have sort of trying to transition to use Thanos. Uh so mostly just self-hosted. Um but we didn't uh we didn't I would say like consider uh Grafana cloud um or like any other

login vendors uh because we felt like self-hosted was the right way to go. Is it any more questions? Awesome. Thank >> Nice to meet you. >> I know uh Eric Louie used to work with him at Second Spectrum Sports. >> Work with Nick, right? >> Yeah. Yeah. Work with Nick. Tyler Lawson >> Oh, yeah. Awesome. >> Are you using? >> Yeah. Yeah, we're using every day. >>

Are you using on or >> um so we use graphic cloud? >> We already had graphic cloud a while ago. Our logging journey was elastic data cloud. That's >> Oh, awesome. >> Yeah. I mean like we I mean, but like we're mostly on on And we didn't >> Yeah. Seven day retention fees. That's brutal. And you couldn't even get like infome >> Hello. Does it work? >>

You want this one? Oo. Hello. Oh, that worked. I don't think anyone >> and then >> Oh, thanks. >> I think it works. No, it is. Hello world. >> Worst case, I'll grab that thing. >> Hello. Can you hear me? Can you hear? Oh, it's working right. Okay. >> Yours is >> Like the moon landing. >> You turn it on and off. You just turn it on

and off here. >> But if we switch it mid, it's gonna >> shut. >> Um, no. Other way. >> Yep. Yep. On top. Yep. But I'm not sure that one works. Hello. Hello. Hello. Can you hear me? >> It works, right? All right, I will go up first. All right, thanks everyone for waiting. Sorry for about the wait. We had some technical issues but like appreciate for

like everyone for to fix this. Uh so today we're going to talk about for this presentation we're going to talk about containers all the way down. uh what we learned running containers in containers for AI and more at Meta uh for about the speakers today. My name is Fong. I'm a cl I'm a production engineer here at Meta. I'm with the with the cloud foundation apps team

and this is my first job out of college. Been here for a year and a half. So far so good. It's it's getting a bit hectic but everything is fun so far. And yeah, this is Sean Hopper, my tech lead. Do you want to introduce Joseph for a bit? >> Uh, hey. Um, Sean, I've been at Meta for about eight years. Um, spoke at scale like three

times. So, it's good to be back. Yeah. >> Uh, a little bit of quick intro about our team, Cloud Foundation app. So, we run the cloud native stack at Meta. We got kind of like the bridge between meta and the public cloud, right? even though we have our own data centers but there's going to be some use cases like for the public cloud and we do OCI

containers why like the rest of meta is mostly on Lexe containers about the agenda for today so the first part would be about what is container we would try to deconstruct like it's a very very like you know like big ecosystem here and the second part we would focus about the usern the namespace and the rootless aspect of container which is will be like kind of like

the main primitive that we're going to look at today and we we will try to understand more about like what is the special thing about user name space and like you know how how is it like leveraged inside the container ecosystem and the next part will be kind of like the star of today's like presentation the container in container use case and yeah this is like a

container in container There is a very very like complex definition. I would say there are a lot of dimensions underneath of that. We would look into like the rootful and the rootless matrix and you know trying to like you know unpack a lot of the terms there and the last part will be like probably like the best part of the presentation. We will actually talk about like

you know the use cases that we have at meta and like how we solve them and what is like our approach Moving on into the first section, what is a container? I will try my best to deconstruct that definition. So first of all, containers are not magic. Uh to understand like containers, I will try to walk you through like a series of simple simple examples and some

definition to kind of like build the understanding here. Uh starting first we have the container formula like you know like container as a very high level. The first component would be the Linux namespaces which give you isolation. Uh Linux namespace is what going to like you know petition the kernel resources. So each process sees only a designated slice of the system. The second part would be croup

which enforce resource control. You would allocate it. It kind of enforced the thing. So you can like allocate, prioritize, limit resources, things like CPU, memory or IO for process groups. And the last one would be overlay FBS. The union file system overlays are like onions. They have layers. So like on so it essentially merge a read only lower layer with a writable upper layer. It mechanism. So

it ensures that what changed always stay in the upper layer and like the lower level will always be like read only. I think this is one of the thing that like makes like you know OCI docker container really successful because of like combined with this capability and it has a really good like distribution system through things like you know like with through like registry like docker hub

quay things like that. uh let's move in into like the first like you know the first example right like trying to understand like the the container definition so we have the pit namespace let's try to understand the pit namespace by running like a simple like workflow here let's assume that we are all like docker beginners like when you start with docker I guess we all do like

docker run enginex at some point in our life right so when you do that you essentially have launched an in a very standard enginex web server. It looks like a normal process but is it? Uh to check where is it running I would do two steps here. So inside the container I would do something like docker exact container like passing like the container ID and like do

ps inside the inside the container you would see that the engineext process is running as pit one. Okay. But when you but like moop out of that when we trying to do like ps on the host and try to grab for an enginex process the enginex process is actually running as a non it's a not like a pit one on the host right it's just like one

of the pit among like thousands of the pits on the machine so the key insight here is the pit namespace isolation the like when you use like pit namespace the kernel going to create a separate IPS like you know like the process ID space inside the container and the process inside the container thinks that it is pit one but it's actually you know like yeah it just

it's it's just like actually like you know like this one of the process among like all the process on the host the next stage you got the container running you want to do things like curve maybe like local host port 80 to kind of like see your engineext page right but with what you just did in the first step you cannot see the page Right? You might

ask yourself, I can reach my enginex uh engineext page. How do I do that? So, we would do something like docker run-p mapping port 8080 to like port 80. Uh some the command like this one. So, essentially this one going to create a tunnel to forward the traffic from the host port 8080 to the container port 80. So, the key inside here is the network namespace isolation.

the container would operate like it has its own dedicated network stack isolated from the host. So even if like you know inside the container like the engineext the engineext process is getting like listened on port 80 but it's not actually like you know listen on the same port 80 of the host. You have to like map it out. And the next one would be the mount nameace.

Okay you got the container running. You got the concur like the engineext page but now I want to serve my own HTML page. Let's say that I create a very nice like hello world. HTML, but like when I curl my enginex like container, I only see like, you know, a default page like from the from the '90s. And in order to like customize your uh your enginex

page, you would do something like docker run-v and map like you know the pad of your custom html file into like you know the engineext container like HTML default HTML page. So essentially when you do the command that I show here, you are leveraging the mount nsp space isolation because the container sees like you know its own view of the file system kind of like isolated from

the host in order to like expose the files from the host to the container. You have to do something like Last but not least we would look into croup. So for example like you know we are trying to like you know we start serving traffic to this enginex container right we get a lot of traffic we probably don't want to like overload the machine making the machine

blowing up because of like you know the enginex process eats all the resources so this is where you probably want to leverage croup which is a kernel feature to allocate you know and limit resources right in this example uh I'm limiting I I'm trying to limit the the memory and the CPU usage of the container. Okay. So to sum up from the examples we just look at

there are seven pillars of Linux namespaces. Um the first one uh is the pit namespace which target like the process ids. Uh what this means for the container is that each container will has its own view of the process tree and like the process inside the container would completely blindly to the host processes. It thinks that it is it thinks that you know it is its own

thing and the second one would be the network name space. So each container will kind of have its own like network stack its own like you know pod space routing table etc. And the next one will be the mount name space. Uh so each container will have its own view of the file system essentially you know it will not like you know have the things that on

the host unless you explicitly you set it up and the next one would be croup for resource limit and the last one which is probably like you know the most interesting one we have the user namespace which target the UID and the group ids uh what does it mean for containers uh this namespace going to do the UID and GD remapping So for example the UID0 inside

container it doesn't necessarily the same it's not necessarily like the same UID zero to the outside of the container and this will be like you know a very very important primitive that we're going to leverage in the later stage and besides that we have the UTS namespace and the IPC namespace I also like want to like take a quick quick step back to like look at the

whole ecosystem up OCI container or like docker container if we say that uh because of like you know like a lot of this we're going to like mention in later stage I just want to like you know go over really quick so the lowest level we have the Linux kernel right we have the Linux namespace cgroup seccoms capabilities of processes and above one level we have the

lowlevel runtime things like runc or c run from red hat those are like you know the binaries that actually like you know in like interface with the kernel to create namespaces, setup croup and exact the processes and above that we have the container engine things like docker or podman CLI right those are like you know the higher level API to manage container life cycle interacting with the

registry and one more layer above we have orchestration layer uh you would leverage this like you know things like kubernetes docker swarm when you want to run containers on a large scale in a distributed manner, right? So, this tool helped you to manage the life cycle scaling of the containers and helping you to like set up the connectivity of the containers across the nodes. And this is

it for kind of like the intro into containers. I would yield the floor to my to my colleague. Thank you. Uh is it working? All right. Sweet. Um cool. So as Fong mentioned, we kind of covered the the basic uh namespaces and croups required to create containers. Um but we wanted to focus on the user namespace specifically because it enables um a different type uh of Um,

username spaces themselves, as Fong already mentioned, uh, pretty much just remap, uh, UIDs and GIDs, which sounds boring until you think about the implications. Um what it really enables you to do is map UID0 in uh a lower namespace uh route to just some higher order uh UID like on the host or in our case when we're nesting these containers uh what we just call the parent

namespace. Um it's not always the host. The way the UID mapping works comes down to two files really uh Etsy subid and Etsy subg. And the the files are very simple. On the left side, you have a Unix name that's not like Shopper, that's S Hopper, that's me. Um, and it basically says like SHpper can uh take UID in this namespace, which is probably the host, but

when you're nesting these can be the parent namespace. Uh, 100,000 and then 65k extra UIDs from that range. Um, and it works the same for groups. The reason we give each user something like 65,000 is because we want to allow that user when they create a username space to su to whatever uh set of users are mapped down there. And as we run nested containers, something as

simple as like running the Ubuntu container off the shelf will have uh an Ubuntu user in it that's probably UID 1000. So, they already need 1,000 UIDs just to start the container. And depending on whatever uh is in that container, like we don't know what the Etsy password file uh is inside there. So, 65K is just kind of a number we start with to to start giving

out uh UIDs. Um where it gets really interesting beyond just creating the the username space is that when you create a username space um you actually become root in that username space. So we have the mapping set up uh as mentioned in the previous slides but you also gain capsis admin and if you have capsis admin you have the ability to then further create other namespaces. So

all the namespaces Fong was uh walking us through in the intro, network, mount, PID, etc. Um you have the ability to do that inside of that user namespace. And it's really powerful because you don't have to be a privileged user to create a username space. Um this is going to show up later in like our multi-tenant environments where we have many users and none of them have

root but we want to give them the access to create uh user namespaces and thus um other namespaces within Um, so I was kind of alluding to the fact that we were going to start nesting these. Um, but in order to sort of explain the trade-offs and how to do that, we sort of have this uh matrix we want to walk you through because it it's it's

going to start getting complex really fast. Um, so we call this like our rootful rootless matrix. And uh we'll kind of go through what we consider the most dangerous way of running Docker to the most uh secure. Um on the top left in red, we have probably what if you run Docker you're doing when when you're not thinking about it. Uh you might have like the Docker

Damon running. It's probably running as root and you just Docker run pull something off the shelf and you don't pass a flag that like overrides who the user is. Um, in that case we call that rootful, which means like the Docker Damon is running as root and you're probably root inside the container. And since you're root inside that container, um, if anything goes wrong, you're also root

on the host. Um, this is kind of like where we started with Docker, but like over time as new capabilities u sort of emerge like better practices kind of uh fell out of that. Um scenario B is the same except uh when you start the container maybe you don't start that container as root. So at least that process running in that container is uh not going to

be root but you know if if a privilege escalation happens in that container and you can become root you're you can effectively have the same privileges as root on the host uh if you get out of that container. Um the next column C and D are where things get uh considerably more secure and also like enable a lot more creative uh use cases uh especially when it

comes to nesting. So in scenario C uh we have the docker damon which is not running as root. So this is what they call like rootless mode. Um where the Docker Damon itself is running as an unprivileged user and the containers that are started from it are all considered rootless containers. They all uh heavily like you have to use user namespaces to create those containers. Um but

you can still be root inside that container. It's just what we call usernamespaced root. Um, and that's nice because if anything goes wrong and you do escape the container, you don't actually have access to root on the host. And then scenario D is like the most mature version of this. Um, if you're gonna run an application and you kind of know like like complete principle of least

privilege, what it's going to do, what it will always need to do, uh, the best way to do this is you run the Docker Damon unprivileged and then you make sure that the container you're running is also running as an unprivileged user. Um, in in our case, we mostly stick to scenario C because the things we're doing inside of containers are highly interactive and even userfacing. Um,

but scenario D is better for like, you know, your your stateless web app where it's you're just running one process inside of a container and there's no need uh for anyone to be root whether that's username spaced or not. Um, yeah, kind of alluded to this like why are we even getting into this? Why are we taking on all of this complexity? Um, why are we going

through sort of the headache of figuring out like are we a username space route or not? Um we kind of have two standout major uh areas of focus which is we want to better support researchers in our multi-tenant uh HPC environments where things like research and AI training happen and we also want something to sandbox agents um either in those environments or not. Um and in the

next slides we'll kind of go through both and how they intersect. So here's some case studies. Uh there's a lot of headache we learned along the way and we kind of tried to distill it here uh something you could take with you without having to go through it yourself. Um just as a a precursor when we talk about research clusters uh we're usually talking about slurm clusters.

You don't have to know what slurm is. Just think of it as um it's a scheduler that allows users to isolate and share resources in like a cluster environment. Um it's like heavily common within academia, but um we we use it too and we run it on top of containers. Uh the thing we'll focus on the most is what's in pink, which is the login environment and

that's sort of those uh very interactive and like userfacing containers I was talking about earlier. Um, here's a zoomed in view of what we call a login pod. And as I just mentioned, like the the login pod is the um the interactive environment. It's basically a dev server for like a researcher. Um, and we chose to completely containerize the research clusters uh using Kubernetes. um as platform

engineers like to us there was value in uh putting everything on top of Kubernetes making everything a container things became portable um predictable and things like that. Um but what happened was we sort of like made the abstraction so uh transparent to the users that they also wanted to run containers. So if we kind of like go through the the nesting of what's happening here, we have

like a node in green and that's a Kubernetes node and then on the node we have a pod and the pod is a personalized um environment per user. So like each username has their own pod and in that pod we have a few containers and that's what you see in blue. The main container that like researchers and users interact with is called the SSHD container which as

no surprise is running SSHD. And that's the thing that um they they actually interact with. Um there's a few other containers in blue in Kubernetes. It's it's common to run multiple containers in a pod. They share certain name spaces um and can communicate. Um and for a while it was just the blue box or the blue boxes. Um, but then they wanted to run containers themselves because

isn't it nice to just pull uh Reddus off the shelf and run it without having to think about what RPM to install or Ubuntu package or building things from source. Yes, it is. So, we got to work figuring out how and that's what we have in yellow. So, those uh boxes in yellow are usernamespaced uh containers. Um, one thing to to call out here is that in

a research cluster, um, NFS is ubiquitous and the whole like idea is that as you SSH into this cluster, um, you kind of have access to a supercomput and like any node you jump to, you have the same um, access to the file system, you're the same user and things like that. So, it's it's kind of like this illusion of one big computer. And the reason that

comes up is because the NFS being everywhere has significant consequences when you're nesting username spaces. Uh and with that we'll get into the the first uh case study we have which is um that Etsy subuid file we mentioned earlier that maps users to uh ranges of subordinate UIDs um doesn't mean a lot when you have NFS in the picture uh and the reason is NFS is not

username space aware so what that means for us is if each individual login pod has an Etsy subuid file. We have to make sure that the UIDs we allocate to each user do not overlap in any way because if we do uh it basically means they can start a container su to a common user write to NFS and we won't really have a trail of who did

what. Um fortunately we caught this as we were testing uh rolling out nested containers. Um, but this would have been quite a headache for us to figure out down the road. Um, there are other options in NFS to get around this. Things like Keraros uh seem to come up in discussions. For us, taking on the complexity of Keraros wasn't worth it. So instead we just came up

with a UID mapping scheme where we um just multiply the user's UID by how many UIDs we want to give them sort of map them into this uh int32 wide name space and now we're good. Uh so we we know for every set of you know 65,000 UIDs uh who could have done what on NFS at that point in time. Um for us that's better than kerros.

If you have experience with this and kerros I'm interested to know. Um as mentioned this is the fix. We have disjoint subuid ranges. So instead of uh what you see on the left where like Alice and Bob might be allocated into either the identical range or an overlapping range, we just have to make sure that uh they never overlap and then things are fine. Uh another use

case we had for running a nested container in that login pod was jailing the agents. So uh we started with cloud code but it's the same for all of them whether it's like Gemini or Codeex or whatever. Um the idea is like these agents are super powerful. They have access to the internet. They have access to your codebase and they might have access to sensitive data. and

having access to all three at the same time is sort of a recipe for a disaster because of things like prompt injection attacks. Um yeah, so we don't want to allow these agents to have access to all three of those like major uh areas of concern. So we came up with a way to shove the agents into a nested container tried to limit uh its access to

two of three of those. The way we did it is we have podman running inside of the login pod. And really uh what what what happens is when the user like invokes the agent, let's say it's claude, uh it actually invokes a wrapper script of ours that will invoke podman which will create uh a clawed container. And before that container boots, uh we went through in the

previous slides how once the user namespace is created, a network namespace is created. Um we actually just run an IP tables rule that says you know absolutely no traffic can leave from this network namespace except through let's say port um 443 on this specific IP. That IP happens to be the inference endpoint for the agent. so that the agent itself can talk to the inference server but

cannot talk to the internet to go fetch arbitrary data or anything like that. At the same time we have a mount name space so we can bind mount in the user's you know home directory and things like that from NFS. Um and yeah at that point uh claude can then start um it could even start as root. We don't do that but it's username space route so

you know it it's probably not a good idea but it's not the end of the world either. And um we have slurp for nets which handles all the uh username space and uh rootless networking um that we set the IP tables rules uh to. So, if you don't know what slurp for nets is, uh it's basically a project that allows you to set up network devices and

um like v pairs and things like that, uh at least it pre presents the illusion that that's what you're doing uh without needing root access. So, since all of our users in these clusters are just themselves, they they never have root, whether that's user uh namespaced or not, um at least at the pod layer, uh slurp for nets lets us set up the network the way we

want. Uh the lasta uh case study we have inside of the login pods is not only do users want to you know pull containers off the shelf with podman or run the agent uh via podman indirectly once they have podman the expectation is they can use podman to build containers as well. So like in testing we had very simple docker files that just did you know from

x copy y to y and okay the container is built and we thought we were done. Uh well very quickly after uh we tried to run a more complex build. Pretty much any build with a run instruction will fail with this error. Mount proctorock oper operation not permitted. And it was kind of a like devastating wall to hit because if you think about all the permissions you're

supposed to have, you're like, "Hey, I'm a user in a user namespace and I'm root and I have capsis admin. Why can't I create a new PID namespace and allow proc to mount to proc?" Um, and the reason is not obvious, but it's because the kernel says you can't. Um, again, we're running inside of a pod in Kubernetes, which means we're started by the cublet, and the

cublet is very opinionated. And it says, "Oh, I see you're starting a container. Let me helpfully bind mount a bunch of junk over proc for you just in case you ever become root and try to do something dangerous." Um, and that's good as long as you're not nesting containers, which we are. So, we have a different problem. uh which is we need a way to have proc

that isn't masked. So after many like long nights trying to figure out what to do, we randomly tried bind mounting /rock to just a random path which we've called pure proc. Uh and the trick here is to make sure you don't do it recursively. As soon as you do that, all of the sudden like you can create pit namespaces and your run instructions will work. We don't

know what we didn't know why that worked for probably a year, but it went uh it went to production because it worked. After some time, we did figure out what was happening. Basically, the kernel has this uh functionality called mount to revealing. And what it's what it's checking is is is this mount like from process going to be too revealing? And you want the answer to be

yes, but usually it's no if you're inside of a pod because the cublet has helpfully bind mounted a bunch of junk over proc for you. So by nature of creating this bind mount to literally anything, that function will then say, you know what, you would have been able to figure it out anyways because you didn't recursively bind mount all the junk with you. So now you can

create that pit namespace and things that seem like they should work actually begin working. Uh that's it from within the login pod and I'm going to pass it back to Fong to talk about some other cases that uh are not in the the HPC clusters. >> Thanks Sean for sharing about the HPC use case. Those are definitely crazy. I guarantee this one will be less crazy. Uh

so we have seen like you know how we do like container and container for HPC use cases and now we have a different one for non-HPC uh which will be built building containers at large scale. So the case study here is that at meta specifically my team we have a centralized container build platform a share service for internal team to build OCI container images without managing their

own build infrastructure. Our customers just have to define three things to us. The first one would be the build config. Uh tell us what are the internal packages that you need. What what are the dependencies that you need? How much like resource that you need like how much disc how much memory and what is target architecture. We support x86 and ARM. And the second one would be

the Docker file which is like you know the recipe for an Docker container, right? This is like pretty obvious. And the last one would be how how to trigger your jaw build. There are two ways. We have the internal CLI for ad hoc builds and we have the CI/CD system for automated production builds. Uh for this slide I would walk you through kind of like the high

level architecture of this built system. So TLDDR it start from a local built contact assembly to a remote execution build happening on AWS EKS. So let's start from the left side. So inside meta internal the developer would trigger a build job via the CLI or the automation. The container built system would assemble the built context pulling the built inputs from our internal VCS and from different artifact

resources. For example, our blob storage and then internal RPMs and then it will kind of like you know like bundle everything into a tab and it would ship that table to S3 and then on AWS we have an EKS built cluster. The build system would start a Kubernetes pod on that EKS cluster and that in inside that EKS pod uh the pod would pull the built context

from S3 execute the build and publish the output artifact to our container registry which is hosted on AWS ECR and the special thing about this E is this Kubernetes part is that it runs something called build kit which is which is an open open source tool for building OCI containers on Kubernetes uh a bit of history lesson here. Before BuiltKit, we were using a different tool. It

was called Kiko. It servers very well, but we were hitting the ceiling. So the Ko GitHub repo was archived last summer back in June 2025. So no more patches. We needed a tool that could keep up with our scale and the requirements. And built kit was like you know the best choice uh for two two main reason. First of all it would be the speed. Uh built

kit allows parallel execution right. Built Kit would transform a Docker file into a grab and stop it concurrently while Kenika would like go through like every step inside the Docker file sequentially which is pretty inefficient. And built kit leverage like native kernel overlay AS to so that it can detect like instant file system change instead of Kiko which would like you know kind of like brute force

the file system walking through like the file system after like every run command. And the second and the second second aspect here is security. Uh built kit offers more like you know deployment model. Uh it offers like rootful username space and rootless mode while ko only offers rootful mode which is not that great and we this will be like a very important part that I will jump

onto later on. And also like another thing is build kit enforce stricter dockify semantics. So okay we have mentioned built kit the built system where is the nested container here. So built kit is actually is actually where this nested container model happens. So like I mentioned we run build kit inside a kubernetes pod right which is a container and inside the build kit container it has a

process called the process of the built kit demon which essentially will create nested sandbox a new username space for every run instruction in the docker file. So essentially like you know like you know every time like you know you do the docker like you build a docker file right you run you embrace like a run command like that run command will be get executed inside usern name

space which offer an extra layer of isolation and build kit leverage run c which is like the lower level container runtime to create the sandbox on the fly and yeah and like I mentioned there are different like security models for deploying build kit rootful root list and user namespace and we pick user namespace build kit uh for multiple reasons. So to run to to use this model

we set on our Kubernetes postpack host user equals false and privilege to true and host users to false essentially like you know would run your container would run your part in username space part username space mode and we have this diagram kind of like break down what happens. So when you start the contain like when you start the pod continuity would create the name space which is

use a name space like 1 2 3 like the the yellow box here and then like you know it will create the pod the build kit part and inside that pod we have a build kit demon process running inside the pod uh the UID the UID for that build kit process would be zero but actually it's just a fake root because we are running the container inside

a usern the name space it get mapped to the host. So on the host like the actual actual UID of that process it will be like you know a nonroot UID for example like you know UID 100,000 something like that all right and then like when uh when the build kit demon execute a docker file it runs some commands right so every time it bumps into like

a run a run instruction it will create a usern name space and the run instruction uh like you'll create a username space like you username space 456, right? And then every time there's like a run command, the run command will get executed inside that little like username space. So even though like the run command like UID inside the username space is zero, it is actually like a

nonroot UID to like you know the pod to like the build kit part and then like two and then you know it still have another layer of like mapping to the actual host. So there's like you know two layers of isolation here which is pretty great. Uh yeah so before like ending this section I would love to walk you through like some challenges that we hit along

the way. So the first part is that when we were like you know evaluating this situ like this solution we look at we also look at rootless build kit because of like you know ruthless is kind of the dream but it was too tricky I would say we look at the rootless build kit like ruthless on like the pod level like doing things like run the pod

as a like a non-root user we look at this but however it requires disabling the process sandbox so essentially what what it means is that you would run the build kit demon on the same username space with like you know whatever like run command that you would execute. So technically if you get like a malicious dockify like you know that command from the commands from that malicious

dockify can break the docker like can can break the build kit demon and like you know do bad things which is not that great and another kind of like you know another model we could think about it's like the rootless end to end right maybe like run everything rootless from like the cubelet level all the way inside this is kind of like you know very very like

this is kind like you know I guess this is like the dream war right but however this would require a lot of like note note level runtime config and kubernetes feature flag changes unfortunately we are using manage kubernet distribution EKS in this case which is not that easy to do that I know that EKS you know doesn't allow people to use like alpha or beta feature I

think they only allow GA under certain requirements as well so yeah this let us to pick like the user namespace model and the second one would be the dockifi compatibility. So when when we were with Kiko everything was rootful. So you were root technically you could do anything not too much like limit for you but when we move to like use a namespace model things would change

like you will no no longer allowed to do like you know operations on privilege file system like proc or cis and like you know we have seen cases of people doing like a very like open grab or like a file command in the docker file that touching like the proced file system which will not work when we move to build kit. The solution would be you know

like yeah we we kind of have the feature gate for people so they can switch you know they can like decide whether they want to move to the new build kit model or they stay with Kiko and do like the rest of the migration in the meantime. So yeah this should be this last slide in our presentation and thanks everyone for listening. I know it was a

lot of content and thanks autocale organizer. Thanks Meta and AI for making this presentation happen. >> Do you have any data you could share on what were the performance gains that you saw from switching to build kit from Kiko? Uh yeah, we actually we did some benchmarking internally like uh I would say like you know we have seen some cases like for example like you know like

we we mostly focus on like you know the kind of like the built time kind of like thing. So we have some giant container which would take like an hour when you use like you know vanilla ko right no tuning I know there are some tunings for ko but then when we move to like you know vanilla build kit it would be down to like 20 minutes

which is great I think most of the bottleneck is like how it deals with like the file system because like Kenneo kind of like traverse like the whole file system checking for like the changes and that's that's part is pretty inefficient. Yeah. >> Yeah. that >> um could you talk a little bit more about how you provision NFS within your cluster you make it available everywhere? >>

Yeah. So is this Yeah. Um it's simpler like running NFS is a pain no matter what but it's really just NFS for us. Uh the the biggest like trick we have for doing it within Kubernetes is we have a basically a little chrome job on every node that is constantly looking for new exports and and mounting them and then we bind mount like the highest parent into

the pod so that the pod never needs to restart. So as the exports come and go, as long as that parent mount was bind mounted in when the pod started, um we get we get a lot of flexibility that way. But as far as how we provision, um another chrome job somewhere else is just basically reading a config file saying, "Hey, it's a new export for me

to make needs to be this size on, you know, this shard." Um and things like that. But it's pretty vanilla NFS for us. Anything else? >> Um, it's both. So, in some cases it's four fronting V3. Um, but most of the time it's V3. >> This might have been answered by the previous question, but one of the things u I run into with like running a scalable

like build system is like managing the Docker build cache. So is that handled by the NFS layer? >> Uh you >> uh in our case, so we had caching with Kico and it was it's already faster with build kit to not even have caching. So we haven't put a lot of investment in it yet. But in that build cluster, we don't have NFS. I think Fong has

ideas for how we're going to do caching though. So like in terms of like you know like like in our build system we actually like you know like you know that like there are some manifests right we actually like do some manual stuff with like the manifest. So in order like we were using like docker v2 manifest or something with kigo it was happy with it but

I think when you move to like build kit I think you have to move to like OCI manifest something like that. There are some changes there. I'm still like you know like that's like kind of like the plan for like next to like look into it and maybe do the migration so that like the cing would work end to end but overall like build kit is already

like pretty efficient so like you know Yeah. >> Oh yeah more. So is your system like engineered for arbitrary level levels of nestedness? And secondly, do these researchers that you described as end users of the system like use arbitrary levels of nestedness? >> Um, no. It's it gets really complex really fast. So they just go one layer deeper most of the time. >> But I have done

it as a concept. It does. It will work. I've run podman and podman via the podman container in the login pod just to make sure everything worked and it did. So, >> does your security concepts get repeated >> Yes. But it gets so complex it's like impossible to figure out what's going wrong. So, >> Yeah. like the it's basically just like the pod is started rootful by

the cublet and then the users are never root and then from then on it's all username spaces down. So if you have the stomach for the complexity you can make it work but you're never getting real root anywhere like that boundary never gets crossed. >> Uh how many inner containers do you have inside one outer container? How many containers in what? >> Ah not many but it's

also up to the user. So since every login pod is like a personal dev server they can start. I mean, as long as they don't fill their temp FS tens, 20s, it just it's really up to them depending on like what compose file they ran that like started those containers. Yeah, >> we probably have time for one more if >> All right, thank you very much. >>

Cool. Thank you. Turn it on. Flip. Wait two seconds and >> you'll have you'll have audio. >> Okay, good. Um, let me get started. Um, I'm Tony from LinkedIn. >> Directly in front of it. How about now? How about now? Is it good? Okay. Okay. Hello everyone. Thank you for coming to my talk. Um I'm Tony. I work at LinkedIn and working on analytical databases and I

like anything in between databases, distributed systems and yeah so on. So today's topic is zero downtime Kate's migration of our 14K Apache Pino nodes at LinkedIn. So how many of you have heard of um OLAP? How about Pino? How many of you have heard of Pino? Okay, two people. That's I'm going to start with what Apache Pino is and how we use it at LinkedIn and um

there are two prerequisites that we implemented before we start migrating. So those two are the availability zone aware shard placement also known as aware shard placement and containerization. So to run application on container you have to containerize first. So there's a second prerec and then u I'm going to dive into the migration work orchestrator itself. Uh to eliminate the human toil entirely we designed and implemented a

m migration orchestrator built-in temporal. So I will go over the design and the implementation last. Okay let's start. What is pino? Pino is a real time distributed opp data store. It's built for low latency and high throughput analytics. So how we use it at LinkedIn? Um first you might have used it or at least know about it a little if you have used LinkedIn before which is

the member facing and enterprise use cases. Um probably the most well-known example is who view my profile. It says who's view your profile for same thing. So when someone checks your profile, it shows you how many uh people check your profile last 90 days, 30 days, 7 days, you can uh filter differently. But this is powered by Pino. Another use case internally at LinkedIn is the internal

products in BI. Uh this example is we use Pino to basically check our growth for the last one year or two years. So these two main use cases are um how we primarily use a LinkedIn. Um we don't have to know all about the Pino architecture but I'm going to go over really uh surface level at least. So let's start from the top. When people query Pino

they query broker uh it is the routing layer. So this is the reading path starts and we go down and this there are two separate path for right the one on the left is the real time it's what we call real time you can think of Kafka or Kinesis there are other flavors of this but when you want the real time update in Pino this is how

you insert data I guess this is how we ingest data from real time another right path is the offline path it's from the right is from the offline sources and there's some kind of offline job you can use to ingest pen into the offline servers. So we have two different types of servers depending on the use case. For realtime path we use realtime servers. For offline u

use the offline servers. You can pick and choose which one fits your use case. And this green box here is zookeeper which we use for metadata storage which is connected by our controller which also does other uh admin API as well. So I'm going to talk a little bit about the scale of pino at LinkedIn. There are roughly 300k QPS and 10,000 serving hosts including realtime and

offline. And there are 2,000 broker/outing hosts and we have I think currently over 5,000 Penino tables. This is slightly outdated. So this is the scale we operate at LinkedIn for Pino. Um so the question is how can I migrate 14 stateful hosts? And a very reasonable person may say can you just turn off LinkedIn for two weeks and it's easy. you just turn off. So the hosts

are not used. You can migrate and everything's great. Um the real challenge here is we can't do that. So how can I migrate 14 staple hosts with no downtime? That's important. And we also want no query latency impact. Before I reveal the secret of how you do it, I'm going to go over a little database 101 in case you are not familiar with databases or um you're

new to tech. So I'm going to start us with shards and replicas. Think about this table. Um this I just named the students. It can be anything. And there are a lot of data. There are 80 terabytes of data for the students table. and it has data from grade one to six. Um the problem of this is it's kind of hard to operate when you just have

a one big table. It can go down anytime and you just lose access to your data table. So one technique uh is known as sharding. You it's kind of how you divide data. Instead of having one big from grade one to six, one strategy you can think of is how about just have a different shard per grade. So you have grade one, two, three, four, five, six

uh in a separate shards. Each shard has a different data. So grade one has 13 terab of terabytes of data and grade three has 16 terabytes of data. This is kind of natural because sometimes I mean data is different based on the actual uh characteristic of the data itself. So in this case they're slightly different but now you achieve this separation. Then what is replicas? Replication is

something you do for u if you want high availability. So instead of having one copy the primary database on the left, you can just have the exact same copy and you can have another exact same copy. So in my example, there are three exact replicated across uh yeah replicated across three. So there's grade one and there's also grade one. There's also grade one. These three are the

same exact data holding the same exact rows. The only difference is now you can scale. So instead of going queries to this one single table, you can go to the copy table one or it can go to the copy table two. So you achieve two um distinct goals with replicas and sharding. Now if one go down let's say copy one is down because the underlying host has

issues it can still go to copy zero which is the primary table or it can go to copy two at the same time you can serve more customers because now you have let's say three distinct hosts serving more queries than before so you have essentially have more capacity okay with that in mind I'll go to the uh the next the fourth prerequisite which is availability zone aware

shar placement. Have you guys heard of a also known as availability zones? It's pretty uh common concept in public cloud as well. This example I just use a random data center in California and there are four different zones. Each zone is physically isolated. They have their own com they have their own uh power they have their own networking fraud. So in theory one zones outage should not

should not impact other zones. So if one zone A goes down it should not impact zone B, zone C or zone D. Um okay simply enough. So the question as a user who put data into the data centers and as it's how to put shards into this um different zones can I just put like this is one strategy you can think of I'm just going to put

grade one in zone A grade two in zone B grade three in zone C and so on. So there are four grades with two replicas each. But another way you can think of is you can kind of distribute so that in zone A we have grade one and two. Zone B has two and one and C three and four, four and three. So the advantage of the

right side uh the placement of the right side is if one zone go one zone goes down you still have one up replica. So zone A goes down can you query grade one? Yeah you can just go to zone B. Same for grade two you can just go to There may be slight latency impact but at least it's not fully down. But for the first case, if

zone A goes down, you cannot query grade A at all. So this is uh important when you are placing shards into. The only problem is for bigger table we have 10 K shards, 40 replicas and 10 A's plus. So the problem is a lot more complex than the toy example I just showed you. And this is just one table. We have um 5,000 tables at LinkedIn. There

are kind of three variables we can control. One is a the other one is host and the last one is shards. And you can think of uh there are two distinct knobs. One is how to put shards into hosts. The other one is how to put hosts in as let me start with the first problem. How to put shards in hosts. I just used 12 hosts as

an example and we have grade one to grade eight. The strategy we are using at LinkedIn is we just group four hosts into one group. So we have group zero, group one and group two. And this is how you think about groups. One group serve all all grades. So group zero serve grade one to eight. Sorry, replica group replica group zero serve group one grade one to

eight. Same for replica group one. They serve group grade one to eight. The exact placement of shard into host can be different but at least one group serve all grades. So replica group zero hey I serve or grades replica group one hey I serve all grades and same for the replica group two. So this is replica group and the only problem there is it's not predictable. So

here server A has three shards, server F has three shards, but server G has one shard. We want to store shards predictably into The strategy we came up with is think of it as an array. This Python, Java, doesn't matter. Um within a group we start with index zero, two, one, two and three because there are four servers per and within a index all servers serve the

same shards. So index zero the pink mirror server is what you call it. We all have the same shards. We all all have the same data. Index one, server B, F and J, they have Same story for index two and three. So if I actually put shards into this mirror server sets, it looks like this now. So mirror server set index zero now has grade one and

eight and horizontally they all have the same shards. This is what I meant by mirror. It's kind of mirror each other. Same for the red mirror servers at index one and the purple and blue. Now we can put shards into host predictively. This is this was the point of mirror server set. That was actually just the first knob. Second knob is how to push host into we

keep the same mirror server set but the only thing the only additional constraint is the a within a mirror server set within let's say pink mirror server set there should be no overlapping a for safety so here I just put it zero one two no overlap same for the red mirror server set 0 one two. There's no override. why do you want to do this? It's similar

to the toy example I showed you. Let's say a1 Are we losing any data or is there any data not queryable? Not really. With this strategy, even though one a is fully down, the data is still available in other servers. So for the first row pink mirror server set server A is down but query can still be served by server E or server U This is why

uh we combine the a diversity with mirror server set. Oh by the way this uh is not tied to one tables one table. So far the example was the students table but it can apply to uh multiple tables. So student table, courses table, staff table, they're all served by the same um replica groups and if one go down it's still safe because there are other copies available

in other AS. Yeah. So this is why with a diversity and mirror server sets one a going fully down is safe for all tables served by a cluster. Second prerequisite to run a container containerized application we need to containerize. There are multiple SKUs at Pino internally and LinkedIn. So this is how we manage it. So there are more than four types. These are the four main We

use the Kubernetes label. So for the first example is Pino server, 64 GB of memory and SSD. When there's no um SSD size, we assume is 3.2 2 which is the standard SSD size at LinkedIn for Penino and look at this example the blue boxes now the memory size is 128 but it has a lot bigger disk it has 6.4 four terabytes SSD. So we label them

these two as well and we use node affinity. So depending on the workload requirements, we can just use a different skew and if their characteristics change, we migrate to a different skew type also using the node affinity. And before migration um Pino, TML, LinkedIn used to have bunch of puppet models to do different things. We had a kernel setting. This case for the M map, we had

the max map count. We also had the SSD mounting logic, NFS mounting logic, engineext configuration. There are probably more in this new world. Before migration, we eliminate the needs for uh a lot of kernel specific settings. For instance, we introduced M advice feature in the OSS pen code. We used the persistent volume for SSD mounting and NFS mounting. And for engineext we move to an embedded we

we embedded into the container image itself. I want to get back to the architecture a because it will help us to find the correct migration strategy. Um remember the there are four main components. We have broker server real time server offline and we have controller which kind of controls the whole coaster but not all of them are stateful I'll say broker which is the routing layer it's

um pretty much stateless they accept queries yes and there are routing layer but they don't have any data on controller as well. They are also helix controller. They're managing uh other pino components. So broker server, they're all managed by controller. They also have some admin API to rebalance or uh create a table, delete a table, but they do not persist any data on disk. So they are

also kind of stateless. Realtime server, they host charts and they serve queries. They do persist data on disk and same story for offline So broker and controller they are stateless applications. So we can use a migration strategy that fits stateless application. For real-time server and offline server they are stateful. They have data on their disk. So we use the stateful application uh strategy for migration. One thing

I haven't mentioned but very important is host maintenance or any kind of maintenance. This it's new for the Kubernetes world for us at least Kubernetes is aware of the pot health and the host health. One thing they are not aware of at least for our use case is the the partitions the shards how the shards place into hosts is something Kubernetes is not aware of. So we

built some um there my colleagues built ACM I think it's called application cluster manager. So what they do is for deployment uplift host swaps all these different applications someone will ask ACM hey I want to stop server A and B they're unhealthy or we found a hardware issue I want to stop so ACM talk to the to sh states in zookeeper so it kind of check are

there enough replicas up um if there are only one replica up if you take this down there's no more replica and query is going to fail. It also verify the enough replicas up before stopping server a check sorry second step was checking all the states. Third step is verify if there are enough and after checking step two and three ACM is like hm we can stop server

B but you cannot stop server A because there not many replicas are up. This is the role of ACM. Uh there's a actually another talk by my colleague at a coupon. You can check their I linked in the slide so you can check their talk Okay. So we covered two prerequisites before we starting migration itself. And how do we actually migrate? We use the which is kind

of a um because we have 5,000 tables and I'm sure I'll make mistakes if I have to migrate 5,000 tables manually. So um this orchestrator is built in temporal and it's fully automated and then um it's designed to like continuously retry if there's any failure in the process. So there I mentioned migration strategies a couple times when I talk about stateful versus stateless. Um this is what

I meant by the strategies. You can either do in place or out of place. What are the differences for in place? You take down one host. I use server A as as an example and use the same exact host in Kate's stateful set. So the data is not moved. You have the same exact disk. So you don't have to move data with the in place migration. For

out ofplace migration, you just throw a bunch of servers in Kate's. So my Kate's deployment has four new servers. Server E, F, G, and H. and then you just take down the legacy VM hosts. So there's if there is data in this legacy VM, it's lost. They have to be replicated before they get deleted. There are some pros and cons between these two migration strategies for in

place. They don't require spare hardware because they're moving the exact same hosts and they don't require data movement. It's the same exact data SSD, but they're slow. You have to do one at a time or couple at a time. You can't just turn off everything and move. And it's also tricky to handle bad hosts because it's data involved and you lose the data during the migration if

there's a bad host. For out ofplace migration, it's the mirror polar opposite. It's super fast. You move the whole thing at once or you know a few steps. And handling bad host is easy. Just replace it. It's a new host. But the problem is they require spare hardware compared to in place migration. In out ofplace migration, you needed the same number of hosts in Kubernetes. and they

also require data movement if there's any data in the legacy VM this is why for stless applications we decided to use the out of place migration strategy for stful applications we use the in place this is all great and the question might come up how do you ensure the safety and performance during migration because there may be performance degradation in between. This is where the aaware sharp

placement from part two becomes handy. Now we know if you take down wall one whole MZ it's safe. There's no impact. There's no latency impact. So what we can do is even for imp place migration take down the whole MC. So one two three four take down four servers at a time. So you just need three iterations for migration to finish. So take down a zero, make

sure they're and then take down a S1, put it up in the Kubernetes, make sure they're up and then move on to the next In terms of performance, we build the validation in the orchestrator itself. So for each host migrated, we check is the host healthy and we log validation. Are there any new errors from this migration? is the host um ingesting data just like before. Finally,

we do the metric validation is the query latency in expected range is the CPU usage in expected range and I believe there are a couple more uh tests we do for each post. finally the mini part. This is the temporal workflow we built for in place migration. We let's say you want to migrate the students table. We gather workload requirements for student table. It can be oh

yeah I want uh 24 virtual 64 gigabytes of memory and I want six So based on this requirement we write u helen values and we commit. There's a separate migr the manifest apply pipeline. So the next work the next step we call it uh temporal activity it just keeps pulling and wait for the manifest application is done. Once it's done we trigger the in place migrator. So

this is what it um so going back to the previous slide it's pretty much do the same thing it just go by go a by a take down move it to kubernetes make sure they're good and move on to the next a constraints we put on just to um be safe we only migrate during business hours and And we also have other constraints if there are any

active incident that's impacting latency or availability we don't migrate. So we had a constraint there as well. And once everything's done we just add cluster as a deployment target. So for our next deployment it will be included there. We have a similar temporal workflow for autolex migration. That one is a little simpler than this. And the beauty of this, this orchestr can migrate many tables concurrently. It

doesn't have to do one at a time because they're uh all served by different clusters. It's safe to be migrated concurrently. So in this example, we migrate this central orchestrator triggers some workflow to migrate students table, but it also trigger workflow for child workflow for STE table and parents This is just a screenshot what of what we used. The main orchestrator workflow has a lot of child

workflows. So it shows 215 u tile workflows. This means at some point we migrated 215 tables under this Okay. some lessons learned in the future. It's kind of counterintuitive, but you will save migration time by spending more time prepping. Um, we actually planned and implemented the design for I think almost a year, maybe a little less than a year and that saved a lot of headache during

the migration. the mirror server set concept and uh containerization. We also did a bunch of improvement in the codebase itself. So they actually save the time for the actual migration of three different data centers. Oh, you have to be careful of deploying during the migration because during migration you temporarily have less replicas and if you are not careful you can take down the cluster. some feature penod

stacking support is not there because of some complication. So this is what we are actively working on as well as the kates operator to um manage pen clusters. So we want to rebalance and uplift downlift we want all these things to be done by pino kates operator in the future. Okay that's it for my talk. Special thanks to my teammates Dino Kja and my managers who made

it happen. Sha Sid and Omar. Thank you. Uh so in the ACM part that you guys did sign um it it mentions that your infrastructure is separated into different servers that kind of share different charts. But in terms of that, how does it makes the decision of I need to shut down server? I can shut down server B, but I cannot shut down server A. Like what's

the decision between why not A or why not B. Um because in terms of shar placement it could be the case that server A has I'm going to use the students table example as let's say grade two and server but there are already um deployment going on or some kind of operation going on on other hosts serving grade A sorry grade one. So if you take down

server A, there's no more copy of grade one anywhere. So you cannot stop server A anymore. But server B, maybe all the other replicas are healthy and up. There's no other operation going on. So it's safe to take down server B. So in this case server A is not safe because of overall replication status but server B is safe because it also check other uh replication status

of sharing the same shards. Are there any other questions? Okay. Thank you for attending the talk. Give it a couple Check. Check. Good. Yeah. Hi all. Uh my name is Shinathan. I'm from a company called ZAS IO. Uh anything as a service input output and I'm here to present about uh building an AI factory with pure upstream opensource projects. This talk is not about uh you know

creating an large language model or doing post- training or pre-training. This is purely a talk for building AI I have four points which I will cover today. First um we'll talk about why we think uh you know there needs to be an uh integrated opensource AI factory and we look into the architecture of the AI factory which which we are going to present today and we will

look into what's the hardware infrastructure utilized for spinning up um this and you know this demo and finally we will just look into the data flow pipeline of how you know from an object storage we are able to push it into an generative AI or an agentic AI without using any publicly available APIs right um so first thing um if we look into the evolution of infrastructure

platforms Um before I begin, I just have one small question to the audience. How much how many of you guys here use AI in your own infrastructure? Quite a few. Great. Good. So uh one of the things which I've been observing is uh from 2008 till 20145 there was this virtualization era where projects like OpenStack cloud stack open Nebula eucalyptus created a framework for cloud computing you

know base uh virtualization and infrastructure as a service. Then for the cloud native era right Kubernetes came in. Kubernetes set standards for microservices and you know all the good things and that continues to be the date but one thing which I've been uh observing we've been observing is in the AI native era there is no Kubernetes there is no open stack you just have uh fragmented tools

like uh cubeflow mlflow ray a VLM and many more. But there isn't an integrated orchestrator for AI native workloads. And uh we strongly believe uh this AI native orchestrator should be similar like what Kubernetes did for cloud native. There needs to be a new kind of a Kubernetes for AI native workloads. That's the you know at least that's the thought, right? So with that as a thought

um we decided you know why not we build a reference architecture and build the reference architecture with only upstream projects and glue it together and see what it looks like and one of the things which uh you know I'm very uh you know let's they uh focused on is you know how is this going to get evolved over a period of time because uh we have to

uh take into consideration Kubernetes kind of platforms were built by um consumer consumer web gains or big tech companies like Google who had uh you know tremendous amount of exposure in working in webcale computing but uh as an uh hobist as and uh you know start a midsize organization you do not get the exposure of what big tech companies get but uh fortunately uh we've been working

on you know kind of a mid to large scale systems especially when it comes to cloud native and virtualization services. So that wisdom has uh made us think you know let's try to do something here and I'm going to just uh present a reference fabric. I'm not just going to just do a theory here. I'm we as well have deployed a small demonstration and I hope that

demo is going to you know as well work well. So to start with um we are having a unified control pane. Uh so this we have built by using a open uh uh you know by using cloudscape. Cloudscape happens to be Amazon web services uh user interface. It's Apache 2 license and we have taken that and for the core infrastructure we are using Kubernetes which is running

on top of KVM. Uh KVM itself is orchestrated with open Nibbla and I you'll be surprised why we are using open nebula on the next slide. The hardware footprint is very very minimal. uh keeping harvest in mind and uh third thing is um you know we are not just speaking about an AI uh serving infrastructure but we are also looking into data because if you don't have

data you don't have proper AI and especially people who practice private AI knows data is the most critical component to make an AI work. So we are having a data lakehouse based on Apache iceberg which runs on top of CHF S3. Then uh for the uh AI platform and MLOps we are using cubeflow, mlflow uh feast uh and we are as well using lang flow and v

code. Uh then we have embedding and future pipelines. Then for interference we are using VLM and VLM is connected to open web UI. Then uh for the vector database we are using Mulvous along with semantic uh retrieval. You can always use PG vector that's not a problem. Um then we have tried to integrate all of the tools with a key clock single sign on engine. So you

do not have to maintain different credentials for different systems rather you have a single unified SSO binder and if you're an enterprise uh you can connect it to your active directory system or your directory services wherever they are running and finally we have an observability system which is basically based on graphana and promeius and the whole idea here is uh you know there are point tools which

are great but we have not seen an integrated stack and uh we are just trying to push uh the data stack and the AI stack and the orchestration and management pane as a single open-source AI factory. So that's the idea. Now that being said, the demo which I'm going to show you in the next few minutes, it's running on just two machines, two servers. That's it. Uh

obviously, it's not going to give us give the best results, but this is good enough to understand uh and build the integrated stack, right? So, we are using HTN. Uh again, uh we are not using any hyperscalers. uh we are only relying on alternates uh because there are two reasons uh I love hyperscalers because uh learned a lot from them and they contribute a lot to open

source but at the same time uh I'm not able to afford them because the cost at times are is very high and um uh and and and you know you don't get to own anything after a period of time you know it's it's a subscription it's like grid uh so when it comes to data when it comes to AI uh I think uh especially when it's you

know in different parts of the globe uh sovereignty matters ownership matters right and in order to um uh you know embrace those values we have intentionally uh looked for systems uh which do not have anything which only have the base and and then you're building everything on top of it. So uh here we have two nodes just two nodes uh one is running in um uh you

know in a data center in Finland and other one is running in a data center in Germany. Both are in isolated uh data centers because they were cheap. We got it in auction and and and we just want to push the limits of software engineering and see where it goes. And I think we have achieved something. Um so um we have 64 uh vCPUs and 384GB RAM

and 2TB of SSD and JBOD. Uh this HCI node fundamentally runs a complete Kubernetes uh uh workload and the Kubernetes workload itself is split with uh KVM cluster which is orchestrated with open nebla. Then we have uh the fourth node which is joining in Germany in uh and and we were forced to take this node because this was the only node available at that cost. Uh so

so there's no 10 gig here uh it's a pure 1 gig uh van bandwidth and uh remember we are in van uh so anytime anything can go wrong so forth we block the complete internet created a secure wire guard uh VPN tunnels between both the environments and uh this uh fourth node happens to be one of the Kubernetes worker node to the Kubernetes cluster a stretch cluster

and uh we are having it with an RTX 4000 SFFF ADA GPU. It has only 20 GB of VRAM. Uh but that's good enough if you're a uh you know developer and harvest. You can do a lot with it. Uh and the base operating system is based on Ubuntu. uh uh but uh we can as well use uh rocky Linux, Alma uh and many other um and

the virtualization platform use here is open nela and the reason why we use open nela it's very lightweight uh to manage uh power on power offs we can obviously use proxmox as well uh or we can even use lib without any of this orchestrator but this is convenient for us um so all of this end of the day is powering up uh cube flows service uh MLflow

service. We are as well running a VLM service which does the uh model serving. Uh we have a lang flow. Uh the vector database is on malus. The AI chat interface is on open web UI. Uh the data lakehouse is on iceberg. Uh container orchestration is on rancher. We using uh rancher kubernetes engine. Uh the storage is on root ch. Uh networking is done with wire guard.

Uh identity analysis management is key clock. Uh the monitoring and observability stack is Graphfana and uh the only thing we have built here is the uh unified console uh which fundamentally uh wraps all the different opensource projects as a single layer. It has the capability to do orchestration as well. But this platform itself uh you know it's still not fully matured. Uh we are still building it

and uh we want to release it to the community and have the community drive it further on. Uh so now uh this is how the flow is going to look like. Uh um uh if uh you know you've been using kind of tools uh when you upload any kind of documents uh it fundamentally stores the document in the Olama memory right uh but the memory itself is

not persistent. uh now if you want to let's say upload uh one pabyte uh file storage uh to a kind of system and it becomes a little bit challenging. So that's where we believe the current AI factory framework which we built uh creates a difference. So fundamentally we have an object storage where you can have uh any number of you know uh documents or uh content both

structured and unstructured and there's an injection pipeline and there's an embedding generation created which then injects into the vector database. The vector database itself is queried by a VLM interference engine and then which is presented with the open web UI chart interface and we will look into the live demo of it what I'm trying to mean here. So now with this being said, let me uh go

to my demo environment. I have pre-opened many of the windows just so that I you know I I I genuinely uh not trying to stage the demo here. I'm uh just uh trying to ensure that you know this is how it's been working. And unfortunately because we are running with very little resources at times the API calls are too slow to respond. So that's why I pre-opened

the windows. But nevertheless uh we are going to have this uh code published uh uh with Apache 2.0 license so people can test it out in their own infrastructure and give their commits. So uh you can see very clearly uh we have a uh so the first thing is this we have a landing page and this is completely built on cloud cloud uh cloud space and uh

once I click on uh sign into console uh it fundamentally get goes to uh key key clock service and because key clock is already been authenticated that's in the cache and uh you can see here uh we have a similar hamburger menu and if I click on AI factory and click on AI factory dashboard uh it provides me all uh you know the all the uh links

to connect to different services like MLflow, Langflow, R Studio and so on so forth. Now for example let me click on domains uh let me open the workspace uh so you can see um here I have uh let's say running instances ah here you can see so you can see there are u different instances running here there is a VS code running there's a jupyter lab running

so on so forth now similarly if I click on malus uh you can See there's two instances of malus database running. So if I click on default it opens the R2. So everything is integrated here similar like how you work in uh a typical hyperscaler. Um now similar like the malous we have the VLM. So if I click on VLM here uh we have the complete uh

model which has been used for this VLM. Uh we can as well see the configuration of it. Uh then we can as well see the manifest of uh that VLM so on so forth. Uh then uh we have uh the um open web UI uh which is here which is opening now. So this gets loaded up. will take some time but I have preloaded um the the

uh open web UI here and I've done you know done some uh chat with it as well. Uh so let me let me just show you a few examples of what I'm trying to um just give me a second. Uh perfect. So if I show you uh the connection string of this open web UI uh if you see the settings and if I click on connections you

can see uh we are not using any public APIs. There's no public APIs, right? And um all these APIs are connected to uh internal Kubernetes uh endpoints and uh internal Kubernetes ingress. That's it. Uh now if I go to my chat, let me try to zoom it back to normal. Just a sec. Perfect. Here we go. Okay. So uh I was um just uh you know having

an conversation uh with the uh langraph engine which is as well built in here. So let me show you the langraph flow which we have built. Uh so this is the langraph flow which has been built. Um so let me try to zoom in here. So there is uh injection workflow and there's a retrieval workflow. So we are fundamentally um capturing the data from an object and

that is uh having a PDF reader in it and then there's a text splitter and then uh we are having some um embedding generation happening where we are using BGE small model then we are injecting that data into malware. Now if I go back to my chat interface I just um put a quiry towards the langraph rag model asking it explain AMD GPUs. Uh it's not a

generator to AI. It doesn't have general intelligence. It's it's pointed it's a private AI. it's pointed towards a specific data set and we intentionally have it in this way. I'll as well show you a generative AI example uh with the QIN model shortly. Uh so you can see here the provided context does not contain specific information about AMD GPUs. So it's fundamentally saying um the CHF S3

backend which I'm having is not having any of AMD uh documentation. So what I'm going to do is I have already searched for certain set of AMD data sheets. So I'm downloading that. I'm I'm putting it in my downloads folder. I'm going to my S3 browser. Uh you can see we have no AMD data sheets here. I'm going to upload a I'm going to uh upload this

uh MI350 GPU. It's uploaded right now. Let me go back uh to my browser and I'm going to go to the VS Code to this is add to then. Yeah. And I'm just going to run two commands. This can be completely automated, but I'm just trying to showcase you the flow. So I'm initiating the injection pipeline, right? um uh in in real world is is to be

automated but I'm just showing you the flow and this injection pipeline is the lang lang graph uh you know lang flow pipeline which I showed you and you can see right now the injection is starting and because we are running with constraint hardware the injection happens through CPU not GPU the interference happens in GPU we're running just one GPU so you can see uh uh it's starting

Now uh let's just wait for a couple of minutes. Uh now in the meanwhile uh I have um Grafana Promeius monitoring system but unfortunately in the last couple of hours it's not been working but I have created a backup for it uh where we can see the performance of the GPU um through the CLI and you can very clearly see uh the GPU memory is uh almost

filled up. We have 20 GB of uh VRAM. 18 GB is filled up. Um the GPU power the the the GPU cores are still not used. Now uh let's let's let's see what's going to happen once we put the AMD query. So let me go back to my browser. Let me go to VS Code. Here you go. So, it has started to execute the job. You can

see there's the Yeah, you can see there's the AMD PDF and it has been it is as well having a 21 chunks here. So 40% is complete. Probably in next minute or two it should get completely done. After we are going to uh inject this data into feast uh it's a feature store. Then uh let's go to the open web UI and retry the same context which

we have given. um the same um set of software with a larger hardware and we uh you know let's say um 25 gig or 40 gig uh network should you know should should should give you realtime results but we are using one gig band and you know very limited hardware so please bear with me and I suppose uh most of the people who want to do testing

and genuine software development and engineering um you you we have limited resources so we need to be a little bit frugal so u here we go so the injection of this thing is done now I'm pushing it to the um feature so feast is getting applied and documents are getting updated as park boom it's done it's successful Now chat. I am going to just regenerate query. Now

if I go back to my CLI, you can see the GPU is right now running at 99%. The GPU is fully utilized. So I'm not That's the message I want to tell and I promise I'm not doing it. The intention here is what u OpenStack did for KVM you know Kubernetes did for containers we intend to do for AI. That's it. Now something is happening here. You

can see GPU's uh load has gone down but I am expecting some result. So previously it was not able to give us the answer. Now you can see it is able to give the ML350GPU answer and it is as well pointed out through which document it is giving the answer. Right? So uh this is one of the possibilities of uh building an AI orchestrator. uh I think

uh it's high time uh you know some of the community members gather together and think about how we can build and uh opensource AI orchestrator and opensource AI factory and and and and you know uh virtualization uh all through the market category was created by VMware uh by large OpenStack had a big say in its from day one has dominated cloud native. I hope someone in opensource

is able to dominate AI native as well. That's that's the intent. Yeah. So, uh with that being said, uh you know, I'm thankful for everyone of your time and thanks for, you know, coming here to hear me. And this is mainly possible because of a few engineers who have worked tirelessly and uh and put their you know additional time beyond your professional time to have this done.

Uh Vijay, Dignes, Vishnuard, Gokul Raj and Gokul Krishnan and Vishwa. Uh they have been pivotal in building the stack. Uh and it took us a couple of weeks to get all the things. Uh but uh we had to burn a little bit more hours to have it done. So that's it. Thank you all. That's it. Yeah. >> Thank you. Does anyone have any questions for the speaker?

I'm not an AI expert. I'm an orchestration expert. >> This is not a AI question. Can you tell us about the auction that got you your servers? >> The auction you said >> hster it's a German uh u alternate cloud platform. >> Okay. Thanks. >> Thanks a lot. >> Yeah. Because uh there are many uh engineers who are specializ in post- training, pre-training. I do not do

that. I only do the orchestration of it. >> and thank you for the talk. You mentioned that uh you're planning to open source this under the Apache license. Is it when can we see this code and get access to it? >> Our target is to deliver that within next 90 days. We want to do it within 90 days, probably within 30 days as well. >> Anyone else

have any questions? Hi. >> Yeah. Hi. >> Oops. Uh, I understand this is a you're just starting this, but are you already planning on it to be kind of for multi-tenants? So, >> y is is that already part of what you folks have created? >> It has to be multi-tenant and as well as a role-based access control kotas as So, um, for folks who already have pieces

of this infrastructure, the the reference infrastructure that you had today, are you also planning on on this being able to be plugged into already existing infrastructure? So, we have Seth, for example, we also have rancher, we have, you know, a lot of the things that you were talking about. Can we leave our infrastructure as is and and put this orchestration layer on top of an already existing

infrastruure? That's a that's a great feedback you have given. This is the architecture we need to work on. Uh because uh by default if you take Kubernetes or OpenStack they are meant for uh green field deployments. Uh your ask here is uh let's say I'm an enterprise I already have a brown field. Can you guys integrate with our stack? Uh that means uh we should have an

orchestrate you know like cloud management platform. we should have a uh API only assess you know either you deploy a green field or you discover a brown field so that's something we can build I think that's very possible >> thanks >> anyone else >> great thank you so much for attending the uh talk and please give another applause to the speaker >> yeah thank I don't know.

Check. Check. One, two, one, two. Hey. Hey. One, two. Check. Check. One, two, one, two. Check. Check. >> Yeah. No. But >> no, I think it was him. I honestly think it was just the way he was talking and like the what he was do. >> Yeah. You remember him? It was like crackle. We had Okay, good. Sounds good. Okay, it is 6:15, so I'll go ahead

and get started. Uh, my name is Justin Fry. I am a senior data engineer working for RX Benefits, and it's randering in case anyone's wondering. Um, just a quick intro. Uh, I'm a forever CIS admin. This is considered or I've been told it's sort of like a junk drawer cis admin. Basically, you have multiple skills in multiple areas and wearing multiple hats. I started my career as

a SQL Server DBA. Uh technically a QA engineer and then eventually moved to SQL Server. Uh I'm actually from the Chicagoland area, but I moved to Boyisee in 2024. Um so still Midwest native and I created a Postgress health check function um that I had a talk on previously yesterday um that is an easy to deploy zero dependency uh Postgress health check function. So I tried to

make this funny uh as much as I could uh mainly because the entire journey that I went through uh was very comical at least to me because at the end of the day um I basically did a full circle. Uh but I thought the best way to describe it was to mirror it off of the five stages of grieving. I know that sounds kind of dark, but

it's really not. It's it's supposed to be uh kind of something funny. So just kind of to go over that denial. This is just another resource to deploy. There's stage two, anger. Why do I need three providers just to initialize a database? Three, bargain. Bargaining. Uh, if I just add one more provider, I'll be able to do Depression. My module's too complicated and I am the only

one that understands it. And acceptance. I'll just build a single-use blank database deployment. So, that should be a little bit of foreshadowing. And then, bless you. at the end uh there's kind of some uh quick lessons learned and uh something very ironic I guess uh when we'll talk about that. So the scenario I worked for I currently work for a healthcare company. So uh as DevOps was

ramping up they wanted to try and basically restructure the entire CI/CD pipeline. And so they started to work with infrastructure as code. Um and so for whatever reason they decided let's use the database admins to start with infrastructurees code. I feel like that was very backwards but I'll take it. uh I had previous experience as uh from another company building and deploying with infrastructure as code um

but it was very bare minimal so we took on the task and said sure we'll do it the stipulation was that the team our database admins would own it the modules themselves but DevOps would do all the deployments so we were just basically uh blessing the modules and then giving them to DevOps to do what they needed to since uh we are an AWS top we wanted

to create two uh basic modules one for my SQL and my Aurora MySQL and one for Aurora Postgress and Postgress RDS. We already had you we were already using Anible for initializing our database. We actually used uh Ansible automation platform or Ansible tower. Uh so we knew we could at least bootstrap that part of it into the pipeline or at least we hoped. And then uh it

sounded pretty straightforward. And after I had come out of this meeting uh I went to my mentor who was just my manager and uh he laughed and I said what's why you're laughing. He said we did this exact same thing at the previous company and it basically ended up with them creating a Python script that deployed all the resources. And I as somebody who doesn't like to

be told it's not going to work, I said no, let me prove it. So, with that said, we'll go into the first stage of denial or the first stage of uh grieving, the denial stage. I'm denying the fact that I'm being told I can't do it and I can certainly do that. So, has anybody deployed or looked at any of the providers with uh Terraform and Open

Tofu? Anyone have experience with that? which shouldn't be a surprise, but some of the bigger providers like AWS, uh, GCP, Azure, most of the documentation that you'll find is great. They've got great examples. You might see some obscure, uh, references, but ultimately, you've got a clear path on how to deploy the module. Uh, however, Anible is not as straightforward. Uh, as someone who is very familiar with

it, it it didn't take much time. Uh but if you look at the provider uh there's very minimal uh examples and it actually requires multiple resource blocks to just to execute a um playbook. Uh we already required we already had a runner created because of anible a or the aap. Um, so that was already something that was going to be included in our workflow and uh the

flow itself wasn't a flexible enough that for our CI/CD pipeline at first we explained okay uh this is what we're going to do to the DevOps team. Uh and unfortunately that is really small but I wanted to map out that ma the main core part of the deployment was going to be the my SQL version or Postgress version. we would create a KMS key for the encryption.

Pretty straight straightforward. And then for the DNS record, we were going to just create a route 53 uh CNAME just because it was a lot easier for the developers to uh use that for endpoints when they uh implemented that. Then we would have a new module for Ansible playbook. And then we had a secrets manager tool uh that had a provider uh that we would use to

create the user uh credential and then pass that into the playbook. So uh we presented this to our DevOps team and they seemed very confused. Uh to me it felt straightforward. Um, we're going to deploy a database, encrypt it or use encryption key, create a DNS name, initialize it, and use a our already existing secrets manager. Uh, what was really disappointing was, as we said, the other

modules didn't have or other providers didn't have great documentation. Uh, this was no different for our secrets manager documentation. It actually uh just provided examples with zero uh words. There was zero descriptions. It was just here's a resource block. Uh, and this is an example. We didn't have any um parameters that with descriptions of the parameters and it was really disappointing. This isn't a secrets manager tool

within AWS. This is a complete thirdparty tool and I'm just not going to throw shade at them. So, I'm leaving the the name out of it. So, lessons learned on our the first stage of uh grieving. Uh, a AWS providers are polished. others providers not so much. You're going to get what you get. What works also locally on our CI/CD uh locally might not translate to CI/CD

pipelines. I didn't mention this, but we were already just building uh we were building this deployment method from our local machines. We were not using a CI/CD pipeline. We were just working on it locally because we wanted a proof of concept and uh that will bat us in the butt later on. So, enter the next stage of uh grieving anger. Uh so, there's a quick diagram I

wanted to put up and of course it's really small so you might not be able to read it but what it's saying is uh we wanted to map out the different types of scenarios that we would get from uh when somebody would request a deployment. Um so just a quick recap we have one provider for deploying the database, one uh provider for initializing the database and then

we have one provider for creating the credentials for the database. But our DevOps team came in after we displayed the proof of concept and they asked what about restoring from a snapshot? It's like um we didn't think about that. Uh it's pretty straightforward but that means there's going to be more user uh input at this point. And then there was the okay well if we're restoring from

a snapshot are we restoring from the latest snapshot or from a point in time and those provide different paths essentially for like latest it's a data source that will just look at the latest uh snapshot that was done but because of how our snapshots were uh being scheduled that snapshot would would actually be midday um which is not ideal for a lot of situations. So it ended

up being a lot more complicated than we needed to do uh we needed and it was not in scope. We didn't set standards. We didn't say hey we're going to start with our initial dis uh deployment with a blank database. We didn't clarify that. So that's on us. Um so now we have scope creep which everyone deals with. So lessons learned here you do your due diligence

before you agree to scope. Uh research what others have tried. define what's in scope and out of scope explicitly. Sounds very uh probably similar to what you hear from a PO. Uh but we should have focused on a blank database deployment first. With that mindset, that would have saved us a lot of time and a lot of struggles trying to develop a uh within Terraform and Open

Tofu uh a better way of handling this is a snapshot restore, this is a point in time restore or just a blank database. Okay, stage three, bargaining. This is probably my favorite uh mainly because we've already created now this KMS key module route 53 module and our RD our database modules and our secrets module and our ansible module. Um but what we were finding is that one

the uh deployment or execution of the Ansible playbooks was very uh hit or miss mainly because something that we couldn't control uh was that our Ansible runner one day decided to have all of the Python Python dependencies removed from uh the the VM which did not help because if anybody uses Ansible there's those are dependent. So that made it really difficult for us. Uh and that's something

we couldn't control. Uh and ultimately the initialization of the database was proving rather difficult within testing in a CI/CD pipeline because one uh we didn't have a uh we had a tech gap or a skill gap in our dev devops team including our team. And so at the end of the day we said you know why don't we just trigger the playbook after the deployment by hand

which defeated the purpose of being a CI/CD pipeline but uh we had to do this in order to get the thing work working. Uh and additionally too the idea of trying to figure out how we can automate this. Uh this is what the um X XKCD diagram shows is the amount of time put into automating something and how much it actually saves you. We were trying to

go through that mentality of is this really worth putting all the time and effort to automate something when it literally just means we can point and click. So lesson, sometimes companies lack technical expertise to integrate existing workflows into new ones and you can always compromise on perfection by keeping the pipeline moving even if that means manual intervention. Uh that's something that I really don't like to do.

I'm really very heavily invested in automation as much as possible. So the fact that we before we even got a working CI/CD pipeline had to make a next step of manual execution was uh Okay. So, enter. Oh, wow. This is very hard to see. I am so sorry. Uh, so I I'm actually embarrassed to show you this. Uh, but this is what everything uh looked like. So,

our our model was we had a local TF file for all the parameters per environment. We had your typical main uh TF for all of the deployments you executing for the modules and what have you. Uh and as you can see the far left if you can see is the local variables. There's lots of them which means a lot of input and kind of defeats the whole

purpose as well as the large amount of modules that we have to call on just to deploy a database. Uh I presented this to my manager or mentor and he said this is great but he was lying. Uh it's too much and I looked at it and said you're probably right. But something that I realized uh was that I had created something that only I understood and

I like the ability to have uh the ability to change parameters whenever I want and have that option. And here this was actually the worst thing I could have done for a CI/CD pipeline. So I'm embarrassed to show you this, but I promise that's not what it ended up looking like at the end. So enter stage of depression. I had realized I had became the gatekeeper and

I'd overengineered something. Uh what my local TF looked like a Costco grocery list. Yeah, you go in and you're thinking I only need one or two things and it ends up being, you know, a foot or two feet long. Uh this was exactly what happened. And we actually had variables for variables o and overrides for those variables because of how our infrastructure was uh created. Not our

fault, right? So what we ended up doing uh when with the feedback we got from my my mentor my manager and by just simply bringing in the team u to be more involved with this. I'm on a team or was on a team of uh four DBAs at the time and was really the only one pushing this method or pushing this effort. So I ended up creating

some one-on- ones with the other teammates and we had regular check-ins to make sure we were all aligned. But that meant we had to rebuild every single module. Uh which was a good thing to be fair. Uh we focused very specifically on variables based off of data engine or sorry DB engines and that'll be uh explained a little later. And we agreed on what we should have

done in the first place uh defaults for all of the configurations whether it was Aurora uh my SQL or Aurora Postgress. So after those several one-on-one feedback sessions, we started to see the light and it started to look look pretty good. So lessons here uh know your customers. In this case, it was no my fellow uh teammates. Uh parameterization is good, but setting defaults to your team

standard configuration is best. Having strong defaults for version instant class security groups make modules much easier to use. uh which I think is pretty straightforward that I should have realized this but as lessons are I learned them. Okay, my favorite step because there's ice cream. Stage five acceptance. So we went from five modules to three KMSs key route 53 and Aurora non Aurora module and I went

down to seven inputs. So uh I think there was about 34 before and now we're down to seven and there was a lot of internal logic in the modules to do things based off the default uh engine. So we defaulted to Postgress if you want if it was uh at new Postgress it was usually 15 or 16 and then with that came an instance class that was

default a parameter group that was default and things like that. Uh, we had one subnet I think subnet group. We had one security group because why repeat a security group for every deployment when it's literally going to be the same port 5432 or 3306. Um, we had a manual playbook execution. Not great, but it worked and it was working before. So why I have to try and

figure out another way? And that meant that I could eat ice cream. I celebrate anything usually by eating ice cream. I'm an ice cream snob. And that is my favorite flavor by Tilmok. So, if you haven't had it, it's really, really Okay, so now this is what it ended up being. Uh, I'm really sorry it's really hard to see, but on the far left is the former

was the locals TF. So, you can see there's only a few inputs. It actually looks a little longer than it should only because there's a inputs for tags and I don't count that because tags are important. And then in the middle is the main TF. So, we got down to just three modules and then our outputs which were pretty straightforward. Um, a lot better and I was

really happy with it and it was actually really doing well. Uh, we deployed about 50 plus databases which isn't a lot but uh they were successful and it was pretty easy to do for all of my teammates. So, lessons learned databases are the backbone of it. Implementing them in I infrastructurees code could be much harder than infrastructure. uh technical expertise gaps, industry turnover and organizational changes are

impacting will impact pitch achievable and something that I learned in a decade of being in IT is it is all gray. There's no black and white. Uh it's never one way or the other. It's always in So let's talk about what happened. We experienced change in our organization. We deployed 50 databases and we didn't even manage state file. uh we didn't manage uh any states because we

basically just deployed it and then managed the entire instances with Ansull um and it made it a lot easier for us. So it defeated the purpose of having a CI/CD pipeline but um my boss was right uh and never we so we never fully implemented what we were supposed to and at the end of the day I ended up rewriting the entire thing in Python. So what

uh basically uh about eight months of doing this back and forth, the end result was the exact same result that my manager had. And uh yeah, he was he was laughing pretty hard at the end of that. So key takeaways, research first, uh scope carefully. Something I think everybody hears and we always laugh at, oh, we couldn't possibly have scope creep, but it always happens. Um providers

are always very they vary widely. Uh I've I think only found a few smaller uh providers for like uh things like digital ocean or uh len node now ai that are pretty good but most of the time the big three are going to be uh Azure AWS and GCP which are really good. So two or four I guess uh know your audience. So build for what your

team's expertise are actually that that are actually within your team. Uh that's something that was a hard lesson learned for me because uh database admins nowadays it has ve it varies and it has changed quite a lot. Uh we've become more of database reliability engineers rather than the traditional let's go through queries optimize queries. Uh we're just doing all that and some. So just adding on to

it and that creates a lot of skill gaps depending on um the experience of your team. Defaults matter. Something that I really can't emphasize more that if you've got defaults for your variables, uh they there's a reason why they always stress it because then you don't have to provide any input if you're just asking for um what is expected. It's okay to compromise. Manual steps are not

failure. uh in this case they kind of were but uh the CI/CDA pipeline still lived on outside of databases and we were still able to provide an easy and efficient way of deploying a database when requested and context changes or org changes priority shift and roadblocks appear you just got to go with the punches and that's actually it um so I was really happy to go through

uh my struggles I know it is at the end of the day so I really appreciate you guys all being here I'm hoping the rest of scale for tomorrow goes well and I am happy to take any questions or comments or suggestions for uh future. So thank you. No questions. Yes. They're recording it, >> but I could hear you just fine. >> Hi. Uh, so you said

ultimately um you moved from Anible over to Python. What's the current state of your Python? Uh what does that look like? >> Yeah, so we >> just just at a high level. >> Yeah. Uh we moved from using Terraform and infrastructure as code to using Python to deploy. Uh currently uh that's a good question. I'm not on that team anymore, but from what I've been told, it's

been doing great. Uh what's nice about it is for the configuration and standards, it's in a YAML file that they can just take a pull request and change things. But yeah, it's it's working, it's running, and they're executing it locally just fine. Um, the DevOps team has moved on to other things for infrastructure as code. Good question. >> Just so I'm clear, like you just mean like

you wrote a script that's like BTO stuff. >> It's essentially just the API calls. Uh, and the BTO3 documentation is really good. It's very verbose. Uh but yeah, it it's just calling specific API calls to to deploy uh and with the inputs that you give it, which are basically just defaults. >> Uh would you be able to comment on what your DevOps team is moved on to?

>> Uh my opinions are my own. They're not my b my uh employers. uh but from what I know they're using they're still using open tofu terraform uh they just have moved to uh not using ours or using the database teams. So last time I checked they were still working with their infrastructure team trying to implement that that pipeline and uh that was about two years ago.

So So I'm curious for the you mentioned database initialization in your experience because I I recognize it varies. How far do you take that? Is it like uh table creation, index creation, all of that fun stuff? >> Yeah. So obviously each company is going to vary on what they need. uh since it's a medical or I work for a company that's in um the medical industry medical

uh the things that we had to focus on were simply just creating um certain restrictions on for postgress specifically uh schemas that could only be accessed by certain users. Uh there wasn't any table creation. We did create a DBA uh schema to like hold metrics that we would eventually collect but at the end of the script it was just generating service users for the app that it

was supporting or service and then the uh local DBA accounts. Um we had six or four DBAs officially. So it would just generate users for us automatically. So once we got that endpoint from the deployment we could just log in and it should be set. I I guess the exception would be the point in time recovery kind of scenario where suddenly >> it would be a restore.

Yeah, we wouldn't even touch an anull playbook at that point. Um but yeah, that would have been more manual uh click ops to get that done. We we were we used uh AWS backup with a capital B and how that's handled is a lot different than AWS backup with a lowercase B which is just the built-in AWS backups that you can initi set up when you deploy

an instance. And so uh it needed more manual intervention for doing the restores. It yeah I'm trying not to go into the weeds on that one. It's an interesting thing. You should check out AWS backup. Yep. Any more questions? >> Say we have some time if you want to go into weeds. okay. So, AWS backup. This isn't I guess this isn't really relevant, but I'll talk about

it anyways. Um, AWS backup with a capital B uh is used for being able to back up not just B databases. It can do S3 buckets, EC2 instances. It's really meant for CIS admins or IT infrastructure folks to make sure that they're covering all their, you know, dotting all their uh eyes and checking off all the check boxes to make sure that we've got some sort of

restore uh or um disaster recovery for all the deployments or all the resources. Uh how it works is it's not like the actual scheduling you can do for an RDS backup with a lowercase B. Uh it you give it a time zone or time window that you can back up and it uses an algorithm to calculate if it can actually do it within that that time or

that schedule. And if it can't, it'll pick one for you. And that means that we were getting snapshots in the middle of the day because uh because of how many databases we had, it wouldn't be able to finish that within that time window. But good question. >> Other questions once, twice. All right. Well, thank you >> Thank you guys.