From Open Source To Enterprise Scale and Back: A Journey... Michael Kuhnt & Gabriel Adrian Samfira

This talk covers the journey of building a managed shared service for internal development teams at Mercedes-Benz, leveraging the open-source project Garm. The speakers, Michael and Gabriel, discuss the challenges they faced in creating a centralized CI/CD runner platform that addresses the issues of duplicate efforts and security. They outline their initial vision and the importance of documentation in guiding their project. The speakers share lessons learned about engaging users, reducing friction for adoption, and ensuring transparency during incidents. The presentation highlights the growth of their platform, which has achieved millions of workflow runs, thanks in large part to the open-source community and contributions to Garm.

So, let's start. Hi. Thanks for joining our talk from open source to enterprise scale and back, a journey with Roadrunner. So, I'm Michael. I'm software engineer at Mercedes-Benz Tech Innovation. I'm Gabriel. I'm engineer at Cloud Base Solutions. In this talk, we want to share some lessons we learned while building a rather large managed shared service for our internal development teams at Mercedes-Benz based on open source project

built by Gabriel and his team called Garm. But of course, it's not the idea to share you another awesome success story. It's more about the winding and uncertain path that it actually was building such a platform with a relatively small team and now it's used by thousands of users in our company. And in the beginning, I think nobody even knew that we needed such a platform. And

take it a step back. So, maybe you've been there. Maybe you you've seen that too. You you have you see that opportunity in your department, in your company. You see an awesome new technology or a tool or a workflow that could help and enable a lot of developers. But you don't have the time, the budget, or team, or even the mandate to do so. That was exactly

our situation. So, how do you start? How do you not let this opportunity pass by? Let me show you our challenge, our problem. So, at Mercedes-Benz, we have GitHub Enterprise instance. And of course, it is not the only forge that we have at Mercedes, but I would say a larger portion of our source code is in this GitHub repository. instance. And it comes with all the bells

and whistles as you would have it in github.com including GitHub Actions, which is GitHub's internal CICD and workflow platform. But for us, it didn't come with runners. So, the actual physical machines that would run these workflows. And this was a big problem. Because I mean, teams needed that and they got creative. And of course, they started building up their own little runner setups, which is totally okay

in GitHub. That is possible, but it creates a lot of duplicate effort. When every team does it themselves, you have basically no room for standardization and also from a security perspective, this is rather concerning. So, um we knew we had to do something and we knew we could do something because just looking at github.com, you have runners available in every repository just ready to use. we wrote

we had a vision and we wrote it down. Of course, it sounds super simple and it is simple. So, you start with a design doc. You want to state the what, the why, and the how. And in our case, we want to build a common managed CICD runner platform for our internal enterprise instance, GitHub Enterprise instance. Why we want to do that? Yeah, we want to have

less duplicate work, more standardization, and security. And we would let our engineers focus on what focusing on what they really want to do and not babysitting any runner infrastructure. And how we want to do that? We want to provide ephemeral VM based runners to every repository, ready to use, no long configuration, no approval processes, and whatnot. Just go. And this document, maybe the first draft took um

one day to to to detail out. This was very important because it became our reference point. So, whenever we had to make a design decision, we could come back to this document and see if we are still on track with our vision, what we actually wanted to build to help our developers. And on the other hand, it became our pitch deck. So, when we would get feedback

from potential users or get backing from management, we could share this document and see tell them, "Hey, this is what we're trying to build. This is the idea that we are looking for." And this is already our first lesson for today. It sounds simple, but it's always the first step. You detail out your vision and you get feedback on it. And you want to state the what,

the why, and the how. You want to make it shareable and you want to get it out there to get feedback because you can't get feedback on a vague idea in your head. This is what we did. We shared it internally and everybody who who cared could find us now and by this, we collected a lot of awesome feedback that we could incorporate in our initial idea.

So, now we had our vision. It was time for a proof of concept. But first, of course, you want to do some research and this is also what we did. Because if you have a good idea, probably somebody else had it too. So, we found Garm, the GitHub Action Runner Manager. Already the title sounded very promising. And we checked it out. It was a very young open

source project at the time. But it had a pluggable architecture. We could easily extend it to fit our needs and it was written in Go, the best language on the planet. But still, you don't just want to pick something from the internet and build your new awesome platform on top of that. You want to check out what you're buying into here. And as it was open source,

yeah, we reached out to the creator. And this is super easy. We just started with one little pull request. So, this is actually a screenshot of the first change we made at Garm or the first pull request. It was one file and it's not so much about what we actually wanted to change here. It was more to get in contact and to see how is our contribution

valued. Is that looking to for a good collaboration? And what an answer we got. So, Gabriel replied the next day with this long reply and it was really welcoming and we knew that this was looking like a good yeah, possibility. So, we said, "Thank you." And we even scheduled our first call and we checked out if our use case aligned with Garm's vision and if this is

a good match. So, this is the next lesson. Before you build something from scratch, of course, look around, but you just don't take everything. You want to understand if you you align with the vision and if So, maybe you say something about Garm and how you guys started. So, [clears throat] when we started Garm, it was to make it possible for the Flatcar Linux project to move

their in-house Jenkins instance to um to GitHub workflows. Back then, they had their own internal Jenkins. You would when you would contribute, your pull request would start running inside their own Jenkins instance, but you wouldn't be able to see the logs that that were generated. This made it extremely difficult because somebody with access to that Jenkins would have to shuffle shuttle information to and from your pull

request essentially and tell you what went wrong. So, I started working on trying to integrate the Flatcar Linux workflow to GitHub runners, but that took more than 6 hours to run and after 6 hours, GitHub would just cancel the job. So, it was basically clear that we had to move to something beefier essentially. So, I looked around. I found Arc, which is amazing, but it gives you

a container runtime. Now, Flatcar Linux requires an actual VM to run on because the whole SDK runs in Docker and while you can do Docker in Docker, it's kind of like baking a cake with a barbecue. You can do it, but yeah, all right? And you also had have to have access to device mappers and everything else because you're building an operating system and you're trying to

test it. So, LVM comes into play. Device mapper is important. So, I started looking around for others runners runner managers that allowed you to spin up runners on virtual machines and while I did find a few, they were very specific to one platform or another. Now, the Flatcar Linux project had their own servers just enough to spin up something like LXD back then in 2022, but not

enough to build something like OpenStack, which didn't make sense for the for their use case. So, I built the the initial version of Garm consumed events from GitHub via webhooks and spun up LXD instances, virtual machines, and injected the runner, spun it up, joined it to GitHub, and we had our first run of Flatcar Linux workflow run inside the self-hosted runner running on our ARM64 and AMD64

machines in our data center. So, that was that was a huge win. But I wanted to be able for the Flatcar Linux project to be able to move their work workflows in any cloud. So, I come from an infrastructure background. We deploy a lot of OpenStack and most clouds have the same pattern. They give you a compute resource. They allow you to inject some user data. They

can create virtual machines or containers or bare metal machines and you can just use them for whatever you want. So, there was no reason why we couldn't add an Azure provider, an OpenStack provider, and an an LXD provider in the same instance of Garm. Now, the plugin is simple. You have an executable. It works just like the CNIs in Kubernetes. You basically call it with a bunch

of environment variables and some input for the create part of it, and it goes out and talks to the cloud and makes sure that your instance is booted up with the correct user data to make that runner join GitHub in your repository, organization, or enterprise. uh it was about this time that Michael's team reached out uh and they had their own GitHub Enterprise Server instance. Now, there's

no reason that if you support GitHub proper, you can't support GitHub Enterprise Server because it's essentially the same API. You just point it to a different set of APIs, right? So, we added support for that. Then they added uh the Kubernetes provider because it's very easy. You can use Bash to write your own Kubernetes provider if you want to or any kind of language. The idea is

that you have an executable you give the Garm for Garm to to call into. So, we added Incus, of course, because that split off from LXD. Uh we added the uh Kubernetes uh uh Michael's team actually added a Kubernetes provider. Uh the Flatcar Linux project added Akamai. Uh and to give you a sense of how it works, this is very simple. Essentially, you have uh GitHub sending

you workflows. You record You record that job in a database. Then you have pools which essentially create uh homogeneous runners of the same type for the same provider on the same repository, and you consume those jobs uh and spin up runners based on those jobs. Um and you Of course, you can create pools uh for one repository in one uh uh cloud or another. You can mix

and match them however you want. And recently in main branch, we also added Gitea as a source forge. And I'm hoping that after a few uh the community in Forgejo uh merges a bunch of PRs, we also get that as a source forge. So, you will be able to to link uh Garm to multiple source forges using the same instance and create runners in any cloud you

can write a provider for. And the uh the the the the complexity of writing a provider is very very low. So, anyone can do it using any programming languages that feels comfortable to you, including Bash. And I'll hand it off to you again. Yeah. Um >> [clears throat] >> for us at Mercedes, finding Garm was crucial because it was doing the initial heavy lifting of running, starting,

orchestrating these runners with our GitHub instance. Now, we had a prototype in no time and we needed users on board. And this was probably the hardest part. And um I'm afraid there is no simple plan getting users on your new shiny um that always works. But we identified two things that matter most for us. First is you want to reach out everywhere. You want to spread the

word and explain why your new tool solves their problems. Uh do it on all your messaging channels. Uh talk to everybody on the coffee machine, and you want to join every competence group you might have in your And the second impo- important part is you want to remove friction as much as possible because if you convince users to try out your new thing, then you probably only

have one shot. And users need this aha moment uh to see and understand, "Oh, yeah, this solves my problems." And not after long configuration, not after reading a ton of documentation. They needed to have this instantly. Sometimes you can also use a little trick. So, the way uh runners are picked by GitHub in in GitHub Actions is by label. So, if you have a workflow and you

say runs on Ubuntu, then GitHub would choose the next available runner with the label Ubuntu and run your workflow on top of But as I mentioned earlier, we had a lot of teams building up their own little runner platforms, and we saw they all used the self-hosted label. So, we thought, "Hmm, maybe our road runners also serve on the self-hosted label." And by this um as soon

as the uh users um registered or used our runner pool and allowed our runner pools in their repositories or in their organizations, they would immediately run their workflows also on our runners. Less friction, and this led to more adoption. And they didn't need to change their workflows. They didn't need to do anything, and they would just benefit directly from our platform. Um of course, [clears throat] it's

not only about luring the first users on board and then be happy. You And And this is what it's actually all about, right? You want to build this platform to share value immediately to all the users, and you want to add new features and new awesome uh bits and pieces to help everybody. And to give you an example of what uh in our case, we uh fixed

the proxy configuration. So, if you have ever worked in a company behind a security proxy, so where every tool needs to go through that security proxy to reach the internet, then you might know how difficult and cumbersome it can be to manage these uh proxy configurations for each and every tool. Uh this can be really frustrating. And what we did is we installed this little component in

our runners that would route all HTTP traffic directly through this security proxy, fully transparent to the user. And I would love to be able to measure how many work hours this little feature Garm solved uh uh or saved in our company. And I think most users don't even realize that this problem existed on this platform. Another nice feature that I really like um debugging these workflows is

difficult. And we added this remote shell access. So, whenever you have a faulty or buggy workflow uh and yeah, usually you have this super long round trip to get to the to the error and then debug it. Um yeah, you can use the remote shell uh to jump directly into the runner um and debug it live on the VM and see what's happening there, what's going wrong,

and fix it. So, this is the next lesson. Getting first users is hard, and you want to the word, and you want to remove friction so that users see the value for them immediately. So, things were going good. Our platform was growing, and then of course, eventually things break. Um we had our first big incident. The platform went down. Users were blocked, and not a single workflow

would run anymore. The reason was our infrastructure layer, so this thing that spins the actual VMs, it failed to delete all discarded runners. And so, Garm's runner uh pools were full of already consumed runners, and it there were no left to spin any new runners. Nobody could run a new workflow. Um once we understood the problem, we fixed it, and crisis was over. Now, what? Maybe you

know where I'm going here. So, of course, you can pretend it didn't happen, and maybe it was on the weekend anyway, and hopefully not too many people noticed it, but no, of course not. So, you want to be transparent, and you want to write this postmortem. And I know everybody keeps telling you postmortems are important, but they actually are. And it's usually not very difficult, right? So,

usually it takes 30 minutes to write it. And there are good templates out there. So, you want to add a description and a root ca- uh root cause what happened, including a timeline. And then you want to detail out how you will make sure that this particular problem doesn't happen again. >> we did that, and users thanked us for that. And of course, they didn't thank us

for the failure, but they thanked us for the honesty. Um because people don't expect perfection. And I think probably we have all been there. Things break, and um users expect that you improve and take their problem seriously and make um that this doesn't happen again. So, this is our next lesson. Learn from your failures and be transparent about it. And when things break, you write this postmortem.

Yes, and you scare this monkey away and because this builds trust. And here's the kicker. So, this is not a one-time decision, right? You have to write this postmortem again and again after each outage, and don't um yeah, let it pass by. we were growing, as I said, and it took us roughly 4 months to get the first 100,000 workflow runs. Our platform was gaining traction, and

after the first year, we had the first million runs. Um looking at the numbers now, uh after a couple of years, we have over 12 million runs in total, and roughly 700,000-ish runs per month. And the number that uh our users, of course, are most interested interested in is how long do I have to wait for my runner. But also this number looks quite promising um between

3 and 9 seconds even we if we are on the M-based None of this would have existed without open source, and none of this would have existed without Garm. And um that is uh one big puzzle piece of how small teams can build big things. And this is already the last lesson. Um So, if you tap into the open source community, you want to consider giving back.

Um And this doesn't have to be mass- massive. So, we started with small bug fixes, and this helps. But we also shared insights in Yeah, we run GORM in a little bit bigger setup than you guys probably had and we could show you edge cases where GORM runs into that you probably never saw. And you can share this and you can also share tools. So, some awesome

colleagues here built a Kubernetes operator so that we could easily manage our GORM in our Kubernetes setup and this thing we open sourced it and gave it back to the community. So, now everybody can use it. And yeah, this were some of the important lessons we learned from our journey. And of course this can't be a complete list. We could probably go on for another hours here,

but maybe you saw something that encouraged encouraged you to also get on and maybe yeah, some things could help you, But I think now it's time for a demo. So, This will be a live demo. Things might break in interesting ways. So, bear bear with me, right? >> So, in the recent version of GORM we have We haven't released it yet. It's not GA, but it is

on GitHub. I'm going to show you how the the Kubernetes operator integrates with GORM and allows you to create multiple endpoints, add credentials for your source forges, define your repositories, define some pools of runners that will spin up actual runners. And for in the case of GitHub and GitHub Enterprise Server, I'm going to also create a scale set because we also support that as well. So, this

is the operator already running. Now, we need to create the endpoints. So, kubectl Now, once we create the endpoints they just pop up. The web interface that you see in the background is basically connected to the GORM itself. It's a web socket interface. So, as the operator creates the resources in in the GORM, they pop up in the background, right? So, for a second I'm going to

go to credentials now. Now, we need to add credentials for for these two endpoints. So, and the endpoints just so you see it it's like pointers to your APIs, descriptions and names and that is about it. So, credentials for obvious reasons I'm not going to print to the screen, but you can add personal access tokens for both Gitea and GitHub. And using these credentials we can now

define our repositories. You can add repositories, organizations or enterprises. So, GORM can operate at all levels. So, repositories for this uh demo. They pop up right here. Now, before you if you're using pools of runners that leverages webhooks from GitHub and Gitea. And that means you need to install those webhook endpoints in your repository, organization or enterprise. Now, you can do this manually if you'd like or

you can allow GORM to do it by just clicking this button. The operator doesn't do this automatically. Uh this is something that we can add, but it has not been added yet. So, let's install those webhooks. Of course, the personal access token or application needs access to do this. You can give it the least amount of access you would like. Now, we add the pools. So, this

will add several pools for GitHub and Gitea and you'll see that some of them are running on LXD, some of them are one of them is running on Amazon and one runs on Kubernetes. Now, if you go to runners, you can see that they're already spinning up and the ones that of course run on Kubernetes spin up quickly. You can also access them using a shell from

the web UI or the CLI. This is a Kubernetes pod, so there's not much running inside of it except for the GORM agent and this is completely optional. You can do without the GORM agent, but you don't get shell and instant notification when a job has been picked up. And of course, you can add scale sets. So, let me just runners. I forgot to refresh this beforehand,

but no worries. It popped up. Now, as you if you look here for a second, there's no scale set yet, but if we create it now, scale set was created. Now, if we refresh this page, it popped up. It's still offline, but if we refresh one more time, it needs to create the session to GitHub and start listening on the session. And once that happens, you also

get the run the runner that was spun up here. So, these are all joining your repositories. As you've already seen, the ones in GitHub are already up. Some of them are offline as they still need to install This is on Amazon, so this needs to spin up here. If we click refresh here, it's running, but it still has to run user data to to pop up, right?

Now, if we want to If we refresh here, we'll see the ID changing. It's 224 now, but if we want, we can just go to actions and we have a runner here now and if we click on just run workflow, we'll see that one has already transitioned to active. Uh you can see the workflow running here. Let me check. Ignore the the commit message. It It just

prints a bunch of information. This is already running on a runner that was spun up via GORM. Uh and once the runner is active, it will spin up one to take its place. So, it does this by lever I mean, we have a min idle runners that we can set on the pool or scale sets and GORM will always attempt to spin up at least that many

runners to be idle at any given point in time until you reach the max runners settings that you can also set. So, if you don't if you want to have a have a cap on runners, you can do that. If you want to have a min idle runners, you can do that as well. If you want to have zero runner pools, you can do that as well.

So, you don't incur costs in case you want to spin up runners with GPUs which are expensive or FPGAs or whatever other specialized types of runners that you would like to have. >> [snorts] >> But if you have Kubernetes runners running in Kubernetes pods, you can always max out your pools and have them available at any given time. And that's basically the demo. Yes. if you are

now super eager starting with your own runner setup, here are some helpful links you can also find in the presentation. And that was already it. Thank you. >> [applause] >> And I guess we have time for some Q&A. We have some time for questions. >> If you have questions, you I shall tell you that you go to the microphone and then if not, enjoy the conference. Thank

you, everybody.