Pull Request Wars: The Flux Awakens - Ephemeral Kubernetes Environ... Matteo Bianchi & Stefan Prodan

In this talk, Mat Bankei and Stefan Prodan discuss the concept of GitOps and its application for creating declarative, pull request-driven preview environments in Kubernetes. They highlight the challenges faced by platform teams, such as inconsistent CI/CD pipelines and the desire for developers to have self-service capabilities while still maintaining governance. The speakers demonstrate an open-source GitHub organization setup featuring Flux, which automates environment provisioning with a single pull request label. They explore how Flux integrates with CI workflows to deploy applications without requiring direct interaction with Kubernetes configurations, ensuring teams can dynamically create and manage environments efficiently. The session also covers the use of AI tools for debugging and enhancing development workflows, emphasizing the synergy between modern development practices and observability.

Welcome to Por Request Wars, the Flux Awakens Ephemeral Environment Strike Back. So, this talk is about Star Wars. No, I'm kidding. It was just uh there to uh for cloud, you know, for getting accepted. It worked. So, my name is Mat Bankei. I'm a solutions engineer at GitHub, CNCF ambassador. I'm a former startup CTO and a semi-retired Devril. I've been speaking too much lately. open source contributor

in Kubernetes and former release me release team member of Kubernetes upstream and today with me Stefan Stefan Prodan principal engineer at control plane creator of flagger how many of you know flagger or use flagger yes po amazing uh core maintainer of flux uh and you know 20 plus years in software engineering and also the mind behind the flux operator and also the flux UI. Tomorrow he has

another talk about that. So if you wanna you want to join him, it's really cool. Platform teams in the AI era. How many platform engineers we have in the room? Okay. Almost half of the room is a platform engineer. Amazing. >> How many are sales? >> Yeah. So the more you tighten your grip in CI/CD pipelines, the more developers experience slips through your fingers. This is a

quote from probably OB1 if he was a platform engineer. Uh the challenges today are various pipelines are hard to maintain a lot and inconsistent across teams. It's very difficult to have everyone using the same tooling and so we need something to standardize our workflows. Share staging environments a nightmare. a lot of maintenance, a lot of downtime as well. Uh long review cycles because of course you need

to fix bugs. It needs to be quick and you need a quick feedback loop and that's not always possible. App teams want a lot of self-service because of course I'm a developer. I'm lazy. I don't want to do anything with infrastructure. I just want to get stuff for free. Good. That's amazing. But platform teams need governance. They need everything centralized. They need control. They need observability. they

need to know their things you know and then AI and agentic coding which is great it helps in a in a lot of ways but it also creates a lot of slop uh and so we need to actually test in a real environment what we're doing and the answer to this and this is what we're going to talk about today it's a declarative prdriven githops native preview

environments owned by platform teams but made for developers that can self serve by applying ing a single label on a GitHub pull request. We have three repos in an open source organization. No worries, you're going to get a link after the the demo. First repo, it's a platform repo that contains the flux operator config, the cluster bootstrap, all the infrastructure resource set uh and it's owned again

by your platform team. Then we also have the app. It can be any app. That's why it's called app X. Uh it could be your website, it could be an API, it could be whatever you want. Uh and this is the only repo where you have your developers working. They don't need to know anything else. They don't have access to secrets, to environments, anything like that. And

then you have Elm charts which have of course uh also owned by your Demo time. Time for Stefan to come. >> Thank you for calling Akbar. It's a trap. >> Yeah. So, we've we've set up this um demo GitHub organization. Um you can actually create your own organization to test all of this and clone push all these repos up to your org. They have um little bit

of automation, GitHub actions with workflows that are doing only the CI side. Yes, >> we build and we push everything to the container registry. That's the only thing we do. >> So from from CI we do the CI part. We don't touch the cluster. CI doesn't even know there is a cluster. Um as we said we have like three repos. The one that I'm showing here is

just the demo app. This can be many many app repos or maybe you have a monor repo with many microservices. The idea that that what what we want to showcase is that without having you know any kind of uh Kubernetes configuration in my app repo I can instruct flux to deploy it on a cluster as an FML environment and uh why is FML because you know p

request FML is like you open it you want to deploy everything there you want to see how it goes then if you merge it or If you close it, what should happen with all the workloads? They should magically disappear. >> Disappear, >> right? >> And remember to delete the branch as >> And delete the branch. But the branch for for what we are showcasing today is not

that important. >> Doesn't matter. >> Uh flux operator actually talks to uh GitHub, GitLab, um Ga >> G. Yeah. >> Uh Asia DevOps. So it's not only about GitHub. This kind of workflow will work uh with most providers out there. And if you want to do gitless GitHubs and you don't want to you know pull things from the repo, you you can also use GitHub actions. We

have a special GitHub action in Flux where you can push the manifest to GitHub container registry. Um let's start. We are late. Okay. So what I'm going to do is um make a silly modification just to see a pull request going. I'm not committing to the branch directly. Test good stuff. Not Yeah. So when I open the pull request, what will happen now if GitHub works? It's

not down. It's green. It's all green. >> Okay. So we have a CI job that has started and it does the usual thing, a Docker build and a Docker push. Of course, here you should run your, you know, llinters, uh, unit tests, end to end tests if you have that kind of thing with Kubernetes kind. We do that a lot in the flux organization. But no matter

how much testing you are embedding here in CI, it will never cover uh you know what actually happens with the application once it runs in Kubernetes. >> It receives traffic you know >> resist traffic connects to other uh microservices in there and so on. So ideally I would want after this build finishes it should take like two seconds not two minutes. >> Yeah. Something. >> Anyway, okay.

It's done. first part is done. Um, so how can I deploy this? I have these labels here and one of the labels is called deploy flux preview. Um, you can set your own labeling and so on. This is just an example >> all customizable. >> What what what happens now? I have a kind cluster running locally that has flux operator on it and I I I configured

as me as the platform engineer this time not as the dev. I have configured in the platform repository. Hey please connect to this organization. We are using a GitHub app authentification. Scan all the pull requests in the appix repo and if you find a pull request with this particular label create a hand release inside the cluster. Use the image that was built here and deploy this particular

image by patching patching setting in the values uh the tag equaling the um the git sha of this build. Let's see if anything works. I'm going to show you the Flux UI. Oh, is this connected? One second. Need to start again the Yeah. Or for Yeah. So this is how Flux UI looks like when when we look at a thing called the resource set input provider. So

this is uh the API where you tell flux which repo to scan and how to pull from their um pull request. This is how the configuration looks like. This is what it is in our platform repo. It's quite simple. Um we have we've set up some default values for our um uh generic helm chart. And what's important here is the URL where you point it to the

app repo. You say it's type GitHub. So it uses all the configuration and so on and the filter I'm configuring the label that should it should watch. Another important thing are the limits. So you can limit how many preview environments you have at the same time. So you can avoid like Kubernetes costs in case you're not using your local host of course. >> Right. So by by

default we have u uh 100 we we'll create a maximum of 100 preview environments for a particular uh app repo. So now depending on how much money you want to spend on previews you have some control here. Um okay so what happened at this point if we look at status we see that okay it it has uh succeeded it has pulled some information down and if we

look at what information it has discovered from the um GitHub API we see here that it has resolved the uh commit of the head of the branch in the pull request. So we can use this information now to uh bootstrap our um and to do that we have a different API object called resource set and I can show you quickly how this looks like it's a templating

engine and the important part here if we look at the helm release uh where is the helm release LR hand alert. >> So we have uh we we basically template here a H release. That means for each pull request we are going to install a H release. Then while I'm modifying the pull request, I'm going to keep upgrading the hand release to the latest image built from

the pull request. And how I'm setting the image, I'm setting the tag to the convention I have set up for this app repo is PR minus the input ID, which is the name is the ID of the uh pull request minus the short SHA for the head commit. So with this I'm getting you know unique images for each commit in a pull request and here I can

uh you know say from this particular pull request I only want to target a helm. So the helm will keep upgrading with each commit that I'm doing. Um this is golang templating. Maybe it looks weird to you. It has like why does it have those? Anyone knows? So you can put this in a Helm chart, right? Because if we are using the same uh templating as Helm,

you are going to mess it up. A bit too crazy. Yeah. >> Yeah. You can put a resource set in a H chart. Okay. Let's see uh let's see what happened. Um we have here a graph view that showed us that this resource set has generated a h release called app 9 and nine comes from here is the pull request number nine. Now if we look inside

the pull request the developer does not have to interact necessarily with the flux UI right you can go to your pull request and flux will post here all the things that have done on the cluster in this case the app has uh has been successfully deployed here you can enrich this message with preview urls with all sorts of things that are helping your developers to reach out

the app themselves or I don't know based on this you trigger some other automation like an end to end test from or a low test from the GitHub action and so on. So it's very powerful you can uh you can actually trigger more events based on what's happening here with this comment >> and if you were at Flaxcon you know it's a recent addition in uh the

latest release. So >> yes >> use the latest release. >> This is a flux 2.8 feature. Another thing here we so besides commenting on pull request we also integrate with uh with GitHub deploy API. So the the the push workflow has added here on the commit status a green check as I push. Then also flux operator marked here on the repo the helm release that I created

for this environment. Uh this one has been uh successfully installed as well. Um, I'm going to do next an improvement, another pull request. I'm improving uh my engine X up by seeing ingress. Major improvement. Yeah, I'm going to create a second pull request now. Going to add the label from the start. And what this will do will do the same thing but instead of touching the previous

end release it should create a new one with my major improving switching from engine X to ingress engine. Okay, while we are doing this um let's see this app running on the cluster. I is pull request 9. I'm going to create a port forward. We were too lazy to choose um a successor for ingress engine X. So that's why we are using port for okay so now

this app that I have deployed with the first pull request should be Okay, it is amazing. So, we have built vibe coded a little game because of course these days I'm too lazy to actually write a game myself and we're going to play it in the meantime just not to wait too much. So, you also be able to play. Oh, I got a graphana here. Amazing. I

have more power. That's great. >> I also have sounds, but you don't hear them. So, it's like Okay, get a little more power. Yeah, I have a couple of waves. It's going to be very quick. As quick as making our little PR fail. Oh, I'm not too good at video games. Maybe someone in the crowd is. How many of you are good at video games? >> No

one. >> No one. I I know at least one person. Okay, cool. Last wave coming. You know, for being a vibe coded game, OPUS 4.6 did a great job. I don't use any other model, but don't say don't say too loud. Yeah, of course, it's not too difficult. I can make it better, but then I think we're going to have a little surprise at the end. I

don't have enough power to do this. I I I'm not sure I can do it. Oh god. Oh no. So remember, by the way, uh, >> cheat codes. You have cheat codes? >> No, I don't. I don't. And you know, >> you didn't vibe the cheat code again. >> Never. Never. >> Okay. So remember to actually take down Ingress engine X. It was archived on stage uh

by by my friend James. So you know, just like take it down. It's not needed anymore. >> Poor forward everything. We don't need >> directly. Yes. Cool. Okay, let's let's hope something failed in the meantime. >> it takes a while to fail. It's like by default I think it's like 5 minutes. >> Yeah. So we can we can actually see what's happening for now. Uh the only

feedback I have on this pull request is that it has been successfully built. Right? I I made a change and improve in my Docker file. CI will not actually detect that because build works. there is a bug somewhere in the configuration. Um now if I go for example to my uh resource set input provider here I should it should detect that there are two pull requests running

and I can kick it off. So this is these are actions that we allow in the flux UI based on custom arbach. I'll I'll talk tomorrow more more about it. But you can do things if you if your platform admin has actually set up arbug. So you can actually do these things. It's not like not everybody can do whatever in the UI. Um so what we see

here now we have two different outputs right for each pull Then if we go here we should see that it's already uh has already reacted to it. We have a new HM release, right? The the one with the game is nine. Works great. Now we are deploying 10. Um, we can navigate to the pull request. It says zero resources. Time out 5 minutes. Let's see when it

wrote this. You can also see the values here. So what I was showing you from the templates, maybe here is more obvious. We set the repository and the tag using the inputs from uh from uh scanning the the pull request and also determining what is the head shot of the of the branch and you can set other defaults here like I'm setting replicas too just to you

know uh uh make uh make uh >> make me pay more money. Yeah, thank you. It's my account. Okay. Okay, so it should fail at some point. What's happening now is so we have made a typo in the docker file, right? The pods are trying to start, they can't start, they they are out uh and helm controller has a timeout by default is five minutes where it

tries to you know is this workflow ready? I'm not giving it up. So you you have total control in flux over all this kind of timeouts, retries and so on. The default is five. Um the thing has finally crashed. I don't understand anything from this error. Let's go back here. >> I never looked at these logs. I'm too lazy. So you know it's just >> Yeah. So

here it kind of says the same thing. So the developer is now aware that whatever they did here is not actually working like why is not working. They have to dig into it. So instead of that let's copy. No we are not going to copy anything. Do you trust copilot to debug this? >> I trust copilot to debug this. Yes. >> With opus. >> Okay. So we

have copilot yolo mode here because we always live on the edge. No, it will going to ask me questions. >> No, autopilot is fine. Well, I'm going to ask it. How is Flux doing? Doing fine. You know, praying the demo gods, praying the LLM gods these days. >> Yeah. So, what happens now is like I have the Flux MCP installed. Um so copilot can use the MCP

to connect to the cluster and now it um it we we when it runs the MCP commands uh it will discover slow it can slowly discover everything that's running on the cluster like this question that I put it was like how is slugs doing it should tell you like an overview of your um uh GitHubs pipeline and yeah it tries here a bunch of stuff. He's using

a skill to understand. >> Now, he loaded the skill. He knows like okay something is uh bad things are happening. >> Skill says use in case of failing like that's why it just pulls it. Okay, we have some advancement. He said like found an issue. It has identified number 10 which is the pull request that was failing. Um and let's see it gets confused by the NCP

to move. >> It tries again. >> Found the root issue. Okay. >> Okay. Okay. Okay. It's a crash loop because ingress engine X was not found. >> Docker entry point. It works. So it kind of figured out what we wanted to figure it out, right? It it kind of it understood that. Okay, it's a Docker U SH that's running there. Uh we we intentionally did that typo

and it told us what's wrong. Now, >> I mean, let's try, you know. >> No, I don't I don't trust this stuff to commit. No, >> it's yellow mode. >> What if it commits to your account? >> No, of course. I only use yours. Now, by the way, on on a serious side, if you use the copilot CLI, it has already a GitHub MCP, which is using

of course your account to commit and everything. So, be careful. Uh, and it also works with any other AI tool like the the MCP is just very agnostic. Doesn't matter which one you use. And, you know, it's just a nice tool to debug your Kubernetes cluster without actually putting all of the effort. And if you're a developer, really, you shouldn't be putting any effort into it, honestly.

Uh but yeah, if you're a platform engineer, sure, you should still know how Kubernetes works. >> So ideally, Copilot will be smart enough to, you know, open a pull request against your own pull request. You'd merge it in your branch and that's how you'll solve the issue. Now, it's not the way I would do it. Uh I would prefer it to tell me the error so I

can fix the issue. But you can you can go full on uh if you you know combine multiple NP servers, you run everything with YOLO and the job is done. >> Yeah. Right. >> And you can use it in the ID too. So if you don't like the CLI for some reason, you use it in VS Code, Jet Brains, whatever you use. >> But let me ask

you something, man. >> Ask me >> like >> ask away. >> This copilot stuff lately looks like clo was happening, >> you know? uh it's just that uh let's say our road maps converge a little bit you >> okay >> we have the same product management in a >> and of course like we all know right now the models are uh uh on the side of entropic

but maybe one day Gemini will come out with a better model so you know you never know which model will be the best so you should use whatever tool is the best for the job I mean like using Argo versus Flux right Yeah, cool. Let's go back to a couple more slides just to summarize. Uh we have a little uh workflow diagram that we designed. Uh it's

just about like knowing what happens in our repos. Okay, of course I'm a developer. Uh in my machine I have Copilot installed and VS Code and I clone the repo. I push the changes to my pull request. Flax operator pulls from the pull request and checks all the data. Flax operator then uses the Kubernetes API to actually use deploy the ELM release. A preview comes up, comments

on the PR as we saw. Uh and then for debugging you use the Flux MCP server which actually reads your cube config. So again be careful the agents can act as you. Uh so you know like keep them in check. Yeah, we have um read only flag for the MCP server. So even if you don't want to create a dedicated service account for it, you don't want

to run it in the cluster and so on. Uh in the MCP JSON config that you have in your repo, you can set minus minus read only. And then it can only suggest fixes. It will never go and do anything on your clusters. So if you are adopting this technology, my recommendation is don't do yolo mode everywhere like yeah maybe keep it only unless you until you

get more comfortable with what it can do. Um our MCP server in Flux exposes all the tools that you would want to interact with Kubernetes is not only about Flux. It can delete resources. It can modify resources. Uh I made it delete all my nodes on a cluster. is it's really good >> self feeling you know >> yeah but um of course you can restrict it with

coarent that's the whole point of running and not you know uh connecting uh uh through other means um so we we at least have this advantage for from the NCP side when we deal with Kubernetes because we can we have all the tooling uh through our back to restrict access and you should be investing in this if you are deploying MCPS no matter if it's flux or

something else. >> No 100%. So these slides are available online. Uh I'm just a little explanation of what we already saw. Um I have a little bit of resources to give. So the MCP docs flux operator repo and the demo or in which you can then pull all of your um all of your rep repositories that we showed today. Uh and you can play with it. You

can use different applications. You know how it works now. And the docs from our friends at Contrplane are really good. So, you know, read the docs. >> Yeah. Or have the AI read the docs. No, no, no. Read the >> I don't understand anything and >> save tokens. Read the docs. They're expensive,