Project Lightning Talk: Argo CD Source Hydrator: Rendered Manifests Made Easy! - Michael Crenshaw

This talk covers a new feature in Argo CD called the source hydrator, which streamlines the rendered manifest pattern for GitOps users. The speaker explains how the traditional pre-rendered manifest pattern, where users make changes in Git and Argo CD renders manifests, often leaves the rendering process opaque to users. The source hydrator allows users to push rendered manifests directly to Git, facilitating easier understanding of changes and maintaining an audit trail. The feature simplifies Argo CD's deployment process with clearly defined application manifests and enables additional automation such as performing security checks. The speaker encourages users to engage with the Intuit team to share feedback on Argo CD and participate in a survey to help improve the platform.

My name is Michael Crenshaw. I am a senior staff software engineer at Intuit and a lead maintainer for Argo Project, specifically Argo CD. And today, I'm just going to talk about one relatively recent feature called Argo CD source hydrator that simplifies the rendered manifest pattern for GitOps users. So, pre-rendered manifest pattern GitOps was pretty straightforward. A user would make a change in Git and I just have

an example here of someone bumping a dependency chart version in a chart.yaml. Argo CD would render those manifests using Helm, JSONet, Kustomize. We also support plugins. And then that manifest, which is flat and on the right side you see just a snippet of thousands and thousands of lines of YAML, that's what actually gets applied to the cluster. And the problem here is that middle section is kind

of opaque to the user. That rendering happens, it goes out to the cluster. They don't necessarily really see or understand what's going on on the back end. So, the next evolution has been that people have shifted to the rendered manifest pattern where you make the exact same change, but then you have some process that runs Helm, Kustomize, etc. And then pushes that to Git before finally handing

it off to Argo CD to apply. And at that point, Argo CD's job is very simple. It's just sending out a flat manifest file. But the problem is you have to maintain that render column. That's custom internal code. At Intuit, it's Jenkins. We run Kustomize build and then push to Git. And it doesn't come with a lot of the niceties that Argo CD does. You don't have

a beautiful UI necessarily. It doesn't retry on transitive network failures, things like that. So, we wanted to bring this column as a first class into Argo CD. And that's what we've done with the source hydrator feature. So again, same old same old on the left side, you're still just making a one-line change in a Argo CD still does the exact same thing that it would usually do

to render, but now it has the ability to push that rendered manifest to Git. And it's literally just a file called manifest.yaml. That's now available to users to read, understand what has changed between each iteration. And if there are issues, you've got an audit trail. You can go back and say, "Okay, this is exactly what was deployed and this is where the problem came from." And of

course, it still relies on the same old Argo CD code that's already really good at applying that code. So, enabling the feature is really really simple. Today, your application manifest in Argo CD probably looks something like the left. You've got a source with a repo, a target revision, and potentially a path. You would just copy and paste those three fields into spec.sourcehydrator.drysource. And dry just means don't

repeat yourself. It's pre-hydrated manifests. That's still your core source of truth, but now you have this additional field called sync source. And this is where your hydrated manifests are automatically pushed to and then synced from. I'm using sort of a variation on literally what Intuit uses today to deploy Argo CD itself. So, we've got 50 Argo CD instances. One of them is literally confusingly called Argo. So,

that's the instance I'm deploying here. And we deploy it in six waves. This is wave zero. And then we would just have the other waves distributed across or the other instances distributed across the other That gives you this really nice additional toolbar in the UI that will sort of walk your user left to right. You've hydrated your manifests. It shows you where they were pulled from, where

they were pushed to, and then the synced panel will show that it synced from that location where we just pushed the Source hydrator also has this extra cute little feature called hydrate two, which allows you to push hydrated manifests to a different branch than what you're syncing from. And this gives you the opportunity to basically provide a gap where you can add additional automation, you can perform

security checks, etc. At Intuit, we use a tool that we recently open sourced called GitOps Promoter to basically use this gap to automate environment promotion by automatically opening and merging pull requests from that hydrate two branch to the sync source branch. So, that's source hydrator. You can come talk to me at the Intuit booth at Argo Con all day today or my team. They can show you

how we deploy Argo CD and use source hydrator. I'll also be at the Argo booth at KubeCon for much of the week, so definitely swing by and talk to us. One key thing I want to ask from folks here, if you would do me a huge favor, that QR code takes you to a link tree. It's got a few Intuit open source links. The top one is

a survey for Argo CD users. We do this every year and the information that you provide in that survey as an Argo user really helps us understand what we need to prioritize in the project, what your pain points are, and helps us focus our efforts. So, if you can't do that, definitely come visit us at a booth and tell us what you love and what can improve

about Argo. Thank you so much. >> All right, thank you very much. >> [applause]