The 10x DevOps Engineer’s Toolkit: Argo CD + AI-Dr... Alexander Matyushentsev & Leonardo Luz Almeida

This talk introduces a new open source project called MCP for Argo CD, aimed at enhancing interaction between large language models (LLMs) and third-party services like Argo CD. The speakers, Leo from Intuit and Alexander from Acuity, explain how MCP serves as a universal protocol connecting AI clients and services, providing a structured way to handle commands and responses. They delve into the functionality of MCP, detailing its simple protocol based on JSON RPC, its use cases, and real-world applications at Intuit. The talk highlights how MCP has streamlined processes for engineers and facilitated assisted troubleshooting in a complex Kubernetes environment. As they address the project's current implementation and future directions, they encourage community feedback and participation in the evolving MCP initiative.

Hello everyone. Welcome to the Argo maintainers track at KubeCon. And today we're going to be presenting a new open source project that we've been working on, which is called MCP for Argo CD, as well as how how we're being using this project to solve real world problems. My name is Leonardo. I usually go by Leo. I'm a staff software developer at Intuit and also an Argo maintainer.

Okay, hello everyone. Let me introduce myself. My name is Alexander Matushchenko. I work at company called Acuity. I'm a co-founder and chief architect. Also maintainer of Argo project. I've been working on rollouts, workflows, and Argo CD for almost a decade. let's get started. I want to start from just giving you overview of what this talk is about. So I will cover half of this agenda. I will

talk about why MCP is useful, why did we even choose to build MCP project for Argo CD, then I will cover the project itself, and share with you ideas we had and the use cases we were hoping to solve using MCP for Argo CD, and then I will get it will get it give it back to where he can talk about how Intuit uses this project and

how they what results they got and how they were able to help engineers to get more productive. And with that, let's start from a little bit of theory. I want to explain why did we choose to even use MCP and what problem it's solving. >> [clears throat] >> as you might know, large language models LLMs cannot really talk to third-party services. They cannot do anything in real

world. They simply get request in form of text and they produce text in response and that's all they can do. So it's kind of boring and not so useful. But there is a simple way to make it more interactive. Request might specifically ask MCP to provide in response some metadata that explains an action that should be executed. And then AI client such as Cloud Code or Cursor

or your favorite client supposed to parse the response and then do the action, call this third-party service that you want to integrate with and provide this response back to a model. And pretty much every company that >> does integration with LLMs already knows how to do it. And it's really not a difficult. There is a little bit of code that you need to write. The problem is

there are so many third-party services that it's nearly impossible to build integration with all of them in a client. And that's why MCP was born. MCP is just a protocol that is intended to be this universal connector between AI client and a third-party service. And so on this diagram I'm trying to kind of visualize it. As you can see, MCP server is sitting between AI client and

a third-party service. In this case it's Argo CD. And so MCP kind of works as a as an abstraction. It encapsulates all the logic that is related to third-party service. And MCP is responsible for making requests and then AI clients work with unified protocol, which is MCP. And in addition to this abstraction, MCP also provides some guidance about transport level. In particular, you can you have it

solves kind of two use cases. One use case is when MCP is sitting next to your client and then in this case you can use simple STD error to send requests to MCP server or MCP can be deployed next to your third-party service and in this case it is possible to use HTTP. There are two forms actually. One is server-sent events and second HTTP2. >> that is

making MCP pretty useful like very I wanted to show a bit more details about MCP mostly to explain that it is really simple protocol. In fact, it doesn't even have its own model. It simply uses JSON RPC message format. So it's a text-based protocol and instead of explaining this format, I can just show an example. So in this example I'm demonstrating how client can execute sync action

of Argo CD and payload is super simple. It just has a name of the action and some arguments. In this case, name of the application that's supposed to be synced. And the response is even simpler. It's just a text that will be bypassed to model itself. So it's really simple. Um and the next question is how does client even supposed to know that it can sync applications

or do other Argo CD um actions? And so this is covered by three kind of MCP principles, entities, tools, resources, and prompts. And so MCP server is supposed to implement so some kind of discovery API that returns metadata about those primitives, tools, resources, and And simplest way to describe tools is uh I would think about them as simple functions that have name and parameter structure. Resources are

kind of a bit more complex. It's a function but it represents a list of resources. It can Calls to resources cannot change anything. They are read-only and they support um events. It's possible to subscribe to list of changes on every resource. And finally, MCP support prompts, which is nothing but a few sentences about a tool or resource that explain to LLM how to use this tool and

what it does. Um I hope um you more or less understand what MCP is and the last theoretical topic about MCP is authentication. At this moment, at least there is actually not so much related to authentication. It's actually optional and there is no authentication if you run MCP as as a process locally and use STD error. And if you use HTTP, then MCP has a way to

use OAuth 2.1 for authentication. But from our experience, the most common um way to authenticate is simply bypass the token to your third-party service. Basically, MCP do not perform any authentication. It simply relies on third-party service to do the And enough of this theory, I want to introduce the open source project It's called MCP for Argo CD and it's available under Argo Project Labs GitHub organization. If

you never heard of it, Argo Project Labs is a home for everything related to Argo. We literally use it to kind of host experimental features of Argo related projects and just optional components that make Argo more useful and and powerful. It's a very new project. We have CNCF channel in Slack called MCP for Argo CD. I think it there is only 16 people. So please feel free

to join. It's very cozy right now and we're looking for for feedback about the Next, I kind of want to show what we've done so far when I'm applying the same model and then want to show which primitives we implemented at Argo CD MCP server. So so far we chose to rely on just tools and do not implement any resources. And tools are pretty much we have

one-to-one mapping with everything you can do with Argo CD CLI or user interface. So you can sync applications. You can ask AI to inspect resources. It has access to logs, live Kubernetes manifests, and I think it's fair to say it's more or less feature complete. And then at this moment we're supporting still working on supporting I would say maybe edge not edge cases like more advanced Argo

CD features such as application in any namespace. But the happy path is already implemented and used in production at Intuit. And we do have prompts that explain various tools, explain how to use these tools. and then final slide related to kind of shows you details of um Argo CD MCP works. So this is an example of a life cycle of a typical request. So if your client

if you as a user ask you through your client to sync application. So AI agent will make a request to MCP server, discover list of available tools and then it will pick the tools that matches and then send the request that asking MCP server to execute the tool. MCP server send request to Argo CD get the response and then it it's responsible to understand what the response

means and it translates it back to a text that's understandable by LLM. So next I want to show the use cases that we kind of tested and we know they're working and I'm hoping you can use them as inspiration to build something more you know that fits your your needs and in your environment. So I come up with three typical ideas. One is kind of I would

say very simple but it demonstrates what MCP can do. So this is a screenshot of cloud code and where I am as a user I'm asking um my AI agent to create an application and just giving it in a text in a free form where to take take manifests, which branch to use and where manifests supposed to be deployed. And um behind the scene cloud code execute

this little dance. It talks to my MCP server, realize what it needs to call and then present me with the result and application is created. Next use case is more kind of real life use case. Here I want to ask my MCP server additionally look at the old directories in my Git repository and create application for each one. And I I'm kind of excited about this one

because as you can see my prompt is still very simple but result is pretty I guess for me as a human it would take longer to create those applications than just type this sentence and send it to my AI [clears throat] final use case I want to show you a mini demo. I was too scared to rely on um AI to execute it perfectly so I just

pre-recorded it. Yeah, yeah, I learned it through experience that. So what I'm doing here is I have and I'm asking my AI to deploy a new version of my application. I knew I know that new version might break and so I'm saying please deploy it but keep an eye on application for some time and if it degrades roll back my manifest to most recent version. And so

as you can see sync just completed. Application did degrade and AI agent keep checking the state of application until it realizes that it actually degraded and now it's roll back AI to previous version. And so that to me is you know convincing enough that AI is not just a toy. It can be used to save time of engineers. And now I want to let you talk about

how it worked in in real life in real organizations. Okay. All right. Thank you Alex. So yeah in at into it we've been using AI quite a while. So we've been trying this to to come up with some use case to use AI with Argo CD a little bit more than a year and a half ago and we wanted to use a use case that was addressing

a real a real pain point for us. And we came up with the assisted troubleshooting tool. So to understand this I want to show you this slides, right? So this in this slide we we have a notion of how big into it operates. So at at the company we have a little a little bit more than 350 Kubernetes clusters. We run more than 3,000 production services between

microservice monolithic applications as well. And that totalizes more than 50,000 Kubernetes namespaces. So this is the size that we operate and I don't have to mention that support is a bottleneck, right? So every every development team at into it we have to provide a support channel in Slack and those channels are full of developers. So you you saw the the the numbers. So you you you we

face all sorts of problems. So support is a bottleneck. Troubleshooting is very time consuming because you have to go and look at the logs, look at the desired state, the live state and understand what happened. There's a knowledge gap. Not every development team have very knowledgeable DevOps engineer. So it's very complex infrastructure that we're dealing with nowadays. So we have to understand about Argo CD, Argo rollouts.

You have to understand about Kubernetes. So this there's a problem there as well. And data silos in the sense that the data is spread across different tools, right? We have data in Splunk for inspecting logs, Wavefront for extracting Prometheus metrics, Grafana. You name it, right? So we're we're all aware of those those issues. So what was our initial solution? So we all understood in the team that

Argo CD is a very powerful API. So from API we also we can almost address the the silos issue. So from Argo CD API we can extract application logs, we can extract Kubernetes events, we can extract the live state and the desired state. And this is very powerful. It helps a lot in troubleshooting. So for example if I want to compare my live state with my desired

state, this is something that we can do. Kubernetes sorry Argo CD API provides that capability. It also allows it has a very simple API to extract narrow log logs, right? So you can provide information like your pod name, your container name and it gives the log for that specific And last but not least smart aggregation. So what I mean by that is we wanted we were really

curious to understand what would happen if I extract all this data which is already available through Argo CD API and we make that available to an LLM and provide a prompt instructing this prompt on how to reason about this this entire data set and explain why this application is broken. Right? So that's that was our So we came up with this. So this it has a lot

of text. You don't have to read through this. That's not the point here. So here I'm just wanted to highlight that we came up with an Argo CD extension. Unfortunately this is not open source because it uses some proprietary API. But that's the first stage that we got into. So this is also not using the Argo CD MCP. So we were using the Argo CD API directly.

And we came up with this result from the the premises that we have before, right? So we're extracting all this data from Argo CD API, providing that to an LLM and see how it behaves. So here in this specific demo application we deploy it with an invalid image tag. So that that the image tag doesn't exist in the Docker registry. was able to provide this type of

answer to to this particular application. So the specified image either doesn't exist in the registry or the cluster doesn't have the correct credentials to pull it. Another interest interesting thing that output is showing us is the ability to consume Kubernetes events, right? So this specific message here is only possible to be provided if the the solution has access to the Kubernetes events because the image pull back

off is just available there. Next example. In this case we deployed an application extrapolating the limits that is allowed for that specific namespace. So it it we define in the manifest it to use more memory than what it should and that's the result that we got. So the resource limits defined in the rollout pod template exceeds the maximum allowed limit. And to be honest when we face

this result I was personally very very well amazed. I thought it was it would be a big hit and it would be a successful project. Unfortunately it wasn't. People were not going to Argo CD and really clicking on a specific button, waiting certain amount of time to get this AI processing the data and getting this result. So the adoption wasn't as good as what we expected. And

we understood one problem. The users that are going to Argo CD are most likely the users that are already familiar with Kubernetes. So they're not really willing to go to a button and a few minutes to to get the the results from from an AI. They go directly to the logs they already know, right? So, the users that are really that would really benefit from this are

the ones coming into our support channel. So, that that was our understanding at that point. So, we wanted to experiment with something else. We wanted to experiment with a different use case which is hooking up somehow Argo CD API in this bot that we use that we run in our support channels and see how it would behave if that bot had access with the same data that

this Argo CD extension has. And at that point it came up we came up with the MCP solution that the open source solution was already available at that point. So, for us it was a no-brainer. So, let's let's give this a try. We had an agent running in one of our support channels that were able to integrate with MCP solutions out of the box. So, that was

our our idea at that point. So, what what was our or what what was the benefits, right? That we were looking for at that point. So, we understood that the MCP would address a problem that we have which is we add into it we run a bit more than 40 Argo CD instances. So, with the MCP we could deploy it as some sort of a reverse proxy.

So, we would have only a single service available able to communicate with all the different Argo CD instances that we're running in the company. So, this is one thing one of the benefits. The other thing is making that entire capability avail available to developers workstation. So, Intuit is investing a lot on allowing developers to have access to all these AI capabilities that are there today. And with

this solution that would be also very easy to provide. Our main idea wasn't to provide an MCP that developers install it locally but to have an MCP that runs remotely, right? So, it would address both problems. The developer problem as well agent to agent communication service to service communication which is the case of the bot that I just explained. Tool listing like Alex explained moments as MCP

provides that ability to describe when to use specific operations. It would also help the developers while using the MCP service in their workstation. And simplify security because at this point instead of dealing with all these different 40 Argo CD instances now we have a single entry point for the MCP service itself. And which which helped us to address the concerns. All right. So, what we came up

with. So, here I extracted some some data from our Slack channel. Mainly for for reading. You don't have to read through through the whole thing. I added in bold the parts that I think are the interesting ones. So, in this particular example here and I'm adding the the time stamp so you you can have an idea how long this conversation is taking and progressing. Um in this

initial example here someone come comes in our support channel and say, "Hey, I'm looking at the Argo logs. I'm seeing those warning messages and I'm not sure if this is a critical issue or not. Can you help me?" And the bot which is powered by the MC Argo CD MCP service now it were it was able to reply to to to the user that this is not

a critical issue. The class I redacted the class name is attempting to cast a byte array into a string during schema schema validation. So, it is only able to reply something like that because it has access to the log. But the interesting thing is also suggesting the user how to address the problem because the the stack trace is right there in the logs. It's providing the user

the the the the developer, right? The the the directions about how to to fix the issue. So, the class I redacted the class name again. Implementation in line 76 and understand when and how the validation of course make sure that this realization order is respected. So, you should go from bytes deserialized object to a schema validation. So, that was a good one a good interaction with the

bot. And the other example I wanted to bring is the when this user came to the same channel and said, "Hey, I have a production build issue and help me to understand why I'm not able to deploy and release to production." And this is the Argo CD link. So, in this case 1 minute later the bot replied to the user saying, "I I don't see one but

two different failures in your application. One with the recent deployment that you try to to execute and another error in the current running version. Which one you want me to investigate?" And the user replied, "Check both." Um after through two and a half minutes more or less the bot was able to reply that um the application cannot retrieve its configuration from Spring Cloud Config server. So, at

Intuit we run a config config server. So, users developers can have things like feature flag configured for example. So, the failure number one there's some bad request. I don't want to dive too much into details but there's a bad request somewhere while communicating with this config server. So, the config server is returning an empty configuration for some reason. So, yeah, I have a hint there. And then

it continues with the failure number two. The current running version is degraded. It also has a a bad request issue and it actually finds the root cause. So, the application is configured to run a URL that is coming from that it points to an E2E environment while this is a production environment. So, it was able to provide the the step-by-step procedure to to the user at the

bottom saying the application is configured with this URL to reach the configuration service while it should be using this one. Okay. So, this is the the level of details that we're able to provide by leveraging the MCP. But now we're facing a new challenge. We There are different use cases in the company that we want to to address. And the recipe in order to guide the the

agent on how to troubleshoot a specific application is now duplicated, right? We have that similar recipe living in that first extension that I that I showed before. And we have a similar thing happening here in the bot. And the new use case will will require us to replicate that recipe elsewhere. So, right now what we're doing is experimenting with agent skills. So, maybe the new kid on

the block here. Uh so, we're experimenting here. So, the the the agent skill is basically a new specification. So, with this specification it allows us to define markdown files and agents will able to consume this markdown file and act upon. So, you can be as precise as you want using agent skills. So, we're mixing right now the usage of agent skills and MCP services. So, in this

particular example here I'm showing I'm instructing how an agent would be would need the the steps that the the the agent would need to do in order to extract the the Argo CD base URL as well as the application name. So, those two informations are necessary in order to communicate with the MCP service or the Argo CD MCP service service expects the base URL to be provided

as a header and app name needs to be provided in some operations as in any parameters. So, basically here in form of a skill guiding the agent on how to do that specific task. The And the other task that it does is this one which means Right here we're instructing the agent on this specific recipe that was that that I was mentioning. So, here we're basically saying

whenever an application is degraded this is the steps that you need to to to execute in order to understand and extract the most out of it. Um to in order to to provide the the results from the user. So, wrapping up As I said before, right? So, we started with that initial approach with Argo CD It didn't work. We needed something more more dynamic more flexible. We

started to address that use case with the bot. And now we are having to duplicate to to to replicate this recipe on how to extract and troubleshoot applications into different agents. And that's the direction that we're going. But maybe this is topic for another talk for another talk. Yeah. please consider us giving consider giving us feedback. Please scan this QR code. You're going to have access to

this presentation. You're going to have access to the repo that Alex presented the open source project as well as access to the URL to to give us feedback. Okay? Thank you so much for attending. >> [applause]