Kill the Ticket Queue: A CNCF Blueprint for Self-Service Platfor... Bhavani Indukuri & Aparna Prabhu

In this talk, Bhavani Indukuri and Aparna Prabhu from DigitalOcean discuss their experience in developing a self-service platform to streamline developer environment provisioning. They address the challenges faced with traditional ticketing systems, where requests for environments often lead to significant delays due to back-and-forth communication. By leveraging technologies such as Argo Events, Argo Workflows, and vcluster, they created an automated solution that reduces provisioning time from several days to under ten minutes. The presenters emphasize the importance of guardrails implemented through tools like Kyverno to ensure resource limits and security compliance, while also discussing the integration of observability tools like Prometheus and Grafana to monitor the system’s performance. They demonstrate how their approach enhances the developer experience by allowing for immediate access to environments and ultimately freeing platform teams to focus on innovation.

Hello everyone. We really feeling very happy to see how it's full of audience and thank you so much for you know coming this far to this room to attend this talk and we are not going to disappoint you for sure. So today we are going to talk about how we kill the ticket queue and build a self-service platform that helped us to provide a the best developer

experience to our developers. So this is Bhavani Indukuri. I'm a senior software engineer two at DigitalOcean. I'm also KubeCon India 2025 co-chair and a CNCF ambassador working as as a organizer for women in CNCF. Hey hi everyone. I'm Aparna Prabhu. I'm a senior engineering manager at DigitalOcean where I manage the storage and platform teams. I've also spoken at a few CNCF events and I was a keynote

speaker last year in keynote in KubeCon India. Okay, let's get started. So this is something probably many people here have I hope have experienced at some point of time. So as a back-end engineer, you have merged a new feature or in it's in you want to put it in staging, you want to test it and you need an environment. You need a customized environment for your feature.

You want to test it. So what do you do? You submit a ticket. There is in most likely there is no specialized queue for this ticket. So it joins a queue where it is competing with other like production issues, feature requests, what have you. Once it is the turn of your ticket, there is a lot of back and forth. Now people need clarifications. What kind of environment

do you want? What kind of CPU limits do you want? How much memory do you need? What kind of database accesses do you need? What are your networking needs? And then there is also the security review. Here's where things like Rback, IAM, firewall rules, all these come into the picture. So as you can imagine, this back and forth takes considerable amount of time and it happens asynchronously.

Okay? And quite a bit of time has elapsed from the moment the developer realized that they need an environment to this point. And mind you, we are here the actual provisioning of the environment has not even started. So finally, after all these clarifications are done and like four teams are involved, the developers team itself and then the platform team, security team and then back to the developer

team, whatever. Finally, it's time for the manual provisioning. Now the manual provisioning itself is not a very laborious process, but it is done by hand. Somebody in the platform team is stitching all these requirements together and they're able to provision you know within a few hours. Kubernetes is not the problem, but the operating model is flawed and that is why it takes it takes so much of

time. So let's let's put [clears throat] some numbers. We were following this model at some point and it used to take seven days to provision because like I explained, there is a lot of back and forth, a lot of clarifications that you need to seek and then finally you have to set up your environment. At the rate of 200 plus tickets per month, this was putting a

lot of load on the platform team and platform team was not really spending considerable amount of time doing what they are supposed to do, you know, doing innovative work. They had basically become a support team for the other back-end teams and as you see that's not a good use of their time. And it's not just the platform team that was getting sabotaged. There were also other teams

like I explained, security team, the developer team itself and you know, a lot of back and forth. So this is not scalable. This is not a good use of their time and every new feature that you add or every new whatever if you want to add something to this setup, it adds a layer of complexity and overall the time increases a lot. So this model is flawed.

well whatever the Aparna was mentioning that so the legacy model that we have followed is like more of the manual human switching between the things. But we were thinking of like okay, we need to eliminate this process. Then we thought of self-service. So self-service looks very fancy and everyone wants to implement it, but if you don't have proper guardrails implemented, then the system become you know, becomes

chaos. For example, in our old legacy system, so those guardrails, the security review and all of that was manual. So that's the reason we we were not able to scale that solution. So now you know, I'm going to explain about how we have designed the system to provide developers their developer environments using all the CNCF projects. If you look at the architecture diagram, so so we have

our in-house you know, we are from DigitalOcean, so we have our in-house DigitalOcean Kubernetes cluster. So we have used the DigitalOcean Kubernetes cluster to host you know, this platform and we have a control plane platform cluster and the worker DOKS cluster and where the platform DOKS K is DOKS cluster is responsible for all the major components of the you know, this platform is being deployed there and

the worker DOKS cluster is dedicated for deploying all the virtual clusters on top of it. The initial place that this architecture starts is you know, providing the developer providing a template like the golden path that we call using [clears throat] the backstage you template and and that emits a webhook event. So we have Argo events in place that that will recognizes the event happening and using the

sensor, it is going to trigger the Argo workflows. So here we have switched from backstage to an event-driven automation and we have a workflow in place that is going to call our V cluster you know, creation template. So finally, we we have come from backstage to Argo events to Argo workflows and we have deployed the virtual clusters here. So I have been telling like self-service creation is

easy, but how do we make sure that it does not cost us a lot and provide us a disastrous you know, solution at the end. That is where the guardrails is very important and and we have to somehow put some limits to the developers to not overuse it and all of all the stuff. So that is you know, that can happen using Kyverno and we are using

that to make sure that you know, all these guardrails in place without the manual intervention. Finally, so we have deployed the platform, we have enabled the policies now. We we need a place where we need to know how this platform is actually performing. How it is creating V clusters, how everything is performing from Argo events, Argo workflows and all the stuff, backstage. That's where we have built

an observability platform. I mean, we have added this observability as a platform feature using Prometheus and Grafana and we have been working towards improving this through integrating backstage and OpenTelemetry and you know, providing a multi-level dashboard using VictoriaMetrics. That is in progress right now. So finally, everything is declarative and we are using GitOps and Argo CD is in place. It's like a lot of things stitching together,

but this really helped us to provide the best platform Cool. So let's spend the next few minutes doing a little bit of a deep dive into each of the components that Bhavani explained in the architecture diagram just now. So let's start with backstage. Backstage is a unified developer portal. It is where your developers come to create, discover and manage all the infrastructure and all the environments that

you have. you may ask me like why can't I just have my own UI it you know, in this day and age of AI it's not hard to spin up a quick bunch of templates using UI and use that. Like why do we need backstage? Because backstage is much more than just a UI. It has a service catalog. It has many templates. It has documentation and it

also has plugins. So you can >> [clears throat] >> integrate your entire experience in one So that's how backstage sort of it set literally sets the stage for changing the narrative from dependency to self-service. So moving on from backstage, all right. So your developers go to the you know, they fill a form or whatever. I mean, in one way or the other completely that is set up

by you. They define what kind of an environment they want. But you need something to constantly be watching these user actions and to trigger workflows that actually provision the environment, right? That is where Argo Events comes into picture. So, um the thing about Argo Events uh I'll I'll I'll tell you later how Argo Events and Argo Workflows together works like a nice event orchestration engine. But first,

let's talk about Argo Events. Now, please bear in mind that this is all very decoupled. So, your UI does not have to know about your event orchestration engine and none of them need to know about your actual infrastructure. So, it becomes very easy to maintain. Um another thing uh we like about Argo Events and another reason why we used Argo Events is because uh it not only

supports Backstage, but it also supports multiple places from which >> events can be triggered. So, it could be a webhook, it could be a cron job, or it could just be uh Kubernetes itself like some change to the configuration that triggers the events. Argo Events also provides reliable event processing. So, your events are not lost. Uh there are, you know, reliable retry mechanisms in place and of

course, like any good event orchestration engine, um itempotency is also provided. So, at an Argo Event level. Um so, this is how Argo Event nicely fits into the process and takes this entire system one step further in the dependency to self-service journey. Okay, so you have something that is watching the user events. Now, you need something that is actually going to work with the infrastructure and provide

the um platform or the um environment that is being desired. This is where Argo Workflows comes into picture. Argo Workflows uh is um if I may, the um heart of the entire system. It is the actual workflow And uh Argo Events together with Argo Workflows um it it they're able to manage all the in a [clears throat] um very scalable and very robust way um with all

your retry mechanisms, itempotency, uh etc. in place. Uh so, think of it like Temporal. Though Temporal and Argo Workflows is not really an apples-to-apples comparison, uh but the reason I'm bringing this up here is because like Temporal, uh this also works at scale the setup and uh you know, your events are not lost. Uh if something happens during the process, then safe retries ensure that um uh

the system works properly and uh itempotency further adds robustness to the system. So, okay, before I forget, uh Argo Workflow also supports DAG. So, whatever is asynchronous can be run Uh so, it saves time as well. And finally, everything is declared as code. So, you know, um I don't have to go over the benefits of why something has to be declared as code, but this nicely fits

into uh your all GitOps best practices. And >> since it's declared as code, uh you can review it, you can version it, uh you can audit it, and everything is transparent. So, together, this Argo uh works as a very nice, beautiful event uh which is not only fast and reliable, but is also able to scale, as you can see in the next slides that we're going to

present. Yeah. >> Now, um so, we come from Backstage and we are now at the place like where we actually need to spin up the environments. Um so, you can think like why can't I create um you know, one cluster per developer, but we were thinking of more like cost-effective solution. Um so, that that's when we thought like vcluster really fits in into our platform. Because it

will give a Kubernetes control plane isolation from your host cluster and it it makes sure um you have that isolation in place and of course, it is really good for multi-tenancy. The reason why I'm telling about multi-tenancy is um you can think of like I can deploy different developers into different namespaces by putting uh you know, uh namespace quotas, network policies for the namespace, or providing those

role-based access for the namespace, but it still uses the same um cluster level uh I mean, custom resource definitions and it is going to uh share um you know, the entire cluster level. So, if something happens to one developer, then it will impact the whole system. So, that's when we thought we we we need to go with a cost-efficient solution here, but also making sure we provide

that, you know, really good uh user isolation. And that's when this vcluster is what we really explored and we have implemented. Uh vcluster is very helpful for you in terms of one thing is like it is very lightweight that you know, you can think of it like as a virtual machine how we came in from physical machine, but it's more of like it's a layer of isolation

in uh you know, Kubernetes control plane level, but not the resources. But uh if you think if you compare this with providing an entire cluster to a developer, you're still saving a lot of resources to uh those the Kubernetes uh control plane resource and everything you're saving and you're using um you know, kind of resources that are very shared that you can definitely, you know, differentiate using

labels and all of that. So, this is how we feel uh vcluster is, you know, very uh useful in this in our platform journey. And of course, this is cost-efficient and it's easily provisioned and it's very lightweight. So, yeah. Coming to Kyverno. So, uh so, I'm going to show uh you know, a demo at like how this end-to-end flow in the end of this presentation, but I

want to make sure that we can use Kyverno at a three levels. One is like we can mutate, use it for mutating, which means like we can update the um existing resources, we can actually generate new resources, and we can actually, you know, uh you know, you know, before we create the instance, we can check like like the validation part of Kyverno. So, we can check whether

it's going through certain uh criteria or not. So, these are few of the criterias we have listed, but we can actually we have a lot of other things in production uh to fit into our needs. Few of them are like definitely mandatory CPU and memory limits because they the user shouldn't use them, you know, overuse the resources so that the other uh users will, you know, get

affected. So, we have those limits and we have the ownership and label enforcement. So, we uh we we should make sure that these are everything comes from a trusted container registries and um definitely the, you know, we should block the privileged containers and network isolation. And finally, you know, this part I I'm going to show you in the demo, the time to live. Because so, you're giving

um your user, right? So, you're you're giving a user an environment, but who should take care of cleaning up the stuff? So, when it is done, uh we need some some automation to make sure that uh when when the usage is done, it should be auto-cleaned. Again, there there should not be any human being there to clean that stuff. So, that's where we enforce this TTL-based uh

lifecycle management that which which I'll clearly show you during the demo. And finally, um determinant deterministic image versions is another part. So, this way like I just want to show Kyverno makes these guardrails as code that helps you automate the process for your platform. And finally, the observability side um because like here you know, we we need to make sure like how our platform is performing, how

many, you know, vclusters are getting created, how they're being utilized. >> Those metrics are really helpful to understand and improve ourself as a platform team. That's what we tried, uh you know, we have done with observability using Prometheus and Grafana and we keep improving this um you know, observability platform, I mean, feature in our platform. Finally, so for every uh journey of uh achieving something, we always

have the lessons. And we always try to improve things. That's what, you know, the experience user experience uh that we give using uh you know, by uh taking those lessons that we learned during the process. One of the first point I would like to make um you know, point out here is design for failure. Because when we think like we have we have deployed a platform and

everything will get successful. That that won't be the case because we we always, you know, see some failures and we should our system should be resilient to, you know, take those failures and how it automatically react is something, you know, it's a you know, biggest benefit of being uh really good platform. So, so there is one example when the workflow uh sometimes got timed out. Uh it

it it used to create uh because it gets retried and it used to create a duplicate environments. There are certain scenarios that we um we have learned to do a lot of really good testing and then we have implemented as a our platform as a And lifecycle discipline is already I have mentioned about the time to live. So, so how we have automated that process is something

I'm going to explain next. And finally, we keep evolving those policies. Initially, we we just been very strict with policies, but later like we just need to make sure that we should give the, you know, a freedom to developers by making sure the mandatory security guardrails are in place. So, that's how we have learned the lessons in this journey. Cool. So, let's put some of these impact

whatever Bhavani spoke about in numbers. The provision time as I was explaining earlier was several days. Now it is almost instantaneous. It's less than 10 minutes. And since it is fully automated and everything as you will you're going to see in the demo, that's right after this slide. Our developers are unlocked almost instantly and platform teams are no longer overwhelmed with huge number of support tickets. There

has been a 90% reduction in infra Mostly the tickets that we are getting now it's for the corner cases and for people who want exceptions to the guardrails. I know it says 50% infrastructure cost reduction, but these slides were made a few weeks back. As the adoption has grown significantly, especially over the past few weeks, the infrastructure cost reduction is actually a lot higher than this. This

is mainly because of people are we are enforcing guardrails, so people are provisioning exactly how much they need and the TTL is also making sure that resources are cleaned up once they're no longer being used. But beyond all this, these are numbers, but you know, I'm sure all of you can relate. Engineers like it when they're challenged. And expecting platform teams to constantly keep provisioning environments was

not a Now with this much of automation in place, we are not only unlocking the developers instantly, but our platform teams are also using their time in doing more innovation and you know, building better platforms. So that is intangible and you know, that cannot be put in numbers, but according to me, that's probably the most impactful aspect of this automation. Yeah, so finally I'm going to show

you live demo. So I hope it it works fine. Yeah, this is not recorded. It's a live demo, so fingers crossed. this is the screen So let me share the screen. Okay. So yeah. So first time I'll be starting it from So now this is the backstage UI. Where I can just sign up. I'm sorry. I can't see. >> [laughter] [snorts] >> Guardrails. Yeah, finally we are

in. >> Now you see this template. And this is the template that we are talking about. So let me choose this and I'm going to fill few details. So So here we are giving you know, what is the profile they need to know whether it's low resources and development or production or multi-tenant. So right now this is a demo, I'm just putting the minimum resources. This is

the time to leave that I've been talking about like how we we're going to make this thing you know, till the end to make sure we are applying the time to leave. Just putting one hour. one second. So now I just created it. I'll just go back here. And I'll show you how this is reacting in the background. So if you see here So this is the

actual log of I hope you are able to see. One second. So this is the actual log of the you know, Argo events where it is showing an event has been emitted and it is successfully dispatched the event bus. So the Argo event was able to take that event and it it went and you can see a workflow is getting created here just now. It's been 20

seconds since it is right now it's created this workflow and you can see here these are the actual you know, once the workflow has created, what are the different steps it it has done to create the actual pods and to make the V cluster up and running is something you can see here. So these are happening at the moment. See those are getting created and I'll show

just maybe one second it will be done, so I can show the actual logs of them. One of the deploy V cluster. So now you can see if you see this there are a couple of steps as as we shown during the workflow, what are the different steps we have put. We have put you can see here there is like deploy V cluster, set up namespace and

verify the cluster. So all these steps have been run during this process and once you look at the logs of the actual deploy V cluster, it it it is saying that the deployment is completed. You know, that I can connect this V cluster using this. Now let's connect to this. Now actually it is switching to that V cluster context. So now we are inside that cluster. So

for the demo we have deployed this in a similar cluster, in the production they so users will get you know, our system will give them the cube context, so they can just access that as a normal Kubernetes cluster. So they they don't even know whether we are using V cluster or anything. It's just an isolation that we are doing from the platform team side. Now I am

inside this cluster. If you see So I have only like my pod doesn't have any other things. Now let's deploy one one pod. See the pod got created and this is like they got an environment. Now they they are ready to use this environment. And you can think like what if the scale goes high because in our environment a lot of teams requesting for environments like if

one D case cluster is not able to do that, so we have an automation that will actually spin up the you know, other D case clusters using there is like just one command that helps us to spin up number of D case cluster once we reach the maximum capacity of the cluster. So at the end user level they just don't know anything. They just get a cube

context file. They just log in and they just use the cluster. So that is how this this has been designed. Now let's go to the Kyverno side how that is being implemented. If you look at you know, the form that I have shown in backstage, it has a time to leave field, right? So if you look at this thing, now I'll I'll just see. Okay, I'll just

go out of this Let me do. I'm just out of this thing. Now I'm inside my cluster. I'm just trying to Get cluster policy. I'll just show whatever the cluster policies we have. We have two cluster policies. One is the mutate cluster TTL labels and then second is the validate cluster TTL. So we are just applying each V cluster namespace with some you know, annotations that I

will show you now. And we are just validating whether this V cluster has those, you know, annotations or not. So, let me show you the 1 second. Yeah. I'll show the name space of the V that we just created. So, this is the and you will see Sorry. So, this is the name space. See how Kyverno has provide added all these TTLs and everything. And it has

put at what time it gets it gets created and what is the, you know, type TTL duration that it has to be. Now, uh so, this is how Kyverno has added. Then, how Kyverno has actually made this to delete the environment. If If you just see I'll just show you. So, this is the Kubernetes jobs. You can see those. So, we have made an you know, enabled

a a job that keeps looking at at these, um name spaces and it looks through those annotation that are added by the Kyverno. And it based on, you know, it will calculate the expiry date and if it reach the expiry date, it is going to clean up. So, if you look at here, I'll just show the logs of it so that you'll understand. I think these are

done already. Yeah. Okay. So, these are done, but it will it will So, these are run before that. That's why it's not showing it, but it will show these gets cleaned up. It's been, uh you know, um 56 minutes till this environment is up like now that kind of log it will show. So, this is how we have implemented everything in terms of Kyverno and the platform.

Finally, I will just show Bear with me 1 second. This is live I'll just show the monitoring so we are good. Yeah. So, let me open the monitoring So, now you can see this, right? So, I just did some magic in the background. Yeah, that's fine. So, it's like as I shown in the slides, you know, the monitoring looks like this where you can see, uh Okay,

let me put it back. So, where you can see, uh you know, at each level like Argo events, Argo workflows, and backstage, and as well as um you know, V cluster side how metrics and everything is got provisioned. So, finally, I would say So, we just want to make this point. Tickets don't scale, but platforms do. If you think of a platform as a, you as a

product, then you keep iterating through, you know, improving the platform to make those user experience the best. And to give them a better developer experience so that they can actually think of what matters to them the most. So, now they can happily ship features. And this is not only This whole workflow is not only work for this environment creation. You can do automate a lot of stuffs

by adding a lot of templates and you can do a provide a unified experience to your end user customers. Thank you. Thank you. >> [applause]