We Deleted Our Observability Stack and Rebuilt It With OTEL: 12 Engine... Yash Sharma & Kunju Perath

This talk covers observability within DigitalOcean's managed Kubernetes service, known as DOKS. The speakers, Yash and Kunju, discuss their experiences managing over 20,000 clusters across 13 cloud regions and the importance of detecting issues before they reach customers. They explore the challenges faced in telemetry and observability, including inadequate audit logs and lack of visibility in worker nodes and control planes. The implementation of OpenTelemetry significantly improved their observability architecture by providing better metrics coverage and simplifying data processing and storage. They also address the benefits of transitioning to an architecture that ensures quick issue detection and improved performance, ultimately enhancing the reliability of their managed services.

So, let's kick kick off our today's session. And today we'll be covering about the observability part that we do at DigitalOcean uh for our managed community service. So, before going ahead and we start the session, I would like to give a quick introduction about me and my co-speaker. So, I'm Yash. I work at DigitalOcean as a developer advocate and I'm also a maintainer of CNCF projects as

well. So, I've been very very uh contributing in CNCF for a while as well. With me uh I have a great uh engineer as well, Kunju, uh from the DigitalOcean managed communities team itself as well. So, what we will be covering today, we will be covering like what actually DigitalOcean managed communities service is and why observability and uh the title like why we went with the hotel

and everything and uh our telemetry part. We will go through the architecture as well, which will be more like a case study. And uh we'll be talking about the hotel and at last you will see what are the lessons we have learned throughout those years while working at such large scale. Moving forward, I would like to give you a quick idea of uh at what a scale

we are going to talk about at today's talk. So, to give an idea, the DOKS team is of 18 people right now, which manages more than 20,000 plus clusters running inside DigitalOcean across the 13 cloud regions. We are upstream certified with the CNCF uh certified host for the communities as a management platform as well. Uh we power open source. We love open source as a DigitalOcean. I'm

not sure how many of you know the Hacktoberfest, but we love open source. So, we are powered by open source as well. And uh we are achieving 99.97 uptime for our DOKS platform. Uh at DigitalOcean we uh promise for 99.95, but last year we managed 99.97, which was really really great. And uh dog food side is like we build on our own products. So, if you know

the droplet, it's like an instance uh for DigitalOcean and we use our own products to build the products. That's it. now you have got the idea of at what a scale in today's session we'll be talking about and why actually it matters and it becomes really difficult when you're talking about the observability of 20,000 clusters. So, just to give an idea why observability for us, it's like

detect issues before it gets to actual customers, right? You want to make sure that we are able to detect the issues before even customer coming us to us and complaining about it, right? So, that's why observability is really really important for us. It's like self-critic. Help us improve by ourself before customer comes to us and says, "Hey, this is this is not working and what's going on?"

So, observability this is for our managed backend services, the observability part we're going to talk about. So, now I will highlight two things which are like what was working and what was not working. And what are the changes team made uh throughout those years to actually make what's is not working we need that and how we did it. So, just to give you an idea like what

was working. Uh our control plane metrics, we have no issues. We have like uh we have good observability on our control plane and uh we have the control plane longs uh long-term storage as well. But, the only issue with that was like it was not searchable. It was very very large file-based database. DOK backend services with searchable logs uh we had. So, what do I mean by

uh DOK backend services? These are the internal uh tools and services which are responsible to provision the underlying infrastructure within the DigitalOcean itself. So, we had the good observability on that part as well. So, if I give you an idea on what what not what was not working, so we were not able we don't used to have very good communities audit logs. So, let's say if customer

comes to us and says, "What's going on? What happened like 5 days ago?" Uh sometimes we were not able to answer that question because we used to have a file-based uh storage and we used to lose them uh when communities cluster restarts. Searchable container logs for the control plane components and the communities events logging we know for the control plane as well. Uh we were lacking on

that as well. Telemetry in the managed part of the community and customer data plane. So, as I said earlier, we want to make sure we are answering our customer when it comes like, "What's going on?" So, uh we were lacking the telemetry on the customer data data plane managed side of the cluster. Uh telemetry for the communities worker node uh which can help us avoid asking uh

requesting our customers to SSH in their systems so that we know what's going on. Uh if we have a better observability, we don't need to do that and we will catch the errors and issues before even customer knows. So, now I hope you all got the idea of about, what was the actual issues we were facing, and what was we were lacking. Now, I'll ask Kunju to

come over and talk about the architecture, what are the changes we have made, and this time, and how we are able to achieve those uh remove those issues with at at our particular platform. So, Kunju, over to you. All right. Hello, hello. Um yeah, let's talk about the the actual underlying architecture and how things were and what we had to change to uh improve our setup. Um

so, yeah, just a quick thing I wanted to just to kind of give like cuz I know you guys are kind of coming from the customer perspective, right? Um so, you're like, you know, what do you get after you create a DOKS cluster? Um Yash mentioned earlier how we had that DO those DO managed services that are all responsible for provisioning cluster and stuff like that. But,

what about in cluster? Like, what do you as a customer see after you create a DigitalOcean Kubernetes cluster? um with the architecture, I've kind of simplified it down kind of into two different parts. It's a lot more complicated than that. But, our goal here is to talk really about the observability and hone in on that. Um yeah, so like you as a customer, you just created your

DigitalOcean Kubernetes cluster. You downloaded the kubeconfig. You're inside the cluster. You do If you do kg pods, you're really not going to see much else. But, then you might go into the kube system namespace. Uh you know, and where you'd expect to see all your like uh components and stuff like that. And then you'll get a different view kind of because you'll notice that uh you'll see

some services here uh like Cilium connectivity and whatnot. And uh but you won't see etcd or API server. Um so, you might be like, "Where did those those things go?" Well, they're with us. Uh because uh we manage the control plane and so we have control over uh those things and um it just makes it easier to have that kind of separation. Um so, yeah, the control

plane is actually hosted in another Kubernetes cluster. We use a cluster API. Um I'm not going to get super into cluster API, but the idea is at a high level, it's basically we wanted to uh basically manage uh these customer clusters, your your Kubernetes clusters, with another Kubernetes cluster. And so, it's the whole idea of managing Kubernetes with Kubernetes. It's called kubeception as if Kubernetes wasn't already

complicated enough. But, it uh it's actually becomes very intuitive once you drink the Kool-Aid, I guess. But, yeah. So, let's talk just kind of like the architecture. The two component The two areas I really want people to kind of just keep in mind is there's the customer data plane. And so, the Kubernetes clusters you see at the bottom there, those are the they're just like many different

customers uh Kubernetes clusters, basically. And then there are their control planes are all hosted on another Kubernetes cluster. This is the control plane cluster. That uh is what we call it. Um so, as an example, you know, we have cluster A, B, and C and whatnot. Uh so, for cluster A, it's inside the control plane cluster. What's going to look like any other Kubernetes cluster. Uh it's

going to inside a namespace that's basically the cluster's name. Uh it's going to have like a cluster A's API server, it's etcd, it's cluster autoscaler, all the other components that we have there. so, yeah. Um so, how did we do telemetry before? Basically, what we did was we just had like a big cuz we have more than one control plane cluster, uh especially when you have DigitalOcean,

we scale it to like different regions and stuff like that. Um but in inside a single each and every single control plane cluster that we had, um we just had like one big Prometheus instance. And uh it was just yeah, we had one big instance that basically just uh took all the telemetry that we were collecting. And I want to I want to also reiterate uh the

telemetry we're collecting is from like the managed control plane. Like DigitalOcean like stuff like that. The stuff we're managing. Um and the stuff that's in the kube system that namespace that you saw earlier, that is like that's controlled by us and managed by us. Uh so, just to reiterate, not the customer application logs, not the customer data, not that stuff. Our stuff. Um and so, yeah, we

would literally shove all this into a big Prometheus instance. And um And this is how we collected like control plane metrics and stuff like that. So, I think the thing to kind of keep in mind is like like what was kind of wrong with that? I mean, it it didn't like it wasn't really the issue of like having a single Prometheus instance. Like there are ways to,

you know, uh scale out Prometheus, right? Um we we also had the compute cuz we're DigitalOcean. So, we had like dedicated uh like we had like a dedicated uh node a machine just to run Prometheus and stuff like that. Cuz we do collect a lot of telemetry just from all the Kubernetes clusters that we have. Um so, that it wasn't really just the issue of scaling Prometheus

that was really blocking us. It was more of getting It was more of like getting telemetry at different layers of the stack, right? It kind of goes back to the the whole idea of cluster API. Those three turtles are just like the the cluster API logo, basically. So, it's like we wanted to collect telemetry under in basically in the control plane cluster, which we were already doing,

but we also wanted to collect telemetry in the kube system namespace inside the customer cluster, the stuff that we manage cuz without that we don't we don't really know if something breaks. Um we also wanted to know about we also wanted to collect telemetry on the customer worker nodes um because if something went wrong with the machine or whatever and later subject ideally we catch it but

if something uh some new kernel update comes in and we don't know what's going uh like we don't really have alerts or telemetry for that. So it was just not ideal. So it wasn't really about just like the volume of telemetry it really was about it was it was really about getting telemetry from different parts of our stack and having multiple Prometheus instances in different stuff just

it just that was that was getting expensive uh very expensive I think. Um it's also and it's also not just about collecting metrics it's also about collecting logs and stuff like that and as Yash mentioned earlier our uh like audit logging telemetry was just kind of just not where uh like we were just not where we wanted it to be it was just like a file-based uh

audit logging as he said earlier and we wanted to have a way to retain that better. So uh yeah like I was saying these are so we had some key gaps with this setup right no worker node telemetry no kube system telemetry uh a lack of telemetry in the in the control plane like we we had metrics but we didn't have persistent like searchable audit logging right

we had audit logging uh but it wasn't easy to search it was literally stored in our object storage solution so it was one of those things we dreaded having to look at cuz then we had to go download it and search through it so it was just not good and um also no uh like event logging which is much easier to look through than uh uh the

container logs for all the different services that we have running so that's kind of where our metric gaps and so how do we solve them? So there's definitely many ways to go about it we went about it with the open telemetry and so we decided to kind of just change our setup to use just embrace hotel right? Use hotel to cover these metric gap areas worker nodes

kube system and then port over what we had that we liked from the control plane telemetry uh but now just in hotel right? yeah so thankfully this was actually a very good experience um embracing open telemetry was not actually that difficult there's it's just it's the project has been around for a while and it has a lot of good documentation uh good community and uh a lot

of plugins this was where I think it really helped us out and made the transition so much easier. We used uh what's called the open telemetry contrib image we have links at the end um but it basically is an open telemetry uh when you it's basically an open telemetry image that has baked in basically all the community plugins. So during the initial development we don't have to

like uh basically like make sure oh did we properly add that uh dependency and it's all baked in so during initial testing and like building it was very convenient to just play around with the technology initially. as I said a lot of like wide variety of open source plugins for pretty much every uh imaginable telemetry and for anything that was lacking there was a way to build

it and add your own custom collector and this is something that was very useful for us when we founded that we needed Kubernetes audit logs to be done better basically. So yeah um we have like a uh like a single an example uh just cuz uh some of you may not may not be familiar with how open telemetry has structured its um kind of like what an

open telemetry pipeline looks like but it's basically kind of separate into three simple ideas the receiver the processor the exporter. Uh the receiver receives telemetry from whatever source and then we have a processor that actually processes it changes it you know you can add in certain fields we use it to add in certain managed fields or filter out certain telemetry and then export it. Uh and so

I don't know if I mentioned it earlier but when we collected from the into the in the old architecture we had that Prometheus instance and then we had our DO backend basically scrape and pull that telemetry in. The new architecture we basically just send it over to the DO backend and that's where like our our basically our observability team basically you know manages it polish polishes it

and put it puts it in nice dashboards and stuff and stores it long term. All right yeah so oh yeah just a quick another thing about the whole open source side of things these are some plugins that were really really useful for us and made our job easier. One was of course being able to uh port over what we had already in terms of like Prometheus uh

in terms of our Prometheus metrics the Prometheus receiver was really useful it just you know it's kind of it's it's very similar to just having a like in Prometheus where you can define like a Kubernetes service uh discovery configuration and you can just redefine what you already had um prior to this I think we were using kube-prometheus and so that was uh it was using CRDs inside

the cluster uh we kind of switched over to just using almost like very simple traditional Prometheus configuration um but it was very easy to set up it was almost one-to-one and uh so that way it made it very easy to port over what we had transform processor this is what we use uh basically like those custom attributes right? Like an example is that uh you know for

our customer cluster I want to know what control plane cluster it's the it's control plane is hosted on right? And so that telemetry I can add in a field basically like a control plane cluster ID and tell me like what it belongs to so that's really nice there um filtering out any other attributes you don't need is a filter processor batch processor very very important to have

it early on so that you're batching your telemetry not just spamming and you know lower package overhead as well with batch processing and memory limiter so that we are not OOM killing the uh the collector something I wish we were paying attention to earlier but definitely something we were only really paying attention to once we had some OOM kills um and then of course OTLP exporter uh

the the format the telemetry data is stored in a different format it's not Prometheus exposition format it's it's called the OTLP uh format by open um and so we needed to export it to basically the DigitalOcean observability backend which supports that and so we just literally pass in an endpoint be like hey this is where you want to send it to. and so that was that was

very useful and then also S3 uh exporter which is uh what we what how what we have a DigitalOcean equivalent is called DigitalOcean Spaces and so that's where uh we want to have like basically uh that's where we basically store like our telemetry for like long-term storage not like super long or anything but you um just so we have a way to quickly get to it. Um

yeah so like kind of adding onto that example you know you see the the first green uh block there you have uh the Prometheus receiver you know it collects literally from the metrics endpoint that those services are uh sending their telemetry to we're just scraping and pulling them in we're transforming them right? We're adding in um uh we're updating the resource attributes. Uh hotel actually has this

ability of there's attributes on the actual raw telemetry and then there's something called resource attributes which is just identify the resource. Um I I'll I'll definitely get into a little bit more about that later but the idea is we wanted to add more metadata and so we could do that very nicely with the and uh yeah and then we export it to our DigitalOcean backend. uh the

pattern that we used uh it's a decently popular pattern uh it's just called the gateway agent pattern the idea is that we basically have open telemetry collectors that act as uh uh lightweight agents that are just relaying telemetry their job is very much the responsibility is just grab the telemetry like that pipeline earlier just use a receiver pull the telemetry send it off and so then we

actually have a gateway collector that is uh stateless and basically uh just it's it that's the one that's actually going to send it to DigitalOcean observability so agent collectors send it to the gateway the gateway sends it to uh to DigitalOcean observability. you'll notice that there are the collectors there's collectors in the control plane these are responsible for grabbing the stuff that's in the control plane like

ETCD API server that stuff those are there's open telemetry pods running that are uh collecting data there and sending it over and then on the data plane uh worker nodes is an area that I mentioned and that's because on the worker nodes running as like a systemd service is actually uh another hotel service that's running there and it is collecting worker node logs and telemetry like kubelet

logs and stuff like that and um and uh sending that off basically over to uh to the gateway as well. yeah like the setup is honestly like like pretty straightforward I think a lot of what took us some time is really just it's telemetry we have to be careful right? We wanted to make sure that we did we properly uh what's it called we go into a

lot of security reviews that's really like I think the length of the project mainly came from security reviews and stuff like that making sure we weren't exporting any kind of PII any kind of personally identifiable information we don't we don't want your data we want to just focus on our managed data and make sure that if something goes wrong we know about it we get alerted on

it and we can uh fix it. Um so yeah so I think one of the things is that hotel provides a lot of great tooling I mentioned how it's been around for a while it's open source a lot of plugins really great I think we got a little too excited and got a little too carried away um with inside the Kubernetes cluster we have our own operators

that are like all kinds of operators one of them for example is responsible for managing like the the uh in cluster life cycle and stuff like that and uh we we saw hotel and we're like all right cool there's an open telemetry operator uh Kubernetes operator let's just add it in and it'll be great. Uh the thing is you know we suddenly had operators kind of fighting

with each other because um open telemetry offers its own custom resource definition uh for instance it's it's literally just called open telemetry collector and then it that resource definition is uh managing uh like a deployment uh you can set it to daemon set other stuff but basically the the thing is if uh for instance I used one of our internal operators to change something with a deployment

uh it wouldn't work like for open telemetry because you have to update this custom resource definition. So this is where suddenly now we have to support this open telemetry uh uh dependency in our other custom operator code and so uh I don't know if uh how many people know this but DigitalOcean runs basically it's a go shop we have a big mono repo and so now we

have this open telemetry dependency that we have to like keep up to date and hotel is updating very very fast so uh that just became kind of like a pain point for us. Uh what we should have done is just kind of instead of going the custom resource definition route, hotel lets you just pass in like a bare deployment like configuration. And so, you could just use

that. I think the benefits of this is that it can like validate and vet things. But, um yeah, that was that was one of those things where I felt like we could have uh we we should have stopped and uh didn't get too excited and kind of think about it a little bit more. Um what else? So, yeah, we should also spend more time understanding the new

telemetry data format, right? Uh initially just seemed oh yeah, there's attributes uh here there's attributes in front of me. Yeah, cool. It seems pretty straightforward. Uh there was definitely some points where uh debugging debugging was a little hard because I didn't understand uh I I got confused with resource attributes versus like particular attributes. So, resource attributes actually tie to the actual resource uh that you're collecting the

telemetry from. Then, the attributes are literally like the the actual telemetry fields and stuff like that. So, resource attribute would be like the Kubernetes cluster uh name uh for example. Uh and then, you know, like attributes would be like the request or whatever if it's like some kind of GRPC uh endpoint. Um so, yeah, and also another thing is having observability for your observability early on is

very is a very good idea to do. Uh OpenTelemetry does also export its own metrics. Collecting those is a good idea. Uh I feel like that's something we also should have done much quicker on because we definitely had situations where we were kind of rapidly developing this thing. We had collectors crashing and we weren't 100% aware of it. We would check on it and be like, "Oh,

it's crashing." But, it would be good to just have that telemetry already just running there, right? So, having that in place is very important, I feel like. So, you know when you're loom killing and you need to, you know, change things up, right? Um but, yeah, like I think the main thing the big win here was just having telemetry coverage across like the basically our entire stack,

right? We have metrics and logs for all managed components. That means stuff that's in our control plane, you know, hosted on the control plane cluster. Uh also stuff that's in the cube system namespace. Uh those services I showed you earlier that that we have telemetry there. And also things happening in the worker nodes, right? Um and so, basically we we covered every those gaps that we had.

We we were able to fill them thanks to hotel and we were able to do it in a flexible way. Uh we were able to deploy hotel in a lightweight uh uh way to these different parts of our stack. And I I just I personally thought that was just it was just much more intuitive, much easier to do uh than try to uh set up basically a

bunch of Prometheus instances at different layers. And it just it just didn't seem practical, especially at the worker node level. Um and yeah, yeah, overall less compute to run. We don't need to have a dedicated machine anymore just to run Prometheus like to run a Prometheus service. Uh we can just use uh it can literally pull from the node pool that our other services are using. Um

but, yeah, once again, kind of like the whole more scalable architecture. Now, we have we have these re uh agent collectors that just collect on the telemetry and pass it off. Then, we have a gateway that just uh literally can scale on its own statelessly to make sure that it's uh properly exporting all the data. Um yeah, oh yeah, then that that transform processor was really helpful

because I know we before this we had like our own go plugin basically. Our not go plugin, the go service that was running that was literally cuz we didn't like the way I think how Prometheus was properly like merging uh our telemetry. And so, we wanted to do it differently. And so, we had a dedicated go service just for that. And we were able to just replace

that with basically just uh this transform processor and add in some YAML. So, we traded some go for some YAML. Uh some people might not like that, but uh I think it it is it's just less to maintain, less to deploy. Uh I'm happy with the route we went. Um and also, yeah, our mean time to detect issues went from hours to minutes because no longer do

we have a situation Well, I'm not saying it's never going to happen, but now we don't have a situation where we have customers putting in a to porous ticket to us or telling our our support team that, "Hey, this thing has been crashing for a while, but it's like it's out out outside the area that we had visibility into." Right? Uh so, something like, "Hey, the worker

node's acting up or something in my cube system is crashing." We would know about that. We'd have customers tell us, right? And so, there was just the whole to talk to support back and forth, finally get it to the DigitalOcean Kubernetes team for some of someone to look at. Just the the we just we we we were able to get rid of that cuz now we were

a very uh aware of it and we had uh and we'd get alerts on it. So, yeah, no and no longer having to download and search through object storage for historical data because now we have proper dashboards that we can actually search through auto logging very quickly and and find things. And so, we no longer dread having to go find auto logging cuz sometimes we do have

customers being like, "Hey, uh can you tell me what happened like 3 days ago? I think this might have been us uh or maybe it was you." And it just being able to easily answer that question like, "Oh yeah, that happened here. This is what happened, yada yada. Uh this is the user that did it, yada yada." Just being able to like kind of do that, I

think uh was it was honestly just it's just made things much easier for us. Uh and we can just offer a better product to you. Um and yeah, that's that. Thank you, everyone. Thank you. >> [applause] >> 4 minutes left just in case if someone wants to ask a question. Uh we are open for questions. Am I audible? Yeah, yeah, please. It's not working. Is this on?

Yeah, yeah, we can hear you. Great. First off, uh great talk, very entertaining. Thanks. You mentioned that you now finally have a searchable logs. >> Mhm. Um did you talk about um how you were indexing them or what workload or service you were using for that? Um yeah, so we were using we're using OpenSearch internally just to like host uh services. And then, we index them uh

based on uh like kind of our our data model internally, which is we would index them basically based on the the log type actually. So, we'd be like audit logs would be in its own index and stuff like that. And that that and then, we'd also have like another index to kind of host like the other telemetry So, you were ingesting them from these hotel collectors into

OpenSearch. >> Yes. Okay. Yeah. Thanks. Yeah. Thanks. Please. Two-part question. Okay. The first part, does this does this framework new framework for you guys offer any sort of boundaries to prevent you know, overload conditions like getting too much data? That's the first part. And then, second part, did it also address any presentation uh you know, dealing with better presentation? Like do you still need something like Netdata

or something else to uh visualize the the what you're gathering or is that also uh addressed with with your new new approach? Okay. Uh yeah, so the I think that I'll I'll start from the second question. Um about the presentation part. It I think the OTLP format is slightly different. Uh like uh we basically have Grafana dashboards to like view this stuff, right? And I think it

really was just a change from like underscores to periods or something like that. Um and also uh I believe uh what's it called? Yeah, it's OTLP supported. So, the queries were it was not hard to change the queries at all. So, yeah, that that part of the presentation part was very smooth. It was it was it was a very it was a quick very quick update to

our queries and we were good to go there. Um sorry, could you repeat your first question again? About like overload conditions, getting too much data. Is there anything to limit that? Yeah, so um that's that's a that's a good point. So, we actually it actually tells you like OpenTelemetry hotel has its own metrics that tell you like basically like, "Oh, this is how much data you sent

over." Uh and so, you can that's the one thing you can use to like add alerts on basically to see like, "Oh, okay, we're sending a lot more than we need to." Um at the memory side, you can of course set like you can of course have the memory limiter to basically like if you notice that it's like uh like you can set like like a burst

threshold basically. And if it reaches like a certain point, you can drop that data. And not only that, when you drop it, that metric also gets tracked. So, you can see like if certain telemetry like if a large amount of telemetry is being dropped, then you might want to like want to know what it is, right? So, you can get alerts based on that as well. Thanks

very much. Thank you. >> Yeah, of course. Thank you.