Cloud Native Theater | Cloud Native University: A Simple and Practical Guide to Obser... Diana Todea

This talk focuses on improving observability for Kubernetes, discussing the challenges associated with managing numerous tools and metrics. The speaker, an OpenTelemetry contributor, highlights various sources of metrics and logs, addressing issues such as inconsistent naming conventions, high cardinality, and the lack of standardization. Important metrics are identified, including API server request rates and CPU usage, along with insights on monitoring logs and events. The speaker emphasizes the significance of structured logging and tracing, and the necessity of refining metrics documentation and alert systems. Recommendations for developers include implementing structured logging and automating alerts, while also encouraging community involvement in improving Kubernetes observability standards.

Uh yeah, thank you so much for being here. I'm going to go quickly uh on introductions. Uh I'm an open telemetry member contributor. I'm also cloud native Romania organizer, co-lead of merge for neurodyiversity. But today I'm going to speak about observability for Kubernetes. Um Kubernetes obviously it's very well known, it's powerful, but observability can be quite overloading and confusing. There are so many tools, there are so

many platforms around it, too many metrics, too many dashboards. So how we can make easier for contributors and end users to onboard, right? Um so if we have like metrics, events on one side, logs, traces, um and a whole bunch of things. Uh so we need to focus right on Kubernetes metrics who produces uh the Kubernetes metrics. So we have like API server, we can have app

coming from an application, cubelets, C advisor, control manager or scheduler. Um so yeah from this side with the metrics we know how this works right what happens for the Kubernetes metric stability. So at the alpha level we have no stability guarantees. Uh so you know if uh they can be modified or deleted at any time um and you know Kubernetes all Kubernetes metrics implicitly fall into this

category and we have stable metrics as well that can be guaranteed not to change except that the metric may become marked deprecated for a future Kubernetes version. So we may ask ourselves what metrics actually matter right. So there are important metrics like for example coming from control plane like API server request rates API server latency API errors nodes and workloads for CPU usage, memory usage, resource requests

versus limits or pod start restarts and scheduling for scheduling latency or pending pods. But here comes the deal, right? Um there's normally around metrics a common problem. There are inconsistent naming and labels, high cardality, duplicated metrics or dashboards with no purpose, right? So it's very difficult sometimes for end users to to navigate around this and to like decide which are the metrics that they could start with

and uh use them. If we go next into the logs, uh we have also um you know a bunch of sources for the logs. We have the application logs, the Kubernetes events, the control plane logs or container runtime logs. So if we follow the same uh kind of narrative there will be also a common log problem. Logs are not standardized. Um for control plane we you know

think about the structure JSON. For application logs it could be anything. Uh for events you know they focus on structured objects. For audit logs we have strict JSON schema. So Kubernetes logs are generally structured and our application logs are our responsibility. We also have another category that is called Kubernetes media. Uh and events are in general extremely useful. Uh audit logs actually tell you who did what.

Um contain sensitive information like APIs, role names. So think about an example here, Kubernetes event exporter. So I'm going to go in more depth about this idea. Uh you can scan the QR code, it will take you directly to uh to the repo. Um and I'm going to show you visually as well the the log the demo. So here the audit log stats uh application uh that

it's in this uh GitHub repo parses the Kubernetes audit log files and sends them to Victoria blogs. Anyway can this is only like a open source uh data uh data source. Uh also it's paired with a graphana dashboard uh that is used to render statistics derived from the logs. Um so let's see what what can we do with it. Right. So if we are focusing just on

audit logs here and we use graphana um and we can also have like a bunch of other you know very useful information that we can get from this type of of um of exercise. Yeah we can have all the verbs. We can have different resources name spaces uh structure. You can customize it as you want. This is like maybe like for you to understand like what is

the extent what can be done with the with this with audit logs and we go we could go into some visualizations here um to get information around the uh blog etc operations per username um and we'll see right now other uh possibilities This okay so um here obviously you can do this later if you want to check it but there are lots of possibilities and visualization that

can be done and can be useful when you're actually troubleshooting something or want to get more information around the audit logs for your Kubernetes cluster. uh you can have like operations you can have like you know uh all around like operations per resource or per username uh per verb per namespace you know so uh a lots of possibilities around that going forward like what's important for us

right also for the developers but also for the end users and actually the maintainers of Kubernetes so advice for the developers try to use structure logs right and have instrumentation from day zero and focus on the alerts. Right? So which alerts would you like to see the alerts actually automated? you know they will lead you to use specific dashboards uh and also they will lead you to

use specific metrics and it's very important to define that and try to uh use that in a coherent way like there are lots of uh there's a lot of um importance and I think it's always good to think about a bit forward for further than logs and metrics and think about tracing why because there are some benefits around tracing it's a vendor neutral standard you have common

tools to work with um and tracing yes is grouped by stability at the moment but we would like to see in the future group them by criticality right so it's something that we want to see a community being more involved and you know trying to change a few things um right so if you think about in terms of community the perfect uh working group or the perfect

let's say project for this will be SIG instrumentation sig instrumentation is actually working on better metric documentation, group metrics by severity, uh how to improve dashboards and alerts, how to improve structure logging, how to create observability guides and also like provide some consulting on best practices. So their let's say focus is quite broad right. So from our side, we would like to contribute more to SIG instrumentation

and especially when it comes to uh the observability signals. Um we created an an issue speaking with the community and speaking with the maintainers. Um so we want to add more visibility into um you know the on boarding for end users uh to create more tutorials, documentation uh and I'm going to say why this is important. uh we want to improve the instrumentation developer experience for Kubernetes.

So for example you have the metrics reference you go there you have a huge list of metrics uh that are not necessarily grouped by uh specific categories and it's it's interesting to actually give them a bit more love and a bit more focus. Why? Because it's important for the end users and developers alike. So one idea that we had in mind as well to classify the metrics

for particular use cases like you know from beginner, intermediate or advanced. Uh then we can also improve the standard recording rules, the alerts and dashboards. Uh and you have the GitHub repo for for that that handles the this specific uh use case. Uh and also we want to create more guides on how to interpret dashboards. I think this is something really important that we can always like

we can bring from our side you know but uh and then uh help the community shape this together. So this is like where we at to the moment. Um we are like trying to to how can I say bring more collaboration into SIG instrumentation from your side as well uh from obviously observability vendors uh and also help a bit more um collaboration with uh with the Kubernetes

community. Uh they have tomorrow actually um some meet and greets. So meet and greet you can meet them uh Kubernetes community and you can ask them a bunch of questions if you are uh really interested I you know we give some pointers like where this is leading uh also the open telemetry if you are interested more in tracing they have a bunch of um like you know

the community is present today and tomorrow so you can always do that from my side you know uh you can check the resources that I give you obviously the slides will be uploaded later and uh you can check them Uh yeah, and thank you. I'm not sure if we have any time left at the moment. If somebody wants to ask maybe one or two questions. Yeah, I

don't know. If not, anyway, if you don't have any specific questions right now, you can find me later or you can ping me. Um and yeah, I'll I'll be around at CubeCon. Yep. Thank you so much.