Planet-scale Dashboards at Google with Grafana

This talk highlights the challenges and solutions related to monitoring at scale within Google, particularly focusing on the development of the Planet Scale Dashboard. The speakers explore how reusable dashboards can help manage thousands of systems without incurring prohibitive costs. They describe the importance of scalability in observability systems, ensuring rapid response to outages. By using dimensions and filters, the approach allows users to easily navigate and interact with dashboards tailored to specific scopes, enhancing both usability and performance. Furthermore, the collaboration between Google and Grafana is discussed, showcasing a shift towards leveraging industry standards to improve user experience and community contributions in observability solutions.

Thank you. You want to have that? >> Thank you. >> My name is Carl Bergquist. I'm a principal engineer at Grafana Labs. >> My name is Claudia Jorda. I'm working in the SRE in Google within the monitoring team. And with this presentation we want to show you what means for Google planet scale dashboard. At Google scale, it's definitely impossible to monitor and maintain tens of thousands dashboard

or hundreds of thousands systems. This would become immediately cost prohibitive. And this is the reason why Google developed his internal system that is planet scale dashboard as a single pane of glass for observing it the health of its internal I will walk you through the journey that made this a reality. To do this, we are focusing on three main areas. The first one is about reuse. So,

to reduce the toil of maintaining this huge number of dashboard. The second one is about scalability because when we have tens of thousands dashboard, you need to ensure that your observability system is scaling in terms of performance when you monitoring this to ensure that you have a quick response to outages, for example. And last, an outlook. What you can do with the system. It's true that in

Google we are working a large corporation scale, but we believe that this is can be useful even for smaller realities. So, to do this, let's take a scenario you can be you may be familiar with. Let's assume that your developer team put together in a production today a new wonderful shiny system and they managed to put it off all the monitoring stuff. So, they ignore all the

persistent email to set up the monitoring, they postpone the dashboarding work relying on the fact that the system is stable. And then the inevitable happens. At 3:00 a.m. your pager screams to life. And after the initial shock, you are thinking, "Oh my god. So, how much toil manual toil is ahead of me? What should I do?" And this is where the magic happens. Imagine that you're landing

on your monitoring page. And instead of scramble to find the right query or to debug the smash dashboard, you can select only the systems or the or the services that you're looking for. And immediately you get provisioned with um high signaling dashboard fully provisioned tailored for what you're looking for. Wouldn't this be amazing? Or let's have a look to how this dashboard can be. This is a

screenshot from the an internal observability dashboard in in Google. This is a standard dashboard, let's say for a front-end job, uh showing the HTTP metrics in which you get an overview and the list of out-of-the-box dashboard that you can navigate on the left-hand side. And on top of that, you can have additional capability like filter and group by specific dimension. Now, this looks amazing, but it can

be a problem. Let me explain why, but before let me give you some numbers. Uh in Google we are dealing with 190K Googlers according to the last uh earning calls from 2025. That is equivalent of the large urban center. We are dealing with a single monolithic code base repository. We're speaking about billions of lines of code that leads to the use and sharing code. And on top

of that, we are dealing with internet scale application and systems. So, this means that we're leading with mmh apps distributed all around the globe. Now, keep in mind these numbers and let's go back to our dashboard. Imagine that somebody creates a dashboard for his front-end service for this specific metric. Fine. The dashboard is ready, fully uh containing the information needed. And now, what the engineer does? Okay,

I need this dashboard for my back-end, for my middleware, or for whatever you have. Then you take the dashboard, you copy and paste the dashboard, or then imagine this happening at Google scale. Only for this metric, we're speaking about order of magnitude of 100k dashboards. This is something that we cannot sustain. It's impossible. You're getting lost immediately. So, this is the kind of problem that we need

to solve now. In which way we are going to solve this kind of problem? So, in Google we have something that we call dimensions. Uh you can call them, I think, in Grafana terms, uh variables or template variables. you're able to provide a fully qualified query only at run time when you inject the variable into your dashboard. You're making your dashboard reusable because you can reuse the

dashboard for all the services that apply for. And this is the way in which you can immediately scale in terms of reusability of the dashboard. This looks great, but there is another problem behind. Then this apply to each and every system that you have. So, what if your system is running on a Java Virtual Machine, and this dashboard apply to something else? It doesn't It's It's not

relevant, right? So, you want to be able to filter out in the how-to the box dashboard that you have on the left-hand side only the dashboard that are relevant for you. So, now we You two problems. How do we inject the scope, the job, and how we filter out the dashboard on the left-hand side. And this is where Planet Scale dashboard plays an essential role. I told

you that we have dimensions variable. So, we lift up one dimension to be more equal than other, and we call this dimension scope. The scope at the end is a filter. It's the filter for your job, or better, is the filter for whatever you're interested for your investigation. It's a filter for your scope, and this is the reason why we call this scope. So, now, whenever you

select the scope, you get the dashboard fully provisioned with all the navigation field for the scope that The second problem, how we filter out the the the the dashboard that relevant for your scope, we solve in a slightly different way. We ensure that your jobs expose the property related to himself. So, if you have a job that runs on a virtual machine, we make the job expose

a metric, let's say, run on J V V M, and we ensure that whenever the scope is selected, the relevant dashboard for the Java Virtual Machine are showing up. And this way, you get the filter out of the box for 10 of thousand dashboards, and you have a list of canonical dashboard that you can use for the full organization. >> [snorts] >> What do we get so

far? First, the dashboard are reusable. So, we reuse the dashboard for any single system, and there is a single dashboard for concern. Second, the dashboard are navigable. The list of dashboard are still filtered by the system that you're looking at, and we made this generic. Let Let me give you another couple of quick wins that you can get with a navigable If you can inject the job

into the URL parameter when you are left files and you get page, you get a deterministic link to the dashboard without having to search anything. And with the same parameter injected in the URL, you can easily navigate through all the dashboards related to this specific job without doing anything else. This saves time, money, maintenance. A quick word that I've been told to tell you, so first of

all, I'm referencing a huge number of technology, but it's just to make sure that it's clear that it's broadly applicable. It's not a sales speech because this applies only to what we have internally for internal observability at Google. And this is a preview of a paper that's in submission, so it can be imperfect. It will be published soon, and then you can get access to the paper

and get something out of that. Now, where Grafana enters into the game. Let me go back when Google started its observability journey. So, at the time uh there was nothing on the market. And Google decided to build everything itself. So, we have the front end, we have the back end, we have the storage, everything. Uh but in the last years, there was a pivotal uh strategic shift

because the world has changed. We have now a lot of industry standard, and which Grafana, for example, is leading one of this kind of the industry standard. And it can be beneficial for Google to make sure that we are aligning with the industry standard for multiple reasons. First of all, we can unify the flow, ensure that the experience from Google is taken into an account, the experience

from Grafana, for example, is taken into an account and create a new, let's say, enriched platform. Um it allows teams to cooperate easily because people are coming from the same technology stack, will be easier to share experience, template collaboration, and it's easier to onboard people if they're already familiar with the technology stack. In which way we are doing this? So, right now we have uh a partnership.

There's a partnership between Google and Grafana, and we are running Grafana Enterprise uh for our internal usage with broader use cases and largest use cases. Uh the reason why Grafana was well positioned for Google was the architecture is modular and this is an excellent entry point. They have a lot of uh visualization capability that can be very useful for Google, and Google can shift the focus on

developing the UI, for example, towards things that are more unique for Google and make sure that we are using industry standard and share this broader with the community. And we focus on integrated solution and services. Right now, we are using this internally. We are collecting feedback from users, many feedback, uh to ensure that all the feedback is captured and we can deploy together in similar Grafana in

an even a better version of the system in order to ensure that all the experience coming from Google is shared across the community. Before handing over to Carl for uh specific detail in what we are doing right now in together, let me tell you something that I personally believe that is the proof why this system is needed. So, in this chart you can see the grow about

of the number of system monitoring with the traditional dashboarding system and with have learned scale system at Without the system would not be possible to monitoring to monitor the number of system tracked here. You see the growth is more than linear. It goes towards to be exponential. And would not be any way to monitor this without a system like that. And with this I think I will

hand over to Carl. So, what I want to cover now is how we implemented the idea of scopes from Google into Grafana. So, set the scene a little bit. Let's imagine you're at Google. You have one dashboarding solution. You have 50,000 engineers, 100,000s of services, and that number is just increasing as Kacha just mentioned very rapidly. You have an ocean of metrics. You have all of these

shared infrastructures and different layers of software, hardware, and other internal So, wouldn't it be good if you there's was one dashboard for each layer or at least one dashboard built by experts? And wouldn't it be really good if when you look at that dashboard, you only saw the telemetry data for your service? And that is kind of what scopes enables with just one click. So, take it

to Grafana and Prometheus terms, a scope is a named set of filters. It could be any set of filters in Prometheus. But for the sake of this presentation, it's usually easy to think about it as name spaces. The user journey when using scopes doesn't start with a dashboards. It start with with you as a user selecting the scope. You might not know what metrics you want to

look at, but you don't know what system or name space or scope you want to care about. Grafana then injects the scope into the dashboard query at run time before it's sent to Prometheus. And by extracting the labels that you care about right now from the dashboard and injecting them at run time, the dashboard becomes much more reusable. And the dashboard that is built by experts, they

don't have to care about what labels the metrics eventually have that they want to graph. Those are disconnected. By moving the filters outside of the dashboard query, you can look at metrics in various different angles. So, you might care about go metrics at a cluster level if you're rolling out a new Kubernetes new type node type. And you might care about them at a name space level

if you care about an application is having problem. Once you know what service within that name space is having problems, you might want to look at the job label. And all of those scenarios are covered by the same reusable dashboard. The scope that you selected initially in the user journey also stays when you navigate dashboards. So, if you selected this name space, you can look at go

metrics, RPC metrics, Kubernetes metrics using the same filters always applied. So, let's see that in action. So, the scope selector exist up here on the left side. So, let's zoom in a And how you select your named several labels. So, you as a user don't need to care about the actual labels. You just care about the name. What happens then is that the labels are added to

the filter box. The dashboard itself doesn't care about that. They're using that generic. So, exit edit mode. And then Oh, no, we don't. I want to save it. Exit edit and discard. But with if we look at the actual query sent to Prometheus, we see that the name space is injected. And we do that by parsing the Prometheus query in Grafana and injecting the The new filter

box also allows you to inject uh filters that you as a user just want to apply. So, if we take something like like take cluster, those are also injected. So, this dashboard is much more usable because it doesn't care about the labels used in your system. We also added support for adding group by on the fly or at run time. So, if I want to group by

job, I now get a uh are able to aggregate the go process memory based on the job that I'm filtering on. The menu here on the left side is also new. It add navigation to the dashboard relevant for your scope. And when you navigate these dashboards, both the scope and applied filters remain selected. So, this makes it very easy to drill on in to something, navigate different

dashboards, and maybe remove filters if needed or not. But it all stays because it does the system that you care about. And I know some of you maybe felt like the the clicking journey here the beginning was a little bit cumbersome. And Google engineers very much thought so. So, we added support for finding scopes through the quick uh navigation item as well. So, you can apply recent

applied scopes, or you can just navigate to scope using the keyboard. So, this is a quick way of zooming in on the infrastructure or processes you care about, and then having the relevant uh dashboard show up for you. So, let's go back to the slide, please. Sweet. And and the dashboard navigation uh item list here on the left side is really one of the key features of

scopes. Uh because this allows you to quickly navigate uh different dashboards, as I said. And you might think, "What's the difference between these links and the normal dashboard links?" And the fact is that these links are managed automatically by Grafana. Based on the selected scope, it figures out if it has relevant metrics for you or not. So, if you selected a namespace and there go metrics in

that namespace, you will see go dashboards. If there's Java metrics, you will see Java dashboards. So, you only see the dashboard relevant for the scope you selected. We do that by adding metadata to dashboards that describe what metrics they are designed for. And then we check if the if those metrics exist within a scope in the background. And this is precomputed. So, getting this list of links

is blazing fast. Um and this query is just an example for how to do it. At Google scale, this kind of starts to break down, and there are other ways of doing this, but this is how we're getting started in Grafana. Managing scopes is also something you can do based on automation. So, you do it based on metrics queries. So, if you use clusters, services, or namespaces

that start exist in the as a result of the metrics queries, new scopes are created for you. So, neither you as a engineer or the observability platform team needs to think about it. You need to think about the query that defines what scopes you want generated Which makes it quite flexible depending on how you decide to run around your infrastructure. We don't need to take care of

like have an opinion about that at all. It's all going to be tailored to Once infrastructure is decommissioned, it will eventually also delete the scope based on TTL. So, what scopes gives you at scale is a way of zooming in on and only see the metrics and dashboards relevant for you. It greatly increases reusability of And it allows your platform observability observability team to manage these as

higher order functions. They care about how to generate scopes and how to connect scopes and dashboards, but not each of them individually. And I'll also ask you this question, when are scopes suitable? Because this is a feature designed for high scale. So, it's really designed for companies who are run have one metric database. A lot of this experience is based on the metrics, so it has to

be one, otherwise it's going to be quite cumbersome. You need a central observability team that can manage these configuration at scale. Someone needs to understand enough about the infrastructure that they can configure it. They don't need to understand all of it, but enough. It's also more suitable for organization that have experts building dashboards for others. So, if your organization have all of the teams completely siloed and

independent, the reusability is less compared to if you have experts building layers or services for each other. So, if you have a database team who build the database dashboards for others or own that, then this is more useful Scopes are currently in our experimental stage. Um, please sign up if you're interested. You can also find a QR code at the ask at expert booth later. It's quite

early in the development process, but we're just too excited to not share it with you. Uh, it's also going to be a enterprise and cloud feature. But, that said, most of the feature that we're building based on Google based on feedback from Google are going into open source. So, the improved ad hoc filters, variables, group by variables, drill down per panel, and section level variables, those are

all based on feedback from Google and they are going to into open source. So, this is not just exciting for us and Google, this is exciting for the community as a whole because they're making us up level dashboarding in general. So, we're very excited about this, obviously. Uh, the Google SRE book is a big part of Grafana culture and we're very happy to take the expertise from

Google and build it into Grafana and make it available to Scopes enables you to drill down on what's relevant just for you at high scale and ad hoc filters and group by variables make dashboards much more usable even for your home lab. >> We are excited to have this collaboration with Grafana. We can level up the whole community by introducing these new concepts and we can work

together to get a better observability board. >> I think that's it.