Cloud Native Theater | Data on Kubernetes Day: From Billion... Victoriya Kalmanovich & Shahar Azulay

This talk focuses on Kubernetes monitoring at scale, elaborating on the complexities introduced by microservices and ephemeral infrastructure. The speakers, Shahara Zilai and Vicki Kalmanovich, share their personal experiences transitioning from monolithic applications to microservices after a significant acquisition, highlighting the observational challenges encountered during this shift. They emphasize the necessity of a robust observability platform to manage the rising telemetry volumes that accompany cloud-native environments. The discussion also introduces the 'bring your own cloud' solution for observability, which combines the advantages of self-hosting and managed services, allowing teams to maintain control over their data while enjoying the operational simplicity of a vendor-managed solution. The presentation concludes with insights into logging, metrics, traces, and the increasing importance of AI in telemetry.

Hey everybody, great to be here. Um, before we present ourselves, just uh presenting what we're going to talk about. We're going to cover Kubernetes monitoring at scale, which is kind of uh what a lot of you people are, you know, about right now in in this big Kubernetes conference, right? So, we're going to talk about technologies around it, how to do it, and a few interesting suggestions

that we might have for you. Um, as we were presented, I'm Shahara Zilai. I'm the CEO and co-founder of Ground Cover. ground cover is an observability platform full stack uh dealing with a different type of solution that we're going to talk about today for high scale Kubernetes >> and my name is Vicki Kalmanovich I've been an engineering leader for the last decade I worked for various industries

and I really love building teams and building different types of products and I guess our story today begins with a startup that I worked at a few years ago it was a very very small startup and we were acquired by Walmart So all of a sudden we had this whole new world of new requirements and a gigantic scale that we had to support and we got the

mandate uh you take your monolith you split everything to microservices and you support that scale. You do whatever you need to do to support that scale. The move to microservices and the ephemeral nature of Kubernetes infrastructure introduced so much complexity that our team just completely lost the ability to see what we were doing and what the system was uh uh what what was happening within our system.

And we tried to create all the microservices at once. And there were so many dependencies between the microservices. And just essentially this really tiny app that we all knew so well by heart suddenly became massive. Troubleshooting became guesswork which is a horrible thing. And debugging a production issue that used to take us minutes suddenly took us hours because the observability tooling just didn't catch up with the

new architecture. the visibility the visibility problems they got so bad that at some point the solution was just to move things back to the monolith and not because the microservices themselves were wrong but because we just couldn't see into them and that was the moment that we realized observability was not a nice to have infrastructure it's a prerequisite to operating microservices >> what uh Vicki kind of

talked about right now is a personal example right a personal story. This we is something we see with a lot of different companies and a lot of different teams from you know small scale to very big scale. Eventually Kubernetes gives you a lot of power to scale both your teams that can work now independently and build microservices and kind of you know scale up scale out as

a team and as a company but also the ability to scale your infrastructure right uh in very different and very uh agile ways. So it's very very powerful but on the other hand the telemetry you need to collect and store and monitor so you can actually know what's going on which is kind of what Vickiy's theme uh experience is also growing significantly much more than you used

to thinking about before when you work with monoliths and this data can either help you or drown you which is what we're going to talk about on the background of that uh there's a growing change in how uh cloud native impacts telemetry volumes we're seeing telemetry volumes like logs metrics traces and so on that people use to monitor their platforms doubling every 15 months and this is

before a much much bigger acceleration we're going to talk about with AI right so this data is growing it's doubling every few months and eventually you need it to troubleshoot which was which is what we're going to talk about today so why is cloud native environment so complicated to monitor um probably most of this crowd is using kubernetes at some form of or another and also a

lot of different other cloud services Eventually the infrastructure behaves differently right there's different size of host there's ephemeral pods there's autoscaling there's so much things going on nothing is permanent like that old you know SSH into the server uh experiment kind of approach that we had in the past everything is moving everything is a firmware everything is scaling everything is temporal um and deployments are also complicated

right resource configuration independ interdependencies configuration things can have a lot of impact uh on other things right that you didn't have before. Infrastructure can impact application application can you know choke the infrastructure uh impact other applications and so on. The application itself is different, right? We're talking about hundreds of microservices, varying replica sets of different sizes growing. What you know, one one morning it's a single pod,

the next morning it's a thousand pods doing the same thing. And it's really hard to keep up with what happens at each one of these different scales. And also user use cases are different. Uh there's just more services which creates different and unexpected use cases, more edge cases, different behaviors when it comes to that, you know, new distributed world. And eventually each layer in this stack can

affect the other layer which is why it's so complicated, right? The explanability of things become more complicated. You need more data and more understanding about each of these layers to figure out what's going on. Um and so you need a very good observability platform, right? Or a very good set of telemetries. You have your infrastructure and your application and you need to monitor that. And there's endless

different verticals for observability and how you actually look into cloud native applications, right? You need infrastructure monitoring to understand the resource consump consumption of your different uh deployments, right? You need log management from your applications to collect all the logs into one centralized place. No one expects you to, you know, log into the actual pod and see logs being emitted right now, right? You need to look

back. You need to investigate. Same for APM tracing between all these different microservices, right? you can sometimes get into 20 30 hops deep to figure out what's going on. Real user monitoring to how your uh users are experiencing the platform, right? So there's high cardinality multi- signal correlated stack that you now need to work with. All these signals need to live in one place correlated to one

another. If you're looking at the resource metric from a CPU consumption of a container, you need to correlate that back to the to the log emitted when the container, you know, did something or the application did something, right? So, a lot of different signals correlated together is kind of the world we're right now working in. And we we mentioned it briefly in a second, but to be

honest, this is just the beginning, right? Um we moved from monitoring like 10 years ago, right? That was the world we we all used. It was more um kind of, you know, passive monitors with different predefined rules that you've set to what we call today observability, which is kind of what I've talked about right now. more reasoning, more correlation, more telemetry which is deep into the understanding

of what's going on and we are moving into a different world called called autonomy. Right now your AI workloads that you're going to all build in a second or either building right now are going to create 100x more telemetry. Right now we've moved from two 20 hops API calls with 20 microservices which were considered a lot to AI agents doing 50,000 spans in two hours right in

front of some other tools or uh LLMs or whatever they're using right and there's also growing need for this telemetry before you already all all you needed was to troubleshoot with it to wake up at night to figure out a problem to investigate your deployments mostly post production right now you're going to want to use telemetry to build code, right? You have coding agents that need context

from production. You want to figure out what's going on earlier in the development life cycle. So, more telemetry, but more use cases and more consumers that are going to sit on top of it. And you you want you will want to have all of it. And you might say it's just more noise. And for some cases, you might be right. But we should be able to have

the right tools to be able to sample the right data and to find that needle in a haststack. And um it's it it really reminded me of something that happened to me a few in in one of the previous places I worked at. I was uh we owned a B2C product uh mobile app and we had a bug in production and um usually the culprit would be

an SDK problem and so it was the same thing that we thought this time. Okay, it's probably an SDK problem. Let's create all the tickets to the SDK teams and we bring in all the relevant engineers. And we worked on the wrong culprit for days. And Sentry was just blowing up with like so many messages you couldn't really understand what was going on. But after a while,

we started feeling that maybe it's not really an SDK issue and we really need to uh uh test that hypothesis that it's not a like a problem with the SDK. a problem with us maybe with the app side and we did have all the proper data with Sentry and we started really looking into like the historical data and starting uh comparing between uh what was going on

like the previous behavior to the current behavior and maybe understand if there's a problem there and we found that the problem the solution for our problem required a really huge refactor on our end on the app side and uh which really proved the hypothesis that it's not an SDK issue but We spent days working on the wrong And observability for us was the key because we found

the right signal that eventually saved the day. And eventually under pressure, which was that situation, we really needed to be able to trust the data that we have and what the system is telling us. >> So where does all this telemetry go? Right? We just talked about an inflation in telemetry, you know, an inflation in logs, metrics, traces, and a lot of different verticals that that come

on top of them for like advanced troubleshooting and observability. Where do you keep all that, right? You have the classical fork in the road. There's two options, right? That uh each one of you sitting down here right now chose you either go self-hosted open source. The promise is full control of your data. You have data sovereignty, cost predictability, right? You're in control. Um the problem is that

you're managing it right and on the on the right you pay a vendor right you go to a managed sus solution the promise is simplicity speed to value no operational burden you can basically just enjoy uh you know the services being fully managed these are the two options that most of us have right now right >> wait a Both paths that Shaka just introduced, they they fail

at scale, right? They break, but for different reasons. Like if we look at the self-hosted reason, if you if you choose to build it yourself, then maintenance gets complicated really fast. That simple stack that someone deployed in two hours is slowly becoming a source of frustration and pain. And I I'm pretty sure that 99% of this room can relate to that pain. Performance is also not stable.

And it gets worse during incidents when your entire team is just quering the data like crazy. And changing your observability stack requires efforts requires attention and everything. You know, those are the the resources that you have within your team. So if you want to ship more features or you want to support customers, you now have to understand what is more important to work on this observability stack

and change it or ship features. and uh poor decision- making when you change your observability stack leads to tech debt that you also need to uh to support. So so on and so forth. On the other hand, if you choose to buy it, cost obviously explodes uh with data volume and you have almost no control over it. Data sovereignity is a serious compliance risk. logs, traces can

contain a lot of customer sensitive data and vendor lock in uh data isn't yours and you're in a long-term marriage with your vendor. So, both paths force you to give up data in order to survive. When we look at teams that choose to uh that choose to self-host uh for their observability stack, they spend precious engineering time on maintaining it. 40% of senior S sur time spent

maintaining the observability stack instead of what? Instead of improving reliability, instead of shipping new and exciting features and what really sucks is that it's not your day job to build observability stacks. And on the other hand, teams that choose uh to use a vendor, then budget is going to be your limit very fast, right? 70% of telemetry is being either throttled or siloed to control cost which

actually limits troubleshooting and increases downtime. Most teams I've seen, they don't use many of their observability observability stack features. They throttle data or they silo it between three, four, five different vendors just to optimize the cost. You never plan for observability to be your biggest infrastructure project or your biggest budget line but eventually that's what it becomes. So um the obvious question right what if you can

have both the we mentioned that for fork in the road open source or SAS managed right what if you can have the best of both worlds have that sus experience it's fully managed you don't have to worry about it you don't have to build it as Vicki said it's not your day job to build observability platforms maybe some of you it is your day job we love

you right but most of you don't um and what if you can also get that self-hosted control right control of pricing control of data control of data sovereignty. This is exactly what um bring your own cloud for observability does. It's a very unique concept that is currently um you know spreading out to different parts of the industry from the database arena to uh different data streaming platforms.

It's very new to obserability but it's going to it's going to be what shapes obserability for the next few years and security as well for the next few years. It's basically the concept of it's basically challenges the concept of what is SUS right if you're using cloud and the vendor is using cloud what exact what ex what exactly is SUS right it's not the data data center

versus you know remote sus cloud managed offering anymore you're both are running on cloud right so if we can solve that and basically host the vendor's uh backend or data data plane on your cloud environment instead of the vendor's cloud environment then we can have the best of both worlds. This is exactly what BYC does. The manage account in your uh in your uh environment AWS, GCP,

Azure, whatever it is, is basically hosting a data plane that stores the logs, metrics, traces, real user monitoring data, whatever you're sending. This is hosted on your data plane, stored and controlled on your premises. And now the vendor doesn't have to take the data cost of your entire telemetry, right? and you can scale properly but also secure and make your data more private. one. Well, this is

all being managed remotely for you under the hood in how Grover does it for example and of course there could be multiple implementations. We actually host that backend with databases that are open source that you know right we use uh two different databases to operate that remotely automatically at the customer's premises. The first is Clickhouse which is a commoner uh engine that basically allows us to store

logs, events, traces, everything that fits that type of database at very high scale. It it's modern. It's native to Kubernetes. We can scale it. We use our our own backend is basically scaled on top of Kubernetes at the customer's VPC. We can use it at very high scale, high resiliency, whatever we need to do from a perspective of uh uptime and guarantees of performance. And we can

also create a high compression rate as click house as a modern database provides and offload to object storage and everything you need to do to kind of support different retention tiers for a a really long period of time in case you need them to troubleshoot to to get compliance approval or whatever you need to do, right? Uh clickas also provide that subsecond uh approach to queries when

it comes to billions of rows, right? And uh no matter what we do, log management and tracing will always be a very heavy part of telemetry. And when AI walks into the door even harder. So this is a database to uh kind of be built for built exactly for observability workloads like ground cover provides and other provides. On the uh on the other side of that uh

we have uh Victoria metrics which is a Promedius uh compatible highly scalable data source. uh it's built for everything that is time series metrics based right now in in ground cover and also this is this is a highly horizontal scalable 10x more uh better in resource consumption than you know Prometheus and other um similar projects and you can use whatever you're used to from the open source

domain right if you're used to building on LGTM stack and you you know like working with promql or anything from that world it's it also fits that world so we have two different open source databases for different two two different reasons But both of them are being scaled and managed for you as the observability back end maybe that you always wanted to build right in your VPC

highly scalable and highly managed. What happens in real life is is kind of where bring your own cloud uh actually operates and actually kicks in. Right on the left we have data streams walking into the bring your own cloud back end. It can be uh different sensors. We use EVPF for example. It could be open telemetry, cloud integrations, whatever you need, right? Those are all being centralized

and stored on your bring your own cloud back end in the in the click house at the Victoria metatricx databases we just talked about, right? This entire orchestration is basically being um run on your VPC from data pipelines to enrichment to correlation. Everything is being stored, standardized, uh you know, sampled, whatever you need to do on your VPC without ever leaving your cloud premises. on the right.

This is where the SAS experience kicks in, right? You log into a SAS platform querying data live and data has been fetched by an automated control plane remotely which is SUS from your bring your own cloud instance. It can also be multiple bring your own cloud instances per region, per cloud, whatever you need, right? The the UI orchestration or the control plane basically does it all for

you. So it feels like SUS, it's comfortable like SUS, but it's the exact advantages of on-prem. So if we kind of look at uh you know the benefits of bringing a cloud in in know kind of a high level summary from the ROI perspective right there's no more hard data budget trade-offs if you store the data and you pay for it then you can push the limits

of what is that trade-off point right if before you had to sample traces really hard or drop logs from a specific uh type that you didn't want to store you're still paying for the infrastructure but it's 100x cheaper than paying a vendor to mark it up for to you, right? So now like storing it on prem, you can push that uh you know trade-off a bit further.

It's it's native to data sovereignty. If you care about the data of your customers, if you want to store more deep telemetry, right? If you want to um look into data more deeply, specifically when it comes to AI and the user prompts and everything like that, this is staying in your VPC compliant to the region where the data was created, right? Data transfer costs are also contained.

The data never leaves anywhere unless you query it, right? Those ten of terabytes of of logs per day are being tracked from your clusters to the the the in the same VPC to an actual endpoint that runs on the same premises, right? So data never leaves on egress if you don't want it to, right? If you don't query it. So it's very different as well from that

perspective. >> But about that last one, you can say that it can be solved with a self-host, right? The data transfer costs. Um but not exactly because cloudnative networking uh become is is really tightly coupled with uh resiliency requirements and that's where it becomes a bit more complicated. When I was working at Booking.com we were uh building this new service and we really wanted it to be

like super resilient and deployed across multiple availability zones which is fine. It sounds like a really good practice that you really want to do if you want your service to be super resilient. But then you look at the costs and you see that especially for uh for observability if you have like telemetry data in high volumes shipped across from from different availability zones to a database that

is stored somewhere completely different then it starts to get like really really expensive and it's something that can also happen when you self-host uh unless you pay really specific attention to it. So bring your own cloud you know works really great as we just mentioned right you store telemetry it's been shipped and stored on your cloud premises but the question that part of you might be asking

right now is yeah that's great for logs metrics traces I get it right but what about AI I'm using external services you know outside of the cluster how exactly are you solving that right you can't do that from within my cloud right so we actually can and this is an example that I think we need to click the middle Sorry, this is actually an example that you're

going to see here of uh how how seamless that looks like. This is the agent inside ground cover as an example. Basically, the all the data you see here on the right, this is telemetry stored on the bring your own cloud back end. All the investigations that happen on the left with the AI agent is basically operating on top of the bedrock in this case of the

customer. So telemetry is being stored on the cloud premises but also AI operations are basically being managed and served from within the cloud premises across the hyperscalers cloud uh native AI capabilities whether it's you know bedrock or vertex or whatever you guys are using. So you can use the telemetry, you can use it with AI and you can uh use it very privately and very securely even

if you let the entire R&D team for example query very sensitive data like tracing and logs through AI platforms uh and through LLM. Right? So this is also a very big leap forward even when it comes to how you operate with AI on top of this telemetry. It doesn't have to stop with the normal methods that we know that we know from before. So just to kind

of touch on uh you know how it looks like from your perspective, right, as people that use the platform, it looks complicated, right? We just talked about we talked about so many different moving parts being deployed, being scaled, being managed inside your cloud premises. But to be honest, this entire thing is self-s served. You can install the bring your own cloud back end in a few clicks

on any cloud provider that you would want. It's elastic. It's scalable. It's fully automated. You don't have to manage anything. If you if you want to figure out how it's working, you can. But it can also be a black box if you don't want to like any sus service that you provide. This is the future of where this is going. Storage behaves differently. It's very cost-ffective. It's

offloaded to object storage. It's tiered. It's exactly the best benefits of how you can operate. And it's also aligned with your infra spend, right? You're paying for this infra. You already have saving plans, EDPs, right? You know how to buy infrastructure. This is fully correlated with how you spend your cash today on buying infrastructure from the hyperscalers and you can benefit from that and and save even

further cost through the through those plans. And from an engineering management perspective, if we have uh if we have a 3 a.m. incident when we don't bring our own cloud, then you self-host your stack and performance might be a problem if 50 engineers suddenly log in and start to query the data like crazy when to find out what's broken. And in other cases, you might find the

data is missing because you sampled it because you tried to distribute it or stored it partially or siloed it. and that may easily cause uh root cause analysis to take hours and that's uh a direct effect on your customers. On the other hand, when you do bring your own cloud and you have the same 3 a.m. incident, every signal that matters is available to you. You store

it even deeper than before with the right cardality to help you investigate. And this can easily cut troubleshooting time to minutes and then you can go back to sleep. Basically, it means that you shouldn't have to choose. The trade-offs engineering teams need to take nowadays um when it comes to observability are just too hard. You dedicate endless time for maintenance of observability stacks that you're just not

trained to build. Or you need to decide every day what kind of data won't wake you up at night and won't haunt you down if you miss it during an incident. And these trade-offs go all the way uh to to teams deciding not to cover critical observability verticals like tracing or even not to cover entire environments. There's a better way to monitor high cardality cloud native stacks.

And hopefully this third option that we presented you today, you like it as much as we do. >> Thank you so much. >> Thank you. Any questions? We have one minute, I was told. Perfect. We're We're going to be around and we're also going to be in the ground cover booth. Uh you can spot these yellow t-shirts if you want to ask me or Vicki any questions.

Thank you very much for your