Kubernetes-Native ≠ Cloud Native: Avoiding Architecture Theater - Prerit Munjal, Groupon

This talk focuses on the distinction between Kubernetes-native and cloud-native applications, highlighting the misconceptions and complexities surrounding these concepts. The speaker, a senior technical product manager at Groupon, shares insights based on his experiences managing CI/CD, cloud, and Kubernetes infrastructure. He emphasizes that cloud-native definitions can vary widely among organizations and that Kubernetes is not a one-size-fits-all solution, particularly for mid to large-scale companies. Using examples, he critiques the operational challenges and overhead of maintaining Kubernetes, arguing that often simpler or alternative approaches may yield better results. The talk aims to provide realistic action points for attendees to consider when implementing these technologies in production systems.

Thanks for coming in guys and uh I think we got about 950 RSVPs. So I I think we have about 300 if I'm not wrong if my math is correct. So thanks for joining. Uh the talk's title is Kubernetes native is not equal to cloud native uh and avoiding architectural theatrics. uh some of the people uh will not uh share my opinion like the opinion that I'm

going to share. So we may differ but uh in the end our goal is simple uh that uh we need some good action points from this agenda because you are giving your 30 minutes to this talk. You should gain at least one or two action points so that you can implement them in your production systems. Okay, so another exaggerated introduction. Uh I am the senior uh technical

product manager at Groupon. Has anybody heard of Groupon? Okay. So I think just 10% 10%. So at Groupon uh we believe in things uh that real things happen in offline fashion. So at group on uh uh we are totally against this this particular doom scrolling thing. So um uh whenever we travel by metro right so everybody is just scrolling scrolling scrolling scrolling scrolling and we believe in

that in in the agenda or philosophy that real things happen in the physical world with the right people just like we are at CubeCon. So we all could have just watched the videos online on YouTube but we chose to come we chose to uh visit the booths we chose to add this session to your agenda. So this is our vision. But coming to my introduction uh I

I manage all the teams uh CI/CD, observability, cloud, kubernetes, uh your databases, CI/CD and like everything regarding infrastructure and platform engineering. I create content on YouTube and LinkedIn uh and I have got about like 100K uh of followers and 20 million of yearly impressions. Uh I like sharing content or tokens because these days everything is a token. Um everything is a token. So uh I speak at

a lot of conferences, CubeCons, uh every CubeCon, Open Source Summit, GofferCons. Uh so I really love sharing my opinions and my uh thoughts with all of the all of the people. I love dogs. I I have a golden retriever named Jerry. So he's missing me. He's not eating food from past 2 days. So after this conference, I'm just going to going back to my home and Amsterdam

is really really windy. Uh I don't know why every time every single time I I'm here in Amsterdam, it's it's really windy. So I came here in September, it was windy. It was rainy. Today it is windy and rainy. So uh really uh very different environment than India. So it it's pretty hot in So a disclaimer that none of the things that I share because Groupon is

a publicly listed company. Uh so such disclaimers are important. Uh so this particular talk actually originated with uh when I was talking with Kelsey Highar in open source summit one year back and and I asked him a simp simple question. Give me some tips for my YouTube learners to learn Kubernetes. and he said that why do you even need Kubernetes right so uh this talk actually originated

from that statement so we had a exchange of thoughts for about an hour and u this uh this talk was originated there so guys can you please scan this QR code so this is a slide and we will just see uh what actually cloud native is to you because I personally feel Right. Cloud native has different principles for different people, for different organizations, for different different people

with different cultural backgrounds, uh different u kind of uh uh uh backgrounds. So, uh can you please scan it uh and share like what is cloud native to you? Uh because for some people it can be about using cloud, for some people it can be about cloud portability, for some people it can be about scalability, it can be about resiliency, it can be about disaster recovery, it

can be about any number of different parameters. So I hope everybody has scanned it and let's see okay so cloud is complicated. Okay. Cloud is sovereignity overengineering. Agreed. Resource manager declarative scalable availability agnostic. Okay. Very very different opinions. Right. So the this particular talk's agenda is exactly this that different people different organizations have the different definitions for cloud native. Here you can see okay money uh work

modern f freedom opensource but no cloud is open source uh using CNCF projects transparent but actually you can use CNCF projects in onrem as well so uh this is kind of not relatable uh useless yeah art of building scalable modern apps. Okay, so guys for this particular talk, I actually built a Pac-Man game. Okay, so it works. It is not like it doesn't work, but it it

works, right? It works in some weird fashion. Uh but it it actually works, right? So I'm not a game builder. I'm not a developer either. So I'm just an engineer like infrastructure engineer. So uh this actually works. So can anybody tell me? Uh so what I did was I I have kind of you just have to guess is this application cloud native, non-cloud native, cubernet is native

or AI native. Don't see my other tabs. Yeah. Anyone just speak out loud, right? Because on YouTube channel like when people will be watching they will be hearing our uh like comments they will be getting more context. So anyone cloud native non-cloud native cubernetes native AI native AI native okay anyone for kubernetes just one hand for non-cloud native okay okay nice uh yeah so uh let's go

back to the slide I I will show you like who was right and who was wrong but let's go back to the slides now let's present yeah so this is just what I right now said particularly said that cloud CNCF particularly doesn't have any any solid restrictions uh any solid definition on what cloud native is but from a very high level it should be kind of uh

agnostic uh but for example let's pick pick particular opinions means here, right? So when we talk about portability, designing an application portability, how many of you have ever migrated uh like about any application just after one year of building from AWS to GCP? Migrations are okay, two hands. So migrations are really really hard, right? So uh I think portability is nice to have but that that has

nothing to do with cloud native way because how for example at groupon right we have about 423 microservices who use AWS we use GCP and uh it really uh gives chills in my spine when I even think about migrating any of the service from AWS to GCP. So generally cloud native can be anything for you. Uh but these are the four or five bullet points that generally

cloud native revolves around. When you talk about Kubernetes native uh your app should speak Kubernetes but we all know right Kubernetes is it's I just had a conversation with a gentleman right so he said that most of people here don't know even about Kubernetes. So though we are at CubeCon and I I I like I I teach people cubernetes on my YouTube channel and it was the

most highly watched cubernetes series in Hindi which is a regional language of India. uh and I really was shocked that people uh don't even know like like basics like they they don't they particularly know the basics of your pods containers your u different different highle policies or other stuff but they don't know actually what needs to be done to get a app from your developer environment to

production. So but in general these are the five different categories. So if you align with these five you are kind of and if you have worked with Kubernetes right at scale uh I have a very strong opinion uh just like Kelsey so I I personally believe Kubernetes is not meant for everybody and anybody I think this is not either for scale companies so like how we operate

at big scale how Uber operates I can't speak for Uber but I I feel like it is just for midscale companies companies uh for for high scale companies it's it's not uh good because u we feel like uh for example let's imagine we we want to build a company right so first we will get a product idea then uh in terms of kubernetes you have to write

your yl files the funny part in kubernetes is secrets are not secrets so there's this space 64 encoded uh um keywords then you have to have config maps then you you need to have your deployment then you need to have your service then you need to have your ingress now ingress is kind of uh uh deprecating so you need to have gateway API then you need to

have CRDs you need to have operators you need to have like bunch of different tasks or different mechanisms just to lift off from the ground uh I sold my previous company so uh it it was cube cloud so um Particularly we were we had a thought that okay we should be uh going Kubernetes native the first day and from our perspective it was just running on Kubernetes.

So no topology spread constraints, no good labels, no uh nothing like no KDA, no HPA, no VPA, no multi-dimension scaling, nothing, no service mesh, no mls, just just using GKE, deploying your workloads and just being happy, sleeping good at night. Uh so when we look at from a product perspective, it's really really hard to build a Kubernetes native application. So I will show you the stats because

I built actually four different applications from four different perspectives. So Kubernetes native, AI native, cloud native and non-cloud native. And I ran 12 kind of benchmarks. Uh I agree this particular the application that I built were not perfect but similar is the case for production environments. They are not perfect, right? So uh that level of ambiguity will be there but uh I have like really really good

stats for everybody in the room. So uh it uh I will just share in 10 minutes. So uh this is the learning c of kubernetes right. So you you go up okay I know gateway API then after 3 months there is some new feature then after three releases there is some new feature and uh everybody wants to manage their Kubernetes on their own. So at Groupon we

manage our Kubernetes on our own. Uh we don't uh need GK autopilot. We use GK standard and u uh from like I can't share the real numbers because I'm not allowed to but we spend fairly fairly fairly good number of engineering hours just on the KTL part. So keep the keep the lights on. We spend really really high number of engineering hours just to pump up the

Kubernetes machine. Uh we have about like I think 400 nodes. We have about uh 10 like 10,000 pods on average. Uh for peak I think it'll be 50k. Um and this is the life of a Kubernetes engineer. Uh uh I have seen people going bald with with with the with years. So I've spent like six seven years in Kubernetes. My friend sitting in front he has spent

like 5 years. He's already losing his ears. So uh thankfully I I have mine uh good I so this is the pay bill for cloud native. I have nothing against cubecon or again cl against cloud native but everything that cloud native sold us for is actually not a problem uh for most of the companies. So portability ask from your CEO or from your executive leadership that do

you even have a plan to migrate right because when we talk about portability if you don't commit very very huge chunk of usage to your vendors they won't give you massive savings they won't give you cards they won't give you rais they won't give you special discounts like enterprise discounts on top of your billing so portability in itself doesn't make any sense yeah u so think from

a business perspective, right? So you are running a company and you just need you really don't care if it is running on AWS or GCP because eventually underlying stack is exactly the same in in GCP we have GKE in AWS we we have EKS so it it really doesn't matter which cloud you are using the only thing that matters is your cost savings the next one is

scalability so I have really really good stat for scalability and in my tests the scalability on a multi-threaded VM or a multi-threaded machine was really really really efficient than compared to your GKE because uh GKE not GKE but Kubernetes in general because Kubernetes if you know there are there is something called as balloon pods so you you need to have some empty pods in order to scale

it very rapidly or you need to have very less end user latency not most of the people know about it uh and Like for example, you are running a business in in US Central 1 and for like tomorrow the entire region goes down, you will go down and so will your competitors. So does it even make sense to like spend another $1 million just on DR and

HA? Uh in my perspective, no. And when we talk about DX because it like DX has now converted to KX Kubernetes experience. I personally feel like the more Kubernetes layers we add, the more Helm charts we add, the more YAML files we add, it is not going to make your developers efficient or you're not going to even have a bit of impact on your SDLC cycle. This

is the CNCF ecosystem, right? So first it was a problem which not which was kind of a problem for some of the people but then a chain of problems were created key management authentication and authorization secrets config maps storage databases uh security then your CI/CD then your like this is how economy works right you first of all create a problem and then you create a solution so

why gold is expensive Because we know that because we all want gold, right? We all want silver. That's why gold is expensive. So tomorrow we all forget about what gold is. It is just a stone. That's it. So first we create a problem, then we create a solution, then we create a market. So uh we have a lot of sponsors. Thanks for sponsoring CubeCon. But uh first

it was a problem which was not even a problem for most of people. And then uh there were a bunch of co like samed so just uh after this talk just search cost optimization for cloud and you will find I think 100 companies building the same thing. So pro scraping the same prometheus metrics having a bare thin agent or engine and which will be giving you some

insights. Again uh we all start with very simple cluster but uh as you progress it is about Yagger it is about distributed tracing. Yeah in some fashion these are really really nice. So I think distributed tracing tracing is really a good good skill uh not skill but tool or or kind of technique uh which has kind of uh given us a lot of things and uh uh

this is uh kind of a reality check that uh first we thought that okay uh infrastructure is a problem then we created platform itself a problem. So now we have something called as platform engineering. I personally feel like it is it is not even a problem. Uh yeah, platform engineering is there. Okay, we have a platform engineering team. I lead platform engineering. But uh I feel like

it it is not that big of a problem. Ideally, it should be that all of your infrastructure should be maintained by the developers, right? Because there is a friction. So for example, your developers know nothing about your CPU. They know nothing about your request limit and uh other things for for example topology spread constraint they know nothing about you. Similarly, your infrastructure or platform engineering team doesn't

know a thing about the application or your back end or your particular micros service. And uh at coupons we have a lot of uh aspirational talks right so somebody's selling crossplane uh somebody's selling like previously was it was terraform then it was bsl then uh it was open to everybody is just trying to mug up a lot of tools together build a good UI or a particular

platform not a portal and then sell it to you but you you just need to ask do you even need OPA do you need even your cyics uh Falco, do you need need even these tools? Do you even need crossplane? Why do you even need it? Uh again, I I feel like these days code is not an asset. It is an liability. The more you have, the

more you are in trouble. The more scanning, the more security part, the more maintenance. And with these AI agents, I feel like uh having less code or having meaningful code is more of um uh kind of useful. So uh what do we really want? We just want our applications to be up and running. They should be low cost. But with Kubernetes, I don't know like if if

there is some company which has kind of achieved the peak cost optimization thing for their infrastructure. I personally don't feel like that because it is really really hard at scale because you never know which node is getting shuffled which you you don't know even if you are on spot right spot creates another big problem set uh we had a se one sometime back I wouldn't share it

here but I'm thinking to write it about a blog so it was related to spot so something got shuffled and it was a tumbling down effect uh so there are two kind of companies right startup and enterprise Right. So 200 engineers, 500 services still can't deploy on Friday because they have different problem like because of stability reasons they don't deploy on Fridays. But startups they have three

engineers, two services and they have really everything which they don't even need and uh this is kind of a very generic problem statement that your app is. So uh when I teach Kubernetes I also do the similar mistake. So I I tell them to deploy the resumeum on Kubernetes. So because it is the most low lowh hanging fruit right. So it is always very easy to achieve.

Uh and this slide is about platform engineering. So I actually feel like developers should be owning the infrastructure. So there are a lot of new technologies like uh encore.dev. uh I'm not they're not paying me to say the name but uh you the developers actually maintain the infrastructure with Typescript. So that is really really a good thing uh because it will completely eliminate the again a layer

right there will be again ticket ops there will be again uh no matter how much you optimize there will be always ticket ops there will be always friction there will be always cost uh bumps there will be less communication less like more problems so and then talks right uh even I I think my talk will make sense for some but for for others it won't uh similarly

like for all the talks. So DX is going to kx and abstractions on abstractions on abstractions. Uh again uh it's good to have abstractions but uh in the end uh there is somebody who needs to maintain these abstractions. Uh it's good that uh we have all these abstractions but um ideally it's not good. Uh so we solved a problem somewhat for some companies for a limited set

of people but we created 15 different problems uh which for which we need to have 15 different tools for which we need to pay for 15 diff 15 uh different tools and this is how I think economy works. U so humans have a mentality to hold things right to test out things. So I I think it works on that and uh I think it is ever evolving.

every every like in in a year there are two releases from Kubernetes uh uh there are a lot of new alpha beta and stable features uh every time there is some breaking change I I think this ingress uh deprecation they have been notifying us from past I think 3 years or so we are not ready nobody's ready for gateway API because we don't have time uh we

already are struggling with ticket ops we already have a lot of backlog we already have our peak coming in we have uh kind of some migrations going on. Similarly, similar situation is for all the companies. So, uh we actually don't have time if some something replicates and then we need to have some more engineering hours and then this AI wave. We need to have more engineers working

with AI. Um really really really tough. you need to laugh on this please. Okay. So this is the illusion um that I really talked in in in in the beginning. So if you need to take a um kind of a screenshot then uh please uh take it. Um so uh yeah so um 80% of the incidents were actually caused by operational complexity. I think you will find

the link to the particular survey in in the slides. I've already added the slide in in my u uh like session so you can check it out. Um so so the problem here is that we are really far far far away from the actual problem. So the actual problem was infrastructure and devs. Somehow it became infrastructure and devs and operations. So uh I partic particularly feel like

platform engineering was very close to solving this problem but building a portal or building a platform even uh you actually can't solve that problem because the problem is uh again uh for some it may work because it is not a cultural problem but most of the companies who have legacy workloads it is a cultural problem and uh in order to change it you need to have devs

really really close to your infrastructure. Um um I I particularly feel like if your feature team has less number of people than platform teams, you are doing it totally wrong. Uh infrastructure was again what was the benefit of cloud right managed services, managed solutions on demand solutions, you get to have better cost analysis in the first hand. You need to have you can have other different metrics

beforehand. So uh I feel like there is still a lot of things that we need to uh do and uh let's let's actually now I think I I just have 3 minutes. So let's go to uh this stats right. So what I did was I asked claude I gave the uh claude are my particular access to all these three four different tools four different applications and this

is uh the kind of output it returned right so uh number of files I think you should count it too because the more code you maintain the more uh kind of liability you carry forward to um so it it has really some good starts uh the the GitHub repo. Um I I you can just uh I will just link it to my website so you can find

it. I have not uh uploaded it yet. But uh if you see uh lines of code for Kubernetes, it's 3x the times that non-cloud native way works. So um uh if you go here, I think this is entire code base. Uh I'll just share it so you can have better overview of how it works. But lines of code were 3x, total files were 3x. uh dependencies were

high. Your YAML docker and other related configuration files were very high. Uh in terms of benchmarks also um startup time u also it lagged as compared to non-cloudnative way. Uh request per seconds also it lagged as compared to non-cloudnative way. So really some good stats. Um um and uh but but in terms like how better it can uh for some use cases I personally believe Kubernetes is

the way uh for most of the cases. For example, if you're comparing it with cloud native, cloud native doesn't make any sense. I think you should be picking the middle path cloud native that means the uh the five categories that we mentioned before uh you should be following that and outside of it I think there is no need to complicate your stack there's no need to complicate

your operations um and um yeah so I think uh in in terms of DX as well if if developers have really really good access to infra I feel like they can do magic instead of creating like uh again giving them a platform building a portal asking them to self-manage it actually doesn't work. So it should be near near in the code that they built or they code

or they maintain. So it should be in that. But in terms of production readiness like it ran a bunch of tests resiliency test your it kind of have have had a particular uh I had litmus testing it. So it was kind of chaos testing it was load testing. So it had about six tests that it ran. Uh and uh I think the the takeaway from this particular

talk today is that uh also like I I think I missed a couple of slides as well. So um these were the non cloud native way that our application was working in. It was totally garbage and it it is not meant for production workloads. The cloud native way is meant for production workloads. So it actually solved a lot of problems, a lot of real problems as compared

to Kubernetes way. So Kubernetes is I feel like it it was an overkill for most of the things uh most of the real problems that uh most of people don't even know that uh exists. So uh I think this is the agenda for this CubeCon as well. let's keep cloud native moving because uh once AI agents come into the picture so I I actually don't know how

many people use um agents or cloud skills and other stuff but um they're really eating up Kubernetes market so all you need to do is um give devs cloud skills for infrastructure and they can just manage the infrastructure based on cloud scale or maybe cloud sub agent they have multi- aent teams as well so this is how uh I look at this particular architectural theater and um

yeah u feel free to rate the session feel free to ping me or have a conversation I think we're already over time so uh I I would love to have a chat if you if you think um different than what I just uh showcased yeah thanks