What Survived Production: Operating Game Backends at Million-Player Scale - Berkay Uckac

This talk focuses on the backend experiences of a gaming company as they navigated the challenges of maintaining and scaling their mobile game, Merge Gardens, over three years. The speaker discusses the evolution of their operational strategies, including live ops, which emphasizes the importance of ongoing player engagement through content updates and analytics. They detail their infrastructure built on Google Cloud Platform using Kubernetes, alongside tools like MongoDB Atlas and Prometheus for observability. The speaker shares insights on their choice of microservices architecture, the utilization of decision journals to log architectural decisions, and the simplicity of their system design in response to the needs of a small development team. Their journey also involves identifying and mitigating scaling challenges as user traffic fluctuates, along with lessons learned from past incidents.

So, welcome everybody to KubeCon 2026 Amsterdam. So far, I hope everybody's having a great time here. Uh my name is Berkay. And today I'll be talking to you about our back-end experiences over the last 3 years and what we built. And hopefully uh the time is going to be enough because there's a lot to go through over the the things that we've done over the last 3

years. So, it's it's quite packed. Hopefully, there's going to be enough time at the end to ask quick questions. So, uh if not, then you know where to find me in in the within the uh conference or after the conference. Uh so, let's get started. Uh I wanted to I guess contain this talk in general to the small teams or even if you are working at a

um bigger place where it's like you you might need to expand your horizons a bit because we kind of gone through few challenges. Um and and there's also a phenomenon these days that, you know, there's an understanding of of of course like we do love CNCF. And the main reason that we are here is because that I guess we do love that. Um but there's also the

perspective of um you know, within the industry that you can just spawn up a history PS and it's going to be okay. Um and and I I think the reality is going to be like somewhere in between for uh smaller organizations or teams like us where you need to kind of think about the the operational complexities. so, I'll like to start off with our story a bit

and, you know, why we are here today um without going into the full details about our, you know, game or anything. Um so, our game in focus today is Merge Gardens. It it it is a mobile game that launched in 2020. Originally, it had a promising scale and it had a promising numbers. Everything was going well. And then after a while, like with everything else, the metrics

were kind of plateaued over the years. And in 2023, we decided to revamp to Gardens. And, you know, with new marketing, new ads, new everything that you might think about. And of course, as a result, um everything was perfect. Um we had 1,000% more downloads. Players are just like running in. It's all great. And the reason that we do that is for those unfamiliar with like the

gaming landscape here, uh to go I guess this is a really high-level summary of what live ops is, but it is a time that is after a game has been launched these days. So, uh in the past, you would have a game, you launch it, and that's about it. You make your sales. And these days, even the paid games, you see a lot of live ops elements

like new content uh to make sure the players are coming back into the game and then experiencing the game that they love over and over again. Uh so, that is what we can call live ops in a uh high level. Um again, we're not going to go into the details of live ops today is not a talk about um you know, product and business uh analytics and

things like that, but what this means is building um platforms for companies um uh I mean, you know, internally most likely, and then we did that as well. So, and then we called it James. The name's a long story, but know that it's called James. Um the and then this tool does a lot of stuff. Um so, the main thing here, I guess, within live ops platforms

is being able to AB test things and you know, seeing the impact of what you're building is is is a essential pillar of uh live ops these days. Or campaigns and segmentation, known as also targeting as well, where you kind of um make sure that okay, within these times, to these players, you want to deliver these experiences. Or analytics, I guess everybody has analytics, um but it

is, you know, again, quite important within live ops. And this can be also, depending on the company, uh multiplayer features as well, like we have here with tournaments where players can go in and then play the game and collect points and compete against each other. Um, and let's kind of go into now that we understood like what we built, how we built it. Um, so with all

that marketing challenges and the I guess the art and the story and everything that is beautiful and good looking, um, the question is of course raised towards the tech side. Okay, what do you guys want to do? because, you know, the live ops doesn't really happen on the front end. It happens on the back end side quite a lot. Um, and and at that point Future Play

did not have any kind of in-house tooling and the the question was, "Okay, like, well, you're starting with a greenfield project, right? You know, greenfield projects are easy because you don't have to maintain legacy things and then you don't have to deal with the the the previous engineers' faults." Um, not quite uh because then the the questions change um, you know, from, you know, where you can

blame the issue to the previous engineer, now all eyes are on kind of on you where you are like, "Okay, what is going to be the timeline and what is going to be the ROI and what obsolescence and how long this thing that you're going to build today is going to last?" And all these questions that we asked or we got asked. And then we came up

with this list. Um, and then we said that, "Okay, um, due to our I guess like kind of past experiences that we know that high availability is the most important thing for our players and um, making sure that they can enter the game wherever they are from the world at any time. Um, the game doesn't sleep. It's just 7/24 online players, you know, from different time zones

and different countries. Um, I'm not going to go through the every single item here, but the other things here that is um, important and also the scope of this talk again is it needs to be managed by the small server team. In the past it was just me and of course we grew up uh over the years uh so now we are up to three people. So

that is a huge increase. Um, but uh still uh it it needs to be like, you know, really small and contained thing. And um thus more technicalities needed there. And we wanted to stay as vendor neutral as possible. And a programming language that the developers will like. And then we came up with this. Um So, this is actually the current state of it. I took a I

guess a snapshot of what we have uh from last month. And this is the whole thing that runs Future Play and millions of players back end. And you just to go over here because we can spend hours on this alone. Um but a few details that to go through. Uh we are using a single cloud provider, GCP, on a single VC VPC. And we have three GK

clusters. Couple clusters on the MongoDB Atlas side as well. We do have Confluence, Kafka, and then the most standard observability stack in the world, Prometheus plus Grafana. Um and that's about it. Really. And and then like you know, again to if you if you kind of take a peek within a cluster, you can see that we run the whole production with just three nodes that are four

core each within those nodes. Uh so, it's quite lightweight platform that we built for our players. Um So, let's slowly build this now and go through the decisions that we made that that is maybe helpful for you and you're on your journey. Um because I I think starting off with this is important to, you know, for other decisions that we'll discuss today because the the most important

tool that we have is or was, still is, um is the decision journal methodology that I guess there's different variants of this, but we also kept this simple. Um so, starting off with every decision, every major decision, not like what you're going to name your variable. Uh but it's just like the, you know, architectural decisions needs to be logged here and then discussed within the team. And

then every decision has an ease of change estimate. Uh whether it's easy, medium, or hard. Um and then these decisions are revisited depending on, I guess, whatever the circumstances might be, whether it's like every few months or quarter or a year or every time we want to question this. Um so, the first three decisions Well, now, of course, the list here is like quite long. We are

not going to be able to go through everything. So, I kind of hand-picked, I think, the most important ones why we kind of kept it simple. Starting off with the the architecture. Um the the So, we decided to have microservices, but those are in like, quote unquote, status. Uh because like like you know, I guess everybody knows that there's like you can spawn up 10,000 different microservices

and have them everything async. I think if that works, Godspeed to you. But for us, it it was like Yeah, with that small team, do we really need that? Do we really need like a new microservice for every single um endpoints? And the answer was like, "Hell no." Um and then we were just like decided on an approach the that we kind of call micromonoliths. I don't

know if this I guess is an official term, but uh so, these services are kind of decided in terms of the scalability concerns. So, for example, we have a service called tournament service that is quite self-explanatory. And then when we have tournaments running during the schedule, we can scale that up differently compared to I guess a different service that we might have. Um so, those are the

main, I guess, reasons. And these are mainly stateless services. And us choosing Go as the programming language made the container station quite easy and default. Um we are not really planning to get off Kubernetes anytime soon, but even if that is the case, we can just chuck those images somewhere else and it will still keep running. Um speaking of Kubernetes, I guess this wouldn't be a talk

in KubeCon without why Kubernetes um state. So, we all know why Kubernetes is great and all all the stuff, but we have our I guess own reasons that you know, well, not unique reasons, but the important reasons that why we chose the Kubernetes. The high availability again self-healing. Um It that was the the biggest I guess deal for us here. Um the the vast ecosystem that it

has and provides us for a small team like us is is quite important where we don't go ahead and you know, try to find in a random forum from 2006 like figuring out the issues. Um instead it's just open open source ecosystem is like really great and communicative so far. Um and also some amount of operational familiarity over the years within the team as well. Uh with

that said, I think the the minuses are not so minuses, but the trade-offs that we made over the the years and then we like accepted and there is no point in denying them that the the initial time sync is real that I uh we were kind of in a unique position where we already had the players and then we built the infrastructure on top of it, but

if you are just starting out from scratch without a single user or player, then do you really want to spend your time like building the infrastructure first or do you want to just build the application and validate it the idea. That is bit of a business decision, I guess. Um but it is real um and the infrastructure and operational cost is not too high, but it is

something to uh be mindful of. GKE platform limitations, it is not the end of the world, but worth mentioning here that the there was different control plane versions and how Cilium behaved between these versions um I guess either we completely misunderstood the documentation or there is some kind of a mishap happening there. Um but we are still on GKE. That didn't change anything. The the I think

the another the the third major decision that we took to keep the communication simple as well. Um so for I guess good 1 year or 1 and 1/2 years we just used HTTP wherever we can. Like not as in like HTTP, of course it's HTTPS, but it's the technology. Um the the main reason for that we noticed, okay, you can spawn up those 10,000 microservices, you can

put a you know a big cluster of Kafka there that is perfectly viable if you really need it. But a small infrastructure like us, do we really need it? Because what we noticed is that the reactions that we have from client to server is always almost in sync. We don't do a lot of async stuff, so it's just request and response. And then do we really need

to send that through Kafka if it is just already waiting for a response or can we just simplify this with a HTTP call? That's what we did. Although when needed, we did introduce the Kafka side and the the the message brokers. Um especially um within few functionalities that I'll get into in a second. Um and then we also introduced web sockets to the client side for more

real-time communication later on. Speaking about web sockets and message brokers, I think I just wanted to um again take a few seconds here to explain like the main root cause that we or not cause but reason that we decided to do this was mainly because of in-app purchases and like purchase systems where it happens more async and we don't want to block users while that is being

processed. And then we also needed to push items into the client rather than that request response attitude. Um so yeah, that was kind of the main reason. This is not a Go talk, but I still wanted to mention the uh the beauty of Go here as well. The implementation that we have with web sockets, we have like five to six digits of open connections at every given

second and it just like is nothing like when it comes to registering the resource usage with CPU and memory. Um so we've been also quite happy with it. Um although again, even though our Kafka is managed through Confluent, I would I would still say that there is a overhead there as well when it comes to the operations. cool. The let's go into the data side a bit

why we chose Well, we first chose that if we wanted it to be a document database, not specifically. kind of came later because we had previous experiences with it. And the reason that it's a document database, again, due to the previous experiences where we do believe that the document database is a great choice for player-centric data schemas where it might constantly evolve or change and like that

the constant thing is only, you know, the change is the constant kind of thing. Um so that's why we decided to go with that. Um and we'll use Atlas again to as a managed service to, you know, reduce the ops burden a bit as we as we're a quite small team. uh this is the only thing that we have at ease of change hard in the whole

journal. Um it's not impossible, but we need to have a really good reason to change it. Um so so far we haven't. Then the the the latter part for the keen eyed of you you seen the the graph earlier with the architecture, you can you could have seen a ClickHouse there uh hiding between the notes. Um that is a recent addition that we had for um it

is not as a permanent analytics data storage, but it is used as a hot storage of some sorts for few tooling and then more easier access and uh lower queries and all that. Um and and the the the scale is well, we don't have like, I guess, trillions of events like it's been advertised. It's just a few hundred million per day and it has worked us fine

even with our smaller nodes capacity that we have. I'm not going to spend a lot of time on the observability because there's literally nothing to talk about. It's just I guess everybody here knows about what Prometheus and Grafana is. Um maybe few quirks is we do use GCP managed Prometheus and then we use cloud login like it's a bit of a aggregation slash search parameters thingy there

and then we have PagerDuty for just managing the on-call and alerts that everybody likes to get at 3:00 a.m. Um They We are not a again to remind again, we are not a huge team so we don't have a 8,000 people supporting us when it comes to the developer tooling so we build our own tooling as well, which we don't luckily need that much because we also

manage what we build. Um so we kind of are already in the system. Um and they the only thing that we have is after a while we started to notice especially after few people joined the team, okay, we need to have some amount of like segregation into what we are building day-to-day and things that like up-and-coming. Um and then we decided to have some kind of a

multi-tenant system and that is a fancy name for just few namespaces and then our back rules. Um and then it's actually a bash script behind the scenes. So it has that like 2:00 a.m. fancy all that but it's everything is a bash script in the end. With the few times that I guess like halfway through the talk here, let's go into the things that we think that

not really failed but also failed um with the incidents. So how not to read this? This is an interesting one. I guess people usually like, "Really?" the the reason that I guess many people usually is that of course there's repetitive like responses that you give and then you don't want to go back to the database every time and then lower the burden there, faster access and everything.

But most of our responses to the client are actually quite dynamic. So, then when the user requests once, the twice, most likely that is already changed. The content is so we don't have a use case there. But we almost got close to using Redis few instances. Um with campaigns caches, we were like, that would be a good use case. And then we did that though that why

is this not in an application memory because it's already just a few megabytes of data is just few JSON lines. and then we decided to do that. We didn't introduce network call there. And then segmentation cache, same thing happened again. Okay, why is this you know not in the application memory because the application overhead is already low. Like it's okay to have 1 megabyte per pod. Um

the the final one was like the player session information. Okay, I'm like, this is the time we're going to get Redis. And then we're like, okay, we do seriously need sometime like it is a centralized cache now. Uh and then the the kind of the idea there was like, okay, but you know, if you installed Redis then it kind of comes with all these, you know, fancy

stuff that is next to it which is good if you're using it or walkie, I guess whatever you want to, you know, use it Um but we don't need all that. We just need a key-value search like as simple as it gets. And then we just built this. It's like a turtle line go application that is written in 5 minutes and then we promised that we will

improve it. It's running for 3 years and it hasn't crashed once. yeah. Um I mean, um don't take this as a copy paste. Like there's better ways to do Um yeah, it's just like a kind of goes to show you that that 5-minute fix kind of turns into you know, your whole infrastructure. Um and it's working fine. nothing wrong with it. I guess so far. the the

the incident wise, like I'd like to of course be transparent here that we are not perfect. Um we had three incidents, major incidents in total over One of them was caused by GCP network endpoint failure. That is completely out of our control. Um the two of which is we caused it. Um the first one is the the class A, which is also referred as like I guess

player save data that we save to our database for And the there was an interesting case turns out in the client side where the at given hour of a given week of a given minute of a given second, the request just jumped 300%. We didn't catch this of course before launching. Um and and then, you know, we were like, "Okay, what's happening here?" because like it is

so sudden that it happens within a second at a given time and a um that it just brings everything down. so yeah, so we fixed that of course. And and of course that doesn't mean that, you know, clients cannot make mistakes. It sure can, so we need to be ready for it. Thus, we also change things on the server side. We make the uh traffic flows better,

make the dependencies more resilient, and and then the critical path because by this time we already had the Kafka server, might as well you use it to, you know, get some benefit out of it uh of that operational cost. The second one is I guess is is a good summary of the the well, you might call like why keep things simple because even as simple things as

HPA, the horizontal pod auto scaler can cause uh this chaos. I didn't know how to write this down, so I just typed it as a chaos here because it was quite a chaos. Uh so what happened was uh we did a series of optimizations to our config service so we can reduce the request um that we had within the resource values. We deployed that, everything is going

fine, right? We didn't adjust HPA. So, what that is like I guess most of you already noticed is that it kept thinking that it doesn't have enough memory even though it had with the new values, but we didn't adjust that. So, normally we have five config pods. We had 115 at some point. It is not fun. Of course, we noticed it just within seconds because of the

amount of pages due to all the static throughput. Because that we were spawning more, it overwhelmed the admin site because that it admin site couldn't distribute the caches. Then the new config pods wouldn't start because they were not ready. It kept spawning more. It kept overwhelming the cache. The loop continues until you manually fix things. And more things that I cannot mention today I guess due to

the time again constraints and bad assumptions I guess is bad. I don't know. It's just there's no way to make all the assumptions before launching, so you're going to make mistakes and I guess that's fine as long as you learn from it. Scaling mistakes was the you know, HPA itself is I think great and sold a lot for us, but then those things where we know, especially

we coming from the gaming industry where okay, this event is going to start at 10. We know players are going to come in at 10. So, why don't we rather than be proactive about scaling rather than reactive with the HPA. So, we just built a quick tool around it and the complexities that we never actually used and still have no idea where we can use it is

one of them is service mesh. We hear a lot about it. I I'm sure it's brilliant. I'm sure it has its use cases for us. I have no absolute idea that you know, where can we fit that. And VPA because we are already thinking about going down to the lowest score possible in like GCP, I don't think we need to be Um just to recap this I

guess um talk here a bit. Um I think it's a practical examples and maybe as a summary. the the best strategy that we had was simplicity. Like keeping things as simple as possible not because of like fancy principles, but because of like we were the ones who are going to also operate that same thing that we added there. So, we don't want to complicate and and also

the just you know underlining the decision journal here a bit again because I think that saved us multiple times from making, you know, expensive decisions when it comes to complicating things because human memory is kind of fragile and then years later down the road even if you are there, you kind of tend to forget things. Okay, like what was actually the reason that we did this? And

more of course um technical wise. Um you might ask like, "Okay, you you talked bunch and then like is there any kind of numbers you can share?" And these are the reliability outcomes that we had over the last 3 years. Uh so, we had roughly 10 million players playing our game. We had countless amount of AB test campaigns. I don't even know the count. I tried to

find it. I cannot. Um and then the SLO that we kind of set ourselves, we don't have any kind of contractual boundary here, but it is something that we aimed for was four nines. I do believe that we kind of exceeded that quite nicely over the years. Uh 2023 has an asterisk because it's like it's not a full year. Uh so, it's maybe like not representative of

reality, but it was still I guess technically 100. Um what next? Uh so, what are we thinking as a team? So, we are really happy with what we have. So, I don't think there's going to be any major shift except one thing which is hybrid workloads. Uh because we know our baseline uh workload now at this point after all these years and after the you might be

in search of okay like how can we combine some kind of an on press on prem or co-located hardware with using still like communities and CNCF technologies but also the reactivity of the cloud platforms. Great this question mark. We have some features upcoming that uh useful but no promises. And also one thing that I didn't really mention at all today was the CICD it is actually absolutely

bad. Like I don't think I can use the word that I want to use here today but yeah it's it's bad and we want to fix that not by just introducing something single but as a revamp of like how we want to think about our CICD in general. I think that's about it. So thank you very much for listening. We have 5 minutes for any questions that

you might have. You can use this microphone or you can hit me up on email. Hi, I got here first. Thank you very much for the talk. Um for context I would love to hear about what sort of game mechanics you're supporting with this back end like are there real time multiplayer aspects or is it async or like what's what's that kind of loop look like? >>

Uh cur- cur- currently um I would say it is not really real time. Um so the the actual game mechanics was the tournament examples I'll I'll just make everybody's mind confused and go back. Yeah this one. Uh I would say this is the major majority one that we have where you know our PMs and designers can design some kind of a leaderboard slash tournament system and our

system like is responsible for collecting the points, distributing the rewards and everything along with it. Uh real-time this wise, we do have a real-time like analytics, but of course that is not a game mechanic. It's more comparison of like am I beating my friends at this game yet? Yeah. Yeah. >> That kind of mechanic. Yeah. Cool. Cool. Cool. All right. Awesome. Thank you so much. Thank you.

Um yeah, I can imagine that during the week and maybe during certain intervals during the day, you get more spikes like maybe around lunch break, people are playing more and stuff like that. Do you have a smart way of like scheduling your auto scaling in that way so that you are ready for these bursts, but you are not waiting for them? Yeah, it's it's it's definitely the

challenge that made us build this tool. Um it's it's like that's that's the proactivity part that I think I was like trying to mention here a bit. Um it's it's because the like you said, you know, there's different people across the world and we have different event set. And even with those different time zones, people sometimes like stay up late and play the game even and then

this you know, causes that spike that you mentioned. So essentially what this tool lets us do is just go there, schedule the workloads that we think it's going to be like, "Okay, it's going to be roughly this from past experience." And then it scales up during that spike period and then brings it down slowly to whatever you might want.