From Creepers to Clusters: Evolving Minecraft Into a Cloud Native... Jaden Walderich & Alex Mizerak

In this talk, Alex and Jaden, platform engineers from ZAX, share their experience in managing the infrastructure of CubeCraft, one of the largest Minecraft servers. They recount the server's evolution from a small project founded in 2012 into a platform managing over 30,000 concurrent players. The speakers discuss the challenges they faced with scalability, performance, and system failures due to the explosive growth of player numbers. They describe their journey of transitioning from a monolithic architecture to a microservices-based platform using Kubernetes and the Caucus framework. Key focus areas include handling network latency, managing player data, and optimizing server performance to accommodate a large user base. Finally, they introduce the Rocket platform, designed to streamline server management for CubeCraft and potentially other game studios, while contributing to the open-source community.

All right. Hello everyone. Um, welcome to our talk. My name is Alex and I'm here together with my colleague Jaden. Um, we are both platform engineers at a company called ZAX. Um, our talk is basically a journey of one of the biggest Minecraft servers. Um, so you've all probably heard about a game called Minecraft. Some of you have probably played it. Some of you um maybe seen

the movie. probably around 50% of you have at some point tried to play the multiplayer version of Minecraft. Um, and maybe around 20% of you have tried to run their own Minecraft server. But have you ever wondered what happens if there's suddenly thousands of players trying to join your Minecraft server? Well, that's what this talk is about. So, it all started in 2012 when Keepcraft was established.

Originally, it was supposed to be just a small Minecraft server, a safe space for a few friends to play together. But it quickly became popular, and that's where it all started. So, between years 2013 and 14, the server started getting more and more popular. Within just three years, we've managed to hit around 30,000 concurrent players, and we were not expecting this. Um so it is important to

mention that at that time we've all been just teenagers we didn't have any degree in a computer science or we didn't have any formal experience and so our infrastructure looked something like this. Um so since we weren't having any jobs we were just teenagers we also didn't have a budget to buy a fancy powerful servers so we had to work around it. Um, we were overcooking overclocking

our servers to make sure we can allow more players to join the game. Um, and when you look inside what the ser inside the servers, this is how it looked like. Um, each server was running a single instance of a Minecraft server that was basically the the copy of the game that allows the players to join the server and play um on our world. In addition to

that, we had a separate instance which was also running a Minecraft server, but there were no players. That was our backend solution. So, we just had a Minecraft server with a different code, different plug-in that was basically communicating with all the other servers through IRC um to make sure players can play the game they want. Um, so we use no frameworks, no orchestration, no containers, nothing at

all. Just the IRC network, um, and few micro servers and that's basically it. Thank you, Alex. So, one of the things we found is that we were having explosive growth with CubeCraft. And it's never a bad problem to have too many players, but you never want to fail from your own success. And so as we started hitting those numbers and as we started seeing all these players

joining CubeCraft, it was very clear to us that we needed to scale. At this point in time, we constantly had people fighting and intervening with our systems just to keep things online. And we knew things had to change. So Microsoft reached out to us and they were asking us to partner with them and to bring CubeCraft and its infrastructure to the console and the mobile editions of

the game. We wanted to support more players dynamically and we wanted to scale and eliminate these single points of failures. And so coming back to this infrastructure graph that we had at this point in time, this was our backend. We called it masterhub and it was nothing elegant. It was a bucket plugin that handled effectively everything you would typically want a backend to handle. And we had

very different points of failures. anything from MySQL to IRC which yes we did use IRC and we needed to change things. So what was the the kind of clear solution to that? Microservices. We knew we wanted to break apart this monolith. We wanted an actual server orchestrator that could dynamically scale and we needed to eliminate that single point of failure. With MySQL it was a bit more

trivial with replication. With IRC we could scale up to multiple instances. But breaking apart this monolith, we realized with MasterHub was not as easy as we thought it would be. So we thought this would be a little bit more of a painless process. It took us four years. Our monolith was quite interconnected. We spent a lot of time building solutions by ourselves and we didn't use Kubernetes

and we didn't do it for a variety of reasons. But at that point in time, we designed the entire stack ourselves. We even created our own common boot application and all these different things in order to get rid of this master hub thing we had. It took us So we got to 2020 and we hit our goal. We got rid of masterhub. We got rid of this

monolith we had uh with homegrown solutions and we're like this migration's done. But was it a success? Not quite. We effectively just replaced it with equally difficult to maintain solutions. We had these applications scattered everywhere. We had no sort of orchestration, but we solved some of the scalability problems. We created some new tech that was able to scale to support so many more players, but it didn't

quite achieve what we wanted to. We did have high game density. It was very fast, but it was very disorganized. Uh, and as these technical solutions scaled, we realized that sticking to our current stack mean we have to do everything ourselves. And that's not the position you want to be in. And so after we kind of re-evaluated our our scaling and what we had done in 2020,

we knew we wanted to to change. We needed to do it right this time. Yes. So we wanted to get it right. Um we didn't want to just reinvent the wheel again and we didn't want to create something that's difficult to maintain and also difficult to pick up by new developers. Um so instead of building just a backend services, we've decided we need a platform. We wanted

to build a platform that is able to manage our workloads as well as provides tooling that allows us to see what's actually going on. Um so in order to understand what we are trying to build, we first need to understand what problem are we solving. And so a typical Minecraft server usually usually has a lot of players connecting from all over the world expecting little to zero

latency. A popular Minecraft server is usually also an um a target for a lot of network attacks. Um and they have to deal with it otherwise they won't be popular anymore. Um a large MA server has usually also a lot of dependencies whether that's just configuring through configuration files um pre-made worlds or different uh plugins. And so our shape of what we are trying to do was

quite complex. In addition to that, um, because we were already growing, we were trying to solve the typical scaling problems. Um, so we had had a lot of bandwidth because a Minecraft server also needs to send a lot of data to the player. We are basically sending all the textures and the actual world to the player. It's it's not just a simple um request response connection. Um,

in addition to that, we also acquired a lot of compute. A Minecraft server is very memory intensive application and requires higher frequency CPU cores. Um lot of players also meant we had a lot of player data we need to deal with whether that was um player statistics, leaderboards or player saves. Um and you know and in the end it meant we had a lot of features whether

it's in-game features which are sealed by different APIs or web panels that help us manage our infrastructure. Um so understanding the requirements helped us to realize what are we building. We understood that we are not the only one um trying to solve this problem. Pretty much any game studio needs to solve it on their own. And therefore we've decided to create a universal platform that would allow

us or any other game studio whether that are small or large to focus on building what they love on building the games rather than focusing on creating all the supporting infrastructure. And with that idea in mind, a rocket platform was born. >> Thank you, Alex. And so, yes, Rocket was what we came up with in 2020 after we realized where we had come from. And so, we

decided to enter the cloudnative ecosystem. And the first thing you see here is there are so many different things to choose from. And often times this invites a lot of choice paralysis. The cloud native ecosystem is quite vast and you all and you always have to figure out exactly what you're trying to do and which tools solve it. And so especially when it comes to different distributions

you could use anything from open shift, you could just use vanilla kubernetes and then you can use other tools like rancher and docker swarm and all of these. They each have their pros and cons especially when it comes to all these workloads. However, the choice we landed on was OKD. And so just some background with OKD, it's effectively an open source version of Open Shift and it

is an opinionated Kubernetes distribution. And so it makes a lot of those choices for you and helps eliminate a lot of that choice paralysis that often comes when it when you're entering the cloudnative ecosystem. When it comes to designing our microservices, we chose the Caucus framework. And so having built a lot of our own infrastructure and microservices on our own, we we already had a good understanding

of how a lot of that tooling worked and coming from a Java background, it seemed like the obvious choice for us. And so Caucus is very similar to Spring, which we had experience with as well, but it is effectively designed to run in Kubernetes and in the cloudnative And so when it comes to designing the actual platform, we've decided to split into two parts. A game platform

which is what we already had the experience with and then a data platform which is something that already has solved the problems for us. Um so the idea behind the game platform was that it should be a simple platform that allows us to run the workloads our Minecraft servers. It should run on both cloud and bare metal instances and it should come with some tools which allows

us to effectively debug what's going on um and have the overview of uh well basically what's going on. Um and then the data platform which of course deals with data um that means um anything regarding the games any any supporting features whether whether it's matchmaking any social features or so on and of course it also has to deal with the actual data storage that means saving any

um play data such as the worlds their inventories and so on. Um so we will need to keep the design simple. The data platform um as I mentioned is something that uh is all solved. We've decided to go with a verified design. We wanted to put GraphQL um as of a data interface in front and in the background uh the data interface or the gateway would be

then communicating with our microservices using gRPC. Um and then of course the game platform is where we officially uh originally had our own solution and wanted to build something better. So we've decided to go with Kubernetes. Of course um when it comes to Kubernetes there is a lot you can do but we wanted to keep it simple. So we decided to have some kind of deployment which

of course has a lot of pots which would be our Minecraft servers which could be autoscaled. In addition to that, we had our uh custom solution for the load balancing because if you compare games to a normal micros service, of course in games the connection is long lived and so uh that's slightly different. Um one of the problems we still didn't know the answer to was how

exactly are we going to run our Minecraft servers in Kubernetes. So, we knew we wanted to have some kind of autoscale deployment and we knew we wanted to have um bots running the Minecraft era, but it wasn't as simple as that. As I mentioned earlier, our games were always quite demanding. They required a lot of CPU, a lot of memory, a low latency networking, and this was

something we didn't have a solution for. So, eventually we ended up breaking the Kubernetes model slightly. Instead of having individual pods communicating through services to different pods or maybe outside of the cluster, we ended up attaching a physical interface, a part of the physical interface to individual bots. We managed to achieve this by using multi- secondary networks which with combination with SRV allows us to attach a

a virtual function which we could say is a part of a physical interface to the individual board and in the end that means we don't have to deal with CNI we don't have to deal with any standard Kubernetes networking um it just behaves similar as if it was running on a bare metal server. which um allowed us to meet all our requirements about the low latency and

uh multicluster configuration when it comes to networking. We also uh wanted to have a look at the node density in our legacy deployments. We are running around 100 micros servers on one physical server. And when you run around 100 servers uh around 100 containers that are running the same workload um you will notice that a large amount of data in the memory is duplicated. So we are

basically wasting our memory. Um in order to solve that we've decided to go with compressed swap memory or zam um which effectively allows us to run more of these Minecraft instances on the same server. Now, when it comes to Kubernetes, swap is a relatively new concept that has not been really popular before and now it's slowly making its way to um the stable release. You can only

use it in the latest version, I believe. And um therefore, that's something we are currently experimenting with. However, Zam is still something we were not able to get um to work with just yet. Um so, that's something to have a look at. Thank you, Alex. Um, so since 2020, we have iterated on the original design quite a bit. So the original core ideas and concepts are still

the same, but how we actually work with it and how we have built out this game platform is that relationship has also changed. So we take the approach of almost like a client relationship between CubeCraft and the Rocket platform. When we first started building out the platform, we tried to be everything. We tried to create a platform that everybody could use. And the downside to just jumping

into that with this very very wide opinion of everybody can use this platform is it's very hard to actually deliver something. And so our scope was very big. We were constantly iterating on different designs and we were not able to deliver very much early on. And so then we decided having CubeCraft that already exists that already has a demand, a huge demand, a player demand that we

want to integrate into Rocket, we took the approach of having CubeCraft effectively being the flagship client and we started designing stuff for CubeCraft. And that on its own also invited its own set of problems because you're trying to build a game platform for a lot more of an agnostic group perspective and you're also trying to build it for an existing client. So this immediately invited some tension

into how we build out this platform. And what we found out is this tension is actually quite healthy because it helps us iterate on the actual design of what we're actually delivering and actually creating results that CubeCraft can use, but also that we could later bring to the to to other potential partners. And so helping us maintain kind of an agnostic design standard while also delivering real

results for CubeCraft meant that we were actually able to build out this platform and deliver real results. And so f and we can focus on developing this tech right away. And also given that CubeCraft had these scaling needs, it meant that we had to create a platform that scaled right away versus just throwing a client or throwing CubeCraft on a platform and then trying to scale that

up later on. And so as we built out this platform, we have actually been able to build out for multiple different partners. And so we not only have CubeCraft, but we have a number of other people using our platform as well. And we've also gotten to the point where we can do small R&D projects with other games. Um, some of you might have heard of Hightail. We

able to do a small R&D project with that. And we're able to grow this platform to support more titles than just Minecraft. And so, as Alex showed earlier, this is a bit of what our architecture looks like today. It's a relatively simple setup here. Um, however, this is just a part of it. If you zoom in a bit more, it's actually quite a bit more complex. So

we have more of like the regional cluster and the engine name space. We have our data cluster and all of these have different interminglings with how they communicate with each other and how that scales up and like the lifetimes of different pods, how you request these. And so it's a relatively kind of complex interworking of everything once you actually look into it a bit more. And so

we did end up deploying stuff. We've successfully deployed our data platform and ever since then it's been dealing with around 5,000 GraphQL queries per second from all our uh 1600 online instances. Um we're also dealing with around 16 terabytes of player data on a daily basis. Um and in addition to that we were also able to deploy our part of our game platform. We were successfully able

to deploy our new version of the load balancers which allowed us to effectively validate the concept of the secondary networks with the SRV and um also allowed us to have more um flexible deployment allows. So when we have a look at our road map, we of course want to continue growing the platform. We still have a lot of work to do on moving all the cubecraft workloads

to the archet platform. Um and in addition to that we also intend to continue with our efforts contributing to the upstream communities. We have over the time identified few issues uh with the uh Quarkus library and we were able to uh sub in some patches upstream. In addition to that we are also involved in the OKD community. We're helping out by building different operators and therefore making

the platform better. Um and of course we are looking to grow as I mentioned and therefore um looking into more uh bigger multicluster management um and scaling. So the lessons we've learned we've learned that it's important to firstly understand the problem and then think about how to solve it. Uh and when you do so, you'll probably find out that there is um a solution for your problem

or there is only a tool you can use and therefore you don't have to reinvent the wheel again. And lastly, we've also discovered that if if you free if you if you free your platform as an internal platform, it will became your own product and therefore it will change how you build it. And that's basically all. Um if you have any questions, feel free to ask. Uh

there should be a microphone at the front otherwise thank you for listening. >> Hi first of all thank you very much. Second of all the fact you used IRC that is awesome. I love that. Um I am old. I remember IC fondly. um also work on running game servers on Kubernetes. So I've got some background on that. What I've seen in the past with people trying to

run Minecraft is issues around making sure people get the same volume mount >> so that they have the same cuz storage data on disk like a consumer like server. How did you handle that? Cuz that's a tricky problem. >> Yeah. Um so we are basically running mostly the miniame servers which means we don't necessarily store the data as the world on the server that starts and so

we at the beginning of the game uh the server is able to download all the configuration the world or it's also bundled in the actual docker image. >> Nice that makes a lot of sense. >> Thank you very much. Yeah. >> Any more questions? Um, one thing I'm curious about is, uh, there's another open- source, uh, project by Google to schedule game service, Agonies or Agonis, I'm

not sure what the pronunciation of that is. Uh, did you ever look at that? Was that not even out when you started or >> Yeah, we had a look at it uh, at Agon. Um, the most the biggest problem is that it doesn't really fit the kind of uh, workloads we're running. is more focused on basically one game server having one map. Uh we are trying to

focus more on being able to run multiple games within the same Minecraft server and therefore there's a difference in how the worlds uh run as well as the actual scheduling around when the game starts when the game finishes internally. >> Okay. Thank you. >> Hi. Um I had a question of why you decided to go with the combined gRPC and um I forgot what the other uh

yeah was. >> Yeah. As opposed to just gRPC or just the other one. >> So I think the the biggest uh benefit is that with with the GraphQL you are able to have different mutations of the queries and then in the gateway um you are then able to distribute the load between different microservices. So we can make one large query saying I want to get all the

player data but in addition to that I also want to get all the um data from maybe the leaderboards of that player which might be a completely different service but it's one large graphql query and then it goes to different microservices for gc. >> Got it. Thank you. >> Hi. Thank you. Nice talk. Uh I host my own small Minecraft server and do you maintain your own

images or >> which what was the question again? >> Uh the container images you run. >> Yes. Uh we we do have our own fork of paper if you know it which is like a Minecraft server software and then of course we build our own container images for it um with lot of extra tooling on top of it. >> Hi. I love your story. Um, how old

were you when you started and what kind of team did you use and who contributed to this? >> Um, so we're not the original team who started this, but I believe the person who started the server was 14 or 15 when they first started >> with Minecraft. Um, and was also learning how to code at first. And so a lot of us have a similar journey coming

into Minecraft where we either learned coding through Minecraft and uh, yeah. started with Redstone and then went to >> Yeah, we started with Redstone and we're to Kubernetes. Y >> cool, man. >> Thanks. >> Hi. Uh I'm curious uh have you revisited the choice of suv especially for the gaming servers themselves? Like I understood or I understand it is useful for waterfall or for bungee but does

it make sense to for the game servers? >> Yeah. Um so our idea of course is to have the load balancers, the waterfalls to have the SRV on IPv4 and then from there on we want to do IPv6 to the game service through SRV. Um and still the main benefit is that if you compare it to a normal CNI a Kubernetes network, it is not guaranteed that

it will take the shortest path to the server. But if you if you take SRV into account, you are basically able to attach it to a physical network on the physical server. So maybe it's like a VR on OVH or maybe it's like a private network from head snare whatever and then you will have the guarantee that it will reach the the bot in the shortest possible

path. >> No worries. >> Hi, good talk. Um you briefly mentioned that you are uh planning on support of hightail. Uh do you have any kind of I I don't know uh estimated date. I'm a big H high fan. Anyone is here. >> We all we've done so far is effectively an R&D project and so that was just to validate the platform. So yeah, that's all we've

done so far. >> Thank you. Have a good day. >> Um second question, um one thing I'm curious about is uh you started off with a picture of your server rack. >> Are you still hosting everything yourself? Because I do know like you often times I know some server providers that even overclock CPUs or whatever else just because Minecraft is so reliant often times on just core

frequency. What do you do there still? >> Um so we do not host everything ourselves. Currently we have um we are basically renting bare metal servers all around the world in Europe, America and Asia um from OVH and then of course we do have uh different Kubernetes clusters where the actual uh what we call data platform runs. So that's the other part and that's on our own

metal. Yeah. Yeah. And then another official question. Um you mentioned DPDK and that you use that for better network performance. Uh that's for the Minecraft servers or for your data plane. >> DPDK I I think we No. Well, so what we do is we attach the secondary interfaces to the game servers. Um and that's how we solve the network problems there. And for the data platform, we

of course just use basically standard Kubernetes model. >> Okay. So you don't use DPDK. You just forward the raw network interface. So there's not as many translation layers. Okay. Thank you. That explains it. >> Oh, good. >> Great talk. Exactly my history how I started with it. >> One question. What's the recommended I should have to a server? >> So it usually depends on what you're actually

doing on the server. Of course, if if you're running mini games, >> it might be as little as two uh memor 2 GB of memory. If you're running a single multiplayer server, like a vanilla world, it depends a lot on how many players you have, how large a world is. It might be as little as 1 GB until 64, maybe possibly more.