Redis on EC2 to Valkey on Kubernetes: A Zero-Downtime Case Study - Joe Heyburn, Braze

In this talk, Joe Hayburn discusses his experience with migrating nearly 300 Redis instances to Kubernetes at Braze, a customer engagement platform. He explains the successful execution of this migration without any downtime and outlines the key technologies involved, including Redis and Valkey. The speaker shares insights into the architecture of their in-memory database, including the use of Sentinel for monitoring and failover, and details the complex challenges faced during the migration process. He highlights significant performance improvements achieved, such as a 90% reduction in P95 latency, and addresses ongoing efforts to develop a community-led Valkey operator for Kubernetes to streamline future deployments and enhance scalability. The session underscores the importance of real-time data delivery in customer engagement and the technical strategies that enable seamless migrations.

Let me actually get started and introduce myself. My name's Joe Hayburn and I work at Braze as a staff engineer on the in-memory database team. And in 2022, we decided to see just how crazy we could be by migrating nearly 300 Redis instances to Kubernetes. And we did all of that without any downtime and a rollback path at every step of the way. But more importantly than

that, I want to show you all here today what that migration to Kubernetes allowed us to achieve, which was a completely seamless migration to Valkey, which helped us along the way to unlock a latency improvement of up to 90%. For those who don't know Braze, we are a customer engagement platform. We help brands to send the right message to the right person at the right time. And

when I talk about messages, I'm talking about highly personalized real-time engagements that can be sent over a number of channels. Like, I think we support emails, push notifications, SMS, and I think we even have some customers that hook into letters, physical letters. Are they still a thing? This customer is using them. The messages can be sent as a result of the data that we ingest via our

APIs. So here you can see how the typical message flow works here. And this helps to enable campaigns such a welcome email to a new customer on signup or scheduling a message to a customer that might have abandoned their basket. And that message could be a push notification with a coupon to engage them back onto the platform. Even we can even cancel those scheduled messages if the

customer comes back and completes that purchase without before that message has even been sent. And we're doing all of that billions of times every single day. Braze has over 2,000 customers that expect us to scale with them. So we have got to make sure that Valkey can scale with them, too. When I talk about Valkey at Braze, though, I'm not just talking about one workload. I'm talking

we use it for a number of different things, right? We use it for rate limiting, distributed locks, message deduplication, and as well as that, Sidekiq. And if you know Sidekiq, then you know that Valkey is at the very heart of it, So when I talk about Valkey at Braze here, I'm talking about the entire backbone of the Braze platform. Just to give you a bit of an

idea on the scale that we're running at here at just this month, we have about 600 highly available instances, which we call shards. And between all of them, they process over 36 million operations a second, With a total memory capacity of 6 and 1/2 terabytes. So we're working with some big numbers here, right? Now, I want to take you back to the Stone Age at Braze. We

were running Redis on EC2, which was managed by Chef. So let's dive in a little bit on what this kind of looks like. So in our legacy topology, you can see that this is just one type of uh of a Redis workload, right? And you know, we sharded it till kingdom come. Like we sharded it for caching in Sidekiq, et cetera. And each type has can have

one or more shards to allow allow it to individually horizontally scale out. Where each shard is a highly available pair of a primary and a replica. And they are managed by a cluster-scoped Sentinel. Just I just really want to quick see a quick raise of hands. Who here uses Sentinel? Okay. All right. See, it's not a dying product. We need to keep it going, then. So the

Sentinel is what uses is going to be doing the the monitoring and acts as like the watchdog to check on the primaries uh to see, are you still healthy? And if you're not, it's going to then uh trigger a failover onto the replica, But in our environment, Sentinel is also used by the clients to be able to for them to be able to determine what the primary

IP and port is that they need to connect to. So the clients connect to Sentinel, and the clients decide what shard they need to write to based on their own hashing logic, which is baked onto baked into the client side. So they'll might hash on like a key or a piece of data, and that then decides what shard it should be written to. Now, we already had

a greenfield cluster that was running on Kubernetes, and we had deployed Redis to that with a Helm chart. And that had a slightly different topology to the one I just showed you. So let me just run through that now. So we deployed we had a mapping of a stateful set to one shard, and a stateful set would have three pods in it, where you'd have one primary

and two replicas. Within these pods, we have Sentinel that runs as a sidecar container. and also here, Sentinel is only monitoring this one shard. Right? There isn't like a cluster-scoped Sentinel, what they what like what we had in our in the legacy setup, right? But the thing about Sentinel is when you're working with IP addresses, it kind of needs the replica IPs to remain static, right? Which

when you mix that in in a Kubernetes world, you're then going to have the pod IP change if that pod were to roll. Which then if you see in the map of the replicas that I've got in the top right, it's going to see the old IP addresses being down. So now we have stale replicas, and this isn't good, right? You want to really keep your Sentinel

up to date with what the most up to date uh names of all the replicas it has. So that's bad. We want to avoid that. But for Sent- uh so what we do to get round this in Kubernetes, is we introduce a service for each pod. And that service is going to have a cluster IP that is going to remain the same for the entire duration of

that service's uh life as long as it's been created. Then we have an init container that goes off and discovers what that pod IP uh what that cluster IP is, and it announces itself at that cluster IP. And you can see the example of that, I believe, on the top right. Uh it's now using the cluster IP instead. And the thing to remember here is whilst these

pods can all communicate within the cluster, cannot be these cluster IPs can't be accessed external to the cluster. Which really presents like a really difficult migration challenge. How then do we go about migrating Redis to Kubernetes? Well, for us to be able to understand we need to first take a look at what our requirements are for the migration. And our requirements were just two very simple ones.

The first one was we have to be able to do this without any downtime. And I think we can all agree here that downtime is pretty bad. But for Braze, where it relies on real-time message uh decisioning and delivery, any downtime is going to delay those messages, and the customers won't be happy with that. I know I wouldn't be if I was a customer. So we have

to make sure that things keep running. The second one is we need to be able to roll back to the previous state if for whatever reason when we're running on Kubernetes, we're not happy with how things look. We need to be able to roll back to the previous state, how things were on EC2, without us having lost any data that had been written whilst we're on Kubernetes.

So two very simple requirements, right? Let's now have a look at some options. So Redis and Valkey as well allows you to persist its state to disk in the form of an RDB file. Couldn't we just take that RDB file from the EC2 primary, copy it across to the volume of the pod that we want to become primary on and then it will just reload the state,

right? That's pretty easy. But when we're copying the RDB file, we're not we have to pause the clients such that they don't write any new data whilst the copy is in process, right? Cuz then otherwise, we're going to lose data. So, whilst this is simple, right? There is some downtime in this, which already we said that we weren't going that we weren't able to do. And this

kind of reminds me of one of my good friends, Indiana Jones. In Raiders of the Lost Ark, he's trying to take the idol, and he's trying to replace it with that weighted bag, But, he just missed it by a fraction of a second, and that was his downtime. And I think we all know what happened to him after that. Okay, well, knowing that the data can the

pods can reach the VPC, couldn't we then just configure the server to replicate against the EC2 and use Sentinel to be able to coordinate the failover and promote the Kubernetes pod to become primary itself? I mean, yeah, we've pretty much got our no downtime, but remember, the cluster IP is internal to the right? This EC2 primary instance is not going to be able to access that IP

address. So, as I said, we get our no downtime from Sentinel, we lose the ability for us to roll back without any data loss. So, that's not good for us as well here. The trick then that we need to do is to be able to get the EC2 primary to be able to replicate what after we've done the failover and promoted the Kubernetes pod to be primary.

The Redis protocol operates on layer four, right? So, what we need is some routing infrastructure that sits outside the cluster and then routes it to the Kubernetes pods. So, for this, we can use an AWS no network load balancer or an NLB. And then and have the server announce its availability at that NLB IP address and port. Such that when we do a failover, the EC2 primary

now has a way of being able to replicate inside the cluster. Well, that's it then. I mean, we've got and we get our no data loss on rollback. It's a little bit more complicated. But, we're engineers. We like solving problems the right way, don't we? All right. So, now let's take a look at what that looks like on a pod detail level. All right. So, allow me

just to walk through what's going on here. So, we're going to have our empty cluster, and it has a pod in it. That pod is then going to have a service that fronts it. And we're then going to introduce a node port for that service. And in this case, I've just picked 31,000. We're going to tell port 31,000, the node port 31,000, to always route traffic through

to the server container. We then bring in our NLB. We then also add a listener on that NLB to say, all right, port 6380, you are now going to be routing traffic to the target group, which is all of our Kubernetes nodes in our cluster, at port 31,000. And then here, we've got a complete start-to-end flow of traffic being able to reach this Kubernetes pod, or the

container, rather. And then the last thing that we need to do is tell Redis to announce its and the listener port that is going to route traffic through to that through to that container. For Sentinel, we just repeat the process. We add in a new node port of 31,001. listener port of, in this case, 26380. And that is going to route it through to the node port.

And then we just configure Sentinel in the same way that you have to announce yourself at the NLB IP address and the for which you're going to be receiving traffic on. So, for the actual migration itself, we just went for a free phase approach. So, for this, we just tell told our Helm chart that you are in a migration mode where you're going to be setting up.

And this is what it does. It provisions the pods, creates the service as it does before, but there's an NLB that exists as well. And we tell the existing uh we tell the Kubernetes pods that you need to at the EC2 server, right? So, that's all set up now. For the migration itself, we used a script to manage this for us, right? Where we would perform some

health checks, making sure that everything is like exactly what we expected it to be. Number of replicas that are healthy, number of Sentinels, and also that the config is the same between the two. Because when you're re-platforming, you want the only thing to have changed between those two to be the actual platform in itself, So, once those health checks were all done and we're happy, we would

then execute the migration itself. And there's a color change that went from left to right, and the server has now announced itself as the primary. And what we're doing here is we want the Kubernetes pod that sits in the same zone as the existing EC2 primary to become primary itself. Because at Redis, we use a lot of AZ affinity so that we can reduce our cross-AZ data

transfer fees. Okay, so we're now running on Kubernetes at this point. Uh so, now all the traffic is now replicating through the NLB because it's announcing its its availability at the NLB IP address. Now, we don't want things to be running through the NLB indefinitely, right? We want to be able to go back to the end state of that uh of the Kubernetes topology that I showed

you earlier. So, we need to tell the Kubernetes pods that you now need to announce yourself at the internal service cluster IP. But at at that point, nothing is going to be able to communicate with that So, the EC2 instances, they go they go bye-bye. We say ta-da, see you later. Then, we reconfigure the pods such that they announce their availability at the internal cluster IP. And

then at this point, nothing is going through the NLB. So, we say ta-da to that as well. See you later. And that's our migration complete of how we migrate one shard. But, there were some things along the way that we didn't expect. And I've got too many to list, uh so feel free to ask me about them more at the end, but I just want to highlight

two that we kind of came across. The first was that we did some analysis into how much it would cost for us to uh data through the NLB indefinitely. And we looked at what the data streams were, how much data was being written, how much was being uh read, how much was being replicated. And with the amount of replicas that we have in this migration setup, we

looked at Prometheus for how much data was being traversing through these servers, and we estimated that we would have over a hundred thousand dollars a month in NLB data transfer fees just on that. So, this told us that we just had to wait until everything moved over for us to remove the NLB and advertise at the The second one is that, okay, so we had when we're

in the migration state, we've got the three Sentinels that are running on EC2 on a cluster level, right? And then we've also got the three Sentinels that are on the pod level. And now these are running on different platforms. So, what happens if there's a connectivity cut between those two? Those three Sentinels are then going to decide, oh, I can't access the the primary. I now need

to invoke a failover. So, you're then going to have two primaries up within the same shard. I mean, this is your classic split brain scenario, right? Where you're always going to get data loss in that scenario. we just added a seventh Sentinel. This would exist on the EC2 side, and it would only be up for the duration of the And then we just updated the quorum Sentinel

to be five, so that there would always be an absolute majority of agreeing on when uh when the primary's gone down. If it so happened that it was at a time when we weren't monitoring it. Well, the migration ran between February of 20 uh 2023 through to May of 2024. As I said, we did nearly 300 shards all in one go across all of our seven clusters,

and we did all of that without any data loss, which was just absolutely incredible for us to do a complete re-platforming and our customers not even notice at all. Now, at KubeCon Paris a couple of years ago, some of you might remember that Redis announced its license change during that conference, right? I mean, I was there. And the that same day, we were on the shop on

the shop floor talking like what's going to happen with Redis now? Like this is another rug pull. There are a number of companies that they decided right, we want to still keep Redis open. So, they forked Redis to create Valkey. Braze wasn't directly impacted by the license change. But, we wanted to continue going where the community was going. So, we did some benchmarks and we thought, all

right, this looks pretty good. Why don't we put to the test just how fast we can move now we're on So, we first looked at what we needed to So, our helm chart before was hard coded to use the startup command for the server container to be a Redis server. Right? Now, in Valkey, this changes to Valkey server. So, we just had to make uh like a

small little templating change to say what image are you deploying? Is it going to be Valkey or is it going to be Redis? And you it would use that so that it would then dynamically set what the startup command is. So, that was the first change we had to And the second change, right, get ready for it cuz it's a lot. All right, I lied. It wasn't

a lot. It was literally just these two lines that we had to change for us to migrate. We just had to change the image name and the version that we would that we wanted to deploy as well, right? It really was that easy. And what made it even easier was the fact that Valkey is completely backwards compatible with the Redis configuration as well. Right? We didn't have

to change a thing. We all we had to do was say, right, we're going to be migrating these shards on this day and that was it. So, for the actual migration in itself, Braze had grown a little bit since our Kubernetes migration. So, we did 350 shards to migrate across our 10 clusters and we did all of that in just 6 weeks. All thanks to the velocity

that Kubernetes allowed us to move at, which was just absolutely incredible and such a good thing to see. But, what about the performance? What did we see when we were on there? Well, this is the headline figure. On one type in one of our busiest we saw a 90% reduction in P95 latency just from the migration. And I'm going to remind you again, we didn't have to

change anything on the config. This is on a like-for-like configuration. I mean, this is incredible. that's the headline figure. That's like the best case. What did we just see on average? Well, for that, we had across all of our types, across all of our clusters, we had an average P95 latency improvement of 15%. Right? And I mean, our tail end latency just pretty much just dropped because

of But, I mean, that's the that's the 5%. What about the majority? What about the 50%? Well, you can see as I'm I mean, modest gains, 5%. Okay, all right, I'll give you that. But, remember the scale that Braze runs at is in the billions. In fact, I think even this we had just recently had our annual earnings announcement a couple of days ago. We're running in

the trillions now. Right? So, that 5% increase is multiplied over and over and over again. And it creates such a such a much better experience for our customers. All right, so we replatformed to Kubernetes and we migrated to Valkey. Well, what's next? Well, we've we're never done with improving the state of Valkey at Braze, right? We've been running on Kubernetes for a couple of years now. So,

there's a few learnings which I'm going to just take away and show you the kind of problems that we're facing now what now that we're on it Kubernetes. The first is that we need to identify a way to better distribute pods on on our nodes such that we don't have them all cramming all the primaries cramming together on one node. You know, we're then going to run

into hot nodes problems. Like we've got uneven CPU utilization. We're getting unequal bandwidth being used across the two of Such that this hot node at the top, that would be experiencing AWS uh network No, what what's it called? Sorry. The bandwidth allowance exceeds, I mean, these are problems that we can solve with just adding more capacity, right? But, that costs money and we want to do solve

this in a much more intelligent So, we want to be able to define rules such that we can say, okay, I want no more than two primaries of a psychic shard existing on the same node such that we get nice even distribution. And it kind of helps us to reduce the amount of of troubleshooting that we have to do to be able to investigate really what's going

on. The secondly is that Braze is always scaling in and out all the time, up and down. I probably the whole time that I've been speaking to you today, we've scaled in and out probably 100 times. And I would love to see a world where we can get our Valkey databases to be with that workload as well. We're kind of running things static. And there's been huge

improvements in this over over the last few months. Like the with the vertical pod auto scaler, I think has uh taken on the in-place pod resizer to now be graduated to stable in the last Kubernetes version. And such that if the node has capacity and we need more memory, we can just add more resources to the container without us having to recreate the pod. So, that'd be

really good to see. But, this is the puppy right here. We want to create a much more easier way to be able to run databases on Kubernetes, right? Of Valkey in particular. So, a group of us all got together and we said we need to really create a community-led Valkey operator. And so, this is it here. We are working across a whole number of different companies and

we really want this to be a best-in-class operator for running Valkey on Kubernetes. And it's in development today. The talk before already gave me a little bit of a warm up to it. But, here it is. We would love to have as many people as we can get to come and contribute and let us know your ideas for how you want to deploy it out. And when

it's ready for production, we're going to aim to support cluster mode, standalone, replication, whether that's sentinel or not sentinel, and even the concept of cells, which is pretty much the topology that I've described to you today. And we've got exciting things on the road map as well. We want to put in auto scaling, custom resharding, plugins. But, we want to hear from you and your ideas. So,

the QR code will take you through to the GitHub repo. Please go in there, have a look, and let us know your ideas that you might have. Well, that's it for me today. Uh if you've if you're interested in helping Braze solve these kind of or if you've got any feedback for me for this talk, you can find those QR QR codes on the screen behind me.

My name again is Joe Haben. You can find me on all the socials at jdehaben and have a fantastic last day of KubeCon. Thank you.