Vitess: More Data, No Problems - Matt Lord & Rohit Nayak, PlanetScale

This talk focuses on Vitess, a database clustering system designed to scale MySQL databases effectively. The speaker explains how Vitess allows applications to transition from a simple single-server architecture to a partitioned, sharded environment that can support massive data loads without compromising user experience. They discuss key concepts like keyspaces, shards, and how Vitess retains a MySQL-like interface for users even as the underlying architecture grows complex. The presentation also covers important features such as automated recovery, query throttling, load balancing, and health checks that enable stability and resilience in large-scale implementations. Through examples from major tech companies, the speaker illustrates how Vitess addresses the operational challenges faced during data migrations and scaling. Overall, Vitess is presented as an essential solution for businesses looking to maintain performance while managing increasing data demands.

All right, well, hello, hello everyone. Thanks for coming. So, today, myself and Rohit are going to be talking about Vitess and how it allows you to scale up your data layer to as you gain more and more data to not have more and more problems. So, we both work at PlanetScale. Um we're both maintainers for the Vitess project, so that's why we're here. And I'm going to

start with just an overview of Vitess, what it is, how it works, why it is hopefully relevant to you, certainly if you are a current or potential future user of MySQL, uh you have a successful business, eventually you're probably going to bump into some scaling challenges. So, the basic value proposition for Vitess is that it allows you to go from where you started, which is probably on

your laptop uh with a single MySQL server. Eventually, you your application goes in production and you have a single MySQL server for the most part, certainly a single one that just takes all of your rights. And that may be an RDS instance, for example. Um so, it's a pretty straightforward interface there for your for your internal or external users, the application developers, there's a single MySQL server,

very standard, pretty straightforward, something they've probably are very familiar with. But once you need get to a certain size, you realize you're going to need to start to figure out ways to partition that data because you can't scale up a single instance anymore. So, Vitess makes it it gives you a lot has a lot of features that we're going to touch on, which make it easy to

do that. But the other key part of it is for your users, internal and or external, for them it still looks like a single MySQL server. So, you don't lose that key property for your internal or external users. And Vitess is is very battle-tested. It's used by a lot of the largest web properties out there today, so if you So, you're probably interacting with Vitess indirectly on

a daily basis, whether you realize it or not. So, whenever you send a Slack message, you know, when you wake up in the morning, for example, that actually gets stored in Vitess. When you open up your GitHub PR or you comment on somebody else's PR, that's actually stored in Vitess, as well. Similarly, if you stop and get a coffee, they have a Square terminal, you call an

Uber, these are all These are all [clears throat] companies that are leveraging Vitess for their data layer. And so, we're going to talk about some of the reasons why they do that and how Vitess makes it pretty easy uh to do that with a relatively small team. So, just some basic concepts. So, when you have a single MySQL server, you have a database or a schema, the

MySQL system database, you might have your customer database, for example. A key space takes that logical concept one level higher because when you use Vitess, that particular uh database is going to be partitioned across, you know, hundreds or even thousands of nodes. So, the key space is just that logical concept of a database spread that's managed by a bunch of different instances. And each of these subsets

or shards of your data is managed by a MySQL replication group, so a typical primary uh one or more replicas, and that's what we call a shard. That's just a partition or a subset of your data. Vschema and Vindexes are Vitess-specific things that allow you to define how that data is distributed. So, that the query layer knows exactly where each row um is supposed to live. And

here's You can see the basic architecture. So, remember that for your application servers and the developers of those applications, it's just a it's a just a standard MySQL server as far as they're concerned. So, first it would typically go through a load balancer, so you might have something like an ELB, which you have in different geographical, excuse me, geographical regions. And then you can have these VT

gates, which are the stateless uh MySQL protocol servers. So, those are the ones that actually receive the uh select star from customer where ID equals one, for example. And you'll have those typically in the same geographic region that you have the ELBs. So, your application traffic comes in, it it would go to a VT gate pool, which also lives in that same geographical region, and those VT

gates, using the the Vschema, the Vindexes that we just talked about, can know, okay, this particular customer record, you know, that customer table is sharded on the ID field, it knows how to build a hash for that ID of one, and it knows, okay, that's actually on shard three. And as I talked about, the shard is just a subset of your your data for a given key

space, which remember is a logical database. And this just a standard MySQL replication group, so you have a primary and typically two or more replicas. We're going to touch on some of these features. We're not going to talk so much about the MySQL compat- compatibility part, although just remember that's of course a key part because it allows you to scale your data layer virtually infinitely while not

losing that key property that you have for your internal external users, that for them it's just just the same old standard MySQL server that they're used to. These are just some examples. This is mostly for people that that download the slides. You can follow these links. What these are blog posts that talk some of the usage of Vitess at at pretty large scale by a number of

these uh users. This just gives you a little bit more kind of a sense of the scale of some of the larger Vitess clusters. And I think it's important to note that all of these customers that we've just talked about or such as the users, some of them are customers as well of PlanetScale, but these are all Vitess users. Vitess clusters being managed by a surprisingly small

uh team. So, we're going to talk about some of the design principles that have gone into Vitess, but a big overarching theme there um automation, right? So, being able to manage this massive data layer, which can consist of hundreds or thousands of nodes, being able to do that with a small team. So, of course, that's going to require uh a lot of resilient features, a lot of

automation. Because when you get at a certain scale, you're going to bump up into a number of So, certainly if you're on a single a single server, an RDS instance, and let's say you have uh 500 terabytes of data, various things become quite challenging, from schema changes to being able to even just backups and restores, the personnel costs can go up per server just because you end

up with so many different kind of operational challenges that you're going to face on a regular basis. So, now I'm going to talk about basically what what would happen when you get to the point where, okay, we realize, hey, this single RDS instance it just is it's it's becoming problematic. And that's when you would look at Vitess as a a means to then partition that particular instance

into some number of shards. So, when you are importing into Vitess, you can also shard at the same time. So, remember that you you probably got to this point because if you're not already having problems, you you see them on the horizon. And so, you probably do want to shard uh your data when you're importing into So, you can start with two or four or eight or

however many shards you think make sense. We have a rough guidance of like, you know, up to 250 gig uh per So, just just as a rule of thumb, if you if you want to start there. But the move tables command allows you to do that. And all of these commands that we're we're talking about allow you to do these perform these data migrations uh without downtime.

And to be able to to cut traffic over and reverse it if you see a problem. So, you may cut traffic over, so now all of your application traffic is actually no longer talking to the RDS instance, it's talking to these to the target shards. So, a reverse replication stream is set up automatically so that you can switch back if once you do that, you realize, hey,

some of our common queries just got like 10 times slower uh or some some queries start throwing errors. So, these all all of these workflows allow you to to perform these pretty complex data migrations in a very easy way from a user's perspective and without incurring downtime, gives you a lot of flexibility around how how long you want to execute this for and building up a comfort

level um that everything has gone well. And we'll talk about VDiff, which gives you even more uh confidence. Once you've So, you may have started from, hey, we've got this single uh RDS instance, we've sharded it into, let's say, four shards, but we also realized that we've got some discrete subsets of tables. So, maybe we have uh customer-related tables and we have product uh related tables. You

can then do vertical or functional sharding to split those particular tables off uh into its own key space and that you again you can use the move tables command for but then at some point you realize okay products table our key space rather that needs to be we need to split that up as well so we've got some shards and it could be just one shard that's

become particularly large or particularly hot or it could be both of them so with the re-shard command you can then partition that particular those the data in a different way so you can go from two to four shards four to eight or you could just take one particular hot shard and split that. The V diff command is one of the the key things that gives you confidence

in in doing these things so remember that these you're kicking off this background data migration you still have to serve production traffic right you don't want to impact production traffic as that's going on so Rohit's going to touch on some features that [clears throat] allow that to be done without having a negative impact on your actual production traffic. So these things are happening in the background they

take a while of course if you've got you know petabytes of data for example it's going to take a while to move these things around. Once we've reached a certain point where we've done the initial copy then we could we continue to replicate changes so that we can live in this state where we have this data migration for however or you can run a V diff command

which will then orchestrate [clears throat] consistent snapshot using GTIDs on the source and target and actually allow you to compare all of the rows in the tables that you've been migrating and let you know if there are any differences. Materialized views is another key feature that so as you've gone from the single server right you you may have fact tables there like states and country various other

tables that are typically joined by some common queries. The materialize command allows you to maintain materialized view so the results are stored in a table they're kept up to date for and one example is a would be fact tables so let's say you have a country's you you join that quite frequently for your customer records you can have a reference table which is just a a specific

type of materialized so that that country's table exists on every shard so you can claw back some of that data locality that you naturally use or lose rather as you partition your or shard your data. Lastly change data capture this is kind of table stakes I would say for for any sort of data team or data layer today virtually everyone has some sort of data warehouse or

analytics system where you exfiltrate not shouldn't say exfiltrate but you copy data regularly out from your transactional system that your users and your application developers are working with off to some separate systems that could be big query or big table for example or red shift which then allows other people in the company to try to gain business insights from that data without impacting or interfering with your

application So we have the V stream API which if you've if you've been using my SQL you may have been using something like the BC and for example and that is actually one of the ways that you can leverage this V stream API there's a the BC and connector for the test but it allows you to then >> [clears throat] >> again just outside perspective to have

a same the same kind of interface that you would for CDC as if it was a single my SQL server. The VT gate V stream will ensure that it's pulling in records from all the shards in a given key space for example. And with that I'm going to hand it over to Rohit to talk a bit more about scaling features. Thanks Matt. So what I'm going to

talk about now are the principles in Vitess so you've seen that a lot of critical infrastructure is powered by Vitess so how do we achieve the scaling and the resilience right? So these are standard principles of distributed computing that Vitess we think has executed well and one of the nice things is stuff that also works for resilience also works for scaling so many of these properties some

of them are only for resilience some for only for performance but usually they're the same. So isolation is to ensure that failure of certain components do not bring down your system and you usually do that with stateless or shared nothing architectures. I'll be giving a few examples within Vitess of what these mean. Of course you want to keep multiple copies of data over provisioning your systems as

load increases or to take care of failures. Now if components fail you can either bring down your system entirely gracefully degrade. So what that means is that you provide higher latency to the users it continues to perform. Of course there are limits there right you don't want to have your PR take five minutes to load or a slack message one minute for to be sent or if

you're on a shopping site you want to continue you know things will start timing out. So at that point you bring in circuit breakers which essentially you can do one of two things one is throttle lower perform lower less priority items like in a shopping site whoever has come in you make sure that they can continue performing don't allow new users for a while. Automated recovery is

very key because there is no time so when you're talking of four five six nines it's like seconds of downtime per day or minutes in a year right? So you really don't have time for a human being to go in and do any forensics unless it's some catastrophic thing that happens. So self healing is important auto we'll again see how some of these things are done. Observability

and testing are obvious things it seems but they're really hard to do at scale in distributed systems. I'm not going to go over those today. So isolation is achieved by having failure domains which we call cells. A few months ago they were US East one went down for several hours but many of the Vitess sites continued to perform because they were spread across the other several availability

zones. For extremely critical stuff like health care infrastructure or nation wide payment terminals you can also have your AZs across clouds. There not too many of them because of cost and complexity involved but there are customers who have some on AWS and some on either GCP or Azure. So even an entire cloud going down means you continue to perform. So the query serving layer VT gates are

stateless they do not depend on anything else of course the database has to be up the state which is in the topo is not needed on a continuous basis. So the topo goes down for short periods of time you continue to serve queries right? You might have a little stale state. one thing I realized when I was preparing for this is we keep talking of failures you

know as if it's happening all the time. The important thing here to note is you cannot when you want five six nines you want to pick up so these are failures that happen very rarely for very short periods of time. Like if something is done for a few seconds you know the CEO gets paged within Planet scale for example right? So it's it's taken very so we

need a lot of design and coding to ensure that that doesn't happen. Sharding which is intrinsic in Vitess very key to isolation as well. shards think that they have the entire data actually so they do not know they're not aware of other shards. It's the query serving layer that will route it to the correct shard or aggregate from different shards. A key thing that the Vitess designers

initial architects planned was not to build our own storage layer. So that you see a lot of other distributed systems just build their own storage layer. The aim was that we use a proven robust my SQL itself directly. That also helps us with compatibility significantly gives us the power of my SQL replication the durable rights that are provided semi sync through group replication. Sharding is automatically builds

in redundancy. Let's say you have 100 shards one of them goes down only 1% of your customers are affected. These are some interesting techniques So I think Matt touched upon it. Let's say you have a concert and uh it opens up. And everybody wants to buy tickets. There are tens of thousands of queries coming in. Many of them are read queries to give information. So, what is

there at the VT tablet is it looks at the fingerprints of the queries. It sees that a a herd of queries has come in. It only passes one through. It holds the others and passes on the result of one to everyone else. So, that's a significant benefit. Instead of 10,000 database queries, you only have one. So, you don't need to over provision. You can handle load more

easily. Connections are expensive to set up and in databases. So, there are various different connection pools possible in VT tablet to So, you can just reuse connections. Because of cells, you can also if domains fail, you can send reads into the local cell. So, essentially you're trying to reduce pressure uh the system by using these techniques, but continue to perform at scale. Back pressure is when you

notice failures or sudden surge in performance you start throttling less important things. Like we talked about migrations. So, we're constantly large systems are running online schema changes. Uh sometimes there'll be re-shards running to bring things up or down. Those are not time sensitive. there's a throttling mechanism. Uh we'll see how that's implemented uh soon. Query throttling, you can have priorities. You can send query commands saying, you

know, uh different priorities. Uber uses that. Uh They've actually built the a new query throttler for this. Because there are different types of apps using the database. So, you want to make sure that uh the critical ones uh uh in case of extreme load. Uh load shedding both time out the connection levels, of course, but also if there's a bad query that returns a huge number of

rows, for example, 10,000. You can configure all of this. And uh you can just return errors to that user. And people fix those their bad queries without affecting the rest of the system. Now, when there are failures or when you let's say you have a re-shard and you're moving from the old configuration to the new configuration. For a short time, you have to stop writes because you

need to synchronize uh the two sides. So, at that time, VTGate, the query serving layer, can see these uh cluster events and starts buffering. And when the event is done, it executes those queries. So, these are transparent to the end user. You will see a small short blip in latency, but there's no hard down time. You can do replica reads in Vitess where you can say, uh

I'm okay with a slightly stale. When we say stale, we talk about like milliseconds, hundreds of milliseconds. So, if a replica is lagging more than 100 milliseconds, don't consider it send maybe send it back to the primary. how do we do self-healing? Right? continuous health checks that are run the VT tablets, the database are sending back health information using pub/sub. VT Orc is an orchestrator that also

is run in a distributed way, which is constantly monitoring the cluster health. And whenever it sees a failure, it can let's say a primary goes down for some reason. Within seconds, it can bring back another one. And how does it bring it back? Backup restores. You know, some of these sort of boring old operational things are extremely critical at scale. We have fast backups, uh incremental backups.

So, some of the more critical systems have like 5 minutes every 5 minutes you have a backup so that at any given time your data on the disk backed up is not more than 5 minutes old. You can bring back a new server from the backup, catch up with an existing replica, and within seconds you're serving. Smaller shard sizes help. Sharding is a key thing here. I

mean, if you have a 20 terabyte database, it's going to take hours for uh restoring, right? So, typically you have 500 GB to 2 terabytes per shard. And that also helps with backup and restore. So, you see that all of these things are interconnected uh in the architecture. When something fails, replace the nodes. Okay, you don't spend time trying to figure out what it is. Don't try

to reboot your MySQL. We have found that sometimes you think that just restart MySQL things will work, but it's faster overall and more reliable resilient and reliable to start from scratch. Just bring up a new server. And then you move uh the other one into forensics, what we call the drain mode, so that people can look at it, but all that is done offline. Um I won't

talk about the first two, but updates for those who run systems at scale, you realize you always want to uh when there's a CV, you want to fix it immediately. you have lots of different packages. They keep coming up with newer versions. You want to stay on the latest version for performance or for security or whatever. The application itself. You'll have minor versions, major versions. You want

to update. All of that happens with zero down time in Vitess. So, the way it happens is So, for query serving layers, typically VTGates you'd run like thousands of VTGates behind an NLB. So, you can of course bring some down, bring newer ones up with the new For the database you upgrade the replicas first. Then you promote one to be primary. Again, that is zero down time

because of so, you don't have any down time for it. Easier said than done. I'm not sure if anybody here who's maintaining that, but you'll realize how tough this is. And this is one of the reasons many people like So, there are different reasons people come to Vitess initially, and that's one of the reasons. there are a lot more things that are there, but uh I just

want to summarize performance and resilience has come in from very deliberate design decisions right at the beginning of the Vitess project. Of course, we have uh improved it significantly over the years, added new some components, modified stuff, the key things still stay. The use of MySQL as the storage layer, using VTGate, stateless VTGate for routing, sharding, the sharding there's a lot of complex support for sharding. We

do not do auto sharding, for example, because in many most cases auto sharding doesn't work, right? When you have a real serious system Okay, I'm I'm being unfair, but many systems can work with like document databases, maybe, but a lot of people come to us because they can actually configure like the SREs can configure their sharding schemes. And that helps both for resilience and performance. Intense focus

on backups, restores, a lot of those follow these principles. So, the mental model of the principles you can use to map what we have built. Uh but of course, it's iterative. Um I'll stop here. Uh there's information here. There are more stuff on the slides you'll find online that we spoke today because of time. uh we have a Vitess booth. There are some few maintainers there. There's

a PlanetScale booth as well uh has been very successfully deploying Vitess. If you have any questions uh stuff.