Project Lightning Talk: Run Your Data Layer In K8s With Vitess - Matt Lord, Maintainer

This talk introduces Vitess, a CNCF project that allows users to scale MySQL databases effectively. The speaker, a maintainer at PlanetScale, highlights the importance of Vitess in managing MySQL data layers within Kubernetes environments. Vitess enables the distribution of data across multiple nodes while providing a single-instance facade to users, simplifying operations and scaling challenges. Key features include data sharding, online schema changes, backup and recovery, and automated monitoring through VT Orchestrator. The speaker emphasizes that companies are already using Vitess indirectly and encourages audience members to consider it for their direct database needs, especially as they scale their applications.

Thanks everybody for coming. So, I'm going to be talking about Vitess. And if you have any follow-up questions, we also have a booth um where we will be there for the first half of each day. So, I would encourage you to come by if you have any follow-up questions. Real briefly about myself. So, I I'm a Vitess maintainer. I happen to work at PlanetScale um and that's

why I'm here. Just a quick show of hands, who has in the past or is currently using MySQL as your database? Okay, fair number. So, you're kind of the target audience here. Um for the rest of you who haven't, I I want Vitess to kind of be lodged somewhere in the back of your brain so that in the future, when you come across a case where you

where you're working, you're using MySQL and you're running into some scaling challenges, that you remember, "Hey, Vitess is a thing. It's a CNCF project that's uh allows people to scale MySQL. Maybe we should check that out." Um how many of you people who are have or are using it or actually running MySQL in Kubernetes? Okay, not too many. And that And that's why I wanted to give

this talk cuz it was a bit surprising to me that so many shops that are all in on Kubernetes are running everything in Kubernetes except their database. The it's It was quite common and still is today to have uh an RDS instance that is sitting outside of your cluster rather than running that directly in the cluster. So, just a very quickly, the the value proposition for MySQL

is that it allows you to go for Vitess is that it allows you to go from a single MySQL instance. So, you have this single RDS instance, let's say. And you can scale that out as needed. So, you can end up distributing that data layer across hundreds or even thousands of nodes. But for your users, internal and or external, for them it still appears as as if

it was a single MySQL database instance. So, some of the key features that it provides, of course, the compatibility is what gives you that facade to your users so that it looks just like it was a single MySQL server. But, for you as an operator, it provides a whole suite of things that allows you to manage this large data layer. So, you can shard and re-shard your

data. You can add materializations so you can gain back some of that data locality that you lose when you start distributing your data. So, for example, you can have a fact table which exists on all the shards. Uh cluster management, of course, is a big one as you get into this more complex setup. Uh so, it has a lot of features for that and has things to

do allows you to easily do things like online schema changes. Remember, that has to be coordinated across, you know, Uh backup and recovery, and then things that also allow you to handle unpredictable load. So, being able to shed load as necessary, apply back pressure to certain things. Uh query consolidation is another one that if say a really hot row and a million people come in and request

the same row, uh new Taylor Swift tweet or whatever the case may be, um it will only have to query the database once and then all of the clients can get that same um result. Uh you're probably already using Vitess indirectly uh a lot of the times throughout your day. So, whenever you send a Slack message, that's going uh through or getting stored in Vitess. Similarly, if

you open a PR, you add a comment on a PR, all of that gets stored in Vitess. Uh if you go and get a coffee and they have a Square terminal, Vitess. Uh if you buy something at a Shopify e-commerce site, Vitess. Call an Uber, again, Vitess. So, my goal is that hopefully in the future you're not only using it indirectly, but using it directly as well.

So, the the automated recovery is the single biggest thing that you're going to worry about as you scale your data layer. And so, there's two main components that really help with that. So, first is VT Orchestrator or the orchestration component. And that will monitor all of the nodes in the cluster and ensure that everything is is healthy. So, if it sees an issue that needs to be

addressed, it will go ahead and address that. And that's combined with the operator, which will of course manage the infrastructure part of that in Kubernetes as an extension of that basic process. There are a number of design features that also make it ideal for scaling out your data layer. So, handling failovers gracefully is is an example of degraded graceful degradation. So, your users will see a slightly

elevated period of latency rather than a bunch of errors. I'm running a little low on time, so I'll skip over this, but this just gives you a sense of a lot of the features that allow you to scale your data layer without having to scale your data team. So, all those customers are users of the test that I mentioned before, they'll have a very surprisingly small data

team that's managing that. Here you can see just an example of installing the operator. It has its own repo, but I just used the the Vitess repo for this example. You can see what an actual Vitess cluster CRD looks like. And then you can have it installed, and then just a couple minutes later you have a fully functioning Vitess cluster that then you can play around with

and see how it works. And that's it. Thank you. Awesome. Thank you very much, Sugu.