За тази лекция
This talk delves into the development of PlanetScale's Postgres as a service, discussing the complexities encountered in achieving reliability and performance. The speaker explains the importance of durability, particularly focusing on synchronous replication and its implementation through Patroni, which automates standby management. Challenges are presented regarding the potential loss of durability due to user actions, such as transaction cancellations. The speaker emphasizes the significance of efficient wall archiving using tools like wallG, while also outlining a robust backup validation process. Additional strategies to enhance availability and mitigate user-induced downtime are shared, showcasing how PlanetScale aims to deliver a dependable database service.
Пълен транскрипт
All right. Last year in April, Planet Scale decided to build a Postgress as a service with managed replication backups and failovers. Our goal was to make the service as fast and reliable as we possibly could. And we wanted to launch at the end of September. To accelerate the pace of our development, we adopted a bunch of open source tools. And along the way we discovered that as
well as these open source tools faults and design choices which are at times at odds with these reliability performance objectives. At other times these objectives have been difficult to reconcile with the needs and expectations of users. I'm Max. I'm a software engineer at Planet Scale. When I joined, we were mainly known as the company that builds and operates Vatess, which is a system for horizontally scaling my
SQL. Planet scale's largest customers run on Vess and their largest databases have hundreds of terabytes of data with hundreds of shards serving millions of queries at P99 at uh P99 latencies as low as 5 milliseconds. There been of interest uh in planet scale from the postgress community but I don't think we ever took it seriously until after we launched planet scale metal. Metal allows our vitess customers
to run their databases on locally attached NVME drives which is much faster than the common alternative network attached storage. The surprise for us was that after the launch of metal, most of the interest in metal was coming not from my SQL users but rather from Postgress users even though we didn't have a Postgress product. So at that point we decided we couldn't ignore the signs any longer
and got to work building planet scale for with the eventual goal of bringing the performance of metal to the Postgress as a service market. Something I found interesting working at planet scale is that when users are comparing managed database services, they tend are to look at availability, at performance, any number of features that they may care about, but they tend to take durability for granted. And that's
fine for uh people who are shopping around for a database service. But if you're building a database service, you absolutely cannot take durability for granted. It's got to be honestly the first and most important thing you get right. For planets, we consider the foundation of durability to be replication with synchronous commits. We allow users to run their database service with uh their planet scale postgress cluster with
just a primary and no standbys. And I think of that as yolo mode. But if users want to be covered for um service level agreement, they have to run in our default configuration which is a primary and two or more standbys connected with replication and synchronous commits. This guarantees or the invariant that this that you're trying to preserve here with synchronous commits. Is my microphone cutting in
and out or is it okay? >> Yeah. Lower. It's testing. Okay. Um the invariant that we're trying to provide with synchronous commits or preserve with synchronous commits is that anytime before a transaction completes on a primary, it's been flushed to at least one standby. This guarantees that if the primary crashes irreoverably, that all the copies that were on the completed on the primary, all the transactions that
were completed on the primary prior to the crash are also present on at least one standby. Setting up synchronous commits is really just a matter of populating synchronous standby names. And if you're operating a database as a service, you have to do this dynamically because users will request to add or remove standbys or hardware failures will force you to rebuild standbys from the ground up. And if
you're operating uh tens or tens of thousands of clusters, you don't want to be doing this manually. Postcrest won't do it for you. So you need to find a tool that can automate it for you. And for this purpose, we adopted Patroni. Patroni does a lot more than managing just synchronous replication, but it also does this very well. I highly recommend it as a tool. When a
standby joins a Patron cluster, Patron will add that standby's name to synchronous standby names for you and it will create a replication slot if you're interested in using uh replication slots. If the standby goes offline, Patron does the opposite steps. it will remove the standby from synchronous standby names and it will start a timer and once that timer elapses it will drop the replication slot. This timer
is governed by governed by a patron setting called member slots TTL which defaults to 30 minutes. This is no no doubt a fine default in many cases but we ran into some edge cases that made us question whether this setting is suitable for a database as a service context. The first issue we ran into is that Patroni will leave behind stale replication slots when a standby is
terminated. To see how this can become a problem, consider a user who has seven standbys, three logical replication clients which are fully occupying all of the replication slots on the primary. The user one day decides they want to operate in YOLO mode after all. So they terminate all of their standbys. A second later, they regret this and bring back two standbys. Even though we can create the
standbys for the user, those standbys are not able to get a logical are not able to get a physical replication slot because all the replication slots are still occupied by the logical replication clients as well as the stale slots that Patron is waiting to clean up. You could just choose not to use replication slots and that's a perfectly valid way to run Postgress. Uh it has pros
and cons. Don't have time to get into them. Uh but let's say you're committed to using uh physical replication slots with replication. In that case, you're going to want to come up with some kind of answer to this problem. You could say to your users, you cannot set max replication slots below some minimum threshold. That gives you some headroom to sort of sweep this problem under the
rug. And I think that's a perfectly fine option if your users let you get away with it. Some users may object on the basis that they want to set max replication slots as low as they possibly can. They may be concerned that a connection storm from their own logical replication systems could take advantage of the excess slot capacity and overwhelm their primary and steal precious connections from
other applications since connections are precious commodity in Postgress. Another option you can consider is simply decrease member slots TTL from 30 minutes to something much lower like 3 minutes or 5 minutes. If you do that, you'll run into another obstacle as we quickly discovered which is that Patron will prematurely drop replication slots if when a standby goes offline. So let's say there's been a network partition. Now
the standby appears offline to the other Patroni members. If it stays offline for if there's a network partition persists beyond a few minutes, let's say longer than member slots, then Patroni is going to drop that standby slot. So that when this standby comes back online, it's not going to be able to resume consuming from the live wall stream at which point it's not participating in synchronous replication
at which point it's not eligible as a failover candidate and really isn't adding any durability to the cluster. One way you could adjust this is by having a fallback wall shipping so that when the uh standby slot is dropped, it falls back to recovering walls from archive and when it reaches the end of that archive can resume consuming from the primary with a fresh slot. This is
a good option as a fallback, but I don't know that I would rely on it because if there's any delta between the rate of wall archival and the rate of wall recovery, the standby is going to fall further and further behind and uh extend the threat to durability while it's not The excuse me, the conclusion we eventually reached is that member/ctl is not appropriate in a database
as a service context. It may make a lot of sense for Patroni as a standalone system because Patroni cannot distinguish between a standby that has gone offline temporarily from one that's been permanently deleted. As a database as a service, we are aware because users inform us through UI clicks and API calls whether they want to terminate a standby or take one offline temporarily. For example, to upgrade
it from a 4 CPU cluster to an 8CPU cluster. And all that's left for us to do is connect this knowledge to Patroni which we ended up doing through Patron slots parameter in which you can tell Patroni the names of all your instances in your Postgress cluster as opposed to it discovering them automatically for you. When you do this, Patroni will create slots for your standbys, but
it will never drop them no matter how long the standbys go offline for. And all that's left for us to do is take responsibility for dropping slots, which we do um with our Kubernetes operator. Just add a few lines of code there, but you could put it anywhere. So now when a standby goes offline, Patron does nothing, which is exactly what we want. And when the standby
comes back online, it's able to resume consuming from the live primary wall stream. And if a user signals to us that they want to terminate a standby, we go ahead and do that with our Kubernetes operator and then drop it slot. So now we solve both problems. We no longer have stale slots accumulating, potentially blocking cluster operations and we are able to um not worry about uh
standby slots getting prematurely dropped by Patron. Even if you tune Patron exactly to your satisfaction, there are other threats to synchronous replication. And the one that surprised me most shocked me is that Postgress will uh allows users to opt out of synchronous replication simply by canceling an in-flight transaction. So if a user has a client application owned by a user has sent a commit to the primary
and the primary is waiting for an acknowledgement from a standby for from any standby and that act is not forthcoming for any reason the client application may have a client side timeout after which they may send a cancellation message to the primary saying I want to cancel this The primary is not actually going to cancel the transaction. It's already been committed on the primary. So instead, what
the primary is going to do is simply stop waiting for the standby to acknowledge the transaction and complete the transaction. This violates our invariant that we're trying to preserve with synchronous replication. And the consequence is that the client application can now query and update this committed data on the primary, potentially showing their users this data on an orders page. And if the primary crashes irreoverably and we
have to promote a standby, when the user tries to reload their page and see their orders, it's going to be gone because this transactional data never made it to a standby. So in other words, the client has observed data loss. If users come to you angry that their data has been lost and you tell them it's because they canled their transactions and that they shouldn't do this
anymore, they're going to laugh at you because it's not a serious answer. Postgress users who are shopping for who who pay for a Postgress service most of them nobody in this room I'm sure but most of them have no idea what synchronous commits even are letting alone that cancelelling them is going to opt them out of the durability guarantees that you're trying to provide them as a
database as a service and that's not a knock on Postgress on people who are shopping for a Postgress service that's just you know they're they're paying you to be the expert not them I think a much more serious answer is to take the patch on this mailing list this mailing thread and apply to an internal build which is admittedly a pain to have to run your own
internal builds but I think if you are aiming to build a durable by default Postgress service this is what you have to do even if you apply that patch there are other ways that users can threaten the durability of their own clusters one way they can do this is by through questionable choices in invalidate the slots of their own standbys which they can do for example by
setting max slot wall se keep size to a ludicrously ludicrously low value and then running a command like an index creation that generates a ton of walls at once which can't be synchronized to standby quickly enough resulting causing the primary to invalidate their slots. At this point you could rebuild the standby from backup but that's slow and expensive. A better option is now to rely on wall
shipping where you configure the primary to ship walls to some wall archival medium and configure standbys to restore from them. For this purpose, we reached for a tool called wallg which lets you which manages well it gives you um a nice API for uh managing your walls. It integrates with uh Google uh sorry AWS S3 and Google uh the Google equivalent Google cloud storage and it also
does backups and we use it for that purpose as well. The challenge with wall shipping in general is that uh the primary and this primary invokes archive command um sequentially. So it won't start one archive command until the previous one finishes. Which means that if you couple a slow process to archive command, your rate of wall archival is going to be slow. Same applies to restore command.
And if your rate of wall archival is slow, then you can easily have your rate of wall generation outpace it, resulting in unarchived walls accumulated on your primary, which threatens the disk uh health of the primary and uh also means that standbys can't recover uh all those walls because they haven't been archived yet. We ran into this problem when we initially configured WG in a naive fashion
by having archive command invoke it directly which meant that with every invocation the postgress was spawning a new wallg process which was opening a new connection to S3 and then uploading whatever object it was trying to upload. So we coupled a slow process to archive command and our archive rate was not able to keep up with the rate of wall generation. The more sophisticated way to use
wallg is to use its built-in Damon which runs as a separate process from uh Postgress. And the way this Damon works is it scans for walls in the background that are ready to ship and then uploads them concurrently to whatever your wall archival medium is using a stable persistent connection pool. So that by the time the primary invokes the archive command, the work it's asking the wall
G damon to do has already been done. So this allows us a way to keep up with uh make sure the rate of wall archival keeps up with the rate of wall When we introduce concurrency into the system through the wallg Damon, we now have concurrency problems to think about. And there are many there are different dimensions to uh concurrency tuning. I'll just give one example uh
in the form of wallg upload concurrency variable which governs how many walls wallg damon will upload at once to your object storage uh to your uh wall archival medium at the same time. Its default value is 16 which is totally fine for many workloads but it's not great for all workloads. If your users are paying you to use a single CPU or a fraction of a CPU,
a value of 16 is going to mean WG is doing a bunch of work that is going to be stolen from Postgress. And if you have users paying you for a high number of double-digit CPUs, then this value may not be enough. And if users are generating walls at an exceptionally high pace, uh 16 concurrent uploads isn't going to be The simple but effective solution is to
scale the value of WG upload concurrency up and down with the number of CPUs users are paying you to use. And um you'll need to do this for other dimensions of concurrency um as well as and you'll need to do this on both the archival side and the restore side on Even if you get um wallg tuned exactly right so that the rate of uh wall archival
always matches the rate of wall generation, users may end up corrupting their own data by accident. Uh just they deployed something bad and deleted a bunch of data they didn't mean to. And when they do that, they're going to want to restore from backup. And when they do, it had better work. You're never going to know in advance when users are going to try to restore a
backup. And you're not going to know which backup they're going to try to restore. So you need to have a very high degree of confidence that every backup you take a user is going to be able to And the process we really like at Planet Scale and that I highly recommend if you haven't uh thought of a backup process like this is self- testing backups. The way
this works uh for us is anytime we take a backup we create a backup job which is a process that runs separately from our primary or and our standbys. And the first thing this backup job does is download the most recent backup and then starts a hot standby on top of that backup data. And then that hot standby recovers all the walls taken since that last that
were archived since that last backup was taken. And what we're looking at here is a restore process. This is the same process that we use when a user wants to restore a And once this process is finished, we then use WG to take a new backup and push it up to our archival medium. We run this process twice a day for every backup in our system. Which
means that within 12 hours of I'm sorry, we run database in our system, which means that um for every backup we take, we have validated within 12 hours. And because our restore process is embedded inside of our backup process, we get continuous feedback from production if there's any problem with a backup or with our restore process. And that gives us a users will be able to restore
any backup we take. We've been using this process for years in our vitest product. Adapting it to Postgress had a number of challenges and the one that annoyed me the most personally is that while the backup job is running and the hot standby is inside the backup job is recovering walls, a user might choose to increase max connections on their primary and that may that will that
change will enter the wall record and when the hot standby tries to apply it, it's going to exit with a fatal error. Recovery abboarded because of insufficient efficient parameter settings. Maybe you've encountered this when you've uh tried setting uh your primary to max connections greater than a a standby. We have a hot standby running inside our backup job. So the same constraint applies. That's just how Postgress
works. One way you could address this is by setting max connections on your hot standby inside of your backup job to a value greater than you would ever allow or expect users to set it to. I think that's a perfectly legitimate approach, but it does mean you have to overprovision your backup jobs on memory. And if you're taking tens, hundreds of thousands of backups every day, that
cost adds up to a lot. And you're going to have to either eat that cost or pass it on to your users. A better option is to instead of running a hot standby on your on your backup job to start it as a cold standby, which isn't subject to the same constraints as hot standbys. The problem you'll run into next is that when you try to run
wallg to take up a backup against that cold standby, it'll fail because it's trying to run a query and backup uh cold standbys don't serve queries. But that's easily addressed by configuring your cold standby to exit once it reach reaches a recovery once it reaches a recovery target and then once it exits to restart Postgress as a hot standby and then finally take and upload a One
of the nice things about having invested so much into durability is that we now get some availability properties basically for free. If a primary's availability zone has a experiences a power loss, because we're using Patroni, it's going to promote a standby in a different availability zone. And because we're using synchronous commits, we have a very high degree of confidence that any transaction that was completed on the
old primary is now present on the new If you do nothing else to have a high availability story, you'll be a in a pretty good spot. And if you were to choose one thing to do, I think this should be it. But there's definitely more we can do. What's left are a series of small improvements that marginally improve availability but have a meaningful impact uh on users.
And the two areas I'll talk about are trying to minimize the duration and impact of planned restarts and try to uh mitigate the impact of downtime that users inflict upon themselves. When you're building a push as a service, you're going to eventually ask yourself, how much freedom should we give our users? Should we even allow them to set max slot wall keep size to a ludicrously low
value? For example, in other words, how much rope should you give users to hang themselves with? At Planet Scale, our approach is to give users as much freedom as we possibly can, but to couple that freedom with automatic safety systems that kick in to prevent bad situations from becoming total disasters. One way that users inflict damage upon themselves is by causing Postgress to run out of memory.
If you're running Postgress on Linux in its default configuration and uh a backend tries to allocate memory that uh the kernel can't provide it, the kernel is going to send a sig kill to that backend which Postgress is going to treat as a potential memory corruption event and force every other backend to terminate. So now we have a bad situation. one backend tried to request too much
memory became a much worse situation. Every backend was killed which is going to have you know a widespread impact on on the application using this Postgress server. One thing you can do is simply disable the Linux killer. If you're running a single Postgress server on a single machine, this is a great option. But if you're trying to run a multi-tenant Postgress environment where you have multiple user
workloads running on the same machine, you're not going to be able to do this. And that's because if you're running a multi-tenant Postgress environment, you're going to want to make sure that each user's workload can't eat the other workloads memory, which means using Linux croups and ensuring that each workload has a fixed limit of memory within the total machine Unfortunately, this system croup uh memory enforcement limit
enforcement requires having the uh Linux killer enabled. So if you want to run a multi-tenant Postgress environment and you want to prevent users from oming themselves into oblivion, one thing you can look into is a user space oom killer like Facebook's omd. When you run oomd inside of a croup with Postgress and a Postgress backend tries to request memory from the kernel that the kernel can't provide
right away, the kernel is going to generate something called a pressure stall information event which systems like UMD can subscribe to and which you can react to however you want. You can configure OMD to find the backend using the most memory and uh terminate it with a SIG term which doesn't ruffle Postmaster in the slightest and all other backends will be left undisturbed. So here we've taken
a bad situation uh a single backend using too much memory and mitigated it prevented it preventing it from becoming Another way that users inflict damage upon themselves is by using up all of their disk space which is uh kind of bad on the on in an obvious way. Uh if it's on their primary, they're not going to be able to send more uh inserts, updates, and deletes.
But we've also seen it cascade into worse problems like uh checkpoint failures leading to panics and primaries being unable to recover from a panic. One way that we help users out here is by automatically increasing their disc capacity when their disc usage crosses a certain threshold. And if you're using an a network elastic volume uh like on Amazon AWS, you can do this with a simple API
call. Unfortunately, some users choose to opt out of disk autoscaling because they're sensitive to costs. And um if you're using a storage subs subsystem that does not do in place uh resizes or can't do it quickly, then the user may run out of disc space before that insize in place expansion can complete. So you're going to want a fallback to protect users. And we reached out to
a tool or an extension called PG readonly which when enabled blocks inserts updates and deletes which sounds similar to what happens when the disk is out of space but we view it as valuable because it prevents those uh that disc being totally out of space from cascading into those worst worse issues that I talked Unfortunately, it is limited. It doesn't block logical logical subscriptions and it doesn't
block um commands from thirdparty extensions like PG squeeze. But uh the way we deal with this anytime our tools don't do everything we need is just put the missing logic in our But what we really wish Kubernetes had is a native uh readon mode like my SQL has which would enable us to uh protect Postgress from a disk being completely full. It would honestly be a a
Postgress operator's dream if if we had this. Users get annoyed when they inflict downtime on themselves, but they absolutely hate it if you, their database service provider, inflict any downtime on them. And the only exception to this is if you are performing some amount of downtime to fulfill uh a user requested configuration change like an increase to shared buffers or upgrading if they request to upgrade from
a 4 CPU But when you do this, they're going to expect you to do it in the fastest and least impactful way possible. And one thing to keep in mind when you're restarting a primary when a user asks you to perform some kind of configuration change is you don't want to do any kind of long running process within the boundaries of stopping the primary and starting the
primary. To illustrate this a bit more, imagine a user requests to upgrade from a 4 CPU cluster to an 8CPU cluster. The way you would typically do this is by terminating one of their standbys, provisioning a new HCP machine, starting up restarting the standby on that machine, and then re repeating the process with the next standby. The question then becomes, what do you do with the primary?
If you try to repeat the same process on the primary, it's you're potentially going to run into trouble at the point when you try to provision new hardware. If your underlying infrastructure provider like AWS or GCP is not in a perhaps it's in a bad mood that day, it may not provide you the new hardware you want in a reasonable amount of time. It typically takes a
couple minutes, which is already a lot of downtime to inflict on a user, but we've seen it take tens of minutes, even even hours and really uh outlier cases. So what you should do instead is stop the primary and then promote one of the upgraded standbys and only then perform the hardware upgrade on the uh former This is called a switchover. And if you're looking for a
tool to help you do this and you're already using Patroni, you should check out its built-in API call which works really well. The only challenge that we've run into is, and it's not Patron's fault, I think it's Postgress's fault, is that when Patroni asked the primary to shut down, it can get stuck waiting for misbehaving backends to terminate. We've especially seen this with wall sender sending data
to a remote logical replication client. And while Patroni is waiting for um these stuck backends to terminate, it's not able to promote another standby. So you've sort of extended the duration of uh the the switchover and the downtime that you've that you're inflicting on users. The simple workaround here is to use a patron setting called primary stop timeout which governs how long patron will wait before uh
before it starts aggressively terminating misbehaving backends. One last example of how you can improve yourility story is by presenting downtime in a different way so users perceive it differently. Typically when you take a primary offline clients are going to perceive that as a spike of errors. If you allow users to route their queries through PG bouncer then you can present it in a different way. You can
pause PG bouncer when you take the before you take the primary offline and then when the primary comes back online you resume PG bouncer and clients will perceive the downtime as basically increased query time. The reason that users might prefer this is because may lack sophisticated retry mech mechanisms which would hide errors from their users. So uh if they don't have sophisticated retry mechanisms, they may surface
the errors that they get from uh at the client level up to let's say a web page and their users will see a 500. If you present downtime as latency, their users may perceive it as a longer webpage load time which some might say is is a better user experience. One way you can stitch a system like this together uh is if you're already using pachone, you
can use pachony callbacks invokes at different points in the uh postgress life cycle. So call back is the before stop callback. Um and uh Patroni invokes this before it stops a primary. And there you can call PG bouncer pause. And uh when is ready to a new primary, you can call all uh use the on roll change hook and send uh the PG bouncer resume command from
there. Another thing that users pay a lot of attention to in addition to uh high availability is performance. So if you want to make a good impression, you're going to have to invest here. You can cut network hops, make those hops as fast as possible, and you can tune Postgress params to be performant by default. And if you do that, you'll be about as good as anybody
else. Planet service because we saw opportunity to have faster storage. Most really all of the uh major Postgress service providers on the market use network attached storage, which is great for a number of reasons. It comes with a lot of out-of-the-box ergonomics that make it very convenient to build a database as a service on top of. It has APIs to increase the storage the capacity of the
network storage volume in place and it has APIs to attach multiple storage devices to a single machine which is great if you're trying to build a multi-tenant Postgress service. But all of this convenience comes at a significant cost. Anytime Postgress wants to read or write to disk, it has to try make a roughly one millisecond journey over the over the network which is a long time if
in the in the context of a high performing and uh cloud storage cloud infrastructure providers also impose IOPS and throughput limits on uh these devices. So if you try to do too many read or write operations at once, uh your read and writes will get throttled which is going to manifest as latency to your users. You can pay to decrease this latency a little bit and raise
these limits but it gets expensive very quickly. Contrast this with local NVME drives. They're extremely fast, maybe a 20 microcond roundtrip from Postgress to NVMA NVME back. And there are no artificial limits, IOPS or throughput limits. If you were to try to approximate this this performance with a network attached storage device, you would pay through the nose. The downside of all this performance is that you get
none of the out of the box ergonomics of network attached storage, making it harder to So if you want to do that, you're going to have to find a way to map some of the semantics of network attached storage onto local NVME drives. To illustrate this a bit, consider that cloud ser infrastructure providers will typically offer a limited menu of options for the kinds of machines that
you can use to uh run network attach uh local NVME drives on top of. So they're going to offer way more than the two options on the screen, but they're not going to have everything users want. Users might want a 4 CPU machine with a 4 TBTE NVME drive, but the nearest option might be, you know, more or less than that. So the user is going to
have to make some sacrifice in one direction or another. They go for the bigger option, they're going to be paying for capacity that they don't And unless you figure out how to run a multi-tenant service on top of NVME drives in which you have multiple Postgress workloads uh potentially owned by different users occupying the same machine and occupying distinct slices within an NVME drive, a shared NVME
drive. And if you figure out how to do that, you'll need to think about how do you tie together the life cycle of NVMe data with user workloads. If a user uh deletes their workload, you need to make sure to delete their data as well. We um the tool we ended up adopting to help us work through these challenges is a Kubernetes plugin called Topo LVM. When
you install too on your Kubernetes workers, it's going to discover the NVME drives on the Kubernetes worker and advertise it as uh capacity to Kubernetes, which you can then request by creating a persistent volume claim. So let's say you create a persistent volume claim uh asking for 1 TB of NVME storage. Kubernetes is going to coordinate with Topo LVM to find a machine with enough free capacity
to satisfy their request and create a logical volume slice on top of that NVMe drive. And if you delete the persistent volume claim, it's going to go in the opposite direction. It'll clear that u it'll it'll delete that uh logical volume slice. You can also top VM also supports running multiple slices per NVMe which uh per NVME drive associated with um tied to different persistent volume claims.
So out of the box with Topo LVM we get flexible sizing. Users can request the amount of storage they want. Uh we get uh we're able to tie the life cycle of the persistent volume claim with the uh logical volume slice and we get multi-tenency. The last thing I'm going to talk about today is how do we support resizes? Sometimes users are going to want to resize
uh increase their storage capacity. The good news is that Topo LVM supports this out of the box. If a user requests to go from 1 TB of uh NVME storage to two terabytes and their existing slice on the their current NVME drive has enough free space, then Topo LVM will just expand the existing slice to occupy the remaining space. Unless the user requests four terabytes and their
current NVME drive only has two, in which case too LVM can't help. And even if the NVME drive does have four terabytes, there might be other users occupying that same NVME drive, in which case again we can't we can't satisfy the request. So we did what we always do when our tools don't do exactly what we need and implemented uh something we called surgebased resizing in our
Kubernetes operator. And the way this works is we find or create new machines that have user's request. We start new uh Postgress instances on top of those machines and then restore from backup. And once the new uh Postgress instances are fully caught up with the older set of instances, we scale down the older set of instances. So this is admittedly slow and more expensive than an in
place slice expansion, but it allows us to solve the problem and it means that users can get really flexible choices in terms of uh which what side their sizing and um but with the raw performance of of local NVME drives. I'll leave uh this up on the screen uh and happy to take any questions. This is something we implemented in our own Kubernetes operator. um we don't
have any plans to open source it and I don't know of any open source tool that does this but um yeah the the concept's pretty pretty straightforward um but uh it's it's kind of the equivalent do you use Kubernetes at all okay yeah good choice um but uh you know it's kind of conceptually similar to if you're using like a deployment and you want to like go
from three replicas to six you can choose your rolling rollout strategy if you uh max unavailable zero. Um it surges new instances. So it's it's kind of copied from that, but it's for uh stateful workloads. Um yeah, there are other um Kubernetes operators out there for Postgress like uh Crunchy Data has one, Zolando data has one. Um so in theory, you could like take that, fork that,
and implement some kind of process like this. But no, I'm not sure of anything out of the box that does this. Yeah. Yeah. >> Mhm. All right. Not seeing any more questions. Oh, sorry. What? Oh, contact slide. There we go. Thanks, All right. Thanks everyone. I really appreciate your time and attention.
