etcd 3.6 Updates and 3.7 Roadmap - Arka Saha, Broadcom & Josh Berkus, Red Hat

This session focuses on recent updates and future plans for the CD (Cluster Database) project, essential within the Kubernetes ecosystem. The speakers, Josh Burkus and Shaha, discuss the improvements made in the recent versions 3.6 and upcoming 3.7. They highlight key features, such as the newly introduced ability to downgrade to previous versions, enhancing the stability and user experience when managing potential issues. Furthermore, they address a recent security release aimed at fixing authorization problems related to the CD off feature. The talk also includes a live demonstration of deploying the latest version of CD with support for certificates and an overview of upcoming enhancements, including performance upgrades and documentation restructuring.

Welcome to the SCD maintainer session. Um I want to go over a little bit of what's been going on with the SIG in the project. Um uh I'm Josh Burkus. Um CD documentation approver. Been messing around with CD for 12 years. >> I'm Shaha. I'm a software engineer at Broadcom and I've been contributing to CD and recently became a CD operator reviewer. >> Awesome. Okay. And we

have a brief agenda here because it's a brief session. Uh talking a little bit about what went into 3.6 um and some updates in 3.6. Um a little bit of 3.7 roadmap um info. Uh some other news. Um and um what we're hoping to get out of this is that we're hoping that at least a couple of you will become interested in contributing to CD um because

it's an important project, a key part of Kubernetes and of other things. Um and we are always looking for more people to get involved. So uh with that uh I want to briefly have an interruption. Um how many people noticed that we had a security release on Friday? Okay. Well now you know we had a security release on Friday. Um the um I to explain the security

release I have to explain a particular feature um that many of you may be unaware of which is called XCD off. Um and I and remember I said this that it only really applies to one person because um CD off is something that you use when you're not using CcD as part of the Kubernetes control plane optionally which is an authentication and permission system forced uh but

it is not used by the Kubernetes control plane which controls access to CD itself. Um however for people who are using SCD off Kubernetes some of them are using off and we had a number of community members report a number of problems with CD off um not enforcing authorization for some resources. Um and so uh on Friday we released a new version new patches to all the

supported versions um which fix these off holes. Um like I said for all of you who are running the Kubernetes control plane you know upgrade at your next scheduled maintenance window. Um but if you are using CD off you should probably upgrade after we get done with this talk. And with that let's talk a little bit about 3.6. >> Yep. Uh so it's almost been an year

since 3.6 got released. So let's see what all features it comes with. The first being the ability to downgrade and it's officially supported. So why do we really need the downgrade feature? So the first thing is since HCD is such a robust key value store, we would want to constantly assess that okay there is no data inconsistency that is happening and any kind of bug that gets

reported we uh very actively uh fix it in the patches and across uh the minor versions. So for the users to adopt uh these fast changing fast releasing uh patches we want uh them to easily able to upgrade without having the uh hassle or the um worry of not being able to uh later uh roll back in case of any um regression if it comes. So that

was the main motivation behind uh having the downgrade uh available. So what we did is that we actually added uh the version attributes so that uh a specific uh level of downgrade is possible only one minor version at a time and that's like from 3.6 to 3.5 and for that we did add annotations to the raft protocols as well as uh validation checks uh in the right

ahead logs to allow that. And basically to uh do that the first we do is the uh validation that whether 3.6 on which we are currently on is able to be downgraded to 3.5. And once the validation uh gives you a successful message we can then enable it. And what it essentially does is that it does the migration of the uh database. And what basically happens is

that all the 3.6 six features which can be used those features get deactivated so that you can um uh roll back to 3.5 without having any issues and uh basically uh what we do is that uh we use the downgrade API and we uh basically status uh poll whether all the members have agreed upon it uh and it's a rolling downgrade so one by one Each member

gets downgraded and that's how uh we go ahead with the downgrade process and uh so basically this is uh how the command looks like. So what we uh do is that we check whether the HCD cluster is healthy or not to start with and if it is healthy then we uh do a validation and once the validation is successful then we go ahead with the downgrade enablement

and we see that here the storage version although the uh uh it was 3.6 six, the storage version has been downgraded to 3.5 and now the V2 store is >> Yeah. And I apologize, we plan to have a live demo for you here, but one of our speakers couldn't make it to KubeCon. Uh but we'll have a different demo later Uh so yeah, so basically uh now

let's see what is the key difference between uh 3.5 and 3.6. So first uh the or it originated with v2 store which was the initial format in which the uh data and uh the logs were getting stored and uh there was a unit uh action that used to happen that on every 100,000 log rides it used to do a snapshotting and that uh used to be relatively

slower which we fixed in 3 uh uh v3 store uh which basically uh is the source of truth for 3.6 six and it's much more reliable. we uh reduced it to uh 10,000 uh log rights and we will snapshot it only uh on the basis of membership change and that uh basically led to the deprecation of uh v2 store and uh from uh basically uh 3.5 although

it is using uh v3 store uh it internally uh uses v2 store as the source of truth but in 3.6 six we have moved it completely to uh v3 and that has come up with some known upgrade issues which we will briefly discuss here. So what happens is during an upgrade uh whenever there is a 3.5 to 3.6 upgrade uh it it's a one by one. So

whenever the leader is uh going to be demoted as a learner and that gets uh caught up with uh all the logs and it gets upgraded to 3.6 six that data is persisted in the v3 store while the learner which is uh still in 3.5 that data is uh stored in v2 and that inconsistency causes a problem of having too many learners. So uh the fact that

uh there is more than two learner which is not allowed uh is the issue that has happened and uh that has been fixed in multiple uh releases. Uh and one more other issue is that the old snapshot uh data gets forwarded carry forwarded while doing a rolling upgrade. So that also has been fixed in uh 3.5.24 and above. And uh so basically coming to what uh the

uh advantage that we got from uh V3 uh store is basically uh instead of doing 100,000 writes we do 10,000 writes and that actually uh makes the processing much faster and uh it's less data inensive as well. So other than this this we have also uh had other enhancements which goes under the hood. So like having a kept style proposal for any kind of et advance uh

uh enhancements. uh then migrating experimental features into feature gets and uh we will we have also migrated from using flags to basically uh using cubernetes style uh feature flags. So uh yeah another thing is basically having uh live Z and ready Z uh endpoints which is also aligned with the Kubernetes uh probes and yeah >> okay so let's talk a little bit about what we're working on

what's what's coming um the um uh so this is is what's in development so far for 3.7 um which is going to be our next version um one of the big ones that's uh just about done um is CD range um because who here like I don't know how many people have actually queried at CD directly even in your control plane who here has like tried to

accidentally tried to pull a million keys out of CD it's not a great experience yeah yeah um so uh we're doing something that we're implementing something that's standard um all over the database industry um and data applications industry which is you know chunk that data up and give it to the client in chunks so that we don't lock up the client for 10 minutes and then have

it crash um which is the current behavior. Um and so that you will actually be able to if you have some legitimate reason to query a million keys um uh you could theoretically actually collect that data. Um uh this is a cap in Kubernetes. So if you want to keep track of progress and the stuff that that isn't done, that's cap number uh 5967 and you can

actually keep track of the implementation and and like other Kubernetes features will have a you know a beta and then a GA etc. Um the uh next we have a lot of cleanup. Um, for one thing, um, we hadn't updated how we use gRPC for quite a while. Um, and have not been keeping up with the upstream project. Um, and also with, uh, Protobuff. Um, and so

now we're actually getting caught up in doing a lot of cleaning that up. Um, for one thing, like a lot of other parts of Kubernetes, we were depending on outdated and now archived ProBuff libraries. Um and so those are being removed um and replaced with the current maintained protobuff. Um we're trying to remove all of the features marked as um because we've been told by the upstream

project that a lot of those will not ever become not experimental. Um and and for that reason we shouldn't be depending on them. Um and unfortunately that among other things involves us needing to implement our own client side load balancer um uh for the SCD client because we were depending on an experimental one from uh uh from uh in Protobuff that is not going to continue. So

if hacking on something like that sounds fun to anybody um we really could uh use somebody who's interested in spending some time on that um as well as in removing cmox um uh and cleaning up after that and a bunch of other miscellaneous protobuff gRPC related now we have two performance features, uh, two major performance features proposed for 3.7 that we've investigated. Uh, um, but chances are

they're not going to be in 3.7 unless we get somebody who with a performance engineering background who's really interested in taking them on. Um, uh, one of them is making use of raft async rights. Um the um as you all know uh if you have a really large Kubernetes cluster uh sync right for the CD cluster become your one of your main scalability bottlenecks. Um uh because

that you know ties you to the speed of whatever your storage is. Uh um uh the raph folks have added some async writing features. Um, so far when we tried them out, we haven't really seen much real performance gain. Um, and since any change to how we do data rights comes with a lot of risks around, you know, maintaining consistency and and not having data corruption, um,

we're not going to go further with that unless we can actually see substantial performance gains. Um, the second one here is, um, somebody proposal to, uh, make the bolt rebalance threshold configurable. Um and so uh this is when you know we're rebalancing the data structure etc. Um defaults to a very arbitrary 25% in the code. Not currently configurable. Uh we tried making it configurable. Can't come up

with alternate values that for any use case we know of actually improve things for users. And um unlike some other databases, CD feels like if a knob isn't going to do anything, we won't expose it to the users. Um so again, somebody really interested in performance engineering, you could jump on either of these and see if you can actually come up with a schema where these are

valuable features. Otherwise, probably won't make it to 37. Um also speaking for the docs team, um we are working on a documentation overhaul. Um, the structure of the SCD documentation is more historical than anything else. Um, it wasn't really planned. Um, this can make it very hard for somebody who's brand new to CD to learn it and to find the information relevant to them. And it's even

hard for the docs team to figure out what's still missing from our docs because stuff is so all over the place. Um, and so we're trying to sync up people's schedules to reorganize it into a role-based structure the same as we have in Kubernetes. Um, and one of the things we're considering to make it more maintainable is possibly getting rid of versioned docs because right now we

do a lot of backporting stuff to the earlier supported versions. Um, and we're not sure that it's actually valuable to keep doing that. Uh, but we'll see. And that's so far the plans for 3.7. Um, uh, which will be out at some point later this year. Um, and then we have So first thing um we haven't really talked that much about support windows for the versions of

CD partly because we went through this was what was it three four year period four year period of not releasing any new major versions um but uh importantly um is anybody here still incd 3.4 four. Okay. So, you should really be planning an upgrade of your CD database because we're going to stop patching that in two months. >> yeah. So, um, we did the the new downgrade

support he was talking about that does go back to 3.4. So you can in fact if you have a bad upgrade you can roll back um the um I the rest of the the sort of windows are going to be kind of dependent on when 3.7 comes out. Um but we will continue to support 3.5 till at least next May. It might end up being longer if

3.7 does not get released for a while. um the um but we're going to start publishing these so that that people actually know when they need to expect to upgrade and hopefully releasing a new version at least every Yeah. Uh so uh bas basically we also adopted the go workspace uh and since city has multiple uh go models within uh the depot so it actually turned out

to be a lot of uh go vulnerability uh issues which were false positives. So this was a journey like uh adopting to the go workspace and for that we actually have in the uh main branch we have done all the work although there are few uh jobs that are still uh basically for the automation and the bumping uh go version bumps uh it those scripts are still

in progress but the adoption has already been started and it has been going forward. Um, we now have official images on the Kubernetes registry. Um, so if you want to actually start using, you know, feeling like, okay, hey, we're going to use the officially distributed project image, um, that's where it's going to live. Um, the um um that is um what we'll install with Kubernetes for new

versions um will be those official images. Um uh so if you're installing Kubernetes 135 or later, they will be the official CD >> Yep. Uh so basically we uh released 0.2.0 a few weeks back uh forced operator. So uh what it basically does for the city operator is uh we will be able to bring up a smallcd cluster through an and manage it through an operator within

the kubernetes to support your uh application workloads not for the kubernetes itself as uh starting now and for with 0.2.0 release we have the certificate support as well as uh upgrade support. So uh how the road map looks like is basically we had two releases and in the future we will also be adding uh recovery and backup with the one release uh doing a helmchart release with

the for the operator itself and uh so a little about the 0.2 release uh about the certificate thing. So basically we have the auto certificate uh which is basically not uh recommended for production use but for development you can uh just uh have a uh CD cluster uh up and running just to play around and not have any certificate manager uh to do with. So should we

do a live demo? >> Yeah, go ahead and go to >> Yeah, we have enough time. Yeah. >> So let's Yeah. Uh so we'll bring up a kind cluster and we will uh deploy the latest 0.2 image uh that we have and yeah this is the official image uh that we have published and we'll just We'll just confirm that we have the controller manager up and running.

Still need to wait. Yep. Now we have them running and we also have the CRDs installed. So basically what we will do is we will see how the uh cluster object looks like cluster object looks like and uh this is basically the very basic strip down version of it. So first we have the city version that we want to want our uh city members to be on

the size of the cluster and then finally the uh TLS provider which is auto and the validated duration. So let's apply and then we will watch. So yeah, this is the uh ATCD cluster object and we see uh each member coming up one by one and we'll also actually try to look at the secrets there. So yeah, if you see we have the client, peer and server

secrets and as soon as each member comes up, these secrets will be mounted to each of the members. And yeah, now we have all the secrets. So now uh for the production ready uh usage we will be like uh we have already implemented the uh search manager and that also we will see it in a live demo. So yeah, we bring up the contest again fresh and

we deploy. So we will need search manager to be So that is something that uh I'm also as a part of it. And so we will for for the search manager we will need an issuer which will be able to issue this certificate. So that is again a search manager requirement. So it can be any issuer. Uh the uh user might be already using issuer for their

other workloads. They can use that. uh here since we don't have any we will be uh applying that. Okay, I'll probably the parts aren't ready yet. Yeah, we might need to wait a bit. Let's look at the YAML while this is getting ready. Yeah. So uh this is the city uh like for the search manager configuration. So everything uh at the top remains same. What changes is

basically the provider which is the search manager uh the validity duration uh that we can configure and the extra fields that we have is this part which is basically you mention that what issuer what is the name of the issuer and whether it's a cluster issuer or it's a namespaced uh So we'll quickly check if the ports are ready. Yes. And now we can apply. So we

will now apply the yes this and we will check how things are. So yeah uh so we have the city cluster object. We have each members coming up one by one. We have the issuer that we created and we have all the uh respective certificates that uh have been created and the related certificate secrets which will be mounted to each one of the members that are coming

up. So yeah just one more member to come and yeah we have it here. So that was a quick demo. Okay. And like I said, u one of the reasons that we do this um is we're hoping that some of you will get interested in messing around with CD as a project and potentially contributing to it. Um we have a lot of things where we would like

to have more people involved with CD. Um I uh performance engineering as I mentioned earlier. Um we're doing a lot of overhaul of old code including Protobuff overhaul. Um the um uh we could use um new eyes and new work on the underlying raft consensus uh support uh for the raft sub project. Um as a matter of fact um if somebody wanted to get really involved and

become the new lead for the raft sub project of CD um that that position is open. Um the um uh we want more people uh Merrick would love to have more people working with him on robustness testing. Um uh which has been awesome, but for a distributed consistent database there is literally no limit to the amount of robustness testing you could potentially do. Um the um we

want for the operator um sub project. Um we'd like to have more people working on that. Um, and of course speaking for the docs team, there are plenty of unwritten bits of documentation um that you could write uh if you were so inclined. So some ways to get started contributing um we have a lot of help wanted issues um in the various CD repos um undercd.io um

you can join our slack and chat with us. You can join the mailing list. We're one of the few SIGs who actually uses our mailing list. Um uh we have community meetings on Thursdays uh generally um although the docs meeting if you're interested in docs is on Tuesdays um in kind of an ocean friendly time. Um I take a look at our existing contributor documentation which is

a bunch of and there'll be a link to later on. Um, look for the announcement of our next mentorship cohort. Um, because, uh, we do that about once a year or so. Um, and we mentor a bunch of people, uh, looking for people to level up their ETC contributing skills. Um, become reviewers or approvers. Um, as a matter of fact, we got two of them in the

room with us right now. So, Arco was in the last mentorship cohort. In the >> Um, and so was Wendy, who's over there. Hey, Wendy. Um and and both of them have leveled up to be um uh reviewer contributor. Yeah. So that could be you. Um and with that uh we have like two minutes left for questions. Um and a link to lots of ETCD project resources

there. Um so with that, do we have one or two questions? >> Uh go for it. We got a mic right there. So you'll be on the recording. Um, so we've had some issues with the it's um it's a bit midway between Kubern being a Kubernetes issue and an STD issue where um uh in a multi-enant Kubernetes cluster with multiple users each on their name space um

one of them can go off the rails and start creating multiple resources in a way that um fills up the CD cluster. completely. So then it locks it locks and you it locks into a read only mode and um that's clearly an issue because don't have observability on the um the you the space user the database space users per name space that's not really uh something that

um that is relevant to itself but uh the way Kubernetes uses it I guess and um we can't limit it either per per name space or per prefix in the um in the database. So what are your what are your thoughts about that and um I guess there could be um some eventually some way to uh try to limit this. Yeah, I mean that sounds like a

Kubernetes feature request. Actually, two different feature requests in fact, right? Because you have two things you want, right? One is you want to get some idea of the space consumed for a set of objects. >> And then second, you want to be able to put some kind of limit on the number of objects per namespace. >> I mean, the second one sounds like something that might actually

be possible with other plugins to Kubernetes, you know, with like policy stuff. >> Yeah. Um I have not had to solve that particular problem myself. So I couldn't tell you off the top of my head how how I would solve it. But um the um Merrick any suggestions >> you ask about the first thing Kubernetes 134 comes with resource size metric for per resource. So not namespace

because there are infinite number of name spaces. So I don't want to blow up your Prometheus or monitoring stack. Uh for the second are you asking about multi-tenency like incd so like or are you asking kubernetes per resource? Um um because the name space because the uh each resource basically maps to a is stored in a path in the TTD database which contains the name space and

also the resource type but mainly the name space. I guess my idea initially was to uh use that path or a reax of the path in order to um assign a size limit to uh again kubernetes had a feature for koas I don't think it well went well uh for atcd that's more of multi-tenency problem of like how do you make sure that one uh tenant doesn't

break other so currently we don't have a good multi-tenency story it's something that we are like I know considering but it's very uh Uh I know I would say pretty hard because everyone uses Kubernetes and doesn't use like multi-tenency. >> Okay. Thank you. >> Okay. Well, thanks. And with that we are out of time. We're happy to answer other questions but we'll have to do it in

the hallway. So thanks everybody.