The Road To Strimzi 1.0 - Jakub Scholz, Cloudera & Paolo Patierno, IBM

This talk discusses the journey of the StreamZ project towards its upcoming 1.0 release, featuring insights from co-founders Paulo and Jakob, who work with IBM and Cloudera respectively. They highlight the project's history, which includes eight years of development and over 300 contributors. The speakers mention significant changes introduced in version 1.0, including the removal of deprecated APIs, specifically those related to ZooKeeper, and the establishment of a new version 1 API. They also introduce an API conversion tool designed to aid users in upgrading their custom resources more easily. Future enhancements like dynamic quorum support, third-party certificate management integration, and automated scaling are also outlined, along with an invitation to a forthcoming virtual conference centered on StreamZ.

Okay, I'll say let's start. So, hi everyone. Thank you very much for being here. I hope that you are enjoying CubeCon and yeah, uh together with Jaku today, we are going to show what was you know our journey finally to release the stream 1.0 release of course. So, let me introduce myself. I am Paulo. I am an engineer working for IBM uh on Kafka and Strezy because

I'm one of the co-founder of the streamy project. So one of the core maintainer and uh I am also a CNCF ambassadors and together with me there is Jakob. >> Hi everyone, my name is Jakob. Uh I work as engineer at Cloudera and I'm also one of the streams maintainers and uh earlier this month we actually celebrated eight years of streams. Uh we celebrate the birthday as

the day of the first release which was following the operator pattern which was actually not the first release, it was the 0.2 release. Uh so eight years is a very long time and uh today we want to celebrate another thing as well and that's that we finally got to the 1.0 release. Are there any people who contributed to streams here in the room? Raise your hand. Great.

I see a few hands. So uh we have over 300 contributors uh during this whole time. So maybe we can give an applause to them. So thanks everyone for your contributions. Uh and uh let's talk a bit about the 1.0 release. And uh maybe the almost word there might be interesting for you. So uh to be honest, the 1.0 release is not uh not out yet. Uh

it's almost out there, but it's not because of some unexpected delays. It's pretty much goes uh by plan. So uh originally we wrote and approved the proposal for the 1.0 release and for the version one API in September last year and then in November we did the stream 049 release which contained the new API. Uh and then we continued this year with uh 050 release and 051

release. Uh to be honest with you the 051 release was slightly delayed just uh just by one week so nothing serious. And then in April we plan to do the 1.0 window release I think uh yeah we are sticking with that plan so there should be no uh delay and uh yeah so that's all as it was planned on the beginning but uh you know cubecon isn't

happening every week every month so that's why we decided to celebrate uh it here a bit early hopefully it will not bring uh bad luck uh let's talk maybe about the main changes in the 1.0 0 release and uh to be honest it's not necessarily a change in the 1.0 release because the v1 API is something what we already introduced in uh the 049 release because uh

it's not that easy to kind of switch from one API to another just like this. So it needs more time. So that's why we introduced it three releases earlier and uh the change in the 1.0 release will be actually that will not contain anymore the old APIs. uh uh the new API did kind of a big cleanup of the streams custom resource definitions and the APIs in

them. Uh we obviously had a lot of old stuff which was deprecated and not used anymore. A lot of that was related to zookeeper. Uh so all the zookeeper stuff is gone. Uh we also used to use stateful sets in the past and had some configuration options for them. That's also gone. Uh but we did some other changes as well. So we for example decided to remove

some specific APIs for example for configuring the off authentication the keycloak authorization or the open policy agent authorization. Now don't worry don't be scared these features are still there and can be used but uh to be honest the APIs we didn't really design them very well. They had basically no validation uh whatsoever. Uh they had a lot of maintenance effort related to them. So instead we are

kind of trying to move to this uh what we call type custom APIs in streams where you can use these to configure the or of authentication or of authorization open policy agent uh and actually many other authentication or authorization mechanisms you might want to use because uh yeah this is much more flexible and uh uh gives uh us uh much less freedom and uh yeah let's be

honest also a bit less stuff to to maintain in uh in every single release. Uh so hopefully that uh uh will work better. Uh when uh designing the the 1.0 version and the upgrade path, we try to be kind of as give give all our users as much flexibility as possible. So uh we didn't want it to say something like okay this release is something you must

upgrade to and have to use it then month later you need to upgrade to this other release and so on. So we try to kind of give you this flexibility and we also uh created this API conversion tool which uh tries to help you with converting the the custom resources but also kind of before you upgrade to streams 1.0 or 1.1 or whatever you need to upgrade

the CRDs kind of modify the CRD status it's it's not completely user friendly in Kubernetes so the tool helps with that as well uh to talk a bit more about the the flexibility so imagine you start on some older streams version like uh 048 uh there's uh one rule that you need to be on craft if you are still running zookeeper based cluster you first have to

migrate to craft and only then you can uh kind of start this uh transition. But otherwise you can for example run a cluster on stream 0 46 47 48 whatever you are running and from there you have to go and upgrade to one of these versions 049 050 and 051. You can of course go through all three of them if you are upgrading regularly. That's the best

thing. We love those users but uh uh you don't have to. So if you have your upgrade schedule that you upgrade every six months for example and it fits best for stream 051 right now for example then then do that go there directly and uh that's kind of one of these free versions. It's where you can get familiar with the new API and you can start converting

your custom resources to the new version one API because these versions they can still work with the old APIs but they can also already work with the new APIs. And then finally once you are moving away from from these kind of in between versions that's where you have to use the conversion tool to upgrade the the CRDs what you really need to do there is the Kubernetes

is tracking in the in the status subresource of the CRDs which versions were used and you have to remove the old versions from there before you can actually remove them from the CRD spec section. So, so that's kind of what the conversion tool helps you with. And that's pretty much something to do right before you do the upgrade. And then the upgrade should be either to StreamZy

1.0 in April uh or again if your next upgrade in your schedule would be in the summer then maybe that would be 1.1 1.2 uh whatever is out there uh by then. You always have to use the upgrade tool but it can be some older 1 something version as well. uh to touch a bit more on the on the conversion tool. So the conversion tool kind of

can do uh three different things for you. If you are using some githops tooling and you are managing your custom resources for streams in files, then basically you can just kind of check out your git repo with the resources, you can use this convert file sub command to convert these yl files. then you can commit it back into your uh your git repository and you can have

your githops tooling roll it out into the kubernetes cluster. If you manage the resources in kubernetes cluster directly there is this convert resource uh subcomand which kind of just edits the custom resource directly in the kubernetes cluster. So that's how you can kind of convert the resources. And then finally there's the CRD upgrade command uh which uh is the part you do right before the 1.0 upgrade

to kind of remove uh the old versions from the CRD status and that's after that you install the new CRDs with the version one only. Uh now uh let's talk about this thing as well. Uh eight years uh that's a pretty long time right? uh 51 minor releases. Uh that's quite a lot as well. Uh it was actually almost 7,000 uh PRs that were merged during this

time. So uh what took us so long, right? Why did it took us eight years to get to the 1.0 release, which uh it's maybe not completely unique, but I think it's pretty unique and it would be hard to find a project which take uh takes such a long time. uh what it was not it was not waiting for streams to get production ready right for for

years we had a lot of users in uh in uh production this is just the small subset of the users we have who decided to add their logo to our website uh so you can see there are some pretty big companies organizations uh if you are using streams in production and your logo is not there then please uh consider it and open the PR It helps us

uh it might help you because then in the streams community we will know better who you are uh when you have some problems need some changes or need to help with something or something like that. We have also bunch of vendors uh again pretty major companies including uh clouda where I work and redhead and IBM where Paulo works. So there are vendors who kind of trust streams

for years now uh enough to kind of build their products on it. So it was not production readiness what was kind of holding us back. It was uh a bad decision than long time ago. So long time ago we all knew that the zookeeper will be removed from Kafka and our thinking was like yeah that will be a big change. It will have big impact on the

APIs uh uh we will need to roll out a new version because there will be a lot of deprecations and so on. Let's wait a bit with the 1.0 no release until that's done and do the release then right and uh was two years three years four years and there's something called uh some cost fallacy it's it's often about money which you invest into something and then

you don't want to kind of just write of the investment but counts for time as well right after you are waiting for three or four years for something you are like h it's going to happen next year it's now let's let's stick with it and let's wait and then maybe it's one more year so so that was kind of our bad decision why uh why it took

us eight years to get here. Of course, it's easy to see the bad decisions when you are looking uh back. It's not always that easy to see them when you are looking forward. But uh the good news is that uh I don't think we really lost that much because it's not like we were sitting there and doing nothing uh the whole five years. We obviously work on

the on the craft stuff. We worked on zookeeper removal. We worked on the streams pot sets and Kafka note pools which have a lot of different features than just uh uh just the zookeeper removal. We worked on the things such as horizontal autoscaling of the Kafka clusters. So there's a lot of work which we have done during that time. Uh so yeah pretty much the main wasted

time was when we had to explain to someone that we don't have 1.0 release and yet we are production ready. uh but uh yeah if we made a better decision long time ago then today we would have been probably standing here and uh talking about the 2.0 release and uh the version two API uh uh now obviously with the 1.0 the old release. Uh that's not the

end of everything. We are planning to continue. We hope that we will continue for a long time and that's why Paulo will now talk a bit more about uh what's next. >> Yeah. So thank you Yagub. So with Jagu we saw what was the journey right that uh brought us to have finally the stream Z10 release next month. So but what's coming next? As Yagu mentioned it's

not the end it's the the beginning I would say. uh I will show you what are the main big features that we are um planning to have across the next months uh across this year. uh how it works in stream in general when you have big features to be added to stream project you go through some proposals it works like for kafka for example so you have

this streamy improvement proposal where you propose something you define what are going the changes to the API and things like that and then if it's got accepted by uh the maintainers by the community then you start implementing that feature so the first one that I'm going to mention is about the support for the craft dynamic quorum so as Today within um Kafka you have two ways for

defining the quarum which is the static quorum and dynamic quorum but everything is moving towards to use the dynamic quorum only. So with study quorum you are going to configure what are the voters forming your quarum. So the controllers within the configuration and then you start tle the controllers together and they are able to to reach each other and forming the the craft quorum. But it's not

that useful for having uh scaling. So for for scaling your controllers up or down if you want uh there are problems about downtime of course and you you are going yeah to lose maybe some availability because without the quorum your Kafka cluster is kind of not uh usable. So for this reason uh they moved to use the the craft dynamic quorum. With the dynamic quorum uh there

is a different way uh how the controllers are discovering each other. uh they are using u um a specific voters record which is a message that they are exchanging through the metadata topic. So the controllers are storing all the metadata within this metadata topic. There is replication across the controllers and that's the way when after you start up your uh cluster with the initial controllers then you

can actually scale up your controllers pool by adding a new uh controller. this controller will be able to start replicating the metadata getting this voters record to know what are the other voters in the quorum. Uh it will start up as an observer and then you have to run the registration process or command in order to have this uh controller to join the quorum. uh it would

be the the the opposite of course if you want to scale down you have first to unregister the controller from the quorum uh and then scaling down otherwise you are going to break your quum right so there are some operations that you have to do manually if you are running Kafka on bare metal of course with streamz uh there is a proposal open and we are looking

for feedback which is about automating everything so uh what you will have uh is a kind of craft quorum reconciliation within the operator. So the operator will take care of okay what's the current request from the users in terms of replicas of the controllers. So if there is a scale up or scale down if the users is adding controllers or removing controllers and kind of trying to

reconcile what what is actually in the Kafka cluster. So what are the voters and so on. Uh of course when you are going to scale up you are increasing the replicas in your Kafka node poolool with the controller rule. uh and then after that it will be the operator to register the controllers for you. So uh making them to join the quorum and moving from to be

observers to be voters in the quorum. Of course on the other way around if you want to scale down and you are going to reduce the number of replicas for the controller Kafka node pool then the operator will unregister these controllers for you and then shut down the controller. So everything will be uh automatic. Uh, of course it's not just about scaling up or scaling down controllers.

Uh, you can also start from some combined nodes where you have a broker and controller rule together and you want to remove the controller role from these nodes. Uh, or the other way around if you have some dedicated broker nodes but you want to have the controller role running on this node as well. So moving to combined node that also support for handling the Jbot because um

in your cafka nodu pool when you have the controllers running you have configured one or more Jbot discs one of them will be the disk hosting the metadata for any reason disk failures or something like that you want to add a new disk and using that disk for handling the metadata. So even in this case there is a kind of way of it's a way of changing

the controllers in your quarum. So the new controller with the new disk will be incarnation of the same controller which needs to be registered and you have to unregister the old one. Of course everything it's kind of automatic with stream C. Uh and also um the proposal is about allowing or providing an automatic migration from static to dynamic. If you're running your Kafka cluster today and it's

using the static quorum of course when you upgrade to the future stream having this kind of support it will just migrate automatically your uh controller quarum to be dynamic and not static anymore. The next big feature is about the third manager integration. So for this the proposal was already accepted approved. So the implementation is happening. uh what we wanted to do is uh actually um delegate to

somewhere else the life cycle of the certificates because today for people using stream in stream you can use a stream operator itself to generate the CA certificates the cluster and client CA and then it will sign all the brokers and controller certificates etc for you or you can bring your own certificates within the secrets that are used by streams for that but it's better having um something

like specific tool like search manager in order to handle the life cycle of this certificate for you. So we are decoupling all the cafka related operation from the operator and uh we are removing the certificate handling to and handing over to the ser manager. Uh of course it will be the operator taking care of the communication with ser manager for getting the certificates and use them within

your uh Kafka cluster. The other will be uh improving the auto rebalancing. So with streamy you have a tight integration with cris control. Cris control is used for rebalancing your partitions across the cluster. The way that you do that today in uh stream is uh not interacting directly with the cris control. So you don't have to take care of the crease control API. You have this kafka

rebalance custom resource that you can create in order to specify the goals that you want to use for rebalancing your cluster. Um but there is also other than the possibility to run this Kafka rebalance um on demand. manually whenever you want uh to run some auto rebalancing when you are scaling up or scaling down like for example you enable the auto rebalancing no scale up you can

scale up your uh brokers and then the the new brokers are actually empty they will be filled by new topics partition but if you want to rebalance the existing partitions uh on the new brokers as well uh you will not need to do that manually by using the the cafka rebalance custo results but the the auto rebalancing will take care for you. And the same when you

have to scale down. When you have to scale down, you have brokers or partitions. You can not just shut down the brokers. So the auto rebalancing will move partitions off these brokers that you are removing and then the brokers will not host uh any partition anymore. So it will be shut down. We want to improve this by adding uh the the the anomaly detection. So in cris

control you can detect anomalies and goal violations and at that point the operator by leveraging cris control anomaly detection will start the rebalancing by using the cafka rebalance custom resource again. Uh we are not going to leverage the self feeding because cris control as a way for starting automatically the the fix of the anomalies because we want everything to be controlled by the operator by the operator.

We don't want to have cris control at some point starting some process moving partitions around and the operator is not aware about that. So uh the operator yeah will get some uh info from cris control that the uh the anomaly was detected there was a goal violation and then we'll start up the uh rebalancing. So it's kind of complete the rebalancing features that we are restrict as

I say on auto rebalancing or scale up or down even moving data across the disks on same node and fixing the anomalies and for this again there is a proposal which is upstream open and waiting for for feedback. Uh last but not least and there is a proposal actually opened a couple of days ago by Yakoup the integration for gateway API. So as you know the in

uh Njix controller was um retired. It happened yet uh yesterday. We was at the session where they archived the project. So for your listeners within stream that are using ingress and using the Njix controller. You are not getting any more update from controller for CVS. So of course the the the idea will be the the path forward is just moving to use the gateway API. So we

are working on integrating the gateway API within stream. uh it will be about of course on your side bringing your own gateway because the there are several uh different implementation of the gateway. You can see some names here. So invoice and selium but then uh you will configure your uh new type TLS route listener within the listener. So your Kafka cluster and the streamy operator will take

care for you of creating the TLS route custom resource and configuring it in order to to to work together with the gateway. Uh it will support the both the TLS pass through or the TLS termination. So the TLS pass through it will be just connection one client with the Kafka broker on the other side passing through the gateway and using the TLS certificates provided by uh through

the Kafka cluster by stream. uh the other will be the TLS termination where you have the clients connecting by TLS to the gateway and then it will be playing between the gateway and the and the Kafka brokers. So that will be all yes handled by the the streamy operators I said. So these are the main big features uh that are coming I would say in the next

months um across this year within stream. Um the S manager is uh the implementation is going on for the others. If you want uh to to give us any feedback, you will find the the proposals open in the stream proposal repo. Now before leaving uh I would like to highlight this uh conference which is coming on June. It's the third editions. It's the the stream. It's virtual.

It's free and it will be of course all about streamy. And main thing is that uh the call for proposal is still open. It will be open uh in uh it will close actually in one week. So if you have any you know any proposal that you want to to submit meet it will be great. These are the main topics that we cover during this uh this

conference. So of course streamy if you want to talk about some internals about streamy how some part of streamy works or case studies. So showing your use case if you are using streamy in production in your company and what's the way that you are using it integrations. So if you are integrating stream with some other CNCF projects or even other open source project but outside of CNCF

so we can see integration with you know open telemetry promeus or graphana any other kind of project that you are using together maybe with streamy and the operations. So how streamy is helping you in order to deploy and manage your Kafka cluster within Kubernetes. So it will be great to hear from you. If you want to submit ideas, you have one week to do so. And uh

yes uh we started yesterday afternoon. Uh we had a lot of people coming to be honest. It was really great to see how many users are really using uh streamy uh even way more the logos that you see on the on the slide before with Jakob. Uh you can visit us at the project pavidian. This is the kiosk uh P1B. So we will be right after the

session at the kiosk. You can come and say hi. Uh if you are using streams, if you want to have more information, if you have any issues, we can talk about And yes, before uh leaving you, these are all the reference that we have to engage with us. So starting from the official website where you can find actually all the documentation and everything about stream all the

quick start guides there is also the main repo. So this is the stream organization where you can find the main repository which is about the operator of course. Then you can find all the other components in the stream ecosystem like the http bridge the component the quota plugin uh the access operator and so on. Then yeah Twitter and LinkedIn account that we usually use for advertising and

for sharing the new releases uh uh etc. and YouTube channel where you can find reference to the conference talks uh cool demos uh um the what's new series which is about uh we always uh uh publish a new video every time there is a new release explain uh what's going on in the new release and you can also find on the recordings of the community meetings but

if you are uh using streamy it will be great that you will join us during the community calls that was all so thank you very And I think if