What’s new in Spring for Apache Kafka 4? by Tim van Baarsen @ Spring I/O 2026

In this talk, Tim van Baarsen presents the latest features of Spring for Apache Kafka 4, particularly focusing on the migration to Spring Boot 4 and Spring Kafka 4. He shares insights from his experience at ING Bank, where they process millions of messages per second through Kafka. The session explores new enhancements such as the Kafka rebalance protocol and introduces Kafka queues, allowing multiple consumers to share partitions, which facilitates dynamic scaling. Tim emphasizes the need for code migrations and adaptations using tools like OpenRewrite and discusses potential trade-offs between the old and new protocols. A key aspect of the talk is the demonstration of practical implementations of these features and the advantages they offer, including improved message processing and acknowledgment mechanisms.

[music] >> Let's let's start. Good afternoon. Thanks for joining my talk. And today we're going to take a look at what's new in Spring for Apache Kafka 4. My name is Tim van Baarsen. I work for ING Bank. We're a global bank. We're active in more than 100 countries. We're serving more than uh 41 million customers around the globe. Uh we have 60,000 employees where roughly 1/3

is in IT. For myself, I'm in the global Kafka team. So we're a platform team. We're responsible for running Kafka in production. We're able to do this for more than 10 years. Uh and there in the bank there are around 1 1/2 thousand development teams somehow depending on Kafka in their applications. Uh and we're roughly processing uh 2 and 1/2 million messages uh a second that are

coming in into our clusters And the majority of our Java users are on Spring Boot and like to integrate with our Kafka services using Spring Kafka. So what's on the menu today? First, we're going to migrate to Spring Boot 4 and uh Spring Kafka 4. You might still be on uh Spring Boot 3 and Spring Kafka 3. So uh before we can leverage the new features from

Kafka 4, we first need to be on that version of course. And once we are there, we're going to take a deep dive look into Kafka queues. And we're going to also check out the new Kafka rebalance protocol that you can opt in now with uh Spring Kafka 4. And at the end we uh have some conclusions and wrap up. And if there is time left, we

can answer some some questions as well. So my goal for today is to uh show you uh code, running applications. Of course, there are a couple of slides to explain the concept here. Uh but you can apply this in your own projects using uh Spring Kafka in action in the demos. And you will also understand the trade-offs between the new features and whether or not you can

start using them in your own solutions. I'm from the Netherlands. Um you already saw in the agenda there are some KIPs. What's a KIP? In in uh In Dutch, a KIP is a chicken. But obviously, we're going to not going to talk about chickens today. It's all about the Kafka improvement proposals. And this is an official way in Kafka to propose new features, new ideas. Once they

get accepted and discussed, they will be implemented. And we will see uh two of the most important KIPs that came out with with Kafka 4 in action today. Of course, using Spring Kafka and Spring Boot. So um just a 101 on Spring Kafka. Um we all integrate with with Kafka in our applications. Uh for that, we are using Java. So we're using the Java clients. you have

the consumer API, the producer API. Maybe you're doing uh real-time streaming using Kafka streams as well. And on top of that, we like to use the abstractions in Spring, especially the ones in in Spring Kafka. And if we're using Spring Boot, there is a lot of auto configuration for all the factories that Spring Kafka is doing for us in the background. And then you have your own

codes where you're going probably to produce to a topic using a Kafka template. You have a method annotated with a Kafka listeners or a consumer. And you might use [music] uh the Kafka test library to do some embedded testing. So that's something you probably recognize from your own code. And today we will also see some new APIs in action, especially around Kafka So um in a nutshell,

this is what we're going to see uh today. So let's start with first a practical migration path. Spring Boot 4 came out of course last year. And also with that one, we will have Spring Kafka 4. And the main big difference uh is that we now have nice Spring Boot Kafka starters. And also for our test uh dependency, there is now a nice uh starter for uh

for Kafka test as But the most important thing is of course the underlying Kafka library because that will bring the new features for Kafka 4. So to leverage those new features, we need to migrate. I can of course show you a lot of uh boring slides of all the things that changed. But maybe it's better to just open up the IDE and we're going to migrate a

application directly into the new version. So let's start here. Um this is the code base I prepared for all the demos. So I have three demos today. So let's hope everything goes goes fine. So let's take a quick look at this application. Make it a little bit smaller. I hope it's visible in the in the back. Uh this is a Spring Boot application. Uh it's still on

uh 3.5.0, which is not bad. But we would like to migrate this to Spring Boot 4. And we uh see we have the uh old Spring Kafka dependencies in here. Um we are a producer and a consumer. And in this case, we're using JSON as a payload. So we're using Jackson 2 here. We would like to migrate that as well. We have the actuator in there. And

we also have the let's say old Spring Boot uh starter test. Sorry, uh the Spring Kafka uh test dependency in here for the embedded Kafka annotations to spin up an uh in-memory Kafka instance under test. yeah, we have a small uh consumer and and producer here. Um so let's go into the uh configuration a bit. We also in this application protecting our application against the poison pills

to not uh end up in an infinite loop if we're running into um a poison pill scenario where we cannot deserialize the message. So we're using the Spring Kafka feature for this. And uh we're of course to make use of that, we need to override some beans that normally Spring Kafka is under the hood using um Spring Boot. Um so let's run OpenRewrite. We are um created

a small example here. Um and if you don't know OpenRewrite, it's basically an open source project to migrate your code using recipes. And there are already recipes of course to migrate our Spring Boot 4 application. There is also a specific recipe for um Spring Kafka to migrate to that version. But it's basically included as part of the upgrade of Spring Boot to Um I noticed that the

recipe is not covering everything. So in this example, I also have a unit test that's still using Zookeeper, which is also dropped from uh Kafka 4. Um so I can quickly show that as well. Um so I needed So we have some Zookeeper properties here. I needed to create uh a small recipe myself to uh to do this. But it's uh it's fairly simple. I can quickly

show it here. I will share the the code base afterwards so you can also take a look. Um so for the things that are not covered, uh I I wrote my own recipes. But they are fairly simple. Uh for example, the um JSON uh starter was not replaced by Jackson. And some properties for the embedded Kafka uh were basically not taken care of. So by creating a

little bit of a YAML, it's pretty simple. You can say replace uh some properties and um some dependencies. But the rest is managed by OpenRewrite recipe for Spring Boot. So let's uh let's give it a start. Um so we're going to open up the uh terminal and we're going to run OpenRewrite. So there's the OpenRewrite Maven plugin is in this project. It will scan the project and

will take care of the migration. And then we're going to compare before and after. So it takes a couple of seconds to execute. It's validating the recipes that we specified in the Maven plugin. And it's going to execute the recipes one by one. Okay, it was fairly quick. Um if we scroll back, we can see all the recipes that OpenRewrite executed for us. And also next to

migrating automatically to Spring Boot 4, it also changed all the Spring Kafka related um uh code refactorings for us. So if we compare this, um let's take a quick look. Of course, let's start with the POM. We now have instead of uh 3.5, we are on the latest Spring Boot uh 4 version. We also have the uh nice new starters in here. Sorry. Um so we uh

let's make a little bit more room here. So we have the new Spring Boot uh Kafka starter. We have the new Jackson starter for Jackson 3. Um and we also have the new starter for the Kafka test dependency. Um next to that, there are also new serializers and deserializers. So OpenRewrite basically took care of that. So we migrated from the left side, from the JSON serializer to

the Jackson JSON serializer, which is basically using Jackson 3 underneath. And also to consume the message, we need to deserialize the JSON back into our object Java objects. So also here we see that uh yeah, it's it's migrated for us. Um also some package names were changed from Spring Boot 3 to Spring Boot 4. So the auto configuration package changed and that's also done for us here.

and next to that also our test got refactored. So we got rid of Zookeeper here. All the properties related to Zookeeper are not there anymore, so they are gone. And let's now start this test to see if everything is is still working. So this test is basically producing something to Kafka. The embedded annotation starts a in embedded Kafka instance and it's consumed back in the test and

apparently everything looks So we're now basically on Spring Boot 4 and we are going to use Spring Kafka 4 to take a look at the new features there. So that was very simple. Okay. Uh one thing to remember is that from a Kafka broker perspective, there is a new baseline for the minimum Java Kafka client that comes with Spring Kafka of course. And the minimum version is

2.1.0. And if we translate this all the way back to the version of Spring Boot, then we go back into version 2.2.0. So that's pretty old. It's from October 2019. There should of course be no reason to be on this version anymore. But that's on the other hand it's also still impressive that with a new broker that's just released Kafka 4.2 is just released last year, your

application is still working. Of course you cannot leverage the new features, but it's still there. But the minimum baseline for the client and basically for Spring Boot and Spring Kafka are those versions. So it's good to remember that. Now we're on Spring Kafka 4, so we can start using or investigating the new features that are released in Kafka 4. And one of them is queues for Kafka,

uh also known as share consumers. So let's take a look what we're going to dive into. this feature is GA, so it's production ready in Kafka 4.2. Be aware that the current version of Spring Boot and Spring Kafka are still on the Kafka library 4.1. So the next version of Spring Boot and Spring Kafka that will be released in May, next month, will use the underlying 4.2

Kafka clients that are coming with Um we're going to check what's the difference between normal consumer groups and share groups, when to use them, what are the tradeoffs. We will go through an example implemented with We going to check some important settings. We will also see some limitations. And we will take a look how we can acknowledge messages using Spring Kafka as well. And of course we

will start scaling the number of instances because that's the whole point of the Uh then we do a demo and we take a look at the tradeoffs. But let's first start Um in the middle you see a Kafka topic. It consists of three partitions. And by default from the beginning of Kafka, every partition can only be assigned to exactly one consumer in the consumer group. So the

consumer group is a group that works together to consume all the events or messages from the Kafka topic. And this is your fundamental scaling in Kafka. So we can scale up another consumer. There is a rebalance going on. The partitions are reassigned to each consumer. And after this rebalance is is done, the group consumes continuously messages from the topic. But if we scale up another consumer, this

one will be idle because we cannot scale beyond the number of partitions. And this can cause some issues and some limitations. So what we see in the bank that people that have to deal with a burst of incoming messages, always ask us to have more partitions on the topic because at some point we need to be ready to handle that message. So they overcommit the number of

But in most of the use cases this only happens when the stock market is open or on a Black Friday. And the rest of the year or month, those partitions are sitting idle in our clusters taking space, taking energy, taking uh also money, right? So that is basically what the share groups in Kafka are going to fix for us. So there is a new concept that's called

a share group, a share consumer. And the nice thing is that we can with multiple consumers from multiple partitions and multiple partitions can be assigned to multiple consumers. And we can scale beyond the number of partitions. So this is the first time in Kafka you can actually do this. So it allows us for dynamic scaling on the consumer side because that's what people actually want. They don't

care really about the partitions. They care about scaling and be able to handle this load when it comes in. And that's now possible with this new Another advantage is with the share groups in comparison with the consumer groups, if there is a message that's either a a poison pill or it takes a lot of time to process that message, you cannot go beyond that point. So we

call that head of line blocking. So the consumer is blocked at that point. With share groups, there is already another consumer in the share group that can already pick up in this case from partition two an event that's sitting at offset five. So it it it also allows for use cases where sometimes it takes a bit more time to process. So this is is more flexible than

the normal consumer groups. So you can see the share group basically as a pool of consumers concurrently consuming from that topic. Uh and now it allows us to decouple the storage, basically the partitions, from the compute. And it allows us to start dynamically scaling our consumers when needed. And it also brings Kafka queue semantics to Apache Kafka. let's explain this because you need to understand how this

works before we're going into the demo, otherwise it doesn't make a lot of sense. So let's now focus on one specific partition. And there is a concept of a share group Underneath it's still a Kafka partition, right? There is nothing different there, but this this is just metadata and a concept to talk about it. So like you already know, every message produced to a partition in the

topic has an offset. Nothing changed there. But each individual event or message written into the topic has a state. >> [music] >> And by default when it's written to the topic, it's available to be consumed. And next to that there is also a delivery count. So by default it will have the value zero once it's produced. Then let's dive a little bit into the states because it's

important to to know There are a couple of states. Like I mentioned, the first state is available. When the message are set of messages are delivered to the share group, it will be marked as acquired. So it's in flight and it's going to be delivered to the to the share group. Um then there is an acknowledgement. So if if a message is successfully delivered, it's it's acknowledged.

If there is an error, then it will be marked as archived and it will not be reprocessed. So this is an unsuccessfully processing of the there's also the delivery count. So when it is acquired, delivery count will be incremented by one. And there is also a lock. So the consumer has 30 seconds to process that message. If everything goes fine and in Spring Kafka, we will see

Spring Kafka will take care for you to acknowledge the message if your consumer method returns, then everything is fine. So this is the happy path. And the message is successfully By default, there will be retry delivery of five times. So it also avoids preventing infinite loops that's built into this into this feature Um also the state of the messages are maintained per consumer group, right? So one

consumer group can acknowledge the message while the other one says, "Okay, for this use case it doesn't make sense and I will reject it." It's just like consumer groups, there can be multiple share groups for different use cases. If we then take a quick look at the um failure scenarios, we can also um mark a message as uh for redelivery. So, maybe you're calling a downstream service

which is not available. So, it's not a transient failure and it will say basically to the broker, "Hey, I cannot handle this message now. Um try again later on." Um the other part is where you can explicitly reject a message. Maybe there is a data coming in and for that use case it's invalid or you cannot handle it and then it will be basically marked as archived

and there will be no retry uh and there is no permanent uh yeah, it will be permanently flagged as as failed. So, those are the states and we will see this in action in the demo. Uh so, when the consumer pulls messages from the share group uh partition then those messages are basically in flight. They will be marked as acquired. They are delivered to the consumer and

um one by one, so each message is acknowledged individually. So, that's also different from a consumer group. So, in this case everything goes fine. So, the messages you already see here are not handled in order as well, right? That's really important to understand. They are handled out of order. But concurrently by all the share consumers in the group. And then this sliding window moves forward and that's

how the messages are consumed from this And here we see some messages are handled pretty fine and other ones are maybe archived um after a couple of retries because they could not be uh handled. So, that's in basically how it works. Um in the demo I'm going to show you actually the uh Spring Kafka 4.1 snapshot because we found some bugs in the current version. So, to

show a nice demo uh we will see this in action and they overhauled also the acknowledgements in the new version. So, for me it made more sense to show you absolute latest version and that will be released um end of May with Spring Boot 4.1. So, then it will be will be there. Uh we talked already about the acknowledgements. So, Spring Kafka will take care of the

acknowledgements for you. So, if your method uh returns successfully, the message is accepted. If there's an error, if there is any exception, your code is throwing exception, then it will be automatically marked as rejected and it will not be retried. Um there's also an implicit mode and then basically the Kafka broker um acknowledge this message for you and it doesn't really matter then on a success or

on an error, everything is accepted. For me it's not really useful to be honest um but I think the most important one is the manual way of acknowledging and then Spring Kafka gives you an API to say, "Hey, I accept uh a message or I reject one or I will release it so it will be uh redelivered again to another um share consumer in the group. Uh

so, you have more fine-grained control over when you accept or release or reject um those events. Okay, let's now show some code and run the demo to see how it works in action. I'm going to show um some important configuration settings. We will see the acknowledgements in action as well. We will scale beyond the number of partitions. So, the in the demo the topic will have three

partitions and we will scale five or six uh share group instances uh um consumer instances in the same application. So, if you have multiple instances, there will be even more um share consumers in that uh in that group. Uh and we also will check some limitations that are still today. Okay, let's open up the code base. Um let's also uh start uh single node Kafka cluster here.

So, let's quickly go into the second demo. Yeah. So, I'm running a uh single node Kafka cluster here with a schema registry for Afro, but that's a bit little bit outside of the topic of this talk. And let's now open up the code base uh for this demo. Um and we have a producer so we can already start the producer here. But of course the most important

part of the code is in the uh share consumer. So, like I mentioned, uh we are on the 4.1 snapshot which brings basically Spring Kafka 4.1 with all the fixes. Um yeah, the rest already showed uh in the in the other demo for uh for open rewrites. I'm not going to waste your time on on that one. Uh but let's now take a look how we can

actually start using those share groups in in action. Uh when I start with uh with a new Kafka project, I always start with my properties, but unfortunately there is no auto configuration yet. my first try was, "Oh, maybe I can use this reuse the consumer properties." But there are some properties you cannot configure for share groups. So, that throws an exception. So, okay. The only thing we

can do now uh today is basically uh create some beans ourselves. Of course, in the next versions of Spring Kafka and Spring Boot, this will be solved. This is also a first version. So, bear with me, but um for now we need to create a little uh of configuration here. So, that's the first limitation. the concept is more or less the same like normal uh consumer groups.

So, we have to have some configuration, right? To connect to our broker. And we need a share consumer factory to create the underlying share consumers and Spring Kafka is taking care of this by creating this bean. So, it's just a couple of simple properties uh here we have and then we can make this uh consumer factory. In the code base, I have examples for each uh acknowledgement

mode. So, implicit, explicit, and manual. Uh because we don't have a lot of time today to show all of them, I will show you the manual way how to acknowledge a message. So, let's take a quick look uh how we can how we can do that. So, one other bean we need to create is the um share Kafka listener container factory and that's basically creating the listener

containers that we will reference in our Kafka listener annotation like you normally do with And here we see some uh important properties um because by default the acknowledgement mode is explicit and in this case we would like to have more fine-grained control over our acknowledgement so we have to set it to manual. That's the first important one. Um the concurrency by default is one. So, there's one

share consumer in a application instance and just for the sake of the demo we would like to show you that we can scale beyond the number Also pretty important, if you're acknowledging the message yourself, you need to make sure that you either accept, reject, or acknowledge a message. Otherwise, you will basically not move forward. And um this can be quite annoying uh but there is a nice

property to set that Spring Kafka detects if you're not doing the uh acknowledgement in the way it should be and then it will print a warning. So, at least you're aware that you're not uh yeah, in your code path not acknowledging um messages or rejecting. Um um there's also property to um commit back the state to the broker, right? Because that takes care of it. You can

do this asynchronously or synchronously. Uh by default it's it's synchronous. Uh it's a bit slower. but you can also do this in async way, but then you have to be aware if your consumer dies before it sends the uh offset uh or the the state basically back, then you will receive that message again and um yeah, you have to be aware of that. Another limitation is that

there is no observability yet. So, there is no distributed tracing with Micrometer yet and there are also no metrics exposed yet, although the underlying Kafka uh has those metrics already, but they're just not exposed using uh Spring Boot and Micrometer. Yeah, so let's uh now take a look uh how put this all together. So, we have a small uh component this is like you're used to. Use

the Kafka listener. You can set an ID for this listener. You going to consume from our uh transactions event topic. And here you see here are we referencing to the actual container factory that we had to create manually because Spring Boot doesn't have auto configuration yet. Um and we can also give it a group. This works similar like normal consumer Uh and if you want, you can

also override the acknowledgement mode uh and the concurrency using the annotation. So, you can do it either on on the factory or you can also do this um [snorts] Um because we are using the manual mode, we have access to the share acknowledgement. And this is the API that gives us fine way way of controlling whether or not we acknowledge. So, the service is processing the incoming

event, and then here we basically acknowledge that message, and we log it for the demo. Also, in the new version that will come out in 4.1 in May, there is also an API to renew the lock. So, if your consumer needs a little bit more time than 30 seconds, it can also call this renew method. So, it basically tells the broker, "Hey, I'm still busy. Don't throw

me out. I I need a little bit more time. And I don't have an example for this, but that's also available in the next version of Spring Kafka. And then here is an example if there is an exception we can recover from, we can release the event. And in case where we can basically not recover from, we will reject the event. So, we actually really acknowledge message

by message. So, that's really different from a normal consumer group. All right. Let's let's start this application. let's now go into the requests. I expose a small REST API for the purpose of the demo to trigger a couple of scenarios. So, let's first go into the scenario where everything is fine. We are here processing payments. Um they're basically individual pieces of work. So, we can do this

out of order. So, let's uh see what is going on here. That's the other example, sorry. take one more look here. I probably start the wrong demo, sorry for that. Okay, so we started the producer, and we are not now going to restart our share And let's now produce some traffic on the on the topic. So, we're first going to go into the happy path. It was

uh successfully processed. So, here we see we acknowledge the message. Then let's go into another failure scenario where we have a negative amount. Of course, this is not acceptable, so our service will reject this, and then our code will explicitly um reject the message as well. And in this case, of course, there will be no redelivery to another consumer in the share group because this is just

invalid data, so we um move this one out. So, it's rejected, and it won't be retried. Um for the purpose of the demo, there is also a 50/50 chance that this message will be either accepted or rejected. So, let's also run this example. Um of course, it And let's see. And here we see um that for the purpose of the demo, the first time it was basically

released, we see it's in offset three. So, that message got redelivered to the to the group for reprocessing, and this can take up to maximum five times. After the message is basically rejected as So, that was the demo about Yes. Some trade-offs. normal consumer groups are still the way to go for event streaming and event processing, especially if you're of course using stateful processing with Kafka Streams.

Um consumer groups are still the way to go. Um but it's an interesting feature that Kafka now brings a more point-to-point like queue like messaging. This is only for units of work. yeah, more consumers than partitions, so we can we can scale, we can handle this in parallel, and we have more flexibility in in scaling. It also gives us the queue semantics by explicitly uh acknowledging messages.

But there are also some trade-offs. If we compare the consumer group with the share group, there is no guarantee in ordering. So, this is really important if your use case requires ordering, then the share groups is not suitable for your use case. Um this goes a little bit deeper, but if your Kafka broker is deployed over multiple data centers or ACs, then with share group with the

consumer groups, if the consumer is deployed in the same data center, it can actually fetch it from a follower. So, it will not go through the primary partition or the leader of the partition to fetch the data. So, there is less AC traffic between the data centers or the availability zones. With share groups, it's not yet possible. So, if you're going to use this, you might run

into some extra cost because it will go to the other data center if the leader of the partition is sitting in on the other side. Also, statically assigning partitions to consumers is of course not possible. Uh you have to be aware of this. And at the moment in time now, you can also not use a regular expression to consume from multiple topics. That's not supported with share

groups as well. Um and you can also there is no API to consume in batches. So, with your normal consumer group, you can either say I would like to have from a coding point of view just one message by by the other or a collection of it, and that's also not possible with the with the share So, that was the Kafka queues and share groups. Let's now

dive into the other um yeah, big tip, and that's the next generation of the consumer protocol. And to be completely clear, we're going now back into the normal consumer groups, but in the consumer groups, there is also an improvement done in So, let's talk about that. there are now basically for normal consumer group two rebalance protocols. Uh the one is classic that you're used to, and there

is also now a rebalance protocol that's called But let's first understand how it works under the hood. Uh because with the classic consumer protocol, yeah, it's it's basically to assign partitions to consumers in the group. And a rebalance happens when a consumer joins the group, leaves the group. This can be a crash, it can be a rolling deployment, or if there is something changing at the broker

side. For example, if the number of partitions is changed, then also those um partitions are reassigned to the consumers in the And generally, this is a quite expensive operation in Kafka. Um and in some cases, depending on the configuration, especially with older versions of of Kafka, basically the whole group stopped consuming, which is quite annoying. Uh and it can become quite problematic if you have topic with

a lot of partitions. So, if we're overcommitting partitions, then the more partitions we have, the longer it takes to rebalance because it needs to hand out all those partitions again. Of course, there were some nice improvements in Kafka 3 already to minimize this, but still it takes some time. And also, if you have larger consumer groups, then it also takes more time. So, there are use cases

that cannot really afford those longer interruptions of consumption, especially if you're in environments where you either deploy quite often or you do auto scaling. So, maybe you're also exposing a REST API, and by bringing up another instance that also is a Kafka consumer, you trigger basically a rebalance. that's why the new rebalance protocol for the normal consumer groups was introduced in Kafka 4. And a lot of

this assignment logic that was basically in the client is now moved into the broker. So, a lot of this group coordination back and forth and the barrier of yeah, make sure that all the consumers in the in the group are back is uh is basically gone. So, there are no stop-the-world rebalances anymore there. Um and you can leverage from really faster rebalances in the group. To start

using this, we need to specify group protocol. So, with the classic one, it's classic, and with the new consumer protocol, it's consumer. The nice thing is that this is backward compatible, but the default in Kafka 4 is still classic. So, you have to opt in, you have to configure your Spring Kafka consumer application to start to start leveraging this new rebalance protocol. But at some point, this

protocol will become the default in in Kafka. Probably it will be Kafka 5. >> [clears throat] >> So, let's see how this works. And I'm going to show you also in action, a live demo where we're going to do a rolling upgrade without any downtime, and we're going to migrate the group from an old protocol, the classic protocol, to the new one without downtime. Um so, we

start out with three consumers in the consumer group, and they're all configured at the start using the old protocol. And then, one by one, we're going to scale up the same application with a different configuration for the uh protocol to use the new one. And one by one, we will start new instances. And the nice thing is that the Kafka broker detects that a new consumer with

a new protocol joins the group and automatically converts all the things in the background into the new uh the new protocol. So, you can migrate this without any downtime. And then, one by one, we will start the new instances. If something goes wrong, there's also a path back. So, if you're migrating and at some point in time you figure out, "Hey, this is something is off, or

you would like to go back." You can also go back. So, once a consumer with the old protocol joins the group again, it's automatically transferred back into the old protocol. So, there is not only a path forward, but also a path a backup plan. this. We can we can stop those two applications, and we go to the last demo before we going to Um so, I already

dockerized the consumer and producer application. So, let's start that. Let's uh take a quick look. Yeah. So, we restarted Kafka. We have clean topics. Everything is is fine And we start three instances of our consumer application using Spring And they start consuming from this topic. There is also one instance of the producer, so it's constantly creating a stream of of data that's produced. And to emphasize a

little bit in the demo, we will see that this application is configured with the classic protocol. And after a couple of rebalances that are taking place now, all the consumers have just one partition to consume. a tab on the broker, so we can actually see what type of the consumer group we are running. I hope it's big enough in the name of the group of the consumer

group is my consumer group, and it's of type classic. So, we're still in the old style of the protocol. But let's now spin up one instance of the new protocol. And we can do this by basically using the Spring Kafka property for the next to the three classic ones, we will spin up one with consumer protocol. So, we're going to opt in on this one. Um let's

see. So, we're going to do the live upgrade Let's open a new tab. And we start this instance. So, now the instance is started. It joins the group. But like I explained, this is the third instance, right? So, it will not get any traffic. But the broker should already detect. It misses a C, but you get the picture. I don't know why. Maybe because I zoomed in

a bit, but it automatically transferred our group now from the classic one into the new one. We didn't consume anything yet, but it already detected it. So, now we can basically stop one of the instances. Of course, there's a rebalance going on. And at some point, this new one will start receiving traffic from that topic. this one should use the new consumer protocol. Now we're confident we

can uh stop more, and we can now scale up to of the instances here. so, let's let's make it three. So, we're actually basically doing a a rolling upgrade here now, simplified with Docker Compose. Also, the idea is to not leave this running for long times, right? The idea is to do this as soon as possible. If you feel comfortable, stop the other. So, don't mix new

ones and old ones in one group, because there is some overhead, of course, on the broker as well to Um so, now we stopped all the classic um in Spring Kafka, and everything is now running on the new protocol here. So, without any downtime, we are now on the new type of the consumer group, and we leverage the new features. And have faster rebalances as well. So,

that's what I wanted to show We saw it in action. a couple of limitations and trade-offs, because there is no free lunch. Although it was nice to see the migration happening without any Because a lot of those um logic for the rebalance is now moved Some configuration properties will not apply on your consumer anymore. And there are also new metrics to observe your consumer both on the

broker side, but of course, also on the Spring Kafka side. The nice thing is that those metrics already exposed using Micrometer So, but you have to do some work there to change your observability stack to Um and like I mentioned, a couple of of the properties are now moved into the broker, so you cannot specify them on the client side anymore. And one of the most important

ones, if you're using custom assignment strategy that you implemented yourself because your use case requires that, that's something you cannot use anymore, because the assignment is completely done on the broker side. So, be aware of that you first have to get rid of that before you can leverage it, or decide to stay with the old protocol. Okay, we are nearing the end of the of the talk.

So, let's wrap up. Yeah, we saw Let's see. I think I'm missing one slide. Anyway, um We saw the Kafka queues in action. Spring Boot is still using the one that's in preview. Um so, I if you would like to use this feature, then use this feature in combination with the upcoming Spring Boot and Spring Kafka release dropping in in May. Also, be aware that the share

groups is not a replacement for the normal consumer group. It's just an alternative way of a consumption model, and it gives you the queue-like semantics. Um be aware again of the trade-offs. There is loss of ordering of messages. Um it's not supported to follow to consume from a fetch from a follower. The static partition assignment is not possible, and also subscribing to topics using a regular expression

is not yet Uh you saw it's it's easy to get started with Spring Boot. It's more or less the same programming model you're used to. But it's also really an early version. There is no auto configuration. There are no metrics exposed for share groups. And also, distributed tracing is not yet yet working. Also, be aware there is no way to set an exponential back-off. So, if you

release a message, it will be immediately consumed by another consumer in the share group, right? So, there is no way to say, "Okay, hold on for maybe 10 seconds." That those features are not there yet. And after five times a message is basically gone. And there is also no dead letter topic support to write that message into a dead letter topic, so you can inspect this later.

So, it's really an early version, but it's nice to play around with it and see if it makes sense for your Kafka use cases. I think I have should start with this slide, but anyway. We also saw we migrated Spring Kafka application. Of course, it's a quite simple application, but it's it's pretty straightforward. I always like to do this once myself, then use OpenRewrite and write some

recipes myself to do this for other services. For the next generation consumer rebalance protocol, it's backward but you still have to opt in using the configuration property to specify the The protocol is way more resilient than the old one, the classic one, because it shifts a lot of complexity to the broker and also makes your Spring Kafka application lighter. We saw in the the demo that live

upgrades are possible. There is no downtime needed. And from that point on, you can use the fast rebalances that make your Kafka consumers more reliable. On the other side, custom assignment strategies are not supported. And there are new metrics, so this will have impact on your observability. This was the end of the talk. If you would like to play around with all the demos, yeah, this GitHub

repository is there. If you spot any improvements, feel free to raise a PR. Thanks for joining today. >> [applause]