Julien Jakubowski: Apache Pulsar: Finally an Alternative to Kafka?

This talk introduces Apache Pulsar as a viable alternative to Kafka, focusing on its unique features and advantages. The speaker discusses the limitations and pain points faced by Kafka users, such as scaling issues and high operational costs, particularly in maintaining clusters. Pulsar is presented as a cloud-native messaging and data streaming platform that supports elasticity and real-time data processing. The talk covers key concepts like data streaming, messaging, and the need for systems to handle high throughput, order processing, and efficient resource management. The speaker emphasizes that Pulsar combines messaging and streaming functionalities into a single platform, allowing for seamless scaling and efficient resource usage. The discussion also highlights Pulsar's architecture, including its use of broker and bookkeeper nodes, and its support for multi-tenancy and disaster recovery. Overall, Pulsar is portrayed as a robust solution for modern data streaming needs.

[Music] hello again and uh here with me I have Julian yakubowski who's going to talk to us about Apache puls are finally an alternative to Kafka question mark let's see how the question is going to Aral in the next 40 minutes and um over to you Julian thank you so hello everyone thank you for attending my talk so today when you need a data stream platform what

probably comes to mind is abashi kakka and uh if you've been using kfka for a while you might be facing some you know some limitations or pain points in your daily operations such as a lack of flexibility when you need elasticity when you need to scale and additionally maintaining your CFA cluster can be very expensive in terms of network and storage cost and also if you're using

traditional message bus Solutions like a rabit mq for example you might also be facing certain limits but I have good news for you there is an alternative that is gaining popularity apash PSAR by the end of this talk you you will have discovered how PSAR differ from kfka and traditional mq Solutions and hopefully you will want to try it out for yourself to make your own opinion

so a few words about me I'm developer advocate in Europe I work as TR native managed apash provider I've been a software engineer for more than 20 years now mostly using Java and spring I've been distributed scalable and evren systems for several Industries including retail and financial services I live in Lily in north of France and I'm one of the leaders and founders of several developers communities

such as the jav user group of Le for example uh so feel free to use the QR code to join me on LinkedIn and you can also follow Twitter so what is Apachi Apachi is a cloud Native Messaging and data streaming platform so well I'm throwing out some jargon here but don't worry I'll explain it right away so Cloud native means that P is designed for running

in containerized environments it's designed to run on doer and kubernetes and all that stuff but Cloud native also means elasticity on an elastic system you can add resources quickly to adapt to a growing workload and you can also remove resources when the workload decreases to save on cost you will see in this talk that elasticity is one of the main things that make p stand out and

pan is both a messing and a data sing platform as we will see in this presentation so I'm going to explain those words as well what is messaging and what is data streaming so p is a messaging and streaming platform but what is data streaming data streaming is the process of transmitting data continuously and in real time from a source to a destination so the data is

sent in small pieces and it's processed or analyzed as received these small records these small pieces are often called events EV because they typically represent something that happened in the past for example my stock level at 10:00 a.m. was 42 and at 11:00 a.m. it was 41 so these are two events and in the context of data streaming the events must be processed in the correct order

they are interl if they are not processed in the correct order something will be broken and bugs will be created these data these events they may be transformed aggregated or replayed replayed and this is why you may need to persist the data for a very for a long time you may also need to ring millions of Records very quickly and perform big catchup breads right so you

need throughput these platforms can handle large amount of data and can process thousands of messages per second so they can scale horizontally which means they can manage workflows that are too long for a single node to process and they do that by spreading the work Cloud across multiple nodes so the goal is to process these data streams and typical stream processing use cases include data ingestion that's

the process of collecting and importing data from various sources for storage or processing and analyzes and other use cases this can be about real time analytics real time analytics is about you know making the right decisions based on fresh data so you know that's when you don't want to wait for a batch process to be run tomorrow you need to make the decision right now on fresh

data for example if you need to implement a dashboard that is updated in real time or when you need to implement fraud detection right so the most common streaming platforms is Kafka but I can also mention Amazon kesis in the manag world and of course poar is a great sing platform now let's talk about messaging and messaging message Queen is quite different there are some similarities there

are producers and consumers and a first in first star structure that we can call Q of topic but the goals are completely different let's say that sometimes a service needs a task or several task to be done but it can't run then this task itself for various reasons maybe the task takes too long or it can be implemented within the service for many reasons for example this

could be uh I don't know an e-commerce website an online selling website that sometimes need to perform a long running payment task or let's say call an AI right some long running task so the service can delegate the task execution to another service by sending to the this service a command so the service standing the command is the producer and the one executing the command is the

consumer and the producer can't wait for the task to be completed because this task might take a while and could also fail that's why we need to decouple the producer and the consumer and we also need to ensure reliable execution making sure that the task eventually gets done even if it initially fails to do that you can set up a work queue so this queue contains messages

from the producers and each of these messages is a task to be performed by consumer instances so message broker manages this message CU accepts the producer messages and delivers them to the consumer instances so they can perform the task the broker distributes the messages among the instance to balance the load across them so task one will be consumed by instance one task two is consumed by instance

two and so on and you don't need to perform this task in a strict order for example let's say the task involve um payment validation for an online store for example and let's say that you have two customers Alice and Bob and Alice place to know two seconds before Bob then it's not a big deal if I validate Bob's payment first it's not a big deal right

unlike when you stream data because when streaming data breaking the order cause problems but in message queing it's not such a big deal so it means that the expectations for message queing are different from those for data streaming so Q qes can grow faster than consumers can handle so one expected feature is the ability to add new consumers to consume the queue faster and also remove those

consumers when the cost another expectation is when a consumer is busy processing a message you don't want to wait for the processing to be completed the broker will deliver the next task to another consumer stands without waiting for the Cur task to be complete when the consumer fails to handle a message because you know there is an error or timeout you may need the message broker to

redeliver the message later to retry the task right and you also need to remove some messages that can be you know undeliverable these undeliverable messages you need to remove them from the queue and move them to a dead letter Q so because of that messages May then be consumed in a slight different order than produced but as we said the order is not a strong requirement here

so rabit mq and Amazon sqs match these requirements these semantics because they implement the message Q semantic and we cannot expect the requirements from a streaming platform these platforms are non the for that however P implements both the streaming requirement and the message queing requirements as we will see so data streaming and messaging require a different set of features so for the data streaming requirements you need

the data to be processed in order that's not a big deal in the messaging world for the data streaming requirements you need to inest large amount of data you need data retention at RIT mq and other traditional message broker are not designed to retain large amount of data gafka is designed to retain large amount of data but not quite as we will see later um kfka is

designed to handle cat up GDs that's not the case for bmq because they are not designed to retain a large amount of data and on the right side you need to add or remove consumer dynamically kfka not very good as that for doing that on the right side for the messaging queing you don't want to block the queue when the consumer is busy or FS so it

means that rabit mq is quite good as managing that that's not quite the case for kfka and you also need to redeliver FID messages later so on the left that's where kfka is good at while on the right that's for rabit mq active mq and all those traditional message Q broker so many companies end up with these two different Technologies for these two use cases meaning two

different broker Technologies to operate in production and two different SDK to learn for developers so now you can say well why not use kfka to implement a messageq because you know kfka is also about producing and consuming messages right so you may think using a stream platforms to implement a work Q is a good idea I used to s that but when I try to do that

well that look looks a bit like this right so I've done my best but I realize that kfka is not designed for that so let's try to do that together right now but first let's explain how it works with an mq broker like rabit mq or active mq so here we have a producer and a consumer we are in a c scenario with a single consumer instance

consuming message from the Que now what if I need to consume my messages faster I will then add a new consumer instance the message broker will then distribute the delivery of messages among the instances it's automatic I don't need to do anything on the broker side I just need to scale up my consumer system or to scale out my consumer system now the messages are consumed twice

as fast because I have doubled the processing power on the consumer side so that's that's pretty straightforward what if there is an incident now for example a message causes a consumer instance to crash or the task corresponding to the message long then I'll be able to re try delivering the messages easily the messages that comes after the problematic one will still be processed so the cube won't

get blocked the broker will just try to deliver the faulty messages again a bit later so this might cause the messages to be processed out of order but as we mentioned earlier that's not a big deal for messaging unlike data streaming okay and what if a message is completely invalid it's a message that I will never be able to consume for example because it has I don't

know an invalid format or because I've retried consuming too many times in that case the broker we remove the message from the que and move it to a dead letter Q so the que is unblocked the main que is unblocked and we can review the faulty message later to investigate and to figure out why it wasn't consumed now let's try that with Scala so we start simple

with one topic composed of one single partition and one single consumer instance I'm not including the replicas in my diagram to keep my explanations as as clear as possible so now let's say that I need to consume the queue faster because the queue is growing too quickly then I need to add a new worker a new instance a new consumer to the CFA consumer group however when

you need to add a new Active consumer to the same consumer group in kfka then you need to add a new Partition to the topic because you can't have more Active consumer instances than partitions however adding a new Partition to a topic is far from being trivial it involve reorganizing the data in the topic and if there is a lot of it this can be a long

tedious and costly operation it also requires an operation on the broker side which is often managed by another team meaning that we need to coordinate with that team and if and that's not always very quick on the other hand with a broker I just need to add a new consumer and that's it now what if a message can't be consumed because it's invalid meaning I'll never be

able to process it this situation is called a poison P it blocks the queue and there aren't many options in CFA one approach is to manually adjust the consumer group ofet to try to skip the invalid message but that's a bit tricky and often that that's of a manual operation manual process so the other option is to add the new consumer however to scale out consumers I

need to add a new Partition on the broker side which brings us back to the same problem before okay so what if I plan two partitions that should prevent these issues right or maybe not so now I have my two partitions and my two active consumers we are about to find out if problem is solved or not let's say that one message blocks again then task from

partition one will be processed which is better but partition zero is still blocked so you have to wait for Consumer zero to finish before processing the remaining task in Partition zero so the problem is not really solved here all right so I intended to illustrate that using a data swing platform for message queing is not ideal these platforms are not designed for that this is not the

same use case not the same semantics messaging is what message Brokers such as RIT and Q are for does it mean that you have to choose between messaging and streaming or do you have to manage two different platforms one for messaging and another streaming well with Brar this is not a dilemma you can can do both using the same platform the same Technologies so you have only

one hdk to learn and you have only one broker to manage in production p is indeed a unified messaging and streaming platform this is one of the key features of Brar now I want to talk about Brar scalability and elasticity scalability and elasticity is different elasticity means you can grow or Shrink resources quickly to adapt to workload changes so we can save infrastructure cost by avoiding over

provisioning so some data s platforms like Kafka and plar can scale very well but plar is both scalable and elastic the scalability requirements are determined by the bottle neck you have to address at a given time so if a bottle neck is the number of messages remaining to be consumed in a topic then you need to scale on the consumer side when the bottleneck is the number

of topics or the number of connection with clients then you need to add more processing power to the Brar cluster and when the Bon neck is the storage capacity well then you just need to add more storage capacity to your to your right so let's dive into the first bottle neck which is likely to be the most common one you will encounter suppose you have multiple consumers

consuming a topic what happens if the number of messages to be consumed grows faster than the consumers can process them then with P you just have to add new consumers to what we call a shared or key shared subscription so you can increase the throughput by adding new consumers with scafa when you need to consumer group you need to add a new Partition to the topic that

can be painful as I said so you have to anticipate and plan carefully butar doesn't have this issue and of course when you have few messages you may end up with underutilized consumers you can also remove them to sell infrastructure cost moreover and that's super cool if you use the Brar key shared subscription feature you can benefit from this elasticity while preserving the ordering warranty so you've

got the best of both worlds right unlike traditional messaging Brokers that is super cool now that we explore the most common bottom neck let's explore how prar can handle a rapidly increasing number of producers but first I need to explain the unique architecture of P this architecture is more sophisticated than other platforms so it brings many benefits in BR there are two types of nodes the broker

nodes and the booking nodes the broker nodes are responsible for managing all the communication and the processing of the topics so they are stess they don't store data so broker not delation won't impact the data in contrast The Bookies notes are responsible for storage they have state they store the messages The Bookies are apachi bookkeepers notes so let's say that I need more processing power on my

cluster I'll add more broker nodes and that's it because their state is stored in the bookkeeper here I can add a new broker node and that load is migrated to another broker node so basically you just you just add new broker nodes and boom your number so some of the magic of P that it will take care of all the moving of connections and the moving is

transparent to your application if you compare this to kfka with cafka you will add to add a new worker you will had to manage the movement of all the data to another broker for it to rebalance it because in cfast storage and processing are coupled in contrast with P that's decoupled and you just have to add the new brok owners and voila it's done much easier and

when when we need to store more data then we'll just add more bookies as soon as you add a new booking note it's going to be eligible for getting new messages right away it's immed immediately becomes functional there is no data movement PSAR does not have this issue when you need scalability on your data so how is all this possible is it magic nope it SS to

the partition free storage model of P so let's me explain now as a reminder here is the CFA storage model so messages are sent to a topic which is the first thing style structure that hold these messages the messages are distributed across multiple partitions each partition is stored and managed on a separate node to enable scalability gafka guarantees that messages are consumed in the same order in

which they are written to the partition new messages are distributed across partitions either in a run verbing manner or using a key in the latter case or all messages with the same key will end up on the same partition allowing messages with the same key to be consumed in the correct order so what does this design imply well imagine you have a broker in this situation a

cluster in this situation CFA cluster let's say that you have three topics one is divided into three partitions the second is also divided into three partitions and the thir has only one partition at this regard you know replicas for the sake of simp Simplicity now let's say that you need to add a four topic that will consume quite a bit of data the topic D here in

red the topic is growing it keeps gr and then what happens the left note crashes because its dis is full but it's a shame because the total available storage capacity on the cluster is actually enough to store the data but this issue happened just because of how the partitions are spread noes with scaa I often have to worry about how partition are distributed across different dis and

noes I need to worry about these details I need to manage and anticipate this problem there's a lot more planning indeed there is a lot more planning and needed with scfa compared to BR so life of a CFA administrator can be a bit complicated what if I could have done this what if sorry what I could have done to avoid this problem that would have been to

move topic C to another node before adding to p d depending on the amount of data involved this can consume a lot of network and our resources and take a lot of time but I would have avoided the problem because now I can add the topic without any issue now let's talk about the poar model this model is completely different usar doesn't distribute data into partitions but

only into segments new messages at the end of the topics are written into a segment when the segment which is a certain size it is closed and a new segment is immediately created to store this segment ring allows for some pretty cool things as we will see so here is how Brar will have handled handled this previous situation remember topics a b and c from the previous

situation rousar will have divided these topics into multiple segments and Brar would have distributed these segments evenly across the different nodes and automatically so as a result when it comes to a accommodating my new topic D it's very simple P will also distribute it automatically and will be able to use all the the available storage space in closer this in this way the problem I had previously

with cka doesn't exist anymore when I use p as long as you have enough total storage space it just works much less management to do much less hassle so let's try now let's try scaning a CF cluster in the case where there is not enough space avilable on the cluster to accommodate new data so we need to grow topic D the red one the red one so

all I have to do is add add a new node and that's it right well no not at all it's not that simple if I let my topic grow this is what will happen an incident in reality I shouldn't have done that let's roll back to the previous situation I should have followed a multistep plan first of course I should have added my new and empty node

to the cluster but then after adding my node I should have reassigned the partitions of several topics as if I were you know playing Tetris you know the Tetris game to do this I have to split topics four into four partitions inste of three and I also have to move some partition to the new node to get a balanced distribution of the data and only then can

I start to accommodate my new data and the problem with that is that all this partition reassignment operations all the data movement can take a long time we're talking hours or even days for the largest clusters it has a very strong impact on the performance on the entire system because it consumes a lot of network and IO resources and since it's consume resources it can also be

very expensive so it's a shame with Spa it's completely different so as a reminder when you need to add more processing power you add a new worker node and boom it's done and when you need more storage you add a new booking node and boom it's done breting forward in the previous situation when it comes to providing more storage space for new messages in topic D since

new messages are added to new segments the new segments are open on the new node and we have the storage space immediately available we don't need to play tetrace to move partitions around we don't have those problems with SP so now let's talk about how data is replicated because there are no partition replicas but instead there are segment replicas those are attribut these replicas among the bookkeeper

notes so in my example here I have a topic containing five segments and I have configured the replication factor of three this means that each segment is replicated on three different nodes so if I lose a node I haven't lost any data in my example segments for four and five contain data that is currently being written or read and this is recent data that needs to be

read in real time with the best possible latency so it's beneficial to store it locally on high disk however the other segments contain C data that will be read later by a bat process for instance in several weeks or even months we don't have the same requirements for this it's C data but it's a waste to use high performance and th expensive local dis to store this

C data once the data becomes cold it advantageous to move this data out of the cluster and transfer it to cloud storage like S3 for example which is much more cost effective and virtually unlimited this things is called teered storage so unlike the open source version of CFA the teed storage impulsa is production ready and has been used in production for many years so we've seen what

happens when you lose a node but what if you lose a whole region or a whole data center never happens right yeah it does happen that's where GE replication steps in Z replication provides disaster recovery so we have several clusters deploying different regions or different data centers and if you lose a region you can recover from it RAR can replicate the data to different regions automatically in

a bir directional way and this is a buil-in feature so you don't need Myro maker or cluster link LinkedIn or any other additional software setting this up is basically configuration now I'd like to introduce another very feature of P multi tency multi tency allows different departments or teams within an organization to share AAR cluster while keeping their data isolated for example I don't want the marketing team

to have access to the finance teams data and if one team consumes a lot of resources I don't want that to impact other teams using the same cluster when using Kafka we tend to serve this by creating a different cluster for H Team but that's not cost effective the benefits of multi tendency includes easier management as you only need to operate a single cluster for multiple teams

plus sharing resources can lead to a significant reduction in the number of nodes in your infrastructure which save on cost prar is designed for that this is a built-in feature so now you could say well Julian you know P has impressive features but in my company I have an existing software ecosystem I'm sending or consuming messages with scfa rabbit andq I have a bunch of microservices and

I can't rewrite all of them to migrate P well I have good news for you to has a high level of compatibility and I'll explain that right now so messaging and streaming involve clients and a broker they communicate using a protocol Brar provide its own binary protocol but with the addition protocol handlers Brar becomes compatible with scafa clients R mq clients and mqtt clients so by leveraging

your existing apps you can avoid the need to rewrite everything and ensure a seamless migration path to Rite client application using the Brer native protocol and benefit from all features you have many Brer cence libraries available you will surely find one for your favorite language so don't hesitate to check out hub. stream native. where you find the wide ecosystem of connectors libraries Etc when it comes to

choosing a technology open source is key some of the benefits of choosing an open source technology are sustainability avoiding vendor locked in and community support and plar of course is open source because it's Apachi PSAR all these features are presented are available in open source so if you download P you'll have all of them this is great because you don't depend on a specific vendor you're free

to call a vendor to provide P the service like native or you can manage a p by yourself there is no vendor locking now about the community there are more than 600 contributors to Pur and there are more making contributions to the ecosystem surrounding ground P the entire prar coas in growing year over year that's amazing the number of slack members is about to reach 10,000 actually

already reached 10,000 there is a really rich and growing body of knowledge of questions and suggestions so who uses P Brar adoption is quite large with more than 1,000 companies using it all around the globe okay so now let's take a look at the history of Pulsar glance Pulsar was developed by Yahoo in 2012 as their Cloud messaging service to overcome both Kafka limitations and traditional M2

limitations it then went open source in 2016 by 2018 P had graduated to a top level Apachi project since 2019 there's been a surge in the prar community's growth with rapid adoption and contributors and it's worth mentioning that P has been in production for over 10 years this means that it's a battle tested project it's proven to be robust mature so here is a quick recap prar

is a UniFi messaging answering platform handling both patterns both semantics at the same time so you can have only one platform to manage Brar is doing great at both scaling and being elastic because of the separation between compute and storage and because of the partition l storage modelar ensures the durability of the data and can flow to external cheap and unlimited storage Brar has du replication builtin

which is great for Disaster Recovery RAR is natively multitenant is compatible with a software ecosystem and all these great features are available as open source so you have no vendor locked in all right so that's the end of my talk I hope you enjoyed it so thank you so much for attending if you have any question I'd be very happy to answer so feel free to scan

the QR code on the left to contact me and to get access to additional resources feel free to try your p on by downloading it I'll be very happy if you try out p and let's connect on Twitter or LinkedIn so now I'll be very happy to answer your questions thank you for uh the lovely topic Julian it was uh very interesting um one question I have

is why do you think a company would stick to Kafka even when presented all of the arguments that you presented in your topic what would make them pick that over a p sir well would they make stay on cka right yeah question well I would say uh because they they they not person I would say because of the uh the kind of an inertia when you when

you when you're acquainted to given technology and I would say also because the kfka cury is a is quite is a bit larger than the P Curr so that that can be a bit you know you you you might be not so confident of of the side of the count of P but while there is it's important to know that there are thousand of organization adoped P

but that's still still still theity is not as large as the CF one for sure uh and uh behind the cka world you've got uh huge and recognized vendors like Confluence for example there there are some some P vendors but they are not as I we say as huge and well recognized for now compare with kfka ecosystem for sure but it's important to know that despite of

that uh you've got a huge and very Vibrant Community around person so should not be afraid of picking the right technology for what you need okay and we have a question from the audience as well what are the downsides of Pulsar that's a very good question uh I would say that uh so prise the right technology for sure but I would say that F uh has an

architecture that is is you know uh a bit more sophisticated than Kafka because you've got you've got two layers two different layers Got The Bookies the bookie notes the and and the Boker notes so that two kind of notes minimum you also need a metadata store and coordination distributed coordination system so we are we are we use zookeeper but we are going to you know to switch

to an alternative but there we still need the coordination and metadata store it means that you end up on three different no Technologies so that can be slightly more complicated to manage because of that so on one side you benefit from all the do p but on the other side you've got quite a bit more of of complexity to manage compare with a small C installation so

if you need if you have a predictable workload you don't need so much elasticity and you only doing data streaming then kfka might be the right pick but if you need multi tency to save on on cost on large scale and you and you also need to save on cost because you will end with only one platform to manage then you should definitely consider P so it's

it's it could be a bit daunting that you have to learn a new system as like a barrier of of Entry uh I was I wanted to check um at some point during your talk like how would I run this on AWS and I noticed that uh stream native is actually offering that as a as a managed pay you go service to manage the apach P for

you so that should make it a bit easier because when you know with the cloud everyone's used to convenience because vendor lock in is one thing but I would rather have cap capex than Opex I don't want to spend uh brain power and people on operating sure for sure yeah my talk was more focused on the on the the open source of course feel free to feel

free to uh to ask a vendor to manage your C installation because that's the that's the goal of those vendors to you know to simplify your life and to simplify this this this management so when you manage a PO using the hosted manage offering of stream native well you don't need to worry about this uh infrastructure complexity because all of this is managed for you and at

the same time you benefit from you know the multi tency the UniFi messaging and streaming the elasticity which is very important and stuff okay awesome um so what advice would you give to people who want to run this on their own without using a manag provider so let's say um um they have um a private Cloud that they want to run this what would be a good

way to up skill so that you can run it yourself um so you've got you've got two options I would say the first one is that you can also ask vendors like stream native for a private Cloud offering so stream native can can provide you with software that will help you manage your cluster but that's your own resources private cluster right and the second thing is if

you really want to manage all of that by yourself then feel free to ask the community you got we we've got documentation on p.org and we've got a community slack with more than 10,000 members that will be very happy to to help you get started okay so basically documentation Community uh I'm assuming there's on online video trainings available as well yeah yeah you can you can you've

got for example we've got some vide for free freely available videos and the stream native YouTube channel for example but you can also find you know video videos and feel free to also check out my my profile the QR code on the on on the left you will find some interesting links okay adding it back all right well Julian thank you very much for the very informative

talk uh thank you to the audience as well for the questions and now we have a 1H hour lunch break after this talk so see you again in one hour see you bye thank