Lovisa Johansson: Real-world Examples of Message Queue Usage: Insights from RabbitMQ

This talk explores the practical applications of message queuing, specifically focusing on RabbitMQ. The speaker discusses how message queues function as a means to efficiently manage communication between various services, particularly in microservices architecture. Examples from the Adidas training app and a food delivery service illustrate how message queues handle increased loads and facilitate scalability. The speaker also delves into RabbitMQ’s features, including its AMQP protocol, various exchange types, and the handling of failures with dead lettering. The session highlights the importance of message queues in maintaining system reliability and flexibility, especially during high-demand scenarios like the COVID-19 pandemic. Throughout the discussion, the speaker emphasizes that adopting a message queue infrastructure can enhance performance and decouple service dependencies.

[Music] ladies and Gentlemen please welcome our next speaker Lisa Johansson presenting the topic real world examples of message Q usage insights from rabbit amp hi everyone I'm really glad to be here it's been nice days uh I've seen a wonderful Cate of wilus and have really great conversations I hope you're feeling okay after yesterday's get together and that this session will not add any post-party headaches for

you since forever we have communicated we have used all sorts of uh communication methods we have called each other texted each other Via SMS even used career pigeons and smoke signals in the tech world the situation is similar communication methods constantly evolving including the increasing use of message cues a message Q is just like a waiting line in the store but for information tasks or messages instead

of a waiting line for people a queue opens up the possibility of easy scaling just as when there is a new checkup opening at the grocer shopping and you be you become really happy uh you can add more services to your system and become really happy because the data flow is increasing because these new Services can help out clearing the queue on a beautiful day when you

go out for a run you might be running with a workout application when youve completed this run you check the leaderboard see if you set a new personal best on the course ran and the IT architecture in Adidas training app rentastic is divided into different parts parts where various services are working together behind the scenes and these Services could be tracking the run it could be updating

the leaderboard or it could be displaying the route you have been running building each service in this architecture is usually the easiest part but most time you most you need these parts to communicate and need to coordinate them and this is where threads of complexity starts to unreveal as far as Communication in especially microservices goes to popular approaches graes the stage we have a synchronous https and

a um synchronous htps and asynchronous lightweight messaging even though https might seem simple at first it has some downsides because of its uh synchronous nature it requires immediate response to every request that is sent but in reality uh every request doesn't need an immediate answer so if you think about the leaderboard in the workout app it doesn't have to be finished the exact same moment as you

has have completed run to be able to handle millions of users the communication in between the services at runastic takes place via a queue also called a yob q message Q message broker and in Adidas case through rabbit mq when the gym closed during the covid-19 pandemic a lot of more people started running and the use of workout workout apps increased a lot cues in this case

helped lining up the tasks this running tasks so even if a lot of more people started running all of a sudden the data was just waiting in line and couldn't be handled when convenient now you could close your eyes for a second and think about new New York City the non-stop City where everything happens and the sound is shouting all the time um here we have relay

delivery which fits right in as a food delivery service with cyclist navigating the street with a phone in their hand trying to find your door so they can deliver the food to them in the heart of R relay delivery we have R rabbit mq of course that takes care of everything in the all the communication they have relay delivery and rasic are just two example of how

message Q can be used for internal Communication in microservices but as we have seen rabit mq is also used in Legacy system integration we have seen it been used when you manage work cues when you need to stream data um when you need to enable RPC Communication in iot architecture shes between raspberry pies and and your back end and so on they are endless we have seen

also that ritam Q is used almost everywhere in society uh so I will show you a video maybe it could be nice to have it on full screen no I bet she doesn't know it's kind of cool that she doesn't what am I talking about well the infrastructure used by her workout application to send and receive data and it's not just this one application in fact thousands

even millions of users benefit from it daily the bank the food delivery service or the retail app that sends confirmation emails seconds after purchase the use cases are endless and the key to all this is called message queuing highly scalable built to handle it all from when data is created until it reaches its destination ask yourself is your infrastructure ready for an increasing number of users is

their data handled safely and can they count on your application working effortlessly to give them what they need if Prov highquality service is at the top of your mind message queing and rabbit in Q should be too because even if she is just one of thousands going on a jog at that exact time all she cares about is that her app working so now I will do

a short introduction of who I am why I'm standing here um then will talk about rabbit mq a little bit about amqp amqp protocol and I will combine this and mix this up with some real world use cases and it stopped working what did you say control shift go back ah perfect so a short recap of everything here so my name is Louisa I'm uh from Sweden

and we love q and ordering that's why I work at cloudm QP where we provide rabbitmq as a service I love also kite surfing uh this is where I kite surf in the North of Sweden in UMO need a big wet suit but I also been in Spain Brazil pH Philippines Morocco and so on it was my biggest interest for a really long time I have dogs

chickens and also two kids which take a lot of time now the youngest is 8mth old probably or maybe you saw him yesterday he was here mingling around uh I've been involved with RIT mq for many years I have helped out on stack Overflow I have um written hundreds of blog of blogs around rabbit mq and even two books about it uh I've also tackled plenty of

support questions or thousands of support questions I've also helped up building a training tool for rabit time how many in here are have used or are using or familiar with the message Q wow that's a lot nice so H I will start out with a use case where a message Q is really handy and it's when you need to make sure that background tasks are reliably completed

so imagine that you have just built an application for rentals of uh houses so apartment rentals and you want landlords to upload um information of the apartment to this app sadly uh a lot of landlords upload really high uh quality images and your app becomes slow so you decide to handle the resizing of this uh high quality images and you have chosen to use a message cue

here to prevent that from crashing if many landlords are up uploading a lot of images at the same time so here we have the base architecture of a message queue we have the publisher who adds an image scaling request to the message queue then we have a subscriber on the other side called image service that takes the message from the queue scales the selected image and adds

the image to the website when this uh app is growing more and more people start using it and even more people are uploading images at the same time you can add more image scaling services that will help clear the queue and so increase again increase the data flow of of your system rabbitmq is built on a protocol called amqp so let's delve a little bit into that

um the default protocol is 0.9.1 which specifies both over the wire uh protocol and how clients and the broker should behave in order to be compatible it has to live up to a few rules regarding how messages are routed and how they are stored in the bro uh in the broker and the protocol is also very specific on how connections are made and for example how you

should declare a que declare a queue other defining features are reliability like acknowledgements and security like encryptions and authentication so in amq you first have something called a connection it's the what you set up between your service and the queue uh this um connection um takes care of underlaying network tasks like IP resolution and auth authentication and things like that within this connection you can set up

something called a channel and a channel is like a virtual connection but when you use a channel you forego the need to reauthorize and open a new TCP stream which makes it better you use your resources more efficiently and this uh setting up connections instead of adding channels is one of the most common like errors our customers are doing which takes down their server rmq also simplifies

uh routing cases compared to other messaging system Kafka so you can um routing makes sure that a message can reach a specific q without your need to add any additional code to all this so messages in rabbitmq are actually not published straight to a queue they go through something called an exchange and you can think about an exchange as a mail delivery post desk ensuring that the

message will end up in the correct queue so messages are then routed dep depending on bindings in between the exchange and the queue and apart from the main content uh in the message you also have a routing key and a set of headers and those headers could could include like expiration time RIT andq comes by default with four different exchange types we have the uh direct exchange

the fan out exchange the topic exchange and the exchange and I will try to explain those uh by using the example with the post office so imagine you come into a post office and you're going to send an email or a a message then you can use you have four different options to which desk you should go to so if you go to the direct exchange you

take your message you write the name on the envelope and you hand it over to the to the desk they check on this name and gives the message to the person that needs this envelope super easy uh here you can think about the name as the routing key and the person receiving the uh the message as the Q so for the direct exchange it goes directly to

that person then we have the fan out exchange here you come in the desk uh to the fan out desk you give the give the letter to the person standing there uh this person takes this letter makes a copy of it and give it out to all uh all persons in that post office area regardless of what you have written on this email so it could it's

almost like span or something like that uh then you have the topic exchange here uh you come to the desk you give the letter to the person behind it which looks at a category or a topic on the mail it could be for example if you want to send to the development department in a company you write that as a category it takes this message and send

it out to all those people subscribing to exactly that topic and then finally we have the Heather Exchange which which is kind of similar to the topic exchange but instead it looks on we could um pretend that you look on a stamp on the envelope and sort the email out of that instead it sorts on headers uh oh I forgot the topic exchange if we go back

it's also pretty uh smart it can use Wild Card matches and U combine them with patterns and and send emails uh send messages like that I would like to direct uh our attention to the topic exchange because it's the one that holds a specific uh pref I think it's really good and that's the one we use the most uh so here is an example of how we

are using the topic exchange we work with as I said our software as a service for rabbit mq but also for Lavin mq what we do is that we when someone buys server rabit cluster or server from us we set up uh we buy this from Amazon or Google set it up in their Cloud We Set uh set up rabbit mq configure it as the customer wants

us to configure it uh automatically or of course and once this server is up we want to notify a lot of other services in our system about this successful task so what we do is that we add a message exchange and we have many services subscribing to this completed finished setup server message one of these is the server metrics which in turn uh sets up uh take

do a does a connection to the newly created server and start to receive server data like CPU or dis space and so on another of these Services is the rep mq metrics which which does the same thing takes the message sets up a new connection to this newly created server and starts to receive rabbit mq specific data which makes it possible for us to set up alarms

and stuff like that around mq and all these new Services can simply as subscribe to the messages without the the first Ser service needing to know about what are other services doing with my data that I'm sending away Adidas runastic is taking advantage of the same type of routing here you can see that or you remember that uh they subscribed to the lead the leaderboard service that

updated the leaderboard uh so here they send a message that their run is completed and other services just subscribe to that that completed session and handle the data uh so we can imagine that we are doing a new training app that will evaluate the Run s session that you yes did so all they need to do is you all you need to do is deploy that service

connect it to the queue uh and you will be able to just keep going we keep moving to something else uh messages in transit might get lost in case of a connection failure and need to be retransmitted amqp is uh very clear on when responsibility is transferred from one part of the system or uh to the others and this is done via something called so you can

confirm that a message has been successfully transferred or processed once it's sent out or when the service want to say now I'm finished with this with this so you can move on um so an explicit acknowledgement if a consumer cannot process a message for some reason your desire uh outcome could be that you want to recue it to let someone else take care of of that message

or that you want to process it in a later time however your message will stay safe in the queue meantime and once an acknowledgement is received it's discarded from the queue many of us are probably doing backups of a databases this is usually a long running task it takes time especially if you have really huge databases or thousands of databases which we have had um so here

we do like this we add a database backup request to the message queue then we let a backup service take care of this backup do the backup of the database the queue holds on to the message until this backup has been completed and is then act and removed from the queue and now you uh might think what happen if we have like we're doing a we have

a database as a service for example and we have scheduled thousands of databases and all of a sudden you want to let your customers do a backup by the by an API or something like that luckily then Revit mq supports something called priority cues so you can let this message that the this trigger triggered via the API you can let that go before all other messages because

it will have a higher priority and then it will be consumed before all the other messages we move over again to something else this is uh dead lettering it's the key part of almost any messaging system out there some messages might be undeliverable or unhandled even if they are received by the broker and this can happen when the amount of data a message has spent in the

queue has been exceeded or when a queue reaches it cap its capacity and then it or when it's negatively acknowledged by the consumer and such a message is then called uh a dead message setting up a dead letter exchange and a dead letter queue let you handle this because you can take care of these orphant messages and they can be stored and processed again from the dead

que using a dead letter is very beneficial in for example e-commerce system when you take care of order processing because this dead letter can handle message that can't be processed after uh that that can't be successfully processed after a certain amount of times and uh this will take care of yeah unexpected Payment Processing failures and after that of course the administrator or someone else can take care

of this message and check what's wrong with it how can we make this not happen again uh and yeah that's it there is also use cases uh where you have um unrootable messages like if you send a message and there is no binding set up between a queue and an exchange example and here you can decide if you either want to silently drop this message uh or

send it back to some other queue here comes a use case from a logistic application um that takes care of rabit mq for order fulfillment an alternate exchange is set up alongside the order status so if a message can't be routed for some reason with an invalid order status for example uh it can be handled directly from the alternate exchange as a safe um fail safe measure

and so far I mostly talked about I've talked a lot about cues but I haven't said much about them uh so here comes the main cues in rabbit m q it's classic cues quum qes and streams Quorum qes and streams are the most interesting one so those are the two we'll focus on now we start with Quorum cues which come in handy um when um high quality

um High availability and data safety is non-negotiable so each Quorum queue here is replicated you have a leader and you have several followers uh known as replicas all client interactions go through the leader and the leader replicates is responsible for replicating various commands to the followers and the follower itself doesn't have anything to do with client interaction it's simply there for uh pure redundancy if a broker

becomes Avail unavailable and other has to step up and quum qes they have their name because uh uh all operations like message replications uh and leader election and so on require a quorum a majority of the replicas to agree uh streams uh is the most recent added Q type to uh RIT mq it's primarily designed for other use cases uh like large fan outs or when you

need to have multiple consumers uh amongst others uh and the key features here is that stream cues uh can read the data multiple times without being removed from the queue and this is something that uh not all Brokers have uh active mq doesn't have support for this uh but of course it comes from Kafka which makes rabbit mq a really good alternative if you are working with

with an rabbit mq environment but you want to add Kafka like features into mq it's good when you need to process a message multiple times or uh from a lot of different consumers while so far uh I have talked about amqp mainly um I want to mention uh rabbit mq is a multi M multiprotocol extending its support Beyond amqp it does also have support for mqtt stomp

HTTP web sockets and so on and you can choose protocol that you can choose the protocol that align best with the technology that you are using for example a lot of our uh customers are using uh mqtt in Internet of Things uh and while talking about mqt we have an example from farmbot uh this is an open-source um farming robot that you can have in your garden

which can Watering your plants and um taking photos it can even scare the birds away um this uh this application or this robot has um sensors and drivers which Bridge the connection between your physical garden and and the soft for a layer that you have or they have a message broker is used for messaging between the database the user and the devices in this system and we

can see they are using mqt from the respir to the broker while they are using amqp in many other parts of their architecture in the early days of the internet we have a web pages they were static uh like digital brochures later Along Came um Dynamic content with server side scripting and uh that Ena enabled personalized interactions and as the Need For Real Time interactions grow finally

websockets came along and they changed the game enabling two-way communication uh between client and server and rabit Q among other message Brokers have support for websockets uh which which is ensuring a robust and scalable real time uh communication here we have an example from visman I not sure I pronounced that right I hope so uh but it's they are leading uh leading industry in the heating and

cooling system and they are using websockets and rabbit mq together when they do for example live up for their live update feature so they let messages from the heating system uh go through um rabbit mq through the websocket through uh and end up in their front application so they can receive realtime updates from their system we will move on to Rabbit mq uh how it's used the

main use case I would say for um for rabbit mq is within microservices here we have parkster par parking app company in Sweden they divided their software into a collection of small isolated Services where each service can be deployed uh and scale as needed and function independent of each other by breaking down their fre uh uh their app they gave developers the freedom to use whatever technology

they wanted for different parts of the system uh and this means that one part can evolve independently be written in different programming languages or maintained by different teams and we at cloud and QP we do the same we have most of our applications written in Ruby some in Crystal some in go and so on uh and we have around hundred of microservices communicating through our own cues

because we are also using repm Q in our own uh architecture so even if one of uh um these Services goes down if part of the back end is delayed or broken the whole app could still be working so even if in this case you parked the your car and you think yay the payment service is down you could you will still have to pay because the

message will stay in the queue and be handled when convenient or when the payment service is back up again many of our customers are sending thousands of of emails uh even handling email automation Automation Services thank and they of course if you have a automation service service you have to send campaigns with the and you can have changing demand sometimes you send a little bit sometimes you

send a lot and um by adding a message cue you can you can kind of they couple the production of your emails to the for from the service that actually sends the email we are using message cues for this of course uh we send our mails to the message broker and let email Services take care of this trust is another company they work with um uh they

let your company find an if you have like a product you want to sell you find influencer to that product this influencer is supposed to write a review review of your product so in different stage they have to send a lot of emails to all of these candidate candidates that should be able to give you a good product what they did is that they decoupled their they

had a monolith synchronous application they broke it up into pieces and are now using as synchronous messaging and are sending much more messages per second another company using RIT mq for microservices is R which work with a workflow solution for Manu it's a for manufacturing so there workers could be standing on the floor reporting safety aards um observations or even if they have an improvement idea for

the company they can improve it directly through their phone and the whole SI the whole their system is built with a message Q as a ground uh in the our infrastructure and they are sending hundreds of thousands of messages through this system per day to keep up with unpredictable command we have multiple retailers using rabbit mq mainly because they can scale when needed like when it's Black

Friday they could pump bump up the service pretty much or they can add more consumers to the servers uh and make sure that that orders are efficiently and quickly process assessed there are also many taxi and delivery firms using uh using us um sending all their data as um a lot of they are constantly sharing GPS data systems um we have also one of the largest breweries

as a customer uh and a newspaper or man in newspapers that add their articles to the queue and take care of them after that and one of these newspapers is a newspaper with 65 million readers a month we have one of the world's largest Payment Solutions uh and several of the world's most famous car brands as customers and it's so interesting because in my job I get

to travel to these customers talk to them and either help them set up their infrastructure W with a broker in the middle or just talk to them and see how are they using how are they using Us and how is a message Q used and again it's very clear to us that message cues are used everywhere as I said in society government retail education and of course

for uh different use cases it Bridges Communication in your infrastructure it's used with streaming iot as a general pubsub message broker on so so now when I have talked a lot about this you might think when uh should I not use mq in my infrastructure and I would answer like a someone in the European Parliament uh it depends depends on what goal you have with your architecture

uh but in most simple use cases you don't need a message CU and it's easiest to exec ex exclude a message Q early in your development process but when you build or aim to build large complex interconnected system or when you know that you will have a need to scale then the need for a message Q will come sooner or later so uh from what I've seen

or what we have seen is that it's easier to implement a message broker early in the process than to try to add a message broker when already a lot of data is Flowing around ironically you need little extra complexity to handle more complexity I have not covered all the features in rabbit mq um shortly mentioned Federation shovel it's when you you can set up uh servers in

different uh clouds or different regions in the clouds and you can let data flow between them you can also do clustering you can set up uh one three or five nodes and combine this together uh and you can shovel messages from one server to another there are features like that and there are much more features and you can come to our booth we are outside if you

have any questions and we will try to answer your questions or uh help you if you are working with rabbitmq and need some help finally uh I want to uh tell the three things I think you should remember from this talk um and what I think is the best things with using a message queue firstly uh it's that you improve scalability it enables seamless integration you can

uh easily add additional consumers uh when there's a peak hour in or when there's a sudden need for more capacity like during the pandemic and a lot of more people started to run or as I said during Black reliability and fault tolerance uh the introduction of a m message CU Fosters a synchronous communication decoupling your services which enhance reliability so for example in an event of a

failure like when the payment app payment service was down in the parking app other services could continue processing without destroying everything or yeah everything else could continue working and lastly flexibility uh message Q facilitates loose coupling H so you can Implement new changes or new feature without having to interact with the whole system uh you can write in different programming languages if you want and so on

like in the training app uh if you wanted to do this new um service that evaluates your your run you could EAS easily just set it to the combine connect it to the queue and start to receive those messages yeah thank you so much for listening thank you a lot for your talk very informative I think and we have a very heated discussions and a lot of

questions so let's see how many can you answer so let's start with the awarded one how availability and data replication work for rabbit you touched briefly on this but I guess you can elaborate a little bit more if I send a message and a rabbit node accepting it crashes what are the scalability and high availability models uh you can set up uh like I mentioned little bit

in the end you can set up clustering so you can have one two or three five nodes or more uh but it's uh if it crash it depends on what you have done if you're using Quorum cues you have one answer if you're using classic cues you have a really bad answer then you should change away from classic cues and shift to quum cues because it's much

more stable and we really recommend it uh if you have classic cues you can miror cues um which you can miror the message from one Q to another uh and if you have Quorum cues you replicate the messages but classic qes have had problem with crashes because when you miror and if the node goes down they start to synchronize which could be uh make your node go

down for a really long time which is yeah don't use the classic cues okay don't use the classic qes uh next question you said that you were operating on mqp version 0.9 and apparently there is a higher version available why don't you use that uh 0.9 one is one protocol and amqp 0.1 is a completely different protocol it's the name are alike but they are behaving completely

different to each other so it's a breaking change between the major B yeah but there is support for amqp 1.0 as a um plug-in and uh in version the next big ma major version in rmq 4.0 1.0 will be much more more involved okay but they didn't change it because it's it's not protocol despite the name major updates when would you suggest to use rabbit over Kafka

and vice versa uh I would say that uh I would always suggest rabbit mq that's a bad question no uh we have actually been hosting Kafka for a really long time as well uh and we suggested all P people that are send this is my most answered questions on on stack Overflow I have so many points for this from this question um it's when you send large

amount of data that needs to be streamed I think you should go with Kafka and general Pub sub message broker microservices rabit mq got job uh what are the benefits of fanout exchange comparing it with a topic um I which just say that the fan out exchange will um kind of when you have the streams you can if you're doing really large um fan outs I think

you should do a stream instead because it will be just one message sent out to a lot of them uh the fan out is easier we are always use we're almost only using the topic exchange because it's it's really good for us but do you know when we use the fan out would exchange the most um tough question the best benefits I will think about it to

find the best benefits for it all righty then next one how to configure or is it possible to configure QE or message to have a special message as a blocker so if the message is not acknowledged next message will not continue to process uh yes it it if you send a message with you can either send a bunch of messages or you can send one single message

so I guess if you I'm not 100% sure but if you have just one single message and you send it and you then you block it until you have acknowledged it and but then you have I don't I don't remember the name of this but if um you can also decide to send like 10 messages or hundred of messages to each consumer and let them handle that

and then acknowledge that back but if you just send one you will only send one but then you also have the problem that you might have many consumers so if you send one and you have more consumers of course more messages will go out because it will try to process messages as fast as possible so one consumer and only send one message at a time then then

it will work is it something like circuit breaker maybe in rabbitmq that says oh this message was not properly processed I do not accept anything until the problem is resolved on the consumer end no because the r mq will always try to send out the messages but if you have one consumer then then it will work okay next one how do you handle rabbit upgrades and or

migrations without downtime bonus question how do you deal with rabbit version being dependent on airline version I start with the second one uh Airline version um actually we built a new message broker because rabit mq is dependent on the airline version uh because we have so much trouble trying to F find employees for that uh that can work with Airline we have thankfully Airline Solutions which helps

us a lot a lot with consultant hours around air Lang but uh we built another message broker uh to be able to not but maybe that's not the question is it around Airline updates I guess so so basically do you have problems with language you said yes not too many days we love air Lang and um but how we handle upgrades and migrations we are doing in

this in different ways uh I can give an a a huge uh description of that uh in the booth but I don't think everyone else will listen but Eric he is a professional on our migration from one version to another uh we're using a lot of federation and we are swifting in and out discs and so on okay for more detail lecture on this please uh visit

rabbit boo he is our support engineer okay let's have two more small I hope Blitz questions do the stream cues have support for something similar to to kkas groups I don't think so correct me if wrong again open to a discussion uh afterwards and the last one who owns the cloud service you or the customer and who can access message content stored on this servers we can

access it if we want to but we are not allowed to because we we don't do that we we have written contracts that we shouldn't shouldn't use it uh the cloud servers is owned by AWS we buy them uh and we set up the infrastructure but uh usually our customers are in the same cloud and we use vcp pairing to those servers gotcha