Consistency and Coordination Patterns in Event-Driven Architecture, Emanuel Trandafir

This talk covers challenges in software architecture, specifically focusing on the dual write problem encountered in distributed systems. The speaker discusses various design patterns aimed at achieving data consistency and effective coordination among multiple services. Key solutions such as the saga pattern, orchestration, and choreography are explored to illustrate how to manage complex workflows. The speaker emphasizes the importance of compensating operations to maintain consistency, especially when dealing with eventual consistency and different types of errors that may occur during transactions. Additionally, techniques like the outbox pattern are introduced to ensure reliable message delivery between services.

Hello, welcome. I'm Emmanuel. I'm a Java developer from Bucharest, Romania. I'm also an author on Bildang. Um, I'm trying to contribute a bit to open source to some projects. Um, outside of work, I'm trying to stay active. So, I'm doing a few sports, uh, rock climbing, tennis, triathons. Uh maybe like 10 years ago when I was doing this a bit more seriously, I had a favorite race

here in Bulga. It was a 10 km race in Velico. It was called Crossy Vilo. It was pretty tough because the route was very hilly and the competition was very tough. So I went there, I finished sixth, then I went home, trained, returned back next year, I finished second, then went back home, trained more. Um, and then next year, well, next year I learned Java, so I

stopped running. So okay, so why architecture, you know, why have I chosen this topic for this day? Uh, why event driven? Why all these patterns? Um well, now that we generate a lot of code with with AI and we have all these nice tools, we don't spend so much time just typing code, just writing. So we can use this time somewhere else, right? And I think there

are some interesting areas where we can use this time. Um apart from teams meetings, uh I think observability is an interesting one because we want this feedback loop, right? We wanted to to see how the generated code is behaving. Uh but today we'll discuss more about design and architecture simply because we generate a a lot of code. We integrate it more often and we need to stay

on top of things and make sure that everything fits into our structure. And it's also because I'm from Bucharest, you know, from Romania. We are in Balkans as well. We I saw some really bad architecture. So when I was 10, my parents took took me to Constansa to the seaside and I saw this. Um, and because this building was really close to the city center, um, you

know, the they tried to decorate it for the Christmas fair. So we had the Christmas fair, they tried to decorate it to put some nice Christmas decorations, um, you know, just to make it look festive. So they put some teddy bears and this was the result. Now, I don't know if you worked with monolithic, legacy, old applications and you tried to put some nice new features on

top, maybe reactive streams, I don't know, maybe AI. Now, um, was this the result? Well, for me, it wasn't. So, that's [laughter] that's what we ended up with. Yeah. Now, how did we ended up here? How do we have code that looks like this? Um let's start with this bit of code. It's the only bit of code that I'll share. So but it's we will assume we

are working for this e-commerce application, right? And this is the Java method that is creating an order. And as we can see here, we are saving something to the orders table and then we are also updating the the products table. Um and at this point we would like to make this transactional to wrap this in a database transaction. Right? We want to be atomic here to have

these two operations atomic. Unfortunately, this method is doing more than just this, right? It is also publishing a message to our message broker. Um, and in fact, this is a domain event. So, it's a way for our order service to communicate to the rest of the system that look something happened within my bounded context, the context of orders, and the rest of the system needs to know

about this new order that was created, right? Then it's also saving something to the local file system. Uh it's also sending a post request to a different microser this loyalty service. And finally it's also sending an email using this email service bin. Now as you can see this code is doing quite a few things right. Uh it's definitely not single responsibility principle and uncle Bob will be

really upset with this. Yeah. Now, have you seen code like this or wrote code like this? Uh, what about your colleague? Um, let me give you a uh like a hint, a helper uh you know question. Have you seen tests like these? So, what's wrong with this? We have mock mock mock mock and then the test itself is just stubbing one call to the production code and

then verifying mock interactions. So this is a way for the test to give us a a hint, a clue that our design is a bit wrong, right? We are doing a lot of different things and we do all sorts of IO things and we are doing all sorts of side effects here. So the test is kind of trying to tell look something is is weird here. Um

and if we look back so the problem why why I said we cannot really or like it's it's tricky if we just mark the whole thing as as a single transaction. Um well what if one step fails? What if the email uh sending the email fails? We will roll back the changes from the database. But what about the Kafka message? The Kafka message is already out, right?

It's in the broker. Maybe consumer applications already received it. Um same for the file and same for the post request. Maybe this loyalty service is already processing that request. So we'll be in an inconsistent state. Yeah, transaction works for the database changes in this case but not for all the other changes. Um, so we have two types of problems here. One is this data consistency problem that

we discussed and there's also a coordination problem. So we have five different systems that we are trying to coordinate and we are having some some issues with this and um if we summarize this our challenge was writing data to different systems and this can involve yeah changing something in the database publishing a message um sending HTTP requests and so on can there can be many examples of

this. Now this particular problem or challenge is also known as the dual right problem uh and it's pretty common in the distributed systems. So this is the the thing that we are trying to solve. So today we'll discuss different uh consistency and coordination patterns that will help us you know find a better way of of of doing this and also it's dual right I guess if you

have two systems we had five so I don't know how we can call now out of these side effects that we are doing which one would be the easiest one to undo if you want to undo it so we sent an email we sent a Kafka message we were saving something to file system and we send a post request. Yeah, if we just want to to deal

with this side effect like just deleting the file is the easiest one, right? We can just this is very simplistic but we can just catch some exception and just delete that that file and we have some consistency again because we something went wrong we rolled back the the data in the database but we also deleted the file. So between these two it's more or less like a

consistent state again. Uh now why can we do it here and we cannot do it with Kafka or with the message broker. So if I if Kafka slips this should be just uh applying for any message broker. Um well we don't have this control over the broker right but if we have control over the consumer application maybe we can do something similar there. So the idea is

that something went wrong, we just go and perform an operation that tries to undo what we've done. Yeah, this is called a compensating operation. So let's imagine this this case the order service receives the uh place order requests. It's going to save the order into the database and then it's publishing the order created domain event. Now the inventory service is listening to this is doing its own

bit of logic and something breaks here. Yeah, let's say we are out of stock. Now in this case we can publish a another domain event message saying that look we had an issue here. Yeah, we have the stock unavailable problem and order service can listen to this and perform some compensating operation. In this case it can cancel the order. So we are in a consistent state again

between these two databases. Uh now cancelling the order needs to be item potent. So we we should be able to retry this again and again until we succeed to to cancel the order, right? So that's why we say we are eventually consistent here. It can be out of sync for a a while for a bit of time but eventually we should be in a consistent state again.

So this is good. It works. here um order service knows about inventory and inventory service knows about order created. So they both both kind of know about each other. Um and there's this almost like cyclic dependency here. So what if we can can we try to to avoid this to break this cyclic dependency? Well, we can if we introduce a coordinator. Yeah. So we have this e-commerce

application which is just coordinating uh this this workflow. So now the e-commerce application can just tell the order service look you need to create an order. So this time this is no longer a domain event right it's it's just telling it to do something. It's a command passive aggressive saying you need to create an order. Uh and then it goes to the inventory service and it yet

again sends a um a command yeah to to update the stock. If something fails here, the coordinator can go to the previous steps again and um send another command to do this compensating operation in this case can canceling the order. So and of course we can have multiple components in this workflow, right? So there's another one there. And if something goes wrong here, of course, we go

to both of the previous steps and undo them. So these two were like two flavors of the same pattern. Uh and the essence of this pattern is just that we do something and then if something and we perform a local transaction. Yeah. And we move on with the flow. And then if something goes wrong, we just go back to all the previous steps and perform this compensating

operations to to be in a consistent state again. And maybe do you know what's the name of this pattern? Yeah, somebody said saga. So this is the saga pattern. So the definition from uh the microservices book is just you know that saga is just a sequence of local transactions where each local transaction updates its own database and then we we publish a message that triggers the next

uh local transaction from the saga and then if something fails we just trigger this series of of compensating operations. But that was the saga right this helped us um you know it's one way to tackle this distributed transaction and this dual right uh problem that we we saw earlier. Now what were those two flavors of it? So this was one of them. So here this is also

uh named uh this is named this pattern is named orchestration. Yeah. And this this coordinator is actually an orchestrator. Yeah, it's like the agent orchestrator like talks but this is the real deal. Yeah, the initial one. Um, now what's interesting about this is that um all it's well first of all it's really simple to implement saga here. Yeah. So the error handling is is pretty simple and

also what I like here is that if you see the order service doesn't know about inventory service and and so on. inventory service doesn't know about shipping service. So they are very decoupled. They don't know very independent from each other. Right? So when I saw this I was like really happy. I said well look this is a system with no coupling. Right? Um but it's not true

right because we what we do is just to move all the all the coupling to the left hand side to the orchestrator because the orchestrator needs to know about all the the component it orchestrates and hold all this logic about yeah the the business workflow that we have. Um, and it's also harder to implement new components because if we implement a new uh component here, a new

micros service, let's say stock replenishment service or something like this, it's not enough just to deploy this new new component. It will do nothing. If we just deploy it, we also need to go to the orchestrator um include this new component into a domain workflow and redeploy the the orchestrator. So there are a few more more steps to do. So this was orchestration. Yeah. The essence was

that we have this central coordinator. Uh that tells the the orchestrated components what to do, when to do it. Well, it doesn't tell them how to do it. This is still responsibility of each uh component and all these orchestrated components they don't really talk to each other directly. So everything happens through Okay. So this is the orchestration. Now the alternative would be this no central coordinator and

all the components are just listening to domain events doing a bit of business logic and then publishing their own domain events and this allows the system to grow faster. So this is choreography. Um, so what's nice here is that um it it makes it much easier to implement new components and because we no longer have the central coordinator, we there's no bottleneck or no like single point

of failure. So it's a bit better from this sense. On the other hand, we still have this complex error handling. So if something goes wrong here, it's pretty hard to notify everybody what what happened. And uh yeah we have the cyclic dependencies that we talked about. So this was choreography. Yeah. Services are loosely coupled uh so they just communicate through domain events uh and they decide by

themsel when when to do uh what to do and when to do it. So we had dual right uh the main changes we are trying to solve. We discussed about saga which is a consistency pattern helped us keep everything consistent and then orchestration and choreography there are two coordination patterns. So they allowed us to to coordinate all these different systems. And now when when I had this

talk before uh somebody asked me like a friend of mine asked me like okay man this this is pretty cool thanks thanks for that but just tell me which one should I use on Monday I don't know orchestration choreography you know just tell me code is also not telling me so just say one hand you know and it's not that simple right so if this is our

system it really depends maybe we have one part of the system where we have a complex error handling and we want to be able to use orchestration in there and for the rest of the system maybe we still want this evolutionary architecture where um you know it's it's pretty easy to add new components. So we can use one the other a mixture of both. Then what if

we spend too much time implementing sagas and dealing with all this inconsistency? Well, in that case, maybe there is a sign we broke the system into two small pieces, pieces that are too small. And maybe you can just have a just a monorit. I don't mean this monorit. Uh just a nice modular monorit. So if this is interesting to you, you can also read about spring moderate

that allows you to to keep monits a bit more like tidy and nicely decoupled. So until now we had we used the saga to to keep everything consistent. But we we always started with this assumption that we can send the message and the message is like the consumer is listening to this and undoing something. Um what if our message doesn't even reach the broker? So this is

my message trying to reach Kafka. Yeah. So if this was our code, yeah, saving to the database then trying to publish something. Um this is just a diagram of the same what can we do here? So one option would be to try to wrap everything in a database transaction again like we discussed before. And this is a naive approach. It doesn't really work. But the reasoning is

is pretty simple is that okay I'm trying to save the then I'm trying to send and if sending itself fails we'll have an exception and we'll roll back the transaction. So we are in a consistent state here. Yeah. But it it doesn't really work. One second. We actually tried to implement this uh with just wrapping everything in a transaction. So the issue we had was so we

were like just creating this batch of promotional codes that the user just some string values that um customers can just put in their application and get some discount. So we created a batch of these codes. We then published the domain event that okay this this batch of of promotional codes is ready. other applications started picking them and like distribute them distribute them to the users. But then

something failed. So the the insert into the database failed. Yeah. Because because now it's it's a transaction. So the commit happens at the end and everything was rolled back. So it was looking like this. Everything looks all right. We try to commit, we fail and we roll back the data from the database. So there are like no promotional codes in the database. But there are these promotional

codes in the rest of the systems and we are already distributing them distributing them to to our customers and it was really hard to debug cuz they look real. They look very like exactly the format we would use. But even we couldn't find them in any audit. It was pretty tricky to to understand what happened. So just doesn't really work or it's tricky. What if we stop

here? No transaction. We just stop here because after all just just publishing the message itself, it's a pretty simple operation. So maybe it doesn't fail. Yeah. Um well, this is actually at most once message delivery, right? So we can either send the message or we can fail to send it. We don't have any guarantee that the message will be fa uh sent. And in some cases it

works. Yeah. Maybe it's just good enough. Maybe maybe if we are sending some promotional emails or something like this, we don't want to over complicate things for some tiny edge keys. Um but in other systems it doesn't really work. So for example, right now we are working in uh financial markets and we have a low number of of transactions. So not many so the throughput is not

not very large but the value of these transactions is very high. So we cannot afford to lose any message not even in this edge cases. So at most once is not good enough in this uh in this case. Now if we stop here for a second I want to tell you a small story. Uh I told this to my father. My father said son you're old enough

now. You know, you need to know that Santa Claus is not real. You know, Tooth Fairy doesn't exist and there's no such thing as exactly once message delivery cuz I wanted exactly once, right? I wanted to to be sure that. So, I [snorts] was really upset at the Christmas fair. All the kids were getting presents from Santa and I was just, you know, just very upset. Um,

but guys, you need to remember I'm from Romania. Yeah. to 20 25 years ago, Christmas didn't look like this. I don't know if it it was like this in Bulgaria, but [snorts] for me, this was the Christmas fair as you can remember. And this was Santa. So, in fact, I was, you can see me there, a bit worried, but after all, I was like pretty happy that

the guy is not coming to to our house, you know, to bring presents. Um but I was mostly concerned about Kafka. I was like what do I do with this right? So at most once is not good enough for this case. Exactly once it cannot it doesn't work. So we need to find something else. And something else is just trying to to have this at least once

message delivery. And by the way when I'm saying that exactly once it is not possible I mean this context of saving to database and then publishing the message. Yeah, cuz just because your message broker probably supports exactly once but not in this context of the distributed uh flow like uh invoking the database as well. So we wanted to go to at least once and we have a

pattern for this. So maybe pattern maybe you heard about it. Have you used it? Yeah. So if you haven't used it, you probably used an email that uses it. Yeah. So when I'm trying to send an email to you, it first goes to a special place called the outbox. Then a separate process is just looking there taking emails to be sent and then it's sending them to

the intended receiver. So it's breaking this into two different steps. So we'll implement exactly this. Instead of trying to do it everything al together, we are breaking this into two pieces. Um first just saving the order later publishing the domain event. Then this outbox space is going to be a new table in the same database. It's very important to be in the same database because we want

this to be atomic. So now everything is a Another good thing about this is that now this part of the code it's it doesn't keep the database transaction open for long. Yeah. So it's just saving like inserting and then releasing the the database connection. It doesn't keep it open for other IO that we were doing like sending to the message broker or like doing requests and things

like this. So that's another thing. Then how do we connect this? We can either have like some change data capture mechanism or like just polling. We take the messages one by one. We publish them and finally we update the outbox table just marking the record as published. And it looks like we solve this dual right problem, right? Because because if you look here, we are only talking

to the database. So it's it's not the dual right anymore. However, on on the other side, on the left right side, um we are still publishing the message and then trying to to update or or remove from the database. So, it's still the dual right. Yeah, we just moved the the problem further away towards the the boundary of our application, right? But this is good actually because

it allows us to do the following thing. So, it it works like this. If if we cannot publish something for any reason, the next time we pull this message, we will send it at some point. The problem is that we can still publish the message but fail to update or to remove it from the database. And this means that the next time when we are pulling the

messages to be sent again, we will end up with a duplicate message. I will send the same message again. Now this is something we we knew it can happen. Yeah. This is why it's at least once. Um but it's it's easier to dduplicate the messages later on than to try to recover messages that were lost. So and we can even do one more thing. We can help

the consumer application to understand that this is a duplicate. The second message is duplicate. How can we do this? Well, in the outbox, we can have a column which is called item potency key. Um, you can give it another name if you want, but this will be an unique ID and it will become a header to our message that we publish. Now, if we send duplicates, all

of them will have the same identity key header. And this is a way for the well look I already processed this message so I can just discard the second one and so on. Uh additionally we can we can add all sorts of things that may come in handy like uh an observed at header. So this is more that um in case we want to reorder messages later

on in a given time window but is a bit more complex. So we can have all of this. it brings a bit of extra latency when it comes to publishing the this domain events. On the other hand, the business side of things here, it's going to be faster because it's not dealing with other IO. So yeah, it can produce duplicates as we know, but it guarantees the

delivery now and also it brings quite a bit of complexity, right? We have a new table. We need to do this polling. We need to check quite a few things. So when is it worth using this? So we can use it when our operation um we can use this when we cannot really afford to lose messages. So if it's very important like the case I described before

especially if fall to tolerance or like resilience is is very important. Um yeah and one example would be in financial markets where you don't really want to lose any message. Uh we can skip using it if at least once is good enough at most once is good enough. So maybe yeah in some some cases where you are having a maybe you're reading from from a sensor let's

say you have quite a few messages and just losing one doesn't really impact things because the others are coming very fast so we can just drop drop that one without worrying too much Um and about the boiler plate code and all the complexity if you want to use something to to help you uh spring modulate has support for the outbox pattern. Um and yeah you can use

something for the change data capture as well that should help managing this. So we started with a dual right. Yeah. So the issue that we had like the challenge that we encountered when we're trying to uh write to different systems. We discussed the saga basically was about trying to perform a compensating operation when when something goes wrong. Yeah. we just go back and then undo what we've

done. Then for coordinating all these systems, we discussed orchestration and choreography. Two different ways of of coordinate co coordinating them. And we discussed the outbox to make sure that our um message actually reaches the broker. And by the way, you can use outbox with the other side effects as well. So you want to make sure a send request is is sent successfully or something like this. So

it works with multiple um yeah different implementations I guess. Um okay we discussed we we said that we are guaranteed to publish the message but we may have duplicates. What do we do with those? Yeah. So this is the consumer receiving a bunch of duplicates because we implemented the the outbox, right? Okay. So we can do something pretty similar. This is called the inbox. Yeah. Um and

it's pretty similar because we broke this into two different pieces as well. We have we first take the incoming message, we save it to the inbox table in our database. Then we pull these unprocessed messages from the inbox. Uh we process them one by one and finally we mark them as as now if you remember about those headers um now having this inbox is really convenient because

we can have a column in inbox to store the item potency key and we can put the uniqueness constraint on this column. This this means that we delegate to the database the responsibility of u the dduplicating this this incoming messages. And of course we need to make sure that um if if we fail with the data constraint because of these duplicates we we just catch this exception

and we acknowledge the message. You don't want to get stuck Um then obviously we can um we can have we can differentiate retribable and non-retable exceptions. So a retribable exception will leave the message in the inbox so it can be pulled again. A non-retable one will either mark it as already processed or we'll just remove it from there. Um, we can try to use the the observed

at to reorder messages within a gi given time window. But it yeah, as I said, this this can be a bit more tricky. Um, so yeah, this one as well adds a bit of extra latency because we have this new table that we go through and we pull and we do all of that. Now if we drop all of this again and think a bit where it

makes sense to use it. So basically we can use the inbox to it important consumers right consumers that can uh deal easily deal with duplicate messages. So if our operation is not naturally item important we can use this to to help to make it important. Now if we if we feel like it brings too much complexity again new table new polling um we can avoid it and

we can use a certain like combination of features. So we can either use we can either have operations that are naturally item important or we can you know change our design to make them like this. So for example, just inserting something if it doesn't really already exist and using a business field or or column to decide this. Maybe the client can send the unique UID and we

can have this in the payload and use this to to understand that we already saved this. Uh we can have retribable topics again to be able to retry messages without blocking the main topic or Q and we can have a dead letter Q just for audit and like to be able to manually inspect the messages that were um you know lost or we exhausted all the retries

for for them. So I think these are pretty much the things that I wanted to discuss. So we we started with dual right [snorts] saga orchestration choreography outbox and then inbox in the software architecture the hard parts in the beginning of the book the authors are even talking about some decisions that you you need to take when you want to connect to the different like components. Yeah

you create this link between them. Uh and they are saying there are like three axis for this decision. um you have the consistency. So it can be either atomic or eventually consistent. We were like in the eventual our patterns were in this eventually consistent uh part of the this 3D space. Uh we have the coordination which we we already discussed about choreography and orchestration and the communication

style which can be there synchronous or asynchronous. So everything that we discussed all these patterns are somewhere in this three-dimensional So we started from the dual right this had to do with updating different systems. Then saga helped us keep this consistent by having this um way of of communicating that something went wrong and performing some um compensating operations. And this is the saga in a orchestrated environment.

Orchestrator had the central coordinator that coordinates everything and it can send commands to to undo previous steps It's very convenient because it allows us to to grow our system pretty fast. Yeah, it has this evolutionary architecture. And we can set we said we can use one the other or even a mixture of both or even none if if it doesn't make sense. Then if we have problems

with like losing messages and we want this eventually eventual consistency and guarantee message delivery we can use And in order to deal with the duplicates and implement the potent consumers we can use the inbox pattern. and I think that's pretty much it from from my side. Uh, this is a QR for feedback if you want to give me some. And, uh, I'll post the links slides on

LinkedIn a bit later today. [snorts] And we have quite a few minutes for questions if you have some. [applause]