Michał Ostruszka: Things I Wish I knew Before I Started with Event Sourcing

This talk focuses on the concept of event sourcing as a software architecture pattern, which emphasizes storing state changes as a sequence of events rather than just maintaining the current state. The speaker, Miteska, explains foundational concepts of event sourcing, such as how commands trigger state changes and how events are processed to reconstruct application state. He outlines the implications of using event sourcing for data integrity, time traveling through historical events, and eventual consistency in system design. Additionally, Miteska discusses the importance of versioning strategies for event streams and the need for careful domain modeling to avoid complexities related to performance and storage. Lastly, he explains how event sourcing can enhance auditing capabilities and the flexibility to answer future business questions.

ladies and Gentlemen please welcome our next speaker miteska presenting the topic things I will shine you before I started with event [Music] sourcing okay hello by the way day I Massac my name and that's what I was expecting uh to be honest uh so once again again my name is mka I'm from Poland that's why probably it was the uh hardest task for thei to uh to

go with uh I work at software Mill uh we are Software House Company doing crafting software for our clients and today I'm going to talk about some things and some facts and and traps that I learned uh about event sourcing uh and things I wish I knew uh about that that that Paradigm when I started with that uh I barely see you but the first question would

be how many of you is theoretically familiar with sourcing raise your hand please okay most of you and how many of you have had a chance to use it on production okay slightly less cool so let's go for for those who who are not not uh that familiar with let's go through some of the basic concepts of human sourcing I ask my body uh CH GPT I

guess most of you these days have the same body we have the same body uh and I ask I ask GP to answer in two sentences what is event sourcing and it told me that event souring is a software architecture pattern where State changes are stored as a sequence of events rather than just a just the current state so basically uh in kind of typical application typical

crude based application you have your entities your business entities whatever it is let's say uh user bank account you have that entity in a database as a form of let's say one row one logical row in a database and then whatever action comes in to modify the state of the account you just update the uh you just select the the current state of account you do an

operation and you issue an update on the on a database to persist the current state the even sourcing kind of flips that uh idea and dites that idea uh for saving the state of the application as a save of the business entity as a series of events as a sequence of events so every single action that you execute on the account on the hypothetical account uh you

store as an event you store as a fact that happened on that account as a kind of log happened only L and that has uh kind of big consequences uh and big impact on the way you handle the application and the way you the the thing and the possibilities that you can do with your data uh these event streams are a single source of Truth for your

applications so there's no there so there is no State like in a database that you you keep in some extra place you keep State and then you keep uh sequence of logs a sequence of events the sequence of events is just application but basically coming from the text to the picture how it works uh in order to execute any any action on the hypothetical account or any

other business entity that you may have modeled as a as a event sourcing one you might need to uh execute a command because if you query the system it doesn't change anything by Design queries should not change the system of a state it should be just readon operation but if you want to change the the state of an account for example I don't know deposit money withdraw

money you need to issue a command which is an intent of changing the state of the of the account like decreasing balance increasing balance depending on the operation and when the command comes in and you can you can you can think about the the uh rectangle uh as a as this account uh as a picture of this account entity when the command comes in what happens in

the typical crowd application would be that you would just load a single row in a DAT from the database you would just select account by ID and just have that account State uh to work uh on it to update uh uh to update to to execute the operation on it in even source application it's slightly different or or very different uh you have this events journal and

you can see that uh when command comes in there's series of events that are fetched from the from the database from the event journal and they are applied to the state to an empty initially empty State they are applied one by one uh in order to constitute in order to build the current state so if you apply all the events like user deposited 10 euros then withdrawn

uh 5 and then again deposited I don't know 20 all these events in order are being applied to the initial State and the final state after applying all these events is being built and that cloud the state builds from events uh Cloud picture is uh here on purpose because that cloud is kind of symbolizes the thing that the state is not persisted anywhere it's a state that

is built entirely for uh and exclusively for this command handling uh thingy so after the command gets handled this state gets uh uh gets the commission gets gets removed so if you have the state build only then we can apply the command because we have the current state we have the all the events applied we can we can apply the command and issue a business Logic on

that and after issuing logic uh like checking for example if user can uh withdraw given amount of money or if user can deposit given amount of given amount of money depending on the business rules uh after handling the command you issue uh you produce one two more events depending on your on your logic and these events get saved in a in an event journal and on the

next command uh all these events all these three events from the top and plus two these these two events from the bottom are uh streamed again replied and then next command gets handled so this is the the the right side of the event sourcing one it's not that complicated I mean it may look scary uh until you kind of shift your thinking to that to that Paradigm

uh but after you issue a command after you do an execution uh do an operation on your account you may want to do uh some query to view the state of the accounts be it user interface and user interface be it a reporting for for internal business needs or any other kind of view projection you name it and for that there is a second component of a

of a event sourcing it's called read side read model projections it depends on on on the technology framework or even language uh the name the name may name may be different uh there's a read site which kind of listens for the events that have been persisted to the event journal and builds specialized projections specialized views for the uh at the data you have a bunch of data

you have like event streams for uh every single user account and you can build uh views that focus on particular aspects on kind of you look at the data from the from the particular perspective and there are also event handlers because our applications don't live in a vacuum uh usually when when we decided and we produced an event that that user is uh allowed to withdraw I

don't know 10 from his account because he has enough balance we usually want to communicate with other systems to let them know that uh that that request was handled that that uh withdrawal comment uh was handled that the withdrawal event was produced and uh these are the event handlers for and the thing they do is usually doing some input output thingy communicating communicating with other systems uh

calling synchronous methods uh sending uh messages to message Brokers and all that stuff uh and the uh the most important thing here uh to notice is that all the yellow part of the picture is one transaction like saving the events to the journal uh is one transaction but usually reading the events from the read site and reading the events for the for the event handlers is the

other transaction and maybe executed in completely different service completely different ecosystem completely different database even uh that's why this uh the the event sourcing and cqrs paradigms are usually linked with eventual consistency and basically that looks like the the event sourcing uh idea looks like this uh uh so the first thing is that that I would like to talk about is that you may not need a

framework although it looks like complicated until as I said until you you shift your thinking uh to that Paradigm but you may not really need the framework uh that's again the same the same answer of my body uh but just underlined the other thing uh other quot in there it's a reconstruction of past states by replying these events and this is what we basically need to do

in order to hand the command first so if you code uh if we like show it in the code it looks like that if you are into this is by the way the Scala code um what what we need to do first on handing command is to rebuild the state fetch all these events for the entity and rebuild the state so we fetch the events in the

line two we fetch them from the load event stream for given entity ID because we know this is a user with a given let's say bank account ID then we have this initial state which which initially is empty it's like default state with an uh default settings and a balance of zero and then we take all the events and uh iterate through them applying all these events

to the state uh and we apply every single events to the state that was that was built from that is that is built from uh application of previous events if you are not into this uh functional thingies or the fold left thingies uh that may be more uh clear to you it's just a for Loop we are just iterating and like changing the state and then returning

the state at the end so this is the part when we load all this events and reconstitute this cloud from the verif picture this is the this is the state at the end this is the state that we that we are going to operate on and handling command is basically rebuilding the state first this is what we just did doing the business and doing the business doesn't

do any IO doesn't do any contact with with uh external services or or stuff like that it just is some events that we decided okay the user is allowed to um to withdraw €1 from his account and at the end we just POS this events so we we open this these two events from the previous picture to the uh uh to the database and that's basically it

that's the simplest implementation of the of this read of this right right side of sourcing so you don't really need a framework for that what about event storage you may ask because like we need to store event somewhere so the database pretty much every single database can handle it happily it needs just it just needs a way to append the events and basically every single database if

you throw at it uh problem of just appending events not modifying not inserting not updating stuff just appending new events we'll be happy to scale a lot and uh be happy to be performant uh in that kind of load so we just need a way to append new events to the new rows uh to the database uh for each event and we also need a way to

listen to the events from the from the read part from the read site and from the event handlers uh point of view it can be done via I don't know right headlock from the postgress Via CDC like theb if you want uh or even via simple pulling every I don't know second or two or five seconds you may just pull for the new events from from uh

uh the database because you know the previous events have been stored and they are not modified they are they are they are immutable they are there and once you read them you're guaranteed that they are not going to be changed so this is the kind of read part of the uh of the event sourcing it may need a bit uh of a boiler plate to to write

that stuff but it's not really much so you basically don't need a framework for that but for the for the storage for the storage itself there's a x dedicated database which is called event stb uh which is written and founded by uh Greg y who an author of The Event Event uh sourcing uh concept and he's um he's building this event stb there's a free version of

it there's also commercial version uh in a cloud that you can that you can use that you can subscribe to but also relational database as postgress so we are not limited only to events Tob uh we happily used the post press on a really high load um Event Source systems I know people using MySQL for that I know people using Cassandra we used Cassandra in in one

of the projects back then uh people are using Dynamo DB there are tons of Frameworks uh Integrations for example in a Scala world we use uh we mainly use acup resistance for that or pup resistance now uh they have tons of the adapters for different database backends so if you use the framework you don't have to worry about it but C in one manual is not a

big deal really uh you may ask okay it's about the events so it may look like Kafka is a good fit for that it's actually not uh and I mean it can be built on Kafka with a lot of gymnastics a lot of Plumbing on top of it uh the main thing is that while Kafka can store the events for you there is no way if you

stream the if you store the events in for example one topic for all the entities there is no way you can cherry pick uh the events for each entity when you want to replay when you want to replay uh all the events for given entity uh to to handle the command you are forced to reply all the events from from the entire topic and discard most of

these so that's why that's one of the reasons why kfka is not a big uh not uh not a good fit for that we can talk about it later if you Wonder's a lot of reasons uh why I wouldn't use it for for that for that actual purpose time traveling this is the big thing and this is the the stuff that I really enjoy and really value

in in Event Source systems it gives me basically it gives me a kind of debugger on my production data uh because we are humans we make errors we ship features and we sh bugs together with the features and happens and it may happen that you have an aggregate or an entity that is even s and you have serious of events as on the picture and at some

point you know that users you find out that users is raising a a ticket to your customer um customer support that he cannot log in he cannot uh access the the account his account showing wrong amounts or stuff like that and then what would you do in a CR in a in a standard crowd application you would go to a database select the account by ID select

the user and try to look at it but you wouldn't really know what happened because everything you had is just a state like current not of the state the current version of the state with with even sourcing you have entire history of the facts that happened on the account the events that have been recorded for that account starting from the very first deposit uh through all the

I don't know configuration changes all the option changes through all the deposits withdrawals uh ordered over time and what you can do like in a debugger just go through every single event and try to find one that was that was applied incorrectly maybe this a wrong event maybe that's a event that has a wrong payload maybe for example you have an event that has two I don't

know integer Fields uh or or two decimal fields for balances and you just flip them around and this is what's causing the issues there with this time traveling thingy you can go through all these events one by one and locate and find one that was uh that was wrong but throw event stream like if you go to the database rent stream is pretty much unreadable it's a

bunch of rows in the database uh sometimes it's a in a kind of plain text it maybe in Json like even stb likes to work in Json uh in my projects we usually work with uh Proto Buffs uh binary formats I'll talk about it the reasoning uh and the motivation for that uh a bit later but the the row event stream is pretty much unusable it's just

a bunch of events depending on how long your stream is uh it's just a bunch of events and applying them like looking at them uh from just PL select query from the database is not a not not a really good not a really good thing because it's it doesn't tell you much that's why pretty much every time very first tool I build for my Event Source system

is that kind of debugger thingy uh just to kind of command line tooling or script or whatever it is just to go through uh the journal uh even journal the event log just to be able to navigate and go through all the events to be able to replay events up until I don't know one month from now and discard all the rest to see how was the

state of the entity like one month uh one month ago and that kind of tooling uh that's one of of the tools uh that I usually build over the course of the projects there are different ones for building projections like ad hoc projections custom views on the data uh analysis of the journals and all that stuff so building tooling for even Source projection for even Source systems

uh is really valuable and it's good to build the Tooling in a way like like Build and Grow the tooling it's not to treat it as a onetime throwaway code just build and curate that stuff uh for you because it can be really helpful for for troubleshooting issues but now when we have the bu uh located we know what happened we know that there is something wrong

with within uh within our business logic we need to fix the bu somehow and if we know that the event for example was flipped the the the the payload was flipped the the big decimals were flipped in the in the payload it may be tempting to overwrite the event like just go to the database issue a query for example if it's a Json issue a query update

that field like payload field payload column in your postgress just uh change the uh change the payload of this faulty event for this given user but please don't do that I did it twice and I was scared as every time time I was doing that uh why because the the events are the single source of Truth and if you stored an event remember that there are this

read sites and event handlers that have already processed your event they may have built a report out of this event of this faulty event so the report may be wrong but still it it handled that event uh even helders might have sent that to the message key eveners might have might have uh sent uh request some operation from another system synchronously with that faulty based on that

faulty faulty event and if you override that manually there is no information uh sent down to the customers down to the re site and down to the event handlers that something changed because remember if event when event is saved it's guaranteed by like like by the conception of event sourcing that is immutable it should not be changed once anything process that event it's it assumes that that

event is done that event should not be changed in any way so you have no way to inform the customers the inform the clients in from the read site in from the event handlers about uh about the change that you have just done manually in that that may be not a big deal but assume uh that there is an uh National investigation or or trial or something

uh and there is auditing required for that even so EV Source systems are uh very often used uh I mean even sourcing is very often used for systems that require heavy auditing that require the base for auditing that require single s of Truth to be to be uh written as a as a sequence of events that is easily inspectable that you can that you can reason uh

from the series of events a events uh and there if that was the basis for for uh for your Event Source system and if there is an investigation audit or whatever it is uh and it happens that you change the you had the developer that changed the event manually uh you may have serious issues if there a I don't know governmental uh system or stuff like that

you may have serious issues really about it uh what to do then if we know that okay there's this faulty event and bunch of our customers are uh are down and cannot execute anything because the the app is the app is down for them uh send compensating command and produce compensating event it may look like a tons of work because let's say we have I know million

of customers and like 100 of them is just impacted by the issue uh by the issue it would be easier to just issue a SQL query to override the event rather than producing new command producing new event handling this new event and deploying all this stuff to to work but uh the really big advantage of that is that again you have this fact of a correction recorded

in your event stream so every time anyone wants to re replay the stream it sees okay this faulty event and then bunch of other events maybe and then Somewhere In The Stream down the stream there is a correct correcting event that corrects that that faulty faulty balances and you have the record of that fact and you have the information about that that correction happened moreover having the

event being saved in an in an event stream forces the event handlers and the read sites to handle that event somehow because if they don't have a Handler for that new event they will fail they will Panic they may want to discard it they may want to ignore it that they don't care about it but they are forced to explicit detail that and that way you may

uh correct the reports in your projections in your Rite you may send send again the proper request to the other systems you may send uh proper uh message once again to the to the to the message Brokers and everything is fine assuming the the clients are are okay with like applying the with with uh using the the new version of the event but it's up to them

it's not your fault it's not your Not Your Role to to care then uh this is about the bux in in um right logic how to fix the business logic that you that you write but how what if you have the box in projections let's say you have a a complicated query to to calculate some Financial metrics out of your data and there's a all events are

good in all this all the strings but you have a boxing projection some some calculation was was done uh uh wrong it's even simpler because you take the projection you throw away the projection you deploy the new version I mean you deploy the new version of a code with a fixed logic and you throw away the old projection and you rebuild the projection from scratch because you

have all the EV since the very beginning it may take hours it may take sometimes days so may you may want to throw away the the old projections a bit later but you have all the events and what you do is just interpreting the events uh you just interpret the events stream the new way with a correct logic in here uh this is something that we've done

several times when we discovered usually in a in a very first phase of the projections building when there were some some tiny bugs we've been just throwing away the projections rebuilding them uh usually at the beginning there was not many events so it was blazingly fast uh so just replying the projection having all the event having all the source of Truth in the form of uh event

stream uh it's it's it's really good uh it's really good thing for it's really good thing to have to to to rebuild uh uh to to build correct projections to build correct view uh on your data and speaking of correct views on the data powering insights is a uh yet another big feature to me uh of the event sourcing this is the one that I value a

lot this is kind of a meta thing because I'm quoting Oscar who is quoting Anita uh but anyway uh by the way the Oscar D if you are from Poland maybe you may know him he's a kind of king of event sourcing in Poland uh basically if you in po polish it if you talk about event souring is basically by default Oscar uh but Anita said on

one of the conferences that event sourcing is architecting for is architecting for tomorrow's questions and basically that's really true and that's a one of the best kind of short explanations of event sourcing to me because you have all the information baked into the events like you saved all the data that were that were meaningful when when you handled the command you save them in an event you

have all the data in your database uh you should not limit yourself to not to store given information if you have doubt if whether you should store given information event you should storage is cheap and that information may be Priceless later uh during the business Evolution business needs change over time now they may want two reports two kind of reports that you can happily build but in

few months they may want the third type of report or few more types of report looking at the database looking at the data uh for example for your accounts uh for user accounts from a very different angle if only you have the data in your in your system system uh in your event stream you are good to build that projections you may need to you may run

it and it may take days to to uh rebuild the projections but you have a way to do that you have the data in order to build that projections and that's really valuable and that's one of the cases that we had uh not so long ago in a project that we had a project running for four years now um uh for five years now and we had

tons of events in there and at some time like year ago or something the business came to us and wanted one particular view of the data like to analyze some stuff to to to to get some metrics to get some insights on it what we did we just buil the the new projection it was running through I guess two days in order to be uh to be

complete in order to catch up with the latest events but hey we had the data we could build the projection it was just a matter of time of running it having the the the CR uh way of storing the data so having just a snapshot of a state we couldn't do much really we could only tell what's the current state of the application and that's it so

yeah build any projection you want answer any of the business questions if only you have the data in the events and as I said before the event streams are a really good base for for audits because the event stream you can just like format the events in a kind of human readable way and show it to like hand it to to Auditors this is what happened on

the given account uh we have the full history of of that data consistency any guitar players maybe in here no oh okay uh this is a kind of one of the guitar pedals it's Trion Volante which does a kind of delay thing like Echo thing is so like delaying the sound a bit so I was thinking that's a good uh illustration for the data consistency eventual consistency

and and all that stuff so basically that's the same picture uh as before but with this this two clock icons added to the event handers the side because remember IAL uh I mentioned that the that the yellow part is one transaction so the events are being stored in one transaction and the read site and event handers are usually eventually consistent usually because it's something it's your another

process that can connect to your application just pull the pull the database uh every I know second to get new events uh to apply to site or to to uh apply event handlers to uh it may be CDC it may be rer head log but still that's totally different transaction from from uh the right side uh this gives you awesome benefits of uh inform of ability of

scaling uh that uh that that read site because if you have so many read site you can scale them horizontally as crazy and also you can and with without impacting the right side because it's totally separate transaction totally separate database and and uh all that stuff so all the benefits of cqrs apply here but the eventual consistency may be kind of scary and sometimes uh you may

come to the point when the strong consistency is really needed I mean my hint and my experience is that usually if you ask business if they want to be strongly consistent they will say yes but if you ask them what's going to happen if the data is going to be there in like 2 seconds 5 Seconds ah not a big deal it's fine uh then it turns

out that you can happily live with an eventual consistency and uh make use of all its benefits but if you really need a strong consistency there is a possibility to do that especially if you handcraft the event s uh Thing by hand and you do this do this all uh even saving by hand without any framework uh you can have EV journal and your read model that

you want to be eventually consistent put it in the same database put it in the same um uh same transaction and once you read once you write the the event you may also update the read model and now you are strongly consistent after the transaction saving events you have the strongly consistent read model that's one of the options that obviously limits you to having the read model

in the same datab base uh so you cannot really use uh all the benefits of different storage system for different purposes like if you want this read model to be used for heavy analytics you may want to have I don't know columnar database document database uh but if you have events journal in postgress you are forced to make a trade-off to to have it in the same

the same database to be able to uh put these two in the same transaction but there is a kind of risk with that and I saw that in one of the projects that events have been saved uh consistently with read model first it was one read model then it was uh then then it were two read models and then it ended up with five or four read

models and that causes a risks uh because first uh every single additional update to storage uh increases the latency and you want your read s you want your right sides to be really fast because it's just appending and saving events is just appending to the log it should be really fast to handle the commands for from from users and introducing this updates to the read models uh

uh May introduce consistency may may introduce uh another latency and also the fact that every single logic in a read model like interpreting the events uh may be buggy and then if any of these three uh uh read models fails you are failing your event uh persistence transaction you are actually invalidating the comand rejecting the comand because of the fault you don't want to do that your

primary goal here is to handle the events from the user handle the comments from the user and produce the events and what should be done on the read model should be independent from what happened in the in the in the events Journal that should be totally separate processes so there is a way to do a strong consistency but use it uh use it wisely and limit that

to as few read models as you as you can if that's only uh and usually uh embrace the Embrace eventual consistency and its benefits uh it as I said usually most of the businesses uh can go with eventual consistency on board it's all fine there's an option to be strongly consistent shift of thinking to eventual consistency way uh for example there are ways to model I don't

know user email uniqueness uh in in eventual consistent way uh using event s agregates there are some some talks and some some articles in the internet in the in the internet about it not going to talk about it because this lack of time uh and the next thing performance of Event Source systems uh usually Event Source systems are the ones that are kind of critical uh and

that need high speed high performance at least the right performance of an event like high ingestion of of uh user commands uh and it may happen that you have a long stream of events you have a user that is uh trading or making Chang on on his account that is that is uh that are so frequent like several or seever several hundred trades let's say per day

so if you have that account modeled as a user account your event stream May grow and grow if you have t if you have 10 events it's fine if you have 100 events is still fine to load them in one go from the database to reply them in the memory and and apply the comment to that but if you start having thousands millions of events for one

user uh and it's not um uh and it it may happen that if user has an account for several years now and it's pretty active it may have millions events there's a way to do that to overcome that but it's usually kind of an optimization thing there's a way called snapshots uh that some uh some people use for that what you basically do with snapshot is you

have this event stream and every let's say 1,000 of events or every business condition that is that the account is in you make a snapshot of the state you save it in a in a site database and next time the command comes in you First Look up the the newest snapshot for the given account load the snapshot State and apply all the events uh that happened since

this snapshot uh so you don't start uh with a empty stream as we started before at the beginning you start with a snapshot and then apply only these few events that have been there this uh technique dramatically increases the the the the load time and decreases the load on the database because you don't have to load millions of events uh you just need to load I don't

know 100 maybe few hundred of events together with a snapshot but be aware that it's an optimization pattern only it's like it's like fixing like hot fixing the the the Event Source system hot fixing your thing because it is a sign that you may want to take a step back and see if there is no hidden concept of a business entity that can work well can work

way better uh in your domain than than this one giant user account for example for user accounts it may be that you may want daily user accounts so for example on every single day there is a roll out of the user balances and then you have short streams with I don't know hundreds of events for the given user for the given day and you just roll out

for the next day when the command comes in then you have multiple shorter streams which are way faster uh to uh uh which are way faster to uh to reply when the Comm when the command happens because for example you don't have you don't have to care about the previous events from the previous days because you have the initial state of your entity to be this this

uh the draw out from the from the previous day so maybe modeling your Domine slightly different uh maybe that that that snapshot need for the snapshot thingy kind of uh uh gives you a hint for the for for the new model model that you that you haven't yet discovered in your in your domain it also introduces state to maintain because for the snapshots we need to sture

the state of of the entity so we kind of do this kind of crude thingy we stay save the current state of the of the user account somewhere in the in a database and we have to maintain it then over time we we don't have a single source of Tru in in form of an event streams but we also have the snapshots to to uh work with

uh with h entities that are that are uh very often uh very often touch and there's many operations that uh executed at them again thinking about the the domain and thinking about the the boundaries of your of your entities may be beneficial maybe there's something you can you can do better uh it's an issue because in the stateless apps that we used to build it's the best

way to build the applications that are scalable in a stateless application uh if you have several nodes the command from the user can not can land on any note and then any note should be able to uh wake up the the user account entity reply all the events and execute the command and you can imagine for for hot entities there may be tons of stuff uh tons

of operations on a database uh to do that if the entities are are frequently used and there's a lot of a lot of commands for for given entity so database becomes bottleneck and command handling slows down because first you need to load all these events and then you need and only then you need to handle the handle the apply the commands to uh to this event and

also the concurrent modifications if you have a user account and they suddenly two commands from the user suddenly land on two different nodes how do you manage the concurrent access for that maybe optimistic locking maybe versioning but then introducing optimistic locking uh and versioning kind of uh amplifies the load on a database because if there's a conflict if there are two operations pretty close to each other

uh if the first operation succeeds and the second one fails you need to refetch all the events again and again and again in order to to uh find the correct version and find the right spot to to uh save the event with a save the new event with a correct version so we are amplifying even more amplifying the the load on the database uh what's the solution

for that what if you keep the entities in memory for some time that's doable but then if we have several nodes there's a there's a problem with that how to synchronize them uh so then how to how to guarantee the exclusive access to them to the to the entity so that if you have if you have a account on two nodes there's no such thing that you

have a count on two noes if the requests land on two notes they are directed to one single instance or a copy of of uh of entity that handles the uh the commands for the user there's the thing that is called single writer principle uh and it it does exactly that it moves the complexity from the database it kind of offloads the database uh to the application

Level it introduces some complexity on the application Level itself uh but fls uh the load on a database and while database cannot be scaled that easily as applications like by adding more nodes application uh it may have significant uh it may bring significant significant improvements uh so basically what it does is it for example in in our Scala word we use usually AKA cluster with with cluster

sharding with AKA persistence as a Event Source implementation so what it does it the the nodes in of the application form a cluster and it's guaranteed with a with a cluster sharding is guaranteed that that uh all the that within the cluster there exist there there is only one instance of a given account uh user account it may be brought into memor into the memory and kept

into the memory serving commands very fast and guaranteeing the exclusive access to uh uh to the to the entity but again that offloads the the load from the database but introduces some complexity on the application Level Evolution and maintenance uh of the Event Source systems they usually live long uh if we invest into the Event Source systems they are usually uh they are usually kind of critical

and they live long they are not like one time throwaway applications so think about your event versioning strategy since the very beginning uh think how you are going to version the events how you're going to add new events how you going to the commission fields that are no longer needed in a given event and so on and so forth choose a payload format that supports uh the

evolution this is why I'm for example choosing the Proto Buffs because in proto Buffs there are clear rules how you how what's the compatibility of the of the uh uh of your event rather than in Json which is kind of structured but there is no clear rules on how you can how you can evolve the Json uh Json schema uh choose carefully what goes into an event

um you may want to save as many uh information as possible in your event but not too many I saw a project where people were saving entire state that was built in this in this previous Cloud they were saving it as a part of an event for basically no reason I don't know why it was done but but it was kind of to the extreme uh and

on every single change if you want to uh do the change uh on a business logic think whether you want to have it as a new field or new version in your event or new optional field in your event or maybe it deserves on a completely new event maybe the event structure is the same but it's totally different business process and it's then it's worth having totally

new event with its own name uh with with its own meaning rather than just having event like version two or event with a kind of yet another optional field and a keep event handles for the past events even if you don't use the feature if even if you decommission the feature uh keep the events because if only the event has been uh if only the event has

been uh written to the database uh there is no way back you need to handle that uh forever then because if someone wants to reply the stream if there's no Handler it will Panic there's a great book uh of Greg young it's not finished but still versioning in an Event Source systems I highly recommend to read it even if it's not done there a lot of strategies

a lot of issues with that that that may have happened uh that you may benefit from even if you're not using given Source systems how to version payloads how to migrate payloads from the database and all that stuff uh scaling R side the quick one uh on the events Journal if we assume that we have this polling for example we have multiple cons consumers of this polling

polling the database the more consumers we have the more load we put on the database from The Rite uh thingy and that kind of limits the number of consumers or read site models that you may they may want to have again um a problem every pretty much every single problem it can be uh can be solved by adding yet another abstraction layer so it's in here except

of the issue of too many abstraction layers uh so in here there's a message CU between that so if you put for example Kafka which is a one consumer that reads from the even Journal then on the other end of the of the topic we can scale as crazy we can scale as to as many read models as we want and private versus public events uh it

may be tempting to produce uh with the event handlers to notify all the uh customers all the all the consumers of our uh clients of our systems with all the events that we have I wouldn't do that I would usually keep all the events that we have private and only publish the events that we have that we know that we want to publish that are business uh

that are meaningful from the business point of view that we have a strong contract with it enables us to uh doesn't work uh it enables us to to evolve the internals evolve the event maybe sometimes uh just completely ditch event sourcing while still uh maintaining this outer layer of of API uh compatibility so don't tie your customers to to your private events only publish what what what

you want uh what your other what other services wants from you to be uh to publish and oby contracts uh this gives you yeah the ability to change the internals change the events migrate the events maybe ditch ditch the events souring out so the summary EV souring is not really a new thing it's like many years old uh it can it can be implemented without specialized framework

uh Embrace eventual consistency it may sound scary but that's a way to scale your systems like crazy if you want to scale have the versioning strategy from start because if you don't think about it uh you will have hard time when then changing your requirements your business requirements changing your your flows changing your your events payloads uh make use heavily of the Rite power it has incredible

power it can look at the data from very different perspectives that you may even not think of at the moment and challeng your have with something and have fun thank you thank you a lot for your talk very like a lot of information so the the problem is we're a little bit out of time there is a panel starting if somebody wants to go just go there

otherwise let's go and take a little bit of your time out of the lunch I'll be there I'll be there somewhere maybe in the in they ask me anything Corner after that but if you want we can go over let's say top three questions first one you did present the snapshotting but are there other things how does one handle increasing replay times when the system gets older

with time years for example uh that's that's the the thing with snapshotting is a kind of cure for long streams so if you let's say you didn't model your domain let's say properly we even sourcing we should Target for shortly for many streams but that streams that are shortlived so for example this this this thing with an account instead of building one entity for user account that

is like spanning long long years you may build one that has I don't know span of a month given user and let's say it's May of 20 2024 and then next entity of the same user ID and June 2024 rolling out this current balances like starting the new entity for the new month with a with a balances or the state from the previous uh previous month for

that you uh that when the when the comment comes in uh in June from the user you only need to reply the June part of it all your views all your read s sites are still correct because they are built from the very beginning of the of the event stream so all the balances are correct on the read Sid you have all the data you have all

the events so you still can build all the uh all the new projections from from this data thank you how often have you had a feeling you are over engineering the things with using the event sourcing uh twice uh twice uh twice and once we dited event sourcing after some time after the system was running for half a year on production uh we dit that because uh

it became complicated we we we went with a domain modeling totally wrong we did this all this uh uh errors with a with a long streams and all this uh we did that because there was no uh there was over the course of the project it was around a year I guess and there was no single use of uh any of the benefits from event sourcing basically

so we we decided to disit but yeah it's it's really tempting if you have a Hummer like everything looks like an L so it it may be you should question that if you really need auditing if you really need this time traveling if you cannot go without it that's that's not easy pattern actually to be honest okay and the follow-up question again I said we only do

top three you said you had to fix um events manually two times why why did you fix two you said that normally you have the complimenting um yeah comp compensating action yeah but you had to implement uh the fix on the database manually two times yeah that was for the the okay one was uh the the project that was built with an event souring I don't know

why really uh but it was internal project there was no need for auditing there was no need for for you know projections and all that stuff so it was easier just to go go to uh to do that and the other the other time was when we just didn't think that we could do that and there was not many consumers on the other hand like even handlers

that that we could screw so it was it was time that okay so no since now since now I don't do that it's like doing that on production is like you can sweat a lot doing that like it's a neurosurgery creating parallel timelines yeah something like that like you cannot change the past so gotcha all righty again you can find our amazing speaker outside of this Hall

um the currently is the lunch time so try to catch him and ask more questions in details