Daniil Doronkin: A Software Engineer's Journey Through Microservices Data Sharing

This talk highlights the journey of transitioning from monolithic architecture to microservices, with a primary focus on data sharing techniques among services. The speaker shares personal experiences and challenges faced, particularly regarding the evolution of communication layers necessary for effective data exchange. Various data sharing methods, such as synchronous and asynchronous communication, are discussed along with their trade-offs. The session dives into common issues that arise with shared databases, emphasizing coupling concerns and future scalability challenges. Additionally, the talk explores a framework for decision-making in selecting appropriate patterns based on specific use cases, characterizing the importance of understanding the unique needs of a project.

[Music] hey welcome back everyone second round of talks here at the death days Europe and uh I'm your host FL and rle but I'm not alone with have an awesome speaker coming up here Danel doronin hi how you doing hi I'm good how about you I'm also fine thanks a lot so you brought for us a really nice talk uh a journey through the uh microservices data

sharing um what is awaiting us in your talk oh I hope you enjoy it uh here I just want to reflect on my personal experience how I work with this challenges when you're switching from monolift to the microservices you know it's always question about data how do you exchange it and uh in this talk I try to wrap up my experience and what I learned from the

past I yeah hope it's gonna be also useful for audience today awesome great yeah this is also one of the things I've had to learn the hard way I mean going back let's say 10 years already I mean the if you construct a microservice uh solution from scratch it's always the case so what are you doing with with data sharing and uh how self-contain should be your

assistance and I think therefore hearing from your experience especially at booking.com I think it will be highly appreciated by everyone so therefore I want to give you the stage have fun and uh thanks already yep thank you so much thank you for introduction and uh such a nice way to start a working week yes with a such great of event and um another good way to start

your day is has some controversial topic about the mon verus microservices I think argue a lot about what they of consistency should it be strong or we need to switch to eventual one what kind of communications can we use synchronous and synchronous or and why and finally should we give up on the model Simplicity in terms of microservices independence level uh those are good questions uh in

my case as I said before it was always um how to evolve the communication layer between Services when we switching from monolith to the micros service because now it's getting more complicated so today I want to discuss it with you and um yeah my name is Daniel I'm a software engineer who was born in Kazakhstan many years ago but last four years I'm living in the Netherlands

and currently working booking.com I work mostly with the distributed systems and I find it really excited and uh it's always new about that you can learn a lot but still have huge amount of things ahead which you have to cover and stay tuned which is I think great and aart from then also Tech talks fan both is attendan presenter after this talk I'm hope to join a

couple of sessions and uh hear something from other speakers here as well before we dive into the details I want to say that I don't want to cover uh model versus microservices and proc and cons here also how to split the modelet into the number of separate services and we don't cover disb transactions and sagas all of that topics are great and deserve the separate talks and

some of them already well covered so instead I want to focus more on data sharing techniques how can we exchange data across subdomains what kind of communications we'd better apply synchronous and asynchronous and why and talk about the trade-offs because yeah software engineering all about tradeoffs and we also have some here a bit of disclaimer here in this story I'm not going to talk on behalf of

my company name or the one I used to work for previously it's my personal experience which I want to summarize and present here and also reflect on it a bit and now we are ready let's go and start our journey and let me introduce Alex Alex is my friend but more importantly he's also a beckan engineer who did a lot of nice things in the past he

buil some systems solve some solutions complicated challenges and once he went to Italy have an enjoyable trip and fall in love with one of the pizza here and from that day he started to experiment with recipes finding for the best place and set up a mission from that day he decided everyone in the world should try such a nice Pizza War wild and how can we accomplish

it so being software engineer he can create some kind of platform in that case he created the pizza as a service a monolith app providing some features like easily unboarding the new product for businesses who wants to produce Pizza can just go to this platform and make the uh suggestions here also some common uh features to utilize like Warehouse deliver analytics Etc and from Another Side he

want to provide the outstanding user experience for the audience so they can go to the app website and do whatever they want and get whatever they want and in turn people like the idea a lot and more and more businesses start to important on this platform and not only pizza but other kinds of food it's become like Netflix for food streaming that sounds great because mission is

close to be finished however Alex faced a number of challenges from one side he has customers who want to see new feature released as soon as possible they want to process more orders because obviously just increase the profitability of the company and they also want to make it fast so the platform should work fast the process of onboarding should be fast everything should go smooth from Another

Side he has development team who have to ensure all this requirements and they want to maintain the level of independency they want to independently de develop test and deliver Cod they probably want to utilize the best paron approach on the market like even store for some cases and uh they mentioned microservices one of the option to go towards this direction to ensure this independence level well what

we do when we don't know a lot about the technology go read some books about the demand driven design about the microservices check a couple of reads maybe have a crash course from Zero to Hero and now we are experts in this topic for sure but at least get better what Alex did and came up with the brand new design instead of the monop he has a

number of well-designed and bounded Services um but there is something suspicious here and I believe you already noticed that all of them still using the same shared database in the middle and if we go check uh best practices for data for microservices implementation or ask your colleagues or simply just ask CHT you will see that it's something supposed to be antient have to be abandoned aing costs

and it has its point however is it really that bad and if we can leave with that approach uh let's take Alex and his team and uh help them to figure it out because there are clearly some benefits from using the shared database first of all we have AC transactions and we can run complicated operations across subdomains and make them atomic that just that is good we

still have strong consistency here second it's easy to quer and join data because we can utilize the database sension which designed to be efficient and provide such kind of operations from the scratch and finally we don't have fur development efforts you remember we still have some features to be released as soon as possible so we can invest more in the product development instead of this huge refactoring

but don't benefits go with the drawbacks here and first of all is coupling and uh that's probably already enough to switch approach because coupling is something we try to avoid when we go from the monol to the micros Services approach let's take an example here imagine to have users in Delivery Services Reading data from the same user table and uh it's fine until one day delivery service

decided to split Building address into the three separate Fields it's just easy on their side to process data so they came up with this idea make a merge request push roll out and after that user service got breaking because got broken sorry because on their side the code base is still waiting for the old database schema it worth mentioned that in production it's barely uh going to

be a problem because you have some smoke test between services and you have some pre-production environment where you have to catch this problems or maybe have some kind of code review process which doesn't allow you to do such kind of things however there is still uh necessity to consolidate changes across two teams and if it just two it's fine maybe it has five six seven it may

be even more complicated and uh it affects time to markets so we need to figure out second we actually can have some kind of isolation level even on the shared database level by utilizing the specific schemas we may say like user schema is for user service delivery for delivery we can have some kind of rules access Etc and don't allow to read from uh wrong places and

we can even have some public views to combine the complicated queries and reuse them across Services still coupling will be there because these views are construct from the tables and if you have some changes on the table level you also have to update views and do it in the non-breaking way so but worth mention that ability is exists the second drawback I see is the database being

bought on NE here uh you know in few weeks we have a European Cup in Germany and I would expect more people is going to enjoy pizza with the friends and see some games so we may have more traffic on delivery site and how can we handle it okay let's just uh add more instances now we have a separate service for that however when uh Peck days

come we may see problems on other services and that's probably because there are still same database handling all the requests from all the services and if we need to ensure the traffic is handled properly we also need to scale this database because it has its own CPU memory network configuration that defines the bandwidth so yeah that's something also have to be adjusted and if we're talking about

the vertical scaling getting more resources to note possible but we don't know the exactly lot we want to handle we don't know how many services going to scale Etc if we're talking about the horizontal scaling in any more instances uh that's possible but also challenging especially for relational databases they supposed to be single node ones there are solution here like services like a Persona cluster or maybe

Cloud Solutions but yeah that is something we need to deal with um if you have shared database it might be hard to evolve in the future if you want to utilize a different data management approach or just change database type maybe you want to go to Cloud now you're on board adio or Azure so you want to go there but if it's shared and coupled it might

be challenging and takes more time and um yeah finally the polyot resistance that is something about using different database type by different teams up to the use case for example someone want to work with the even sourcing and they prefer even store some of the team has like graph like data and maybe now forj is better Feit for them if you have shared uh I don't know

SQL database they still work with that but it might be not that convenient and uh not the best option in the market something also mention and just to wrap up this part I believe shared database has its own benefits and uh they are good but significant drawbacks such cing and bottleneck may drive us to change the decision and that's what happened with Alex and uh his team

they decided to embrace the big refactoring and switch approach and having database per service pattern here looks good and they're about to plan work and uh start next Sprint with the splitting however the question is now we don't expose database API Cross services how can we get data from different subdomains because we have a feature in the backlog that's asking for review cards uh in the menu

item interface and here people want to see some data about the menu item and at least four Services part of this card like we have user photo we have review score we have some us about the menu item itself like a name and when order was processed so four Services combined here to provide this user interface data interface um obvious solution here okay we don't have database

API exposed to public but what if we do that on the application Level and services themselves provide a public API so in this case we can say review getting data from all the parts do the aggregation prepare the response and reply to the frontend with the suitable data to consume that's fine we can even use already existing technology like maybe restfully API or grpc maybe have a

graph code on board it but also we need to maintain the level of uh complexity for example we need to take care of configuration service Discovery because review need to know which Services have C under the hood and also for tolerance experience here what I should do if some of the services respond with the error should I make a retry should I have a kind of fallback

should I partial response to the client or maybe fail the entire request that is something to figure out that may be even more complicated when you're talking about several services do the same for example analytics collect data from the same parts and also have to implement all this configuration code here well there's something to think about and uh yeah the first idea maybe we can have something

like service mesh which encapsulate this infrastructure work Ure the fa tolerance and configurations here so we can just run it like a side car close to our container and just utilize and reuse the same features might be Improvement however business part still here we need to get data and combine them maybe prepare for the response so instead we can reconsider approach and have a API composition pattern

here by introducing the service in the middle which is responsible for data collection and aggregation which keeps the configuration complexity and also ensure the same small full tolerant experience and uh yeah talking about this orchestration like approach I thinking about the graphical Federation as an example for example you can have number of services which expose a graphical API to the public as a subgraph and this subgraph

may be unboarded to the Apollo Federation register inside and now the rotor also part of the Federation analyzing the incoming request smartly naate to the proper part to collect all the data synchronously synchronously depending on the request itself there are some dependencies and provide the combined response to the calling site might be an option might be an option because it feels like we maintain the service independence

level here if you have to on board your one you just in create a service create a graphical API and uh regist Federation you don't have to think about schema teaching because it's working out of the box by Federation and this solution also has good orentation and examples and also paid uh subscription which goes with the support but yeah there are also drawbacks here and first of

all for sure you need to lock on the graphical dependency if you don't have it it may takes time to bring it and I know some people who don't like it so might be a problem uh you can't find Ro Behavior inside it's working based on some internal mechanism and uh you don't go inside it's a blackbox at least I didn't find a way and I don't

think we should do that if you want to make a request fast yeah not much options there and also it has its own learning curve uh you have to invest time to learn graphql how Federation War Works how to set up environment etc etc so if you go on this way there are some investment and time and skills have to be here as well so trade-off getting

back to composition U now we have a single point responsible for uh collecting data and providing the best experience for calling services that also mean is getting the critical point if it fails everything fails behind it also we adding extra latency because we have one more extra Network call we expect more time to get data on the front and S side and also it's not clear how

do we scale Downstream Services here because if we handle more traffic on the composition level we may need to scale Downstream correspondingly alternatively we may think about caching uh but it's also complexity where this cach should be on the composition level maybe it should be distributed across Downstream Services maybe on all the sides and it's also have to be invalidated so that's a also takes time to

implement and needs Investments yeah just let's WRA up this part what the real time data access when we're getting them on demand calling the services uh when we have some requests we kind of have a composition way which allows the smooth behavior in terms of data aggregation which keeps the the photo lears experience there but yeah we know that's getting more critical service which have to be

maintained properly we also increase latency because it's a network trip and we don't know how to scale it efficiently in independently and uh the question from Alex which is actually good if latency and scalability are big concerns what can I do about that for example he has a kind of customer face application which should work really fast and the front end should get the data as soon

as possible and provide the smooth and uh best use experience for the um users well what if we query data from the domain as we already have everything need and for that we can actually bring data in the background by utilizing integration events as a public contracts anytime many users or orders have some updates they put the message into the queue into the topic and the subscriber

or consumer in this case it's a review service just subscribe on these topics updates aggregate data somehow make some calculations maybe translations and put into its own memory in that case we getting data as synchronously not on demand but just in the background so we kind of eliminate Network um dependency and uh don't worry about latency anymore that much we also can scale uh this review service

independently because now the downstream services not call on the real time and we can just do whatever we want with the review domain itself and yes since we're getting data inside we can make them ready to be consumed by frontend we can make the translations of them we can make them better in terms of efficiency worth mentioned that there is a case that the data here is

stale because we don't know when the event is processed we don't know how many events in the topic to be processed and we don't know if something went wrong in the middle of the process because there are many parts like produc producer consumer the topic itself infrastructure layer maybe some data get lost and we don't have them in this datab so they kind of not synchronized with

the um source of true what Alex also noticed is with that setup it feels way more decoupled now Services know nothing about each other and they just interact with um each other on the topic level by consuming and producing the events as a public contracts and that's true however the events have to be designed carefully let's take an example imagine Manu service has just one topic universal

one where all the menu item updates published so every time FS the change we just send the like old new state and provide it for consumers consumers can subscribe on it and uh do whatever they want it may feel quite convenient because it's just one topic not a complicated setup and also you don't have to change a lot for the new use case for example new service

have to be on boarded and consume this topic it's already exist just use it and uh that's here however this menu item updated schema very similar to something we have internally for the menu so it's kind of maybe a copy of the database scheme of this item and we kind of leak in internal implementation dets into the public and when you change the menu item database schema

on the menu domain you may need to provide corresponding changes for menu item updated meaning that consumers also have to be updated and now we're getting back to coupling despite of using events and no direct dependencies coupling us here we need to do non-breaking changes we need to do it maybe incremental in a few steps probably the better way is having more granular and specific business oriented

events instead of data oriented and it's time we have a new menu item name we just put the new item name event if promotion is requested we can send the item promotion event and if it's out of order just send the even not available and from right side if service interested in this specific update it can subscribe on it it doesn't have to listen for all the

topics also you may notice that the even contracts they are very small and barely change so if it's item id it's must probably stay the same for a long time it feels more safe and uh well organized from the de coping perspective and when I'm talking about the events and that approach the change data capture approach came as an example um some kind of CDC tool May

listen for database updates and provide the uh transaction Lo events updates to the broker broker May distribute them across the consumers so in most cases it's mentioned as the way to fill in data inside warehouses or make some audit tools probably invalidate cash but I think it also works for other domain services but yeah keep in mind there still same concerns here uh in terms of EV

translations um we don't want to expose the internal implementation data of the specific domain into the public we want to translate this message into the uh easy to consume public contract that's just and no events come for free because you need to have a infrastructure level which you have to maintain it's maybe I don't know Kafka cluster or maybe some service solution you B you have to

pay it you have to ensure the high level observability and the error handing of the system because with a synchronous Communications it's hard to catch the problem and it's hard to react on them so we need to have some kind of res resiliency logic here probably as well there might be cases there no real data uh real- time data exists as we covered before something went wrong

or even not processed yet so if it's critical for your use case might not be an option yeah and uh if you already have a huge amount of data before events are here uh you may need to transfer that data into the new domains and um maybe just scan database once and put all the events from the scratch to re process them or maybe we can have

some I don't know migration tool which request data from the database transfer and put it to another one that is possible but that's a concern you have to take into account and yeah after this meeting with Alex and team it feels like even not easy to use and I agree it's uh definitely increased the overall complexity of the system um but maybe we can have kind of

incremental steps and make the easy approach easy to implement and uh we actually have an option instead of utilizing events we can have kind of materialization job which runs when it's configured gets data from different places and bring that to our domain in that case this motorization job still can use service to service Cod I know restful API by getting data and it can uh do the

data translations before put to the storage it might be the background task of your service it might be the crown job which is running inside kubernetes for example it might be serverless solution you can configure the L you can config Crown schedule so it's up to you and um in this case we don't use events but we still uh make this mization job and reviews internal implementation

details so we can scale review independently and we don't depending on the downstream services on demand because the process getting uh happening in the background and it's also totally fine to combine the approach imagine you have a long running job which takes one hour to process huge amount of data so you don't want to run it every hour not to pollute your resources so maybe you can

run it once in four hours and if some of the services already exposed events or just started to expose the events you can also subscribe on them and just uh make them your data more fresh so that's totally fine to do this combination if you find it useful and to wrap this part with a syn data access we don't have Network travel on demand because we're getting

data to our domain from all the places in the background we can scale independently now because that's the only concern of the specific service now we don't scale the underlying service down stream once and yeah we qu data within the boundaries however we need to pay costs like infrastructure for example for events we increase the system some complexity for sure and there might be possibility to work

with the stale data it might be not up to date it might be not synchronized for the entire system it's kind of eventual consistency here and that's more or less it about the options I wanted to discuss today and the key um point that I noticed when preparing this stuff and talking today is there is no best solution we have different options and uh up to the

characteristics up to the necessities and the use case it might be better fit or might not be if something went wrong for one te it's not necessarily your solution and the question from Alex now okay we have too many options we already increase the system complexity how can I actually choose the right one what should I do to understand that and that's a great question which I

also ask in myself a lot of times um let's try to came up with kind of decision-making framework because we're not guessing when we create a system design we rely on the specific characteristics functional non-functional requirements let's investigate what are the requirements for our system let's bring it together and for the every specific use case we can prioritize what are the critical characteristics we need ensure let's

take an example we have a request from business owners that they want to see the menu item performance report uh within the hour within week day week month any time range as possible and they getting data from menu orders and reviews so we need to combine data and provides a nice dashboard for side and in terms of characteristics it's critical so time to implement should be fast

since uh the short period is possible for item performance we prefer real time data because even like two three missed items May uh change the picture yes since we don't have a lot of time to implement we would make the system complexity on the uh maintainable level now let's put all the options together along with the critical characteristics and run the trade-off analysis about the real-time data

we know that service to service C an API composition uh getting data on demand on the fly so just getting them from the source of true domain this should be real time but might be an issue for integration events and data mization because there might be St data here so now we can choose between first two and uh forget about last two options time to implement service

to service calls feels easier because we don't have to introduce a new service we don't have um do some infrastructure work the deployment work etc so just feels like less work here and for the system complexity we already have some restful apis here the only thing we need just to call the endpoints and get data here uh for API composition we may need more work and it

might be more complicated setups so we decided to go with the first option I want to point out that despite we choose this one for this specific use case it's not necessarily the best solution because in the time the apad composition may be better because we can reuse it across other services and uh yeah that's just better way to organize things in your system might be better

way but yeah running the trade of analysis here considering the circumstances and the limitations we decided to go in that way but there's another example now the application users they want to see the menu item reviews you don't remember this review card which you have previously now we're getting back to this jira ticket uh users want to see some comments about the menu items and you may

notice that exactly same Services providing data manyu orders and reviews but now the characteristic may because that's a customer face application which provides data for the front end and they have a high requirements for latency we may need to scale it because of the peak hours and also special request from product owner that's not a feature which uh gives us money directly and that might be experimental

so can switch the approach in the future let's not invest a lot into the infrastructure costs so those are three we want to analyze between and they do the same latency is the better for last two events integration events in data mization because we don't have extra Network trip and the same for scalability we don't have to think about the downstream Services uh scaling that's the only

concern that the specific domain we're going to use for this feature in that case reviews so now we can choose between the last two options and uh infrastructure costs feels better for data materialization because we don't have to on board Kafka cluster for example we don't need to get any Cloud solutions to that um so we just create a job or create some background task on the

existing service which do the work and provide the data for us so in that case we decided to go with the last option and uh yeah you see we have a same data providers but we have different use cases and based on these different use cases we had characteristics that actually drives our analysis and uh yeah that's just to wrap this uh decision working framework decision- making

framework uh first we collect the characteristics next we Define the use case and filter all the critical ones the round the performance uh performance trade of analysis sorry and finally make a decision uh I think that works that works for me should works for Alex and I use it not only for um business cases also so when I need to choose the specific database option or refactoring

or whatever I can run it I can see the benefits and the trade offs and that actually make the decision more strong and confident and also helps to get feedback and provide your visibility colleagues and the key takeaways here are the context is important if something went good for one team it's not necessarily your best practice and each single use cases may be evaluated separately and may

be specific so we need to keep this in mind the second key point is characteristics drives the analysis we try not to guess about the system design we try to came up with the meaningful number of requirements and uh build our solution based on that and finally there are trade-offs there is no Silver Bullet no single solution that fits everything and works perfect even from mon microservices

if you want to go the microservices way you have to give up on the Simplicity over the independency and yeah working with microsources brings lots and lots of challenges I have to with and I want to say thank you everyone for participating today uh was great not to be alone and uh describe the idea for Alex and his team and the more important now he's ready for

his mission to make everyone happy with this again awesome thank you so uh there are not many questions but uh let's uh kick off with a question from your personal experience what would you say was the let's phrase it like this the most difficult part learning so maybe difficult in the sense that you know that uh um you already tried something and maybe it it looked good

and you went even to production with it but then something failed and so that was a let's say a hard lesson to be learned right because it wasn't obvious in the beginning yeah yeah sure I think there were a number of things like that I don't recognize exactly ones but this changing from the monolith to microsof approach definitely because I mean if you start let's say eight

years ago you don't have a number of tools already you don't even know like a best practices maybe you don't have such experienced colleagues around you who can help with something for example very first approach I've tried with micros service was kind of crowd oriented I just make some like Crow Services put them in network and say that's a microservice but that's actually not because no domain

boundar is here it's just uh kind of separate depl playable unit but it doesn't make it easy in terms of design so applying domain driv design is challenging I guess and uh for me personally because it's um it takes time it needs you be either domain expert or you have to you have to have domain experts that helps you mhm so that's one of the challenges I

had okay great uh then also a question from Maxim uh when to choose patterns when to choose design patterns that's a great question I think it's better to choose them before you have already some kind of implementation and um from the past experience I can share some approach how can we does it in the previous working company and the current one we had the some kind of

propos let's say from product owner we need to do that that that so then we start the investigation what do we need to involve what kind of Integrations we need uh we collect all the sites all the feedback and organize it as a kind of document with all the things now you can uh check for challenges check for tradeoffs you can provide the high level solution to

that collect the feedback from colleagues if you need to involve something outside maybe some architecture could help you you can also do that it's ready uh so then you can start with implementation but I would also say it depends on your company depends on your use case if it's really simple and you have no DBS about it I don't think such a big process required so if

it's a simple solution and you already does it in the past if it's something which is already used widely in your company and you know for sure will be in that way so yeah just might be like this all right great then we have another question from SAS uh can you tell more about event schema management management um I think that refers to the slides we have

about this menu item uh yeah the maybe the main point again to try make this public event contract independent and non containing any internal implementation details because you have to change it eventually and you have to make a non-breaking changes and all your services have to be Consolidated later um but yeah that's apart from that I think nothing special just work as as usual with the events

uh maybe if we talk about some infrastructure also depending on the company if you have dedicated team who takes care of your let's say Kafka cluster or support the Cloud solution they're always here for you to provide the best practices and you can ask them if you implement by yourselves yeah just try to follow the meaningful approaches here um that's what I would recommend also about the

events maybe what I what I was struggling a bit in the past is distinguish between Commons and events when you send synchronous things uh so yeah maybe that kind of theory on that regard also might can be useful all right um we got two more things uh to wrap it up the first would be another comment I guess from Maxim who says that service to service API

composition integration events data materialization need to have experience for teams to chose I would say I guess you agree right or what's your take on this what's my take on this what would I choose you mean yeah I mean I guess it's more a comment right need to have experience for for teams to choose so yeah so it's only done with experience of course right there's no

yeah that's for sure okay now another question from gitis um is it usual to mix these patterns in a single microservice for instance having service to service calls in one micros service and consume events from another micros service I think it depends on the design but I would say yes and one of the things why might be also limitations for example you like the EV approach and

you kind of utilize them but if the service you need get data from doesn't have events it doesn't expose anyone you don't have much options so maybe you can just use the public API to get data uh if they also can handle your load for example you get the guys you said okay I need data from your site do have something they said we have just a

restful end point call it I say okay I have like I don't know several thousand RPS can you handle my law they say no please do it in the background in that case I will go with mization approach if they say okay we already handled a lot lot of RPS and you several thousand doesn't change it a lot just call it on demand okay maybe I can

do that if I don't care about latency so yeah yeah I think it's possible and also talking about microservices it's not necessarily a small it might be like a self-content system which do a number of things under the hood so we may expect number of Integrations and all this Integrations may have different uh communication approaches all right maybe one final question question see does once again to

have more clarity in event schema validation so um specific question is do we use a schema registry uh I think it's a good question first of all and uh I can't give much details about it because I'm not specifically working around this event infrastructure setup but uh yes I think schema Registries uh required here here and for example if you work with confluent for KA I guess

it's called like this confluent they also have this schema register which provides so it's exist for validation it makes sure you don't introduce a breaking changes and it makes sure all the clients already work with your events they can continue working on that when it have some changes so yeah definitely definitely that is something useful uh I think it's not only for events I think the same

work for graphical Federation for example I also have schema registry something which is created specifically to uh keep this I guess you need to have it especially of course you already talked about testing so contract based testing you want to have that you want to ensure that this thing keeps on working so I guess that's the end of the session a lot of questions here at the

end so thanks everyone for participating uh thanks for the great talk and uh everyone let's have a great conference going from here on so yep thank you so much a great day everyone bye see you