Panel Discussion: DevOps Project Failures - Lessons Learned and Strategies for Recovery

This panel discussion focuses on the failures encountered in various tech projects, particularly in the context of DevOps and backup solutions. The speakers, experts in their fields, share personal experiences with failures and recovery strategies. They emphasize the importance of collaboration, proper adoption of technologies, and understanding the roles and responsibilities within teams. A notable story was shared about a customer who initially decided against implementing a data backup solution, only to be hit by ransomware shortly after returning a test device. This incident highlights the necessity of backups and the need for proper communication between vendors and clients. The panelists agree that success hinges not just on technology, but also on involving people and maintaining a proactive approach to potential issues.

[Music] hi hello everyone welcome on our panel discussion Devil's project failures lesson learned and strategies for Recovery my name is Gregor but I used to introduce myself as Greg as I believe my name is is pretty hard to pronounce I'm chief of R&D at zpro software and G protect which is Backup Service producer for more than 15 years together with me there are four gentlemen and I

believe everyone can introduce himself so greos let's start from you okay hi hello G from Poland uh currently product owner but in the past four years for building from scratch devops operations competence Center and generally a lot of failures during that time to share thank youer so Manuel would you like to introduce yourself yeah thank you very much Greg so Manuel uh I represent two countries because

I'm French and I live in Italy um a lot of experience in devops projects I've been working for companies like Serena micr Focus cloudbees uh we Pro more recently and uh so a lot to say on devops projects and uh failures uh for and and the recovery for those uh those failures yeah we cannot forget about the recovery so thank you Manuel and now let's jump to

Max hi everyone I'm Max I'm the founder of liquid reply and uh I call myself myself a yeah Cloud native technology advisor um but what we do in the end is we build with our team together for customers platforms so we do Platform engineering the last five years before that we we called it some way devops but we try to get away a little bit from it

and maybe we can discuss about why I believe devops is not the best description anymore thank you Max and last but not least Phillip hey hi from Slovakia to everyone uh my name is Phillip I teach developers about testing and testers about development and I work for a company called replay IO uh we build a browser that records everything that's happening on it uh and that means

also what's happening on CI so what happens on CI will no longer stay on CI so if you have flaky tests uh that's that's where replay IO comes uh comes in uh and I'm excited to be here and to talk about devops and all the problems it brings hopefully talk about Solutions today yes yes let's talk mostly about how to solve the problems we met thank you

philli by the way and yeah so as we used to say we we used to learn on mistakes but on only Geniuses learn on mistakes of the odors so that's why we would like to share with you some nice stories about mistakes that happen and how to prevent them in future or maybe how to recover from such a mistake if you had or if you cause any

so let I would like to share with you some short story about one of of our customers because as I said at the beginning zos soft Where We Are backup provider for more than 15 years and in our offer we've got also Ser we've got also products for Onsite backups and we had one customer who wants to check in live how our solution works so we sent

him our test device he set up the entire environment he checked it for more than two weeks and everything was Fa went very well we were all the time in touch with the c customer we get very positive feedback about our solution and the sales team was pretty sure that the deal is already closed and we don't need to do anything else to to make this customer

safe and and his data stays with him but at the end of of the PC process the customer states that you know it's fwell product everything works as it should but I'm not sure if we already need a data backup because we never lost any data we have probably some more time to to implement backup in place and right now there are some more important topics for

us and of course we totally understand the decision but it was pretty weird that such such a huge company it was has no backup in place in general there was in total no backup implemented but at the end of the PC process as usually we asked the customer to erase all data from the test device to pack device and send back this device to us and probably

you may think what happened uh literally hours after he packed device and passed to to the posting company uh the company was hit by ransomware so a lot of data were encrypted the customer lost access to to his resources he cannot access his dat bases some important documents but luckily he didn't follow our instructions to remove all the data from the test device so so hopefully for

the customer we were able to send back him this device together with all his data and luckily he was able to he was able to restore all his data so from this Theory we can learn that back is always necessary and you should follow manuals from from the backup vendor even if he asking to erase all your data from test device otherwise the backup vendor or old

customer may get access to to the data so even if it helped in this case please do not follow that remove always your data because privacy is very very important but keep in mind that always you have you must have implemented backup which you are able to restore and even when you have the best solution uh chosen it's very important to work with the right people to

to implement to develop the solution and I believe Max has something to say in this area yeah absolutely so um yeah I mean it's like not always is the is it about the the solution itself right it's about the the people and the mindset and the the approach you take to um come to the the way you work and and how you you implement um what we

faced in a in a few years ago um was a also fairly large customer introducing a huge jtic citus cluster need to run in in three different continents uh needs to be operated also from uh a restricted area and so on and so fors having different um certifications which you need to have in place to to operate it so it it was fairly complicated to to set

it up and follow all the rules right it would be easier just to have somewhere piece of metal put into the seller and and and put a big fat lock on it that that would be better um than to go with a huge Club um and it was actually not the the first time that with that customer we we implemented something like that so we we tried

it before um and we very fast find out that something is wrong and we couldn't give it a name right it was like the tools are working somehow um the platform is deployed everything is fine the first applications migr to it it's all okayish um but continuously we were not able to to finish any any of the implementation um always yeah exchange and replace a couple of

of tools which were before decided on uh putting back in putting back out um because someone else come with some other concerns and so on So like um the whole team was a constant move and we also felt it from the customer side that um they continuously switch people in and out um from the own team from their own team side and due to the complexity it

was not something which which uh fits for everyone and um with yeah a couple of uh good workshops um we identified that there was a very big misalignment in the understanding of what does devops mean for the team at the customer side versus what we were hired for and as I said nowadays we call ourself platform Engineers because I believe that's the better a more precise definition

of what we are doing rather than we take care some somehow from for everything from top to bottom and as ver fogal said 2006 you build it you run it I believe that's you build it you maybe Run part of it but you don't run everything in it um we we saw that this was a very big misalignment in cultural misalignment and and a mindset misalignment in

the end on the customer side which you have to to to tackle on so how the customer people understand themselves was either they are devops and are implementing things or they are devops and are operating things um where this comes from obviously is because it was two different departments which were merged together and one of the part of the department comes from an operational perspective and the

other part comes from development perspective and suddenly their boss practically expected like hey you switch roles and you do this one and uh you go in a weekend and troubleshoot and whatsoever and that leads to a very high level of frustration um within the team and it leads to that there was a continuous discussion going on on places where we couldn't observe it anymore right it was

like off the table it was not in any kind of architectural decision record you didn't plan it in somewhere some some other technical discussions it was always like on the coffee machine two people had a hard discussion or in the evening when they went for an event they had hard discussions but it's always off the Record and so there we were learning like okay wait something is

seriously wrong but it's really really about the Team Dynamics about the mindset and about how you introduce to that team their new role um many of them were stressed out as we found out because I like got a huge stack from cloud provider um putting on top of kubernetes putting on top all the different parts you need to to to run and maintain it putting applications on

top and then you you have it extended to different locations around the globe so the people had like a huge list of things they should have a look on and obviously they freak out because it's too much they came from a world where you have like your application your single piece of component you took care um one two years fast forward um same customer different department um

we switched the perspective just taking care about this big platform around kubernetes not taking care about the infrastru structure below below it's just an API you're talking to nothing more nothing less it provides you capabilities um the application on top you don't care you provide just like the the runtime environment you talk with your customers obviously who are running on top of your platforms um but it's

not your responsibility to operate and maintain the application you don't have to be the observability expert you just integrate observability you don't need to be the security expert you just integrate the security and so on and this perspective from a silo towards like more like a a factory um or reliable Foundation helped in this case um yeah shift your mindset put a little release on the on

the mind of the team and really focus on um building the best-in-class platform for them which is now still running after 3 years and does a thing and has couple of million of devices connected to it as a as n back end okay thank you for for sharing this story Max so yeah it's very important to to work with with the right people I keeps my finger

cross for for the environments to running until the end of of the world uh yeah from my experience I know how how important to have pretty well team and and how One S one person can affect the the rest of the team and how those chats next to the coffee machine may affect the entire project in positive but also in negative way so in the meantime horor

joined us too so horor hello and and nice to see you we are just in the mid of our panel discussion but hor can you hear us okay yes perfectly we can hear you too Jes so it's time for you to yourself okay uh hi hi guys good morning here I am in Peru South America so it's morning here uh well first of all thank you very

much for for this invitation I'm very happy to be part of this group um my name is for Castro I work as an transformation leader uh in entity data which is a Japanese company um I have more or less 12 G 12 12 years of working experience working with uh data transformation programs like divos transformation a transformation and so forth with different clients telecommunication Banking and so

forth so currently I'm working on that I'm leing some transformation program with my clients and um I'm here because I like sharing you know I feel passion about sharing knowledge um communities so yeah okay perfect so I believe you have also some nice stories about about failures and and how to recover from from such failur so far I shared my story about about date backup H as

well as oh guys sorry as well as as Max about working with with the right people the right team he let us know uh about about kubernetes deployment and changing the team which help him to to solve the problem in general but I believe if the story is way deeper than we have the time to to be told to to be discussed so maybe now let's jump

to to Manuel because I know that Manuel have some nice history about adaption Filer and as we know it's not only about having the best solution because kubernetes is one of the best or even the best solution in in this area H it's not only about working with with the right people but it's also also about the right adaption or implementation because you might have the best

solution the best software wrongly implemented and then May happens does happen now just just to comment before before talking about uh adoption and the experience um what you were saying at the at the beginning Greg did did ring a lot of bells in my experience also uh about backup Solutions about um and I I think I will always remember uh some kind of joke when I was

at at the beginning of my my career uh my boss uh told me one day if you want to know if you talk to serious people when you go to a customer site uh if if you want to uh to know if they're really serious just propose when you log in uh to their uh machine to do uh RM minus RF slash which is basically uh for

those who may uh not know or or not understand because of my accent remove the complete F system of your machine if the guy said yeah if you want to go through that okay it's not necessary but we can do it then they are serious because they have backup and they have the process to uh restart from the backup in in minutes so that uh you can

you can still work if they if they think you are crazy then people yeah as we used to say you know there are two group of users one who have already backup implemented and another one who will have backup yeah exactly and and you don't want to work with the one who will have backup it's much more dangerous so uh going back to uh to the experience

yes unfortunately I have a long list of projects where um the the the solutions of devops project Dev secops project nice project uh where where you really believe that you have the right set of uh processes you have to right combination of tools you've been studying a lot of uh things about communication about synchronization but well all the technical stuff is clear in your head it's clear

on the paper it's nearly clear on the uh on the machines themselves the tools are working you have a wonderful solution you have a list of benefits of that solution you you're convinced that the return on investment will be uh terrible I mean in the in the positive um meaning of terrible it will be awesome for your customer and something is not working what what people don't

want to use your solution whoa whoa I have the best solution on Earth okay and you don't want to use my best solution on Earth why why because I I don't know what your best Sol solution is about I'm not sure I have other things to do okay I have my application features I have my own customers I've been working like this for 10 years and it

works fine I don't want to okay um I've I've gone to one of your meetings where you talk about the benefits it was half an hour I had a phone call I have it's all about uh so long story short it's all about ad option and uh involving the people who will use the solution the people for which for whom you're working to define the benefits are

they aware of the benefits are they have they been involved in the definition of the benefits do they understand the benefits uh it's all about sometimes I've heard the uh the term uh arrogance okay it hurts because you believe that you bring a very good solution you've been working hard on that solution and people tell oh that group of people bringing that devop solution that release automation

solution that deployment automation they are all arrogant no we we are not I mean we want to bring you something but this important uh still from the experience and I have no doubt that each of you could comment and say yes I also have that kind of example um to involve uh people in even in building your solution okay you don't have uh magic things I mean

you may have more experience uh more skills in defining develop processes because you've been studying that for for years but if you don't people uh taking into your solution the way they working uh the benefits they are expecting because uh the people management people uh you've been talking to may have a slightly different definition of the benefits the benefits could be Financial benefits while uh the development

teams and operation teams could have morale benefits okay they want to be uh more uh comfortable when when coming to work in the morning it's part of the uh of the important things in a project uh so adoption which is a very generic name uh but adoption is a very very important aspect of the project uh I don't know if I mean it it does it resonate

to uh to some of you in the so you know I totally totally agree with you and and from the vendor perspective I know how older vendors used to say that their solution their software is the best one always fits to the to the customer needs but I used to say usually during during any presentations any demos but there are just a few vendors on the market

which cover this area so you can easily test every one of them and choose the one which fits your needs even if I'm trying to to promote on any way my software I know that older vendors May in this particular cases do their job better that's how Market Works no no absolutely I mean we Mo most of us I think have been I've been working for for

vendors and uh yeah when you sell a solution a commercial solution you know that the competitors are between 85 and 90% they do the same I mean doing CI doing CD or I mean the things that I I know in in depth uh well there there there are some tiny differences but at the end of the day it the Ci or the CD after that you have

some other considerations and driving the project is uh much more important uh at least in in my experience much more important than the the tool itself yeah I believe har has something add no no actually I agree I agree um um something that I notice is for example about providers you know uh um something that it's it's unfortunately very very er um something that happen very often

very often is that um when you are going to hire a provider basically they the provider some of them um prepare the business case right and all is about tools right I'm going to work with jira with the Microsoft whatever right a lot of tools a lot of um success stories but um in most of the proposal the part of the people it's missing okay what what

what what you want what you are going to do to change the ways of working of the team right if you want to uh implement devops or Cloud microservices whatever uh what is your plan to change the ways of working and also uh how you how how you can be sure that you are going to understand my business needs you know I think that that is another

problem that I noticed most of the pro some of them uh providers are interested more in that technical part the tooling part but I think those those aspect are quite important yeah so that's why between between venders and customers they must be some integrator who knows pretty well the customer his needs the way that the customer is working and can find the right vendor to fit to

to the customer needs so so yeah it's very as you said also it's very important to to know how your team Works how how the people you you are working with follow the the projects how they use the software to to help them to to provide the best and the right thing and in terms of working with with the right people in terms of of following the

the project Jos has also some nice stories to to be told so gor this stage is yours I think I Willer to your backup case at the beginning because I have at least two of them one was the the the more physical than Cloud ones because one was regarding the maintenance of the main power line to the data center and unfortunately during the maintenance the second line

for was has a failure so there was no power to the to the data center for a few hours because of that and it brings me to the question how many backups do we need if one backup is enough yes because Also regarding for example the physical firewalls uh I had a case in in different project that the first the primary one uh has stopped working because

of the hardware failure not the software one and the second one just few minutes later do the same and we were lucky because our backup was uh just uh by the desk because we got the third one that we could install immediately in the data center and that was really the backup uh Max said about platform engineering which I love because I believe that we when we

are starting our devops journey we are devops team but after having the competences growing of the number of the people we are starting to be the platform engineers and we can do something else not only the PE devops which is the basics for the platform Engineering in in in my opinion Manuel said about business yes I believe that if you are doing some tasks even if you

are rescuing the prod because we got some critical situation understanding the business and oper operational context is very important and this is not only about the technical part of knowledge but also about the business and operational to understand what exactly we are going to do what we should do first which service which micros service which which cluster is more important at the beginning uh after that H

H has said about about people I personally believe that people are first and uh the technical knowledge and and and resolution will come if we have the commitment in the team if we had the faith if we have the common understanding on the goal and so on and I was leading a team of 10 devops Engineers providing not only the devop services but also 247 uh support

for life saving processes and we were hit multiple times but by by multiple issues on production and we had to decide if it important the current security layer or the application that is go working on production and uh we were hitted by external Jenkins team that was making some maintenance and for example for one day we were not able to deliver any build to production yes fortunately

with without any need for that yes but if we will have a critical situation we will have the only possibility to deliver the manual uh deployment and most commonly the end user is also the case uh can also bring the failure to us I had this history that someone reported that the whole application is not working and was we were not able to reach that person for

two hours and for for devops engineers were working because works for me buted that it's not at we discovered that the person goes to the market to buy buy some vegetables or something like that and he forgot his phone so there are some cases not only on the technical layer that can bring us to the critical situations and what is more important than specific cases is what

to do before during and after the critical situation so before we should focus on being prepared so for example do we have the proper documentation do we have the knowledge of the process processes do we have the awareness of the procedures did we perform some trainings did we do some outage tests and so on did we share the knowledge that we have yes because potentially only uh

one person will will be on call during the the weekend and he don't have in his mind all of the knowledge yes what to do when we really receive that critical situation we should identify what is going on in integrate the team that is possible to help us with the solution H and at the end inform everyone else because someone may got the hint what may be

done here and after it is also sometimes we don't have time for for for cleaning after yes we do the hot fix everything is working so we move forward for the next task and we are happy that it will never hit us again and you said that about the the usually hits the fun and we got really problem after that so uh after we will survive this

critical situation and failures in the project on production we should gather an knowledge why it happened perform some retrospection fix the issue if that during the situation was just a hot fix and at the end update the process and procedures what exactly should be done in the future I had also the situation that there was some critical situation small hot fix uh we documented it and it

happened after two years no one from the team remembered exactly what step by step we did to resolve it but we had the page on the Confluence so we resolve it within few minutes not few hours again yes Boomerang issue as I used to say yeah yeah that's I I hate that issues which which came back as a boomerang which you throw out and then months later

year later you you're are hit exactly by the same issue and the worst scenario is when no one knows what to do to fix that issue that was solved before because the be the person for example who solved it is not already in the team or or it wasn't written anywhere on Confluence jir or or anyone where else so yeah I hate such such issues and and

situations okay so so G thank you also because you you you've covered a lot of topics including even my area which is date backup and honestly there is not all in terms of number of backups that we need there is always not enough number of of backups in terms of continuity and and even as a backup vendor recently last week we made maintenance of our internal VPN

we had some backup but this backup also failed due to Hardware failure so so even even even if you are a shoe Master you're working without shoes as we used to say in Poland so so we've got also a few more gentlemen to to share their stories and both stories are tests related from my perspective as chief of R&D tests are one of the most or even

the most important part of software development life cycle because because this is the last chance for us to stop any issues any any failures to bring to the production so philli it's time for you yeah uh well we see how that works in real life right because we all want tests we all like to have them we all hope that they will give us the confidence we

need but when it comes to real life sometimes the story is little bit more messy right I I what Gregor said about uh uh having sort of a retrospective on incidents and make sure that things don't come back around uh in my uh when I was working uh for the previous company slido maybe you know it it's a Q&A platform for conferences like this we would like

to say that there are no second chances for Live Events right so for example the platform we are on right now if it went bust right now then not only us we couldn't talk to each other but the the whole conference would be would be ruined right so when the stakes are really high the the pressure on making sure that you deliver the thing you want and

not break anything is is much higher and nowadays it seems like teams worldwide uh are moving towards a continuous integration that's like the lovely term we all like to use but we know that sometimes the integration is not so continuous right sometimes releasing takes a long time you can have like a couple of hours to release something uh sometimes you need to roll back and that that

will take a lot of time right uh so how do we usually mitigate this right how do we how do we make sure that we can uh uh deliver continuously and the answer is tests right there are important part uh of the release cycle but also tests are slow right uh but when I say they're slow I don't always mean the test execution of course the test

execution can be the frustrating part but even the more frustrating part is when tests fail for NO real reason right if this functionality did not work we would already know about this if if this like QA would would catch it right or or something else like tests when we are talking especially about UI end to end tests they tend to be slow they tend to fail a

lot and uh they tend to be slow not in the test execution per se right because you can have 100 tests run in two minutes and because we have the tools now right we have playright Cypress all these new tools can we have we can front test in parallel uh but uh it's the test maintenance that's often slow right and especially when it comes to like unstability

and Flaky tests and trying to make sure that everything is green that's that's a long story so recently I done a LinkedIn poll among like the testing Community uh asking them what takes bigger part of your time whether it is writing your tests or is it maintaining your tests and it was like 72% I I believe was maintaining the test that takes bigger part of my day

every day and yet like we're talking about which tool is fastest on CI well it doesn't really M matter does it uh so and the reason why the maintenance takes a lot is because when a test fails on CI we're kind of blind to it right what we usually do if you find yourself in a in a role of a test engineer is you try to take

that issue that happened on CI and try to replicate that locally it's sort of like you're trying to catch your own tail uh you're running in circles until you know it takes too much time and we're frustrated and say something like oh you know what CI is not like a real user so let's not treat it like that uh we're going to mute that test or delete

that test or just not care and I'm sure everyone uh here online already heard the term alert fatigue where oh this is always red I'm just not going to pay attention until an incident happens right uh the the red light has been blinking blinking at us for too long we're just going to ignore it and uh uh and yeah I mean there are nowadays tools that uh

I mentioned playwright in Cypress they added some functionalities to add visibility to CI which is great we have Trace viewer in playwright we have the test replays in in uh in Cypress and they did bring visibility but there's one crucial thing that I think they're missing and that is that you see only like the dumb snapshots of command that was executed but you don't see the whole

story you don't see the run time ideally what you would like to have is to have the trace of everything of what ran on CI and then debug it maybe like half your uh Chrome Dev tools right in the in the CI and be able to to examine that um and I I don't want to be too long because I know we are sort of reaching the

end and uh and we still have one one speaker to share the story so I'm just going to give a small teaser to my presentation tomorrow because I will be talking about something like this uh I think we're entering a very exciting era of of having a better V visibility of what's happening in our tests in CI so yeah if you want to learn more about that

then uh then I'll be happy to share that to with you tomorrow so I made a commercial out of this sorry about that but yeah I'm going to pass pass on the the word yeah so I believe hor has something more to be added in terms of of the testing I can't wait about this story because it's combined with gam gamification and it's another one factor I

I feel lack in the test yeah um I well basically basically the problem was that you know Common problem right um when you when you try to um transform your business with devops agile whatever right any kind of new technology and so forth um all is about people because your engineers your engineers need to learn new skills and put it into practice that is the challenge right

one of the challenge um in my story the challenge was about that uh we were facing a de transformation and to be totally honest our testing was totally manual we had a lot of manual testing so with this transformation we need to first of all build new skills for the team you know integration test um uh microservices test and so forth a lot of T automation new

skills you know for continuous testing as part of doops adoption um and of course doops practic right for the engineering team U developers testers and so forth so that was the challenge the first approach was the traditional approach you know trains a coach a mentor a engineer a technical lead doing doing a lot of training but it fails because uh no matter if you provide a lot

of training to people you know you open some platforms of learning like like udmi and people is not to attrac is to change right it's quite difficult right I actually that that is that is something that happens in real business so we noticed that our team that most of them were very young people you know people who likes uh you know um social networking who likes gaming

on internet and so forth so we notice that so we create uh we apply gamification to for testing we create this this this um game name it the game of testing similar to Game of Thrones but in this case was the game of testing and basically in this in this game with Jenkins we create our some rules of the game and basically depending of your depending of

the complexity of your test depending of the number of backs that you get with your testing depending the number of your uh build that you break with your testing um and depending of the the level of testing right if you create unit testing integration testing functional testing security testing and so forth you get points for for for you as individual and also for your crew for your

tribe or for your team right for your value stream okay or for your product so we create we we we run this game right not only in South America also with South America us and Europe and and India and we compete we compete in a kind of global tournament you know uh we create this de Cup in our company and actually it was very nice you know

because um people start to collaborate each other right collaborate to to comp compete but in a good way right thinking on the product thinking on try to robust robust the test and so forth so I in my experience it was a a very nice experience using gamification to try to build new ways of working and especially testing skills okay do you have maybe this story also written

somewhere on on your website on block to to get to get deeper because because I get really really interested in in this topic and and I believe it's something also I would like to think within my company absolutely I I can share I can share with you the I I post a blog about it and I have a couple of um speaking um about that you know

perfect if you can put it in our on our internal chat and I believe our moderator maybe somehow will be able also to share it with other attendees will be great because because probably not only me not only I'm interested in in this topic so guys we've got also I know about one question but Max letting me know that there is also one more question so probably

we've got two questions to be answered so let me read the first one I have I can see there so guys how to deal with technical depth when you live something for let's fix fix it later and you don't have resour afterwards so how to deal with technical depth I can I can very short answer because if you don't have time to fix the technical Dept you

will need to handle the issues and consequences from that technical Dept in the future so it is up to you which option do you prefer on which may be more cost effective or at the end for you yes that that's my Approach yeah when you take technical deps it's always like a it should be a conscience decision and you should track them but if you don't have

afterwards resources for it then um well you can still collect your technical depths but you will not find a way to solve them over time which would be problematic towards your environment you working on you should always find some time to not immediately but like I don't know six months later nine months later reiterate through your technical depths and see if there's maybe meanwhile a technical solution

for it I mean the open source market is moving so insanely fast um theoretically you can give on the same topic every year the same talk and it will change um because tools are improving approaches are improving and so on so forth so um you need to ensure that you have capacities later available else you have a problem in poisoning your your thing you're building and and

just to make just to make an additional comment on that we we're talking about I mean the question was about Technic depth the answers are are on how to handle technical depth doing a small uh step back knowing the technical depth is not something straightforward also so um one of the first step I think is have a correct knowledge a correct evaluation of what your technical depth

is from that moment you can make a decision uh solve things or uh postpone or but knowing is also important okay okay so the best way is just not to make the technical depth in general but I know that it's it's impossible in in many many cases even from from my experience so we are two minutes over time I heard that there is one more question but

we didn't get this question posted here so I believe later we will be able to answer this question over the email I get also the links from Jor but I can't share with the audience those link but guys if you will write in Google game driven development uh hor Louis Castro toribio definitely you will find his his lesson his lecture about this topic so everyone who's interested

can join also hardhead there on on the YouTube and for now I would like to say thank you to every attendees who join us I believe you have learned something from from mistakes from from failures we we discussed about and you won't repeat them so at least I keeps my fingers crossed for that and gentlemen's also thank you for sharing your knowledge and stories it was really

nice to meet you there and also I learned something from you so thank you guys and hope to meet you again thank you thank you