Viktor Vedmich: Ideal Blueprint Versus Reality for CI/CD Pipelines

In this session, Victor Vedmit discusses the importance and intricacies of CI/CD (Continuous Integration and Continuous Delivery) pipelines. Drawing from his ten years of experience in building CI/CD projects, he elaborates on various branching strategies, emphasizing the significance of source control and how it relates to effective CI/CD practices. He shares insights into defining continuous integration, the essential phases in the build process, and the role of artifacts and feedback loops. The speaker also highlights different deployment strategies like rolling updates, blue-green deployments, and canary releases. He stresses the importance of metrics in assessing the efficiency of CI/CD processes, including deployment frequency and lead time, advocating for a systematic approach to measuring performance. Overall, Victor provides a comprehensive overview of how to architect robust CI/CD pipelines and the best practices for automated deployments.

[Music] ladies and Gentlemen please welcome our next speaker Victor vedmit presenting the topic ideal blueprint versus reality for cicd pipelines uh hello and good morning everyone as you heard my name is Victor I'm from Ms and yes definitely today we will speak with you about cicd uh how many of you you already know everything about cicd please raise your hands so my question why are you here

why you decided in 2024 just join to cicd talk and speak about cicd so uh first of all before I joined T best I work as the engineer more than 10 years and honestly I spend a lot of time to do different type of cicd projects and know maybe 20 25 projects I had to build and create from the scratch sometimes working with Legacy cicd process but

anyway so it sounds like I have some experience I believe I hope I have some experience and my goal for today just like to share my experience with you maybe to show something new to systemize the knowledge what exactly CD why it's so important what's exactly difference between CI continuous delivery continuous deployment what kind of best practices do we have and what's most interesting SP from my

perspective how we can measure our cicd process because you know if you don't measure most probably you don't control if you don't understand what's going on with your continuous integration with continuous delivery process most probably you don't control this process someone did this amazing job but you don't what's exactly happening under the hood why we have so long build process or deployment process and I will share

with you my like ideal blueprint pipelines how it should looks like again it's uh from my experience perspective I'm not going today to speak with you on implementation because some of the people prefer to do it based on the junkins some of you most like to do through the GitHub actions maybe you still working with creas control who knows what is cras control please R you know

what is c three four four people you see how I old yes so that sounds like you also not very young people so this this was my first uh continuous integration tool cruise control I recently figure out that s work uh company made it uh s work from s work was released grease control it was wonderful product so anyway uh we will start from our review life

cycle in general what's exactly continuous Integrations how it's related like what the first step the second one and for sure when we start to create our project we start from the source yeah it's very very important part because a source where we store our Cod and the next steps like it's a build test production sounds super easy but uh why still so popular devops area when we

have to make a lot of automations continu integration delivery it's very important part I still remember this time like when people spoke about hey we have five deployments per day it's like amazing job and nowadays like people have 100 thousand deployment especially if you look at the ad I know know in we deploy like 100 times per day different Series teams but still like it's amazing numbers

but everything started from Source it's so important because sometimes people to forget that continuous integration uh started from source code because if we speak what's exactly continuous Integrations means in this in this two words we have continuous and Integrations everyone understand continuous it should be happened every time when we do some part of the job but what's exactly Integrations some people said hey uh integration means like

when I made the build run my unit test and see the result in reality it's not true because uh continu integration started when you made the selection uh of branching strategy so I hope you all of you use currently G uh it's a main source like where you push your changes not as F anymore I hope so so uh but I met recently people still working with

this one I I understand but like my preference G so why I want to spend a little bit of time on branching strategy because you know like it's uh ridiculous when like it's it's so dramatically important in the future because based on your branching strategy you will Implement your deployment strategies how you will delivery to production and what kind of currently options do we have the first

one it's not option please don't use git flow if you don't have some specific reason for that why because first of all uh gitflow was uh designed and implemented by Vincent recent in 2010 it's like 14 years ago and the main goal of gitf flow was to develop a product which can be delivered through like compact dis for example we working on some game or office or

I know something what we distributed through the compact dis and delivery to our end users not through the web applications but somehow to install and work in with this application my experience with gitf flow I Join one one of the project and open the list of the branches and started to scrolling I'm scrolling one screen second screen third screen so I stop on the one like Achi

10 screens or and it was only branches I started to spoke with uh developers and ask hey people what's what's going on why do we need so many branches uh you know like we create this Branch because we wanted to make refactoring uh this branch is 3 years old okay uh and what about this Branch this is was the second iteration of refactoring we decided to refactor

our front end this is 2 years old branch have you merged this changes to the like main branch no we just like started to work on that and forgot about them so git flow a little bit difficult I mean like from implementations but I know some sometimes of the reason when you need to have some strict definitions what will be released on your production how you exactly

release how you test sometimes get flow will be best options but most probably in 80 or 90% in my experience I would say no please don't use use kit flow it's a little bit over complicated especially if you work in uh when you develop application which is distribute like web like you deployed to cloud or to kubernetes or to someone like super quickly and every day what

other options do we have uh most commonly currently uh use in development areas it's GI Hub flow uh what's the difference so first of all we have a main br branch is our main line where we try to support it as stable branch and uh we can deploy at any time to production for the main branch so uh one of the most important part from git uh

Hub flow that it's said like we need to have only shortlived branches and another option is like C review you know uh before uh like my my preparation I did some couple of the researches what's the current state of continuous integation delivery in the world and uh s work again like it's huge company they uh release every six months uh technology radar it's a pened view what's

going on with it in general in Martin Fowler working for S work radar today I S like a lot of times with work not ads uh so as you can see not any implementations on ads so uh he updated recently his huge article about continuous secreation delivery highly recommend to review if you want to go deeper like on 400 level so what's the problem with court review

Court review initially was created for open source when you have different developers who distributed across worldwide and you don't trust them so it means like it's an environment when I am like for example as a senior I'm done with my job and I don't trust a person from other country which is based for example in the United States I don't know what's your level for example example

of the knowledge on in development process and so on but how it's affected to our process in our life so sometimes we implemented quart riew for so long so long time and we stuck with qu riew please trust your teammates please trust in your environment and make Court review faster and quicker why important because sometimes we create a GitHub flow and we don't have at all continuous

integration because we just stuck on review of our code so we be very careful with quote review it's very important part to grow as engineer to learn from each other for example from senior to the junior but at the same times if we create a lot of qu review we can like waiting for for a long time to move on the next step what else uh branching

strategies can be also trunk based uh development super popular in startups uh if you need very quick development process so what's the main difference so first of all we have not like a main trunk and trunk uh Branch we merge or update trunk uh Branch every day it means like for example we have two different uh developers like for example Anna and Victor for example and when

we do any changes as soon as we not done with a feature as soon as we done with some part of our job we try to merge this change to trun it means like trun we try to save as stable as possible but at the same time we don't guarantee that trun can be deployed to the production we try to do as our B like our best

so we try to commit to trunk to every day so it me like at least every day uh the biggest challenge with strun based development how I can commit my changes if I'm not done with my new feature and the answer will be feature toggle it's here so what's what's exactly feature Tuggle uh sometimes people call it feature flag you create some configuration of your uh application

and create like a flag please not enable this functionality I'm not done with this functionality as soon as you're done you just enable it through the feature flag and it means like you can enable this feature exactly on the production that's exactly how for example AWS deploy most of our new features to production system we deploy it like very very often like every day but we not

enable all of the functionality we enabled exactly when we made all of the test blah blah blah and ready to run for our and future for our customers also super quickly uh interesting part of git I know most of you knows what is git but maybe you have never heard about what is true if you're not so what exactly is you have to try and if you

need to mer one for example commit you have some BS for example on production you figure out about that made the changes on the drunk and you can Che repeat or merge only exactly these changes to your release Branch like release 1.1 so this is exactly when we start with continuous Integrations again continuous integration it's when we merge our changes to uh our source code to our

git or s and it me like we integrate our job like you me work together in one application in one microservice for example and we merge our changes we integrate our changes that's exactly where continuous integration started we going to build this blueprint you see some you know like boxes so we will fill it like a little bit later so uh what's next do we have so

for sure like the next one it will be test so we need to run some test and it's where most of the people started to seem like it's continuous integration why because so first of all what's the main goal of continuous integration we have to automate everything as much as possible so as soon as developer finish with uh job like I'm done with my functionality I commited

push and automatically starts the process of build the main goal of uh two main goals I would say from continuous integration the first one it's most important one it's feedback loop uh this is the main reason reason why continuous integration was created in general why we had junkins GitHub actions C Cal from made Scot build from metab again team City for example or many other tools what

you prefer so to have a feedback loop uh feedback is means you again like you done with your job and run and run in one central place test Court review and everything else and this is exactly what feedback is you run and figure out that you don't fit to unit test or for example some static C analysis said for you hey you have an issue you store

some secret data like password in git so be very careful and this is feedback back to you as a developer or to you to asops engineer and said hey you have to fix it the second one is artifacts so uh we prepare our artifact and store in some artifactory how this looks like from from diagram perspective so first of all what we add from the previous slide

we have cicd dashboard uh is again it can be your junkins Some central place where you see what's going on with currently with CI what's the build Q what's the result of unit test and so on and what the step we should to add to our continuous integration cot analysis again it's static quot analysis mostly it can be just linters for example or maybe more advanced tools

like sonar Cube or something from ads or something from third party like up to you what you prefer run build like for example prepare like run and build on java.net uh go like depending what's language do you use for sure like if you run in the python most probably you don't need to run the builds but maybe you need to like to download some dependencies and unit

test uh again uh unit test just like to be aligned I understand most most probably you understand it but I want to say it out loud and make sure that we are on the same page what's the difference between unit test and integration test unit test never will go outside of the system it means like we in unit test we never test that we store something in

our database or file system or someone to some third party Services Unit test is just to core test core function ality so for example I have a functions which just calculated two numbers or preed uh collections of the data so I have in input two and two and I need to have a sum of these functions so the result of the unit test like to make sure

that uh two and two it's like four not five not six so it's what's unit test again unit test don't store anything on your file system on database on your cache on someone some other place you need test only inside in your application not outside and uh you also hear see here like I have quality gate in every step I have this quality gate in means like

if I failed for example with static code analysis or with my unit test I stop this process of continuous integration and EXA exactly I got my feedback from my system from my continuous integration to developer and say hey John can can you please fix it my integration like John you have a problem with this functionality can you please look at this because you not fit to like

pass all unit test for example and the output of continuous integration for sure will be our artifacts artifact repository we store our artifacts which we have to deploy to all other environments again please try don't create artifacts for every environment if you need some configurations for Productions make it as configuration part not to create a separate build for production environment for staging environment you create one time

your artifact and deploy this artifact on the environment and also don't forget that you need to make some version of your application I know I know that's most probably you would say I know what the semantic versions uh super quickly to explain if you're not familiar so in semantic versions We have three numbers uh major version it's like like when we have uh dramatically changes like for

example python 2 and Python 3 so we cannot use quote which is re written on python 2 and Python 3 versions like you understand what's the logic minor changes like when you add some new functionality patch it's like your build my experience show me that it's very good practice to put in patch version also hash of your git commit why because in this situation you will very

quickly to understand what's the current version from qu perspective on your station environment on your production environment on your any environment because you will have this traceability which is match to your git repository not only some magical numbers 1.2.5 what exactly is what the state of the code is of this version so if you at uh hash commit of your in your patch version you will be

able to trace very very quickly so we're moving on so we currently cover it like Source build and test and the next one is you can see it's continuous delivery so what's the main goal of continuous deliv first of all it's continuously after we done with continuous integration so we done with our artifact so we prepare this artifact and we deploy these changes like we start to

deployment activities uh we promote our artifact to some environment so it means like we run this on some environment let me show you from again from diagram perspective so first of all super important part again like based my lesson learn from 10 years of experience infrastructure provisioning also should be part of your continuous delivery what I mean like if you work for example in your own premise

application it can be your anible script it can be your puppet script if you use still puppet or I don't know Chef for example if you work in Cloud most probably it will be terraform or cloud formations or maybe pulumi or maybe something else doesn't matter currently doesn't matter but your infrastructure provision and configuration how you create this environment you also have to should be a part

of your continuous delivery so you create this cot also as a part of the whole development process and create first environment like in my case it's death environment it's my first integration environment and where we run additional couple of the tests so in my case it's integration test and acceptance test and again as you can see I have quality gate which is return to my continuous integration

dashboard to my for example genins to share with me what the current status of Integrations test currently when I work with integration this means like we integrate with external world like for example with database with other micros services on so uh for sure in continuous delivery most probably you will have another environment and this is why I said you need to create artifact and store your artifact

in artifact repository and after that promote it to other environments for example to DF environment in my case the next one will be QA or it can be in your case I know staging uat uat user acceptance test or prro doesn't matter how to call it but like next step of your environment and on every different environment you can run you scope of the test or type

of the test so in my case as you can see I have on uat or preproduction I'm going to run performance test so performance test this means like we check the performance functionality so in my experience again how we did it like we have absolutely exactly the same amount of resources on our stag in orro environment run through the GM performance test and look at the analysis

with previous uh test PR previous releases for example and compare it uh for sure it can be different type of a test like I don't want to say that you have to run all of them maybe one of them mandatory test for sure it will be unit test you have to write unit test I don't want to say that you need to cover 100% of all of

your code but at least I don't know like maybe start from 30% or 20% it will be good start but try to achieve like magical numbers like 80 or 70% but it's not like main qpi so again different type of test depending what's the how much of how many do how many resources do you have and so on so we move on to the next step it

will be continuous deployment and what's the difference between continuous deployment and continuous delivery the main difference it will be automatically deployed so it means like we don't need to wait for some approve we don't need that someone will say hey I'm happy with this release can it move to the production but there is a challenge how I can deploy anything to the production without affecting my customers

my end users and I prepare what the current uh deployment strategy exist uh not like on the market but in general what kind of deployment strategies exist and what their you know like uh parameters what their uh functionality or something like that so first of all one of the most important part will be zero down time so it means like we try to avoid effect on our

end users and we have uh five different type of deployments uh I will go deeper for everyone like super super quickly but as you can see here I create this characteristics like zero downtime real traffic testing Target users Cloud cost roll back duration impact user and complexity of set up so first first first of the strategy it will be recreate most probably you see that Financial Service

like Banks still working on recreate strategy what this means like you get some SMS for example in your mobile phone and your bank said like during this tonight we are not going to accept process through the ATM so you will not be able to get your cash for example or we will have some issue with transactions or something like that so maintenance it means like we just

stop previous environment or delete previous environment it will the best options because in this situation we apply immutability of our infrastructure and create from the scat the new environment but my experience showed me that most probably you just stop the environment apply new change deliveries these changes maybe you will see some configuration drifts but again like uh best options will be created from the scratch if you

have your um configuration management or infrastructure provision part of continuous delivery you can create um recreate but if you don't have you have to do it like manually just like stop and apply new version rolling update or rolling sometimes so in this situations what's going on uh we have some small part of new new version and we transfer like start one instance of new version and uh

if everything looks good from automation perspective this instance up and running we start to send the traffic to this new new new version and down on our old versions and continuously rolling like add new resources to the new versions controls through the load balancer like super quickly we can move from old version to new version not super quickly because we have to do it like step by

step we can create very quickly from the scratch new version but we move pretty easy out of the box available in kubernetes World in many Cloud providers if you have some manage services and you need to make some updates Run update most probably will be available out of the box because it's super easy to implement the next one uh blue green uh so like two environments uh

one of them uh drawbacks of the blue green that you need to support two environments okay support it sounds not not so incredible but you have to pay for two environments this two environments should be absolutely the same so if you have in your production like 100 uh virtual machines so this means like in your blue and green environment you will also need to have this 100

virtual machines and you have to pay for all of them so controls through the lot balancer one of the Main benefits from blue green so first of all you can test real production like real Cod so when rolling update you're not able to test it in blue green you can do it and what interesting part is that you can super quickly switch from version old to version

U or for example if you need to make some roll back just through the load balancer you can do it Canary Canary deployment interesting picture yeah so I tried to decide to go deeper and understand why Canary what what why Canary it's a bird everyone knows like Canary it's a bir super small bird and I figure out that in Spain miners who works under the uh like

you know like in the places where the quality of the air sometimes can be not so good not like Bitcoin miners I mean miners uh yeah so how they test the quality of the air so they use this and put it to the place where they working and if this bird is alive like super small part like super small bird and if this animal like is alive

it means like everything looks good from air quality perspective and the miners can come in so it's very transfer to what we have on Real uh production so what is scany deployment we move to real new version of environment like 5% 10% depending on what you prefer in my diagram you see 10% and I sent real traffic to new environment and see what's going on like based

on my observability locks uh Trace metrics understand everything looks good my customer is happy we don't have some performance issues or something like that and we can move all other traffic version the next one the most interesting and more complex version of deployment strategy it's AB testing so first of all AB testing most probably uh we will need only when we need to have some business uh

changes like for business metrics where we control business Matrix let me explain what exactly is AB testing so first of all in AB testing we select a target of group of real users so for example you have some uh game application like create a G mobile game yeah and you want to selective people who spend in your application every day 30 minutes or 1 hour and you're

going to do some redesign or Implement new Quest or to do something new and you want to test how how long people will stay with new functionality if you release only for this group of amount of people because you don't want to deploy it for everyone you want just like like to test on some small amount of group of people with specific characteristics like in my case

who spent 30 minutes every day in my game and I want to test this functionality so I selected this group and send the traffic uh to my new version of the applications so it's a no on the market currently does not exist very good solutions to make a testing so most probably you will need to do it manually I mean like to create your custom Solutions or

integr wa with some uh some third party solutions to select these Target groups let's back to our uh table so characteristics of deployments so what's we reviewed so first of all recreate uh downtime please don't use recreate it's like you will affect your users uh rolling so good options because zero down time super like it's super cheap because like you create for very short period of Time

new resources uh rolling uh if you need to to do roll back it will take the same amount of time when you may create new deploy new versions you will most probably not impact of your users a little bit and complexity super easy uh blue green what's the main options is like uh so first of all uh rolling uh roll back will be super quickly and maybe

you will affect your users because when you make this network s to the new version if you have some sessions on previous environment maybe you will affect like previous your current users so uh Canary I show it's can be a little bit complicated to implement and AB testing so first of all uh real traffic Target users and one of the most complicated versions of deployment what you

can do production okay we moving on to the next part metrics uh as I said on the start if you cannot if you don't measure some system if you don't measure your like performance of your continuous integration most probably you don't control your continuous integration continuous deliver on the market exist very popular metrix which is called Dora or it was created by U company not like company

by research devop research and assessments is currently the part of Google so so what the metrix is the first one deployment uh frequency so how often do you deploy new functionality to the production not like to the def environment or Q environment to production the first the second one it's lead time so it's not lead time you know like uh so my understanding of lead time that

when I have an idea to implement something and lead time it means like from idea to production no lead time here it means like how how much time will need it as soon as I committed all of the functionality to for example to the main branch in G and all of this procedure of running continuous regression my tests or for example qu review again remember C review

you remember like you can stock on C review all of this stuff and when I will see these changes on production the next one uh CH change failure rate so how often do you have an issue when you deploy to production you know like I see the correlation if you have more frequent deployment to production most probably you will have less uh failure rate and time to

restore so uh again like how much time you will need to make this restore to a normal situation for production uh and what these people said like if you uh if you have deployment frequency like on demand it means like you can deploy at any time or for example as soon as you're done with all of your deployment development process you can deploy to the production uh

your lead time less than an hour so it means like this time when you commit it to the main branch so it's will take only less than one hour to deploy the to production and fail rate very close to the zero you like El light Elite software development teams in the latest researches from 2022 and 2023 they didn't uh reported about the this metrix in 2023 they

mostly spent more time on generative EI and how it's affected on devops area so but still I believe this metrix is very important but not ex that what I wanted to share to share with you so my personal metric what I created when I work with very difficult customers I would say uh as customer didn't trust to us as a company uh because it's like very huge

Germany and you know like when the customer said like okay you are sounds like a good company but you know like we don't believe that you can do some good cicd okay let's try to measure what we currently have and what we do with our continuous integration and do we have some improvements or not so the first metrix what we had it was build failed percentage uh

so again like you committed something you make some changes and how often it will be red you know like we prefer to see the green light on our continuous integration so it's like what's the percentage will be red my recommended values will be around 5 10% not the more because in other situation most probably uh developers and other teammates will not trust your continuous figuration system because

if you have like every second build is red is failed so what's the reason to look at the continuous integration dashboard uh time to feedback it's very very very important guys so like I hope you will learn something from my session today at least remember time to feedback less than 15 or 30 minutes I put it here as a number 30 minutes but my recommendation will be

15 minutes why because we are human we are not genpi robot who can control everything and so on we are human and what's what this means like a human so I'm developer for example or I I work as a devops engineer and I'm done with my terraform Cod I done everything I commit it and push it to my main branch so and waiting for like I know

going to drink something like coffee or speak with my colleagues and during these 15 minutes I can remember all of this context what I did in my task what I did in my ticket after 15 minutes or 30 minutes most probably I will take another task so this means like I will switch my Concept in my mind to new code to new functionality and it means like

I will need if I need to back to my previous ticket I have to spend again this time to look at this code remember what I did here and so on so try to make sure that you have time to feedback less than 30 minutes it's a Time window when people most probably switch to the next uh task to the next some you know like responsibilities or

someone and so on percentage of unknown to known so uh it means like you don't have any issues on your infrastructure in your continuous integration um infrastructure I mean like you have junkins and sometimes junkins can be not very stable junus is amazing tool is continuous integration but if you have a lot of plugins sometimes it can be terrible to support it so like this this metrics

show me how often I have an issue with uh Q on your build serror maximum Q build terror is less than five so if you have like long Q I know like 20 50 uh Bild is currently in the queue it means like developers waiting for when the result their cord will be tested run through the my continuous de creation and it has huge correlation with time

to feedback so B your Q on your build here bigger time to feedback average Q time so again like less than 10 minutes so me like very very quickly can be proceed and the last metric previously I call it like pull request count so how many pull request do we have uh it means like to show that your continu integration proceed during the day I don't know

like 100 pull requests you run 100 continuous uh integration bills and so on currently I would say I would add one more it will be called like pull request cost with the same abbreviation like PRC but with a little bit another definitions pool request cost so in nowadays when like economy situations it's a little bit I would say difficult uh and we need to control how how

we spend in clouds because clouds it's amazing place for scaling but at the same time uh you can have I know dramatic Bill like Bill at the end of the month or for example at the end of the year so please control how you spend your money in the cloud so what will be our blueprint uh to like to summarize everything in one place again uh very

quickly we start from our Cod I want to uh like remind you again that continuous integration started from your Version Control you go through some process so again it can be dependent on your configurations it's you know like to show you possible way to inspire you for example or show you where you can have some you know like maybe opportunity to Implement in your configuration in your

continuous integration uh so in continuous integration we run like code analysis like up to you what the tools to use unit test run artifacts please remember that the main goal of continuous integration it's artifact and feedback feedback to developer to devop person like to to show like hey you have an issue for example uh we provision C infrastructure is also should be part of continuous delivery run

some amount other test like performance synthetics integration acceptance test and will have you will need someone who said like let's go to the production I approve these changes or you have like release life cycle every two weeks like for example in my experience mostly we Deploy on Wednesday because it's not a Friday please don't deploy to production a Friday evening so today you will have after party

so today at after 5:00 p.m. production and you see I didn't mention anything about these blue blocks uh but I want to spend like my last two minutes to to set about that continuous preparations or monitoring it also delivery continuous integration why because you have to control not only your production From montoring perspective From observability perspective but also low environment why because for example on Q environment

you for example can work together with QA teams or with other microservice teams and if your QA environment is broken it means like other team cannot use your environment please control monitor your low environment and create the instructions what you should do if for example database does not respond on your query if you have this instructions on low environment most probably you will have the S you

will have the same instructions how to fix it on production that's it uh thank you very much for your attention and it was cicd uh yes thank for real amazing speech but uh we have a lot of questions here and uh a lot of okay yes somehow you are popular this morning okay I don't know do you see them I think that we can go and read

them is it possible to present it to my scin the policy for removing fish flag I see questions uh maybe so first of all maybe we have some questions here inside in the room yeah like I I see we have let's start one question from room and one question from online yeah let's do it this way if you do not mind one question here uh what is

the proper way to measure Dora those other metrics in distributed systems because different uh Services could have different build time and even code reviews could have different requirements for core services or or minor services and so on so how do you make an average or something like that from all those metrics first of all want to pay attention that Dora as a metric exist more than three

four five years something like that and interesting fact that I don't see good implementation on any cons like gitlab C GitHub actions steam City Azure devops none of these uh from ads least none of these tools don't want to implement Dora is a metrix at all so recently again it you be reference to S work radar can you please look thew work I saw one of the

paid version of the tool I don't want to advertise anything but I saw one of the paid tool can currently do D metrics to collect what your deployment frequency what your failure rate to connect it to your uh bamboo GitHub uh and other uh tools and your monitoring system to collect everything in one place this is how you can do it but it's paid version I don't

remember what the tool and I don't want to advertise here on the conference you you can look and find by yourself so work are like it's is like also a little bit is advertisement but anyway so it's like where you can find answer uh currently honestly I don't have like to say you should do in this way uh so question from let's go to the questions because

we don't have too much time what about end to end tests would we be getting a merch uh end to end test can be run and know most probably it's a continuous in deployment process like delivery process so you can run your endtoend test there on this environments like QA for example uat and so on uh should we run this test before you merge to the main

branch it's a good question so if you can implement it and do it you know like on the way that for every Branch you create a separate environment it will be great to have entry and test before you merge to the main branch but if you don't have these capabilities uh basing on the cost basing on your tool set and functionality I would say you can run

your endtoend test on this like QA PR production uat envir yes okay actually we are running out of time and when new presenter is coming I really believe that we have a lot of questions you see it I think Victor will not disappear from this conference soon so you can catch him in the yes okay so first of all I would like to thank you and to

give you a small present thank you very much thank you thank you thank you and warm Applause to Mr Victor I hope you learned something new today it was my main goal thank you very much