Koray Oksay: Continuous Deployment: The GitOps, The Pipelines, and The Ugly

This talk covers continuous deployment and GitOps, addressing traditional pipelines' challenges and proposing solutions for smoother operations. The speaker, Corai, emphasizes the importance of automation in the deployment process to avoid manual errors common in several environments. He discusses CI/CD practices, particularly continuous integration and delivery, as critical components to streamline deployments. The talk also explores GitOps principles, comparing tools like Argo CD and Flux for managing Kubernetes configurations declaratively from Git repositories. Through hands-on demonstrations, the speaker illustrates how changes in Git repositories automatically sync with Kubernetes clusters, ensuring consistency and a single source of truth.

[Music] ladies and Gentlemen please welcome our next speaker cor o presenting the topic continuous deployment the get offs the pipelines and the Ugly yeah thank you very much um welcome everyone um well I I just landed this morning um to V I haven't have much time to see around but uh it looks like a very nicity so far I hope I will have some time tomorrow to

uh see around um today I will talk about uh continuous deployment and giops and the our traditional pipelines uh and some good and bad stuff uh around those um my name is corai I work as a communities consultant and instructor at kuber Matic uh we have um automation for managing multicluster multi cuber clusters on multiple uh or different clouds uh we also do consultancy and training as

well um um I I live in Istanbul um work remotely uh the company headquarters are in Germany Hamburg um I am a cncf Ambassador for the last month so with the latest version I have some um certificates around kubernetes also a contributor for the C Kates infra um I'm also kdul organizer which happened last week actually in istanal which was a very good event and I also

know it's how hard to organize an event and um I want to congratulate all the all the organizers it's um it's a really hard job uh so how many of you use kubernetes on production oh almost everyone yeah um who uses already okay very nice who considers to use it wants to use but doesn't want to spend so much time Myra thing changes Okay cool so you

are in the right talk I hope um so for releasing and deployments um there are some anti patterns um worst or the the biggest one uh is the manual deployments so uh when you deploy a stuff to first test the environment then maybe staging how many uh environments you have you do the maybe development and then testing maybe staging or acceptance and then you go to production

if you do all those deployments manually um well by definition it they won't be the same so if you just change one step if you you forgot to export a variable uh in one step and then they won't be the same and they will probably uh outcome with a different result all the time so we definitely don't want that so that's why we have the whole pipeline

stuff and make everything automated uh when we say manual deployments here I mean really like checking the scripts uh running commands manually uh so of course if you have I mean for for production you might want to uh you may not want to do the deployments all the time maybe some specific times Etc but this is like triggering the automated platform or automated thing uh whatever so

here the automation is the key point it can be manually triggered but the deployment itself or all the running the commands should not be uh manually um the second an pattern is to uh deploy the production like environment this could be acceptance this could be called preo this could be called staging whatever it is uh a bit late so most of the time uh people do it

like before actual release like a few days before maybe maybe a week before to make sure that the product owners or whoever responsible uh to check the if this is what they actually want but but this is a bit late because you just about to I mean you are very close to deadline you go there production like environment and you have problems so the ideally you need

to deploy um those type of environments uh more frequently so that if there are issues with the scaling maybe or um other performance stuff or related to permissions you need to see them early uh in your development stage and the third one is the manual config management the configurations must be in a centrer place everyone I mean you should not change the production configuration manually it should

be stored somewhere and it should be the same as the live environment at that moment um so the cicd stuff comes to uh our help here so when we say cicd it's a continuous integration so when you develop something you push your code and then some integration start you run the the something runs your um tests like the um it could be integration test it could be

unit test it could be uh performance test whatever those type of Integrations runs and you or the development team makees that um the code works as they want as they as it's required um the second part is the continuous delivery or continuous deployment there's a difference there but um I here mainly mean the deployment uh part um when you test your code when it's ready to go

production it should also need to uh be delivered automatically or continuously uh of course again it might wait for some sort of um uh approval uh but still the whole process should be done automatically so cicd helps us uh with the the the three problems I mentioned um previously um and it solves almost all of it so this is how a pipeline looks like in general so

developers um sres platform Engineers devops Engineers operation Engineers whatever you call them uh I think they are the same people with different names uh depending on the company so they make uh some code changes um for for your application they also make some configuration changes could be AML file it could be in file depending on um your uh application uh your uh language Etc the code is

pushed to git or gitlab or whatever GitHub here but then the pipeline runs um so because I will be talking about github's it's the in this will be in the kubernetes context so a container image is created it's sent to the registry and then with some Cube cutle commands or maybe some different ways uh this change the new container image uh tag is pushed to the uh

related cluster so that's that actually solves a lot of our problems uh but it also has some issues here um so one of so there are four main issues uh one of them is the correctness so uh with the whatever pipeline script whatever tool you are using you need to make sure that um it is it does automic deployment so if some one step fails then you

need to make sure that either everything is roll backed or it will be light again and everything will be um in a desired state so that that part is important you need to make sure that uh within your uh deployment pipeline um security is an issue because in that model you basically provide uh credentials to your cluster to a third party in the previous picture it's GitHub

I'm not say saying that GitHub is I mean you can you can trust I'm not saying that it's not trustable uh but in the end it's a third party it's out of your organization so um you are just giving away some credentials to to some um also the current state is not exactly known so you might your your pipeline might run everything worked very well it deployed

the code your your new uh code new container image uh but you are not aware if someone else made some manual changes on the clust this could be the case so yeah yes when you check the pipeline you think that okay this version is now on production but when you check the actual cluster it might be different because someone else um might have might have done some

changes and well um about the uh developer experience so there's also debate about devops if uh about shift left um if the developers should do the operations Etc that that stuff but in the end um I mean in general Dev developers do not want to deal with the operational stuff they don't want to deal with the um deployments check if deployment is successful Etc uh they just

want to um deal with their own code that that's it so for the development experience also um in in the previous or in in this uh form they they need to understand why the pipeline failed what happened if the um there was a problem with the cluster if there was a problem with the connection if there was a problem with the script or authorization Etc so uh

regarding the developer experience side uh that push model is also not the best again solves a lot of problems if you I mean if you don't have anything this is great uh but also it has some some issues with it as well so in the real world uh this this happens with a pipeline most of the time I I mean that happened to me um so basically

especially if you make um if something fails and then start to make um manual TS on the production system then it gets more and more complicated but then you can just add the LinkedIn skills like a devops Automation and troubleshooting you can def that troubleshooting but yeah it's um it doesn't mean that um you solve the real problem there so what git Ops provides us is a

different um approach it's a pull method this time so instead of pipeline pushing the changes to the cluster something in the cluster pulls the changes from git itself so in that case um again different people make different changes and when the CI pipeline runs it creates a container registry so it creates a container image and pushes pushes it to The Container registry but then it makes a

configuration update like a DML update or something um and when this is done uh it is read by or it is just pulled by um the tools it could be Argo CD or flux there are some some other tools but these two are the main ones and they basically they apply those changes uh to the cluster so instead of someone so something running inside a kubernetes cluster

just guess the changes but of course a recommended practice would be uh this way so you have a repository for your code and another repository for the configuration so that that makes more sense um instead of a monor repo with code then altogether uh this one again CI makes a config config update to the um configuration repo and then these are uh pulled by the uh GitHub

tools and applied to the so so why do we have giops or what are the principles around it so um well kubernetes already uh brings us the declarative approach right so so you when you want to run a workload you already um Define it declaratively in the cluster saying that hey I want this and that uh that many replicas that version um if the load increases increas

it to that many replicas etc etc um here with giops it's also deorative so basically you change the ml files and then these changes are applied cluster um because everything is in in Git it is vergent and immutable so you know who made what change when and if that was that caused a problem or not um another thing is that again uh whenever you pushed something to

a certain brand Branch um so with a PR or you directly do it again depending on your setup um those changes are pulled automatically uh to the cluster so uh there is no manual intervention could be there could be some manual intervention um that is also possible but um by default there is nothing and uh cluster just check uh pulls the changes and then applies them and

again with this it's uh continuously reconciled if someone goes there make a manual change to the cluster then uh um the tool just reconciles them okay so on git side it's three replicas someone made it five replicas um then Argo or flux just takes it back to three because yeah you can't make manual changes and you shouldn't be so with this reconciliation you make sure that um

at a given time like the what you put on git repo and the cluster same so that brings us the uh single source of truth truth um basically you can just trust whatever um on your git repo uh it should be uh on on the cluster itself as well another question is is it perfect um not necessarily uh so this is um so we uh have a

um a project with cncf uh and we provide support for the pro clusters of kubernetes project um and we were I mean on those eks uh Pro clusters um there's flux we wanted to do some changes um so this is like um public you can check what happened there but basically we made some some wrong changes uh with my friend uh Marco and I realized it uh

while the uh PR is not merged um but then we decided somehow decided that it will be fine but it wasn't then the flux just removed like removed itself from the cluster but the good part was that uh we just uh got back to the previous state like in two or three hours and the long time was because of there were some um manual uh Secrets external

Secrets needs to be defined by by some people so that was the problem if it wasn't there the cluster was going to be up again in I don't know 10 15 minutes or so um so in the end if you make a mistake uh then things go wrong again with kops it doesn't mean that everything will be perfect there won't be problems at all there could be

problems as well um now it's demo time it didn't take much nice uh so yesterday I did uh polls on LinkedIn and uh Twitter uh people wanted to see Argo is there any preference here if anyone wants to see a demo with Argo I have both of them prepared doesn't matter I'll go with okay then I want to do flux but um people don't want it so

this will be just a a uh Argo instruction um is it visible B enough okay um so to to to install Argo CD so um I don't know if you know this but there's a uh platform called killer Coda um basically now here I have a kubernetes cluster with um two nodes I can manage it from the web browser and I'll do uh run some stuff uh

with it here um you can also go there and try this out it's um free um to to to deploy Argo CD uh they provide some um manifests uh by so I made some changes to this one um to to fit this demo and my environment but um basically you can just go to the documentation and um see those manifests so the install these are non High

available ones and you can also install per name space or there are also High available ones um there are also uh Helm charts but these are uh to my knowledge they are Community um charts so not from the Argo project project itself so what I do here I first create the Aro CD name space and then install this install yo file as you see it deploys a

lot of stuff uh some crds deployments uh Network policies Etc and um if you see actually there are three uh main crds applications um applications application sets and projects these are the the important ones and let's wait for Argo to be ready um so there will be some PS so they are all ready now great and if you check so there is the application controller um server

repo server where is for its uh personal uh like a argo's own configuration Etc so it deploys some stuff to the um if so for the people who don't know it or are not aware uh Argo has a UI or like yeah a UI tool where you can manage everything from the UI but you can also manage everything from the CLI it also comes with the CLI

on the other hand flux uh is like has only the CLI there are some uis for it but I would say they are read only uh they are just to see things around but not to manage the um flux itself so these are the main differences I would say so I will also install the um CLI tool and um I go CD yeah it has some we

will use those commands um init so um you can create application so these are called applications um when you want to deploy a workload it could be a public hand chart it could be like pritus or something it could be your own uh application um but these are called applications um in the uh Argo so this is what a Argo application looks like or basically you just

say that hey this is the name this is the uh name space I wanted and I want to deploy to the very cluster um what you can do you can have a Argo cluster and manage other clusters from from that management cluster I would say but you can also uh have the or do the um management from for the local cluster there's a you can also make

projects there's default project I did not create more and this is the main part so you just want to say you say that uh from this repo I want to install whatever under guest book uh path um let's check this one also out this yeah Okay cool so here um I just fored it this is the from the Argo project itself um so there is a guest

book path and whatever under here so these two yl files will be applied to my classer so that's simply what this application will do and during the sync I also said that um if there's a new name space needed create it as well if you don't put this and you want to let's say uh deploy this application to Dev namespace and if it doesn't there then it

won't be able to sync because the namespace doesn't exist you need to create then uh create it manually so let's create this Aro application it is created and let's get it so you can just take it with Cube cutle commands so let's do it once more um so by default because I did not um add sync options it's out of sync and the health is missing out

of sync means that um there is a difference between what is on the cluster and what is on git side basically here uh nothing is um installed uh I think it was going to install it on guestbook name space nothing is installed so there are things that we need to uh install on git but they are not um on the on the cluster at moment so let's

uh this is my admin password let's see the UI are you with me or are you all like wanted to go home this is the last last presentation so this is the this is the application we just created and as you see that the same information here so if you're a CLI guy then uh CLI person uh then you can do it the other way around um

so as you see it says that I need to create those um this is the app it will create a service and a deployment but right now it can't um Do We sync it yes let's let's sync those so here you can just click on sync synchronize so here you see that these are out of sync right these two things and let's synchronize them and magically Argo

will be creating those yeah because this is a deployment then it created the replica Set uh because it was a service it created end points Etc and now a pod is um just coming up so it is synced so the same configuration or the same y files are now applied to the cluster progressing because the um pods are not ready yet or whatever you deploy are not

ready yet not all of them so as soon as pod is up running um it will just say that the app health is um app is healthy we can also check it here with Cube cutle it's sync progressing same stuff so again depends on how you want to manage and here you see that oh you see that in the guestbook name space guestbook UI is container creating

I think it will be up soon yeah so it's running and here it's healthy yeah the UI is yeah now it's healthy everything is running fine that's the first part you can create the same application using the CLI so initially I created with with with a yl file you might be you might hate yl files um I sometimes feel like I'm a yl engineer and then my

eyes are are bleeding and it's not nice um there are two more ways so you can create an app here from the UI directly give the name give the um repost URL everything whatever I put the um yo file or um you can also use the CLI command line and um uh with Argo CD app create blah blah blah you can also create it uh I just

need to define those first so I just log to the Local Host because yeah this is actually a local machine on like um on the cloud killer Coda so here I created application this time it's named gbook app and you see the gbook app is also just here again out of sync I didn't put uh sync options so not running at the moment if you also list

um so guestbook this is the first one it is sync healthy and the other is out of sync and missing and Argo CD app sync now I want to sync the second one in the app app D2 name space it's progressing and it's done this this one is faster sorry yeah so the IM the the container image was downloaded with the first one so it just started

the image so initially it took time to download the first image so that's that's the whole uh stuff and you see UI there um sync and now healthy everything again CLI UI whichever you prefer or the Y files um it's up to you um so let's do some GitHub so so far I didn't do anything on the git side I just created some stuff kind of manually

um just just to show you around but that's that's the um more important one so let's go one of the applications um this one so I want to change some settings here and I want to enable autosync so this was disabled by default I also want to prune resources and sell Feld the prary sources is that um let's say you made some changes for example you removed

the Ingress so in the very first version you had an Ingress object but then later at some point you removed it um with this prune uh option it will also remove the Ingress as well otherwise um by default maybe I should say Argo will not remove those sort of of resources directly the prune needs to be uh enabled so um let's go here and change some settings

in the deployment I want to change the replica account to tree Comm scale to three now comment here hopefully short time so um by default it checks it every 3 minutes so 180 seconds but I made it change it to uh 20 seconds for the demo otherwise I did it once and I waited very long time uh so you need to get some lessons from your changes

um now we have three pods you just made the change on the git side and it's just applied here in the um we comitted pushed and um what I can also do is so I created both applications from the same config which is which is it might be okay so you might have different uh organization like different um teams working on different name spaces with the same

uh application um then this one because I did not enable autosync here uh this one says that it's out of sync because in git it's three replicas but here it stayed as one replica because it wasn't Auto synced you can also do that for production um to make sure that you made those changes in a given time after getting some approvals from the management or what however

things work for you um you can also make some manual changes so I just change the um number of replicas to five for the second one and as you see here there are five running and it should also be the same please yeah there are five ports running but again this was not um Auto um enabled um auto sync so if I enable it as you see

now it's removing the port and make it three again so simply you cannot make manual changes um to the cluster if you enable auto syncing um so regarding deployments uh what what I did was just to I deployed some yumo files for my uh applications right but you may want to deploy some customized uh applications or applications that you normally deploy by customized um that is supported

uh you may also want to install or deploy H charts um that is also supported but there is one catch with helm um that is um Argo CD does not uh so what Argo CD does is that before um deploying the helm chart it runs Helm template creates manifest and then apply those manifest so you simply don't uh have a you don't see anything with Helm LS

or similar command so actually a chart is not deployed but the Manifest about for that chart is deployed of course you can reach the version Etc with the annotations and everything but um you basically don't see the ham chart itself it's different for flux flux actually uh manages it as a ham chart um it has its own ham release thingy uh another object but it treats them

as a different object so these two are the main differences between Argo and flux I would say the UI um and how they uh treat the H um let's deploy a let's check our customization thingy so um another one um so uh in the repo there's a customized guest book so this is it's the same application but it's with customize um where is it here so the

same deployments but it has a customization y that's that's the um the main difference so if you want to use customize for your different environments you can also uh do it with with Argo and also with um um flux as well so let's create this one now I should have a new app K demo is here um for this one um as you see sync policy automated

um and prune and self heal are true so for this one I just applied it and directly it uh got all the changes so it didn't wait for me to manually update things and let's check the tag of this here with the K demo um when we go to the Pod you can just see the live manifest and it is version 0.1 right so let's make t

change the tag to 0.2 um change tag version let's Commit This and hopefully within 20 seconds there will be a new p and this one uh actually there will be a new uh replica set as well because there's a new version by and it yeah here you go so there's a new replica set with a new version um 0.2 I hope yeah here you go so you

just deployed the new version um to the cluster and I yeah mentioned about Helm it just applies Helm template and everything uh um there are um a lot of documentation around it best practices how you need how you should be um organizing your giops repository with what sort of directories names Etc both for Argo and flux um you can just check them um these are you can

just um do those scenarios as well if you want and in ker Koda there are many more um you can just check them out and um see how um so this is all I have to say um yeah this is the way and uh hopefully um you will be start using Ops I think it's um it will help you more uh with your deployments thank you very

much yeah thanks for the presentation but now we still have like questions at least six at this moment so I will just ask maybe any live questions from here if not I will start from the most rated yeah I see two two likes like for two questions so the first what would be your guidelines recommendations and having to decide between us cargo CD and flug CD um

I kind of answer that I think so the main difference is if you want to use the UI or not to manage things uh if you your team needs a UI then you should go with Argo CD because it doesn't exist with uh with flux um but that also has a complication um last week as I said KD stanbul um we have a company called trol in

Turkey um say EC e-commerce company um they were using Argo and they had around thousand applications or something and I mean for different teams and everything because like they were managing so they were they had a um Central Argo cluster where they were managing um other kubernetes clusters um they said that at some point the sync time uh increased to 7 minutes or so and uh they

had a lot of issues with the UI because the UI was not able to load those thousand applications which is U I think expected so um you could end up with those kind of uh performance related issues if you have um that sort of large system you might need to uh change your uh architecture but I think it's up to your preference uh both both tools are

fine um another point with flux is that maybe you know uh wew Works was the the main uh developer for flux uh now they are out of business um I know that the maintain some maintainers are already um um like sponsored by some companies but and it's also flux is also a um cncf graduate project or C project as well so there is a support behind it

but it is also a little bit um you a hard time I would say um I think both are fine okay let's go to another one well uh you already read yeah yeah yeah yeah the the ugly is uh people I think here I mean if you make a mistake in in whatever on I I think the the the the good part is the automation you should

automate things it doesn't matter if it is like getop actions or giops the auto automation is the key Point uh the important part I I think this is all what devops is uh about um but whatever you do you need to be you need to test stuff and make sure that they are working and you shouldn't go around what you build so if you have a pipeline

but I mean I mean you you should use PRS you shouldn't directly commit to uh main or Master branch that sort of thing because you are the admin you shouldn't go around the uh and do do uh stuff uh out of policy let's say then things will be fine I think yeah one more about TGO uh yeah uh what is the recommended way integrating cargo CD with

multiple Cas clusters one instance per cluster or one Argo CD for multiple clusters for the maintenance yeah yeah so that I think depends on how you want to um do things so if for example you add new clusters all the time or they are coming and going I depending on your environment maybe having a one Central Argo and um adding new clusters that would would make more

sense um in the end if you have let's say 10 kubernetes clusters and in install uh Argo in 10 of them then you also need to maintain those 10 uh as well so probably um maintaining uh all those clusters from one Central space would make sense but as I said with the transal uh case uh they had a lot of clusters behind Argo and at at the

time then they had um performance issues so depends just like other answer I see that it's already like time but we have one uh also quite high like rated question short one why preserve resources on deletion is not uh on by default on Argo CD I don't know I mean I honestly don't know okay thanks for your honest uh answer and actually presentation I uh will ask

like audience like to give and share your warm uh applauses yeah thank you much yeah thanks thank you very much