From Jenkins to Tekton: Our Journey Toward a Kubernetes-Native CI... Mustafa Barış Ege & Özge Aygül

This talk covers the journey of migrating from Jenkins to Tekton, emphasizing the transition to a Kubernetes native CI/CD approach using Argo CD. The speakers, Ozge and Baris, discuss the scalability and management challenges faced with Jenkins, such as plugin management issues and inconsistent pipeline practices. The session emphasizes the need for a declarative infrastructure and ephemeral execution capabilities, which Tekton provides. The speaker highlights the architecture of their CI/CD process, focusing on how Tekton's pipelines are designed for efficiency using Kubernetes resources, alongside an integration with Argo CD for continuous deployment. The talk concludes with a live demo showcasing their CI/CD flow, demonstrating real-time pipeline execution and deployment.

Hello everyone. Welcome to our presentation. Today we are going to talk about Jenkins to Tekton, our journey towards a Kubernetes native CI/CD with Argo CD. Thank you for joining us. Today we want to share journey. It's a story about migration, sure, but mostly it's a story about moving our mindset from managing servers to managing pipelines. We start with our journey began and we evolving this CI factory

and we talk about how we met Tekton. After that we explain the Kubernetes way of CI/CD and go through the technical strategy. We briefly talk about our CI architecture and our CD architecture with the Argo CD and the image updater. will continue with the cultural challenges and lesson we learned. After that we wrap up with a live demo. I am Ozge. And I'm Baris. >> [sighs] >>

We are a team working We are a DevOps team working with generally Kubernetes and cloud native technologies. Our goal is not just to run pipelines building our mindset. Sorry, I'm so excited. >> [laughter] >> Thank you. >> [applause] >> Our goal is just not run pipelines but to build sustainable, scalable and standardized delivery process to feel our developers sleep well at night understand how we got there.

Baris is going to show you how our nightmares used to look like. All right. We had a similar Jenkins structure as you all know and love. We all love Jenkins, of course. It was fine at first because we had like four repositories or something like that which was fine. But in whenever the repos grow and teams get got larger, we needed something better. We had problems with

the Jenkins. Of course Jenkins is a great tool, we all know it. I'm just repeating myself. But we had some frictions about [snorts] the First, of course, we had scalability issues. The static agents couldn't handle bursts loads loads efficiently. And we had management problems as well. We had plugin hell as we say which caused a lot of trouble for us. We had a lot of plugins which

were kind of good sometimes but with the upgrades and etc. It was a it became a full-time job. And also we had a consistency problems. Something some some like we had like a lot of repositories in Jenkins files and the Jenkins files needed to be changed. Whenever we changed the Jenkins files, the should be go to the PR stage and we needed to approve that etc. So

this was a huge problem for us. Now, what we what we go to the next stage. We really realized that we had Kubernetes applications and cloud native applications but we had static agents running in the build time. However, we needed to change the CI factory. We needed a declarative infrastructure as a pipe pipeline as a code structure and also we needed a ephemeral execution with the zero

waste containers etc. Like Kubernetes provides us. Also, we needed a Kubernetes API control tool because we don't want to manage the role-based access controls or securities etc. via different servers architecture. And also we all also need that GitHubs compatible tool. That's how and why we met Tekton. Now I will talk about the Kubernetes way of CI/CD. Firstly, I want to mention that Tekton use unified pipeline architecture

which means we have one pipeline and one webhook and we can build our projects with that and allows decoupled logic meaning that this pipeline logic separated from the implementation details. Also, it's event driven and it's case on demand. In the efficiency parts um Okay. pods has it has ephemeral execution which means it's uses zero resource waste when it's in the idle state. Also, Tekton is Kubernetes uses

Kubernetes native secrets and role-based access mechanism. Now I want to talk about the Tekton deep dive parts. custom resource definition and it's fully Kubernetes native. the smallest unit here is step. It runs just a specific scripts and or a commands and these steps grouped into task and runs these tasks runs as pods. we go through the pipeline. If we compose these tasks, we can define the pipelines.

We use triggers to initiate a pipeline run and triggers here has trigger template and the trigger binding eliminates and detects the related parameters for the pipeline run. up and running. I just mentioned the event listener. Event listener here is our cluster's bouncer we can say. It detects the webhook and extracts the related parameters like commit chars or branch names whatever extract these parameters and creates the trigger

binding and the trigger template these trigger templates launches pipeline with these parameters and creates the pipeline run and after that task run. These steps These parameters can be with these steps with using the workspaces. All right. Now I'm going to explain the, you know, picture I think. There is a not a thing that's a pipeline as you see. We have one pipeline. We don't have a lot

of pipelines. No, it's not manageable for us. So as we say how we tackle the pipeline and we say the one pipeline to rule them all. We need to the Sorry, I closed that I think. Okay, that's fine. >> Sorry. We need the container task approach with demonless with build it Kaniko. You can use builder or whatever, it's fine. And we had the profile logic fixed like

we did build part for example in Docker like NPM builds or whatever you need to do that. So it became a standardized task for us in the Kubernetes structure. This technique allowed us universal adoption. Even non-Kubernetes projects can still be built in Tekton because you could just use container task to the create libraries for example but also you could just, you know, write a Maven YAML or

Maven task or whatever. Also, of course, we had a trade-off. We lost the flexibility. We lost the every team had to do some flexible teams things for the developers and let's be honest, they didn't like that. But what we gained was a massive standardization for companies or whatever we can think of. Okay. Now we have a diagram as you should see the diagram. We have a developer

that commits to the Bitbucket server in our case, you could use any Git server and webhook triggers the event listener. The decides decides the pipeline and path. Like whenever it's a PR pipeline or, you what can I say? A development stage pipeline, production stage pipeline, it's the one decides that. We had logic in the event listener part. Then we had some Tekton triggers and pipeline runs as

we mentioned earlier. I'm not going to repeat. Then the the clone task is first created. Clone task is similar Git clone. And so we went to the build tasks. We have similar build tasks as well as we mentioned container task etc. But what I'm going to talk about the common tasks. This allowed us the universal adoption or you know universal standardization. Common tasks could be extendable or

can be extend extendable for us and we could add any as many as we need for the company. Like Sonar scans or you know some grep or SC checks whatever you you could think of. Also you could add add alert alert management systems, notification systems and something like that. This this was the important for part for us. And the whenever the build is pushed image is pushed

to the harbor registry we were fine. Now we had the image and we will we have we're going to push the image to the somewhere. We needed a CD pipeline. As we will talk us with that. Um As a developer or a DevOps engineer we the helm charts for the deploying our application into the uh Argo CD and the Kubernetes. Uh we uh you push the that

manifest file Kubernetes manifest file to the our Git repository. Uh Argo CD checks here the Git states and if any changes happens there it's automatically syncs the Argo CD application and also the Kubernetes resources uh redeployed with these values. Also we have the image update in our cluster. It's watch our image registry and here it checks automatically the image registry if any changes happen uh or any

version changes happen here. It automatically update the image tag and redeploy the Argo CD application and also redeploy the Kubernetes resources. Um to some to summarize you can see the whole here. Uh firstly we talk about the Tekton part. It's if any changes happen in our repository Tekton pipeline automatically boots and creates the container images. And the Argo CD detects that and deploy in the dev environment.

We use that Git here is the single source of truth. Uh but we didn't just blindly auto deploy to the production or the staging. we use a human approval step. Uh if it's okay for our developers or us uh no writes any manifest file or manually deploy uh anything. It's automatically deployed to production environment. Of course getting there wasn't the a easy challenge. It was not a

technical challenge per se. I mean it was kind of but it was more of a cultural challenge for us. The idea was to create a standardization developers didn't like the way we think. The developer thought that they they were losing the control. They wouldn't change the architecture or the pipeline. They said what if we do something else? Our solution was so shifting the responsibility. The old team

has and now has the authority and the responsibility for the CI pipelines. Of course easy to prove. First we what we did was we created pipelines and we standard standardized the pipelines and uh let's let the developers see that. And whenever they see the CI pipeline is not not failing in any time or even if fails it's the responsibility is ours. They they like that and they

that's how they create or win the That's how they win their heart. Trust. Trust us. You know. Of course we have edge cases sometimes but it's fine for several microservices or a teams for us. All right. We're going to go with the lessons we learned. This was the easy lesson to learn for us. Uh this scheduling trap by default Tekton uses affinity assistant and you know creates

we build our pipelines in same node which is fine for their logic. We are not judging that because workspaces or PVCs are by default the same thing. But in our case we create we disabled the affinity to not don't stall the pipelines or don't wait the pipeline build time. >> [snorts] Uh in the second part I just want to mention the canonical caching problem. Uh we face

a problem caching the image layers and it's we found it's unreliable because it's what we are looking for. What we are doing is that for example in the Maven builds or uh NPM builds we use uh their directory Maven. Uh we use M2 directory in node modules. node js we use node modules directory to cache the uh dependencies and we use here persistent volume claims to uh

this cache mechanism. Uh it's directly build reduces the build time in our um environments. Also we had last but not the last but what we're going to talk about. Last is the common package problem. microservices we have common libraries or common packages which is fine to do. Uh but in our CI cases we they needed to push common wait then push the app to application doesn't fail

due to the newly registered codes. However the developers didn't do that at first and we were trying to fix that by creating more and more and more yamls and it was really you know fragile. However uh we we learned that the communication is better to solve sometimes not just writing more yamls or writing more codes to it. Just communicate. Like with we communicate with teams developer teams

and they communicate with each they themselves and we found our path. It was not the best solution maybe but it was the great solution for us. Sorry. Okay. Uh at the end what we learned and what we thought. We planned for distinct pipelines for every single possible path. We had a lot of pipelines to be honest and we are not we weren't going to create them. However

we built a single unified pipeline as we showed. And we thought that we will be finished someday because you know you should keep moving but you know you have you need to have some things to do. But reality is definitions keeps moving and changing. we have we what we call this the unfinished success for us. The goal is not a perfect state. The goal is a manageable

scalable even changeable path in our structure. All right. We I think we have a time so we're going to show a live demo and it will break I think but it's fine. Okay. >> Uh As we will show you that I think. Okay. Uh we have a local Minikube cluster and kubecon demo in our uh up and running in our cluster. Uh now I will show you

our demo app in our browser. Here you can see it's blue background and in the version two. Uh I just want to change the background and update the version to two but I just copy my ASCII codes. Uh here you can see uh firstly I should show you the uh Tekton and the Argo CD, sorry. Uh let's me check my minicube status. It's up and running right

now. And also I want to show you uh my We just have a simple setup for the demo purpose, of course. You could create more of the things. Uh here you can see our Tekton pipelines and namespaces. We have dashboards also. The Tekton uh CRDs pods are up and running. And also I want to show you the Argo CD parts. Uh here uh we have Argo CD

server and the repo server. And also here you can see Argo CD image updated. You can see it's that right now. Okay. Uh now I just changed the background color to green and update the version to two and uh commit my changes. She doesn't use VS code's commitment. I don't like it. Yeah. >> And there's a typo. Of course. >> Okay. I just push my uh changes

to the um GitHub and we will check the Tekton dashboard here. As you can see, can I No. as you can see here, uh my pipeline is uh that takes the webhook and uh starts immediately. Uh it's fetched the Git resources and now it's building the image. Uh we use here Kaniko builds. I think it's going to be ants uh in a minutes. I don't know. And

after uh the build is uh successful or failed, uh it reports the status to the GitHub and also the uh Teams notification uh step we added. I just want to show you the webhook uh in the uh GitHub in our repository in the settings. We in initialize our webhook here. It's up and running. It's in the successful status right now. I just go to the code And

I am using my test branch. Let me check. Yes, build is successful. And it's will It will trigger to notification tasks. >> Yes. I just want to show you the notification is comes here. Yes, you can see it's success. Uh and uh I want to show you the Argo CD application here. Um >> our Argo CD pods using the 7b3 to uh sha. If uh image updated

detects the changes from our Docker registry, uh it's automatically syncs our cluster and updates uh the new pod, terminates the old one. >> I want to be quick. Can I refresh? I think you can refresh, I think. It's just waiting to sync. Uh okay. All right. It think Maybe uh in that >> It sync. It sync? Yeah. In a minute a minute? Yeah. Oh, yes. The sha

is changed now. Okay. Uh now I want to refresh my browser and voila. >> It's It's a simple app. And now in the notification part, I just show you the Teams notification. Here. Um it's old time, but it's successfully notifies our Teams. Thank you for listening and joining our And I just want to show you our QR. You can scan and connect with us or see the

the demo itself. We have links there. Thank you for listening. >> Uh if you have a question, you can ask us in the microphone, please.