Tutorial: AI on Kubernetes Without the Chaos: Building Reproducible ML Environmen... Nourhan Mohamed

In this talk, Norhan Muhammad discusses the integration of AI with Kubernetes to create a more organized and reproducible machine learning workflow. The speaker, who has extensive experience in virtualization and Kubernetes, highlights the challenges of reproducibility in machine learning, discussing issues like data leakage and the importance of version control. Through the use of tools such as Kubeflow, Argo workflows, and MLflow, the session emphasizes the necessity of tracking experiments, managing pipelines, and maintaining an efficient model training process. The significance of a well-structured pipeline that avoids chaos is underscored, with a focus on validating models before deploying them into production, ensuring that teams can manage resources effectively while minimizing errors.

My name is Norhan Muhammad and I hope today is going to be a very beneficial uh talk on AI on Kubernetes without the chaos because all our lives are about chaos in it. So let's help reduce the chaos and thank you all for being here. So let's start. So this is just a little bit about myself. My name is Norhan like I said and recently and thankfully

I was able to continue my journey towards the golden cubeot. I finished that and I am a devops lead at my beloved company code cloud and uh my expertise is about virtualization. So I started off at Dell as a technical support engineer. I built up my way virtualization, containerization and then Kubernetes uh as a manager of Kubernetes platforms in VMware and then I moved to COC cloud

later. So uh by now I have a good amount of knowledge about Kubernetes and cloudnative platforms and as we are all new uh with AI and ML. So we are struggling everybody's struggling in this environment and in this uh platform. So let's try to explore options for all the problems that we are seeing. So let's dive in. So let me start by a number 300. Take a

look at that number. And 300 is the amount of machine learning papers that detected data leakage affected by reproducibility. So 300 is a big number in the research world. And this uh paper has been published in 2023. But if you have been around today, if you have attended the uh keynote this morning, our friends over in Nvidia were just saying something like that, then they still struggle.

Some people still struggle with reproducibility until now. And we're still researching options into how to improve that, how to build reproducible and reliable machine learning workflows. So today we are going to explore some options into how we can fix this. So what's about the reproducibility problem? What do we mean by reproducibility? So just to make it easy, I'm going to stay away from machine learning for now

and just tell you a small uh story about my friend Bob. So my friend Bob has a sweet tooth. He was sitting in his room and he remembered that last week he ate that cupcake that was amazing at one of the restaurants. And that cupcake, he kept dreaming about it every day. So he said after lunch today, I'm just going to head back to that restaurant and

have that cupcake again. So last week when our friend has gone to that restaurant, Chef A was on duty. Chef A had some ingredients. They mixed it. They put it in the oven. And then after they put it in the oven, they got that lovely orange cupcake. And then today when our friend Bob went to the restaurant, that happened. Chef A wasn't available really. So, Chef B

actually went into the kitchen, used the very same ingredients, which we can see here, but tweaked it a little bit. So, maybe used a little bit eggs, increased the flour a little bit, stuff like that. And then added everything in the oven, but used the pretty lower uh temperature, but ended up getting a nice cupcake, which was green. Our friend Bob got to taste both one week

difference but he doesn't really feel like this thing is the same that he tasted last week although we have used the very same things we have used the inputs used the same machinery but produce different outputs so this is reproducibility but again why would we care about reproducibility especially in machine so in terms of data leakage. Just a little bit about machine learning. Machine learning is that

we have a data set and we'd like to infer uh or knowledge from that data, know some attributes about this data, study this data and then train a model and this model can help us later either double check or fact check stuff that is new that data set new data sets that we get exposed to against the trained model. or we would really need to infer new

stuff based on the information that we really need. So what we do when we are training models is that we get a known data set and we split it. We take a portion this portion is going to be trained by the model and then we hide another portion. This hidden portion is going to be our control. So we are going to test the model afterwards and see

if it actually behaves like we want. But what happens if the model actually peaks on that data? If the model peaks on that data, then our experiment is ruined. The data that we have been gathering, everything that we have been working on is now useless because the model knows the answer. It knows the final answer. So useless. We didn't do anything. It just can get give you

the predictions that you already know about. We would like to eliminate this and to be able to eliminate this we need to have a framework that we're going to work with to make sure that everything is reproducible while ensuring that data leakage does not happen. So again why we would reproduce stuff and why do we have to run multiple experiments? Why do we have to care about

that? Can't we just train a model once and that's it? Actually no. Why is that? Because our aim is not to just run the model and train it once and that's it. Our aim is to get the most efficient model that we have the least cost and the highest performance. And to be able to do that, we would need to feed it different in different inputs and

then get different outputs and then evaluate both and see which run actually yielded the best results so that we can deploy that to production and actually use it later. So as you can see here if we could not have like a list of runs with different parameters and different associated outputs we cannot compare anything. If you have some apples and some oranges you cannot really compare them.

We need something that has to track everything from start to end and then we can compare stuff and make sure that this model that we settled upon is the best. Without this reproducibility, we cannot compare runs and we cannot achie a achieve this. So let's look first on the traditional ML workflow, the one that has been used for a while and the one that is actually causing

problems and see what problems do we have. If we know what problems do we have then we get to enhance uh our processes our workflows and even achieve the results that we want. So to start machine learnings have been when when machine learning have started we have all used those interactive nice Jupyter notebooks. We would have a notebook we would add some inputs and keep running and

getting the nice charts and the nice outputs. But there are some problems with these. So if we can see once you just close your laptop these are gone because they just live in the memory. So I restart my laptop that's gone. I have a colleague at work they restart that tomorrow. There is no way to know if I ran this. If they ran this I don't recall

which parameters I used yesterday. They don't recall which parameters they use today. So that's the first problem. Coming next, the same thing that I was saying, manual tracking. So let's say we are going to fix our issue by having a spreadsheet and maintaining it. So today I'm going to run the model with parameters A, B, and C. And then I'm going to track that into an experiment.

And then comes a day that my colleague decides that he's going to lose the bus today and he would like to go home. So let's just skip doing that today and then after tomorrow somebody else decides that and then after tomorrow somebody else decides that. So we are missing runs and maybe I forget my eyeglasses today and just decide that the 0.5 is 0.05. So there are

errors that happen when humans are responsible for something. Human errors happen. Third thing is have you ever seen that final final v2 before? So this doesn't even happen in models only like in powerpoints in models in anything that we have. We do if we are not using something that is virgined we keep on virgining stuff and appending underscorefinal final endlessly and then we get lost which is

the actual final version. So that's actually a problem because we cannot pinpoint that final version where did it come from and if we cannot pinpoint how did I get that model I cannot actually reproduce it. Next thing, okay, I have the model. Fine. I copy and paste the model on my environment and then I run it. And then because everything does not work as expected, it just

crashes. But I don't have a virgin. I don't have virgining. So what do I do? I don't know. I just freak out. You know, page of beauty, wake up someone at 1:00 a.m. And that's it. They hate me but okay. if I actually want that to be good then I need a version to be able to roll back. If I don't know which model has been deployed

then I cannot roll back. Simple. Last but not least we have all seen the it worked on my laptop issue but here it's actually a major problem. So it worked on dev. No, it worked on staging. Oh, production. Wow. Did we just push to production now? It's not working. Oh my god. So, how do we if these are the problems that we have? How do we actually

work on fixing them? If this is just uh deployed locally even not even on Kubernetes when we are going to deploy that on Kubernetes and start adopting the state-of-the-art things uh that the world have been creating beautifully then this becomes even more chaotic. Why is that? So number one you have virgin drifts. So you try something locally or you try something on dev or you try something

on production and each one has versions packages different OS is different stuff that is different. So I have stuff that you cannot compare because this version produced something that this version is not going to produce. Second thing I have a broken pipeline. I don't have an automated pipeline. So I might run step one then two then three. My colleague can opt for step one then three then

two. My other colleague can opt for step four then five then one. So we don't know what actual steps we took to be able to produce something. All these things that happened uh can be summarized into two main problems. Like I said, environment and version drifts. Packages are not pinned. We don't know which packages we used, which versions we used, which environments we used for each build.

And the most important thing is lineage. What is lineage? Lineage is to be able to pinpoint and trace the life cycle of a workflow from the beginning until the end. from the minute that I entered some parameters, trained a model and then got an output so that I can attach each model to its parameters to its models. So let's imagine the best way the rubric that we

should follow the framework that we should follow in order to actually achieve what we want. So in theory we would want to have our inputs virgined. What do I mean by that? So if I'm going to run run 0 1 I am going to use parameter 1 and two and three then there should be a no a known and uh reliable way of knowing that run 02

runs parameter a b and c instead. So if I have that virgined, saved somewhere, a database, a git repository, then I can track it back. Second thing, the pipeline itself. So whatever I'm going to run has to be virgined. I need to be able to know that this model was produced by this pipeline and this other model was produced by another pipeline. So that even if the

inputs are the same and the outputs are the same I can know that this is can be reproducible but this is not entirely exactly like this because we used another pipeline. Third thing metrics have to be tracked. So I put some parameters into a pipeline and then the these gave me a model like we said model_final_final that thing. So we have to track these artifacts. We have

to save them somewhere that are virgin as well so that I can refer back to the versions and I can roll back if I want if I want. What if I passed in some parameters and then the model that got generated is buggy? It's not good enough. I do not want to push that to production. I need to have an automatic gate that would assess my model

based on criteria and then validate whether this is actually a good model or a bad model. And finally, what happens if I have done all that? Everything is fine. I generate a model, I put it on production, but I go on a sick leave next week. So my colleague takes over and then they don't know what I have done because I just fell sick and didn't show

up today. So they did something else and then they pushed it to production and now I come back from sick leave and I don't know which model is deployed. What happened? So there is nothing that would denote my actual like um desired state that I want to be in the cluster. I don't know what happened when I was out. So we don't we want to avoid that.

So if we have something that audits uh my roll outs on the cluster I can roll back I can know that nobody tampered with my environment that the state I want the cluster is in is actually the state that the cluster is in. So this is the framework this is the framework that we wanted to talk about and have a me have a mental model on. So

up to now we spoke about reproducibility. We spoke about how reproducibility would help with data leakage. We spoke about why we need What uh the team struggled with and the mental model or framework that we would like to implement. Let's go and think about how to abstractly implement that. So basically we would like to go through a defined pipeline. So this pipeline is going to take it

from the start to finish every single time in an automated way. First I'd like to train the model. And to train the model I will be able to track my parameters, track my metrics, log the the artifacts that get produced and log the metrics that are actually produced by this training. And then next I will register that model. Why would I register this model? It's for virgining.

So I would register version one and then version two and then version three and later I can roll back or I can compare them. Third, I'm going to validate that model. Do I want it? Do I not want it? If I don't want it, then I can just skip. Or maybe like we do with for example security enforce enforcements. I just want to audit what metrics are

produced in a certain way. Some quality checks, some audit checks. So this validate can be just there to log that for me so that I can compare if this is the actual correct threshold that I should put later and then I can go into enforcing mode like okay if I get this uh value then just drop don't continue don't build we've reached to the build step now

that we know we love our model we would like to build it so this model is on a tool registered and I would like to package it into a container to be able to deploy it finally on Kubernetes. So I use a container builder. I build this container and then I have to deploy it. Again, like I said, if I just went and deployed it the way

that we roll out deployments on Kubernetes, this is not tracked. So in theory, we need something to be able to continuously monitor our desired state and make sure that this desired state is actually on the I'll take a moment here just look at that pipeline because we are going to implement it in a while. How can we implement that pipeline? This pipeline is a full pipeline that

introduces the concept of MLOps. So in DevOps we are monetary produ we are actually actively producing applications and then deploying them. In ML that's the same. We are training models and then pushing them to production making them live. So it's the same. First I'm going to have my orchestrator. My orchestrator here is Qflow. All the tools that we are going to be dealing with today are open

source and cloudnative. So it's accessible for anyone and it can be scaled to any production. My orchestrator here is going to be Qflow and it's going to do the full uh for my uh pipeline from start to finish. How is it going to be able to do that? It's bundled already with Argu workflows and it's bundled with ML metadata. ML metadata is going to make sure that

everything along the way from the minute I start a pipeline until it goes on production, everything is tracked. Everything is known so I can reproduce that later. And Argo workflows is going to give me a very important thing. It's going to divide my stages like the train model, the register model into containerized uh environments so that each container cannot take a peak on the data from another

container and this will help us minimize or eliminate data leakage or audit it if it happened. Next step we are going to use MLflow. MLflow is a nice uh framework agnostic tool that we can use in order to plot our parameters, our metrics, our registered model and serve that model at the end and be able to compare runs, check uh all everything that we have achieved so

far in terms of machine Third, we are going to use a versioned backend min.io O here but for production of course we can scale that up to something like Amazon S3 or so. Next we are going to use scanno for builds. Uh this helps us build containers into a user space because for docker we need a privileged uh container space and this is not possible in orgo

We are going to keep everything that we need versioned into git. So the model serving deployment that we are going to have, we are going to leave it in git or github or gitlab or whatever software that we aim to use. And last but not least, we are going to have Argo. Argo is our hawk. It monitors our cluster. It sees if the desired state is different

from the actual current state of the cluster. It detects drift. It autoheals and syncs the models and e even if we would like to deploy a model we are going to just add it here in the repository argu is going to pick it up and then it's going to deploy it for us and that's how we achieve a full fully automated pipeline from start to finish that

we can reproduce that involves minimum human interaction. I have spoken for a lot of time. I'm sure that some of you are getting bored. But we are going to get to the interesting part. We are going to see a live demo of whatever I have seen now. And we are going to try and do it together to see it in action. So here I'm just using a

data set, an open source data set. This data set has 39 numeric features and it's a simple data set, a fitness data set. So this fitness data set it represents uh the diet and heart rate and exercise data for a for human beings. It has 20,000 records and it is an unsupervised it's unlabeled and the expected anomaly rate that we expect in this data set is 5%.

So inherently when we train our model we are going to ask it if the output is between 3 and 5% sorry 3 and 8% that then this is acceptable else not. So this is going to be a our validation quality gate. Again this might look very very simple but this is the essence of the demo. It's going to be simple but you can replace its parts because

the framework is the same. the framework is going to always be the same and you can replace tools, you can replace uh the the implementation of each uh stage, but it's going to work the same, the very the architecture. It's a very big diagram. Let's go through it. So, we have our training data here. Our training data is going to be mounted into our five stage pipeline.

So if we can see here in the orchestration layer like we said Qflow is going to be our maestro here. So first we are going to train an isolation forest model and this is just picked again for the simplicity of it for the lack of computation requirements of it but you can change that it's fine. Then we are going to register that model and for both of

these stages we are going to use ML flow like we said. Next comes our validate. We are going to validate the produced model. See if it passes our gate of 3 to 8%. And if yes, then we can proceed. If no, then we can hold. Next, we are going to build the container. Cano is going to build that container and push it to uh an incluster container

registry. Again, let me remind you that this is a demo. We want it to be light for the simple the simplification fact but in a production environment you would opt for something like u GCR or u harbor or stuff like and then we are going to deploy it. How are we going to deploy it? Like we said we do not want to just throw a model in

Kubernetes and not know how this happened. So that deployment is going to go on GIA which is locally hosted on our environment. GIA is going to see okay this is your new desired state the new Well this is this is watched by Argo. Argo is going to detect that you are trying to deploy a new model. It's going to sync it into the model serving Kubernetes deployment.

And that's everything how everything is monitored. ML metadata is watching like like a hawk. It sees what happens from start to finish. Argo workflows is the execution engine and it makes sure that no container would peak on the data of the other container. So that's how we just achieve our goal of having a fully reproducible automatic automatic pipeline for ML flow for ML uh That's a big

architecture overview but in a minute we are going to try to achieve that together. And because this is a very uh comprehensive uh diagram, it has a lot of pieces. It has a lot of components and it's going to be hard to deploy that with all of you on on each on local. So prepared a lab for you. So with the amazing help of my team in

code cloud, we have prepared an environment for you here, a hands-on lab. This is the link. If you have your laptops, please get your laptops out if you'd like to follow along and access this. My team is going to be uh in their CodeCloud t-shirts roaming around the room. If you have a problem, reach out to any of them. They are going to help you. Uh, and

I'll just give you a minute or two to acquaint yourself. Open your laptops, get everything running, and then we're going to start. Uh, I have the um demo on on my um laptop. I'm going to share that and we can work together. So, just giving you two minutes and let's get started. So since we are a lot of people here, if you face any timeouts, if there

is a little bit of slowness, excuse the internet and let's get around in retrying until that works. Okay. So, uh I'm going to put up on the screen the link again just in case somebody missed it. And you should see a sign-in screen. If you're not signed up, if you have never signed up to CodeCloud, this is just going to take a minute or two. So, get

through it and you can start. If you have any questions, either shout out or uh reach out to anybody that is roaming around you in a ColdCloud t-shirt. Let me try and open up the lab with you guys. So for the sake of this demo we are going to run a pipeline together explore all the aspects of everything that I have uh shown you. Um, I'm just

going to check with you guys in the back. If you can see, uh, well, raise your hands. If you need me to crank up the font, don't raise your hand. Okay, is that better? Everybody can see. Thumbs up. Hey, people in the back. A little bit more bigger font, guys. Love me a little bit. Okay. Okay. Everybody ready? Or should I wait for a minute? Slightly larger.

slightly larger. Good. Good. Okay. Okay. So, I'm going to go quite slowly. Uh if you need me to stop for any reason, just wave or tell anybody. Okay. So, just to wake you uh you walk you through the environment that we have. So we have on the right hand side uh some instructions that give you the overview of whatever we have been speaking about this since this

morning. Then we have uh VS code environment on the right hand side. The right hand side is going to be our main environment. It will uh we will use the terminal to be able to deploy and and compile pipelines and stuff like that. And we have some of our um uh files that we're going to use. I'm going to show you around everything slowly and then we'll

have some uh time for questions at the end as well. Again, if you are stuck with anything, uh reach you in a cold cloud t-shirt or if you're near me, just wave. I'll I'll stop. one thing. I'm going to be uh going to the laptop a little bit and coming back because of the interactivity. I'm a little bit short, so if you can't see me, it's fine.

I'm not a ghost. So let's go through the instructions. So that is what is included in the lab. It's a full MLOps pipeline. So train, register, validate, build, deploy. The one that we have just been saying uh git ops with Argo CD to be able to push to get and sync with Argo. We have an incluster git server and we have an incluster container registry and we

have ML metadata embedded in Qflow pipelines. We have Qflow pipelines of course MLflow minio and we have three main files. We have the deploy stack. So I want to show you guys how the deployed stack look like how everything is deployed. How what different components that we have in in here. And then we have a pipeline. The pipeline is already provisioned. You have the code. And by

the way, this lab is going to be available on the same link until next Monday. So if you go home, you'd like to play with it, you'd like to break it, make it, that's going to be staying. So it's not going to go anywhere. last but not least, we have a run KFP. KFP is Q flow pipelines just for short. We have a run uh KFP py

that we are going to use to deploy pipelines on cubeflow. There are two options either CLI or GUI. You can try the CLI. I'll just give you a hint of the the UI just because it's uh more interactive and and good to see. There are some quick instructions here um that are going to guide you with this lab if you're not in this session or if you

decide to see it later. So first thing that we are going to do is copy this and paste it here and let this deploy. So like I said, let me just uh make this a little bit better to So this is going to deploy the infrastructure. It's going to deploy registry, minio, uh mlflow, kfp and githops. And while this is running, this is going to run like

for five minutes or so. So while this is running, let me go and show you how this actually goes because you know, it's going to take time and I don't want to waste your time. So, the first thing it's going to do is uh it's going to deploy a local container registry and then it's going to wait for that registry to be up. It's going to configure

minio. It's going to wait until it's up and then it's going to configure a bucket and this bucket is called ML artifacts. We are going to use this bucket to uh save our modeled uh so sorry our uh versioned models later. So we are going to deploy MLflow. We are going to uh point it to that registry of the the uh M IO and then we are

going to crank up the Qflow pipelines in an actual production environment. All these will be already installed. I'm just showing you the infrastructure just for your information. So Qflow pipelines are going to be uh taking some time. It takes a little bit to to deploy. Once it's deployed, it's going to configure everything. Service accounts, config maps, uh, arbback, everything. And it's going to later provision some persistent

volumes. Again, this is a lab environment. That's why we're using persistent volumes. In a production environment, it's not wise to do that because you have multiple nodes and you don't want something to be rellyant on a node. You can use persistent volumes if it's uh like creating uh an external cloud storage not a local one. For here for the simplicity we're just using host um host based

ones. After that we are just giving some uh role based access control access for cube uh flow pipelines to be able to navigate and manipulate the the cluster. We are aiming to um verify that our gy server is up, our Argo CD is up and then we are going to push a base um image deployment of the model. This doesn't this doesn't have any uh containers running.

So because it doesn't have any containers running u we did not train any models yet. So that's why that's fine. It's it's not going to be a problem. And then we are going to train the model, deploy it and see as the pods recover and the model is deployed. Last but not least, the Argo CD application. So we are going to create an Argo CD application to

monitor our Git repository and may be able to sync Let's go back to our lab. So for some reason Yeah, let's give it a minute or two. So let me just uh recap what I have said while this is running. So for now we have only set up the environment. We have installed all all of our architectural diagram uh components. We have configured them to work together,

but we haven't done anything related to the pipeline yet. So while this is deploying, let me show you the pipeline that we have. So this is a very very long file. I'm not going to go through it all, but let's just showing you around how this looks. So this again because the font is big. Let me just pop that out and show you in a better way.

So this pipeline it has the five stages that we said. First thing oh sorry. So first thing is train and then we have the register and then we have the validate and then we have the push the build and push image and then we have the deploy to the githops. So as per whatever we have discussed now this train model should train the model. It's easy. And

then register model should actually version and save my model into an artifact that we can refer to with versions after afterwards. Validate is going to be uh the validation gate the automatic checks that we are going to impose on a model. Build and push are going Kiko is going to build us our container and push it to the incluster uh container registry and then deploy model is

going to deploy it to git and Argo is going to sync it. So we are not going to push it directly to Kubernetes but rather we are going to deploy to git. This is going to be picked up and reconciled by Argo. Let's check one of these. So to start with we have something called the DSL component. This is I'm not going to go into details of

that because this is Python and and details but just for you to get a rough idea. So the DSL component is going to uh install a container when Argo workflow uh cranks this stage up. So I'm going to spawn or Argo Workflows is going to spawn a container with these packages installed. And as you can see, each package is pinned to a version. So if I run

this today, if my friend run this tomorrow, MLflow is always going to be installed by version 2.17.2, which makes sure that we are not having any environment drifts when we are not having any package drifts. And then we get here defrain model. This is our stage, our first stage, the train model. And we have some parameters. And these parameters are going to be processed somehow. I'm not

going to go into the the code, but you can uh check it on your laptops. Uh and then it's going to output something that we might use as an input for a future stage. Let's go back and see if infrastructure has been deployed successfully. Great. So this is telling you that the infrastructure deployment has completed. The services available are these services. And then the next steps are

the following next steps. Before we go further, I just want you to notice these links above. So these links above are our tools KFP, uh, Gitia, MLflow and Argo CD. You can just click on that and it's going to open the relevant window uh so that you can see the UI of the tool. They are exposed via note ports on production. Basically, you're not going to use

note ports of course, maybe load balancers or stuff, but for the sake of simplicity, we're using note ports here. Uh, if you scroll down on the left hand side, you can see credentials if you need credentials for any tool as well. And let's go to our next step. Now, we have the pipeline. We are going to compile it. a compile pipeline uh command just Python 3 and

compiling. Let's paste that here. And before I paste that here, I want to show you something. So on the left hand side here, we don't have any YAML files, right? We just have a fulfill pipeline py. And cubeflow pipelines do not work with Python. They work with YAML. So once I compile this, it is going to tell me that it's compiled successfully and it's going to generate

me the AML file that I'm later going to use in Qflu pipelines. As I said, you can follow along with me, but as I said, uh you can use u CLI or UI. CLI option is available here. So I can just take this command and run it here. But for the sake of uh visuals, let's see how this work with we with UI. I'm going to right

click on full pipeline.l and I'm going to download this file locally. And then I'm going to click on Qflow UI. This is the UI for Qflow. So on the left hand side I have pipelines, experiments, runs, stuff like that. So I'm just going to click on upload pipeline there and call it KFP and then upload the file that I just downloaded and you would see that this

is a full pipeline six because I have downloaded this a lot of times and then I can click create. Once I click create you can see this uh lovely diagram. So this lovely diagram in Argo workflows is called a DAG if you know anything about Argo workflows. So DAG is directed as cyclic graph. Directed asyclic graph means that every step is going to have a relation with

another step. So for example here as we can see this train model is going to give an output to the register step. It's going to give another output to the model step here. And then this register model out step is going to output something that is going to go into the model the validate model. This validate model is going to either succeed or not succeed but it's

not going to give any outputs. And then once this succeeds, the build and push is going to be cranked up. And then this outputs the image that it generated to the deploy model git ops uh pipeline. And that is going to give my image into the u desired state of the cluster which is git. Argo cd is going to pick that up and sync it to the

Let's go and create a run. So up top you have the create run. Click on create run you would see here all the run par. Let me crank up the uh font. So you would see here the run parameters. So all these are things that we can modify. We can change the u gate um anomaly max rate, anomaly min rate for the validation later. we can uh

change passwords and usernames. Of course, in a production environment, we are not going to have passwords like that. This is uh not good. We are going to use Kubernetes secrets in a different way. But just for the sake of simplicity and you seeing that it's in plain text here. Contamination is something that we can uh pass as a parameter to a model to see u and vary

different contamination rates and see if the anomaly rate is going to be cranked up or down. uh the data path, our fitness data and some more parameters. You can see here something called the random state. Random state is used in machine learning just to make sure that every time I'm going to give these inputs with this uh specific seed, the output is going to be the same.

So it ensures reproducibility in a machine learning sense of u thing. So I'm not going to change anything. I think I'm just going to go with the defaults. If you would like to change anything, feel free to, but don't change passwords because this is going to break them. And let's start this. So, this is going to give uh to move us to the run tab. And the

run tab is just an instantiation of whatever we had. So we had the pipeline, we have the rubric that we have and then we instantiate uh an instance from it and we pass some certain parameters to it and it's going to pass these values throughout everything and show us the uh actual output. This is going to take like two to three minutes. So while this is running,

I have prepared some um run that already finished and let's explore it while this is happening. So after the whole pipeline is finished you are going to see so up top here beside the pipeline uh name you are going to see a small uh dot this dot can be blue which is indicating that this is running. It can be like a clock and this clock is like

it's scheduling it. It hasn't started yet but it's scheduling or it can be green uh the pipeline has successfully completed or it can be red. So this pipeline has failed for some reason. And then for each stage you are going to see this uh green tick if the pipeline is finished and successful. It can be like uh let's see it in the other instance. So it can

be like this uh loading thing if the stage is still running at the moment. And the beauty of uh Qflow pipelines is that you can just click on and you can see all the parameters that I entered. So these are the parameters that I entered the contamination rate, the uh N estimators rate, the random seed, everything. And then I can see an output here. This output is

going to be something called a run ID that I'm going to feed later into the register to make sure that this is the run that I would like to register later. So this model is going to be used for registering and I can click on logs here and see exactly what input parameters did I have, what outputs did I see, what are the artifacts, what is the

generated model, everything. So we have everything logged, everything is tracked and this UI is all because of the beauty of ML metadata that is embedded inside of Qflow pipelines. And each one of these run in a separate container like I said. So the train model is running in a container. The register is in another container and every single one of these is in another container. This folder

that you can see here is dropping all the models versioned onto Minio. And this can be a three, like I said, it can be a three bucket. It can be anything that can version models. So let's just take a look at the inputs and outputs. I don't want to worry you with a lot of logs, but you can have a look on your on your side. So

the train model is going to generate the run ID. Like we said, this run ID we are going to feed into the register. So if I look here, it's the same run ID. So it started with CAA, it's still starting with CA. So the register knows that this run that had the run CAA whatever is going is having a model as an output and I need to

register this and it's going a number. This number is the version of the model. So first it would generate v_sub_1, second v2, third v3, stuff like Now I'm going to validate this model. So as you can see here, this model failed. It has an anomaly rate of 50% and this is outside 3 to 8. So this is not what we want. We want it to crash. We

can want it to be like that in case we are just running to make sure that we log everything and later on decide on the threshold that we want. So for that sake I didn't make it crash but we can see in a bit how to make it crash. And then you have the build the build image the build image took the model name. It took the

registry that it's going to pull to push to it and it's it's going to return a URL for that image that it's going to do. So I took a version two model and I generated model serve colon v2. So that's the new image that I'm going to push to my registry. And this we are going to use to update our uh git ops. So we are going

to push these things. We are going to push that git ops and we're going to see that in a bit. So we're going to open all the UIs and see that in a bit. Is that fine? Does anybody have any questions so far? All good. Okay. So let's go and see that pipeline that we have run. Sometimes you need to reset or restart the the pipeline or

refresh or wait for it to refresh because it's a little bit not real time. So if you click on something and you see error, it cannot retrieve the pod logs, just give it a refresh or wait for a few seconds, it's going to uh give you something. So this is still, like I said, this is this takes a little bit. So this is still on the image

and push. So, uh let's go back to my pre uh run. And if we go here, we can go to runs, sorry, runs. And we can choose two runs for example, and we can just click compare runs. So compare runs can actually tell me that on the first run I used a contamination version a contamination uh of 0.5 which is different from the default value. So it

keeps track of the default values. It does not output automatically all the default values. So the default values are staying default and it just gives you an uh idea of what changed here. So on this run I changed 0.5 for the contamination. For this run I changed a 200. And like I told you from a bit we can actually use the validation gate as a make it

or break it. So what I did here for the third run is I added these lines. Let me uh get you the code. So I added these lines here in the If we scroll down back then these lines were not there. So I just told it if the decision is failed just raise an error and that breaks the the model the pipeline at this stage and it

does not advance to the build stage. So when I compile and run this you are going to see something like uh sorry you're going to see something like Again, give it a second. Validate model. And I can see the logs for that. And this is an error. The validation failed because the anomaly rate is outside the range of 3 to 8%. And it's not going to deploy

this model. It's just going to stop there. So for now, we have explored Qflow pipeline. Let's check the other um uh views. So Qflow pipeline like we said it's our Maestro. It orchestrates everything. What if I would like to compare uh the experiments that I have run using MLflow. So let's open MLflow. MLflow is here and we can just go uh to the model training tab. So

if we can just click on the model training tab. Ah and then select that cube con demo that I have experiment that I have set because I have ran this in in uh in the background. I I started that before the demo. I have two runs not just one. So we can click on the um checkboxes here and then ask it can you compare these runs. So

it gives us some metrics. It gives us the parameters the metrics that we are going to compare with. It gives us like a nice diagram here and you can check uh the run details. So the run ids of each CAA is the one that I showed you uh from a bit. And then the run time, the run name uh this gives it a funny name that is

autogenerated. This is not me. And then the start time, when did this run got executed exactly? And then the end time, when did this run end exactly? And then how much time did it take? So it took three minutes and a half here. And then you can go down you can see parameters and you can see I ran here a contamination of 0.5 here and 0.05. So

this is the one that failed later the one that I uh crashed the validation on purpose for the N estimators changed here the end features changed here and you don't have to uh worry about the numbers here. So these are just for demonstration purposes and these are going to be relevant per uh data set. So this is just for demonstration. We are not focusing on machine learning

here. We are focusing on the actual framework metrics. The same anomaly rate here is 0.5 which is 50%. And anomaly rate here is 0.05 which is 5%. This is the accepted model that we have. Artifacts is going to shave to show you all that in models we can uh view the models generated by a run. So I can click on the run and then I can move

to model registered here. It's going to open my model registry. that this is the anomaly detector uh model. I'm keeping the name the same but I'm changing the versions. So this is version three. The last one that I registered is version three. And I can see uh previous models. I can roll back. I can do that. So that's for MLflow. uh Git Ops. So let's check Gitia.

As I told you, if you scroll down there, you can see the credentials for logging in. this pipeline that we have been running is finished. So this one that we have just cranked up a little bit uh later um from a little bit it finished. We can check what happened here. Let's go to the lab and click on get here. U the first time you open this

it's not going to be logged in. So login the username and password are MLOps and admin 123 again if you can see here this is a created repository that was created 19 minutes ago. This is when I launched the lab with you. So this is an initial commit that happens or every time you crank up the lab. So this created a directory a repository sorry and then

later if you can see that from four minutes ago it deployed model version one and if we can see that we can just go to the repository we can go to the model serve deployment. So this is the deployment that we are using to uh serve the model. I can click on this. Let me just close this. I can click on this and I can do a

history for the file and I would end up seeing two versions two sorry commits. The first commit is the initial commit lab startup and the second one was done by cubeflow pipelines the one that happened when we ran the pipeline and it deployed model version one. If we can check here what happened what changed. So this added some annotations. It said that this is deployed at this

time stamp. It was deployed by Qflow pipelines and it has model version one and it changed the registry down. So the new image is going to be local host is because the registry is deployed local host. This is going to be your production registry if this is in So now I need to go to Argo and I need to login again. The login are in the lab,

but it's admin. Admin 123. If I clicked on that, it's going to show me that I have a synced status. So, my application is healthy, my model is healthy, everything is healthy. The sync is okay. And this succeeded 6 minutes ago. Again, this is when the pipeline has finished. And to make sure that this is that, if you see the last uh author, it's Qflow pipeline and

it's the same commit that we saw in git, it's deploy model version one. And if we can see here, this is my deployment. It has two pods. And the two pods are up and running. And we can verify that from the CLI. I can just uh come here and click uh cubectl get pods. The models are served into uh the default name space. So we can see

here two pods like we saw in argu and these two pods are in the running state. So what if I want to query that? So I would like to query the model and see if it gives me a prediction or not to make sure that this is actually working. So if you scroll down here you can see that there is a command that I gave in the

lab. This command invokes uh it give a row to the model and it queries whether this is anomalous data or not. If it's anomalous it gives me a negative one. If it's not it gives me on. So if we can run this here again the sample data is given in a file. If you can check the VS code but this sample request.json JSON that I'm giving should

actually be a good data. So just to give you an idea again on that, it's the data for a 35 year old healthy person that eats fine, that has a good training uh So there you go. You have predictions equal one. So now let's just recap everything that happened. We went from giving parameters to training the model to reproducing everything without any human inter human human interaction.

We have uh succeeded to be able to achieve all these steps in a very automatic fashion and we can reproduce this. So you can on your laptops you can run another run with the same parameters and see if that model is actually going to be reproduced or not. So let's go back to our slides because I to give you some key takeaways as we are approaching the

end of this talk. So five key takeaways from this talk is if we would like to achieve a fully automated reproducible machine workflow we are going to follow these five rubrics. First one is to pin everything. As we saw all the packages, everything the environment everything is pinned to a certain version and every time you're going to run your pipeline, this version is going to deploy to

deploy. Second thing, track lineage. So the lineage is the most important thing for you. ML metadata achieves that for us. You know what goes in, what happens and what comes out every single time. You are auditing everything. Git ops. This makes sure that nobody messes up with our production or our environments. This makes sure that our desired state, our wanted state is the one that always runs

in production. Validate before you ship. Make sure that your models are not faulty. Make sure that you run tests, extensive tests, and break before if if that is not what you want, break before deploying it to production. Finally, immutable artifacts. Make sure that if you are using git, if you are using virgined S3 like storage, everything is going to be the same. It's as if you're writing

with a pen, not a pencil. Nothing can change. Everything remains the same. Everything is the same every time. And that's guys how we achieve a fully reproducible uh workflow using Qflow and using the help of Argo workflows and ML metadata behind the scenes. Uh again this is Nam Muhammad. Thank you so much for being here. Uh I am very grateful for you as an audience. And last

but not least, some questions. These uh material are all available in the GitHub uh repo. I'm going to upload the presentation to uh the SKED uh QR. Please leave me a feedback. I'd like to improve on on everything. And if you have any question, please we have some time. So shout out and we can go ahead. Yeah. Thank you. Anything? Sorry. Yes. Yeah, sure. >> Yeah. >>

Thanks. >> Yeah, thank you very much. It was super nice. Um, I was asking myself what you need on the cluster side. So you you have the all the installations I show I saw quickly for cube flow of course argo you showed what else comes into it that that'll be cool to to get answered and just quickly a second one also for MLflow for example like in

our case what we intend to do is host it also centrally and I guess yeah Argo is also the same all these things this would also fit in in the same category right we could do the same Thank you for your question. So for your first question, uh I am using a very basic Kubernetes deployment in the background. So everything that we deployed is in the deploy

script. Nothing is different. It's just uh Kubernetes uh version 1.34 uh in the back end and then we deploy everything in the script. So nothing is hidden, everything is there. Uh for your second question uh like you said if we centralize everything we are having everything passed as a parameter or embedding everything as in the pipeline. If you go through the pipeline you would be able to

either tweak the parameters to be able to add your external uh repository for example. So we're having here an in cluster docker repository. If I have harbor then I have a link I can change that. If I have an externally hosted argu I can change that. So this is going to roll over for anything that is centrally hosted. We are just using uh a mini a mini

version for our demo just for the sake of it being light. Yeah. Did that answer your question? Yeah. >> Yes. >> Hi, good afternoon. Um, my name is Winn and I saw your presentation and I think it was a a very very good actually excellent uh analogy with the um uh the comparison with a a chef that is preparing a certain meal. Um, and I think that

was that was very clear to to a lot of people. So, thank you very much for that. Um, what >> Thank you. >> You're welcome. Um and and and what what uh strike me was um that you said well there was a another day um I think it was Bob or Rob that >> it was Bob >> um and then there was another chef but then it

was very clear to me that you said well he used the same oven and um there I think Um um well perhaps when you're when you're using Kubernetes um that might be true but in other cases um uh it it is also very important to use exactly the same hardware. Um >> definitely yeah >> and there could have been another oven >> of course. Yeah. So even

in Kubernetes you might have different nodes with different GPUs and this is scheduled on that node this is scheduled on that node. So it's it's important to keep track where you are actually scheduling your workloads which is going to be given to us here by the lineage as well from ML metadata. So, so I was I was just wondering why you haven't touched upon um that part

of the reproducibility aspect of uh also perhaps drifting or deviating away from uh certain hardware aspects and that could also result in a different uh uh result when you're keeping all the other aspects uh constant. >> Uh yeah. So just to reiterate uh in my example I just said that they use the different temperature but it it applies the same. So if you use the different oven

if you use the different temperature this all comes into the pinned versions. So or the pinned hardware or the different hardware that you that you can use. So your point is very very valid. Uh that's true. If we are running on different hardwares we need to be able to actually track this and and make sure that it's the same in each time to be able to compare

both runs. and I'm a very very novice uh person when it comes to Kubernetes but uh what is there in place to actually make sure that that aspect is also reproducible. So uh first off you can uh choose to run your workloads on a certain node. So you can enforce that you can use tain and tolerations to be able to know that all these workloads are going

to be scheduled on for example these GPU nodes. All the others are going to be scheduled on other GPU nodes. So you have some Kubernetes specific uh configuration that you can have in place to make sure that each workload is going to be scheduled somewhere in in general or you can go full separation and schedule each workload on different clusters. So you have a cluster with high

CPU and high GPU. You have a cluster with medium. You have a cluster with low. So both works. >> Okay. Thank you very much. >> You're welcome. Thank you. Questions. Did you guys enjoy the lab? >> All good. >> Hello. >> Wave to me. Love me guys. >> Sorry. Also one question for me. Uh thank you so much for the presentation >> um so I'm not really

machine learning engineer but it was very interesting to see the whole uh the whole flow but uh I was wondering how relevant would this be if you would be using the same model but you would be comparing two different data sets together. uh well I wouldn't advise comparing different data sets together because you would generate the same thing for the same data set as as you see

I didn't change the data set at all but if you change the data set uh your output is going to be different but maybe this is the use case for your company so maybe uh we get uh output from for example uh Garmin watches in comparison to Apple watches in comparison to something else so maybe you are going to gather that and see the mean uh varian

or anomaly rate in Garmin versus Apple versus that. So, this can be a use case depending on the use case that you want to implement for your business. >> thank you. >> Yeah. Um, >> thank you for the presentation. It was really interesting. uh we are also using Qflow in our company and I wanted to know how can we configure the pipeline to use the GPU for

LM fine tuning for example >> so you might not want to configure the actual uh Qflow to that but you can enforce tens and tolerations you can also use node selectors in the in the pipeline I'm not proficient in Qflow But we can check this together after the the call and see uh what's the exact Python scripture that we need to use to to enforce this. But

this is definitely something that we can uh enforce to make sure that these kind of uh containers run on a certain note selector. >> Any other question? Any other questions? Counting down three. Thank you all. It has been my pleasure being with you here. Thank you so much for your time and I hope that it was an interesting presentation. Thank you so