Pay Less for More: A Practitioner's Playbook for Kubern... Malgorzata Widelicka & Lukasz Ogrodowczyk

This talk introduces best practices for optimizing Kubernetes infrastructure in a highly regulated pharma environment. The speakers, who are DevOps specialists at Roche, share insights from their experience using tools like Carpenter for autoscaling and cost reduction. They discuss challenges such as financial waste from idle resources and stability issues due to incorrect resource allocation. By implementing node pools, proper request limits, and event-based scaling, they demonstrate significant improvements in efficiency and cost, achieving up to 90% reduction in cloud expenses. Additionally, the talk emphasizes the importance of observability in managing cloud infrastructure effectively, highlighting the balance between optimizing costs and ensuring compliance with regulatory practices.

My name is Ma Gojata and here is my colleague Gash. >> Hello Kipola. >> We both work at Rush. Uh we are a DevOps there and I am specialized in building cloud platform at scale. >> I've been in IT for 18 years in Rush. I work in pharma R&D division. So it's very high highly regulated environment. Um this is my first CubeCon and today together with Mosata

we will tell you about the playbook we created based on our experience with Kubernetes and autoscaling. Uh we are from Rush. ROS is a global pharma company with over 100,000 employees worldwide. It was founded in Basel 130 years ago and uh our company believes that cutting edge science and cutting edge technology can and will improve humans health. >> We both we both work at different project. I

am working in envelope platform where we are supporting uh data scientists in a drug discovery process. And to also provide you some numbers um monthly we are generate generating roughly 150 million of predictions and also we are retraining 300 models. >> Uh my platform is called ocean. It is for ROS data scientist. It is data and analytics platform. It is for GXP and nonGXP clinical trials. So

for those of you who are not in pharma, GXP means good practice. It is set of rules and regulations to ensure that clinical data is safe for it. It means that every change should be tested, should be documented and should be traceable. And to give you insights about the scale, we are processing about half pabytes of clinical data and metadata. My team is responsible for the cloud

infrastructure and qualification tests. Uh so both those platforms contribute to the drug development journey. >> If you can think about what will happen before the drug will be introduced to the market. So there is a first discovery process. So this is something that I'm supporting. And then clinical trials. So the step that Wukash is >> Yeah. So we are here to tell you about the issues we

had with the infrastructure setup. First pain point is about financial waste. Um paying for readiness instead of executions at and keeping infrastructure ready just in case is waste of money. The second point is about stability failures frequent out of memory issues um because of lack of proper settings for request and limits. >> Yeah. You may also experience lack of flexibil flexibility. If you would like to introduce

for example spot instances, it can be troublesome or require some additional configuration or for example you may reach some quotas. If you are uh working in AWS, we were reaching the security group quotas. And then if you are uh working in multicluster environments, something that is usually troublesome is the maintenance uh upgrade of the Kubernetes also require a lot of manual work. >> Yeah. So when you

think about the change in the highly regulated environment, usually there are two words. Word of DevOps with a mindset to deliver fast and frequently. Worth of automations. On the other side, there is a compliance um where every change is considered as a risk. Um this word guarantees um quality and stability. >> Yeah. And then you can think about that DevOps is one on one side of the

canel and compliance is on the second and then the tools that CNCF is providing is the bridge that you can use to connect them. >> Exactly. So before we jump into how to of configurations let's look at this example of heavy workloads. Heavy workload in our case it is backup job. Backup job of one of our database we hold in the cluster. uh we started with best

effort quality of service without limits set. So the workload wasn't heavy from the beginning but when we reached the limits of the node it started to be a noise enable it caused issues also with other applications running on the same cluster. So settings for of proper request and limits is crucial. Our decision also was to isolate this heavy workload to a dedicated node. Uh we started managing

node groups but the backup job is executed only once per day. So the node is was was um busy for one hour and was idle for the rest of the time. Waste of money. So we moved further and now we provision the infrastructure just for the uh job execution and we scale to zero between those executions to save the money and for us heavy workloads are good

candidates for cluster autoscaling and for that we use carpenter um open source project under CNCF umbrella and during the presentation we will mention about some carpenter concept the first is node poolool which is the intent of the infrastructure. >> Yeah. And if you will think about how to configuring uh the node poolool you can think about there should be some separate note pools for example for the

critical workflows for the heavy workloads or for the ones that have some specific requirements. >> Yeah. So when you assume that the pot the job should be accomplished within given amount of time you can set expire after parameter. This way you ensure that stay pots will not keep the infrastructure up and running for a long time. Default value of this parameter is 30 days. Um we have

expire after set to 12 or 24 hours depending on the use case. >> Um and here also depending on the requirements on your workloads you can set different instance family types. You can set many of them. Just think about that if you will set too many probably it will increase the node churn. If you don't know what node turn is we'll cover that later. So stay in

tune. Yeah. And one instance type is also okay for very specific well-defined use case for critical workloads. And for critical workloads probably the first choice uh will be on demand instance type. It guarantees stability. uh from version 1.7 carpenter natively support uh reserved capacity and take and takes from this pool in the first order um for cost savings uh when using on demand instances you can consider

AWS savings plans >> yeah and if you have some fault tolerant workloads you can consider also to use spot instances and here Carpenter also provides some additional configurations that you can enable and then you will be notified 2 minutes before your instance will be terminated. So this is quite useful. And then generally maybe you have a use case that you would like to have some of the

workloads uh which are uh running on demand and some of them on the spot. This is also possible. You can use note weights and set priority for a different note type. So generally the higher the weight the higher priority. And here you can also say so for the specific note pool I would like to have a static number of ondemand instances and then variable one for the

spot one. And this is also a good setting if you are thinking about maybe you would like to prioritize some of the instances even if there are more a little bit more costly or sometimes not available but then there are much more efficient for your use case. Consolidation is um engine of carpenter engine for cost optimization. Carpenter continuously observe the cluster and try to optimize by beam

packing. So trying to move pots to a cheaper node or by removing nodes and which nodes uh those which are underutilized or empty. That's the default consolidation policy. But you can change to when empty. Uh this way you ensure that the pot which was started on a given node will be processed on this note and will finish on the same note. It will not be evicted even

the note is underutilized. >> Yeah. And here similar uh results you may achieve if you will use do not disrupt at notation on the pod level. And that's something that you would like to use either when empty or do not disrupt notation for some long long running workloads. It could be a training which require GPU. >> Um spinning up a new infrastructure takes some time. So uh

Carpenter um doesn't have to terminate the note just after it's empty. You can set consolidate after to couple of seconds or minutes. And this way you ensure that node will wait for potential new pots, new workloads that will come to the >> Yeah. And then just be careful because if this time will be too long, it may happen especially if you have mixed workloads that underutilized nodes

will keep receiving some of the pots and will be the last one to consolidate. >> Yeah. So uh Carpenter also respects pot disruption budgets set on the pots >> Yeah. But you can also set on the note pool levels that the disruptions uh the budgets in disruption section. And here you can um specify many of the conditions generally the one that is most restrictive will be applied.

And if you are setting this you can think about just what is are the specific uh requirements for your work workloads. And a good example is for example if you are serving the infrastructure for the users then maybe you would like to disable all of the disruptions during the working hours. So we've provided some configuration examples. Um but how to prove that the configuration works as expected

or how to even improve that? Uh let's look at observability. Uh this is uh my favorite panel when it comes to cost optimization. Capacity versus utilization. Capacity is how much you pay for the cloud infrastructure. Utilization can be your memory, CPU and basically the whole space between utilization and capacity is waste. So we would like to minimalize it. On the left you can see example of static

node groups with constants capacity and huge waste. on the right autoscaling with carpenter where when uh capacity is provisioned uh for execution and there is scaling to zero to save the money so can the waste be like zero I would say it's not possible golden ratio between utilization capacity says about 90% in the industry >> and here you can also take other measures to see if you

are succeeding with your infrastructure scaling configuration. Uh so here I'm presenting the difference between cluster autoscaler and the carpenter and you can see the duration of the pots being in the pending state for one of the our use case for one month and uh for one of the environment you can see this is a statistics for around 20k of the pots. So I guess pretty good one

and um what what was the difference we we discovered then that in our previous solution for the cluster autoscaler uh most of the pots so 99% of the pots were ready um a little bit below in the time a little bit below 5 minutes where for the carpenter it this time was reduced significantly to uh to one minute and few seconds and this is also a good

measure if you are thinking about how to satisfy your customer needs and uh user experience. >> Yeah. So also we can check what contributes to the pot pending. That's the real example from one of our application applications. Um on the right what you can see is called startup time 5 minutes. Is it good? Is it good enough? We can we can check further. We can check uh

operator not claim metric. Note claim is request for a note. So operator not claim provides information about different phases of the infrastructure provisioning. So looking at the left those yellow dots at the bottom two seconds um after two seconds in um the instance behind the note is launch. So Carpenter is really fast but then there is a time to boot from EMI to execute user data. So

the node is registered after 2 minutes. Then there is a time for CNI and uh cublet to interconnect the node to the cluster. So two and a half minutes and you have a node ready. Then there is a time for application to pull the docker image to start the container to initialize the application. yet another two and a half minutes. >> Is there any option in your

case to reduce this time even further? >> Yeah. And we did um by changes at the application and infrastructure levels we reduce bot startup time by 50%. Um at the application uh what we did we shrink the docker image we optimized the code of the application at the infrastructure. We updated the Kubernetes um upgraded uh Carpenter to 1.7 and we used now the optimize CMI that gives

us some savings at the infrastructure one minute and the rest at the application level. >> Here we would like to also presenting the present to the cost reduction. Um this is mainly the cost reduction related with the instances cost but in general that's the main contributor to our overall cloud cost and in both our cases we were able to reduce the cost 90% since th those cases

are very different for the envelopes platform we were able to say that the main contributor was the right sizing of the carpenter mechanism. >> Yeah. So it was middle of the last year when we stopped managing node groups and we moved towards carpenter. So the key factor for us for uh cost reduction was to stop paying for readiness and start paying for executions. Okay. Um so cost

optimization is not always so colorful. Now a few things that uh require special attention. The first is hidden cost of Carpenter metrics. After we enabled all carpenter metrics, we realized that the bill for the observability platform increased by several,000. And why? Carpenter uses some metrics internally. Um, it tracks nine over 900 types of instances, their availability and cost. And Prometheus add yet another labels like pot names,

note names and it creates combinations. It creates time series and there is a cost related to each time series in the observability platforms like data dog or um graphana cloud and the issue is called matrix cardality. Fortunately, there are solutions um available solutions to be applied like metric relatrix aggregation. But the key message from here is that observability is not for free. >> And then coming back

to the node churn, one thing is that you may experience node turnurn and then the second thing is how to really observe it. So generally node turn is a frequent creation and deletion of the nodes. It may disrupt your processes. So this is something that it's not expected and generally you may observe it just by taking a look what are the numbers of node events uh versus

time and if there are some occasional spikes probably you are a healthy scal scaling configuration. But if there is more noise or more frequent spikes, this is something that I um I recommend to take a deeper look into. And Carpenter also provides a good metrics for that. So there is a disruption note claim metrics where you can see different type of disruptions for example underutilized, empty, drifted

and so on. And generally if um the main contribution there is uh underutilized which the type underutilized is the uh main reason of the most of the uh disruptions probably your settings uh for the consolidation are too aggressive. >> We've mentioned already that settings um something crucial and for that we use goldilocks. It is CNCF project. Um it is based on VPA but with auto mode disabled.

So when deployed to the cluster it doesn't change anything but can provide useful recommendations um about both guaranteed and bar stable quality of service. >> Um you may think about that this is something like on the beginner level but generally why is it important? So carpenter use those three requests which are set on the pot level to calculate uh utilization. So if you are filled with incorrect

data then you may happen that there will be not enough resources for you in the nodes or still you will be paying for something that you aren't using. Yeah. So that was a part about infrastructure scaling but we thought that generally if you are thinking about uh introducing any scaling option there probably it should be a result of the your application logic. So here we would like

to also present some of the examples. Generally two years ago there was a Nobel Prize uh in chemistry awarded for alphafold model. So um that's a model that can predict the shape of the proteins and that's pretty nice thing. At rush we are also uh calculating similar things. ROS is a part of uh open fold consortium. We are supporting open fold model which is open source and

um when we start introdu introducing it to our platform we were thinking what will be the input for this model. So uh the input is provided by the user and we are using chemical sequence there. So we thought maybe we can scale our infrastructure based on the event whenever this input is provided. So we choose to use SQS with the messages and then uh ketta as a

event based uh approach and scale uh the stateful set where the calculations are performed. Then unfortunately or fortunately there was a requirement that for the specific uh calculation a huge data database is required. So uh we thought that we will attach the snapshot and uh create a volume on the fly using fast snapshot restore with this database uh whenever it's needed. Generally introducing this whole um infrastructure

and also adjusting our application logic a little bit comes up with 100% of the speed up of the calculation and also reduce the cost 10 times and here also if you are thinking about how you would like to use uh the infrastructure uh event based scaling I would recommend to start about what is your current cold start and what are the main contributors there probably your uh

application logic will be a pretty good one uh to optimize in the first place. So generally uh you can think about that one and then also what is the requirements for your use case and what's a good cold start uh to really provide this infrastructure and then the second thing is of course to choose the proper metrics a scalable one. So you may think about what will

happen if there will be any delays on a big scale in production with that metric. And the last one is of course we are scaling in but after some time there is a scaling out event. So then um the after some time it's a keyword here you need to think about what will be a proper time for your use case to really safely disrupt the the infrastructure

at that point that's also quite crucial >> uh in our application for data ingestion and acquisition uh we moved the logic upstack to deal with variable clinical data sizes. You can imagine that clinical data can be a single kilobyte file or very large gigabyte data set. Um so we don't use keta but we have a custom solution. It is based on key value with clinical study profiles

and there is disparture to decide whether the process should be started on a standard node or high compute node should be provisioned by Um when we moved from static node groups towards carpenter we needed to accept this tradeoff of uh latency 2 and a half minutes but that's fully acceptable for long running processing. Uh as a bonus uh with the coupled um application logic from infrastructure provisioning

and configuration. So this way we can adjust clinical study profiles without releasing um new of new version of the >> and on the end of the day unfortunately sometimes there is like more things than playing in with infrastructure and sometimes you need to provide some information about the value that the things you would like to introduce will bring. So one one thing is of course to provide

some cost reduction uh metrics or other PHOPS metrics that you you can think of and the second thing is really about how to prove the value of the changes and here of course we can say Dora metrics but like what in our case uh we can think about in in those metrics that we we can provide on. So the first one is generally about the when you

introducing the changes you should probably think that uh provisioning or upgrading your infrastructure shouldn't be a bottleneck for those changes to be introduced right and also from the user experience how every failure is um is uh is affecting that right so from that perspective uh if you have a proper settings for the drift configuration what will happen if your instance won't be available or uh whenever there

is a spot instance interruption. >> Uh so looking at technical takeaways your infrastructure is driven by applications by your workloads but you can control the life cycle of the infrastructure your nodes uh by settings of the node pools. So like things like consolidation policies and budget settings. >> Yeah. And just remember to customize your configuration based on your workloads needs. If you don't know where to start,

start with the extremes. Then you can also dig uh into the bottlenecks. It could be your AMI, it could be the image size, it could be >> Yeah, observability is not for free, but that's amazing how much insights you can get from Carpenter metrics. So observe the cluster both infrastructure and application and don't lose your phenoms mindset. pay for executions, not for readiness. And that's a great

moment to take a photo of us, but with a nice background, a set of uh metrics we use on a daily basis, but also we used to prepare slides for this presentation. And with that, we would like to say thank you. Thank you for joining this session. >> Thank you. Okay. And if there will be any questions, you can ask them or reach us >> Have a

nice evening in Amsterdam. Thank