The Shell Awakens: Cloud Native Workflows for Particle Ph... Raulian-Ionut Chiorescu & Hannes Hansen

This talk, titled 'The Shell Awakens: CloudNative Workflows for Particle Physicists', presents an innovative platform developed at CERN to handle the massive data generated by the Large Hadron Collider. The speakers, Hannes and Raul, explain how they built a computing environment on Kubernetes, integrating heterogeneous hardware resources such as Nvidia GPUs and AMD processors to support various user workflows including machine learning and scientific computing. They describe the architecture that supports interactive sessions via SSH and Jupyter notebooks, as well as tools for managing machine learning operations like Kubeflow and MLflow. The presentation emphasizes the importance of user feedback and the platform's ability to offload workloads to public cloud resources for better efficiency. The speakers also touch on challenges and future enhancements, particularly in areas like multi-cluster scheduling and resource monitoring.

Okay. Hello everyone. Welcome to our talk, The Shell Awakens, CloudNative Workflows for Particle Physicists. It's great to see so many faces in the auditorium. Um, yeah, my name is Hannes >> and I'm Raul. >> We are both computing engineers at CERN. CERN is the world's largest particle physics laboratory. We are hosting the LHC, which is the so-called large headroom collider. It's a 27 km ring below the

French Alps where we accelerate protons to nearly the speed of light and then we let them collide and then we study what happens in this brief moment of time. When a machine is running it and runs for 24/7 and produces a lot of data. So the output is like in the range of terabytes per second and this is a little bit too much for us to handle.

So in order to reduce a bit the data output we uh employ a so-called trigger chain which we use to filter out uninteresting data. And this whole chain consists of FPJAS but also CPUs and GPUs. And we recently started a R&D project which is called the next generation triggers where we try to employ like the latest machine learning research in this whole to support our users in

this endeavor. We run a platform for machine learning and scientific computing. And here on this picture you can actually see quite a real picture of a working environment at CERN. So you can see a bunch of physicists sitting in a uh oper control room operating the detector running some physics analysis and you can already see like everyone has like their own workflows their own requirements. So you

can see oh maybe someone says oh I want to have an H100 GPU. Someone else might say I want to log into a shell. I need all my in like I need all my physics software pre-installed but I also want to install my own. Someone else might say I want to have a Jupyter notebook. Then someone else comes and says I need machine learning tooling to run

distributed training across all the nodes that you have. And then someone else might want uh to run statistics on power consumption of his workloads. Okay. So in order to cover all those use cases and all those requirements of our users, we had a look into the CNCF uh ecosystem and eventually we came up with this reference architecture that we actually released um a few days ago to

the CNCF. You can check out the QR code here if you're interested. And the idea is uh quite straightforward. So our platform is running on Kubernetes on a cluster with shared compute resources. And then on top of that we have different entry points and different features that we offer to our users. So for machine learning we cover the whole MLOps life cycle using cubeflow ray kerf but

we also offer interactive access to the system via ssh via jupyter notebooks or v code ides. Next to that we also have MLflow for model management and experiment tracking. And we also in include GitLab and GitHubci so that our physicists can make sure the tests are all running through. And now Raul will explain you a little bit more about those shared resources. Thank you Hannes. So how

does our cluster look? Well, it's quite a heterogeneous setup. Uh it includes uh a bunch of Nvidia GPUs like H100s, H200's for the usual use cases, but also L40s for more like scientific computing type use cases, AMDs with MI300X's and Radon Pros, a few specialized CPU nodes with high memory or high core count or high base clock frequency. And all of these nodes have uh interconnects. Some

of them are fully interconnected with Infiniband. So as you can see on the right side uh each GPU has its own uh connect x7 nick and all of these nicks are then connected to the infinite band switch that sits in between the nodes and uh that's how you can run workloads across the nodes uh with very high speed and very low latency. All the other nodes that

don't have infiniband they have rock v2 tunics and also we have a as of more recently a few as well. So how does our storage look like? As you may know, sir needs a lot of storage. So this is the current setup inside the cluster and outside. So for uh when you need high IOPS, we just mount an MTB deer volume that we call it scratch and

people can compile their code there. For sharing uh data across workloads in the cluster, we have longorn deployed and this works quite well for us. Each user has its own like shared PVC and this is automatically mounted to their workloads and they can share data for extracting data outside the cluster. We use Seph. This is another team maintaining it. And uh this is how can they can

get their outputs from running their code. And we had to work quite closely with our users to like inform them and inform them of the best practices of what to use when because if you compile experiment code on a shared file system, you might not have such good experience. So what environments we provide our users? Well, we provide quite a few curated images with CUDA and roam

riaries, all the pre-installed tooling for ML libraries as well and physics software. Uh, as these images get quite big in the order of like 30 40 GB and even for FPGA simulation, they get to like 140 or 150 GB. So, they're very huge. We prepool some of these uh on all the nodes. So, when you start up the session uh the you don't have to wait too

much. Also some users actually build their own images with their custom software if they want to that are based on our based images. All these environments as I said have access to their local scratch space as well as the shared sol systems. Their home directory is set to c other certain storage systems and all the CPU and memory and local scratch space are preset based on the

number of GPUs you select when whenever you want to start your session. We do this with Civer and Hannes will give you a bit more detail about this later. So we might be at the emerging plus advanced track but uh when we first released the platform uh one of the many requests from our users were to give them SSH access to their uh their sessions. So how

do we do this? We use a CSF project called container SSH and this works quite well for us. In this case we use the pod name as the identifier for what pod you want to SSH into. Each user is namespaced in their own namespace. That's fine. And actually the user ID of the CERN users is taken from either their OIDC token or their career ticket for legacy

reasons because each user is isolated. They have pseudo capabilities inside their sessions because they asked for this so we gave them and we also collidal sessions if they're either for too long of course. So how do you set up an interactive session? You can do this in kind of two ways. uh we give users already access to the cluster as well. So they can just uh kubectl

create pod in their own namespace. So this is kind of how a podspec looks is quite simple because we abstract a lot of things from our users with kivero. So here they have a few labels they need to add to their session like mounting all the certain storage systems and what home directory they want. They just select how many GPUs they want and the flavor of GPU

and that's pretty much it. you create you have your session then either you can use the coupeflow UI and this will go a bit in more detail and just uh create a notebook type interactive session where you just select your image and you select the type of GPU you want okay now I have my interactive session so I need to write code physicists write very performant code

so they want to squeeze every last bit of performance of the hardware they run on so we had to do a few p performance optimizations so how does this uh by default in Kubernetes uh container processes are spread across CPU cores and NUMA nodes and this kind of uh causes quite high memory latency and we you can lose a bit of performance. I don't know the exact

numbers but they wanted this. So without topology awareness you might have a case like this where a CPU in node zero is trying to access memory that's bound to num node one and this causes a bit of latency. with the topology manager in Kubernetes you can actually change this. So how it looks uh you do some settings in the kublet and basically you manage to uh give

users access to full course instead of spreading across uh all cores and also you have local memory access guaranteed in your num node. So I'll do a quick demo and I'll show you how this works actually. So bear with me. Okay. So, first I'll create my pod and then I'll go on how uh on to how the pod actually looks because it will take a bit of

time to start up. So, this is how it looks. It's pretty much kind of the same as I showed you in the picture. I have a few labels to set my home and mount some certain storage systems. I want an H100 and this is the default or the most used image that people used. So now that the pod is ready, I will just SSH. Come on, internet.

Unless I misspelled it. I don't think so. Yeah, it's probably the Wi-Fi. Okay, I'll I'll give it a bit more time. If not, I'll try from the other terminal. Ah, okay, I'm in. Okay, thank you. So, I have access to my Nvidia GPU here. So, I can start doing my workloads. Oh, wait. I forgot to install a package. So, I can install a package. And it's that

easy. This will take a while. So I'll go into an SSH session in the other node as well in the other terminal tab. Okay, cool. So as you can see I have some uh scratch space that's pre-mounted here. This would be around 200 GB for this type of GPU that I requested. I have access to my shared space as well. So now I'm going to do a

quick demo of the topology manager as soon as this package installs. So yeah, cgroups. So even though I have uh allocated like 46 cores by default to my session, I can still see all course on the on the node. But now if I run this stress test tool, I should only have access to 60 cores or so or 60 threads. that even if I run it on

384, it should show up that I only run fully on the course I'm allocated and I cannot uh I cannot influence my neighbors in this case which I would say is pretty good. So that's it for the demo and I'll pass to Hannes to present our machine learning stack. >> Okay, thanks. So yeah, for our machine learning researchers we try to cover the full MLOps life cycle

uh which usually begins with interactive development. For this we use cubeflow and cubeflow notebooks um where we provide created machine learning environments pietorch tensorflow scypi the usual packages that you know you can also have ssh based access similar to how rul just uh showed to you or you can attach your local vs code IDE to a running session on the cluster. So how this would look like?

Okay, at the top you see the classic interface uh for the notebooks. Users request the flavor of GPU and the number of GPUs that they want to have. Or if you go the SSH way, you would, as we just saw, create a session, set up your environment, uh set up a T-max session, and then run your training on the container with access to your GPUs. Okay, more

advanced. At some point, you want to scale out your training. We have multiple nodes with multiple GPUs. So you want to run a distributed model training for this we have cubeflow and ray. You can either submit a job like in a batch style um fashion or interactively train using the cubeflow trainer and the ray train SDKs. And for this I will also do a quick demo showcasing

the cubeflow uh training. I will go ahead and already create the training job because it will take some time to run. And now I will show you a bit how this um PyTorch job looks like. So it's a declarative way of um starting your training. In PyTorch you usually have a master which is used to initialize the distributed training and you have a worker pool and what

you do is you set your image. We have either a created list of images or for the advanced users you create your own image and in this case we are using one from our users which is used to reconstruct particles in this brief moment of the collision. You set your entry point script to your training um to your training script. You configure the number and type of

GPU. So in this case we will use uh six Nvidia H100 NVLS. And we do the same for the worker. So in total as you can see we have one master and five workers running. Okay the pots are up and running. That's nice. Let's have a look at the training. The master usually runs as rank zero. And as you can see the training is uh running now

in the background. Let's log in into the master node. Let's have a look at the GPU. Okay, the process is bound to the GPU. We can see it's running. And if you don't want to submit a job, you could also just use this environment now to run your torch run command for example because cubeflow will take care of all the necessary configuration that is needed to run

a distributed pytorch training. Okay, so the training is running in the background. So we can go back to the slides. An important step after training is also the optimization of hyperparameters. Again we use cubeflow kip component as well as the rate tune component. There's different ways of interacting with the system. So you can either use the UI, you can use use a declarative way using the yaml

or again with SDKs. The important part here is that you need to train your users and you have to see what they like and how they interact with the system. Um yeah because there's many different ways and everyone has like different requirements for the orchestration of more complex workflows. We use cube for pipelines. So this is a quite nice tool when you can split your workflow in

multiple steps and then you can reuse them in across uh the different pipelines. What we also do is we pre-built some pipelines for our users. So for example, we have a model quantization pipeline and a FPGA synthesis pipeline that can just be used in the UI by our users and they only provide the path to their model pickle for example and the quantization will just run automatically

on the cluster. Okay. To track your uh experiments and all your model versions that you work on, we have model flow for model management and experiment tracking which we also integrate into our pipelines. And then at the end if you're if you're done with training your final model okay you want to surf it for inference we use cubeflow and ks surf which is a very nice combination

it provides multimodel serving zero scaling and also custom serving runtimes okay now let's have a look at how we do the networking especially how we do GPU inter communication so traditionally if you want to share data between GPUs this involves staging through host memory and CPU involvement and using the default Ethernet uh interconnects. So this works quite well but it's not very suitable for latency sensitive workloads

that we are mostly running at CERN. So that's why we onboarded um some low latency networking networking nicks uh in this case infiniband and rocky uh v2 and uh the setup is done with the Nvidia network operator and the AMD and Nvidia GPU operators and we then expose the nick devices directly to the ports. The nice thing uh of having those interconnects is that it allows us

to use a feature called GPU direct remote direct memory access. It's a very long word short RMA which allows direct communication between the GPUs across uh the um uh infiniband or rocky interconnect. So as you can see there's no involvement of the CPU and no staging through the host memory uh required anymore. Our stack is um based on okay of course the drivers infinip and rocky DMA

buff for the direct memory access between the nick and the GPU and then for the integration with the user workloads we use UCX and MPI to validate the setup we usually run the standard benchmarks but also our user benchmarks. So for example here on this plot you can see a few tests that we run like a standard bandwidth benchmark across two H100 SXM GPUs on two different

physical nodes and we like comparing a few different RDMA options and Ethernet versus infin. And as you can see it goes up to a 50 uh gigabytes per second uh throughput. Yeah, running all those workloads consumes a lot of resources. Um, what do we do if we run out of GPUs on our on-remise resources? >> Well, we have the solution. We can offload workloads to the public

cloud using multiQ in this case. So, how does this usually look in our setup? So, we have a few pending jobs. Our local GPU nodes gets uh gets full. So, we just use multiQ to dispatch workloads to uh other cloud workers in other environments outside the cluster. And this works quite well for us. Also soon we are looking at uh dispatching workloads also to HPC centers. So

we will uh look into using interlink to do just that. So this is kind of how the setup will look and we're still investigating what we'll do. Also scheduling one of the another first let's say feedback uh points we got from our users is that they cannot get access to a full node with H100s. And this happens usually because Kubernetes by default does a spread of GPUs

across nodes. So this is the default policy in the scheduleuler. So this would be like the least allocated uh policy. So we changed this in theuler to use a bin pack strategy. So it actually fills the nodes one by one. So users can access uh full nodes and I will pass to Hannes for the monitoring stack we have. Yeah. Yeah, for the observability we used a typical

stack that most of you already know. So, it's running on Promeus and Grafana. If you want to find out more, there's a talk from our colleagues. Feel free to scan a QR code and check out the talk. We use it mostly to monitor like overall cluster usage. So, for example, here you can see the average GPU utilization, uh GPU memory consumption, but we also track like active

sessions, active but we also have namespace dashboards for our users to give them like a bit more detailed insights about their workloads. And here we show the power consumption of the GPUs uh GPU memory and CPU usage and we also explain them how to scrape those metrics by themselves. So they can use them for their benchmarks. Okay, so now you might wonder okay we have many components

how do we do all of this configuration without confusing our users too much and for this Kano comes to the rescue and we use the Kano policies to yeah extremely simplify the user experience and automate any configuration that we need and this helps us to go from like a very long complicated spec on the left to a very short and concise spec on the right side where

the users only have to configure via labels. So for example, we use the mutation policies um to set the home directories to mount any volumes that we need to do the RDMA configuration that you just saw be before. Um but we also use validation policies to yeah kind of restrict our users from messing around too much on the cluster. So for example, they are not allowed to

set their own resource allocations because this is something we set automatically again via a mutation And yeah and this uh helped us to really simplify the life for our users. But there was a few other lessons that we learned on the way. >> Yeah. So obviously the success of any platform depends on how the users accept it. So we for this to work we had to kind

of glue together things to expose familiar interfaces as I mentioned before with SSH or having access to all of these storage systems. They have their scripts that they're running from like 20 years ago and they still want to run them today. So we had to expose expose familiar interfaces and of course provide known environments. Also we had to we had quite a quick feedback loop with our

users and we had quite a long period for a couple months where we were iterating very quickly releasing stuff every day and getting feedback very quickly. Also uh we had to do a few I don't know uh changes in some of these components and it's not uh feasible long term to actually maintain this ourselves. So we contribute all the local features that we have in the cluster

upstream. For example, for container SH, we implemented the new mode operating mode that just got released a couple days ago. Also, you have to constantly exchange with your users. And I think after seeing all of this and seeing all the feedback, I think cloud native is definitely ready for scientific computing. And as you've probably heard from all these talks, uh these days, it's ready for AI and

ML as well. So to finish off, what are the current challenges we're kind of facing and what's the outlook for multicluster scheduling? Now for now we can only do job like submissions because multiq doesn't support uh remote coupubectl logs and remote coupubectl exec and we are working closely with the cube people to actually implement this as quickly as possible. In terms of low latency networking, we don't

have any network isolation for the uh low latency nicks right now. So we are looking at the srv and things like the aret to handle this problem especially if you don't trust your reser workloads but we trust them so it's okay for now. Also, GPU failure detection is uh not amazing. By default, some device plugins, yeah, they stop exposing uh the GPUs if they're faulty, but this

doesn't happen in all cases and there has to be some manual intervention. But there's tools out there now like NVDMV Sentinel or other tools to and we're going to explore this and make it work. And as the future improvement or what we're going to do in the future is bursting to HPC resources as I briefly mentioned with interlink also we're having a look at the array and

automating partitioning for checkpoint and restore this we would need very much for node maintenance because a lot of our workloads are stateful. So we want to be able to actually move them to other nodes without kind of affecting users too much. And there's this new uh checkpoint and restore working group that was created not too long ago and the guys are working very hard there to actually

make this work. And also we're looking at building a few automated benchmarking suites on different hardware configurations as well. So thank you very much. This was it. And if you want to contact us Hello. Um, doesn't work. Oh, it works. Uh, so you just told how you run uh machine learning models with your platform, but do you have a practice of running predictive models like for example

with open form uh on the similar platform? >> Uh not specifically. Um but yeah, we can have a chat afterwards if you want. >> Yeah, >> like I'm not like you don't know open >> Yeah, it's just a tool. But do you run predictive models? >> I feel like some of our users do uh but I don't know exactly. >> Okay. Yeah. Thanks. >> Hello. Um have

you considered using cube word instead of uh pots that run sleep infinite and also have like 40 GB big images which is I think not the best use case of if uh of the uh resources. So you have also when you restart the pot obviously what you install afterwards is just gone. >> Yeah. Yeah. Well for the curated images we try to fit as many packages as

we want. We have enough storage to kind of handle this for now but uh if it gets out of hand we'll definitely have a look. >> Thank you. >> Hi there. Uh nice work. I had a question about how many users do we actually have for the platform inside of CERN? I presume you're in one of the experiments CMS or Atlas. It's actually multiple experiments collaborating together

in this project. So it's actually CMS and Atlas and the theory department collaborating together. So it's quite a few I'll say >> but like how what's the adoption for your project inside of CERN at this >> ballpark users tens hundreds thousands >> I would say it's in the is the it's in the hundreds for now for the users the active users. >> Are you running any inference

as well? >> Users are running inference. Yeah. >> Okay. Okay. So they're uh you're doing uh management for their models or are how do you manage their models for inference or for example how are you deploying them? Are you just letting them deploy it themselves or >> Yes. >> Okay. So you're not doing management in >> No like that's the idea of the platform that as a

user you can manage everything by yourself. Um you can yeah you can run like any infant service as you want. >> Okay. Of course we enforce quota and those things but we don't manage the models for themselves. >> All right. Cool. Thanks. Yep. Hi uh thanks for the talk. Um it seems like you're giving your users a lot of freedom in running workflows. Do you have any

audit auditing or threat detection uh in place? >> Yeah, we actually have falco deployed and if something weird goes on we get alert we get alerts. So it's uh it's okay. >> Okay. And again we use kano policies to make sure that yeah you don't run with um yeah privileged access for example. Hi uh first of all congratulations on a great talk. Um I have two questions.

One is that uh do you also have requirements you know uh around uh artifacts lineage tracing all these type of things you know do scientists uh you know in your daily work also ask questions like oh okay you know to really get uh these type of uh you know uh predictions right you know which model was used which data sets and so on so forth. And my

second question is that do you guys also have a phops you know um you know type of questions right you know >> so I'll answer the phops part obviously as we have most of the resources on prem like we already paid for them so yeah but we also have open cost deploy to kind of like have internal dashboards on how to how much users are spending even

though yeah they're not spending it actively for the cloud stuff yes we do have it >> and for the first question yeah It's pretty hard like this kind of reproducibility on the system. We trying to integrate like all these tools like cubeflow, mlflow automate as much as possible. So for as you mentioned like kind of yeah have like the model artifact to know it's related to this

code to this version. Yeah, this is a pretty pretty hard challenge but yeah we're trying. >> Yeah, thanks a lot. >> Uh thank you for the talk. It was really interesting. I have a question. Do you think this uh sort of cloudnative infrastructure has the potential to replace classic HPC you know kind of slam based or PBS based kind of uh you know resources for now we're

not necessarily looking to replace it but work alongside it so that's why we said we are going to look at like things like interlink to kind of try to bridge this gap and also maybe things like slinky or these kinds of things for slurm so >> yeah and what do you think is missing Not quite sure yet actually. >> Um hi, thanks for sharing all that wonderful

information. Um a question it's related to what he asked. Uh what made you choose Q over Slinky or some other type of resource manager? Q over Slinky for public cloud or for >> for for private just like on selfho hosted internal HPC uh workloads. >> I I think we tried a few things out and we just had a bit more experience with Q. So that's why we

ended up picking it and we also are work quite closely with the people who maintain Q. So it's easy for us to uh get stuff done in the project if we want. >> Okay. Thank you. Hi, thanks for the talk. Um, I wanted to ask um it's quite a big tool chain already. How do you test all that? Um, have you a dedicated environment for this? So,

how do you handle this? >> Yes. Yeah, we have staging environments. Uh, we have automated testing suites. >> But yeah, you're right. It's a many components. Yeah, we make sure upgrading doesn't break user workloads. So, we always try to make sure that documentation is up to date and everything userferacing is tested before we upgrade to other versions. >> Yeah. And uh we have a like a set

of user workloads more of the common ones that we also run automatically whenever we want to do something new to make sure the same workloads actually run. >> Pretty cool. Thanks. >> Hey uh thanks for the talk. Um I just wanted to know um how many people are working on this big setup? Can you tell us something about the manpower behind that? So you can see the

whole team there including us and Ricardo. >> Okay, fine. Same as our setup. >> Cool. Thank you guys.