Locking Down Ray Serve: How to Secure Ur ML Models? - Kateryna Hrytsaienko, Valtech

This talk covers the importance of securing Ray models and environments in machine learning applications. The speaker, Katina, discusses how Ray operates with distributed computing and the potential security risks associated with Ray jobs, highlighting that Ray does not enable security mechanisms by default. She demonstrates how to harden infrastructure by implementing namespaces within Kubernetes, using virtual clusters, and applying policies to enforce restrictions and mitigate risks like remote code execution and resource abuse. The session also explores the integration of mutual TLS for secure communication among workers and the use of tools like Kivero and Prowler for policy enforcement and system monitoring. Throughout, Katina emphasizes the necessity for ongoing security efforts and the application of best practices to maintain a secure environment.

So actually hey everyone it's such an honor today to stand in front of you. Thank you very much for joining in and for sharing the passion for the security and locking up your ray models. So basically this topics is something that you usually learn in hard way and that was the case for me that's why we are here today. So you wouldn't need to spend so much

time on research and upgrades and everything. I will show you it today all. So before we start actually nice to meet you for those who don't know me my name is Katina I actually do quite lately a lot of envelopes with a security flavor in it and that's why I spend quite share of time making envelopes more accessible to everyone preparing the workshops to create your first

envelop pipeline and etc. So today we're going to dig in into how actually make your way secure and I will in the end also share with you the runbook where you can use all the examples that I showed today. So please don't be worried that you will miss anything and let's dig. So first of all I want to know about you. So please raise your hand if

you heard about Ray or use it. Wow mean like there is the fans finally. I mean like I will still talk about it but that's fine. And who does envelopes for fun for work? Okay. Amazing. Amazing. I see those row. It's my favorite so far. Um, okay. That's awesome. So, just quickly because I wouldn't bug myself to not talk about Ray at all. Ray is awesome. You

all know that it's open source. It's allows us to dig from our shoulders the issue of distributed computing and allows us to manage low-level processes with a high level APIs. That sounds already amazing. So actually also ray translate the needs of your model to the kubernetes via kubray with the three custom resources. That's also awesome and also nice but we already probably all know that just for

making sure that everyone on the same page ray cluster is a huge box where all shenan is happening. Ray service is where actually have ray cluster combined with your model and in the end it's ray job. That's my favorite one. So that's the part where you can submit something inside of your rake cluster for training for example and after the completion resources will be cleaned up and

actually ray jobs is something that concerns from the security perspective quite a lot. So we're going to talk about it as well and here if you also dig into how ray cluster works under the hood you will find out that it's quite similar to the kubernetes as well. So in our case we also have a head node of course where we control our workers. We have our

worker with a raillet process like couplet and etc etc. But what you should be aware of that the head node is as well worker node in disguise. So it means in theory you can actually run the worker process on the head node and that's in general not the best idea and generally the best practice to trim your node from this permissions to avoid the situation when you

do overload on your system and your head node just blackouts and your cluster goes blank. And oftenly when we talk about ray, we also know that ray is flat. But what it means ray is flat in terms of security. And it means that by default there is no security mechanism enabled within the cluster. Everything has access to everything. And while we know that actually ray has access

and can run arbitrary code that gets scary. So basically in the right hands it's a huge power. So anybody with the right skill can abuse your GPUs and execute their malicious code and make your cloud bill explode. And that's exactly what we don't want to have here in our case. And that's actually another issue in ML security because we not only care about data. Data is still

important but ML has a huge access to the resources that are quite powerful. So abusing them means actually making your bills huge and unbearable. So we will dig into it and see how we can fix it. And I just want to show this simple pipeline here. I'm a huge fan of the Google cloud. So you will see a lot of Google cloud stuff today. But come on

ladies and gentlemen, we're doing the Kubernetes. So it should be fine as well. Um in this case it's a simple MLS pipeline that we have submission of some rate jobs from the cloud functions where we do training. We pull some data from them uh data source and in the end we push something in the container registry. Pretty standard everything is as usual. So what you need to

remember about MLOps apps that are still apps and they also ML. So it means that they are burning in all the same places that the regular applications would but in additional as well. So what could go wrong here? As you already know remote code execution huge problem also we can face with some issues due to the list privilege vulations. If you don't do the configurations on the

level of the data source properly, we will also be having the situation when we load extra information from our data source. We also can have issues with um input manipulation. So somebody can inject some malicious input into your object store and corrupt actually your ray node. So it's getting quite scary and quite quickly. And what you should do about it? So what in our case we should

do with everything? Well, our first idea would be throw on some layers and please welcome the security designer. That's what we're going to do today. So from bottom to the top, we're going to throw the additional layers of the security to our ray clusters. So in our case, what we're going to do first of all, we're going to talk about an infrastructure and how we can harden

it as well. We're going to talk about the feeling. So in this case, it's a core identity. So how to make our cluster accessible for outsiders with ease. And we also have a source because of course how we could forgot about it. And that's actually enabling the mutual TLS in the easiest and nicest way possible within Z and the basil on top will be compliance and governance

because security team from my company would kill me if I wouldn't add it to the slides. So the next thing what we need to do is get to the business. So here we're going to talk about the past the first layer that's the infrastructure layer and imagine the situation. Of course, we would like every ML team have their own cluster, be peaceful with it and actually do

whenever they like. But unfortunately, the economy says us otherwise that we need to share and we do share. So imagine in like simple security scenario when we get our node exposed then through the kub operator gates to the control plane and now all cluster is compromised. So what we can do about that? Of course, the first thing that we would like to do and we should do

it's throw some name spaces on it. So basically to bound our array into the some limits where it can survive and function properly and happily for us we already have the functionality to do so. So there is a configuration on a kubray level where we can install kubray into the separate name space allow it to leave it there and what we achieving with this that we actually

binding ray not more on a cluster and giving them cluster level permissions but rather giving them row level permissions so it's completely no differency and of course you would say like kina I have a two teams three teams five teams I don't want to manage kuber imagine new version and everyone need to granularly update do it all in sync I hate it I wouldn't do it ever.

So the next thing what we can do is actually allow ray to have kubray for the multiple name spaces. So in this case each team can actually use one kubray but we can upgrade it equally use it and whatever and in theory everyone should be happy with this solution as well but unfortunately it's not the case especially in the earned price where the limit is quite high

and any platform team member will say to you like kubareet customer source definitions on the level of the cluster I don't think so you wouldn't have it and even more frustrating like when you have the general cluster for every team that means that you need to grab their permissions and put it to the minimum. So basically you don't even have access to getting the list of the

name spaces that you belong to. So it could be getting quite frustrating. So ideal world suggest us to having something like small kingdom within your general cluster. And what we can do in this case to have a small kingdom where we can actually run all our ML processes preferably drop bombs and still be happy with it. Actually there is a solution that could help you with exactly

that to have the individual cluster per each team but still having the same GPU resources and not losing your clas. So the resources we cluster open source solution that allows us to create the virtual clusters within our huge GKE clusters. It's quite awesome. You would think the first idea would be virtualizing our ports, but that's not the case here because VC cluster allows you to create quite

real, but they are under the virtual control plane. So they are constantly synced within the real GKE and the virtual cluster. That's quite awesome thing. And here inside of virtual cluster, you can have your own custom resource definition, your own policies, basically everything. And each team can have a feeling that they have own cluster they're using but still with the virtual cluster you still can enforce some

policies on those teams on the level of the host cluster and that's already awesome and that's the next topic that I want to discuss because okay namespace is good but in reality namespace is just sin boundaries that we put what we can add here with some policy engine like for example as well awesome open source tooling that you can use to provide the policy on the level

of the host cluster and they will be transmitted inside of the B clusters as well. So Kivero works something like this on a moment of the admission on validation of the resources it actually will forbid or mutate the resource that it's not corresponding to your policy that you already declared and that's quite awesome. You remember I told you about the nodes back then when we have a

head node and we shouldn't allow it to have a worker process on it. Kerno is exactly the policy engine where you can enforce this policy to make sure that nobody within your teams actually doing some shady stuff. And here additional stuff that we put and should do is to limit communication between the workers and head node and control plane. So the general rural for a cluster is

head node talks with everyone with workers and control plane. The worker can talk only with worker and head. And maybe another thing that I should say here is that further step to make sure that you not abuse your resources is to put right limit ranges because of course everyone will tell you like hey kuberator uses the pot template. So here you can already put the maximum resources

but limit range is another level where you can make sure that nobody abuses your resources in terms like somebody configuring the job that pushes too far to the other jobs. And well with that we will move into the next layer and this layer is probably most interesting one where we actually communicate with outsiders because our cluster is useless if nobody can have access to it right so

here we can differentiate between the inbound traffic outbound traffic and some cherry on top inference getaway I'm extremely happy I have this slide because it's exactly what we talked on the K node in this case so let's start with it of course depending on the trigger that you actually do inside of your cluster you should consider few options. First option, if you are actually using some resource

that provides by your cloud provider, in my case it was a cloud functions, the best way probably would still keep your traffic inside the network. So what you should do in case of GKE, you should use the uh direct ingress. That means that you basically provide the private API within the network where your main cluster lives and no traffic ever leaves your network in general. So in

this case we have a private API inside the subnet where our cluster lives and we communicate through internal load balancer. Sometimes of course we cannot do that because our consumers are outside for example developer getting to our dashboard which is extremely important because ray dashboard doesn't has any additional authorization as for now and in this case we can use identity aware proxy. So what it does basically

it allows us to create the proxy with a Google sign in in it and with the transmission to the roles and permission and configuring the proper firewall we actually can make sure that nothing and nobody who doesn't have proper permission and enterprise ID get inside of our GK cluster that was a case exactly for us and probably the most interesting part is AI proxying so that what

we heard on the keynote already everyone told you already that hey we can do some advanced caching. We can do actually spreading the load on the port that actually have access to the context of the concrete operation. But what inference getaway in this case also does for you. It prepares for you model armor. So model armor in security terms is the one of the best thing because

you can reuse the general policy from the malicious input on your actual configurations and RA jobs that you submit. So nobody can inject something like ignore all previous instructions and do some other crazy stuff that they would like to do. And that's most important things that as is provided by the cloud armor. And well additional benefits are of course cache routing that we can do. Addition we

can do some advanced Laura adapters and routing between them loading them in the runtime as well. Quite cool feature. And last but not least as I already said is load aware routing. I would highly recommend you check out these features and check out the inference gateway because it gives you huge power in the end. And now we're moving to the outbound traffic. Outbound traffic is a bit

different thing because here we need to think about our virtual cluster, our real cluster and how the traffic goes from the virtual cluster to real cluster and then to our data resource. In Google cloud is pretty easy to configure. I wouldn't say probably so easy but it's possible. So there is a abstraction that calls for cloud identity where you can actually map your Google account to the

Kubernetes service account and as well to your virtual Kubernetes cluster account that will mean that you will have a direct mapping from one another and it's actually not causing any additional overload because it's happening on the level of the virtual control plane and the level of the sinker. So in this case what will happen you will need to make sure that your virtual clusters are in a

proper name spaces. So your EM policies never overlap and then when you are mapping from the virtual cluster to the KSA you will need to also provide the proper emails as a tick text sorry for that as additional stuff and text for the cluster to operate. Um as we moving further I think the next best part is actually bashamel sauce that's where the stuff also gets interesting

and maybe more on the kubernetes level because of course we would like to limit our communication as much as possible but unfortunately in case of ray it's not the case because here we need to think about the workers that communicate within each other and what it means in ray workers often share the model so in cases our model is too huge and we do some distributed processes

Of course our worker should communicate and in this case a scenario can happen that if someone has access to concrete worker and can expand through all system and in the end again our precious resources are burned. So what we can do and what we should do for that of course the immediate answer is mutual TLS but another thing how we will configure it especially so what we

need to do to make it as easy as possible. So first instinct for everyone would be saying like we throw the service mesh on it. But the reality is like which service mesh how it will be what is the concrete type of service mesh and is it actually the best option. So in this case we have a few questions to answer. So if it's service mesh should

it be sidecard or should it be maybe ambient service mesh. So we won't have in even the overload for the communication between the workers with a quick Z tunnel where we actually have the assess with socket open connections. Actually please raise your hand if you know what's ambient meshes. Okay. Okay. I see. So basically let me then explain quick a difference. So sidecar that means that we

have a proxies on each. So it means that each time we talk with one node to another, we have double proxy and double spend time on actual communication. When you're talking about the ambient is another type when you actually have a quick Z tunnel that don't have a proxy on it. So the communication with mutual TS is much much more quicker and in this terms it's only

L7 communication. So the good thing about ambient mesh if you don't need some L7 stuff so you don't need some advanced additional features you can actually start only Z tunnel and it will communicate much quicker. So in general the rule is the bigger your model is the less you will see the difference the set truth and that's exactly why for example huge providers like Google for now

have a sidec car still enabled. So if you want to have a huge fast, if you want to install esto as you manage it and configure ambient mesh, of course you can do it and you will see the benefit on the lower workloads. But in case the bigger models and bigger loads, it will be less noticeable. But another thing that we have here is also proxy process

gpc. So who knows what it is. So it is another approach when you inject the logic for mutual TLS in the libraries OGRPC service itself. So it means there is no proxy at all. But the issue here is that you need to have access to your code when you're doing it because you need to update the specific library of gRPC and that's why unfortunately this way is

not possible with Ray because it allows us to do the distributed processing abstracting all this logic. So you don't have any access to how your gRPC actually communicate with one each other. You cannot upgrade this libraries. You cannot manage the configuration. You just need to live with it. So as far as proxy gpc probably one of the best options for us to actually get the best of

all worlds with ray it's not possible. But what ray says it says like well we will invent our own way. So, Ray provides a way to do mutual TLS on the application level with array. And I see some smiles here and I understand why because actually it's probably one of the worst ways if you want to configure mutual TLS in Kubernetes because what you will need to

do you will need to create your certificates authority. You will need to manage the life cycle of the certificate authority. you will need to inject the certificate authority into each pot and generate on the need container the certificates per the pot. So basically you manage everything and you create the additional overload for each Um why we should do that unfortunately I don't have reason for that. I

think it's additional features that ray has right now to make the future look better and more prospect in terms of the security. But in some cases when we don't actually have access to Kubernetes for example when we run Ray on the VMs or on our own hardware sometimes probably it can actually benefit a little bit. So just a small matrix for you to make a final decision

what you should do and what you shouldn't. So probably the best way to configure the mutual TLS for the array would be using the ETO or in case of Google cloud it could be managed uh control plane from the GKE themselves. It will allow you to actually get your mutual TLS up and don't spend too much time on it. But please be mindful when I'm saying you

won't spend too much time on it. That unfortunately means that you will still need to do a lot of configuration because as you can imagine Ray has own clusterto service mesh do its own L7 monitoring and you need to make sure that knows about the ports and ports that Ray will use. So it wouldn't block it and wouldn't overlap. So you still need to do a lot

of manual configurations and list all ray ports within your configuration to let it know what actually happens. Unfortunately, there is no way to make it quicker to make it a port range for now, but we still keep hoping and we'll still look into the future. So the verdict here would be probably use the ambient mesh if you can to make it most quick as possible. If you

don't want to bother, you can still use some managed solutions like Google cloud provides with the cloud service mesh. Then you will use cardar or you can also deploy your own isto with a sidecar approach. And the next thing it's probably most frustrating slide from all my presentation because it's the first time when I ever will say that I'm a bit shocked with approach that Ray chose

because that's the ray tokens and when the feature just came out I was extremely excited about it because it promised us to finally have a token authentification within the ray. Wow. But unfortunately the thing here is that first you shouldn't get excited because ray tokens are not granular. They just a passcards saying that you can move from one node to another. Second thing, you still need to

generate them on each start of your cluster with a need action. You will need to generate the token that will be stored in a shared memory and then it will be passed between different nodes and workers. And also if you use ray tokens they are transmitted via HTTP nonprotected headers. So it means that even if you would like to use them as a tokens, you will need

to provide a proxy in front of it to make it actually secure. So the fair question is do we really need to do all of that just to still put a proxy still get a fast with all our other functions when we have mutual TLS probably some proxies in between and etc etc. I would say it depends on what you want to achieve and how much you

trust the workers within your actual system. But the general rule of thumb would be that for now while the array tokens are not mature enough. I would say that you shouldn't add it to your system at all for now. And especially important thing here as well that you cannot enable it just on a part of your system. So if you are enabling the rate tokens on the

system as a whole that means that even with communication on your outsiders even in communication between your head node and workers you will still need to provide the tokens in general. And that's finally the slide that gets us also excited because I was talking about Kivero and policies and how you can actually enforce some rules within your clusters. And that's another thing that my security team made

me to add. That's basically the axiom that security unfortunately is not constant and you need to put a lot of effort to just keep it running on the same level as it is. So that's why we need the monitoring tools that allows us to scan our system as a whole and made the reports on it. So in this case that's a Prowler. That's my suggestion to you.

Why? Because it's open source. Second thing because it's quite easy install and also it provides you a lot of resources that you can actually manipulate it. And also we have a lot of stuff that it can scan for because it can scan for the misconfigurations on the infrastructure level. it can scan for the improper permissions within your application and even if we don't have the actual authorization

in BL for our application. So probably brower will complain about your dashboards in rate quite a lot and even the best thing about it that that's basically how it's installed. So we can install it as already predefined set of resources. It already comes with a package as a job roll binding and everything and you can just put it inside of the chrome job that will continuously run

on sometime intervals within your cluster to make sure everything is configured as it should with a proper reports and beautiful dashboard. And in the end they are evolving pretty quickly and allow us to add our own features on top of it. So it's also quite exciting. And that would be all that I wanted to tell you. Actually, please don't hesitate to scan the QR code because I

have the all stuff that I display in here you today with the configurations in the blogs. It's the first part of the section. So, you will also find there some information about how to create your first envelopes pipeline and don't cry yourself to sleep. And also, please don't hesitate to put the feedbacks because I will appreciate it quite a lot as well. And of course, let's connect

because I'm usually quite friendly. You can find me whenever on a set. I will be happy to answer your questions. And thank you so much for being my first ever Kubricon audience. Thank you so much. Thank you.