Your Models Are Vulnerable: How KitOps Turns KServe Into a Zero-Tru... Brad Micklea & Gavrish Prabhu

This talk discusses the vulnerabilities of AI models and strategies to enhance their security, focusing on the KServe and KitOps projects. The speakers, Brad Mikley and Gavish Prabhu, emphasize the need for zero trust principles in AI deployment, recommending practices such as validating model signatures and maintaining chain of custody for training data. They introduce KServe, designed to simplify the deployment of inference workloads across various AI frameworks while enabling integration with tools like LLMs. The session highlights the importance of unifying model artifacts, datasets, and compliance documentation, demonstrating how KitOps facilitates secure packaging of AI assets using OCI standards. Specific use cases, including a logistics company standardizing AI processes with KitOps, illustrate its benefits in managing compliance requirements and promoting collaboration across teams.

Welcome. Thank you uh for spending a few minutes with us. We're going to jump in and talk about why your AI is vulnerable and how to make it less vulnerable. My name is uh Brad Mikley. I am a maintainer of both the model pack specification and the kitops project inside CNCF. Uh I'm also the CEO of Jozu, which is a commercial company that uses the kitops and

model pack technology. Uh hi, I'm Gavish Prabhu. Uh I work at Nutanix. I was one of the founding engineers at Nutanix Enterprise AI. So I'm also a maintainer in case and also now a maintainer in on my AI gateway. >> All right. Do you want to talk about KERF? >> Yes. So Keser has been around for some time. It was a sub project under CubeFlow. Then it

took its own direction. So the main purpose of this project is to help you deploy your inference workloads with any runtime in just like few clicks. We have something called as a inference service which is our custom CD sorry CD and we use that extensively for all our uh selective runtimes. You can choose VLM, you can choose SG lang, you can bring the traditional ML models, anything.

And this is a complete box with both generative AI and predictive AI. So it's very simple just one CR and you're good to go. We also have very good integrations with LMD. There are a lot of talks with LMD. It's the new hot topic this year and we have something called as LLM infant service. It's a new CR. We saw that there is very much need a

different like a good integration with all the things LLMD needs. It can be KV cache, it can be like GI integration, any gateway you want to bring all those things will come in. And now also disagregated inference you want a different PD part good we'll give you all in the LM inference service C and all along with this KSER has also its own uh in-house autoscaling it

will help you with all your networking everything so all's good so that's about Kserve >> all right so Kerve as Gish mentioned makes serving very easy but you're the ones that are responsible for adding security. Um, it is not designed to be an exceptionally secure. It's meant to be exceptionally easy. Um, so you're going to need to verify that the model or agent that is being served,

MCT server, whatever it is, is the same one that was approved. Um, you want to make sure to validate signatures. You want to make sure to validate addations before you go and load weights. You want to track the data set that was used for training or was used for validation. You want to make sure that if you're looking at something running an inference in production that you

can very quickly and very easily without talking to other people or tribal knowledge get back to what was the training run, what was the data set in the event that there are hallucinations or there are mistakes and you need to figure out why those got in. And for those of you in regulated environments, which I'm sure a lot here are, you also will need to be able

to prove your audit ready evidence or your chain of custody was kept intact for the whole way. And again, these are not impossible uh but they're not things that come out of the box with many of the uh open source solutions you might be using. So the idea is let's create zero trust for inference. So in other words, same principles that you would apply to containers in

your microservices, which I'm sure many of you are going to be very familiar with, but now applied to AI workloads. So the first thing is you don't just trust the workload. You have to make sure that you can prove its identity and its integrity before you serve it. You want to verify things at deploy time. It's not enough to just say, hey, they were I'm pretty sure

they were verified or they should have been verified before going into the container registry. Not good enough. Got to make sure they match still when they land at your cluster. And you want to make sure they fail closed. Um, and this is this is a bit of a tricky one because especially with agents, people often want them to work even when they go and disconnect, for example.

Um, but depending on the security posture of your organization, you may want to instead have them that require failed open obviously. You may want to have them fail close. That's going to create an operational pause if there's a problem, but it's not going to create a security issue. So again, that's kind of a choice you'll need to make depending on workload, depending on organization, etc. So, kit

ops is a CNCF sandbox project and it's about bringing OCI native packaging for a IML artifacts. So, you bundle your model weights, your data sets, your prompts, your config into an OCI artifact and store that in your enterprise registry where people already go to get all the different containers that they go to run. So this gives you uh Shaw 256 digests on every layer and of course

on the package itself. Um it gives you standard OCI distribution. It works with any registry. You get the benefits of content addressable storage which when you're talking about gigabytes and tens of gigabytes and hundreds of gigabytes payloads that content addressable storage can be a huge gain over something like just throwing stuff in S3. And you get provable chain of custody. Well, we're sorry. You can get So

the reality is that what most people are probably working with is a situation where your model weights are in one location, your data sets are in a different location, your code is in Git, your documentation is in a wiki or in Google Drive or whatever have you. Even if each one of those areas has proper versioning attached to it, there is no unified package. There's no unified

versioning that is happening to say that the project is the composition of model version three, data set version 16, code from this commit level or this branch and documentation from over here. And that's important when you're talking about chain of custody. You're talking about audit evidence and you're talking about just knowing exactly what went into production so that if something goes wrong, you can answer the question

of how did it get there and why. So, I'm going to talk a little bit about Kitops and how it's being used at a logistics company here in Europe that is worth over $30 billion. I'll leave you to figure out which one that is. Um, but they have 3,000 offices and work in 90 countries. Um, and they're standardizing on KOPS for all their a IML workloads. The

reason why Kitups is such a good fit for this organization is because they said something which I think is probably true for almost everybody which is they want their data teams to be able to use whatever tools make the most sense for their workload for their use case. They are covering logistics in a lot of different countries with a lot of different um kind of compliance requirements

in different countries. So those teams that are local in that area know what they need. Let them use the tools they want. They wanted security and chain of custody across all their operations and they wanted that to be unified. That was something they didn't want to have 90 different versions of. They wanted one version of for the company that everyone can feed into. They wanted it to

be compatible with Edge, with on-rem data centers, with Azure, with Google, with um AWS because they deploy to all of those and probably more that I've forgotten. um they wanted open standards wherever possible. Um it's actually a bit surprising to me when I started really working on this problem about three years ago that there wasn't already a lot of standards associated with this, but it seems in

the IML world there are relatively few universal standards. I think those are growing now and this is something I think we all benefit from. So they're going to use it both in development and in deployment. So let's talk through kind of what that looks like. So in the first phase and I'm calling this the build phase you could call the development phase u but this is where

you're the data scientists or the data teams are selecting base models or in some cases building their own models of course from scratch um curating data sets and running training. So the kind of things they end up with of course is some kind of model, some training data sets, some scripts, hyperparameters, and then metrics out of those training runs. What worked, what didn't work, right? So what

they do at that stage is they put all that stuff into a kit ops model kit. And um if we have time uh well I'll I'll just hint on this and then if you're interested, you can talk to me afterwards. But there's some really interesting things you can do here. I was talking to another organization yesterday about this. Let's say you have um data sets that need

to be kept secure. They are perhaps customer data sets that not everybody should just have access to that maybe need to be deleted when that customer says okay the project is done. Um please delete all my data and you want to make sure that it hasn't randomly ended up on somebody's laptop that you don't know about. So you might put that into its own model kit. Then

you might have your internal company anybody can use it data sets. they might go into a different model kit and then each project would create its own model kit and you can kind of chain these. So this project specific one could say I'm going to use this particular one one of our internal company data sets that everybody can use but I'm also using this very specific data

set from Jozu my my customer and I want to make sure that that Jozu data is kept private. So I have a specific ref to a limited entry registry uh um repository pardon me where that data is kept. Now, if I come on later on and I say, "Okay, the Josu project is done. We've got to delete that data." I just go into the registry and I

delete that repo. And although there are a bunch of model kits out there that might point to that, they don't have the data itself. So, you don't have a data um kind of leakage issue. So, you can chain them like that or you can just throw everything in one. It doesn't really matter. The point is having it in a model kit does give you that chain of

custody. Uh next phase would be like the verification phase. This is one that in my experience is missing from a lot of organizations. My suggestion was it would be if you don't have this, I would add this. Um once you kind of get something that that the data science team feels is ready to go to production, whatever production looks like, you should have somebody who is responsible

for reviewing it. Um maybe they're doing compliance analysis. Maybe they're just double-checking all that work and the artifacts to make sure that everything is complete. they really did look at performance as well as you know precision and accuracy or whatever metrics are important for you. Now this is where it gets really interesting. By having this stuff in an OCI artifact, it means that not only can we

do things like add the compliance docs so that now those aren't in some other directory on somebody's laptop. They're actually with the artifact itself. So the compliance documentation for the model is with the model. That's already cool. But we can add addestations. So now we can add an add a station that says Brad reviewed this and maybe he's responsible for the project. So Brad gave it the

thumbs up. If you don't like it, go fire Brad. Um you can put evaluation results and metrics as an addistation so that other thirdparty tools can look at that and say hey I am comfortable with or not comfortable with this level of security or the level of performance that this particular model has. Um my company Jozu which uses this technology for example is working with a big

um uh security organization. They have about eight different kind of security levels of networks that are all segregated. And so they use addistations on top of these model kits as a way to say it can go into networks one through five but it can't go in six through eight. Um this one can go in through to seven and eight but it shouldn't be used anything below. example.

So things like last is that deploy phase and I've hinted at this obviously here you would have you know optimizations that you need for serving on different platforms that would be you know the kind of case serve specific configuration for example would land here. Um you might have definitions for your deploy pipelines. Maybe you have different deploy pipelines depending on where it's coming from or where it's

going to. Um you can end up with quantization configs, optimized weights, maybe rag pipeline configs. All of those things can also be added to the model kit. And again, your use case would depend on whether that makes sense or not. But you can add those things if you'd like. So you end up now, you can see, with a model kit that has everything from what you were

using and started with through how you developed it to how you are going to deploy and run it. You could add your telemetry stack in here. You could add a reference to the base container that will actually be used as the runtime here. whatever kind of makes sense again for your organization. If you want, you can talk to me afterwards about specific cases. All right, so let's

talk about how you could use this to then add security to what you may already be doing. So I'm going to kind of take this in levels kind of from the easiest to a little bit more advanced to quite advanced. So kind of your level one, what you should really be doing already, but which comes kind of for free with kit ops is at least the verification.

So if you do like what's called a kit pull command to bring that artifact into your Kubernetes cluster um we do obviously the check of the Shaws to make sure that it is the same artifact from the registry that landed in the cluster. That's kind of minimum you got to be doing that. You should really have this thing be signed. So you can do a cosign verify

and say did this actually get signed? Are we sure that this is the thing that I want? Assuming that it p that all passes it just runs it. KServe can go and start serving it. Let's assume that it doesn't pass. Then the pod enters a crash loop back off and it just never gets to run. And you can go through and harvest back those and figure out

why did they break. Level two is now going a little bit further. Now you're talking about applying policies before you go and run. So obviously policies can be as flexible, as complex or as simple as you want. But you can imagine a policy that does something like says, "Hey, if this hasn't passed these three security checks with flying colors, it doesn't go to production." Or maybe if

it hasn't had hit these performance benchmarks, it doesn't go to production because we don't want slow and underperforming models in production. You can control any number of different things. You could have it check those compliance docs, for example. Is there is the license of the correct type or did the compliance include these particular checks that are required for this particular region then great if it does allow

to deploy to that region. If not can't deploy to that region same kind of structure it passes ker serves it it fails crash loop back off and you never get it to run. Level three is actually quite cool and something that um that you can implement in your own registry. We've implemented this in Jozu um is you actually have a policy that is applied at the registry

itself. This allows you to create a kind of quarantine zone and that can be used in a pretty interesting way and again it depends on how much security you need. But you could have a situation where um a model or a project ends up in your registry but you do not want people to pull it until it has been let's say security scanned or until it has

gone through that compliance review. while it is in your registry but before it has had those checks. You can have a pull policy that basically checks for that before allowing anybody to pull it. That's subtly different than the deployment policy because if you deploy you apply the policy at the cluster, people can still pull to their local laptop or anywhere else. This would actually stop even that.

So that prevents this spread of potentially dangerous um artifacts in the organization until it kind of gets a basic check mark. Um that's one that I we see kind of increasingly being used by by our users for for security reasons. Um especially with things like agents, MCP servers. There have been, I'm sure people in this room probably know, but there have already been quite a lot of

documented exploits of people using MCP servers and agents as a way to get remote control, um, privilege escalation, data excfiltration, allowing people to just choose random MCP servers and agents off the net and pull them onto the local machine, not really the safest thing you can do, to put it mildly. um having them nominate them and bring them into a quarantine zone in your container registry where

they can be checked and then moved into a more publicly addressable registry for usage by your internal teams is a much safer way of still giving them choice saying hey I want to use this agent or I want this MCP so I think it will help me but not giving them cart blanch to just go and run random things on your network all right let's get demoing

the exciting part. I'm finally going to show >> I hope you all are ready. So, I'm going to demo quickly how KOPS sits so well with Queso and how it will create your inference as a zero trust inference environment. So, let's get started. So, just full zoom it. Yes. So, initially I'm just checking whether my kits is installed. Well, it's very easy to install. It's a one

command installation. Next, I'm using cosine for all my attestations and verifications. So I'm checking whe those installations are okay. So I'm generating a key for me myself. So it'll be a public and a private key in cosign. And I want to check whether this public key is all right. Okay. So next I'll use my kit to actually import a hugging face model which is GPD OSS 20B

and actually pack it into kit ops using kit ops. So I'll pack this model. These are all the files you would see in hugging face. You can add any code documentation if you like and then I'll say yeah let's good to go. Let's pack it. So here it's downloading all the files and all the packing done. So I'll just skip that part and I'll show I have

two kit already ready. One is GPOSS 1.0 which is 12.8 gig and one is 1.0 unreed. So this is something which will be the unreed uh thing and we will show how when we deploy both how it will act on our So I'm logging into my Jou which is my registry where I'll host all my kids. So I'll push the first model and then I'll push the

second model as well. And after this we'll try start attesting our uh models with uh the cosign key which we have generated. So here I'm giving my name as the approval. So if something goes wrong they can fire me. But it's okay. This model is okay. So we can go ahead. So I have put on my private key to attest it. Yeah. So now the main state

I want to show that because we have two different models. What would be when I verify the one which I've attested and what would be the one which when I'm not attested, right? What would be the response? Right? So this one says yeah it looks good. Your claims are valid. And the next one would say that oh I've not found any attestations your claims are not valid.

So this is the unreed one. So here I I've not given my attestation. So I'll not get fired on this one. So it's okay. So now this is the part where we actually do the inferencing. So we use queso for this. So I'm creating a separate name space for myself. I'm adding my public key, the cosign key which I'll use for verification during the init container state

and I'm adding the script which will make it very easy in the demo to show that oh whether it has actually passed or not. So here you can add more policies more attestation verifications anything which you would like any level of your choice. So in this script we only check whether my attestation is okay and if it's okay it say it says passed and if it's not

okay it says failed. So this script I will embed with my in my case of init container. So in case of you need to do one extra step. If you want to get it from a custom repository right you need to add something called as storage container. So this is the case way of giving you flexibility to bring in any approach to download your model files from

any registry or whatever of your choice. So you just have to create a storage container pass in the variables you would need to and pull those models into queser and just give a prefix which you would use for all this uh in your case deployment. So here I'm using something called as kit as the prefix in my storage URI. So this part is actually deploying of inference

service. This is the case of CD. Here I'll have a init container just for my kit verification which I'm using the script which I generated from cosine and I'm passing the variables needed to uh make the verification. The second part is the main container which is the VLM container. So here if you see I have the VLM serve commands and I'm passing the kit prefixed model which

have been hosting in my uh Jozu registry and I have the VLM 17.1 which is one of the latest releases. So this will be my main container and I'll have a init container just for verification purposes. So here I'll be hosting this in one GPU. I'm passing my public key so that the verification goes well and then let's deploy this model. So as soon as I apply

this ISVC case of controller will pick it. It will start its process of deploying the model. So it will go through an init state first which is my verification state. If it has gone to the first state already the first state is actually verification of my attestation and the second state of init using my cluster storage container to download the files. So if you see the first

state lock says that yeah you're good. You have attested this one. you can go ahead and if you have any other policies you can put in here and that verification can be seen. So now it's on process to do the second state which is downloading it from the joer registry and at the same time I would deploy another ISVC which is the infant service with an unreeded

one. So the ISVC is the same the only thing I'm changing is the tag here and that kit per model which I'm setting. So which is that only that particular item is changed. It's all is same. So unreed is being passed and let me deploy another ISVC. So now I should have two deployments in caseer and the second one should be failing in the init state because

I'm not attested this one. I'm it's a no-go model. So let's check what my uh init container logs are saying. So it says that this one is not attested by your public key and it's not ready to go. So it's saying blocked. So this way your resources are not consumed. You can have a easy fail safe and make sure the deployments are like stopped at the first

state of uh if it's not at state. So this if you see the first one is saying okay and second one is in error it'll always stay in a crash back off it will not get deployed. So next let's see whe how my first model is looking after deployment. Can I like start doing all my inferences with this one? So it is in a running state now.

So in this right side uh terminal I'm showing the logs of logs. So it's running VLM 17.1 and it's all ready to take a inference request. on my left side container. I'm just running the OpenAI Python library for hitting you on chat completions and yeah, I'm just asking explain deep neural networks and it's all okay. So this is how it's so simple to start with your KOPS

plus case of integration and make your like AI deployments, AI inferences zero trust environment. Thank >> Awesome. Thank you. All right. Um, so quickly, uh, Kitops, although it is a sandbox project, it's got a huge amount of traction in the last couple years. It's a fairly new project, but it's already, I think, actually it's now almost 250,000 downloads as of uh, yesterday. Um, and this is, I

think, a good way to think about it. Uh, this is someone from the US Department of Veterans Affairs digital health group who said, "We see KOPS as the gate between development and production. Securing that gate is critical and this is a great way to do Uh, all right. So, things you could do to harve uh to harden Kerve next week. I didn't I was gonna say this

week and then I realized you're all going to be busy this week. You can do this next week. You got my permission. Um, so you can use secure packaging and signing with kit ops and cosign for example verifying it at your uh at your registry. Then days three to seven things get a little harder. Obviously, you can capture approvals. You can add security scanning um for agents

data sets and models. Add admission policy checks for approvals for scanned and safe models and pull all that chain of custody information from your model kit into a cryptographically secured audit log for example. Um my company Joe, shameless plug, does all that hard stuff for you if you want to skip having to do it yourself. Uh thank you. Um now questions. Uh for for questions, please line

up at the mic uh so that that way they can be heard on the recording and and streaming. I'll leave these up so you can learn more while uh while we chat. I think I beat everyone. Um so great talk. um we heavily use uh ker and um I've never uh I don't really have a necessity to do attestations but how would you what what is your

advice for like a MLOps team that is deploying a bunch of models and perhaps is is new to this workflow of attestations like if if even if we don't like are required like would you would you suggest that it's a good idea to just you know make sure that this is always the the path for production and is it just for human in the loop like like

I'm I'm looking at this as like somebody like in our case I have like hundreds of scientists that are building the models so should I ask them to push some CLI and that like I don't I don't get the yeah >> excellent question okay so I'm going to take kind of there feel like there's two parts to that so for your data science teams the goal with

kit ops is to make it as transparent as possible we don't want to make them change tooling so What usually is easiest is there is a something called the pi kit ops py kit ops um re repository which is an SDK. Uh what a lot of the organizations who are standardizing on kitops do is they will create kind of a default a notebook template or cubeflow or

mlflow template and it includes this kit ops kind of at the front and then at the very bottom and the the front just loads the library. the bottom actually does the packing of the model kit and pushes it to the registry. The data scientists just fill everything in between those brackets if you will um with what they normally do and they never need to think about it.

Um second question was really about addestations. I would strongly recommend using addestations for two reasons. One is that it keeps all of the policy and criteria with the actual model. And that's really important because that means no matter where that model goes, it carries with it all that approval, all the metrics, everything that somebody would want in order to make a decision about is this safe, is

this appropriate. There are lots of tools out there that ha that you know can do some of this stuff, but they're almost always going to force you to have a connection back to that tool and they're going to keep all that information inside their tool. That is less helpful because if somebody can't connect back to the tool or doesn't know how to use the tool, it may

not be obvious to them if something is unsafe. Keeping it with it super valuable for that reason. The other reason is addestations are standardized within the the um Kubernetes and container ecosystem. And so there are tons of tools that can read those addistations and make their own decisions. So you don't have to rely on only one vendor or only one solution. >> Yeah. Thank you so much.

>> Hi. >> Um thank you for the presentation. I think the topic of security deserves uh much more attention in the uh AI world and so that's why I think uh what you do is really important. Um actually I missed uh uh two things in your presentation uh um showed it a little bit. Um when you uh when you verified the weight and uh um created the

attestation um I mean nobody can actually verify the weight. Wait, this weight is 0.12 low huge issue. Um, you can actually verify it. >> I think we have to start at the training level where and we have to we have all the the tools in place. We have in toto >> and we we have salsa and we need something like salsa for the training level so that

you can uh actually atestate uh the model at the training level. >> Yes, that I could not agree more and you can do that like kit ops allows you to do that. You would just do your in total addestation. You would add your salsa and you would store all that with with kit ops as well. And I agree a thousand%. You should be doing it at the

training level and at the verification level and at the deployment level. >> You're just trying to make it simple by saying if you're not doing any of that, at least start at the deployment level and then kind of work back. But you're right that starting at the training is best. >> Yeah, it's great that you mentioned that and it's important to point out uh that this uh

is uh open. I mean, let's skip three cubecons ahead. Then we have solar winds with the AI world. Somebody is wipe wipe opsing uh something and uh yeah, then things will change. >> Exactly. >> Thank you. >> Hi. Well, again like like the previous two people, very nice presentation. Thanks a lot. Um you mentioned and I I saw it also in your official documentation or at least

for for KTOS I mean that you can also package among other things data sets right the the training or evaluation data sets in in a model kit I'm curious I I think it's still somewhere on your road map to do it for non-local files right for remote storage I'm curious how that looks like for you for large data sets as well so we have sometimes models trained

on terabytes of data for example >> yeah so so actually that was just merged um Uh it I don't remember if it's in the current version or if it is in the coming version, but you will be able to um pack essentially from S3 to the model kit. Um well, sorry, have a reference, pardon me, to S3 and have it automatically pull it down. What you can

also do in the meantime is you can always just add the reference as a file inside the model kit. It's not perfect. doesn't automatically then pull it, but at least it's there and it's part of your your your um requirements. The way that I think about anything inside of a model kit is whatever goes into the model kit because it has a Shaw digest because it cannot

be tampered with, it is appropriate for compliance auditing um for for anything where you need that level of confidence and security that this was bite forbite exactly the same as what I said it was at the time when I was okay with it. If you don't need that level of certainty, then just having a reference to a data set held somewhere else is fine. At least that

way you know there's a connection. What you can't say is that that data set is exactly the same now as it was when I trained it three months ago. Um, and so that's the thing you lose. It just depends. >> How are you doing? >> I think it's >> Hi. Uh, first of all, uh, thank you for the great talk. Uh I just really wanted to ask

right you know uh how does kitops would work you know uh in synergy with some vendor solutions that actually really also got quite popular right you know among the data scientists like ways of bias and so on so forth right you know have you also considered uh some potential you know uh integrations uh you know interface there and uh yeah just wanted to to earn a little

bit what's your strategy >> yeah we would love to talk to some of those platforms about making full-on integrations. In the meantime, what we suggest is each one of those platforms typically stores their artifacts just on a file system. And because Kit can be composed of anything on a file system, you would just have the kit CLI in like a pipeline say, hey, when there was a

change to this file system, go and move those things into a model kit and push it to the to the thing. So, you can do it today with automation. I would love to see like a weights and biases plugin for kit ops so that you could just say hey I want weights and biases but I want to use it with kit ops as my storage layer instead

of with a file system. So MLflow uh we are working on a uh on a on a storage system for MLflow um if you didn't want to use the Python uh approach that I gave you. >> Fantastic. Thanks. Yeah, >> I think everybody Yeah, I I just wanted to say thanks again too. I'm really happy that Kit Ops is here. Uh I do have just one simple

question. Have you looked at OS and how you could leverage that for the models? >> Uh os. >> Yeah, I think we have. Um, so I haven't myself. There's a different maintainer on kit ops who I'm 99% sure has been working with oras. I actually think that there's a uh feature gap that is preventing us from doing something we want to do. I I'd have to go

look at the repo, but I think that there's an issue tied to Orus now. >> Okay. I'll I'll uh I'll go on to the >> if you see me after, I can we can kind of check it together. Yeah. Thank you again for excellent presentation. Well, uh I was wondering because we can maybe attach data sets to the kittops. >> Would be possible to do like a

package of packages. >> Yeah. So today, so you you kind of can so within kit ops right now within the model layer there is something called model parts. Yeah. >> And so you can have a primary model which could come from a different model kit and then model parts are the things that you would add to that. And so when you pull that this this model kit,

it actually pulls from that one at the same time. The thing I'd mentioned about data sets for S3, it's going to do the same thing where you would have a data set in S3, but your model kit would know that it was linked and it would automatically pull the data set from S3 while pulling down the model kit. Um, so yes, we do kind of have that

right now. It is not generic. So you can't just say whatever is in model kit A is part of model kit B. Um, that is something that we've been talking about. I would suggest if you don't mind filing an issue in the repo, it would be awesome to kind of be able to point at that and say, "Hey guys, look, people are asking for this." >> Yes.

And also for instance, let's say that we have this um kit of or this package from packages. could for instance I just select on this index of these packages from this reference and just pull that part because let's say that I have the data set that was a model kit and then I have a model version and then I have this package of packages if I can

just pull the model instead of the full data set because >> the full data set yeah that's a that's a good thing to add to the nuance of your issue. Yeah, >> thank you so much. >> Yeah, thank you. No, I think these are exactly the kind of things we want from the community is what