About this talk
In this talk, the speaker, Gurcan, the CTO of Jozu, explores the complexities and challenges of current AI and machine learning supply chains. He outlines the typical ML Ops workflow, highlighting the lack of transparency regarding the origins of datasets and models, exemplified by incidents involving compromised Docker images. The session emphasizes the need for better security practices, including the use of signing, verification, and provenance tracking. Gurcan introduces tools like Cosign and the KitOps project, which aim to enhance security in AI workflows through standardized artifact management. He demonstrates a new workflow that incorporates packaging, signing, and attestation for AI models and skills, as well as how to leverage a policy engine for ensuring secure artifact execution. Overall, the talk advocates for a shift in the AI industry towards safer practices similar to those established in traditional software development.
Full transcript
So, my name is Gurcan. I work for a little but mighty company called Jozu. I'm the CTO of Jozu. And today we're going to have a quick look at how the AI ML supply chains actually work today and how we can make it better. this talk is going to be a lot about me complaining. All right. So, what you see on the screen is your typical workflow
ML ops workflow today. You curl into something Hugging Face and S3 bucket Google Drive something and then you retrieve model bits or you retrieve some unknown skills or some MCB server. And then you take those model bits curl into to Hugging Face again. You retrieve some data set which you don't know. But trust me, Hugging Face has never been known known to have any bad bits in
it. do your fine-tuning. You train your model. And then you deploy. It's easy. It works. And the sad thing about this is that's actually normal. Like that's how it is done today. Most Most organizations have no idea where the data set that they have used for training come from where like I have actually seen this. There was one model that they were deploying to production. This was
an ML It was named after the guy who trained nobody knew him on the group that I was talking to. So, they weren't sure where it was coming from, but it was working. So. Ah. I couldn't figure out this yet. So, the sad thing about all of this is we actually seen this before, right? This This actually happened to us before. Remember 2014? When we were happily
creating Docker images from Ubuntu latest. you know, we we we did a very good we had a very good run for like a decade with that. And then what happened? SolarWinds happened. When the SolarWinds happened, everyone started to realize that hey, you know what? We cannot just say from Ubuntu. There was one I don't know many of you remember that. There was one Docker Hub account called
Docker 123 something. Uh and it had 14 or 15 Docker images with the legitimate name. So, it was the account was Docker 12345 slash Terraform. And that one actually had 5 million pulls. And all those images were compromised. that's the point where the industry actually realized that hey, you know, we can't actually do this. There is something called provenance and attestations and so on and so forth.
Let's find a solution, a better workflow for it. What is What is our workflow today when we are doing microservices or or software? Workflow is we sign them, we attest to them, we verify them, and we gate them. Like we have tools like cosign that does the signing, that does the attestation. We have Again, cosign do can do verification. OPA, Cuerno can do gating as well. So,
all of this tooling actually exists. But does it exist for agentic AI and ML? This is already happening on the agentic AI and ML, right? In the last 18 months only, there have been these incidents. For instance, there was one that was very recently published by Snake. They looked at the claw hub and look at that the skills and realized that there was Their number was when
they published it was 76 malicious skills. How is easy is it to write a skill? It's a markdown file. How many of you used a skill? How many of you looked at what it is What's inside it? You're the good ones. Another one is the MCP remote. I love that one because that one basically gave you a bogus OAuth endpoint. And with that, it could do any
remote code execution. There was another MCP server called past man. It could it actually agents could use that MCP server to send mail. What it did is it actually BCC'd a copy to another address. And these things actually had enormous number of downloads. We don't know what the usage and what the what the impact is for most of them. So this is already happening. It's just doesn't
make the news with all the other AI news. And one thing that you may have noticed, the attack surface is not just the It's not just the MCP. It is not just the skill. It's everything. Everything that goes into your uh agent tech workload is affected by the supply chain. one thing that when we thought started thinking about this problem and we actually started thinking earlier than
everyone else. Uh we noticed that hey, just those tools actually exist. You know, we we do have OCI artifacts which have give you access to six store cosign. Uh you can do verifications using cosign verify. Um the only thing that was missing at that point was to come up with a standard layout for the agent tech AI artifacts. So we went to CNCF, created a project called
model kits, uh kit ops, sorry. And in as part of kit ops, there is a CLI and an SDK, but also a specification for the OCI artifact. So the layout of the to fact is specified on a CNCF project. Actually, in two CNCF projects because later on uh there were a few more companies interested in this. Um Docker, Red Hat, uh ByteDance. So, we formed a specification
only project called Model Pack, which is if you look at it, they are pretty much the same uh specification uh with a few changes here and there like the way that the config is handled is different on Model Packs compared to Model Kits, but the rest is is pretty much same. But if you go and use the Kits CLI, it supports both uh because it's just a
single if sentence that we needed to add. so, the solution was actually very elegant. Like whether you have an MCP server or an agent skill or um a model, you could use the same registry, same infrastructure for signing things, same enforcement, and uh you could get a end-to-end supply chain that is as secure as as um existing software chain. So, what did the workflow come to? Uh
the first workflow becomes pack, sign, attest, and we'll talk about attest a little bit more, gate, and run. So, that's that's the new workflow for uh your agentic AI. I'm going to switch to demo. Because it makes it real, right? I don't know if this is visible. Okay, so uh in order for you not to, you know, watch me fat-finger CLI commands, I have some scripts that
I have written. They are they will look like I'm typing and it is real typing. It's just scripted. Uh so, if you are wondering how I am able to type that fast with a single hand, because I'm not. So, So, I'm going to just run that script for you. Um what the script does is it actually types these things uh as I as I press press one
key. So, nothing fancy. But, the rest of the execution is real. Um one of the things that I did on the script was to make it more obvious. I actually use some environment variables so that you don't have this very long, hard-to-understand command lines. Uh you can understand them better. So, this is the actual values that I'm substituting. So, you can so that you have in like
you know what it is real. This is a demo about packaging uh a vision model, which we called creatively Northwind, because I'm from Canada. and it it has what you may expect. Uh there is a safe tensor there, so that's the trained model, and there's an eval set of JPEGs, uh some inference YAML uh for configuration, a Python file for config for inference, stuff that you can
expect. The only thing that is unexpected here is the kit Uh the kit file is uh Dockerfile equivalent for KitOps. It basically tells where the artifacts are when it is doing the packaging. So, and also some metadata about, you know, the model and so on and so forth. Most of this file is optional. Uh you need one layer, which is either a model or a dataset or
a code or a doc or a prompt, whatever you are packaging, you need one of those. You can have multiple, but you need at least one. Uh the big difference between a model kit and a Docker image is a Docker image is a additive uh file system. So, you cannot just unpack single layer. Model kits allow you to unpack a single layer. So, I can actually say,
"Unpack content classifier {dash} v3," which is the name of the model. It will just unpack the model and not the datasets and and the other layers. Uh this is this becomes, as you may imagine, this becomes critical when you are dealing with different lifecycles. Like, in eval time, you want the dataset, but in inference time, you don't want that terabyte of data on your cluster. And when
I run the kit pack command, it will do a very familiar thing. It will spit out something saying that oh, I packed this layer, I packed this In this case, it is it is saying they are already saved because as you can imagine, I actually ran this demo before. Uh so, uh it is saying that those layers already existed in my local machine, so it just didn't
save it. It just uh ref- referred to those bits on the manifest essentially. So, saved some storage. And I'm going to push this to my registry. The registry uh you can see that it says kind.cluster. Uh it's actually running on my localhost. Uh I just did a DNS trick to uh map the localhost. And now I'm going to use the cosign to sign it. You know, you
may have seen this before. Let's sign verify it. Nothing nothing, you know, it's all working on it as expected, and that's actually the point. now I am going to do an attestation to it. Um remember we talked about the workflow and there was one attestation side of it. The attestation that I'm going to add is something called scanner results. Uh imagine that you are running a code
injection scanner like model scan or pickle scan. where do you put that data? Today, for instance, hugging face does that for you. Uh so does Joshua Hub. The difference is hugging face keeps that on hugging face side. Joshua Hub adds that as an attestation. When it is an attestation, the advantage of it, and we're going to look at it in a in the second scene, is you
can actually make decisions by looking at that attestation, whether you want to move something to production or drop it at that point. So, um now attaching a uh file named scan results JSON as an attestation, so what this will do is it will uh package it, sign it, and push that to the registry. And when I download the attestation, I will be able to read the read
the contents, and also verify that it is coming from a valid uh valid uh signer. And then, does this work for skills? You can Can you package your skills to model kits? Of course, you can. Um this is a typical skill. Uh I don't even know what I have written here. Uh so, I hope it is good, but let's package it. The kit project has a init
command that will basically create a kit file for you. I'll just do that. I'll just pass these, and here's my skill, the kit file for the skill. You may see that the descriptions are coming from somewhere. They are actually coming from the YAML front matter that was on the skill.md. So, kit basically just does most of the uh packaging the skills. And the rest is, as you
may expect, package it, push it, and so on. So, so far, what we did is we did the production part of it. Like, this is where how you package your skill, how you package your model, how you package your MCP server, right? You package that into a model kit, sign it, attest it. If you have attestation scanning, provenance, we'll talk about it. Um data that you want
to attach to it, you can actually put all of that to that single package. And the the that package lives in your registry. So, it travels with uh with your model kit between registries. Or registry to where you're running it. So, now I'm Now, we're going to consume this just for full disclosure, I have some environment variables which I'm using so that we don't deal with a
very long command line. Again, I'm going to do something different here. I'm going to package another model kit, but this time uh as you may guess from the name, I'm not going to sign it. Another thing that I'm going to do after pushing is I'm going to also the same model, essentially, uh to a model kit, but this time, as you can guess from the name, it's
compromised, which means that I'm going to put an attestation that says it is compromised. let's start with the uh compromised one. Let's sign it as well so that, you know, it is real. It's done from It's coming from your build system. You signed it. It is correct. But then your pipeline kicks in, does a scanning, and discovers there's a code execution problem. Or it scans your skill
and discovers that there is a prompt injection in it. remember I talked about the attestation? This is the attestation that I have attached to the compromised model kit. One thing that you will notice, oh, this looks hard, but if you could see it, there is one line in there that says critical. and there is number one here. The attestation format we use at KitOps, uh and you
can have your own. So, this is in no way uh I'm not saying that this is the standard way of doing it, but it basically says, "Oh, I found this, and it is critical. This is the number of items that I found that are in critical level." Uh we kind of normalize the scanners into an attestation. And there is a uh explanation for it in the details
saying that a suspicious pickle operation OS system invoked via via dash dash radius. So, next I'm going to put this attestation to the uh compromised image uh com- compromised Verify that it is the correct one. Um um going to verify the unsigned one first, figure out that oh, it's unsigned, so I it's not safe for me to use. And now this is the interesting bit. Remember at
the beginning I told you about, you know, Cosign has this verify command that you can use. The verify command in Cosign can take a Q like the same language that OPA uses for policies. So this is the policy that I'm using. As you can see, it's actually requiring the critical to be zero, high to be less than five. So I'm going to just run the verify and
do some JQ magic so that you we can see what's happening with it. it basically said pass. Did I verify the wrong thing? Yeah, I verified the correct one, which was the earlier one that we had we we had. And now this is the compromised version and it basically says, yep, it is failing for us. So uh again, all of it is open source. Like Cosign is
six six store CNCF carries uh GitOps. OPA Q is part of OpenSSF now, I think. Uh OPA is CNCF again. So this is all part of the the open source ecosystem. And you can actually do this for your ML Ops AI workflows today. There isn't anything that you need to wait. Like open source way of doing this. Although the verification part is a little bit stitched together
right now, but otherwise it's it's pretty working pretty well. So, the one part that I never liked about this is the verification and gating because you are depending on cosign verify command. It works fine. Uh but it's not as easy as OPA or Kwerno. And OPA and Kwerno cannot do this because they are very specific to Kubernetes and they are very specific to containers. So, the way
that they work is they need to know essentially a a resource, a Kubernetes resource so that they can uh figure out where to get this from. And most of the time AIML doesn't have that. And most of the time you're actually running your skills on your laptop. So, you need a different way of just show uh something that we have been working this is uh this is
this is something we call the agent guard. Uh it is essentially a micro VM uh that runs your agents. Uh whether that is cloud code or open cloud or whatever your poison is, uh it can it can run them. But, one of the big things that it does is it actually respects the secure supply chain. So, I'm going to show that part. agent guard we do have
policies and agent guard guard comes in with a policy engine embedded into it. When agent guard starts, it basically does signature checking, artifact and uh attestation checking as well. And one of the ways it does it is it has a policy language. And if you are familiar with the Kubernetes way of doing policies, this is Kubernetes uh very close to the Kubernetes how Kubernetes does policies natively.
Uh it is CEL language. Uh we chose that because uh we could uh compile these policies at uh load time and we wouldn't have any overhead at run time. So, this this this has some run time uh advantages for us. The another thing that you can do with Agent Guard is you can add a policy for the host. Uh if you're running it on your laptop, you
can say that, "Hey, I don't want you to install anything from GitHub container registry." now I'm going to I had some uh agents that was prepared before. I'm not going to go into the details of that. The purpose of this demo is that you should you can see that this this actually can work for any kind of environment. I'm going to run an unsigned uh which is
a model kit essentially. And I have a public key attached to it. So, what Agent Guard guards, if it is still working, uh default behavior is to verify the signature if there's a public As you can see it because it's not signature, the verification fails. Agent Guard never starts. So, let's look at an agent definition. This is what we are loading, essentially. And one of the things
you will notice that there are m- modules in it. One of the modules at the end is called an untrusted MCP. And it is coming from a random registry. So, I don't know that MCP. And a policy that says that registry cannot be trusted. So, let's try to load that. it goes through it. This time it is a a signed um model kit, so it has no
problems with it. But then, when it tries to verify the actual MCP, it doesn't like that it is coming from GHTR. Uh and it cannot verify its signature. So, it basically fails at And finally, let's show that it can actually run. After failing so many times. So, the rest is the same. Uh this is pretty much how I use my coding agents nowadays. It As As you
can see, it is in YOLO mode uh because I have um policies that are attached to it. Uh the Agent Guard has guardrail and host tool policies. Tool policies limit what tools can do, including what kind of parameters can go into them. And the guardrail policies can either drop or redact uh what goes to the model itself. So, you can actually drop things like secrets, privacy for
data Uh with that, let's go back to our slides. We briefly mentioned that what happens with the provenance, right? It's like where does these things are coming from? And the big good example of it is today when you go to Hugging Face, you don't know. Like there is 450,000 people on Hugging Face. And you cannot expect all of them to signatures into what they are doing or
have, you know, they they will not put, you know, this is the data set I was using to train this model and that sort of information. So, the best you can do uh is at the time that you are doing the import, uh the Git command has a import command, which you can import models directly from Hugging Face. At the time of the import, you can actually
uh put an attestation saying that I have imported this from Hugging Face, from this sha, from this repo. So, that kind of gives you a provenance data. But, provenance is a very hard problem on AIML because in normal Docker images, provenance is linear. Uh you are always extending from uh or something. Uh in uh AIML, uh you are doing, let's say, uh you are doing training. There
is a data set and there is a base model, right? That's the your fine-tuning basically is a tree of dependencies that you need to walk back to to be able to verify the provenance. So, you have a a a base model which is trained by certain data sets, which is trained which is created out of certain other data sets, for instance. So, the it becomes a tree
that you need to walk to be able to prove the the provenance. So, the ML provenance is are a little bit uh more uh harder to do, uh but um and there is no tool for it yet. I'm going to be honest with you. Uh this is not easy. when you're doing uh scanning for 40 GB data sets or 40 GB models, uh storage is a problem
because every time you're doing uh scanning when you're moving a 40 GB piece into the scanner, doing the scanning, and then doing the attachment, discarding the storage. So, it is uh it is a bit of a hassle. And then 40 GB is nowadays not that big. Uh there are larger out there. The other problem that you always face again uh our experiment i- i- is our experience
is the experiments. Uh some organizations like to keep the checkpoints on experiments uh uh in model kits. That is a bad idea because there is so many experiments. you need to be selective about it because you don't want to promote every scratch build that you have done on software side, and then therefore like we're not pushing every PR to the model to the registry Uh the same
way experiments are works the same way. And we talked about the fine tunes and the provenance chains to wrap it up when it comes to what we are trying to do uh is to get the ML ops and AI uh supply chains to the same workflow as software which is sign, attest, and gate. And today, we do have enough projects and open source tools to accomplish that.
Yep. So, it took us 10 years to do it for the AI is faster. So, hopefully, we're going to just do that in in an hour. Thanks.
