Securing the AI/ML Lifecycle With MLSecOps: Open Source Best... Bahaulddin Shammary & Andrey Shorov

This talk focuses on securing AI and machine learning systems in production, emphasizing the unique risks that arise from their implementation. The speakers, Baja and Andre, draw parallels between MLOps and DevSecOps, illustrating how lessons from traditional software security must be adapted for machine learning. They introduce a reference architecture that maps different stages of the machine learning lifecycle onto an infinity loop, highlighting the need for security controls at specific control points. The discussion covers the roles of various personas involved in MLOps, the security risks specific to machine learning, and tools that can help mitigate these risks. Ultimately, they advocate for a shared responsibility approach to security throughout the lifecycle of machine learning systems.

Thank you for being here. Um, my name is uh Baja. I'm a lead AI engineer at Openext and my colleague here >> Andre Shoven, senior security specialist. Yeah, I don't want to talk with you about MLOps actually. >> Very nice. I see many people here. So, but how you nervous? >> I I was told there would be less people. So, uh we'll we'll see how this goes.

Um, very good. So together today with Andre, we're uh going to take you through something that matters more each day. So and that's um how to actually secure AI and machine learning systems in production. So as organizations rapidly adopt ML, we're seeing a new set of risks emerging and risks that simply don't map to traditional software. Um with traditional software it focuses on um code, you got

containers, configurations, but with machine learning the that attack surface um uh explodes. So now you're securing um training data um that can be poisoned. Uh you've got models that can be stolen or tampered with. Um you got pipelines that um retrain continuously which can introduce new risks with every um iteration. And you've got the inference endpoints which um when exposed um they're exposed to adversarial inputs uh

which can be designed to uh you know uh fool your model. So the good news is um we don't have to start from scratch. Um the lessons that we learned from dev sec ops like automation um shared responsibility and security as code um those those principles still apply. So, but what we need to do is extend them and that's what MLSOPS is about. It's dev secops for

the machine learning um life cycle. And this talk is based on a white paper uh that myself um and my colleague during our time in Dell and colleagues uh from Ericson um worked on as well as um the a IML open SSF working group. Um so let's get started. Okay. So I'll just quickly walk you through what we are covering today. Uh what we're going to do

is um show you one of those infinity loops that you've all seen when it comes to DevOps and MLOps. Um but what we're going to do then is uh introduce a reference architecture which you can map these stages in the um in the infinity loop. Uh and it can be something like a uh reference architecture or reference uh points um practical ones that you can use. um

in your uh work and and then we're going to approach it the same way we did with with the white paper. Um so basically we're going to layer um the personas so the people that are involved at every stage uh you know who's responsible for what. Um and then we're going to layer on the threats uh OASP threats and then we're going to show you some of

the security controls you can implement as well as uh some tool suggestions. Um so who is this for? Well, if you're here, uh, that's great. Um, anyone really who, um, works with machine learning, uh, whether you're cloud professional or, um, a contributor or an engineer, uh, this is for you. So, let's let's get started. So, uh, we started with, uh, this lovely diagram. Now, a bit of

background, right? So, we won't go too back for um, too far back. Uh, let's say with DevOps, right? It helped us ship software faster. Um so um it automated delivery and broke down silos, but security did lag behind. So that's when DevSec Ops came in and it integrated security right into the pipeline. And so it made it security was made then a shared practice. And now we're

at another inflection point with uh machine learning. So MLOps extends DevOps into machine learning. So now you're automating training workflows and operationalizing models in production. Now the issue is that machine learning systems they do behave differently as you're aware. Um they are data dependent. Um they are non-deterministic and they do change over time because you've got this continuous uh retraining. So drift can happen and so um

the system can keep evolving even after deployment. Um now looking looking at this diagram you have the uh DevOps loop you got the uh data loop and the machine learning loop and in production these loops overlap um so you've got data it feeds into models models models move through delivery pipelines and um monitoring and feedback they trigger retraining so it's continuous so ML suckups is what happens

when we secure that overlapping life cycle end to end. And the assets are not just code in containers. They include data. You got models and the pipelines that build and redeploy them. you can see what's different here is that we added some structure to the loop. Right? So when we were writing the um the white paper uh we started with a reference architecture where you can see

the different stages and you know where most um MLOps systems go through. Um so what we wanted is kind of like um you know I think most people can struggle with you're seeing the loop and you're seeing the reference architecture and there's it's hard to maybe link two. So what we did is we mapped first our reference architecture stages onto this loop so that you can see

the the controls at each stage and we ended up with nine primary life cycle stages onto this loop and that's from planning and design through to monitoring and that way we're giving each stage a name and a place and the reason for this is practical. Um so while the loop is intuitive, MLSOPS needs control points uh specific stages where you can apply those security controls and um

so yeah I I mentioned the reference architecture. Um so let's let's let's have a look at it. So this is the let's say level zero reference architecture. It's bare bones. We started with this and if you can notice is we kind of untack untangled those arrows you were seeing in the uh previous diagram in the infinity loop and we mapped it onto the reference So um this

reference architecture that now connects the infinity loop to real pipeline components and you got artifacts that flow between these stages and it's organized into three domains. Um so you can see the the planning you got the experimentation and development and then you got the staging and operation and the colors do matter as well. So with the yellow boxes those are those are manual processes and with the

blue boxes these are automated processes that can involve your pipelines. Um and also you can see um the flows uh with these arrows. So these show where things move. So you have data test, data sets, pipeline code, model artifacts, and metadata. And every movement is a chance to lose integrity, provenence, or access control if it's not protected. So here's the um the key. These stages aren't just

abstract. Each one is a handoff between people, systems and stores like the source repository or the um artifact store. Um so um every handoff is a control point. So MLOps uh like an MLOps process that um ignores security, it can end up having the same failure points as DevOps before before dev secups. Um so without a uh security layer it can also become a fast path from

data ingestion to a compromised model in production. And so this is what MLKOPS um addresses uh not by slowing down the pipeline but by making security a shared responsibility across machine learning um developers, security practitioners and operations um teams at every stage. Um so what we're going to do now we're going to layer on the first um layer that we say which is going to be personas

and I let my colleague Andre Andre um take over. Thanks Ba. uh when we talk about personas and ML cops in general uh first of all we should realize that uh ML cops is not about technology itself it's about people who doing how the process there's many people involved as you know that data scientists data engineers software developers and so on so Um every person should understand

security responsibilities and uh we should clarify who own what during all this process and all this architecture that Baha showed before. Um then what we do we introduce personas it helps organization to realize what people are involved what they should do and how it's relevant to security of uh MLOps life cycle in general. Um how we do this because we're working with open SSF we extended classic

opens personas and added ML people there. uh particularly we added uh MLOps engineer who responsible for the whole process how you set up all this uh CI/CD continuous training pipelines. Um also we added security practitioner who plays crunctional role everywhere and uh we will look to that uh later also on on mapping and but most important thing uh is that security in ML ops process it's not

just something that added in the end it is responsibility of all people a shared responsibility for every person on every And this what we discuss on the next slide. Okay. If we start mapping uh all our personas to architecture Baham presents, you see these icons uh human icons on the top of all every process. Um and uh if you look to left up upper corner so it's

ML design and there you see security practitioner uh software architect uh MLOps engineer so these people work together to ensure that security is embedded through whole life cycle what what kind of security should be there and following that I mean data engineering uh process you see data engineer and security practitioner that should work together and realize what kind of security need to for specific for this pro

process how to secure uh training sets test set how to inest data to the process and so on the similar for uh all other processes what is important here you can see security practitioner is everywhere so in ML secos process security practitioner pay all this function role. He ensure whole life cycle is secure and also security partition should come to every of this person if you work

in a company and discuss what exactly need to be done what this person who is not security probably explain him what this person should do uh to ensure the data all assets uh during the life cycle is secure. uh then we can go to the next slide and start to discuss risks to uh for ML life cycle. Um actually before we start to introduce security we should

clearly understand what risks are. is classic risk assessment process you know and we start with thread modeling and YML is kind of different a bit different from traditional software in traditional software as Baham mentioned it's where deterministic you write code you execute code every time it's kind of executed the same way and you see you have visibility like what happen and why it happens in ML systems

you have this opacity because you you train your model then it's data driven what data was there not always know and then you run model it's executed produce some results you're not aware why it's execute because it's a kind of black box to you when you run this whole ML ops process it's very difficult to figure out why the model in the end behavior wrongly why is

some drift or something and because of that you should ensure as a security practitioner that every all these processes they are kind secure enough to ensure have some confident confidence that there not because something happened before like at some adversarial uh poison your code or poison your data. problem here is that in general if you look to some sometimes uh companies like our team they oversee the

security of ML specific ML security and they run code scanning and harden infrastructure but there's ML specific threats that you also may know and often it's overseas ML specific threats like poisoning adversarial input uh some privacy leaks during API during inference or some supply specific supply chain like uh compromised notebooks or libraries. Um then uh we decided to use OASP ML top 10 as our baseline and

it's very good practical approach to have some kind of framework that's well known and very well kind of community supported because it gives to your team specific baseline where they can take they can discuss they can do threat modeling and based on that understand introduced to every step as well. It help to prioritize security controls that what what need to be done first and what need to

be maybe dep prioritized because not so urgent right now and it's allow you uh and your team to really practically implement security controls uh during your MLOP life cycle. Um then let's quickly look to ML uh secops uh of top 10 place to this ML sec architecture um the same architecture and you see this red circle there and this is how we map all the different security

risks uh from OAS ML top 10. Uh again if you look to to the design design phase we put like all threats from from input manipulation to model poisoning it makes sense because at this step like you security practice you form all security architecture you understand where security controls needed what's potential risk there during every stage and you start to realize what need to be done in

in every stage from security point of Then for example uh in uh data engineering stage you can face risk like data poison supply chain risk or in experimentation phase you can face with model theft and also supply chain um actually you can notice as well that supply chain risks they almost in every stage. It's also shown like how pervasive the supply chain risks are. It's very important

to track your provenence. Ensure that all uh assets are trusted and you uh following all like you're free from vulnerabilities. Um if you look to the model server um process you see it's kind of very high density of different risks. Um this is a point where your model is exposed in production phase and then adversaries active actively like probe your model and test your security there and

there's high density of different risk uh start from model poisoning or model theft uh in membership inference and many many many different issues there. but how actually do security? So this picture showed how we trans transform ML ops into ML sec ops. How we add securityities here and if you notice a dashed uh green bo boundary around this is your security environment. This is uh this uh

baseline from your basic security you have in your company. Um one example of this is well know is SMS you have uh or if your organization follow ISO 27,000 standards. This is this is security environment without that you cannot do all other things. Then you start to realize what kind of security I need for every stage and um again if talk about design here you do actual

planning based on the previous threat model and risk assessment actually in general you start to realize okay I need security controls like this yeah it's of course CI/CD3 plus C people from Microsoft know it's very well secure by design and by default in uh deployment plus communication risk assessment of course and also important step is and we talk about later bit uh security awareness and I mentioned

personas you should as security practitioner should talk with people with all these professionals explain what security is why it's important and provide all information necessary to protect all the stages then in data engineering for example you have this trusted source It's one of the biggest maybe and less let's say clear way what kind of security should be there. It's very important to ensure with the data you

ingest it's secure. U you cannot just randomly download any training set and just run it within your your experimentation or continuous training pipeline. Then in data in experimentation you need to protect your models. you protect these assets, you should trust them and and following in CI/CD pipeline, you do security assessments. It's classic. So um and version control uh and so on in motor servering you go to

operation phase and operation also uh a lot of stuff that you know you can you can do uh login monitoring uh incident response and all this operational stuff. Then uh again very important point here that it's not kind of security is not kind of just process that you you added in the end um it it's starting from really design it should embed it at shift left like

it where no concepts uh safe left security you start security from the beginning and you making sure all people know about that who participates and all people all team work together to ensure all security controls are in So like Baja, do you want to talk about uh tools? >> Sure. Yeah. Uh what a ride, huh? Um so that's that's the approach we took in in our research

is that we're layering on um one by one. And in the paper, we keep all the previous icons that we had, but um to save your eyesight from uh exploding and seeing all the stuff. So we only added um one at a time. So um we in our paper we we frame it in in the in the way that OASP um helps you define what to secure

against and then open SSF uh provides you the means. So particularly uh for the um a IML supply chain. So on this diagram um if you can see the the the green circles um in this case that would be the OASP tools applied at specific stages and the stars are opens open SSF tools uh providing supply chain coverage across the um life cycle. Now in in our

research and in our paper we're not really focused on um adopting everything adopting every tool but let's say for for the sake of this we're trying to cover four outcomes and that's integrity provenence repo hygiene and um traceability. So take for example sig store um that handles signing and verification and in ML suckups that includes model artifacts and not just container images. um in the case of

let's say salsa um it gives you provenence at stations so you can prove how something was built and what were um its dependencies and then you've got um scorecard and um allstar and these harden your repositories and they reduce upstream supply chain uh risk before it even enters your uh pipeline and um then there's guac I'd say that's probably the least familiar tool here and um I'd

say arguably the most interesting um for machine learning specifically. So, Guac um what it does is is that it connects um artifact and dependency graphs across your uh pipeline. So, in practice um what that means is that you can inspect lineage to help connect a bad outcome um all the way back to upstream inputs. So you're able to identify um which models may um need retraining uh

when a data source was found to be um uh compromised. So um and that's you know that's what you know you turn it into a root cause once you can solve that mystery um through let's say use of uh guac. And in our paper we've highlighted a lot more um tools and we focused on open source ones uh for the most part. So um check it out.

But one thing we do make clear in our research and that's something we found is that um most of the open source tools that um we found is that they they were designed for software supply chains um but they do need extension for uh to cover uh machine learning specific artifacts like model weights and um training um training data provenence. So um the foundation is strong but

the machine learning specific layer um that's still being built and um yeah so what we wanted to do as well is um show you the final uh diagram. So that's how it looks like. Um so we didn't want to keep all those um as we go but basically you have kind of like a heat map or a reference at every stage what you have in terms of

security controls what are the threats you're seeing at every stage and the uh the relevant tools. So even if a tool you like is not there you can find something similar um to um to implement in your in your flow. Um so what is the key takeaway? Well, ML secops is now one control or one tool. It's a structured way to secure the entire machine learning life

cycle and that's by making the system visible uh making responsibility visible as we saw with the personas mapping and then aligning threats, controls and tools to the same control points. Um all right. um four uh key findings and conclusions. >> Yes, >> that's yours. >> Yeah, we we can close our presentation and um yeah, we want to talk about challenges, recommendation and uh listen quickly. So recap

about challenges. So we have ML specific threads that also should be taken into account when you build your system especially ML based. Then we have challenges with retraining. You retrain your model. You're not really aware what happened there. Maybe there's new vulnerabilities introduced. You should take it into into account. Model is very difficult to audit especially if they are uh some some big models. It's very difficult.

You don't know the structure and so on. And it's very difficult to track prominence um and do responsibility. It's a lot of different dependencies. That's where kind of is is a challenge really. Um if you talk about recommendations so um you should automate as much as possible. Uh one big challenge uh to close about retraining. If you have continuous uh retraining loop uh you cannot rely on

manual reviews. It's just not feasible. You cannot review every model after every retraining. So it should be automated. um try to version everything um data sets and yeah it's expensive uh models and so on it can give you some kind of confidence that maybe model drift not because some attack or something um and also u it's very important to embed security roles within uh ML ops uh

teams so security from the beginning and lessons learned is ML Mlops it's not kind of replacement it's it's very important to understand not replacement of current security just supplementary you have your security already you have data security whatever and you already you don't you add in the this ML uh specific things there so uh security should be shared about all team every team member know what to

do and it's also important things and if you integrate security early It's also will help you to reduce cost when you need to remediate when you have some accident. If you know what happened before and you know this this this thing already your data is already secure. So uh it's not because of your data it's much easier to find the root cause and uh we found that

still gap in ML ops and ML security in general. So you should invest in your team and you should invest in training and security awareness. What happens? why it's important and this is one of the thing you should you should do if you want to ensure your mic ops process secure. So this is it from our presentation. Please check out our white paper and feel free to

contact authors. >> Yeah, it's uh it's the end of the day. So thanks thanks for coming and thanks for um uh being here. Uh please check us out on LinkedIn. Feel free to connect if you have any questions. Um uh we're we're happy to take any if anyone has any. I think the mic is over there. And uh yeah, thank you.