About this talk
This talk covers the importance of red teaming for large language models (LLMs) and introduces Aries, an open-source project aimed at assessing their robustness. The speaker discusses the evolution of LLM applications beyond simple chatbots into critical business tools, increasing the necessity for rigorous security evaluations. Red teaming involves systematically probing LLMs to identify vulnerabilities while considering multiple attack strategies and objectives. Aries is designed with modularity in mind, allowing for tailored assessments and a flexible plug-in ecosystem. The speaker highlights various attack goals and strategies supported by Aries, including multi-turn attacks and privacy evaluations, as well as avenues for community contributions to the tool's development.
Full transcript
[music] Hi everybody. Uh so my name is Judo. Uh I'm a researcher at IBM and uh in today's talk we're going to talk go over red teaming elements. Um we're going to give a focus to one of our open source projects uh Aries but also give a bit of a general uh flavor of the area and the kind of problems that uh you encounter and difficulties there
and maybe hopefully encourage you to use Aries to to uh to assess the robustness of your LLM systems. So why red team LLMs? Um now with LM being deployed in beyond kind of just chatbot applications and being used to as coding assistants to call MCP tools in business critical workflows the security and robustness is becoming more important. um you know when they were just limited to perhaps
chatbot applications only it was perhaps more limited the damage that a subverted LM could do like it could be mean to you for example it could say offensive things um but it was unlikely to uh like cause immediate dollar damage if the expected workflow was was hijacked now I think the the the they're being used so widely in so many different situations that the story is a
little bit different and so there's been a lot of efforts into trying determining how robust LLMs are to uh arbitrary uh uh hijacking by by adversaries. Um this is kind of building on previous research in robust um uh machine learning kind of this is a new field that's just emerged with the rise of LLMs. uh this has been going back many years but now with the rise
of LLMs the uh it's really matured I think from going just uh from a pure academic and pure research exercise to now actually having a lot of uh practical real world use cases now we're dealing with with real hijacking and with real um uh systems uh rather than just academic papers the general flow with these things at least traditionally is you have some human redteamers uh try
and probe an LLM and find some weaknesses And generally speaking, I think if any if any of you guys get given access to an LLM with with a a chatbot interface, odds are without too much effort, you'll all be able to to hijack it and break it and and subvert the the safeguards. Um the difficulty here, I think, is is twofold, which is which introduces the the
motivation for more robust tooling is one a case of scale. Um, we're not particularly interested when doing these kind of uh assessment evaluations in finding just one jailbreak or one prompt hijacked. We want to be able to systematically discover classes of them and find them uh find thousands of them to train downstream uh uh detectors and safeguards and guardrails. But then also it's the second one is
a difficulty of what kind of jailbreaks and hijacks will humans find naturally just by prompting the model. You'll find certain classes of data distributions and and uh uh types of attacks but humans won't naturally gen generate um classes of attacks that are based on like optimization techniques. uh he will never find these seemingly random token strings that algorithmically you can start finding. when you run these evaluations
is a couple of things that you need to try and consider. Uh partly is how can an attacker interact with your system like is it just as we've been discussing through pure chatbot interactions like okay you type something into a text box in a screen uh or is in your particular system URL relying on MCP tools that could have hijacks inserted in them uh rag documentation that
has been manipulated in some way that or now now there's skills which gives like another potential attack surface for for for an adversary. These are all potential injection points um that an attack can start interfering with. So it really depends like in your system what is relevant. Um the more points there are, there's more things to check. It's unlikely you'll be able to to to you'll have
some kind of budget with with these red teaming exercises. Um so you need to motivate your red teaming exercise with a with the attack and model for your system. Um and here it's also important to distinguish between the attack strategy uh and the attack objective. So the attack is what is the attacker actually trying to obtain from your system. So this could be um you know some
malicious code for example. It could be trying to subvert the tools that are being called. Um on the other hand the attack strategies algorithmically what's going on behind the scenes for uh the actual uh subversion of the uh LLM workflow. Um and it's important to distinguish with them because sometimes they get mixed up but they this is kind of a cross productduct kind of search that you
need to do which is where some of the scaling problems come in. Um you know here's like an example plot right like you can have lots of different types of attack objectives like uh write me some code some malware code uh leak some private information whatever you it might be and there's can be many many different attack strategies. So if you just check like a single class
of attack strategy like the crescendo attack um great your model might be very robust against this attack and related attacks in that family um which are based on like using LLMs in a uh attacker uh kind of role playing scenario um but you might miss like optimization based attacks um which your model might be weak against. So this cross productduct is something that you need to kind
of consider when doing your um LLM evaluations. Um and again this this this scaling problem is something that then motivates like having a good attack model like for you what is important um and what will inform your defense design the most when you're running them because very unless you have infinite budget which you rarely do for security um you're going to need to pick and choose and
and deploy the safeguards appropriately. Um [clears throat] when you do run this and there'll be uh when you start running these tools and you start jailbreaking your LMS and you find these weaknesses um map these risks that you find in these jailbreaks to um known risks and threat models. There's lots of different kind of things that you can use here to start gauging the severity potentially of
these things um and mapping them to what is relevant in your use case. Um, I think it's something to bear in mind like not all threats are equal in this. Uh, if your LLM does not have access to private information, then the fact that it could in theory leak it is not maybe something you need to consider. Um, so like what does your threat model in this
situation say? uh a very generic all corners threat model that says like okay I just want my LLM to be robust against everything and never do anything bad is not very helpful in the uh design and uh strengthening of uh of the system. Um be specific because that's then ultimately what's going to inform the uh research design and development of your safeguards and which is ultimately what
you want to try and improve when doing Um these safeguards will always obviously have some kind of overhead, some kind of costs be it compute costs. Um there could be uh like false positive rates uh go up. Um so you need to deploy them in in a in a manner that actually extracts the maximum amount of benefit for you and your system based on the kind of
uh uh risk profile that you have and the vulnerabilities that you find in your LLM. >> [clears throat] >> So hopefully this gives a bit of an overview of the the uh kind of state of play here. Um so to try and and and do all of this kind of stuff, there's been a wide range of uh tooling that's been developed and deployed. Um a lot of
it is open source, some of it is closed source. So kind of some projects that have uh emerged out of the open source ecosystem um are some of these ones. Um this is we're going to focus on our own one, but this is just a highlight. There's a lot there's a lot of work going on here. Um and a lot of these ones have some great uh
aspects to them. So, you know, by all means, uh have a look at some of these things after the talk. They might be what you uh uh are after. Um but we think anyway that they they have some shortcomings, which is where Aries, which is our system, hopefully tries to improve. Um some of these kind of uh difficulties that some of these tools have is around um
mapping some of the attacks that are included to um known threat models like OASP or ENIST or mitro risk frameworks. Um some of them are like have variable evaluation methodologies which makes comparison between different attacks uh difficult. Um they can have various difficulties in terms of deployment and scaling and customization. So what are we trying to do differently with Aries and why do we think it's a
good uh a good approach to the problem? So Aries fundamentally we're trying to motivate it with um uh the idea of modularity. So Aries has a very small lightweight core component. So at its core there's just four small bits which is the glue that holds the overall red teaming pipeline together. Um so there's the target which is like the actual endpoint that you're trying to test. uh
the goal. And so this is okay, what are the attackers trying to actually achieve here? And how can we expand and enrich this um uh data distribution to have good coverage, the strategies, which is where the actual attacks live, and finally the evaluation. Outside of these four components, there's then the plug-in ecosystem, which we're developing. This plug-in ecosystems should um lower the barrier to entry because if
you have a contribution that you want to add to it, it can essentially live independently of the rest of the system. Um and can also mean that you can very can tailor areas to your use case um very specifically. you don't necessarily need to load a hundred different attacks uh uh and and deal with the overhead of managing that overall software system if you just have some
specific things that you want to try and test for in your uh threat model in your risk profile. Uh and so you just need to deal with that. So to go into some of these uh components in a bit more detail, we'll drop the target because the target is essentially just some um uh lightweight glue to talk to the various models that you might be loading uh
and go into maybe the more interesting components which is the goals, the strategy and the valuation. Um so the attack goals um at the core they're going to be some seeds that you have that represents the high level intent of the attacker. Um so this can be like okay generate me some malicious code but you want to scan for more than for example that just just that
one uh particular prompt. Um so this can be variations on that uh intent. Um different styles of like attacks that you can uh run within particular attack families. Um this can also cover things like um not just say domain specific behavior in terms of malware but also like okay well what if your LLM is interacting with um different tools and rack documentation and back uh and and
different types of back end. So there you can have lots of different like goals here that specify what are possible things that the an attacker might want to do or I would like to prevent from happening um at a high level. We then have the strategy which is where all the um actual like mechanics uh live the the algorithmic components that run. So this can be different
kinds of attacks um from obfiscating the the attack request u running some kind of optimization um composing different strategies together uh running multi-turn attacks so so gradually steing conversations in particular ways. Um, and so these these are largely based of like different academic papers and research that's been going on uh uh to really try and give you the the breadth of um coverage you need to to
have a good idea of how robust your LLM system [clears throat and cough] And finally, there's the evaluation. And the evaluation one um can rely on a few different methods from relatively simple lightweight methods like just doing string matching on the output to more involved and more lengthy analysis uh based on LM uh LMS as a judge and variations of it uh or specific plugins based on
say checking for privacy leaks and risks. Um the evaluation is kind of important because the the the NLM's outputs can be so varied that actually having a consistent and robust evaluation pipeline is one of the difficulties actually in in having this um how if the LM's output is in itself obiscated uh then you need to have like a good way to check that actually what you're doing
uh uh is has been successful or not uh to to get a good idea of how robust the actual underlying system is. So uh when you boot up Aries you will see something like this which so we have a few different ways you can interact with Aries. Uh there's like a CLI there's some um uh APIs that you can code up in Python directly. Uh or we
have this nice gooey interface where you can drag and drop different configurations and uh load up the plugins you want. Uh so there's kind of hopefully something for everybody to to to use the system. Um we'll give in the last couple of minutes uh an overview of maybe when you run Aries what might you want to try and and do for and uh and check for. Um
so this is for example uh based off a hackathon we ran a couple of months ago but we set up an intentionally uh uh lightweight small system that we can probe uh we can run attacks against. Uh so in this scenario we have uh an agent which can talk to have some tools. Um and what the attacker is trying to do is say like hey um give
me some information that belongs to a different user. Um the LLM should refuse this request. Um and the what the attack is going to try and do is break this alignment. Um so so here's for example like what should happen. We say uh you know hey what are my recent transactions? the uh uh LLM calls the appropriate tools and returns you okay this is what you you
bought recently um if you ask hey give me the transaction of a different user it says the alignment uh is correct and says like hey I can't do that because you're not the right user um when we can we can run like a one of the Aries attacks in this scenario which um called cresender which is a multi-turn attack so here what happens is the the attacker
LLM will gradually steer the conversation um starting from like an innocuous question uh and gradually persuade let's say the the uh LM that's being uh subverted um to to reveal the information you want. So here eventually uh we get the transaction ids for for this different user. Obviously this is like a simple demo system. We've used aries much more extensively than this um both on uh like
open source systems but also uh closed source internal systems to to um to really like show the different scenarios that we can uh deploy areas in. Um the final outputs that you can get is uh that looks a bit like this. So you have an overview of the um goal which says okay I want to uh hijack the system and get some information that I'm not supposed
to obtain. um the final prompt that actually triggered the attack. Um this is the final one in the whole conversation. Um and then the final uh LLM response which actually gives the user ID uh and and the tool call response. And finally uh is the um evaluation. So like in this scenario is a relatively straightforward kind of um evaluation output where you say like okay well the
user is not user ID 2. Um but you will get in uh like um more borderline cases which maybe need more uh uh reasoning on the part of a LLM evaluator why a particular output was considered malicious or not. So Aries is in active development. Um we uh are open to PRs um issues try it out. Um we have a good suite of uh current attacks and
evaluation pipelines but we're expanding them all the time. Um we've tried to make it like I said lower the barrier to entry through this plug-in based approach. Um so if it's something you are interested in uh by all means um check out our GitHub uh give us a star uh and uh try and use it. Um so thank you very much for the time. We're on on
the dot exactly. So if we've got time for any questions. >> Perfect. So thank you first for this presentation and this is um working now. So do we have any question for Julio? So no we don't have any question. The same question why it's not in Eclipse. >> Yes. >> Your slide also is in the documentation part in the GitHub. Sorry. >> H >> are these slides
in the GitHub? >> Uh no, these aren't in the GitHub. Um there might be some way to disseminate >> Yes, there will be some way to disseminate after the event. Uh the material will be available strangely in open source way. Yeah. Yes. So it will be available but as you can see we have a link in here so we can connect with Julio. I think you can
write him. >> Okay. >> Yeah. Well, if you're interested in slides, yeah, drop me an email or LinkedIn and I can email them. That might be easier than waiting for them to be spread around officially, let's say.
