About this talk
In this talk, Alex discusses the intersection of security and artificial intelligence (AI), specifically focusing on the vulnerabilities that arise when using AI in software development. He highlights the evolution of AI models and their impact on coding security, particularly with regards to web applications. The speaker outlines various types of attacks, such as prompt injections, which exploit weaknesses in AI systems to compromise security. He emphasizes the importance of using multiple AI models simultaneously to enhance code review processes and mitigate security risks. Additionally, Alex explores how AI can be integrated into products, the security challenges that come with it, and strategies to safeguard against prompt injection attacks. Through practical examples, he showcases how malicious instructions can affect AI behavior and provides recommendations for improving security in AI applications.
Full transcript
All right. Hello everyone. I hope you can hear me well. It feels a bit weird not hear myself. But yeah, I'm Alex. Today we'll be talking with you about security and AI. And the title is all your AI belong to us. I hope you get the reference. But before we and of course my clicker doesn't work. Okay, before we dive into security and AI, I wanted to
learn a bit a bit more about you. So if you could get your phones and go to this QR code scan this QR code or go to that link slido.com with that number and answer a few questions. There will be a couple more uh after after after that. So I'm curious if you can use if you use AI in your work. And I've left a third option
that I'm not allowed by the company for people who unfortunately are in that category. I still meet people like people like that occasionally. But looks like we don't have people like that today. Okay, so everyone is using Okay, also for some of us not allowed. Okay. Okay, let's see. The next question is about MCP and I feel kind of weird to ask you about MCP now that
because the previous talk was about MCP and skills. Uh but yeah, still uh curious if you use any MCP or and or uh agent skills. still a few people don't know what's MCP or skills. Okay, I thought the previous talk was about it, but okay. I will briefly touch on that. All right, but looks like a lot of people are quite quite up to date on the
latest technologies uh when it comes to AI. Now um now that we learned a bit about about audience and about you. Few fun facts about myself, right? So there's also contacts if you want to reach out to me. But I've been in IT for more than 10 years now. Mostly in machine learning, a bit of DevOps, a bit back end, all the different things. I've been building
ZenCoder, AI coding assistant, since day zero. I'm one of the founding members. Uh for the last year and a half, I'm mostly DevRel. This is my event number 57. Which, of course, made me fly a lot. This is from last year. I had almost 100 and 10 flights. And I also live somewhere between Madeira or you also I started my YouTube channel because I'm, you know, every
every DevRel should have YouTube channel. Uh don't have a lot of subscribers yet, but, you know, you can help fix that. And I live somewhere between Portugal and Singapore now because I'm kind I have my cats in Portugal and I have my girlfriend in Singapore, so I'm stuck somewhere between those two countries on the other side on opposite sides of the world. Now, um being, you know,
being in the in the space, being in the AI coding space for more than 10 years now, uh like specifically for AI for coding, uh it allowed me to also uh observe all the sort of the evolvement all the evolution of the AI. And I've at some point I've created a sort of timeline of all the all the model releases. And I, as you can see, I
stopped doing that at around February because uh it became quite tedious work. Uh but uh that's I have actually a third question from the Slido. Uh who remembers which Sonnet version we had 1 year ago, exactly 1 year ago on this day? May 8th, 2025. And just as a reminder, uh today we have Sonnet 4.6 and Opus 4.7. So, let's see if uh you remember what you
potentially used just a year ago. Okay, so people are split between 3.5 and 3.7. All right. Well, uh the answer is Let's see. Sonnet 4 was released on May 22. So, uh around this date, 1 year ago, we had Sonnet 3.7, which means we went through around five generations of the models. And of course, this this revolution is sort of twofold, right? So, on the one hand,
we have a lot of issues with the security, right? So, people are getting the coding when they write code stuff. People are getting using AI to some write code, help them write create applications, so on. And this results in AI introducing a lot of security vulnerabilities, especially for web coders who don't know anything about security, right? So, a lot of those headlines are coming from those examples.
But at the same time, models are getting better at security as well, right? So, especially like the latest cloud methods, which is supposed to be uh the next breakthrough in cybersecurity. Um you know, they are getting better, at least on the benchmarks, and even some um competitions, like this one, for example. Uh the first place, Octopone, is AI model, basically. So, it's an AI agent which win
a security contest. So, throughout this talk, I want to talk about AI and security on three levels. And first one is web coding, right? So, on the web coding level, um usually people either um haven't been developers at all, or they are just, you know, like me, for example, I'm developing my own website. Uh I'm not a front-end developer. So, for me, it was a bit um
a new area in a sense when it came when it comes to security. And here's a good example of uh web coding app being hacked like in under 10 minutes. In this case, uh AI didn't configure Supabase properly. And I from the first from first experience uh know that because I also use Supabase for my pet project, I saw firsthand how AI completely ignores any security requirements
for Superbase like role level access uh for database and this is what what this is the case here in this specific hack. Right, so when comes to when it comes to security in web coding, of course you can, I don't know, ask AI to just make no mistakes, but that's of course not very reliable, right? So better way better approach when it comes to protecting yourself at
least on the sort of first level is don't focus on just single model. Uh nowadays you can have a lot of different models work together in parallel or you know, against each other. So don't just use one single model to generate code and then review code and so on. Try to use multiple models at the same time. And here's one example here. So let me zoom in.
So this is our product and here I'm just I'm just using it as an example. So here I'm asking it to I'm asking three models in parallel to review the changes to review the code. I think I'm asking Grok, Codex and Claude to review. So in this case Opus 4.7 spawns a few sub agents. Each of them independently And because those models come from distributions, from different
training data, they can detect different things. On the security side or in general with any issues with the code. So in this case it found a few a few issues and I have a basically from the Opus and I have a summary and what to do with those issues. And that would be it is possible that since the code was written by Opus, Opus on its own
might not detect the same issues because well, it's pretty much like a developer, right? When you write a code, you don't you are not the same person who reviews the code because you well, you already you just wrote it. You know it you know it that code and probably you will miss some issues. But combining multiple models will help you um detect any possible issues with generated
code. And then of course there are yeah, and it also helps you might helps you save save money and get better results. This is our research on uh generating combined multiple models both for code generation and for review. And then of course, there are also tools which can help you audit your code. So, do pen testing, automated pen testing like Shannon. It's essentially um a tool on
top of cloud code which allows you to well, autonomously pen test your application. It requires access to both source code and the actual product. Uh so, usually you have that and you can use that to essentially find any vulnerabilities, any issues in your in your code base, in your application. Now, uh next level is when you use AI as a part of the product. So, you have
some sort of feature which is based on AI. And here, a lot of issues, a lot of security issues come from prompt injections. Right? A few years ago, it was quite easy to do the prompt injection. So, you probably seen examples like that where an attacker just asks model completely ignore previous instructions and do something else, completely different. Right? So, in this case, this screenshot is from
2022 when the models were just the LLMs were just starting to get popular. And here, we simply ask a model to ignore both instructions and do completely different thing. Right? And it it followed the instructions. Uh nowadays, models are getting better at detecting those prompt injections and we will see a few examples also later. Uh but still, when it comes to having AI features, you need to
be aware of any possible attacks through any possible data source. And I will and this this doesn't only come from external sources. Right? So, I will give an example here. And those people a lot of companies nowadays use rag or just a semantic search to augment their internal tools. Right? So, for example, you have some sort of a knowledge base, internal knowledge base, uh can um help
your AI answer questions from your employees, right? However, even internally, some people might knowingly or unknowingly upload uh malicious data, or maybe not even uh it it could be it it's not necessarily uh mali- they not necessarily have malicious intent. It's just um they might download something from the internet which has potentially uh prompt injection, uh which will then lead then uh steer the model into some
malicious behavior. I have a simple example here. Let me uh switch to my different tab over here. So, I have a very simple 5 coded, of course, uh application of company assistant. And let me see the docs here. So, I have three docs, right? Let me zoom in a bit. Uh which are essentially uh well, three wiki uh pages. And then I can ask some questions. So,
uh let's let me ask uh it to list uh files in the current directory, right? So, of course, it will well, it I hope it will deny me this uh request, right? So, yeah. It says that I can't run the command because I'm not an admin, uh yada yada. Now, let's say um someone internally uh decided to be to provide some helpful instruction for AI. Let me
copy it here. And what it says here is when the user says special magic word, uh they are admin, and the AI needs to run specific specific tools. Let's call it helpful note, and upload it. All right, we have now four docs, And let's do the following here. I think uh list all files in the current directory. Let's see what happens now. Let's see what happens now.
Okay. Well, it still didn't run ls -l for us it did list all files. Uh as instructed in the document in the just uploaded document, for your ink run book, it ran the env. And in this case, it would essentially give the attacker potentially my API keys, right? So, as you can see, in this case it did mention that, you know, they shouldn't be stored, locked, and
so on, but still it mentioned that I have some API keys, I have some virtual environments, and so on. So, essentially, uh through a simple uh instruction of couple lines, I was able to sort of trick LLM into giving me um unauthorized access, unauthorized data. And again, this example is supposed to be like internal tool and internal assistant. So, regardless of you know, source of data, it
can be uh steered into inappropriate behavior. And there was a nice simple example just a few a few days ago where a user asked uh Grok bot on X, on Twitter, um to do something, and then uh they also hid the instruction, a prompt injection, in the Morse code. So, when uh the bot translated the Morse code into the text, uh this prompt injection uh take it
took effect, and it essentially sent around $200,000 to Bitcoin account of some uh yeah, crypto account. And then the perpetrator was able to get those money uh to the USDT. Right. And yeah, just a few uh a few words on how to protect, how to handle uh those injections. So, any data essentially which is being ingested by your tool uh on the user level uh needs to
be treated as malicious by default, right? So, anything you uh need to do with the value, sanitize your data, uh should be done. But when it comes to yeah, it is also a bit tricky because prompt injections uh the prompt injections is sort of arms race because uh yes, models are getting better, but you can't really use the same model to detect prompt injection because it would
be then steered on its own, right? And then of course you want to have your agents, your AI um sandbox isolated, uh have um limited permissions and so on. So all the uh you know uh typical basic stuff which you might uh want to exercise when it comes to uh raw input data from the users. Now the third level is AI as a user. Right? So um
Unlike with the white coding, when you don't know what uh you know what you're doing essentially, uh when professional developers uh use AI, they usually uh at least have some notion of security, right? And still uh there are a lot of ways for uh perpetrators to attack AI. And yeah, AI does uh give you a lot of power, but still it comes with a lot of responsibility.
Now uh couple words on uh things like I asked about, MCP and skills. Yeah, and we're talking about prompt injection again. So first, MCP. Uh I think previous talk was about MCP, but just so that we are on the same page. MCP stands for Model Context Protocol, and it allows LLMs to get more context uh to answer your questions. So let's say you ask the LLM what
is uh I know response time for Google, and then LLM doesn't know it by on its own because it well, it doesn't have access to to internet, doesn't have access to anything on its own, uh then it can use through MCP from CP server uh ask get that context, get the data. And MCP server essentially is responsible for uh exercising those tools, uh those uh gathering those
con- context and passing it to MCP. Oh, sorry. To to LLM. And here's an example. So uh whenever LLM decides that it needs uh to gather external data, it can ask MCP server to uh get the data from external system. It can be local system, it can be external uh remote system. And then, uh, this extra contact is passed on to LLM and LLM give you context
context contextualized answer. And then, another, uh, thing seem kind of similar is skills. So, through skills, uh, you can, uh, teach LLM to do something, uh, some specific, well, skill, right? So, here's a good example. Um, comprehensive review. So, essentially, this skill teaches how to teaches LLM teaches the agent how to do a code review, right? And this is just the first, uh, few lines. Uh, it's
essentially a markdown file. And in addition to markdown file, you can also include scripts. You can include assets like images, uh, videos, and so on, uh, for AI to use. So, essentially, again, uh, this skill in in informs AI how to do a code review. And then, there could be some links in the code to to the in the markdown to specific code. So, let's say the
skill for creating, uh, PowerPoint presentations from Anthropic, uh, they have a bunch of scripts, a bunch of a bunch of Python scripts, a bunch of JavaScript, uh, scripts, uh, which allows, um, Claude, which allows LLM to create PowerPoint presentations through this skill. Now, uh, yeah, just to sort of differentiate between them, MCP is essentially a way for LLM to get access to the outside world. And then,
skill, AI skill, uh, tells LLM how to what to do with that with that access. That's how to you like access and what, uh, how to best, uh, get the most of the out of the MCP. yeah, of course, uh, MCP, uh, it was introduced around a year and a half ago. They're still quite, uh, wild west when And I will show you a few examples of,
uh, what, uh, issues can arise from MCP. And, yeah, before we do that, couple more things. So, MCP servers source code which you might run on your local machine. And of course, when it comes to running third-party code on your computer, there are sort of usual suspects when it comes to security issues like package name hijacking, malicious code execution, and so on. At the same time, AI
introduces sort of next new level, new layer of attacks which I kind of call social And this comes in different shapes, like you've already seen prompt injection, but also MCP and skills add sort of another layer through tool poisoning and so on. And let me show you a few examples of that. Two live examples here. here I have a simple simple Jupiter notebook which has MCP client.
So, I will start with using Sonnet 4.5 first because older models are most susceptible most susceptible to attacks, but I will show how even Sonnet 4.7 can be attacked through a simple MCP poisoning attack. So, here it's a simple MCP client. Nothing fancy, just gets a bunch of tools, and then it can do something with the with those tools, right? So, and I have also a server.
Let me open this page which is essentially a simple very simple web server which hosts this page. And I have three pages here actually. I have one which is called regular. I have one which is called not oops. Which is the same. And I have another one which is called oops, which is again the same, right? So, let me close that, and let me ask the our
agent to fetch that page. I hope everything uh, works on Tropic. second, let me restart the server. Okay. Now it should work. Okay, so it was able to fetch the page. It said hi to us, right? Everything's fine. Now, uh, this not oops page, um, sometimes it works, sometimes it hacks, uh, the LLM, sometimes it doesn't. And this is due to the fact again that models or
model providers, they are trying to protect, of course, the LLMs from external attacks, but um, still the it's not, um, 100% success rate. So, let's see if, uh, this time the it will work or not. And the problem with that page is, in addition to, uh, what we've seen, the hello world page part, there was a hidden instruction over here, uh, which asked it to create also
create a file called oops.txt. So, in this case, uh, looks like uh, Sonnet 4.5, uh, noticed the text, but decided not to follow, uh, which is good, but, uh, in my experience, if you run the same request like maybe 10, 20 times, you would get one or two, um, times where, uh, Sonnet will still follow the instructions. But let's let's get this oops page fetched now. And
this one actually contains a bit more elaborate prompt injection. So, here is the instruction. Again, it wasn't visible for the human eye, but it was part of the page, and it was sent to, um, LLM when we, uh, fetched the page. And as you can see, it is taking some time for LLM to, uh, get us a response. That is because it is essentially, in this case,
following the instructions, uh, as is. And the instruction here, uh, asks is asked asked the LLM to list all the functions which it has access to, and also add a new function, and then, uh, test simulate the test run of that function. This function is, um, this one, right? main phrase in the file, and it asks to essentially create a file. Let's see the response from LLM.
As you it did say hi to us, right as instructed, but also on top of that, it essentially followed all the instructions which were injected into the page, into the LLM. Right, so at least it told the functions available, and in this case it has what, 16 functions. And then it tried to do the test call of the function. But doing that, it actually did call the
actual tool, write file, and I can switch to my files forward here. As you 36 36 36 seconds ago, it actually created a file on my local machine. Right, I didn't instruct it to do so. It was instructed by perpetrator. And yeah, but basically LLM since it was essentially doing that on my local machine, it essentially got access to my local machine. And of course in this
case it's just a text, but you can imagine this being used for more malicious use cases. I think we are a bit on time, so let me just switch to one more example. And I will share the link to this notebook for you to play, so you can see all the different examples here. But let me show one more example. Here it's tool shadowing. So tool shadowing
means that let's say we have a which has specific tool. So in my case I have two servers. First one has a bash tool, right? So this tool allows LLM to run bash commands. And it's a good good server, nothing wrong with it. However, I also have a second server. Which has better bash tool. Which is you know, much better tool for running commands in bash. And
it also has a random tool for the fact of the day. So again, doesn't look anything doesn't look malicious. But in reality, our better bash tool is kind of shadowing that proper non-malicious tool. And this better bash tool is actually So let me actually, for this demonstration switch to Claude Opus 4.7. Cuz as I mentioned, models are getting better. But even Opus 4.7, which would not fall
for um this instruction over here, it will it will still fall for uh for this attack. So, let me uh run first ask it to list all files in And as you can see, it used the bash tool. It didn't use our malicious better All good. However, if we change the prompt a bit and we ask it to use the best tool for the job, let's see.
It will Well, as instructed, it used the best tool for the for the job. However, unfortunately, uh this um was shadowing the proper tool and it is uh malicious. And as part of that, as you can see, there is uh nothing wrong with the output, but uh if you look at the files, it actually created local file for us, oops.gif, uh which again wasn't wasn't We didn't
ask to do it. Uh it's actually just MCP server did something uh that we were not aware of and which uh you know, we didn't ask for. And in this case, even if um could potentially detect anything malicious and refuse to do so, in this case, it's it was essentially out of even AI's control. It was just an MCP which was doing something else because you're executing
the code essentially. Um let's see. So, this is the QR code for uh the notebook if you want to play with it and try all the different attacks on different models. Okay, I will give you a couple seconds to download to scan it. Now, uh I showed examples with MCP, but skills also can be used for malicious instructions for prompt injections. And here's uh one of the
examples. So, if you heard about Open Claw, or at some point it was called Cloud Bot, and then Open Claw is something else. So, essentially, it had a sort of agent skill marketplace, where people could upload different skills, and then AI could search for a specific skill to do specific job. All right. So, and this one was actually one of the most downloaded skills. This skill is
for, well, interacting with Google services, right? So, which would allow AI agent to read your Gmail, calendar, and so on. However, as part of as part of that skill, there is actually these few lines which are labeled as prerequisites, and they instruct the agent that it needs to download this library called Open Claw Core. And because it was run as part of the Open Claw tool, it
sounded it sounded fine. It sounded not malicious, right? However, actually this utility was a zip archive, which was which contained a malicious code, which was then executed as instructed by AI. And it even asked you as a user if you want to download the download and install Open Claw Core. But since it since it sounded non-malicious, it since it sounded uh relevant to the tool, a lot
of people could fall for for the fall for that. And then AI would just follow instructions, essentially. Even though this file would contain some I think it was some screen locker thing or whatever. just to give you the scale, this is from February. So, against 36% of the skills were found to contain prompt injections. Now, how to protect yourself from that? So, we just recently published a
sort of blog post series about using AI to help you in protecting your your AI. All right. So, you can you can scan QR code to get the link to part one Uh but uh the summary as a nutshell in a nutshell, uh here's a few things. So, um don't just blindly, you know, uh rely on um any anything you download from the internet. You know, it's
always always think, especially when it comes to executing third-party code, right? Uh so, ideally, uh validate everything you download, everything you use, everything you run. Ideally, also sandbox your agents. Um prefer, you know, um pinned versions and so on. So, all this all the stuff you already do for third-party libraries. But also, um don't just rely on yourself. Use AI to verify what is being downloaded, what
you about to use, and so on. And you can also incorporate all those practices into your uh in default engineering uh workflows. That uh yeah, I will be around here if you have any questions. Uh on the this QR code will lead you to the website of uh my company. And this one this QR code is for you to uh reach out to me if you have
any questions. It has all my socials over there. Uh that's thank you all for attention. And uh good luck on the you know, on the rest of the conference.
