Arshan Dabirsiaghi: Who’s Going to Secure the Code Our Army of Robots Are Going to Be Writing?

This talk discusses the intersection of code development and security in the context of generative AI tools, specifically focusing on GitHub's Copilot. The speaker shares insights on how Copilot can improve developer productivity by generating code and assisting in writing new components. However, they also raise concerns about the potential security risks associated with AI-generated code, noting that these models are trained on existing code, which may contain vulnerabilities. The speaker emphasizes the need for better security practices and tools, suggesting the implementation of strategies such as Runtime Application Self-Protection (RASP) and static analysis tools to mitigate risks. They also introduce an open-source project called Cod modder, aimed at automating security code updates and enhancing the security of newly generated code. Overall, the talk highlights the urgency of adapting security measures to keep pace with increasing code production driven by AI.

[Music] hi and welcome to who's going to secure the code our army of robots is going to be writing my name is AR andagi and I'm sorry that we're getting off a little late but I appreciate you sticking around um I've been working in uh you know where code meets security my entire career I develop open source tools I was a practitioner doing code review and Pen

testing and I've made security tools in the past and uh that's the experience that I'm I'm bringing today uh a lot of people are probably familiar with Co pilot co-pilot is what we call a fill in the middle sort of product it's essentially autocomplete it's really smart autocomplete I'm a I'm a user and a fan and it's definitely the most widely adopted uh you know generative AI

tool in uh you know in the development space and uh when uh the big companies you see listed on the bottom here have started to study it they found that uh it helps generate between 25% and 60% more more throughput so one developer is developing 25% more code 60% that range does not feel right at all to me as a power user of co-pilot definitely 25% sounds

more uh more like my experience um and then if you have been uh you know getting the updates from uh co-pilot in your IDE you'll see that they're turning a little bit more into assistance these lightweight assistance that can draft whole new sort of uh components uh more like things like you know give me a file that does this it is sort of one file at a

time and uh you know this is getting more popular but the the statistics on the adoption we don't know um and then what we also don't know how will this help with the throughput uh you know if if autocomplete is giving us 25% more and we can now have an assistant in ide drafting whole files uh that is interesting and I think we you know I'm just

going to click some numbers out of thin air here no matter what number here here you want to come up with you can use for your own mental model I think we're going to St still end up at the same place so I'm just picking 100% uh again out of thin air and then now if you watch the end of uh the GitHub keynote very closely uh

GitHub Universe keynote um they they uh they flashed something on the screen uh very quickly it was a Devon like uh feature where you could open up an issue and have the you have the uh the co-pilot go sort of add a feature um you know across many files across your dependencies across all you know all all of the stuff uh that you might want to edit

in a repo to bring a new feature to um now if if autocomplete gives you around 25% more throughput what can a junior developer with infinite time and uh you know what what can that give you in terms of throughput um where every fever dream that goes through a business analyst Ted can become part of your codebase uh what will the throughput be and not only you

know enabling more less technical people to write code but also as a power user how much code could you put out um and it's definitely going to be a lot more than it is today um and that's not baking in any of the potential uh sort of quality breakthroughs that we might see in AI so uh there have also been lots of studies that talk about the

security of llm derived code um um so it's important to just understand that human code is what these tools were trained on uh these tools are essentially a mirror held up to the code on GitHub and stack Overflow and so if you uh you know if you have bugs in that code then when the llm learns from that uh it is going to produce bugs in the

same code and so this is what the Studies have shown as well but there is some other interesting and probably unfortunate uh facts here too that uh the code looks good um but it is uh more likely to give an insecure solution for one of the for one of the studies and also uh paradoxically developers trust it more uh because you know it's look it looks right

it's uh you know there are comments that uh you know give assurance and just the form fit gives you the illusion that it it may be more secure and so uh you know I've heard I think Caleb siba who described llms as a very confident overconfident 13-year-old and that seems uh really right to me and that's this is the kind of statistic that backs that up so

I think you know the first uh probably the first for The llm Optimist uh they're just going to think well look that may be true now but won't the models just generate secure code the models are anybody who's work at the models can tell you that the more context you give it um the more likely it is to be first of all confused or hallucinate or or

get the instructions wrong whatever but it is interesting to note that a vulnerability is an emergent phenomena AC lots of different things the the code the libraries the Frameworks the runtime and so let's look at a typical vulnerability you know uh data flow vulnerability something like SQL injection user input comes into this system and it goes through a complicated path before it reaches sort of the place

shouldn't and so you uh you know the idea here I think for The llm Optimist if I try to steal man in their case would be we're going to be able to shove all this code into um into a context window and we're going to be able to have it answer the question for itself secure um so uh there there are now million token context Windows um

and so the number of cod bases that will fit into uh context are getting bigger um but there is an interesting point here that a lot of the data flow first of all isn't even in your using embeddings as a replacement for uh complicated you know reasoning is is not really uh sort of a valid approach embeddings are good for search and for other use cases but

asking to reason very deeply about complex things that it has a hard time about in a very micro case um we're not going to be able to use embeddings to get out of that um of course the models you know the more multistep the more um abstract an idea is the more the difficult it is for the for an llm to model it um and first of

all this is sort of a solved if we take a step back this this problem is not solved well but I'd say but it is solved uh from a the point of view of we have uh purpose-built software that we've been trying to use to find vulnerabilities for 25 years we've invested many thousands of of manh hours trying to uh to do this accurately we have not

been able to with traditional techniques you know scan the code build a syntax tree ask that model questions does a flow to B and it's it's very difficult and um it's very slow and there's a lot of branches and to for trying to imagine how a uh a model would come up with uh the right understand all the paths that the code actually takes takes and uh

follow a data flow vulnerability through that is uh seems very far-fetched based on my experience with both the tools and uh the llms so I think it's also interesting to think about the uh the incentives for the generative AI tools so um the tools the llms will be mostly motivated by making developers happy giving developers what they expect uh but we actually we don't always yeah and

so in the other cases in history where you look at does secure code get rewarded by developers the answer is no so as as a lifelong security person I can tell you usually when you go to stack Overflow and somebody puts out an answer um the answer is either very incomplete or sometimes wrong um and so in this case uh you know the developer is asking on

stack oral how do I prevent cros a scripting um in PHP and they give an answer that will work 75% of the time or 80% of the time but everybody it it looks very correct right HTML it says Echo HTML special characters it sort of you know it matches what the developers sort of predicted solution for this might be um but the truth is it's it is

more complicated which crossy scripting control you choose depends on where in the HTML document um the input ends up does it end up in an HTML attribute or does it end up in JavaScript in those cases you'll actually want different solutions and the other Solutions probably won't protect you so uh you know like life is complicated security is complicated and so the the easier answer sounds correct

and so if if an llm gives you an easy answer uh you may be more inclined to use that llm versus an llm that puts out the actual secure answer because it's going to be complicated you're not going to understand it and uh it's not going to fit your predicted model of what that might look like you know good example of that is um trying to stop

uh an attack called external uh XML entities uh it's it's a really bad attack in XML parsers that allows uh people to steal their uh you know the contents of their file system if done wrong and so in this case the developers solution is to um switch their parser for something called diffused XML and they might not know what developer if they get that suggestion they might

not know what diffused means um and so that's uh you know that's going to be tough for the developer to have to um uh to make that decision um I'm getting some I'm I'm getting some feedback here in the chat uh that we might be getting some audio problems I I hope it's working and I if not I hope they're working on it quickly um so look

uh now let's we're going to flip back to the you know the the software development we have today so this um this solution here is or this uh this graphic is put out by Pedra Duty and it describes all the manual processes that are performed in a secure development life cycle and so when and so all these that's what all these little m's represent um I I

love this idea but realistically um if you have a static code analysis scanner that's running on build or on Commit um there's a lot of manual effort there too right we need to look at results that come from the static analysis tool we need to look at the results spit out by the our SCA tools like dependabot or uh renovate and so if there's fuzzing or iast

or dynamic scans or you know container scans all of these security activities we do uh they do require some manual effort the developer has to triage or they have to fix or they have to uh talk about and and so yes there are some activities they've highlighted as as being pure L manual but there's so much manual stuff that happens along the road here and this is

my estimation again I've done a lot of this work this is where I think a lot of the the uh the manual work ends up being um in your in your steady state as well not just your before the thing starts and in monitoring all every day there's there's activities that are um done and another interesting part of this mix is that uh a lot of these

activities are sort of cross-disciplines so a lot of times we do have security people who don't know code very well or code people who don't know security very well and so trying to ask either of them to triage a static analysis finding is very difficult because they don't have the one of you know the either one doesn't have quite the perfect background for for analyzing the others

sort of aspect of the discipline and so uh we've got we've got uh we've got a lot of manual activities here and we're not staffed really well for it um we're not frankly putting out the documentation the the research into the findings that we we should be um and this is really evident if you go to a big company um and you look at all their assets

uh you'll see that uh if you look at their programs you'll see they don't don't have enough humans um some I've heard GitHub say that the developers outnumber security 100 to1 and I I'd say if you go and look at the Enterprises I think that is uh that number is a lot worse again the humans we have aren't skilled and so if you take a look at

the uh you know let's imagine that this is a Enterprises portfolio of applications um I think what you'll see is that they most of the time they rank their applications in some risk Matrix and they say okay well this application is it's our front-end banking application it is exposed to the internet it touches the the most sensitive stuff we have this is a five and then you'll

have other things that are you know seemingly less important and we'll rank those low and then what they the unfortunate choice they have to make is they really only spend their activities time on on those super high-risk um super high-risk assets and so that's really tough we're only doing some things for some uh for some parts of our portfolio now uh you know let's think about sort

of what we started off the talk with which is we're today uh we have these stressors we're outnumbered at least 100 to one probably a th to one in in the the worst Enterprises we have you know we're we're writing 25% more code with co-pilot and there is a there is a you know I don't I I hesitate to use the word future I mean there is

a uh there is a present where we can potentially start writing a lot more code uh 100% to 500% more code how are we supposed to keep up with the robots in terms of security right all these activities all these M are done by humans right now and if we are producing five times as much code how are we going to keep up so that is the

basis for um I'm going to talk about a few Solutions here a few things that I think can scale with the robots um there may be others but uh let's let's just talk about a few because I think they'll give us the pattern we're looking for so look we need tactic strategy uh you know we need we need to think about this a little bit um systematically

and I think one of the things that um can help our engineering cultures uh and at the same time help the robots is to make it hard to be insecure um if it's hard for a developer to write insecure code then obviously by definition we will have a lot fewer vulnerabilities okay uh we did this in our compilers by the way we we had all these memory

corruption vulnerabilities in the early 2000s and we added uh you know we hardened some of the memory corruption um uh attack vectors by adding stack cookies to our our compiled binaries so uh Netflix introduced this a similar idea called paved roads and the idea is that we give uh all of our apis that allow them to just focus on doing their work but that automatically includes security

so you know for instance we might give you an API model to develop on that automatically enforces authentication you don't have to think about authentication that will be automatically provided for you or we might give you an API to implement that uh even if it doesn't do it for you it forces you to think about what is the access control you should be providing you know we

you know we have a maybe a static analysis rule or we'll have something in our framework that fails either compilation or at runtime when uh an access control rule is is not specified so in this way we're making it hard for the developer to make the mistake uh one sort of less uh business logic type of thing like we can we can have a a rule that

you know only we only allow rest plus Json in our microservice and this actually go does go a great deal towards eliminating a very popular vulnerability class called crossy scripting and so um another interesting upshot of having patterns like this is that the robot also learns from your code so a robot will end up delivering code that looks like yours I don't know if you if you've

experienced this with co-pilot usually uh co-pilot wants to write a unit test one kind of way but then if I have other tabs open where it can learn about the other types of tests I'm writing usually I just have to write one and then it starts to understand kind of how I want to do things and so if we're laying out a lot of evidence for our

robots to follow like these are the right ways to do it uh these are the only ways to do it as well then I think uh that's going to help a lot uh another more abstract part of this is is uh you know version of this same kind of thinking is to have a uh you know we have a only a few different types of pipelines in

our company you know and every one of them forces a static analysis on every bill so pave roads great idea not only to eliminate um you know make our make it hard to be insecure when writing new code um but also to help train our robots how code and so we've got some uh some interesting people here to help um another class of tools that I I've

worked on in the past uh have a different strategy which is to make it hard to actually exploit insecure code when when it occurs it is a when not an if that you will write in secure code and so how we deal with that uh how do we deal with that reality um so there is a uh a new class of products called rasp runtime application security

protection and so what we used to have uh was tools like web application firewalls we still do have these tools they sit outside the network and they look for malicious input and they sort of make a go no-o decision when they receive the input and uh this is tough because these tools are executing outside of the context of the code they don't know what kind of input

the applications want and so they end up a lot of times being a log only sort of tool they don't they're not usually blocking because they don't have the they're too far away from boom uh to understand if something is really an attack or not they're very signature-based and uh anytime there's a new attack attack people are putting out signatures uh people are putting out uh exploits

that evade the signatures and uh there's another alternative which is to instrument uh protections into the application the framework the runtime Etc and so here's an example of of a uh an agent that instruments a SQL injection protection into your code when user input comes into the system if it was a a firewall kind of solution they'd have to decide based on the input should they block

this but a agent can look take the input hold it wait for queries to be called and say Hey did I see that input go into that query and if so did it change the structure of the query so did data become code and so you end up with a wildly different level of fidelity on results like this because you not only did you see input that

look suspicious you saw it go into user data and you saw that overlap of tokens where it seemed to be crossing a boundary and uh basically you saw the input get executed and so that's a very different uh uh way of working and uh very effective at removing a lot of different bug classes and so now there uh you know that now there's a lot of vendors

who are offering tools like this so look the the other sort of practical end of this is how can we automate the and by the way I should point out these uh again we similarly did uh we did this already at the OS layer we introduced uh you know non-executable Stacks non-executable heaps we may we used to be a new virus every week um you know when

I was Computing in the early 2000s but we added a bunch of protections to the OS to the runtime of our operating system and we uh we eliminated you know made it impossible to practically impossible to exploit a lot of insecure code anyway all right back to this the more practical um consideration is how are we going to uh deal with all the interruptions from tools when

we have you know we're putting out five times as much code so that means essentially means triaging uh results from security tools and automating sort of the remediation of them and so the the uh you know we can we can use uh tools that we have today to find vulnerabilities in code that um you know that the that the generative AI is creating right that we don't

need to change any anything except use more of those tools as we create code um and it is possible also to triage the the the reports from those tools and it because we can use you know lots of you know we can use orchestration of llms we can use um you know traditional techniques to look at the vulnerabilities and you know decide for ourselves like is this

a real vulnerability because most of the things that come out of tools are not real and so if we're going to have humans look at all this stuff not only are we going to be U wasting a lot of time but it will be a waste because most of the things that they report are not real and so we'll be wasting it for for no reason sort

of the definition of waste um and so we will also be able to put out pull requests right uh we can if we can use techniques to find the real things find the uh and generate the fixes for those things then we will be able to um you know pass along the our feedback uh using a virtual engineer virtual product security software engineer uh to interact with

the other with the uh generative AI produce code and if we can have you know a bot that works with their bot a bot that works with the generative AI tool um I we think that is what um is going to actually get us over the hump that's how we're not going to spend all of our time chasing this down chasing these bugs down and at the

same time it's the only thing that's going to enable it to happen anyway to get the assurance that we need um on all the new code um and so uh I've actually uh released uh some of our research here so I'd like to sort of end talking about that and so um we have an open source project called Cod modder so uh code mods were this interesting

uh idea they were um in I think it was released by Facebook engineer named Justin something who I can't remember right now sorry Justin um and the idea was to build uh build a piece of code a small piece of code that could change a lot of code quickly um and so um it was really cool but it sort of died and now the JavaScript Community has

something called Cod mods but they don't really do a lot and the reason is because uh it's very difficult to build a a Cod mod library that can identify arbitrary shapes of code and uh you know change it to any other shape of code and so this Library orchestrates tools that are great at finding different shapes of code and existing tools that are great at mutating is

existing source code and so this is an example of a code mod or code mod that orchestrates a semrep rule semrep is a very expressive um uh tool for finding different shapes of code and our goal with this uh code mod is to replace random with secure random right insecure sources of Randomness with secure sources of Randomness and so we made it we try to make that

really easy all you do is provide the sem grap Rule and then that the results from that the findings from that are delegated to a lib CST uh a call back with lib CST objects where all you have to do is you know make the change that you want to make and so we you know it's purely we're focused on De delivering the magic between uh taking

you know taking results from these tools uh uh create you know parsing them and delegating them to the right places and and um making uh you know making the changes is making that really easy um and so uh here are some of the links to to uh look a little bit more into Cod monor but you can imagine uh you know having a codon that looks at

a static analysis result and you know has the pre-big solution for uh for what it takes to fix that result because a lot of times security changes are actually not that difficult to make there's usually sort of one right way to do it and it's usually a matter of flipping Security on and so I I think uh this will be a really cool library for you to

check out and build your own stuff on okay uh look we spent a lot of time talking about generative AI I'm excited for the future that it that it uh that's that it's going to allow and we are not prepared today to take on the security burden of a lot more code but I think we have some Solutions in front of us and that's where I'll leave

it uh I have no idea on time because I know we started kind of late but I'm happy to answer any questions all right well if there's no questions I want to thank you for your time today