Scale application security with AI-augmented vulnerability remediation

This talk focuses on the intersection of artificial intelligence and application security, particularly in the context of vulnerability management. The speaker, Nitish Tiwari, a Gartner analyst, discusses how the rise of AI tools in software development has transformed security practices. He highlights the increasing challenge of managing vulnerabilities, evidenced by statistics indicating that a substantial amount of AI-generated code contains vulnerabilities. Tiwari emphasizes the need for automation in remediation processes to keep pace with rapid attacks, advocating for a blend of AI and human oversight in security workflows. He also presents strategies for implementing an automated vulnerability remediation system using both commercial and open-source tools, while stressing the importance of a collaborative approach between development and security teams. The session concludes with insights on how organizations can enact effective governance policies to balance innovation and security.

[music] >> The topic is going to be on application security. It's not a question to anybody that AI is changing in terms of how we build and interpret software. It is also changing the way how we should be addressing security. How we should be taking care of vulnerability management as well. You all have heard about my tools model. You might have seen the latest blog on Opus

4.7 where is it was able to expose a vulnerability in Chrome. So things are changing and that is going to be the primary topic for this particular presentation. Now before I jump into the topic, a bit about myself. My name is Nitish Tiwari. I'm a Gartner analyst and I belong to the practice called software engineering leaders group. My primary role is to talk to CIOs, CTOs, software

engineering leaders and CISOs on a daily basis on a wide range of topics. Now my coverage basically belongs to three top three categories. The first is software supply chain where I mainly cover open source security, SCA tools and SAS tools. Then second is AI in software development life cycle where I cover things like AI coding agents, AI augmented code review tools, AI augmented modernization and vulnerability remediation.

And then I also cover enterprise low code platforms as well. I'm also leading the open source research for the software engineering group within Gartner. Now let's jump right into the topic in terms of why are we here? What is so important about this particular topic? Before I jump actually before I jump into the actual problem, let me tell you a small story. About 3 years back I

was at the Gartner India Symposium where I was about to meet the software engineering leader of one of the largest global enterprise. The topic was open source security and vulnerability management. And by the way, the story happening before chat GPT explosion. So before the AI era. That is very important to note here. So at the time I asked them, "What is the current security process that you

are following?" And to my surprise, they told me that currently the security team uses some tools to find out the vulnerabilities in the in the PR and then they just hand over the list to the software engineering teams. Now software engineering teams manually dig into the code base whether it is first party code base or open source packages manually identifying the vulnerabilities and remediating them on their

own. So I asked her "How is it impacting the developer productivity in their organization?" And she was very out front with her frustration. She told me that her team is always overwhelmed with the new security findings. In fact, many a times they can't even find a vulnerability or it may very well be possible that the vulnerability didn't exist in the first place. Now just imagine their situation

in the AI era. How things have actually changed in the AI native age. According to Gartner's software engineering leaders survey, 40% of leaders are saying that more than half of their teams are using AI tools to augment their software development processes. such as Cloud Code, GitHub Copilot, and Surf of the world are becoming mainstream when it comes to software development. In fact, our survey points out that

writing new code is the topmost use case for using the AI in software But, is it all safe? The Georgetown University points out that 48% of the AI generated code is actually contained vulnerabilities. A Pido also gave this particular quote that AI is generating 4x more code, but is in fact generating 10x more And how does it impact the developer productivity? Well, Harness reports that 68% of

developers are in fact spending more time in remediating vulnerabilities. And it is not necessary that these developers are using manual approaches to remediate those issues like the story I was telling you about. In fact, it is very much possible that these developers have access to some but are in fact spending more time in remediating issues. On the other hand, open source supply chain is also under attack.

Over the past year, we have seen continuous supply chain attacks on the NPM supply chain. Started with the crypto attack, then shy Hulu the tar, jsPDF. The latest one was Etsy OS attack. It even reached the AI tools such as N8N. The second issue that community is facing is the AI slurping. You might have read in the article that the curl maintainer actually had to stop their

bug bounty program because of the heavy AI slurping. A few other articles suggest the same that some other communities were applying the similar measures as well. And lastly, it's not just limited to vulnerability attacks and AI slurping. Developers with access to documentation and code base are recreating the entire open source projects using AI. A prominent example of it would be the Python library called care debt, which

was initially published under the LGPL license, but is now republished under the MIT license. So, this indicates that vulnerability attacks are elevated, new attacks are emerging, and we need we cannot approach it with the traditional sense. So, how is our responsibility actually changing in the AI native age? In the AI native age, we cannot just be responsible for managing Our responsibility is to manage vulnerabilities at scale

at a much faster pace. And that is why we cannot go with the manual approach that the story I was talking about. And that is why the solution comes from AI as well. Organizations need to develop automation processes for remediating because there is no way when attacks are happening in hours we can achieve that scalability via manual processes. Over the next few slides, I will initially talk

about how you can build an AI automated vulnerability remediation system that would be coupled with commercial After that, I will also talk about if and how you can actually build one system using just open source tools and agent. I expect there are some open source leaders and community folks as well, at least a few ones in the audience, who might like that section as well. I published

this prediction last year. By 2028, AI agents and assistants will remediate 70% of the software code Now, to be honest with you I think I was a bit too modest here. We have already seen the results of it and I believe that we will be able to achieve 80 to 85% of results using AI coding agents. The agents are offering fixes via two approaches. For first-party code

base, they are offering direct code fixes. And then they are offering back porting methods for open source packages and libraries pulling down the updated dependency into the current open source package. Now, whenever we talk about the emerging use of AI agents we always ask one specific question. Does it reduce the dependency on humans? Does it reduce the importance of humans? And the answer is no. I published

this this credit the same year as well. That by 2028, 30% of organization will face severe issues due to over-reliance on AI. And this is very interesting because I don't think anybody dis- discuss the importance of human in the loop approach. A few weeks back, I was at the null con summit where I met dozens of CISOs and every single CISO was strongly firm on the human

in the loop approach. At the same time, I still find organizations over-relying and entering into severe issues. The another problem is that human in the loop is important, okay? There is no deny to that. But at the same time, it's not scalable. Just think about it, right? If I have to review every single line of code and every single fix that AI is giving me, then I

will lose all the time savings that I got with code generation in the So, how do we make it scalable? How do we build this system and what approach we should take? This is the answer to how we should be addressing vulnerability remediation in the AI age. We came up with a three-step process. Number one, you still need to use your software composition analysis and static application

security testing tools. These tools are going to help you in list vulnerabilities. In fact, the modern SCA and SAST tools will also tell you about whether a vulnerability is reachable from your code piece. Particularly, they answer three key questions. What are the vulnerabilities? Where do they exist? And how are they reachable? Some of the tools also offer fixes for these vulnerabilities as well. Now, this entirely helps

with building the context. And that context is very important for the second step. You submit this context to AI augmented tools that will offer fixes for these Now, you do have the option to use specialized security code assistance. Vendors such as Veracode, uh root.io, Semgrep, they offer security code assistant. I do have a slide of commercial vendors examples, and happy to take a pause there so that

you can take a snapshot. But at the same time, you can also use the typical AI coding agents. Itaú Bank used Devin along with Veracode. And they were able to achieve 70% results 70% success result for the remediation of vulnerabilities through their AI augmented system. This slide also points out how you should be maintaining human uh oversight as per the uh human oversight, okay? Like I said,

human oversight is important, but it's And to make it a scalable, we refer that you should maintain human oversight based on application criticality. If the criticality of this piece of code is higher, make sure that human eyes are on it before it passes to the next stage. The last step is rather optional. We are seeing increasing use of runtime security analysis tools such as runtime SCA tools

and dust tools to identify the vulnerabilities and their exploitability at runtime. Making sure that you are not leaving any severe issue into the production pipeline by any chance. Now, in the AI native age, we are also seen and we are also recommending that a single organization cannot handle all the responsibilities of security of We have to adopt a shared responsibility. Software engineering teams and security teams need

to take that shared responsibility when it comes to And for that, software engineering leaders and vendors have to prioritize developer experience Now, just imagine an experience. You are working in a VS Code or via agentic IDE, you have integrated an SCA tool with it that is giving you the list of vulnerabilities, and then you have further integrated a security code assistant that is offering the fixes for

it. You are not leaving that entire ecosystem and are still able to push shift left approach in terms of How does it look like for the OSS supply chain? Now, we recommend that you should ensure the secure use of open source throughout the software development life cycle. And to achieve this, we recommend that you should not allow your developers to install anything directly from the internet. Instead,

have an internal repository management system or artifact repository management system. Your AI coding agent should be integrated with this system. Okay, so every package is a scan through the security and compliance check. And if it fits the risk profile of your governance policy, then you're hosting this package on the internal repository system. From there, AI coding agents are bringing it into the application development environment and using

uh and and not installing anything from the Automate the SCA scans throughout the creation and verification phase. Here you would be also using the AI augmented tools and practices, ensuring that uh whatever SCA is given you, whatever the list of vulnerabilities SCA tools are giving you, AI augmented tools are offering And finally, you need to make sure that you are putting the updated patches on the on

the production phase and continuously monitoring the use of open source for the new vulnerability Now, like I said, here is a list of simple representatives. These are not my favorites. These are not from any Gartner Magic Quadrant as well. These are just the ones that come often in our inquiries. Now, some of them, for example, root.io just provides uh AI augmented tool for open source packages. Then

there are tools like Semgrep that offers AI tool offerings for first-party codebase. But then, a larger section of it, such as uh Endure Labs, Lineage, they offer both for first-party code bases as well as uh third-party uh open Now, the dream of AI-augmented DevSecOps cannot happen without the cross-collaboration of uh of teams. Organizations, especially the CIOs, need to put their software engineering, legal, and security teams together

to define a governance policy that will foster innovation in a secure and efficient way. I often talk to organizations where I see that when a single organization is leading this particular effort, it starts acting as a resistance for the innovation that developers wants to do. This is also need to be supported with platform engineer uh platform engineering approach. A platform approach turns out to be highly beneficial,

reducing the cognitive load of developers, and making sure that developers do not have to integrate security tooling for their individual projects. In fact, platform engineering or platform-based approach turns out to be a highly beneficial approach when it comes to enforce security tooling and governance policies within your automated CI/CD pipelines. A real case study would be of Southern Company. Southern Company established platform who were responsible for handling

complex AI use cases. These complex use cases included security vulnerability remediation, upgradation of patches, and upgradation to newer versions. The The platform engine team developed these automated workflows that were served through a centralized portal. This centralized portal was acting as a single point of entry for developers who can access AI tools and services for their development work. For example, they had a security remediation workflow. Developers request

for a security check through the automated workflow through the centralized portal. And as soon as the request gets in, an automated workflow starts. At first, it will copy the entire source code into the security tooling. The second thing it will do is it will select the appropriate prompt based on the transformation needed. In this case, when we are talking about it will choose an appropriate prompt and

apply it to based on the security tooling findings. The workflow is also responsible to remediate all the vulnerabilities that security tools generated. After that, it will use a generative AI-based tool to get the transformed code, and then it will run through the quality checks to make sure that the code is aligned with company policies and company standards. Finally, it reaches to the developer for review. Now, this

is very important. Human intervention is happening, although it is happening at this particular stage. Developers are still responsible in to make sure that they are pushing a secure code and quality code down the pipeline. Now, till now, I only talked about the commercial tools and the case studies where organizations use commercial tools as well to build the AI augmented And to be honest with you all, almost

all the organizations that I talked to are using commercial tools to build an AI augmented vulnerability remediation But, can we also build this system using just open-source tools and agents? The answer is yes. And you do need some key components and some key skills to build the system using just open-source tools and agents. The first key component is context. Context is extremely valuable, especially when it comes

to gathering accurate results from large language models and agents. I don't think it's news to everyone, especially if you are a developer, you do know it better that context is everything. Building those lab pipelines, code base indexers, governing policies, MCP server, all constitute as a con- context through which an AI agent and a model will give better results. Now, over the past year, we were seeing a

trend of larger context window. But, we also saw that larger is not enough. In fact, if you have attended yesterday's session which happened in this very same room, the author who was presenting on a small language model also talked about that larger size was not just enough. And this is important because now what we are seeing is a trend of optimized context. So, you can also train

the open genetic AI smaller language model with the right amount of data and give them the right context and we can get good results out of it as The second key component is the security You still need open source SCA tools and SAS tools to make sure you identify the list of vulnerabilities. Now, they may not give you the exploitability of a vulnerability. They may not offer

you fixes, but they will still tell you where the vulnerabilities exist in your system. This is also telling us what key skills we would require to build the system. You need data science skills to build those lab pipelines. You need software engineer skills to put this all together. Context engineer skills, application security skills. So, with the help of these key components and these skills, you can actually

build this entire system using just open source tools and agents. So, how does this entire system actually looks like? I came up with this five-step approach. implement secure coding principles using Skills offer repetitive behavior for In fact, we have seen that when secure coding principles and policies are applied by skills, AI agents deliver better outputs. Second, use hooks or some coding agents are also providing workflows these

days. But the idea is that as a developer, you take back that control and you run a scan against open source SCA and SAST tools. They will give you the list of vulnerabilities, which is important for the third step. Now, we have got the list of vulnerabilities, but we still don't know about the false positives. We still don't know how to identify the reachability analysis. For that,

you should need you would need a domain specific language model. The domain specific language model will build a dynamic call graph by tracing the functions hit before a vulnerability hit. This model will also help out in And like I said, context is the most important part for AI coding agents. You can use open source AI coding agents, for example, Theia agent AI, Open Code, Kite, Continue.dev, Codium

Code. And you can submit the context to these coding agents and they will offer fixes for these vulnerabilities. Finally, if the criticality is higher, especially in that case, make sure that you review the code and make sure that it is secure and quality Now, this is not some theoretical approach that I came up with. And you might think that okay, we all know those definitions, so there

is no it it sounds so good in theory, but can we build it? The University of Singapore in collaboration with Google actually released a research paper where they build their entire AI augmented They initially used uh fuzz testing. So, they used Google's OSS-Fuzz program that was able to identify more than 10,000 vulnerabilities in more than 1,000 projects. But, they soon realized the bottleneck. And the bottleneck was

manual The mean time to remediate a vulnerability was 5.3 days. Where 10% of vulnerabilities were able to escape the 90-days deadline and leaving the software exposed to attacks. So, they adapted Auto an open-source LLM agent called AutoCodeRover into CodeRoverS. CodeRoverS was specifically designed to remediate security vulnerabilities. It was used in the OSS-Fuzz reports. And it was built in its own exploit input to identify the exploitability. How

it was actually doing was it instrumented a program that was hitting all the functions that were uh that were pulled out before the crash was actually happening. Now, this is this was uh given uh CodeRoverS a dynamic call graph. Along with that, it was given it a context as well. And CodeRoverS used this context to remediate those fixes. Sorry, remediate those vulnerabilities. Uh this system was on

588 C C++ vulnerabilities and Code Warrior released 52% plausible patches. So, it is possible. You can build your entire system with open source tools and Now, before I move to the last slide, I would again like to point out in which situation we are in. Supply chain attacks are elevated. We are already seeing that the window, once a vulnerability is publicly made available, attacks are happening in

24 to 48 hours. Sometimes, they are happening in minutes if if you were all at the keynote yesterday, where we had a amazing session on Ferrari uh sorry, Formula 1. And by the way, I'm not a F1 fan, but that was a fantastic uh presentation. We are right now in the game of margins as well with attackers. If attackers beat us in the game of margin, they

will be able to attack our systems. So, we need to act at high speed. We cannot just focus on yes, I have this backlog and I will remediate these vulnerabilities later. No, we will have to achieve a scalability. We will have to achieve high speed to fix these issues. With that, I come to the recommendations. In case you missed some of the uh content in the slide

or you joined late, the gist of the presentation is number one, you should use AI augmented tools and practices alongside SCA and SCA and SAST tools will give you the and they will help you uh in sorry, AI augmented tools will help you in offering fixes for them. They offer fixes by direct code fixes for first-party code bases and back porting method for open-source code bases. you

need to maintain human oversight. Human in the loop is important, but not To make it scalable, you need to Also, have a cross-collaborative effort to build rigorous security policies and layer your security with runtime security analysis. Third, establish a platform-based approach. The platform-based approach, like I said, is turning out to be very beneficial. We saw the example of Southern Company, where it reduces the cognitive load and

build out those augmented AI-augmented workflows, automating the security vulnerability remediation, and automating those CI/CD pipelines. pilot with open-source tools and agents to build your own cost-effective AI-augmented vulnerability remediation Thank you. I hope this was an interesting session to you. I am happy to take any follow-up question answers if you have. Thank you. >> [applause and cheering] >> You were recommending to have um, internal artifactories or repositories

and then scan them internally to basically know what vulnerabilities are in there. However, that would um, require the companies to set up the infrastructure for doing this stuff, which most open source projects already provide to the public. Um that also would mean that if I fix something internally maybe, like I fix a package, like this would stay in my company, it would never leave, will never go

upstream. So, I would um disincentivize the um people in my company to contribute to open source and maybe make sure that they have the resources, the open source uh package to uh take care of the security part. why would you And And if company A solves that, company B still has the problem, right? And company C and company and so on. That That is a good question.

And uh let me tell you the the real picture is, you know, we we do want and we do recommend uh that, you know, you should contribute back to open source. But many a times they have the expected uh government policies and security policies or something else that they don't do it. That is first point. The second key thing is what we are advising here is that

you should not be installing directly anything from the internet, okay? So, what happens is developers without doing a due diligence installing a package, you know, directly in the application environment. It may be, you know, take an example of log 4j vulnerable package, right? It It's been years since log 4j has released the secure package, okay? That is available for everyone. Even then, we still see that about

I think 40% of organizations still have the vulnerable log 4j package. And it's not me, it's the data. You can check it. I might be uh quoting the number wrong. I think it's much higher than 40%. So, in enterprise things don't happen that way. That is why it is important within enterprise systems at least you should have an internal repository where you first scan the package and

repository making sure that it is uh you know it doesn't hold any severe issues or licensing risk. And if it fits under your governance policy then you are hosting it in on your internal management repository. Uh internal repository and then you are environment as well for uh for development purposes. I do see some change in that where organizations are now happy to contribute back to the communities.

But it's still that's a long way to do. I mean very honestly uh speaking I still find uh I would say you know less than 10% of who are using open source but are also but also wants to contribute back to the community. Many a times we don't see that. And in fact within open source communities updated dependencies are always available. Like for example in log 4j

right? But they they just don't know that updated dependencies are available. Regarding the topic with like a package mirror I'm I'm also not super sure as to how relevant it is it is to the whole security in the loop because the security in the whole software production loop pretty much. Because having a mirror could have multiple reasons but um having to scan especially for security vulnerabilities every

package that you're uh trying to publish to your developers would in my opinion not be really Having a mirror for performance reasons sure. Having a mirror for uh for licensing reasons or generally like um it can compliance stuff sure. But I'm not sure there's a Personally, I can't see a world where you can scan every package that your developers would need beforehand. So, I'm not sure how

it relates. That's a like a statement. So, I understand your point, but uh the key thing here is uh See, if you want to make it a scalable, there are other approaches as well, okay? So, for example, there are artifact repository management systems, something like JFrog Artifactory and Sonatype Nexus. Then you also have uh you know, secure repositories, commercial repositories from ChainGuard, Lineage, uh there's another company

called Eco Software as well. Now, if you are maintaining an internal repository management system, I understand your point that how do you achieve scalability? And for that, you will have to take support of platform engine teams. The organizations who are maintaining their internal repository management system, they have a platform team support who regularly maintains the scanning of these packages. The The idea is that, you know, if

you leave it in the hands of developers, like I said, it's it's a data point, if if the safe package is available on the internet, but you install a uh unsecure package because you were not aware of it, then it becomes an issue for the entire And then you lead into a situation like Equifax. Yep, I kind of agree with the issue. I'm also not kind of

sure what's the solution. For example, with us being Mercedes-Benz, we've got like thousands upon thousands of developers using probably millions of dependencies. Yeah. Um and I'm not sure that there's that there's a repository having scanned every for once having scanned every dependency. Um secondly, having scanned it on time, for example, when when releases are happening happening in um published libraries. And then, I'd also kind of kind

of agree with the first question being asked, even if we are scanning our dependencies on internal system, which is not a not a managed um, service, how does the information that we've got that separately from the open source project flow back into the um, original repo? So, what I'm saying is um, yes, I think there could be benefits in in being super secure and scanning all of

your dependencies, but then again, uh, it's got to be fair somehow, and you have to pay for it based on the scale of your um, company. I I understand your point. I mean, it's not fair for open source communities, right? When you want if if the company has applied a package, they should contribute it. I'm talking to organizations who doesn't even know what exists in their system.

I'm talking to organizations and software engine leaders who doesn't even know that they are using open source, and they are building software. So, you So, you have to understand that point of view as well. From their uh, you know, concern, from their uh, relevance, it is highly important for them to build that system. And secondly, uh, like I said, the updated packages are available from the community.

Most often, we have seen that. Those are not just installed in the uh, in the application development environment. That's why you create this And we have also seen, you know, use of uh, AI coding agents. Like I said, you can connect it with your JFrog Artifactory, using which you can install the secured packages via the secured Thank you. Do you have another comment? Okay, then I have

another question, actually. Um, so, recently, when was recently, I mean, this morning, I read an article about Anthropic's Mythus Mythus Mythus uh, model, and kind of security scanning um, applications in the wild. One of the statements in the article was pretty much um AI usage in in security and for discovering vulnerabilities um might end up being um a fight of who's spending the most tokens. Yeah. Um

Have you thought about this topic before? What's your opinions on this um because um what I'm what I'm saying saying is okay, we can leverage um AI to reduce vulnerabilities, but on the other hand, there's people actively trying to um find vulnerabilities using AI. AI is a waste of Where are we here? So That that is the problem, right? You you have to I mean you have

to use more AI for solving the problems that are generated by AI. It's funny thing at the same time, but uh this morning I was having a conversation at this event only with with someone who are actually using Mythos right now. They have the exclusive access to Mythos. And they told me that it's it's not completely It's it's actually good. And uh all of the vulnerabilities that

it has given them is uh you know, they haven't find any false positive till now. So, it is serious, and it's not just limited to Mythos. There are a lot of open generative AI models, organizations are already attackers sorry are already using these models to identify And that is why you need to achieve that uh scalability as well. So, you need to uh use AI automated methods.

I agree your point, you know, who can um consume or spend the more number of tokens will be able to remediate those systems, but at the same time, you also need to make sure that you have a strong security processes, right? And many organizations that we see, they do not have any security gates. They do not have any firewalls, and they are just waiting for somebody to

attack their system. Okay, they do not have mature software engine processes. So, if you don't have that, then you need to protect your kingdom, right? So, you need to put those walls. Make sure and and and that is how you can actually optimize the use of AI as well. Putting those secrets and data at a safe location. And that's where I agree. I think the baseline for

everything is having hygiene and protocols and mindset and then on top AI can help. And then again, in parentheses, if if you've got the money for it. Exactly. Because of the Mythus thing, I think the number that was thrown around around was um spending 100 $125,000 worth of tokens before it became super super helpful in some cases. Which many companies just don't have. >> But the problem

is not of Mythus, right? It's not like Anthropic is the only company that is building models. It's just a matter of time. Yesterday, um like I I'll give you one example, okay? Last year, I published the strategic trends on the growth of open generative AI models. This year, with the release of Opus 4.6 first, then now Opus 4.7, then we also had you know, GPT 5.2, Gemini

3.0 Pro. We have seen an increase in gap in terms of accuracy again. Okay. But open generative AI models will also catch up. The other models will also catch up. So, it's a matter of time when attackers will get a hand of these models and they will start exploiting your systems. Exactly. Thank you. And that's why I'm interested and that's not the question anymore, but I'm interested

to see where this leads. Yeah. And at some point, it's going to be marginal gains again and let's see where it leads and who wins in the end. >> [laughter] >> Yeah, exactly. That that's what I'm saying that it's it's a marginal race, if you don't Uh, I mean, this this is still comes back to the original processes and fundamentals, right? You still need those strong processes.

You still need uh, strong system design and skills. Uh, you still need to put application security gates. If you don't do that, then you are in a high-risk environment right now. >> [music]