BSidesSF 2026 - From Noise to Notes: Orchestrating SAST with Developers... (Adrián Puente Z.)

In this talk, Adrian Puente discusses the challenges of implementing Static Application Security Testing (SAST) in development workflows and how to effectively bridge the gap between SAST expectations and reality. He emphasizes the importance of integrating security seamlessly into the software development lifecycle by shifting left and empowering developers to catch vulnerabilities early in their work. The speaker shares practical strategies for tuning SAST tools like Snyk and Semgrep to reduce noise and increase the signal-to-noise ratio, focusing on actionable findings and prioritizing critical vulnerabilities. Additionally, he introduces AI-driven remediation approaches that significantly reduce the time needed to address security issues, enabling developers to implement fixes more efficiently and effectively. Overall, the talk concludes with insights on fostering collaboration between security teams and developers to create a culture of proactive security.

All right. So, the next talk is from noise to notes orchestrating SAS with developers through AI-driven remediation. This talk is presented by Adrian Puente. Adrian is a principal security engineer at Remelt Remotely based in San Francisco Bay Area here where he specializes in application security and DevSecOps. So, let's welcome Adrian with a round of applause and then you can take it away. Thank you, Adrian. Well, first

of all, huge thanks for to the B-Sides San Francisco organizers for having me here. This community is incredible and I'm honored to be sharing this with you. So, think about the last time your CI/CD pipeline broke because of a security finding. You got 50 finding, 30 of them are false positives and now your deploy is blocked. You're frustrated, your engineering lead is frustrated, and security we are

just as confused because because we don't know why the tool we paid for is making things worse, not better, right? So, this presentation is is what is that about? How we close the gap between the SAS promise and the SAS reality and how we fix it with a process. a quick introduction about me. I've been working on cyber security for 20 plus years. I started as a

system administrator and then got I got hacked and I wanted to be part of that. Then became a consultant, I did pen testing for eight years across three countries, two languages, and then moved to DevSecOps broad security. And I'm laser-focused on making security work with engineering teams, not against them. Sci-fi reading and gaming, that's my sanity check. You can find me at Chucks, c h z e

r o k s, on Twitter and most social networks. So, let's jump into it. Quick baseline [clears throat] for those new to SAS, it means static application security testing. So, you analyze the code without running. This is the keyword. Like we are not actually running the application. We are checking before it actually get deployed. So, imagine of a spell checker for security. Instead of is this spell

right, you're asking is this code handling data safely? Are there hard coded secret? Is it Is it exploitable? >> [clears throat] >> The why we want to do this is we want to shift left, move security earlier in the security develop in the software development life cycle. So, developers can catch issues during development, not after deployment. That's the promise. Moving shift left as much as possible. So,

now let's let's talk about what actually happened when you roll this out. So, what you get from the vendors, right? It's like you we are going to catch vulnerabilities earliest. You're going to shift left. You're going to empower developers. But in reality, what happens is that you connect your you plug the system into your your GitHub into your GitHub organization and you end up with 5,000 findings.

70 70% of them are false positive. Your developers ignore it all. You're being hated. You're now a backlog. You're not a security program. You're just a nuisance for engineers, right? So, so [clears throat] we went from expecting harmony and we ended up with cacophony, right? A bunch of noise and confusion. This is the story of what happened to us. So, this is kind of the diagram of

timeline expressing music. We started from noise to to notes. this is the agenda. So, the first part is going to be early noise. The problem we have we plugged the system. We ended No one nothing is what working, right? Then we started orchestrating. We start working engineers with the musicians in this story. We find we we look for harmony. And this is how we tune SAS to

work with our workflows, not against them. Metrics per session, workflow embedding. We can part of the workflow of engineers. And finally, we embedded AI the solution, right? Because that's the new thing and you have to be part of that. So, the magic how we use AI, I call it vibe security patching. You can also call it as love security, why not? To close the finding to fix

this gap from weeks to 48 hours. So, by the end, you're going to have a practical playbook that you can take back to your organization on Monday and actually implement it. That's the purpose of this presentation. I want to share you my experience so you can you can get there faster. So, the first part, the early we have a bunch of tools, we need we have a

couple of problems, we don't know what to do with that. Everything's chaos, right? Let's just go over the real challenges. So, choosing the right tool, we [clears throat] needed a SAS tool. So, it's important to know that this problem was already been tried. When I I got the the implement I I got the pro the program, it was already been someone already tried to to make it

happen and it didn't work. So, I have to start with something that was broken, prove that it was broken why, and then start over again. So, we we needed a SAS tool that can integrate with our in-house CI/CD pipeline. I don't know if this is a common problem. I'm probably you're experiencing it, but in my case, we have a very specific way of deploying things. So, in

the developers doesn't need to think about infrastructure. We need to match the engineering team deployment model and the decision was we chose Snyk Pro because it was fast and flexible with native CI/CD integration and customizable rules that could be tuned to our code base. So, this actually helped us work directly with the bug bounty program and the pen testing team. So, whoever the bug the the hunters

they found something in our system, we can actually create customizable rules and then looked for the same pattern in our code and actually reduce the amount of money we were paying to the hunters. So, that's part of the story uh of what we did. So, the rollout, we started very very with 100 repositories, something small. We were just testing it. We tried a couple of companies. We

ended up with Semgrep. Again, POC is small. It works well. Everyone was happy. And then we took 1,000 repositories. It was around we have to be very intentional of what we wanted to scan and with what because we only have a limited amount of licenses. So, we ended up scanning 1,000 repositories and we got became victims of our own success because we ended up with like 4,000

findings. At that point, it looked like that's noise. That doesn't work. It's overwhelming. We don't know what to do. Where what the problem we're going to be attacking at the beginning, right? We need to move fast and and move efficiently. Also, we we were a very small team. We only It was only me and other contractors, so we need to scale quickly and and fast. >> [snorts]

>> So, what do the developers actually care about? Like, is it real? Can I fix it? Will this block my deploy? When 50% of findings are false positive, the answer is no. We don't care, right? So, they ignore everything, even the critical ones. So, we need to actually justify why it was a critical one and why it needs to be fixed. Why engineers need to actually put

time on that. So, without [clears throat] trust, even perfect findings are garbage, right? And you only reveal through source you only reveal trust through developers' experience, not findings. So, at this point, we were like I was >> almost about to look for another job. And then I come up with this idea of okay, let's let's find the harmony in this bunch of noise and not broken things,

right? So, to fix this, I had to turn down noise into signal. We started prioritizing purely by severity, dropping low severity rules to reduce alert fatigue, and focusing on you know, high critical severity confidence. So, this is important. Synk Pro has these uh kind of rules that you can use the community ones, but it's worth it it's worth to you to pay for it because you actually

have access to a specialized rules that have higher confidence on being actually a true positive and that actually is a game changer for for a team for a small team that needs to scale quickly. So three strategies. We start filtering severities. We stop chasing low securities. We don't care about low securities, maybe mediums we can look at them once we figure out criticals and highs. So let's

let's just focus on the criticals and highs, right? With that we actually remove remove the the attack surface that we needed to actually look The second strategy was to use Synk memories. We need to reduce the false the false positive. So Synk actually allow us to add context to the findings and to the repositories. So that actually help us flag certain patterns that engineering were doing like

for example, we have a lot of false positive related to HTTP being used and then we discovered that it was a pattern because a lot of engineers were while while developing they were using localhost. So we don't care about that kind of finding, right? You add that to memories, you get you get fixed. You you actually remove all those false false positives. Now Synk assistant that actually

help us kind of figure out how to fix this problem. This is the the part of the tool, right? That that's the only reason that that was actually help us to approach the problem. So we reduce from around 5,000 findings to 800 but quantity wasn't enough. We also need to to prioritize by risk, >> So we focus on So from all the repositories because of the amount

of licenses we have, we have to actually find those that we care about, right? So we focus on compliance scope systems since we are a fintech. We are we care a lot about PII, regulator regulatory data, etc. etc. Then focus also on system that were re- related to payment processing, auth, that kind of thing that we cannot actually impact our customer experience. Data classification, availability tiers. So,

we didn't care about that system that can fall tomorrow, like it can actually die and no one cares. We care about that that is actually impacting our business, right? So, we ended up having a coverage of 40% of the whole organization, but we were actually covering 95% of the real business risk problems. So, that was the kind of a game changer. Now, we actually have a strategy

to attack the problem, right? This is the big move. You're not trying to fix everything. You're actually fixing what is matter what it matters. >> With that, we then we move we already have the strategy, yeah? We have we we have the attack plan. So, now we're going to go we're going to get involved with engineering into their work processes, their workflows. We start adding our tool

into the PR comments, in-life feedback, uh right there during the code review, right? So, that everyone was like working, you know, you get to the PR and it will actually tell you, "Oh, this is a problem." And with assistant, it will tell you like, "Okay, this this is a recommendation on how to fix it." So, now engineers can actually know how to attack the problem. CI/CD checks,

blocking on critical, but also allowing engineers to break glass, right? Because if we are having like a like a um large security problem, how you call it they call it large-scale event where we are we have a system that is down and someone needs to actually push some emergency code into the system, you don't want to block them. You want to enable them. So, we were part

also of their break glass exception. And [snorts] then we integrate into a Slack and Jira to have real-time severity updates. So, now everyone can actually own part of the problem. >> The then we we became more developer-friendly. Like it's it's not just about throwing findings, even doing it self-service works for certain teams. But when you want to you want to report the the findings, you want to

be clear, actionable with actionable messages, not just like, "Oh, we found this problem." No, it's we found this problem in this area, this is the risk of not fixing it, and this is our recommendation how to fix it, right? That kind of closed the gap between we just beginning to yell at people to actually working with them. So, when signal leaves where developers work and they can

act on it immediately, that's when adoption happens. We also were very lucky because there was a team dedicated on just doing a technical debt, and they adopted Semgrep, and we gave them access for self-service. And at some point they were they fixed like 600 vulnerabilities because they found an old and forgotten library in Oracle that we didn't need for the for the application. They deleted and we

start like, you know, fixing problems at that point. Okay. I think I No, I already went. No. So, turning noise into into Yeah, I think I Okay. So, severity filtering is still chasing. I think I went Yeah, sorry. Okay, so measuring what matters. The adoption rate on high-risk report scan it actually went up. Now engineers were happy, we're having information that can actually work on and remedy

it. The fix rate increased. Now we we are reducing time it takes from a detection to actually fix. And developer engagement was actually We They're they're triage They're triaging their own findings, and they're actually working with engineering. That's That was like a really good success story at that point. Security become an enabler for developers to you know develop code secure by default. So what change? We got

false So just to summarize a little bit the story is we we got fewer false positives with higher signal, right? Faster triaging and less frustration because now we have a process on how we are going to be triaging focusing on just high critical high and critical severities with high confidence and we are just focusing on those systems that will actually damage the the business when um if

if it goes down or get compromised, right? So that's that's something you can actually tell to your executives. It's like this we are working on this and it's taking this long to fix it and we are reducing the risk on this way. >> Clear prioritization and reducing friction. Friction is super important at least depends on the team. You have a really strong security team with a security

with a high strong security culture. You can collaborate better with Otherwise you have to do a lot of influence without authority and you have to negotiate. That's a soft skill that becomes very important the smaller the team and the smaller the company. So now the cherry on top like how we close the last mile with with AI. Right? This is the part I think everyone come here

or at least where everyone is talking about how we integrate AI into processes, right? So let's let's go a little bit back. Where let's go back a little bit from the beginning like where traditional SAS stops, right? SAS will tell you like the tool will tell Sorry. Tells you what is broken and where it is broken, but it will not tell you how to fix it, right?

And the problem with these security bugs is that a lot of people get panic when they see security besides the word bug. It's a still an engineering problem that it requires to be fixed, but sometimes even security engineers we don't we are not software developers and software developers are not security engineers in most cases. Therefore, there's a gap on how we can actually approach the problem and

actually as security engineers, we tell them the problem and how to fix it. what is the usual usual workflow is you read the finding, you report it, you go and Google it, maybe you find something in Stack Overflow, understand the context, how many layers you have to work on and it maybe it's not just one file, etc. etc. You write the fix, test it, deploy it. If

it works, you're gone, high five, the alert is gone. It will take like three like it can take like an hour, right? Uh for every engineer to deploy this. >> Now, multiply that for a thousand It it is not scalable, right? So, how can we still automate these these steps? So, this is where I Vibe security patching comes to. Uh we start with the triaging. You have

a backlog, you actually remove, you know, the noise. You just focus on the things that you care about. You aggregate. It's very important like context switch context switching for engineers is very expensive. Having to to switch from one vulnerability to other it takes time to, you know, dump all the information from your brain, up upload it again, and then work on it. It's just something that you

want to avoid. So, you aggregate things by vulnerabilities. And then what is the vulnerability you found, in which repository, and then you get a list of the files that are affected. You take that code and then [snorts] you do uh you are group you you group similar tickets issues together. For example, you have a 10 SQL injection findings, create one aggregated ticket. And then you take the

AI, you can clone that information. There's different approaches. The the best that has worked for me is using augment with uh with cloth. I think it was Opus or the other one. But uh you can take the code. You you you don't put in your into your uh you make a clone. You start your favorite LLM and you take the vulnerability from Semgrep and you I have

a blog post on that on that subject. You can take a a laser focus prompt that will just remediate a very specific part of the code. It's very important because as we know like the LLMs will take everything and in some cases it will actually uh delete stuff. So you want to have a laser focus prompt that will just generate the patch for that specific vulnerability. And

then you you create a a commit and you do a pull request. And in the pull request I I really recommend you using the words this is a security recommendation. I need an engineer to actually validate that this is not breaking anything. Why? Because you don't want you don't want to be liable for someone saying well security told me to do this and now it's broken, right?

You want to actually uh just pass the that um responsibility to the owner of the And with that you will and then you you you just tell the engineer or you put it in a ticket in your Jira ticket and like I I just proposed this PR. It's it's a recommendation, please validate it. And in some cases in in my case I was able to remediate around

78 high high severity vulnerabilities with three PRs and it took me like a week. So back in the day it was I was able to reduce the time for the remediation one from 130 days to actually 17 days. And now I can say that I don't have criticals or highs and I'm on high confidence in the rules of Semgrep. And now I'm working on the medium severity

and I'm just working and I'm just looking at it, like I actually just give a work to my contractor and the contractor helped me create the tickets. I have templates. I created a GPT companion. Like you go to we have open AI, so we use ChatGPT where I created what I call GPT companion that uh just walks the whoever is working on it is I'm trying to,

you know, first become a workforce, use AI for as a workforce multiplier and also to enable juniors and maybe not that expert uh security engineers to actually do the work of a staff engineer, right? By guiding them with this uh very specialized uh GPT companion. So, you have that, you just throw the the the finding from Sangrep, probably the code or the ticket and it will create

a Jira ticket. Then you clone that code and put it in in your hard drive, use augment or whatever your flavor code or whatever and you use in the prompt. And that the whole process it will take you like hour, 2 hours. So, it's 2 hours where the security engineer can actually come with a recommendation that it will take 10 minutes to a secure to a developer

to actually approve. And you're golden. Um that's pretty much it. Uh So, what is the transformation, right? We have we have all this before AI, we have all this noise that are chaotic, engineering is hating us because we're not actually providing value. We don't know how to report why we're spending this much money on this system. Uh and then after this is study that I just mentioned,

right? Working with the engineering understanding their processes, how they work, their hopes and dreams and then you can actually help them get there in a secure way in a securely way, right? You can help them develop secure by default. And AI is this that enabler that connects just the the to the to the actual remediation. finally, I want to give you some takeaways so you can you

can take this, go back to your work, and look like a champion to your bosses. Um the first takeaway uh SAS adoption only works with developers, not at them. So, empathy through the development experience is non-negotiable. You have to work with them. You have to understand them. You have to make them become your allies, right? So, kind of like uh in this game where you're like "Walla

walla walla." You go to the meetings, and then you bring them to your side, and then you work together, hold hands, and dance Kumbaya. You can sing Kumbaya. Um you have to reduce the noise to signal to find signal. What it means you have to actually find high-quality findings, and then do high-quality reporting in a way that is actionable for the engineer. Right? Like you you need

to add a context. You need to aggregate. You need to focus on the thing that we care. Something that has worked a lot for me because the challenge that I get is "Well, why should I care about this? Like you say it's a risk, but it's a high risk, but I don't think so because we are behind 17 layers of uh proxies. Why should I care, right?"

Uh something that has worked for me is to take that those those findings and use CVSS 4.0, and actually put it in in in ChatGPT or some LLM cloud in this case, whatever you're using, create a whole report. You have to add a lot of context. Like for example, "Okay, this code is actually running in a container within the environment, and only certain people can touch it,

and only certain data is going through it, etc. etc." You add that to the CVSS 4.0, and then you get up and you put that in a ticket as a recommend and you you explain how you're using it. And then the engineer can actually come to the same conclusion. So, you want to create processes around the way you are reporting things in a way that engineers can

actually get to the same conclusion. Right? [snorts] It's not you remove the subjective part. You just build through these processes. That actually has worked has helped me a lot to reduce the pushback. what else? Reduce the noise. Measure the outcome, not the findings. Adoption fix, time of remediation. From there's a problem to here's the fix is a massive hit. It works a lot it really it work

really well. So, what you can do on Monday if you're starting your SAS program, choose the tools that fit your workflow. Right? Uh not industry hype. This is very difficult, but you need to get the numbers. A lot of something that happened to me is that oh, I want you to look at this company because the the cousin of the CTO is the founder and it was

just like a startup with 10 people. Um but then everyone wants to use all executives wants to use the the garden the garden quadrant, right? Oh, I want to use you look at this and then you end up with something that doesn't work and it's it's costing like $20 million a year. That's the complicated part. If you are going to be the driver of this, you need

to get the numbers and you need to show how why why you're choosing this tool. Focus and start and start small. Get a PLC of like 100 repos, something small. You need to build through like is stories of success, right? You you create milestones. So, you start with something small, you show the value, then you have more you you can ask for more budget, more people, more

engagement. Uh you just build your uh credibility in all around the team and you manage to actually be be golden and and you can actually push push for more initiatives. Uh if you're struggling, let's say you you already started your SAS program, my recommendation is audit your noise. Are 50% of the finding false positive? Why they're false positive? Where are you getting this information? Is it like

why is not a high quality a finding? If yes, change your metrics. Maybe you are actually not reporting the right things or the things that you care about. Track adoption and fixes, not raw counts. And and also reevaluating your rules, turn off low severity noise like right? I I you I I think I killed like 30% of the the findings by just ignoring the 30 the the

low severity noise and the informational one. So, we don't care about that one. And [snorts] if you're already running and it's kind of running well, but you're not scaling because your team is very small, I recommend you to experiment with AI Transform your AI access into like a workforce multiplier. Create agents, companions like I mentioned. And then give access to people to to experiment with it. Ask

for feedback and reiterate. By the third time you have like a person working on it, you probably have something really good that help you get there 90% of the time. Um AI is your friend and it's here to stay. We need just to use it. Uh partner with engineers, not just finding like for example, by talking with engineers, I actually managed to get a contact with this

team that was working on technical debt and they loved the idea of having sand rep. So, I gave them access and they become my one of my partners. The other is understanding their processes, the way they have the break glass for emergency deployments. Instead of me imposing the the PR blocking, I work with them. So, they it became part of the break glass. So, by the time

if for some reason they need to deploy something and and there was a PR blocking because of security issues, they can just bypass it and then we can deal with that. Right? But we need to actually keep the company running. Um so, pick one for Monday and commit to Uh any questions? Thank you so much. Um I just want to also give another outlet for questions. If

you want to ask questions online, you can anonymously. It's www.bsidesf.org/q the letter N and then the letter A, Q&A, and then I will read them. Otherwise, you can ask them live as well if you prefer. Yeah, we're going to get But, um thank Adrian Adrian for your my talk. Let's go. We're very comfortable here. >> [laughter] >> Thank you. Uh first of all, thank you for that

talk. It's a great subject, close to my heart cuz I'm a consultant and I deal with software teams that struggle with this. Um and I also I want to thank you cuz in your previous slide in the summary, you said uh ordered by severity, but then earlier you said ordered by risk. Risk is the way to do it, not severity. Severity does not equal risk. So, I

was really excited that you said, "Hey, figure out you know, what the risk is and prioritize that way." That's how they should do it. Severity does not account for that risk. Um the other thing I usually recommend is fix it if you can, right? Like if you can just fix it, don't doesn't matter. High severity, low severity. If it's if it's quick, doesn't break any dependencies, just

do it. If it doesn't, then consider what is the risk, how does it fit into my threat model, should I fix it or not, that kind of thing. But yeah, great talk. Thank you. Thank you. Thank you. Yeah, no, there's always these situation where uh is this going to be more expensive to fix it than actually get the problem? Right? If not, then you don't fix it.

You just keep You just accept the risk and move on, right? To the next thing. Yes, thank you. Great talk. Um you mentioned experimenting with different LLMs. Do you also or have you explored using all LLMs in this case? I think there's a there's an issue of vendor lock-in when when think about experimenting and then using only one for the auto remediation or AI-driven remediation. I'm curious

on your thoughts on using one vendor or a mix of them for your own modeling. Okay, that's a good question. So, in my experience right now, I'm because like again, I'm trying to scale. I don't have a lot of work a lot of time and I'm just jumping into whatever the the company is actually using right now. I think it has to do a lot with the

prompt that you are submitting and the quality of the model well, depending on the vendor, right? So, you can use Opus or Sonnet. And usually Sonnet works really well. It has worked Opus is more expensive. I think it depends on what the company is because at the end a lot of the vulnerabilities at least the ones that I worked with they are fixed with some input validation.

So, an input validation can be remediated with one with one function changing things in several lines on different files. So, some of them are really easy. Some of them are like when it comes to like logic, like you have a whole thing and whatnot and saying it would really good at telling you where the the the faucet the faucet and the sink are. And that can be

more complicated and more tricky. And that's usually you use something smarter, right? Like Opus or Gemini 3.5, I don't know. One of those. I I have only used ChatGPT for companions because we have the enterprise license. So, I can just point people to a browser and they use it through the browser. And they are not very technical. So, I just need them to actually create tickets in

Jira. But for something more specialized, then I use I clone it and I put it in I use augment or Claude and it works. So, yeah. Thank you so much. Do we have any other questions? Any other follow-up questions? Awesome. Well, in this case, thank you so much, Adrian. Again, a round of applause for Adrian. Thank you so much. We do want to remind you that there

is, you know, coffee available. You can get beverages from also upstairs. You can get some food from upstairs as well. This is our last talk for today. So, thank you again for staying and listening to this amazing amazing talk. Yeah, feel free to reach out to me on the internet. I'm more than happy to to discuss this. Great, thank you. >> [applause]