Who Is Securing the Code Your AI Wrote? - Biswajit De

This talk focuses on security challenges within the DevOps and AI landscapes, especially related to vulnerabilities in software supply chains. The speaker, with a background in the security industry, discusses how traditional methods of identifying vulnerabilities are inadequate. They propose a solution centered on producing secure, hardened container images by compiling each package and dependency from source. The speaker also delves into the risks associated with using AI for code generation, highlighting issues such as vulnerabilities in underlying frameworks and the importance of Software Bill of Materials (S-BOM) for transparency. The session emphasizes the need for robust verification processes in managing both first-party and third-party code to secure development pipelines effectively.

About me, uh I I I have primarily from the security industry for the last 20 years. Uh I'm based out of Singapore. And I 3 years back, 2 years back rather, we started this company called Deep Start. And before I dive into the details of this session, which is more on a particular threat, I would like to tell you what we as an organization what we figured

out in the DevOps world and in the DevSecOps world, the biggest pain point is everybody is shouting vulnerabilities. And there are new tools who are coming up and telling this is vulnerable and you need to sort it out. we we figured it out what should we do and what are what is the industry doing? The max the industry was doing was going ahead and telling you that

this vulnerability is exploited, this is not exploited, you should allow me to and should not allow. Uh we figured it out that we have fundamentally solved this problem. And the only way to fundamentally solve this problem is that why don't we give clean container images as such? So, and then we discovered it's not about giving container images because many a time we are dependent on our same

containers, etc. We figured it out we have to source compile each and every package and dependency. We want to keep it running and we will show you what happens through our AIX app, but then we did So, we figured out that we need to give images, we keep a track of the packages, the source code, even the libraries. of course, in gate hardening. So, we figured it

out that uh just giving an hardened image is not solving. So, we started shipping two variants of every image things. One is a dev and one is prod. The dev, of course, is a shared that we write. Prod is contained in the score, only runtime, read only, shell-less. So, developers can the development environment. Perhaps when this copy make it tools for that. It >> [snorts] >> We

also figured it out that there are problems uh it's not as easy as you tell that if you just give an image, which shell-free and and does not write to the root directory. So, it's no you need to give them charts, you need to give Kubernetes manifests, uh how it writes to different storage. So, that's what we do and we are constantly trying to do that. And

of course, AI as well, attestation, provenance, FIPS uh all side parts to it. Do they know who wrote the code and it's like code and they're putting it into the hard to data center. So, it's completely opposite to the whole concept of zero trust, where we are putting it on our employees and other people whom we know we can stand accountable for. But when it comes to

our actual product, even some which we are shipping to our customers, we are clueless about what is happening. And this cluelessness is increasing further with the advent of AI. So, when we talk about the olden days, and I'm talking about 1 year back, and what happens today, uh we all know white coding, we all do white coding, we all love it. Some people who are not great

at coding, like have turned out to be coders, right? And uh And And typically, uh we are And what the AI is doing is pulling the base image, putting the libraries from everywhere, installing stuff you are clueless about, installing dependencies that you are not aware of. And it's giving you a finished product, which is looking great, and it does great stuff. You can give markdown, you can

do low code, but does it not solve the problem? Because if something needs something, it it uses it. I I think I'll shift over here, it's easier for me. So, let's talk about the whole AI, and let's break it down. I mean, I'm not great at simplifying things, and that's what my CEO says about me. Uh I kind of make things complicated, but yeah. So, LLM When

I When we talk about LLM, the model is a brain, it's nothing but a collection of weights. Right? It comes as a dot bin file or any other format, per se. And this LLM sits around a framework. A PyTorch, a TensorFlow, etc. And this model needs a runtime to run on, which is Ollama. Now, when we talk about securing LLM, LLM is a collection of weights. The

best thing that you can get out of an LLM is a model card. What data was used to train it, etc., etc., from a supply chain perspective. It's not you cannot sandbox an LLM. And if anybody tells that they sandbox LLMs, they can't. What you can What you can do is the biggest problem statement is around the frame frameworks. And these frameworks have Forget the frameworks. Let's

talk about the underlying runtime, vLLM or Ollama. If that has vulnerabilities or that is compromised, what do you do? And with this back pressure of organizations have to adopt AI, then there is another pressure that you cannot use open I mean foundational models. And then you try to download your custom model and you do it and you host it, this is what you inherit. And of course,

you cannot use foundational model because you have sensitive data. You cannot let it go outside. And then you have a problems associated with it. Let's associate an attack that actually happened on CVE 37032, right? And it was a simple HTTP request-based attack. An arbitrary code execution that happened. if you and it's just a normal pull that right single crafted HTTP that happened and it went through. and

it was simple way it happened. A A request malicious HTTP request, the attack navigates outside the intended directory structure, and it writes a malicious script to a writable location. what we see is the step six. Now, once it is inside it interprets the planted script and runs arbitrary code. And now it can do whatever it wants. So, inherently you have given access because typically you are not

experts on that Ollama code or you are not experts in that you have not written that code. But when you have implemented that solution out there, you have implemented the vulnerabilities, implemented all the weaknesses Which can drill down into your organization and your stuff. Let's take a look at Ollama today. Because I I tried to use a real-life example so that it is relatable rather than just

talking about this attack is happening, etc. If you can go to GitHub Ollama and this is and Ollama's written in Go and this is the go.mod file. I've highlighted one example, Go Crypto. This is today's page, just taken. He's using 0.43. Can anybody guess what's the latest version of Go Crypto? Any developers here? 0.49. So, what does this mean? That the Ollama maintainer and rightfully because it's

open source is not taking complete his if it's his vulnerability, he's taking responsibility for it. But if it is these library-based vulnerabilities that is used to do compilation of his Ollama, he does not take. But do you inherit the vulnerability when you're you're the package and installing it? The answer is yes. So, we are not talking about AI security out here. This can app be applicable to

if you are deploying a local Kubernetes or Kyverno or OPA, everything is written in Go. You are deploying itself to secure your your Kubernetes. But, it's written in Go and it is inheriting these If the maintainer has not gone upstream and fixed it. what is the And if I give you a uh story out it, this is the What is surprising is not the vulnerabilities. What is

surprising is the date. And this 37032 vulnerability, though it has been detected, it has not been fixed. Though there is a POC available on how it can be exploited, it has not been Today. And the date is when the CV got released. What can you do? You cannot do much. Because if you have to run a model, you have to use Ollama or vLLM or any other

counterpart. And you as organizations, as freelancers, as the community people, cannot do source compilation for everything. Right? The only way to fix it is recompile Ollama with the fixed That's the only way. Right? Because if you take an Ollama package, you cannot install a library on top of it because that's part of the package itself. I I hope and I started off by apologizing. I tend to

think makes things complicated. I hope it's not. So, if you look at Now, let's come down to another reality is the S-bomb. I am a great believer and slowly slowly with compliances and other things coming and not even compliance uh S-bomb, I think, is a simple hygiene. What is there that is included in my image or in my VM or anything in my food? And then only

I can say that I want this or I do not want this. So, I can take an informed decision that should I take this or should I not take this. And the biggest problem in current day S-bombs that it is invisible to us libraries. And how many of you heard of SPDX? CycloneDX? Okay. These are S-bomb standards. Newer standards like in-toto has come. when you talk about

these things, the one of the biggest problem is you are uh How many of you work in product companies that you ship to other customers? Right. Do you have S-bombs and do you see if they have GPL licenses? If they have GPL, that means people can sue you. GPL means it's open source, but it cannot be used commercially. So, understanding this thing like PSD, GPL becomes very,

very, very important. And it's not important until you have visibility to the same. So, S-bomb as a concept is very important. Now, coming back to our Ola my example, we can see these red spots out here. And what it And those are the malicious back Not malicious package, those are the vulnerable packages. And when we talk about the Go runtime, I explained to you I would not

divulge more uh until you have questions. That the libraries And And that is the case for Python, that's the case for Java. That's a That's a case for NPM Right? And we had the NPM Axios based Right? Very recently. And something in October 2025 as well. So, it's not that we are talking we are not talking reality. This is Right? Uh You are depending for code on

somebody you do not know, is faceless. He might have a reputation, but what if he's compromised? His Gmail is And compromising somebody's Gmail is I think pen testers at uh a very demo version, they do that. In today's world. So, so And that's how your uh That's as easily as your maintainer can get compromised. And if it gets there can be multiple ramifications. Yeah. Now, coming back

to AI coding and wipe coding. Now, this was a problem with your developer. But your wipe coder is also a smart But the only difference is he does not tell you what he does. How he does, why he does. You can have markdown files, etc. But, you'll never know at what logic he will go astray. am a lover of AI, but what I believe is it's like

a very smart child. And you constantly need to sit with a stick and tell it what to do. There is nothing called autonomous wipe Yet. So, the answer is it's a very smart developer, but he will do whatever. But, how will you teach him the god rails, etc.? That's very important. How many of us know how an LLM model is trained from scratch? From scratch. I'm sorry.

There's a gun. Like, if I create my own model from No, it's a data about the state of the The first data set that is used to train any model is GSM 8. That's mathematical algebraic data. Second data set that is getting and and that is multiple levels, of course. The second data is medical data. And third data is legal data. For every models. Medical data teaches

it deep reasoning skills. And you build on its reasoning capability, it's not teaching it to be a doctor. Tuning the weights to think analytically. Finding root causes. That's why. And legal data is taught because it gets to know when to understand what not to do. What are the rules to become a rule bot? The Those are the three basic data sets that are initially used. Then you

can take that model and train it on what servers. mathematical data. So, it starts off with pure play addition, subtractions, and then it goes into algebraic functions, Right. So, that's how weights are tuned. Now, what I said is that the agent is blind. You're You're doing a blind trust to the maintainers. And now you have an agent who is coding, that's also blind, pulling in everything from

anywhere. How do you go about it? the NPM attack or the NPM compromise, which has million plus downloads, I wouldn't ask you to please raise your hand if you have traced it in your organization. That's not the right question to ask. But yes. And with DevOps world, with the velocity, a I push Somebody pushes in a malicious library in the DevOps world, within 1 week it can

land in land in your production. Because that's fast, that's agile, that's DevOps. That has been taught to us over the over the last 4-5 years. And that's where we have been going. But what we are inheriting, and do we have a clue? Do we know? That's what it is all about. So, uh in the last 10 minutes, uh and the problems I think um I just have

uh I explained to you the problem statement Now, how do we go ahead and remediate I just told you about an upstream based vulnerability. I showed you this is still nice showing 3 to 21 days. I showed you a 2024 vulnerability already exploited, not fixed. By a maintainer like Ola Amas' In this AI generic word. So, that should be the highest priority, right? But, a developer's priority

never is security, sadly. It only stings when it when it goes to your production and your dev sec ops or your sec ops team flags it. Right? And it cannot go into production. And that's when you realize the problem. cutting the story short. If I wait for a go.mod update upstream, it's not going to work. And what do we do? So, now this is uh, the solution

part that I'm going to talk about. What we are talking about is, first of all, zero shells, completely hardened, no root FS. Then we are talking about minimalistic, so distroless. Distroless is you create an image from scratch and only install dependencies that are needed. And do not install and remove certain dependencies even if they get pulled in that is not needed. BusyBox is a primary example. we

waited for the Busy Box contributors to solve Busy Box vulnerabilities. They did not. So, we went it ended up writing our own Busy Box. And somehow I'm a lover of Rust. So, I made it a Rust-based Busy Box with an ASH base shell, etc. Uh we are going to make it open source. So, if you guys are interested, you can let me know. so, that's what we

are doing. Right? So, minimalistic and of course source compilation. We are compiling from source and fixing But, we also discovered one thing. And when we went into security, when we discussed with the security minds, etc., we figured it out. If I do source compilation also, I'm solving part of the problem. How do I catch an NPM or an Axios kind of attack before it even becomes a

CV or a news? And with that concept, we came up with the concept of the verified source Where we verify each and every open source component, every line of code that comes in. That means we look at the code this, we do an AST analysis, we do a type setter analysis, completely deterministic analysis of what it is, and then we have correlation rules. So, for example, if

it's an obfuscation paired by a web API call at the back end, so, the risk score increases. And based on this correlation and the risk score, what function it is calling. We are also doing a baselining of the maintainer. So, we have three pillars, identity, integrity, and intent. So, intent is gone through code. Identity, we baseline the maintainer. From where does he do commits? How many commits

does he do? What is his code complexity? What is his commit ID normally? Right. So, all of those things. And of course, the integrity part. So, so, we are going towards a very We are doing source compilation, but before compilation also, we are adding that card to do this. what we are trying to do is rather than we are trying to be a single source for libraries,

secure source for libraries, for images, and for packages for organizations to consume. A ton of our images are already open are we a distro? The answer is no, we are not. We are distroless. what do we What do we do? So, if I can fast forward and I can speak on the slide, so, we give we make secure container images and all the components associated with that.

Uh we are doing source code verification, not your third-party code. Third-party code is from your developer. So, you know what your developer does. You have full control. You can catch hold of him. But 90% of the stuff is not your first-party code. It's all third-party code. 90%. If you say that somebody writes a Python program to integrate with single sign-on with AWS and Azure or 0365 for

example and make it on AWS etc. He will just pull in those libraries and those SDKs and write collaborative program which will call those SDKs. Right? How many of you use Grafana? Or have you used Grafana? Now, there's a problem with Grafana that when it ships, it ships with AWS SDK 1 and SDK 2. SDK 1 is phased out by AWS. But that means Grafana still has

the SDK 1 vulnerabilities. How how are you going to cater to those vulnerabilities? And even AWS is not supporting. But Grafana would like you to be backward compatible etc. And at the very end, I think I would like so we are doing these are numbers are misleading. It's around more than a million repos now that it is gone. So, this is what we are doing Linux packages.

But the moment we went to libraries the number of repos that we are monitoring is gone to the millions. The dependence relationship, the advisories etc. and having said that what we are trying to So, I think you figured it out what we are trying to do. We are trying to be a secure supply chain vendor. We are giving hardened so that when you are putting into your

DevOps pipeline it's kind of a vulnerability free. And of course, it's a never-ending A new vulnerability comes, a new image gets shipped. And a new vulnerability comes, a new So, I stay a lot in the US. Uh I was going ahead and some of my Indian customers have told me, "Bis, do you support backporting?" All right. And then I want to end this on this note. So,

said that let's do something about it. And let's brainstorm. Let me interview people on what they think. Like, for example, we give n to n minus three versions. So, if you're on Python, we give you 3.11, 3.12, 3.13, 3.14, on request, 3.10. But let's say 3.9 is no longer supported upstream. And the customer is still using 3.9. Can we backport and do it? My engineer said, "Yes."

Couple of my US customers, Indian customers have asked for it. But when I interviewed my US people, they said that the moment you say that, "My developers will never move. And we need them to move. So, let's not give them the choice." so so the mindset is so different. Right? So, the mindset is so very different. And and I think uh it was it was incredibly the

more people we talked to. Uh I'm humble enough to to that I am still in my learning phase and and I'm I have learned a lot from my engineers, from my freshers, from anybody somebody comes up with something, etc. They teach me a lot of stuff. on that note, I think that you have been a wonderful audience for bearing with for such for 30 minutes, right? I

promised you 30 minutes, right? So, So, thank you and it was great talking to you. Bye-bye. >> [music]