Kalle Sirkesalo: AI-Powered Slopsquatting: Is Your Software Supply Chain Compromised?

In this talk, Kala discusses the growing issues within software supply chains, particularly focusing on the impact of artificial intelligence (AI) and the vulnerabilities introduced by using ready-made libraries. The speaker highlights that many organizations are eager to adopt AI but struggle with its implementation, as a significant percentage lack proper integration within existing workflows. The talk examines how malicious actors are leveraging AI to exploit software weaknesses, emphasizing that significant vulnerabilities often stem from outdated or unmonitored package updates. A concerning trend is that many developers rely on open-source packages without understanding their dependencies, which increases security risks. Kala warns of threats posed by malicious code in popular libraries and urges for greater awareness and proactive measures in software development practices.

So, I just said before that like I had a really positive panel about software engineering, where we are and all of that stuff. And now it's time to destroy everybody's hopes and dreams and everything around software. So, I'm Kala. Uh, I work as a field CTO nowadays. inside chains to grow. But I'm going to be talking about supply chains most of the time and about the software

supply chains and how we're basically being destroyed by AI and it's different use cases. So most of you probably see these kind of posts every now and then and you're like laughing at this. So AI wipe coders being in the internet and being like oh my god I'm being hacked and I'm being like I'm not technical I need help. The sad thing is the same things that

these guys are doing is happening to our enterprise software. So there are most of the time just prompting against AI and it's using readym made libraries. It's making using invented libraries in worst case bit less now and it's getting constantly parad with different kind of CVS vulnerabilities and capabilities. It's what creates this problem that we are seeing like a massive amount of organizations saying that they want

to be faster. They want to make more AI stuff and then in reality uh [sighs and gasps] 91% of the organizations have no idea how to implement AI. There are 85% of them are like yeah it actually didn't integrate at all into our existing workflows. You've probably seen this like you probably have about six AI tools now and you're like none of them actually work on the

workflow that I want them to work and then you try to solve it with cloud code and it becomes this massive headache where the management is trying to figure out where's the AI you're trying to figure out how does it even work because nobody's training you on it and on the other hand the attackers are actually inventing how do you use it and it becomes a massive

headache in real life situations. The fact is they models the whole iss like it's it's vectors in matrix. That's all it does when you're using an LLM. Everything magical happens outside of the GPU calculation actually. So when you see AI doing thinking for example that's a for loop or a while loop. You can think of whichever way you want. It's going at back and forth against the

model and creating a plan and structuring things. And that means that most of the time this is just information that you're giving to a tool. Well, if it's information that you're giving to a tool and stored state of a historical situation, especially with AI, that means one thing every time. It means that while we're automating, the hacker man is there. He's coming in and he's trying to

get into your systems like they've always tried. Like most of the people in security are in security because we had a nice hobby as a young wings where we really enjoy trying different things and accidentally got into one system and we got that rush and we were like, "Oh my god, I'm going to get into security." Then we got into security and we realized that's not actually

fun. But you know, one can dream. The big thing is these guys most of the time they're not hackers. They're still script kitties. They're most of the time just taking a readym made tool from internet copy pasting it and running it on terminal. Now they don't even need to go to internet and copypaste the thing. They can just install cloud code and tell it hack this system

and it will pull tools from the internet and try attacking against you. They don't even need to do the five minutes of finding a hacking tool. Uh why it's problem? Well, it's because when we give bad data into a system that stores data, we get bad data out. So, uh this is Sonatives previous year's thing because I didn't bother to get the this year's data because it's

the same. It keeps going up and it keeps going up and up. We're getting about a million malicious packages per year at the moment to these different systems. So uh you can imagine the amount of attack vectors that are in npm python or not maven central. This is from maven central. So this is from a dead language just like who writes Java today? We we've never seen

Java anywhere. It's still one of the most popular languages. But yeah, so the big thing is the amount of vulnerabilities that we're pushing into the system is ridiculous. At the same time, the whole software engineering practice that we're doing is kind of work in progress. The worrising thing here is this 13% of the downloads of Rock forJ are still vulnerable. We're still like not updating those packages.

We're still like running old packages on different systems. We're still getting them from internet. We're still having this situation where zero 0.5% of the code that is open source doesn't have an update. But if you go and ask an engineer, they're going to tell you that there is no update for it. But the reality is most of the buckets actually have an update for the CVS or

things that are found. We are seeing for example in GitHub if you look at the open source community being like oh my god we're getting all these attacks or like not attacks even pull requests about vulnerabilities and you might be scared oh my god so many vulnerabilities there's 600 security incidents open here it's all the same it's that one vulnerability that your AI finds and it's not

actually even wound ability it's the same when you ran Yenkins and you got a secret now is coming in and they would run against the Yenkins and they would be like I can do remote code execution on Yankees and you're like yeah there is a sale command in Yenkins you can do remote code execution on Yankkins that's kind of the point of this system I don't know

if anybody of you got that but like I had a really fun time with one security auditor talking about shell commands in yeah 80% of the applications depend depend on [sighs] things like human upgrading packages. It depends on someone actually going in and changing a version of the number. Like most of the people have not actually heard of what is a dependabot on engineering side. Most of

these packages are they don't even understand why should they update packages. On the other side, this year, if you noticed, updating packages is getting also bad because if you update your packages immediately when the new version comes out, you might actually get some kind of a worm on your system. We're going to be talking about worms in a bit, but the big thing is you end up

in a situation where most of the time people are not actually updating. So you think you're safe from the works, but you're not because the problem is we're very dependent on these open source systems. We're very very heavily gotten used to installing different packages. You can take white, which is very small JavaScript library. You get 200 packages on your laptop. One of those packages needs to have

a remote code execution, a secret stealing leakage or cyode in it. If you don't know what cihood is, you will know by the end of this session. But 200 packages and that's third biggest JavaScript library. The second biggest is Nex.js that has depending on which you choose 280 to 600 external packages in it. So when you run it and install it a new software, you get 280

to 600 dependencies on your code base. Nobody's going to review those manually. By the way, nobody knows what goes into it. And you can develop those with basically any language and it might not actually even support anything else than JavaScript. It doesn't. And now with Nex.js, they just released the Rust version of it. So now you're RS programmers here who thought that you don't have these dependencies.

You have the same dependencies by the way now. Thank you. Uh fantastic. And then the worst of all the original dupregated React install which most of the React applications have been done had thousand over thousand packages in it. So when you did the React uh quick start you installed thousand JavaScript libraries on your laptop. How many of you actually did that? Because I I for example did

that because I was trying to study how to write react. I don't I still don't know how to make front end. But that was great. Uh there's also research that about depending on where you are, you can end up in 20 to 40% of hallucinated packages because the RAMs are not great at not hallucinating packages. This is less if you're using something like uh cloud code or

open code or github client because they are nowadays verifying against the source but the original lms that you started using and if you haven't improved your usage and knowing what you're doing can just invent new libraries. Uh I tried few things and none of them actually worked in today's harnesses. So the ecosystems that you use but half a year ago if I would have prompted I got

uh three viruses with one prompt because the it just invented a library that is in CER and it tried to bring it in NodeJS because I was trying that kind of things and someone had pushed a malicious code already in those systems. Well, what happens actually when you do this and this stuff happens is you get into these situations like for example Axio. So you start getting

compromised on MBM. So like you get these Marisus versions into the source code. It can be it can be not maricious and then it gets hijacked. So you have like left bad for example. Everybody remembers left pad from their nightmares. Uh if you don't, happy for you. Uh it was a package that got removed from node uh npm and after that it got replaced by Marius's code.

[snorts] Everybody who installed it got hacked. Uh same thing. So Axio, one of the AI libraries got taken. And after that the new versions 1.141 and 0.3.4 for were poisoned. So if you got that into your CI/CD system what ended up happening is they installed the remote access troen and they got access to your So they could get to into any system that this library got deployed

to and because it worked normal it went through the CI pipelines normally and got all the way to production in certain SAP had the same thing. npm package is compromised and it harvested the developers and CI/CD secrets. How many of you have a GitHub token in your uh laptop's uh secrets? Yeah, that's gone at that point. Now it has full admins to your GitHub. How many of

you have uh at the moment for example an open some kind of AI token in the uh environment variable push? If you have that would have gotten taken. Now they can generate code with your tokens. So free tokens for them. They can push it to GitHub with your name and they can uh easily use whatever you have as an access in your environment variable. And that was

through one variable, one package. So fun times this is this is the world we're living at the moment. The worst thing is you have to actually get notice it on your laptop because if you push it to Git GitHub or GitLab, what it does is it takes the secrets from your CI/CD and sends those also to the attackers. And now what do you have as a secret

in your CI/CD? How many of you actually have separate uh environments and profiles and secret fing production versus staging versus comparable? And how many of you actually get all of those secrets? So, how many of you fetch secrets separately? About 110th of the room. How many of you have secrets that are production secrets in your CI/CD? It's my other hand because my other customer uh so all

of those would get taken and sent to the attacker. So now your network security defines are you safe? But if you're using GitHub runners, oh, I have an attack vector to all your source code and the GitHub can actually access to your deployments and I have your GitHub token already. So I can push it also to your GitHub so I can change your GitHubs to deploy to

production. Not great, but that's the current day supply chains. Um the actual actual work that goes into debug debugging and finding these is really troublesome and work work a lot. If you don't believe that's enough and you think your language is safe, go models, ruby gems, same credential theft, PyTorch, if you're writing Python and doing analytics, your analytics teams got get stealing. So it's in any language

that has dependency, we're seeing the same behavior of stealing the tokens and attacking the actual delivery software that we made to make things easier, faster, and better. And just to make it more hitting home, you can also develop it with AI. So when you use right or used for a certain period that got hacked by team PCP and now they have your whole AI gateway managed by

them and they got few of these. So that means they got free tokens to everything. It was a persistent backd dooror. They could decrypt your encryption. they got everything people were asking. So if you for example as one of my customers does uh had an HR system connected to AI because they want to answer people about sick leaves and comparable with AI they don't collect that info

themselves because they are not allowed but guess what the hacker can nobody is against that the hacker collects your cancer or whatever you have and you're asking in your system how do I actually apply for So it's in everywhere. So if you're not scared enough, uh you might have heard of this. That's Sihood from Dune. And we call it now mini sihood because we made it smaller

and it it it's attacking in smaller section. But what this is doing is this is hunting in the word of GitHub. So if you follow that team PCP was behind the previous attack. According to today's sources, they have gotten 4,000 of the GitHub's internal repositories. Not sure if that's true. GitHub is looking into it. But the big thing is they uh what happened here is they got

the worm into one library. They got the GitHub tokens, the npm tokens, and they made it into a worm. So it gets self-replicating in every single repository that did good so that they would have access to every place. So the mini sai who over 24 hours got into 160 different packages got into mistral got into tanstack got into almost every single place that we uh could think

of that like is used in AI code and its whole job was to just keep stealing credentials and get everywhere. And then it sent those credentials back to the uh staff of the attackers and allowed them to get into the systems and start attacking. Actually, how we could actually stop many hood is we would have to rotate every single package manager in the internet, every single token

and every single GitHub token. For some reason, we haven't done that yet. Anyone want to guess why? Yeah. Yeah. It's it's a bit of a lot of work. So, what your developers see when they're working with AI is their IDE, their instant messaging system, and the version control system. They don't see the web UI. They don't go to PP, they don't go to MPM. If they go,

they're very well educated. Uh I wouldn't believe in that. So where you need to tackle against these systems is in systems that are built around these three. We know the instant messaging system they're not going to read any messages that you post. So you have to make a technical systems or make it into cat pictures. They might check your cat Uh so what we start usually is

we built protection around the IDE. So we uh start creating different protections around there. So we start putting some type nexus or J for uh creator services for example. We start creating different kind of prevention to prevent them from pulling systems. We're going to get into those but I have some questions. So raise your hand if you have personal data on your machine. Yeah, that gets taken.

You have admin access to your machine. Yeah. Now the now the attacker gets AI You can run commands to production from your work machine. Oh, that's that's very low actually. But they have no access to that also. You have secrets in your laptop as command line variables. that's the statistics from the same paper. Uh, so it really confuses packages in JavaScript. So game over you might say.

So it's a game over if you pull a language model and the malicious user gets the package and the package gets to your package repository and gets to your normal user. So what you have to start doing is you have to start at preventing it before it becomes malicious. So we need to start attacking systems here in the package registry. We have to start creating systems in

the normal users laptop. So they wouldn't pull those ward packages we are very far behind at the moment in many organizations. So we want to be exploit uh preventing exploitation in many layers and it's not just package housation as we saw because now we've found a way to use the package housation to upgrade ourselves to higher and higher tiers to get actual production libraries into use. So

I made with G and I a really handy nibble of like info how to actually tackle this and how do you actually run commands in different languages to use dependency cooldowns you'll be seeing this and it these talks are recorded but the key thing here is what this means is in different languages you can define how long since it was released in different package registry do I

pull it from? So you don't get it and let it go and get the package immediately. You give it for example 24hour cool down because at the moment if it's a popular package, if you're using a popular package, most of the package managers are hunting these things very tightly at the moment. So if you give it like five days, they probably remove malicious packages from the system.

So you're about 60% more safer still high chance of getting attacked but you have different ways of communicating and protecting yourself. So you can create like minimum release age you can create uh I mean ruby doesn't exist but like net users have a possible way coming in and there is also update their tools where you can define So that's something you can do that's the easiest way

that doesn't cost you any money. The sad thing is this requires it to be in every single developer's laptop and that's difficult. So the other way I have it from the Nexus firewall but it's the same principle in uh Jrock. It's the same principle in any package manager that has capability. new component would be get deployed by someone. It arrives to the package manager. This can be

any component in any package manager and it gets evaluated by a vendor in this case sonotype because I'm using sonotype slide in JFrog's case Jro security team and before it gets allowed into your system they tell it that it's known safe suspicious or it's a critically wound up or malicious code and then it gets moved to quarantine So that way you don't accidentally pull But the thing

is you have to actually combine this with your package manager and everybody has to use a remote How many of you are using uh only JFR or Sonotype or Harper or comparable packages and never going straight to the source? Oh, that's like five people. Yay. So the big thing is most organizations see it too tedious because it requires training staff, it requires network controls and it requires

protecting the end devices so that you are not actually allowed to go to those pages. You can also take scanning approach. So you could have a light listing for the vendor but that doesn't really help. Um oh I have lost my one slide. So I'll go back here. Um the big thing here is do things where you prevent the network access in the CI/CD at least. So

if if you're fine on the developer laptop getting taken. So if you're for example working on a laptop but working on the actual systems where the customer data for example is you can easily prevent internet access from the RDP or something else or another laptop or you could just do zero trust security which is a business case. This creates a really good business case by the way

for zero trust. uh you can prevent the internet and that allows you to really tackle against the access towards how to say external packages from the CI/CD. So what we do with most customers is that we don't prevent the developer environment because that's difficult because then they just go to an external source and they use that external source to pull the packages because developers I'm one of

them. Um so what we do is we prevent CI/CD. We we do a protection in CI/CD. You can easily put everything going through a squid proxy for example or if you're in cloudn gateway or if you're in somewhere else you can put like a access control limits and you can guide the traffic to your package manager and force those users to pull from there. So you can

also do scanning the new programs that people are allowing. Checking an application whitewisting vendors. You would have some kind of whitewisting vendor like dependency track for example. Um you could then check with the external database that they would use and then you would have a manual approval for a new applications and at that point your developer has already gotten the package somewhere. If you could try, some

organizations can do this and then you could uh try and get time and resources to put into this manual process and then you can trust certain developments. This is easy in terms of like work if you have one system and one language. The problem is according to again research and based on my research the 62% is it's about 98%. Uh today I heard about 900 different CI/CD

tools in the pipeline. So like uh 62% of de uh developments and companies using six or more tools to develop software. Sounds small at that point. Uh I have so far averaged about 20 different DevOps tools in an average CI/CD pipeline and that's an average. Some people have more. So when you start putting these packages and package managers, they become part of multitude of points because it's

not just the source code that you're doing because the same attacks can happen against these tools. I didn't put the slide in but uh if you look at for example runtime protection and you're like we're protecting our runtime where 3B got hacked same way as I was saying before. So what happens? since you pull 3B automatically and now you have an remote access available to every single

machine that is running 3B. Yeah, not great. Uh cooldowns, put the cooldowns in. I also run uh DevOps tool chain analysis. So I go and check people's DevOps tool chains. Now I'm also doing AI maturity, but that's a different report. And I wanted to highlight in red, as my colleague in history said, every single box here is a security issue, but I wanted to highlight in red

the boxes that are critical on security. And the big things are most organizations that I have talked to have more than one version control system. And those of you in small companies are like, how is that possible? Like why why would they have this? And then you're like, oh, we have an Azure DevOps. Ah, then we have GitHub. Uh, then we have this guy still stuck in

SVN and then we have CVS in use. And then you're like, what's CVS? Well, most of most of the young links are in what's SVN? But like if you know CVS, then you're like then you then you've seen some stuff. On the other side, most organizations have no idea how their deployment process works. If you missed the talk, there was a great talk about uh from uh

Janis, you can check it on this uh videos that come out about deployment processes. But most people couldn't describe to you how they deploy software. They would have no idea what happens from the idea to production. They will need like six other parties and they will sit together on a table and do use an hour to mark what goes into a deployment process. And after that they're

like this is about 80% sure and then you ask like okay what's the 20% and they're like well we have this other team that is doing this and you're like okay how much do they do? Well that's 50% of our revenue actually. Ah And then you're a security expert on these two and you're like okay I want to update a lock forj package. You need to know

where it is and you need to know how to get it to Those two boxes prevent you from doing that and you have no idea how to get Then documentation is everywhere. So you can't even go and read it. You can't even get the AI to tell you how do I actually release something because AI is really bad at finding multitude of sources. By average, I've come

to a conclusion that it's about seven to eight systems that people put uh documentation in. And you need all of that to understand how do we actually release something. That's going to get better by the way with AI because everybody needs to put something into markdown and then you have everything in the repo so you can actually know about topics. Then you get into okay we are

going to update the package finally uh except the testing and quality is not well defined. So you don't actually know who can say that it actually works. How many of you have a really good data set of test data in the testing environment and you can say that it works after you change the variable in um on remote like u dependency and you see it going up.

I don't see any hands. Uh that tells you about the test data. then we have my favorite box because I keep hearing from people that like security is getting so much money and I'm like no it's not like come on we all know there is no money in security. Uh the big thing is how you get money as a security is not through talking to management about

security. It's going to talking with business and talking with them how could you enable them. Uh that's a separate talk but uh yeah security is everyone's mind and everybody says it's high priority until you ask money. It's my favorite also for sales. Like they tell me that this is critical feature for customers. I ask how much are they paying for it and they look look at me

quietly and I'm like yeah okay that tells me that it's not actually critical. Um then this surprised me. Uh so EU is releasing an cyber security act which requires you to have a software bill of material and when I started doing this almost nobody in our customers had a software bill of materials. So if you ask them what open source packages do go into your system they

would have no idea what system had what packages. They would just have a big list of these are the dependencies we have in JFrog and you would ask where is this package in you would hear like empty room and you are just like okay fantastic. Uh I hate saying this out loud because uh first of all secret management is the wrong term but secret management is after.

So, how many of you would say it's a secret if you uh have a secret uh if you tell it to 200 people? Yeah, that's not a secret. How many of you would say it's a secret if you tell it to five people? It's still not a secret. Like that that's we how many would you say it's a secret if it's told to nobody? That's pretty The

secret managers themselves are capable of doing fantastic work of keeping the secret secret from even us. But us as humans have not engineered like if you ask our cooperatic expert here uh like you can easily integrate any secret manager to Kubernetes and you would never know what goes into those pack BS and you would never know the secrets. How many of us have actually done that? uh

almost nobody. You're supposed to rotate your secrets. You're supposed to not know the secrets. And then most of us find like the password 1 2 3 4 as a placeholder password that went to production because nobody remembered to change it has happened uh multitude of other fun So basically what I'm saying is all of this leads to the fact that the su roots etc have become a

vulnerability for us because we haven't actually engineered any of our pipelines any of our processes or any of our things into a state where we would be able to rotate a secret for example. So now if GitHub informs us all the tokens are removed, how many of you would have a really really bad 24 hours? Okay, three of us, four of us were like, "Okay, yeah, I

I I know it's going to be bad." The rest of you are going to find out if that happens. uh usually I want to say something about what to do in this case and my answer is think about fixing some of these and I know you're sec okay I'm a security expert I can't really fix these things. So how I usually approach this at the moment is

figure out how it relates to AI. Documentation is everywhere. Yeah. Sorry guys. I mean uh we can't really give AI a context because we don't have the documentation related to this topic. Should we think about like centralizing this context into this context management system? Yeah, sounds really good. I got the documentation. You could use green for example or there are multitude of other tools that enable you

to search through multitude of sources and get documentation. Uh yeah, our source code is everywhere. So we don't really have this capability of giving the AI a capability of doing changes across our code base. So you know that like these automated agents are not really working well. Ah hey maybe we should go to one version control. Yes. Oh we don't really we are not really able to

monitor and guard rail this AI because we have so many different admins in the version control. So should we like make a actual like pattern of preventing these code changes from being pushed automatically? Ah sounds good. You can start thinking each of these boxes as an AI enablement. And surprisingly there is actually money at the moment in AI. So AI foundational problems is the number name of

the slide. I'm supposed to be out of time here and I am. So,