Nigel Douglas - How can we abuse AI hallucinations to feed the already bloated software supply chain

This talk addresses the challenges posed by the integration of AI within the software supply chain, particularly regarding security vulnerabilities. The speaker, associated with Cloudsmith, emphasizes the risks of type squatting and slop squatting, which exploit the reliance on open source software and the increasing prevalence of AI-generated code. A major focus is on the significant findings from a report concerning the security flaws that AI introduces, especially in terms of trusting AI outputs. The session also highlights recent incidents within npm and other package managers, illustrating the real threats of malware in a landscape rapidly shifting towards automated code generation. The discussion also includes proactive measures to enhance security, like using tools such as the OSV scanner to identify vulnerabilities in dependencies effectively.

start going. Okay. So, thank you very much everyone for attending today's session. We're going to talk around uh how we're going to abuse the existing software supply chain specifically with uh this high velocity introduction of AI. So, I think the clicker should work and if not it's fine for me just to click next on the So, yeah, I work for a company called Cloudmith. Cloudsmith are a

registry solution put it simply. Um, so you'd see other organizations in this image registry area. We obviously do other things than just software images, Docker containers, those kind of things. We also deal with uh everything that comes from these open source software package um managers. So things like PI, npm, Maven, those kind of hubs. So essentially the world is reliant on open source software written by other

people. We are just that registry so that you can securely consume uh those software packages and components. Uh I won't go into what cloudsmith is but this just gives you a a high level introduction into what my organization is working on today from a security perspective. So we're going to talk around AI. Uh we we did a report it's actually a year ago at this point. Uh

we've done another one since for 2026 but I think some of the reports still hold up today findings. Uh so the first part is around securityurities be well there's a security blind spot around AI. Uh we'll talk around this idea of type squatting. It's an age old security problem. Um but now slop squatting is the new variant of type of squatting and how that really makes sense.

Uh we'll look at existing research specifically in that area of slop squatting um and what that means for the industry at large. Uh we'll talk about in open infrastructure. I think this is a big topic as of well yesterday OpenSF opened up uh an existing paper um around the fact that we need to now think about open infrastructure. Everyone's using it. It's not free. It costs a

lot of money to actually run this infrastructure. We'll talk about the recent software supply chain attacks and again this all ties back to the fact that this is open infrastructure. It's open source and anyone contribute whether good yes but then there's also bad actors out there and we're I'm going to give a demo and I'm going to show really how we can do kind of proactive steps

to stay secure in this world. So going back a year ago, uh I remember someone wrote an article on Financial Times and it was a piece about vibe coding and why it's the DIY of the world where anyone can do it yourself like building your own kitchen furniture kind of thing, you know. And I wrote a small piece back to it. It really was just one of

those little letters and I basically said DIY is great. Anyone can build your own stuff, but if you're anyone like myself who's not very good at carpentry, you're going to end up breaking things all the time. And that's kind of where we're going at the moment when it comes to security. Uh, everyone's writing code, even non-technical stakeholders. Problem is a lot of this is like scaffolding that's

not kind of sticking together. And I'll show some of those examples in a while. Uh, I don't know why, but this very small letter I wrote started appearing in other publications like Forbes, and I really don't know why, but again, it was just that piece around scaffolding, so it seemed to appeal to other software developers who seem to resonate with that topic in mind. If you're not

familiar with the term vibe coding, I think everyone already is. People are doing real coding. That's where you think about what you do and you write it and you kind of have a plan of action. Vibe coding is going by vibes. So you're not really thinking about what are the design. You often just go by vibes and you just generate code and it either works or it

doesn't and you reiterate on that code. That's the simple definition of it. According to Andre Karpathy, it's just that uh concept of again using LMS to generate code. So AI is now a new security blind spot. Uh we ran this report. We had a bunch of different findings came out of it, but some of them were quite interesting and probably make obvious sense now in 2026. But

it's this idea that pretty much everything showing up in codebase now AI generated. I know obviously the heavy majority of our code isn't actually code we wrote ourselves when we think about open source dependencies, libraries, all of that stuff that builds up uh our overall uh codebase. But again of the stuff that is being generated and put into the codebase by us uh it it's pretty much

all AI generated at this point and that was the sentiment that was coming back in these survey results. Uh the other side was while people were saying okay we're generating code and that's all great we're solving the velocity issue most organizations were genuinely coming back from these tools like cursor and claude and saying yeah actually we're having some security concerns. uh when asked simple questions like do

you believe AI will exacerbate open source malware threats the heavy major said yes of course you know I I don't think there was any concern that we were going to introduce more problems because fundamentally if we don't know what we're doing it's the same old security issue before we just don't know what we don't know that's basically it um we asked other questions like how much of

your codebase was AI generated again how accurate these results are is really just survey results uh we surveyed software developers specifically using AI coding tools today. So we hope that the answers are somewhat uh substantially accurate, but again they're they're most were coming back saying yes, you know, we have AI throughout the codebase at this point. Uh when we're talking about where were they using AI today,

I think that's going to change a lot from last year's results, but a year ago, everyone was using it just for code generation. It wasn't in the QA process. It wasn't so much in design documentation and the deployment phase. They were really just generating loads of code. That was the velocity argument. that was the value proposition that came from AI. So again, that will probably change in

later results and surveys. Uh but when asked a question like how do you feel around malicious code and being able to see what is or is not safe within your code stack, I'm pretty sure people would have given the same answer before AI, but most are coming back or a large percent are coming back and saying, "Yeah, we we don't really feel confident that we actually are

able to identify malware within our open source stack." So the idea that we didn't write this code, we're less aware of what we're actually introducing into the codebase. So that's where type squatting and soft squatting comes into it. Um I I use this funny example where you know you take two drinks that look the same, they're stout and they're same color, same top, everything's identical. And people

think of type squatting like this where it's two different brands and you miscommunicate with them. It's more like simple typos. You know, you take two brands, they're almost identical, you swap out an A for an E, something very, very simple. Um, again, the common scenario we see here is that people find a topic squatted package, they panic, they remove it, then later they go through the whole

cycle again. It it's a a repetitive cycle. This has existed for years. This is non AI specific problem. This is just a simple cyber security problem. If you can file something, a website, domain, a software package name, and you can make it look legitimate to someone, you're hoping that they type really fast, make a mistake by spelling, and accidentally go to the wrong website or pull the

wrong package name. Very simple attack surface. You can see type of squatting here. You know, a simple example is you go to wellsfargo.com and you drop an O in com and you go to a wrong, you know, suffix or domain extension, whatever it is, and next second you're downloading malware. That is the basic example of type of squatting. Uh what slop squatting is is this idea that

we now use LLMs and they are blindly providing stuff to us. The attack surface is something very rudimentary like this which is I go to LLM and I say I need hugging face CLI because that is how I'm going to interact with hugging face. For those who don't know hugging face, hugging face is just a big registry for pulling LLM models and uh data sets and anything

to do with building your own open source LLM model stuff. The idea here is I don't know why but chatbt hallucinated and it started giving fabricated package names. It's this thing that happens. They're trying to address it in the the frontier models, but it's still an outstanding problem in the industry today where you ask for something, you'll either get the literal response back or due to complex

requirements inside your request. Um, you're ending up getting again the package didn't exist. Um, but what happened was a security researcher realized, hey, I ran pip install on that package name. And it literally just throws up an error saying that package doesn't exist. So this guy Barl Yardo who I can't remember what the organization he worked at but uh he basically in 2023 uh published a paper

where he said hey I just filed that package name when I realized it didn't exist. I was just curious how many other people were doing the same thing. uh he realized 30,000 people were obviously doing exactly the same thing because 30,000 downloads were made over a threemonth period pulling that fabricated package name proving the point that a simple example of hallucination in the LLM is people are

got blind trust now with LM you're saying we're moving fast we got velocity we're pulling things problem is it's it's not it's not surfing the web it's not going to the package registry and checking is this a valid package is this a valid author it's not checking for provenence it's simply just saying Uh yeah, sounds good enough to you and here you go. If it's not good

again by vibes, we'll just keep doing until we get the right CLI tool. Uh there's no design documentation or implementation around this. So again, simple attack surface. It's just an extended logic on type of squatting. But what people are realizing is if we find package fabricated names from LLMs, what if I just register those set names into the hubs? Whatever it be, Pi, npm, hugging face CLI,

there's a dozens of registries. So research in this area um this was a piece done by trend micro um again data is constantly changing um what was frontier at the time was like GPT 4.0 do uh we have moved a long way on since then but what they realized is across the board what was happening was the more context the more um complicated really the requirements were

within that uh LLM request uh within the payload is what is actually leading to these instances where they see in the heat graph more and more fabricated names. It just simply wasn't a please find me exact package name for a thing. Instead it was like complex deployment requirements and in the process it it it fabricated a package name here or there you know simple little mistakes but

again you got kind of bonded in that scenario. So in this one they went over 100 tasks again foundational models were predominantly finding zero hallucinations. That was pretty common. That's a good thing. That's the expected outcome of these LLMs. Uh but when they started bundling these novel libraries you know you started doing more themes. So you're learning like graph or wave socket. you're doing lots of again

different context into it. That's when it was sounding plausible and it was saying because again think about it an LLM is just predicting next word. That's essentially all it is a next word generator. So the more context you're adding in the next more possible things it can get hiccups on and eventually it's generating those outcomes and again the trend right micro paper is probably the best one

for showing these outcomes. Um there was a load of other things time related signals again if you're asking for specific time periods specific versions when you start asking for more specific things that's when it starts going wrong as opposed to generic uh requests and that that is just a known problem in LLMs today. Um they were seeing like Lama was a model that did really well. Again,

we've moved a long way on, but Lama in general do quite well. Um these open- source models um at addressing the issue of hallucinations and again kind of tailored for that purpose. Chat GBT isn't probably focused specifically on just being a coding agent. Um it's a general purpose. Again, Frontier Model um does pretty well at what it needs to do, but again, it makes mistakes. The next

problem, it's related. So we talked about slot squatting, type squatting and the fact that packages are going into registries and people are filing them and they are bad code essentially that we end up downloading but we download it from these registries. So pi pi the package python package index you know you have node package manager from JavaScript. So all the languages have their own registries. Um this

is a really scary um piece. This this came out yesterday. So again this is part two of an ongoing series from the open SSF. So this is the opensource security foundation for those who don't know and they are the ones who set up kind of governance controls and kind of really work and building open source projects and making sure that all of this is stable and will

work for years to come. So what they've said is look 10 trillion I'm not making these numbers up but you can you can look at the report yourself 10 trillion open source download packages are going to be made in 2026 uh from these indexes. Uh that is that's it's a terrifying thing. We're seeing a billion per hour. The idea now is everyone and their mom is now

generating code. You know, it's a reality. We're all told, let's use AI agents, let's vibe code, let's just be software developers. So, the idea is more velocity, more code generation, but also a lot of code. If I say, generate me a Python script.py file, uh it's also going to just generate a bunch of imports, and it needs to pull these packages uh these requirements, you know, and

they're all coming from these indexes. So, again, it's open infrastructure. It's free for us to use it, but it's not free. It's literally costs a huge amount of money to run uh this infrastructure. So you think about the likes of uh Russ Foundation, Maven Central, Pi Pi, npm. There's a huge amount of these registries for all these languages. Uh the numbers are constantly going up. Uh I

think this is just a scary reality of where we're going, which is more and more demand for AI, more and more demand for these registries. This is the software world we're living in today. problem with it this at the end of the day it's open source these are open registries anyone can commit anyone can get involved you know as we get more velocity from software development so

do adversaries hackers you know they can contribute to these registries as well so again I say the AI boom presents big challenges it's a big scary thing um which is we're seeing yes more and more legitimate code people can uh publish uh packages uh dependencies that they never could in the past which is great or projects they struggled at getting contribut contributors to their project are now

solving some of those pro uh problems that they had that were outstanding. And this is all really exciting. Uh again, I point to that OpenSF piece at the bottom about the hidden cost of running package registries. But the real scary part is as we're seeing more and more packages going to the registry and there's more and more authors, it's really hard to question the provenence thing of

what do you trust? What can you use? Did we even really have trust boundaries to begin with around the software packages we brought into those registries? So again, even though it's all great and good, bad people are I don't know if I did anything to disconnect that. Oh, it's back. That's great. So, uh, yeah, bad people are also publishing, too. I don't know why it's gone. Uh,

still on my screen. Will I unplug and plug back in? Okay, we'll wait two minutes. Does all this make sense so far to people? Uh, yeah. Okay, cool. And everyone's still enjoying it. Okay, great. So, now we're going to touch on the really interesting ones. Uh, recent npm supply chain incidents. I don't want to leave it at just npm. There's so much more than npm. Um, but

node package manager for those who don't know this where uh npm packages, your javascript packages go into them. Shyud was the big incident for those who don't know. Uh, well, first of all, it's the the whole June reference. So the big worm Shahulud uh that's the the campaign that they're running anyway. This malware this um data stealing malware it's it's really scary but you'll see like every

security vendor was publishing piece about Shy Hulu. This went on I don't know two or three months back but recently chainuard.dev they published their threat protection AI page. You can definitely check that out. um it lists all of these packages that were compromised and you can see that in these reports they're basically on a a near daily basis that another package has been published to a registry

whether it be npm or pi um and they're all terrifying in the sense that these software dependencies whether we know it or not the likes of Axios the request package these packages get millions and millions of downloads every day so we're all using it uh whether you know it or not it's a dependency inside our stack so if someone compromis mizes it, they publish a a bad

package to the registry and we say for instance blindly pull the latest from it because we're just five coding. Uh you're you're in without knowing downloading malware, do you know and that is the reality where we're at. So the shy hullet incidents we're looking at uh light lm and pi um there was the trivia incident the vulnerability scanner there was team PCP this organization that were actively

going out to publish this malware into these registries. uh we we've been dealing with a lot in the last uh two months really. It's it's it's spiked. Um and it just it it damages the trust in these registries because people are saying, "Goodness, I just got in uh affected by that incident and you're telling me 3 weeks later I've been I've been impacted again." It it's really

upsetting. Um you think about log for shell, you know, log for shell was a massive security vulnerability. Back then, not everyone was generating sbomb. Not everyone had a full understanding of what software components existed within their software development stack. So they didn't know did I have that vulnerability and if so what were the steps mitigated? Apparently 10% of year's enterprise security that year uh was spent in

just trying to mitigate uh the CV itself for log for shell which was the log forj Apache component. Again we're just going to see more and more of these appearing over the years. That's just a sad reality. But there is a solution. So I mentioned OpenSSF. They're the organization who steward um these open source projects and these registries and making sure open infrastructure is open and stable

for the future. Uh Google created this project, this uh fuzzer called open source vulnerabilities and then they donated it to the open SSF. So it is open source open infrastructure. Brilliant, you know. So here's a blog from 2022. again it's been a few years now um that Google published and they were just saying hey we released this OSV scanner um OSV scanner is awesome for those is

anyone using OSV here or is this no okay you should um for a few reasons it it's free so you have no objection as to use it open infrastructure the other piece is that realistically it's very very powerful there are lots of registries there are lots of uh threat informed reports Google's fuzzer scans the registries so it finds out is there something that is known to be

malware inside those uh packages, whatever defines malware, a crypto miner or type of squatting, whatever it is on a package name that's legitimate and they do all that for you and then they create these reports into this giant report database. So the idea is if you want to scan, let's say in this screenshot you scan your lock file or your um requirements.txt file, what it'll do is

it'll say here are all the dependencies, these are ones that are affected on these versions and here's our uh our advisory that actually tells you like why that's Initially was created for vulnerabilities. Uh but then they created another extension. [music] >> Collecting web data shouldn't be this hard. Scaling equals complex infrastructure. Advanced bot systems blocks you. Heavy data loads break pipelines. Oxalabs can fix this. Curious? Meet

us at booth 00021. Fast adapting infrastructure for web data. See it in action. Realtime data for AI models. Track pricing and market trends. Reliable business intelligence. 177 million IPs, 195 plus countries. Oxalabs fast adapting infrastructure for against it either by hash or they can also do it based on like package name and version very simple stuff so [music] again we don't Google all They scan it from

super awesome across all again they're doing it all for us. There's also other projects that come along like OSN which are dedicated just to look at taxes. [music] at booth 0021. >> Use that API amongst others. So in this case, it makes it easier to read. So I'm basically [music] saying [singing] I'm check it's going to be so in this case [music] that's a simple change right

that's type in its finest I roll the team investigating those two guys and suddenly you get this huge out. The idea here is it's telling us okay [music] >> collecting web data shouldn't be this Oxalabs, fast adapting infrastructure I think if I say cubectl we had this script here actually I think I probably [music] already did it what I'm doing [music and singing] [music] the again wrote

this custom script. The reason for it is I have a running containers. This container is another code. So I want to scan it and I want to say what are the packages. And the idea here is you can run it whatever way you want. It's an open API. So in this case, I can check, hey, I could have dozens and dozens of dependencies that exist in the

container. But if one of those are type of squatted, I want a simple way to query and say, hey, look, in in clear as text, that one is known to be malware. You have malware in a running container. Kubernetes doesn't really have um malware scanners. It's not really a thing that's thought of in Kubernetes mainly because Kubernetes is supposed to be like uh dynamic and pods die

and they relive and they go up and down all the time. But if in your build process you're introducing malware that was type squatted as per you know vibe coding you're just going to keep reintroducing this malware over and over again. So um again it's a really useful way of um of scanning and identifying the malware. Um the other thing I wanted to show off and I

think it's really important to show it is oh yeah this was also how to create the fake malware. It's literally just created a fake name and ID. There's actually nothing in the file. But the thing I really wanted to show off was Oh yeah. So you can just run a post request against the API quite literally again JSON structured output that's all pretty normal. But the other

thing that's I think very important here is OSV scanner. So I think I had an example from earlier that I had running up here. Um again dozens of different scripts inside here. OSV scanner. So I have a requirements.txt file. Very simple. It could be a lock file. Any of those formats are recognized with an OSV scanner. I mentioned Google mentioned that they have this OSV scanner. uh

you can run as CLI or you can run as GitHub um you can run as like u an action you know all of these things can be defined so in this case I have a text file my text file is literally cat requirements txt so you can see it's just requests light ln pandas that's that's it but what it'll do is by scanning those three packages on

those specific versions it will give you the full audit trace to say hey here are the packages you can see light lm requests on the versions You can see what source it was scanned from. Imagine you can scan literally your whole file system and get an advisory on everything. Um or external hosted location. So again, you can use it whatever way you want. But in this case,

when we talked about that MAL ID, it's saying that light LLM on that specific version has both a Pisc advisory but also the malicious advisory. So you can go to osv.dev, go to mal whatever. And that's it. I think I'm done on time. So thank you very much for attending.