Daphne Preston-Kendal – The next xz attack #FOSSBack

This talk discusses a series of sophisticated security attacks on software systems, starting with a historical example from 1983 when Ken Thompson introduced a backdoor into the Unix operating system. The speaker explains how Thompson embedded this backdoor in the compiler to hide it from detection. The session then shifts to a modern attack attempt by an entity named Jia Tan, who manipulated the build process of XZ utilities with the intent of compromising SSH, affecting servers globally. The talk highlights the complexity of both the technical execution and the social engineering involved in this attack, which included creating fake user profiles to build community trust. The discussion concludes with a warning about the scalability of such attacks, especially with the advent of large language models that could generate numerous fake profiles to undermine software projects.

Good evening everybody. The year is 1983. Ken Thompson, the inventor of the Unix operating system, wins the Turing Award. And in his Turing Award lecture, he stuns the room by describing a security attack that he carried out against his own operating system. He messed with the login program. So that no matter what computer that was running Unix, he tried to log into, if he gave in his

username and his password, it would always let him in. But that wasn't good enough, because he couldn't just hide this attack in the source code. He had to hide in the compiler. So the compiler would always say, "Oh, I'm compiling the login program. I'll insert this backdoor, so that Ken Thompson can always log in." But even that's not good enough. He has to hide it two levels

in the compiler. The compiler notices when it's compiling itself. It hides It hides the code that recognizes the the login program in the compiler. And now it is completely vanished from the source code. Nobody will ever see, unless they happen to be reading a disassembly of the of the compiler, nobody will ever see that this attack has been deployed. The year is 2024. Actually, 2021. An entity

calling itself Geatain decides that it wants to diddle Xed utils. But actually, it doesn't really want to diddle Xed utils. We'll get to that. So it uses Xed's build tool to diddle Xed utils. It inserts special magic binary into the compiled finished version of Xed utils. Um but doing it through the build process again, so it's not visible in the source code. But again, Jia Tan doesn't

really want to diddle XZ. He wants to diddle SSH. And SSH is used for every server worldwide and depends on XZ. Everyone here has deployed SSH ever? Everyone who's not raising their hand is a liar. You have all you every server in the world practically uses this. So if Jia Tan had been successful, every server in the world would be able would be accessible by Jia Tan.

This attack was detected and the real world deployment diddle binary was prevented by total chance. A impatient developer called Andres Freund noticed that on his pre-release deployment of this software that it took him half a second longer to log into the to log into his SSH server and he happened to have the know-how to work out what was going on. So this is fine. This would certainly

be worrying if it happened again. But one of the things that is also notable about this attack is essentially not only how complex the technical engineering was, but also how complex the social engineering was. So this this entity called Jia Tan, but it's not acting alone. It has created a lot of fake users that seem to be supporting what it wants to put in the source code.

What Jia When Jia Tan comes along and says, "I want to make this patch to XZ." These users are there to kind of back him up and say to the maintainer, the single maintainer, "Hey, why is why haven't you merged this patch yet? Please please merge it in. I need I really need this." It took a long time for Jia Tan to get this attack in get

the actual attack in. 3 years between Jia Tan's first interaction with XZ and the actual attempt to deploy it into a release undetected. And in that time, Jia Tan and his sock puppets uh submitting numerous real useful patches before and preparing building up the trust of the community in order to to build up the trust of the community in order to be able to finally deploy the

attack. So, is this really scalable? You know, it's a very technically advanced attack. It took uh 3 years. Well, here's a third hack. Anyone remember this? Um hacker at the Chaos Communication Congress last year announced that uh she had uh hacked into a neo-Nazi dating website. And the way she did it was she created a load of fake profiles using an LLM. LLMs drastically reduce the cost

of sock puppeting. Many cheap sock puppets could subvert even a well-maintained project. XZ have to be poorly maintained if you get enough uh LLMs claiming to be users, claiming to be developers, seeming to support uh real uh patches. It could happen to anyone. And there are still under maintained critical projects out there. So, this is fine. I have no solution to this. I'm scared. I hope you're

scared as well having uh listened to this. Thank you very much uh Daphne.