Panel Discussion: Strengthening the Cybersecurity Ecosystem

This panel discussion focuses on the security of open source technologies and the importance of strengthening the cybersecurity ecosystem. The speakers, including developers with extensive experience in open source contributions, discuss the significance of transparency and collaboration in enhancing security. They refer to historical examples, such as the introduction of the seatbelt by Volvo and the success of solutions like Let's Encrypt, to illustrate how open source initiatives can lead to improved safety and security. The conversation further delves into the challenges of maintaining and validating open source software, particularly concerning software bill of materials (SBOM) and supply chain security. They explore the balance between rapid patching of vulnerabilities and the potential risks associated with public disclosures. The talk emphasizes the need for both community involvement and professional code audits to ensure the integrity of open source projects.

[Music] ladies and Gentlemen please welcome to panel discussion about the topic strengthening the cyber security ecosystem open source supply chain and active deterrence all right we good you guys hear me is this thing working oh wow okay I just yell um all right well uh thank you all for for hanging out during lunch here uh we have some awesome guests so we've got uh I'm Rob Carson

I'm going to be uh moderating uh and just chiming in randomly but my esteem colleagues here um we'll start with next to me is Nelson so Nelson you want to introduce yourself a little bit tell us about your background thanks I I'm a developer inang Solutions I work with distributed full tolerant and scatterable systems uh contribute a lot to open source my main line of work is

a messaging server that is open source and also some load testing tools and a bunch of other minor libraries and we do work with um like with a messaging server we work with privacy issues for example with security as well and open source awesome and Thomas my favorite Frenchman F yes Thomas uh I'm French like you can hear with my accent uh right now I'm developer Advocate

at cig the C SEC cber security company I'm mainly doing advocacy about open source projects especially Falco cncf project which has been recently graduated um before that I was Sr for almost 10 years I did five years of on call a week a month for five years um I'm also gong developer you can see that um that's pretty awesome cool well uh and my background I do

the Arts and Crafts of cyber security so I don't do anything cool like these guys I just hold people's hands while they change their passwords for the first time and tell them it's going to be okay U but my background is U Marine Corps infantry so anyhow let's get started with the first question here so and please you know you guys are in here either put in

your questions on the app or just raise your hand shout them out let's let's make this fun like it's not a huge group um so I think the first one is uh let's talk about uh security of uh of Open Source like using open source Technologies how do we how do we secure that that that software build materials let's talk about that a little bit I can

start with that um basically we start with a simple story maybe you don't know but in the 70s a car manufacturer Volvo they invented a new security system the seat belt and they choose from the beginning to open open the patent to allow anyone in the world to use it and protect their users this idea this decision saved thousands of lives over the years it worked only

because the patent was open and free to use this is why I think open source is one of the key components for security having something at least could be free or not but it could be cheap or not but at least open it's like knowledge know knowledge even if you share your knowledge you will not lose it you can share as much as as much as you

want everything you have everything you um everything you know globally it will increase the level of security for everyone this is the same hide that pushed some guy few years ago for his thesis for to complete his PhD to create let's encrypt this is free anyone can use it it's open so anyone can contribute and it it increased a lot the security on internet see the same

idea open source okay but uh so you know with that great you know everyone can contribute right and can be maintained uh there's challenges though with that right like how do you know when you're using open source it's the right tool to use like what would you say about that like how would you how would you validate it's it's it's appropriate I need to say that I

love the story of the Volvo one I'm going to steal it thank you uh I had one more story as well of uh something that was shared to made the open source uh we all use uh one messaging up or another call it you know let's give some names WhatsApp signal slack Facebook Messenger telegram there's so many options and at some point the topic of became like

mainstream and the signal people uh they came up with this uh protocol it's my special they work with messaging and often I'm being asked like can my server be end to end encrypted as pros and cons but the to the point is uh that they also open the protocol they published the the patent everybody could use it and this uh protocol that is called signal protocol it's

divided in two subprotocols it's the same one that WhatsApp uses for example and that Facebook messenger only a month or two months ago finally implemented after so many years this Mark Zuckerberg isn't uh checking out your messages anymore hopefully but you need to update your uh version otherwise he's still like anyway uh so yeah that's another example of when open source helps I think that uh a

good thing about the open source is that more than one person can have a look at it one thing that is always sa in cryptography for example is don't invent your own thing like uh just don't like there are lots of smart people that have studied this problem in a distributed way uh in science is usually say that uh whatever your science paper is needs to be

verifyable and repeatable that's the scientific method from like 400 years ago and that's the thing about open source that there are more eyes looking at it mhm uh and you can think of for example like critical components like a kernel are plenty of eyes looking at it unlike other Kels that only a few selected people are and let's not even talk about bad intentions but just simply

mistakes the chances that thousands of people in thousands of different places without a relationship to each other would make a mistake is slower but it's never zero it's just increasing your chances but where would you go though to when you're looking at open source projects to to find the ones that where there are thousands of people looking at it versus the one where it's just you know

me in my basement going hey I learn how to use Ruby I can do this today yeah you all know this XKCD meme that you know there is like a massive pyramid like the whole economy of the world is depending on this little column here that is a open source project maintained by a random guy in his garage on his free time and like that was a

case for open SSL maybe you remember that two or three years ago something like that they discovered a big cve in op SSL op SSL basically is a the key component for security for almost all protocols for communication SSH primary and they discovered the Civ in that and big companies really with mons of money asked the developer to fix it quickly because it was critical for everyone

MH that guy was maintaining the project by by his own for years without any money without any concern by anyone except that day this is exactly the real example you mentioned with casity yeah so so we had like the good cases like the kernel like the Linux kernel there is so many Geniuses working on that there is so much money invested this is the perfect story of

Open Source gone right then you have like op SSL which is just as critical and this just one poor guy and then the whole world is ranting on him and then suddenly Facebook made their own Fork of open SSL and Microsoft make another and I don't know who else make another because this guy is irresponsible yeah he's he's a little busy yeah been fixing his hair something

pay him yeah pay him that' be uh that's a it's a fabulous idea right um well how do you so you know you let's talk about like software build materials and that security supply chain like where where would you guys give advice or what are the what are the pros and cons of you know open source supply chain and what some of the cons like what are

the what are the challenges you think or are there any cons is it all open source that's the way to go Freedom um first we have to Define what is as bomb software bill of material every everyone among the an know what as or not basically the idea behind is just to list all the dependencies of your project so and the dependencies of the dependencies and the

dependencies and so on you know where that came from drugs from the FDA yeah build of materials like it's like you get your your initial precursors all your drugs they put together that's actually where a lot of that origin I was told it came from construction you know the BS and I it came for for meat to be sure well we've all got different rumors of the

point is is like it came from highly regulated Industries where you needed to know where did this chemical come from where did everything come from or conr by the molecule level everything is documented exactly we want talk about compliance that's a whole another level but it works only if every layers in the CH play the game correctly mhm uh so it's not really easy to do so

uh this is why I think as bomb right now is nice but not really useful okay because we love coners I think most people love love coners but for love what the containers Dockers Anders containers sorry French accent uh love containers I was like we love whatever it is man um but to be honest most of them are just black boxes mhm we don't know what we

are running sbom is one of the solutions to have at least a better idea of what is running inside the CER but you have just an idea of the supposed to be in inside applications and packages but when you build your images you don't know if the compilation if the build process has been correctly made and if nothing has been introduced illegally imately in that yeah and

this is another tricky question this is why we also have one Cas exactly signatures especally we have this case with the lib XZ uh you tell the story much better than me let's let's see let's he that story that's a we had a great lunch discussion been here for the panel you all have heard of the exit uh Library vulnerability from not long ago a couple of

months since familiar Rings B uh just a comment on the containers that reminds me of another joke programmer says it works in my computer and the manager says okay let's ship your computer that's containers in in a nutshell the exit vulnerability that was brilliant I was very impressed and I'm also concerned that like okay we detected this one what about the ones we didn't because that was

very smart so the story exit is a library utilities for compression algorithms so C and SLE and the usual ones that is very popular in the dros Linux and it's maintained by one guy and which project doesn't ever do um compression it's again we have the problem I said before op SSL is super critical but there is one poor guy so in a compression the standard compression

Library again there is one guy and he's tired and he has his main job and his family and one day A couple of years ago a guy appears on the internet making a contribution here and there to different repos and then he arrives to this one makes some contrib tion very good code with good descriptions everything super friendly he starts joining the community and after a couple

of years uh the official maintainer makes him a maintainer because by now the guy really is the maintainer he's like contributing all the stuff so he just gives him access rights on on on GitHub and the little community of people looking at Exit because nobody was coding is very happy with this decision one day ER I been uh in January more or less there is a postgress

developer I don't know who of you was on this talk this morning like 9 to5 and then it uh that the guy was talking that the stereotypical programmer is that one whose Hobby and job are very related and there is a little boundary between what I like to do for fun and what I get paid for which introduces which my introduce on healthy habits anyway there is

a pogress developer that is having his own Farm in his own own home because he likes it so this is the first requirement uh he's not being paid for this uh he is investigating some fun stuff that he did in postgress and something is slow he's running on the latest x86 chips running all the software of materials everything is running on the latest version latest kernel latest

systemd latest dependency everything is just compiled from Master just testing on the Edge and something is suspiciously slower that is not supposed to be he enters uh his uh he SSH to his machines and he realizes that is his SSH connection but what is slowing down a lot of stuff he's like what does this has to do with pogress his job again is pogress and again he's

not doing this at working hours with his working Hardware but his job is pogress he gets curious and he says okay I'm going to investigate the open SSH because something is slow here super complicated he realizes that which the craziest the algorithm open SSH the where BG uses open SSH open SSH requests systemd for compression uh functions systemd provides Dynamic function pointers this is super lowlevel stuff

and syst D provides a pointer to this uh compression Library where this new guy is now the maintainer and now this compression Library executes some code now that guy he made some pull requests like more than a year ago where in test the test were spawning another server and verifying the connection and the compression and so on he used a compilation a linking flag that systemd needs

to like decide when you need to compile a low-level project for so many architectures to like simplify the the whole linking thing he uses the same trick that systemd does to dynamically inject the test into the binary as is shipped in the in the code so the binary is not exactly the same source code but veryify the binary is made hard by a very useful and popular

trick that linkers give you that you cannot get rid of this trick because then you need to get rid systemd and in this way what was being slow is that he was not compressing a movie he was compressing the key so this is supposed to be instant pretty much a n second but it was taking around half a second so this is like noticeably slower and In

This Moment he was just like stealing the keys and sending it to everyone this guy started making a mistake when he so that systemd make a pull request to change the way the linking works not because it's vulnerable but because system they say that the current linking mechanism is a bit hacky we can find find a cleaner way to do it they didn't even know it was

vulnerable to attacks it was just hacky and when the guy from XC saw that systemd is going to break how his algorithm Works he started pushing a lot of people to to rush and that's when he started calling attention as well it's so tricky yeah and it depends on so many different projects this depend on it R runs on x86 it depends on how systemd does the

weirdest stuff how the Linker helps system do that like so then what would you uh there is no single person that works in all of those projects at the same time absolutely and and that's where you know you have a nation state potentially trying to execute something like that it's difficult right so what would you say for the you know you're on the you're on the uh

the IT team you're trying to use some open source you're on the dev team you're trying to uh leverage these things but you have people that are afraid of it for there are risks what what what measures would you put in place that would prevent Maybe some of this you can't stop everything right that's impossible I have a better chance of growing a full head of hair

uh it is what it is like but what would you say like what would you do to stop it or mitigate those risks what what what's one thing you could have done of course we got the chance to have someone so passionate about the topic that he spent his own time to discover this stuff but this this was possible only because everything from A to Z was

totally opened mhm so code base was opened and but also the request from the mainers from the contributors from the users the discussions were also all discussions were public everything was public if totally Iden behind the payroll of Health I think it would be totally um undiscovered yeah missed by by the community by the users and this this wrong stuff will be integrated for sure mhm um

especially when you are a big company and you have to make money it's legit you can't wait for years or for months or for weeks on a so small issue or so small changes you can say okay just two lines in my code base two lines is a test okay let's approve that let's move on because I need to integrate that feature that will make me more

money after um this is why open source is important and this is why I think open source won for a long long time Linux is the most used system for servers for main frames for mobile phones it's Android um for security we have open SSH and so on um almost all Protocols are open for that it's signal but most even most basic ones like HTML um but

this is the first thing to have it's the ability to review and it's thanks to open source then we need all tools to prevent it's the other part yeah we talk about as bone just to know the tree of dependencies but we also need to be sure we use this dependency that is noticed in the tree MH so we have now things like cosine to have a

signature for every package we provide every artifact um this is another level okay we also need to event bad stuff happening in at the CI level so we also need to control what the worker no do U for example GitHub I know for sure not exactly for that but they are using the project I'm I'm maining Falco underlying when you run some actions in GitHub actions GitHub

has thousands and thousand of Nod and they're running Falco to detect suspicious behaviors to protect G not to protect the users but at the end everyone is protected a little bit more this is level um I lost my stream of feelings of FS um think when you a provider of a service where like you allow people to execute code in your service that it's is a Pandora

box like it's like if you allow I don't know storage is Gmail or like or cloud and you put your pictures whatever it's just content but the moment you allow any user to execute in server people can execute the weirdest stuff like from and they will hug attacks to crypto mining I I remember you know in open source uh it's very common that you get like free

credits from distinct um CI environments because it's open source and they promote it and so on and some years ago when the whole crypto boom started I remember so many of our CI that we were being contacted by Travis Circle CI GitHub action didn't exist back then telling that we need to close and reverifying a lot of CI is actually crypto Mining and like crypto miners are

running crypto mining in our offering yeah you know I always tell people look at your one of your best uh security indicators is your spend on your AWS environment that'll give you an indicator of oh you've turned your half your operation to a Bitcoin money operation yeah some expenses start going up you haven't done any you're like hm why am I yeah why am my costs going

up funny things happened to Amazon last week maybe you notice that a company they check their bill after and they noticed really huge hammer and for Amazon A3 the object storage system and they discovered 99% of these fees were related to 404 erors so basically on Amazon when you try to uh pull an object on S3 and this object doesn't exist you are charged Anyway by Amazon

and they disc discovered if someone else than you try to call that object this is the fees are on you you have to pay you and because of some conflicts between the bucket names they discovered for strange reason a lot of systems all around the world were calling their bucket their F3 bucket for missing objects and they had to pay they had to pay thousands and thousands

of dollars for that they discovered that they complained Amazon refunded um um agreed to remove the bill for that and they change Behavior so no we can't anymore do that I think it was a nice way to annoy your neighbors MH uh I think some did that I'm not sure that but this is exactly um the you mention lets your users do stuff and you will see

what you basically the best C engineering tool is basically human so yeah like when you're back on Monday to work check your AWS like just let know your DeVos people to take the bills who it's a crazy thought right but it's like we got all this technology let's just look at the bill right any questions uh while we're going through this here well here we go what's

better to quickly patch and disclose supply chain for owner abilities to maintain transparency re quietly mitigate them to avoid alerting potential attackers oo and this is a problem in open source because the patch that fites the vulnerability is public the moment you public the patch there is uh like an attacker that says oh this patch oh it's fixing this let's attack it before this patch is distributed

and this is a disadvantage of Open Source that has always like made me sad basically yeah but the the other part is like even on the on the commercial side like it winds up being uh security through I mean just like the I look at it it's interesting because I I have this Challenge on the other side right and it's like oh should we should we tell

anybody about this and it's like well you think you're the only one that discovered it right yeah so pushing the patch out yeah yeah okay but there's a good chance someone already found that vulnerability as well because I mean the best uh so I I deal with you know hardcore pentesters people that trade no days on a regular basis stuff like that that's more uh Myspace and

like I can tell you uh they probably already know and it's like Eternal blue like it's like oh only the CIA would figure that one out like no that's not exactly how that works right so it's a challenge but I think it's better I would argue it's better to uh get it out there because the the best attackers potentially already know about it yeah like uh something

that I have seen on the L Kel sometimes like there is a disclosure that works only with a selected people they fix it nobody knows they publish a patch but the description doesn't say that it's necessarily that critical or something so a very smart attacker can like see what is it that is being fixed but but they tried not to call attention at least yeah it's like

apple patches sometimes it's like yeah it's security fixes yeah just first update it'll be fine but yeah the thing on Apple for example is uh well Apple's Kel is public but it's open source but the rest of the system is not so sometimes that that's an advantage when you don't know the code you don't know that a pach is being fixed but again that's security through how

do you call it security through obscurity through obscurity that's not really yeah yeah no security and and it's one of those like own it fix it move on as opposed to I mean I would argue that all day long because like it's good to well the question is and that's thing is like devel the patch it's one thing though you could argue is it worth pushing out

the if there's a vulnerability but you don't have a fix yet right yeah like that might be the time where you're you're not disclosing it because you're trying to you're trying to figure out what the mitigation effect is then push out the solution with it at least then you can go you give somebody it's not hey you're screwed we let you know in a week when we

have a fix okay you're screwed and here's your fix because otherwise you're just sitting there going you know that reminds me of uh that's a good question like you leave it open this uh what was the name this CPU vulnerabilities that was a big deal a few years ago the Spectre mhm that that was disclosed to Intel and intel was like mind blown and the disclosure that

was done the proper way it was disclosed in secret and you have like so much and so much time and Intel had no idea what to do with that expiry happened and then they said that there is a zero day ability in the CPU but they still didn't say what was it but they were very responsible that that was a really big deal that that was a

crisis the whole world got like 10% slower having to fix that it was really harder to fix because it's material it's physical yeah so you can't change a CPU like you change just some lines of code even if at the hand they replace some micro code of the CPU but it decrease the performances it created so much latencies people were a lot impacted a lot of because

of that I I want to mention something a little bit of an anecdote before being airine an airine developer I was working in C a lot and I you have the use of performance compile languages versus interpreted languages and a compil like I have a myth breaking uh line for you all languages are interpreted C is also interpreted the assembly is uh reinterpreted on the CPU the

CPU has a virtual machine executing the xx6 uh code the last time the CPU the x86 was not interpreted was I believe Pentium one or Pentium 2 I'm not sure which one if it was Pentium 2 or pume 3 they wanted to introduce optimizations to the CPU but they couldn't do it without breaking the the binary instructions and breaking like all the that is compiled in the

world so the CPU has a virtual machine inside like C is also interpreted of course it's a different layer of interpretation so there are fixes in the CPU that you can do because it's release a virtual machine that's a micro code that that you hear about hope everyone understood all that it was awesome uh but so some Next Level just to complete the answer to this question

as an open source Mainer uh transparency is really really important for us you lose all you use the purpose of Open Source without it right yeah exactly totally if you try to add something to you you users uh to contributors maintenance or health it's really easy to lose the confidence so it's better to patch quickly and communicate because even if you hide the stuff you are sure

even after years a lot of system still have the CV because because people are not patching their for companies I'm like I don't care how many vulnerabilities I care how old they are yeah like because that's what it really comes down to it's like did you fix it in time because like you think about the rapid oh this vulner blade has been exposed for a week There's

Been stuff that's out there for years that people are still haven't fixed so L for J is is an example yeah another big deal you have all have heard probably L for J Java loger zero day vulnerability W went mad ER there was a study from Maven Central which is where you get like all the dependencies that last year in 2023 25% of the don't loads of

log ja log forj were still of all vulnerable versions so there is still a lot of code that is just like recompiling or whatever running on CI is automated that has not been upgraded is still fetching vulnerable version 25% of all the downloads during 2023 yeah and it's already a more than two years old vulnerability good parts we're all going business uh so I guess one of

the questions we got here is uh what are your favorite tools for minoring vulnerabilities and dependencies on projects using different languages so we'll start with Thomas what's give us your top two trivy is pretty um famous for that uh which one trivy from Aqua Aqua security um it's basically this it scans images um I'm not doing production anymore so for three years now uh I don't have

any favorite tools as long you have tools you can use and you understand the report it's a good it's a good it's probably the best answer is the one you actually going to use yeah because we have plenty of tools but if you just run them without any actions be after it's totally useless security is not an action it's a process uh yeah it's funny I've seen

tools that they're great but they don't integrated into the pipeline or to the they don't they don't create the the request to fix stuff and and you need the human resources to take care of that after absolutely what about yourself um most of my programming is an airline with which uh I'm I'm going to brag about the fact that we have one of the lowest rates of

uh CVS in the top 100 programming languages in the world okay like we are like top one in the bottom so I'm very proud of that we have very little so little to monitor so usually like you know as there is so little you just get the official CVS and they they are usually fixed okay do you find out that uh sometimes they wind up being overweighted

where you get you know like a lot of times like on my side I see like two mediums is actually a high if I combine them together like do you guys run across that where you're like oh this is actually this should be weighted lower weighted higher because sometimes the CV scores like or CVSs they wind up showing it's a high availability issue but availability is not

the issue right another interesting tool is not really to get the V abilities is to be sure you are up to date MH for example if you are using GitHub for free you can use dependabot which is a bot in charge to check all your dependencies and propose to you through PRS upgrades for the dependencies it's pretty convenient it avoids you to think about that annoying step

to upgrade everything awesome another foot for thought that I was thinking uh statistics I like checking them um popularity is not correlated to uh maintainability and one example we have already mentioned a few open SSL or xed they are like absolutely popular every single Linux Dro has them more or less every single one but they are heavily on maintain so that's something to pay attention to when

when checking tools you're using got a new question here this is a good one what's more effective op for for security and open source projects regular code audits by security firms or the community all of it see that's that would be my answer is like I want both I don't trust anyone um basically I will explain um Falco the project I'm maintaining is a graduated project as

a cncf to get the graduation even for to get the level of incubation we have to do these two things first we have to do regular community driver report so basically we have the mners we have some Searchers um doing analysis by the home to write papers after and once a year we have to buy a professional totally independent team to do a more deep a deeper

or more complete audit of the code base every once a year it's mandatory to still have the graduation level so the answer is pretty easy both absolutely well I mean like you remember the security firms those those guys this is what they do on a regular basis so they're Geared for it and they're getting paid to find something like I've got buddies that get paid a lot

of money to find Odes every single year like that's their job right and like there's also Google C team kudos to them like I mean that's the thing like this is this all they do and then like I mean what do you think about uh what's it called uh like bug Bounty stuff like that guys everyone like you ever use that the bug bani programs I haven't

had the luck yeah but I I encourage them I think it's smart uh bug Mones are more are more for companies with project not not for a project because you have to basically to do a bment you need data in the background you need users and so on just the project itself you have to install it you have to do everything by your own uh it's not

the spirit of a bug Bunty um so and you also need money for the Bon you do well then people get upset when you think like oh I'm giving you 100 bucks like that's $1,000 bug like pays a lot they get very upset some days right uh cool it's worse when you get at one of these firms just suing you for attacking their service like I was

trying to help what the I disclose it in anyway people get upset no matter what uh any other questions here if not I think we'll wrap up here we got one more here okay not really quick nice shoes rub right that's right you have skulls on them for the record I'm bringing compliance making compliance sexy every day I can any other questions these are some sweet shoes

I preferred your shirt yesterday hey man these things though man like they got skulls gun you got skulls on your shoes no no exactly that's right badass any other questions all right thank you all for your time [Applause]