Strengthening CNCF Projects: Impact of Security... Eddie K, Bradley A, Justin C, Shuting Z & Orlin V

In this talk, Eddie Knight introduces a panel discussing security self-assessments within the Cloud Native Computing Foundation (CNCF) projects. The panelists, including community managers and maintainers from various projects, share their experiences with security assessments and the crucial role these play in improving security practices. They emphasize the importance of documenting security processes, identifying vulnerabilities, and the iterative nature of security assessments. The discussion highlights the necessity for projects to develop a security champion role and encourages open collaboration between security teams and project maintainers. The session aims to empower attendees to implement these practices within their own engineering organizations.

The way we're going to do this I'm going to yell at y'all to sit down on those chairs, then I'm going to talk for a second and introduce what the heck we're doing here. And then we're going to hop in. Uh for most of this talk, you guys should not be hearing from me. You should be hearing from our panelists. Uh I'm going to let them introduce

themselves. My name is Eddie Knight. I work at Sonatype. I am part of the technical advisory group for security and compliance. We got renamed last year. Um the technical advisory groups within CNCF are focused on supporting projects and enabling uh different improvements. There's uh five different uh groups within the CNCF. Um I'm casting my screen up here. Uh can you tell me what I'm doing wrong to

make sure that my my uh screen gets up there? All right. Um so, within the TAG Security, we do a lot of different things to enable security and compliance within uh the CNCF projects. We coordinate with uh projects directly, also with enablement platforms, tools. OpenSSF is a sister group to CNCF that provides a lot of guidance that we like to implement. Um but one of the biggest

things that we spend a lot of time doing is self-assessments and joint assessments. This particular talk we're asked to focus on talking about self-assessments because this is the thing that is most replicable for anybody. The types of things that we support projects doing, you could do internally within your your engineering organization if you wanted to. A lot of this is replicable. Um but we're going to focus

on what the projects have learned and what kinds of advice we give to the projects. So, could we uh hop in here? Maybe we go from left to right. Bradley, if you can introduce yourself, then Justin, Shu-Ting, and uh William. Sure. Yeah, I'm Bradley Anderson. I'm a community manager for KGB. It's a global service load balancer for Kubernetes. It works over DNS. I'm Justin Cappos. I'm a

professor at NYU, and I um created the process by which we do assessments and wrote the book for it. You're also a maintainer. Yeah, I'm also am a creator maintainer of um the tough in toto uh S-BOM get tough obtain and other Linux Foundation projects. Cool. Yeah, I'm uh Shippings out. I'm co-creator of Caverno and currently a Caverno maintainer as well. I've been involved in the uh

joint assessment uh process with tech security. So, that's why I'm here today. Uh hi, my name is Orlin. I'm representing the project Keteris as community manager, but I'm also newly joined to the new on earth foundation as a devrel person. Yeah. Very cool. So, we're going to hop right into it. Um I'm going to ask this question. And then as we start trailing to the end of

this question, if you guys are thinking about something, uh you can raise your hand. Looks like we don't actually have mics queued up, but just kind of raise your hand if you want to hop in after this question with your question. Uh we can kind of alternate as much as you guys want. So, this is we're hoping to be a little bit more interactive. Also, it cuts

off any rambling that we might do. Uh fortunately, I'm not in a chair because you can tell I'm I'm very prone to that. So, um maybe uh Justin, if you could start and then we can tack on uh as many answers as we have to this. What the heck is a security self-assessment? Yeah, so a security assessment is basically a structured document that a project writes up

to try uh reason about and convey to other people about what's the security of the project. So, it focus on it focuses on things like what are the bits and pieces of your system? Are they separated from each other? Do you have something where if you have a failure in one area, that's going to spill over? Um what's the impact of different types of attacks and failures?

And uh the idea is then, you know, and also things about like what kind of security practices are you doing? Are you doing general best practices? So, it's meant to give someone who might want to adopt or at least understand the security of your project a way for them to know like what are they really getting into. Um and we've also found that that this assessment is

actually very helpful for projects to kind of have to write down and think about these things themselves and sometimes they spot issues. Um this is also a precursor for the thing that we're not really talking about here, which is the joint assessment, which I will just very briefly say is uh basically where the experts from uh our our group who work on and have seen a lot

of these help the project take their self-assessment and do like a more reasoned thinking through it and basically make that document longer and stronger and more um uniform. So, it's not just kind of, you know, you writing your thing up with your impression, but it's it's getting probing questions and uh suggestions from a broader group of experts. Does anybody want to add on that? It's a pretty

robust answer. Cool. Uh now I'm going to scan see if anybody's raising their hand for a tack-on question to that. No? I'm going to keep asking this, so you guys get your get your hands ready. Um okay, so maybe we could kind of move down to everybody but Justin with this question. Um does your project have a a security champion? Yeah, I think that's a very good

question. Uh we didn't really have a formal security champion before this assessment assessment process. Um you know, um before that, uh you know, while going through this process, it helped us recognize that there's a, you know, need for the security ownership across a project and currently I'm serving as a point of contact for all the security-related issues, but I would say it's really a teamwork uh eventually.

Uh for example, Jim, who's our co-creator and maintainer of Caverno, who has been helped a lot during this process addressing all the review comments. And our team actively work on fixing CVEs and security issues, reviewing security um advisories, etc. But, I think this process is really valuable to us because it aligns well with our reality. Because, you know, um as an open-source project, you always have to

balance between security and feature velocity. But, it doesn't require you to be perfect in the in the very beginning, right? Um it give us the framework to identify gaps and then uh pri- prioritize findings to uh fix them and approach them gradually. So, yes, we do have the security champion right now. >> you. But, eventually, it's a teamwork. >> All right. I like that. I like that.

No, we we we um we're very immature in our security posture, so this process was, to echo your point, super valuable for us. It actually showed us what it alerted us to things that we didn't even know to ask. Things that people are looking for, right? So, um through this process, we went from something like a 34-line document that said, "Yeah, we do S-bombs. We sign our

commits." Uh to something like a 230-line document that says, "These are the actors. These are the actions. This is what everything does." The kind of things that make it easier for people to come and look and want to use the project and know what our security posture is, right? So, um uh before a lot of those things were pieced throughout maybe our documentation, now it's more coherent.

In our case, it was interesting because the the self-assessment helped a lot to figure out that we are not exactly compliant with CNCF wants, right? Um K3s is a quite well established distribution. And I discovered we are uh we didn't have a champion. We still don't have a like a like one person for that, but I figured out we're doing that wrong. You know, in a way

that for example, uh SUSE I'm going to bash a little bit the community now, but um uh SUSE is the main one of the main main contributors. The old security um issues were going to uh SUSE internal mail, which is like, what? No. So, that security assessment helps us to identify that and to fix it on time and not to be like not good. So, uh a

security champion is not not always a formal role with a title or or anything like that. So, so so just a tack-on question from me before I'm going to start scanning for hands raised. Um how how have you seen projects evolve from like a hard no on that. Do you have a security champion? No, what do you mean? To like, oh yeah, it's Bob. Like, how have

you seen that kind of emerge? Yeah, I mean, it it's really we tend to see projects as they go through this process. Like, some projects the first 5 minutes of starting this, they're like, I know what I'm doing. There's a little bit of this if you don't know anything, then you often think you're an expert, um but once you know a little bit, then you realize you're

not. Um and and I think that some of that starts to happen and projects start to say, wait a minute, like we do need to have a reasonable way for people to report security bugs or we should actually be, you know, doing basic software supply chain security things to make our software actually come out, you know, in a way that we can believe the artifacts are secure

and so on. So, um a part of what what we're trying to do is not to be like, you're doing it wrong. We're trying to get people to kind of come to the realization that um by asking questions like, hey, how do you handle this or you know, how would this happen for them to come to the realization that maybe they need to do more uh, work

and and need help and then we can maybe guide them a little bit. Yeah, very good. Okay, uh, I'd like to ask if in your opinion it would be better to within the company appoint uh, security champion from within the team that's working on on on the particular service or product or to maybe have a separate dedicated team of security champions or whatever that's external to to

the service. So, sh- sh- should that person work on the project as well or or not? Yeah, that's a good question. I'll I'll weigh in first. I think in general having people from the team is really important because they know where all the rough edges are, where all the bodies are buried, all of the like to do we'll fix this later things and those are often some

of the the problems, um, when you get external people to do that, they tend to do that for lots of different things and they are more tend to do have more like red teaming experience and stuff like that and um, they're just going to have to go immediately to the to the project team cuz they're not going to have enough of sort of background in general. So,

definitely people on the project getting more security awareness is I think the long-term best solution. Yeah, I think uh, the informal definition, we haven't really codified a definition on this, but informally it's who on the project cares the most about security. So, when when we are red teaming, who do we go and talk to, Right? Uh so the next question is why did you do this? Why

did your project perform a security self-assessment? Uh can I take this one first? Um the obvious one was we were aiming for incubation and that was like checkbox thing. But but actually that whole process helped us out to identify the actors and all the roles and actually for me as a person, I don't consider myself a security person in any way or form. That was super revelating

kind of open eye-opening what we can do wrong in so many levels. So yeah, from checkbox it actually came realization that we have to put a lot of effort because a lot of people are relying on our project in like community. I'm not talking about product or companies right now. I'm talking about people who run their stuff. So yeah, from checkbox actually transferred to something that we

had to do long time ago. Yeah, the same reason with Allen. We trying to get Caverno graduated and that's one of the requirement of the graduation process that you need to get involved in a review with the corresponding tax tax team. And for Caverno since it runs as an admission controller, it does all the image verification stuff, security compliance check and etc. So it became a natural

process to you know, do this together with tax security. Uh but you know, my I don't know. It's it's from my personal I don't think it's really a checkbox one-time checkbox thing because you're you're in the incubation sorry, in the sandbox and then you're through the incubation then graduation. So there'll be a ongoing process throughout the whole event process of the you know, the project graduation stuff.

I'm sorry. Yeah, so for us it was the same thing, right? We have We've participated twice in the security slam. We're finalists. Perfect CLO monitor score. We're not a security tool, right? We're just handling traffic at the L3 L4 layer. We're not looking at anything. We didn't think we needed something like that, right? So, it's a good reminder that everybody actually needs to think about security and

it really um we really doing this for incubation really opened up our eyes that it's it is a continuous thing that we need to keep um keep an eye on. Yeah. Uh and for the slam we we double-check. If you haven't renewed your self-assessment last year, we ask you to redo it again. Uh so, it is a recurring thing. So, first one for compliance, great. Let's do

it for fun after that, right? Next question. what did you learn from the I'll go first this time. I'm staring into the sun, so if anybody I can't see anything. This is Yeah, we we learned that uh we we didn't really have any kind of security posture. We thought we did. We thought we didn't need something. Uh we we learned that actually and this is something we

we learned as well in one of the talks this week that it took a little while to get to the point to tell people what we're doing. We learned that we weren't really effectively telling people what we're doing, what the what the value is there. So, it's really helped us to um really helped us to learn to be more complete with those things. Put everything in one

place and say as someone said earlier, what the what the actors are. These are the components. These are what the components do. This is the I think Justin said the blast radius earlier, right? So, uh for us we really learned what we didn't know uh to even think about, right? So, we didn't know we didn't know. Now we know those things and we've it's made um it's

made our position stronger. Do this. Well, um I there's a lot a lot of learnings from this process. Um, you know, governor, it plays such a critical role in your cluster admission uh review process. So, we were focusing on uh the security side since the very beginning. And I think it was a great way for us to build on uh the work that we were already working

on. For example, um, you know, we've been focused on this principle of less privilege um, you know, for such a long time. And in one of our releases last year, we removed the vault car view permission for all the secrets just to be secure as the vault. So, we were not, you know, starting from scratch. But, um, throughout this process, it's been beneficial for us because it

found security gaps that we couldn't, you know, figure out um while shipping the features, right? Um, for example, on the technical side, there's this feature called global config text entry, uh, which is to fetch data externally or internally through within the cluster and then store them for faster lookup. But, uh, we realized that uh we realized the restriction on restrictions and the bounds limits, and we're working

on the fix uh for that right now. And one other example is, you know, um, on the documentation side, um, you know, we we could be much more clear about the security risk of the external data lookups because we are able to reach to the external system. It's important that, um, you know, we do have the support of HTTP uh with the certificate, but it could be

uh much more helpful, I think, uh to have everything well documented in the first place. And for us was more what Burley said. Uh, we thought we are well seated in the security stuff. Um, but we discovered that we think about that, but then we don't communicate that towards the community in the proper way. Like we we don't have a documentation um to say, "Hey, if you

do this, that can can go terribly wrong." and all that. So, I think for us it's just like to to think about what we have to do to make the our customers I'm sorry I'm our users to um to know what are the risks and which are the areas that they have to take take care. Um yeah, so it was kind of self-learning also and teaching processes

of how to present that information out. Very good. Anybody else want to want to hop in with a question? It's It is actually really hard to see you guys right now. I have like glasses, so it's like glaring down in my face. All right. Um Okay. So, if we have engineers who have a project that they're responsible for, we like to call those maintainers, right? So, this

not just talking about open source maintainers, but what advice do you have for security-minded maintainers? And I think Justin, if we could start with you this time and then just kind of go down the row. I mean, really just do the self-assessment. Um that that's that's like the number one thing. Just, you force yourself to actually write down and think about like writing is is thinking. And

um force yourself to have to have a structured thought process and communicate about your what you think your security is. And then if you can, um and by the way, and if you do that, you'll submit a PR to the to the repo to have this added in, and then I will always give you a few things to fix in in your thing. In part because I

want to figure out, are you just like yoloing this over a wall or are you actually going to be responsive if we try to do a joint assessment? And I will also try to give you an idea of this is quite close to like a good version of this document would be for a joint assessment or this will take a lot of work to become that. uh

so but you going and doing that process is is helpful because you're starting to communicate what your thoughts are about it. And for people that are security minded, sometimes hearing someone talk about why their system they think it's secure is really helpful um to know whether they know what they're doing one way or the other. Um and you know, yeah, it is also um internally helpful for

the project. Yeah. And uh if we replicated that to a non-CNCF scenario, this is where that third team would come in, right? Where it's like we have our security team. Uh they would step in and and provide the same kind of things that Justin was just describing. Um Chu-Ting, you want to pick up on advice for for security minded maintainers? Yeah, of course. Um we started with

the self-assessment, of course. Um so the first piece of advice is that don't wait until you think you're ready. Just submit it and Justin and other reviewers will chime in and tell you, okay, you have to fix this and that, and you'll go through the process. Um and there's also an important thing to keep in mind is that you have to be honest with your gaps, right?

This this process is designed to help you identify the the reviewers are here to help, not to judge. So uh you know, just just trying to be honest and with with that mindset, you know, to to keep curiosity and willingness willingness to learn. So that I think that's that's how you can best use of this uh process. I just want to say one really quick thing on

that that's really important. Like every single project has things that if something happens like in this part or in these series of things, you're you're dead. There is no security system that is like unconditionally secure against every single possible attack. You know, so that is okay. That is expected. That is If you say that is not the case, then we are very worried that you don't know

what you're talking about. So, you're just pointing out like, "Hey, if you take this reliable car and try to, you know, make it be a submarine, it is not going to function very well as a submarine, right?" That's expected. Yeah, so I would say, um, as a maintainer of a project, you have to think about what you're trying to do. You might want people to use your

stuff for whatever reason, and this, um, you know, you can help them to use your stuff better, right? When you're in the thick of something, it's often the case that you understand things so deeply that you have forgotten the hard the easy parts. And so, someone coming to the project, they might need some help with those easy parts. Putting all this stuff together in one coherent document

is really helpful toward that end. I have two type of advices. One is for maintainers who are going to do that, don't postpone that too much. And it's, uh, um, it's not a homework that you can do during the weekend. It's, it's a kind of lengthy thing. The other thing is go through the other self-security assessments by the other projects and figure out what are your weak

points in your assessment, what you maybe missing. For me, it was super useful to go through all them like to I'm not a security person. I don't know what to look for in deep, so, go through that. It's interesting reading. And talking about reading, how many of you are I have read six self-security assessment for projects that you're using? Exactly. Oh, we we do have a non-zero

number of hands. It's like Me as well. I I'm in the same bucket. I And I didn't that didn't occur to me well, I did that. Like, we should be reading that and we should like promoting that when when people are trying to use our our projects, like, hey, check this out. Maybe that's going to be super useful for you. And maybe it's not your project. Maybe,

I don't know. So, do your homework and read the self-security assessment. Hi. Uh thanks for this great talk. Um I'm wondering if you have any advice how to increase the general um security mindedness in organizations. Uh from my experience, uh people are not explicitly against security uh things, but they have a very different opinion on what is sufficient to feel secure. And especially, they might even feel

validated because nothing has happened before. So, uh before you said there's often people who are interested in securities and many who are not. So, maybe your general advice uh security in organizations. Nice. Um this is hard. Uh it one thing that Okay, so, to be honest, um when the CNCF fairly recently merged the old tech security into tech security and compliance, I initially was like, this is

really weird. But, as I've worked more, I've realized that companies care a lot about compliance. And a lot of the reasons why they're forced to do compliance is because that is a misguided attempt to try to get at security. And that's like sort of the only way they can do it, and they face hundreds of millions of dollars in fines in the EU mostly if they don't

obey compliance. Um so, that's that's sort of part of part of it is you can often reframe security things as a compliance thing. Um and I hope with the CRA, there will also be more of this sort of um you know, now that we have all these reporting requirements and we have this and that and we have all these other things going on, then you know, we

need to um it would be helpful if we had to do fewer vulnerability reports out and helpful if we had to do less of this kind of work. Um but you know, other than that, the things the other things I found effective is to you know, red team your own stuff and then show your boss things you know, they thought um you people shouldn't be able to

do. If I could really quickly, I would say also um just back to the idea of having a security champion. As you said, it's super hard. I think people are well-intentioned and they know that they are supposed to do certain things. They just don't have the time to do it or they don't have the rope to do it. And so if you assign somebody and say, "Hey,

once in a while maybe you report on this, maybe it gets a little more backing." And and actually I'll say one more thing. Um from being an expert witness in a few cases, um write emails about uh you send send an email. Uh you don't have to be obvious about it, but having those emails will will be useful in the future. I don't want to stay on

that one too long. That's a good point though. Um so this is the last question I have. Uh so following this one, we've got like less than 5 minutes. So if you guys have any other questions, queue them up right after this. Um this is mostly for Justin, uh but anybody else can hop on afterward. How can a project build on a self-assessment after completing it? Yeah,

um one thing is the obvious is to do a joint assessment, which is basically having experts really go through your self-assessment with you and do things. And this, depending on where how complete and how well done your self assessment is will take different amounts of time and effort from your side. Um uh the other thing is you will find things during that will make you uneasy. And

so open issues about them, do things like that, and you'll start to work on them. Um and and you know, you can sort of improve your security posture. Also, if there's things like that you think are fundamentally like a big problem, but you just don't think can be solved or you don't know how to solve them, then reach out and talk to to us or to others

in the community that are working similar issues. I I do think that security is one area where projects are not really that much trying to compete. Like for the most part, projects are trying to be secure. There are a few exceptions to this I will not name, but I think for the most part, um everybody would be happier with all project security being being better. So, um

yeah, those are those are my thoughts. All right. Um I'm going to leave this up while we do the last couple minutes of Q&A. I did not think ahead to put a link to where you could get the materials for your own security assessment. So, instead I'm forcing you to come ask us for a link to that. Uh that was not intentional, uh but we really do

like company. At Eddie Knight. At yeah. Yeah, at Eddie Knight and all your questions about where to go find that. There you go. Thanks. Thanks. Thanks. Um but yeah, okay. So, last call for questions. Um is there anything you think that that we could help with? Any questions you have that would make it easier for you guys to do this internally or on your own projects? I'm

about to just start handing it to some maintainers that I see and just say, "Do you guys have advice?" Um I have a question about the uh um partnered assessment. Forgive me, I've forgotten the the jargon term. Joint assessment. >> Joint. The the self-assessment joint assessment dichotomy there. Um, have you found that the joint assessments are more or less popular as a result of the self-assessment being

made mandatory? You know, do the joint assessments go better at etc. Please tell us a little about that area. Yeah, um a lot of times when a project completes a self-assessment, they feel motivated to then do a joint assessment even if they didn't have to. Um, I will also say uh a few years ago we did not have a particularly high number of self-assessments across important projects

in the landscape. So, I actually as part of a more than a hundred student um graduate security class I was teaching had them go in teams and work with projects to do self-assessments that they were doing as outsiders of a bunch of these projects. And those are of semi-variable quality um because, you know, it's it's hard to read through a bunch of documentation for a project that

you've maybe never used and maybe have would have a hard time even setting up and running in your environment. And that to then try to, you know, list out all the security properties and things like this. But, the students made a really good effort with these. And so, those are all checked in as like starter places for a um a lot of projects that didn't do it

themselves. For the projects that did do it themselves though, we have had a quite a high number of them go and just want to do a joint assessment right away. Um, and yeah, I I think they've, you know, seen the value of of this and even the joint assessment they're like, "Wow, now you know, I thought my self-assessment was good, but now I'm really learning even more

by getting people to give us feedback." Cool. We are at time. I'm going to throw up uh everybody's picture again so you remember everybody's name. Thank you guys for doing this. My name is Eddie Knight. If you guys want to come and ask any questions of us afterward, we've got at least 15 minutes before the next session. So so feel free to come up and bug us.

Thanks guys.