A Bug’s-Eye View: Kubernetes SIG Security Explains It... Ian C, Tabitha S, Rory M, Iain S & Mahé T

This talk presents insights from Kubernetes SIG Security on their recent activities and the collaborative process behind enhancing Kubernetes security. The speaker describes ongoing initiatives such as the update to the OWASP Kubernetes Top 10, aimed at providing current security best practices and community-driven insights on securing Kubernetes applications. They emphasize the importance of external audits, highlighting past audits and plans for future assessments aimed at identifying vulnerabilities in Kubernetes components. The talk also addresses the role of the Security Response Committee in managing security vulnerabilities and incident responses, ensuring that issues are triaged and communicated effectively to the community. Additionally, advancements in vulnerability disclosure processes and the integration of security tooling are discussed, emphasizing the collaborative spirit of Kubernetes SIG Security in improving overall project security.

Hello, my name is Ian Coldwater, pronouns they them. Um, I am one of the co-chairs of Kubernetes SIG Security and you can find out more about my current full-time job at standwithmininnesota.com. Um, you are here at Kubernetes 6urities maintainer track talk, a bug's view. If you are here for a different talk, you're in the wrong room and that's cool. We're happy to see you. Anyway, this talk

will be a little bit different from our previous maintainer track talks. Usually, we just give updates about what we're up to, and we do have some really cool updates for you today. But we also wanted to take this time this time around to go through not only what we've been doing, but how we do it. We wanted to talk about the process. We wanted to talk about

how we think through these things and just kind of like the actual gears and how these things work for anybody who was curious or anybody who wasn't. So, um, we're going to go into what gets done and how we do it with a special guest appearance from the security response committee, who is not technically SIG security, but they're here today anyway. Kubernetes SIG security is a horizontal

SIG that works in collaboration with other SIGs across the Kubernetes project to make the Kubernetes project more secure. A lot of our work is done in our sub projects which we will be hearing from today. Um, our first update is from SIG Security Docs. Rory, tell us the latest. >> Awesome. Thank you. Uh, yeah. So, where are we slidewise? One more. Great. So, I Yeah, I'm here

to talk to you about SIG security docs. Uh, if you want want to join us, uh, especially after hearing about what we get up to, we meet on the first Thursday of every month at 2 PM Eastern. I'm one of the co-chairs. My name is Rory McHune and our other co-chair is just there, Sabatha. Um, and we do a variety of things to try and help out

with documentation. Our overall title is that we work across SIGs to improve security content of Kubernetes documentation as well as independently create security docs in the form of concepts and blogs. And it's one of those projects as one that we spent a lot of time on this year that I really wanted to talk about because it's it's kind of an interesting one and it's outside of the

main Kubernetes doc site. What we've been working on is the OASP Kubernetes top 10. So um a little background on OASP. OASP is the open worldwide application security project. If anyone knows OASP of old, you'll think, hang on, that's not what it stands for. They changed it. It used to be web application security project. They changed just worldwide because they do stuff other than web apps these

days. And OASP is an open source foundation that has a lot of things like software and documentation that generally attempt to improve software security. One of their most popular projects and the one that probably most people will know them for is a thing called the OASP top 10. Uh you'll see it mentioned a lot by people in like as a standard or we comply with it or

a tool says you know we implement the OASP top 10. It has been something therefore that people thought to replicate. So they said we've got this OOS top 10 for web apps. We could have top 10s for lots of other things. And one of the ones that got created back in about 2021 was an OASP Kubernetes top 10. What we realized last year though was that unfortunately

wasn't being maintained and updated. You know, the original maintainers for whatever reason weren't able to keep it updated. And so what we did was said, well, maybe we can help here. Maybe we can improve this. Because what we realized was a lot of people were going to that reference and using that for like how do I secure Kubernetes? And we were concerned like that's not going to

be up to date and they're not getting good advice anymore. So how did we do that? Well, we came together as a group and fortunately I had some contacts with O wasp. So talking to them and the previous maintainer, we got access to the repository which is all O was projects go through GitHub. So you just need access to a GitHub repo. We then wanted to say

right what would our current recommendation for top 10 be. Got together in a meeting and just brainstormed it like said hey what what could we put in a top 10? What do people who do Kubernetes security need to know about if they're new to it especially? We came up with 10 items which I'll show you on the next slide. And then what we did was say okay

let's ask the community because this is all meant to be communitydriven. Can we find out what people think like which of these 12 should be the top 10 and where should they be ordered? Like what's the most important one? What's on people's mind? We got over 150 responses to that survey which I thought was a great result both in terms of ranking but also they had ideas

for other things we might want to look at because those are things people were interested in. And with that done we could go to drafting. And again, it's a group effort because no one wants to have one person have to draft them all. So, we passed out to lots of people in the community and said, "Hey, help us." And they did, which was awesome. And we got

drafted top 10. And then we got reviewed and I want to give a shout out on camera to the reviewers who did a marvelous job. We got loads of feedback, especially we got some new contributors who hadn't been with done anything with us before who helped review that document and that made the outcome better. So reviewers, so even if you don't want to like write stuff and

you want to participate, reviewing output is awesome and it's really important. Even if it doesn't make sense to you, maybe it won't make sense to someone else. So we can improve it that way. So that got us to the point of last week we released it. So we now have a new updated OASP Kubernetes top 10 which looks a bit like this. So we've got these are

the things we think people should start with with Kubernetes security. And this was a successful outcome for us because that's now hopefully a thing that people will find and get new better information. And that I think to me is a lot of what we're trying to do, right? Is trying to provide that better guidance. Um it also gives some ideas for additional projects because there are other

things under the O banner that relate to Kubernetes and containers. Uh we might end up diving in there as well because maybe they're not getting maintained. And that's obviously a list of things you need to think about with regards to Kubernetes security. But to talk about a list of new things you might need to think about in the future for Kubernetes security, I'm going to hand over

to Ian to talk us through third party audit. Hey everyone. In the interest of avoiding a naming conflict here, I'm four-letter Ian. Uh get called that more often than I should do. Um yeah, I'm going to talk to you about audit, which I'm sure everybody is really excited to hear about at the end of a conference. Um so third party audit sub project is a sub project

that deals with thirdart third party audits. That might sound a bit obvious, but I like to think of it as just a nicely well- definfined scope. Kubernetes, I'm sure we probably all know, has quite a lot of people writing code, making changes, developing this thing that we all rely on to keep running because it runs everywhere. And while we we like to think that they all care

about security and focus on security, an external perspective is also hugely valuable. Sometimes having someone who's not been working on the same bit of code for several years looking at it can find problems that others can't. So, as a sub project, we try to work with external vendors to get that external perspective. Before I go too deeply into the what we've been doing most recently, we're going

to take a little trip down memory lane. We've had two prior audits over the last few years. One with a tread and trailer bits in 2019 and one with NCC group in 2022, which weirdly I was on the delivery team for. So I was actually writing the the report, sending it off to Kubernetes. Now I'm on the other side. Still not sure how that happened. Don't volunteer

for anything. You just get more responsibilities. Um so any before we published any previous reports any high-risk and severe findings have already been fixed. There is a private disclosure process which Tabitha will talk about soon. Um but there are there were a number of findings that have been fixed to make security of Kubernetes tangibly better. We like to think however there are still a number of findings

which are still open. I'm sure we've all seen the number of issues open on the K/K repo. There are so so many of them. Uh so before we started looking at a new audit, we went back and checked the previous findings to see what was still open. If anybody really wants the numbers, uh there were 13 of the 37 findings from the 2019 audit still open. The

oldest one was actually a finding from the audit that was closed as a duplicate of finding number 18982, which I believe is Rory's favorite finding. Uh it just celebrated its 10th birthday, so I'm sure we'll see traction on that one any day now. It it'll be fine. Um that finding relates to um certificate revocation in user identity searchs for Kubernetes and there's been a great discussion over

the last few years about if that will ever happen. From the NCC audit there are still nine findings open. Some of them have been closed as um an active decision that this is just how things work and yes it's a good observation but we're not going to make any changes. Some of them were closed with technical fixes. some are still open and that pesky automation robot that

likes to mark things as stale after a few days got quite a lot of them. So, if you are a member of a SIG who had findings which are closed by automation and you're wondering why I'm reopening a four-year-old finding, uh that's because we're just trying to make sure that we get some kind of closure on the previous findings. We're going to be trying to work with

the relevant SIGs to triage those again and either get a yes, we are going to fix these issues or a no, we don't think this is a priority. We're going to actively state that this is probably not going to get fixed unless we get a lot more volunteers. If you do want to get involved and try and fix some actual security stuff, there's a great list of

findings. They're all tagged as wgs/security audit on GitHub under K/K if anybody wants to have a look. And that brings us up to where we are now. Over the last year, we have been working with the Open Source Technology Improvement Fund and Shielder. uh Shielder were the organization who actually performed the audit over the last year. This is going to be the kind of two-minute summary of

what happened. If anybody really wants to know what happened in this um we actually gave a talk yesterday, about a 30-minute talk on more detail, I can see a couple of people in the audience who were there yesterday. I hope you're enjoying this very short version. Um there will be a recording of that as soon as conference talks are published. So that audit started in early 2025

after a vendor selection process which is all documented on GitHub under the case security repo if again you want some extra reading. The audit was uh performed over the summer of last year. There were a few findings. There were some which we have already discussed published fixes for and have all been addressed. There are others which are still working through the process. The scope for this was

a little bit different. I don't know if anyone's ever gone and counted the lines of code in Kubernetes. I assume you haven't done it manually. If you run clock uh to count lines of code in a git repo, it returns somewhere between 2 and a half and 3.8 million lines of Go just in the core repo depending on how you slice it. That's quite a lot of

code and but also that repo had been reviewed quite a lot before. So what we decided to do this time was look at non-core components. So some of the surrounding SIGs, some of the technology that builds and powers Kubernetes and helps people keep it running but isn't the API server, the cublet, the absolute core So we we asked Shielder to have a look at quite a lot

of these. We basically had a list of uh I my speaker notes are so good they don't have the text in the slides and 93 areas that we wanted to look at. We basically asked Shielder to we gave a priority suggestion and we said to them work through it in this order unless you really want to look at something else at which case speak to us and

we can discuss. They worked their way through as many of the components as we did. Raised a number of really good findings and if anybody really wants to know the list of things that were particularly focused on for this audit, there is a list here. I'm not expecting anybody to read this in 10 seconds. I'm also not going to stand here and read it all out to

you. But uh thanks to Shielder for having a look at these components in particular. We did get a great report sent over at the tail end of last year. That report has quite a lot of security related information in that we don't necessarily want to be public right now until it's fixed. So as much as we're a community that likes to work in the open where possible,

I'm going to pass you over now to Tabitha to talk about how security vulnerability information is handled. >> Thank you. Yeah, like like he said, I'm Tabitha Sable and in addition to being one of the SIG security co-chairs, I also serve the Kubernetes community as a member of the community of the security response committee. So within the Kubernetes governance model, there are a lot of SIGs and

as a SIG, you are required to do all of your work in the open so that we can continue to be an open community. Because there are some things that need to be dealt with privately. There are also committees and your name being committee is basically the license to do something in private. So security response committee is kind of like two sides of the same coin with

SIG security. There's the public side and the confidential side. Security Response Committee handles those confidential security issues like security incident response for the project's infrastructure and the disclosure and fixing of security vulnerabilities which of course begs the question what is a security vulnerability and this is actually fairly straightforward. It is any time that the software behaves in a way that violates someone's reasonable security expectation or an

explicit security promise made by the software. Uh, a lot of these a lot of times when you've been working with vulnerability management for a while, you end up saying CVE, which is shorthand for entries in the common vulnerabilities and exposures database because that's one of the most important ways that we publicize these things. So through the rest of this section, if I say vulnerability, if I say

CVE, I'm probably talking about the same thing there. So sometimes a software vulnerability is an obvious bug like one of the most recent ingress engine X CVEes was if you send gigabytes of data to the admission control web hook, you can run the admission control web hook out of memory and either crash the pod or crash the node that the pod is running on. Anybody on the

network can do this because it's possible to do authentication on admission control web hooks, but it's so difficult. I have never seen anyone actually do it. Which means with this behavior in the software, anyone on your network can run an important pod in your cluster out of memory and maybe take a node with it. It's reasonable to expect that installing ingress engine X shouldn't expose you to

mean network people crashing your nodes. And so since that reasonable expectation is violated, that behavior is a vulnerability and we worked with the maintainers, got it patched, released it, published it. Not every vulnerability is due to a software bug. Sometimes some of the worst vulnerabilities are due to proper software features. Another recent ingress engine X vulnerability, this one is CVE 20251979. Ask me how many times I've

had to talk with people about it. Ultimately was because in that in that admission controller, we ran a test for the validity of the generated EngineX web server configuration. This is clearly a good idea because you're taking all of these ingress resources, you're putting them together, you don't want to accidentally break the data plane. So, you should run a test, right? And then some folks made us

aware that the EngineX configuration test can be tricked into loading and running arbitrary code off of the file system. not again like this violates a reasonable expectation of the software which is that you can't just take it over because you're on the network and so that feature very quickly became recognized as probably the most serious vulnerability in Kubernetes history. So we had to deal with that. Triaged

it in in private fixed it publicized it as widely as we could. Because of this sensitivity, vulnerabilities and their fixes have a special life cycle. Because once somebody in the world knows about a vulnerability until everybody knows about it and has patched that knowledge can be used to attack users and you know steal their data, hurt hurt them in some way. And so we we handle these

things in a special way. We receive reports privately through the Kubernetes bug bounty system on hacker 1 or on security at kubernetes.io and members of the security response committee have a weekly on call rotation to catch these triage them give it a sniff test. Does this report make any sense at all? And then contact the security contacts members which are stored in GitHub basically next to the

owner's files. Then with those maintainers of whatever the project is, we ask ourselves important questions. First off, does the software behave the way that the reporter claims that it does? If it does, is that behavior in violation of a reasonable security expectation of the software? Is it a vulnerability? Is it a bug? Is it something we can just ask them to open a public issue for? or

sometimes is it actually just the way the software works and there's there's no follow-up to do with it. And so after we have made this kind of determination, let's assume that it's a vulnerability, then we work with the maintainers to get it fixed and then we publish it. Spread the word about that far and wide. First, we open an issue in the KKK repository and we put

some special labels including official CVE feed onto it and that serves as our sort of central canonical source of what is the public disclosure that we're making about this vulnerability. But then we spread the word much more widely as well. There are three separate emails. one that goes to a few hundred,000 people worth of Kubernetes lists, one that goes to the open- source security email list, and

one that goes to the online discuss Kubernetes forum. I believe that a lot of folks use it, but I'm not personally one of them. And so, if you do use it, I hope that it's helpful there. We post into the announcements channel on Kubernetes Slack. And then one of the most important long-term places is publishing it to the international common vulnerabilities and exposures database. The reason this

is so critical is that a lot of organizations monitor CVE. Some organizations have compliance requirements to look for vulnerabilities that have CVEes issued for them in their environment and fix them. And a lot of that triage is done based on this CVS, common vulnerability scoring system score that the vulnerability has. See a little bit of how the decision tree works. And there's a mathematical formula that, you

know, sets a numerical value based on, you know, whether it requires high, medium, or low, you know, high, low, or no privileges, that sort of thing. This looks like it should be simple and repeatable. And it's more simple and repeatable than just having somebody vibe check it. But there's a lot of interpretation that's necessary to answer these questions. A couple of ones that often produce confusion for

folks with respect to Kubernetes CVEes are the question of do you have to access it over the network or does it happen locally? Well, if it's something that you put into a API server request, then that's network because local means you are sitting at the keyboard and mouse of that computer. And so Kubernetes components almost never have local vulnerabilities because they're almost always running in a data

center. Um, the one that really trips people up is privileges required because if you think about a Kubernetes control plane, no privileges. That's pretty straightforward. or you don't have any access at all. But if you have the ability to create pods, does that mean you have low privilege? Does that mean you have high privilege? And we have to take context into into account here. Sometimes having any

cluster access at all might be considered high privilege. But if you're talking about something where somebody can go from a namespace administrator to a cluster administrator, okay, now we know what high, low, and none are. And these these kinds of things have to be evaluated carefully so that we don't either get people to start ignoring Kubernetes CVEes because they're always rated critical. What does it matter? Or

also to make sure that we don't rate one too low when it's actually important such that people deprioritize it. So we have to use a lot of this judgment. All of this communication to people around the world is a tremendous amount of work. And one of the things that has been happening in tooling helps us to make that more easy. And so I'll pass it along to

Mah to talk about that. All right. Thank you, Tabita. Uh yeah, I'm I'm my um representing uh six security tooling here. Um uh let's see what six security tooling does. So basically SIG security tool to tooling maintain uh code base to help u make communities more secure by writing code like across sigs and uh I wanted to mostly mention these two projects. So Tabita touch like uh

talked about like all the what the SRC is doing. they are disclosing vulnerabilities and uh we published this thing called the official CV feed which is uh a way for people and in individual or security teams to consume what the SRC is disclosing. So basically it's under the form of this web page but we have like a JSON feed or nurses feed that people can subscribe to

if they need to and we also maintain the sneak scan which is um like a scanner like running on KK repo and checking for the dependencies to see if anything uh is vulnerable to any any CVS. So yeah I wanted to highlight uh two things we've been doing lately. So uh one of them is on the official CV feed thing. So basically people approached us saying that

uh they really liked the official CV feed but they wanted to consume the information in a more structured way and um basically right now the way it's done uh the SRC when when they are writing a new CV uh disclosure they follow some kind of templates but it's kind of in a free form way like they don't really uh use uh JSON form or something like that.

So in the end this is what we consume. Tabita mentioned that they are labeling the issue with the official CV feed. Retrieve the information and it becomes uh the official CV feed. So for a while some people have been uh writing the missing information uh that they wanted to add like uh this structured information and it was in the form of this thing called OSV which is

a JSON schema describing CVE and the issue is that because people were doing that manually uh most of the time it was quite late after the disclosure and uh it was not that useful in the end. So we tried to brainstorm how on how to make it happen so that like when a new CV is disclosed we can have this information immediately. So we we we thought

like yeah let's just ask the SRC members to do that for us. And I mean they are already like doing plenty of stressful work. They are reviewing all these emails. So of course it wasn't a solution. So we we we came with this idea of maybe we can try to make their life easier and along the way as a byproduct we can produce this OSV thing we've

been looking for and the result was this kind of thing. So it's it's basically a very simple CLI. Uh it's a glorified form basically. Uh you can browse uh from a few question and answer them like uh in a g commit fashion and at the end you can export all this information on uh different uh templates. So this is basically the thing they have been using forever

like uh the issue templates, the email or the slack message. All right. Um so Tabita recently gave it a try on the many ingress engineics uh CVS that she released and apparently it was useful because she had to do a lot of uh the sameish uh CV disclosure. So using this kind of tool actually made our life better and she gave us a lot of feedback on

how to improve this thing. And so if you go to the uh last one that was released uh last week, even if you scroll down uh the issue, you can see an an HTML detail tag thingy and you will find the OSV feed format here. So basically now the idea is to combine this information that we have like on every new release of CVS with the historic

information that we've been manually writing and and make this into the official CV feed so that everyone everybody can consume that. Uh the other thing I wanted to mention is that uh in the KK repos we have many CVS uh that have been uh uh managed by the SRC and the way it's done is that they open this issue and when it's fixed and everything is done

they close the issue but we have a bunch of these issues that are still open and uh those are like CVS from 2020 and 2021 and when they were like uh registered into the CV database uh they were like marked as fixed as the that was released in the days. So the situation right now is that we have like a like I think it's like three or

four four right CVS that are open on like very outdated version of Kubernetes and they are like technically fixed but in reality they they just we we didn't do anything about them. So the the idea right now is that we would like to update the metadata so that it's correct and that people can see that the CV are still open in the codebase but at the same

time we don't want people to freak out that like their latest uh release uh like their latest deployment with the latest release of Kubernetes is vulnerable to like 2020 CVE. So alongside of that we are releasing a blog post uh Pushkar and Tabita have been working on that to explain uh the situation and and secondly we'll introduce for the future a new won't fix label that some

the CVS will be uh marked with so that like it's clear in the future that those CVS won't be touched won't be deal with. Uh yeah I just wanted to give a few shout outs uh because lately we we we we had a bunch of new contributors to six security tooling. So I wanted to do like some public uh thank you. So Ariad joined recently and uh

he did a lot of good work on the CV feed. So basically it's a bunch of scripts. It did a lot of bug fix and improvements. And uh Sherin also joined and she did a great work on fixing the situation in which we had like actually add some CVs that were missing from the official CV feed. It was spotted by Rory by the way. Thank Um she's

also been interested into adding go like using go checks if you are familiar with that uh on KK and and trying to um reach to the secret release like team to work on that and of course Eric uh he has been maintaining the sneak scan thingy and and pushka and tabita have been doing like a lot with the sik so yeah thank you and lastly uh right

now I think it's a good time to join six security tooling because we have like this b we have a bunch of issues that are open right now uh we meet every other uh Friday basically at the end of the afternoon in uh Europe time and uh yeah I think uh if if you are like searching for simple issues and a way to enter SI security I

think it's a good way to start with and that tuan thank you >> so Kubernetes SIG security speaking of getting involved really prides ourselves on creating a welcoming inclusive environment that um that welcomes in new contributors of really every experience level. You don't have to have a security title. You don't have to be a developer. If you are security interested and interested in contributing to the Kubernetes

project and helping make it more secure for everyone, come to one of our meetings. This lovely QR code will bring you to our mailing list. And if you join that, that will give you the calendar invites for both our main meeting, which happens every other Friday at the opposite weeks of tooling. Um, and our sub project meetings as well. Um, we don't have a membership roster. We

are made up of whoever shows up. So, come show up. Bring your curiosity. Bring your ideas. We are really excited to get to hear from and learn from you. And even if you are the newbiest newbie in the world, you have a lot to teach us. So, thank you so much for taking your time out of your day. Um, we really appreciate you spending this time with

us. Um, we are Kubernetes SIG Security. You can also find us on Kubernetes Slack at channel SIG-Scurity. Thank you all so much and I hope you have a wonderful rest of your day. We have like a minute and a half. So, if somebody has a really quick question, otherwise we will be around and we have a booth at in the project pavilion at P4B if you want

to come talk to us. We will be there actually for the next couple of hours. So, thank you all so much.