Kubernetes Security at Shopify Scale: Automating Security Across an Infrastr... Jie Wu & Pulkit Garg

This talk focuses on Kubernetes security implementation at Shopify, as presented by Jay and Pulkit from the infrastructure security team. They describe the challenges of managing security in a vast Kubernetes environment, where hundreds of services and thousands of deployments occur weekly. To mitigate risks, they introduce practices to catch misconfigurations directly in the CI environment using Semgrep, a static code analysis tool that identifies vulnerabilities and provides guidance to developers. The speakers also discuss the importance of training and educating developers on secure coding practices while maintaining deployment speed. They highlight a collaborative approach, incorporating AI to refine rules and improve detection accuracy.

Everyone, thank you all for joining us. Um we're very excited to be here. Um today we're going to going to to take you all on a journey in how we secure Kubernetes at Shopify. Uh my name is Jay. Um I'm really excited to be here visiting Amsterdam. I flew in from New York City. So, that tells you two things about myself. I really like pizza and I

walk very fast. And I'm joined here today with my colleague My name is uh Pulkit. I'm from Ottawa, Canada originally. And I've been in three different time zones in the last 2 weeks. So, still a bit jet lagged, but super pumped to be And first time attending, first time speaking. Yeah. So, we're both on the infrastructure security team at Shopify. Uh so, what we do is we

help secure infrastructure for millions of merchants. and but for today, our official title is professional cat herders. Um so, what is Kubernetes security at scale? It is like herding cats. Well, except that these cats can accidentally expose your entire infrastructure. So, uh let us show you what do we mean. So, let's set the scene here. Uh welcome to Captain Q's kingdom. You might recognize Captain Q from

Fifi and Friends. Um so, you got server land, you have cluster island, you have deploy road, you have uh helm workshop. Um all of this uh at the center is held together by the monorepo. Uh which we'll dive into later on. Um this is a lot to take in already. So, Pulkit, let's walk through the scale. We run hundreds of services on K8s, making it an extremely

important part of our infrastructure. Our developers push almost 60K deploys each week, which is just going to increase as we keep scaling up and growing. All of this means we have close to 500 clusters deployed worldwide in different regions and zones. With all this massive infrastructure, we power millions of merchants and help run their online stores. Now, with everything this, let's look at how all of this

set up in the back end. So, all those services you just saw, so where is the Kubernetes manifest files that defines them? Where is the helm charts that the developers use them to template? Where is the deploy configurations that helps them roll them out? What are the policies in place to better govern them? All them live in one place, the infrastructure monorepo. Which give uh which give

us one source of truth, but also one source of risk. A single change here can ripple across everything. So, who makes all those changes? And this is where we introduce Dev, which is a uh developer at Shopify. Uh Dev here will regularly make changes to the infrastructure monorepo. Um you can see from the structure here, you have storefront, checkout, payments, um each with their own environments. So,

and this is the similar setup for hundreds of applications, all within one repository. And Dev here uh ships changes pretty much every single day. And that's a lot of changes, but Dev [clears throat] Dev has a awesome partner. Meet Hazel, also from Fifi and Friends. Uh this is the helm the helm hedgehog. Uh Hazel here will take take the um Dev's single helm chart and renders it

into hundreds of manifest files. the these will get shipped out to multiple environments. Uh together, they are a awesome duo. Dev will write it and Hazel will multiply it. Together, they work great. Everyone's happy. They're shipping code. Uh life is good. >> [sighs] >> but not everyone will follow the rules. Um so, not every change that do make it to production is a good one. Every misconfiguration

that ever snuck through a code review, a security review, every bad configuration that someone just copy and paste. And that is where we introduce you to Mischief Cat here. Mischief Cat does not mean to cause any harm, but the damage here is real. So, here is what Mischief Cat will look like in practice. So, we have privileges set to true, which is gives you full host access.

We have resource that's set to pretty much nothing, so there's no limits set. And this is in a shared helm chart. So, remember Hazel from earlier, the helm hedgehog? Hazel will just multiply this and renders it into hundreds of manifest files. So, if there's a one bad line or two bad lines or multiple bad lines, it gets multiplied to hundreds of manifest files. Which is bad because

this is our blast radius. whenever there's a bad a bad misconfiguration snuck in by Mischief Cat, if it is not caught, Hazel will continue to renders it across hundreds of manifest files. If it goes through a PR review and no one catches it, it may get deployed into staging, then production, and potentially impact our what makes this even more scary is that Shopify developers ship fast. Um

there's a over a thousand PRs that gets merged every single week within this one Um so, we don't want to slow that down, but at this pace, is anyone watching Mischief Cat? Um so, we hit a tipping point. Misconfigurations were slipping through, which is not great news for our on-call engineers. Um so, what we do know is what does a bad a bad pattern or a bad

config look like. But we just cannot keep up with the number of changes going through the pipeline. Uh so, we couldn't catch them fast fast enough. So, we couldn't keep on going like this. With all this scale, all the infrastructure, all these merchants, all the problems that we just described, something really had to change. So, we sat down as a team, looked at the problem, and came

up with a plan. First and foremost, we wanted to catch the misconfigurations directly in our CI environment. Secondly, we wanted to add the rules before it is deployed. So, before they make it to our production environment. Thirdly, we did not want teams' productivity to be impacted by our controls. We did want to flag and solve the misconfigs, but had to find a sweet spot between security and

speed. Last but not the least, we did not want to just deploy controls and let developers fix it. We wanted to educate and show developers the green path by adding in documentation and guidance so that misconfigs actually never make it to our checks. So, now let's dive deeper into how did we catch the misconfigs. Here, we introduce you to the Semgrep Cat, the code inspector. Semgrep is

an open source static code analysis tool that searches for specific semantic patterns defined in the rules. As you can see, it can fix both the vulnerabilities that we flagged before in a misconfig, and also keep the Mischief Cat in check. Let's dive deeper into some of the rules that we have in place. This one is to make sure that we have no privileged containers. When a container

runs as privileged, it gets full access to the host, the kernel, the devices, and the file system. Basically, the walls of the container disappear. It's one of the most dangerous settings in Kubernetes. So, three lines of pattern matching, which looks for the exact string and warns the devs to fix it. This one is a bit different. We are not looking for something bad, but we are looking

for something that is missing. In K8s, every container should have CPU and memory limits set. Without them, a single pod can consume all the resources on a node, starving every other workload running there. It's one of those things that looks fine in the YAML because there's nothing wrong to see. The problem is that it's not there. So, if a container exists, but has no resource limits, Semgrep

can catch it. Now, this rule catches three ways a container can access the host. Host network gives the port direct access to nodes' network stack. It can see all the traffic or can bind any port. Host PID lets it see every process on the node, including other containers. And a host IPC lets it share memory with host processes. Any one of these breaks the isolation that makes

containers safe in the first So, three escape vectors, one rule. Again, if any of them appear, Semgrep would warn the dev. So, three rules and three classes of misconfiguration, all caught uh automatically before merge. These are just some of the critical ones that we have for K8s, but there's more that our Semgrep Cat can do. So, beyond the three that you just saw, those are pretty big

rules that you put in place, there are other rules that we can create to better secure our infrastructure. So, there are rules that that we can create to prevent risk like supply chain risk or find missing security policies or missing authentication. So, here in this example here, you might see your image tag is set to latest. So, it flags to see if there is any images that

has been tagged with latest. So, as such, what this means is if there's a new update to that image, you don't really know what's actually released. Someone might might introduce something new to the image, but you never know cuz it's such a leaders. So, this is a supply chain risk. You want images to be pinned to a specific version or shot. So, you know till that way,

you know exactly what's In this next example here, it really depends on how your organization handles authentication. Every every every setup is different, but the idea is the same. If the ingress is missing a required annotation, it could mean that there's no authentication properly configured for your application. You can create a rule for your developers and warn them before they push code into production or they ship

a new application into production. This way, you can ensure that authentication is properly configured for their app. And this is what it will look like in practice for our developers. A developer will open up a PR. Um in this PR, if they have a misconfiguration, our CI will now flag it using Semgrep. Um so, in this case, you see privilege is set to true under security context.

Um developers themselves, they may not have the best context like security context. So, we want to make it we want to make sure the guidance here is easy to follow. They don't have to guess what is wrong. It's right there in clear message. And this is basically our way of setting up the green path that we want them to follow. This is what the green path will

look We did not want to block a bad config, but we want to educate our developers to understand why like what is the best practice and what are the steps that they can take to remediate those type of findings. And this is done via a self-serving guidance that we put in It explains the why. Why is a previous container bad? And that's because a previous container has

nearly the same access as the host, and they can bypass a lot of other security security protections. And it also shows the exact fix. Um so, in this case, you can set allow privilege escalation to false as a way to resolve these type of findings. Um so, ultimately what we want the guidance to be is that it must be easy to follow. Then developers can self-serve and

resolve these findings by themselves. And remember Hazel the Hedgehog from earlier? Hazel did multiply a lot of those bad misconfigurations, but if we do find a good configuration, a secure way to make a change, it will also multiply them across those applications. So, even if we change one line a if we fix this configuration in like in one helm chart, it will get multiplied across hundreds of

manifest files. So, Semgrep will catch those misconfigurations. Dev will then fix that one line, and tada, hundreds of applications is now secure. And you see storefront, payment, and etc. So, when it comes to create these rules, we create a bunch of them. Uh but it's not a set and forget. Um there we write we do write these Semgrep rules and policies, but and but the ultimately we

want to follow a secure way to test them out. So, we always want to test them either in warning or send by a dry run um pipeline to further test them. We want to monitor for any false positives or potential gaps. So, we did work a lot. We collaborate a lot with infrastructure teams or whoever the app owners are to better understand their application and see if

there is any edge cases that we need to um we need to take care of them. So, if a pattern that we have found as we create these new rules, if it's very very risky for these changes to ever be in production, um we enforce the CI to block the changes. Otherwise, it will be surfaced to developers and they can address them themselves. And we basically iterate

this entire process whenever we create new rules. We ultimately treat these type of rules the same way we treat production code. So, some of you might be thinking, uh why not OPA or why not Gatekeeper? For our setup, uh Semgrep wins. So, it scans the the the templates directly uh or the manifest files directly. It runs in CI whenever someone creates a new PR. But if your

configurations come from many repos or you need a specific cluster enforcement setup, then Semgrep might be a better fit for that. For us, the mono repo means that everything goes through the CI. But with that said, we didn't completely abandon OPA. Um so, where does it all fit in? So, meet Alena, also found Fifi and Friends. This is the policy enforcer. So, if Semgrep Semgrep cat here

will inspect the code, Alena here will enforce the rules in our infrastructure. But our Alena enforce it beyond just Kubernetes. It will be in Terraform environment. So, Semgrep is great at getting those files those new changes, but OPA is great for evaluating logic over outputs like uh Terraform plans. So, while they may be different tools, both of them share the same philosophy. We want to catch it

uh before it reaches So, here's a real example of what that would look like. So, this is a Regal parse policy that checks for ABGK node pool changes in the CI. So, if someone happens to use a restricted uh GPU type, um they will get flagged because they do need approval for it. Uh but ultimately, this is what it will look like for the developer. So, it

will flag it, present them with guidance on the next steps. Um And they can go through the process themselves. So, you heard a lot of rules that we created and also the policy we created throughout this journey, we created a bunch of them. Along the way, we did have a little help. So, this helper, I'm sure you must have heard about it and should be actively using

it for various use cases in your organization. We introduce you the AI cat. We use it as a risk finder or the rule refiner. So, just to give you some context, initially when we started with this work, we had close to 38,000 findings because we were using overly broad patterns in our Semgrep rules. We also initially did not account for applications which needs to be exempted for

valid use cases. These many findings are not actionable. They would not only add extra toil for developers, but also might crash our CI So, that's where AI helped us a lot. We asked AI to look at the findings, asked if it really makes sense, and asked it to refine the rule and make the patterns more granular. After going through multiple iterations, we were able to reduce the

findings by 88% to actionable true positives, and that represents the actual risk. Let's dive deep into how we actively use AI to write Semgrep or OPA rules these days. So, we start with a simple plain prompt. AI does its magic. We have a working rule ready in a few minutes. A human would review the rule, and then we ask AI to run it against our infrastructure mono

repo locally. If we are not satisfied with the number of findings, we ask AI to refine it and follow the same process again. We ship the rule only when we are fully satisfied with the number of violations. Not only that, when we ship the rule, it goes through a dry run pipeline first, which does not flag the developers, but gives us insights in a Grafana dashboard for

the number of hits we see in our actual environment. Once teams have alignment on that, only then we ship the rule to production. Now, even if Jay and I work together or the entire infrastructure team security team, we would not be able to develop a rule from zero to the end as fast as AI. Not only that, with the scale we work in in the mono repo,

it could be easy to miss serious edge cases which could cause disruption which AI did help us find. Now, to give you a high-level overview of the entire process, developer and helm chart ship the PR following the best guidelines we have in The Semgrep cat flags any misconfigs. AI cat would help us and the devs fix the violations or refine the rule in case there are any

false positives. And in the end, everything runs securely and kube commander, I think it's pretty happy, but So, you might be wondering, did it all work? What is the current state right now? Are there any gotchas for us? So, just to give you a context, we shipped more than 50 rules within a span of 6 weeks. We also deployed 13 OPA policies for keeping our Terraform code

secure. We have also added guidelines for other developers on how to write Semgrep and OPA rules with the assistance of AI. With that in place, we have seen an increasing number of developers adding to our security controls for various other use cases. Now, when we were writing these rules, we also made sure that we are covering all the bases for various security frameworks. So, we asked AI

to benchmark our suite of rules against these frameworks, and in some cases, it did suggest us new rules which we missed. So, how does the reality looks like now and how was it before? So, just to give you an idea, every single PR had to be manually reviewed. And with the scale we work in in the mono repo, there were times, unfortunately, when some misconfig reached production

and caused incidents. However, with our security controls in every single PR, as we speak, is scanned by our rules. We caught more than thousand misconfigs that were surfaced to developers and fixed. And thankfully, we had zero incidents after we rolled out the rules. It did take some time to get alignment with the teams initially, but once we had it deployed, we did not have to revert any

rule. Hey, look, that's our team. So, we are the security cat in this story. So, the goal was never to just catch violations. We want to make sure security is the default. We want to educate our developers on what is the proper path forward. And this is where our where Security Cat comes in, which is what our team is trying to do. We Our job isn't isn't

to block deployments. It's to make sure that every single change that flows through the pipeline is secure by the time it ships to production. So, when the secure path is the easiest path, developers will follow it. So, with all the character that you have in this story, we have Semgrep Cat that will scan every single PR in our repo and check for misconfigurations before they are merged.

You have Security Cat here, and that will make sure that the pipeline will continue to stay secure. We'll continue continuously look for new ways to secure this pipeline. And we have our our Lena here, which will look at other infrastructure changes to make sure that they are And lastly, we have the AI Cat, which will help us find potential blind spots or edge cases that we may

have missed on during our rule development. And together, we have the Guardians of the infrastructure mono We Well, before we actually roll out all those any of those changes into production, it wasn't always smooth sailing. During testing, we found a ton of false positives. There was thousands of them, and while the rule looks great in theory, we had to look into it a bit more further and

working with the teams to understand their applications and see if we have missed potential edge cases. So, we had to iterate many, many times to get a better understanding of the applications along with their teams to make sure that we're actually capturing actual risks. So, and we also have a a countered a lot of pushback. Developers like we see cases where developers ignore the warnings. Um if

developers ignore the tools that we have put in place, might not be the best. So, we want to make sure that the guidance that we put forward are easy for them to follow the exact what to do and what not And we also had to course correct a few times, so we had to address our documentations fairly quickly, making sure that any new rule that we rolled

out, it is actually documented with the proper steps for developers themselves to follow. We fine-tuned the the rules as soon as we found an edge case. And we aligned with the teams responsible to make sure that they understood the controls that we're putting in place and the proper steps that they can take to resolve the findings themselves. And where are we during all this chaos? In the

middle of in the middle of the storm. uh we did have to navigate a lot throughout this entire project, but we also have learned a lot of very valuable lessons. So, here are the key takeaways from this story. There are four lessons that we learned the hard the hard way throughout this project. So, one, always start in warm mode, as always. If there's there need to be

a better way for us to test out these changes. Um so, either it's staging or driver mode, we don't want to actually impact developers, but we want better idea of what it will look like in production once these controls are in force. Two, we want to make sure that any false positive that we have encountered, we want to make sure that it's to a ideal volume. So,

we want to address them right away, making sure that we're actually detecting detecting the actual that that might impact our infrastructure. So, we want to make sure that the developers can see that these rules are actually accurate. Three, we want to give the developers clear steps to remediate those findings themselves. Um so, we want to better educate them on what is the best practice when it when

it comes to security and Kubernetes. And four, we want to make sure that the guardrails and the controls that we put in place is the secure path and the easiest path for the teams and developers to follow on their own. it all comes down to this. We want to pave the road that the teams will run on. So, ultimately, we want to make sure that security is

built in to the path, and the developers will use it. And with that, So, with that, all the cats are herded along with the other friends. So, we have the developer and the helm hedgehog shipping PRs. We have the security controls actively watching it. We are right by the door watching for new security vulnerabilities as they come up and keep on adding more And the Mischief Cat

is happy and calm. And this is the cast for our story. Uh so, all the characters along with Phoebe and friends, they're all here. uh this is where you can find us. And also, you can scan the QR code. We won't hack you. So, this is totally safe to to scan if you want a copy of the slides. And that's all. Thank you. Thank you. >> [applause]