Invisible Guardrails: Enabling Developer Velocity W... James Elías Sigurðarson & Vignir Hafsteinsson

In this talk, the speakers, James and Vignir from Asana, discuss their experience building a robust infrastructure platform with a focus on security. They describe historical challenges such as the trade-off between security and developer velocity, as well as the scaling issues that arose from using single-tenant Kubernetes clusters. To address these problems, they developed a multi-tenant architecture called 'cells', which supports isolated failure domains for better stability and security. The speakers delve into various security features they implemented, including fine-grained IAM permissions using Crossplane and network policies with Cilium. They also emphasize the importance of automation and the principle of least privilege in permissions management while detailing how their approach has shifted responsibilities towards application teams. Finally, they share valuable lessons learned throughout their process, highlighting the necessity of clear communication and early conversations between platform builders and users.

Hello, and welcome to our talk. Uh my name is James, and I'm an infrastructure engineer at Asana. And my name is Vignir. I'm an engineering manager with one of the platform teams at Asana. James and I, we work out of Asana's office in Reykjavik, Iceland, where we are yeah, basically build weird infrastructure every day. Uh quick intro to our company. Asana is a work management platform, and

Asana's mission at Asana, our mission is to help humanity thrive by enabling the world's teams to work together effortlessly. As such, the Asana product is trusted by over 170,000 organizations with millions of users that depend on Asana to get their work done every day. As such, we need to build infrastructure that is stable, performant, but most of all, secure for those users. And our teams at the

Reykjavik office have been responsible for building that infrastructure. Now, today we want to tell you about our experience of building out an infrastructure platform with security as one of its first-class features. We'll start out by sharing some of the historical problems we were facing and some of the difficult trade-offs we were making in favor of security. Next, we'll discuss how escalating scaling issues presented us with an

opportunity to entirely rethink our architecture, define some guiding principles that allowed us to reject some of those old trade-offs. We'll then dive deep into some of the security features we built into the platform, and finally, we'll share some of the lessons learned from our experience that will hopefully help you on your own journey. So, at Asana, uh security is our highest priority, but for a long time,

we were forced to accept a difficult trade-off, security versus velocity. Some investments we made in security came at the cost of developer velocity, while other investments that were meant to increase both security and velocity often did not scale. At Asana, we started our Kubernetes journey in AWS many years ago by creating single-tenant clusters, which we called Cube Apps, which were fully operated by the infrastructure teams which

owned them. Our infrastructure security team could then treat each of these Cube Apps as a black box, securing it externally using AWS IAM roles and security groups, all managed centrally by them. Now, this this did give our infra teams autonomy, but at the cost of control over their security. This model However, this model broke as our infrastructure grew in size and we had more of these Cube

Apps and teams owning them. Changes to security configurations, such as allowing applications access to AWS resources through IAM, or networking changes through AWS security groups, had to be both reviewed as well as deployed through one single infrastructure security team. This meant that teams met a lot of friction in delivering their proposed changes. The wait for a simple security change could be measured in days, not minutes, as

engineers had to both wait for security reviews as well as coordinate the deployment of their changes with a deployment of security Infra engineers were deferring responsibility to our infrastructure security team. That team became the bottleneck and our engineers lost the ability to fully take responsibility for Now, ultimately, this centralized manual approach broke down as both our organization and our infrastructure grew globally. Compounding this challenge, having our

infrastructure built on multiple single-tenant EKS clusters presented its own set of critical scaling problems. Operationally, each of these clusters essentially became a single point of failure, causing stability issues for our customers. A bug or misconfiguration in one application could take down Asana entirely, while an innocent-looking change to a security group could also cause a full downtime. And organizationally, this meant that a high operational burden was being

placed on each on each team, which all had to become experts in owning and operating these clusters. The same problems were being solved again and again by multiple teams, and we were keeping highly skilled engineers away from operating in their zone of excellence. Our new architecture, which we call cells, was the breakthrough. We built multiple instances of multi-tenant EKS clusters, but crucially, each was designed as an

isolated failure domain, only serving a subset of our customer traffic. We established dedicated platform teams who built out this Kubernetes infrastructure platform, enabling teams to focus on their business logic rather than operating Kubernetes clusters. Now, building out a multi-tenant platform created a lot of new security challenges for us that we did not have in our single-tenant world. It is worth to note that we were running our

own trusted workflows, but there were still many considerations we had to cover. But, this rewrite also allowed us to address some of our historical security challenges while introducing brand new ones. But, best of all, it gave us a chance to reject that old false trade-off of security versus velocity, allowing us to ask ourselves the question, what if the most secure path was the fastest one? to sum

up our challenges, we had sacrificed velocity for security. By centralizing security infrastructure, we had created a single point of failure both operationally and organizationally, as teams were not taking full responsibility for the security of their of their applications. And as we rewrote our infrastructure due to scaling needs, we introduced new security concerns, but along with new opportunities. So, how do we address those concerns as we rebuild

our infrastructure to meet our scalability needs? At Asana, our prioritization framework is simple: security, followed by stability, and then velocity. This means we should always make sure to prioritize security even when it comes at the cost of doing things slower. But we really wanted to do great things fast. So, we asked ourselves, how do we enable velocity by baking stability and security into the platform? We started

off with a couple of principles. Firstly, we wanted to adhere to the principle of least privilege and ensure everything had the minimum permissions necessary in order to work. Secondly, we wanted to be able to deploy permissions with GitHub just like our application code. And finally, we wanted to be able to use automation and not rely on humans or agents to do everything perfectly. The breakthrough was shrinking

our permission unit. We stopped thinking about permissions at the at the Black Box cluster level and started thinking at the application level. This way a permission is now just another part of the app definition. No different than an environment variable or startup argument. With this, we could treat security as part of our infrastructure. Now, let's look at the architecture that made this possible. One of the first

challenges we had to tackle was IAM permissions. In our old model, we had single-tenant clusters for every application. So, we simply set permissions for the entire cluster through instance profiles. But that doesn't work in a multi-tenant world where different workloads work side by side. We needed something more granular. Otherwise, permissions would be over-scoped. In this example, although application A is the only one that actually needs to

talk to S3, application B also gains access. This This breaks the principle of least privilege. As these permissions are also deployed separately as part of a larger permissions deployment, the risk of problems was fairly high. If someone made a bad permission change, it could go out of to all of production at the same time, causing an incident. To solve this, we adopted pod level This allowed us

to describe much more granular permissions and making them smaller and therefore much easier to just answer the question, why does this application need this permission? Again, following the principle of least Instead of implementing them as part of our big IAM deployment, we moved away from separate deployments entirely and put them right next to the application And in order to do this, we chose Crossplane to manage the

IAM policies with Crossplane composite resource definitions. We were able to describe permissions definition, which turned cloud identity into just another cluster resource, ensuring every workload had a standardized, reproducible, and self-healing identity. Also, this just made our dev loop faster because we were able to eliminate the context switch of working with Kubernetes to working in the AWS console or your IAC tool of choice. Next, we looked at

our network security. Just like with IAM, the model we used for our single-tenant clusters simply didn't translate to a multi-tenant In the previous model, network security was defined at the cluster level. This meant that network security was essentially the union of every workload's permissions. Um that is, if one app needed access to a database, every app in that cluster had it. This was deployed as part of

IAC deployments again to our networking stack and a bad change here had the risk of causing problems everywhere. There was also no protection inside the cluster. The security boundary is entirely on the outside. So, app B can just talk to app A whenever it wants, breaking the principle of least So, again, we focused on pod level Ultimately, we shifted to a deny-by-default stance uh using Cilium network

policies. In this diagram, the Cilium icons represent new guardrails around every single pod. This allowed us to reinforce the as network privileges are now explicit, if app A needs to talk to GitHub, it needs to declare it, or otherwise it won't hear anything from the network. We also get to enforce segmentation. So, in this example, app B can talk to RDS, but app A can't. network policies

are deployed alongside the application, um just like IAM, and networking is just another cluster resource, ensuring each workload had a reproducible network permission scope, and again speeding up development workloads. Let's look at what this means for the application developer. As a reminder, in the past world for changes which affected the permission for example, a new feature which needs to use a DynamoDB table, they needed to they

needed to make at least two PRs. One to add the permission, and then one to use it. Uh they need to get each of them reviewed, and then sequence their deployment correctly to make sure the permission is everywhere before they start using it. In the best case, this would take at least one to two days to do, and sometimes more. Further, these permissions would not necessarily match

across all of our environments, um as there was one set of permissions per environment. developers share the global set of permissions, which could be ahead or behind or far down some subtree. So, they would not even match across our production environments because we had more than one. in their new world, the permission definitions are included right alongside the other definitions that make up the application, such as

the deployment YAML. They're packaged using Helm, deployed using Argo CD as part of our GitOps pipeline. In this world, a user which wants to build a new feature which is uses a new can make the necessary permission in the same PR as they add the code that requested. This both means that it takes them fewer cycles to dis- to deliver code to production. But it also means

that it's super clear why the permission is necessary in the first place. Um there isn't like a separate permission PR that he needs to like backlink to the PR that uses it. And then as the permissions live next to the deployment object, they get configured right alongside the application, wherever you deploy it at any stage of the deployment cycle. And this gives us more confidence in the

changes before they're merged. As we know the behavior during building and testing actually maps to how it will behave in production. Um after merging, the application gets promoted through a staged deployment with beta followed by prod, uh which allows us to catch issues before they hit all of our users. We know that what we deployed in beta is going to happen in prod, it's going to behave

the same way. So there's no question around um oh maybe it'll work um or or vice versa. This allowed us to basically eliminate a class of outages that we had where a developer makes a bad change, for example by dropping a permission by accident or something. Now, something that was both a challenge and a massive opportunity was that our previous permission setup was incompatible with multi-tenancy. So

now we had to rewrite everything. Uh since we had to do that, we leaned fairly heavily on automation. Um In order to do this, uh we leaned on building a good way to capture logs to understand the behaviors of Um Hubble, Cilium's logging tool, provides really great L7 observability where it doesn't just show you there was traffic from IP X to IP Y like AWS flow logs,

but it tells you this pod tries to talk to this DNS name or even deeper than that. We were able to do the same thing for IAM with the with the AWS SDK through something called client-side monitoring uh which allowed us to capture the set of AWS permissions that applications were trying to use. We collect these logs and store them. And once we did that, creating the

permissions is mostly a matter of capturing the usage through roughly a 30-day window and converting that into Cilium network policy and cross-plane I am policy Uh full disclosure, there are some gotchas there which are around generalizing the policies to your but this way we can treat permissions as an iterative process. When a When a service first spins up, we don't expect the developer to know every single

ARN or network flow it will ever touch. Instead, we can use a discovery phase. The developer can provide a baseline permission scope and then they can iterate on it and have tooling generate the least privilege YAML for them. This way we aren't just giving them guardrails and telling them to do their best, but we're giving them a GPS to pave the path for them, which makes the

secure path also the fastest path. Currently, this information is pulled by the developer or operator through running the script, but in the future we hope to instead push this information to the application owners. For example, send the developers a notification, "Hey, your app is trying to talk to this service, but you didn't declare it." And even provide them with a patch that they need to add with

a permission if they want to. Uh this focus on automation also meant by accident baked in auditability. Um by ingesting these same logs into our seam, uh we can verify application behavior in real time. Uh this allowed us to maintain a trust but verify model. However, trust but verify isn't always enough. There are some things that we want to be sure will never happen. Um let's look

at the guardrails that we built in to catch misconfigurations at the gate. I click. Yes, there we go. So, uh we still need to set some guardrails as both as part of defense in depth and in order to protect protect ourselves against ourselves. There is a definitive set of permissions where we would like to ensure a review by the security team. As an example, we definitely do

not want to allow any application to delete a production database. For this reason, we used AWS permission boundaries, uh which allow us to define the maximum permission scope any application can hold. We also continue to use security groups in AWS, but now we just use a single security group which defines the maximum network access any application can have. And we also made sure to shift these guardrails

left and bake in validations against the guardrails to let developers know when they're requesting permissions outside this boundary. And this allowed us to effectively set up a high-velocity playground, which is both protected by a fence around the perimeter as well as little bots on the inside, ensuring that the scope of everything is kept small. Now, we did not just build security features into the applications running on

our platform. We also built multiple security capabilities into the platform itself. Uh we selected AWS Bottlerocket uh as the OS for our nodes to have a minimal OS. Uh we use Istio on top of Cilium to handle all incoming traffic into our clusters. And we also use Istio to restrict egress traffic going out of our So, all outgoing traffic goes through Istio, which then defines the maximum

scope of outbound traffic that any single application can create. This also allows us to audit all incoming and outgoing traffic by our And we also leverage Istio to turn on MTLS by default for traffic within the we also introduced Kyverno for validations. Now, Kyverno is one of our runtime gatekeepers. It's an invisible guardrail that prevents accidental and sometimes malicious security misconfigurations from ever entering the cluster, reducing

our attack surface. However, we saw the biggest benefit coming from having policies that guide our engineers away from writing specs with too many privileges. Now, these specs could originate from something like somebody copying something from the internet or what we're seeing increasingly, the output of an AI agent. Now, we deployed a number of Kyverno cluster policies that would end up as validating webhooks that would reject invalid

specs during runtime. Now, having these invisible guardrails were really beneficial initially. But, however, we saw that the feedback loop for our engineers was too long and required them to deploy their changes in order to get them validated or rejected. And this led us to start using Kyverno's testing features, which allowed us to run these cluster policies on generated Kubernetes YAML. This allowed us to shift security further

left and just run this as a linter on our laptops. We had previously also invested in tooling and linting to catch invalid specs created by our templates. And then when we reach the PR stage, we also validate changes to application configuration against running production But, this like kind of shift-left model gave us an unexpected as we were wrapping up our usage of AI agents. Our cluster policies

are now are now in the context of our AI agents, which are writing Kubernetes resources, meaning that our invisible guardrails are also guiding the AI's development flow to a higher security standard. So, to wrap up, we wanted to go through some lessons that we learned from this Um as always, there were some edge cases. So, we have one KCL application that needs to delete DynamoDB tables. Um

do we want to add that into the boundary and make it possible for everyone? Uh it depends. Um but we realized it was really important to think about how we could be extensible and flexible uh while building these guardrails. Um we found that abstractions didn't always lead to simplicity. Instead, they could hide things from the developer when they actually needed to have a deeper understanding. Uh when

initially using network policies, we played around with uh using label-based policies. So, like you'd have an an RDS policy that and you you'd add the label can talk to RDS uh to the pod. Um but we found that this was too opaque to understand uh what was happening as the developer just like couldn't find the policy. Um and therefore didn't understand how it worked. Um instead, we

have a single clearer network policy associated with each application, uh which is easy to find. It's just right next to the deployment Now, uh our move reduced the centralized work for security teams, but it shifted some to our platform teams and our application engineers. And this meant that we had to invest in education, automation, and clear documentation as our infra engineers uh were had to quickly ramp

up on new concepts. And the lesson we learned here was that we should start having these conversations earlier uh with everyone that would be affected. Sharing your vision and like how you believe it will improve life in the long run is really rewarding, but we would often shy away from having these conversations because we were just like heads down focused on building that vision and kind of

wanted to show up with a fully realized implementation. But it is in these conversations where you get your most valuable insight and feedback because you are not your user. one of the more significant problems we ran into was with the guardrails themselves. As they were now much more invisible, they don't show up normally until you break them. That meaning that everyone just forgets about them until they're

forced to remember them. An example of this was when we introduced the first workload to cells that needed to access a new service, the SQS in this case. The application itself was correctly configured, but we just never added the allow statement into the permission boundary, which led to silent failures and and some debug sessions. And one of our takeaways from this was that we needed to make

sure that the guardrail was permissive enough to not be restrictive, instead just preserving the properties that we truly care about, like not deleting stuff. Uh but the bigger takeaway was that even if these guardrails stay out of your way most of the time, there will definitely be times when we run up against them. And in order to cover that, we both needed to shift left, build tests

that could verify these permissions, but also fail loudly and clearly when when we do. summarize, uh when building platforms, we must treat security as a first-class feature. When we're building platforms, we're oft- we often talk about happy paths which we want our users to follow. The more that we can turn these paths into just happy slides that our users intuitively and happily fall down the better. And

we should not remove security from this happy path design just because it feels scary to do so. Another key part of a successful platform is the clear distinction of responsibility between the users of the platform and the builders of the As an example, we like to say that application teams are responsible for the performance of their applications, while the platform team is responsible for the performance of

the platform as a whole. And this does not need to be any different with security. Making security everyone's responsibility by making the split of responsibilities clear and then empowering everyone to fully take on their responsibility is the key here. Now, there is a cost to building platforms this way and the sooner you start paying that cost the better. Invest early in tooling, automation, observability, and evangelization. Don't

just shift left on tooling, shift left on the conversations between your platform teams, your security teams, and your users. That is it. We have made it to the end. Thank you. Thank you. Now, we have some time. There is a mic over there if you have any questions. Otherwise, uh you can find us here and in the hallways. We look like this. So, happy to uh continue

this conversation. Ooh. Hello. Thanks for the great talk. I have a question regarding how do you manage the cross-plane permissions? Like, if the application B repository is compromised, what prevents the attacker from creating a cross-plane resource and I am role that will give itself access to the S3 bucket of application A? Uh um in fact, very little. Uh we what we have is uh we force cross-plane

to always attach the permission boundary uh to the policy, so it will definitely not be able to give itself permissions to delete RDS instances. Um but uh here we're working with our own trusted workloads and in fact, the all of these charts we have them on the repo, so they're all in the same repository. Um so, yeah. Okay. Okay, thanks. Yes. Thanks for the talk. Uh I

have a question because I can't see that you ship the like the policies, network policies, and given a policies together with the application. But, in the other hand, for example, Cilium has a global network policy that you can create it and the application can use it. What was the reason behind this uh that design that you is coming with the application not the platform team deploy, for

example, global network policy and then Um again, it was basically about magic. you having the global network policy means that uh you need to know about the global um and application developers need to know about it. So, if it was um if it was too wide, people could just like, oh yeah, this works, and then they do something else and that's not in there and then just

doesn't work and now they don't realize they need to think about network Um so, we wanted to we do in fact use some global network policies, um but we keep them extremely small. Um Basically, it's uh for uh for observability agents because they needed to talk across namespaces. I think that's That's the only case I remember. Yeah, and those network policies as well as uh some of

the like global Kibana cluster policies, those are owned by the platform teams. So, they they are the ones that are configuring them. Yeah. Hi there. Um you said the permission boundary prevented deletions of resources in production. So, what do you happen when you actually decommission a resource? Is that a painful cross-team exercise or how does that work? Uh no, so we we do have a lot of

resources that we create along with our clusters and those like are contained like those are kind of like attached to the life cycle of of those clusters. So, we allow deletions of those. It's more that like there are some resources that we you know if they are deleted there's data loss. So we we definitely want to restrict those but the deletions are allowed in in some cases.

And in those cases it would be an operator action. So really the operator goes in and says yeah I allow this deletion and that's all logged and tracked and like who did what and so on. But um it's really that uh we definitely don't want to like allow the user facing application to go in and delete our an RDS instance um ever um uh if it were

to be compromised hopefully it will never be but if it you could basically destroy the company. Hello. Um just a question regarding the the purchases that you are creating do you put them directly in enforce mode or first in audit? And what's the process to go to enforcing them? Um is it in in psyllium or in giverno? Uh giverno. Ah um we yeah we we usually start

with auditing but like for a very short time. Um we we've definitely caused issues by shipping policies that uh we're enforcing but uh we we tend to like test with auditing for a for a short time. This is also some of the benefits of a mono repo that um we kind of introduce the policies and then we go and fix all the things. the platform team goes

and fixes all the um and then they can immediately enforce basically within like a day. Yeah hello. Thanks for the talk. It was very insightful. Uh maybe one question how do you I know safeguard or prevent against you know developers working on multi-tenant just notify your the deny by default the policy. So basically just deploying the policy of allow all. Um, do you want to take Oh,

yeah. Um, in in short we currently do very little. Um, like as we mentioned we have the automation stuff we'll look at this. Um, ask everyone to behave and ultimately run the automation script. Um, and like but in fact if they don't then we will Um, and we will go and check that in. So currently that is human. Um, and that that's largely because um, there there's

a little bit of as I mentioned there were like we can mostly generate the policies from the logs that we get but like there's like there's always this edge case of like what if it didn't do this for the past 30 days but now all of a sudden it does or something. Um, so like while we're working out those kinks we're keeping it that way but uh,

once we're once we trust it basically we're we want to move to I think one >> Yeah, I think one thing I will will add to that is that we um, we do everything through our GitHub pipelines. Everything should be coming from declarative configuration and then we audit all human actions. So like nobody should be editing anything in production using CubeCut for example. We're out of time.

That was the beeper over here. Thank you so much everyone. Have a great con.