About this talk
This talk delves into Google's authorization system known as Zanzibar, which is responsible for handling permissions across various services like Gmail and Google Docs. The speaker explains the evolution of access control methods, starting from access control lists to role-based access control and finally to the more sophisticated relationship-based access control that Google developed. He highlights how this system allows for efficient permission checks through the use of directed acyclic graphs, enabling complex relationships between users and resources. The speaker also discusses scalability challenges and solutions employed by Zanzibar, such as distributed caching and handling data consistency. Overall, the session emphasizes the importance of a robust authorization framework in modern applications and encourages developers to explore the open-source implementation, SpiceDB.
Full transcript
Challenge accepted. [music] Challenge Challenge accepted. Challenge [music] accepted. Challenge accepted. P2 is always [music] challenge accepted. All right, folks. It's time to zoom into security at Google scale. You've probably heard of YouTube, Google Drive, Google Clouds, Google. All of them rely on a single authorization system. >> And no, it's not your mother checking if you ate your cucumber. It's Zanzi Bar, the globally distributed authorization system that
keeps billions of requests consistent and secure. Our next speaker has traveled here from the land of canals, cheese, and windmills, Amsterdam. >> He is a lead developer advocate at Oz and has spoken at over 100 international conferences from AWS reinvent to web summit. >> When he's not debugging tricky system, he's probably winning at pub quizzes or throwing frisbes like a world champion. >> Please welcome San Maheshan.
[screaming] [groaning] that was the most hype introduction I've ever received. So, thank you. All right, so as they mentioned, my name is Sohan. I've worked in the industry for 16 years now, which makes me an old fart, as they say. But I'm so excited to talk to you about authorization and about specifically how Google built their authorization system and how you can too. I have a question
to ask you all. It looks like a very young audience so you may not remember this but the slightly older people in the audience like me might know but does anyone remember this tool from around 20101? It was a social few of y'all. Yes. Do you remember what it was called? Yes, you can shout it out. Anyone? >> Your friends. >> Yeah. So, for the young people
in the audience, this was called Plus. And this was Google's attempt at creating a social network. Now, we don't use this anymore, of course. And the idea was unlike Facebook or Orchet or MySpace back in the day where everyone could see everything here you could share things with certain circles. So I can have a group for university friends or I can have a group for family and
share things with certain groups. As a social network, it didn't do very well. But the story goes that they changed how their authorization worked internally at Google to build this. In fact, who here uses Gmail or Google Docs? Everyone? Yeah, of course. You know, uh if you have ever used or if you've ever sent a Google Docs link in Gmail, you'll see this popup appear saying, "Someone
needs access to this thing. Can you give them access?" And if you think about this, this is actually magic because Gmail and Google Docs are two separate apps. But yet there is a permission check that's happening in between. So how this works is exactly what I'm here to explain to you about. I'm going to give you a bit of a history lesson here on a primer on
authorization. In the industry right now when we say the word authorization and authentication which are two very different things. Authentication is identity who you are. So if you have ever logged in with LDAP or octa that is authentication. Authorization is what comes after that. It's permissions. So once we know your identity what can you access? What do you have permissions for? And in the past to do
this, we have used things like access control lists. Imagine a long list of all the people who can read a document. Now you have another list for all the people who can edit that document. It's not very scalable. How many lists can you maintain? So after that we had something called rolebased access control or arbback. Now if you work in a company you might be familiar with
this. You have a role tied to say software engineer and all software engineers can check in code. You have a role that says sales and all sales people can access financial reports. But what if someone from sales wants to check in code? So you have to create a new role. Doesn't happen very often but it could. And again with role-based access control it's not very scalable at
complex use cases. So the industry came up with this thing called attribute-based access control. Here everything is an attribute where you have a bunch of characteristics. So if I'm trying to log into a system, it checks my IP address, my location, my email ID, maybe even my role. And based on all of this, it says yes or no. Now both these systems work. They work well. But
when you want fine grain access control, it starts getting a little difficult. And Google being at Google scale with billions of users and hundreds of billions of objects needed something different. So they popularized this thing called relationshipbased access control in their internal And we know exactly how their authorization works because they wrote all of it in a white paper that you and I can read. In fact,
I recommend you read it. It's about 14 pages, very interesting where they talk about how they built this authorization system. I've read this paper many times and I'm just going to talk to you through this white paper. In this paper which was released in 2019, so not many years ago, they say that they can do 10 million client queries per second, which is an insane number. And
I'm sure that number has increased by now. If you're a developer like me, it's just an API. And I'll show you a demo of how that looks. And like I said, it popularized this thing called rebback or relationshipbased access How does that look? The idea is there is a relationship between different things in your system. So instead of force-fitting a role or an attribute to something, you
describe the relationship between things. So for example, say you my bad. Say you have a document called steal the moon. That's here. And there is a folder and there is a relationship between the document and the folder. Similarly, there's a user here and the U user can view the folder. So there's a relationship between the user and the folder and hence there's a So what can there
be relationships in between? Well, one has to be a user. So if you have a Google account, you have something called a GIA ID, which is a 12digit ID. And in this system, you will have relationships to things like documents, Excel sheets, photos, YouTube videos, maps, any object. And the relation could be anything. It could be a member of a group. It could be an editor of
a document or an uploader of a video. And the cool thing is say tomorrow Google they launch a new product. It's about creating a new object type and a new type of relation. So you have a user, you have an object, you have a relation between the two. How do you tell a system that this relationship exists? The paper describes this thing called a relation tuple which
is a format to describe a relation to a system. So how you read this is start h again start at the hashtag and start at what's at the right. So in this case, user number three is the owner of document 123. And this format is very flexible because objects can also relate to other objects relations. For example, in this case, the group security is a member of
the group engineering. Think of a time a reorganization has happened at your workplace. It happens often. If a [snorts] reorganization happens, you have to change permissions, email servers, who owns what. It's a cumbersome process. But in a case like this, it's about just writing a new relation. You would say something like group security is now a member of group DevOps. And that's it. Everything falls into place.
What are we doing here? You know, we have these relations. What are we doing with this data? The idea is to build a graph where your objects and your users and your are different nodes in the graph. And this is a directed ascyclic graph which means that the direction of things matters. I'm going to take the example of Google Docs. So in this case you have some
document and you have a reader Fred and Kim. They're readers of the document. You have a writer named Jill. The key thing in this graph is this little dotted arrow here in between which indicates that all the people who can write to the document can also read and this gives a lot of flexibility for a company like Google because say there's a new requirement saying add an
to this doc you know add the ability to add an organization it's just a new node on the graph so this node would organization where it's called ACME. You have an admin of the org and you can say that all admins are writers and we know all writers are readers. So you're drawing this graph. Now when you try and access any Google doc and say you're logged
in, what is actually happening behind the scenes? How is that permission checked? Let's say the example is can Jill read some The idea is if there is a unidirectional way to go from some document to Jill via the reader relationship that means Jill has permissions. So in this case yes we can go. So you can start at my document you can go to reader writer and Jill.
So there is a unidirectional way of going through from the document to the user. So yes the permission is there. Let's take the example of Fred. Maybe Fred wants to edit this. Of course there is a graph for a permission check. It's open source again and it's not a general purpose database. So if you're thinking I can use it as an RDBMS like Postgress. It's not that.
It's meant only for authorization queries and it was written in go and fundamentally you can do three things with this right. So you can say something like can a subject take an action on a so view this document or can this person edit this code repository. It doesn't have to it could be any use case. You can also list all subjects that can take an action on
a resource. So give me all the people who can view this document and you can also do a reverse lookup. So say give all the documents that a person can view. I'm going to show you a quick demo of how that would look. So this is a playground environment where there is a spice DB binary running in the browser. So you start by describing your schema. you
know which are the different users and objects. I'm keeping it simple. You have a user here. You have an object here. And you define the relation between the user and the object. A user can be a writer or a reader. For each of these relations, you can describe a permission. So writers can edit the document, but readers and writers can view the document. Now, typically you will
be doing this in code, but this is just a demo. And once you have a schema, you would write relations. So say I create a new document and I'll call it new doc and I need writer permissions on that. So I will say user so on. So there will be an API call to write this new relationship. And then you can actually test out if someone has
permission to access a doc or not. So I'm using a CLI tool but you can do this via code as well. I will say um something like this right uh Z permission check does user Fred have access to first doc. We can see that Fred is a reader of this document but cannot write to it. So that says it says false. Fred does not have edit permissions
but Fred will have view permissions and that is true. So again this is what would happen in the back end and just to give you a visualization this is what the graph would look like. So again if you have ever used a Google service in their back end is something that looks very similar to this. Now the question that I actually get asked fairly often is does
this scale? You know this we need this to scale at the level of Google. What are some of the techniques used to scale? I can bring it down to two things. One is everything is broken down into a sub problem and the system tries to use the cache as much as possible. Now going into the details of this is another talk itself but I'll try and explain
it at a high level. Going back to our example you know can Jill right here read this document. If you look at it this is a union of three problems. The first one is is Jill's name here? No. So that's no. Is Jill's name here? The answer is yes. So that is a yes. And we have the third problem which is here. How the Zanzibar system will
work in the back end is the first check is made and it's like no. And the second check is made the answer is yes. Result is returned to the back end. The back end does not bother checking this third check because it's already got the answer. But interestingly is it knows the answer to this already because it has made this computation. So that is stored in the
cache. Remember this cuz I'm going to come back to it shortly. A question I get asked is okay you're caching all of this but what if there is stale data? What if a permission changes? You know maybe I'm removed from a document. That is a great question and that's described in the white paper as the new enemy problem. So at T1 basically K here has removed Lex
who's an evil person from this document and then K has made and changed this document added new secrets which Lex cannot access because he's not supposed to access it anymore but because it's used from the cache he actually attempts to read this secret plan and he gets access and there's chaos. So how do you solve for something like this? The Zanzibar paper describes this as the new
enemy problem and they solve it with something they call a zuki which we think stands for Zanzibar cookies. It doesn't mention but anytime there is a change that's made a opaque timestamp is attached to the call. So every time a document is changed or created or deleted any new relationship there is a time stamp that's created which is attached to the back end. So if I say
I want data that is fresh as of now the system will bypass the cache and get me the latest data. If I say no no I want a answer that is fast but I don't care about correctness. It will get me something from the cache. And this is how it also gives Google flexibility because in a system like Google there are what is called hotspots where many
people are accessing a document and in the paper they mention we found the handling of hotspots to be the most critical frontier for low latency. So how do they handle that? There are four techniques right and I'm going to be a little technical here. The first one is a distributed cache. If you're in the cloud native space, you might know how Kubernetes works. So imagine something similar
where you have different nodes, each of which has a Zanzibar system running and each of them have a distributed cache. So they each know which ones have solved which problems. So when a problem comes in, they're like, "Okay, that node has solved this already. We will route it there." And they get an answer back fast. And this is done through a system called consistent hashing. The second
one I said before that a time stamp or a zooi is created every time a change is made but they very smartly round up the time stamp. There is a window at which time stamps are created so that the back end is not overflowing with different timestamps. Three, all remote procedural calls or RPCs are internally optimized both when a call is being made and when a call
is being answered. There is cache at both ends. So you know we you have much lower latency [snorts] and Google has an internal tool called slicer which they use for load distribution. And lastly there is something called a lock table. Now imagine 50 people try and access a permission at once. Instead of all 50 traversing a graph, they create a lock table where only one can access
it. The result is cached and the other 49 use it from the cache. Now this might take a little more memory and bandwidth, but it's faster for the other 49 people. And what about the really hot spots? Say a video goes viral or a document is shared with an entire organization. Remember when I told you that the sub checks are cancelled when an answer is received. Google
detects that when it's a really hot spot, they actually go through the full read. So they would rather read all of the different subpros and cache it so that people can access it faster because they know that this is a hotspot. And sure this again uses a little more memory and bandwidth but the subsequent checks for all the other people accessing it is much faster and from
the white paper itself this saves the Google backend about 500,000 internal RPCs and that is a huge number right so all of these different techniques help it scale to the level of and we've seen in the last few years that if you want to build permissions or authorization and you need low latency because authorization is on the critical path, you have a high throughput of authorization checks,
you need global levels of consistency and you need all these different permission models, then a system like Zanzibar really shines. In fact, how many people here use chat GPT? Yeah, everyone. Yes, of course. Uh if you have noticed chatgpt has a new feature called chat GPT connectors right so if you click on that little plus button you can connect your Google drive one drive sharepoint Dropbox etc
and it would read from those documents and you can query this and this is big in the enterprise and open AI actually uses spice DB so this is a complex problem because you need to know the authorization system of Google drive and match the user with the authorization system of Google Drive, you need to do that with One Drive, SharePoint, Dropbox and so on. And maintaining code
for each of this is painful. But if you have any centralized authorization that can model different authorization models, well then that's good for you. And OpenAI actually uses Spice DB to process 37 billion plus documents. So that is a fairly large number. So Zanzibar really works at that kind of scale. So why am I telling you all of this? You know what is the relevance of all
of this? If you work in the industry, you might have heard of this foundation called OASP or Open Worldwide Application Security Project. Now this is a foundation that looks at risks to web apps and LLMs and they released this top 10 list every three or four years where they tell you that these are the big risks in security to web apps and the last list was released
few years ago and the number one on that list is broken access control. Just open up any tech news and just type data breach and you will see so many articles of millions of documents or pieces of data that has been breached because access control is complicated. Permissions are hard as our systems get more and complicated. And a way to sort of fix that is to not
write your own permissions and authorization but think about using you know a third party or an industry standard. So a system like Zanzibar might actually be useful for Well, for next steps, I would say just read the Zanzibar paper. It's a very interesting read and written in English, not very academic. If you're into coding, you know, try out Spice DB. It's completely open source. You can feel
free to contribute as well. And yeah, I have few videos up on the YouTube if you're interested. If you want today's presentation, feel free to click on that uh QR code and you'll get it emailed to you. Um I also have few copies of the Zanzibar paper to hand out. So find me after the talk somewhere out there and I you can read on the way back
home. I hope you learned something new today and enjoy the rest of the conference. Thanks for having me. you. Thank you, Zan. Any questions? [applause] Let's see your hands raised if you have a question or you can also meet Zohan in SAN in the next break in the speakers corner. No questions. So, let's send him off the stage with big round of applause. So,
