How Google Built a Consistent, Global Authorization System; and You Can Too! - Sohan Maheshwar

This talk explores how Google developed its global authorization system known as Zanzibar, which enhances access control management at scale. Soha Maheshwar explains the limitations of traditional methods like access control lists and role-based access control, leading to the adoption of relationship-based access control (ReBAC). The speaker elaborates on how ReBAC utilizes relationships between users and objects for permission checks, making the system more efficient and adaptable. Soha provides insights into the underlying principles of Zanzibar, including how it handles high throughput of authorization queries and the importance of fine-grained access control in modern applications, including those using AI. Additionally, solutions like SpiceDB, an open-source implementation of Zanzibar, are introduced for developers looking to implement similar systems.

Hey there, welcome to this talk on how Google built a consistent global authorization system with Zanzibar and how you can do. My name is Soha Maheshwar. I'm based in the Netherlands and I work for a startup called AuthZ where I do developer relations. I've been in the industry for more than 16 years now in small startups but also in large companies like AWS and Amazon. Uh and

I've really spent all this time in the cloud. So, I hope to teach you something about how systems work at scale today. I'm going to ask you a question and uh the younger folks watching this probably will not know what I'm referring to but um maybe the older folks or people my age might remember. But, does anyone remember this website or this platform from the early 2010s?

I'm going to give you a second to take a look. Again, the Gen Z folk in the audience might not know what this is but this was actually a social networking site by Google in the early 2010s. It was officially called Google Plus but informally called Google Circles. And it was different from some of the other social networks back in the day like Facebook and Orkut, if

you remember that. Instead of everyone being able to see everything, you could actually put people in circles like university friends and family and share things with these different circles. The social network site doesn't exist today at least in the form that we saw it back in the day but the story goes that Google actually changed their internal authorization system based off of how this was built. I'm

sure that each of us have encountered this at some point of time or you've used this internal authorization system and I'll give you an example. If you've ever linked a Google Docs file in Gmail, so on the left say you're sending a Gmail email and you're sending a link across, sometimes you get this pop-up saying someone needs access to this document. And if you think about it,

this is kind of like magic cuz you have Gmail and you have Docs and these are two separate websites or apps and there is a centralized authorization check that's happening in between the two. And how is that possible and this quickly? Well, that's what I'm here to tell you about today. Just a quick primer on the word authorization. In our industry, we conflate the words auth with

both authentication and Authentication is identity, who you are. So, if you've ever used LDAP or Okta, then you know that that is authentication. You know, verifying who uh what your identity is. Authorization is permissions, what comes after that. So, we know who you are, but what do you have access to? And that is authorization and in the past, we've done it in the form of access control

lists. So, imagine a really large list with all the people who can access document one. Now, a different list with all the people who can write to document one. Same thing for documents two, document three and so on. You can see that it's not very scalable. This changed a little bit much later on to something called role-based access control or RBAC and you might use this on

a day-to-day basis at your work. If you are tagged to a role like software engineer, then all software engineers can check in code. If you're tagged to say sales, then all the people can access sales reports. It's it's a widely used system, but here's the thing at scale when, you know, you have lots of different uh employee types and lots of different roles, it doesn't necessarily scale

because what if you as a software engineer have to check a sales report? It's about creating a new role and you're doing this and the permissions become what we call coarse-grained. Uh it it's not very fine-grained, it's very broad. So, the industry changed a little to have something called uh attribute-based access control. And in attribute-based access control or ABAC, everything is broken down into an attribute. So,

say you want to access a system, the system checks all your attributes at that point of time. So, name, email ID, location, IP address, role, department, etc. And at that decision point, if all is good, you get a yes or you get a no. Again, attribute-based access control is widely used, but at scale it can become complicated. And these methods didn't necessarily work for the scale of

Google. We're talking about um millions, tens of hundreds of millions of users, billions of documents, photos, maps, YouTube videos, etc. and spread across the world like literally every country in the world. So, essentially, they popularized a new method of access control called relationship-based access control or ReBAC. And we know all of this because in 2019, yeah, it's fairly new, in 2019, they released a 14-page white paper

called Google Zanzibar. And this describes exactly how their internal authorization system works. And that internal authorization authorization system is also called Google Zanzibar. And again, I'll save you the trouble of reading the 14 pages by explaining how it actually works. The cool thing about this is even back the white paper describes how they could handle up to 10 or more than 10 million client queries per second,

and I'm positive that that number has increased since then. And from a developer's perspective, well, it's just an API. So, if you're a developer in Google implementing the system, you don't have to worry about how this works. You just call an API to check for permissions. And like I mentioned, they popularized this concept of ReBAC or relationship-based access control. What does that mean? Well, instead of breaking

down everything into a role or an attribute, this basically represents a system in the form of a relationship between one thing and another. Here's an example. So, say you have a document, and that document exists in a folder. There is a relationship between the document and the folder, the fact that it exist there. Say that folder, um there are people who have access to that folder, and

hence there's a relationship between the person and the folder, in this case viewer. And this implies there is a relationship and the document. So, everything in the system is represented by relationships. What are the things that can be represented by relationships? Well, one, according to Zanzibar, has to be a user. So, if you have a Google ID, well, you have something called a Gaia user ID. So,

one of the objects has to be that. The other thing could be any object that the Google system has. Now, this could be a folder, a document, a video, a map ID. Uh if Google releases a new product tomorrow, something there, it could be anything. Essentially, and there has to be a relation between that user and the If you use Google Groups, you could be a member

of a group or you could be an editor of a document or an uploader of a video and those are the relationships So, you have users, you have objects and you have the relationship between them. How do you tell a system like a computer back-end system that these relationships exist? Well, the Zanzibar paper describes this thing as a relation tuple. And the idea is if you add

something in this format and add it to the system, you tell the system that there's a relationship So, start on the right of the hashtag and say user number three is the owner of document 123. So, if you've ever created a Google Doc, something like this has been written in their internal system where they're like user you is the owner of document XYZ. And this relation tuple

format is very, very flexible because objects can relate to other objects, but objects can relate to other objects relations as well. So, in this case, group security is a member of group engineering. Now, think back to a time when there was a reorganization at your work and say group security was reorged to group DevOps. That would typically involve writing new access control lists, creating new internal roles

for our back, etc. But, in a reback system, it's just about writing a new tuple. You would just write this tuple saying group security is now a member of group DevOps and everything would fall into place. And how would that happen? Well, what are we doing with all of this data? We have users, objects, and relations, The idea, and this is the key, is to create a

directed acyclic graph with all this data. You're essentially taking this data and breaking it down into a graph traversal problem. I'm going to give you the simple example of a Google Docs that we will use throughout. So, you have some document and you have a reader of this document that is Fred and Kim. And you have someone who can write to this document, which is Jill. The

key thing here, and notice this little dotted arrow, this indicates that anyone who can write the document can also read the document. Now, say your your requirements change and you're like, "Hey, you know what? I want to be able to add an organization to add access control to this." Instead of rewriting your entire code, it's just about adding a new type of relationship. So, you add an

organization, let's call it Acme Inc., and we type we add a new relationship, which is an admin of an org, and let's call Liv an admin. Again, no notice the little tiny dotted arrow between admin and writer, and this implies that anyone who's an admin can write to this document. And we already know that anyone who can write to the document can read from that So, how

does this work behind the scenes? Well, if you've ever clicked on a Google Docs link, this is actually what's happening behind the scene. So, a permission check comes in to check if can Jill read some document. The idea here is if you can traverse this graph in a unidirectional way from some document to Jill via the reader permission, that means yes, Jill has access. And that is

the fundamental principle behind any permission check in any Google's Zanzibar system. So, let's see if that works here. And the answer is yes, because you can go from some document to reader to writer to Jill. You see, there's a unidirectional way to go from some document to Jill via reader. So, the system replies saying yes, Jill does have permission to read What if there's a permission check

to see if Fred can write to the document? Let's see if there's a unidirectional way to go from Fred to writer to some document. So, we go some document writer and nope, we can't go to Fred unidirectionally. So, Fred does not have access to write to And essentially, you're breaking this down into a simple graph traversal All right. So, we saw that example. Now, this paper was

written in or released rather in 2019, and this made shockwaves in the IAM world or identity and access management world because everyone so far were using very different systems. And suddenly, they're like, "Wow, if this can work at the scale of Google, it must be really good." So, at the same time, uh people and companies started building out open-source versions of that white paper. And the four

that are super popular right now are the ones you see on screen. There's SpiceDB, there's Ory Keto, there's OpenFGA, and Permify. And each of them have different, let's say, opinions on things in the paper and have different architectural approaches. I will actually show you a demo using SpiceDB because um I work in a startup that helps maintain SpiceDB, and I've also contributed towards SpiceDB uh code base.

And essentially, it started off as a simple open-source project, but now it has contributions from Netflix, Red Hat, Git uh Gitpod, GitHub, Google, and a lot more. Again, completely open-source. You can see the link there, github.com/authzed/spicedb. And of course, the one metric that matters to us programmers when it comes to evaluating any software is, of GitHub stars. That was a joke. But, uh yeah, the SpiceDB project

has about 6,500 GitHub stars and counting. If you're in the mood, please do give us an extra star. We We always appreciate it. So, what is SpiceDB? Well, it is an open-source implementation of the Google Zanzibar project. Even though the name has DB in it, it's not a general-purpose database, right? So, don't use it as uh RDBMS or a SQL database. Essentially, it is a highly parallel

graph database meant only for authorization queries to do things like permission checks. And what you see on the right is a essentially how it works. So, you have your apps, your microservices, your data warehouse, and the idea is to centralize your authorization. The sort of pattern that is rampant in the industry is writing your own authorization code, which is then very tightly coupled to your app or

to your microservice, and it gets hard at scale. It You have to suddenly maintain it in different client libraries. And, you know, the team that, you know, built that, if they move to a different project, then suddenly you have all this code. So, the idea really that Zanzibar popularized is this centralization of your authorization service. And SpiceDB is a combination of these relationships that I spoke about,

all this data, uh something called schema, which I will discuss in a bit, which is essentially the modeling of your users and objects, and of course, the graph engine, which creates this unidirectional graph. With SpiceDB, you can basically answer three types of questions via API. The first one is can a subject take action on a resource? Like for instance, I'm Sohan, so can Sohan edit document one?

There are also couple of lookups that you can do. For instance, list all the people that can edit document one or the reverse lookup as well. List all the documents that Sohan can edit or any subject can edit. Got to hydrate while giving this talk. Now, I'm going to show you a quick demo just so that you can visualize what this looks like. Essentially, oops. Going back.

Essentially, this is the SpiceDB playground. Again, it's this tool is open source as well. And I'll show you how you can think about building permissions into an app. So, it's about the schema that I spoke about is defining the different objects in your system. Very simple example, you have a user and a document, but this could be anything. And if you click on this, you will see

examples of how you would model authorization systems of like GitHub or Google or whatever. We are talking about relationship-based access control, so you have to define the relation between the objects. So, user can be a writer or user can be a And for each of these relations, you can define permissions. So, all only writers can edit, whereas the people who can view are readers plus the ones

that can edit a file. So, in this case, that would be a writer. And I'll show you how this would look to visualize this. So, this is a system visualization. And typically, you would write permissions to do this. So, here we have written some permissions about users Tom and Fred. So, Tom is a writer of first document, Fred is a reader of first document, and Tom is

a reader of second doc, and this is the graph that you can visualize. So, if you had to make a permission check, you can actually do something like this where you say, let's look at document first who can edit the document? Let's see if Tom, yes. Tom is a writer of first doc, and hence Tom can edit first document. But, let's look at Fred. Fred is only

a reader of first document. That's a relationship we have written. So, you get the answer, no, because we don't have any writer of first document, or at least Fred is not a writer of first document. And you can use a CLI tool to do the same thing as well. I say something like permission um document first doc, view user Tom. Right? And you get true. So, this

is how you would do it. Now, typically, this permission check would be an API that you would write in your application. And again, when you've done something like this in any Google service, something like a permission check like this has actually been written in the back end that, you know, has taken part in or taken place in the Zanzibar system. Now, the question I get asked is,

okay, this works, graphs, read back, etc. How does this work at the scale of Google? You know, we're talking about low latency and billions of objects and users, etc. The performance strategies itself could be an entirely new talk, but I'll try and condense that in the next 5 to 7 minutes. But, there are two main things that you need to remember. The first first one is everything

is broken down into a parallel subproblem. And the second is it tries and reuses the cache as much my much as possible. So, aggressively breaking down a problem into sub problems and aggressively using the cache. So, let's see how that works. Okay. So, I'm going for the same example again. Google Docs, Jill, can Jill read If you think about this, this is a combination of three sub

problems. One is Jill a direct reader of some document? The answer is no. Jill isn't in one of these two names. The second problem, is Jill a writer? Because we know writers can also read the document and the answer is yes. that's good. And the third problem is is Jill an admin of the organization? Because we know that admins can also read. So, this particular problem is

a union of three problems. In the system, first it checks if Jill is a reader. The answer is no. And if Jill is a writer, the answer is yes. And any subsequent checks are then not done because we got the answer already. Here's the key though. We have already computed in this first sub problem that Fred is a reader and Kim is a reader as well. And

that is the aggressive breaking down of sub problems. Because say immediately after this, a question comes, is Fred a reader of some document? You know that answer already because you calculated this in the previous run. So, you don't have to traverse this graph again. So, that particular answer is cached, which is then accessed if a check comes to see if Fred or Kim are readers, and we

know that answer. Here's a question I get asked when I say this. They're like, "Okay, okay. That's good. But, what if someone removes Fred's permission in between the time that this check came in and say, you know, some time has passed. Like, what happens then? If you thought about that, well, congratulations. That's very sharp of you. And this is actually defined in the paper as something called

the new enemy Let's see what it is. So, you have time T1 where Kara removes Lex from something called secret plans. All right? So, you're like, "I don't trust Lex and I don't trust Lex to read my secret plans." And that's when Kara adds new secrets to Now, Lex attempts to access this even though they shouldn't be able to access it. But, based on what I said,

Lex can access this document even though they're not supposed to. And then, you know, total chaos because Lex is then reading the secret plans that they're not supposed to access. This Zanzibar paper calls this the new enemy problem. And to solve this, they have a very interesting approach. And this is key to authorization. And they describe something called a Zookie, which presumably stands for a Zanzibar cookie.

We don't know. They don't say that in the paper for sure. Essentially, this Zookie, think of it as a token that represents a specific point of time, like a timestamp. So, anytime any change is made, like say a new document is added or someone is removed from permissions or someone is added to permissions, this Zookie is added along with that call. So, this means that when Kara

is removing Lex's reader from permissions, there is a Zookie added at time T1. At time two, when Kara adds new secrets, a Zookie is added. And in time three, when Lex attempts to read this, that Zookie can be compared and saying, "Okay, this comparison came after T1 and T2, and hence Lex does not have access. And the cool thing is in the API you can actually specify

when you want to access data. So, say you don't care too much about the new enemy problem, you want something fast, like super fast, you can try and get it from the cache as much as much as On the other hand, for you security matters the most. You're like, it's okay if my request is maybe a couple of milliseconds late, but I want fresh data. I do

not want stale data from the cache. I want fresh data always. You can specify that in the API as well, and you can get whether you want fully consistent or a specific point of time, and all of them using this concept of a Zookie. Cool. So, that is some of the sub problems in caching bit. The paper, the white paper, also described something called hotspots, as the

name suggests, where lots of people are trying to access a particular access control list or a permission check. And from the paper they say, "We found the handling of hotspots to be the most critical frontier in our pursuit of low latency and high of high availability." There are four things that Zanzibar does to ensure this handling of hotspots, and I'll distill it. is a distributed cache. So,

Zanzibar is not running on like one database, or the cache is not running in like this one uh one, you know, um data store. Essentially, the cache is distributed across different nodes in a highly distributed fashion. And the results of previous reads and access checks are stored distributed in these caches. Remember I mentioned the It uses a system called consistent hashing where SpiceDB or Zanzibar itself knows

where the results of previous access control checks are stored. So, say a check comes in for something and it's in the cache, the system automatically knows which cache to hit to get that the fastest. And there are internal optimizations there to ensure this happens. At the same time, there are timestamp optimizations as well. Remember I told you about Zookies? Now, at every read and write check, if

a Zookie is created and you're using say UTC, this can create lots of Zookies and it can actually cause more confusion and more latency. So, what happens is instead of like assuming time moves this way and at every like milli nanosecond or whatever you're creating a Zookie, they've actually created quantization windows of in which you can create Zookies. And that way many requests actually share the same

cache key. And you can batch requests to this different cache, right? Making it a lot more efficient. At the same time, there is also RPC optimization. So, each of these calls, when it comes to a permission check or a lookup check, all of these are RPCs or remote procedural calls. What the Zanzibar system does is it basically does optimization on the caller and the callee side. And

internally at Google, they use this load distribution tool called slicer, which they it basically knows object IDs that are being accessed frequently. So, it uses something called a forwarding key to make this access a lot faster. And lastly, uh there's also use of a lock table. Now, a lock table is basically an in-memory structure to track progress for a particular cache key. Now, you might have heard

of something called the cache stampede problem. So, assume there is a permission check for one document and say 100 people access try and access it at the same time. That is like a stampede. Imagine 100 people trying to get through a door at once. So, what this lock table does is it allows one person in or one access check to traverse a graph, store all of that

um like a cache and then that result is faster for the other 99. Now, this avoids the classic cache stampede problem. It might take a little more memory to do this because you you know, you're getting one call to actually go through all of these permission checks, but it's faster for the other 99 and that's where the lock tables actually come in. And these are for hotspots,

but sometimes you have really hotspots, you know? I'm talking about a video going viral or say a document is shared with an entire organization of 100,000 people. What do you do then? Now, two things happen. One, remember earlier when I said that these subchecks are cancelled once you get the correct answer, once you know that, oh, Jill can read a doc. Well, that is actually delayed in

a really hotspot. super cool, but the Zanzibar system identifies that, okay, this is a hotspot. So, they delay the cancellation of the subcheck and go through the entire graph because that way you can store all of this in the cache for so many other people who are accessing it. And at the same time, you're doing a full read and cache for all these hot objects because subsequent

people will get it will get the results faster. It does use more memory and bandwidth, but it is low latency for everyone else and again, this is a really hotspot. And this is cool because in the paper it mentions that sure the these rates might appear low, but 500,000 RPCs per second are prevented because of this design. Imagine that. 500,000 calls per second are actually, you know,

prevented because of this ingenious And turns out Zanzibar shines when you need low latency, when you have a very high throughput of authorization checks, when you need global consistency of relationship data. And surprisingly, we are actually seeing a lot of the AI use cases fit this. So, if you've ever heard of Rack Pipelines or Agentic AI, now these need low latency because a Rack Pipeline's looking at

a million documents or an AI agent or hundreds of AI agents are doing all these different tasks and returning data, you're having a high throughput of checks, permission checks by each of these agents, and you want these answers really quick. So, any Zanzibar model like SpiceDB is actually ideally suited for any AI authorization that you might be writing. So much so that in production right now, ChatGPT

actually uses SpiceDB and Authzed infrastructure. So, if you've ever used ChatGPT apps, it used to be called ChatGPT connectors, right? Where click on that little plus button and you can connect Google Drive, OneDrive, SharePoint, etc. And then you can query questions from the documents that exist in your or OneDrive. That actually uses SpiceDB and Authzed for its fine-grained permission. And look at the scale itself. 37 billion

plus documents, and this was many months ago, so I'm sure that number's increased, across 5 billion users, right? So, that's the scale that we're actually, you know, looking at when you can use a Zanzibar-like system. So, if you have any requirements that are low latency, high throughput, high number of authorization checks, and possibly an AI, then look at uh building a Zanzibar system or using any open

source Zanzibar systems out there. I'm telling you all this because it's super important. You might have heard of OWASP or Open Web Application Security Project. Every 4 years, they release this thing called the top 10 most critical security risks to web apps. The last two lists, including the one that came out in November last year, number one is broken access control. It's such a problem in the

industry right now. Reading that report is scary because the report says 100% of the apps, 100% of the apps tested had broken access control issues. it's more important than ever to have access control fine-grained in your system. And hence, something like uh Google Zanzibar-like system like SpiceDB is vital, especially in enterprise and AI use cases. For next steps, I'd say just read the Zanzibar paper. You There's

an annotated version linked there. It's It's pretty cool. But, check out SpiceDB. It's completely open source. You can self-host it and run it in production. I know many big companies that do so. And if you liked this video, we have some videos on our YouTube where I talk about some of the technical aspects and also some of the cool industry-changing aspects with AI and And if you

want today's presentation, yeah, just download it there. Or if you have any questions or if you enjoyed this or just want to say hi, hit me up on LinkedIn. I reply to all messages or any any form of engagement on LinkedIn. And it was my pleasure, and I hope you enjoy the rest of your conference and keep learning. See you soon. Thanks a lot. Thanks a lot,

and goodbye. >> [music]