About this talk
In this talk, Alex Livier, co-founder of Serbos and co-chair of the OpenID Orzen working group, discusses the complexities and challenges of authorization management in modern distributed systems. He explores common scenarios where authorization failures occur, such as inconsistent responses from multiple backend services and the fragmentation of authorization logic across systems. Alex emphasizes the importance of maintaining a unified authorization model to prevent cascading failures and security breaches. He proposes externalizing authorization logic and adhering to the OpenID Orzen standard to create a centralized decision point for managing permissions consistently. This approach can lead to more predictable and reliable authorization behavior across applications, making it easier to audit and manage permissions effectively.
Full transcript
Round of applause for Alex. >> Good. Cool. There we go. Good morning everyone. Nice to meet you. I am Alex Livier. I am the co-founder of Serbos which is an authorization management platform and I'm also co-chair of the open id orzen working group which is the standard went final in January this year uh for how you do authorization between uh a decision point and an enforcement point
which is going to be a key part of what we're talking about today. Uh seros is an open source project as well so I'm not here necessarily to sell anything either. So today we're going to talk about the situation where something has happened in your environment. you know, it's passed off and then things have got a bit weird and the realities of what authorization failures look like.
So, I'm sure many of us have been here. I certainly have in previous lives and previous roles. Um, you it's 2 a.m. in the morning, you're on call, you get that page of duty alarm or that off Genie alarm, and you now going to figure out what on earth's going on. The alarm goes off, you scramble for your laptop to try and find where it is. Um
you look at your metrics, you see the alert and the checkout error rate is climbing. Um you know the latency on the order processing service has gone from uh you know half a second to 10 seconds. Your customer bounce rate is massively climbing because nothing's really behaving. So you sort of find the machine you SSH into the node or uh exec into the cluster and the order
service you can see in the logs is starting to make um six different backend calls to process the transaction. You know that's not unusual. You might be filing the actual transaction, claiming stock, doing an uh anti-laundering check, etc. But half of those uh downstream requests are coming back 403. The other half are coming back 200, which is strange because you got the same user, the same identity
making the same request, but you're getting this divergent answer of like actually was this allowed or or this action is not. And then this, as I said, this model is not really unusual. You have this kind of multiple downstream services. And last week, someone went and actually slightly changed the permission in one of these systems. Um, and permission changes are always one of those scary things to
kind of roll out. And you may have had those six different services. Two of them have adopted it properly. Uh, one of them just doesn't even check permissions. Another one is now uh returning a deny or has like a typo from a system that was changed six months ago. Um, service A can call service B. Service C silently skips the check. The next one goes and denies
it for the whole system. The other service tries to assemble some sort of response, get some inconsistent state back, some divergent pattern, retries, retries, retries, retries, your crew grows that, you know, at 4 a.m. the Q depth has got now to 30,000. You're trying to figure out what's going on. Every one of those retry loops hits the same sort of fragmentation and divergence or drift around um
what should happen uh with this kind of impossible answer. So you end up rolling back the permission change. Latency drops, page alerts clear, etc. All come in the next morning. You do that postmortem and it gets boiled down to like a permission roll out regression which yes is true but I argue is missing the point. Authorization logic in this model was now scattered across these six different
services. Each one has implemented it very differently. Uh the system as a whole get six different answers of what actually can be allowed or not at that moment in time. And this fragmentation in some cases has been sitting there for years just never checked in all the code paths because of the way the system behave. It's the same sort of shape of of trouble you might run
into with like a database schema migration that never wrote out properly. But I'd actually argue it's worse because in this case from a security perspective, you've authenticated the user. You you're happy with them, but now the underlying systems are actually disagreeing about what they what they can or can't do. So I've mentioned authentication and authorization already just to make sure we're all on the same page. These
are two somewhat two very different concepts. Annoyingly, they sound the same. At least in English, they sound the same. So when I talk about authentication, this is about proving who you are. You have some sort of credential. This is what happens during the login flow. Um when you start going through token issuance, MFA, those sort of things or you know I arrived at the airport yesterday with
my passport. I am actually half Dutch so I do have a Dutch passport. So I skip the long lines which is nice. Um so I authenticate by having a credential my my passport that says is me. But the authorization decision, am I allowed into the shenan zone was determined by the type of passport I have. That was an authorization decision. So this is where you might be
used to terms like arback, a back, pback, reback. These are ultimate policy decisions, access checks. Can this person do this thing on said system? Going to be focusing very much on the authorization uh side of things. So now we're in this model where we have this very distributed uh type architectures particularly in like enterprise environments. You're going to have some sort of API gateway. You got numerous
app services, microservices, picky poison. We've got data platforms. You might have like um a Trina or something running. You might have a load of background jobs on a queue. They're being processed. It's 2026, so we now talk about AI agents running rife across this ecosystem. And what typically happens we we see is that these authorization decisions, these checks, can this user or this identity because it could
be a workload or it could be an agent do this action on this particular resource. That is where drift occurs. This is where things can actually result in two different answers to the same check because this logic is kind of distributed. And when this happens, one slow check or one diff diverging decision can actually lead to this whole domino scenario. So an authorization call maybe is in
the blocking path hitting one part of the service and then that cascades downstream where now every request has to wait for all these decision checks to be made and one slow check or one incorrect answer or one divergent answer cascading failure scenario. So you might have scattered inline hard-coded if else statements as authorization checks based on a token claim thread all over the place ultimately never converge.
The retries keep building and things go wrong or maybe there's a small change to the logic in one system which is going to lead to a suddenly a burst of deny decisions. So things that shouldn't be allowed and suddenly what used to work doesn't work anymore and so you end up with this kind of cascading failure model which um is representative of how critical authorization is is
a common crosscutting concern inside of any modern architecture. And this is where you'd end up in this kind of diverging uh drifting workflows from a pushing model. So every call in a in a call chain typically will go through some sort of authorization check. It may be a binary. Is this person authenticated and have they got the right scope or claim in a token? Fine, very simple.
Or it might be fine grained. Okay, this person is the owner of this document, therefore they should be able to edit it. And this this logic ends up being split across all these different layers. So maybe in your CI pipeline, you've got some logic that determines what can go to production. you know, you deploy a job, it's authorized for the deploy action based of you as an
employee of the company. The but the policy maybe doesn't distinguish or the logic check doesn't distinguish between the different environments. So now each step of the chain is correct in the sense that it's checking that you are authorized to a deployment, but it hasn't got that context of which environment you're running to and now suddenly you have an environment an artifact that was designed to go to
staging or pre-prod environment in production. Not good. Maybe you have that ETL pipeline job that's extracting data from your data warehouse and pushing it to some other systems. that may now fail because of a deny reason. That's going to lead to a complete retry, going to fill the cues and basically every downstream report and dashboard is now going to be wrong because the data isn't being fetched
because ultimately an authorization decision. And then the worst of all is kind of this drift. It's kind of this like fan out that you see here where the same request is actually start going to get different permission answers from uh different systems depending on how it asks the question or the access pattern to that model. You know, if it's going to go and throw through the rest
API, maybe it's using a token, fine. But if it's another system that has access via a different endpoint to get some of the same data that has a different code path, you could now end up with this kind of divergence. And so really this per action authorization is ultimately trying to decide is this request allowed, but it doesn't actually answer based on the holistic sort of trajectory
or plan or intent of the action being performed. Is it actually still aligned with what we're trying to achieve? This is only exacerbated with agents. So now we have agents being spun up. Um, I like to think of them as like eager interns. They give them a task. They're going to try and achieve. Um, what they're going to try and do whatever they can to to, you
know, please the new boss and that they can do something. But the difference between interns and agents is interns usually have some sort of moral compass to determine like maybe I shouldn't go and access that HR record. That doesn't quite feel right. Whilst an agent will just go and pull every li lever and call every API and make every tool call uh to get to the the
end result. And the other problem here is authority. So if you start thinking about author authorization and identity as the authority kind of boundaries, um you're going to have this fragmentation. So authority ultimately never gets reviewed. If the uh the policy or the authorization check in some service that was written four years ago was written one way, two teams year later, you know, the company's grown that
that service is now maybe a loadbearing service for 10 other systems. the actual authorization logic was written a long time ago and there was no tie back to a particular user or role or um audit requirement or compliance requirement needed to actually manage that and evolve that when there's chains of identities involved. So service Acting on behalf of the user called service B on behalf of service
C the effective authority needs to be that whole chain when it comes to uh compliance and and particularly regulated businesses need to be able to understand and rationalize about those different checks along the way. You know, each one of those individual calls may looks fine, but holistically is that whole chain actually authorized for the actions trying to do. Very important when we're now looking at agents that
are building long trajectory plans of what they want to do. And then ultimately, if those things aren't all aligned, you're going to end up with enforcement points be applications, API gateways, databases that are going to start disagreeing because suddenly they've got diversion logic checks. Um, what makes this kind of scary though is that revocation of access uh can't really keep up with execution. And by that I
mean if you want to go and revoke access for some type of service or some type of user to go and access some sort of system, you're now going to have to go and dig through dozens of systems and services and try and unpick how that permission is granted. Um if it's anything beyond a simple have they got a role in in the token. So when we
when I look at the world now, you know, this is very authorization and permissions is always something that was very much aligned with kind of the security side of the house and you want to make sure your security checks are are kind of in play. I propose that we reframe this more as like an S sur or reliability perspective. So in a security world, we we start
to think about as security is like what can a user do? What what can our identity do on the system? If we flip that over to a reliability world, it's more about what happens when uh decisions disagree at scale, how do we manage that drift? How do you manage the the the retry, the back off? When we talk about preventing unauthorized access from a security perspective, the
inverse of that is we want predictable behavior in underlying that underlies all the conditions that are being made across checks across the system. Uh periodic audits. I don't know if any of you have had to sit through a quarterly or an annual uh standing access review where you have to say, "Yep, this team should be able to access this system. This team should be able to access
this system." Tickbox exercise. We kind of turn it around. We actually what we really want to do is be able to continuously verify that our permissions and authorization model works as a CI step process following all the GitHub style principles. You when you do a security view uh from a security perspective so there's been some postbach type investigation on the flip side we want to make sure
we can turn that around to a pre-execution validation. So how can we verify that our policy that authorization checks actually work and align with what we're trying to do from a security perspective of the business. And then a security view of the world, a bug is a breach. In the SR world, a bug is more like an outage. It's unexpected um behavior, latency, workflows, things like that
piling up. So you may wondering like when or how would this be applicable to me? And I break it down in kind of five different areas. So if um permission denies or permission errors uh differ between services being called by uh different backends, then that's kind of a flag that there's some inconsistency in how you're managing permissions across the system. If you ever end up with the
same user trying to do the same action but gets two different answers depending on their access pattern on their device, those kind of things, that's a inconsistency in the authorization logic layer. If a permission roll out leads to some sort of discomfort, let's say, um, and unrelated services now suddenly start behaving differently even though you just change say what a role does in one particular system, that's
a sign of this kind of drift and inconsistency creeping in. If you ever end up with this like a replay tree, a retry storm with there's this fragmented decisions along the way, getting this authorization logic standardized and uniform is going to be a key driver. And then if you've ever building some part of application or adding a feature and you start mocking uh the authorization logic or
mocking the permission checks that's probably a sign that the authorization layer isn't unified across the stack. So how do we go about solving this? And really there's there's two concepts I want to introduce here. One is about externalizing authorization. Um so if any of you are familiar with things like open policy agent uh CEDA or servos which is my company um then uh that's this is the
idea of you taking all that authorization logic out encoding it as a policy definition in some shape or form. So YAML in our form um which is something that's versionable testable and auditable and externalized from the application. And then on the coordinate side of things you want to make sure there's a single way to do authorization checks. And this is where the open ID or zen standard
uh 1.0 O as of generary um defines kind of the API interface between an enforcement point such as an API gateway or an application or a database and a decision point such as servos or APA or one of those kind of tools. So if we externalize the authorization logic logically not actually how things are deployed but logically it means we no longer have all these distributed different
ways of managing authorization checks each hardcoded into each of these different services. The idea is there's one now central logical place where all the decisions and all the policy is managed. So there's one uniform source uni uh un unified source of okay for this type of access to this type of resource here are the roles here are the permissions here are the conditions under which access should
be allowed or denied. Now for actual deployment that that central service would actually be deployed out and run alongside each of those different services. So it's like a Kubernetes sidecar but the idea is you centrally manage but you distribute out the decisions locally. But the key key thing here is you are centralizing into a single place to a single git repo ultimately here's the permission logic and
the authorization logic across the application and every system that needs to do permission checks calls into this single set of policy truths and then how you actually go and implement that obviously you don't want to get locked in to some vendor that's going to charge you way too much this is where standards come in so uh the open ID or zen standard is as I mentioned that
the spec for how each of these different systems be it a gateway or an app or an MCP server or an agent harness now say code can talk to a decision point a policy decision point engine so this standard is the spec that says here's a here's a subject so a user a workload a token a delegated token and on behalf of token a workload identity here's
the resource so they're trying to view a report edit an expense download an invoice these kind of business objects and actions um um and here's some context additional context and be the IP they're on, maybe they're on a trusted device, maybe they're on a the MDM uh risk engine has flagged them as an at risk user. That all becomes context. It goes through a request to a
decision point like Serbos. Seros then evaluates the policy, creates an audit log, an audit trail of that decision and what comes back is a simple decision, allow or deny. So now regardless of how each of these systems might do permissions that decision now being offloaded to this central um source of truth for how permissions work and that way you have this uniform layer across all the systems.
So when you move to this model you actually flip your system from an architecture perspective for authorization and you gain five key things. First off you get predict predictableness predictable in in type of your environment. So the same inputs subject principle as subject resource action context is always going to get the same decision. You know the protocol in terms of the API format guarantees that the response
structure the central policy engine is going to give you a uniform answer across all those different layers. Next up is testable. So the policies are now being stored centrally and managed centrally. Uh any good policy framework like servos or others has a testing framework align it. So you can write your policies, but you can also write tests against it. And thus in your CI process, you can
actually verify and have tests around your authorization logic that's completely decoupled and not written in Java and not written in Go or anything like that. It's written in something um that is agnostic to the actual implementation where it was called. And that way you can uh test in isolation that your authorization logic is doing what your security team or your entity team or your um regulatory um
body has has deemed uh required. It's also explainable. So now every decision that gets made you can start tracing it back to okay this action was allowed for this user by this resource by this particular version of this particular policy and you can explain it back out which then ties into accountability as well. So you can now point to okay this decision was made because of this
particular this particular um change request or this particular regulatory requirement that landed on us as a business and you can tie and trace order back through and then kind of containable so you can start putting in um those guardrails and those those hard limits. So make sure you uh that you have default deny. Make sure there's a enforcement point at multiple layers of the stack and you
can start constraining the blast radius for every decision down to every layer of the application. So now when the the alarm goes off at 2 a.m. you have one place to look, one audit log of the decisions if it's a permission issue. Um and then you can actually go and replay that logic through through the system. And so you have the same engine, same actor. Here's an
example of a seros policy where we can say, okay, to send a bulk email is allowed if this expression or this condition is matched. And so based on the inputs you can see here we have uh sales rep a CRM service and agent we can start evaluating based on the inputs u some conditions and we use cell in seros at least to define this again this go
and grab it off GitHub especially license you can run away with it. So now if we brought a agent identity in this doesn't actually have to change. We can evaluate the same sort of policy the same sort of unified source of truth for what actually should be allowed or denied. Um regardless of what type of identity or or combination or delegated identity that that comes in for
mission check you don't need to start hard coding this logic into your MCP servers into your agent harnesses into your legacy applications. You can define it once centrally and have that single source of truth for how identity works across the system. So kind of three questions um when you go and look at um your your environment. First off, if there is a permission change, what can you
actually go and replay that check? Is there a way you can go through and figure out what actually led or why did this decision uh become uh true? So can you go and look through your logs and figure out why a permission was blocked? If you can't, this is a pointer to go and use something that's externalized in a failure mode scenario. if your authorization sort of
service um or layer runs of an issue. Do you fail open, fail close? Do you have a cascading failure? Do you know kind of logically how that is actually enforced across the stack? This is actually a key one to understand really what your fail scenarios are. And then dissminism given some input, are you always going to get the same output? That's kind of the key point here
rather than having this scattered logic across all your different application layers. So if you want to go and start play with this, just go and pull our Docker container. Um to start looking at this, first off, go and spin up a decision point, write some policy against it. Go through kind of your application and find the simplest check. This person must be in this role and they
must be the owner of the document to edit it. Define that as a policy. Get your head around the motion. You know, you'll end up with documents that sort of look like this. Um then go and replay a request. Compare the logic from a decision coming from a policy engine versus what was happening in line. Make sure they align. If good, make the swap and move that
particular path code path of your system to an authorization engine. And then once we scale this out, show it to your teams and prove the value. Um, and so hopefully when we'll start deploying and rolling out viod applications, we can put proper sensible deterministic guard rails and policy around what they're allowed to do before we before it's too late, I would say. And with that, I want
to say thank you very much and love to chat more.
