About this talk
In this talk, Remondrozol, a security engineer from Dyne sec, discusses the importance of white box security reviews, particularly for web applications. He explains how having access to the source code allows for a more thorough analysis beyond traditional pen testing methods, leading to better coverage and root cause analysis. The speaker highlights a specific vulnerability found during a pen test, demonstrating how a minor oversight in authorization logic can lead to serious security breaches. He emphasizes the systemic issues in API design, such as misimplemented authorization protocols and inconsistent security practices, which can result in fail-open conditions. Remondrozol proposes that improved architecture can enhance both security and developer productivity by minimizing the chances for errors. He shares strategies for implementing effective security practices and emphasizes the need for global permissions and tenant isolation at the database level.
Full transcript
My name is Remondrozol. I'm a staff security engineer at the Dyne sec. We are a boutique security company. Our main craft is performing white box security reviews of well, various systems. In my particular case, it's mostly web applications. And this is what I would like to talk about today. So, uh white box security reviews means that on top of the regular dynamic testing of the system at
hand, we we also got the access to the code of the of this particular system. And we analyze and perform a security review of the code itself. Uh it's a great way of performing these reviews in our opinion because um thanks to that we save a lot of time on usually used on reconnaissance in in more traditional um pen testing. This gives us better coverage in the
same time. And we can also perform root cause analysis of the of the issues that we are finding. So, first of all, why should you care? Let's go quickly with one of the bugs that I found in 2021, I think. So, during the this pen test, I found this endpoint API identities. And it turned out that it would share with the requestor with potential attacker a list
of all the emails throughout the entire system from all the tenants from from all of the clients. So, it's a vulnerability, all right, but not that it doesn't sound that bad, you know, the the disclosure of the emails is is not that bad. Certainly worth a mention in the report. On top of that, a bunch of metadata. However, I kept digging in the code and I was
able to actually craft a proof of concept Uh, by adding this uh, parameter to to the get request with, uh, value tokens. And what it did, uh, it added, uh, a data from, uh, SQL table named tokens, which was related to the, uh, vulnerable one identities. And what was in the tokens table? Well, there were all the session tokens for all the users, uh, entire, uh, system.
So, as an effect, with one request, I was able to take over all in the entire, uh, system for all the clients, all the users. And conveniently, uh, packed with the emails already. So, so you you you already knew what kind of, uh, company you were hacking, basically. Uh, so, this begs the question, how otherwise, uh, robust, uh, company with pretty robust code can fail, uh, so
badly with the security of, uh, authorization in their API? And this is what I would like to, uh, What kind of architecture and anti-patterns, uh, can emerge in, uh, API code? what to do to avoid avoid these these mistakes. If you hang around, uh, until the end, uh, there will be also a quick, uh, root cause analysis of this particular bug that I showed you. And interestingly,
uh, the architectures that I would would consider, uh, more secure are also the ones that easier to maintain and make your, uh, developers more productive. So, today I would like to also, uh, convince you that, uh, security and, uh, you productivity of your developers don't necessarily have to be do these two opposing forces that you need to balance out somehow, but can go hand in hand in
hand, at least to some degree, of course. Quick disclaimers before we start. All the shown vulnerabilities and, uh, code was actually found in, uh, production of our clients. Of course, I had to anonymize all the code and all the endpoints, so there will be no companies' names, no server names, and even the names of the endpoints were changed into a simple project management API. So, expect APIs
that are pretend to be some kind of, you know, users, projects, and stuff like that. Um last but not least, the presentation tries to be technology agnostic, which means that I'm not uh making any statements about security or insecurity of certain technologies, languages, or frameworks. Um this this pattern can emerge basically with any high-level modern high-level language. Uh but you will notice that all the examples are
in Python. This is for two reasons. First of all, actually a lot of code that we do see is in Python, so naturally many examples that I have are in Python. Secondly, I think Python, due to its it kind of looks like this pseudo code, so it's a great language to uh show this kind of more general statements. So, simple is best, right? So, this is a
made up actually simple controller that implements an API endpoint serving company data. And let's discuss what do we actually expect from authorization code in a simple software as a service SaaS company. So, there are two basic tasks that we want to achieve. First of all, we want to achieve what we call the tenant isolation, which means that when we have multiple clients, multiple companies as our clients,
we want to ensure that company A can only access company A data, but not the company B's data. And vice versa. The other thing that we want to ensure that inside the same company uh there can be multiple users, and these users can have different levels of access. So, you can have an admin, we can have a regular user, maybe a read-only users, and sometimes others. And
this is what this code is doing. So, let's uh, take a look what uh, we actually see here. We take uh, from a from the request data, we read the company ID of the requested object. This is, of course, untrusted. Uh, an attacker can basically give any value Then, from the uh, authentication context, we read the user's role, and we verify whether or not they are an
admin. Uh, if they are an admin, we also verify that uh, the requested company ID is actually the user's company ID. If that is not the case, we raise a permission denied error. Otherwise, we fetch the object from the database and serve it to to the client. So, who would say that this is a good way of uh, performing uh, authorization? Uh, not many people, and that's
good because I think that this code is uh, pretty bad for many reasons. So, the elephant in the room is that um, if statements, the conditionals are written pretty badly. So, let's take a look how easily it can go wrong. So, if we misplaced just one indentation uh, for the return statement, and we will write it like that, let's uh, quickly debug what will happen if a
non-admin uh, requests this endpoint. So, the is admin will be false. So, we will fall through all the checks, basically, and just return the object to basically anyone. So, we are not doing the Neither of the tasks that we wanted to implement is working uh, properly. Neither the uh, tenant isolation, neither the enforcement of the uh, user privileges. And you can ask, "Well, okay, Szymon, but why
uh, one mis- one or two misplaced uh, if statements uh, make an architecture problem. Well, I think that these misplaced uh, if statements are actually just a symptom of a larger problem. And the larger problem is that we basically have our developers re-implement the authorization logic every time they're implementing a new endpoint. So, they are bound to make mistakes over time. uh, if you if you are
thinking, "Okay, well, let's then write a helper and then dry up uh, this this a little bit so we don't repeat ourselves with this logic and keep the logic inside some kind of helper." I would say that it is solving only a part of the problem because okay, now we are not re-implementing this each time, but we still aren't um, providing any guardrails uh, to our developers
which and the authorization is still fail open which means that anytime uh, a developer forgets to uh, write uh, this uh, this function call, the this particular API endpoint will be vulnerable. So, sooner or later we will also uh, see an endpoint like this. And on the first glance it uh, looks pretty scary because after all we have discussed already, it kind of looks like we don't
have any authorization logic here uh, implemented. But in fact, we really don't know whether or not is true because maybe some other team working on the same code base decided to push the authorization logic somewhere down the line, maybe inside the fetch project function, mix it up with uh, business logic. And maybe this is actually secure. But the problem here is that just by looking at the
controller we don't know what is the intended and what is the actual um, access model here. So, we've got a bunch of systemic problems here. We did not provide any framework to the developers. We did not uh, provide any guardrails for them. And as an effect, we got the code that is far from self-documenting and therefore becomes unreadable quickly and therefore is hard to maintain, is expensive
into in maintenance and development. It will be hard or impossible to catch any bugs with static code analysis and it is fail open, which means that in a 500 endpoint API, we have 500 chances of our authorization wrong. If we have everyone thousand endpoints, we will have 1000 chances of getting this wrong. All in all, the only thing that scales well with our API are security vulnerabilities.
So, surely there are better options. So, what are they and what what could go wrong with them? Here we have another case of inconsistent design. Um So, this particular In this particular case, we have this two-layer API. So, there is a proxy view that is client and internet-facing. Uh but it does not actually implement any business logic. The business logic is implemented in these um API views
inside some microservices somewhere in the back end. And the proxy view only well, proxies the request to I do like some parts of this code. So, so what I do like is that we have these attributes like permission classes which define what is the expected access model for given controller. Uh there is also we know that there is also some kind of middleware that will execute it
later. What I don't like about this code is that is it is inconsistent. So, we have this proxy view which says, "Okay, the permission class is allow any." So, we it doesn't care about authorization actually. But the API view, the back end one, says, "Okay, we require the permission retrieve." So, of course, uh the entire stack works well because the request will have to go through both
of them to actually serve any data. But we are consistent and uh it gets uh even more complicated if you if we take a look at another end point where we will see that uh now it's the proxy view which actually uh has this allowed role uh attribute which requires in this particular case uh a root. While the back end one only cares whether or not you
are authenticated. Another problem that you might have uh noticed here is that we have also a mix up of the role-based authorization authenticate authorization and uh uh uh we we we we take a look at the permission base. So, some of them uh care about the permissions of the certain user while while others require uh certain roles. uh the number of possible combinations is rising very quickly.
We can uh require this in the proxy view, in the back end, or in the back end one. We we might care about roles, about permissions, and so on. And uh you need to remember that uh they were set for you for your convenience in the presentation side by side, but what you actually see in in your code base, you only see one of these always. Uh
so, what do what you usually will look at uh in the in in the repository will be let's say project view, API view, with a bunch of business code, and the permission class is authenticated. And you have no idea what is the intended and what is the actual access model for this particular business. And of course, it's only a matter of time where we find a pair
like here where the proxy view will have allow any, the back end will have uh is authenticated. So, the entire stack for this particular endpoint does not implement uh any of authorization at all. So, as an impact we have numerous domain data revealed to low privilege users. There is also some possible cross tenant leaks. Again, systemic problems here because because there is this inconsistent authorization enforcement location
and unclear privilege model role based versus privileges. As an effect we get unreadable code with also we can't still be using any static code analysis tools. So, maybe a better idea would be to centralize the entire thing in one place, some kind of access control list. This is what our client did. So, this is small example of that. So, basically the idea is that we create one
static object per each role in the system and it contains I'm simplifying here an array of all the endpoints that the given role can have should have access to. And in theory it seems like a good idea because the intent with such a design is clear what we are trying to do, what we are trying to allow to access per each role. The problem however is it
does not scale well because of course we are talking about 500 more endpoints so it looks much more like that and you need to remember this is just one of the roles that we have in the system. So, it looks more like that actually. So, we get this uh entire so we get this file spanning for hundreds and hundreds of uh code lines and we have really
it's very unmanageable. We have no idea just by looking at this what is available, what to anyone. And to make matters worse the middleware that was executing these privileges was implemented in a way that if you forgot to put the an endpoint any of these arrays, it would just be available to anyone. So, the impact was over 50 vulnerable end points. Most of them was not that
interesting, but we were able to create, for example, a privilege escalation uh path and some log stealing. We have systemic problems here. Because still the forgotten end points don't execute the privilege check. So, it is still fail open with a 1,000 end point API with 1,000 chances of getting we basically, instead of making the life of our developers easier, we made it harder. Now, they need to
remember about going to at least two places when they are implementing a new end point in our old end points maintenance is a a little bit different beast. Basically, the idea is that imagine that a startup and in the beginning you don't really care about over engineering API with the access levels and so on. You just want to ship your features. Uh so, you do just that.
But over time your clients get bigger, they want to onboard their own employees. So, but they don't want to give them the keys to entire kingdom. So, you need to implement some access levels, admin, user, read only user on top of the already existing API. And this is what was done here. And actually, I do like this code. So, I like when uh the authorization especially access
levels are denoted via annotations. Annotations is one of these rare examples which is very readable for both humans and machines. So, both intent and model is pretty clear from just looking at this. So, it is a great example of how you should be doing this. There's only one problem. >> [snorts] >> You see, developers have this compulsion of never introducing any breaking changes to their API. So,
even when introduced a new security feature, they did not make it a breaking change. So, they implemented this version tool list access level aware on top of the old one which was not. So, a low privilege user could just change the API address from version two to version one and they could access anything still. The impact was that low privilege users could still list all the tenant
users including emails, personal information, modification endpoints were possible. So, we are able to do privilege escalation. Therefore, a guest could take over the entire tenant. And still we have systemic problems here because we still did not create an universal privilege execution mechanism. The other thing is regular housekeeping of your code. So, it does sound like you know, additional work, but in the long run the growing stale
of these endpoints will be making your code unmanageable and and you should be sometimes cutting the dead weight. remember our vulnerability from the beginning. Let's with all that knowledge that we have seen, let's see what actually went wrong here. I think that just by looking at the code you will start seeing the pattern. on the first glance this is a pretty innocent code. It's a function JSONify
responsible for serializing the data just before it was actually returned to the user. And inside this function, someone has implemented the last line of defense to not drop customers. So, we have this line where it is verified that the current user context matches the customer UID of object that you are serializing. First problem was that this last line of defense was actually the only line of defense.
The other problem uh was this code has attribute. So, it turns out that we will uh execute this last line of defense only if we find this attribute customer UID in the object that we are dumping. And so, it is like the question, "Okay, how does the property makes its way to the models?" So, there's this class customer owned, which has this attribute customer UID, which is
a wrapper over a column. And all of the models should be inheriting this customer owned class. And as as you can uh predict, we have this identity model class, which they forgot to to inherit the customer owned. Therefore, there was no customer UID. Therefore, was no ownership verification. So, the obvious fix is of course, "Okay, let's just add this customer owned inheritance." But the real issue is
is of core, you guessed it, systemic uh because the ownership is fail open. So, if we forget about this inheritance, we just allow anyone to to see everything. And we still are lacking a good framework for developers. We are doing some weird checks in the last moment whether or not we are serving wrong data to to to wrong people. Uh another problem here was this close coupling
of the API with the backend data model with the actual database, which allowed us to escalate it into critical by querying the tokens table. It might look like a good idea to to implement such features because it speeds up development a little bit, but in my opinion is uh uh way too easy to be a foot gun in the long run. So, what would we consider a
good architecture? A good architecture and good code is self-documenting. So, just by looking at the single endpoint at a controller, we want to be sure what is access model for it. We don't want our developers have to remember stuff. So, we want it to happen automatically. In the case we they actually forget about something, we want we want it to be fail close. So, any failure on
the developer side should lead to data that is just not available to anyone or just to higher kind of user like admin, not just to to to anyone. so, how do we do that? Well, we want to ensure that these authorization rules are global. So, in the case of previous execution, the best way in my opinion at least is doing this in some kind of global middleware
that is called just before we actually handling the the HTTP request. This way um we know that it has been already verified when we are actually implementing the the endpoint. And we either some attributes or or annotations denote per request what per per controller per handler what is the expected access level. for the tenant isolation, we want to do possibly as close as possible to the data
layer to the database. Um if we our database access object is authentication context aware, it can filter out all the data for a given tenant before any data actually makes way into our business code into our endpoint. And this way, again, developers don't have to worry about it at all, don't have to waste time on uh implementing and filtering that that out. And we can be sure
that any of this code does not uh serve wrong uh stuff to the wrong people. Of course, if our uh code does not adhere uh to these uh rules already, uh we can't just do it overnight. So, what to do with limited time? Well, first of all, especially that you know that uh some of your endpoints might be vulnerable, you want to identify uh the most impactful
endpoints. this vary from business to business, but usually is the stuff that is serving uh any private information. And also stuff that can be used for privilege escalation. So, anything uh for instance used to uh manage users and stuff like that. Then, you refactor these most impactful endpoints for your business. For the rest, you could uh utilize some uh custom rules counters. We are done cycle of
Semgrep. Uh it's uh kind of grep, but uh semantic, that hence the name. So, the idea is that you have this uh tool to search through a code, but which understands the syntax. So, you don't have to uh write a very intricate um regular expression to catch uh a function call. You just try to catch a function call like in this example. So, here we are catching
the NF uh function call from the router object, which means that we catch all the instances of the creating of the endpoint in our Python code. And on top of that, we add uh negative rule pattern not. We filter out any any code that is authorization aware via this uh um parameter. So, as an effect, we get the list of all the endpoints that are uh potentially
vulnerable because they are not taking this authorization um uh code. So, we can utilize this in our custom CI/CD pipeline and make it a control to improve over time. Also, to ensure that any newly created endpoints already adhere to to to our new more architecture. And last quick win that you you can do if your API still uses integer IDs to denote objects, you should migrate to
GUIDs. This way, if the vulnerability already appears, it it is already possible to query any object from the system, also from other tenants, it becomes much harder to be exploited by an attacker. Because if these are IDs, you can easily just iterate from ID 1 to ID 1 million and dump all the objects from your API. If these are GUIDs, which are these random longer strings, you
can't just easily um guess the IDs in the system. And as an effect, the severity of your issue comes down by one. So, anything that would have been a critical will be just a high severity vulnerability. High severity severity vulnerability will will become medium and so on. thank you. That's all I have today. >> [applause] >> If you want to chat about any of these or any
other security issues, uh you can find us at our booth, the Insec booth here at the main uh Wait. Thanks.
