Signed, Sealed, Delivered: Why Reverse Proxies Outperform VPNs - Peter O'Neill & Boris Kurktchiev

This talk discusses the implementation of identity-based control through reverse proxies, specifically focusing on the use of Keycloak and Envoy. The speakers, Boris and Peter from Teleport, emphasize the importance of enhancing security by moving away from traditional VPNs to a model where every endpoint requires authentication and authorization. They outline the risks posed by current network access models and propose a framework for handling these challenges. The session includes practical demonstrations that illustrate how to establish secure identity management and audit logging using tokens and role-based access controls.

All right. Hi everybody. I'm very surprised that so many of you are skipping beer hour and are hanging out with us today this late into the conference. Before we start Peter and I would like to take a quick selfie with all of you to prove to our company that we're not on a boondoggle. All right. Appreciate it. Everybody I saw the people making funny faces in the

back. So yeah, thank you for being here. What we're going to be talking about is sign sealed and delivered. How do we do identity based control with reverse proxies and why that's important and why that's something that you would want to implement and do in your companies and day-to-day lives. First and foremost, why you should you know trust us or why you should listen to us. Hi,

I am Boris. My last name is very complicated. I am the field CTO for Teleport. I'm also one of the leads for the CNCF AI TCG. Shout out. We just published some agentic based um cloud native framework if you will. You should go read about that. And then Peter. Hi. So my name is Peter O'Neal. I'm a sales engineering manager at Teleport. Uh and yeah, this is

this is something like zero trust security something I've been working on for for many years now. So definitely definitely excited to share with you guys. So let's get going. Why are we here? I think AI has done one thing and I have to talk about AI cuz we are here and AI is everywhere, right? It has made it very clear that our identity story is broken. And

more specifically we have been using things like VPNs. We have been using singular points of control in order to enable access to private resources. And the issue with something like VPN is that it's a single front gate and once you're inside you do not have the you know the tools necessary to reduce lateral control lateral access to reduce the ability to crawl the network to discover and

maybe that is just enough when we're dealing with just humans, but now we have automations that can come in and without the right authorizations, without the right authentication, without the right controls can just wreak havoc. I think we've all been reading the news. All it takes is a single identity to be consumed as Trivy discovered unfortunately yesterday and cause a lot of issues. So why or what

are we proposing? Right? The idea behind our entire talk is that you need to do the lift to add authentication, authorization to every endpoint in your stack. And that is that means absolutely everything that your end users or AI agents or CICD pipelines, it doesn't really matter what it is should have a way to be authorized and given giving you the knobs to decide yes, you should

be able to access that data or service or you should be able to do the thing that you are trying to do. Right? It doesn't matter how much something hallucinates if it's never been given the proper authorization to do said action. Go ahead, hallucinate, right? I no longer I can sleep a little better at night at that point. What we'll be talking about today are two open

source projects, Keycloak and Envoy, and we'll be using those in order to build out the architecture that we're talking about. Oh, one thing that I missed. Uh security folks love audit trails. Nobody loves a seam, right? I've spent a lot of times a lot of time in the IT industry and I love logging into Splunk and then I hate writing a 7-mile long query in order to

try and prove out that an attack happened or that to figure out how an attack happened. Um Adding something like a reverse proxy does add the context that you now truly need in order to be able to figure out and what those controls are that the controls are functioning the way you want them to function. Should we explain what a reverse proxy or I guess I guess

on the next slide, yeah? I believe >> Yes, it is. Okay. Mhm. Okay. So what is a reverse proxy, right? What is I'm not going to dive specifically into how Envoy does this, but Envoy is the project that we're using for the reverse proxy. But what does it do? It acts as a singular front gate to your application. So now you're able to scale the idea of

a VPN instead of to a singular front gate to your entire network, you can now put it in front of just about anything. And that anything does not have to know how to do authorization, does not know have to know how to do authentication. Envoy and the reverse proxy will handle that for you. And then handle things like TLS. So another CNCF project. If you've ever used

Istio, one of the nice and easy wins of running Istio is being able to get mTLS on anything that runs within the service mesh, right? Very similar concept. Here I can put it in front of my database. My database some legacy databases have no concept of TLS, have no concept of certificates. I can now actually utilize those and continue um continue doing this and now I also

have a centralized way to do my authentication, authorization, and and getting that logging intent added in. Sorry, lost my uh So a single a single reverse proxy can multiplex multiple endpoints. But because we now have the ability to define the rules and define the intended paths you now [snorts] have the uh kind of the control knobs to make the decisions between I want user A to go

to backend Z, but user B to be able to go to backend A. Right? You can utilize a single point for multiple backends with a single set of rules instead of okay, how do I fight how do I create the controls that we're talking about in something like again just perimeter controls. All right, but the most important piece for me at least is [snorts] the identity aware

access log. So instead of getting oh, an IPA was you know accessed an endpoint I now know hey, Bob tried to access an endpoint and they did not have the permission to do so, so they got a 403. Or they got a denied or a 505 or whatever. I now have the understanding of what path did Bob try to take why were they denied and then obviously

the specific endpoint that they were trying to reach. Then on the other end I have the other side of the house. I am able to prove out my compliance with the myriad of three, four, five letter policies and compliance frameworks that we all have to do. Whether it's GDRP, whether it's PCI, whether it's SOC 2 etc. I can now actually create the the proof if you will

that yes, I have these controls in place and here they are actually working and doing the thing that I'm trying to show my auditor or what have you that we're doing. And with that I'm going to hand it off to Peter to walk you through the actual demo. Mhm. So Boris is always our big picture guy. I'm more of the nitty-gritty guy. So now we're into the

actual like pieces of the demo, the things that actually create that identity, right? So we're going to be using two tools here, OAuth 2 and OpenID Connect or OIDC. Right? With OIDC this is going to provide that identity layer. This is what's actually going to take your user from a system, in our case Keycloak for this example. We're going to take that user. We're going to create

each of the different components that we need to identify about that user, and then add it on to the network traffic, right? This is what's going to give us that audit log that Boris was just talking about. They're going to give us that who, what, when, where, how attached to the network logs that you already have, right? Giving us that full picture. OAuth 2 in this equation

is going to be the authorization piece. uh map what that user can do on the systems that we are targeting here. Uh and so for this example, we're going to have a few I think it's on the next slide. Uh and so with this Yes, and so we have Keycloak doing the authorization. We have Envoy, which is going to do the routing. And then JWTs, I think

that's in the next slide as well. So, with that I'm going to pop I'm going to go check the next slide here. Right. And so JWTs Right. Now that we have the identity information from OIDC, we need a way to transfer this information throughout the system. JWTs here are going to be an easy way and a very standard way to take that information, encapsulate it, and then

be able to hand it from one service to another, right? And we are encapsulating the data here, not encrypting the data, right? So, we're just making this an easy standard format with JWTs. We're not securing the information, right? So, JWTs are not a place to put your secrets. This is a place to put usable information that you're going to want your system to access. Right? And so

this is when we look at the actual payload here, right? These are the the pieces of information that we're going to be wanting to look at, right? We're going to want to look at the issuer, the audience, the username, uh the realm access is where this is valid, right? So, all this information that you want your system to use in order to decide whether a user should

be able to take the actions they're trying to take, right? Giving us those those status codes of 200 or 403 in this demo. And then yeah. Oh yeah, and then the trade-offs here. Right, I already talked about encoded not encrypted. The other one here is that's going to be very nice for us is a short TTL, right? One thing we don't want is to create these authentication

tokens, right? And then have them live forever. The nice thing about using an identity is we have a very short TTL. When a user logs into Keycloak and then tries to do an action, the tokens attached to that action on that request can be, you know, as short as 5 minutes, like we're doing in this demo. Maybe they're a little bit longer, but they're going to expire,

right? Really reducing the attack surface of having any sort of static credential out there that can be lost, discovered, stolen, what what have you, right? And so that's something that having these short TTLs is something that, right, especially now is becoming more and more of a requirement. And so now we're going to look at the VPN approach versus what we're suggesting, right? And so with this VPN

approach here, right? This is just the classic you go you enter into the VPN and now I'm inside the internal network. And so each of these services have no idea or no concept of the identity of who's trying to reach them. If if if a hacker is in here, they're going to just poke around and look at all of the available endpoints and see what they can

see. This is also true for AI. When an AI is inside of your network and given enough permissions, they're also going to start poking around and pulling as much information about the network that they can, right? And so if they could see all of these endpoints, they're going to pull all that information in. And what they do with right? Then they might start acting or manipulating these

endpoints in ways that you might not expect, right? So, now we want to secure this. And so this is when we're going to be introducing Keycloak and Envoy here. Once again, Keycloak is going to provide us the identities. Envoy is going to be doing the routing to the back-end services via the reverse proxy. And so what that means is that when I log when I'm trying to

access one of these endpoints, and for this example, I'm using generic API endpoints, but it does not need to be an API endpoint. You could also think of these services as like a database or another resource, right? Anything that you want to log into you're going to want to be able to say I know who was trying to log into it, how they were doing it, and

what the end results were. So, you can think of these endpoints as pretty much any agnostic resource inside of your environment. So, how we're going to be demoing this, we're going to be using curl and curl with the identities passed from Keycloak, and then Envoy is going to route them. I think we have I don't know if there's another slide on this or not. One more. We'll

go back after the demo. Yeah, and so with that I'm going to hop over to the demo here. While Peter is hopping on, all of this is on GitHub. There's a link at the end. And obviously look it up at the after the talk, you can pull up everything that we did. Mhm. And so really simple, right? I have a self-contained Docker Compose here in the repo

that you can deploy, right? This is going to be hosting each of the services. So, we have Keycloak, Envoy, and then each of the endpoints that we want to access, right? And so I'm going to hop on over to my terminal here. How big is that? A little bit more? One more? Okay. Uh and then so I created a little script that's just going to step through

the commands, but everything that's being run is going to be shown on the screen. All my services are up and running. First thing that we're going to want to do is curl the public endpoint. So, you can think of these as shared resources inside of your network. And if you have And if you have no identity, you're going to want to be able to say that no,

even though you're inside the network, you're not able to access this resource. You're going to want to you're going to want to see these 401 codes everywhere. So, anyone doing any sort of scanning or poking around, you want them to be turned down. You don't want the internal wiki or whatever whatever's holding your information to just be accessible to anyone inside of the network. And so now

let's look at the identity for one of these users. And so I ended up using a username and password here. The like the Docker networking got really obnoxious with OIDC. So, Typically this this will have the OIDC off in it, but I'm using a username and password to kind of show the flow here. And so I'm going to be curling here we see the OpenID endpoint token

to get the JWT token for Alice, right? And so now this token is going to contain all the information for Alice. If we decode this right? Now we can see exactly when it was issued, right? And we can see the issue and the expiration date are very close together, giving us that 5-minute that 5-minute TTL, right? Keeping that very short for us. Where this is valid inside

of my demo client environment. And also what roles this user has, right? We are layering here our back and identity together, right? So, our back our back will provide some of the controls as to what this user should be allowed to do. And then the identity is going to give us that security piece of what the service is going to if the service is going to allow

them to communicate. And then we can also see some of the user information here, the username and email. So, now that we have the users the user's identity encapsulated in a JWT token, we can now use this to try to reach some of these endpoints. Remember this public endpoint here was the one that just gave us that 401 without any identity information attached to it. But now

I'm passing in the Alice token we just looked at as a bearer token. And this is going to now give us status code of 200 and see that we are able to see our internal resources with that identity. And then the other thing that we want to make sure is that Alice can see her own environments. Also being able to get in here, see that 200, see

Alice's private workspace. And so with one identity, right? Now we want to check if a different identity has the same access. And so now we're going to get Oh no, sorry, last one. So, like can Alice see Bob's private environment, right? This is where this is where we're going to want to see this 403. And this is that compliance piece again of is someone doing something on

the network that they shouldn't be doing, right? And so this is we're going to need the tracking for that compliance piece to see exactly when things happened. If someone is scanning your network, is an AI actually rummaging through things? We want to have all of this in a nice neat audit story. And so now we have Bob. Bob's also going to come in here. Bob, we're going

to grab a token for Bob and then use this to hit these same endpoints, right? And so the difference with Bob here is that Bob is also an admin. Right? So, with an admin role, right? We want to we want to have the differences between our back and identity, right? Just because Bob is has a role of admin, he shouldn't be able to change his permissions. He

shouldn't be able to have additional access to just rummage through the network. We still want to identify Bob's traffic to see what Bob is doing. And so when Bob tries to access Alice's information, even though he's an admin, we still want to be able to see that that 403, right? And so this is this is completing that identity picture for us of what these users should be

able to do based on their identities. And then Bob should be able to see his own stuff. You can see this, it means your name is Bob. And then coming to the end here right? The audit logging, this is where all of the pieces come or get tied together. Right? This is where we want to see that who, what, when, where, why. We can see what they're

accessing. We see the Bob access. We see user is Alice trying to access Bob. We're getting the 403. And with that we're going to pop back over to the slide deck. And start the slideshow again. And then is this slide you or slide me? Uh that's Am I on? Are you off? Oh, yeah. >> be off. Okay, well, Boris and my team seems to have disconnected. So,

uh, let's see here. I can I can take this, right? This is wrapping up wrapping up kind of what I was talking about there with the audit logging, right? With the audit logs, you want to make sure that the identity is heavily intertwined with the audit log. You don't want to have just an IP address saying that this IP address had a 200 or 403, right? That

doesn't give us a complete story. We want all of that encapsulated together so that we can know the who, what, when, where, why right when we're looking at the audit log. We don't have to cross-reference IP addresses and people's computers and the timestamp, right? All of that should live in one place at the same time. Is this Okay, this seems to be working here. Uh, so why

is the you know, why are we kind of putting a heavy emphasis on the logging piece? going back to the original bit about now we live in a world where intent is way higher value, like knowing the intent is way higher value than uh, just somebody accessed something. As you you know, again, you go out on the on the sponsor floor, everybody's doing AI, everybody's telling you

the uh, you should run agents, they will secure it. You still have to be able to A, prove, but B, know that your environment is safe. And over simplified way, uh, and which why I started the whole uh, talk with, "Look, the old way, uh, and the way we have been doing authentication and authorization is just broken. Uh, AI broke it and Open Claw and we insert

whatever the next uh, kind of popular thing is just exposing how broken our idea of authentication and authorization is. And that intent piece, carrying that uh, uh, you know, Peter and I did a podcast today and I can't believe I'm going to repeat this. I was walking around the floor and I heard somebody going, "Oh, the new RBA is IBA." I stopped and I'm like listening over

here and I'm like, "What is IBA?" Intent-based uh, access control. I'm like, "This is going to be the new buzzword. I guarantee it by uh, KubeCon US, we're going to have IBA companies." But it is not far from the mark, right? I now have to be able to carry, "Oh, Bob initiated an AI agent and that AI agent did something on behalf of Bob." You have to

have that intent chain and be able to show it and prove it, right? Uh, so that's why the heavy emphasis on uh, that that piece of the story. And I Oh, yeah. So, then just like the key takeaways here, right? Zero trust, meaning uh, we want to be able to trust the action or we would we don't want to trust the network, right? This is where we're

moving away from VPNs and having identity security on a each one of those interactions. All right, I I think at this point uh, we we have we have explained these uh, a little bit in detail. Uh, you know, we have a bunch of time or a bunch of time being a few minutes uh, for any questions. Mhm. And there's a mic and I can run around with

the mic that I'm not using. And then we also have the uh, demo up here. Uh, if you if you if you want to access it, uh, play yourself, kind of test it out. Um, yes, go ahead. Test. Test, test. Thanks. Thanks for the talk. Uh, I love it. Identity-based access. Mhm. Uh, what's your thinking about protecting endpoints which originally, you know, have their own FQDNs and

then if you set up a reverse proxy, then these FQDNs get terminated by other certificates, whatever. So, all these client applications will will need to be revised, right? So, like connected to a database. Well, it it's not not that they need to be 100% revised. This is where Envoy is doing the heavy lifting, right? So, Envoy as the reverse proxy here, you do um, you do have

to tell the services how to trust Envoy. So, the connection is between the services or the resources that you want to you want to be put behind that reverse proxy and making sure that they can connect to Envoy. And then that is what's going to be that that entry point for the users to get to the resources. Mhm. Does that make sense? Yeah, I think it's only

mainly your story is mainly applicable to web apps. And and I don't think it's really about admins connecting with databases, you know, starting up their data studios and it's I guess it's I I I feel we are really still locked in with all these tooling not capable of connecting to reverse proxies. They they they assume direct connectivity. Make sense? So, I mean, in this particular use case

we were showing web apps, but Envoy is not limited to HTTP traffic, right? It it can very easily do TCP traffic in much a similar way. Do you want to do TCP only? Like if the point of all of this is modernizing how your endpoints are being controlled, then yeah, you you have to change some of the behaviors of your end users and of your less modern

internal endpoints, but you can put I used to work for an Istio company. We used to actually put Istio in front of databases for that specific reason cuz I wanted to have Envoy in front. I wanted to use Envoy as uh, authorization engine versus using Postgres' built-in RBA model or what what have you. So, um, yes, there is a little bit of a lift, but I think

the the win you get is higher than the pain point of I have to change my workflow somewhat. Totally agree. Thank you. Thank you. Thank you. Uh, thank you for the talk. Uh, I think my question is a bit of a follow-up on the same idea because essentially what you are trying to say is that by moving from VPN to this uh, gateway style access to pretty

much everything with an authorization layer that is injected there, maybe in OPA format, maybe in some other format, uh, it's a compelling story. But VPN in the end provides you L3 connectivity to a lot of places and you are trying to shoehorn it into L7 or L4 uh, Envoy capability. Is it always even possible? Uh, for example, if you are going to the TCP uh, strengthening of

the authorization with the database, right? You will still have to inject the JWT somewhere and it's not like TCP itself will allow you to support that. So, there is a bigger need of a shift then. Like there is still a component missing, no? Yeah. >> So, we we are we are okay. obviously we're trying to talk about CNCF and open source projects and everything else. We work

for a company that solves this problem at uh, enterprise scale to where it solves that particular or answers that particular question. Uh, but yes, when this is the I think both good and bad about uh, the open source community, right? Like all of the the knobs that you would want to solve the problem that you're talking about are either a little too hard or not there and

somebody's gatekeeping it behind enterprise feature A, B, C, etc. Uh, I don't know if Peter had a different Yeah, and I think you bring up a very good point of you also need that network connectivity. And so, with this, right? This is a little bit of a heavy-handed approach in order to control that access by using identities on every single endpoint. And so, this might this works

very well for like the human users, but maybe for the machine users, you're going to use something like Spiffy, right? You want to be able to enable that network connectivity, but you still want mTLS everywhere, right? And that that is the big piece that we're trying to get to here is just not having unencrypted endpoints out there that people can or people, machines, AIs, whatever, right? Can

start to probe and pull information in. And so, layering these technologies with other things like Spiffy where you can still hand in hand and kind of give you both the network connectivity security as well as the identity security for human users. Yeah, I think that makes sense to me as well. It basically doesn't make a case for VPN going away. It makes a case for VPN with

unauthorized access everywhere in the network going away, which I'm totally behind. So, >> Yeah, absolutely. >> Yeah, I don't think we were trying to say, "No, trash your VPN, do this." Uh, a VPN will always have uh, a place. Uh, that's I don't think that's going to go away. That's all entrenched in how a lot of big enterprises work, right? Mhm. Uh, Again, you have to still

modernize. It's no longer enough to just be like, "I have a VPN," right? I think that is ultimately what we're saying. I you have to think about the lateral piece of all of this. Cool. >> And I think we have only about a few seconds left. So, I think we're going to say, "Thank you. Appreciate everybody's attention." Thank you all.