About this talk
In this talk, Nathan Handler, an infrastructure security engineer at Reddit, discusses the creation of a unified cloud inventory system using Cloud Query. He highlights the challenges of managing multiple cloud resources across different platforms such as AWS and GCP while ensuring data visibility and compliance. Previously, teams at Reddit struggled to accurately determine resource ownership and access due to outdated data and fragmented tools. By implementing Cloud Query, they achieved a centralized, queryable inventory that supports site reliability engineering, incident response, and governance across the organization. The speaker emphasizes the importance of maintaining current, accurate data to enhance reliability and operational efficiency.
Full transcript
testing. Testing. [music] Testing. Let's get it a little closer. >> Yep. Bend it for the in where the double thick area is. >> Double thick. >> Yeah. Yeah. That area is more like for you. Also might it go forward there. [laughter] >> I've tried to grow them. >> Yeah, right. What's up with that? >> Testing. [music] Testing. That sounds pretty good there. Let's just get this. >>
I don't know if they have a wrangler for questions. staff. >> But if you have a friend, you can ask him to pass around. >> Check. Check. Hey. Hey. Thank testing. Good morning everyone. We'd like to welcome Nathan Handler, staff infrastructure security engineer at Reddit for building a unite a unified cloud inventory for reliability. Lessons from using Cloud query. Take it away, Nathan. >> Thank you. Good
morning. So, as you said, my name is Nathan Handler. Currently, I work as an infrastructure security engineer at Reddit, but I've spent most of my career working as a site reliability engineer and pretty deep in the infrastructure and open-source trenches. I realize I'm standing between all of you and lunch currently. Um, I have some slides prepared. They definitely won't [music] take the full time and I'm happy
to take any questions afterwards there. But hopefully you walk away learning something new here. So, at Reddit, we used to get asked, "What resources does my team own?" or "Who can access this?" and we'd have to cobble together different answers using three, four, five different consoles and maybe a spreadsheet. And worst of all, the data would be old and quarters old and not really the most useful.
So, we got tired of that pretty quickly. And we decided to build a unified cloud inventory using cloud query and it's become the foundation for how we do reliability, engineering, incident response, and governance. In this talk, I'm going to share how we did it, the decisions we made along the way, and how this visibility now empowers SR product and GRC teams across the company. If you've ever
wished that you could just query your infrastructure like a database, this is for you. So, what's actually going on here? Modern SRE teams operate in environments that span multiple clouds. AWS, GCP, plus Kubernetes, identity systems, secrets managers, and a long tale of other services. Reliability depends on knowing what exists, how it's configured, and how it changes. But here's the problem. You really don't know where your data
is. And the data [music] is ultimately everywhere and nowhere. Different APIs, different consoles, different config stores. None of them speak the same language. Before we had a unified approach, we were stitching together APIs, normalizing formats by hand, and maintaining separate pipelines for every source. When someone needed a simple answer, what does my team own? They'd file a request or they'd get a point in time expert or
export that was stale before the meeting ended. Well, I like to think that I can use the AWS CLI just as well as the next person. It's not a oneanddone task. AWS is constantly adding new services. And Reddit is also constantly adding new cloud environments. I really don't want to be that person telling finance to go and use the CLI to try and determine which team spun
up that very expensive EC2 instance which would also require them to be granted the appropriate set of IM permissions. So why does this matter for reliability? First, reliability depends on understanding your estate. You can't improve what you can't see. Second, incident response needs the current state. When you're on call at 2 a.m., you need to know who owns a resource, what it's connected to, and how it's
configured. Third, governance and compliance requires auditability. You need to prove that resources meet your standards and that you can answer questions like who can access this sensitive data or which instances don't meet our compliance requirements. This data underpins thrift detection or sorry drift detection [music] dependency analysis and capacity reviews all core SRE workflows. So given these challenges, we set pretty clear goals for what we were looking
for. We wanted a single source of truth, one place where we could go to understand our cloud environment. We wanted that data to be queryable and auditable as well as not locked into proprietary tools. And critically, we wanted it to be useful across teams, not just for SRE, but product GRC and others who need to make decisions based on infrastructure data. This last bit was especially critical.
[music] If we wanted the data to be viewed as a value ad rather than a costly security data collection, we needed other teams involved. So we set out to adopt a framework that would let us extract and normalize cloud software as a service and infrastructure data into relational tables and build continuously updated inventory that provides a shared view of our We chose cloud query as our foundation
following our 2023 security breach. [music] We wanted to improve visibility into all of our cloud resources. That event highlighted how important it is to understand what exists in our environment, how it's configured, and who has access. Cloud Query allows us to pull the data from all of our cloud providers, not just AWS. And its wide selection of plugins along with the ability to create our own meant
that we could get a complete picture of our infrastructure in one place. Just as importantly, it let us expose that data in a way that was actually useful to teams beyond security. Our engineers didn't want to learn yet another tool. They wanted the data available in the same destinations that they were already using to analyze data from the services they're developing. And Cloud Query gave us a
way to consolidate everything into a single consistent framework. So the code snippet that you see on the screen currently is showing part of a cloud query configuration. The configuration takes the form of YAML [music] which fit in nicely with the rest of our Kubernetes resources. In this example, we're showing how you would configure a cloud query source. Um in this case specifically, [music] it's specifying the cloud
environment to collect the data from and specifying We're then in the spec there's a list of tables over there. In this case, we're [music] fetching a single table um for AWSS S3 buckets, but all of the different plugins have their own large collection of tables typically corresponding to the different resources supported by the cloud. So there are also tables for EC2 instances, IM roles, and RDS clusters.
And you could optionally pull in whatever ones are relevant to your situation. Finally, since we're an AWS organization with many member accounts, we're configuring cloud query to fetch this information from each account by assuming an IM role. So, this small configuration is going to go across each of our AWS accounts. It's going to enumerate all the S3 buckets, fetch information about each one of them, and then
ultimately um send it along further in the pipeline. I want to share how I like to describe cloud query to other engineering leaders. Infrastructure as code defines the desired state, what we want our environment to look like. Audit logs show the actions being taken, who did what and when. But there's a gap. What is the actual state right now? And Cloud Query completes that trifecta. It shows
you the actual state of your infrastructure and allows you to readily query and visualize it. You need all three desired state, actions, and actual state to have full visibility. Cloud query runs within our existing infrastructure as a collection of Kubernetes crown jobs. We have scheduled syncs that pull data from our cloud providers and write it to a queryable destination. In our case, this is BigQuery, a data
warehouse that teams were already familiar with and using. That means they can query cloud inventory data using the same SQL tools and dashboards that they're already using for other data. We also no longer have to grant these teams direct access to the underlying cloud environments, which simplifies permissions and [music] reduces risk. The data is automatically collected, normalized, and made available in a familiar format. Earlier we discussed
how to configure a cloud query source. However, data collection is only half the problem. You need to send the data to a destination for it to be queryable. The example on the screen is showing us how you would configure such [music] a destination. As I mentioned, we opted for BigQuery, but we could have just as easily gone for a Postgress database, S3, [music] Splunk, or even something
as basic and local as SQL light or a collection of CSV [music] files. Cloud Query's plug-in model was a big draw. We have around a dozen sources in use currently. AWS, GCP, Kubernetes, and Hashior Corp Vault are likely the ones receiving the most usage. But the cloud's more than just AWS and GCP. We have resources being deployed in countless other systems, and having no real way to
track them was proving problematic. The ability to create our own plugins meant we could extend coverage to internal systems or systems not supported by cloud query. This is how we were able to implement a custom Hashi Cororp vault plugin before cloud query ultimately developed their own. It also provides some protection against full vendor lockin. It's also worth noting um we use the Terraform source plugin which generates
a list of all resources being managed by code which also gives us some coverage for environments where we haven't written dedicated plugins. there. The query on the screen is showing um an example of listing all AWS S3 buckets being managed across our many Terraform repositories and modules. If you look closely, you'll notice that our wear clause is referencing a CQ sync time column. This is automatically inserted
by cloud query and it's a timestamp that represents when cloud query last fetched that particular resource. By leveraging this column, we can configure whether we want to query live or historical versions of our infrastructure. For example, we can revise the current query over here to find AWSS3 buckets that were managed at some point in the past, maybe when we were experienced a particular incident or to find
all buckets that have ever existed in our Terraform infrastructure. Um, all this becomes even more powerful when we compare it against what actually exists in the cloud as you'll see next. So going into this um one thing that we discovered was that we have a lot of cloud resources not being managed via infrastructure as code and cloud query was how we accomplished that. We were able to
use the Terraform source plugin to generate a list of all resources managed as code. And we took it a step further by comparing it against the resources that were being returned by the AWS or the GCP plug-in. The gap, which is resources that exist in the cloud but aren't in Terraform, represents drift or unttracked We can then follow up to get these cleaned up or imported into
our infrastructure as code. This is the kind of visibility that we simply did not have before. The query that you're currently looking at on the screen there is an example of identifying all GCP projects that exist [music] in our organization but are not managed by Terraform. This is likely due to legacy reasons or perhaps an engineer that opted to bypass our established processes and manually create the
project using the GCP console. Going back to the trifecta I mentioned earlier, we can combine this with audit logs to instantly detect any new projects getting created outside of Terraform to prevent regressions. So the idea being you can generate a point in time report of these projects with cloud query. spend some time remediating them and then leverage audit logs to detect future regressions or take it a
step further and utilize organization policies and other similar tools to flat out block people from doing the wrong thing. So, how do we actually use this inventory for reliability? We've built query patterns around three main areas. First, drift detection, comparing infrastructure as code to actual state as I just [music] described. Second, dependency analysis, understanding what depends on what, so we can assess blast radius during incidents or
plan changes safely. Third, change tracking over time, seeing how configurations have evolved, which helps with debugging and compliance. The ladder also allows us to track migrations to ensure that all services are using the latest recommended and supported patterns. All this is done with SQL. So SRRES can run ad hoc queries or we can codify them into dashboards and reports. Reddit uses a large number of AWS services.
EC2, IM, and S3 likely get the majority of our attention. Um, we leverage cloud query though to generate reports around who can access certain sensitive data, which of our compute instances meet [music] the compliance and security requirements and to generally keep track of the countless S3 buckets constantly being spun up. Cloudqu also makes it possible for us to keep tabs on all the services being used in
these environments and to prove to ourselves that we're not using unex on any unexpected ones. Questions that we can now answer are, what resources does my team own? Which resources are not configured in a way that meets our compliance standards? And are any of our resources configured in a way that makes them vulnerable to [music] this newly discovered attack vector. The best part is that the Wii
doing the answering [music] can be anyone in the company as most people are comfortable reviewing a curated dashboard even if they know nothing about interacting with the underlying cloud. Here's a bit of a concrete example. Domain takeover is a pretty common attack vector. If a Route 53 domain points at Fastly [music] CDN, but there's no corresponding Fastly service, an attacker can claim that Fastly configuration and serve
malicious content from your domain. We use Cloud Query to find exactly that. Route 53 domains pointing at Fastly that are not associated with the service. one query joining data from AWS and [music] Fastly. Without a unified inventory, you'd be manually correlating across consoles. And with Cloud Query, it's just a SQL query we can run, schedule, or alert on. When dealing with security incidents, we'll often try to
craft a query to determine the full scope. The results, they serve as a checklist as we go about remediation. By rerunning the query, we can later verify that the risk has been fully [music] mitigated. Um, I will acknowledge this query is a little longer than most of them there, but it is extremely powerful and has avoided a lot of relapses. We expose all this information through curated
dashboards. Teams get visibility into our running infrastructure in a safe and controlled manner. We can choose whether data should be made available to the entire company or just to a subset of users. The inventory data feeds into operational reviews. And we can correlate resource configuration with reliability metrics as well as understand capacity and ensure our security and compliance posture is visible. This isn't just for security. It's
for anyone who needs to understand the state of our infrastructure to make decisions. For incident response and on call investigations, the inventory is invaluable. When an alert fires, we need to know who owns the affected resource, configured. Before that often meant digging through multiple councils, examining the infrastructure as code configuration, or filing a request. Now we can do fast lookup of resource ownership configuration at the time
of the incident dependency mapping and this all just speeds up the triage time and reduces the time to resolution. The data is there when you need it. Operationalizing this required some care. We had to manage rate limits. Cloud APIs will often throttle you or block you when you exceed them. And we're pulling a lot of data. Scheduling also matters. How often do we run these sync jobs?
You have to really consider data freshness versus API [music] constraints. We also have to handle the fact that we have multiple accounts and we're running in multiple regions. So you need to fetch data from all these environments. Even the ones where you might not normally be deploying infrastructure, you still want to be able to prove to yourself that you're not deploying infrastructure into this region in Asia
that you're not expecting to have anything in. So these are the kinds of operational decisions you'll ultimately need to make. Se sync frequency, error handling, retries, and how to scale your footprint as it grows. It's all manageable, but it's not set in and forget. In our case, we ended up configuring a tiered data collection approach. For data that does not change often um and is costly to
collect, we're going to synchronize it less frequently and with a lower level of concurrency. So, it's going to fetch it slower. For data that's rapidly changing [music] and cheap to collect, we're going to collect it more often and with a higher concurrency. We want to get it as fast and often as possible. And then for data that falls in the middle, we adjust the frequency and the
data of the data collection and the concurrency accordingly. So maybe we fetch it at an intermediate rate with a medium level of concurrency there. So we still have the data, we still have it reasonably [music] fresh, but we're not hammering the API. We also work pretty closely with our account managers to try and make sure we're being [music] good partners there and establishing appropriate rate limits to
meet our needs. The inventory ultimately does not exist [music] in isolation. We're integrating it with our monitoring, alerting, and incident response workflows. When an alert fires, we can enrich it with inventory data. For example, if someone deletes an AWS S3 bucket, we can check the bucket's tags to determine its owner and criticality. We also connect to governance and compliance tooling so GRC can run their reports. The
goal is to make this data [music] part of the fabric of how we operate, not a separate system people have to remember to check. The data enrichment [music] also plays back into the trifecta I mentioned earlier. So those events of someone deleting an S3 bucket are coming from your audit logs. So this is an example of the two sources playing nicely together. We also integrate these into
other systems. So we have an automated offboarding flow. It will attempt to look at has this employee that's leaving the company access to any sensitive company secrets and determining sensitivity of the secrets that were accessed requires looking at audit logs for vault as well as additional metadata that we are using to enrich that information that's coming from cloud query. So few operational lessons. My team is quite
small, especially in relation to the rest of the company, and it's impossible for us to directly review every change being made. The challenge is how do we best utilize our limited resources to prevent to prevent, identify, and mitigate security issues. Automation like this inventory [music] is how we scale. But there's a lesson from our pre-cloud query days. We were collecting a lot of data, but it still
wasn't enough to earn a spot in the everyday toolbox of our engineers. As a result, outside of checkbox compliance, it never really provided much value and it was largely viewed as a cost. Data completeness and freshness matter. If engineers don't trust the data that you have or if they aren't finding it useful, they're not going to use it. This drove us to devote a large upfront effort
to onboarding most of our cloud environments to cloud query as well as trying to provide some white glove services to partner teams early on to help them prepare dashboards to visualize the data they cared about. We're also constantly tuning the tables we're collecting as new services are launching and re-evaluating our data freshness thresholds to try and provide the most up-to-date information to teams. Perhaps the most rewarding
outcome is the cross team impact. Product GRC and other groups can now explore and use the same data to inform their decisions. They're not waiting on us to run a report. They can answer their own questions. What resources does my team own? Which resources don't meet our compliance standards? Shared visibility is ultimately enabling faster decisions and stronger operational alignment as well as increased engineering autonomy. We're also
seeing a reduction in operational friction and an overall strengthening of our of our security posture. My team, best of all, is no longer playing the support role, which frees us up to focus on true security engineering and other important tasks while still empowering the rest of the organization to get the information that they need. So, a few things that we I'd like to emphasize here. First, completeness
matters. We learned that the hard way. When our data wasn't complete enough, it never made it into anyone's toolbox. It was checkbox compliance at best. Second, it's important that the data is [music] ending up where people are already working. You can't have it in a siloed sandbox environment that only security can access and people don't want to learn to use a new tool. Our engineers wanted SQL
in the same warehouse they were using for everything else. Third, the cloud is bigger than AWS and GCP plugins for things like Fastly, Terraform, Vault. That's how we got drift detection and the domain takeover query that we saw earlier. And fourth, running it in your existing pipelines. It's if it's a separate system people have to remember to check, it won't get used. You need to integrate it.
You need that trifecta of enrichment [music] of incorporating into the other dashboards. So the next time someone asks what does my team own or who can access this, we now have a solution. We can point them at a query instead of three councils and a style spreadsheet. And that's ultimately the shift that we [music] were able to drive there. So with that, I'm happy to take any
questions that people have here. Beautiful. [laughter] >> Uh, GRC, that's a new one to me. What What does that stand for? >> It's our It's our government um Sorry, I'm totally brain farting. Ask me in five minutes, but it's our it's our basically our compliance team that deals with the regulatory [music] bits there. >> Okay. Um [snorts] >> um so you mentioned that you run the this
on a cron on kubernetes. >> Yes. >> I assume that that can be abstracted from kubernetes. I could run it on a lambda function in a step machine or something of that nature or >> yeah um you can even run it locally. So there's a cloud query binary. It has a sync command that takes a configuration file as an argument. The key thing is you need to
be able to pass in whatever secrets you need for authentication. Um if you have a story for that you can run it anywhere. >> So you're using Kubernetes because that's what you have and everybody's >> Exactly. Got it. >> You mentioned that uh this helps with dependency analysis. >> In what way is cloud query able to provide that notion of dependent resources? >> That's a fair question.
Um that largely depends on how you think about resources and dependencies there. So one example that came up was we inventory all of our Kubernetes resources. So all the deployments, all of the pods, config maps there. So being able to see things like which pods or deployments are utilizing a particular config map like that's a form of a dependency that we were able to analyze in a
recent security incident or being able to see which of our IM roles have policies that reference a particular S3 bucket. So sort of that chaining of resources that [music] are referencing one another >> Got it. So it it it has more than just a superficial knowledge of there are 10 of these things for an AM role. It can say that this role provides access to these 12
other resources. >> It doesn't understand the concept of access. It is essentially doing a list call of all the IM roles and then [music] a get call on each of them. So it has the same data that the AWS API is returning. You have to know how how do I parse this IM policy and find references to an S3 bucket. >> Um one other quick thing I
do want to mention though related to dependencies. Um I'm not sure if any of you are using um backstage at all. Um we have it internally. That's a tool that's designed to be able to like codify these mappings of what service is consuming from which other ones, [music] what depends on another service. And we pull that in with cloud query as well as just another source. So
that's another area that we are collecting that information. >> And you mentioned there were some data that was costly to collect. >> Yes. What what type of data are we talking about and what sort of costs magnifiers are we talking about there? >> Fair. Fair. Uh rapidly changing resources tend to be the main bit there. So if you think of I don't know a Kubernetes cluster where
you're constantly spinning up and down pods. If you want to have truly real time information with no gaps or anything like that, you're going to have to perform this sync pretty much constantly. and perform a sync fast [music] enough that even if you have a pod that's only live for a minute that it's still [music] getting picked up by the sync. Um, that's a bit of an
extreme example there, but hopefully it conveys the point there that it's a [music] constant tuning battle of how fast do you need your sync to run and how often are you running it and then just resource cost there. You do also have some API cost. So if you're hitting certain cloud APIs there in certain environments, you are paying a fee just based on the cost for them
to serve that data. So if you're constantly hammering them, you could rack up a decent bill >> And [music] on the AWS side, it sounds like this is all querying the AWS APIs like the things we would see in the CLI. Is there any are there any hooks onto something like cloudt trailil so that it could go the opposite direction where cloudt trail would notify cloud query
that these changes are taking of effect that would seem cheaper >> it's okay if you don't know >> yeah I don't know offhand we are not currently utilizing that um mainly we want that [music] full point in time snapshot cloud trail is great it does not capture everything there and being able to get that complete snapshot is pretty valuable for us there for some of the initial
runs as well. It's very helpful to get that back fill. Um I yeah I I don't know off hand if they provide it. You could almost certainly though codify this yourself if you wanted to. um it'd just be a little hard to scope some of the tables because cloud query wants to fetch all the S3 buckets. It's hard to say only fetch this particular one. >> Ah,
last one. Um setting this up. I have a I have a sort of control tower style AWS account organization. Do you in [music] in that sort of scenario would you put CL is cloud query an inherent piece of every account or does it become its own account that then like the audit account that uh this sort of best practice with AWS now does is that where you
sort of put the cloud query components or do they need to live in each account that is being monitored? >> You don't need much in the accounts. [music] So there are a few different ways you can choose to architect it. The common one is you essentially create an AWS IM role in each of your accounts. Um within your management account, you give that role permission to assume
role all the other cloud query roles. And then within the member accounts, you make sure the cloud query role has permission to enumerate and read whatever resource you're trying to collect. you don't need anything else within those accounts. Um, so it's pretty lightweight [music] with regards to that. >> Cool. Thank you. Great. >> Yeah. No, thank you for those great >> Anyone else have anything there? All
right. Well, >> we have time. [laughter] We have time. >> Thank you, Nathan. >> Yeah. And if you think of anything else, my contact information is up there and I'll stick around for a few minutes as well. But thank you very much for coming. I hope you have a great rest of your conference.
