Lee Briggs - An AI Operating System

In this talk, Lee Briggs, the Director of Solutions Engineering at Tailscale, discusses the integration of AI into engineering practices within his team. He shares insights on identifying different modes of AI, including automation and reasoning flows, while emphasizing the importance of measuring the impact of AI on productivity. Briggs highlights the challenges faced when AI tools are introduced and the necessity to centralize AI usage through a product called Aperture, which helps manage and understand how AI is leveraged within workflows. He also illustrates how examining AI utilization data can inform decisions regarding resource allocation and improve team efficiency. Ultimately, this session presents AI not merely as a tool but as a system that enhances operational effectiveness in solutions engineering.

uh, the director of solution engineering at Tailscale, a secure network platform connecting remote teams, multi-cloud environments, AI workloads, and more. So, perhaps you all heard of Uber recently where they blew their whole token budget by the month April, I believe. We're in May now. The year lasts a little bit longer. Uh, so teams at the leading edge, they will find increased token spend and a lot of

code. Now, Lee and his team actually experienced several challenges themselves as well. So, they came up with a solution, decided to scale it and sell it to clients who were interested in the model as well. How to use AI as an operating system. So, what will you walk away with? Um, you will learn how to identify different modes of AI, from automation to deep reasoning flows, learn

how to measure impact, uncover inefficiencies, and more. Please welcome Lee Briggs. An AI operating system measuring real adoption in engineering teams. >> [applause] >> Thank you so much. Uh, I got my cheer squad down at the front as well, which is great. So, as it was introduced, my name is Lee Briggs. Uh, I lead the solutions engineering team at Tailscale. Uh, I started my career as a

system administrator, and then that turned into DevOps engineer, and then that became platform engineer. Um, and then around 7 or 8 years ago, I moved over into solutions engineering, which is instead of solving internal problems with infrastructure, I now solve your problems with infrastructure using Tailscale. And one of the things about solutions engineering is that it is incredibly high context work. You, you know, my team of

almost 20 solutions engineers are problem solvers by their very nature. They designed the team is designed to engage with customer problems that you need to solve. The mantra that we have in our solutions engineering team is there is nobody's going to care about Tailscale as a product if we cannot understand the problem you're trying to solve. And that's really, really high context work. It's really difficult for

engineers to context switch so rapidly between customer engagements and customer problems. Now, luckily a lot of those problems are very similar and a lot of the companies are trying to use Tailscale to solve those problems, but it is really, really high high context work and building technical trust and building, you know, building a relationship with people is part and parcel of the day-to-day job. If any of

that sounds interesting to you, by the way, we are hiring on my solutions engineering team. So, if anybody would like to chat about that, please come over to the Tailscale booth in the expo hall. We're hiring right now for for solutions engineers. I will say as well that while all this sounds very technical, it is a sales job and we are selling things. And so, I'm not

here to sell you Tailscale, but if you just sort of decide that you want to buy Tailscale at the end of this, then great. I'm doing my job well. All of a sudden, about two to three years ago, as we all know now, AI just arrived. And you know, it started out as an experimentation. It started out with people starting to make that high context work easier

for themselves. And it showed up in my team when everybody started asking for a ChatGPT subscription or a cloud subscription. And as a leader, the immediate thing that comes to mind is there's risk involved here. There are there are potential situations in which this could disrupt deals, it could disrupt the sales flow, it could disrupt our the customer's trust in us. Um I think everybody in this

room has been able to spot an AI-generated outbound email or a LinkedIn post or something along those lines. It's not hard to to spot those things. And the big fear that I had in my mind when I was kind of scaling the team from three engine solutions engineers to 20 solutions engineers is I don't want customers to think we are just forwarding and proxying AI-generated content to

customers. Because that creates a problem for me in that I could just hire an AI agent. Um up until about 6 months ago, they were cheaper than humans. Um and now they're not, but I don't really want to be in a position where I can't justify how AI is being used as a leader of a large team. So, I have to start think about how I respond

to that as a leader. Is it amplifying people? Is it making them better? Are they using it in the right way? And you know, more importantly, is it helping us as salespeople win? Is it helping us be better at what we do or is it just making us feel a little bit more productive? From a leadership perspective, what I see is a high amount of cost and

a lot of new tools and a lot of new things, um you know, swimming around. But I can't see or measure any impact, right? People will tell me that they feel more productive. And I think as as sales-focused individuals with a technical background, we desperately want to be more productive. We always want to do more. We always want to extend the amount of time in the day.

You know, we are um partly compensated on the success of what we do. And I think the opportunity that the team saw was like, "Okay, I can use AI, and I can turn my, you know, 10 deals that I'm doing this week into 20, 30 deals this week. It was just a really, really optimistic perspective. But again, for me as a leader, I don't really know what

that impact looks like and I don't know what success looks like. So, the first thing that you need to do as engineers, because I'm still from an engineering background, is like, okay, let's approach this from a scientific perspective. What do I need to measure? Where is AI actually being used right now in my organization? Is it being used for a thought partner? Are people asking it questions

and learning more and understanding more about what they do? Or is it being used, is it embedded in workflows that I didn't even know about? Um like if if we decided to turn off our ChatGPT subscription because the token cost was too high next week, how is that going to impact my team? At this point, I didn't know. I had no idea what was going to be

different about the day-to-day. I also wanted to understand how it's being used, which is back to that problem that I just just talked about earlier around like making sure that we don't lose customer trust just forwarding AI information to them. And then, most importantly for me is like, when is it being used? Like, when are we actually bringing these things into a day-to-day usage? And as I

started thinking about these things, I realized something that this isn't a an IDE that somebody wants to get a subscription to. This isn't a a new video recording tool. It's not a tool. When you start to think how you're measuring these things, you don't measure the impact of your IDE usage. You don't measure how productive that makes you instead of typing things out in notepad.text. You do

measure systems. You have to measure them because if you don't measure them, then you don't understand how they're how they're going to be used. And what what it what occurred to me is that AI is not a tool. It looks like a tool because it's a Claude code agent or it's a, you know, a ChatGPT window in the browser. It looks like a tool, but it isn't.

It's a system. And that's where we start to think about what what the hell we're going to do about dealing with a system. And as a leader of a large team, you're dealing with systems all day and every day. So, I had to take a few approaches. The first thing is I had to centralize all this stuff. And this was not just a problem for me, it

was a problem broadly at Tailscale. You've probably seen and heard stories about non-technical people vibe coding applications into production. That's not what the majority of people at Tailscale want to do. We're a security company. We can't just yeet a bunch of code into our into our solutions cuz people aren't going to be very happy about that. But we do want to be more productive. And so we

have to centralize it. We have to bring it into a place where we know what's happening. And, you know, like a lot of other organizations, we actually started solving our own problems first by building a product that we could use internally to solve this problem. So, we built a tool that we call Aperture, and I think the easiest way to see what that looks like is to

give you an example in real time of how my team is using Aperture right now. So, I have a pretty standard coding agent um that I can connect to. And you can see the configuration up here is how I now connect to my AI provider. I'm using a an interim gateway, which is Aperture, which is available at the AI address here. And I can open my coding

agent. And I ask it a question. And I hopefully it doesn't tell me what its name is cuz it shouldn't have a name. Um And if I head over to this log page here, you can see all of this information that's happening right now is being centralized through Aperture. I can see everything that's happening on a day-to-day basis through this system. I can see every tool call.

I can see every um interaction. I can start to understand what's actually happening. Now, for those privacy-focused folks, you might be a you know, scared by this this idea, but as a as a business organization, I need to understand what's happening on a day-to-day basis. So, centralizing all this stuff through aperture without having to hand out API keys, without having to hand out a bunch of different

authentication methods, I know what's I'm now centralized this and I know in my AI usage within my organization. And I can start to get an understanding from this and get insights into what's actually happening, which I'll talk about in a few moments. you provide all of your authentication for the actual AI LLMs through aperture. You no longer need to hand out credentials to to everybody, and the

Tailscale client is the thing that actually decides the thing that figures out who you are because I logged in with my identity provider when I started this up, and I have a Tailscale client on my laptop here, and I'm logged in, and it is aware of who I am and what I'm able to do. And this centralization was step one in starting to understand how we manage

and operate a system, right? Like as soon as we started to bring everything into a centralized place, now I'm starting to understand how these things are affecting the day-to-day within my different workflows. The next step is to start understanding the data and the metrics and the insights that we have in these systems, right? Um so, you can see there from aperture, which I'll go back to in

a few moments, we have now have a mountain of data from day-to-day usage within the solutions engineering team. We have a from our engineering team, who is also leveraging AI agent decoding mechanisms for different parts of their workflows. And once you have those insights, the data is very raw. Like I think as as um um engineers, we've all seen a huge um you know, monitoring stack or

something like that with lots of data and lots of metrics in it in them. And and and that can provide many insights. So I can see a breakdown of the different metrics by model. Um I can understand how many of those um how many of those models are being used for input tokens, output tokens, cache tokens, and reasoning tokens. And each one of those metrics without any

context can be super helpful and can be super useful. But without context, it's not always super useful. Um so you can see here um unsurprisingly um to I think many people who are doing agentic coding right now, the number one use model within my team right now is Claude Opus 4.6. Um I think 4.7 is just about uh out the door or came out very very very

recently. Um but the the majority of the work that my team is doing is with Claude Opus 4.6. Now, if you have used Claude Opus 4.6, you'll also know it's very expensive when you compare it to something like Sonnet or Haiku or something else out of the Anthropic model. And I look at these numbers and I see, you know, 1.3 million input tokens, 8 million cached tokens.

Now, again, I have the context of what that means, but to a um to a layman, that might not be immediately obvious what's going on here, right? So the other thing that I can start to understand is which members of my team are doing the most with AI right now. Which members of my team and I have um you know, I've blocked the I blocked the names

out so that they can remain anonymous, but there is somebody at the top there uh and you can see the total number of tokens consumed by that is orders of magnitude higher than all of the rest of the members of my team. That individual is my strongest performer. And he's already figured out how he can leverage AI on a day-to-day basis to automate things. Now, you see

immediately and and I'm going to bleed into the context here, the total number of tokens that he's using is, you know, 841 million tokens. And of those, the vast majority of cached tokens with the with the AI agents that he's using. I Once we started centralizing this and understanding that, I sat that individual down. I'm like, "Okay, can you explain to me how you're using AI?" Because

then I needed to understand the context. I started needing to understand what is happening with this metrics and data. And what I discovered was really remarkable. Those eight 800 million cached tokens, this individual had automated every single thing he hated doing. He just completely automated out of his life. He no longer He would wake up every day. He would kick off some AI agents that did all

of the manual laborious tasks that he really loved doing. This is a high-context individual, a high-performing individual, and he wants to get more time in his day and not dealing with a bunch of like shuffling things around and a bunch of like manual steps that he has to do every single day as part of our responsibility as solutions engineers. And that's why the number of cached tokens

are so high. Like he's doing repeatable work over and over over again all the time. And then you look at the number of reasoning tokens that you use here, and I had to specifically craft a a bunch of prompts and a bunch of conversations with the LLMs to start generating reasoning tokens. I was using those reasoning tokens and me talking to the AI or talking to the

agent to build this presentation. I had to think about things. I had to build a thought partner. I had to bounce ideas off the AI like, "Hey, does this work? Is this working?" You know, 5 years ago, I probably would have done that with a member of the team, but now I can do it with a thought partner in in ChatGPT. And so, those reasoning tokens are

telling me, and I knew because I obviously have the context, that this particular individual is using a genetic AI to enhance their thought process to enhance their day-to-day. And these are the sorts of insights that you cannot get without um context and without centralization. So, what did this look like? So, before I had multiple individuals coming to me and say, "Hey, can I get a ChatGPT subscription?

I want to do this thing." now I've gone to an identity-based access model. They log into the Tailscale account that they use every single day. They have access to all of the tools and all of the the models that they need to do their job. And I've now controlled the adoption of AI in a way that makes me feel less risk as a Um I can now

understand the usage tracking. Uh before I had the usage tracking, right? I'm like, "Okay, this individual is doing this many things. This individual is doing this many things." Without the context of understanding what that looked like, I had no understanding about the workflows. I had no understanding about the day-to-day. And these are the kind of things whether you're a leader you need to understand, but as an

individual contributor, being able to take that to your boss and say, "Hey, I get it. I used $250,000 worth of tokens this year, but look how much more effective I was at my job." Right? Those are the kind of things that are going to get you promoted as an individual contributor. And as a leader, they're the kind of things that are going to get you a whole

bunch of budget next year to make your life easier. And I can measure that real impact on a day-to-day basis. The before we centralize all this stuff, we had model hype. Oh, wow, Claude Opus 4.6 is out. That's obviously going to be better, right? I'm going to divert all of my workflows to Claude Opus 4.6. What we started to do as a team is I said to

the the high-performance individual that I mentioned earlier with all those cash tokens, I said, "Hey, if this is repeatable work, do you really need to use Claude Opus for that? It costs like this many dollars per token. And if it's repeatable and straightforward work, one of the cheaper, more cost-effective models would likely be a better fit for that." I can now route workflows to specific models. I

can make decisions about where I'm using the budget that I have in a much more realistic way. That's just That's just a spending thing. I think most engineers have used some sort of like cost analysis tool that will tell you, "Okay, you're using this much money on this thing and this much money on this thing." But very rarely do you get insights about how to save that

money. And this is what I got out of centralizing all this workflow stuff. And then finally, um you know, before I understood the token spend because I get the bill every month in my inbox, um but now I'm getting operational insight out of that bill. And I'm starting to really make better organizational decisions. I am one of the few people at Tailscale that is able to justify

more head count based on our AI usage. We need more solutions engineers to make our customers happier. And I think the the narrative, whether it's real or not, is that AI is going to reduce the amount of work. You can't put an AI on a Zoom call with a customer. But what you can do is you can make those solutions engineers more effective when they are on

that Zoom call and make them and scale them better. So I'm one of the few organizations at Tailscale that can justify increased head count because I now measure and understand how AI is having an impact on my solutions engineers' day-to-day work. I believe we've reached a point at Tailscale in the solutions engineering team where this is not an experiment anymore. Um you know, this is no longer

And because we've started to make Aperture available to other companies and we're selling it as a product, we talked to hundreds of customers about this. You would be shocked at how popular this was when we announced it. And all those companies that said, "Yeah, we're still in our infancy when it comes to AI usage. We're just sort of tying around with it and experimenting with it." I

feel like we and the solutions engineering team have got to the point we're not experimenting with AI anymore. I think we've fully operationalized it. Like we're at the point now where I know when a new when a new solutions engineer joins the team, I can actually help them get started with a bunch of really, really helpful AI in in in invocations and a bunch of agents. Obviously,

once I saw those cash tokens with my high-performance individual automating all this boring stuff every day, what's the first thing I'm going to do as a leader? Well, we need everybody to do this, right? Like let's get this into everybody's hands. And I can now do that. I can actually make my solutions engineers more effective quicker than I could 2 years ago. I think that one of

the more important things that's happening right now is that AI is changing the way leaders think. And the sound bites that you hear from earnings calls is, "I need 20% less engineers, or I need 40% less engineers." Or if I'm Meta, "I'm going to spend a trillion dollars on inference, and I'm going to lay off half of my workforce." Those are the scary things that you hear.

But for me, as somebody who needs more people, who needs to hire more, and also operationalizes AI, it it means that I'm now no longer just managing individuals, and I'm no longer just a people manager. Like my day-to-day is not all one-to-ones and asking how people are doing and the way they're feeling. I'm now looking at this from the perspective of this is a system. The AI

is embedded in the system. We're managing all these different workflows, all these different dependencies, and all these different processes. And it makes me feel better as a leader knowing that I can bring something into my organization like this, centralize it in a way that makes sense, and start to distribute it to everybody in the team in a fair and equitable manner so that they can be better

at their jobs and they can get more time back in their day. What I'd really like to get to is that everybody leaves at noon on a Friday and goes and a couple of drinks or goes and does something that they enjoy. That's the ideal world and I'd love to be able to attribute that to AI because I believe that happy people are good, effective, hard-working people,

and AI is making my team happier. It's making them more productive. It's making them more effective. And this is all a part of the workflow and management that I have to think about as a leader. So, that's it. Um I hope everybody got something that they can take away from this as a as a a thought experiment in how you can embed AI into different workflows. If

you are interested in learning more about Aperture or Tailscale as a product, we do have a booth in the expo hall with a bunch of wonderful people who can help you, but I really appreciate everybody's time listening to me talk about how what I've learned from AI, and thank you very