Matt Billman - Agent Experience

In this session, Matt Billmann discusses how AI is changing the approach to web development, focusing on agent experience (AX). He explains that AX will shape the future of development, as non-traditional developers, such as domain experts, are now able to contribute to software creation alongside professional developers. The concept of building autonomous agents that work seamlessly within developer tooling is introduced, highlighting how these agents can facilitate development processes. Billmann emphasizes the need to design developer platforms with AX in mind, ensuring that agents have the necessary access, context, tools, and orchestration to function effectively. By doing so, organizations can empower more individuals to build software, ultimately transforming the landscape of web development.

Just to drop in hot, fun fact. Please welcome Matt Billmann. Fun fact, he vibe coded the entire presentation. So, all you can heart out. Um Matt Billmann is the founder of Netlify, an AI native platform where developers and agents collaborate. Anybody here aware of open claw? Remember what happened when it went viral? Moltbook? So, anybody who heard of Moltbook here? Uh basic Okay, perfect. Do you maybe

want Can you come up here and explain you short what it is? It's uh the favorite social media of my agent. Very good. Is he having a good time? >> [laughter] >> So, Matt will be sharing how AI is transforming the way how we build for the web. From agent experience AX to open and closed agent ecosystems. Agent experience, the discipline like UX and DX, building products

for agents and making them easy to access and a great experience. Thank you. Here we go. great to be here. Uh still a little jet lagged coming in from San Francisco last [snorts] morning. Um and uh I took the opportunity to uh for the first time ever vibe code my presentation entirely with like a mixture of cloud code and Netlify's agent runners and and open claw. Um

so, we'll see how it goes. Um I started Netlify more than a decade ago. We launched in March 2015 with the idea of helping to to build a better web and to enable the world's developers to create and unlock the power of the web. Um, but before I talk more of that, I'm I'm going to take a little detour. Um, >> [snorts] >> how many knows this

kind of car? It's like probably a very American phenomenon. Um, it's a a picture of the car that I drive. It's not my car. Didn't have a good picture of it. Um, and uh, I think I drive it because they managed to make something that serves the purpose of a minivan to ferry around your kids look like a an adventure machine. And sometimes I drive it up

to from San Francisco to to Lake Tahoe. It's like 3 and 1/2 hour drive if you don't have traffic where you can go skiing. Um, I'm very bad skier. I only skied for a few seasons. Um, and normally, um, if I had to go there, I would I would I wouldn't be able to like go skiing and then drive down in one stretch on the same day

just from like not being focused enough. But, um, this Rivian company that that that built my car launched this autonomous universal hands-free mode uh, just at the end of last year. And now I found that if I drive up to Tahoe, uh, go skiing, I can jump in the car, just take it to the large road, and then from there on I kind of like just have

to supervise the car, right? Like before that it could do like cruise control, but now I can kind of drive itself. I still have to keep an eye on it and and be present, but it means that I can take a much longer stretch in one go than and would be able to do um be before I had this like AI assisting me in in driving. And

then in San Francisco on normal days when I go to the office, I take another kind of of of autonomous car um a Waymo, that's a fully driverless car. And it's like very convenient. I can take I can plan on taking my first meeting in the day from the Waymo, right? Like I just jump in, there's no one, there's no driver, there's no distractions. Um I can

sit with my laptop there and work. And it's easy to imagine like when our own car starts reaching this level of full autonomy, how it will change not just that we can do more of what we're currently doing, it's easier to drive a longer stretch and so on. We can plan our lives in very different ways, right? Like I'll be able to take the car up to

Tahoe, go skiing, go to the après, have some drinks, have fun, and then just go sleep in the car on the way home, right? Like um and I think we've been thinking about these cars, the physical world changes slowly, right? Like so we've been thinking about these kind of self-driving cars for a decade now, and we've used to think about like once cars can really drive themselves

around, it'll lead us to rethink how we build cities, how like we won't need parking, we we will think very differently about commutes and where we go and so on. It'll it'll change the places we we live in. Now in the digital world, and especially in our world of of of development, we started sort of a similar journey where when Copilot came out, it was kind of

like cruise like advanced cruise control, right? Like you were still kind of like manually writing all the code, but you got these autocomplete suggestions from an AI that was sort of directly in the loop. we started moving from that to the idea of of AI assistance where you more just supervise these agents that are that are writing the code for you. You're no longer doing it yourself.

For us like for me the big tipping point was sort of like right at the end of 24 when lovable and bold came out and started really popularizing idea of of of live coding. Um later Claude code came out and started doing the same for for professional developers. Um and we have now have like our own agent run a product based on like Claude code, Codex CLI,

Gemini CLI to allow this form of like supervised development. And this kind of supervised development just like the car that can drive itself mostly let us do much more than we could do before. This lets us build much more than we Uh at Netlify we've seen our internal teams from HR uh build software to like this this is an example of like our recruiting dashboard entirely built

by one of our recruiters. Um recently our head of support rebuilt all of all of all of Zendesk inside our internal support tool. Um and it now like sort of replaced our 10-year-old support flow with with Zendesk with homegrown software with a team of like two developers and the rest of the team that builds and maintain that are are are not typical developers but have domain knowledge

and um I I direct contact with customers and so on and contribute to to to this. So we're starting to see this whole build versus buy equation fundamentally shift now that agents can help us build much more than we could possibly do And we're seeing it on the one hand with like these new builders that couldn't build before at all. And I think from the end of

last year we started seeing a similar shift happening where in my company now the most senior engineers, our distinguished engineers, our principal engineers rarely write code by hand at all anymore. And then earlier this year we had this like open claw moment where I think we started getting a glimpse of sort of a different level of autonomy, right? Like where the coding agents that sit in front

of us, it's right there on our our laptop. It feels more like how we're supervising and almost self-driving car. And then open claw still very rough, right? Like it's it's still not quite there for a lot of reasons in terms of using it for for for very real and sensitive things. But I think anyone that's tried it have gotten this sense of like this is what it

feels like when I just an an agent that lives in a computer that it's driving itself and and it can do things fully autonomously. I set up in my open claw on on my old work laptop in in my office. It's a picture of my claw. Um I'm going to do a quick test to see if if Embod is around. Let's see. See if if if it

can see y'all. Say cheese. Okay. I'll tell it to you. Here Here's a photo of the audience. >> Build a cool thank you page. Now, I sent this message to to this club I'd living on my old work laptop and we'll see what it what it does later and if it works at all. But this new sense of like suddenly having this kind of like virtual coworker

in the cloud that's going to start really change how how we think autonomous agents in in in the digital space. Now, the first thing that happened when I set up my open claw was that I gave it its own Gmail. So, it I could sign up for all kinds of services and so on. And a little after I did this, I got this from Google saying the

account might have been created by a computer program or bot and that's against our terms of service. And I was like, yeah, it's hard hard to argue against. It it is a bot, but it's my bot. It's like a friendly bot. It's not an evil spammer. And I was actually surprised when I appealed their decision of banning my bot account by writing that this is a bot,

but it's my bot. It's a good little open claw. And they immediately revoked that ban. Because it shows that they are aware that they actually want these agents as their customers and as their user base even if traditionally we've been trying to to to rip out bots. Internally at at at Netlify, we've started seeing that having the ability of creating these kind of autonomous agents that do

work in the background and so on have changed a a lot of how how we work. Um this is like an anonymized screenshot of our lead researcher that sits at our top of funnel where thousands and thousands of people come in every single day and look at the one that had signal of like starting to have more usage, doing something more interesting, and then sort of really

analyze like what what is this user doing on our platform in a very autonomous way. They can then like pass to um to our support team or our sales team, is there something we should be aware of around around that user? Before we would have bought a bunch of like go-to-market tooling to look at that funnel. Now we can build these agents that can just kind of

pick out what they should look at by themselves. And we've also seen this like massive shift in the landscape as software becomes something that everybody can potentially read and write and and as our space of developer tooling has really gone from like an addressable market of 17 million front-end to essentially a user base of anyone that's technical enough and they're connected enough to to use a spreadsheet.

And that led me to start fundamentally reshaping all of Netlify from the like end of '24 until now around this idea of AX or agent experience. How do we build our product for agents with the idea that like in a few years almost all the code on the web is going to be written by agents, whether it's professional developers that do it or whether it's other types

of builder that do it, and we won't succeed if we don't build a great platform for these agents to build on. UX, we all know, was a term that Don Norman coined back in the '90s as the holistic experience of a program. Used to think about like a product as sort of a checklist of features. And Don Norman and Jakob Nielsen started to really think about it

as like an and as a competitive advantage to build a better experience around the product including like the manual and the packaging and everything involved in it. In the early 2000s, Jeremiah Lee Cockrell coined this term DX or developer experience to differentiate platforms as not just like does it have an API, does it have a functionality, but as a more full-fledged way of thinking how is the

experience for developer to extend this platform and build on it. And then at the start of last year, I wrote this article about introducing AX, why agent experience matters. Sort of coining AX as the holistic experience an AI agent will have as the user of a product or platform. Again, really seeing from our first-row seat at Netlify this early sign of agents starting to build more and

more of the software on the And realizing that we had to build for these agents just that in the decade before we had been building for developers and had DX as our North Star. And I quickly saw the term resonate with our dev [snorts] with with other founders of developer tool companies, right? Like again, I think the first really strong agents we're seeing are all coding agents.

So, it's specifically been like the dev tool space where people have been really attentive around this idea of of of AX early on. Michael from WorkOS was quick to jump on it John Maeda from Microsoft the CDO of HubSpot was quickly out saying you can't have great DX if you don't have great AX. This guy Tobin tweeted that if I got a dollar for every time I

heard the term agent experience and SF I could buy Netlify and I don't know if that's because you heard it a lot or because he thinks we have a very bad valuation. But I'll take it anyway. So how how do we AX? What does it actually mean to build a great agent experience? The first thing to understand is that it's not about like a feature of protocol.

It's not about like adding MCP to your product and and then you're done. When we think about agents the simplest definition of an agent is really this core Loop. An agent is a loop over LLM calls. Those LLM calls depends on context for how good they take a decision. And then tools. And those tools are where agents interact with your product in one way or the other.

And this Loop takes some input prompt and then it keeps going until the LLM decides that it's either reached the goal or can't reach it. That's like the most basic definition of of of an autonomous agent and that's what we're building for when we're building agent experiences. And often when we do that and when we try to look at these agents using the products we built and

that we think should be very simple and intuitive to use we get this kind of experience and I'll demonstrate one of the times I got this experience later. But as we've built this practice at at at Netlify over the last year and a half of of building for agents there's like four pillars that's stood out to me of the way I think about it. The first one

is access. Can the agent actually Is it easy for the agent to access your product? Is it easy for it to get the right permissions? Are there places where the human needs to be in the loop or can we get the human out of the loop and just let the agent do stuff? The next pillar is context, which is really the whole discipline of context engineering that's

becoming more and more important for building for for for agents. If you have a great agent experience, people shouldn't have to think very much about the prompt they give the agent. If they have to do that, it's typically because the agent doesn't have the right context and specifically doesn't have the right context around your product. Tools is the third category. This is how we actually give intentional

tools designed for agents to interact with our products. And the last part is orchestration. Just like agents can come to our product and try to use it, can our product also call out to the agents that users work with and want to use and make them do stuff? Can human and agents collaborate within your products in a seamless manner? When we look at at at access, some

of the first patterns that became really popular, um we introduced early on this idea of like deploy site to identify and claim it later. Um without any friction from like first having to sign up to our service, and lots of other providers have have followed in that in in that path. Here's an example from from Neander Bay that introduced Instagress with the idea that you could just

import Instagress, install it from NPM, and then when you run your code that's all you need to do they'll make sure that there's like a lightweight Postgres database available that you can then go and claim and upgrade to sort of like a more like to one that sticks around and doesn't lose your data and so on if if it if it matters to you. But they want

to give the agent just the way of getting started before the users have made any intentional interaction with your product. We were just a part of Stripe's launch of projects. Um projects is a part of Stripe's CLI. And the idea there is that if you have a Stripe account and you're signed up then their CLI can provision and can let agents provision and and sign up for

and even pay for this whole swath of services. Um so if you have a Stripe account and and and you install projects and give it to Cloud Code you can tell it go build something on Netlify and the agent will be able to create a Netlify account and even upgrade to a pro account and and and do work on our platform and on many other platforms without

any without any friction of getting access. We've started at Netlify to sort of build a whole battery included sets of tools. We latest launched Netlify database and Netlify identity to sort of make sure that agents can just pick the stuff they need to build the kind of applications that our users want to build and and we'll make sure it's there. So if an agent again just imports

our Netlify database library and and write some code and push it we'll make sure that the agent has a database. It shouldn't have to have any friction in that process. And we recently launched netlify.ai. That's our website exclusively for So if you go there as a human it will just tell you to to to copy-paste a message to the agent. And if an agent goes to that

website, it will really just get a marked-out document that explains to it how to access Netlify. Like, how do you install our CLI? How do you install the relevant skills for Netlify so you know how to use our product? Um and how do you get the first user authenticated so you can start building? I think we'll see that pattern for a lot of companies in terms of

like building dedicated entry points The next big piece is context and and context engineering. And one big shift we are seeing, this is a screenshot of the docs for Vite, uh really popular front-end build tool. Um And again, really common pattern we're starting to see is this idea of content negotiation, where when you go to a human as as a human to the docs, you get a

website. But if you're an agent and you sent like typically the agents will actually send an accept header in the HTTP request that says they prefer markdown. And if they do, then we'll give them directly just markdown for their context, not a website, not no navigation, or anything. We do the same at as Netlify's docs, and we publish prompts that people can run to make their own

docs behave in the same way. MCP, everybody here is probably somewhat familiar with MCP. It's a very popular protocol, but like it's really important to think about the context in model context protocol. It is essentially a protocol for giving LLMs context about things. Some of the MCPs out there doesn't give the models any new tools. Context 7 is an example, it just gives documentation. Um And I

think um skill serves a a similar purpose of of getting context in place. in both cases, I think it makes sense to APIs is like endpoints you can do stuff with. Skills and MCP are actually like UIs for If you look at our API at Netlify, our open API spec has more than 100 endpoints for for all of the different services we we we offer. If we

took all of that and took some one of the many toolkits to create MCP servers based on open API spec, we would end up with like hundreds of tools for for an agent that would just pollute its context and make it make it make it confused and make it perform worse at every at every single prompt you would give it. Instead, we really thought about like our

MCP server not as a wrapper around our API, but as a different UI for agents. Because the UI for agents is context. That's how they see the world essentially, right? Like so our MCP exposes five tools that can do anything on our platform and that will gradually reveal the right context for for for interacting with our And whether you're building skills and MCPs, it's the same mental

model of like the context is the UI for agents. The next pillar is the actual tools we give to agents. And no matter if you intentionally get give the agents tools or not, agents are probably already using your product. This is like an early screenshot from Claude's computer use. Now we have Codex computer use, you have browser use, you have all these ways where where an agent

can try to interact with your product whether you want it to or not. So when I talk about agent experience again, it's not something you add to your product. Every product has an agent experience. Agents will try to use your product, no matter what you do. But, the question is is it good? Are they actually having a good experience? Are they having a really bad experience because

they're trying to get to your product through web interfaces of computer use? And sometimes you think the tools you have are good enough for them. I thought that in this CLI, right? Like we've had a CLI for a very long time. This is an example of sort of like the like of how the DX for our CLI used to look like um allowing a developer to deploy

a site to a URL in just a few seconds. And when I first took this CLI and then sat down with Claude code and just told try to create this Netlify project. It It started using the CLI and it got this first sort of like interactive menu in the CLI. Do you want to create a new project or do you want to link to an existing project?

And it couldn't handle the interactivity, so it just started trying to like do weird bash commands to get around it, pipe weird characters into the CLI, see if it could sidestep it. It started like trying random different commands. It did things and thought it could deploy and in the end it spent like a lot of tokens and it never deployed a project. And I thought our CLI

was like really simple to use. This was very much sort of my reaction to to seeing Claude trying to use our So, what did we do? The first We've made many changes since then, but the first little change was like same DX, same interactive menu, but we would add at the very beginning, you'll see a little line of text right before the interactive saying, "Like to create

and deploy in one go, use Netlify deploy." Again, think of the context as the UI for agents. Now, when Cloud Code this it would realize that it could use that command to do things without any interactivity. It would run that and deploy would now start succeeding without any friction in in if just two two tool calls. So, huge improvement um and really just from sitting down, observing

the agents use your product, and think of context as the UI and make your tools better for the agents. Last pillar is really orchestration. There's a lot of products that tries to force you to use their agents within their products, and that's kind of like slapped agents all over. Salesforce used to be my big example of this, and they've kind of caved, right? Like, you saw this

split between companies saying, "We need to be open to agents and let agents just use all of our product and orchestrate agents from within our product and orchestrate the agents people already use, not our agents." And you saw now Salesforce took a big turn and and announced headless Salesforce for that reason. we launched the agent runners as our orchestration layer where we took Cloud Code, Code XCEL,

and Gemini's ICEL. Again, we didn't build like a Netlify agent, but we looked at the agent our users are are getting value out of, and then we put a prompt box into our UI where you can prompt those agents to build stuff on Netlify without leaving You can both use those agents outside of Netlify now also inside Netlify and you think we'll see that pattern in a

lot of different product where we start orchestrating asynchronous agents to start working autonomously for us within our products. So, to recap AX in practice, access. Can agents actually use Can Can they actually get access to our product? I talked to a at DevTool founder that that wanted advice on on their agent experience and said we've put a lot of work into our CLI to give agents a

a great starting point. And then I went and looked at their CLI docs and the first part was before you can use before you can use the CLI you must configure it with a personal access token. That's kind of it, right? Like who knows where to get that? Like how is the agent going to get that personal access token? We have to be really mindful of like

how do we give the agent access to our products? How do we give it the context it needs to understand our products? Things of context as the UI for agents. Do we have the right tools that are really built for agents that we've tested with all the different agents that we run evals on? And do we have the right orchestration where agents can both use our product

and drive the workflow and where we can trigger agents in the other direction? And today AX is really critical for anyone in the infra and developer tool space because like it's very obvious that one of the first parts where autonomous agents are really getting real and powerful is in software development. But that's going to happen in all parts of our industry. So, tomorrow if your product is

not built for agents, it probably won't succeed. And that brings me back to our mission of enabling the world's developers to The beauty in what we're seeing now is that everybody is becoming a developer, that everybody can build web apps now, and that every company is in in a certain sense suddenly be becoming a dev tool company, is suddenly building tools that agents can build software on

top of rather than tools that give fully customize like fully baked solutions. you are now a developer even even if you don't write code. If you build software, you're a developer. And we are seeing like colleagues around developers transform from being stakeholders to being collaborators and seeing teams build software together with a small set of people that are on the team because they're technical and know how

to build software, and a larger group of people that are on the team because they have domain knowledge and can directly drive like what the value should look like. So, to bring you back to that initial story of cars and how they are going to reshape our cities over time when they start driving on their own, as agents become autonomous in in in our virtual world, and

our whole team become builders together, we'll need to reshape every part of of our developer platforms and and our virtual worlds of software that we are collaborating in and and living in. With that, let's let's refresh and see what happens. This was the work of of Embod. Thank you, everyone. >> [applause]