About this talk
This talk covers the integration of AI into applications, focusing on practical implementations rather than conceptual discussions. The speaker, Rik van Zanten, emphasizes the importance of agentic AI, which actively performs tasks rather than merely suggesting actions. He demonstrates how to create an AI assistant within a to-do application, explaining the different interaction modes, including background automation and guided user interactions. The speaker also addresses key considerations for building effective AI tools, such as tool definition, user feedback, and safety measures for potentially destructive actions. He discusses the use of frameworks like Vercel AI SDK and Pinia for managing AI tools and maintaining contextual awareness within applications. Overall, the session highlights the opportunities for enhancing user experiences through thoughtfully designed AI interactions.
Full transcript
[music] >> All right. Hello, hello. And we got sound. Great. Oh, let's dive right in. So, MCP is dead. Just kidding. Just kidding. Just kidding. Just kidding. Um hello, hello Amsterdam. Thank you for having me again. I got a clicker this time. So, I'm going to be walking all over the place, I think. Um before we dive in, I do want to do a very quick show
of hands to get that energy back up cuz we're right in that food coma and I could feel it in the room. First and foremost, who here dealt with AI? I mean, this is this is going to be Yep. Figured. Figured. All right. Who tried integrating some more LLM features into an app or project? Good amount. About What is that? 40%? Who was a bit tired? Who
was a bit tired of it? Okay. Oh god. That is >> [laughter] >> That is a lot more hands than I thought, to be honest. So, get off of X, get on to Blue Sky, you'll be fine. Take care of yourself. Um well, good news. This session is just going to be for you. Um it is not just another There AI will replace developers by Tuesday. Um
it is definitely not a product pitch pitch either. Um today we're talking practical AI, not the let's just have an open claw run my whole life, not the claw is a national security risk. That's so last week. Um feel about how AI however you want. Excited, skeptical, elevated, exhausted. I kind of feel a combination of all of them, to be honest. It's all valid. Uh personally though,
as a product designer, I have to admit there's also a lot of exciting opportunities here to do cool stuff with LLMs in your own products. >> [snorts] >> And that's what we're here to do, right? So, who am I? Uh we had the quick intro just now. I'm Rik van Zanten, Dutch born and raised product designer and software developer. Currently based in Brooklyn, New York. Um as
mentioned, I work on Directus data platform with a view studio for humans and a bunch of APIs for machines. Um next to that, I'm also a part-time professor at Parsons in New York. Where I teach typography and interaction. What that means is that I spent a lot of my time thinking about how people and machines interact with software. So, on that note, what do we do with
AI in Directus? Well, we spent the past year or so trying to get AI to be actually useful rather than just a gimmick. Uh not just summarize a bunch of stuff, not just generating some filler text, the usual, but actually help our users get their own job done. That exploration obviously quickly led us into agentic AI, which is a great buzzword. Um you've probably heard the term
floating around a lot. You know, it's everywhere nowadays. I mean, even if you use a little cursor or claw or whatever, it's in the bottom left, it says agent mode, whatever. But what it means kind of depends on who you ask. To me, just for the sake of this talk, AI chatting, you're just talking about stuff. Agent, agentic work, it is doing the tasks. It's doing the
work. Think about calling APIs, modifying data, using MCPs that you build yourself for your view dev tools. I mean, to do so, it is actually very, very similar as an MCP world. You use these things called uh to make these agents do things, you give them access to the various tools that they need either through MCP or directly. As we saw before, it's a function with a
name and a description, uh some input parameters, for example, here. And then that execu- execution flow within your app looks a little like this. It's fairly straightforward. The user prompts, the LLM sees what that prompt is, figures out what is the right tool to call for this, sends a request to call that to the app, and then the app returns the result to the model. That allows
the AI within your product to interact with the real world, both internal and external to your own product. Um whatever piece of functionality you previously made available in your UI or APIs, you can make available as a tool to LLMs as well. So, I really see it as a sort of third interface, right? Where we had just your regular user interface for users, your API interface for
well, programmatic interfacing. And then now is a sort of a third uh interfacing method here. Um MCPs, ACPs, A2A, there's all sorts of standards for it. Without tools, the LLM can only really talk about work and help you think about it. With tools, it can actually also do Libraries like Vercel, uh AI SDK, Tenstack AI now make this pattern fairly easy to implement. Uh honestly, it works
great for a lot of cases. So, the usual current pattern that you see for that is the user is clicking around your app, asks something in the little chat sidebar, the LLM then calls a server-side tool more than a minute. Um the server creates the result, and then the UI updates, hopefully automatically uh in your To make that a little bit more concrete, let's look at a
small example. The most overused demo app in the history of programming, a little to-do app, of course. You've seen it before. The user can just create a to-do. Slowly. You can check to-dos. It's magic. You can even delete them. Isn't that amazing? It's great. Uh big shout out to our friends working on Next.js UI. This was put together in absolute rapid rapid speed just be- because of
the great building blocks that this community provides. So, thank you for that. Um now, imagine this app with an AI assistant built in. So, for little bit of my background showing Uh we have an AI chat sidebar now. What should happen when my user tells it to create a new to-do? Well, there's kind of two options you can do. Option one is just do it. Um the
AI calls the server tool for creating a to-do. The server executes that tool, and the to-do now exists in the database on the server. Then on the client side, you can respond to that event. You can refresh the to-dos and just show it on the UI. Um it's fast, it's simple, it's direct, gets stuff done. For smaller, some obvious interactions, kind of like adding a to-do, it
could be good enough, and it's fairly cheap to implement. So, for example, that would look something like this. >> And there it is. Magic. Um that being said, it doesn't really help the user understand what was just happening, right? Cuz all of this stuff kind of happened behind the scenes. There's some distracting stuff happening in the sidebar when it's streaming the output. There's just a to-do that
magically shows up with no sort of indication. There's no feed forward, there's no feedback really happening Um so, it can be a little overwhelming and even a bit scary, especially for less technical users. Now, there's also a second option, which is instead of silently calling that API internally on the server side, you could have the LLM take multiple steps on the client side to just walk the
user through that path. So, for example, in this case, it could open that create modal on the user's behalf, fill in that title field, type it out, then click save on the user's behalf. Um at that point, your user will see that behavior happen in real time, and that really helps them with learning workflows and the discoverability of your app as well. Cuz now they both know
how to do it themselves, but they also know what the agent is doing on their behalf behind the scenes. So, let's take a quick look at what that could look like. So, this is the same example, but this time I'm asking it even more stroopwafels on the way home, but show me how to do it. So, instead of just do it, show me how, right? And then
it could open the modal, type it in, and then hit save. It's already a lot better for the end user now that we have this sort of step-by-step approach, but it can still be quite jarring as that AI will be flying through the interface. Right? It doesn't have to really search for the right button. It doesn't have to do that kind of stuff cuz you've programmatically told
it where it is. there is an opportunity here where you update your view component to artificially slow down what these interactions look like. Just to help your user out. Cuz at the end of the day, that's what we're doing here, right? Helping our users. Uh you could even take it one step further and show a virtual cursor that really explicitly points out like, okay, I'm going here,
I'm clicking this, I'm typing this out, etc. Uh I think there's a real opportunity here to modernize a lot of the onboarding experiences of our apps. So, you've probably seen this all over the place, right? Where you sign up for a new app, and then you have to go through the little box that says, okay, and then you go here, and then you put your avatar, and
then you go there, and blah blah blah. Usually, I find those a bit obnoxious cuz I'm just trying to get you know, just let me do it, and I'll figure it out. Um but this is a very fun middle ground to me, where you could just say, okay, we'll let you loose, but if you have a question, we can show you inline what the answer is to
your question, but only that, without forcing you through the whole UX flow. For example, it could look like this. So, once again, I'm asking it, show me how to add a new to-do in my little to-do app. But this time, I'm having it render a little cursor that just goes to the button, another little cursor that goes to the input, types it out with a bit of
a delay, and then click on save. Same job done, but this time it is very, very, very explicit to our end user, to the human in the loop here, what just happened. Um there's two interaction modes here, and they these two approaches really map nicely to the two different kinds of AI experiences that we've seen come out of the field recently. So, on the one hand, it's
a background agent. It works autonomously. You give it a task and it just goes and do it. import all the CSV files in this folder is something you just want to do on the background and not be bothered by. There's also that foreground chat, however, which is way more interactive. The user is right there. It's actively watching what the AI is doing. And it's often better to
show it very explicitly. So you have that sort of collaborative nature to it. Um both of the approaches are valid. There's no wrong, there's no right. And there's also no reason why you couldn't support both at the same time. By making all the options available, you can just let the model choose itself. In this example, if it has a create to-do tool available, and you tell it
to create a to-do, it will just use the create to-do tool. If you have a show me how to do it, and you have a couple of tools for in the UI with a good system prompt, more on that in a minute, um you can get it to go step-by-step through your user That was way longer of an intro than I intended, but here we go. So
how we built this actually? we found during our experimentations, uh shout-out to the funny guy, you know who you are, uh that the easiest way to reason about any of this stuff is to register the local tools inline in the view components where you want to use Then you want to manage those tools somewhere centralized, that's where Pinia comes in. Um and then lastly, pass all of
these tools to the LLM in whatever chat abstraction that you're using. So first and foremost, we'll just quickly put together a global tool store uh using Pinia. >> Um that creates a really nice safety net because it means that the AI only sees the tools that you have registered in a moment. And because we're registering from the components, if a button is invisible in your app, the
AI wouldn't be able to press it. don't worry, I don't expect you to remember any of this stuff. I'll I'll share the link to a repo after this with these implementations as well. Uh in this case, we have a fairly straightforward Pinia store, two little util functions to register and deregister a tool by name. And then that's that. Uh shallow ref cuz I don't really care about
tracking the the nested information here. So for that second step, registering the new tools inline, we just want to have a simple composable that we can use. Something like this where in a component, I can just say, "Okay, define a tool." Maybe some of you have ever experienced with like define shortcut in Next. It's just a great utility for doing keyboard shortcuts. It's kind of the same
idea here. I just want to have a define a tool for my AI in my component. Uh similar to the store, the definition of that could be very straightforward. It's basically just saying, "Whenever this is mounted, register it. Whenever it's unmounted, deregister it." Therefore, you have that magical kind of out of the box, right? The component unmounts, the tool disappears. So the AI is no longer confused
by it. Therefore, the AI can only ever use what is on the screen, which also helps with a lot of accountability and security problems there. Um then last but not least, we got to pass the tools to the LLM. This is a little bit more depending on your exact architecture and what libraries you use. Um in my little demo example, I'm using the Vercel AI SDK chat
abstraction cuz it seems to be somewhat of a standard-ish, however much that exists in this industry right now. Um but in this case, you know, I'm just passing all of the tools in the body of that request coming from my Pinia store. And then when a tool is being called, I just scan through it to see if it still exists and then execute it. Um that is
the basic plumbing that necessary for making these client tools accessible to the LLM. Um with that basic tool usage in place how do we make them good? Cuz that is crucially important, as you can imagine. So elevating these tools from just a gimmick that clicks around and reminds you to buy stroopwafels, um takes a lot of considerations, right? First and foremost, you got to pay attention to
the tool definition as it is effectively the API for the AI. So the same care you put into designing a good component with the good props, um it it applies here as well. So there's three major things to to think about. First and foremost, naming and granularity. This is a mistake we made day one. Um models perform way better if you have specific, well-defined tools rather than
one giant Swiss Army knife. Uh you might be inclined to make less but bigger tools to avoid overloading your context window if you've been experimenting with this. Uh but there's better solutions for that problem. Um at Directus right now, we're basically experimenting using a tool catalog approach on the back end. This is a bit out of scope for a Vue conference, I'm aware, but the idea there
is basically that you only ever give the LLM three different tools, uh a tool search tool, a tool describe tool, and a tool execution tool. And therefore, every other actual tool uh only exists on your server and not in the context window of the LLM. If you want to learn more about that, just find me after. I I feel like I'm one of two people in a
purple sweater, so that should be easy enough. Um second thing here, descriptions do the heavy lifting. We heard it earlier that there's an input schema that you can pass it to inform the LLM what the form is of the data that it needs to execute the However, in my experimentations, it's really the description that affects how it behaves way more than the input schema. And really treat
it as writing the docs for like a very very junior developer who takes everything at face value, right? It's like everything you tell it is the only thing it has ever known about the history of the world. So you got to make sure that your description is accurate. Um and last but not least, you got to make sure that the LLM knows what is going on on
the screen right now to make good suggestions about what tools to Uh for that, we can update our little Pinia store to also track the context. And what context is relevant really depends on your app. Uh in our case, we could uh um track the selected to-dos in the app so the the LLM knows which ones to use. If you're saying, "Okay, now delete my selected to-dos."
It now knows which ones you had selected. Feels obvious until you try it and it doesn't work. Um with that additional context that we then pass in, the model knows things about the state of your app. I'm actually kind of curious and I might have to do an experiment with this to see if we can combine the MCP idea with this where it can read it from
the actual state of Vue. Although, haven't tried that yet, but please do try it and let me know if that works. Um without that context about your app, the model is effectively flying blind, right? It has the tools, but it has no spatial awareness. It's It's like giving somebody a TV remote, but first putting a blindfold on. It's like, "Okay, I have the tool, but it doesn't
help. I don't know how to use it." Um so with these things in place, the AI can now call the tools in your front end. Great. Should it? Is that a good thing? Do we want that to just go wild? No. Um I don't think so, at least. There's definitely a spectrum of trust here that we have to think about and that we do think about a
lot. For example, it really depends on what you're doing as well. If I'm just reading some content that I have access to and not really doing anything else with it, assuming it's not private information, of course. It's relatively safe and we can just auto-execute it, right? Skip the step, skip an approval, that's fine. If it's a little bit more risky, but something you could still easily undo,
like archive some completed to-dos, as long as you can unarchive it, right? It's medium risky. You might decide to just go for it. You might decide to add a tool approval thing. However, once you have things with side effects or things that are destructive, like email my whole team a report of this thing, I definitely want to, you know, preview and accept it first. Um luckily, Vue
makes it quite easy to do so. Um when a tool call comes in that requires approval, we can pause that agentic loop and render a confirmation component. In this case, I just use a little pseudo show confirmation utility function. Um you can imagine what that implementation looks like. Notice how I'm returning an error string instead of some sort of structured object. And that is just because all
of the LLM input and output is just strings at the end of the day, right? So I'm using a sort of pseudo structure that could be anything. It could literally just be the string user declined. in my experience, that object shape with an error property seems to work the most reliable, but that is as of this week. We'll see what that looks like next this pattern also
lets users build that trust gradually, right? Cuz you can start with approvals for everything and then relax them a little bit as the confidence of the user grows. You see this in a lot of coding tools nowadays as well where you can say, you know, approve it once, always approve this one particular flavor of this tool call. You could do the same in your app. Um other
than that, to make your tools good, we have to talk a little bit about the system prompt. And to be honest, I could easily spend another 30 minutes on how to make a good system prompt and what that looks like. So I'll I'll try to keep it brief. Also because the big scary clock is going down fast. Um the system prompt is where you tell the model
sort of who it is and how to behave, what it can do, how it should do those tools. Think of it as a sort of onboarding document for your assistant. Um this is where, you know, I I don't like humanizing it too much, but it's the best analogy that we have. Think about it as a new hire. You wouldn't just drop them at a desk and be
like, "Here's the 40 different buttons, figure it out." Um you'd explain what the app is, what are the workflows, when when do you use which one? Same goes for a system and same goes for it is crucial to have a well-designed system tool here in a system prompt. Uh especially when your amount of tools grows, you have to make sure you instruct your AI uh to know
how to deal with it. So a minimal system prompt for my little demo app might look a little something like You know, we're saying, "You're a helpful assistant." Don't know how much that helps anymore. It was a big thing 3 months ago. Nowadays, it's kind of irrelevant. Uh the relevant bit is definitely the the third uh the second, third, and fourth paragraphs here saying, "Whenever the user
asks you to do something, create, edit, or delete, prefer using the UI so the user can see it and only do it directly when the user tells you to do it in the background." And that really gets that sort of handoff going here. That last one, always confirm before deleting, kind of gets to the point of make no mistakes. It you know, it's it's a suggestion, not
a rule, and you have to treat it as such. Right? So, I compare it myself to to sort of client-side form validation. You want to have it cuz you want to make sure that your user sees but you should not rely on it exclusively by any means cuz there will be a point, probably prompt number three, where it just goes off and does it. And if you
didn't build against it defensively, This type of prompt engineering is, to me, interface design. So, designers, myself included, spend decades designing interfaces, you know, of pixels and components, but now there's a new design service that is just English language. You're not writing a code, you're not writing component specs, you're writing behavioral specifications just in English prose. And the model will follow sometimes, usually, uh mostly nowadays, which
brings us to something very important is that the quality of the system prompt directly affects the quality of the UX of your product. As more users will start relying on these chat interfaces, we are in charge of making sure that the chat knows how to behave to help the user out. So, we have to make sure that we treat the system prompt as part of the product
design as much as the actual user interface of the Um writing a good system prompt, therefore, requires the same exact skills as designing a good interface as well. You have to have user empathy, you have to have that clarity of intent, and it takes a lot of iteration as well, which is kind of pain and suffering sometimes with this. Um but it is what All right. With
all of that knowledge in our back pocket, let's take a beat to talk about some of the tradeoffs of this approach here. Obviously, as with any new technology, this is a very, very new design space, and that comes with a lot of unknown unknowns. Um in the last couple years, we've seen some recurring problems emerge though that we should account for. Um that I just wanted to
go through here. First and foremost, tool explosion. So, when you have a complex app like the Directus Studio, instead of having three little tools like open a modal and put some text in, you might have dozens, if not hundreds, especially if you connect third-party MCP service into it, you might have thousands, and it gets a little crazy. Um just as humans, models will degrade if you give
them too many options to do. So, just for like human users, we have to we have to handle that appropriately. Second tradeoff here is latency and choreography. So, LLMs, as of right now, are still quite slow, and if you use the fast mode, it's just unaffordable. Um that also means that if you have it use a server tool in a foreground interaction, so the one where the
user is actively looking at the chat, you're creating a lot of visible latency. You're making people wait for the spinner to finish, and then something happens. In this guided approach that I just showed, you can kind of treat that little cursor that moves to the button as your loading state. So, while the LLM is doing the work under the hood, I can basically distract the user for
a second by moving that little cursor and typing in some text. And it becomes quite seamless. At that point, it doesn't feel slow cuz it feels like the thing is just doing work in the meantime, right? Um this is this is product design all this time, frankly. Um actually, I did a a talk a year or so ago about how to make your app feel fast even
if it isn't. It's the same thing over here. Um users are quite patient when they understand what's going on. They're not very patient if the UI just stutters for no reason and just sits there waiting. So, you have to choreograph this whole song and dance very carefully. Then third and foremost, third and foremost, that's new. Uh then third, the AI can now do stuff in your app.
And that also includes destructive operations. So, we have to think about the guardrails for any automation here. Confirmation steps for destructive actions, making sure that your app supports proper undo, uh making sure you scope what it can read and when through permissions cuz eventually the model will do something unexpected, and you have to design for Fourth on the list, determinism or lack thereof, rather. Uh the same
prompt won't always produce the same tool calls or the same results or even the same output, and your UI needs to handle how the model behaves at any given day. Um it might very well choose a different path than you expected at first. So, defensive programming applies double uh for this topic. Excuse me. And then last but not least, testing. And this is one I haven't quite
figured out myself yet, either. Um cuz how do you write unit tests or integration tests or any other sort of testing for a system that is so non-deterministic by design? Um the tools themselves that you make available are just functions at the end of the day, so you could just unit test those separately. That part is familiar and pretty straightforward. Uh the tricky part is really that
integration layer with the model itself. The model decides what tools to call, and you don't really know when and how it does So far, I think there's kind of two options here. One is snapshot testing based on mocked model responses. So, you run it once for real, you just check what it did, copy-paste that in as a test, go with it for a little bit. It works
to a certain extent. Uh definitely no guarantee that the same actions happen with the same input prompt. And as the model gets updated, the whole thing the whole thing might be gone. Um at present, the models update on such a breakneck pace that this is quite tricky to sort of keep up to date with uh in your app. Alternatively, you can go with more behavioral testing. So,
you don't assert against what tools were called in what order, but you assert on the end state. So, you could say a test just puts in the prompt, create a to-do called X, and then you wait until it's all done, and you just check if it exists. It's a lot more resilient cuz it's a lot more real, but also back to the determinism problem, right? If now
the LLM suddenly decides to do it slightly different one in every 10 goes, your test becomes flaky, and it's really hard to um rely on. A lot of issues with that. Um it will also be very slow to run in CI, which is then extra expensive. So, the uncomfortable truth here is that we're shipping software with a very probabilistic component. Um it feels a little new for
us right now, but it's kind of what we've been doing for a long time, right? We do it for humans all the all of the stuff that we do for human testing, we have to do here as well. All right. With 2 and 1/2 months on the clock, what do we take home from this? Well, first, the view component life cycle is actually a very natural fit
for some of these agentic tools that we saw registering deregistering. Uh the framework solves a lot of the scoping problems. Make sure you design the tools the way you design component APIs. Start with server tools cuz they're easier and cheaper to implement, and then add client tools where they're earned to keep. Chat is just the beginning. So, in my demo here, I'm using a chat interface, but
there's really no reason it has to be constrained to that. So, for example, you might right-click a row in a table and and have that trigger an LLM under the hood. Or there's a toolbar button to say auto-categorize the selected items. At the end of the day, the the Pinia store doesn't really care where the trigger comes from. The tools are registered, the context is there, so
you can trigger the prompt under the hood on the user's behalf even without the user seeing what prompt it is, and then use the output from that. Um the underlying plumbing for that setup is is the exact same. And last but not least, and if you ever feel the urge to, take it from me, do not build your own CMS, please. Please. Uh we're doing it, we're
slaving away on it so you don't have to, and take it from me, we've done it for a gosh-darn decade right now. Don't don't do it yourself. Do not do it yourself. And with that, all that's left to say is thank you very much. I'm Mayra van Zanten, you can find me on @mayravanzanten online, or like I said, one of the two people in the purple sweater
here later today. Uh my little demo app is the mayravanzanten/view-agentic-ai. Take a picture of it or ask me later. You can obviously find Directus on directus.io. It's early days in the industry, so if you mess around with my demo and you see stuff that it's like, "Oh, it'd be neat to try this out." Please do open a PR. This is all open source, that's all why we're
here. Uh I'd love your suggestions for improvements. Let's experiment together and use this as a testing ground. It'd be great. Um love to chat about what you're building as well, so hit me up after the fact. And other than that, I'll say enjoy the rest of the conference. Thank you.
