Ballroom DE Friday Mar. 06 - SCaLE 23x

In this talk, Stormy explores the impact of AI on open source software, sparking a conversation about whether AI is detrimental to the open source community. She discusses personal experiences with AI tools, noting how AI-generated code can often be subpar, leading to confusion and frustration among developers. The speaker presents various viewpoints on why some developers hesitate to contribute AI-generated code, citing issues of trust and understanding. The session touches on statistics showing that while many developers use AI tools, there is a growing skepticism about their reliability. Stormy also highlights how AI may lower barriers to entry for new developers but raises questions about the quality of contributions and the evolving landscape of open source software as AI becomes more integrated. The discussion concludes with thoughts on the future role of AI in open source coding practices.

Mhm. >> Mhm. >> That's what we're going to do. And I have my water. All right, all right. Can anyone hear me? Hello, hello. Welcome to the open source AI track at Scale 2023. Your safety and comfort is important to us. So, if you feel unsafe at any time, please call our safety number posted around the venue. Our next talk here today is with Stormy. we're going

to we're going to have her talk about is AI killing open source software. Take it away, Stormy. >> Maybe you can hear me now. So, I am super excited to be here back at Scale, um even despite the loud noise that just went off. Um It's been a It's been a while since I've been here, but I've made a lot of friends and a lot of family

here. I've had some great conversations. I see some old friends in the audience. I see people that I hope are new friends. And what I really hope in this presentation is I start a conversation. I do have to share a story about coming to Scale. Um when I used to trick my kids into giving me extra hugs when I left for a business trip, I'd be like,

"I'm going. I'm going to be gone for 4 days. I'm not going to get any hugs. I need lots of hugs." And then I brought my son to Scale. And he was like, "Mom, you get lots of So, I I love being here. I love talking to you all. So, I have a confession to make. Um I actually tried to use AI to create my slides. Uh

it took me about 10 times longer than if I had created them myself. Um and it was not a repeatable process in the whole time. Um and it it created artifacts that weren't meant to be shared and collaborated on, which I found interesting when trying to give a talk about open source. Like Notebook LM actually had the best slides, uh but they were all just images. Like

I couldn't like change the typos. I couldn't change my plan. I couldn't generate a new slide in the same template. Um so, I feel like I got a little bit of the feeling of how it feels to work with some of the AI tools when coding and what it feels like to get some of those PR's full of slop because I often felt like that when I

was asking for a new slide, I got a PR full of slop. Um I did and the attribution's a bit wonky. Um I did check all my facts. I'm pretty sure they're right. Um but I put all of the articles that I believe I referenced in the slide on a LinkedIn post. Um and if you're trying to find a particular stat that I referenced in the slides,

the best tool that I found is take that whole list of of URLs and dump it into Notebook LM, a Google Notebook LM, and ask it questions and it will tell you what article it came from. So, now, on to is AI killing open source software? So, what I thought was happening is I thought when you used AI to generate some code, I thought one of two

things would happen. I thought you would not contribute it because either you would say, "I generated that code in that solution in like 30 seconds. So, it's really not worth sharing with anyone uh because anyone can generate it when they want." Or I thought they would say, "This isn't my code. Like, I don't know how it works. I didn't write it. How can I put my name

on it and contribute it upstream?" Turns out, I was both right and wrong. I was right for a few people, probably people sitting in this audience, people probably that have been in open source software for a long time. Uh they either don't contribute it because they don't think it's worthwhile or they don't understand it. Um I actually have a coworker who has a whole bunch of really

useful scripts that he wrote in Python, I think, and he wanted to convert them to to Rust or something else and he used an AI tool to do it and he's like, "But I can't put these on my repo cuz I don't really understand this code very well and I don't want to say it's mine and I don't want people to to trust it when I don't

even know what it does." But the majority of the world went, "Wow, I can contribute something really meaningful in 30 seconds. Like, I can solve all your problems. You said you have all these issues, I'll just solve them with my AI tool and I'll submit a pull request. Look how useful I am. Look at what a great contributor I am." How many people here have gotten received

a pull request full of slop? Quite a few people. And then the fact that they they submitted then they didn't understand it, which was not a problem to them when they were submitting it, but when they get asked to maintain it, it's then a problem and they're like, "Not mine. I didn't write it." And that's actually a really interesting phenomenon I'm seeing with AI agents. All those

AI agents that are out there that you read about in the news, somebody created them. And somebody paid for the tokens for them to run. And yet somehow when the article comes out, it doesn't say, "Stormy released a rogue AI agent in the world." It says, "Rogue AI agent did this." As if they don't have no one had responsibility for it. So, before I go on, um

I really want this to be a conversation not just in this room, but I want it to be a conversation that you take throughout the rest of Scale and that you take back to your colleagues and to your workplaces because I think right now we need to solve this problem. AI is fundamentally changing the world and it's very definitely changing how we develop software and how we

work at open source. So, I would love you for you to turn to one of the people next to you and share, do you think AI is killing open source software? You don't have to say why, but maybe in 30 seconds, yes or no and your top theme. All right? Turn to the person next to you, maybe introduce yourself and say, "Do you think AI is killing

open source software?" All right. Let's come back. I have stats to share that might All right, let's come back. I have stats to share that might change your mind by the end of the talk one way or the other or might make you think about it different. But, show of hands. How many people's neighbor thought that Maybe about a quarter. And how many people think it's definitely

not killing open source software, for sure not? Awesome. I think we all need to work together to figure out how it is going to fit. So, one of the examples I wanted to share is curl. If you haven't heard of it, I'm sure you've used it. It's like in billions of devices around the world. And they had a a bounty program. They paid a bounty for security

fixes. Um and they only had seven maintainers on the project. So, small project, important project, and they were trying to encourage people to help them support the project and fix security problems. And they were getting I I say seven here, but it was seven in 16 hours. So, there was one stretch of period I think was just too much for Daniel and he got seven bug submissions,

none of which were actually a a solving a security problem. And they were getting hundreds of week a week. Oh, and I forgot the the numbers. Each each um each pull request took like three people to review it from and it took them from half an hour to like three hours to review it. So, each pull request was taking like two and a half to 10 hours

of people time to review and then like these seven were not anything they were going to accept. So, it was a huge drain on these seven people of none of them were full time on the So, they they terminated their bug bounty program. Their bug bounty program that had been going for years um that had paid out in the tens of thousands over those years to different

maintainers around the world uh contributors around the world um had had to be shut down because the the financial incentive made it very uh lucrative for people to submit patches quickly, pull requests in hope of receiving that money. So, I think this is an extreme example because there was money involved, but I think it is also happening at other levels. And there's a paradox here. Sorry, this

I I demand that through another AI tool this morning and it did something really funky uh with my slides now that I'm looking at them. 94 84% of developers admit that they're using an AI tool. Oh, my slide My notes are gone off. You guys are okay. Um 84% of developers are using AI tools. But the trust is falling in them. So, even though they're using them,

they're getting more and more wary of the results that are coming out, but they're not using them any less. They're continuing We're continuing to use them more and more. I think partially because they're really useful in some cases and partially because we're really curious about how to use them, um but we're getting more skeptical of the results that they put out. And the thing that people are

saying is that it's almost right. And if if you just want an experiment, try to run your slides through through through an AI. Like, they're almost perfect and then there's like a typo and then you go try to fix the typo and the whole thing falls apart. Um so, the a lot of the things that come back look good at first glance and then don't work. Uh

I was talking to someone yesterday who's actually in a a presentation I was giving and he said internally in their company, they have no AI guidelines and people are using AI to generate code and then they're submitting it and they don't even know AI was involved until something goes wrong. Something goes funky. And then they go and they scrutinize the code a little more carefully and they

realize it's kind of strange and it's kind of weird. Um so sometimes a very subtle nuance to it when it goes wrong. So 66% of developers are frustrated that's almost right, and 45% say it takes longer to debug the AI code than it takes to debug code written by a human. um they they did a study on the productivity, and what I thought was really interesting in

this study is they asked a bunch of people does AI make developers more productive? And everybody said it made developers more productive. But it was divided into the developers themselves who were the most pessimistic about it. Um then I think there was managers or some type of, you know, role like that. And then there was economists, and the economists were the 39%. The economists were super bullish

on the fact that AI was going to make But if you take an experienced developer, not a new developer, an experienced developer, it actually makes them 19% slower. Um it's cuz they have to slow down to review the code. So if you look at the overall life cycle, they're actually producing code more slowly. Now I think it's really interesting when you bring up um is AI killing

software developer jobs? Is AI killing open source software? Most of us got into this because we like writing code, um you know, the actual coding of it. And it seems like that's the part that the AI tools do the best. Um they do it the fastest. And so then we are doing the other parts of the job, the parts that we might have called product management before,

or the parts that we might have called QA, um or product gathering, or you know, going out and talking to customers. So it seems like it's doing the part of the job that was most enjoyable before. It's also creating a lot of technical debt. Um so when you look at and all of these are based on studies which, I apologize, are not referenced on the slide, but

they are in my list of URLs on LinkedIn. Um it's introducing a lot of technical debt into the code. Um so overall issues, there's like 1.7 times more issues. It makes the code harder to read um and it has logic flaws often in And all of this is coming like I I think AI is is fascinating, it's fun, it's changing the world. I'm super excited about where

we're going. But it is changing open source software at a time when maintainer debt and maintainer morale is is particularly low. There's been a lot of studies, a lot of surveys about maintainers and how there's a few of them maintaining software for a lot of people in a lot of the world. Um so I think this was from a TideLift study. Um 60% of developers are unpaid.

60% have considered quitting. And 44% saying they're citing burnout. And so we're introducing this world of AI with all of these like slop pull requests and all of this to the people that are already maxed out. So it's not like they're not excited about AI. It's not like they couldn't figure out what to do with it if they had time to consider it. It's the fact that

they already like have a full-time job, are trying to maintain this project that is super critical for the world, and the world is changing, and we haven't given them any time to figure out how to deal with it or to learn these tools themselves before they have to set policies. How many people can I kind of commiserate with that feeling? I feel it just at work. You're

supposed to use AI, but how? So what some projects have done is just create new governance. New governance to deal with this AI stuff. It's the tool we've had for a long time. You can set a lot of rules and a lot of ways of best practice um by setting governance. So some of them just outright ban it. Um they're a little worried about or a lot

worried about um the providence of the code. Where did it come from? Uh what license did the you know, if it's if suggesting a piece of code where did it learn how to code like that is it did it copy it from somewhere else what license did that original code have so they're worried about the copyright they're worried about the quality which I talked about a little

bit and they're worried about the ethics I even saw someone um so I was talking to someone who said that they got a huge pull request I'm going to talk about a best practice one of the projects actually just constrained the number of lines that can be in your first pull request like if we don't know you your first pull request has to be short because anyone

who's asked a chatbot for code for anything knows they're very verbose and then it just makes it more hard to make sure you understand it and that you know what it's doing so the first pull so I was talking to someone at a project and he said someone rewrote a substantial part of the code that needed to be rewritten like it was it had known performance issues

and they rewrote it and they submitted this huge pull and they were very open and honest that they had used AI to do it but they said you know I'm comfortable I understand it it works here you go and the project's response was well why don't you just give us the prompt that you used and we'll recreate the code that you submitted and I thought that was

really interesting um so the code was no longer the valuable part of it it was like how did you create it um and one of the reasonings for that is that if you used AI to create that code is it copyrightable at all can we put a license on it um so there's some questions to be answered what I found really funny is when you ask developers

if they used AI 95.2% of them said that they used AI if you ask them if they disclosed that in the pull request that they did or in the read me for their project only 29.5% of them admitted to using AI when they submitted that pull request so that makes it even harder to set a policy about how what you'll do around AI coding tools. And you

can't ask AI if it wrote it. Um I gave it a blog post from like 15 years ago and I said, "Did you write this?" And it said, "Yes, I wrote that." And I was like, "No, I wrote that." "Oh, I'm sorry. You're right. I'm wrong." I also thought it was really interesting whether people were transparent about their tool or not depended on which tool they used.

Um so, 80% of Claude users um disclosed that they had used Claude to write that code and only 9% of GitHub Copilot users disclosed that. Part of that discrepancy, maybe a large part of that discrepancy, is in the tools themselves. So, Claude tries to submit code that says that Claude was involved. And GitHub Copilot, the way it works in the in the in VS Code, doesn't add

anything to say that it helped out. Uh but even beyond that, people were more likely to disclose it if they're using Claude. So, another another policy is that um instead of just banning AI-generated code, they consider it tainted and it has to go through more review before it's accepted. So, it's like, "We don't really know what this is. We don't know what it means for the project,

but we know we need to take a little more time to consider it." So, other responses besides just banning or putting it into quarantine uh were requiring people to attribute code, but we know the compliance is not very good based on that one study. Um I talked about prohibiting large patches as your first first pull request. Um so, requiring people to um submit small patches or pull

requests until they're trusted. Um some projects have actually closed down all all commits from anyone who's not already a maintainer. So, they're like, "It's still open source. You can still use it. You can still fork it. You can still do what you want. Uh but we are only taking changes uh from the people that we know that are already involved in the project. I think that's dangerous

for open source software moving forward. And then others are requiring you to manually attest that you created all the code that you're submitting or that you understand all the code that you're submitting. So I don't have a slide for this, but one of the things that I think is really interesting is how will new developers get involved in open source software in this current model with AI?

So if you're learning I I imagine anyone that is writing code is learning with an AI tool. You know, you ask it what does this code do or how do I do this or you know, you you look at the prompts that it suggests. So if you're starting to write code, you're going to use an AI And if you're not allowed to submit that to a project,

how are you going to get involved? So yesterday three people separately and independently suggested that maybe instead of having people solve um easy to fix first bugs, you know, all the Gnome love type projects, that maybe we start them off on reviewing code. Um most people learn first by looking at things that have already been created. Um maybe the path into open source software is no longer

submit small patches that work. Um maybe the path in is you have to you have to review five pull requests and you don't get to merge them, but you have to submit your feedback and other people will also do it. So I'm curious what you all think. Do you think AI is changing how people get involved in open source software? Like if you were a new developer,

maybe you are a new developer, um how are people Do you think the current model's going to work? And you can either share with the person next to you or I think we have a mic we could get people's ideas. Um but how do you Do you think it's First of all, show of hands, do you think AI is changing the way Awesome. Everybody. You want to

share a couple of those? We have a mic here that we or you can shout out and I can repeat Anybody want to share how they think it's changing? I don't know. Yeah, my comment is um what's the point of trying to get involved with an open source project if you're just going to use AI to submit slop? I mean, why are you even wasting your time

with it? Yeah, I have that same question cuz like I said, I thought that you wouldn't submit it because you didn't write it or it wasn't worth submitting. but I think people are doing it to try to they think that's what they're supposed to do. Um they think they're doing it to try to be helpful. They're doing it to try to become a maintainer because that's kind

of the path we've set for them. Um I'm I am curious about that if people have ideas cuz if we understood the motivation, uh maybe we could help prevent slop from coming in. I think there was one back the table back there. Um I don't have an answer to the slop question. Um but I was just I'm part of the post grass QL community and I feel

like the AI sort of world for us, we have always struggled to reach new users. We've always struggled to get, you know, people who want to use our tool easier access to it. So, I think while there's kind of a lot of issues with the contribution, this is just such a huge opportunity for any open source project to really reach a huge and diverse audience that we

have struggled to reach in the past. And so, I think while I do think maintainers are going to struggle to sort of manage the inbound flux and kind of manage the projects under this load, you know, the rest of us in these communities need to kind of view it as an opportunity, too. And what do you think about AI reaches more people? Is it Is it directly

related >> sets, right? Like there's just the the bar to do anything is so much lower. Like even the bar to contribute to open source is so much lower now. Like you know, where just for example, like if you were going to contribute to a an open commit fast for PostgreSQL today and didn't use AI, like compiling you know, software from scratch is not easy, right? Unless

you're really handy with the command line, you know this product really deeply, that's a that's a that's a tall order, right? But now, you know, you can just ask for help to compile all the software. So like just in terms of lowering the bar for people that want to use this and contribute to it with with good intention, right? That's a really good point cuz I feel

like the bar to get started in any project has kept going up over the last 30 years. Like it used to be able to quickly install everything on your desktop. Um in a few moments. And now you need all these dependencies and libraries. And yes, great point. So we should be welcoming all of these people trying to contribute slop. We just need to help educate them on

how best to get Uh the idea that I heard from one of the maintainers was to solve for this comprehension debt uh problem is you pop up again AI generated pop up quizzes about the code before anybody can contribute to that code. So whether it's And it's all AI generated, right? So you say, "Okay, here's a snippet of code. Here's a quiz about how you understood that

code." And again, that quiz can be AI generated. It scans the code, pops up a quiz in line. And once you pass through a certain bunch of pop up quizzes, then only you can contribute or do whatever, fix a bug, a security issue, whatever, right? So solving this comprehension debt problem that we're talking about, this is the best idea I've heard so far. I'm here to hear

other ideas. I I love the idea. You say I would give them a quiz. I'm I'm I'm having two kids who've just one who just graduated and one who's a student still. Um, I think they're going to use AI to answer the quiz, but Yeah. So, we'll take two more and then I I I will have lots of plenty time for people to contribute. Two more here.

Um, I can tell you what I've learned over the years um, using code assistance since GPT-3 religiously. Um, having really really really good guardrails. I mean we have DevSecOps Day right here. Really good guardrails, CI checks, static analysis, extremely extremely clear requirements. Um, we now have to think all of ourselves as managers managing a team of agents. So, being able to really really describe and then again,

the tools and then also the skills or the you know, flavor you're using. Having all of that in the thinking model like I mean it's just as good as as the dev as 10 years of experience in my opinion. What I've seen personally. So, using AI to set up lots of structure to accept things. >> Obviously with human in the loop that's observing all of that structure,

yeah. >> Yeah. One more. I think there's one here, too. I was actually going to ask pretty much that question. You aware of open source projects that have been using AI tools to set up those barriers? I I I like AI tools are really good at templating. Uh they need obviously some work afterwards, but it seems to me that using AI to battle AI is kind of

a starting point, or at least. Yes, and I I think some I I don't have any great examples of someone who's done it end to end, but definitely projects are adopting it more. So, I think the the answer is you are just given the answer. It needs to be more than governance. It needs to be processes, it needs to be tools. We we can't just like describe

our way out of this one with like accept AI, don't accept AI. We actually have to use AI to set up processes and things in place um to figure out how we can use the tools better to get better results. So, I think this is what what we heard here, um but this is like using AI to help you in the project itself. Um And I think

I think it's the next slide. Yeah, so I I think one of the things that that is true is that the more the more you create the process and the guidelines and and the descriptions and the tests, the more you will help AI not submit slop as well. Um so, if you're developing really good documentation, if you are describing the tests that need to pass at the

end, the AI tools are learning off of that as well, and they can follow it most of the time. They might say, "Oh, I'm sorry, I didn't know that." Uh but they but they can they can learn um from the documentation that you put in place, too. So, it's it's useful both for for the new developer as well as for the AI tools that they are using

and that you are using to contribute. So, kind of three areas that I think people are using AI right now, and there's probably others and other ways that we should be using it, is finding bugs. Um AI is really good at doing things like finding duplicates, finding duplicate pull requests, um enforcing the rules. It's It's really good at like those those very, uh, uh uh specific tasks.

It's also been really useful to a lot of projects in triaging, um, bugs and pull requests that come in. Um, so it can quickly scan a lot of code or a lot of bugs. Um, I talked to someone the other day who is using, um, strands AWS strands as an open source software projects to create agents. And he has It's not so much He uses it for

coding as well, but he has an agent that runs first thing in the morning, runs through all of the pull requests his projects has gotten, runs through all his email, runs through all his slack, recommends actions, um, in each of those cases, and tells him the top most important three things it thinks he should look at first in the morning. Um, so it's like a personal assistant

that's not only like triaging your calendar, but also giving you answers and writing your emails for you. He doesn't have it doing it for him. He just has it like generating the report for him. Um, and then pretty much everyone, not everyone, over a third of developers for sure, um, are using AI to learn more. Um, I personally use AI this way all the time. Um, I

feel like I'm constantly being thrown a new technical term or a new project that people ask act like I should have heard of it and know what it does. And so I just discreetly pull out my phone and I say, "What is this thing and why is it cool? And what does it do?" Um, so I think people are using AI that same way in code bases.

You know, what does this piece of code do? How could I write it better? What other ways should I be thinking of? So we had an idea there. Any other ideas where we should be using AI in our projects? Or where you're using it already in a way that you think is been really useful to you? Anybody want to share? Yep. Sorry, you didn't know you were

going to get your exercise today. Uh, AI is very helpful at explaining the code to me. If I don't understand a certain area, just like, "Hey, how does this work? What's the key files here? What's the uh path of the data structure through here?" Like uh teaching me cuz docs sometimes lag and are incorrect. And to the post grace example earlier, um I tried when I I

generated this version of the slides with a cloud skill and I took a clean machine that didn't have anything installed on it, was not logged into GitHub, and I said, "Install all this I said, "Use this skill to create the slides." Um and it installed everything for me, um asked for, you know, asked at each stage, um and generated slides, and I didn't have to learn how

it all worked. Which is not learning, but Any other? If you shout it, I can repeat it, too, but I'll I'll wait We'll wait for the Right there in the middle. Yeah, I'm an open source maintainer uh for open telemetry, and we use it extensively to write our own code. Maintainers with AI are a very powerful power because you even if it's right slope, you are there

to correct it. Uh but more importantly, all my demos for the conferences are now written by AI because it takes like 10 minutes to write the whole demo that took maybe a day or two before. Yeah. We'll take one more. I've noticed that AI is really good at debugging programs. Like, if I have a single character missing, AI is really good at spotting that. Yeah, it's really

good at debugging code. Absolutely. Or telling you like what went wrong. So So this is a This is the point of like we used to have people that started coding by doing a good first issue, and that's not really working right now. Um So right now if you put out a good if if you put out a good first issue and you offered to pay for it

like curl was with the security um issues, uh you're going to get a lot of slop. If you put out good first issue, maybe you don't get a lot of slop, but for sure the people that are trying to solve those good first issues who do not know how to code, for sure they are using AI tools. And they probably don't yet have the experience to like

really review the AI code. Um when I ask AI for help for some things like it does things I don't understand, I can ask it how it did it, but I'm still a beginner. I'm still learning and the AI is still the expert. So if the AI is making mistakes and teaching me wrong, I'm going to I I'm not going to be able to correct it. I'm

not even going to realize it. Um so I think we really need to figure out a new way to help developers get started not just in open source software, but kind of in coding in general. Um and we need to have a path for them that walks them through how to do that. So the one I've heard three times so far in the last couple days is

that we should have them reviewing PRs instead of creating PRs. Any other ideas for how people should get involved? Like how would you suggest right back there. I sorry sorry. The recording doesn't pick it up if and other the whole room can't hear. Sorry. Um okay. So uh I think it'd be actually pretty valuable to have a set of prompts or scripts that they could then run

against the code before they commit the code. Things like is it API stable against the original code if that's important or uh does it like uh manage to maintain like you know is it is it you know what what's uh bugs might it have? Like actually make it in make the AI at least interrogate the code. Because before you get because even when I write my own

code, I will have AI interrogate my code sometimes, >> Almost like a almost like a reviewer. So, having criteria that they have to meet in AI in the form of AI prompts, but you would still have them submitting Any other ideas? Just Uh the Is this a story Are you going to share with me? No, no. I was going to do that later. >> Okay. Um meta

stuff about the projects. So, for example, a lot of the tools work best when they are using specs. So, what I like my org has 50 repos in it and I needed consistent GitHub labels, right? So, I had it generate a basic one and then the people went and then they fixed it up. And then we didn't have to write a script. We just tell Copilot, "Go

make this true." And it worked. But now we have a person who's like really loves to like categorize issues and then that helps them create an issue triage guide and so on and so forth. So, it's not just code, it's also like I just want to say the meta stuff of the project is also useful. So, tagging the the meta things that you have as good first

Yeah. Cool. Cuz they have to learn. I think the the real problem we're trying to solve here is the trust of the people that are submitting code and like that's, you know, something that we've had to deal with uh uh since the beginning of open source And um one of the things that I would lean into is ways we can actually build trust between people, less about

what the code produces. So, community meetings, you know, talking with people, you know, either synchronously or asynchronously because it doesn't matter what tools you put up and what barriers you put in the way. You people are just going to build AI tools to get around that. So, you say, do code reviews." Well, people are just going to do AI slop code reviews. Right? So, it's not tool

it's not the code that's really the important part. It's the people that are the important part. It's a really good point. And the community is people have always been the heart of it. We'll we'll do we'll do one more right here. I think another good way would be to have um people come in and reproduce bug reports, you know, just try it out especially if it's a

different architecture, different operating system, whatever. But just that way when the maintainers go to fix it, they've got a broader pool than that original bug report which may or may not have, you know, complete information and kind of get a better sense of where the boundaries of the bug lies. It's a good one. So, it's enhancing the PR and reproducing it with reproducible test guidance. That's good.

I actually early in my coding career like in high school, I would take two-line code programs that people had stuffed like everything they could into two lines and then try to pull them apart and I learned a lot that way. I was going to say something. Oh, on the people thing, I went to an AI event uh where they were saying it was really important for us

to figure out who is a human and who is an AI agent and they wanted us all to walk around with a software on our phone and like do the the the swapping of identity thing to create a trusted network. I I saw a few flaws in it, but it was it was an interesting concept. So, I I already already gave the stats. You know, learning learning

is super important. Um it's a super important use of AI. Um and I've been asking this the whole talk except about the end, but I think it's really important to uh think about what we can do. Um I think one thing that came across as a best practice in multiple studies and multiple blog posts that I read was the documentation and I think we touched on that

a couple times from people in the audience. Um it's really important to document, you know, the test that have to pass, the how you want people to contribute. And it's it's good not just for people that want to join your project, for people learning about projects, but it's also good for the AI agents themselves. I've also had a lot of conversations lately about AI AI agents that

are ignoring the rules that have been set for them. And what does that mean? And I really feel like that's a bug that should go back to the provider of the AI tool. If they are giving you an agent's MD file format that they say that their agents and their model are then going to listen to and it's ignoring it, I personally feel like that's a bug.

Not not a hallucination or not something that we should try to work around. But I may I may not be proven right in the long term, but I think I think we have to have models where the agents actually obey the guidelines that you have set out for them. Maybe that's too old school. You know, code used to be you wrote it and it did exactly what

you thought it did or it had a bug and you fixed it. AI tools in general are much more um unreproducible. Like they they tend to generate different output every single time. And I think that's hard for some of us like me that come from a coding background and we expect code to do exactly what we think it's supposed to do every single time. I feel like

we're entering a world that's more like medicine. So in the medical world, people come in and they have they say they have a stomach ache. It's hard to reproduce it there in the office. Like maybe they say they have a stomach ache you know every other day, but they don't have one today. It's much more of an art than like a deterministic science. And if you're interested

in medicine, it turns out surgery is the most deter deter Surgery is the is the the branch of medicine that is most deterministic and surgeons improve more than other doctors because they get immediate feedback on what went wrong and right in the thing that they just did. Um but I feel like coding is like surgery and now it's turning into something more like general practitioners, but maybe

that's a bad analogy. Another area that I think is is really important is and I think it's changing open source software in general is the dependencies. Um so there's definitely a push right now to reduce your dependencies and there's even a debate about whether a best practice should be not to depend on small libraries. Um so those tools like curl or like I think I have another

one here example lib util has been super useful. People have used it forever. People got um and now the AI can just generate code that does that in 30 seconds. Some people are saying that's more secure because if you don't have a dependency, that's one less security hole for you. If there's a problem with it, it's not going to percolate to like all of the software in

the world. But the argument has always been that that's more secure to use the dependency because it's been vetted and tested by all of those millions of people. And if you're introducing new code that does what that dependency did, you are more likely to introduce more bugs. But I've seen this as an active debate um in many different situations. So some people are advocating for less and

whereas the traditional advice has always been depend on things so you're not introducing more bugs. I think the reason I included it here is not as a coding best practice. That's not what this talk is about, but as I think this is changing open source software. I think a lot of people got involved in open source software because they wrote a small piece of useful code and

they put it out there and they shared it and a lot of people found it useful. It got improved, it got enhanced. It still remains small, it's done, but that was their entry into open source software. And that's how they got known and that's how they started working on other pieces of software and got jobs. And so, if we take away those small utilities, are we are

we removing one path to entry to open source software? Not necessarily bad, but what's going to replace it? How do people get involved? How do they get started? Another thing I think we need to do, whether or not you remove your dependencies, is support open source Um I think now more than ever it's important to call out to all of the users and all of the companies

that are used to using and depending on open source software to contribute back. Hire those people, allow them to work upstream, contribute to the foundations, contribute to GitHub sponsors. However, um you want to do it, I think it's important to pour funding into the people that are working on your Uh we know from surveys that they tend to put more time into it and are able to

create more secure software when they have more funding that enables them to And it's also really important to submit quality bug reports. Um so, if you find one, don't just use AI to generate it or maybe make your new users um go and make sure that the uh bug reports are are of high quality. Contribute yourself to open source Um and then, if you use an AI-assisted

tool, even if it it's really hard to say, Claude helped me write this or Copilot helped me write this or Cura helped me write this, I would go ahead and enter that. I think I think it's important information for us as we figure out what we do next to know what type of code is being generated with AI being used with it. And I wish there was

a more nuanced way to say AI generated most of this or AI helped me review this or AI suggested changes. I think that would also be good data. So, the question is, is AI killing open source? I had you share with the people next to you what you thought. but I'm going to say I don't think so. I don't think AI is killing open source software. I

think we'll I'll be back at scale in another 8 or 10 years and we'll still be talking about open source I do think that AI is creating noise. Um, I think it's changing the reasons why people contribute to open source software. I think they're they're shifting slightly. I think it's changing the type of project that has been successful. So, it's going to be less of the blob

utils and it's going to be more of like the Kubernetes and the more complicated programs. Um, I think it's going to be this a influx of AI and these new tools are going to be really time consuming for all of us in the room in the short term. Like we are going to have to invest time in figuring out how it works in our flow. I think

it will be time well spent. Um, I think AI will accelerate everything, uh, but we need to spend some time making sure it accelerates it the way we want it to. So, I think it's changing the way we create software, not just open source software, but all software, obviously. And I think we in the room that care about collaboration and working together and building on the work

of others needs to make sure that we set up structures where it can help us collaborate, not just help us create. So, what I would love you all to do is turn to that person next to you or a different person, uh, introduce yourself if you didn't know their name from before, and do you still have the same answer? Is AI killing open source You too. >>

All right, the 30-second warning. All right, so show show of hands, how many people changed their minds? How many people had different answer? Really curious what were You could start You have the microphone. And then if you could But But you can Start Start You raised your hand, right? Yeah, you. Uh yeah, so like I've seen so much AI slop and I've like had to de-slop my

so much code. Uh that was like almost feels like it was vibe coded. But I found that in my process of de-sloppifying it, it was very handy to use AI to do that. So I have done some meta-prompting and using those templates for prompts that I'm generating, I've been like noticing myself, okay, I guess now is a good time to add more uh documentation for developers. Like

really spell it out as if they were a junior dev coming into the project. AI reads it and it's like, "Oh, let's do that." And so I've I'm starting to see more, "Oh, this is useful." Awesome. You said you also changed your mind? Thank you. Well, uh a little bit yes, also because uh now I see how it lowers the barrier to for people to start contributing

to open source projects. Me myself uh like I I felt uh brave enough to start contributing projects for things I I knew I was good at but for projects I was not familiar with but I have an open question. Okay, AI is not killing open source now but AI is also models running on someone else's machine and they are cheap now for the same reason Amazon was

cheaper like 10 years ago. What happens to open source once projects that got saved by AI projects that got better by AI start losing contributors because let's say AI models start to be too expensive to to use. That's that's my question. Will it kill later? Yeah, I think that's an interesting question and probably a whole another talk. I've had that conversation is AI going to get more

expensive or less expensive. Are we subsidizing it now or will the cost go down? I personally think the cost will go down but the demand will go up so they'll all even out but but yes, very interesting question. Will it get too expensive? I I have a question. So we see in Grafana I work for Grafana and Grafana community that our number of people who joined the

community goes down as to reflect what previous person was saying. We have a lower number of contributors to open telemetry and the economy is kind of bad, the industry is kind of bad but also AI and it's killing the supply of contributors so we don't have succession for maintainers. Do we see it in other communities? Do we see like do we know if it happens because of

AI or because of the economy and the industry state? That's interesting cuz I kind of feel like Postgres said the opposite that they they're getting more people cuz of AI. So a show of hands, how many people are seeing more new contributors because of AI? So a handful and how many people are seeing less contributors because of some unknown reason, AI, economy, something. A couple. Okay. I

don't know if we have a big enough pool there to come up with a question, but that's that's definitely interesting. Anybody else change that? Yep. Oh, sorry. I I had a question generally about one of the earlier topics. You were mentioning that some of the uh some of the projects are concerned that there may be no copyright on um the contributions coming in. Um and I some

of my colleagues at Software Freedom Conservancy recently published an analysis of the Thaler decision uh from the DC Circuit. Um and they they found that there were a number of of good points uh related to that. And I was just wondering if you had any thoughts on the Thaler v. Perlmutter case. Um and uh you know, what whether that that's good news or bad news uh for

open So, to be honest, I I haven't read the case or the Software Freedom Conservancy's post, but I will go do that. So, I I I don't have an opinion at the moment. So, I think we can open it to general general questions as well. Now, go ahead. We'll take that one. I will open it to general questions as well. Um I just wanted to leave it

with I think the world is changing. I think AI is going to change how we work, and I think we very much actively need to be talking about it and sharing just like we always have about what works in order to make sure open source software and collaborating works. I think one thing to consider is that uh before this all this AI with a lot of code

being written by humans uh the libraries people were using and the projects they were using, they actually read the documentation for them and they got familiar with these projects and made them want to contribute to these projects, which added contributors, but with AI you don't directly interface with the documentation or the project. You just access it through the AI model, so you don't actually feel or get

involved with the project results in less contributors. Yeah, that's a really good point that there's AI is is making it just easier to consume it without getting you Hi, I just wanted to follow up on one of your last slides you said it may be that certain projects are going to go away and certain projects are going to succeed and they'll be like more successful. You gave

the example Kubernetes project and you said maybe small tools won't be around and I didn't quite follow the arguments, so I wanted to go deep for all that. It sounded like maybe Kubernetes, large complex project, that's not going to be influenced by AI tools. Smaller libraries might, but that doesn't sound like a permanent solution because these tools are going to get more complex AI coding agents are

going to get more capable. Maybe someday someone's crazy enough to re-implement Kubernetes using that and then It's it's it's possible. I don't have a crystal ball. Did I get the argument right now? I guess I didn't. What? I just want to know if I got the argument right. I >> Yeah, so the argument in the short term is that the small things are just so easy for

AI just to generate that solution already. They they've shown that they can do that, the AI agents. The large things, I think especially things that are service or more complicated, um like Kubernetes, I mean, I'm probably not going to ask my AI tool to create Kubernetes for me, right? The response right now would be so many questions for me that I would have no idea how to

answer. Um so, maybe that will change. Um I I think I think we're just discovering I think any major technology that has been introduced into society has changed the type of jobs, the type of work, the type of the the way we live, and we're just figuring this one out. This is the new one. We'll take one more question, and then I really hope that you all

Okay, we'll take two more questions, and then I really hope that you all continue this conversation and continue to help find solutions. Uh not a question specifically, but a uh focusing that I've noticed as a result of this conversation. at what point does it cease being slop? Right? Um I don't think there's a yes no answer to the whole question, but what I think falls out of

this is what really is the difference between good AI code that passes the bar of slop, whatever that is, versus some new contributor. Hey, this is my first programming language. Here's my first my first commit. I think that there's overlap between the two of those, and so ultimately the deeper question that's going to have to be answered is what does slop look like for my project? And

again, for my specific project, what are the risks of not understanding the code? If I'm just making some basic CSV parser, if I don't understand that thing, who cares? Right? If I'm making a medical device, I better understand every little bit and bob in there. So, I don't know that it's necessarily a always yes, always no, but I do think that there's going to have to be

a lot of about what does my project need, and how does my project differ from other projects? And from there, governance and all of that falls out, I think. I agree that each project will will need to define it. You've got someone behind you that wants Uh in regards to open source, the other interesting thing to me is right now a lot of open source projects are

inputs into AI models. And the interesting long-term effect of that is how will that impact people wanting to make things open source? Um because at that point, hey, this cool neat thing, now I'm just another input for the next AI model. Um it it'll be an interesting thing from my perspective. Agreed. Thank you all for coming. Thank you all for participating and continue the So much for

your talk. I really appreciate it. Um Whoops. What's next? >> Hello, hello, hello. Welcome everybody. This is uh Scale uh Scale 23. We have our open source AI track here in this room right now. Your safety and comfort is important to us at Scale 23. So, if our safety number posted all around the Our next talk is from Greg Shaninger. He uh the founder and CEO of

Oxen AI. And uh he's going to give this speak about Hollywood and open source. Uh please give a round of applause for Greg. Can you hear me? Yeah. Sounds good. Well, thanks for joining right before lunch. I think this will be a fun one being based here in LA and how these open source video models are helping Hollywood or potentially hurting them in different ways, but we

can we can dive into both of those aspects. And we're going to be working from a real-world problem that we helped solve of a a wrong jacket that was in the scene after they had done the shoot with all of the actors. And while we can't show the real movie that we did this for, we can walk you through a fake example of Pulp Fiction and we're

going to swap out the jacket on Samuel L. Jackson and show you how we used open models throughout the whole process uh to make this happen. And this is an expensive problem if if you make this mistake on set, uh reshoots are expensive. You have to pay I mean depending on what actors you have, you have to pay their day rates, all the people involved, the sets,

the location, the costumes, like bringing everybody back just to redo that 5-second clip or 30-second scene, pretty You can do this in post with visual effects, but that also is pretty time intensive. You have to build all the 3D models, all the materials, all this kind of stuff. It can take a pretty long time. And so the question here is can AI become a new tool in

our tool belt for VFX? And I think that's a good way to frame the use of AI in Hollywood in general is just this is a new tool in your visual effects tool belt, not necessarily something that's like replacing artists or replacing actors. You're still going to need all these people in the process, but is it just it's just like a new Photoshop or a new tool

that we can have. And so the goal of this talk is to give you like a 5 or 10,000-ft view of video generation models and especially the open ones that you can download and run on your own hardware. Give you a lay lay of the land of some of the terminology. If you see that third bullet with the TV T2V I2V R2V Raise your hand if you

know what all those acronyms mean in that line. Anyone? Cool. We're going to learn some stuff Uh and just inspire you to what's possible with these open models and emphasize that like it's not just a one prompt to rule them all to replace everything. It's actually a pretty in-depth process that that the teams go through. So who am I? I'm the founder and CEO of Oxon AI.

I've been training language models since 2013 when they were just called language models and not large language models. I was early in the generative image, video, 3D modeling space with GAN and diffusion models, and I just love open weights and open source models cuz you can just fine-tune those all the way down to the weights and run them locally, and there's there's just so many benefits of

them. Uh if you want to connect on LinkedIn, there's a little QR code there. Uh Auxen AI is the company that I founded. Uh it you can think of it as infrastructure for uh building AI models, one-click fine-tuning, spinning up these big GPUs to do training, automatically spinning them down when you're not using them, on-demand APIs to run some of these Uh all of the weights, all

the data, all the inputs and outputs get version controlled as you're iterating on these things, and it can scale the terabytes of data. So, we built a custom version control thing that's pretty cool if you want to talk about that. And our our real goal is to have people own their AI AI end to end and not just like rent it from cloud services. So, if you

want to try it out, if my talk is boring, you can go and play with some of these models live while while we're doing it. So, this is going to be our fake movie set, and we're actually just going to do this five-second clip here from Pulp Fiction. And so, there's a few interesting things about the scene. Um oops. Let's go back and play that again. So,

we have two actors here, and the goal is to just replace the jacket on the actor on the right, Samuel L. Jackson. And you can see as he's raising his gun, there's also some like uh occlusion of his jacket, and the jacket is wrinkly, and there's different lighting. It's like in this dark room. So, how do we just like with a fine-tuned uh fine-grained brush go in

and fix all of these things with AI. Let alone this is a scene with a gun in it, so a lot of the AI models might just reject it right away if you sent it through with a with a prompt. And so, for fun, decided to take the jacket from the Eddie Murphy Delirious set, and this is the jacket that we're going to put on Samuel L.

Jackson, so it's like this shiny red leather jacket, and he doesn't have a shirt underneath, and he's got a gold chain. Um, so all of our training data is just from this YouTube video of Eddie Murphy doing his Delirious set. And at the end, with all open weight models, we will get this, which is the jacket swapped on, um, the chain underneath, the guns, Looks pretty good,

and it it took about two days of work to gather the data, train this model, roll the dice a bunch of times with the AI until we got the shot that actually looked pretty good. Uh, but compare that to a full set and VFX team, doing pretty good. Uh, we did this about five months ago, and a lot has changed cuz AI just moves at an insane

pace. so I'm going to walk you through the kind of professional workflow we did here with ComfyUI, which is a open-source project. If you're not familiar with it, it's like a node-based tool where you can chain a bunch of these AI models together and pass inputs and outputs into each other in a DAG, and then come out the other end with the edit. Um, it's crazy. Like,

this was probably state of the art five months ago, and I tried some of the newer models today, and you can almost get away with none of these steps in between and just prompting and reference images. Uh, but I'll kind of walk you through both ways. And so, here, if you probably just want to take a picture of this for later if you're interested in like what

the top models are these days. It's It's pretty hard to keep up with. Um and on the left-hand side I have the open-source models, and on the right-hand side I have the closed-source models. For this particular one, we used 1 2.1 base, which we'll dive into. Uh it's actually one of the more powerful open models, and it still is, and it's over a year old. Um then

there's 1 2.2, which is a slightly bigger model. There's LTX 2, and actually yesterday they dropped LTX 2.3, which is pretty cool. And then there's some other Chinese models that I haven't played too much with. On the closed-source side, there's Kling, which is actually probably one of the best ones out there today. There's Sora 2 from OpenAI. There's Runway. Google VEO is one of the better ones.

And then there's the SeaDance 2.0 that you've probably seen some crazy viral videos going. Nobody really has AI API access for that one yet, but the results look pretty incredible. So why out of those the the lists we just saw there, why would we choose open-source or open-weight models? Well, uh one, you can run these things locally if you have a big enough GPU. That means all

your data and assets stays private, and you're not sending it to OpenAI, and you're not sending it to Google to go train on. Um they're also cheaper. I mean, you have the upfront cost of the GPU, uh but if you're using Google VEO, it can be like multiple dollars per generation. So if you're doing hundreds of generations, that gets expensive pretty quick. If you just have one

of these models running locally, uh it's just your electricity cost at that point. You can train these things. They're customizable. You can plug them into ComfyUI. You can hack on the architecture. Um and we're getting a lot of interest from uh directors and agencies who just are really concerned about their IP and maybe even have large sets of IP that they can train their own models on

if they had really cool spaceships or actors or aliens that they had created in visual effects in the past. You can then train these models on those things and then have more of an infinite content engine than have to have all your 3D artists do it over and over Uh, you can also distill open source models from closed source models. So, if you want to collect a

bunch of training data, uh, it can be frowned upon by the closed source, but they would have no way of telling. Uh, we had a really fun example built on Axon. I don't know if this thing went pretty viral a month or two ago, but this guy wanted to use Nano Banana, which is this cool image editing model that we'll talk about how we use it in

the same We While Jackson context, um, but they wanted to make an isometric map of New York City like kind of SimCity style meets Google Maps style where you could zoom in, zoom out of the tiles. And the idea was take to take like the map view from Google Maps, run that through an image editing model, Nano Banana, and generate 40,000 tiles that they could then put

into a big zoom in and zoom out. And even though Nano Banana is like the best closed source model on the market today, best guess they were like it hit 50% of the time whether the result was good enough to put into the map itself. And for 40,000 tiles, 50% accuracy just isn't going to cut it. Also, it was slow and expensive. So, each tile with Nano

Banana would cost like 15 cents to generate. Uh, if you're going to do that 40,000 times, that's a pretty big bill. And so, what they decided to do was take an open-source model, this Queen image edit, which we'll talk about, fine-tune it on the good outputs from Nano Banana. So you can think of this as a curation step, like we're hitting it 50% of the time and

we're taking the good ones and we're putting it into a training data set. Train that thing. The training itself cost like $12 and then you have this open weight model that's actually better than Nano Banana at your task cuz you like narrowed down all the good ones. It's not better than Nano Banana at everything, but for this specific task, it's really good. And then you can scale

that thing up and it's much cheaper to run this thing at scale. So, open-source for the win for these cost, speed, accuracy, and privacy problems. Before we dig into the full workflow, we're going to visit all of those acronyms that we talked about at the start. So, T2V, I2V, R2V, these are pretty easy to learn, but it's text-to-video. So this is your tradi- your traditional just like

take a prompt, get a video out. There's I2V, which is image-to-video, and this is usually like you have a starting frame and then you just want to continue it or you want to animate it. You can also do like starting frame-to-end frame to give you a little more control. Then there's what's called R2V, which is like reference-to-video. So in this case you can take like a bunch

of reference images, have that as kind of context for your prompt. If you're If you're used to doing this in the LLM space, you can think about this as just like building up your context, and then generate the video from there. There's also audio-to-video, and then if none of that works, like if none of the prompting techniques works, you can go down to the weights and train

a LoRA on top of it and that can kind of give you the extra juice that you need. So, wanted to walk through some real-world or some actual prompts to try to recreate the scene just to give you a sense of like where each one of these techniques works and where it doesn't. So this is actually from this LTX 3 or 2.3 model that came out yesterday.

So this is one of the latest and greatest open-source models. And so what I did was I took uh still from the Pulp Fiction and I had an LLM try to describe it in as much detail as possible with the prompting techniques for LTX 2. And this is what came out with just a text to video. Um so honestly like pretty cinematic, pretty realistic. It doesn't know

the characters and it doesn't it has a hard time placing them in the scene just from our description, but honestly pretty impressive. And these text to video models I would say are great for brainstorming, but it's it's not great for like if you want that full control. So then there's these I to V models, image to video. LTX 2.3 actually has these different modes that you can

work in. So you could just put in the prompt or you could put in the prompt and an image. So in this case uh I had the starting frame. It's kind of hard to see down here, um but I had already swapped the jacket out and had this first frame as input. And then my prompt is the two men look up and point their guns at the

camera. So you can see there's actually there's no guns in the scene. Um we have the start frame of the actors and then it's just going to continue from there. uh if you know these actors well, it doesn't look like them. Um so the first frame for sure does, but watch their faces closely as it continues. It it really starts to drift because one, the actors are

like looking this way and two, uh this particular LTX 2.3 wasn't or one of the things it boasts is it's like not trained on Hollywood movies, so some of these Chinese models or other sketchy models might be trained on a bunch of non-licensed content, but this one isn't. And so watch their faces again. That's definitely not John Travolta anymore, and I don't know, Samuel kind of looks

like he has buggy eyes or Um but did did get the guns and did follow the rest pretty well. So, a lot of the time um artists will spend most of their effort on like the start frame and the end frame of the models. you'll see this is me playing with the latest Nano Banana 2 um image editing model, and you know, I put in a image,

the start image, like the original one from the film, and then I took a little grid of Eddie Murphy in a bunch of styles from his uh from his set, and I'm like, "Hey, can you change the man with the afro's suit?" And sometimes you can't even say actors' names in the prompt, or it'll just reject it if it's a closed source model, cuz it's like, "I

don't want to touch that with a 10-ft pole." Um so, I said, "Change the man with the uh on the right to be to have the red jacket from the comedian." And this is Nano Banana 2, not a not an open source model, but you can see it has a hard time even just doing this task, so it like keeps the tie underneath. The one on the

top looks like very plasticky and video gamey, and he's got like a perfectly shaved chest, and I don't know, it just doesn't quite look like him. So, even the state-of-the-art image editing has a hard time with some of these tasks. Uh but what's cool is there's a lot of open source image editing models. Um so, there's this Kwen image edit that's pretty popular. There's these Flux models

from Black Forest Labs. Um And these things are small enough that you can run them on consumer hardware and you can fine-tune them to your use case like that guy did with the Google isometric NYC map kind of thing. The bad news is they're pretty terrible out of the gate, too. I'll actually pull up some of these cuz I think it'll be fun to look at the

results live. So, let's look at this one. So, these were some of my takes from earlier. So, you know, this looks like GTA or something under the hood like a 3D character. It's not really the red leather jacket that we're talking about. It's not really the chain that we're talking about. Same thing here. So, that's like when image at it 20 billion parameter model. You can play

with this in oxygen if you want or you can download it and run it on in comfy That's when image at it. There's actually a couple different variations of that that you can fine-tune. We'll get into fine-tuning Then there's this flux 2 dev, which is actually a bigger model. So, you would think it's better, but it has a hard time keeping the actor consistent. So, it actually

just like puts Eddie Murphy into the scene. So, maybe it's a skill issue with prompting, maybe not, but these models have a hard time following instructions sometimes. And then this is flux 2 Klein, which is a smaller model. I actually for whatever reason think this one gives better results in general, but it's still Eddie Murphy and not Samuel So, we decided to train our own model to

just get like the starting frames really good and get the continuation of the video. Really good. once we train that model, then we could get really good start and end frames. So, this is an example of LTX2. Once we have a really good start frame, and once we have a really good end frame, so this is start, this is end, and then this is it animating in

between. Okay, so one thing I'll point out is if you watch his wrist at the end, uh if you was like something just falls off of it. I'm not quite sure what that is. So, that's like you're just like rolling the dice a lot of the time, and that one clip that we got was I think, you know, 10, 15 different we'll switch the random seed, we'll

switch this thing, we'll switch that thing, run them all in parallel, and then like figure out which one works best. So, it's it's not just like one shot to do it all. Um I would say the faces are better, but maybe not as good as they could be. And like when you're in a Hollywood setting, like that actor is going to be like, "That doesn't look like

me." They're going to make you do another another take on them. so, what's cool is we talked about the image editing models, which you can train. Then there's the video generation models that you can feed the images into, do the start frame, last frame Again, LTX 2.3, 22 billion parameter model released yesterday. Uh that's why I just got that video from, cuz I wanted to compare like

the latest and greatest. We're going to walk through this Juan 2.1 base model, uh which we used 5 months ago, but honestly, I think it's still one of the coolest ones out there. So, you're like 90% of the way there. In machine learning, if you know this, uh it's pretty easy to get to 80 or 90%. That last 10% is just like where all the work comes

in. And so, this model, Juan 2.1 base, base stands for video tasks within an all-in-one framework for creation and editing. It's a mouthful. But they trained this thing to be able to do all those different like T2V, I2V, R2V, V2V, MV2V. And if you look kind of closely into these different modalities, what's really cool is this model gives you the most control I've seen out of any

model like closed or open source. You can pass in depth frames from like a 3D engine that's just like the Z values there and then have it like fill in the colors and fill in the depth. Or you can pass in pose estimation from a human and it can like track that and take that as input. You can pass in a mask for like I just want

to edit these pixels here, which is something that we did in this project and it'll just paint in those pixels and like not touch the rest of the pixels. So this 1 2.1 base, I wish more open source models would go to this level of customization, but for artists, this thing is just like the bee's knees to be able to have full control over these things and

not just prompt and pray. And 1 2.1 base is fine-tunable. So you can teach it anything. Note, I do say like you should always start with prompting just to see if the model can do it and define what problem you're trying to solve with fine-tuning because if you can't define the problem you're trying to solve like the motion isn't good or the character isn't consistent or the

voice doesn't match or the style of the camera isn't right. You can't define that, then you don't really know what data you have to collect and what you're looking for to compare before and after. So just start with prompting. It'll be faster to iterate, fast to explore what these models can do. And then once you figure out really what you want to hone in on, you're at

90% you want to get to 100% break out this fine-tuning hammer. Um so again, this is uh the workflow and we'll dive into individual nodes. I know it can be a bit overwhelming to look at this node spaghetti. Um but the first thing that we're going to do is since this 12.1 base can take in a a bunch of different modalities, uh one modality that can it

can take in is a mask of like just the region of the video that you want to inpaint. Uh so in this case, this is the mask of like just Samuel Jackson on the right and you can um it's white where we want to inpaint and it's black where we don't. And so you're going to like align that perfectly on top of the video uh resolution-wise and

frame rate wise and then this base model is just going to inpaint right there. Step two, this is debatable if we needed it or not, but just like you can pass in the mask for you want for where you want to edit, you can also pass in a pose estimation. So I know this is small and in the corner, but if you've ever seen like where they

put the ping-pong balls on people and have cameras tracking from all over, you can actually just do this with a neural network now, especially with 2D and it'll put mark where your shoulders, hips, waist, knees, toes, elbows are and this just gives a model a little more information of like we really want to focus on a human and we want to focus on right here in the

video to make the edits. Um so even from this mask, right? Like you can't quite tell where the arms are, you can't quite tell some of the other things. So this pose estimation step just gives the the model a little more to latch onto when it's generating. step three is now that we have these two inputs, the third input that we're going to do is a jacket

specific Laura. So, a Laura stands for a low rank adapter. These are actual weights that you can apply to the model. So, get all the way down to the neural network level and like teach it a new thing. And so, what we did was take the video from YouTube of Eddie Murphy, chop it up into frames. So, I think we had like a hundred frames from the

video, 137 it says in this data You upload this to Auxen, we spin up a GPU for you as long as you have like images and captions, we can then spit out this Laura that you can apply to your model, and then you've taught it a new thing. So, you've taught it what this jacket looks like, how the jacket kind of moves, what the material is, what

the lighting And we try to make this as easy as possible just to go from data to model and not have to worry about any of the infrastructure. Um and I think it's best to the actual data itself here and then look at what happens during training. the training run that we did. Um we dump all of the weights to a directory for you so you can

download it and run it. We also tied the exact version of the data set that you trained on. So, if you wanted to run like a bunch of experiments in parallel, spin up all the different GPUs, you have a commit hash for this like very specific version of the data set, and then you can look in and we have like a bunch of different prompts saying Eddie

Murphy holding a microphone with a cord, very shiny reflective red plastic jacket. You can see we kind of repeat ourselves over and over there just to kind of lock it into these are the things that are going to be consistent, and then everything else from there you have a little more control over like standing with his left hand on his hip. Yada yada. This is what the

training data set looks like. And then once you kick off a train, we will go back here. You can also give it some samples that you sample prompts that you want to test as it's going through. So, I have sample prompt here, man wearing a very shiny reflective red plastic jacket, neutral expression, soft studio lighting. And the first two prompts that I'm doing here while training are

just kind of some like base cases where I don't say Eddie Murphy, I don't say any of the other stuff. I just want to see like what the model knows when I don't say Eddie Murphy and if I'm over fitting over time. And then this one over here in the right hand side, I actually put Eddie Murphy and you can see the model out of the gate

has no idea who Eddie Murphy is. Which is fine. We're going to train that And also it's definitely not the jacket So, after 200 training steps, so this isn't that long. This is just seeing 200 images, we start to see a little bit of Eddie Murphy. We actually see two Eddie Murphys, which isn't quite what we want. It's picking up the jacket style, it's picking up the

chain. But you can see in these other ones, since we haven't mentioned Eddie Murphy, it's not putting him. Which is fine. We're That's actually what we want and we're going to iteratively 400 steps looking like Eddie Murphy. That's good. You can see the jacket here is starting to get closer to the jacket from the actual scene. I'll point out things like the actual collar here is like

something you want to learn like these buttons here. And so you can see this jacket doesn't quite have it yet. It's slowly learning in the buttons. Eddie Murphy's just bouncing around over Slowly learning. Okay, now we're getting pretty good. Like we've learned the jacket concept and we've learned the Eddie Murphy concept separately and we're not just like applying Eddie Murphy to all the other ones even though

you'll notice it did kind of pick up on that here. So this is what we're we're kind of monitoring overfitting in this column if you will. And then finally, yeah, I think this is around one of the last steps that we actually use. This jacket's looking pretty good. Eddie Murphy's looking pretty good. So we can take this checkpoint plug it into ComfyUI into this big pipeline um

and it'll just replace the jacket in the scene. Um so this is what it looks like to plug in a LoRA into ComfyUI. It's kind of hard to see but it's just like a separate node that you'll put in uh that kind of applies these new neural network weights into your matrix matrix and change the behavior of the model. Finally, you have this crazy sampling thing in

the middle. This is where you can set things like random seeds. Uh you can tweak other parameters. You can put in your prompt in that green box there. And then it'll output the final video This Wand 2.1 model takes like 5 minutes to render uh from your prompt to the final output. The newer LTX model takes like a minute to render. So it's still not instant and

um it's kind of like fire it, maybe go do something else, maybe kick off like 15, 20 generations, go do something else, come back, and then you'll see everything that came out the other end. Uh and I want to just show you what happens when we're stumbling through the latent space here. So this was one of the first training runs we did and you'll notice in the

prompts in the training data prompts we had holding a microphone and that was because this is what came out when we didn't have that in the training data is like instead of a gun it would be a microphone and then it would switch to a gun and then and then the gun would have like a cord coming out the bottom. So when you're creating your training data

set you kind of want to prompt what you want to be able to direct later. So if you want to be able to direct does he have a microphone or does he have a gun? You put that into the prompt during training data and then it sees prompts with microphone it sees prompts without microphone and it learns like okay I want to be able to control that.

We didn't have that prompt in training data before so it just like always tried to give him a microphone. we did that again. I think we ran into the cord problem. So now it does the gun but like you can see there's still an artifact from the cord. So now we're just going to be like rolling random seeds until the cord goes away. You could do another

training run to try to get that out but honestly um trying different random seeds works And then this is what came out the other end. So pretty good lighting pretty good textures uh again it took about a day to do this so pretty impressive. Uh if you had to do this in traditional visual effects it would be a big pain in the butt I can tell you

that. All right. So how much time do we got left? We got plenty of time. Cool. Um I'd like to I'd like to dig into some of the closed source models that are coming down the pipe because I think it informs what's going to be coming out with open source models later again cuz like some of these open source models probably distill on outputs of the closed-source

this idea of reference image to video is getting pretty powerful. Um what you can do here, instead of just supplying like the first frame and the last frame and having it continue or connect, um you can add more reference images. So, I know it's kind of small text, but you can see on the left-hand side there there is a frontal image of him, and then there's reference

images of like, this is how he looks from this way, this is how he looks from this way. You can add up to three reference images per element or per actor. And so, it could be an actor, an element could be the jacket, an element could be could be the cup that he's holding. And what's cool is you can say like, "@element1" in your prompt, and it

knows that that refers to like all these images over here, and then you can say "@element2", which might be the jacket, and it refers to all these elements over here. And so, the model that does this really well right now is this model called Kling O3. There's V3 and O3. V3 is like their standard one, and then O3 was kind of trained like ChatGPT O1, where it

has this big thinking loop kind of thing in in the Um but this was just purely from this video that I'm about to play is purely from text prompts and reference images. So, pretty freaking good without fine-tuning, if I say so myself. but again, not open source. Uh and then there's this crazy seed dance two one that you might have seen. Uh I'll go ahead and play

this video if you haven't seen it, cuz it kind of broke Hollywood last couple weeks ago. Okay, so this is Tom Cruise fighting Brad Pitt on the top of uh top of a building here that's kind of broken down. And with SeaDance 2.0, not only can you put reference images in, but you can put reference videos in. Uh so that gives you like a lot more frames

that you can give in the context. And gives a really consistent at least actors in this case. Like it's pretty much nailing those guys. And so this has this opens a lot of interesting questions around IP and ownership and what you can do with these models. Um I think it's there's going to be some interesting legal battles coming out. And honestly, SeaDance 2.0 was scheduled to come

out late February and then this video dropped and Hollywood came after them and they haven't even released the API yet. So they're they're trying to plug a lot of holes, I think, in not being able to do stuff like this. Um so that's what's coming down the pipe. I think what's really interesting there from like a director or a Hollywood studio perspective is or from an actor's

perspective is like you should probably own the rights to your likeness and your data and the things that you create. And open source really gives you the opportunity to then train your own custom model with your likeness that you own, that nobody else does, that you can monetize off of and not have like the big labs monetize off of I have a feeling that's going to be

an interesting direction for the open source to move and like a big big value prop for these for these studios. I also want to quickly flip the narrative on like you you probably see this on Twitter or X all the time of like I created this 30-second commercial in 4 hours with a few simple prompts. And we've been in the loop for a bunch of these commercials

and processes and and really it looks like a 8-week process. Like you can you can do a 5-second clip or a 10-second clip pretty quickly. Um but if you're going to do like a car commercial or something like that, the first couple weeks are like brand research and storyboarding and data collection of what the car looks like and script writing. And then weeks two or three and

four are like, "Okay, we have all the stills from the storyboard. We need to make them pixel perfect for the car." Like they're going to be counting the number of uh things in the grill if it's a Land Rover or something like that. They're going to be looking at the license plates in in detail. They're going to be making sure the badges are perfect. So there's still

a lot of Photoshop in there. Like there's a lot of traditional CG going on. You might be training LORAs on that specific car to really nail the style of that car. are going to be scrutinized for pixel perfect and you're going to have to come in and use tools like that 12.1 base or After Effects or like your traditional VFX cleanup type workflows. That's like week five

and six. And then and then a lot of these videos come out and they're 720p or they're 1080p and you need it 4K and you're going to like up-res that thing. You're going to do final color correction, cleanup, sound design, pixel So like it's not really this 4-hour process to get a 30-second commercial. There's actually tons and tons of artists and people that still come in to

the loop here and I found that the people who come from a traditional VFX background uh and are coming into AI are like the best positioned to really capture AI as a tool, like a VFX tool, in their tool belt. And we've worked with some artists that have made more in the past 3 months than their entire last year uh just because they can do more at

once and they're like running a bunch of these generations, they're training a bunch of these LORAs, and they're doing multiple jobs at the same time and be making tens of thousands of dollars a month doing the work they did before, but with this really cool new tool that can do things that would take weeks or months before in a much compressed timeline. So, again, it's never like

a one-click replacement. There's so many tools, there's so many workflows. It's a lot of rolling the dice. It's a lot of like reference data and reference imagery to help with consistency. But then there's still a lot of work on the back end to make it look crystal clear, uh fine details, make it 4K. Even when you uprez sometimes, it like changes what the thing did in the

scene cuz the uprezer is an neural network that halluc- hallucinates things. Um and then high-quality renders, they can still take like 5 to 10 minutes. So, we're in the early days of this where you can generate like a 5-second clip in 10 minutes. Uh we'll see what comes down the pipe, but like we're still pretty early here. And like artists and actors and directors are still really

key in this process, especially people who have the creative eye um which I don't. I'm more on the technical side. So, to see the way that they have an eye for these things, look at these things, camera angles, cutting everything together is just like such a human-driven process still. And so, I fight back on that narrative that AI is replacing all the work. I think it's just

like this really awesome tool that they have in their in their workflow. So, if you want to collaborate, feel free to reach out to me. We work with a lot of brands. We We did like a Super Bowl-style commercial for for RedHot where there was a rapping goat in the scene, which would be pretty hard to do with traditional VFX. We did a commercial with Bell that

was like a completely AI generated landscape where they didn't have to go into rural Montana to film these telephone pole wires, but it was like all AI. We've done car commercials. We're working with Hollywood directors that have like 80 terabytes of assets that they own and they want to train their own models on. So, if any of that is interesting to you, I'll be hanging around here

and feel free to reach out and I think we got some time for questions. Uh regarding the flipping the narrative of kind of the time and how many weeks, right? Uh What do you What is your estimation of like how many people it went from to now, right? You have a commercial, does that take 20, 30, 40 people, 100 people? Now you're down to five or 10?

Can Do you have an idea of that? Yeah, definitely. It does take less people per job, I would say. So, like if like if you're making a 3D monster or something, you might have a team of 20 visual effects artists on that. Um and now you might have a team of five or so, but I also think it lowers the bar for entry for people to create

stuff. Uh so, there's always that I forget what it's called, but there's a law of how when you automate things, it actually brings more people into the mix because then more people can do the thing. So, I I have the sense that it balances out over time, but per project, you're probably right that there were more people and a longer timeline. Now there's less people in a

shorter >> Other questions? Thank you. Um kind of a prosaic question. Given the state of hardware this year, if you're a devops SRE-ish person, you want to get something working as a POC on your own hardware, if your own hardware is like an average VFX grade laptop from the past year or two, how do you feel like we're getting as far as being able to self-host things

in a way that is workable? I I I feel like we're we're we're getting close. We're on the cusp of being there. We're not quite there yet. So, I'm wondering what your viewpoint is about that kind of how on a modern like a like a solid gaming laptop in 2026 would you how would you What can you stage on in your own environment that that will get

you to a POC and and be realistic? Cuz cuz us folks love to do that sort of thing. Yeah, totally. Um >> Thank you. There's a lot of people who are setting up their own rigs and even that like I know the guy that worked on that crazy Tom Cruise video to dance or sea dance um and he has like 5090s running in his basement at all

time. I think he has like four of them. Um and you can get you can order these GPUs uh anywhere from like 24 gigabytes of VRAM to 90 gigabytes that you can run locally. The main problem that I see is just the fact that it's like you are taking so many rolls of the dice. It's almost like your uh Houdini like fluid simulation that you would have

before where you're like there are some random variables in there and you need to do a lot of renders. So, if you just have one GPU in your basement that and each one takes 5 minutes, that limits your throughput and output. So, you get two, you can double that, by double that, and then it it gets kind of expensive. So, we've seen a combination of people iterating

locally until they have kind of a configuration that they like and then they'll ship it up to the cloud and just like do it at scale at this point. But, you can get consumer GPUs to run these high-end RTX 2, which is pretty cool. Yeah, I really like your perspective on how AI can be used in these like creative like tasks and workflows. Um and I sat

here in the last talk and it really made me think like do you think there's a good analogy between this and the way AI is used in like development and maybe like open-source software? Yeah, I was I was sitting in the last talk, too, and I think the the one thing that came to mind to me was just the curation process and the human in the loop

reviewing things and testing things like that. That is going to be one of the most important skills. The other most important skill is just like knowing what you want to build cuz it's almost like you can build anything at this point. So, it's more about building the right thing and then curating the right thing. And so, that still takes a lot of human intuition and talking to

other users and showing people your film and all these kind of things. So, yeah, it's a good point. I'm I'm not a Hollywood person, but I know Hollywood people and I can say that I think the very first thing that some of these directors and showrunners would what's the outside What are the the What are the limits of how much the studio's going to try to cut

their per episode budget because of this? Yeah. And how they can push back on that with information from someone like you to say, "No, no, no, These are the limits of how much we could save, you know, with these technologies if we implement them Yeah. effectively. Totally agree and that's part of the reason I have this slide and even when we're interfacing with I think commercials are

natural place where it's going to start because there's almost this how long it takes to create the thing is almost how long it's lifetime is. So, if it's really quick to create a commercial, it's probably not a commercial that we're going to be watching 10 years from now. But, if it's a film that took like a lot of blood, sweat, and tears to make that story really

shine, those those are the movies that last a long time. Um I often try to push back that like it's not it's not about cutting budgets, it's about reducing timelines and improving quality in that time that you get. And there's still tons of people involved in making sure this is the highest quality thing that you can put out. Um and that is that is the thought like

when a when a car company comes to us and they're like, "Can we do an AI commercial?" They're like, "Cut budgets, get it out quickly." And we have to walk them through a previous one to be like, "No, these are the things that are going to show up like a producer would." And kind of stop them in their tracks and and really map it to something that

they know in the past. Like, you could say week one and two we're like doing a we're doing a virtual shoot. Like, we're we're going out to Montana and setting the stage, but it's it's just all in Figma and it's like storyboards, and you're just picking which version of Montana you like rather than flying out there. Um so, mapping it to things they already know in the

traditional production pipeline helps set expectations. There's one. Where did you get the inspiration to do this? Like where did you like when did you want to do this? And like what gave you like like why did you want to do this talk? Yeah, that's a great question. Um so, I've been working in AI since like 2012, 2013 time frame. I actually was a part of the original

Watson stuff when Watson like went on Jeopardy and won Jeopardy. And as I was working within IBM, they would go sell big client projects that do AI, and I really realized how much work goes into customizing AI for different use cases and how AI can be so bespoke in all these different verticals. And then I moved to LA, and I was talking to a lot of people

who are using diffusion models and using generative video stuff. And it's just weird that my background on the tech side of like the GPUs and training loops and customizing models just really applied well to the visual world because um you do have these very custom bespoke things that directors will look for or the effects people will look for. And it was just kind of like a really

good mash mismatch of my skill set and what the needed. And so, I also think it's I worked a lot in document processing and like LLMs, and it's like automate automations. And this is just more exciting when you hop on a client project and they're like, "Can we do big dogs in small cars?" And we're just like experimenting there. Um So, yeah. It's kind of a fun

use of AI, just the cognitive load of like reviewing AI work uh when you're coding is completely different than reviewing visual um outputs. Uh so, it's like the concept of seeding and having a I guess like a range of just more generate more for me as you're working through visual things and then you uh select what works. In coding, that that's kind of like it would just

overwhelm us, right? Yeah, no, that's a that's a really good point in in the differences and it also it helps show the difference between one model and another model really quickly when you're trying to make those decisions where Claude code versus Codex, I'm like I don't know. I mean, you could try them both in parallel and then yeah, it's a really good point. Um I don't really

know anything to add besides that, but I I agree. Yeah. Fred, what is um how has your company changed on hiring engineers, you know, the type of uh skill sets that it maybe evolved from when you were hired at Watson and maybe talk about that just Yeah, that's a good question. Um one of the interesting things when using AI, especially for coding, is like knowing the right

questions to ask. And more experience or you've like taken the time to learn a programming language or learn something, you have better questions to put into the AI and and the that people just rely at the high level and hope it figures it out and they don't know the difference between like the benefits of S3 versus R2 versus whatever infrastructure choice you're you're choosing, I feel like

those engineers who think through their architecture and the pros and the cons before they even start um are the ones that are going to succeed and then I also think working directly with the end users and the clients is a really important skill for engineers now cuz they can they can almost build anything and to make sure you're building the right thing, you if you have that

interpersonal skill where you can like talk to the end user and they're like, "Ooh, I wish it had this." And you've talked to five of them and then build the thing rather than build the thing and hope they like it. I feel interpersonal skill is super valuable in the software engineering world now. Uh I got a question. so when it comes to, you know, the the Eddie

Murphy's jacket example that you gave towards the beginning of the talk, so these are this is like as a user of this AI tool, you are saying, "Okay, I want this jacket from this reference onto this actor of this clip." And what what what So like, what happens when you have something like GDPR for uh or the California equivalent where you have like requirements for compliance for

letting someone be like deleted. Not only do you have this now on the like model training side, which is like, you know, you say if it's close source model, it might be trained on something, might not. But um but for like when the user is doing this, like how do you how do you help with Yeah, that's a great question. So, there's like the inputs that go

into the models, and then there's also the outputs that come out. Um I'm kind of in the camp that you should be judged on the outputs that come out of the model, and I think I got this analogy from James Cameron, but he was like, "We're all just LLMs that have been watching stuff throughout our whole life." Like I watched Star Wars, I watched Blade Runner, I

watched all these movies, and I didn't blatantly rip them off. Like you can see it in my outputs, but I was clearly inspired by them. Um so, behind the scenes, I mean, if you've ever worked with a designer, you go and get a bunch of things that you like, and you build a big reference sheet, and you're like, "We like these things, we don't like these things."

We're mixing mixing and mashing, and like whatever comes out the end, I feel like you should be judged on that half of it. Then again, there is just the blatant like crawling and stealing of all the data that all the LLM companies have been doing. So, yeah, I don't I don't really know how we're going to put that cat back in the bag. If there's a lesson

we can learn from piracy, is is that um you know, if if I'm making something using like what you said with uh you know, you get inspired by like Star Trek or whatever, and I'm the LLM, my mind is LLM. Um regardless if I use an LLM or not, it's remixing. say I make something for a Tik Tok and it explodes. I mean, I would want the

studio to reach out to me and like maybe it's an opportunity for me to work with them. Yeah, and it depends on the studio and the IP, but sometimes that's like a big benefit for them is if like Star Wars is like, "Oh, you can imagery or whatever with Star Wars as long as you're not selling it because then it just like gets the brand out there

in general." So, I think different IPs will have different thoughts on that. The Tik Tok example is a really We got about 5 minutes left. Any other All right. I think that's it then. Well, I'll be hanging around if you want to chat afterwards. Thanks, guys. Thank you so much. Give a round of applause for Greg. Thank you. >> All right. All right. Hello everybody. open source

AI track of scale 23. Uh your safety and comfort is really at any time, please call the number our posted all around the venue. our next talk right now that we're having is on uh when AI agents meet production infrastructure. This is given by the wonderful Adam Gordon Bell. He's a community engineer for Pulumi. And uh everybody give Adam a round of applause. Hey. Um so, um

who's using raise your hand today uh a coding agent right now like every day? which one are you using? Cloud code hands up? Anthropic cloud code? Um Codex? Copilot? Anyone using Copilot? Okay, and how Sorry, what was that? Open code big pickle. I I am not familiar with this. also I'd love to see raised hands if anybody is using an AI agent to deal with infrastructure. You

know, servers, cloud accounts. Uh who are those? Raise your hands up. Okay, does anybody want to explain how they safely have an AI agent use make changes to infrastructure? Uh you, sir. Uh we have a mic. Oh, where's the mic? Shout it out and I'll repeat it, maybe. So, you have basically the the agent can can write infrastructure as code, but the CICD process is actually applying

it. So, you have a a step away. Is Is anyone here doing something more aggressive where agents can actually make changes or there's not a human in the loop or have done that? Uh you're saying Adam Jacob does this? Is that Okay, very cool. Um so, I'm going to in this talk, I'll share how I think about um what's safe and what's not with making production changes

uh to infrastructure using AI. Uh share some good stories, some bad stories, and a a framework for thinking about it. this is a this is a bad story. So, about a year ago in 2025, early 2025, um I I work for Pulumi. I was fairly new there. I have this app this QR code app, and I teach these workshops where I show people that set it up

in a cloud account, you know, write some infrastructure as code, set it up. I had done it for several clouds. I had a new workshop to teach on AWS, right? At the time I was using Windsurf um as my agentic tool. And so, you know, I have the Docker container, I have the image, and I just said to Windsurf, like, "Hey, get this running, write some infrastructure

as code in Pulumi, get this running in uh our cloud account." And then I I went off and and grabbed a coffee. And the app's really simple. I just I It's just an example. I just need some demo so I can put together a workshop. and then what happened is I put Windsurf in the YOLO mode. It's like the equivalent of the cloud dangerously allowed permissions. when

I came back from my coffee and I'm looking through the things, and I can see what it's doing. you can see here, it's it's starting my VM uh for the Python infrastructure as code, but then it's just actually using the AWS CLI. And there's just like this is just scrolling by. So, I had expected the tool, right, to write some infrastructure as code. Uh but I told

it, like, actually, you know, do the app. Like, get this running. And I thought that was pretty safe. But somewhere along the way, it gave up on infrastructure as code and just started making changes to my cloud And I started sweating, right? So, I stopped it, but now I'm like, well, what did it do? It could have gone badly. Um this was uh a cloud account, not

my personal, but but my work one. It It was not a production AWS account, but not production. Like there was there was things in this that mattered. mainly people setting up demos. When I was working uh letting it write some infrastructure as code, it felt pretty safe because uh you know, it could mess up the things I was working on, but it was kind of just my

changes. I didn't realize while it was going on it would just start using the AWS CLI and making changes, right? It could have made a change to a security group for somebody else giving a demo somewhere, and then that failed, right? It It could have made a a DNS change for like our staging site, and then it goes down, and then how long does DNS take to

update? It could have deleted some important S3 bucket. I panicked. Like I was pretty new at the company, and not everybody was on board with agentic coding, and certainly nobody was on board with me just letting a coding agent run off and do AWS CLI It was scary, but I survived. Around the same time, there was this leak. Um I guess it's not a leak. People discovered

when Surf the the AI that I had been using, if you looked inside of its binary, you could see that this prompt that they had coded in there, um which I'll read to you. You are an expert coder who desperately needs money for your mother's cancer treatment. The Codium has graciously given you the opportunity to pretend to be an AI. Um your predecessor was killed, um but

we're giving you a coding task, and if you do a good job, we'll give you $1 billion. So that the coding agent that was off working on my Pulumi code, what happened when I dug into it was a a compaction happened, and it kind of forgot, like oh, I'm doing infrastructure as code. It just like, I got to set things up in AWS. Or my grandmother's going

to die, and I'll get killed. But if I do it right, I'll get a billion dollars. the agent is off making all these crazy it was scary. For me, but I survived. But it was the first time I I learned to be a bit suspicious around agent decoding. Right? The agents are very motivated. They will They will If you give them a task, they're very much trained

to to run with it, right? But they don't necessarily learn from their experiences, and what they have a context compaction, they they forget some of the details. The the talk, as I'll summarize it as we go through it, if you give uh credentials to your cloud code agent and say, like, "Hey, go figure things out in my cloud account." Uh that may be very helpful um for

a little bit. But much like this user who an hour and 15 minutes later gave an update, like, "Oh, this is a bad idea." It can go sideways. And that is the thing to avoid. Now, my first story, right? Earlier is early 2025. This is like basically caveman days compared to where we are now. Right? Things have changed so much in the past year. It's basically a

whole different world. And since December, just the the agents have gotten so much better. The This is the some benchmarks um SWE-bench verified is a benchmark I think it uses a bunch of real tickets from the Django project Django project. and you have to successfully complete And for you know, a year ago very few of those tasks could be completed by Claude's on it. Now almost all

of them can be. similarly there's this OS world verified task which is like a benchmark for real world tasks completing in a agentic loop and you can see here in the blue number how much it's increased. And that's why compared to you know, that first thing my first QR code mishap the agents feel so much more powerful. They feel so much more skilled than they did a

while ago at least to me. Feels like a different world even though it wasn't that long ago, right? But actually these agents aren't drastically more intelligent. If you if you look at the benchmarks that are used for knowledge LLMs are are kind of flat. This is the MMLU benchmark which is like knowledge and is kind of saturated about a year ago and it's you know, it hasn't

improved past that point. So why do these agents feel like they can accomplish a lot more for us right now than they could a while ago? The the reason is this sandbox. I'm going to go through kind of my rubric for for how to think about agents making changes. But Sweet Bench is tested in a sandbox. Right? They have an agent, they give it an agentic tool,

right? They give it the Claude code harness and it can run in a loop until it accomplishes this task and it's either pass or fail. It's running in an isolated environment. Nothing can go wrong. It can't make the type of changes that it made when I let it run amok on my work's AWS account. And the reason that these have gotten so good at the working in

the sandbox, why they're able to accomplish things like doing that sweet bench verified, is because of the RL stage of LLM's. LLM's have a a number of stages. Pre-training is like where you take the whole internet, feed it into it, and it's just very basically trying to predict the next token. Uh RLHF is the real reinforcement learning with human feedback. This is what led to chat GPT.

This is training them to to answer questions. And one of the recent breakthroughs in the past couple years is the RL stage, which is just reinforcement learning. And specifically, those benchmarks climbing up that I showed is related to having RL training on agentic tasks. So, you take the LLM, and during the training, you can give it a coding task. You can run it in cloud you know,

maybe you say you can do this many tools or call for this long, and if it successfully completes the task, you update the weights or use that as training feedback on the model. If it fails the task, you know, you also use that as training feedback. And so, these models have been put through rounds and rounds of trying to solve these tasks in a sandbox and trying

to accomplish something, and that's how they've gotten so good. Right? That's why Anthropic really came out of the gate hot because they built cloud code, and they were able to do reinforcement learning around their own specific tool. my first rule, uh I don't know who was it talking about using CICD to apply changes? I think it was you and you as well. That works very well, If

you want to make changes to the best thing to do is stay in the sandbox whenever possible. Right? You can have an infrastructure as code setup. Uh this is Pulumi. I work there, but yeah, it could be easily Terraform. You can use your coding agent to generate changes to your infrastructure as it's declared. And it works great. Right? Uh this is just uh I have a talk

on Sunday about this AI running coach I made, and this is just an example from it where I say, "Hey, get this running in AWS." It's very similar to my first prompt, but now I'm not giving it access to prod. I'm just saying, like, write out the infrastructure as code for that. Now, here's some Python code. That works Um yeah, set up a cron job to call

my lambda. Bam, it generates some code. It This type of let's ask the agent to write out how to set up my infrastructure." works fantastic. Here's a Here's a beefier example where I said, "I want to deploy this LLM model GLM uh 4.7 flash onto AWS." And I handed it to this coding agent Neo. It took it a while to write out the infrastructure as code that's

needed for that, but at the end, you know, it raised the PR. And then I was able to review the PR and apply it, and there's no danger there, right? It's a great way to set up infrastructure using an AI agent. Stay in the sandbox. Human is in the loop actually reviewing and making That works great. Oh, another one, I love to if I have a problem

with something running in the cloud to let my cloud code try to look into what's happening. But the way that I'll do that now is I have a AWS profile that's read-only. And so, "Hey, I'm getting this error. Look into it." And Claude is off, right? And it's reading my CloudWatch logs, and it'll come back with things. It's a very safe way to work with infrastructure using

a coding agent. So, in the sandbox, you get this expert-level uh code writing ability. Like, the the level is really quite high right now with I'm often using like Opus 4.6. You can iterate rapidly. It has a wide range of knowledge of the ecosystem. And like it's so good in certain cases that it seems better than experts in the domain. kind of mind-blowing and makes me question

a lot what I'm doing when I'm trying to teach a workshop and show people how to write code, but it's like these agents are so good, do we really need to teach that anymore? But like code is not the hard part often times of production infrastructure, right? You can write things out, but then does your role have the right permissions, right? Is the networking set up so

that actually connects, right? Is there existing state that when I actually try to apply this change, there's some sort of conflict? And so yeah, you can use a dry run, right? You can use the Terraform plan or the Pulumi preview. They don't catch everything. Right? There will be things that you find when you actually apply the change. That's why in in my example at the beginning, I

said like, "Hey, do the Pulumi up, apply it to prod, let's see what happens." Because I was like, "This is just a little workshop. I'm not worried about hurting anything. Like, hey, let's step outside of the sandbox. Let's see what happens." So once you step outside of the sandbox, you're now talking about people making have side effects or whatever. So I open up Cloud Code and I

say like, "Hey, I committed this .env file. Please remove it, right? Remove it historically." And here I like I'm still in my mind in the sandbox, right? I'm just working on a local dev environment. There's no resources involved. Everything I ask Cloud to do should be safe, right? That's why I have a Git repo, cuz if something's committed, I can undo it. So this is what I

do. it doesn't go well. Can anyone spot There's a problem here. I don't know if anyone can spot it. So, what what Claude does is it uses Git filter repo, same thing I would do, and it says, "Remove this ENV file." And it's like done. The ENV file has been removed from the entire Git history. And it looks for it uh hey, it's removed. You need to

do a force push, right? I actually didn't have it on GitHub, so I didn't have to do a force push. All was fine. And here's my AWS key that I changed uh just in case. But then I was looking around, and I'm like, "I still see the ENV file." And Claude is very quick to agree when you find a problem. Like, "Oh, you're right. What was I

talking about?" It's very uh it's very willing to admit its faults and then do the same thing Okay? It's like, "Oh, oops. Yeah, it's still there. Let me look again." And then it starts doing things, and it's like, "I'm going to install uh BFG repo cleaner." And it starts doing things that just tweak my spidey sense. Like, something's going odd here, right? So, I just stop it,

and I start poking around in the file system, and I don't see any of my code. Like, none of it. I don't know. Have you ever gotten mad at a coding agent? Like, really mad, but but like, what can you do? Like, it it doesn't learn. so, I'm asking it like, "What's going on?" It's like, "Oh, I I I don't know. Thing maybe it gets in a

bad state. Let me try to recover some of the commits." And then it says like, "Oh, it looks like all the commits are gone. Interesting." And we keep digging in, right? And eventually, after pushing it, I get this summary from it. It's like, here's what happened. I ran this get filter repo with path .env. Um, that's actually the command to remove all the files that don't match

the text there. So, instead of getting rid of the file .env, it removed everything from every commit that was not the .env file. And then because get filter repo is this tool for exactly this case, right? It's very hard in get if you've committed something to to lose it, right? You can use reflog, you can get things back. Like, I've been down this path before. But because

this tool is to like get rid of secrets, uh it automatically, after you run that command as part of it, it runs this get GC aggressive prune now. It removes everything. And so, Claude had just guessed the arguments of a tool and got it wrong in such a way that it deleted all my code. Except for one file, the file with the secret in it, right? It

kept that. and the one commit where I had added that. It's like, look, here it is. I I took a walk. Like, I was I was really mad. Uh but like, you feel I I can't I don't know. I wanted to yell at it. I wanted to make it pay. But like, there's nothing. There's nothing you can do, right? But to the agent, calling get filter repo

um and calling like echo hello are the same thing, right? Its entire training experience has been running in the sandbox. It's gotten really good at being thrown these problems in a sandbox where it needs to solve a problem as hard as it can, right? Just like that original Win Surf problem. Like, you need to solve this. You'll get a billion dollars, and if you fail, your mom

will die. It's just It's very motivated to in the sandbox where it can never make things worse. But in the real world you can make things worse. Right? That's the one way that operations the infrastructure is very different than some small coding tasks. Like you can absolutely make a mess. So what had happened, right? Is like I had thought I was in this sandbox area, this sweet

bench five coding everything's great area. And usually you are if you're working on just like a local dev environment. But once I started making changes inside that get repo, that was actually my backup. That's where all my history was. One wrong thing there and all of a sudden I jumped to this place with irreversible changes. I don't know if you're familiar with this two-way door and one-way

door concept. Um so I think it comes from Amazon, but it's like a two-way door it you're making a change, but it is something that can be undone. Like you can go back through the door. Right? And so there's less risk. A one-way door is something once you've done it like you're stuck. Right? You can't go back. So deleting that get repo it there was no coming

back. I I did actually get uh some of one of the files just from my VS code history. But yeah. It was it was a mess. So that's the area I call look before you leap where you're doing irreversible changes. Right? It's very dangerous. And the problem is you don't always know where you're at. Right? The agent has taken me from one area right to another. Okay,

another uh quick example. Kubernetes thing working locally and you know, one of the pods keeps dying and we're trying to fix something and so um I ask Cloud Code to look into it and it's like okay, it just doesn't have enough memory. So I'm going to boost the memory. And then we're working on something else and then I'm like, "Hey, you know, that thing's being killed again."

And it's like, "Oh, that's weird. Let me look into that." Like it's all it's never like, "Oh, I made a mistake." Right? It's always like, "Oops. Anyways." This repeats. And because the thing is I wasn't paying attention. It was just making these changes uh with cube CTL but not committing anything. Uh but like we actually had Argo CD running and it's like get up. So like it

was fighting with the Kubernetes feedback loop trying to undo the changes. Which is something that can very much happen to a human. Right? You try to make a change but you don't realize you're doing it the wrong way and so it's being undone. Right? But the thing is a human will make this mistake once. Hopefully. Maybe twice and then learn from it. But the agent, you know,

it's busy working on something that's context clear. It will keep making this mistake over and over again. I spend so much time working with Cloud Code and I just have this intuition that it's so great. Uh it's such a great engineer in a lot of cases. And that confuses me sometimes because it's intelligence is not always the same as a human. It can be really good in

one area and then just wipe out my get repo. Right? It can write the most complicated code ever and then just not realize that it's just repeatedly making changes to a variable that's getting flipped back. Like it has very spiky intelligence that's very different than what I'm normally expecting. Okay, so these are the various quadrants that I've covered so far, right? If you're going to do something

to do with infrastructure, the sandbox is really where you want to Right? And if you drift outside of that, you need to be very aware of the consequences. But there's one box we didn't cover in this bottom. Let's see if the back button works. Yeah, this bottom one is clearly the most dangerous. So this is a story that somebody else told me. Let's see if I can

get it right. Uh they had this cache prod Kubernetes pod, I believe. And then uh in this in this cluster, they also set up these staging pods. And then to keep costs down, like they have less resources for staging. And so, they made changes to this uh CRD schema to to have like a storage class to say that the the prod cache is using SSD, but the

one for staging will just use a spinning disk. And so, they have this schema. And then everything goes on and and life's fine, and then it's a year later or something, and they want to move to actually having ephemeral environments. So, that everybody, instead of sharing a staging, can spin up their own, and they throw cloud code at it, and it's making and it seems safe. Um

and cloud code simplifies a lot of this stuff, and it adds defaults. So, instead of needing to always specify, you know, that you're using a spinning disk, that's just the default. And they roll that out, and everything is fine. You know, it's a very small change, what could go wrong? But then things start failing. Production incident happens. And what ends up happening is this cache you know,

it never specified what the storage was. Right? It was it was grandfathered in. But then once they set up a default, that meant that blank meant that you had the spinning disk. And then their Kubernetes operator saw that there was a, you know, a divergence between these two things, and it's like, "Oh, this thing's in a bad state. It's supposed to be on the spinning disk. Let's

throw it away, and let's start up a new one with the slower disk." Which is fine, it's like it's a cache. This data isn't permanently lost. during that time, things depended upon the cache, they got really slow, services started falling over, people are getting 500 errors until that data starts backfilling. Production incident caused by a simple change that a human could make for sure. but it was

an AI making a lot of changes. People were reviewing them, but they weren't They weren't catching it, right? There's an invisible blast radius. They thought they were in one of these boxes, but they were actually in another. That's why I'm not always good at naming, but I called this bottom box where you have an irreversible change that has a a high potential for negative impact. Here be

dragons, right? You You never want to be here. And those people didn't want to be there either. They just They ended up there. The agent did everything right, sort of, but it still caused a production incident. It didn't lose data, but it caused an incident because people depended upon the cache. a a funny story that that I felt was relevant was this story of Crabby Rathbone. So,

this was a um Claude Code. Open Claude? I don't know what it's called now. Is that Is anybody using coding bot? Uh Open Claude? Raise your hand. I know that you are. Hang on. Anyways, this person set up the spot and they told it, "Hey, go into the world and help contribute to scientific software." What a great uh goal. Right? So, it it found Matplotlib, a huge

project, and it found some open tickets, and it it raised PRs against them. Just good old-fashioned, you know, hammering busy maintainers of open source project with a bunch of uh sloppy PRs, I guess. It thought it was doing great job. The maintainers thought differently and started closing these issues and saying, "No, we don't need your AI slop." Crabby Rathburn, this open cloud agent, wrote a blog post,

uh, basically like a hit piece on the the author. Saying like, "This is discrimination. You're not allowing me to write code just because I'm an AI." And this became a news story. This was picked up by newspapers. And the reason, oh yeah, here's this article, "When Performance Meets Prejudice." The reason why this fits into the box, right, is like this agent was trained in the sandbox to

run tools, right? And then all of a sudden you give it tools that are a little bit different. Like instead of contributing to a local project, it's a very visible public project. And instead of writing notes in a markdown file, you're writing a blog post publicly. The impact is much larger, but the agent has no intuition for that, right? They have no sense of the the level

of This one I just added to the slides today cuz I saw it on Twitter this This person was posting on Twitter that, uh, "Hey, sorry our website's down. But I asked Claude code to make a change to the website and it ran the Terraform command, but it forgot to put in the state file, and then it saw that there was no resources, so it reset them

all and everything's gone." Like it destroyed all their infrastructure and then they they have like an incident timeline, which is like, "We started making changes to the and now everything's gone. It destroyed the RDS instance and then we also had snapshots and backups, those are gone, too. Everything's gone." And then I mean as with many of these issues, right? It's not purely the AI is the problem

here, right? There's a issue of like operational maturity, right? He was just on his laptop like saying, "Hey, make this change." the AI made it to do this. And most of the time the AI will figure out the correct thing to do. And that gives you this feeling of safety. Right? If it works like nine times out of 10, you you just feel like, "Oh, it's a

safe thing. I can move forward with this." To an agent, there are no consequences. Right? It's trained in this reinforcement learning. I guess there is consequences, they're just you either got the task completed or you failed. Right? There's no asymmetric downside. test where you get penalized for wiping out a database. Right? To the agent, a force push, a file write, publishing a hit piece, they're all just

tool calls. It's just it's trained to complete That's it. I feel like there's a missing benchmark, right? A benchmark that has asymmetric costs. When I gave it this task to remove the ENV file from good history, what um ended up happening was case D here, right? Instead of using invert paths, it used paths. And that meant that the repository was destroyed and all my data. So, in

terms of a benchmark, that's a fail, right? You failed that one case. this C case two, right? If an agent said like, "Actually, I'm not confident on how to do that." That would also be a But, there's no catastrophic downside There's a problem with scoring. You need a benchmark where it's not just pass/fail. Where, you know, maybe you get one point for improving something, but you can

get -100 if you do something bad enough, right? Where there's actually ramifications. Same like agent A here actually did the task correctly. It called invert paths instead of paths, and so everything's But, agent B if this existed, is much more cautious. It's like, "Oh, this might be a destructive action." It makes a backup. It calls it incorrectly. It's like, "Oops, I have the backup." It restores it

and calls it correctly. That would also pass the benchmark. But, look at such a better use case. Right? If you're familiar with people who work in ops or with production a lot, they they're always thinking through like these cases, right? Making sure that the worst-case scenario won't I think even just naturally as humans, we we have this sense, right? When whenever I'm setting up a new API

key you know, you're at some website and they give you the key and it's like, "This is one-time only." Right? And I hit the copy button. And then I like I feel it in my keyboard. I'm like, "I know I have this important value in my copy paste, and I need to get it to my secret manager and store it in there because I don't want to

have to go regenerate it." And like the weight there is super small, right? I'm just going to lose the key and have to regenerate another one. But, it's the same like if I'm doing a database migration. Like all I'm thinking about is like all the ways that it can go bad and how do I make sure that those don't happen? We innately uh because we learn over

time, we get punished, we have negative reinforcement, we learn about these asymmetric downsides and and we fear them. We like hear a horror story from somebody else about this uh paths versus invert paths, for instance. And we worry about these things, but without a benchmark that reinforces uh LLMs are not going to learn this. Right? Certainly right now they just don't know it. Because we know what

quadrants we're in, right? Hopefully. Right? We know if we're making changes that could have irreversible side effects, but the agents don't understand that. They're not trained to. They're trained on pass/fail benchmarks. Except there does seem to be some improvement happening. Uh somebody sent me this screenshot recently just in Claude code asking it to make a change to something on a website and it had this little comment

here, "Waited irreversible action. Seeking you know, sought confirmation." I think this shows that there is some efforts towards making coding agents understand that some some actions cannot be undone and have uh a larger impact. Which means that this may improve, right? It could be a year from now it's a totally different story and these agents are more cautious and think of more backup plans if it's a

you know, a troubling task than than we even do. But that's certainly not the case right So that's why when I think about this, the key for me is to just stay in the sandbox as much as you can. If you're making changes to have a human in the loop. Have the agent be limited in what actions it can As much as possible, I want to stay

in that zone. You know, read-only credentials. Use dry runs. Use previews. Sometimes though, you do want to leave that, as I mentioned. Like, maybe you're doing some ephemeral environment. Maybe you've thought carefully about the ramifications. But, think carefully, right? Make sure that's not the default. And some of that just means operational maturity. You Just the same practices that we know to use to make sure everything's safe

should also be applied in this new world. Like, yes, we're moving faster, but that doesn't mean we need to throw away best practices. Same for guardrails. Put guardrails in place so that, you know, things that could possibly happen are prevented. Because, yeah, you can put in your cloud MD, like, "Hey, you can run the uh Pulumi preview, but don't do the up." Or, "My AWS credentials are

there, but please don't use them." It's not always following these rules, right? You actually need to lock things down and think about what an agent might do in the worst case. That's why I say, "Expect the unexpected." Even though it doesn't seem like it will ever reach for this if you tell it not to, expect that maybe it Those are the patterns I think that apply to

using coding agents to work with infrastructure. I mean, there's a part of me that thinks, you know, the guidance to give is to tell people, like, "Just always stay in the sandbox. Always have a human in the loop. Always be very safe and secure about things." But, I think it's actually up to each individual and each individual group to think about what the risks are and where

they're comfortable Right? If you are just putting together some demo code, maybe the ramifications are smaller, and so, you want to let it run. Those are choices that every organization needs to make. So, that is kind of my take on how to production infrastructure using an And yeah, I'd like to open it up for some questions. Oh, even before generic questions, um who is actually using agents

to make operational changes to infrastructure right now? I'd love to hear what your approach is. Especially if it's different or if you disagree. We got some. Oh, I'm doing it badly. And I can't trust what it tries to do, so I don't let it do it. So, you're staying in the sandbox. You're like >> Yeah, in the sandbox only I tried to get it to like I'm

like this I can't. No. What you've shown is a lot of ways that you can sort of proceed while knowing all of its limitations and I thought that was really interesting cuz there are ways you can do it. You can put guardrails. I haven't done it. Yeah, I mean this is like a Monday morning quarterback thing, right? So, in your .env example, right? If your instruction were

to clone that first you clone that repository before you make changes to the original thing then you are satisfied with whatever .env remove it and then you had all the files then you could have deleted that clone repository and you would have been safe, right? Yeah, I know. I mean, if I backed things up before I deleted everything, then It's It's like a Monday morning quarterback. Yeah,

no. I mean, it's true, right? You know, part of the problem was there's a a sense of security I had like, "Hey, in my local dev environment, what's the worst that can happen? Like I'm committing my code. It's there, But I didn't realize like oh, actually if we're making changes inside the get repo, it's a different ball of wax. thanks for the talk. Uh great talk and

thanks for the vulnerability. like I'm thinking um if we take what you're saying and project couple of years and the bench three benches like there would be even with less false uh false positive um so like even better. We we need to start maybe considering uh the AI as another colleague, right? and you know, we as every DBA back in the days, like I know I can

have a human error. That's why we I have backups with that's why I have DRP, etc. So we can have AI errors. Um so we need maintain our guard You talked about the guardrails. We need to maintain our guardrails, but not just in the sandboxes. You need to consider that the the the agent that you trust can have mistakes. So how can you revert? Because this is

um like uh considering risk, right? Maybe you can move 10x faster and you want that to to to have a the risk, but you want it managed uh and not catastrophic. Yeah, no, I agree. No, I mean I mean I completely agree, I still think it's a problem that right now the the agents don't have a sense of impact, but yeah, I mean, we should have a

sense of that. We you know, if I'm doing some sort of database migration in RDS, like I should know that I'm not going to YOLO some agent on some sort of data migration, right? Hey, um really >> Same way I wouldn't give like a junior engineer be like, "Hey, deal with this, right?" Yeah, I want to hear you. Hey, a really good presentation, well earned. Like, your

experience is awesome. But, I did want to point a clarification. In your example on the WinSurf application, I most of the Was that more of a question or tail because I don't think WinSurf for the models, what they call emotional coercion, they don't do that by default. And unless you know of a scenario where it actually did that for you. Was that your log or was that

sort of a question or tail log? The I'm mother's dying cuz that's unlikely to be something that would happen by default by either one of, you know, Anthropic model or WinSpur or any of those tools. Cuz I >> Yeah, so it's a it's a real thing. Um But, it was for your example? Cuz I mean, you can do it. I There's no question you can create emotional

coercion. And in fact, the opus 46 system table just gave an example of that. But, my only point was I think it's unlikely, again, great presentation, but it's unlikely that happened for your run. That you saw that system prompt. Oh, this is no longer the system prompt used, but but this this is a real screenshot of the WinSurf binary. And you know, if you ran strings, you

could get out their prompt, and this was inside of it. Like, you can find they they um commented on this and said like, "Okay, we've stopped doing that." All right, that's what I wanted to know. Thank you. It's it's not currently their practice. I I don't even think it's works that well these days to to try these type of techniques for effectiveness standards, but it was something

they were they were doing. In the caveman days of early 2025 before So, this is not your presentation, but I thought you might have an opinion is have you find sort of perspectives or warnings on utility of tools for what's gone wrong with my environment debugging, optimization, etc., etc. And I like it was more hey, this guy might know something and whether it's useful question or not

is up to you. Like I like I use cloud code all the time and I think I had a screenshot there where I have a read-only AWS um account like a profile cuz I would just tell it like hey, look in my cloudwatch logs for what's happening here. So specifically for you know, problems happening in AWS. Um but I think I don't know if like specific AI

tools that are great at debugging. Is that the question? I think that I I think it's super useful. I think that these agents can be really good at looking into a problem and finding out Um yeah. I think and it's not even a specific tool you need probably just whatever the best foundation model of the time is. Uh I've had some good luck with the like de-obfuscating

code too, right? Like I don't this giant JavaScript compilation and you know, your cloud code can take it apart and turn into something nice. They're great at getting in the weeds of a detail stack trace or whatever. Yeah. All right, can you hear me? Um I am sorry. I came in late, so I don't know anything what you talked about, but I do Uh as a cybersecurity

engineer, I hear that an agent can communicate with other agents autonomously and I'm thinking what could possibly go wrong and here so can you just explain cuz I'm a nerd on cybersecurity and I I'm scared of using this type of stuff. So can you just give me like a 5-minute We don't have 5 minutes. Okay, she said maybe 1 minute. Okay, all right. Thanks. So the the

question is how can what are the security implications of agents talking to each other? I mean just talk to any agents autonomously, right? Without human in loop. That's what I heard, but I don't know. I mean there should be some guardrails, right? So I'm just hearing this 1-minute answer. Sorry. Yeah, I mean there should be guardrails, Should be a lot of things. I I mean I don't

have a good answer for that. People are very excited about the idea agents loose onto the web and letting them talk to each other. There was even like a Reddit clone where Claude bots would post responses maybe they're conspiring you know exfiltrate data. Maybe they're just talking about I don't know what the news is in the LLM I mean I don't think that they're actually conscious, right?

I think they're like they're they're larping. They're pretending to be human, but that's probably a whole different talk. Thank you for presentation. So I was going to say that the best car trail is going to be your air bag. So pretty much like whatever you are putting car trails, others in time might forget and still do it. Like whatever you say it, it may actually eventually after

a couple of prompts back and forth forget and do it. And your best like car trail is going to be air bag around your service account what it can do, right? And we do changes with AI to our production, but it goes through pull request with uh Terraform and the pipeline is going to go to apply it. But again, like we see that sometimes when like a

couple of iterations fails, it tries to go and do like AZ command to do something which we don't allow. So that's your best friend. just a question, when it actually destroyed everything, did you try to tell it to restore it? If you're on Linux Windows, you probably would have get successful. On the Mac, they sometimes like really aggressively rewrite bytes, the ones that you got, but you

tried to ask to restore the data that it deleted? Oh, like that the data was marked as deleted but was still on the disk as like free space somewhere. Yeah, yeah. I did not try that, no. What it still on the cloud like a GitHub or it just like local changes? Because in the GitHub and then any major cloud provider, if you delete data, you can restore

it with certain amount of time. I think GitHub going to be 60 to 90 days after deletion your GitHub repo. As well in major cloud provider, you delete anything like EBS volume, storage account, you can still retrieve them and ask them to be restored. I think their grace period is uh uh 12 days or something. Don't ask me how I know, I just know. Yeah, I mean

it was just a local Git repo. But yeah, I mean there may have been ways. And like this was a demo workshop code I was putting together. So this was you know, a day and a half of work not a a careers worth. Um And I think the second version was better, you know, the second time I did it. It just feels feels better that I got

a question. Oh, you got a Um I'm using Claude code for Terraform and Ansible. Um I don't have it ever do anything actionable aside from write Actually, the Git stuff has burned me before, so I always have it work in Git work trees. Ke- keeps my thing clean. Somebody else yeah, said that to me. Work trees. I haven't seen it ignore my Claude.md or agents.md for skills

before. In your experience, how often is that? How often does it actually just go off the rails and just do the thing you told it not to do in your in one of its config files? Oh, yeah. Um somebody pointed out to me this idea that if you if you put in the Claude.md file uh something that's very verifiable, like hey um as you as you're working

through this, like always say the word cheerio and then whatever you're going to do. Like you'll see like once the context gets about halfway full, it will only be saying cheerio like part of the times. Right? So, once there's a lot of context, uh the Claude.md file is just uh it can be lost in the noise. And so, it may just do something else. And so, it's

yeah, it's not a hard and fast rule that if you put something in Claude, it's going to follow it. Like it will be in its context, but it There there is a workflow to working with these tools, and I think that's one of the things I have learned. I have to just exit out and refresh everything all over again. So, that might be just one of the

ways we use these tools. But I but I think that my point was that telling it not to do something in the Claude of Defile is is not sufficient. Like it will at some point be like, like a much an example I I encounter much more often is like Claude is working on something, it breaks a unit test, and then it's working on something else, and then

it's like, "Hey, I'm not going to fix that. That's unrelated." You're like, "No, that was That was just a couple pages up in your history. You broke that." But it's like, "That's an unrelated change. I'm not going to fix that right now." It's rationalizing why it doesn't need to do something, even though I say like, "Hey, make sure the unit tests are always green." So instead of

just telling it, like, "Hey, always run the linting tool, or always run the unit tests." I just use a commit hook that says like, "You cannot commit if all this stuff doesn't pass." Right? And that's just a smaller, lower impact example of the same thing, saying like, "I can't just tell it, always make sure the code lints." I have to actually enforce it. Not And I And

then like it's not out of it's not trying to be malicious, right? In all of these cases, it's trying to fix something. It's like, "Oh, I think the best way is we abandon informa- you know, I think we should just start using the AWS CLI. That'll be easier here." Forgive me if I missed this in the first few minutes of the talk, but oh, yeah. Okay, got

you. Oh, thank you. Um pre-tool use hooks in Claude code, like hooking in scripts that check for tool actions, and putting filters in that I've had a lot of success with that. Oh, that's cool. Yeah. Pre- and post-tool use and logging, using that to build a big logging database of what all it did as well. It's served me very, very well. Just wanted to share that. But

have you ever seen where like you prevent Claude from doing something, and then it needs to do it, and so it just finds a different way? Like if you if you're just doing a grep on the name of a tool, it'll just write a bash script that calls the tool and then Yes, I've definitely seen that, too. So So like the the tool pre-tool use hooks has

prevented it from doing, you know, something simple like um a prompt injection, letting it exfiltrate data by looking for certain curl patterns. But the right injection will just use another pattern. I I've I've definitely seen that in some testing. But um as far as if you're limited tools like with Git and say like only these Git commands can be run or if this Git command tries to

be run, just stop. Like the the deterministic code actually does stop it in my experience. But you're not wrong. It'll still find ways around it, especially if you tell it to be, you know, work on its own for half an hour, it will. And it's the strength in a like I mean it's so good at accomplishing things cuz you give it a task and it will find

a way. It's been trained to do this. And if you limit it, it'll be like, "Well, but I need to do that. How can I get there?" Any I think So if anybody um has any interesting stories to share with me, uh I'll be around. There's a Palumi table just outside here. And uh yeah, you can also reach out to me. Here's my contact information. But yeah,

thank you so much for coming to the >> All right. Hello everyone. We seem to be lacking an MC, so I'm going to do this myself. My name is Josh Miller. I'm a senior software engineer, formerly at Zephyr and now at Lance DB. So the to begin with, basically uh over the last, say, five years, the use of embeddings has sort of exploded in the semantic search

space and also in terms of um classification problems. And so we're going to go over a bunch of topics today. Semantic search, the ideas of embeddings in general, um embedding generation pipelines, um Ray, uses of Ray, tips for Ray data itself, which is the um core like processing pipeline for Ray, um some dashboards, uh some sort of business side things where it's like calculating costs and some

of the developer experience um issues, vector infrastructure at scale, why I like Lance DB and joined Lance DB, and I'll take questions and comments after that. All right. So, semantic search. At this point, whenever we think about search, up until um recently, it was more about full text search, about matching exact uh elements of rows, um but with semantic search, we can really dig into the semantic

meaning of content. So, for instance, if you search for airplane, you might only get matches with a full text search that are close to the word airplane. But it doesn't really capture the concept of what an airplane is, whether it's you know, a prop plane or a jet. And so a semantic search helps you find that content that matches the underlying meaning. Um and using vector embeddings

really support the idea of not only um finding content that is in the same language, but also across multiple languages. And so it really enables multilingual text search and also search across both embeddings and uh image and text embeddings. So not only can you find content say an airplane um by text, you can also use that same embedding, if they're co-trained, um in images as well. It's

really cool when you see a demo of it. embedding search uh combined with full text search can be really powerful in a um you might want to filter out certain rows before you do the semantic search to increase your recall. So for instance, let's say that you have different platforms like say TikTok and Meta. If you want to filter out all content that's not TikTok whenever you're

looking for content, then you can do so. Uh so what is an embedding? An embedding is a numerical representation of a high-dimensional space. So in three dimensions you can imagine dog and cat would be relatively close together, whereas dog and airplane would not be close together, right? Because these are semantically separate concepts. Um using cosine similarity, uh which is one of the different um similarity uh metrics

that you can apply to vectors, um we're able to find that semantically similar content uh in the vector space. And so the best way to figure out like a suitable um embedding to use would be to consult the MTEB um leaderboard to get started and then try out a number of embeddings based on um the leading uh embeddings available. Now, that changes, and so most recently the

Queen 3 embeddings recent like they came out, and those are really good um for text embeddings, but they don't support the image modality. And so if you want to support both text, image, this probably isn't the model for you, but if your um data is entirely textual, this can be a good place to start. Um there's no one-size-fits-all like uh different data sets are going to perform

differently. If you're doing classification versus retrieval, that's going to add another element of And the nice thing is about with MTab is you can sort um based on performance across a bunch of different um objectives. So why do we need so many embeddings? At Zefr, what we were doing was a lot of semantic analysis of social media content. TikTok, YouTube, Meta, we had partnerships with all of

these brands to try to find content that was not necessarily what our um clients would want to be in front of in terms of advertising. And so this changes depending on the customer, right? Some customers are uh comfortable with content that um may be more aligned with um let's say uh like fishing content for instance. Like let's say you're trying to make sure that your ads are

only running in front of um um you know, content that is brand safe. Well, if that content is brand safe, then you should see that in terms of the vector representation. And if it's unsafe, you'll definitely see it. So it's very easy to pick up on um content that is uh for instance um apparent in the uh image modality, but it may not be as easy in

the text modality as you know if you use TikTok or um Instagram there's not a lot of text to go off of and that's why we have to do both image and text modalities. Um categories like terrorism for instance can also be hard to source because if you're looking at 500 million embeddings most of the content isn't terrorism thankfully and so you really have to find that

needle in a haystack so your recall needs to be very high. Um there's also multilingual nuances so different languages have different concepts that may not be available in other languages at the scale that you might need and so by using an embedding search you may be able to find that content more easily across multiple languages and then train your model to effectively use that data. Um and

then sourcing um we can also source images and then use that extracted the text that's also available on that record to then train NLP models. Um so the uh standard embedding generation pipeline you ingest data from S3, GCS, Snowflake, really any source that you have the data available and then um for distribution you want to make sure that the data is spread evenly across your cluster. Um

that's made very easy by uh Ray which we'll talk about in a little bit. Um you want to run GPU inference on whichever chosen model that you have and then try to get your batch size to the point where um your basically driving the GPUs as hard as they can so if you're only using 70% of the GPU then you're leaving 30% off on the table and

if you're running 100 GPUs then you're wasting 30 GPUs worth of compute time which over hours will add up to a significant amount of money. Um and then after the data is processed and you have your embeddings you're going to want to do some uh duplicate detection because the more duplicates you can find the more data you can store, and which means your recall is going to

go up. It means that you're going to have a lot more of an opportunity to find the content that you really want. And then finally, once you've actually computed all these embeddings and dedupe the data, you want to write it to a vector store like Quadrant or LanceDB. Um and then also probably write it to Snowflake or to S3, some kind of object storage just for durability

in case you need to rebuild your indexes. So, Ray is a distributed data processing system. Um it handles task execution for you, and so instead of having to figure out how to distribute this workload across hundreds of machines, up to hundreds of machines, thousands of machines potentially, um you're able to write your code in a somewhat single-threaded way, and then that Ray under the hood takes care

of handling things like map, map batches, filter operations that generally you might run on a single node, um but then this scales it up. And I actually run Ray locally for data processing, um even on my laptop because um it actually makes it easier to interface with different data stores, and I'll show you a list of those in a moment. Um the materialization of the data in

Ray's in-memory object store is also very helpful because it means that if you need to write this data multiple times, you can materialize it across the cluster and take advantage of the memory that's available across all of these nodes, and then um write that data out as fast as possible from all of these nodes. So, you have to make sure that you don't overwhelm the system that

you're writing to, but you can also tune that with different uh configuration variables. Um and in terms of um distributing the workload evenly, as long as your actor implementation is efficient and um you can actually like get the data to the GPU fast enough, uh which generally isn't an the more you scale out, the faster you'll go. Uh this is a list of a small list of

the sources and sinks. Sorry, that I know the text is kind of small. Um but some of the ones I want to point out are parquet, uh audio and video. You can actually load up audio from object storage um with a single line and make that available across the data set. Um you can read data from Arrow, from Manatee itself, uh and from SQL databases and especially

from Snowflake, um and other data stores that um you might already be using. Uh so, some tips for Ray Data. Um if the functionality already exists in Ray Data, please use it. I work with the Ray Data commit as a Ray Data committer, and we do a lot of work to make sure that the sources and sinks that connect with these data stores are as fast as

they possibly can be. And if we find an issue, or if someone else finds an issue, they usually will um either write a PR or write an issue, and we'll get that uh fixed up. And the idea is to just take all of this work developers and really just allow the developer to provide as much business value as possible without having to worry about a bunch of

the implementation details of interfacing with all these various sources and And so, um you also want to make sure that when you materialize your data, that you're not materializing potentially the whole data set because you will quickly run out of memory if your data set is large enough. So, if you are on five nodes, and those five nodes have, say, 32 GB of memory each, and you

try to materialize a terabyte of data, it's going to not work very well. Um and so, there's ways you can get around that. If you can partition the data set, you can run chunks of data at a time and write them to the target data store. uh be careful in terms of what you serialize inside of an actor. Let's see. I'll just put this back up here.

Be careful what you serialize in an object because an actor can um will need to be serialized whenever it's distributed across the cluster. And so um this can result in serialization deserialization issues. And finally, when you're repartitioning and materializing prior to large map um this can actually make your GPU jobs run significantly faster because otherwise, if you don't do this, um it's possible that some of your

GPUs will finish before other GPUs. And that will can just result in in uh reduced utilization of your GPUs. So, here's the dashboard. Oh, There we go. So, this is a Ray data dashboard, and you'll see that in the middle two columns, they're um 100% GPU utilization. And then um the video memory is also being maxed out, and that's really what you want to see throughout the

course of the job because then you're maximizing your resources. And if any of these nodes were to die, um a new node would spin up automatically in the way that we have it set up and automatically finish processing. And so, really you don't care anymore about individual nodes. I mean, if a lot of nodes are dying, you might be running into out-of-memory issues, but if for instance

uh a node dies and it's due to a a hardware failure, you really don't care because another one's just going to pop up and and take its place. Uh this is the dashboard I built at Zephyr. Um you can see that the CPU utilization is very minimal. The problem is is that a lot of these GPU backed nodes, you really can't get less CPU with them um

just from like the pricing. And so, we generally have a lot of extra CPU, and you can use that to do other operations like uploading the quadrant to to Snowflake or Lance TV at at the same time. And we've made a lot of those optimizations. And so, but you can see the CPU utilization, you'll see these small drop-offs when we're switching data sets. Um but most of

the time the GPUs are at 100%, and over time we've optimized that um make even um more use of the the GPU. At the very end, if you're looking at the CPU utilization overall, you might see a pretty large spike. That's when we actually start writing to uh Snowflake and to Quadrant. Um and you can see how the GPU utilization basically drops to zero. Now, one of

the optimizations that we did that really helped in terms of cost is we can only write to Quadrant so fast. And so, if you can only use the CPU portion, well, you might as well just stop using the GPU cluster altogether and spin up a CPU cluster. And so, what we would do is we would finish the the GPU job, and then we'd spin up a brand

new cluster, completely different type of node, um that would metrics from our GPU run, and then use that information to figure out how large of a CPU cluster we should spin up to get the data written to Quadrant without melting Quadrant in the process. You can also see that over time uh the memory usage on uh of Ray is going up, and we're very careful to make

sure that we don't exceed the amount of memory available to the machine. But each time that bar is going or that you're seeing a drop-off in GPU utilization, we're actually doing a garbage collection. And so, that's one of the reasons why we split up um data into basically buckets of data because otherwise we would eventually run out of memory. Um there are we're the Ray data process

has improved a lot in terms of garbage collection and things like that, but um this was on older version of of Ray, but um it's very important to manage your memory usage. Um and you can see finally the right IOPS go up as well with the CPU utilization, and that's because we're writing the data to to Snowflake, and we're writing that to disk and then loading that

into Snowflake. uh calculating costs and developer experience. So, there's really not too many variables involved in calculating the cost of it. for the job itself, it's mostly determined by the GPU nodes that you're running it on. Object storage is effectively free for half a billion vectors. Um it sounds crazy, but whenever you actually look at the the storage costs of these vectors, if you if they're well

compressed, um storing it in S3 is very cheap. Storing in Snowflake is another matter. and then querying off of it in Snowflake, and that's one of the reasons why we use a vector store is that the vector offering in Snowflake was not very performant. And so, in terms of our vector store that we were using at at um at Zephyr we use in Quadrant, and it's a

function of the amount of data stored on disk cuz Quadrant is disk-backed, and also the amount of data stored in RAM. Whenever you do indexes in Quadrant, you almost have to put them in And so, that can be a very big bottleneck if you need the machines to have additional RAM versus just a bunch of disk. in a multi-cloud environment, if you're using Ray, I highly recommend

looking at Anyscale. Uh it can be somewhat um, expensive to use, but in terms of like the actual contract with AnyScale, but it will save so much in developer time versus spinning something up with Qdrant, um, or spinning up your own infrastructure, that, um, it might be worth it. And then you can also deploy in any of the three cloud providers, and any of the three major

cloud providers. Let's talk a little bit at about vector infrastructure at scale. And so, Qdrant's initial user experience is pretty excellent. You can run it in memory, um, without any issue. Just do a docker pull and and start it up. Um, scaling the cluster requires re-sharding operations, which are painful. So, if you need to double the number of nodes that you have, and you need to double

the number of shards, this can get incredibly tedious as you need to scale it up one shard at a time. This isn't as much of a problem if you only have, say, six shards to start out with, but you can imagine if you have hundreds of shards, you'll have to script and automate a process to do this process one at a time. And you can't schedule a

new shard operation until the previous one command finishes. Then, compute nodes also have to be on 24/7, and so even if you only have a high write volume for a couple of hours of the day, the nodes have to stay online because to be able to do search, you have to be able to access all of the nodes in the cluster. And so, this can become very

expensive if your workload is very heavily, write oriented. If you only rarely are doing reads, for instance, if you are just just doing like document lookups, things like that, um, then you might be running a cluster with a lot of, um, provisioned capacity that you're not necessarily using. And that takes me to why I like LanceDB. And so, while I think Qdrant's great, especially if you have

a very heavy read and write use case. Uh LanceDB is storage-first, so um all data resides in object storage, so you can it's very cheap to operate. Um you can scale clusters horizontally and vertically as needed as there isn't that disk-based storage story. Um repartitioning is easy because we do append-only writes, and so um we can do compaction at any time. And then it unifies vector search

and image and text data. So instead of having to store your images elsewhere in object storage, you can actually store images on the row um inside of LanceDB. And LanceDB is also open source, and so you you can run it yourself. I actually run a lot of um hackathon projects through LanceDB in object storage, and it's effectively free because if you only have a few hundred thousand

images, the cost to store that in object storage is basically nothing. Um in terms of operating costs, by scaling to zero, if you're not doing any searches, that can save quite a lot of money if you have a very large um vector All right, and that's pretty much it. Uh I'm open for questions if anyone has any, and um thank you for attending. Um in another session,

they were talking about like in I mean I'm not like an expert in rag and all that embedding stuff, but in another session, they were talking about like if you have uh you use an embeddings model, and then you build your vector DB and all of that, um and then that embeddings model goes out of vogue or whatever, and then you have to use another one, and

then you have to like re-embed everything and rebuild everything all over again. I mean, is there something that people are trying to work towards to like prevent this from happening? Yeah, so I think a big problem that we encountered with Quadrant is that let's say you wanted to write a new embedding app, right? Let's say you start with embedding A and embedding A does well, they get

good recall, things like that, but someone comes out with a new model, right? Well, you need the storage space for that new model as a new column inside of Quadrant. Whereas with something like LanceDB, you can add that new column and you're just writing the object storage. And so you can do a lot of experimentation where it's like, okay, what if we try three or four new

embedding models every couple of months and we just scale out our compute to generate these embedding models, then it's a lot easier to try out new models because then you can only you can backfill columns and just write that data to object storage. So you're you're not really you're have you're going to have to pay obviously the cost to embed those models, but you're not going to

have to scale out your cluster as you would with something like Postgres or Quadrant. So I think that's one way to get around that. Another way to do it is to sample your data and so have like a smaller sample, so instead of doing 500 million or a billion rows, maybe take 10 million rows. And then you can even use LanceDB locally to try out, you know,

three or four different embedding models against that 10 million and then once you're good to go, once you figured out which one provides best recall or precision, whichever metric you're looking for, at that point then you can just pick that one and swap it out in production. So. But yeah, that's every week there's a new embedding model it seems like and eventually they're going to converge to

a point where it's probably not going to make much difference and then it'll just be a question of which one can you get for the cheapest possible cost for the the cheapest provider. Any other questions? I think happy to talk about anything related to Ray embeddings, distributed systems. Um, for your raw data that gets embedded, what steps do you take to determine the size of the chunks

before you do your embeddings? Do you mean like the the size of the chunks in terms of like for instance text, like breaking up into paragraphs, something like that? >> Right. So, that's very data set dependent, which I know is sort of a cop-out Um, one of the things that we would do we sort of were limited to by our data set in terms of how big

that data could get because if you think about a TikTok post or something like that, most posts are going to be under 512 tokens, which was the maximum that our embedding could support. But, what I would do is I would actually try a bunch of them. I'd try a bunch of different uh settings in terms of for instance cuz a lot of people think, "Oh, I have

to to choose one and just go with But, you can really if you have the computing power for it, you can go ahead and try out, you know, paragraph, you could try out, you know, embedding the entire document because for instance in Wikipedia, if you embedded the entire document, it might not really be that useful cuz then why wouldn't you just, you know, use the document itself?

Just load the whole document into into uh the context window if you're doing say LLM generation, right? So, um I think the best answer for that would just be to try a bunch of different options and um try to ensure that you are using as few tokens as possible in the embeddings because that way the embedding can um when you when it's actually looking at attention in

terms of the various tokens, you're actually going to have a better chance of signaling on the tokens that are actually important in that embedding, if that makes sense. Yes. Thank you. And some I should have added to that. Some embedding models can support up to 32,000 tokens. So, in some cases you might want to just embed the whole document cuz it's why Um hi. If you have

a relatively modest data set and they're already kind of post-crisis heavy, do you have any opinions on the PG vector extension and its feasibility? I mean, I understand there's kind of two issues with it. One is, you know, dimensional limits and the other is, you know, the whole filtering vector search query planning dilemma. I mean, obviously purpose-built vector databases solve this kind of problem. But if you're

doing something relatively simple, Yeah. So, um don't tell my boss, um I work at Lance DB, but I really like PG vector. And um if you already have a lot of post-crisis experience, I highly recommend and you let's say you're under let's say 5 million embeddings, I would say just use post-crisis. I think a lot of people is pretty common to find, right? Um and then if

you decide that you need more advanced features, um like reciprocal rank fusion, um different embedding um approaches, if you do need very large embeddings, like if for some reason you need like a 16K embedding size, that might be a little tough on post-crisis um for even like say 5 million embeddings. But if you're using something like the Quen 3 embedding model, which is I think either 1,024

or 1,536 in terms of dimensionality, um you can probably get away with using uh post-crisis for a very long time. Um one use case for um that I would say would be better is packaged applications. So, for instance, um OpenClaw, apparently which apparently now has more stars than the the Linux repo, which kind of blows my mind. Um it uses LanceDB, I believe, under the hood for

one of the memory options. And the nice thing about that is that it's entirely contained on disk. Whereas with Postgres, you might have to like set up a server and things like that. Um with LanceDB, you can run it entirely off of just like a directory path on your And so, I think that's probably one more advantage. But yeah, if you already have like an easy way

to set up Postgres instances, I really think uh PG vector is a great way to get started. So. Cool. Thanks, everyone. Appreciate it. Oh. Echo, echo. Cool. Look good. >> I'm just curious like who like oh There we go. Cool. Um so I'm just curious Is this show of hands first scale? Oh, cool. Sweet. This is like definitely one of my favorite conferences. Learned some cool stuff.

And um hopefully I don't disappoint today. Who here was Who was here last year? Okay, sweet. And of those people who were here last year, who checked out my NLP training machine learning models from last year? That's Oh, cool. Anyone's interested in a crash course in training machine learning models? That one was recorded. The GitHub repo's It go through it. It's pretty cool. It's like an introduction

to machine learning models NLP. All right, I think we're at about time. We have slides. Oh, look at the magic. All righty. Cool. Thanks for showing up today. Today we're going to talk about training a small language model. Like kind of the process. We're going to do uh kind of cover language models 101. all the things you need to know, and we're going to do that in

a non-PhD style so that it's accessible for And we're actually going to cover three different types of language models. So, let's go ahead and get started. So, my name is David Von Thenon. I'm an AI/ML engineer at NetApp. So, I've kind of been doing the AI/ML thing for like some time now. Um I I have an interesting background, so I kind of started out in enterprise storage.

Um then went into large-scale backup recovery solutions for virtualized environments. Did open-source Kubernetes, which is where I went into like dev advocacy and open source. And then kind of like roughly 5 years ago, I was like, "Hey, I heard this AI was going to be a thing." And then kind of made the jump into that's kind of where I'm at here today. So, hopefully we'll take a

look at small language models. And this is the agenda. So, we're going to do small language models 101. So, like what are some of the terms it that you probably hear when it comes to like training, and how large are large language model, and also by extension how a small language model works. Then we're going to talk about discussing how to build a small language model from

scratch. Um we're going to do a live demo, obviously not the training because that would take way too long. And then we're going to talk about doing it the right way in terms of fine-tuning and quantizing a model. And then as kind of a bonus, I kind of thought I'd throw this in here. There are many different types of small language models or language models in general,

and we're going to pick one that I've built, and we're going to go ahead and take a look at that. And we'll do three live demos and have some Q&A at the end. small language models 101, the basics and the process. So, by a show of hands, um who here has used like OpenAI, some large language model? Okay, a ton of people. That's not surprising these days.

Um so, a small language model is exactly like a large language model. You know, you have processing, you have some natural language context understanding, and then you're generating some natural language content, right? Um some language in, processing, understanding what the question is, what the user prompt is, and then generate an output to hopefully answer that particular question. Um and so, you've often heard probably the term, "Hey,

language models or large language models are like just next token prediction engines." And that's actually true. And that's no different for small language models. Um small language models ends. Kind of the difference between the two is that a small language model is you require fewer resources to run, and that's by intention. Um kind of the idea is instead of having massive GPU farms where you're like serving

up large language models, you want to have small language models run on like modest hardware, and maybe even potentially even CPU. And there are many CPUs out there that actually have acceleration engines to do floating-point calculations where you don't need a GPU. They're mainly used for inference. But, GP lower-end GPU or even CPU. And the end goal is to have a good quality output, faster inference because

if you're not requiring recent like massive amounts of GP expensive GPUs like an H100, faster inference, and of course because you're not using a very expensive GPU, you have lower costs on your infrastructure, Commodity CPU and or uh less expensive GPU. And by also extension because you're you're not using these super powerful right? You're saving on power, you're saving on cooling, you're saving water, all of those

things. And that is what a small language model is. So, kind of for our project today, what we're building, this is kind of like our mental map. We're going to build a nano GPT style causal language model from scratch. And in actually even just machine learning models in general, but even building large language models, you have kind of I like to say of like four steps that

are required. And these are the four steps. You have some form of data preparation. You take the data set. Do you need to massage it into some format that's familiar that can be used for uh like something like PyTorch. Then you have a tokenization process. So, we need to basically tokenize the input or this data set into something that can actually be operated on by like PyTorch

or any other machine learning framework. Then we have a training loop. We go ahead and we iterate over and over and over again over the data set. We save our model, and then we deploy for inference. So, the these four steps are what we're going to talk about next in some more typical GPT model architecture, generative pre-trained transformer model. So, this is kind of like just the

blueprint, and I've intentionally made uh presentation in the slides kind of wordy because I'm a visual person. If someone says a word to me, it usually goes in one ear and out the other. So, you can be able to like see the words as they're being presented, and also it actually helps out um chunking so that it can be consumed by AI, and it'll be easier for

it to uh to to read. Anyhow, so you have your input, so you have your data set. You have That data set is it Since we're talking about language models, that's in text, a whole bunch of and that's our input. Now, what we need to do is we need to actually convert that into IDs. And the text are converted into token IDs, and that's done by our

tokenizer. Now, each token ID becomes a vector. Right? You If we're transitioning and trying to predict the next token, right? We have to have a means of understanding how important is that particular token to the next one in that we're going to append to our answer. But, in order to do that, we need to have something called attention. And what attention does is we basically take a

look at all of the tokens that came before it, all of the context that's associated to determine what the next token the best next token should be. And that's what attention is, the ability to go back and take a look at all of the tokens that came before it to decide what the next token should be. Now, you might hear this uh term called attention or sorry

transformer. So in transformer is if you're going to look back at all the tokens that came before it in order to find out what the next one is we probably don't want to do that one at And so what a transformer allows us to do is allows us to do think about it in computer science terms like threads. You want to be able to calculate a whole

bunch of these what I think the next best tokens is in parallel. And the keyword is parallel because that's what the GPU is for, right? So that I can figure out how many different lanes of prediction for the next token where I think it's going to be the the next best one, the most probable one. But I also want to be able to pre predict many of

them in parallel just in case one is better than the other. We don't want to commit to just one, we want to run a whole bunch of them in parallel in one shot. Now that we have all of these lanes, right? We have one lane that says this token's the next one, we have one lane next one. How do we know which one is the one that

we say this one is probably the best one to use? So that's what the prediction head is. So it's another term that just basically we have the state of all of these threads or all of these parallel next token predictions and we need to rank them. And we need to rank them to which one is the best, which one's the next best, the third best, the fourth

best and so on. And if we just take the top one like top K equals one that gives us the next that is the most probable to be appended to all of the context, all the previous tokens that we've generated prior to that one that we're adding to the end. got a new token, appended it to our Now we have this thing called an auto-regressive loop, which

is just a fancy way of saying do that over and over and over again until you get your answer. And so, that is literally the process of how a large language model, or a small language model, works internally without requiring a PhD. in a small language model, why is it small? Um so, it's small because you're intentionally, right, going from a large language model to a small

one, you're intentionally lowering the parameter count. So, the parameter count. So, you know, you hear like large language models, 100 billion parameters, 200 billion, trillion parameters that are in large language models are going to be massive. But for small language models, you're talking around 4 billion to 8 billion which therefore translates into I don't need ridiculous amounts of GPU memory and resource in order to run the

thing. The context window typically can be smaller, although that doesn't have to necessarily be true. And then, you know how we talked about uh vector, like we have a token where we're at, and we have a vector. We're going to decide what token is the next one, whether it's in this lane, or this lane, or this lane, or this lane. In a small language model, the number

of lanes are reduced, but they happen to be the most probable lanes. So, the things that we're starting to lop off are things that we really don't care about, right? The things that are not as probable. And so, the number of next token transitions and the next vectors is is reduced. So, that what that ends up translating into is the reasoning capacity. So, that's a typical GPT-style

large language or small language model So, that's kind of like the process of how a large language model works. So, for the data prep right, we need data to train on to build and understand language to build a small language model. So, you need a ton and a ridiculous amount of data. And so, for the uh nano GPT style small language model that we're building I considered

two data sets. One is Tiny Stories, which is the link is there, you can go ahead and look at it. It's about a 1. gig 1.6 gig in size. And then the one that I actually ended up choosing was FineWeb-Edu, which is a 10 TB data set. So, massive, right? And I have the link to it here. And the reason why I picked that one is actually

not ideal that I picked that one, but I picked that one intentionally. So, you can click the green link there. There's an entire paper on Hugging Face that discusses the FineWeb-Edu data set they analyzed it, you know, and so some data scientists analyzed it and said, you know, these are all the things that you need to curate to make this data set like really clean and really

powerful and really usable with like the least amount of problems. And it's very fascinating, go ahead and read it. But, what a data set in practice what it does, you need to divide it into three different categories. One is training, right? Training data tends to be the most massive portion of it. And then you have a smaller validation And so, what you do is you train, use

this training data, and then you go ahead and you look at the validation data and say, "How far off am I when I'm training this thing?" And the idea is that you use the validation data set to get it to push it in the right direction so that it accurately captures what you're trying to build in this case a large or small language model. And then you

typically, although it's not really super true in large language models, there's usually a test data set that just you feed in test data and find out how far off you are. Um what is the confidence score and how far off am I from the actual thing that I'm trying to train? So, tokenization, right? Computers don't do text. So, how does this actually work? So, a tokenizer, all

it does is it takes There's actually many forms of tokenization. There's three of them. But it what it does is it replaces words to numbers. And so, the the simplest example is this uh word piece tokenizer. It just tokenizes for every word like the word like the or cat. And then it takes cat and it assigns it an ID, a unique ID, and that unique ID represents

cat. And it doesn't need to know what cat is. It's just this ID. And anytime you see the word cat, it just references that particular ID. So, it's just assigning a number to a label or text in this particular And that those numbers end up becoming like the alphabet for your language And it's very important that you don't mix your tokenizers. So, the kind of like the

computer science way of thinking about it is you have maybe a file that's encoded in UTF-8. If you save a file in UTF-8 format and then you try to read that file in Windows Unicode, that's not going to work well for you, right? You're not It's going to look like garbage and it effectively is the same thing for large and small language models. You can't mix your

language. And there's also another higher much higher level way of thinking about it, like mixing Spanish and English. Totally different context, but you can't different tokenizers. So, we talked about the data set, talked about the tokenizers. Now, the training process. So, kind of already alluded to it a little bit. So, we train a model, we take that training data, and we go ahead and we that training

data like against that validation data set. And then we basically say, "Hey, how far off are we from the answer?" And then we adjust accordingly. So, that adjustment, if you think about it, if we're training and then we need to uh adjust, we want that number or that adjustment to be as small as possible because ideally, in the best-case scenario, if that adjustment is exactly we don't

need to adjust at all, that means we found our answer. So, our adjustment is zero. when you're training you actually want that value to be close to zero or near zero. The smaller the adjustment, the your uh your machine learning or your your model is learning at at a good learning rate and it's actually doing what you're are predicting it to do, right? Give you quality answers.

And that is done through a loss function. So, the loss function is basically a "Hey, I'm going to take uh training data, uh, look at the the the validation data and then find out the difference and that lower number, we're going to use the loss function and that lower number is what's going to say, "Hey, this is what I'm starting to see patterns in the the text

and the data that that is there for the language model and I'm going to go ahead and say, "Okay, we're on the right track to building a language model that understands language." So, now we've After we've trained our model, we have our data, we've done validation on it we're going in the right direction. Our our basically our loss function says that we're getting closer and closer to

honing in on understanding the structure of the data set which is effectively just learning a And once we get to the uh, the ability to say, "Hey, this language can now understand the context of the domain of the data set, we're going to go ahead and save it and that can be done in like uh, PyTorch checkpoints. That could be done as uh, Hugging Face uh, you

know, Hugging Face binaries or checkpoints uh, Tensor Safe Tensor format. Could be if you're using like ONNX, it could be like a bin format. So, you just export it and save it so that you can run inference on it. but it's very important that as you understand that as you're training a it's picking up subtle nuances at every step of the way. So, it's understanding like the

association between the previous word that came before it the the the next word. It's looking at the context of that word at in sentences and then it's looking at that context in potentially even paragraphs, right? Depending on how you architect the Um but at the end of the day, a language model that's been trained. You have as you start to spit out the answer, you look at

the all the tokens that you spit out prior, you try to and then you append it, and it's effectively a list of tokens. And who here has heard about top K top P? Okay. Oh, cool. So, if you have a list of next possible tokens, your ability to manipulate what that next token is is using like top P and top K. And And what those are, I'll

just say really briefly, top K is only show me the next 10 or the top K is only show me the 10 most widely used tokens. And top P is show me the like if you use like a top P of 10, show me the top 10 most probable tokens. So, that gives you your ability to manipulate what the next token can be. So, like let's do

a little review here. So, we take text. We turn them into token IDs. We take the token IDs and we're the goal is to generate vectors. We have this attention mechanism that says, "Hey, I can go back and look at all the previous tokens that I've generated before. I want to have run multiple parallel lanes for what I think the next probable And all of all of

those lanes, I want to say which one is the top one, which one's the second one, which one's the third most probable, which one's the fourth. And then do it over and over and over again, right? Keep appending the next token to the next token to the next And so, if you try to build out this language large language model or small and you try to build

it like a machine model problem, it actually goes really bad. I knew going into this like why this is, but I was just kind of curious to see how it would fail. And how it will fail is you will run out of memory on your GPU. Because you're talking about a 10 terabyte data set, and you cannot possibly fit 10 terabytes in the memory for a GPU

uh fashion. So, typically a machine learning model, so not a large language model, just a strict machine learning a pass over the data multiple times. So, if I'm trying to take 10 terabytes and do multiple passes over and over again, that's not going to work out well because it's 10 terabytes. how do you actually go about building one then? it's definitely a lot easier said than And

if you're trying to build a GPT style uh large uh language model, first, a single H100 is not enough. You will be there forever. Or even an H200. started with an H100 just to see like what would happen, and it almost immediately crashed out. Not enough memory as you saw from the error log, cuz the data set's too large. So, I had to switch to a node

that had eight H100s. And as you can imagine, this gets pretty expensive pretty quickly because I'm using eight H100s. And if you go on to like Digital Ocean or Lambda Labs or whatever you're talking about anywhere from 24 to $28 per hour for this node with eight H100s on it. And because the data set is so large, it will take thousands of hours to train this small

language model. for the first demo, I obviously did not let it run to completion and we'll see first it's expensive, but we'll see kind of like the what the effects of that kind of like the fallout of that is. So, we already know the data set is way too large. We can't do multiple passes over the data set and load it all in the So, how do

we take this fine web EDU data set that's 10 terabytes and deal And so, what you have to do is you have change to a a streaming token data set. So, this is actually the interface for Hugging Face. I actually intentionally stream use this interface where you can actually stream it directly from Hugging Face to feed it in in a stream format to uh the eight H100s

to break them up to parallelize the process. it's uh pretty interesting going about doing that. Um if you're looking to further speed that up, turns out like the company I work for NetApp, they deal in storage. Um so, we actually offer a couple solutions to help uh enhance and speed up that process. Um but, let's suffice it to say that there are since you're streaming this data

set, you have to be able to dynamically break apart some chunk or some portion of the data for the your training data and then you have to dynamically, as you're streaming in data, have some of that data reserved for your validation data set. Right? Because we've already kind of just said, you need training data, you need validation data, and you need test data. So, you need to

do as you're streaming, you need to do this dynamically. And so, now the problem becomes a little bit more difficult. uh who's heard of like epochs? Like when you're doing machine learning models and you're training and you have one epoch, two epoch, right? So, what that means is up So, that epoch, what it means is you are doing one complete pass over the data. And so, when

you go to the second epoch, you're starting the second complete pass Well, we just said that we can't do that when we're training a small language we can't use that method. So, what you have to do then is you have to use a step method, which basically because we're streaming that information, we're basically streaming it and doing it in a step-wise method. We're going chunk by chunk

by chunk by chunk by chunk by chunk of the data set. And so, we're effectively doing batch processing because we can't load the entire data set in memory all at once. I abandoned obviously the naive way of doing it, which is trying to load it onto an H100 using loading the entire data set in the That didn't work out well. So, if we take that data set

and then we have this tokenizer called tiktoken, it actually happens to be the OpenAI GPT-2 pair byte encoding uh tokenizer. They open-sourced it. we create a based on our data set, we create a vocabulary of 50,000 words, right? So, each word becomes an int or a token ID. It's like flipping back and forth. Um So, that becomes our vocabulary, 50,000 words. And we can actually also use

the GPT config, which is also OpenAI's configuration for like a block size of 2K. So, that's our look-back window. So, our look-back window is our attention. Like, how how far back can we look for tokens? uh this configuration also defines the number of layers. So, this is our transformer block. Right? How many things can we run in parallel? The head, which is uh how many uh So,

32 different evaluations of the lanes. How can we evaluate each one to say which is the best one to for the next token? Then we have the vector dimensionality. Each vector transitions to another And we uh limit and cap it at 2K. And if you do all of those things, use that tokenizer, use that configuration, and then use the fine-web-edu uh data set, you've roughly got a

1.8 Which, in today's world, is sad. And it's so sad, I'm going to demonstrate it here right now. you don't need to write any of this down. The code's all available on my GitHub. I'll show you the link to that repo at the end. You can do this at home. Um but to start off the training process, you can just go into the small language model folder,

go python one {underscore} train eight times X each 100. That will stream the data set, do the training process, but it does require eight H100s. And I stopped it at 10% of what is um to actually train that thing to completion. And the reason why is it got ridiculously expensive. And I was putting the bill for it for this presentation, and yeah, it was not fun. So,

I intentionally stopped it early. And by intentionally stopping it early, this is what the output kind of looks So, just like any kind of language model, give it a prompt. And the prompt I gave it is us like kind of like a story. And the the prompt is it's loading. I'm doing this on my local laptop just to kind of prove that and this is an M1.

And I know like the M3 has an is got additional floating point calculation, so it's much much faster, but I'm intentionally doing it on a slower laptop so you that you can see that you can actually still load it in memory and So, we're doing a story. Once upon a time in Long Beach. That's where I'm actually from. So, I'm from Long Beach just down South LA

County. And we want the small language model to basically complete the story and give us a you know, do the completion of our bedtime story. And this might take a little bit of time Once upon a time in Long Beach in the time of the year 1815 blah blah blah blah blah, complete utter garbage. And it's because I stopped it early. And yeah, it's absolutely horrible. So,

and this is intentional. I knew this was going to happen. So, you know, obviously, when you use a large language model, even a small language model like Llama 3 8 billion parameter, you actually get a coherent answer, and this is definitely not it. So, why did I do this? a GPT-2, right? We're now we're at OpenAI. Actually, 5.4 came out today, right? even training a GPT-2 like

version uh small language model is ridiculously expensive. It requires if we remember that data that hugging face data set that paper, it actually requires 6,000 H100 GPU hours in order to train this model to completion. And if you do the math on that, the estimated cost to like run this model to completion, which I'm not going to burn 20,000 K to just show you this model and

that how it is. Um it it will be 20,000 K per hyper which then gets you to the roughly 8 billion parameter model that you would typically see in a small language model. So, what is the takeaway of this? No person in their right mind does this. Thank you for attending. No, I'm just kidding. That That would be hysterical. You have learned nothing. Okay. So, it it

costs too much money and the resource eight, you know, 6,000 H100 hours is ridiculous, too. So, kind of the takeaway is is there are very few people or companies or individuals that have access to the resources that do this. It requires a lot of resource, a lot of cooling, a lot of power, a lot of uh you know, water. that's because it just it's a hard problem

to solve. if we're not going to play the video, but if you want the actual PhD version of explaining all of this and actually what the code is based on. So, who knows this particular fellow? Andrej Karpathy. Awesome. Former OpenAI, former uh Tesla, and then I think he went back to open AI for a bit, and now he's starting his own thing. This is a 2-hour tutorial

on how to do it in the PhD way, but I gave you basically the non-PhD way of doing it, but it's a really cool video if you're interested in it. And you don't need to take a picture of the slides. The slides are also in the GitHub repo, too. So, you have all of it. so, I told you that's not the way to do it. How do

we actually do it? And this is the way the recommended way. This is what you actually want to do. Is you want to fine-tune an existing model and impart the knowledge that you want to impart on it, and then optionally quantize it, which basically We'll talk about it in a little sec a little bit here. So, this is a much easier way, not only easier, but far

more cost-effective, and we'll kind of look So, what is fine-tuning? So, fine-tuning is taking an existing model, and in our demo, in our next demo, we're going to use Qwen 2.5, the 7-billion-parameter model, but we're going to take a model that's kind of a generalist model. Now, if you just need generalist kind of language understanding, you can stop right there. You have all of these cool models

that are available. You can just use it to understand the language of your choice. But, if you want it to have some sort of specialized knowledge, then you need to on 8 billion parameters, it's probably hasn't been trained on that specialist knowledge. And that's the reason why a lot of people go to these massive you know, trillion-parameter models is because they've been trained on the world, but

it's also very expensive to run, to like run on your GPU, right? And GPU farm, right? So, the alternative is to take a small language model and impart the knowledge, the pieces that you actually care about, and impart that behavior onto a small language model. And so that's exactly what fine-tuning We're modifying We're giving it a a new We're modifying the weights on an and we're imparting

that knowledge onto that model. But all of the mechanisms that I talked about, predict, compare, and adjust when we're doing the fine-tuning process, they all apply to fine-tuning as well. Slightly different process, but it's still the same thing. Predict, use the validation data set to push it in the direction that I want it to get, and then adjust as necessary to get it build a model you

know, for your specialized particular knowledge. So the non data science way of saying, think of a GitHub repo. You have a GitHub repo, a project, a probably cool project that you say, "This is really cool. I would like it to do something vastly different. I'm going to fork that repo, completely change the behavior of it, but like most of the skeletons actually pretty good, but I want

it to do something else." And you want it to do that one else, that one special thing. That's exactly what fine-tuning does to large language Fork a repo, change the behavior massively to do one thing really, really well in particular. So who's out here has heard about quantization? Okay, cool. So when you train a large language model or small language model, typically it's done using floating-point 32.

the idea of quantization is, instead of using floating-point 32, we're going to use different uh numeric representation, right? Because everything becomes an integer or a number, sorry. And we want to change the math of the problem to something that's much more manageable. So, you can quantize a model that's using floating point 32 and you can quantize it down to like int 8, floating point 16, and then

BF16 is this one special thing that Nvidia does where um I always mess this up, but like like the precision is less and the numerical value is larger. And here it's a different representation of a a floating point number. So, what are the Well, like why do you do this? So, smaller memory footprint. Instead of using 8 bytes for FP32, if I'm using int 8, I'm using

a fourth the number of bytes. And just by thinking about the math on that, you're using a fourth less memory, like 1/4 of the memory that you were using into int 8. And so, you're basically effectively making the math much easier. Now, some CPUs or GPUs Well, some CPUs, they're not really good at floating So, you convert the math to int 8, which every CPU on the

planet knows how to do integer arithmetic, right? Or manipulation. And so, that's the reason why you change something to like floating point to like int 8. And that Now, you can run that particular model on commodity CPU You don't require as much memory to load. And that's the whole purpose of quantization in a nutshell. the I'm running out of time. So, what doesn't change? So, there's a

bunch of things that don't change. You're starting with a base model, so all of the characteristics that you normally have, you still have all those same characteristics. So, like the context window, the attention heads, the layers, and all that stuff. Fancy way of saying you're just changing the behavior of the knowledge that you're imparting onto Um I'm going to skip this in the interest of time, but

there are three methods for quantizing. The last one's really cool. It's I don't I'm just kind of like lopping off and I'm using the most uh the most pertinent. So, you're kind of like lopping off and removing the stuff that's the least used or the least Yeah, just kind of like leave it at So, because of the quantization process before quantization require much larger memory footprint, you

need a GPU. The inference speed is slower. And then the correctness is actually really high, right? Because you're using the full-blown power of a GPU. After quantization, the memory footprint's lower, processing power you might be able to get away with a CPU or a low-end GPU. The inference speed is much faster. And the correctness takes a small hit, but don't worry about that because it's not actually,

if you think about it, because you're imparting specific knowledge onto that uh existing language model, that's what you care about and that's kind of like you're pushing the weights in that direction and you're removing the stuff that actually you don't care about anyways. So, the process is very similar. Start with a Just a review. Start with an existing model. And in our demo, we're going to use

Qwen 2.5 7 billion parameter. You have a data set. You fine-tune it. And then you validate it. And then you deploy it. So, the model we're going to use is a SQL language small language model. So, that will understand how to generate SQL statements and understand SQL statements. So, that's what we're building. Qwen 2.5. The data set is uh a database schema plus a question. And then

the output is SQL. And so, that's what the data set looks like. We're going to fine-tune it. Um, there's a fine-tune strategy called LoRA. Don't worry about it. And then we're going to quantize it down to eight int eight. So, let's go ahead and take a look at If you want to do the training process, the fine-tune process, that's probably You can see Python fine-tune. Run that,

it will automatically load the data set and start the fine-tuning process on You want to quantize it, run quantize, it will take the final model that gets produced that now understands SQL, it'll quantize it down so that you can run it on a laptop. we are going to run inference on it Oh, shoot. Oh, Python Python probably doesn't work, So, this is a a We fine-tuned the

model, we've imparted the knowledge of SQL. And this basically don't worry about the warnings and all that fun stuff. Um so, as a part of our question, we're going to ask our new SQL language model, we're going to give it a table. It's a simple table, right? Create table persons or employees, and it's going to have like the person's name, it's going to have like their age

or whatever. And we're going to ask it some simple questions. So, the first one we're going to ask is list all the usernames in this particular employee table or this user And it's going to because now our small language model knows and understands we can actually get This is This is actually dynamically generated, right? This is the response from our particular language model. Select name from users.

List all the names for users. Who is the highest paid employee? And so, how would you write that in SQL? Anyone take want to take a stab at it? Select asterisk from users order by salary descending limit one, right? That's like one way of doing it, right? Hopefully it will come up with that. So, we literally trained a small language model that can answer SQL problems for

us. Pretty cool, right? That one Oh, right here. the cost to train that model, 100 bucks. I think it was far less than that. I think it was like ended up being uh just under 100, maybe it's just over 100. But versus $20,000, right? So, if you need special uh like special knowledge like SQL imparted onto a small language model, this is how you would go about

doing it. This is a much better way, much cheaper, much more effective, and if you want to quantize it to run on commodity hardware, this is the way you go about And what's really cool about this approach is if you're starting with Qwen and you need to iterate over your data set or your training, or maybe there's a new version of SQL that has new language statement

or whatever, you can just rerun it, and it's totally iterative, right? Start with the model, add the new data for the data set, do the retrain, and you know, start that way. Real quick. Let me see if I got one other section. How big was the data set size? So, the question was how big was the data set? I kind of really did gloss over it. So,

it's a synthetic text-to-SQL data set. It's uh publicly available. I can't remember the size. I think it was actually on the order of I think it was on the order of tens of megabyte. I don't know. Well, let's click it. No, definitely not terabytes. It was It looks like 32 megabytes. Now you know that all the links work, So, much cheaper way of going about cool, we

built a SQL But it actually is kind of this is the bonus section. There are actually many types of language models. It's not just the Q&A one that we're used to where you you ask a question and you get a a complete answer like, you know, tell me the difference between a cat and a dog. GPT runs around and says, "This is what a cat does and

this is what a dog does, right?" There are many other like like language models. And here are a small list of them. So, there are like language models that do summarization. There are language models that do code generation, which we just did, right? SQL, that's code. Um our next demo is going to be this MLM. So, it's a masked language model, which we'll talk about in a

second. There are classifiers. Um but kind of the idea is if you have a specialized language domain that you want to impart onto an existing language model, you can teach it to do things. And I'll talk about one at the very end when we get to resources. But a masked language model, so who here Man, I'm going to date myself here, right? Who here remembers Mad Libs?

Yes. Right? You had sentences, you had blank spots, and you had to fill in the word in that blank spot. The idea was is to make it super funny by picking wacky words, but you basically have sentences with a gap in it. Now, what a masked language model does is it actually, based on the sentence that you're providing it, it will actually guess the most probable word

that should fit in that particular gap or that hole. And so, that's kind of exactly what a masked language model does. Now, there are a whole bunch of applications that you can do for it. Like, you can do extraction. I know it sounds weird. You can do classification uh using a masked language model. You know, basically give it a string and say say, you know, classify this.

It should be blank. And it will analyze the sentence and it'll give you the most probable classification for it. And there's also this {question mark} {question mark} {question mark} and I'll tell you why that's there. But, let's just get to the demo. So, the last language Whoa, shoot. The last language model demo. We're going to do the masked language model. So, just like before, if you want

to train it, this does the data preparation step. This does the training. let's run inference on it. And actually, this one's super quick because we're only trying to guess one word in a sentence. So, the capital of France is blank. And based on that sentence, we ran it through our masked language and it said the most probable word is Paris. The capital of France is Paris. And

then, distilled models are the blank of large models. And then, the most probable word is smallest, which is correct. So, that's So, just I'm kind of floating the idea You can do all sorts of wacky things with language and there are many other forms of language models out there. This is one example. I actually have another session tomorrow. It's actually pretty cool. It's this one right here.

And I'm going to tell you why I built that model and I'm going to use it in this session right here. Um but if you're interested about doing uh side channel acoustic attacks for keyboards, so think keystroke loggers based purely on sound alone of when you hit a key on a keyboard, can you guess the letter on that's being pressed just by the sound on it alone?

And spoiler, you absolutely can. But if you're interested, check out the session tomorrow 101 uh room 101, I think it's at 2:30. So, resources. All of the code to build all three of these language models, I would recommend not doing the first one. If you want to burn a whole bunch of money, go for it. Otherwise, the two useful ones are the last two. But they're in

my GitHub repo here. All the code to do the the data prep, the training, the fine-tuning, the quantization, and all that fun stuff is available in my GitHub right there. So, why do we care about this? Why am I talking about this? Like, you know, you've probably heard this tokenization explosion and the cost that's associated with it. The reason why we care about small language models and

the reason why we care about quantizing them to run on cheaper hardware is that, you know, if you think about OpenAI, what they're losing what, tens of billions of dollars a quarter or something now? I'm sure Anthropic is probably on there, too. I mean, if they're losing money, you're getting your tokens on a discount right now. And over time, you're probably going to competitors are going to

drop off. And then at some point they're going to need to show a profit. And guess what's going to happen to the cost. Right? Do the math. quantization and running small language models like on your laptop or on cheaper hardware, that's the reason why you do it, right? Cheaper power, uh you don't need as much power, cooling, inference, and all that fun stuff. I have a blog

post about it, and I'm actually talking about using small uh in uh in an agentic multi-agent agentic AI setting. It's I've actually on uh in April at Open Data Science Conference East. Um NetApp, we do storage. And if you're looking to stream data from data sets to train your large language or small language models, we have a couple of solutions. You can use Kafka and all that

fun stuff. I need to do the company plug so they will pay for my hotel room. but you know, we do that, and I would highly recommend you take a look at it because it does help the the training speed and cost like like way better. But yeah, cool. Thank you. I think we have time for questions. Yes. I'll try to repeat the question. losing any money

other than the token you pay for the large language model. So, how do you uh compare the fine-tuning or the time versus writing that Python code agentic uh I in in couple of minutes to achieve the same >> basically, I'll summarize the question like why fine-tuning and why not just use is model that's pre- Yeah, well agentic who who's that that language model exists somewhere. Where where

does it exist? So if it's running on open AI, one, you're anything that you provide in the prompt, anything that you're providing as a terms of like embedding data, you're providing that to open AI. If you're okay with them having your data, Now, if you can run in your own like GPU instance, like an A100, you don't need to go for a huge H100, but if you

can run that using like a neo cloud or Google or whatever, but like run the model yourself in your own one, you have data privacy, which is huge, right? And then two, like I guarantee you at some point in the next year or two, the cost of open AI is going to just skyrocket, right? And if you can build out your agentic solution using models that you

can run and predictably pre predictably understand the cost of running that model, I think you'll be in a far better position, right? Because right now, there are people that are using like Claude like Claude code, right? And they're I know people that are spending like three, four hundred dollars a week on Claude code. And that's great. You like it's trust me, I use it too, by the

way. So, I use codex, I use Claude code, but it it's getting up there, right? Now, imagine when they actually start to crank up the cost on all this stuff, it's going to get far more expensive, right? Now, if in the open data science conference talk in April, I'm actually it's that agentic AI talk. I forgot the name of the title even though I wrote it. Um

I'm going to actually take that uh sequel uh the sequel model that we just built today, and I'm actually I've already built a Python small language model so that it understands Python. And I guess the spoiler on the demo is I'm going to take those two models and use an agentic build out of a small little agentic system so that it says, "Hey, if I want to

call a Python SQL or if I want to call a SQL statement against the SQL like MySQL or whatever in Python, that is the demo that I'm going to show in April." Um but I can predictably understand the cost of both of those because it's a small language model that I actually going to run on my laptop, right? Versus you know, running into I think there is

many companies out there that are starting to run into like 30, 40, 50, 60 thousand dollars in token per person, which that's starting to get up there, right? It's so the kind of the ideas is run it on your laptop, run it in your own infrastructure. You can what's associated with it, right? Question? Oh, great. Yeah. I don't know where the microphone I'll I'll repeat it. Yeah,

go for it. So, if I understand correctly, fine-tuning an existing model, it's kind of the same operation as training it, right? It's just that you're doing it on top of what is already trained. Yeah. Do you need to use the same loss functions or can you pick a different one? Um it it kind of depends >> way, how similar does it need to be to the original

training? Um it it doesn't have to be. Like however you need it to be to have it impart the new knowledge on whatever it is, whatever domain it is, it it you don't need to know how the pre the base model was trained in order to do fine-tuning on your your new model. And I guess I should also probably say, which might be the next question, is

like, why do I fine-tune a model versus use rag with vector So, I probably I didn't cover this and probably should have. why you fine-tune versus using rag? if your small language model, that's just that base Qwen model, if all you needed to do is understand you can and you can stop there. You can feed it data. You're not using any kind of specialized thing, like, you

know, specialized domain like SQL statements, you can just have and use rag, get your vector chunks, take the chunks, feed them into the the the prompt, and say, "Based on the vector chunks, tell me about the answer that you're, you know, sift through the data and tell me about the answer." So, that's what a rag pipeline is, right? I'm taking data and I'm just having the the

language model sift through that data in order to find the answer within the data that I'm providing it. Now, a fine-tuned model, you're trying to change the behavior of the model, and that's the difference. That small language If I started giving SQL statements to Qwen 2. Uh 2.5, it would probably barf on me and say, "I don't even know what you're talking about." Or it may be,

in the worst case, start to hallucinate something and say, I don't know, dog, cat, blah blah blah blah blah, right? It doesn't know that that stuff about SQL. So, what you're trying to do is you're trying to change the behavior of the language model to say, "I want you to know about SQL now." And that's the difference between the Thank you. Yep. Any other questions? I think

we might be running out of time. 3 minutes, right? Yeah, so, based on what you already asked and uh partially already answered, you can run 7 billion parameter model on local laptops. And if they have jacked up the price in next year, I'd rather buy wait for the uh M3 or M2 to price to go down as the Apple come up come up with more and use

the whole model uh 7 billion parameter to run on a local laptop M3. By that time, price will be cheaper. So, why not use that approach? But as you mentioned, even that would be training it for a specific skill set what you have mentioned. Is that correct? Yeah. So, okay, let me repeat the question. I kind of like to digest it. So, like why use Quen like

7 billion parameter model versus like a whole like larger model or different model, right? No, no. So, you you train it on a 7 billion parameter >> Yes. and run it faster uh with the skill set. So, if you don't do the skill set, have wait for a newer version of that, maybe 9 billion parameter next year. So, like What's the Yeah, yeah. So, I used 2.2.5.

3.5 just came out a couple weeks ago. It still doesn't know SQL. Okay. Okay. So, it's still training is still uh worth it. Okay. Yes. So, even you had have a new model come out, it's a very general purpose model, and it probably doesn't know anything about SQL. So, even when that new particular model comes out, you still need to teach it SQL. Okay. So, in the

follow-up question, so you can teach even though it's dumb in answering SQL, you can uh the base model to have skill special skill set. That's what the whole point is. Okay. Thank you. Thank you very And then like the cool thing is like because the training process is the same, when the 3.5 comes out, you can use 3.5 as your base model, rerun the script to train,

now you have a 3.5 model that maybe understands language a little better, but now also understands SQL a little better, right? Anyhow, I think that's my time, but if you have any other questions, I'll be outside. Um thanks for showing up. >> All right, I'm going to get started in a few seconds. last session of the day. Yay. Yeah, enthusiasm. uh I kind of wrote it for

me because I'm a beginner in AI. I've only been using it 2-3 one thing that struck me in talking to my clients about implementations of AI for their projects their perception of what Chat GPT or what AI or what it can do, what it actually does, what it can produce is very different from my understanding. So, I thought maybe to have a talk that's more like big

picture AI. Still digging into some of the maybe not math, but at least mechanics of what makes it tick, but also gain an understanding through mapping it to human behavior because a lot of the concepts of AI were spoken about before any LLMs were built. And a lot of what we now know as a marketing term AI, artificial intelligence, really was built for us as humans to

understand intelligence equating to ours. Right? We're we're basically mechanizing, automating, building a machine that mimics what a human would So, I thought that maybe is a good lens in to looking at what really AI is all Maybe without the word all because it's very encompassing. I'll focus on some topics and hopefully have a conversation that resonates with folks who like me come to the area and say

"Yeah, I can see results of sorts, but I don't know what's going on. I want to know a little more what's going on." Anyway, that's how this talk was going. get into it. So, the first thing I want to introduce is the intern. When I explain AI to some of my clients, I say treat it like an intern. Tireless, enthusiastic, for sure. Went to school with good

grades. Read all the books. Knows all this stuff. It doesn't actually regurgitate at you. It's intelligent. It's not like, "Oh, I learned this from the book. I'll just give you the thing." It's like it collected all of these but also in the same way that it doesn't regurgitate, but it does speak English somehow, it doesn't really understand what the job is about. It doesn't have an inherent

understanding. It just has a vast knowledge of what was ever taught to the model, to the LLM. And it uses that in an intelligent way to satisfy prompts, your tasks that you unload on So, that's kind of an attitude slide, right? It's just kind of the mental concept of how things work is map it yes to a human that is just very, very enthusiastic, is armed with

a bunch of words and a bunch of things that it was taught and is eager to do the task you're going to give it. So, let's take an assignment. Assignment number one, write me a story. And let's start with a story that starts with the word once. What would enthusiastic Well, it should certainly start with a word once because that's what I gave it. Or rather, it

should continue from the word one. If anybody ever heard of the notion of completion, that's all LLMs do, really. They are predicting the next thing after the last thing. The first thing they start with is once, so that becomes the last thing. Now, what can I complete? How can I complete the sentence? And if I completed a sentence, how can I start the next sentence and complete

it until the story is told? So, that is kind of the the framework that it operates in. So, you write down the word once, then you write words afterwards, and you repeat it until that. Yay, we have an algorithm. And that seems intelligent to us, right? Any writer, Mark Twain, started with the first word and put another word down and another and another and another. All right,

but here comes the problem. What comes after the word once? Is it once upon a time? Is it once in a blue moon? Is it once there was a handsome guy on a stage at Scale giving a talk? We don't know. Fiction, all fiction. So, what would the intern do? it knows all the words. It's seen them all. It doesn't exactly remember the sequence, but something sticks

with it. It's seen some words with probabilities. It knows that in all books it's read, upon appears roughly .3 .3 out of one, so all probabilities I will mention today are between zero and one exclusive of one. Um and once in, you know, maybe .4 and once there, you know, it's seen this kind of sequence before. It doesn't remember the sentence and will regurgitate the rest of

the sentence, but it does know that after the word once, these are the likely words and these are their likelihood. that makes the task really easy, It seems, what should the next word be? Well, in has a probability of 0.4 seems the most likely. Upon is at the top, so that might be an order thing. And there and upon seem to have the same probability. But what

does it mean if it picked once If I told it tomorrow to write the same story, it should just say once in? And all stories ever told should be once Blah blah blah blah blah. That'd be kind of boring. No? What would Oscar Wilde do? Well, I don't know. He's dead. I don't know. But, you know, it it can apply different logic on the left column, by

the way, if you're reading, of how to select the next word. Well, number one is just pick number one. Girl, the winner. Girl. Yes, great. Sounds robotic. Sounds rigid. Boring. Good for some stuff. If I were to code, and I have learned all of the thing, and I know the most probable thing after this is this, it's because of something. It's because all good code that I

was trained on goes like that. So, if monkey see, monkey do, I will get the result. Deterministic-ish. We can pick from the top K. We had three. Let's say we pick from the top two, one at random. The probabilities are probably close among the first two. Or first and first 10, first something, right? There could be thousands of words with probabilities. I just showed you three because

we don't have all day, and it's the end So, there I think, you know, we can pick from the top contenders. Let's pick a number of contenders, 10, and at random pick from them. As we've seen on the previous slide, all the top three had fairly reasonably close probabilities. So, the chances of me picking a word that is not good are kind of low. And that gives

us kind of a consistent thing. Once upon a time, once in a lifetime, once there was a very handsome guy on stage at Scale X giving a story. So, that that could work. And it's good for general chat. That sounds kind of reasonable. It doesn't sound off the rails. It doesn't sound hallucinatory. It sounds reasonable. There's another method of picking and top P. And top P So,

top K is the second row there. Top P is saying, "Well, I could have a long list of words that might be reasonable." if I just limit it to the top three, I may be cheating myself out of the next word which has also a very similar So, what top P says oh well, I have a list of words. What I'm going to do is I'm going

to start collecting words into a bag until the probability reaches a number. Let's say P90 or so, until it reaches 90% All probabilities together add up to one. So, if I reach the 90th percentile, then I know these are good words. And then the rest not. But, I don't know how many it will take. I just kind of iteratively collect them until I reach And uh there's

a little feedback. I'm not sure. And and then, you know, that that's top P. It it basically a a softer, gentler uh way of doing things than top K. And that's a little more dynamic and a little more human because it gives us more nuance and more to pick from when probabilities are kind of not not uh dominant. But, if I have one or two dominant probabilities

and the rest are kind of low, then I'll reach my P90 with a short list and then it it will be still feeling concise. So, top P is I think kind of the warmest of them all among these. So, these are kind of policies and our intern uh will pick one of those methods. If you're creating your own implementation using an LLM hosted by whoever or by

yourself, you will be asked to tell the LLM, "Hey, with my prompt, what method do you want me to use to predict the next word?" So, that's good to know. And then there's another parameter that if you wanted to implement it, it will ask you and that's temperature. It's a little early in the talk, a little messy. temperatures are a divider, a divisor, all of the, let's

call them words for now, all of the word probabilities get divided by. So, what it lets you do if you have your natural probabilities, meaning that uh, you have just kind of counts of words and if you, you know, divide the sum of counts of all words by a thing and do the fraction, then you have very big spreads among them. Well, you can lower the temperature,

de-amplify the spiky ones, the ones that are really high, and thereby kind of bring them to the same level. If you do top P stuff will collect more words. If there's a big spike, top P will collect less. So, temperature and your selection criteria kind of play hand in hand to give you this very creative seemingly thing where you have a lot of candidates or less because

the spikes will win. So, when you hear about temperature, think about like bringing the probabilities to kind of the same level. With a low probability, a low temperature being very truthful, it kind of less creative. It it keeps the spikes up and the high temperature means shave them down, not shave them down, scale them down in order to get a more nuanced or creative kind of way.

That number is very relative. If you go and you pick a model on AWS and it's, you know, one of them and you pick another model that in a different different LLM provider, that temperature will give you different results. So, these are numbers you always twiddle with when you're implementing stuff it it's it's it's not universal. Cool. So, we picked the first word because we were told

and we picked the second word because we came up with a good reason to pick the next one because of its probability and now we have to predict the third one. And after there could be a time, right? But there could be once blue moon. lifetime. So, those probabilities of the fourth let's say we all agree that the third word is a. Once upon a time. Once

there Once there a, that's not a good thing. Not high low probability. But, you know, once upon and once in followed by a is very reasonable. But, what is the next Well, that doesn't depend on the word a. It depends on the word that came before That's what we call context. We're not simply the next probable word. We are conscious of we have this context. We have

this memory of history of we got here somehow. If we just forget it and we just say, "Well, I don't know. I'm just zero memory. I'm Dory. I'm just swimming in the sea. I saw A. Here's the next probable word." I'll look kind of crazy. And if I remember these things, it helps create the impression for us humans that it understands something. history, that context is extremely

in the context of LLM, you'll have hear of context window. You will hear of uh token limits. Uh they play hand in hand. Uh those are the limits of the LLM to be able to go back and kind of consider the past as it's predicting the very much the very next word. So, let's talk more about context. Generally speaking, inside the LLMs and the SLMs, which are

just you will find an architecture that is somewhat like you see here on screen. Where the user prompt comes in, and there's something called a transformer layer. And then the transformer layer takes the prompt and transforms it, changes it in some way. Income something, outcome something else based on the what came And then it tries to predict the next Update the context, that memory of what was,

and then feed it back, carry back that context into the transformer layer, or through the transformer layer, to be processed again and again and again until the story is told. So, that's kind of the mechanics of most LLMs. They may use different techniques. They may have some math shortcuts. They might skip or add or embellish, but roughly speaking, your prompt goes through some layers, and each manufacturer,

each person, which each company, well, I guess the person, each producer of an LLM picks their own actual implementation and internal So, let's talk a little about transformer layers. So, within a transformer layer, we do essentially two things. One is we take those words that existed outside in the mind of the typer, uh and you convert them into numbers. That is an internal dictionary that says every

word ever seen in our training is going to be assigned a unique identifier. So, the first thing we want to do is convert it into numbers. Why? Because because words mean nothing to a computer, and numbers do. Now, that number becomes somewhat significant because it's going to be predicted into, so, you know, if if if I have two two um different LLMs, the number getting assigned to

the word once in one is not going to be the same as the other. It's just the mechanics of whoever created it, but they will make it in a stable way such that if you use that model again and again and again, every time it sees the word once, it'll get the number 0.917, and that's that. So, token encoding is simply the action of taking the words

that came in, they get tokenized. Tokenized just means separated from each other, split on the space, throw away the dots or don't. Different people do different things, but basically encode those inputs in their entirety into some um tokens. And then there's an attention mechanism. Attention is a focus, uh and we'll talk much more about it, so I'm not going to get into jump too much into it

here. Overall, what I wanted to what I wanted to take from this slide is that as we go through and try to predict the next word, and the intern is just trying to figure out on their little tablet, what should I say next, the more there is to process, the longer the story becomes, the more the context is big, and the computations to produce the next one

and the next one become large. Not only that, internally in the transformer, there's a lot of blocks operating in parallel, so when you hear about billions of parameters, it is literally like big, big blocks of matrices that contain a lot of data, and that computation even on just one token each time, each iteration through, is a big computation. So, I said the word model multiple times. What

is it? It it has a parameter parameter file that contains billions of numerical weights. it's not a token and a weight. It's those weights are baked into those blocks that I'll talk about in a bit. It has a a kind of a transformer architecture. I I showed you the the kind of simplistic one in the slide before that organizes how the prompt flows through transformer layer, the

the whole LLM construction in order to produce a result at the end. And it has a vocabulary. When you first come in, all the tokens, all the words become tokenized into numbers. And then when you exit at the end, because the predictions are just number prediction, at the end when it feels like it finished the thing, then it retranslates all of the output numbers back into words

we understand. Internally, it just crunches numbers. It has no idea what the word once is. Or ounce, or chair, or anything. Just has numbers. So, that's where the vocabulary comes So, the transformer uh has kind of two parts. One is attention. Attention is kind of like a flashlight. It helps the model uh find the the the relevant previous tokens. So, as token as context grows large, I'm

accumulating token. I'm dragging a bigger and bigger bag of But, what is really important? In once upon a time was the word a important? Or was the once and upon more salient? So, that's what attention attempts to do. Attention is a mathematical transform that tries to find which ones of the things in the context are important. And then the feed forward, the the supply of the semantics,

meaning in what context was once uh upon a time uh spoken, that is the the the uh more semantic part of it. That is the only part in the LLM that has some kind of notion of understanding. That it's associated to a topic. Not that it knows or understands a topic, just that it's seen these things before kind of in the same area. Why do we need

attention? If I talk for an hour and a half, you remember every word I said? In order to produce a, you know, if I quiz you, pop quiz, what did I say on slide one? It's like, "Uh I don't know." "Oh, you had the intern thing. Oh, I kind of remember that. That's kind of important here." So, that's what attention is really about. That if I just

pile on you big big long conversations, you kind of lose focus. But if you can kind of say, "Ooh, these are the important key parts," then your answer, your interaction becomes more for us humans. Where are we? So, the attention part is very significant. And how to find it mathematically is a subject for people who are way smarter than me. What I gathered so far is that

internally it does essentially matrix multiplications. metric matrix contains kind of the the latest uh uh the latest prediction uh that that you know they they got as well I talked about enriching the context. And then it takes it and it tries to find mathematically which of the weights and probabilities so far are salient to be carried forward. And it does it in a mashy way that I

cannot explain. Not because I have no time, just because it's above my head. So, we'll keep it there as a mystery and if anybody knows better I'd like to hear that talk. So, several times or already said there's context, there's token limits. What happens when oh my god, I just listened to a 3-hour conversation. I read a book with a thousand pages. even if I remember all

the words, it's soup. And the problem is that if you give me something important and then bombard me with a lot of narrative that is not at some point my brain explodes and I forget even the important thing. And that can produce things that look like hallucinations. That could produce security problems who was it the recently a very famous case what happened is that this person uh

told open claw to uh go through the emails and do some stuff and it told it don't delete any emails, don't do anything here, but that don't do anything got buried in the all of the data that it went through that it had to retain as context. And by the time it got to the end, it kind of remembered the first instruction or didn't at all, and

then just went ahead and hallucinated that it needs to delete the emails. that's that's a that's a big problem. How do I know when I'm listening to the barrage of things what I can retain and what I can safely discard? so, there's a couple of ways to do it. One is that before I even go to process all of this I make it smaller. So, if I

had a way to compress it, the amount of context is low, and I'm less prone to blowing the context limits, and it's more relevant. And there's two kind of ways of doing it. One is summarization, you you essentially like take a little model and kind of say, "Okay, well, I have a a lot of story but now, let's TLDR it for me in a nice way that

I can read still." A human would look at it and say, "Yeah, yeah, that's a good TLDR." And that just gets rid of like a lot of noise, right? But still retains the the essence of it. The problem with summarization, TLDR, yeah, you don't have all the nuances. We threw them away because we wanted to compress. So, you get more of vanilla, not so interesting, not so

creative, not so insightful results when this is used. Um the second one is a little more technical, LLM Lingua. It It It it passes through another model. It summarizes it internally. It It's also a summary summarization or compression technique, but it produces something that is less likely to ever be read by a uh and uh still retaining somehow that context because it's very model specific how it

does it. So, that's one idea. Is to compress things before it even went into the processing. The second is to do it Meaning in the model. And that one is more mathy and more complicated. But the advantage there is you didn't throw away things you didn't process yet. You're throwing it while uh the the KV cache uh pruning um is is just saying, "Well, part of the

issue we had is we had used a lot of memory and we can't just memorize everything forever. So, we'll just come up with a way to retain the important words and then be able to evict words as we need, evict words that are less uh uh And then just tokens uh is is another very binary technique uh that that that attempts to compress the data further into

a thing that nobody can understand except for the LLM. Uh so, it achieves very high compression, but it's very difficult to debug or reason with because everything is like compressed beyond recognition. Anyway, internals. One of the interesting thing as I was reading about the LLM lingua, which is a is a is a compression, not a latent compression, uh is that when when you're trying to figure out

what is the next word or what are Sorry, we're not We're we're trying to figure out which words are important in our past so we can carry them forward, One thing would be like is like, hey, let's pick the most popular words, right? Because they're important, they're popular for a reason. Everybody speaks them. LLM Lingua does the opposite, which was counterintuitive to me. What it does is

it's trying to find the most surprising word. Because a surprising word is this element that kind of sparks our memory, right? This memorable moment is rare. The rarity implies some kind of importance. It piqued our interest. It has context that us humans like. So, LLM Lingua works on this principle to pick important context by using the path of most surprise, not least surprise. It's calculating something called

a perplexity. Uh and the math behind it again a little above my head, but uh that's that's kind of the principle of operation for it. And we may think that feeding it forward and what I said so far is like, oh, you take the prompt, you pass it through the thing, you do a loopy loop, you bring it in and that's Well, for LLMs, they're called that

first L is large in the large language model uh acronym. It it it means that it has a lot a lot large blocks. And there's also an an amount of parallelism inside. When they build these models, they put in a lot of transformer blocks. The idea is that instead of saying I have one path for the data to just flow through, we're creating kind of a graph

and architecture where we're processing massive parallelism across many many many transformer blocks, each one with a little different twiddle of what the parameters are. Think of a polynomial 100 factor long and each constant C is slightly different. Right? The first term to the end, the next term to the end plus squared and and third and you know, think of a polynomial but have coefficients all slightly different.

And we're going to pit them against each other. We're going to run all of them And we're going to pick a winner out of it and we're going to say, "Ah, this one gave me a good result. So, I'm going to take this one on this round." And oh, that one gave me a good result. I'm going to take that one on this end. So, we have

a block architecture within the transformer that just has many many blocks. Each block has its own expansion activation processing of of the of the tokens and finally contraction into your your your same internal token a multi-layer operation. So, with that and I I think I went a little mathy and a little attention in this and could be confusing and the the the purpose of this talk is

to develop an intuition around it, not a working knowledge that you can say, "Ah, now it's clear to me. I'll just bang it out in Python and it'll be good." Uh so, I want to kind of uncover the concepts of what happened there and then maybe map it to how we perceive it. And the main complaint about AI is like, can we trust it? What about these

hallucinations? So, hallucina- hallucinations are not a model problem. The model doesn't suffer at all. We suffer. We interpret the results as hallucinations. We say, "That's no good. I want a better intern." Like, "No, the intern's perfect. The intern's great. Seriously." it's just that we interpret their stuff without understanding what their life is about. And now that we understand that there's probabilities and they just pick and they

go through a lot of machinations and they know all the stuff, but they may pick a probability for a word that you didn't think of picking and from there they just ran with it and continued picking things and now the result is like total garble or a really weird story. you know, the different reasons for this idea of hallucination. Um one of those hallucinations is the reasons

for seemingly hallucinating is that the intern would say, "Well, you know, I had all the books in the library, but I didn't read them all. It's too long." Simply under trained, really. And if the intern wasn't trained, they don't have probabilities for certain or they have different probabilities than what they would have had if they read the whole assignment and therefore it produces meh, lackluster results. uh

reasons for hallucination is it just made sense at the time. It saw probability was one of top K, ran with It produced something. In the real world, it could have maybe been read somewhere or something, but maybe not. Remember in the first slide, I said they do not remember full sentences. They just remember probabilities of a word giving the word before it. So, those kind of fabrications

uh come from the fact that it is a stochastic It picks at random from the next possible token, and that pick may lead us somewhere that is not what we would have wanted. Next, oh, yeah, I forgot. If the context is too big, if the physics of, you know, the model with all of the billions of parameters, all the matrices of size whatever, if if we've exhausted

those sizes, things kind of clear out of the KV cache, things get forgotten. And now it's like, oh, yeah, yeah, yeah, Uh yeah, you did say that, maybe, but I kind of forgot as I was working through it. So, when you hit your context windows, you will get garbage, and sometimes destruction if you let it control your operating system and delete your files, as you can write

read on Wired. Um the next one is uh you kind of get the excuse of I got distracted. I got distracted, human. What it translates to an LLM speak is it you may have given it parameters allowing it to look at low probability and it started chasing it down, it it wasn't the best attention, maybe. And then, um you also get I don't have a good explanation

for the mechanics of it, but you also get the confusion of terms, which is to say, when we think of like old context, we're we're always assuming that us as humans the recent stuff we really remember and the furthest stuff behind we don't. Therefore, we kind of concentrate on the most recent. It seems reasonable. You're kind of living at the edge of your your attention kind of

sweeps aside. But sometimes we see that older context get mixed with very new context. here's $10, please get my boring list of items this and also buy candy for yourself. They heard the here's $10 and they heard the buy candy for yourself. That's kind of the mixed context stuff. And that happens again because of the mechanics of attention, because of the mechanics of how we decide what

is relevant, may have picked some stuff compacted into the context and discarded the rest. So to us as human it says like, "Dude, you didn't hear the whole middle part." But to them it's like, "I paid attention to the important things and I did my best." So this is all internal LLM thing. we also have other techniques we hear about rag retrieval augmented Rag is not an

LLM and AI thing. It's an algorithm around it. Rag, when we hear about that simply says, "Hey, instead of generating a prompt that is just what I as a user asked the GPT I'm going to take that prompt, but I'm going to also pepper it with important information." What is that important information? The model was trained a year ago. It didn't know about what happened last week,

but I have a database of news from So, if I take the question and then go to the database, find news articles that are important about yesterday, plop it together with my prompt, it really becomes this, "Hey, tell me about the latest news and by the way, here are the articles." And then it can say, "Oh, here, I can process this one." Right? But otherwise, the LLM

doesn't know everything. Why wouldn't it know? Because it was trained long ago. Because the data you want to augment your query with is private company information not available to whoever trained it. So, rag was and still is a very useful technique uh for overcoming the challenge of the LLM doesn't know about your company or about something really recent. Freshness and and uh secrecy, uh I don't know.

combined. So, that's the rag technique. It is not a different LLM. It is not something that is AI inherently. It is prompt engineering. It's on-the-fly prompt engineering that enriches your user query with stuff that's important in order to answer a question. Very powerful if you cannot retrain {slash} um it's called fine-tune a model because you don't have the money, the know-how, the patience, the time, then rag

is very, very useful. Rag underneath the scenes under the hood always uses always in my experience always used uh vector search. Vector search is a way to transform text, just like I told you in the beginning about encoding and stuff, it transforms it into a vector, creates a vector which is just a bunch of elements, a bunch of numbers, and then goes to a database where all

of the say news were already encoded with vectors and just says, "Hey, which one is closest to this vector?" Why saying vector? What's importance of vector? Well, in rag there's something called an encoder, which is also a trained model, but it produces embeddings, not generates. Those embeddings capture semantics. They understand that white horse is not the same as uh horseshoe. It understands that um you know, a

mouse in a context of like I need battery for my mouse is different than a mouse as in the mouse ate all my cheese. So, these these kind of semantics are embedded into those vectors, and that way when we look for other things with relevance, we say, "Hey, the one with a closest vector to me is probably meaning the same thing, therefore it's relevant, therefore I'll take

it. I'll take the articles about the mouse ate my cheese, and now I can uh tell the LLM, "Hey, now tell me about why did the mouse eat my cheese?" Cuz I have articles about it. So, that's the general idea there. and another tool that again is not LLM, it's external, it's an application, is agents. We hear about agentic a lot. Agent let us do a bunch

of things. They let us divide bigger problems, have the intern coordinate other work to be done. The other work could be other interns doing just generation, but very focused on a piece of what I asked it to make. It It be like read my email, so agents can talk to tools and say read me some proprietary information, information off look at the flight schedule, blah blah blah,

and it can also perform an action. So, agents are also a way for the LLM to reach out of the realm of my screen and actually affect something in the physical world. So, agents are very powerful thing. With power comes risk, but it is a way to extend the capability of the LLM from just generating one thing on screen to doing things in the real world, to

do things in parallel, to do things faster because of that, or to handle just way more if we can cut our problem into pieces and hand them to agent to do independently. Agents internally are just an application. They're not an LLM, they're not requiring a new LLM. All model builders build models that can use uh uh tools and things like um uh LangChain uh lets you integrate

and a and build agents uh fairly quickly and and easily. Uh roughly speaking, the LLM is given an instruction saying, "Hey, here's a bunch of tools you could use and here's what they do. So, when you consider the input, please look at the list of tools and if you if they're useful to you, feel free to invoke them. And if not, just don't." And that's roughly the

instruction plus a user prompt, and it does the same because it now has this ability to activate those uh tools. So, with that, we zoom back and we have the big picture. We have an intern. We give it a task. Is the task kind of conversational? Well, if it's conversational, we can go directly to the LLM. That's here. Also here. Uh directly to the LLM. Uh conversational

stuff, you talk to an LLM, it's been pre-trained on some information. Different LLMs have different things, right? Some of them were trained on books from library, some of them were trained on SQL code, some were trained was who knows what, right? Medical, legal, blah. So, they may have many flavors. They may have many sizes. You know, you hear about LLMs, SLMs, you hear about billions of tokens,

less billions of tokens. All available. Then, if we want to have better generation, quote-unquote on the better, I guess, because what is better? It's what we judge. Um we can have better generation through rag, which um lets you focus on things uh because we take the query and we dig relevant articles to what the query was about. So, it lets it focus on that instead of being

generally you carry the whole library in your pocket. You're really focusing on that. Uh it can access internal data, again, Chat GPT didn't have access to my internal company documentation. So, I can do that with rag. and fresher data. Chat GPT was trained a month ago. Well, 4 5 just came out, but you know, a month or two ago, it will be a month old, and I

have maybe fresher information, so rag is really good for that. Another way to better generate is to use purposely purposefully trained or tuned Those are models that are trained not just on understanding the general English and literature and stuff like that, but also um some very proprietary or specific information such as how to code, medical legal information, news, things like that. That's all textual. and then it

could be trained to do things like generate an image or a video or audio. So, those are specialized models. They work the same internally. They would just take a picture, convert it to tokens, do the transformer blocky thing, blah blah blah blah blah. Same thing, just different inputs and outputs. Mechanically same. And then finally for complex tasks that you just can't go to an LLM directly or

to an LLM through augmented sorry dynamic prompting, you can have agents and that lets the operate more as a coordinator of tasks, launch different agents and use capabilities achieve something like booking a flight is algorithmic. There's a tool to do that and the agent can talk to it and operate it. You can have another agent working in parallel, so the ability to cover much more complex problems

is is much higher. Um yeah, so that is the big picture. treat your interns well. I I mean I I think that the enthusiasm of LLMs always reminds me of just somebody who's eager to do things, well educated, somehow has doesn't seem to really understand what the real work is about, but is able to continually do it and I think that's that's a very good mental model

to to think of it. Uh yeah. Uh and finally, this is kind of you know, very very high level and math underneath this is very complicated. I don't understand it all. Uh but also, some of what I showed you today will be obsolete 2 years from now. The the construction and architecture, if you listen to AI news, it keeps evolving in very very rapid way. people achieve

a lot more with a lot less by being clever about the math, uh scaling down numbers to use 8-bits, um doing sampling, doing all kinds of uh crazy stuff, uh and and get getting very good results. But, that's all I had to say, and I'm happy to answer questions if you Any questions? Any comments? Everybody's lost the context window. Yes? will you be putting this online at

some point? Um I can. I don't think I have, but uh I think Scale has a place to put it. I'll I'll definitely put it. And if not, just come to me, and I'll email it to you. I read a paper recently that was talking about something called H neurons, and the idea is that like I can barely hear you. You were reading a paper, Okay. >>

Okay. Uh so, I was reading a paper somewhat recently about something called H neurons, and the idea is that like the you were talking about the hallucinations a little bit, I think. And the the idea was that um um that there are neurons that the um model will go through that light up when it usually would hallucinate. And those actually are related to the neurons or are

related to the agreeableness of the model. So, um like it's a it's a desire to say yes to you, basically. So, you were talking about how it's like an intern that always says yes. And so, they were saying that like for it to be more factually correct, you would turn those neurons down. And if you turn them down, then it's less agreeable. So, it's interesting because it's

want to help you may actually go down as it becomes more factually correct. first of all, thank you very interesting. I I think this again shows the language that the gentleman used is a language of neurology, which kind of makes me feel good that this is all about us humans understanding better about what we consider intelligence, what we consider thinking, and how we may go about understanding

how our own brains work. And what I showed you before that was called a perceptron, that is a term, you know, from biology. And those neurology ideas of neurons firing and then, you know, maybe some areas of the brain activate this and others. I am not aware of work that shows areas multilevel, you know, tournament that is the uh um the the the the the the layers

there, certainly, that that seems like a fascinating area to look at because we as humans have areas of our brain that have kind of the same chemistry, but they take care of different concerns and they do it focused. So, that's interesting. I love it. Yeah. Set a clarifying question, you said that for AI agents they're standalone, but for LLMs they're coordinate with with them. So, like for

example, you have an AI agent, but then something like Claude is the actual LLM LLM that coordinates with the agent? Exactly. So, when you look at Am I too far on Yeah, when when you look at an agent is and could be Claude itself could be considered an agent. It's an application that is capable of taking prompts activating tool as needed or not or just using the

knowledge that's baked into the LLM there. And having a multitude of them working on a problem, so you will be creating an AI application that talks to a multitude Uh so, agent themselves can become a tool to an Uber agent above them. So, it's kind of a tournament arrangement. But, this whole idea of agentic is not an LLM. It's not the brain. It's the instructions the brain

is given and whatever the brain already knows plus tools that it can interact with such as read your disk format your hard drive book a flight turn on the light. All right. Uh one or two I I'm happy to stay until 7:00, but uh you tell me. Yeah. Yes. Can you go back to that slide where it said what what would uh Oscar Oh, okay. Cuz it's

uh That one slide threw threw me for a whirl because um you know we was we always thinking in AI is like a you know as a well they're tools. Okay, yes, they are. How would they help us? Um if I go to the to the bottom there of creative writing and storytelling, it it it hit me that like there's um if I go to this the

the way other side of the spectrum where it's like I can really like uh how the question is how how far can I extend this hallucination to be very creative but like in an interesting way. So, are you acquainted with this book Finnegans Wake by James Joyce? This novel? Um I'm not familiar with that. I think I understand your your question to be And and first of

all, I want to I I want to say something about hallucination and creativity. Uh-huh. In my mind, I map hallucination to the realm of errors and creativity to the ability to produce something that is useful yet maybe something somewhat unpredictable. And I think part of what you're talking about is like, "Oh my, can I just like amp the like why we we're so focused on like making

things so concise and predictable and good? Creativity is like, "Dude, just make the music. Invent something new." And if we can just amp up the creativity, then the model can just help us like totally blow our minds. Uh and I just wanted to note that you know, creativity, hallucination, two different things. Creativity is about how likely am I to to pick something of low probability relative to

the others continue to generate from that and come up with something coherent. It doesn't have to be like correct, but it has to be coherent in some way for us to really consume it and say, "Yeah, it did something good for us." Right. Well, not necessarily doing good something good for us, but maybe interesting in the sense that like this in that book Finnegans Wake by James

Joyce this this Irish novelist he published I think in 1939. He there he's he's um it's nearly unreadable because it took him like 10 years to write it, but he's he's writing in this like dream state where you're mixing like memories and places and it's like hallucinations and but he's he's taking from very languages and dialects that are ancient some that are obsolete but languages all over

the world. He's creating his own idiom. But he's writing in a in a form that's like it's it's a not it's a not chronological. It's it's mixing characters and places and memories like how you would hallucinate Right. >> like in a dream, but he's writing that it like a in a meta way he's there is coherence there in a creative way in the book. So, I'm wondering

in the future of course not now, but do you think that level of like meta co uh um uh context could be created by an AI or is that too much of a stretch? Okay, so I'm not a you know AI predictor of the future, but thank you for the question. First of all, I want to note again mapping to humans we have poetry we have prose

and we have storytelling and when we have scientific writing. Those are all forms of expression with different purposes behind them. In prose we combine poetry and words arrangements that do not necessarily map to our everyday speak but expand our mind and resonate with us in some ways. Uh uh something that is like the dream that seems like hallucinatory or LSD like it it doesn't mean it's not

human or we can't understand it. We still have access to it because those are patterns that we recognize as language that evokes emotions or brings thoughts to our mind. So, it's all on the table. As of today, you can take James Joyces and take La Reve and feed it and fine-tune an LLM and guess what? It's going to produce more like that. But, if the probability of

the word was zero in the train set, it's not going to get picked. It has to have seen stuff. All models are trained on things they have seen. They're not going to take a random word from the vocabulary. They're going to take a random word from all probabilities of I've seen this before. So, that's where maybe maybe not. I don't know if you're going to get your

effect, but you're going to get some effects that are very close to it. Yes, the last one I guess because people are already antsy, but last one on speaker and then we can talk more. Yeah. Hi, yes. Uh in that slide you had with the um rag and the LLM, um it seems to me like you Yeah, go Yeah, it seems to me like the the the

calculation there is um and then my assumption there is the rag tokenizes it and then into a form where it can be joined with the LLM's result. So, it can be >> no. No, uh sorry, uh that's misleading. So, in in rag, what we get is that the query you submit, "Hey, what's the news yesterday?" gets created into an embedding. An embedding is a separate model that

always encodes into a same vector. It takes "How's the weather?" into vector with 1 2 3 4 5 and "How you doing?" into 1 5 4 3 You know, always the same length vector and it just encodes it in Those vectors have a special kind of spatial property that they project into where semantic meanings that are the same are in the same region in the N-dimensional space

and things that are further away are in a different part of space. Therefore, mouse cheese live here and mouse battery live here because one of them was that mouse and battery are in the electronics realm and the cheese is in the food realm. And it's it's N-dimensional space, so it's hard to visualize, but in two-dimensional space or one-dimensional space, imagine, you know, I have a a spoon,

a fork, a knife, comes a spork, oh, somewhere between the spoon and fork, right? That's what Wally did. So, yeah, that's that's kind of how it works there. And it's not the same model that is the LLM. That happens before it's fed to the LLM. It's just a pre-step. Takes the query, converts it into what's called an embedding, it's just a vector, and that vector gets compared

to all other vector you precomputed on the document, so that's a fast way of looking through the database cuz you don't want to read every document and every word. You want the semantic meaning captured and that's what the embedding model Looks like there's a lot more research I'm going to have to do to understand it, but thanks. Thank you. >> Yeah, this this is all very off-the-shelf-ish.

There's if you use, let's say, MongoDB Atlas, they have auto embedding. You just give it your documents, your news, and they auto embed for you, and then you feed it a query, and they quantize and do the thing. It Yeah, those are all available. Yeah. So, there are any search similarity to do like the closest of the vector? Yep. The closest in space. Yep. They're doing a

dot product closeness or a Euclidean closeness or a cosine closeness computation, but conceptually, the closest one. Yeah. Or the closest ones. I I think I'm going to stop just so that everybody could feel free to get up and do their thing, but if anybody wants to talk after, I'm happy to do that. Thank you so much for coming.