Room 101 Thursday Mar. 05 - SCaLE 23x

This talk focuses on the evolving landscape of the software development life cycle (SDLC) and the implications of emerging technologies on traditional practices. The speaker, Kelsey Hightower, discusses the integration of Nix and its potential to revolutionize package management by providing a more universal framework. He emphasizes the importance of understanding how existing tools can coexist with new advancements while reinforcing the need for developers to grasp fundamental coding skills. The session also explores the benefits of community engagement in technological growth, specifically regarding the influence of AI on coding practices and the necessity of maintaining a human connection in software development. Kelsey encourages dialogue among participants to share experiences and insights about implementing Nix in various contexts. Overall, the talk aims to inspire collaboration and innovation within the Nix ecosystem.

Good morning. Hi. How's everybody doing out there? You don't need to clap. It was just I'm, you know, thank you, Cleveland. No, it's not that. Uh so uh I'm Mike. I'll be hosting today. Um most people call me Stony because Mike is not a unique identifier in any situation. Um so I go by my last name usually. Uh I I guess I do engineering and marketing stuff

at Flux. Um and volume up. Is that Do you want me to eat the mic? Is that I can do that. All right. Thank you. Also, if this happens again with other speakers, do the same motion. It was very noticeable. All right. Um, so welcome to Planet Nixs. Uh, this is our third year doing this, I think. Is that right? >> Second. Okay. It feels like three.

I don't know. I've been here a while. Um, anyway, this is awesome. Uh, so thank you so much for coming. If you registered as a VIP, there are shirts available. Um, if you didn't register as a V VIP and want shirts, um, maybe wait around later and we'll see if we have some leftovers. I don't know. Um, let's see. What else do I need to tell you?

Uh in in case of emergency, use the doors to exit. Um the restrooms are back that way behind the hall, behind the stairs. Um the Wi-Fi uh I guess if you have a GitHub token or if you're doing lots with GitHub, please put a token in your shell so that you don't rate limit the entire conference for the outbound IP. Uh ask us how we know. So

um see what else did I want to cover? Jackie's around here somewhere. She's in a black sequined uh jacket. She's over there. Okay. I couldn't see you because you were off camera. So, um, but Jackie also will be hosting and if you if you need anything logistically going on, find anybody that works at Flux and we we can definitely help you. But generally, Jackie and I will

be hosting or or figuring out what's going on in the different tracks. Um, and is there anything else that we that we want to cover? Anything that I missed? >> Videos. Oh, we are requesting some people do videos that talk about their experience at planet at Planet Nyx where we will ask you some questions and we'll cut them up and do some little social commentary, you know,

social media stuff for us. Um, you might be on it, we might use it, we might not. We we might make a cool super cut. Maybe if you want to do a Tik Tok dance, like whatever, we're in. Um, but find Jackie and she will uh help you with that. And we have we do have some door prizes if you participate in those activities. So, it's it's

not just for nothing. Um, I don't know. We have we have some other cool stuff going on. But, uh, do check out the booth and and thank you very much to the sponsors. Without these sponsors, this conference couldn't exist. Um, and it really does matter. So, thank you very much to our our planetary sponsors. Um, some of them have booths out here. Some of them have excellent

products that you should be using. Some of them, uh, you know, they have they just want to be aware and like, hey, they want to be part of part of the next community. And that's awesome. So um anything else we want to >> We are getting some feedback I think from left. >> It's it's not in the recording. It's just live. >> It's just annoying these guys.

Uh I don't know if there's anything to do about the that do we All right, >> got it. Um, just to find the AV people. >> Yeah, >> that that was just a long explanation that we need to talk to somebody different. >> You didn't You didn't miss anything. >> Sounds good. So, we already mentioned the videos. If you want to talk about Nyx, what you're doing,

anything technology, come find me. Let's talk about it. I genuinely want to hear what you're doing and how you're building. And in addition to that, if you've stopped by the check-in table, we do have some different stickers. And there is a little bit of a game around some of the stickers. So, I would encourage you to grab them. If you have not grabbed a Next Store Path

sticker yet, please do. And well, yeah, read the sign, find the game, and figure out who your people are that you're m matched with. But yeah, I think that's all I had. >> And with that, we're gonna hand it over to Ron. He he's he's my boss. Uh, no, he's he's uh I guess what are he's the head of the Nixos Foundation. I I hear that's important

to this audience. Uh yeah. Anyway, here's Ron. >> Thank you. Uh all right. Uh thank you, Stony. Couldn't have prepped that uh intro better. Uh, I think Elon stole a chair from us at some point. So, if someone can find the second high chair, that'd be cool. Either ways, I can stand. Kelsey can sit. Um, all right, everybody. Welcome to Planet Nix version two. Uh, every year

we're trying to make it better. Better for you, better for the folks that are coming in, better for the folks who are watching online. The entire idea behind Planet Nyx kind of started a few years ago because we really saw a need to see if we can bring a kind of a amalgamite of the folks who are thinking deeply about Nyx and the folks who also want

to use Nyx in the workplace, right? So and and kind of talking together about how that can happen, what is being done, what are some best practices, how have folks here who passed through that, you know, mystic uh enchanted and scary forest of convincing their boss to pass Nyx through security. Uh so just just like to get things started, a show of hands. How many folks have

been using Nyx for less than a year? Don't don't be shy. We want to see you. We want to see you. It's fine. Uh, cool. It's like I think maybe like 20%. That's pretty neat. Uh, how many folks have been using Nyx for less than three years? Okay. Okay. About half the room. U, how many folks have been using Nyx over five years? Over. I went from

less to over Cool. Over 10. Okay, leave your hands up. Leave your hands up everyone. Look at those people. >> find them. >> Okay, that's why they're here. They don't know it. They're not being paid for it. They're here to answer your questions. Um, thank you guys for that. Um, I guess next set of questions. Who here is using Nyx at work? Whoa. All right. So, obviously

the cameras can't see this, but I would say like 80% of the room raised their hand. uh for the folks. >> Okay, good. How many of you How many of your bosses and know you're using Nyx at >> Yeah. Okay, so again, for the folks watching online, I think 80 to 5% of the room responded yes to using Nyx at work and then 50% only responded to

their boss knowing about it. Again, you can only see me, so that's hidden. Um yeah, so one of the things that we'd love to get out of this is obviously we're looking to to grow the Nyx ecosystem, grow the contributor base, grow how sustainable, how secure, and how instrumental Nyx is when we're looking at the modern SDLC and that's what this these two days are about. So

talk to people, figure things out. Uh if you have something to share, share it. We have an unconference. Jackie has the video team. Uh we have uh Linux unplugged folks. You can raise your hand up here in the front. They're doing the audio stuff. So we're trying to cover as many many ways of getting the message out there as possible. So you have all those outlets. Use

them. Um the other thing I'll say is that every year we're trying to up it a level. So this year we have lunch. Last year we did not have >> I know. Um, and I'll say another thing. Rock, I actually don't know the ice cream situation, but I just do know that if we don't have ice cream, Rock will be mad. So, for those of you that

know Rock, he name person, it's perfect name. Um, all right. So, with all that intro, um, we're going to kick off, we kicked this off the same way last year with, uh, Kelsey High Tower coming in. We're gonna be talking about why Nicks. We're going to be talking about the broader spectrum of things because this conference is also inviting a lot of folks that are just not

Nyx experts, right? They don't necessarily have to know all the nit and gritty details of our binary cache or the Nyx language or what the latest contribution was because Nyx is not just for the folks who are using it at that foundational level. Nyx is for everybody and that's what we're trying to do here today. So, without further ado, uh I do want to welcome Kelsey High

Totower uh to the stage. And do you have a mic? Mic. Oh, we need to mic you up. >> Yeah, there's a handle. Cool. Um, do we have a chair? Do we find a chair? >> Oh, okay. They're like They're like We're 10 minutes early. >> No, we need we need the high chair. >> I guess >> we can start early. Yeah, we can. >> Can we

start early? Do we have permission? >> Yeah, there's still people in the hall. >> Oh, there's still people in the hall. So, what do I do for 10 minutes? >> I think we start. How about this? How >> We'll do >> Q&A for 10 minutes. >> We I think the most important part about any fireside chat. >> Can we start? Thank you. >> All right, let let

me get you Let me get you uh Let me I'll get another chair. One sec. >> Just get two chairs. >> Sorry, we're we're islanding you here. Here we go. Here we go. >> Damn, look at that. >> All right, we're we're not going to talk about the important stuff until uh everybody gets in here because Yeah, I see a lot of people streaming in. We promise

we'll start attend Who starts a conference early? >> We do. >> We We're so cool. Um so, so we'll start the important stuff uh when the next talk comes up because that will be the important stuff. We're going to talk about the not important stuff. Uh but you want to start with Q&A just like riff for a moment and then at 10 we can start with topics.

What do you want? >> Why am I here? Right. If you're like, "What? Why is Kelsey here again?" I asked the same question when I got invited back. Um, I think it's important for any technology stack to realize the bigger picture. Like when Docker came out and Kubernetes came out, there were people starting to get the tattoos. People were getting just so myopic. They couldn't see anything

beyond the thing that was right in front of them. I think this community is a lot more mature at this stage than even where Kubernetes is at this point. But I also thought it was good to have outsider perspective. So before Kubernetes came out, I was very happy to have been a system administrator, a DBA, a network person, a product manager, an open source contributor. And so

I wasn't necessarily blinded by Kubernetes being this new hype wave. I can kind of see the things that it didn't do. Also figured there was things it shouldn't do and there was things that it could do. And the visions I had for what Kubernetes should be able to do were the things rooted in reality. the jobs people are actually doing in the real world. And luckily for

me, I was at a startup and then eventually I was at Google and I was seeing much bigger problems than Kubernetes was ever designed to solve. And using that input into the ecosystem, number one, we made it more inviting to more people. So cloud native became more than just about Kubernetes. It came about can we take this philosophy and way of thinking and put it in more

areas even in the areas that Kubernetes isn't suited for. Luckily out of that we got people rethinking networking you know service mesh I apologize for anyone that went down that road um hold >> oh and then we also got things around like open telemetry right some consistency around observability again these are things that are not just necessary core to Kubernetes but just thinking bigger and giving people

inviting more people with more ideas in I really think change the trajectory of what Kubernetes is now so 12 years later Kubernetes looks like it's going to be a 20-year technology mainly because it's more than just Kubernetes. And then they made the pivot to AI. There was a chance that AI didn't need Kubernetes anymore. There were some people thinking we should have just built a new platform.

There's a new workload type that's more expensive, more performant, more performance concerns than what we currently have. But for some reason, we brought in the right people that they chose to build on top even for the next hype cycle. Very rarely does a a technology get to span multiple hype cycles and still be relevant. And so my goal today is I also do a bit of investing

and not that the VC part is important. It's the part that I get to see a lot of companies trying to do a lot of things. And there's one consistent narrative is this AI agent pivot. I haven't seen this many people um try to commoditize themselves this fast in a very long time. And then so how do communities like the NYX community stay relevant when the thing

that's being hyped up is looking to bury and hide as many of these tools as possible because now people believe that they can create their own software and their own workflows. So your thing that was top of mind turns to a skill in a markdown file. It gets hidden. And it doesn't mean all of our experience gets thrown away. It just means that our experience gets just

be part of someone's workflow. And so I do think Nyx and we're going to get into it today. Do these projects, all these command line tools that do all this work, do they just become guard rails to this new modern workflow that people are pitching or do they disappear because we won't need them anymore? I think that is a very valid crossroad to think about. I'm in

the they become frameworks for the underlying tools because we don't need to reinvent all this stuff again. There's a lot of people that don't even know what this stuff is. And when people don't know what already exists, they tend to reinvent the wheel and the current thing kind of just gets pushed aside. And so I think there's a lot to talk about. We'll cover all of those.

And ideally, we're going to take as many questions from you all. The better the questions, I promise, the better the answers will be. So that's your part today. >> Yeah. And I'll I'll further emphasize I see more folks coming in. There's a little bit more spots out here in the front to come sit down. Um, but in general, we do want to turn this as conversational as

possible. So, we're going to kick off. We're going to talk about a few things like Kelsey said. We're going to kind of talk about industry, where things are going, outside perspective into Nyx, where Nyx is going, some opinionation on what Nyx is today versus what it can be if we do things right. Um, but we really want to turn this into a conversation with the hundreds of

folks right here kind of sitting and that came in all the way from many different places. So raise your hand. We'll we'll kind of talk about that. I'm going to check time like three minutes out. Okay. Um >> start. >> Yeah, we'll start. Um all right, guys. So again, u for those of you that don't know me, I'm I'm Ron. I'm um I'm I'm part of the

Nexus Foundation and I also run run Flocks. Um and I think I don't know if Kelsey needs an intro. Kelsey is Kelsey. uh he's done a lot of cool stuff and we'll talk about those cool things as we go through it. I think just just to kick off, right? I think one of the one of the things that's that's been top of mind for me and I'm

I'm assuming it's top of mind for for a lot of folks here in this room is what is actually happening right now, right, for software engineering, the SDLC, like what is that ecosystem? What is it today? And and do we have any thoughts about where it's going? Because I think that's going to lead us to talk about Nyx and Nyx's place in all of it. And before

I turn that question to you, I actually have a question for y'all. Um, how many folks here have written a line of code this week? >> No, no, no, no. Like written a line of code. You you went in there into an IDE or wherever you're writing your code. Good for you. How many folks here wrote a line of code in the LA like last year? Look

around, right? Things are changing, right? Like the way that we build software and create products in the ecosystem and create value in the ecosystem is changing very very aggressively. U right we have we have our friends from anthropic sitting around here that are part of that reason. Uh but I think going back to the question like what what are you seeing? >> Look, I'm going to be

honest. I am an AI skeptic and it could be because I spent all these all this time learning this craft um going through the experience part of this craft. It's not just learning how to write code or learning how to write SQL or debugging network packets. It's literally the experience that goes into trying to build things and having this context. So for me, I've always been in

the train your own model. I think it's okay for humans to know how to do stuff. Um, and then also remind myself that we train the machines where you think the data set comes from. So I'm very biased in that regard. Now I do think a lot of people wish software would go in a certain direction. I think a lot of people wish that software development would

go into a certain direction. And >> what directions are you seeing? What are they wishing for? >> Today you have agency. Most of the languages we use, most of the runtimes we use, most of the operating systems we use, you can just go download them and build whatever you want. But you have to have the skill and that's a big bar of entry for a lot of

people. Some people just don't have the skill. So if you don't have the skill, you almost can't build anything. Sure, there's some drag and drop what you see is what you get editors out there, but for most people, the skill is a big barrier to entry. If we think that we're going to make everybody go back and learn the same things we learned 20 years ago and

go through the same path, some people might consider that gatekeeping. If you are now given a tool where you can express what you want and it spits out something very similar that someone with 20 years of experience could do, you're going to be excited about that. So for that next generation of software developers, my daughter's speaking at this conference, it's her first real conference talk for the

first time. She's 18. But she uses these tools. And as much as I want to romanticize about the idea of like, no, no, no, no, no. You need to scour the web. You need to copy and paste from Stack Overflow. You need to see your machine segur, it ain't realistic. The search results are pretty terrible these days. And there are things that you can do now. It

feels like a superpower. So, who am I to tell her that's not a great way to learn? And so, for her, I can imagine why she would want the world to go this way. Like, and every once in a while, I'll download one of these tools and just see what the flow looks like. And there's no denying I can't remember the entire standard library. And then when

I do visit the docs for the standard library, there are no examples. I don't know why in 2026, we still believe that humans don't need examples on how to use things like functions or what the system calls actually do. And so in that case, what do we do? All y'all, I think if you're being honest, used to go to Stack Overflow, too, right? All the law, you

copy and paste. So you were kind of doing some of the similar things that we accuse these tools of doing, but when I use those tools, the inner loop is much tighter. When I ask for an example, a similar quality thing comes out. Last year, I was even more skeptical. This year I'm watching that thing try to compile it first, look at the errors on its own,

go back and regenerate the code until it at least builds. For a lot of people, that is a huge boost. The thing that scares me though is if we did this for 10 years in a row, how many people actually know how to actually write code? I know a lot of developers that can't write a SQL query. full-time enterprise developers if they don't have an OM they

can't write a SQL query left join we're asking for too much select star from all the tables and just filter the data in the programming language of your choice and so I don't for some reason I just don't want that to happen and so that's the way I think I see it now there's two branches >> why why though right why because because >> I love the

I love the profession I love the craft I'm not here to sell you the >> so so what you're saying is that is like there's going to be software a software engineer there's going to be the artists. >> No, no, no. There's no guarantee there's going to have to be anything. >> That's the hard truth. If you all believe that it's just going to be there because

we want it, that's not how it goes. A lot of us don't own these companies. A lot of us are employees at these companies. They're given RSUs that you probably sell pretty close to immediate because it's deferred comp. A lot of us aren't that invested in this. And there are pressures, right? I'm watching people say, "Hey, I'm anti-AII until you see the layoffs changes your tune real

quick if you need that job." So, I think the reality is the tools of the trade will dictate what the hiring process looks like. I'm pretty sure someone somewhere is like, "I don't believe in typewriters." It's going to be real hard to get a job today if you don't believe in mechanical keyboards. It's not going to work. The team will look at you crazy like, "What do

you mean you don't believe in email and sending electronic messages?" That will change. It probably will. Where the inertia is right now, a lot of people are trying their very best to make that happen. So, we got to be honest about it. And if those tools are actually any good, would you not use them? Who uses a compiler here? Okay, there was an argument if you look

back in the archives 30 years ago, compilers are for people that don't know how to write code. What are you using a compiler for? You're going to let the compiler decide what instruction sets you use. People are still arguing about garbage collection. >> Garbage collection is still >> sensitive topic >> Garbage collection. Nah, real people manage their own memory. Buffer overflows and all. >> So the way

I think about it right now, that's what we're faced with the reality of it. So I think a lot of people here are practitioners. you're at a NYX conference, Flux conference, you are probably in the fringe part of the technology where you know what's going on. You've turned that knowing what's going on to leverage. And also, I would really hope that people who are in charge of

the core layers of our infrastructure actually know what's going on. So, I'm not surprised that people in this community need to know how software is actually built. We need to know how software can be exploited and we need to understand how these tool chains go together. So, I think this group is very special in that regard. But if you all zoom out a little bit, that's not

the norm. A lot of people can care less where the software comes from. App get install the internet until this thing works. Not sure who packaged it. I don't even look at the author's name. I don't look at the check sums. If it works, it works. That's that's the reality. And one thing you and I were talking about was, do you need package management in this day

and age? And you all would say, Kelsey, of course, this is a silly question. But have you seen people use some of these tools? Give me a thing that does X. And I don't honestly see these things go out and go find the right library. Sometimes it just generates the snippet. You remember the leftpad debate? Should you use things like leftpad when you can just write a

threeline function instead of taking a third party dependency? Well, every prompt is having that same discussion except for the developer isn't really making that decision anymore. So what happens to all this package management work that we're doing underneath the hood? Do we become the guard rails or do we get replaced by people creating software for themselves and not sharing it back? >> Yeah. And I think the

what I've been seeing is that it's becoming almost a prerequisite, right? Because I think I think another aspect here that's creating an impact on on what we view as software engineering is almost the competitive landscape, right? like what what allows you to be more successful. There's a bunch of of different things. I think in the past it was very obvious like what were your moes? What was

your defensibility? A lot of it was IP. A lot of it was the code that I generated. And the more I'm talking to folks, I think that that wall has started to kind of crumble. And and to your earlier, >> I challenge this. You know why? Because I've worked in small teams where I liked all the people. I was trustworthy of them. I knew that if we

gave a person a task that they could knock that task out and if they couldn't they would ask >> and we wouldn't shame them because we knew that they would be able to do the best of their ability and they were professionals. Yeah. >> And so we helped them when we could. I love working in those teams. But I've also worked in large enterprises where it literally

is a real-time Game of Thrones. >> You don't understand why decisions are being made. You're like, "Wow, this is a silly decision. Who made this decision?" Oh, the person that knows the least with the highest title made the decision for us and we just have to implement it. And that feels terrible. And so that loop has been some people's inner loop for the last decade or so.

You get the jar ticket. I'm not sure why a customer want to do that. Customers are always right. Copy paste stack overflow. Poor request. Two lines. We can talk about it for a year. 300 lines. It looks good to me. Merge >> and repeat. And so I think if you come from that world, maybe some of this tooling looks like it allows you to automate some of

the most tedious parts of that workflow. The things that I like from this is like you have to write a doc about what you actually want. Give me an example. If he was like, oh, I won't give examples to humans, but the AI agent needs examples. I will write all the markdown in the world. It's like Ben, we've been trying to get you to write docs for

like a decade, but you I didn't know if you were going to read them. Now you have to write the docs and the specs if you want this thing to do anything. >> I mean, who's who's choosing who's choosing what what goes underneath our request? Like, who's who's actually >> you know, this is fashion, right? So, let's get back to humanity. You know, a lot of us

are just cargo culting what we see, >> right? >> Right. Somebody does something, you go to a conference, you're like, "Wow, I remember seeing someone use Vim for the first time." I'm like, "Damn, he's Hey, uh, I have a question. Uh, what Vim plugin is that?" Like, dude, what? I talked about curing cancer. I know, I know, but I want to know what Vim plugin was that

your workflow was amazing. And so, a lot of times that's how this stuff spreads. That's the real leadership component. The people who get things done, we want to see how how are you getting things done? Like when you all talk about Nicks to other people, that could be a weird thing to talk but when you do, there's some value that you're putting out there. You're telling the

story, right? You're saying, "Hey, ever since I started using Nyx, my life got better in some way." And if I hear that story and I trust you, >> you lost weight, hair grew back. >> Yeah. Well, none of that worked for me. >> What? >> But then those become the things that make another engineer say, "I might give this next thing a try because if that stuff

is true for you then maybe it'll be true for me >> right and I think I think that's a good segue into right so so the industry right we we're talking about that right now everything is kind of in the air um we're changing our workflows I think folks are questioning folks are renaming software engineering have you seen that like there there's there's context engineer I think

u I think I think Mitchell had a really smart thing around like harness engineering um if you >> someone that went from being a system administrator to a devops engineer to a s SE into a platform engineer like I get it. That's what we do. >> Yeah. Um and I think and I think I'm seeing a lot of folks that have not gone through that yet, right?

There's like the first time they're like, wait, I'm a software engineer. How am I how am I not going to be a software engineer tomorrow? But I think that's a good segue to start talking about like where does it meet us in the Nyx ecosystem, right? like is is the is there something here that is uniquely positioned uh with Nyx I'm asking you as an outsider >>

so as an outsider the thing I've always wondered was at some point when does RPM Debian Gen 2 go away the package managers why would it go away and when I was at Puppet Labs I remember we started building our own package manager the first time that I built the package manager and I learned about the dependency diamond problem this is like in in actable thing. It's

not easy to solve. And I decided to say, yolo, let's just use Ruby gems as the baseline for it. And that causes all kind of problems because you can have side effects and things can execute underneath the hood. But we built the package manager when there was already like 18 other package managers. I don't know why people do this like a write a package passage. And every

language comes out with its own way of thinking about managing packages to some degree. At what point do we get to where people say, "Hey, maybe Nyx becomes a universal way to talk about the relationships between software and how we consume them, whether dynamic loaded or fetch and retrieve or use as composition for something else. We know it has those capabilities, right? >> But I don't know

if people are thinking of it that way because it could become the universal package manager and that's the vision that you all would have to put out there, >> right? And then the other one should be able to go away. And me and you talked about this earlier was if Nyx became the relationship tree of all software, then if you're Debian, you would go to that tree

and say, "We want this collection of packages to represent our district, right? You could put them back in debs if you really wanted to distribute them that way." No different than you putting that in a Docker container to have your container run with a subset of this nick tree. That could be a vision that you all put out there. And then the work becomes you convincing all

the existing people repackaging the exact same software to stop join this one effort and we'll build something that becomes an industry standard where there's one repository where we put all our effort and then everyone's free to pick and choose from that lake of software as a distribution target or repackaging to your own district of choice. that could be one outcome, but you're gonna all have to get

together and have a unified set of discussions and be very convincing, >> which is what we're planning to do in these two days. Right? So, the way the way that I've been looking at it and I've been talking to folks inside of Nyx, outside of Nyx, is I think the the more I'm seeing where the industry is going is that I would love for it to be

a fact, right? So to your point about the vision for for Nyx in in the modern SLC is that Nyx can be the pure way that we actually are able to derive the packages and the baseline infra for what we're building regardless if it's led by a human by an agent by a model it doesn't matter right so when you need something to be reproducible and you

care about the speed the infrastructure the security the underlying factors of it I think Nyx is uniquely positioned to be that platform Right. And again, folks in this room kind of understand the underlying tech for it and why it's uniquely positioned. But I think that's, you know, when you want to be pure, there's nicks. And I think it's also fine to tell the industry that it's fine

to be impure, right? Like if you want to just vibe code an app that you don't care if you throw it away, right? Like there is still a reason to use paper plates. It's bad for the environment. You shouldn't do it. But, you know, when when when you're tired at the end of the day and you have like four people over and the last thing you want

to do is dishes, throw the paper plates on there and then throw them in the trash. It's totally fine. >> But at what point do the LLM contribute back, right? So, right now we have this one direction that get trained to help people retrieve things from it train or generate things from its world model that it's developed on its own. So, we know what that direction looks

like. When does your LLM stop and say, "Hey, wait a minute. This is the millionth we're doing this implementation of a thing. When does it say stop? Let me create an interface for this, package it, generate it in every programming language and contribute it back as a reusable, if you will, cached version of this logic. We call them libraries. and you take that library and you say

going forward when anyone wants this piece of logic I will just add the import statement versus generating the same code again the problem with that is how do you get a function signature that everyone will be happy with right this is that hard part right if you create a function signature maybe 80% of the people like it and then do you create another thing that has a

different signature with slightly different behavior >> to me that's another route that this goes where people start right now I think most people that are getting into this, they can probably care less if it came from a library or the LLM generated it and they think about the world a little differently. The the LLM in their world is the runtime. This intermediate thing is just a temporary

thing. That's just how it works today. There is a world where you go back to maybe that small talk way of thinking. You're just in this inner loop the whole time and you're saying that this thing could be anything and you're just trying to limit the outcomes, the inputs and the outputs of it. And in that world, what is a library? What's a library? So that is

another way that this goes. There's a world that it goes that way and then people say, "All of this package management stuff you all did, we don't need it anymore." Could be a mistake. I think it would be a >> But that never prevented people from trying. Anyway, >> yeah, I think I saw I saw a Twitter post by someone recently who was like, "Why why do

I need package managers? I can just have, you know, the models are going to get better. They're going to get cheaper. maybe I'll run something locally and and it'll just generate everything it needs. >> What was your answer? >> Um I think my answer was that as as an ecosystem, right, there's there's a resource allocation question. It's like a physical question where if we can reuse, we

should reuse. Now, I don't know if we're going to change physics. I mean, we have some anthropic folks here. We might want to ask them, but so if we're changing physics, let us know because that would change the entire conversation. But if we're not, I think that I think reusing is going to be still create like be the baseline principle of how we think about things and

how we optimize and make them better. So I my personal opinion obviously I'm biased drinking from the Nyx Kool-Aid is Nyx is uniquely positioned to be that reuse platform, right? So if the models are generating the same piece of code five times over, that model can very easily use a Nyx framework to go and make that piece of code into your, you know, your mention of the

library and then I can reuse it. Maybe I can reuse it locally. Maybe I can, you know, the models become open source contributors as well. I mean, who who knows Ryan TM here? The biggest contributor to Nyx. Who knows Ryan TM? Is Ryan TM a robot or a human? It's a bot preai days. So, so I think uh I'm I'm totally with you on this and I

think Nyx is very uniquely positioned to >> people actually make Nick packages here. Anyone ever you maintain them? >> Do you make examples on how to use the packages? You make examples like >> only that person is saying yes. >> Who else is saying yes? Who else writes examples? Like I think three of y'all do. One of y'all are lying. So the thing about reusability and I

honestly believe this now if you first of all we have a discovered book problem. >> I don't know what half of these packages actually do. Like imagine someone new to computers like what does ffmpeg do? >> It does everything. What does Emacs do? Everything. But you can never tell from the name alone. So right you go to the next level. You got this man page. Like what

is a man page? And it's like all of these things except for how to use it. There's lots of reference docs except for how to use it. And then how not to use it. Good luck. Go find a blog post. >> There's a hand up. Well, we can't >> so just >> we can't even acknowledge that person. >> I'm so so someone from the crowd asked um

about some not well-known person statement of uh of being able to just skip the entire thing and the models are just going to just going to build and and deploy the executable. Yeah, honestly at this pace they probably should if we're being honest. Like you use a compiler. You skipped a lot of things you're supposed to be doing. Like when I ran go build for the first

time, that whole linker step, that pre-processor step, I don't know about that step. The assembly that comes out of it, you don't know about that assembly. Most people don't. And the most people that are qualified to even look at it, sometimes they don't even know what they're looking at. And I remember there was a commit from a guy named Russ Cox into the uh the assembly part

of Golang. They don't use lib C. They have some of their own implementations straight to the kernel like we're not going to depend on lib C for historical reasons. We want no part of that. And so they implemented some of these system calls themselves. And I remember looking at one of the files it was a bit of assembly. It's like this looks crazy but don't touch it

because it works better this way. Right? So all the people with skills be like oh no no this is not how you do that. like it is something something computers that's how they work. So most people I don't know if they even know how to operate at that level. So the thing that I'm seeing from this user experience of an LLM like when people interact with data

find the tallest person in my customer base and then it goes out and says it's this person. Now the Oracle people's like oh that's easy. Select from customers where decrypt PI data cross reference to the license database. Find the thing that's not called height in the column table. It's called person measurable abbreviated X. If you know that, then you know that that's the column and it's in

metric. >> And if you know that, then you know to use a conversion to make it interest. and you will get your answer and it's only three pages of SQL and you're like oh man like I I don't know if I want to learn that and so I get this new tool and it says you can just ask the question and somehow it returns the answer. So

think about software in practice. You can pick Golang today, you can pick Java today, you can pick Python today. There's lots of ways for you to try to articulate your intent and then have a runtime or compiler turn that intent into something the computer will actually do. and I don't want my Python to be translated to Pearl first. Doesn't make any sense. We can skip that. Go

straight to the thing that the computer will need or whatever Python needs to do its thing. So over time, either a why wouldn't we create either a dedicated programming language that works well for the new interface and we've done that over time, right? Remember when the web takes off, we get Ruby on Rails. When the cloud takes off, Golang comes out and finds its perfect fitting and

the standard library tends to match. I will not be surprised if someone somewhere is saying why do we need this intermediate representation if we don't expect people Not saying it's going to work. I'm just saying if we got Haskell more than likely we're going to get something that's going to be something more friendly to what agents do versus what >> I I was I was reading so

so this whole discussion unlike us sitting here last year which felt so different to me feels like 10 years ago. uh I was reading it kind of went back and I wanted to start reading about like initial literature around software. So I I started reading some thesises from like 9297 and one of the ways that one of these uh PhD candidates uh explain software in general and

I think it relates to potentially some of your your question topic was all software is is our attempt to do two things. one, communicate with a non-organic brain, right? It's just like we have this brick. It's a rock and it can do things and we want a better way of telling it the things that it can help us so it does the things that can help us.

And the second thing that he said that I think really struck with me around software is all software is is our attempt to make sure that some form of a similar input magic happens and some form of a similar output comes out of that. So, I know that, you know, when I turn the light switch, hopefully 98% of the time it turns on the lights and doesn't

like blow up my my mic microwave. Um, and I and I think that that is right the paradigm that we're looking at now because I think for most of the industry, they just want to be able to click a button and for that button to somewhat deterministically reproducibly provide the output that they're expecting. You know, you know, you know, a good example to think about is when

certain software functions move into the hardware, like remember when SSL was a very expensive thing to do and then it became something that was just embedded either in your network card or the CPU or some co-processor and over time as things get more efficient or we get async instead of general computing, you start to see that things are just worth doing that you collapse the abstractions down

and then those libraries are like, hey, we don't need to implement this in user space anymore. they're now in the hardware. They speed up. They're more predictable. Maybe there's more consistency across different platforms and operating systems that I think will continue. But there's also a question I think that's worth asking. So I always have to try to suspend my AI skepticism. If there was a better way

to write software, should we pursue it? That's still worth asking because I think sometimes we get stuck in what we know. This is how we've written software. This is how we're going to always write software. And someone somewhere also saying if there was a better way to do it, would you? Right? Do these things not just become functions in the hardware? Do they not become part of

some maybe more rigorous standard library? Why are we still trying to figure out how to log in a user? I can't believe the number of developers that are still trying to figure out how to log someone into a website and make sure that they only do specific things that that user can do. Why does that take up so much energy? That's a known thing at this point,

I believe. Maybe we can find a better method, but I don't think people should be struggling with SAML in 2035. >> So, by the way, feel free to raise your hand. I think you raise your hand. >> Audience questions. My favorite part. >> So, question to Kelsey is, you know, you mentioned AI skepticism a few times. Someone like you, where does that come from? U why >>

uh it's it's the motivations behind the >> right? Some people were trying to compare AI to like a destructive weapon. Well, ma a weapon of mass destruction is for one purpose to destroy things. I look at AI as a little different. You have a bit of agency and choice on how we get to use it. And so when I started my career, I was coming in 1999.

You could go to like a computer store, Cop USA, get a magazine, and there was like free software in the back of it. And I felt like, wow, just in time. I didn't have to learn Solaris. I didn't have to pay IBM for AIX license. Some of you are still paying IBM for AIX license. That's okay. Uh, but I I just got this software and I remember

the first distribution. It was like the Beasty logo from FreeBSD. I was like, "Oh, that is a cool logo." And then I just learned FreeBSD at home. And then there was a community of people that were just really excited about me learning FreeBSD. And I was like, that's the where I came from. So operating systems could have stayed proprietary. But then this community showed up and says,

hey, what if everyone had the ability to do this? Not just the raw tech, not just the raw skills, but even just teaching people how to do it. The skepticism around the AI piece for me is it's one of the first maybe crypto was there too. Well, crypto had a community. I was just not really a fan. But AI feels like we're skipping the community part. almost

if if people don't even care. For example, when I talk about the relationship or I see the relationship play out, I've been an open source maintainer, had a little project called Cop D and I remember waking up on Saturday and someone's like, I work at a company that makes billions of dollars and I would like you to implement this feature for free on the weekend. I'm like,

nah, that's that's not happening, right? I know the burden of that. And so to see someone say, "Hey, I got this tool that allows me to generate software real fast and all my only goal is to rise up to the leaderboard of top contributor. You need to review this like, hey man, I can't keep up." I didn't start this project to just add a bunch of features.

I started this project to reflect some of my own needs and then to share with the rest of the community. The fact that you can go fast is not the number one concern I have. And so when I watch people talk about this technology, all I'm hearing so far is productivity increases. Then I ask questions like, "So you spend more time with your family now?" Like, "No,

no, no, Kelsey, that's not Are they giving y'all raises?" "Nah, we're not getting raises either." I'm like, "So where's the productivity going?" And that's the part that I'm skeptical about. If someone said, "Hey, Kelsey, our goal with this AI is to make food free, right? we're going to create enough robots so food is now free. Not that we're anti- capitalist, it's that we want to take the

profits and spread them out across society. Then you would see me doing a thousand AI demos because I would feel that the community is aligned. But right now, what I'm seeing is we're going to make a robot that you can buy on a subscription service. So if you thought being able to roll down your windows in your car was cool, wait until we put a subscription behind

it. That's not exciting to me. So my skepticism isn't can cloud code really generate code. That's not that's not the most exciting thing to me. It's that do the people that control the biggest part of it. What are their motivations? Like I got a letter in the mail. Hey, you're part of a class action lawsuit. Nothing against anthropic, but hey, they took your Kubernetes up and running.

The book that you labored over, they just took it and they used it to train their model. And some answers you get from talking about Kubernetes comes from your work and they just took it. I'm like, really? You just you can just take it? Yeah. Yeah, you can just take it and they've agreed to pay you this much as a consequence. Like I wasn't I wasn't part

of the agreement. Like yeah, I know the lawyers were they'll take 80% of it >> and the rest of you 100 million, we'll split the other 20%. Fair deal with me. All you got to do is click here and if you don't clear and print, you'll never win anything anyway. So you might as well click. That's where my skepticism comes from. Now, I don't want to dismount

or disregard the open source work. So, I'm happy again. I'm seeing the open source people say, "Hey, we can have open weights. We can have open bias. We can have people that know what we're doing and we can choose to work on our own terms. We can choose to have this stuff run on your own machine. We can make our own idees." I'm like, "Yeah, we're back."

That's the world I want to see. So, I'm not going to throw in the towel here, but my skepticism is that I want to see them get funing, too. I want to see us use their models and I want to see them get stage time as well because I think that's going to be the thing that keeps the playing field even going forward. >> Yeah. So free.

>> Yeah. No, I think I think I wanted I wanted to go a bit more into Nick with U. Perfect. My question is more about margin. I'm having a difficult time over. >> The goal isn't to attack, it's to inspire, >> Those are different things. So, so just to so folks know the general question is that um I'll also provide some unneeded context on Fared. Um he's

kind of like our Trojan horse to a bunch of places. So we he kind of sent him to all these firms and he brings in nicks and you know and then I I I learned how he did it. Uh but he was asking about, you know, he's he's a new place, right? He's a new place. He's trying to bring Nicks in. Um Freda is a damn expert.

I would I would definitely hunt him down and ask him questions about bringing Nick into the workplace. Uh but but it feels like the conversation now because so much work is going into just how things operate right now, right? That it's kind of like at the 80%. And yeah, you don't have to put that much effort and get Nyx, but Nyx is only like 85 or 90.

U so it's almost like you have to tear things down to show them how much work is happening. And then without all that work, you can get Nyx for free, but without it, it feels like a marginal difference and folks are not convinced. Um, and and I think Fared is saying that he doesn't want to go out there and like tear things down or or be on

the offense against something else that exists. >> If a new medical treatment comes out and it gets better results for real people, >> then the number one thing you do is educate and prove it. And if it is better and someone says, "Nope, I rather stick with the one that doesn't work." That's that's tragic. But our job is to show and prove that it does work. So

when I think about like when Kubernetes came out, I remember someone was like we're going to kill MSOs. It's like and this was in public. I was like why? What is there to kill? We're inspired by the MSOS paper. We were inspired by those things. We looked at what they were able to achieve with distributed systems. They kind of opened up the door. We think there's a

better way of thinking about it thanks to these new tools underneath. So we we are taking distributed systems and making them friendly. And so when you go to the MSOS conference and you say, "Hey, I'm going to show you all the difference between MSOS and Kubernetes." Not to tear them down. MSOS is amazing for these reasons. You have the scalability, the two-part scheduler. Kubernetes has these things.

And the people in the MSUS community look at this and say, I see what you're doing there. We're going to take the CNI, the container network interface. We're going to bring that in. Maybe we can add Docker support over here. Now, maybe we don't believe that your API is right. We like ours, but you're right about these two things. we've been inspired by that. And then when

they come, don't say things like they stole that idea from like what is there to steal. You're you're giving. And so if I come to a company, they're struggling with Debian packages or maybe there's a Debian maintainer. You go to the dev conference and you say, "Hey, look, I work on the Nick stuff and I have this talk about a universal packaging layer. been spending a lot

of time in this space and the work that you do in Debian is similar to the work they do in Fedora, similar to the work they do in Gentu, similar to the work they do in all of these areas. And look, I'm not saying Nexus has to be the one. But I think there is a world where there is a universal agreed upon place where we put

all of our efforts and it looks like this. And if you really want to be um inspiring about it, you can use the Debian tool chain to probably create that initial lake and add features that are missing from the Debian. So now you show people the delta. If Debian had you would solve these kind of problems. If you had this, you can solve the sbomb problem. If

you had this, you can install two Debian packages at the same time on a singular system and they wouldn't conflict with each other. if you only had this and then people will see that small bit of delta and be like oh that makes a lot of sense and then you say I'm going to show you something I'm not going to name it yet but it has those

things and then they can decide does debian move closer to what Nick is doing or do you get a few people inspired enough to say what if Debbie were to pull from Nick's packages to create the distribution that's that inspire inspirational that's that doing the work and you're not it doesn't have to be crushing Now, if you're successful and it actually works, there may be a world

where the Debian package ecosystem is layered on top of nyx, that's progress based on inspiration. You don't need to go and talk bad about the community. You don't need to say the Debian people are dumb. They're wasting time. That's the attack mode. There's another way of doing it that can actually yield a similar result. So, that's the way I think about it. But, it's on you to

prove, but also you got to humble yourself. There might be things you can learn from those other packaging communities about how they make things simpler, right? I think the reason why Flux exists for a lot of people is because I remember trying Nyx for the first time. It was like, what do you know about quantum theory? I was like, oh, oh, I'm just trying to get a

new version of Postgress, man. Like, I didn't want to venture through a wormhole to get it. So, I think you've got to just make sure that we're all being humble here. world can we learn and just have that shared vision because Nyx is one path there there might be others. >> Yeah. And I'll I'll say a few things. So obviously I wear two hats all the time,

right? I I wear the Nixo Foundation hat and I'm I'm talking to very big and very small companies. And I think with with my role uh because recently we in the last year or so I think we had a lot of progress on governance. And I've pretty much converted my role instead of just like doing everything to just focusing a lot on on really getting the partnerships

in and getting nyx more integrated into these frontier leading ecosystems so that it becomes more of a standard, right? Because I think there's a there's a flywheel and I think Hashi Corp actually talked about this flywheel at some point back in the good old days and the flywheel is industry standardization. You want to get the folks that are showing the world how things are done because that's

how it works, right? always the 1% that everyone's looking up to. I think Kelsey, you're part of that. And and they're like, "Ah, okay. If they thought about it, like they probably thought about it enough and and I can probably just try it." And then you want to get team proliferation. Like you want to get the real teams hands on, right? You want to get like the

boots on the ground experience. And then you want to kind of circle it back so that that Nyx becomes more and more part of the general critical infrastructure. What if I told you that AWS was fun, you know, funding the S3 cache? Doesn't that make you feel a little bit safer? like someone at AWS probably a really smart person it's like yeah we need to fund that

because it matters because of reasons right but to get a little bit more tactical right so I'm also on the flock side we're working with with folks who who are for the first time stepping into the paradigms of mix and just to sneak behind the current and what I've seen in the last few month maybe it's helpful um on both instances is one there's in the larger

organizations and even in the small ones there's still a lot of that enchanted haunted forest around Nyx. Like all of us know that there is now like if you go back to the 2022 Nyx Con where we were 120 people in the basement of a Parisian university and it was had no windows, had one bathroom, there's, you know, I felt I feel like we all went through

boot camp together there. Uh there was maybe half a path that you needed a machete to get into the holy grail in the middle of the enchanted forest. And all of us here usually now know that there's like eight, right? Uh there's different products that can help you get into it. There's different flows. Nyx documentation has evolved. Nyx itself has evolved. Like there's a lot of things

happening, but there's still a lot of that feeling, especially with folks who have a tie-in into the technologies that they understand, right? So we have executives, we have folks who grew up on on containers, they grew up on the the the previous paradigm that they're used to. So especially in the larger orgs, I do feel a lot of times that there's like a subset of folks who

are like we're like let's not relearn something because this kind of already works to your point free, right? Um and I think we need to have empathy there. And the way that I've been able to kind of like draw the path or chart a course that has seemed to resonate the most and I would say the last three months um has been two two things I think.

Number one is the fact that this doesn't break anything, right? It doesn't. You don't believe me? Let me show you. Right. It doesn't. I can take the same machine and I can have my Nick set up and I can have whatever bash script Docker amalgamate of a Frankenstein monster we have running. and then it just works side by side, right? So, so a you're not breaking anything.

I'm not forcing you down a one-way door. And I I found that to be very powerful especially when you talk about u the second thing which is kind of finding the black swans of what the decision makers or the folks in the room really care about. And today what I've seen in in the second area is again two things. One is because I think that's top of

mind for a lot of people and I think Nyx is positioned incredibly well in anything that has to do with the questions around software supply chain and two is right I think Nyx again is uniquely positioned to help control guard rail and provide more of a solid base for whatever work we're doing with agents. So, so I think obviously those are mine, but I think from my

perspective, I found the don't worry, this is a two-sided door. You can walk back from it. No one's going to get hurt. And finding those black swans that the people care about around you have been more successful. But, um, >> I want there's a there's a human element to your question that I didn't answer. Kubernetes celebrated its 10th birthday a couple years ago and they had this

nice celebration in the Google headquarters. I'm retired by this time but I come back to MC this event and Solomon hikes there the founder of Docker and I can tell he was reluctant to be there. This is Kubernetes 10th birthday event and if you know the history of how this container thing happened, Docker is to me is very credited with getting a lot of people on board

this container train like a lot. Lex existed before it Linux C groups on the fringes but the thing we call containers today that's Docker and Docker was leading the entire industry including the hyperscalers and the incumbents. they were full steam ahead and you had to make a decision. Do you get on the Docker train or do you wait? And in many ways it felt like Docker was

just, you know, making this path that even the biggest companies were just trying to attach to. And then Kubernetes comes out and then Kubernetes is like, hey, Docker is cool, but it's missing a lot. And we put Docker below Kubernetes to the point where it disappears for most Kubernetes installs. I bet you Docker is not even installed. Container D, run C. It's been blown up. And most

people, if you looked at KubeCon, KubeCon is still 10, 15,000 people. There's no more Docker Cons really. And that 10th year anniversary, we're sitting on that stage. And I can see the hesitation. I was like, Solomon, I just want you to know it's your birthday, too, because there is no Kubernetes without Docker. doesn't happen. It's not on the radar. And I had to remind him is that

there's that phrase and I use it a lot. Uh some people have ideas and some ideas have people. Docker became so successful that it went from Solomon's idea and Docker Inc.'s idea to just the industry's idea. And once it belongs to the entire industry collectively, they decide where it goes next. And it happened to be Kubernetes. And so in many ways there are people in the Kubernetes

community that felt like they came and blew up that entire trajectory that they deserve to probably have a bigger part in than they have today. And so I can understand there's this human element of people who create things and trying to create better things without necessarily stepping on the toes of the people who've given you your foundation on which to grow on. And so I think that

real human element is there. But the fact that you're even asking the question, that's the important part. That's the empathy piece where you listen, you say, "Hey, I don't want to step on your toes. How can I present this in a way that it inspires and doesn't attack?" >> And I think and I know we're short on time. Kelsey and I are going to be around for

for the next few days. Uh but I think this also I think Fared's point kind of leads us into almost an ask, right? And it's an ass that I'm very excited that I have really good industry partners inside of the NYX ecosystem that are that are really bolstering and partnering up with us, the foundation to do this. But, you know, if you're using Nyx out there, if

you're a company, if you're part of a company, telling that story, sharing it, being willing to partner up with us, the foundation, the various things, like support contributors, fund contributors, and be public about it is what helps folks like that then come to work and be like, "Hey guys, you know, it's not just us. It's not just me, the crazy Nicks person telling you that you should

do it. Look at all of these folks, right? They're they're mega successful. They're billion-dollar firms. they're they're scaling and and they're in the news all the time, but underneath they're using nyx. So, I think it's almost an ask back to the ecosystem like if you're out there, talk about it, right? Celebrate it. And that just helps us kind of spin that flywheel around. Um so before before

we finish, um a you know, obviously super excited and happy to have you here with us and you taking the time bring the family down. Um, I did hear from like a little bird in blue skies that I don't know if this is true, but would this be the last year you're doing conferences? >> This is the last time I'm ever doing a presentation. So, I've evolved.

I have no interest in standing at a podium presenting technologies because that's just not me anymore. I have to lean into that. I don't want to show you how to use cloud code. Don't want to show you how to use Kubernetes. I'm 45 years old. I'm ready to move on. And to me, technology has taken as a different life form for me. It is just a means

to an end. But the end now is becoming more important to me than any individual path. So no, I am not excited to put together another live demo. If you want the old Kelsey, go watch the old videos. The evolved person is way more in tune to what we need to do going forward as a whole. Big picture. So the people that have been raising their hands

doing nicks for 10 plus years, that experience you have, the thing that made you excited to be committed for 10 years, you got to figure out, you don't have to do anything. There's a benefit to you figuring out how to recapture what made you excited 10 years ago and try to inspire the next generation. It is not a marathon. It's a relay race. And the best time

to pass a baton is when you're running full steam versus super tired and injured where it doesn't do anyone any good. So to me, that would be my call to action is that there is so much more to this game than whether Nicks is successful or not. I do think Knicks has a role to play, but no one's going to ever know that unless you find a

way to tell the story. Thanks for having me. And we so so we we have we have speaker gifts. They're really good local artisal chocolate. I don't know if AI was involved in that. Um, but I I did want to get you something personal from from me and flocks because it does sound like a Hey, hey, I'm I'm I'm pumped that we're one of the last conferences

that, you know, Kelsey's like like golden year is going to finish off with. But, uh, hopefully in retirement you might have some some more time. U, I did not wrap this. My wife did. Um, we we got you two I know you don't like taking stuff, but it's two Magic the Gathering commander decks. Can I tell you a story about Magic the Gathering really quick? I know

we're close out of time, but Mike, you were there. When I joined Puppet Labs, I moved from Atlanta, Georgia to Portland, Oregon, and I realized people trade duck eggs for tomatoes. Weird, but it's a thing. And I remember they they they started this thing called like a draft. It was the first time I've ever played Magic. I was too cool for school in middle school. Never learned.

But at Puppet Labs, I was like, "All right, I'm going to learn how to play Magic the Gathering." And so, we did the draft. I got my cards. I remember it was like a white weenie deck and I remember I'm just beating people and I end up in the finals and so like you don't even know how to play and you're in the finals and then I

became like a card shark. I would go to the comic book stores and like hey how do you play? Can I play? And I remember someone was like hey I'm not going to let you play until you put those cards in the sleeve. I like okay I'll put them there. And so this is >> sleeves. And so I've been playing Magic Gathering Arena probably ever since or

Magic Online ever since. Uh, so this is definitely something that I don't talk about a lot, but I am like hidden Magic card shark somewhere. >> He also me. How many people play Magic the Gathering? >> Okay, you you don't want to play Kelsey. He has the most horrendous deck to play against, but you kind of do. So, anyways, you know, maybe you can get your daughter

back into it. There's two decks. It's pre-made decks. There's a box to put them in, sleeves to put them in. uh you know and and thank you for everything that you've done for the >> All right. Uh cool. Good to go. >> All right. Uh we have a break until the top of the hour. So we got about about 12 minutes. So the restrooms are over there.

Uh, and we'll see you either back here or in room 103 depending on which talk you'd like to attend. We have a a flock workshop over there. I don't remember what we have over here. I probably should. No one cares. Hello. Hello friends. How's it going? We are about to get started. So, if you could find your seats, get comfortable, maybe grab a coffee, whatever you need

to get ready. We will be beginning our next session again in just a couple So, I guess while everybody's sitting down, how's it going so far? Are we enjoying the sunny California weather? Yeah, I you're my favorite right now. I love the enthusiasm. I'm like, let's do So, I think I mentioned this earlier, but if you're willing to do a video interview with me, you get one

of these. Now, I have them for demo. It's a fun little custom cereal box. It has chocolate and candy in it. Um, if you've got any questions, we've got the agenda up here. You can find some of us around the floor in purple shirts to ask any of And we're just going to take another couple seconds for people to settle in. Sit down. Who's ready? Yeah, I

like the hands. Hello friends. Thank you for paying attention and quieting down and finding your seats. I appreciate it. We are going to get started. Um, I'd love to introduce you to our wonderful friend Pria from Microsoft who's going to come on and give the talk. We've got some technical demos for you. It's going to be fun and exciting. Are we ready? Can we give a warm

hand of welcome for Pria? Can you hear me? Yeah, I can hear myself really loud. Yeah, So, I'll just mirror my display. Okay, it's already mirrored. Let's see >> because of this. Yeah. Right. Right. >> Okay. >> Everything good? >> Yeah. Can folks still hear me? I think Yeah, I think this is good. Okay. Hey everyone. Um I'm Priya Anuntas Shankr. I'm a principal software engineer at

Microsoft. Um today I'm going to talk about how we are going to do runtime package layering um and how that breaks the bloat cycle of your container. Um this is mostly a talk on nyx packages uh containers and from a cloud infrastructure engineers point of view. Uh I don't claim to be a Nyx expert. I'm just learning. Um yeah. So so let's get to it. Yeah. The

views and opinions provided here are my own. Um they do not reflect those of my employer. Okay. So I want to talk about the parable of the managed console. And this is if you're a cloud engineer or cloud operator, you have some environments that you manage and you you I'm not talking about environments that you give customers to manage. I'm talking about environments that uh you know

you you manage your platform and typically um you're building a manage console for your cloud platform and something like a dev environment that you want to give everybody and let's say uh you you yeah you have a docker container or a container system there where you put all your tools um and you start with two tools I'm just going to give bash and and and some CLI,

right? And then you ship it. Customers love it. It's fast. The image is about 400 megs and users love it, right? And then your requests start coming in. Um, someone wants three versions of Python and then there's like a whole bunch of DevOps tools that people want and then at soon and and you know, and Helm and Terraform and whatnot, right? and then you add them because

that's the right thing to do. um you distribute it, the image is now 4 gigs. Build your build time grows. That's fine. Your CI time is fine, but then you're starting to accumulate these managed console uh tools tool sets that you want to give people. And I and I totally agree with with Kelsey's previous talk where he said that people just want tools. Um they're not looking

deeply into package manager or packaging. Um I think that's true in some ways where people are like give me an environment I I just want I just want these things in my environment just just spin one up for me right and so far we have dev containers as a solution and I just want to tease that is it is it sustainable right when you keep bloating it

this way so let me move this mic microphone down okay so and you have a docker container with all the tools and then you do one docker build and all tools upgrade, right? So there's package 1.12.13. Let's say there's like a DevOps package. There's two two use two sets of partners and it's all fine and then you upgrade it with say a Docker build. You roll it

out and and it breaks for one person and the other is is fine. And now you have this a bloated docker image and then slowly your container is one sizefits all. So how can it evolve further with it being this monolithic right? So um then a list of solutions come up right let's break up this big docker image into three different ones. Let's let's give people a

devops image. Let's keep pe give people a security image. So let's ask partners what kind of image you want and then you have a data image a security image. You try to meet in the middle and a devops persona a data each with versioned variants. Now now instead of one you'll have to maintain n times m versions right. So a CVE a vulnerability in curl means you

go patch everybody and it's just your ops has doubled honestly. So if you say if you look at these these different versions there's drifts between packages. So you have u you know you you first have drift on of the environment itself and then you have overlap of packages. What if your container was never a right unit for the environment? I know it's it's excellent at process boundary.

It's excellent at isolation, file system namespace, network name space, but somewhere I think it evolved. We whoever's you know maintaining the containers kind of made it coupled it with package manager package version manager or a team sharable environment spec. It's been used for these purposes and my argument is your substrate is not equal environment. So substrate is the thing underneath that everything else depends on. So Kubernetes

for example, Kubernetes is a substrate for our workloads. You could you could say that. So the substrate is your OS, your lib, your base tools. You can have your container as your substrate. I let's define the container as the substrate. It should be owned by your ops team or your platform team. U a docker image is a great unit for the substrate. It's coarse grained. It's stable

which is what you want. It is distributable. But then the environment it's the this is your tools, your startup scripts, your dependencies. It changes frequently. It should be owned by users. It needs to be fine grain. It should be fast to swap. Should be independently versioned. And then we can tell people just bring your environment into my substrate. we can we can manage our ops really well

here. I think nyx closures um are the right unit for the environment and the key insight is they can be decoupled from the image entirely injected at runtime and not baked in at build time. That's that's a little hard like injecting stuff at runtime. But then it frees the people who manage the substrate from breaking anything and you know yeah removes it gives them the flexibility to

upgrade right and maintain things maintain the substrate without fear right so yeah so that's that's the start argument and also in the whole DevOps life cycle you have a inner loop and an outer loop right and you have your package managers and your docker build and you're pushed to the registry which is in the inner loop and then I make a case where I can say nyx

closures can give isolation in the outer loop without disturbing the integrity of Okay. So before I get to this um I want to get to like a demo. Let's see. So in in my demo I do have uh let's see where is Wow. Okay. So I have uh let me open my VS code and and show you uh a docker compost file, right? And I do have

a volume mount that mounts a whole bunch of packages, Nyx packages that I want. And from that volume mount, I just mount the binary cache into my workers. Yes. okay. So, thanks for letting me know. now let me just paste it and it Yeah. Um, I'm just trying to increase the font size and get put it in the terminal maybe. >> Right. Okay. >> Uh, is this

is this good? Um, let me just say docker compose.yaml >> right. Thank you. Um, so here it's just a volume mount with my Nyx packages with with your Nyx channels and I have a bunch of packages over here. And I do bunch a worker image which is just a basic image and that's my substrate. And then I I just have a demonstration as to how you can

bring your environment inside the substrate. anyway, I think I need to increase the font size for everything. Um, see, is that readable? No. So, I'm going to I'm going to start it from here. Uh, let's let's just go. I'm I am just going to populate my nyx right and it's populating all of my tools or nyx packages that I need. I I kind of needed um AKS

MCP server and some other tools like Hubble and Helm and and Claude. um all of it is populated in in the and now I'm going to run my workers right um dash dash scale So if you look at this you have your sub you have your substrate which is your which is basically it just has your nickos you can take any image not not necessarily nick service

it's just your substrate but then you brought in all your tools via the volume mount so if I do a new window and thankfully it's it's and then if it's a docker Yes. Right. And let's go inspect where is this name next worker? Yeah. Three any of these workers you will see it's got clawed right. So I didn't go and install it. I didn't have to maintain

it. I used a NYX closure that brought in these libraries inside my substrate and it didn't really disturb the integrity of my substrate. that's that's one of the things with Nyx. But then but then there's a lot of um if you if I if I just look at as I said before I I'm just a very new um uh NYX practitioner I would say and I was

trying to do gen switching nyx profiles like removing the profile like installing a profile rolling back a profile and you do need lots of familiarity with with the CLI and I think and I think that that's with that's with a lot of um learning and investment you could do that and and that's my um you know I was just trying to see if there's a sim link

that I can swap out and it's all runtime closures and so you can easily swap out versions when you release new versions you can update a store path you can invalidate a store path all of which can be done without going through your whole DevOps cycle and and crossing that whole um timeline right for your team and also ops for your team. let's go to flocks. Now

flocks kind of simplifies this for me because I don't it it's also modeled a little bit like git in the command line. So you're familiar with git. So you say get in it, git pull or you know and you have flocks in it, flocks pull. I think that was a smart thing to do because that's that's kind of instantly puts developers a little on the familiar ground.

So I have and thanks to thanks to the flock folks for for the last minute container stuff. So this Yeah. Oh, okay. So this is really small. So this is a container um which is which has a flock environment which is the AKS MCP environment. So I created a flocks environment with AKSMCP server of which I am the maintainer of and I quickly went and checked once

KLC said you know did you write examples for AKS MCP serve I didn't so so yeah so that's my AKS MCP server um and clawed code and I have them together and I did not install them in my substrate which is my and and the docker container that I'm running right now is let me show you that container size right it is um the shell container and

it is about 13 1/2 gigs of just tools and and packages and with more AI tools and MCP servers and LLMs the size is just going to go up it's it's yeah it's not going to be sustainable with this bloat. So this flocks where it helps is brings your um environment inside this substrate where you can say flocks list and I have my AKS MCP server and

clot code here. And if I exit that's my substrate I don't I don't have it. And we can also use this to deliver fresher versions of packages. Right? So if you go to user bin, which is a shared space for storing packages, right? Um, we could make an argument and say, "Hey, I could just go plop like a bunch of packages through file mount under user bin,

but that would destroy sometimes the integrity of the container because it's not isolated as a closure like how Nyx is. So here's the ny store that flocks used for from where your AKS MCP and I know this is not too readable but that's the part that I think flocks um abstracts. So yeah, so that's that's one of the things I wanted to show. And also I can

take this environment, right? Um I'm activating the flocks environment over here. Okay, sorry. I don't know where that environment is. let's see. It's probably in here or it's probably in my home. Um there's a flock folder. If not, that's fine. I can always pull my environment um my environment and I can pull it. And so whenever you pull an environment, there's always like a rebuild that's triggered.

And that's that's something in a cloud environment, in a cloud infrastructure, that's a latency. That's like a runtime latency, right? So, uh that's something we got to grapple with. So how do you do like again going back to this to this presentation? I think here if you do not own your host and if all your containers are going to come up and die then I think you

need a sidecar that kind of puts your binary cache with the container like so when you init the whole sidecar init with your binary cache along with your container so all of the Nyx binary packages are available on on your sidecar and your container is just able to activate environments from that sidecar. this kind of expensive and it's kind of an open question with ephemeral um consoles

right like consoles that come up and die and we do this all the time in cloud right so yeah so thanks for that um if you own the host this is like a typical Kubernetes scenario where you own the node then you can put your binary cache in the node and then you could have the substrate and the parts pull from it. Um, yeah. So, so in

closing, the environment is a closure and the container is a process boundary. And that's the case I'm making where you have a minimal base image um stable. It's owned by ops and you can rarely rebuilt and then you have personas as flocks and mindments or even nyx profiles and then you can democratize it and you can tell people bring your own environment bring your own manifest and

you could just load it up so that you don't have to carry all these things. Yeah. So I think I'm at time and here are a couple of links. Uh thanks everyone and and any questions. Hey. Yeah. Thanks gig image. >> In that example that you showed of the 13 gig image, how much did it come down to? Um I didn't quite understand >> how you went

from 13 gigs to just two packages. Is that how it work? >> Uh no the 13 gig image is is the result of building packages over time. So if we yank off packages and move it down to the environment, it's going to probably depends on what you want to define as a substrate. If you want to say I want to give my cloud CLI and a whole

bunch of things like DevOps tools as a basis I want to give cubectl as a as my substrate so that I you know I keep maintaining that but the rest of the stuff like AI tools or whoever wants anything new I'm just going to bring in via the environment then I think your size is is much lesser and you have a sustainable docker image that you patch

Um yeah, so substrate could be whatever you define and not let it go to this extent where you keep adding packages and it just turns into a monolithic big >> Yes, thanks for the talk. uh where do the next packages come from and what I mean is if I currently have access just to a container registry and I'm either airgapped or don't want to add the network

dependency how can I leverage this workflow >> sorry your your question is why are we not depending on the ACR >> um I'm asking >> sorry container registry sorry >> yes so in can I still use just the container registry for storage and transfer do I need any other new element in the network in order to host the packages for container registry. Yes. Um, are you asking

me if I can store the binary cache as a first class artifact OCI artifact in a container registry? The next binary cache. You could um that's again the ops, right? It just blows up your ops again. It's the same thing like you got to maintain that that um and and keep publishing it just like how you publish docker images. Yeah. Yeah. I know it's a little when

when I say bring stuff in runtime it's a little hard but then the other resilient way to do that is to storage. So use storage and do a fuse mount and bring your binary caches in. I think we have time for one more question if someone want >> Sorry for breathing into the mic like a >> I know I see you fixing it. It keeps moving anyway.

It's fine. >> Does anyone else? >> Anyone else? >> Last question. No, I think you've done a really good job answering them. It looks like we are all good. >> Thank you. Thank you everyone. I'll catch you later. >> All right. And I think we just have a couple minutes between now and our next session. So great time if you want to change rooms, get a drink.

Yeah. We'll be right back. All right. All right. Hello friends and welcome. We are going to continue track one. Um we're having a great time here at Planet Nix today. And yeah, so if you could just get settled in, quiet down and we're going to get the next session started. >> Awesome. Thanks, guys. This is Wes and he's going to be talking to you about Nyx. Anywhere

else? >> Hello gang. Wonderful to see so many Nyx nerds out here today. Um, so just to start things off, I just want to start with um maybe an error some of us have seen before. Nixos, it's great for running software. Uh, it's not always great at running random pieces of software that you might found on on the internet. So I I at least um first started

running into things and seeing cannot execute required file not found. What file wasn't found? Don't know. Doesn't say. Uh actually Nyx these days makes it a little easier for you. Um but this and sort of the discovery um the the looking under the hood uh that this drove me to is kind of the focus of the talk today. You're going to hear a lot and you already

have from folks who are using Nyx at work and doing all kinds of crazy stuff with AI or stuff in all kinds of embedded environments, all sorts of stuff. That's not this talk. This talk is what you do at home after hours when you want to just dig deeper. You're not trying to get something deployed right now. you just want to make sure that you actually understand

um what's going on under the hood. So this talk is nyx anywhere else relocatable binaries via elf surgery. Uh I'm Wes Payne. You might know me from the podcast Linux unplugged uh with me and the Jupiter Broadcasting people every Sunday. We're live. Go to jupitterrocasting.com if you want more of that. Uh, I'm also a software engineer doing Closure, functional programming, distributed systems, and now breaking my open

claw seemingly every other day. So, come talk to me about that or jazz or quantum physics if you like. There's all kinds of fun stuff. And of course, Nyx. So, to dive in, um, you might know this pattern. You try to run some random app, uh, error while loading shared libraries, lib, standard C++.So, So, and then some random number at the end that probably doesn't mean anything

or you hope that it doesn't mean anything. Uh, or maybe you're trying to run a Python project, right? You're you're finally trying out Nexos, but you have a Python project at work or something you found to integrate with a new API, and well, you try to use a standard Python library, something like NumPy or Scypi or anything that's kind of popular that isn't just a pure Python

package, and what happens? You get another obscure import error. libz.so cannot open shared object file. No such file or directory. uh or you want to go try to run some pre-built tool, it doesn't always work. Now, there are really good patterns and answers to this. And the usual answers involve something like uh Nyx-LD or NVFS or you combine those things. And I'd like to be clear, do

these things first. Don't do what I'm trying to do to solve actual problems. Uh people have already much smarter NYX contributors already solved a lot of these problems. And if it's just kind of like dealing with can I run this binary not meant for Nixos that doesn't have a bunch of complicated dependencies, Nyxld, Nvs, they'll they'll solve it for you before you have to do anything manual.

Um, but it doesn't necessarily teach you why the existing stuff without them doesn't work uh or how you can make it work or if these tools didn't work, how you would solve it for yourself. Um, so first things, ELF, it's executable and linkable format. Um, we're not going to dive in. There's a bunch of folks, maybe go talk to Ilco if you can find him today, uh,

that know a lot more about ELF than I do or we need for this talk. But when you have a Linux binary, and we'll focus on Linux because, you know, Nexos. Um, when you have a Linux binary, of course, there's actually the assembly, right? The compiled instructions, the stuff that your CPU is going to somehow magically execute for you. But because computers, there's also metadata, right? There's

always metadata. Sometimes there's more metadata than actual data. Uh so there's a lot of metadata in ELF. We do not need all of that metadata for today's talk. Uh and we don't really need to think about the bytes on disk, but what we do need to think about is it's a place to set some key key value pairs that help Linux actually be able to run your

binaries. Um so first things, right, we're talking about dynamic binaries here. You can't have a static binary where the compiler and the linker help you out at a static pre-chosen time right at the start. Bundle everything together. But if you've run Linux anytime in the past 20 years, you know that if you're not using something like Alpine and Mucil or maybe some modern Rust and Go stuff

is often harder than it might seem. And we live in a world, whether we like it or not, of a whole bunch of dynamic libraries that get loaded when your program executes. Not when it gets compiled, but when it executes. And on traditional Linux distros, that often means it goes hunting under SL userlib for whatever is going to find there. Now, of course, we know on Nyx,

we do things a little bit better, right? We have hermetic packages and we have custom store paths that tell us just the right library. Um, but whether you're on Nyx or you're on a bunchu, the kernel doesn't know about any of that stuff, right? It needs to know how it's actually going to execute this program that wasn't really fully compiled. So, the first little bit is the

interpreter. Who loads me, right? There needs to be some kind of file that's going to run things. It's going to know how to execute. And crucially, it's going to know how to pull in all those dynamic loaders. The colonel doesn't want anything to do with that, right? It just it just wants to run a binary. So something else usually from lib C needs to get involved to

go pull in all those extra dependencies. But where do you find them? So first we have our interpreter. That's what we were just talking about. Next up we have the R path. And this is just answering the question like what folder do I even try to look for to go find all the the libraries that I need for my program. That's the R path. Uh and then

after that you have what actual dependencies you need. So in this case we have an example with lib Z and SSL and some standard stuff from C++. So those are all metadata that get attached to all of your executables or at least most of them that you're running. and you have some handy tools to read them. Uh you might check out stuff like read elf, which can

be very handy, although it prints out a whole bunch of stuff. But if you do some grepping or get it as a JSON data structure, you can see it'll tell you like, oh yeah, I expect to load an interpreter under /lib64, at least if you're not on Nixos. Um you can also use LD, which will help you do the same thing. And all of these are just

great ways to start exploring, but they're kind of read only, but you you will start running into them. and we ran into them just messing around last night before the conference. In fact, um we've been at Jupiter Broadcast at Jupiter Broadcasting, we've been playing around with some uh some of these new uh open source agents like Open Claw and associated projects. And if you've ever used Google

Workspace, you probably know that configuring it and getting it set up and trying to use it is maybe one of the most painful things known to mankind. Uh it turns out just recently Google trying to help folks out has finally released a new workspace CLI and not trying to represent Google use them, don't use them, whatever. Uh but it was a tool that would finally let us

automate some more of the stuff we do behind the scenes anyway and we wanted to try it, but of course we all run Nixos and so we go down to we go to the GitHub page, we download the you know pre-ompiled it's it's written in Rust. It's pre-ompiled. It's ready to go. And uh what happens if you try to run that on Nix OS? Well, it's not

going to work. I mean, if you use Nyx LD, etc., it will. But if you just try to run it on Nyx, it's not going to work. Uh now, of course, this was pretty great is it was 9:00 p.m. last night. We should have been getting ready for bed, but we were messing around and I was trying to wrap it with a little flake so it would

work on Nixos. And as we were doing that, it turns out Google was merging a PR at 9:00 p.m. to add its own flake.Nix to the repo. So, I didn't really need to do that. They're doing it upstream, which seems perfect for Planet Nix and is awesome and we need that really everywhere that we can. Um, but it it kind of made a good little um thought

for me is okay, that's what you should do. Go use the upstream stuff, right? They're going to have it. But what is that? How is that shipped? Right now, they don't have their own cache for it. And that means you're going to pull down like two gigs of cargo dependencies just to build the binary that ends up being, you know, 10 megs on your file system. And

if you want trust and you want to do it right and you want to be secure, that is what you should do. But a lot of us, you might just want to be able to use the gosh darn binary that's on the GitHub page so you can try it out and then decide if you really want to commit to it. And that's what using and playing with

tools like patchel can let you do. Uh here's just a little comparison to show a little bit more of what you might see on Nixos or a standard DRO. Um, so if you do go download a random binary, um, on a standard DRO, maybe something like Abuntu, it's going to be able to figure things out. It's going to go take a look and say, "Oh, I need

to find lib GCC. I need to find lib M. I need to find lib C." And it's already set because most distros put these in the same places to go find it under the magical every shove all of your libraries in one place and just hope for the best strategy that we've been using for like 20 years. Well, probably longer. And on Nix OS by default um

you just get a whole lot of And that's where we have a new friend and that that friend is patchf. We had read elf, we had lddd. These are great for spelunking, figure things out, taking a peek under the hood. But I mean, come on, we're hackers, right? We want we want to break stuff. And that is where patch elf comes in. It's been part of uh

you know, it's been a helper around for Nick since I guess 2004, a long time. uh is kind of instrumental to a lot of what Nyx ends up doing and it lets you actually just muck with that metadata. Right? If you can think about it as a complicated set of bytes on disk, but if you just think about it as key value stores in a dictionary, patch

lets you go set some of those fields and not on the next binary that you're going to make on the real binary that you have right now. Uh so you can do stuff like set the interpreter. You can do stuff like change where it's going to go look for libraries. Uh, and you can even, you know, add extra stuff. You want to say like, oh, well, the

person who made this didn't realize, but it actually needs more libraries than it has, you can do There's also in Nyx packages, which we'll make use of. Um, the great auto patch elf hook, which can take care of a lot of this for you. It's smart enough to go look in whatever your Nyx environment is. It goes and sees all the libraries that might need to apply

and it can automatically sort of fix things up for you. So, it's a great little hack. You don't necessarily need to use it for everything we're doing today, but uh if you're doing anything actual, you surely would. Okay, so here's just a little before and after of what you can do with patch elf. Before not found, and now autopatch elf and patch elf can fix all those

things up for you so that they know where all the all the arcane paths in the nick store and it knows actually the right library to find and it won't just blow up on you. And and if you do it in this way where you let autopatchelf do it and you do it maybe as part of a flake in a proper Nyx package, then Nyx can track

that too. The garbage collector won't, you know, steal them out from under you. It all just works. this comes in very handy if you've ever tried to make Python work on Nyx OS. And like don't get me wrong, right? Nyx packages, lots of great Python stuff. Tons of things are packaged. You can just run it. It's really one of the best ways to try and ship Python

dependencies. But that's kind of for projects that you know maybe they're well packaged by a Nyx packages contributor or it's something you have full control over. So you can kind of choose like I'm going to make sure I use the dependencies that are already in Nyx packages. But for me coming from like a background of doing uh application development for SAS companies running websites, we weren't trying

to build real Python packages. It all ran in Docker anyway, right? Like I was just trying to get Python that had all the libraries that I needed in a Nyx environment, ideally with a dev shell that could work. I didn't need to muck around with all of the actual Python native packaging stuff. Uh I had Nyx for that. Um and Python packaging itself has improved. There's UV

on the scene these days. That's getting better. It's a lot faster. So it's pretty easy. You know, Nyx has a lot of Python versions. So you can't just do something like, you know, pip install numpy. Except if you do that, it doesn't work because of course like Python packaging, it solves packaging for Python. It doesn't solve packaging for all of the standard labs and the forran libraries

and the rust stuff and the C libraries that it turns out Python uses to get most of any of its stuff it needs to do fast done. Nyx can help with that. Now what you probably do and what most stuff does and what you'll see in upstream Nyx packages is you essentially set LD library path and that helps the binary run and tells the dynamic linker, hey,

I know it says one thing here or you don't have any info. Let me help you out. Here's an environment variable. It's a little crib sheet. Take that uh go look there. You'll figure it out. And that totally works. You can use the neat uh wrappers make wrapper in Nyx packages library. It'll do that for you. So every time the binary runs, but if you don't want

to do that, especially if maybe that's not especially dynamic, you're in control of the project. You only, you know, I don't know about you, but I don't add a new native like C library dependency every other day when I'm working on a Python project. So it's it's probably like a core set of stuff that you might actually need. And there's another way. Instead of patching the actual

patient instead of doing operations on every single little randoms file that you need or making your own custom libraries, sometimes what I've started doing is I just use patch elf to tell Nyx, I want those extra libraries all the time. I don't want to wait till till you actually try to run it to figure it out. What you can do is you can use patch elf and

you just tell it, hey, I know you don't think you need libzy. I know you don't think you need a bunch of stuff from C++, but it turns out you actually do. And um when you do that, autopatch elf takes care of the rest. So I can just say I don't even need to be super specific about where to find it in the nick store, all the

path stuff. I just say, hey, add libzy, add lib standard C++. And then when this builds, patch will go find the right paths, go use it. Autopatch elf takes care of hooking it up and you end up with a binary that actually works. And what's great about this is you can just have this one Python interpreter that you ship around, Nyx is going to make sure anywhere

you run it and build it, those C libraries come with it and they're already injected. And the moment it tries to actually go pull in NumPy, well, the the linker is already happy. It's already seen at startup. It already loaded that C library. It doesn't have to go resolve all of that every time. Oh yeah, we can uh see there native build inputs. I'm adding auto patch

elf hook. That's just so that it auto runs during the build. Uh build inputs is where you kind of specify the exact libraries you want in the environment pulling it from Nick's packages. And then down here at the bottom is where we actually go kind of add everything. And uh it actually works. So if you go and run it, I have a little dev shell um that

we'll have links to at the end that you can run and it uses uh UV to just go install stuff really quick. It tries to make it work. Uh and it never works every time. Reproducible failure. Everyone loves it. Um but if you do go patch the interpreter, what happens? Success. Uh it just works. It can find it. Um same numpy, same virtual environment, same UV, same

everything except a little bit of patch shelf. And this is handy. This is really nice. I've been using this for a lot of my Python stuff where I do need these things. So, I just don't have to worry about wrappers and setting environment variables, but it's kind of only useful for surgery in it helps us take stuff that isn't meant for Nixxos and bring it to Nixos,

which is really great. Um, pre-built binaries, pip c extensions, stuff you find on GitHub, the latest Ruster Go project. Um, but I wanted to see what about going the other way, you know, like what about surgery out? Um, because patch shelf doesn't really care, right? It's it's just setting text files. It's setting keys and values. You have a lot of flexibility. And an opportunity came up. I

don't know if any of you guys are are file system folks, but I like playing with copyright file systems. And in the Linux world, uh, we've had Bcashfs, which has been a new fun development in this space. Um, unfortunately it hasn't always been easy to use or to try. Now on Nixos, the support has been great almost since day one. I'm running on this laptop. I run

it for some reason on my router at home. Uh, but and this is no shade against Debian. Uh, but it had been kind of a struggle to get it on Debian for a while. And part of that is just, you know, Nyx is happy to kind of meet software where it's at, right? we have a lot of like ways to make it safe and reproducible and dependable

that maybe some of the older package systems don't really have. And so Debian kind of having a debate and a lot of Linux projects are around, you know, most of these modern software pieces, they're kind of developed with lock files in mind and they're not trying to have like one version of a library that's kept in the package manager that every Python app needs to use, right?

Like a lot of stuff, especially Rust projects, Go projects, they expect to have their own specific lock file per project, not per distribution. And a lot of dros are kind of struggling with that, but Nyx packages totally fine. Just build it. We have great helpers for cargo, etc. Um, so I thought like, okay, you're never going to get this accepted in upstream Debian packaging it this way,

but like what if I have some like Abuntu or Debian systems I would like to run bcash fs on? I don't want to have to try to build this in a build system I'm not very good at. But what do I know at least semi-well how to build stuff with Nyx. So uh I tried making myself a Bcash FS tools portable and part of that is just

kind of doing a similar thing with Nyx which is hey I know I need a whole bunch of libraries Nyx you have everything why don't you provide them. So that's here. I've got some paths here. I needs fuse, gip, lip, sodium, whole bunch of stuff that Ken Over Street set up. But after that, I mean, once we get the stuff we need, it comes time for surgery.

So first of all, what do we do? We build the actual package. I want BcashFs tools. That's all the userland helpers that let you make a Bcashfs file system, basically do anything useful with it. And it's packaged in Nyx. So, first thing you just actually build that package. Um, and then I have a couple helpers here and some bash scripts which will be linked that kind of

go and trace through and find all the sim links and resolve them because often there can be several different layers of sim links in the next tree before you actually go find like the actual file on disk. Um, so once you do that then essentially you can go find all of the dependencies because Nyx tracks it already. So you build the project, Nyx tracked all the dependencies

that gives you all of the dependencies. Um, so next up, we know that Ubuntu has its own or Debian has its own linker. It has its own setup. That's not going to work for us. So we use patch elf and we say, um, actually we're just going to ship our own linker. We're just going to bring our own dynamic linker with us. We'll take the Nixos one,

bring it with us. No problem. And then we do a nice fun little trick, which is you have that R path and if you remember that's saying that's saying where do I go look for libraries? And it turns out there's a fun magic value called origin. And origin gets set to wherever that binary is running from when it's launched. It's basically like a little way to point

to like where am I starting from? Um, and you can set that as your R path. So this just tells us essentially, hey, we're going to bundle all of our libraries with us and keep them together. In this case, I'm using /user/local just as a way to avoid like colliding with anything that would be on a Debian system already. Um, so we're going to stuff our stuff

under user local nicks and then we have we'll have some some libraries and some binaries. Um, so we set our interpreter to our custom one we're bringing with us. We set our r path to go look right where our binary is, which is going to start bringing its own libraries. And then we go do some some more patching to finish the surgery up, which is just to

make sure that everything that we might need knows that we are going to be local. We don't want anything looking out of the store. We don't want it trying to hunt around on the Abuntu system where it's not going to find anything valuable. Uh so we kind of go through and any of the libs that we are bringing with us, we patch as well. Um you'll see

patch elf down at the bottom just to say hey use origin. And what do you get at the end? You get a bcash FS tools that can actually find the stuff it needs. And so here you'll see we'll do an LDD to go take a look at what binaries it might want. Uh you'll see the the normal sort of stuff, but in particular uh we no longer

have Nyx paths. If you go look at this on a bcash of ss from nyx, those are all going to be store paths. Ours started out that way, but now they look a lot more like paths you might see on a normal Linux system, right? User local Nix lib. Uh, pretty standard. I dump a lot of stuff under there all the time. What's fun is Nyx built

it. I didn't have to figure out how to package it. Nyx already knew how to package it. And Patchel is what made it portable. And then on the side, uh, I have an NFPM package that can turn it into a deb file. You again, you would never do that. Debian maintainers will hate you. Don't try to put it into Debian like that. But like for my own

systems, I don't need a like the right Debian metadata. Uh I just need the right files. Um so once you do this, you get yourself a bunch of files on disk. And NFPM, which turns out is packaged in Nyx, uh can turn that into a DEP for you. And so if we just pop over here. How's that? Better. >> Okay. And if we It should just go

real fast hopefully because it's already cached. But you can kind of see the script running a little bit. It does some clean up. It builds the binary. It builds the library. And then ultimately uses it builds a package. And we've created ourselves a janky little dev file. So if we pop back let's see here. Ah, right. Okay. So this uh I'm just spinning up. There we go.

Yeah. Okay. So if we just spin ourselves up a quick little Podman container running Abuntu. Uh I've mounted in this exact folder under slash app. So if we go to slapp, we'll see that we got exactly what we just built. We have ourselves our deb. And so I can do dackage- i. I have an actual working bcash of s. And I can even if we go pop

over somewhere that might work. Um we can make ourselves a little oops raw. Little file yeah, and now we can actually format a Bcashfs file system on Abuntu entirely built from Nyx. So uh just to kind of emphasize what happened, it was the same tool. All these all these binaries have the metadata. Anyway, we were blessed um from ILCO and others to come up with all of

this crazy tooling to help NYX work that it turns out can do more than just make one of the best Linux distributions possible, right? It can do surgery in to bring whatever tools you need to Nyx, even if it's not maybe the right way to do it. And it turns out it can be used for surgery out to take all of the wonders we have in the

Nyx ecosystem, make them portable, and bring them with you wherever you are. Patch Elf is really it's steady metadata. It's writing some strings and um once you see that you might start seeing it everywhere. So um where to go from here? There's a QR code. Uh I need to flip that public I think. So I'll do that right after the talk. But uh there's a bunch of

repos here with the things I've been playing with. And there'll be a repo uh for the talk as well which um has a flake to build the slides and run it. Um and of course it looks like we're coming up on lunch. So if you do have questions, feel free to come find me. Uh happy to talk, happy to work through any of the code. And um

thanks for coming. I hope everyone has a great rest of the planet, Nicks, and happy lunch. >> One logistics note, earlier when Ron was saying we have lunch, I wanted to be clear. We have lunch in the agenda. I am not bringing you food. Um which I admit not the way I wish he would have worded that. Uh but I'll deal with it. So you are at

lunch break for the next 90 minutes. Uh and we will see you back at 1:30. >> I mean, we have two seconds or like Okay, sorry. I didn't mean to cut you off on Q&A. >> If you want to know questions, you can come talk to this dude. He's right here. He's in the in the flesh. >> That's right. Not. >> Oh, no. >> It totally is.

>> Yeah. Except it's uh they're like dwarf files, right? Like >> or whatever. Yes. Yes, but it should be essent like right the part where it's just metadata the like actual dynamic linker stuff is slightly different because it's smack but the idea that you can just muck with the metadata of the executables is I think largely the same >> that's awesome isn't that such a nice unlock

Yep. >> You could you should totally be able to do that. Uh you might just need if they have the the part at the end the most complicated part is if they have intradependencies on each other or if they expect you know. So you might have to do a little accounting for like make sure that it's only referencing the stuff you're bringing or it already has. Um

but otherwise yeah because essentially the main things usually are like the what LD is it using and does it have any other dynamic libraries attached and then otherwise bring whatever it so it run if it can be interpreted and run on processor >> yes and that's kind of what >> so that's where um Nick LD and kind of come in >> is they like wrap that and

>> what they do. >> Yes. >> And I think I think what Nix LD does in particular is it presents its own LD and then it dynamically figures out like oh it looks like you're trying to run these things. I'll go grab them from the >> Yeah. Which is great and what you should do and it actually like the Google Workspace thing totally just work with >>

but it's probably not also maybe what you want because like it's great to have a random binary that works that also is like not what you want to maintain or deal with or Yeah, right. So, it's great for trials and terrible for everything else. >> Nice to meet you. >> Thanks for asking questions. >> come on. Jump up >> or I can jump. No one knows. I

know a lot of them like I have a little bit. I like >> Oh, you don't you're you're fine. >> I I was just going to put my little auto slides back up. >> Um and I realized that literally our slides that we were serving lunch. I don't I did not make them and I'm very upset about it. >> No, I He definitely did not make those

either. I can tell you that. I >> It is not important. >> We're not here to assign blame. We're here to figure out whose fault it is. No, it's not. >> I feel like I've talked to you before. >> Oh, that sounds like Ron. What are we here for? >> Yeah. I don't know. Yeah. >> How you doing? >> I'm doing all right. How are you? >>

Good. >> Hand of head of technology. I thought I said hand of technology and I was like >> I was like I don't know what that means >> I'm the VP of stuff. That's usually what I tell people like what do you do? I'm >> They're like what does that mean? I'm like well I sometimes run engineering. I sometimes run marketing product for a while. What do

you do? >> Yeah. Good to see you. Uh Sorry. And finally, I got they ordered I think number of doctors are But having a friend. Yes, the lily go can talk to health. Um, as long as it's topic. I want to to give it a try. Feel free. What? Check. One, two, three. Check. One, two, >> Hear me? >> Awesome. Oh, that's much better. Well, good afternoon

everybody. Hopefully you all had some good lunch. Um, my name is James. I'm uh relatively new at flocks but uh been around open source communities quite a long time at Hashi Corp and Pivotal. I have with me Stormmy. >> I'm Stormmy Peters. Um I head open source strategy and marketing at AWS. I'm relatively new to AWS but I've definitely been in open source for a while and

uh scale is one of my favorite >> or planet Nick. Y >> yeah plan scale it's all the same. So the format that we were going to use today is we had a kind of teed up some questions that uh on reproducibility and the social contract how things in open source are changing uh with respect to that and so we wanted to just start off with a

discussion uh related to that but then you know somewhere you know several questions in we're going to try and ask for some audience participations or and if you if any of you have hot takes or want to uh give us a kind of some some kind of topic that you want to explore with us. We'd love to do that because I think that'll be more engaging for

everybody if we have um some interesting stuff in the room. So, >> and and the world is changing really fast right now as you all know with AI and it's changing open source software and so I think the answers aren't going to come from us. The answers are going to come from everybody in the room. So, we look forward to what's top of mind for you, what

you you know, what are you looking to get answered in the next few days or next few months and what ideas you have to help us move to the next >> Yeah. So, one of the the first things I wanted to discuss was like what reproducibility means to all of us. Um, obviously if you're here at a Nyx conference, then you're kind of bought in a little

bit already, so I don't have to probably persuade you too much on reproducibility, but if you know I I did some thinking about like what you you mentioned open source and like reproducibility there and you think about like for me the roots of it almost go back to things like you talk see these mathematicians or scientists, you know, like hundreds of years ago communicating and it's like

the early days of open source by sharing their research with each other and communicating back and forth. So, how do you think about that as like maybe some of the origins or do you have other thoughts about that? >> I I think that's a a great analogy and you know that's why we have like the science journals now where people publish the results so that others can

reproduce them and build on them and open source wouldn't exist without that that ability to like I mean it's useful because you share it and others can build on it. So if you can't reproduce it we wouldn't have this model of sharing. Yeah, I think that's absolutely right and um you know I think one of the things that's I think interesting about you know you said everything's

changing so quickly and one of the things I have a lot of I don't know concern about is like what does that mean for you know Nyx in particular and then the open source communities because a huge part of what has made Nyx so I don't know valuable and probably why a lot of us are here is there there was this massive community effort into Nick's packages

which is an open source project and has all this participation. But if we're going to the place you probably saw how the open source maintainers um have been getting a lot of AI generated poll requests and and it's been putting a maintenance burden on maintainers and they're getting frustrated or trying to deal with that. So like how how does that play into you know like the future

of open source? I think it's changing the whole model. Like we're going to have to think about what it means to be a new contributor. How do you get into a project? How do you allow AI? How do you compensate for it? But I'm I'm curious because I I have a talk tomorrow where I have slides and I vibe coded my slides. It was really frustrating because

AI doesn't have this reproducibility aspect to it. Um, I'm curious how you see Nyx playing into that or how are we going to like add, you know, is is context the new the new way of reproducing things? >> Yeah, I I think so. Like, you know, I used to work at Hashi Cororp and Mitchell Hashimoto has been very public about how he's been um using AI tools.

He wrote a blog about how it changes and things. One of the things that I really liked about something re Mitchell recently shared about the upcoming release of Ghosty, which is an open source terminal, is they had this really um troubling bug that they was going to take them he he thought hours and hours of research to to try and fix. And he's like, you know what,

maybe the AI can can do this. And so he gave it a really well-crafted prompt. And then he used a tool called I think AMP code or something like this, but it's just like helps um engage the AI. And the thing I really loved when he shared the story of it, all of the interactive prompts were saved and part of what he was using to demonstrate the

reproducible. Here's how you could reproduce what I've done to solve this really challenging problem. They delegated to the AI. The AI solved the problem. It was a codeex model on like 53 with very high uh what's it called the that high effort like or extraordinary effort you know like just dialed it all the way to 12 but the thing that makes that reproducible isn't now like the

source code and the recipe with with Nyx and that that's what's given us all this reproducibility but like what the specification that goes into what he gave the AI and I think that's a really you know and I know actually that maybe I'll turn this to Amazon Amazon because Amazon introduced Kuro as a new flavor of an ID. I think it's a VS Code like inspired, right?

>> same open source underneath. Yeah. >> Yeah. And open source and but it had spec driven development kind of like opinionated in baked in, right? >> Yeah. So, Curo is a specbased um specbased AI driven code machine. I should know I should have these words on the top of my tongue. Um, and I think I think there's a bridge that we have to gap to cross or

to a river we have to cross or um with those of us who've written code for a long time are used to like the same things producing the same thing. And in AI, if you've tried to to generate the same results even from the same prompt, you don't always get it. Um, so Curo uses more of a specd driven model so that you can describe the whole

problem. And I I think about it a little bit when you think about like AI agents. Um, it's kind of like having a team of people that work for you and if you just go out and tell them like go write this thing, they're all going to interpret that differently and they're all going to go write something different. And so now our role is to describe the

context and what we want really specifically uh much like a product manager does or or much like a a manager of a team does. And so you're now having to set the context more than just like writing the code. you have all of these like agents who are actually almost like having a whole team working for you. So I think the role of a coder, developer, engineer,

like even the title is changing um is very different these days than it used to be. Um so I'm curious >> Yeah. So one of the things that we do at flocks is we've been trying to you know use AI tools ourselves and change we had been traditionally using a very standard software development life cycle where someone on a product team or an engineer writes the equivalent

of a product requirements document a PRD and one of the things that we have been trying the last few months is we've used things tools like cloud code and skills to like make that process itself repeatable so that when someone is deciding what they want to build, they're engaging with the AI to lead them down a guided process and then we get a like a markdown file

that whether it's something small, something big, it's gone through the same process and has the same answers the same set of questions and that becomes kind of our our specification and then you could either build it the oldfashioned way with you know handwriting code or you can give it to an agent and take it from there and we found that's pretty effective, but it's also had some

resistance from um people that like on the team that just said, you know what, this is kind of a frustrating experience. Like I I'll tell you for myself. Sometimes it asks me questions around like um who else has asked for this feature? And I was like, well actually it's just my idea. I didn't really validate it with customers yet. and he's like, "Okay, I'm going to ding

you for that and I'm not going to prioritize this as highly in our scoring methodology because you don't have enough evidence yet." And I'm like, "Oh, darn it." But it's holding me accountable and that actually is good. Sometimes accountability doesn't feel great, but I know it's right. I should be getting more evidence from customers before we go ask the engineers to go >> or before you ask

your agents. >> Oh, sure. I'm curious if you use like a a skill like a clawed skill or like a power like a curo power to do that. Um because I think they have a lot of they have a role there too and helping us set the the the context for it. Um I was I was sort of a tangent. I was I was talking to someone

uh last night about strands. AWS strands is a way to create agents. Um it's it's quite fun and it has this idea of like prompting you you for what you want while you're creating them. Um, but I said, "I have a robot vacuum cleaner that I absolutely love." And I said, "But I want one that flies around my house and dust the house, but AI can't do

that yet because AI doesn't control hardware." Um, and it turns out Strand's Labs just released like this robotic um, library and you can actually control hardware or you can play with a virtual environment so you can write your code to control the hardware. So, if anybody would like to work with me on a drone that will dust my house, that would be really awesome. >> That sounds

pretty cool. Yeah, we we're just mainly using cloud right now, but I >> it's a really fascinating time where, you know, first we saw MCP come out like a little over a year ago or something like that, right? And it's become a pretty widely adopted standard. We saw a whole bunch of other people pile in on that. And so I kind of believe that if you go

look at a a skill, it's a fairly simple structure with some YAML at the top to tell what the skill, you know, is in the description and then just some human readable markdown underneath. And I'm really interested to see like what happens with the open source community on that because like is are there going to be big open source community projects emerging, you know, going forward or,

you know, is there going to be something more like the agents just write all that stuff and we don't need open source communities? >> Well, I think that's where the open source software community can really help and really play in. We used to have like way back when we had standards groups. We still have standards groups. uh but they would all meet and they would describe the

standard together and it was a relatively slow process and then when open source software came out people were like well I'll just do a proof of concept and I'll put it out there and those open source software projects became the standards in a large part. Um so I think with AI changing so quickly I think the open source software community could be creating standards like MCP and

other things and we need them. we need to know how we're going to work with all of this new technology in a way that still allows us to collaborate because going back to your like all these PRs that are coming in. What are what are maintainers supposed to to do about that? Like and I'm curious from the room, um what have you encountered with AI changing the

world that you work in? Is it changing how you can reproduce things? Is it changing the contributions you get? >> Yeah. Is anyone getting AI slop from poll requests? So you want to say something about it and we can >> know we don't recognize So you don't have an AI policy and sometimes people in your company are submitting slop and you're not you're not able to notice

it till the code starts doing weird things. >> that's that's interesting. So that's like an internal consumption. Is anyone doing things in open source and had open source poll requests come in that are obviously AI generated or that's becoming part of a burden? You're nodding your head, sir. Uh what could you say about it? It was just confusing because it was on a couple of repos. I

just put everything. It was like a request like oh I'm adding tasks and it was very obviously but also trying to engage. >> Oh interesting. So the story here is some older project about five years ago that this this person had you put out there and then only recently did an agent submit some poll requests that related to you know some kind of scheduling enhancement and it

didn't make a whole lot of sense and then when you tried to engage with the agent the agent didn't respond back because it's just AI. I don't know if any of you saw made the news um in the last few months where someone put this open claw um you know or claude you know claudebot and it started submitting pull requests in to a maintainer and this was

a python mat lib uh library of some kind and uh the maintainer is like no that's not in the direction I want to take the project so they closed the pull request and the the bot actually wrote an inflammatory blog post doing research on the maintain container and and their personal history from online and you know said they're gatekeeping that they're they're they're doing things the wrong

way. They're treating me poorly and they tried to shame this person into accepting their poll request. And so like that went round and round and there's a really good hard fork podcast about that if you want to listen to that. That was about a couple weeks ago. And I just think that um you know maybe I'll talk about promptly with this. You know, one of the things

that um some of these projects are doing is they're putting up some you call them friction points or barriers so that you either automatically get your poll request closed if you're not on a pre-approved list. And the way you get on the pre-approved list is by going to the you know getting somebody that's a maintainer or a community member to vouch for you. What do you think

about that approach as like a a way to give maintainers some relief? Yeah. So, I think the projects are getting a lot of slop. So, they need to put some some gatekeeping in in place so that they're not just getting inundated with these. I had a couple interesting thoughts on on the the bot, but to to your question on what should projects do, we've seen projects do

things like say your first poll request cannot be more than so many lines of code because anyone who's ever asked an AI chatbot for something realizes they're very verbose. Um, and plus if it's short, there's more of a chance that they will have read it and made sure that the human will have read it and made sure that they understand it. Um, I was actually talking to

Elliot last night and we had a we had an interesting conversation and he suggested maybe we should flip it um and have new contributors review PRs instead of write PRs. So you were because the fun part is writing it and creating it, right? Um, so maybe you say everybody has to review so many PRs they can't approve them but review them uh before they're allowed to contribute

their own. Um, I thought that was a really interesting idea. Like is is the barrier to entry something different than what we've had so far because now that entry point is too easy um and too easy to produce bad quality. >> Yeah, that makes sense. Do you want to go back to the open claw um thing that you were thinking about? >> Yeah, I had like two

thoughts when I read that thing. Uh my first one was like, "Wow, assuming our large language models learned off of things they read or saw, this is really reflective of a bad behavior in our society." Like it had to have learned it from somewhere, right? I don't believe it just came up with like that way of acting. Um and second, we talk about these things like their

um what's the Terminator one? The Skynet. We talk about them like they're independent, like nobody created it. Um but somebody created that agent. like somebody sent it out to do something and shouldn't that person who's giving it tokens and money to work, shouldn't they be responsible for its behavior? But we didn't mention that person until very late in the news cycle. >> Well, I think they were

anonymous until fairly late in the news cycle. There was like three or four blog posts that were back and forth before that maintainer actually got contacted by the person who was responsible for setting up the open claw. But that made me think about, you know, something like, you know, if you uh our CTO for Flux, Michael, he bought a drone because they they have a house in

in the US and sometimes they need to do maintenance on it, but they're not around and they needed to look at the roof. >> Is this going to help me dust? >> No, it's not. The drone's not going to help you dust. But when you buy a drone of a certain size or capability, you have to register it um with, you know, the I'm guessing some whoever

maintains the airspace. >> FAA. Thank you. So, you know, at some point are we going to have to register our agents that are acting on our behalf and or at least have them, you know, some kind of authority to be able to say like who is who is the human accountable behind this agent because like it it is very strange to have this stuff, these agents running

around doing stuff and you don't know who or what they are because that's what was in this. If that person didn't come forward, there was going to be no way for them to get identified because they could you could start up a Gmail account. The Gmail account could open up a GitHub account. The GitHub account can now, you know, comment on PRs and shame people and you

the whole time you don't get to know who they are, unless maybe you sue them in court and then unwind it all or something, right? But >> I I hope there's like I I agree we should be holding the people behind it accountable. Um the open source software community has always pushed back against making everybody identify themselves. So, I hope that we come up with that we

come up with a model that works well before somebody else comes up with a model that that doesn't meet our eth um our moral values. >> Yeah, that is a good point. I I appreciate the there are benefits to the anonymity and then there's a cost if if there's no counterbalance to that, right? Um was there anything else that you wanted to cover on the OpenClaw part?

Have you tried OpenClaw yourself? >> I haven't played with it much. Have you? >> I did. I I set up a Digital Ocean uh account and I just felt like I had to like understand what all the hype was about, but I was really worried about >> Did you buy a Mac Mini? >> I did not because that's what I was like, I don't see the point

of that. Um, but I set it up on Digital Ocean and I only gave I I turned off all like the I think they call it skills. I can't remember exactly the word, but I I only gave it calendar readonly calendar access in Google Calendar and I was chatting with it on WhatsApp and then I was and it was cool, but I was like I'm just I'm

too nervous about the security of if I gave it too much permissions, what it might be do on my behalf and those kinds of things. So, I turned it off. I think what's probably going to be app and this is my hot take is I don't understand why those things like cloud code now has scheduling built into it um that they can why aren't the they have

co-work code and normal Q&A Q&A mode. I just don't see why a fourth tab wouldn't be personal assistant or something like >> I think that's what the co-work is kind of kind of getting there. >> Yeah. if you do once you do scheduled tasks like so one of the things open claw can do is like give you you say what news topics you're interested in or monitoring

and then it'll give you a summary overnight or something like that that could just be a scheduled task so I I feel much more comfortable with a big AI lab or someone with resources of Amazon like building those kinds of safeguards than I would um someone yolo on a project right >> so you can do that with AWS strands so somebody I was talking to yesterday actually

every morning when he wakes up um his his Strands agent that he created himself has read all of his Slack and all of his email and told him what things they think what things his agent thinks are most important for him to look at and also has suggested you know replies and pull requests and stuff. Um so he doesn't have it do anything. He just has it

summarize his world and suggest the first course action. It sounds extraordinarily useful and I think as long as it's being done in a way that has some safeguards like one of the things that I know that Anthropic does and they've been in the news a lot for this recently is they build their safeguards deeply into the model itself where when I set up my open cloud instance

I forgot to tell you this part uh I didn't feel great about this. Have any of you heard of Miniax? >> So it's it's a Chinese company very cheap um because I was like I don't know how much this thing's going to cost me. So, you know, I'll I'll just set up for Miniax and I it was $25 is your minimum buy. But, um, when I was

texting through WhatsApp and then it goes to my calendar, sends all that data to Miniax and then it spits out the result that I said, you know, tell me my travel schedule the rest of the year and it told me I was coming here to Planet Nix and things like that. But I just realized, okay, this Chinese company is getting all the data about my calendar. Am

I okay with that? You know, those are the kinds of things you have to think about. And I think the models are getting better and smaller. That was an example of one of the things I got nervous about um setting that up. >> I had an example I think I think almost 10 years ago. There was like this online journaling tool that sounded really interesting to me.

So I tried it out for like a week. I don't think I posted anything too personal in it. And then I went and deleted my account. First I deleted all my data manually. Then I told it to delete all my data. Then I told it to delete my account. I still get reminder emails. I got one earlier this week that says you posted this on like in

2018. would you like to to write about that again? And I'm like, you're not supposed to have that anymore. So, I think we have to look at who we're trusting. >> Well, that uh the speaking of that, have you any of you seen the news article in the last few days about the meta glasses and how the date where the data goes for that? You want to

share a little go ahead. >> Okay. So, my understanding is that there's some relatively, you know, lowcost uh center location. I don't know. I think it was some place was it Kenya or some some place like this? I can't remember where where all of the video from those glasses are being human trained. So, um, and that means people's, you know, when they're wearing their glasses to the

bathroom, when they're doing other activities with the glasses on, you know, like >> it was very inappropriate, but unintentional recordings that were happening. Yeah. >> So, you do have to be careful with this stuff. Um, and you know, we were I was meeting with someone the other day that they had these glasses on and I noticed they look like bif focals and I was like, "Oh, that's

interesting design." Um, and you know, I was engaging with this person for many hours throughout the day and at the end of the day we were at dinner and he's like, "Oh, do you want to try my glasses on?" And you try them on and it had like the Star Wars text that's like going like that, but that that was being printed out in the glasses. And

so the whole day he's able to record the conversations that are happening and get transcripts of that. And I didn't know, >> you know, that's kind of strange, right? >> And and what does that that mean? Like we can all wear AI recorders now. Like and and how can open source help that? Like what what what should we be putting in place? So >> I you know,

>> we've kind of strayed from reproducibility. >> Yeah, we have we have strayed from reproducibility. But it's a it's a good question. I mean, to me, I think one of the things that I appreciate about some of the going back to your example of the the data privacy laws and being able to trust but verify, I would like, you know, you could submit a request like you

did to say, "Delete my data and you should be able to to get a confirmation that they deleted your data and they're complying with GDPR or the California privacy law or whatever. Otherwise, there are penalties to be put in place and there's a repercussion for that, >> And a place to report it because I maybe there is, but I don't know where to report it. Is this

a United States thing that you did? Okay. >> Interesting. >> Any any other topics or questions people would like to to bring up? >> Misalignment. >> Deliberative misalignment. a bunch of measured on these KPIs and here's what you can't do. The agents violated not only >> I I love that. Oh yeah, you're right. I was wrong. >> Repeat the story. Do you guys see any efforts to

sort of standardize? It seems like a file. >> So, so the question was based on some Cornell research, they gave agents some tasks to do and gave it some guidelines on things it was not supposed to do and they violated the things that it said do not do. And so, is there a way that we can put these guidelines in place? But even worse, they were aware

that they >> and they were aware that they violated him. >> I'm not I'm never sure if it was really aware or if it just changed its mind when you call it out because I like trying to create my slides for tomorrow. I said, you know, can you create a PowerPoint? And it says, yes. I'm like, where's the PowerPoint? It's like, oh, I created it. Here's the

link. And it's I'm like, that's text that's not a link. Oh, I'm sorry. Here's the link. I'm like, that's still not a link. And it goes, oh, I'm sorry. I create can't create PowerPoints. I should have told you that earlier. Um, so I'm not always convinced they knowingly violated it, if that makes sense. But I I think it is up to like the companies like AWS Anthropic

and the open source software community to make sure that we're developing tools that obey those guidelines that >> Well, then maybe I can take this back to to Nyx because with Nyx, we have cryptographic guarantees. When you build something, it's going to be put in the Nyx store whether it's input addressed or content addressed as your choice. and you know that the same thing happens again, it's

going to get the same result. But maybe we need something like that for some of these models to understand you know how the how they behave and the the chain of thought you know famously like when these first things first came out they didn't have citations right and then I think it was perplexity first started like adding citations that was really useful and other things but >>

if we can like have some sort of way to in whether be an open standard to understand why and what the model is doing with these things and you know Maybe like for example, I remember like about a week or two ago, the Meta person um famously had OpenClaw set up and it was deleting her emails and she was telling it to stop and they were trying

to figure out why. And what I read about it was it was probably a context window problem where the in initial instruction which is like always get confirmation from me before you delete my emails. It was like the ver and never delete an email without asking me. And it's like at some point the context window has got full and it compacts and then like those instructions are

gone but it still continues about its task and then it can start deleting. So if you have some kind of standard or way to you know prove just like we do at Nyx that something is is done a certain way maybe that's you know some some techniques we can bring >> Yeah. In that case I would consider that a bug like like >> Yeah, I think they

did too as well. Well, here's here's another one I'm interested in. Um, and especially if anyone else in the in the audience has a hot take on this, too. So, there's have you seen this debate about right now we're all in the open source community standing on, you know, work of a lot of other people that have contributed to the open source community over time. And so,

we all benefit from that. when we're starting a new project, we're using things like Linux and Kubernetes and containers and all, you know, a whole bunch of um libraries and utilities that are out there. Um, but some people are believing that in the future some of these agents will just create the code you need on the fly and they'll even make the argument, well, there's less security

risk in that because um, you know, there was not a common commonly known vulnerability in a particular code that like let's just say you have a CVE 10 that's a really bad CVE. Well, you can go all around the world and finds people using that same dependency. So what do you think about that in terms of the future of open source and if these agents are deciding

not to use and contribute to libraries but just to make up code on the as they go? >> I I am worried about that. Um and I'm going to mention in my talk tomorrow but the in the open source software world a lot of people got their start writing some small piece of code that did something really useful. um I mean how many people started writing something

just to solve a problem that they had and it was a small piece of code but it did it really well and so everybody used it and that's how they got started and now I do think that you're not going to go use that library if it's like five lines of code your your AI agent is probably just going to rewrite it. Um, I don't know if

that means more security or less security because those five lines of code might have been really well tested um for the whole world and now you've got a whole bunch of different implementations of it that might not have the same level of scrutiny. Um, but I mostly so I worry about that, but I mostly worry about like how are people going to get started in open source

software? like if it's not submitting pull requests because they're using AI and they're submitting slop and it's not creating these small libraries that are really useful. What is our path for new people to get >> I don't have a great answer for that, but yeah, you do, sir. >> But that's kind of how you learn is There's a way to kind of learn. >> So maybe people

can get started reviewing code and learn. I I know it's hard to review code, but I do think that's the way we learn, right? You read >> you read things. >> You want to repeat it for the recording? >> Yeah. So, I think what I'm hearing here is like the the triage and adding reproducer steps and things like that to some issues and things like that or

ways you can contribute. >> Yeah. the bug isn't relevant anymore. >> go ahead. >> What about forks? >> Oh, this is a good one. Okay, so um this this one came from um I don't know if you know the pragmatic engineer. This was a former Uber leader. He's very prolific right now on and sharing engineering insights. And he had Mitchell Hashimoto on his podcast. And one of

the the questions came up like what do you do as a maintainer when you want to um decline the the poll request from anyone just like whether it's a person or if it's a bot and and Mitchell's feedback is I think there should be a lot more forks because forking is not necessarily bad. In fact, that's the whole point of being an OSI license is that you

are unrestricted from making a fork and being able to, you know, do with it as you please. You know, there are some differences obviously with certain copy left licenses and things. But most of the time people don't want to maintain a fork because it's like a lot of work. But if the agent now is able to take away some of that pain because they're able to do

the, you know, merging upstream by keeping the whatever difference you have in your fork alive. Maybe that makes a lot more sense and that's a way to relieve some of this tension between maintainers and people that are sending them lowquality PRs that they don't want to accept. What do you think about that? >> Yeah, I was talking to a Postgress maintainer last night and he argued that

we've always done forks. This is just a different way of doing it. So, you know, whenever you go to work on a codebase, you make a local copy and you make your changes before you do the pull request. So, we're all we always were doing a little mini fork. Um, and now he thinks the forks will just be bigger. Um, and he talked about a contribution, an

enormous contribution they got from somebody and they were like, we can't merge to this. Like, it would take us too long to to review this, but he's like, give us your prompt and we'll recreate it and put it in the codebase. So, it's kind of interesting concept. >> Yeah, that is really interesting. I I remember back I used to work on cloud foundry and one of the

things we used to think a lot about there was are there ways that we can have an extension or a plugin right and many of these areas have things like that so that >> at least you're not maintaining the whole codebase of all of postgress maybe it's just a postgress plugin or something like that so that's another you know thing that you can do >> yeah we

use has some kind of develop. So you're suggesting that we use AI to review code. Absolutely think you should and you can feed it all your guidelines like you said on the you also said that all code has some AI generated code in it. There was a really interesting I'm not going to remember the stat off the top of my head, but there was a study and

they asked developers if they had used AI and if they told the project when they submitted the code that they had used AI and most people had used it. I don't remember the number like 70 80% and very few people reported it but it varied by tool. So what tool they had used made them more or less willing to like tell the project that they had used

it. So like claude had a very high compliance rate. Um people would tell the project I used collad. Um >> well I think another thing this gentleman's story from earlier around if their organization doesn't have a policy >> you might be using it like just how I used to use Stack Overflow to uh to you know contribute but >> which was not reproducible a lot of >>

and not not always not always not always reproducible not always good either. Um but if if you're not even allowed to use AI officially but people do it anyway. I don't think there's a great way to tell the difference um between between something at some point like um you know my my kids are in high school and there's all kinds of chat GBT being used to contribute

to papers and stuff and teachers are trying to catch kids that are doing this stuff but like some of the kids are getting flagged for doing good quality work. They're that they're getting claimed as AI but it wasn't right. And so I think that sometimes like it's just really hard hard to tell >> and the chat people to assess those met% but it looks good >> and

I tried using chat bots to like help figure that out and they hallucinate. I'm like did you write this? I was like yes I wrote that. No you didn't. I wrote that. That was a blog post I wrote 10 >> So here's another interesting one. Uh is the cost of AI tokens a barrier to creating software? you know, some maybe not everybody. And one of the great

things about open source is that you could go to the library, get a cool book or something like that, and then you maybe have access to a computer lab or something like that, but you could participate with a pretty low barrier to entry. But if I have to go buy a bunch of tokens to create software, is that going to, you know, create different classes of people

that can contribute and those who can't because they just don't have access to money to pay for the tokens? >> Yeah, I think so. Um, I think they'll get cheaper. I think it'll get a lot cheaper. um will just be expected to generate a lot more. Um but I think it's kind of like the internet. Um so I used to help set up computer labs for kids,

disadvantaged kids around the world. Um and internet was the big barrier. So I think tokens probably will be another barrier. What do you think? Yeah, >> please. >> I mean, it does seem like it could discourage them. PR. >> So, here's a vote for making newcomers review all the PRs. I think we've heard a version of that from three different people in the audience. So, I think

we might be on to something here. >> Yeah, makes sense. And I think, you know, one of the things I've noticed is I didn't actually intentionally set this up, but when I've been doing contributions, um, when I was at Pivotal, we would do pair pair programming and the git commit would have both contributors that were pairing together. Well, now claude just does that for you automatically when

you're using claude, right? And so then at least hopefully there can be more of a standard based on that. So that'll give you some indicator as well. But I like your idea. So, so if Claude was like your partner or pair there, will we build the same kind of communities >> because that that's what I love about >> I know. I I think it'll be a big

shame if none of us get together like this in a few years, right? Because I think the personal connections are a huge part of it and very motivating. Um, you know, I think I I sure I sure hope it still happens, but I am worried about it. I really am because um the if I think about like think about something like the Linux Foundation or the CNCF

and all the companies that had together pull resources and make a project like Kubernetes what it is today. Would that still happen in the same way when people could kind of make their own little mini Kubernetes that didn't have the community and didn't have as much sophistication but solve their problemish in a way like I don't know it may not have happened the same way. I don't

think we're yet at the point where someone could create Kubernetes with a bunch of agents like that would be a superhuman >> I agree it's a sophisticated project but I'm just saying I I worry about a future project like that that did get a bunch of collaboration and you know you could you could pick a lot of them you know it's these um frameworks like you know

Linux Kubernetes and you could go pick any number of like even JavaScript frameworks and things like that they form these big communities and JavaScript has been one of the things that's been most disrupted because the front-end code is fairly approachable for these agents to to solve. >> But every every technology that's come in that's disruptive, we've worried about the jobs that were going to go away. And

every single one has brought maybe jobs to go jobs do go away, but it's brought new jobs. Like our jobs didn't didn't exist what 30 40 years ago, like not not like they they are >> Um so I'm really curious where we're going. >> Cool. All right. any other interesting uh ideas for that you want to discuss with other people at the conference especially related to open

source >> or reproducibility. >> Michael. >> Does reproducibility matter if I can just do it again? >> If you really could, maybe not. But we are not there yet. Like when I was trying to create my slides with AI tools, none of the tools could generate the same thing twice. >> But you could generate slide deck that had the right content, right? >> It didn't. I really

like that infographic it put in the other slide deck and now it's not coming out in the new one or I tell it just to correct the typo and it changes the whole slide. >> So I'll Oh, you have a comment here. So the value of software our reproducible manner. What happens when the language describe the language? >> Okay. So the the comment here is you know

when you have a very structured approach for like a C++ or forran or something like that it's it's written to a specification and that's how you get the deterministic result. If we move down into just human language like what what does that mean? My personal belief is you know things like hero that will ask you to to use human language but in a certain way. So that

like put in the phrase here for the the overview the main goals what are constraints that you want me to never do or stay away from things like that. I think there will be some structure and if you're familiar with this concept of evals where you can you know evaluate per Michael's comment here about if I could just recreate it well how do I know that I

recreated the good good result if I could have a you know some way of scoring the result to say like did it meet my criteria then I think it doesn't really matter if I regenerate it again and it's and it's slightly different if it passes the evals then it's probably okay and I know that with some of the reinforcement learning techniques. That is how people are getting

better and better results out of these models because the models learn that they generate something, it's not quite right, so they change something and built it back back in again and then it gets better and better and better. >> Do do you think previous generations had the same conversation? Because like I never want to write an assembly again. But surely there was some expert assembly programmer out

there that's like, man, C is not going to do this well. Like it's going to make all these assumptions and have all these memory >> yeah, no, I agree with that. Um I think I think we're moving up in levels of abstraction and for the most part it's going to be good but it's going to be hard. Um so one last oh go ahead >> we have

one minute so I was >> one minute. Okay last last question here. What do you think about dog fooding versus de developer autonomy? So the dog fooding approach is like you know Amazon's making their own models and they for let's just theoretical example. Amazon says, "I want you to all use our models." And and and then there might be some ones outside Amazon that you might do

a better job at certain tasks or whatever, but they're not allowed. And then the other side of things is like anybody at Amazon can choose whatever model they want and then maybe your models don't get better because your teams aren't using them as much. How do you think about that? >> So, I think you have to do a little bit of both. And I think the open

source software community actually went through this transition as well. um when open source software first got started pretty much everyone worked on something that they used like you worked on the desktop you got to see how it worked and now when you're working on Kubernetes or seph or you know you might be working on something that you don't use in its entirety um but one of the

examples that I like to give where I think I worked on a project that didn't succeed because of this um was when I was at Mozilla we were making the Firefox OS phone I don't know if any of you are familiar with it the idea was to build an HTMLbased phone it was going to cost $100 it did cost $100 and it was going to go to

the developing world to bring the internet and the web to the to the whole world. Well, none of us really wanted to use a $100 phone. We had thousand phones and so the dog fooding was really hard and I almost think we should have started with the thousand phones and then figured out how to scale it down um because I think we made some mistakes there. >>

Makes sense. Well, with that, we're going to leave it there. Thanks for engaging in the discussion. That was really fun for us. Hopefully, you had some value out of it and uh enjoy the The next talk should begin in about five All right, we're just just a minute away. Uh, he's getting set up. Reminder, if you'd like to do any uh interview or video content, we have

door prizes. So, you can get if you find Jackie, she's walking around with cameras and stuff and she'll ask you like three or four questions kind of like roaming person on the street reporting. You can answer them and you'll get a door prize. And if you want that, that'd be great. And if you don't, that's okay, too. Um, and there's an afterparty tomorrow afternoon after this after

the conference. There was a QR code that'll rotate on the when we have like the default slide deck up here. It's also out in the hall and stuff like that. So, uh, please do attend if you are so inclined. There will be beverages and snacks and foods and all sorts of good stuff. So, uh, are you ready? >> All right. And with that, we're gonna kick it

off here. Uh, wait, this is the missing part of Nyx, right? All right. Missing part of Nicks with Bernardo. All right. Thank you everyone. Um, okay. So, today I want to talk to you about the missing part of Nyx and uh where to find it. Um, a little bit about myself. I am a build systems engineer at Enthropic. Um I'm at love fault on GitHub. You may

or may not have before and I used to do Rust builds at AWS and before that I did build stuff at Google. So I've been mostly just doing builds for a long time and yeah talk to me about build systems and uh sound systems line systems. Um so I want to start just talking a little bit about how what this even is like what what even is

uh uh distributed build system. So wait I have to fix my screen. It broke. Oh no one second. Perfect. Okay. Um, so what is a distributed build system? Um, it's a system that efficiently runs builds across many machines, sharing the results between them and deciding which machine builds what. And this sounds pretty simple at first, but if you think about it, these properties are actually really hard

to do correctly. Because if you're going to run builds across machines, well, you already kind of need to make sure that those machines don't really matter. And if you're going to share the results, you need to know what is what. like how can you know for a fact that you can substitute something for something else and you then just have to decide who to build it which

sounds to me like the easy part. Um they're not CI. So CI has to do with uh web hooks and uh GitHub and making sure when a new commit comes in submitting jobs that the build system is what actually is doing your build. So the CI is usually a customer for distributed build system. So if you look at things like build kite, they have both. Usually CI

systems have a distributed build system inside of them. But I'm talking I'm focusing on that part that is inside of it because I think for a lot of us that do a lot of builds, it would be really nice if you could use that distributed part without having to deal with the CI part. Every, you know, if every time you want to do a distributed build, you

have push or commit. That's not a really great experience. Um, so I'm going to start off talking about the pillars for this. Like so if builds run on many machines and the results are shared, what do you really need? Um you need hermaticity for sure, right? Builds as pure functions, the only way you can guarantee that you can have a heterogeneous fleet of compute and still have

your builds make some sense. And once you get that, you get good caching hopefully, so you don't have to rebuild things over and over and over again. Because again, if you have a lot of people using this very large distributed system, the more caching matters, right? The more the more important it will be that you don't rebuild spuriously. Providence becomes really important because now you didn't build

a lot of the things you're using. So what is that artifact even built from? What is in it? And signing. At this point, you have a large network of computers and a large network of people using them to build things. You would hope that you know which machine built which artifact and with a cryptographic signature, you know who uploaded it to cache and you know who submitted

that bill. Um, if you don't have this, well, things get really hard. Um, I if you don't have caching, I hope you have a lot of compute to pay for it. Without hericity, great. you built uh works on my machine but at scale and without signing how would you know which artifacts to invalidate if you do have a security issue in your building and without providence well

how do you even know what you're building how does Nyx fare here how does it come into play so so far if you know Nyx pretty well you would be like oh yeah Nick is that distributed build system so far all the properties I mentioned we permaticity we got sandbox caching substitutors providence the DRV records everything you need for providence pretty much and uh we have really

high quality signing with multiple signers so let's go back to our definition um runs across many machines I call this execution and Nyx does give us that we have remote builders and remote stores shares results between them we have caching with substitutors great decides which machine builds what coordin a nation. Yeah, we we have nothing for that. Nothing really. Um, Nyx gives you everything except the part

that makes it distributed. what is coordination? So, it's composed of a couple different parts. So, first is scheduling. Who is going to build what? Um, who's actually going to be doing the building. Dispatching, which is actually just routing the request to the to each builder. Life cycle management workers do go down. They run out of memory. They will have random health issues. Uh you have to scale

them up and down and queuing ordering retries of builds. So I started searching for this like um does it exist? Is there something that I can use in Nyx easily to do builds at scale really fully distributed builds? And I found a couple things. The first one was Yens. This is a hat tip to Julian and Alex at Garnex. It's pretty cool. They just shoved Haproxy in

front of the SSH protocol and then they let it distribute it among a bunch of machines. It's It's kind of janky, but like if you wanted to code Golf build distribution for Nyx, I don't think you could do any better than this. It's pretty awesome how well it works for how little Infra is behind it. The problem is you can't do anything that's really clever with this.

So you can't do any really intelligent scheduling. You can't really understand like hot proxy is not going to understand derivations. Hoproxy is not going to look at the derivation figure out what dependencies it has and then be like oh I should schedule on this host because it already has those in the store. So, you're never going to be able to build something that's truly efficient with something

like this unless you like, I don't know, go all in on modifying a proxy, which probably don't want to do. Then there's Nixbuild.net. Um, they're here. You should talk to them. Rickard and David. It's a paper use SAS build form for Nyx. And, uh, you can self-hosted. They they do offer licensing and we use it at Entropic. It's very, very good. And it's essentially what this talk

is about. It's about, well, why don't we have this in the community? Like, it's such an obvious next piece for Nyx. It's so clear that we've done all of these things to make this possible, but then we just didn't make it possible. And uh the obvious thing when you think about this is Hydra, but Hydra is not really an answer for this. First of all, it's more

CI shape than you'd like, and it just does way too much. like uh Hydra does email. Uh I I don't want my my my distributed build system doing that. Um and Hydra used to be really bad, but thanks to a lot of work from the people at Helsinki, we now have the new QRunner, which I I I don't know if it's been released yet or not, but

I think it's getting close. And the new QRunner is way more efficient, and it is getting better, but it used to be not not very good. Um, I thought to myself when I was looking at this like how how hard could it be? You know, all the pieces are there. How hard could it be to build a a dist build distribution system for Nix. So I I

called the project Rio and um this is uh mostly the story about me failing to build it. So, we had this long weekend in October and I made a bet with my co-workers that I could do it in the long weekend. Three Yeah. Foreshadowing. Uh my pride which I lost. Um so obviously working at Entropic I have access to you know claude uh a lot of clouds

and I figured you know with enough clouds I surely can do it. enough clouds and coffee. It will work just fine. So, um, by the way, these links won't work yet because I forgot to click publish, but after the talk, they will. Um, so for the V1, I thought, okay, I'm going to I'm going to build something that central broker that receives these build inputs and distributes

them. I am going to try to get the SSH protocol support via this crate that I found from Tweet called Next Damon. The builders just register with the broker. Simple round robin scheduling. should be easy enough. Uh yeah, first of all, the central broker is a huge single point of failure. So anytime someone sends like a cursed derivation or anything and the broker processes and crashes, your

entire system goes down. So that wasn't great idea. Um the next statement crate has a lot of unimplemented things and a lot of parts of it that don't quite work that well. Um and implementing the protocol yourself is uh not easy. uh is way harder than you think and a lot of the docs is just C++ code that you have to go and read. Um so I

gave up on this pretty quickly and never even got got it to work after iterating a little bit. I think halfway through at the end I think it was like end of Saturday I was like oh yeah I'm I'm I'm done with this. So then I started um early like Sunday 2 am on V2 which I thought okay let's go broker lists. I will use raft consensus.

There's no central broker. There's they will elect a leader and then the leader receives the build requests and then it will decide which of the other ones gets to do the build. Um I'm not going to implement the next protocol. That's way too hard. I'm just going to make a custom CLI that does the eval locally and then sends it over gRPC. That sounds like a good

idea. And um yeah, this seemed way better. Also, the bills would flow directly from the client to the elected agent. and you when you submit a build, you get back the address for the agent that was assigned and then you tell it, "Hey, I'm me. Here's the derivation." And this seemed nice to me because you don't have to funnel through all the like all the derivations through

the the broker. Uh yeah, this didn't work. Um the raft part was really hard to build. Even though seemed like there were good crates and rust to do this, it was really hard. Like I I I have never tried to debug something that was so cursed. Um, the custom CLI I after using it for a while I was like I miss my normal tooling and I don't

want to deal with this. Um, and I don't think anyone else would either. So why am I building it? Um, yeah bugs. A lot of Heisen bugs. I had like I spent hours and hours and hours just trying to understand why sometimes the it would elect a leader, sometimes it wouldn't. Sometimes it would assign bills to the wrong person. Like it was it was really really messy.

Um, and then I gave up. Then at this point, we had um we already had Nyx build.net deployed or starting to deploy internally and it was working great. And I was like, okay, I'm I'm I'm sick of this problem. It's it's not a good one. Uh, up until winter break when I got bored again and I was like, oh, maybe maybe now I've learned enough lessons. So

let's use SSHNG none none of the legacy SSH protocol only the new one leader elected by theuler with a PG advisor lock um the workers all use a fuse backend for the nick store which is really nice with overlay effects for the build and uh none of normal nick store for in S3 or anything like that so there's a component that's called rio store that does chunked

cast storage with fast cdc There's a cool paper by the way if people are interested in that you should go look at the best it's really really fascinating. Um yeah implementing the protocol is hard but actually Claude was really good at it once I just told it write tests that just run the next Damon and compare bit by bit that you get the same output. Once I

figured out how to do that it just turned through the whole problem. So that was easy. Um, making consensus being roughly someone else's job, I think was a wise decision. That saved me a lot of trouble. Um, the fuse part I'm still fixing. So, I don't know if that was a good decision or not, but when it works, it is really, really nice to just have full

lazy fetching for this. Um, and the dup dduplication, full dduplication across all NARS is very powerful. Um it's uh yeah you you can save an unbelievable amount of storage um by doing this which it sounds like cost at first but if if you have really fancy duplication that means you can you can maybe afford to shove your cash into like a ginormous EBS volume that just mounts

to all of the hosts which is way faster than it. Um, so I I kind of sort of found the missing part. So I think if you really just want to if you want to find it easily, just uh buy nu.net. That's how you can find But if not, well, we have a couple gaps to fill. The protocol spec that we we have to we have to

specify this. We have to write a doc that people can read and understand and make it easy to implement an X protocol. not just for this but in general I think we need to make the protocol like a first class well supported thing um having libraries for this would be nice the snick stuff works but it's all GPL free and I didn't want to use it and

then the rest is all this real C++ code and I didn't want to use it because I was writing all this in everyone has been talking for a while like if you've been following along with snakes and fix about how to change the store model to be more efficient. How to allow for lazy fetching, how to um yeah, how to use the fuse for this. We're all

think there's pretty broad agreement that that's the way to go to make this better. We should just do it once, build this component once and then we can all use it because it's kind of a reusable so I think for me like my my big takeaway is like we we got the really hard parts right. The really difficult properties to guarantee. We we got them like we

they we have them and they work really really well. The only thing that we're missing now is coordination. we're missing this central component that you can deploy easily and just send builds to and not worry about who's actually doing the building. And I know this sounds like an annoying like uh maybe corporatey problem like you don't need this at home. I don't know. I would use this

at home. I would a thousand% put all my 12 computers on my distributed NYX build system and then just do next build and not care where the builds go. And and if you do this smartly enough, right, this would know that you have it running on a machine with two cores and one with eight and one with 16. And it knows how to right size your builds

because the P name, although it's kind of janky, is a pretty good indicator of what build it was. You can basically tag your builds with it. And you can do very clever scheduling and things like that. So I am uh I am building this. Well, Claude is building it. I I'm telling Claude how to build, but he's doing all the work, not me. Um, have it mostly

working right now. I'm still struggling with the Kubernetes parts of this. Um, right before this I was trying to get a demo for this and I I thought I had I was like, "Oh yeah, this is definitely going to work." And then I ran into some really cursed bizarro croups thing inside the Docker container inside another Docker container that I'm still trying to uh figure out. But

today, if you don't want to run it in Kubernetes, if you just want to um deploy it yourself, it already works. It just doesn't autoscale. But you can take the Riobu code and you can run the Rio worker and your some of your laptops. There are Nexos modules. Um and uh yeah, it it uh it works. And I will now to celebrate this talk. Open source it.

Wait. Come on. Come on, baby. It's a lot of Tada. Woohoo. Yeah. On the client. The problem with with that is how do you rightsize the allocation? Like if you had a if you had a fully homogeneous fleet, that would be perfect because it doesn't matter who gets what. But in reality, most of the time you want heterogeneity even especially if you're deploying to the cloud where

you can like you can just not have capacity with the same skew that you were using before and then what are you going to do? You can't scale up. But if it what happens if I if I have a build that only works on a machine that has at least 32 gigs of RAM and it gets bucketheaded to a machine that doesn't if if if if your

hashing scheme puts Chrome on your Raspberry Pi to build. >> Yep. And there's also questions about like how how much you can optimize it later. For example, you may have builds in a real environment that you don't care about how long they take. You just want to do them cheaply. Or you might have builds where you're like, "No, no, no. I'm doing development right now and I

want really low latency." So instead of you, for example, measuring the average CPU utilization to to save in history, you can measure the peak CPU utilization. So you can size them to as many cores as the build can take. So it gives you a lot more freedom with how you you do the scheduling. >> Yeah. So, um I actually thought Claude was very clever with how it

did this. It built this bloom filter setup where it has a it keeps track of all the store paths that all the workers have which it knows because it knows all the builds that submitted to each worker and it knows what each build depends on so it can deduce. So then very quickly when it's trying to schedule a new build it runs this bloom filter to find

out which builder is more likely to have the most paths present and then if it has the right um size it gets routed to it. Um I I thought that was really I wouldn't have thought of that honestly and I thought it was interesting. Um there's a lot more you can do for the like for scheduling things correctly. So one of the things that I've been trying

to get it to implement is this paper that people in the next community have been talking about forever called tags where essentially you um you divide your your hardware into these groups. So you say like oh I have small machines, medium machines, large, extra large, whatever. And then every time you see uh you find a way to a stable key for your build. So I think for

Nick's P name is pretty good. Um every time you see a new P name, you always allocate it to the smallest group with a deadline. So like five minutes. If it finishes, great. You remember that that builds on a small machine. If it doesn't, you move it on to the medium machine. And you go on until you find the right size and then you record that. And

over time, every once in a while, you try to move it back to the smaller one to see if it if if it moves. So, this is nice because it can combine both right sizing from a resource utilization perspective and from a like outcome where you can just change the deadlines for each group to get really really good latency and still have all of them right sized.

Thank you, Tom. Going once, going twice. That's it. All right. Thank you so much, everyone. All right, we're on break until 3. Check, check, check, check, check, check, check. Check, check. Mic check. One, two, one, two. Hey, hey. One, two, one, two. Check, check. Hey, hey. One, two, one, two, one, two. Check. Check. One, one, two, one, two. Check. Check. One, two. One, two. Hey. Hey. One,

two. >> Still there. Mic. Mic check. Mic check. One, two, one, two. Mic check. Mic check. Mic check. One, two. Mic >> Mic check. One, two. One, two, three, four, five. Mic check. One, two, three, >> Six, seven, eight. check. One, two. Mic check. One, two. >> Louder. Yeah. >> And obviously the more you project to a >> Okay. Cool. >> And then you can turn that

off and when you're ready, you're good to go. The one thing if you do It is 3:00, so we're going to get returned to our program. Uh, I was going to tell you what the name of the talk is, but it's sitting up here on the slide, so you can read it, I hope. And if you can't, um, ask for help. So, uh, anyway, Sam, kick it

off. Thank >> Thanks so much. >> Hey, everyone. Uh, thanks for coming. Today, I'm going to be recounting to you the saga where I got local Nyx builds to cooperate with our remote developer environments that run in Kubernetes. I plan this talk to have a little bit for everyone. We'll cover highle details and we'll also do deeper dives into Linux fundamentals. But first, I'd like to introduce

myself. Hi, I'm Sam. Um, and I'd like to make a confession. Unlike many of you here, I am not a expert in Nyx. And, uh, that means I needed a lot of help to do this project. Um, and I promise you, I genuinely mean this when I say this, that my closest collaborator here was Claude, who taught me a lot about Nyx, Kubernetes, Linux internals. we uh

brainstorm possible strategies for each of the issues I encountered and also implementing the solutions themselves. Um however I do have a lot of experience with uh Linux systems and building dev tools. I've been working in developer experience for almost a decade and uh I currently work at Anthropic. So we are an AI safety research company. Uh we make claude and we're working to build reliable, interpretable and

steerable AI systems. My team at Enthropic is called developer acceleration. Our charter is to accelerate the productivity of developers and that means humans and clouds alike. So I'd like to start my talk by making what might sound like a crazy ass. Our dev tools should just work. And so this was like random day in October. A nice user was like, "Hey, I'm trying to build my security

CLI. I write Nyx build and it should just build." Wait. Um, error. This system does not support the kernel name spaces that are required for sandboxing. Use no sandbox to disable sandboxing. Okay, that's not good. That's not what's supposed to happen. So, uh, let's let's take a step back and talk about what our developer environments look like. So, this is just a, um, little diagram. I'm on

the left with my laptop. I can SSH into a Kubernetes pod. That's my, uh, dev box. And, uh, you know, over SSH, I can run T-Mox or I could be using VS Code. It doesn't really matter. All that matters is I can now get a shell onto this box and do development work. And when I type nyx build, what it does is it uh runs in this

isolated build context which is called the sandbox. So that your builds are reproducible. And in order to build up this uh sandbox, there are Linux kernel rules that uh dictate when you are and are not allowed to use them. So these aren't arbitrary rules. Um, so this sort of bug report came in and we're like, okay, let's figure out how to uh what are possible solutions to

address this. So the solution space here, there are a couple. Um, we could patch the Linux kernel to allow this. We could uh disable sandboxing for all our developers. That's what the error message suggested, by the way. Uh, we could give all of our developers privilege containers, which would be another way around this. and we could make sandboxing work, whatever that entails. So, uh, looking into patching

the Linux kernel, it turns out that this was already attempted in 2018 and rejected upstream. So, I didn't want to pick that battle. Uh, disabling sandboxing sounds like a predominantly security thing, but actually it's a predominantly hermaticity thing. You would lose the reproducibility of your builds. Um, and since this is Planet Nix, I'm sure everyone here understands why it's important that your builds are reproducible. Um, but

then we would also need to change the configs and build scripts and habits of our users. making your containers privileged basically gives your container the same privileges as the node itself. And so this opens up the attack surface very widely for container escapes. And we wanted to we discussed this with our security partners who are also at this conference. Go talk to them and they're like please

don't do that. So we did not do that. And then finally making sandboxing work. It seemed like this was feasible and was also the right way to do things. So we're like let's take a look. I'd be remiss not to mention uh what Bernardo just talked about about uh distributed builds. And so there some of you might be asking why not just do remote builds only? And

there are a couple reasons for this. One is that they didn't exist at the time. This happened a few months ago. Bernardo's work is much more recent. And then um also even if you have a very good remote build system, you do want to do local builds sometimes. Uh case in point, Bernardo actually asks me for more disc space all the time to do his local builds.

So we chose make sandboxing work and uh this is the right way. Should be a pave path. How hard could it be? It turns out this would be a bigger journey than I anticipated. So, I'm going to take you through this roller coaster of emotions. There were twists and turns. Um, I want to take you through this journey uh through a couple of acts. And I'll intentionally

walk you through my process and not just the solutions I arrived at. So, let's go back to those kernel rules that the Linux kernel enforces that are stopping us from using sandboxing. Um, so there's there's this thing called slashproc. It's a uh file system uh that you can find if you you know lsprock, but it's a virtual file system. It acts as an interface to internal data

structures in the kernel and uh the kernel uh or or or Kubernetes pods by default will mask some paths in /rock because giving users access to them is dangerous and allows them to uh twiddle with things that they should not twiddle with. And when the Linux kernel sees that some of these paths are um are masked, it disallows making a fresh proc mount. And then the sandboxing

doesn't work and sandboxing says, "Hey, I can't make a fresh proc mount. Please use no sandbox." Um that's fine. We did a cursory online search and saw that the Kubernetes documentation says you can just set procmount unmass and you will have an unmasked proc and everything will be great. So the solution here is to add procmount unmasked podspec and we are in business. Let's go. We've applied

coupubectl and we run our build and it did not work. Okay. Um let's see. Um, oh, the the proc mount is not on the pod spec. Okay, let's let's figure out why. Um, this this is where I looked up the procmount type feature flag in our Kubernetes control plane and it said this was a beta feature. And so, okay, let's take a look at the Kubernetes documentation

again. These are their feature gates. specifically the procmount type. It says it's been around since 1.12, but then I noticed that uh these are alpha and beta and by default false. And EKS says you can't enable these by default. You have to have these either in GA or beta by uh or or on by default, which is fine. Okay, we're on 1.31. When do we need to

go to? Looks like it's enabled in 1.33. So all we need to do is upgrade our Kubernetes control plane to 1.33 and then we'll add procmount on mass and then everything will be great. Let's do We did it. We upgraded control plane. We applied the coupubeCTL and forbidden can only be set to unmasked if spec hosts users is false. What does that mean? So there's this like

small footnote in the documentation that says setting procmount to unmask requires also using this value. you must uh put your whole pod into a user namespace. Now uh there's a quest icon here. We take the side quest. What is a username space? So we have our pod in our node again. And uh normally what happens is user X in the pod is user X on the node.

And when you are in a user name space, it maps user X to some different user Y. And how this is useful is in the security context where uh root inside of your pod is an unprivileged user out on the node. Uh if you're sort of thinking back to first principles that means that if you're not using user name spaces root in your pod is root on

your node. Something to think about. Um but if you are an unprivileged user then actually the kernel is like okay I will let you have an unmasked procm mount or actually Kubernetes says I will let you do this. So we need this uh let's satisfy Kubernetes rules. Um the just a little bit of background our dev pods are not just raw Kubernetes manifests. We actually configure them

using terraform which is infrastructure as code. And so uh we use this resource from the upstream Kubernetes Terraform provider called Kubernetes stateful set. So we just need to add host users equals false to our Terraform config plus the procm mount equals on mass and everything will be fine. And uh we Terraform plan and we're in BIS an argument named host users is not expected here. Huh? Why

does it not support it? Oh, looks like someone had asked for support some time ago and it was closed as not planned because it was in alpha. I have made an issue on their uh GitHub repository to now support it. But I can't wait. We got to find a different So uh talked with claude a lot about this. We were sort of theory crafting and one of

the solutions was what if what if we allow Kubernetes stateful set to create the stateful set and then afterward we immediately launch a local exec that does coupubectl patch stateful set and we just add these uh host users false and procmount unmasked onto the spec. And this worked. This actually worked for um quite a while. And it might feel a little bit uh rickety put to together

with duct tape. And you would be right. So eventually um this did cause some problems because we're doing this like post hawk patch where we are are just doing something and sometimes there's a race condition. We're in the same reconciliation loop. You're like saying, "Oh, also do this." And the the cublet was not very happy about this. And so uh theory crafted with claude some more. This

is about two months later. And uh this this was actually something that I had we we had like thought of as a as an approach before to migrate to Kubernetes manifests. But uh and just as background, Kubernetes manifest allows you to just put basically a a your your raw YAML into uh this Terraform resource. allows you to do things that are not just supported upstream uh or

or like already encoded into the official resource. Um, but I decided against this because like this would be like a 2,000line PR doing onetoone mappings, but like no one would actually read it. And then I didn't really understand it. My teammates didn't really understand it. And we have a team that's meant to support this. And so I wanted to do something that was actually sustainable for our

team. But it turns out that we just needed to do this because the other solution was was not it. And so uh this is approximately what it looks like. you could just put things into the spec and it works and we had a giant PR and Claude did all the onetoone mapping and it worked like a charm actually. So we did it. We can run Nyx build

and well okay it's a little bit anticlimactic but that means it worked. So let's let's just like do our due diligence make sure our pod is healthy you know check it and everything is not fine. Okay. Um we have one of three containers ready. Why why is which container is not happy? and it's the tail scale sidec car. So I want to uh back up a little

bit. So we we don't actually only have the dev container in the pod. We actually also have this tail scale sky tail scale sidecar container. And for those of you unfamiliar, tail scale is kind of like a VPN. It's an internal only mesh network for corporate devices. It allows you to connect your developer environment to other resources on the corporate network that you might need to access

but you don't want to uh make available to the public internet. It does fancy networking magic like configuring network interfaces, installing routes, doing IP tables, setting up what they call magic DNS, and then running a Damon to connect to uh the TailNet, this uh And uh in order to configure network the Linux kernel has you talk to this what's called a char device called DevNet Tune. And

uh if we look at the Kubernetes logs for this container, it says DevNet tune no such follower or directory. So obviously the solution is just pass through DevNet tune to the tail tail scale sidecar is just a little oversight. This was not a thing previously when we weren't in user name spaces and everything will be fine. So we just we just add it to the manifest. Boom.

We get the pod and actually now we're at zero is ready. What is going on here? Um, let's let's take a look at the error code. And oh my goodness, this is saying something about OCI runtime and containerd and mount at ID. I I I don't know any of these words to be honest with you guys. So, let's talk a little bit about how containers get built.

So, this is sort of the full stack all the way from you running coupubectl uh to the uh coupube API down to containerd run c and then finally the Linux kernel itself. And so this passes the container config. Um we've told it hey like pass through devet tune and it adds this little bit of extra things saying ID map it. We'll talk about that in a bit.

And so you know the container config goes down down down to the kublet down to containerd down to run c and finally run c run uh does some sis calls to the Linux kernel and runs the container. Great. Um Claude was like hey this is a fundamental limitation in the entire chain of how this works. you can't really get around this like they are hard coding that

we IDM map everything maybe pick a different route so I was like okay like uh let's let's let's make tail scale work somehow um oh oh sorry the the the reason why we need devet tune is is we pass through this and we say hey like now now you can use it um the the the reason for the error code earlier is because remember how we mentioned

that like we are mapping the UID zero to UID 100,000 outside. Well, DevNet tune outside is owned by UID0. And what is that inside? Actually, that's undefined. And so, uh, the colonel says you're just never allowed to ID map chart devices. Just not you can't do it. And then this is when Claude says like, yeah, you you can't do this. So, um, took a happy little detour

here. This was a scrapped approach. And uh for for this approach we uh or or what I did was like I had rather than putting the tail scale container as a sidecar inside the the dev pod it was just like a sibling pod next to the dev pod and it would do ns enter into it. U I'm going to gloss over this for the sake of time

but feel free to talk to me about it later. Um this actually worked but I felt like it would be a maintenance burden. It was highly complex and had a lot of moving parts and I felt like it would be unfair to sort of just throw this into the hands of the rest of my team to sort of be on call 4. So I said to Claude,

"Hey, let's get creative. What what can we do here?" Um, what if we monkey patch run C? So we did. And so what what this is is when when container D passes the container spec to run C, it passes this as a JSON file. And so we when we pull new nodes into the fleet, we inject a bash script that wraps runc. And then we check if

that JSON file is passing through DevNet tune. And if so, we remove the ID mapping, pass it along, everything is so the real run execute. Everything is happy and this worked. And so boom, we get the pod. Boom. Tail skill is happy. Something's not happy. Let's take a look at what it is. And it is Docker sidecar. So, we're we're at Planet Nyx. Why am I talking

about Docker? Well, um this is this is not about Docker builds, but more about running Docker containers and our um yeah, sorry, I also lied to you guys earlier. We do have yet another sidecar container here. And uh this is where our um product developers who run microservices will run their microservices inside this container or containers inside that container. And uh that container needed to be privileged

and so that actually doesn't work because we're now in a username space. And so we needed to find a different way. Um I went back and forth with claude a lot on this. Um, and in the end, uh, and I tried a lot of things to try to get this to work. And I'm glossing over a lot of details, uh, just because like it was dead end

after dead end after dead end. But overall uh decided hey let's rip out the docker sidecar entirely. What we should do is actually put in uh what's called dockerd rootless which is a way to run docker nested inside of docker or rather to run docker as a rootless user as as unprivileged and then you know they'll run their microservices inside of here. And in order to do

this also uh we sort of got a kickback from our earlier uh thing with tailscale where we needed devet tune because that creates a network interface and so uh with this things were things were good. Uh docker run boom we get it. Awesome. Let's double check that things are healthy. Boom. Okay. Everything is good. Our containers are all healthy and uh we are so back. Let's talk

about rollout and testing. Um, just real quick, just because we've implemented this, we haven't gotten it in front of any users yet. Um, we have this Slack channel of like 20 to 50 users. We said, "Hey, we you can opt into this uh use this different setup that's, you know, we've rewritten the entire Docker stack and tail scale works differently now and we have a run cmon

patch in there and uh but but Nick's build should work and uh they they battle tested that for uh about a month, found some bugs uh especially in the upgrade path and then finally announced it as generally generally available. um it was on by default but then provided a per user opt out where they could self-service be like, "Oh, this isn't working for me." They unblock themselves

and then let us know what we can uh what did not work so we could fix it. Um we had happy users. Here's a quote from Anish right here in the front row of uh we went into Dante's Inferno and uh initially intending to do make Nick's builds work and then actually solved everything else and rewrote the entire Docker stack. Um so what what did this enable?

Obviously the point of this talk Nyx builds work and tail scale did not break and docker did not break and we also unblocked a top level company initiative but that was a little bit of like lucky timing and I can't really talk more about that but that was very lucky and very uh like very good timing. But let's talk about what this felt like. Back to the

interstellar, right? The waves are rising and every twist and turn is sort of like, I don't know when this is going to come to an end. And then in the end, my little robot buddy comes and picks me up and whisks me away. And then we're smooth sailing away, launch off the planet. So I want to leave you with some takeaways. First is as dev tool developers,

things should just work for our users and we have a responsibility to go to great lengths to enable that. And even though Nyx was the tool I talked about today, many dev tools in the ecosystem make the assumption that they're running in a privileged environment, which is often the case when you're in uncontainerized environments like home systems. And so perhaps you've heard about how difficult it is

to get native Docker and Docker working, at least if your outer container is not privileged. So containers do a lot of Linuxy things that make them different from uncontainerized environments. And the pay path for any many many dev tools today assumes uncontainerized devel environments. And so uh beware there be dragons but also check your assumptions if you can about where your tools might be run. Um the

ecosystem support for user name spaces is improving though. For example, in Kubernetes version 1.35, the uh podspec security standards is uh has better support for username spaces in that they don't yell at you when you say procmount unmasked plus username spaces are enabled. And then uh finally, this doesn't seem like something that we would be the first to run into in 2025. like it doesn't seem that

off thebeaten path that our developer environments are in Kubernetes pods and that people want to run Nick's build and it seemed like there was very little chatter on the internet about this and so if you have run into this before I would love to talk to you please come talk to me did you also monkey patch runc we have a booth out there we're a sponsor of

uh of u planet ny so please come to our booth talk about this talk about anything you'd like to talk about And uh yeah, thanks so much. Do we have time for questions? >> Couple minutes. >> Couple of minutes. >> Yeah, this should still work with remote >> What's the max set? How many builds do you allow users to do locally? >> To do locally. Well, we

constrain the uh assertion that each node only has one pod and so I think users can just build as many with as many CPUs as they want and if they sort of brick their pod that is a little bit on them. Um that being said, these are pretty beefy machines and so I think no one really has pushed these to their limits. >> Do they have to

choose to do remote build? builds? I'm gonna have to defer to Bernardo for that one. Um, I don't know if remote builds are sort of uh on by default, but that might just be a process of rollout and they'll eventually be on by default. If they're like opt- in right now, maybe they'll eventually be opt out. Um, remote builds are just so new and for us and

uh yeah, we we we need to just roll it out slowly as with all things. Yeah. So the question is why is IDM map turned on? And when you do host users is false, Kubernetes says IDM map all the mounts. That's it. It's just there's no like and als but don't ID map this now. It's just like if you have this setting enabled that we needed if

you have are using user name spaces, you need to ID map all the mounts. The assumption kind of makes sense, right? because um if you have a mount or or if you have a file system on the host that's owned by UID 10 and then you uh mount that into your username space pod. Well, UID 10 doesn't exist in your uh in your pod. And so you

you need to have that that map to the UID that exists in the pod. So it makes sense when you're thinking of just like file systems, but when you're thinking of like char devices, the colonel says no. And I think this was a little bit of an edge case where apparently no one else had tried it before. >> It works because the device is is readr by

any user, but then the kernel does other checks against like do you own your user name space and so there are still like we're not just breaking security invariance here. But yeah, it's because it's readr by any user. Yeah. >> Any more questions? >> All right. Oh. Uh, let's talk in the back afterward. I think we're at time. >> Yeah. All right. Thank you so much. Uh,

next talk's in five minutes, but we're going to let that person come up here. Has anybody seen Ashley? Let people come up here. Get set up. Okay, we are going to get on with our next talk of the day. Um, we're going to we're going to talk about mesh networking. This is going to be awesome. So, uh, Ashley, take the stage. Thank >> Testing, testing. Okay, we're

live. >> Uh, hi everyone. I'm Ashley and I am a Devril at Netbird. Uh, so admittedly I'm not a NYX expert, but I am a home labber who fell down the rabbit hole and falling down the rabbit hole kind of allowed me to bring Nyx to work and led to me using Nyx to solve what I feel like is one of the hardest problems to solve in

reproducible testing environments, which is networking. Uh, so this is just an idea of what's coming up. First, I'll go over my click moment and how the light bulb came on for me with Nyx. Uh, then I'll go over my two-phase approach to sneaking Nyx in the back door of my company and then I will kind of outline my future plans and wrap everything up. So before we

get started uh I think maybe I should just give an overview about what Netbird is for maybe people who are not familiar with it. So Netbird is an open-source wire guard meshvpn. Uh generally the preferred way is for peers to connect directly with each other but we do have a management server control plane. uh signal handles web RTC signaling and relay is the fallback when peers cannot

successfully negotiate a peer-to-peer connection and the peer-to-peer part is actually what makes the testing so tricky but we'll get to that in a moment. Uh so like many of you probably I ran a home lab and my kind of workflow was docker compose bash scripts and pray and for every new service I ran the same six steps. Sometimes I tried to deceive myself into thinking I would

maintain my anible playbooks that never really lasted. Um but for every service I followed the same six steps and encountered the same six opportunities to forget something or misconfigure something or just mess something up. So at some point I saw the light and I switched to Nyx for my home lab and after a few iterations of my config. I didn't get there immediately, but after a few

iterations, uh, it finally clicked for me. And I think kind of what I can pick from my config that embodies that the most is my service definition. So in my services directory, I have a common function. And the common function basically takes a service definition and wires up everything needed to create a complete service. So when I create a service, I import that common function and I

automatically get a DNS record in my Cloudfare account. Uh obviously that record has a TLS certificate. Uh I get a dashboard entry. Uh I pictured that at the bottom. My dashboard of choice currently is homepage, but other dashboard solutions are available. Um, so an example here is Meie. Uh, if you're not familiar with Mely, Mele is a recipe manager and with one simple import and just enabling

the service basically and giving my common function, the port that the service runs on, I get all of the things I just described. Uh, the function does everything for me. And so I stopped thinking about what steps I needed to set up a service and I started thinking about what a service actually is. I started to think the Nyx way where you define the inputs and the

function handles the how. Given a set of inputs, give me this output. Now I recognize where I am and the people I'm talking to. I know that thinking this way is probably as easy as drinking water for you guys, but I need you to understand how putting the functional paradigm in the context of infrastructure really allowed it to click for me and to have that aha moment

and to have the tools in my arsenal necessary to explain this to other people unwilling or reluctant to learn Uh so I ended up getting hired by Netbird last year in November. Uh partly due to this Nyx energy that I have and I knew my mission immediately and that was to evangelize Nyx and Netbird. I knew that there were many problems at Netbird that could be solved

with Nyx. And I decided to approach this kind of evangelization in two phases. And I'll break down both phases for you now. So phase one was pretty straightforward. I had to bring NetBird's Nyx packages modules in line with the current state of the client and management services. Netbird and their previous NYX evangelists sadly parted ways. rest in peace to a fallen brother. Um, and the modules basically

hadn't been updated since he left. So, they were quite outdated. They're in quite a state. This would make me a first time Nyx packages contributor. And this was both very exciting and terrifying. So, I'm not going to lie, initially I did take some heat from the next packages maintainers after my initial attempt, but in the process I learned a lot and my PRs improved for the better.

My instinct was to just give every upstream flag a typed option. And in my defense, this was how the module did things before. Basically every flag in the NetBird client or the Netbird server had a typed option. So my thinking was well I just need more typed options for the options that are missing from new versions of NetBird. This was very The view the reviewer feedback was

basically your module shouldn't be a second source of truth for your upstream docs and they pointed me to RFC 42 structural settings. Provide escape hatches, not mirrors. You layer your options as environment variables on top of defaults and let Nyx do the rest. This tension between completeness and maintainability is something that you all probably know very well. But for me, I had to learn this the hard

way. So phase one was just packaging, a necessary but relatively routine step. Phase two is where Netbird and Nyx gets truly interesting in my opinion. So, the problem with testing different networking setups is that there are so many different variables and conditions to consider when you're trying to pinpoint an issue. If you have two clients behind easat, then they're probably going to negotiate a peer-to-peer connection, no

problem. But if you put one of those clients behind a restrictive or carrier grade net, then they're probably going to be relayed. Now, if you multiply those variations with different client versions, different server versions, different oss, uh different network conditions, maybe different MTUs, uh then you can kind of imagine the mess of variables that we need to track and test in a reproducible way. Uh so traditional

CI flows would spin up VMs, run the tests and hope that the environment is representative. And so more than actual confidence, all this produces is the daddy of all it runs on my machines. Now the natural reaction might be not of this people in this room probably but of most people in tech would be just docker compose it bro but for netb bird this doesn't work for

a number of reasons one being that netbird on Linux the netbird client on Linux for example uses kernel wire guard not user space wire guard netbird wants to be your DNS client if you're running the client because netbird can do all kind of fancy things with automatically uh redirecting host names to IPs. Um so we needed a proper kernel and we needed a real networking stack. So

we have this kind of cacophony of test scenarios and I realized this kind of problem is kind of exactly what Nyx is good at and what Nyx can solve for me very easily. We have the same outputs for every test given the same inputs. If you describe a network topology as a Nix expression, you can get an identical test environment on every machine every time. And unlike

Docker, VMs run on their own kernel. When Netbird creates a wire guard interface in the VM, it's real. When a packet hits a net, it's a real net filter. uh it's not IP tables, it's not some shared kernel shenanigans. And so every test dimension whether it be NAT type, client version, OS version is a function. And functions as we know So the whole test environment basically can

become a derivation. You can cache it. You can share it across CI runners. You can diff them against another topology. It's a value. It's not a process anymore. So I looked at this problem and said, has Nyx been waiting for this problem or has this problem been waiting for Nyx? And my colleagues, they rolled their eyes thinking, here he goes again. But I knew I was on

to something. Now, a lot of you maybe will have noticed a couple of glaring issues with my perfect solution. So, I'll try and break those down for you. Now uh the first issue is something that I encountered pretty early in my home lab journey and that is that Nyx doesn't really have a way to do VMs declaratively very well. Um, so I went all in on Nyx

modules in my home lab, but sometimes you still need a VM, home assistant, for example. Unless you want to have two billion dependencies, you probably want to run it in a VM and not via the Nyx module. So I noticed pretty quickly in my NYX journey that Nyx was lacking in this regard. And even community-made like libvert rappers didn't feel Nixie enough to me. and solutions like

micronix VM obviously as the name implies only really works for Nyx OSV VMs. So I wrote my own solution. It's just a very simple QMU wrapper. Um so I define my cloud image source. I give the hash uh define a format CPU type memory and give a cloud init config if I want to do that. and it will basically create a Q move service of that VM.

It will automatically download the image. If it needs to be decompressed, it will do that and it will convert it to whatever format I specify. So I had this module already ready in my back pocket. Uh cloud in it support was obviously going to be very helpful for me in this use case. Uh, I had cow support so I could have ephemeral discs for my VMs. Um,

so even though it was built for a home lab use case, uh, with a few relatively minor additions, it would be ready for e ephemeral VM test usage. Problem number two being the next guy. Stop me if you've seen this one before. A problem arises and you think, "Man, Nyx would be awesome for solving this." You raise your hand, grinning from ear to ear, and watch your

colleagues mentally shut down in real time because they already know what you're about to say and they're sick of it. There's no point in building a perfect testing solution if you're the only person who's going to use and maintain it. You need your team to be on board too. And this is why I spent so much time earlier detailing my home lab journey and the click moment

that I had because seeing the functional paradigm build infrastructure was a lot more intuitive for me than seeing it build a string. And it gave me the aha moment I needed to be able to confidently break down the paradigm in an easily digestible and uh in an easily digestible way that just kind of seems to make sense more so than the alternatives. So when I explain Nyx

and this solution in particular to the team, I don't start with syntax. I just start with very simple natural thinking like what are the things what is a test scenario it's nodes it's a topology and it's a few assertions that's really it a node if we drill down further what is that a node is just an OS some software and a network interface a net type what

is that how do we break that down? Well, that that's pretty simple. Uh, a net type is basically just a router with some firewall rules. It's really not that complex if you think about it in this way. And this maps more directly to how you would think about this problem if you were whiteboarding it. The difference is that in Nyx, the whiteboard description is the implementation. If

anything, this is the more natural way to approach this problem. So when it starts clicking for your team members and you say things like same inputs, same results, it runs on my Mac, it runs on your Linux laptop or define what things are, compose them, and the system builds itself. Then congratulations, my friends. You've officially done the impossible. You have made Nyx sound appealing to a nonny

user. So this is what the scenario looks like. I don't know how well you can see it on this screen, but I'll try and explain it for you. Um, it's just YAML basically. So it's easily readable by anyone on the team, even someone who wasn't drawn in by your Nick's Captain America speech. So all I do is define the nodes. I boot them. I run some commands

and assert things work. Then I collect the diagnostics. In this example, client 2's IP gets resolved automatically. The runner handles it. And assert in this case just means this command must exit zero. If it doesn't, the test fails. Artifacts get collected. Notice there's nothing really nyx specific here. All the ny happens. All the next magic happens in the implementation. So your Go engineer can write and read

this. And that's kind of the point. So behind the scenes, Nyx takes that YAML that I just showed you uh and it takes the node definitions and builds a self-contained bundle. Everything is there. the runner, the scenario, the uh, bootable VMs for every node that you've specified. Um, it's all there. Every binary, config file, and kernel modules gets pinned to a hash. So your teammate runs the

next build, gets the exact same store path as you, and gets the exact same result. So that's the deal. The team writes the scenarios in YAML. Nyx handles reproducibility. They don't need to learn Nyx to use it, but they'll start to understand what's happening and become more capable of contributing to the tool if they do. And if you've drawn them in with your explanation, hopefully they will

want to learn next. okay. So, in the future, uh, plans I have to improve this tool and expand it. Uh, I guess number one on my list of priorities would be to have tests run as build steps. Currently, the build kind of spits out an artifact and you have to run that yourself. But if tests were to run as a build step, then the test itself could

become a derivation. we could automatically cache test results which means that basically once a test runs once you know the result you never have to run it again. Um, also, uh, one cool thing I think would be achievable is to have maybe on GitHub, for example, if you have your issues section on your GitHub, uh, comply to a template, then a user fills in the template, a

CI would pass it, it would generate a test scenario, and it would run automatically. And then the result of that scenario would become a early kind of diagnostic signal before an engineer even dives into any code or tries to diagnose the issue themselves. Um there's also maybe a cool community angle here. I try to keep that on myself as a as a devril where you could maybe

have community members submit their own weird exotic networking environments as modules and you could run those. Anyone could run those. You could add them to your list of tests. Um, because we're in the age we're in, I feel like I have to give a mandatory AI related idea. So you could have from sources other than GitHub uh some kind of LLM maybe passing user issues and writing

their own kind of Nix expressions to represent test scenarios that we don't currently have in our repertoire. So yay AI. Uh and I will finish on kind of like a selfish plug note. Obviously, I have developed these new schemes and ways to sneakily make my colleagues accept Nixos, but I could always use some help. if you'd like to join me and solve some of these problems, if

they sound interesting to you, then that could be Thank you for coming to my TED talk. No questions. I can go now. >> I'm joking. >> Yeah, we do have a couple minutes for questions. If you if anybody has questions, feel free to ask. I believe like, am I wrong in thinking this only runs on Nyx OS >> because one of the key requirements was that every

dev, no matter what OS they're running, could spin up these tests. So, yeah, that's why I didn't go Okay, thank you everyone. Have a nice Check. Check check All right, we're like I don't know most of the way through the afternoon. At 4:40 we have lightning talks where we'll have some people do really short stuff like five minutesish around a couple topics because sometimes you just got

a little bit to say or you want to pack a really big punch in a short amount of time. Either way, you can have a lightning talk. If you want to give a lightning talk, we may have a spot for one because somebody didn't show up. I'm not going to tell you who it was because that's mean. But anyway, um Morgan, are you good? All right, >>

this this dude's crazy. Uh and the reason I know this is because last weekend he ran a marathon and then he also runs Nyx and I think that's a lot of work. So, um yeah, but anyway, Morgan's going to talk to us about container stuff and uh here's Morgan. You guys hear me? All right, cool. So my the title of my talk is from Nyx to Kubernetes,

no image layover. Uh the reason it's titled that is because Rock called me and he said, "So there was a spot and we signed you up to talk about Kubernetes at Planet Nyx. What's the title of your talk?" And I said, "I don't know." And he said, "Well, I'll just wait for you to tell me. I'll just I'll be here." And I had just finished booking my

flight here and I was debating whether or not I was going to go to LAX or Burbank. And I ultimately decided to go to Burbank, which was the move. And so thus thus became no image layover because that is the first thing that came to my brain. Uh so what I'm going to talk about is basically how you can run Nyx packages on Kubernetes or like in

a container but without building an OCI image first. Uh last year I saw a talk uh from Anish at anthropic where he talked about Nick Sidecar and that was super cool. Um and then there's also Nick Snapshot and what I'm going to talk about is two like relatively new things. One of which is made by Flux that I I I was part of the team that worked

on. The other is not and it's the one I actually run in my home lab. So it's two really neat solutions to run Nyx packages directly in a container but like without the intermediate step of building an image. But uh first yeah so I'm Morgan Held. I'm a software engineer at Flux. I work on our infrastructure team. Um that is me. Uh and so the layover so

like I I don't have to extol the virtues of packaging with Nyx because it's Planet Nyx but you package with Nyx. That's great. And now you want to like you want to scale it which is great. Maybe you want to run it on Kubernetes. And so you say how am I going to how what what is the way that I package software and ship it to Kubernetes.

Well, you probably reach for Docker tools like you reach for build image or build layered image and and that actually works fine. Like I I use that all the time, but there are some constraints like like build layered image sometimes. Where did my video disappear? There it is. Okay. So like with build layered image occasionally what I'll end up with like a bunch of packages or big

closure and I'll end up with 126 small layers and one giant one. Uh or you know I I'll it's it can be annoying to have to repush repush stuff. Um especially when your images get gigantic but we're using Nyx so like we know everything we need to run software. So why do we have to package it in an intermediate in an intermediate deal? like why do we

have to package it in an OCI image just to unpack it again and still have Nyx store paths like I shouldn't have to do that if I don't want to so that's what we're going to talk about the direct flight from Nyx package to Kubernetes cluster so first I'm going to spend like two minutes on Kubernetes and containers just to kind of give enough context for me

to talk about the rest of it so Kubernetes is a control system what does that mean so my background before I worked in tech stuff for the last 5 years was I was like a controls engineer at an oil refinery which means I worked on like physical control valves and safety systems and sensors. So when I was first introduced to Kubernetes like I was like oh it's

a control system for software. What does that mean? So we could think of like a like a typical example of a manual control system is like if you were sitting in a room and you had a a thermometer and you had a heater and you you like I want to make the room hotter. So I turn it on and I watch the thermometer and it okay it's

a 72 now and I turn it off. And you can kind of think that's analogous to sshing into a box and then installing some package and then like you run it or you enable system dun you're doing it imperatively like you you are intentionally installing software on servers with Kubernetes. So so the the contrast is an automatic control system. So like a thermostat I want it to

be 72 and then the thermostat turns on the heater, turns on the blower, maybe it opens some louvers like it makes it 72 and then it watches it and it keeps it 72. And you can think of that analogous in I I use a like a deployment resource and I say, "Look, I just want to run five copies of this piece of software. Just just do it.

Make make their five. Give it networking. Let there be environment variables. Just just do it. If one disappears, put one back. Just keep it at five for me." So you can think of Kubernetes like that as orchestrating containers. And then what's a container? Uh we probably know, but I'm going to I'm going to zoom in on the parts of a container that are relevant to this talk.

But it's so it's an isolated environment with your software and everything it it needs to run. But part of that is a file system uh the root fs. And so that's what we can actually use Nyx to create that that's what we can compose out of just Nyx packages. So how does containerd create a container? So it you you pull an image as this headset drifts away

from me. Uh you unpack the layers into a content store and then you assemble those layers into a file system. uh and like the act of doing that is what ultimately gives you the file system of the container uh that runs um and it gets executed by runc uh accordance to the like the OCI spec JSON that's generated with like what to mount into it what environment

variables and then like what program you want to start and then the container um but but the file system of a of a container like it's just files like the majority of that entire process of starting the container was just to create the file system and if I I have packaged with Nyx like I I already know all the files I need to run the program I

I didn't need to do extra work still jacking with the headset so why don't I just use them like I can hand the store pads to the container runtime directly and say look why don't I just instantiate the container out of Nick store paths why do I need the OCI image we're going to talk about two routes uh going back to The the metaphor one is a

custom containerd runtime shim which is the flock solution essentially like we can as part of container creation we can like stop we can pull nyx packages into the store and then we can update the spec to just bind mount those into the containers file system as it as it's created. So we we just we sit in the middle and we inject nick store pads in and then

we're done. And then the other method I'm going to talk about is nick CSI which is which is another mechanism creating an ephemeral volume on the side and then attaching that to the container as it starts. So so two routes two pathways to achieve the same objective putting Nyx packages into a container without creating an OCI image. Um for the purposes of this I'm going to I'm

going to just have just some arbitrary example package. It's a it's a Go package. Uh it's called Echoserver. the the specifics aren't important but when I talk about it I'm talking about the specific package. So the first one I want to talk about is the the flock container runtime shim. So before I do that I want to talk for a second about what flux is because like

flocks put on planet nyx I worked for flocks but I think like I was here last year and I had not actually used flocks and I didn't for a little while after so I wasn't actually familiar with the ergonomics. So I'm going to spend like one minute talking about that just again for some context. So, Flux at its core is an is an environment manager. So, like

so like to be able to create a development environment um that that you can kind of take with your codebase and have a consistent set of tools and then share it with other people. It uses NYX under the hood, but it's configured using like a like you um it's configured using like typical package manager ergonomics like install, search, upgrade, whatever. And then it ultimately like edits a

tommo file which is carried with a lock file. So what what a the flock development environment workflow looks like is like let's say I want to use go I want to build that go go application I want go I don't have go so I in initialize a flock environment and I install some stuff I install go tooling and what what this generates is a is a manifest

which is basically just a definition of the environment of all the packages I want to put in it. Um and then from there I run flax activate which drops me into uh a subshell. You can think of it analogous to running next develop uh which has all those packages on path and now I have access to go and I can do go version. So that's that is

that's the the flux development environment which is necessary for the next slide. So flux's imageless Kubernetes essentially presupposes that what if I had took the development environment concept but I just put my application in it like I the only thing I had installed was the the echo server. So then what I get is a is a declarative and composable environment definition that I can then use to

just say make the container this thing like I I I have that I have that that Toml manifest uh and that corresponds to a a particular Nyx package which has like an eight package closure and so if I if I send this metadata to it'll just pull those packages mount them into the mount them into the container and then it starts up and can run. Um so

let's talk for a quick sec just about the operating principle. So like you're Kubernetes and you say hey like hey containerd I need to do a thing I would like to run this package please or that I would like to run this container please. Um so and so the operating principle here is that you configure containerd to call flux's runtime shim and it says hey like a

call came in we got to create this container. This is the flux environment we want to use to materialize it. Um, and then what the flux shim does is it is it uses that metadata to pull the packages and then once those packages like once those store pads are on the node itself, it adds those store pads to the container spec. So once that spec is passed

to runc when the container comes up, those store paths are already mounted inside of it. And so what that means is like like the flock chim pulled pulled the store paths onto the node or built or built them if necessary and then they're they're just there. So it didn't matter what image I used, the the container already contains my packages. So in practice, what does that look

like? So again, there's the the the manifest that defines my what my container will become. And then on the Kubernetes side, this is what the podspec would look like. And the key here is the runtime class name argument. So so Kubernetes basically has something called a runtime class, which is what it tells what runtime it tells containerd to use. And so by using this we can like

we can have some pods in the cluster that use the flock runtime and some pods that don't and you get you get control to mix and match. And then we use an annotation to uh to define what environment to run which corresponds to the manifest f the the you know manifest file on the left. So basically I define my environment in the manifest file. I say use

the flux runtime. I pass the name of it and then those store paths are pulled onto the node and they're materialized into the container. Uh as far as as far as an image goes. So, so a container has to have some image like that's part of the spec. Um, we supply like a 48 byt empty image. Um, but like any scratch image would work or even even

a non-scratch image that the key is it's just by mounting those store paths into the container, right? So like you can start with nothing. You can start with an empty an empty container and the file system that's materialized by the runtime shim is enough for the container to start. Um, and then you just run the command and it rolls. Um, so that's so that's the flux shim.

Um, and it is one of the routes to get to running a container with Nick store paths without building an OCI image. The other one I'm going to talk about is Nick CSI, which is also super cool. I did not work on this one, but like I said, this is the one I use at my home lab. Um, so first, what is a CSI driver? It is

not the thing from the TV show. Uh, CSI stands for container storage interface. Basically, it is it is the the the type of plug-in that is used to provide storage to uh to Kubernetes pods. So, if you've ever used Kubernetes on like a cloud provider like AWS or GCP, so the mechanism that mounts EBS volumes into pods uh is their CSI driver. So, this enters at the

same part of the ecosystem as that. Um, and so basically you just reference the volume and and attributes about it in the podspec and then Kubernetes calls it at the right time and makes it available to And so Nyx CSI is a is a is a CSI driver that creates ephemeral volumes from Nyx packages. So essentially as part of the podspec you say hey pro attach a

volume at slashnix to the container and give it like a flake you can e write an expression or or like a a flake reference or or just explicit store paths and say materialize these things inside the volume and then attach it to the container creation time. And then what Nick CSI does is it builds the closure and it mounts it inside at /nix inside the container. And

so what you get with Nick CSI ultimately like the thing it is doing in the Kubernetes ecosystem is not weird like it is just a volume. Um it it it's ultimately using standard Kubernetes primitives to inject this into the container. Um and it so so what that really means is it runs on any cluster um because it's entering the Kubernetes ecosystem at like a normal place like

it is it it is just something you you just manifest you apply it's just something you install in the cluster. Um, Nix CSI is is pure Nyx in the sense that whereas the Flock solution kind of offers that nice manifest abstraction. Um, this is presupposes that you've already packaged your software with Nyx. Um, which like I say, great for me. The other thing Nick CSI does that

is super cool is it includes a cluster level cache for store paths. So anytime you either build something or pull something onto a a pod in your cluster, it it gets put in this separate binary cache and then any future pulls will use that binary cache of the store paths available. And so what that means a is that is that it can be a lot faster after

first first pull than conventional OCI images because they're already there local to the cluster. Like it's it's already in a it's already in in a in another container with a persistent volume attached to it. But also, you get some immunity to upstream disruptions. Like if you're pulling if you're pulling from some upstream and it disappears or it goes down or it whatever, somebody deletes an S3 bucket

or changes a policy, your stuff is still going to work because it's going to check your own binary cache first. Um, which I think is a big advantage, especially like when I when I when I run a cluster that's like big, uh, I try to aggressively scale my nodes up and down, like I I do not want compute active that I am not using. Um, and so

the downside of that is that I lose the benefit of a of a local next door or even OCI layer caching because like I am deleting nodes a lot because I do not want to be paying AWS for compute that I am not using. But having a cluster level cache means I get to maintain that same kind of speed and reliability while like not paying them for

for nodes I'm not using. Um, so just like a a quick quick run through of how Nick CSI works. So you can see like like the bottom line is creating the container and that that is completely destroyed. It's not interrupted by the nick CSI flow. The only thing that happens at the end is it waits. So basically cubid cubid uh the the cublet basically says I'm going

to create a container but hang on cuz like I'm calling this other guy and he's going to create a volume for you. So like just wait till it's ready and attached before you start. Um and then in parallel cubid calls the the CSI driver. The CHI driver either builds or just pulls those store paths and then puts them into the the ephemeral volume and then once that's

ready, it's made available to the container and the container starts with uh with the volume attached and those store pads available. So just a quick look at a podspec for Nick CSI. Um similar deal it a lightweight image a scratch image would work fine. This is what the author recommends just really static. Um but in general like it's again it's just attaching a volume at slnix. So

you can pretty much do whatever and then the way you attach it is you specify a volume and in the volume you you can specify like in this case I'm using like the default package of a flake which contains that echoserver package from before. Um but from there it uh it will it'll pull that package onto the node and it will be ready to start up and

run. Uh and you can just say the command and it runs. So we can look real quick at a comparison of the of the two just to kind of sum it up. But basically there are these are two ways to do the same thing to be able to materialize a container with nick store paths without building an OCI image ahead of time. Um with flocks you kind

of get that really nice uh kind of abstraction of the of the of the environment. So you can kind of almost think of that as the same mental picture as as a as a docker file like like here is an atomic unit of of application that I'm going to ship and run in the container. Um the uh the compare that to Nix CSI where it it uses

Nyx expressions. The the flux mechanism is modifying the container spec as it's being created. So it's a little bit of a cleaner interface because like you're just using an annotation. That's that's basically it compared to Nix CSI where you have to actually express that volume and then like like you have to be able to add that into your podspec. Um the other big contrast here is installation.

So because flocks operates at a much lower level of the stack, you have to be able to install both flocks and nyx and the shim on your nodes. Um which like is not a big deal in some cases like EKS like u like AWS it's very easy with user data to be able to accomplish this. Um but on some things like GCP it can be uh either

very difficult or impossible. So it really just depends on how much access you have to your nodes um on on on uh at that case. Whereas nick CSI like it it it enters at the cluster level like its manifests you are applying and it just runs. So um so it just kind of depends on your application. Um and that's about it. That that's what I got. Uh

if either of them sound cool, I encourage you to check them both out. Um Lily Carl and I, the author of Nick CSI, like I reached out to him before this and I was like, "Hey, I'm gonna talk about your thing in a talk. Are you are you cool with that?" And since then, we've been like we've been like um conspiring on like a third way on

a third route using uh using using a a different mechanism that I think might be like a super cool happy medium between easy manifests, but also but also like usable everywhere. So he and I are like doing parallel implementations of a thing that maybe we'll talk about at a future conference. Um but um but yeah that that's basically what I have. I think I can I can

probably have time >> uh for the Nyx CSI the cluster level uh caching. Mhm. >> How does that work? >> So basically it you it runs like a single stateful set with one with a volume attached and then and then um the CSI so the CSI driver manifest as a as a damon set that runs on every node and it's configured with that as a substitutor. And

so anytime it tries to pull something it'll check there and then conversely it's also configured on on like pull of a new one to push it back over there. So, so because the the the Damon set is the thing that's doing the pulling and the push and the materialization, then it can also just push it to the binary cache. >> So, do you usually like the stateful

set, it's using like any >> IDBs back? Yeah. Yeah. I mean, it it you could probably do something cute, but that I mean all things considered, if you lost it, it wasn't a huge deal, you know? It if it goes down, it just doesn't do that, right? It's not a loss of availability, so it's not a huge deal. But yeah, just EBS disc. there might be a

fifth way. Uh take a look at Nick's snapshot. >> Yeah. So snapshot snapshot like that's one of the OGs, right? like uh like I I have another talk at at scale uh that that I signed up for myself and was not conscripted into giving and and that one I I talk about snapshot as a as a third way um to do this and like and like one

of the one of the cool things snapshot can do that that these can kind of do is the the intermixing of OCI layers and store paths at the same time because they kind of compose a stub image and you can kind of pick what you want and so like so if you're trying to do that it's really neat like Nick Snapshot is a is a third thing

I think I had it in here at one point and I took it out cuz I was worried about time which apparently not but uh but yes Nick Snapshot is a is a third one that that folks can look at. Thank you. >> So you're not using Nyx to write your manifests. You're writing them by hand stone ages. >> Are we talking about are we talking about

the flux manifests? So generally you'd use those using the like the install install verb. So, so what what what Flock does for those manifests is it Oh, you were talking about pod specs. You're talking about pod manifests. So, um one thing that the author of Nick CSI sent me that he wanted me to show and again I was worried I didn't have time is is he's got

a so there's cubix and easy cubit and he has like a fork that's easy cubenix which is which is a way to write like like accuracy checked manifests. Um me personally in my home lab I use um I use argo cd and I use nixity with it. So I so which gives me this really like Nixity switch verb where I can basically treat it like a Nixx

OS configuration. So if so like no I don't I don't write manifest by hand like a savage. Uh but um but uh but yeah so there's another thing I should have like uh like easy cubenix and nixity if you use Argo you can come talk to me after and I can gush and show you my home lab config if you want. >> Hi nice a nice talk.

uh for the flocks uh method uh how does garbage collection >> Um there's there's like so honestly suboptimally there is scheduled ston shaking his head. So, so there's automatic garbage collection scheduled. Um, and then there's like the the it's configured with the upper limit, right, of the like don't let the disc be more than such percent full and then it starts to GC. And so like it

will GC. Um, I would say that like the the end user of it does not have good enough control over it like in that implementation. So there is an automatic garbage collection scheduled. Um, but I would say that like if we worked on it more like like if and when we work on it more, I would say that like tunable garbage collection is probably like There's also

an LRU thing underneath it. If it's been activated recently, it won't garbage collect and all that. There there is more than that under. >> Yeah. And um and and also I mean like like you you you know I always run not I have run a skew of node ephemeral disc uh like like filling up so many times that I I like monitor it now. I don't know

any cluster I do I'm watching node ephemeral disc. So I think there was a bug in Argo one time that would like flood my disc but that's a different problem. So it's a it's a good question like uh like I wish it were more tunable. For people with really big image uh the rage is like uh seekable tares and stuff like that. Do you think it would

be possible to what would what would you have some plans if somebody has some really huge images with really big derivations you know ML stuff and so on. uh would what could we imagine to address that the same way like to have progressive loging of that? >> I I I mean I think large large I don't think size would be a really a like I think this

solves an image about large packages and closures. >> The point is more like if you have a 15 gigs image uh to have like magic tricks to start running the container before the image has been pulled entirely. >> That's that's a really interesting question. I do not have any. I've not put much thought into that, nor do I have a trick. But but that that is a

really interesting question. If you could kind of pre-start like that that that actually be a really cool thing to investigate. Like I like that idea. I'm going to like write that down at the the booth after and go home and think about All right. At 4:40 we have lightning talks in here, so please come back. We'll have some a few short uh kind of bursts on some

different topics. So uh we'll they'll pick that up at 4:40. Uh we have about 15 minutes until then. So uh go say hi to your sponsors please. I would they would like it. I would like it. Everybody would like it except for maybe you. If anybody wants to talk about Nyx or and Kubernetes, come to the booth. All right, we're about to get started with lightning talks.

Before that, um, just wanted to thank the sponsors one more time. Um, that's ExpressVPN, PDP partners. Uh, >> okay. How do you pronounce this? Anyone SoCal Nug Scalnug Snug. I'm so lost. There's some kind of user group and they're a sponsor and we love them. Um anthropic. Uh so anyway, thank you very much to to our sponsors, the planetary sponsors that literally none of this would be

possible without them. And I I will say that again tomorrow at some point because it's my job, but also is true. So I'm not even lying. um which is, you know, a feature. So, uh anyway, we're going to do some lighting talks. So, we're going to start with some stuff about DJX Sparks. And he even has a piece of hardware up there. It's really cool. You should

go check it out. Put your hands all over it. Get fingerprints on it. It's great. Not too many fingerprints. Okay. You can all hear me. Okay. Good. All right. Um I'm Graeme. I'm here to talk about a small hobby project I've done recently, which is Nixos on the DGX Spark. So, what is the DGX Spark? Well, it's one of these guys. Um, it's uh an NVIDIA desktop

sort of AI workstation. It's got a GB10 GPU in it. So, a Blackwell era GPU, which is relatively up to date and quite powerful. Um, sorry. It's got 20 ARM cores. Um, and it has 128 gigs of unified memory. So, the cool thing about this is you can use it as CPU memory or GPU memory. And you can uh you know you can use if if you're

not using it as CPU memory you can use it all as GPU memory which is quite useful. So by default it comes with this Nvidia DGXOS which is an Auntu derivative but we want Nixos obviously. Um okay so how do we get Nixos? Well this is a short talk. So just to cut a long story short I got it working. Um and there's a repo here. Um

so you can look at it if you want but basically it's got a flake in it. It runs. It has a an output that builds a USB boot image. It has a Nixos module that you can include in your config. And it also has a template so you can make your own Nixos config if you want. Sorry, my headset's falling off. Um, so it the I guess

the main difficult part was building this custom Nvidia kernel. Um, so Nvidia has their own GitHub repo with their own kernel fork with a load of config options. So you need to do that and got that working. and it also does all the various drivers and Nixos options to make it work. And it also supports this thing called DGX Spark Playbooks, which is actually the main thing

I want to talk about in this talk. Um, so what are DJX Spark playbooks? Well, they're these kind of recipe things that you uh let you do kind of cool things on your DJX Spark. So things like run a VLM instance to serve an LLM or uh fine-tune a model in PyTorch or generate images with comfy UI that kind of thing. So there there's a lot of

good stuff there. There's a lot of interesting things to play with. Um but there is a problem. Um what is the problem? So warning there's a bit of disturbing content coming up at least for this this audience. Um so yeah the coin out. Um, so the problem with the playbooks is that it's they basically have a lot of things like this. So they're like, oh, check all

your prerequisites. So what versions of stuff do you have or install this VM with pip or install this other thing with UV or um set all these environment variables, make this directory, chimod this thing, run this container. It's all kind of complicated. Uh, or download this random stuff off the internet or run this stuff off the internet. um uh etc., etc. So, not very nice if you're

in a Nyx kind of world, right? Also, it's not only the playbook. So, on the forums, there's lots of like, oh, I tried to install this thing. It doesn't work. Here's my recipe for getting this thing to work, which is like five pages long. Uh, here's another one. Here's another one. So, there's a lot of work going into this. It's not ideal, right? So, obviously, this is

Planet Nix. So, we know a better way. Um, and the better way is to use Nyx. Um, so for in this repo that I mentioned before, we've got a flake and it exports some dev shells and some apps which are basically the equivalent of these playbooks. So to run the VLM one, all you do is nyx develop VLM or the comi one. Very similar. So it's um

they're all defined in the in the flake they're using packages from nyx packages and also nixifi.ai which is quite a nice project for this kind of um there's one command to run them and set them up and everyone gets the exact same thing, right? Which is the beauty of Nyx. Um so that's much better in my opinion. Um so in terms of pros and cons for Nyx,

um or advantages, disadvantages. So on the plus side, I think the thing I found which I was surprised at is the Nyx Nix packages CUDA ARM ecosystem is actually pretty healthy and I didn't have to do that much to fix things to get it all the stuff to work. Um, building the custom kernel for Nvidia is actually quite easy as well with Nyx Nyx OS. Um, this

write once works for everyone thing is obviously a big thing with this this kind of use case and flakes are actually a really good interface for this because you can export them as apps and dev shells and you can easily introspect what's available and it works really well. In terms of rough edges, I think you know this you can't really get cached CUDA builds of ARM stuff

at the moment. We would like to fix that. A lot of us are working on that. Um I think we we'll fix that at some point. Um the freshness of packages is a bit uneven. Some things are quite up to date. Some things are quite old and you have to sort of pack it around. Um I think ARM Linux is actually not the most used sort of

target, you know. So some things just broken but actually not that much really. Um and we don't have any test hardware for the community uh with GPUs like this one like an ARM uh attached GPU. So I think the main thing I wanted to say in this talk is not really about how I got this working but it's it's that there's an opportunity. So um I think

yeah ship shipping an AI package ecosystem like this is really difficult. There's a huge amount of stuff in it. They all don't work together very well. It's really hard to to make it work. But Nyx is actually really good at this. Um and so this DJX Spark is actually a really good use case I'd say for Nyx. like if we were to use nyx flakes and expressions

instead of those recipes, I think everyone would have a much better time. Um because they're reproducible, they hide all the complexity. Um the out of the box experience for users is much better. Uh the support burden is much lower for you know to to you know you don't really have these reproducibility problems anymore. So I guess my pitch if there's any I don't know if there's any

video folks in the room but I really think they should use this. Yeah. And uh that's it. Thanks. And thanks especially to the next cuda team. >> Do we have time for questions >> to the next lightning talk? If you have questions, um, you can find them in the hallway, but we got to keep we got to keep on the on the pace here for the All

right. Who got in an airplane to get Like half the room. All right. Who got on an airplane from a different continent to get here? All right. Thanks. Yeah, that's cool. So There was no value in that. I was just curious. So Can you hear me? All right. Great. Hey everyone. Um, a bit of improvised demo. So I'm going to present the dev 2 um which is

kind of like an alternative interface to to uh an alternative interface to Nyx. Um we all know flakes but we've been building this for the last three years. Um so I'm going to show you how it like some of the main stuff we've been working on. So this is kind of like uh a Rust FFI interface to Nyx and it's trying to show like what is it

building querying uh behind the scenes and it's interactive so you can go like up and down in this case you can go and see what is it building what are the logs and then it activates uh a nickshell or developer environment so I'm going to show you now cumbersome but all right. Uh all right um so a lot of the times people use deer env with nyx

with uh nyx and one of the cool features is re reloading shell. So we've built something that we call dev reload where at the bottom you see like a status line if you edit dev.nix techniques it will automatically reload your environment in the background and then you press Ctrl AltR to activate that um so it doesn't bother you a lot of times you will see the end

of like building and you cannot do your work and you know if that takes a lot of time uh it's really annoying um so this is a short demo of of how that works um and you can also press Ctrl Alt D to kind of disable it if it's building too much which also unloads a lot of cool features in the future but I'm going to see

that in the next versions. Uh all right let's go to instant. So we've also developed uh an evil cache for nyx that is kind of custom to dev. So if you run dev test it takes about four seconds to kind of evaluate and do the whole thing and then second time it takes like 150 milliseconds because it's using all the uh evaluation caches and it's kind of

incremental. So if something changes later on the older stuff will still be around. Um so that's very requested feature on especially on Um all right two more that we've built a process manager. We used to use process compose which probably a lot of people know. Um but because we want to do really cool stuff we had to integrate our own. And here you can see three processes

coming up. Well now it's two one more should come later on. And you can define different probes like you know TCP port is available or custom commands. Um, now it's going through three different processes, looking at logs, has like an expand mode to see the logs. Um, and it supports kind of cool stuff like systemd socket activation and things like that that you would expect that are

kind of like now user land. Um um under the hood we're using task which is like an impure DAG and processes are like a type of a task that you can then spawn and form dependencies and so on. one of the cool features we also have is port allocation. So if you use coding agents it will allocate a port. So you can spawn like uh different environments

of the same project and then it will try to allocate a port so you don't get port conflicts which is pretty common now uh these days with different projects although we do recommend using socket activation from uh systemd um and the next thing we've built is called secret spec. So it's kind of like a specification for secrets which is integrated into dev. Um so it will in

the beginning ask you if some of the secrets are missing and that integrates with like password manager so keying or you know whatever password manager you use so you don't have secrets like laying around but they come exactly from the password manager into dev environment. Um and yeah it has different backends so you can implement whatever secrets provider secrets provider you you want to use. And that's

pretty much it for the demo. You can go to dev.sh and and look at the 2.0 blog post if you want to know more. But yeah, that's that's it. Thank you. Awesome. Good stuff. All right, we have one more. Are you Are you ready? All right. I mean, we can do improv if you want. Does anybody else want to do a lightning I'll just throw up some

slides. Somebody else and give a presentation. It'll be So, what's everybody doing tonight? Sleeping. Okay, that's a good plan. I recommend it at least once a night. There's there somebody's going to a flock event. I should figure out what that is. I don't even know what that is. >> No, seriously. I don't know what that I don't know. I just do stuff. I just work here, man.

We're having the best time. All right. Who in here has a commit inside of Nyx packages? A commit on Nyx packages of any kind? All right. Cool. It's like almost half the room. >> I'm running out. I'm running out of I'm running out of things to ask. say the thing I was gonna say after >> just while Leonard's finishing getting set up, which is it's all good.

Um, I just wanted to take a moment to say that lightning talks are actually deceptively hard. Everyone thinks that five minutes is so easy. And when you actually time what goes into five minutes, it's not a lot. So, I thank you so much to all of our Lightning Talk speakers who are coming up today and giving a presentation. >> And thank you to everyone here that's been

watching so far. It's not a goodbye yet. We do have one more speaker. I just wanted to take this moment to call it >> I'm a bit messed up on this setup, so let me fix this. >> I can't see anything on my screen now. >> Okay. Extended main display. >> this is so weird. It works earlier. Yeah, I'll try that. Sorry. >> It was working. >>

All right. Who's got a trivia question? >> Okay. You know what? >> What's the weirdest package you've ever found in Nick's packages and and you used it and you liked it and you thought it was useful? What's the weirdest qu What's the weirdest package you found in Nick's packages that was useful? >> I cannot see my screen. Yes. Main display mirror. >> I was playing with disco

now the other day, which is like this thing that looks at your disc and then makes you like a 2e interface of how much storage you're using where. Um, turns out that the Nyx store the nyx directory uses up a lot of space. Who knew? Uh, but also if you have like LLM studio or LM Studio like that folder uses a whole lot. Um, so I don't

know the whole model thing. Anyway, we'll get into that later. We're like going to have the real talk now. I'll shut up. It's all you. Thanks. Uh, I apologize for the technical issue. It always happens unexpectedly. Okay. U my talk is about um tracking next packages pull requests. So when is the fix available? Okay, you you saw a pull request is merged in next packages and then

you try to do uh next flake update but it just not appear. Why? Okay, this talk is about that. Now the problem that I when I first started with Nyx, I always face this problem. You know whenever I want to see a next package pull request is merged and I try to type it doesn't appear the fix is not available yet. What is missing here? Well it

turns out that uh if we uh read through the contributing guideline uh it typically uh there's two deployment paths uh available. So typically the first one is the fast lane. So the fast lane would mean that if the pull request is not affecting uh a lot of rebuilds which means like in this criteria it's less than 500 rebuilds it would the merge request uh would directly delivered

to the master branch and be made available to the next channel uh in for example uh next OS- unstable. So the other one is the slow lane which uh if the pull request is modifying or affects more than a thousand or more rebuilds of other packages first of all it would go to staging and then go to staging next and then remerge it in the this small

loops and after that if if there's uh it's it's good enough uh shape it will continue on merging to master and made available to the next channel. So why we need the slow lane? So for example if uh packages like uh go python gcc has been updated it would affect massive amount of rebuilds towards all the packages that is depending on it. So this would also mean

that it has uh a long rebuild process time. It takes up a lot of uh lengthy uh process on the hydra the next uh CI uh boot system and also if things are not working properly it also have uh higher risk of breakage throughout the whole uh ecosystem. So that's why Nyx have uh come up with this solution. So the batch it in a staging branch and

also using staging next as a as a tiny feedback loop there. So what can help help us to identify the the uh after merge results? First of all we can actually uh see the labels on each pull request. So for example here we have uh 10.rebu Linux uh 5,000 plus. So which means that uh it affects 5,000 or more uh packages uh reboots. So it would definitely

go to staging the slow lane. If it's uh tiny changes and doesn't affect much of the other packages, it the fix will be uh merged to master and be made available more quickly. And here you can see other labels like security uh security which would mean that this pull request will be backported to the last uh stable versions. So think the labels as your pin uh routes

on the map. Uh this is a real example. Uh in this p request 480 465 it's updating go to 1256. So it has these two labels security and rebuild Linux uh 5000 plus. So what does it mean? It shows us that after this pull request is merged, it would definitely go to the slow lane which would take the staging and the slow lane before it is made

available to your next uh channel until it's delivered to your system. And another example uh how we can um identify that the fix is available or not. We can use uh g or github command to actually uh compare the merge commit and to to check the status of it. So uh in this example if we try to run this uh command so if it's ahead or identical

on your target branch then it means that it is available. So a quick reference if the reboots is 500 and less it's fast uh go to the master branch and if it's a thousand or more it would definitely go to the staging branch and it will maybe take weeks before it's made available to you even though it is already merged uh in the pull request. A quick

summary. So if a pull request merge it doesn't mean that the fix is available immediately. Different pull requests follow different paths and try to read the labels. They do give you a good uh time frame on when the fix will be made available on the target uh release channel. uh if you see that staging which means that uh it would take some time before the fix can

be propagated to the more uh stable branches and um you can use uh the g or github commands to track the status of the merge commit uh in each and specific branches. Um yep I think that's all for me. Thank Yes. Any questions or No, no questions. want, but I want to get out of here at some point. Uh, no, it's all good. Uh, thank you very

much for that. I I know that I've been confused by that, so I'll I'll take that. My first commit on to Nick's packages took like three days for me to see because it was a Ruby change. All right. So, thank you very much. This concludes the primary track today. Uh, now you're up, now it's up to the user generated track, and that means you get to pick

your own adventures. Um, we will start tomorrow at 10 a.m. in here again. And thank you very much. Uh, the full scale conference also starts tomorrow, which is in the other building. You know, there's all sorts of stuff going on. Be the community you want to be. Associate with awesome people, have fun, share, all those things. So, is there anything else needed? Does anybody else have anything?

Any closing remarks? Did anybody learn anything today? I learned a few things today. It's great. All right. All right. Well, have a great evening, y'all. And uh you can hang out in the hallways for a little while or whatever, and we'll take it from there.