Room 106 Sunday Mar. 08 - SCaLE 23x

In this talk, Justin Garrison discusses the concept of immutable Linux and its various implementations through different distributions. He highlights his extensive experience with immutable Linux distros over the last decade and explains the core idea of immutability, which is that once a system is deployed, it should not change. Justin explores several approaches to achieving this, including CoreOS, Flatcar, Chyros, and Talos, detailing their unique mechanisms like overlay filesystems, partition management, and handling updates. He emphasizes that while these systems aim for immutability, they often include ways to handle changes and customizations, which can complicate the notion of being truly immutable. The talk provides a candid examination of challenges faced in using these systems and practical advice based on real-world use cases.

Yeah, we're good. >> All right, we are just about at time. So, welcome to the systems and infrastructure track here at scale. We are here welcoming Justin Garrison from Cedaro Labs to talk about the immutable Linux, the state of immutable Linux. As you can see, couple notes. Uh we are taking questions throughout the talk. Justin's comfortable doing that. So if you have a question, please raise your

hand. I will try to run the microphone back to you. And there's props. There's cool props. >> We're getting it. >> Justin's gonna have a good time. So let's give a very warm welcome to Justin. [applause] Everyone having a good scale? >> Might change now. So this we have like what 25 people in here, which is about 24 more people than I expected. So thanks for coming.

It's a Sunday morning. Um yeah, I'm Justin. I do work at one of the vendors that create a immutable Linux intro, but this is not a thing for what I'm what we do. Um, I put ours at the very end, so if I don't get to it, you know, it's not a sales pitch. I'm going to just try to cover as much data and stuff about the

other ones as possible. And, uh, while I try not to hold back my opinions about things, I've been doing this for a little while. Um, I I intend to throw no shade against any other projects or developers or maintainers about these things. Everyone builds their things in their own constraints and they have their own ideas and their own tools and that's how we get further in life

in a lot of ways. So, uh I've been using immutable Linux distros for oh gosh uh more than a decade. Um but uh yeah, so we're just going to kind of go through some of it and and just just talk about it. And um the front line here, you might be in the splash zone. Just FYI, I do have some props. So fair fair warning. Um, so

this is the thing that like I like every marketing is like we're immutable, so that's good, right? And it's just like what is what is immutable? What is what do people expect from a dro that says it's immutable? Any any guess of what would you say if someone says this is a mutable dro? What does that mean you can or cannot do? >> Read only. Right? Like

you can't write it. You can't modify to it. You can't change the dro. And that should give you some asurances. There's some things about, oh, if I can't write something to it, then a virus can't write to it, right? Like that's kind of the goal here of what we're trying to do. And then also, we have some assurance that the thing that we deployed in production is

the thing that we built in in CI/CD, something like that, right? So, it's like, oh, that I can trust these things might be the same because it shouldn't change in production. That's it'd be mutable if I did that. And I'm here to tell you they're all lies. [laughter] Um, unfortunately, Linux isn't designed to be immutable. It was never designed to start up and run and not change

anything, right? Like we have to put things in memory. We have to run things on CPUs. We put log files for debugging. Those are all mutations. Those are things that mutate the world. And so, anytime someone says we are 100% immutable, they are lying to you. They're trying to get you to a goal of maybe this is better than what you used to do in most cases.

And if we look at it in that framing of like well I used to go on every server and run yum updates and yum upgrade and then I didn't know what happened like if it broke maybe it was in some state but like hopefully it got rid of the security whatever like that's that cycle is what this is mostly changing the the thing that we used to

just like change the whole world in place is kind of what we're trying to get to here. So the immutability side of it is is fungeible in it's mutable that definition. I'll say that. Um oh and also 10 years of working in big enterprises has taught me that to hate slides. I absolutely hate making slides. Mostly because every company that I worked at required lawyers to review

my slides. So I'm sorry that these are like the lowest effort slides you're ever going to get. And I try to do as much possible in live demos because yeah. Um, so yeah, this is these are my my design skills after 10 years in corporate America. Um, mutable immutable dros need some way to run software and install software and they all have different ways of doing that.

If we're looking at strictly the server side of things, a lot of the world has settled on containers as being the thing, meaning specifically OCI, Docker containers, whether that's Kubernetes or whatever, you run a container and that's the mutable bit. like you ship a a tar file with lipstick to a server and then it runs and that's that's the thing that we're doing but that's not limited

to all immutable dros. Um flatpack is is coming around as as a more people are aware of it. I was using flatpack when it used to be called XDG app. If anyone remembers when engineers used to get used to name things um it was called XDG app now it's called flatpack. This is what you're getting in more of the like desktop class like hey I want to

run a a guey app like on a basite Linux or you know like a a steam deck or something where it's more I'd say user friendly but it is solely focused on a UI that has you know a person clicking around. Do not try to run command line utilities from flatpack. You will have a bad time. [laughter] It's not designed for that. Uh snaps are exactly the

opposite where snaps were designed for services. For some reason, iuntu decided that they didn't like regular OCI containers. They wanted to own the stack and they just kind of did their own thing. And Snap Classics really sucked and they were invasive. And Snap new ones don't have as many features. And now they also run guey apps. I have no idea what Auntu is doing with snaps. But

it's really confusing. Uh as someone who has stopped using Auntu mainly because of their persistence on everything being a snap and I I did not like that and I I moved on, but you do you. App image is actually one of my favorite uh portable packaging things. Um it's it's guey focused. It's something that you would run on again I'm running um Ublue or not as man

what's it called now? Fedora silver blue um blue fin. That is the one. They're all the same. I'm running blue fin on this laptop. And I like running app images. I also have flat packs. App images are basically like what if we shipped you software in a CD but that CD was digital. That's literally what it does is it has like a loop back interface that mounts

the software and it uses Fuse to like here you go, here's a CD and now you can run software from it and it's that's all it is really. It's like a live CD. Um, I like it just because of the portability. There's no Damon requirement. It's just portable, but it doesn't have all the integrations. Sometimes things break like Nvidia drivers. That's hard to package because there are

things on the host that need to be in the application. Same thing for containers, flat packs, all that stuff. They all have those same problems. There are things on the host you need to match in your container. Uh, Brew has been one of my favorites as a as a long time. That's the most ridiculous worst package manager I've ever seen in my life to oh my gosh,

it has all the software. It has it up to date. U for probably five years, the ma the vast majority of my open source contributions was to put Linux packages into Brew [laughter] because I was like, I want this tool. I'm going to put it in Homebrew so I can use it. Uh and and I converted some companies to do the same. I said, "Hey, you should

package like AWS. You should package this as a Brew." Um they said no. And I said, "Forget you. Here you go." Uh I did it myself. Um so there are things that I love about Brew. It is it is the best worst packager I've ever used. It's all based in Ruby. It's it's getting faster, but it's still like let's just duplicate Python 30 times on your file

system. This is cheap before 2026, right? [laughter] In the in the era when disk and RAM was cheap. This is fine. we can just duplicate all that every time and you just have a whole file system for every package and it's okay, right? There's not a lot of sharing there. But Brew is really easy. Um, systemd uh cisex is again engineers naming things and putting sex in

a command line tool and not even thinking twice about it. It's just every time I look at it, I'm like, you can tell an engineer did this. And no marketing person ever read this as a let's go out with that. That's fine. Um, these are system extensions uh specifically for systemd. These are just like raw like volumes like snapshot a drive and like mount it anywhere on

your file system and like systemd will manage it. As someone who's carried a pager for a long time that scares the hell out of me. [laughter] That is one of those things I'm like there again you have to match things with what's on the host, what is in the kernel, what's in the application. Like just take a raw volume and throw it anywhere you want and then

load libraries from it is really scary to me. Um, but also people do it and it's working for some cases and I get that you have to have a lot of controls. It it is just a flexible mount the volume and have fun. Uh, you need to set up your own process and utilities to test and make sure that that thing is a little a little more

aligned unlike flat packs and brew and those sorts of things that kind of give you some of those nice additives to make sure that uh something is working before you try to run it. I'm just going to go over a few questions on each of the dros. I'm going to cover uh just to give you a little bit of background about like who owns it. Um that

is important. Uh what what does it use under the hood to be immutable quoteunquote? Uh where do you put your customizations and then kind of just where how do you maintain something this long term? Because you can read the spec sheet and say cool it's immutable and then at the end of the day you're like actually this kind of sucks. Um, I will also say that if

you're looking for an immutable DRO and you already have like a vendor ecosystem, pick that vendor's immutable DRO first. [laughter] Don't try to shoehorn someone else's immutable dro because you like it more. A lot of them have different technologies. A lot of them do it in different ways. But if you're trying to put uh flat car into a rail ecosystem, you're going to have a bad time.

It's like you're going to have to do more work. You're going to have to integrate it more. So, just FYI, if you want to save some time and you're like, "This is going to solve my problems," just stick with the one your vendor makes. As as someone who makes a a third party agnostic vendor uh version of this, I still recommend people try the immutable DRO that

their vendor ships first and see if they like it. See if they have problems, see if there's gaps. So, I'm going to start with Core OS. Um Core OS was like the OG of immutable Linux distros. Uh anyone here have Android phones? Anyone use Chromebooks? It's all based on like core o was based on that technology. Um so and also unfortunately core o is was renamed to

container Linux and then uh core o the company uh got bought by red hat so it was and then it was forked and called flat car container Linux by IBM. Um so because flat car also got bought by Microsoft or sorry yeah that's Microsoft anyway it's a long history of it's kind of the same thing. It's basically an Android phone for your server. Um, let's let me

just show you what it what it looks like. And this crashed my terminal so many times this last week. We'll see how this works. Um, car start. I am just running a VM on my machine with each of these dros. Uh, if you Yeah, scoot forward if you can't see it because I don't want to make it any bigger. Let's go. flat car SSH. So I am

now inside of a core OS machine or or sorry a flat car machine. Um the technology behind flat car is something called uh Nebraska protocol which is XML. Don't look at it. Don't try to do anything with it. It is like you can tell how old these things are because this is just legitimate like XML sending back and forth. And um they do one really cool thing

where uh the AB updates have like binary diffs from one version to another. So instead of saying here get the whole file system and do an update, it's like oh I know the block sections that you need difference to do an update. Um if you've ever seen one of my talks, I like to do props. Um so I think of a partition like a big bucket of

stuff, right? So, we have a bucket of stuff, and this is my uh my 3D printer trash. Um, which I kind of equate to a Linux distro. Um, it's basically hopes and dreams that you throw on the file system and you say, "There we go. We're going to boot that. That is now our Linux distro and we're ready to go." Um, the thing that, sorry, I I

need another bucket. The flat car and and originally coros does is they have two partitions. They're like, you know, it would be cool if we could have twice as much trash and they just say, hey, actually the second partition here, we're going to it's wanted to fall. Um, we're going to while while we're running from this first one, we just throw all our trash on the second

one. And then when we reboot, we tell Grub to boot that one. And that's it. And that's all we have to do. We said, "Look, we have 80 partitions and we can boot from one to another and they're immutable." Not really. Um, but that's not the whole file system of your computer. Like there actually is what if I have changes? What if I want to do things?

What if I have container images I pull down? I don't want to pull them down every time I reboot. I need some shared space of storage that is going to like always retain those containers. So there is other partitions that will still mount like var and whatnot that has variable information. Um those will be separate from these two AD partitions but flat car is that it is

it is two partitions that we put stuff into. Uh, and if I do, uh, you know what I have? I wrote myself notes because of course. Oh, I don't have what I also don't use this laptop very much. What is the command? Here it is. lsb. So if we look at this, we will see a user A and a user B. Literally two buckets on the file

system. That is it. There are two drives that we just throw everything into. They're both about a gig. Uh and then they have these other weird things like OEM and OEM config. A bunch of other dro started copying this where I think core o thought that like every vendor was going to go this routes and so they're like let's just make an OEM area where they can

customize whatever they want. And if you run in AWS or you run somewhere, this is where like your customizations for that environment go. They all go into OEM and it's like your scripts to bootstrap and start it. Uh sorry, core OS also heavily relied on cloudinits. Everyone know cloudinets? It's like the world's worst config management. Like iuntu wanted their own config management and at the time it

was Puppet and Chef and Sol all these really cool advanced systems that can do a bunch of stuff and they're like what if we just ran Python with a bash script like well and you could just run it every time and it would just reconfigure the node every time. Like well like does that work everywhere? Like I don't know. Uh core OS went that route and said

yeah absolutely we're going to go that route because we think that the cloud and VMs are the future. So every time the machine starts it looks for user data and cloud init and it runs that and then it again it just does this AB partition but like I said a mutable dro can't write to it right hello oh I need to it works it's not immutable there's

nothing about that is mutable I can even uh pseudo I think they have v uh let's See, let's go in and change something. Fail. It works. This is not immutable. Like, what is going on here? Like, why why do they say this is an immutable DRO when I can obviously mutate some core files in the file system that I shouldn't be able to? And the reason for

that is this magical thing called overlays. I'm gonna move these. And uh let's say I have Etsy engine X. All right. This is my Oh jeez, I made a mess of it. Um, [snorts] get out of here. Let's say I have Etsy engine X and I need to put my hello file and I want it in the Etsy directory. How would I get it there? It's immutable.

You can't, right? Overlays are this like fancy trick where we just like pretend that literally overlays one file system and we're like boop, look it, we win. We can we can put one file system in. So literally it just layers one thing over another and then the view you see is I have all my files. All my files exist because that got shoved into some weird place

that's an overlay and we can see it where there's my overlay. I'm going to make that a little bit smaller so it doesn't there's these things called a lower dur and an upper dur. That is the lower dur is the immutable parts. The upper der is you made changes. And so we can go find it. And we can go ls cis root. No, sorry. This the make

it smaller. work. That's where it is. It actually it's really funny because your upper dur goes into like a temporary work folder first like it goes in cis it goes like lower does never touches lower but it goes in Etsy work and then it goes into your upper um so it's really fun uh to try to like troubleshoot these things if I look at yeah there's where

is my cis root you gota trust me on this I did it last night cis root pivots to root and then it's user SH flat car. You know what it is? Oh, it's hidden. It's a hidden There it is. Etsy work. and then you get this weird work. >> Is it a RAM disk? >> No, it's a partition. It stays. It's literally mutable. Like, it will write

to a partition. If I if I parse this output somewhere in mount, I can find where that's actually mounting it. But >> yes, absolutely. I'll repeat. Yeah, it does persist across the boots because it it's on your file system. When I looked at that um uh what was it? LS block. There is the OEM config. There's my root VDA 9 uh will have some overlay in there.

So those folders end up showing up in this route. So you're still mutating the whole thing. Like there's there are sections that you can't touch that don't have an overlay, but in general they put down here's our file system. You can do whatever you want on top and then whenever we whenever you do an upgrade, we're going to take your bucket of trash and then your view

of it is just like here's your overlay very nicely right there. And that's it. That's what you see when you run the system. So you still have all the changes you make. You still have a home directory. You still have stuff that you need to configure. everything is still there and so it it it feels like the way I always describe core OS or flat car is

it is just a Linux distro without a package manager it's not really immutable it's just like well what if we just don't have app and it's like oh we run containers so it's good I love core OS and flat car I ran it back in the day um the thing that core o was trying to do which is like their their goal as a company was to

secure the internet so they always said over and over again and by trying to secure the internets. They said the things we needed was an OS that automatically updates. That's was it. [laughter] It was just and and how they got there was they needed to make updates more sec more stable. So they said actually we need to make this overlay file system so we can always write

the underlying file system. And they built a distributed database because they said we need a cluster machines. We need five machines together to only reboot one at a time. And that was how they were going to get there to secure the internet. They built those two core technologies of we need to just make updates easier in general across a fleet of machines. And that's kind of how

we got to fly car Linux. It didn't it didn't do anything else. And again, it stole it from Chrome OS. Chrome OS was doing this at a large scale already. They said, "Well, we just do it on servers." And they took all the technologies and they they did that. Did I I don't think I have another note. Oh, an uh update engine. Um this is how you

update it. Uh sudo update client I think client update. Uh if you in 2026 are writing command line like utilities with underscores in the name. >> You're a cop. Like I'm sorry this is like [laughter] don't do that. Like I I but like I know this is this is this has been around for a minute. So, uh, I don't even know if I have internet, but anyway,

that's like the update is just literally you're going to go a partition, B partition back and forth, and you just you have two and you just throw away one when you're going to write the other, and that's all you're doing. And you're giving those overlays to give files that you want to change. That's the whole basis of flat car Linux. Chyros, that's the next one. So, Chyros

does things a little bit differently. see if my terminal crashes. Every time I would start one of these VMs, it would crash. Uh, wait for it to start. Oh, I forgot one thing with flat car. customizing flat car. So we have our cloud init which will start like in initialize or customize the system when it's booting. That's not enough. They created a thing called ignition because they

realized that systemd when it starts assumes the file system is in place and assumes the files for services are there. And so a lot of people for a long time in their user in their cloud innits would like write a file reread the file system from systemd start a service and it was really hacky back and forth and some things didn't work that way but also what

if you need to partition a drive what if you need to actually lay out a file system of some sorts. So they wrote ignition which is it's basically the same thing that runs in your inits. We have to explain what an RAMFS is, right? When a computer boots, the BIOS her UFI initializes the hardware goes over to the bootloader. The bootloadader reads the kernel in a nitram

RAM FS. These are supposed to be the kernel is how to initialize the hardware. A nitram FS is a tiny piece of RAM that initializes that file system so it can start running the real system. A nitram FS mounts a bunch of stuff and then pivots to the real file system. That's how the process works. So what they decided was actually in that little initial before the

machine fully boots, we get PID one. We can do a bunch of stuff to the hard drive. We can format things. We can drop files. We can do all this stuff because the OS doesn't think it's not fully available yet. The kernel has not started. PID one has not started. So they give you this little shim of JSON that you shove in place and it kind of

looks like cloudonit uh that just like, hey, let's just do a bunch of stuff to the drive before it starts. And that's a thing that flat car and cor did later, but that's how they kind of initialize bare metal servers that like, oh, I don't have user data. I need to format stuff. A lot of other distros again use that same sort of pattern. Um, so I'm

on a Chyros box now, and Chyros is a meta Linux distro. Um, similar to how Bluefin is a meta. It's not it's not the actual DRO. Chyros takes another DRO and immutabilizes it. It says, "Oh, do you want Iuntu? Do you want I guess you wouldn't take a core sus but you want red hat you take another distro and then we're going to wrap it in immutableness

and then deliver that to you mainly for like compliance and onboarding because training people is hard and if you tell someone they don't have iuntu anymore if if they're used to it they're like oh well now what do I do right like that's the hard part of moving forward with things so they kind of took this approach of what if we took the other dro and repackaged

it as something that was immutable And if I and and one of the things they did was they packaged that they packaged that distro as a container image. Again, it's a tar file with lipstick. Okay, this is this is all it is. And so if I look at my image, this one's a abuntu. And then if I go over here and let me if I put it

in my notes. No, of course not. Oh, was it? >> Thank you. I still didn't put in my notes. no, it's a crane export image tar. You think um this will download that that container and show me what's in it. It's just a file system. It's this. It's just the trash. They just shoved it in a container image. And what they do to make it run is

they repackage this as this literally like in the technology. It it mounts as an img file which is runs as a loop back which is a CD image on the disk. And so instead of having two buckets, Chyros doesn't need to do two buckets. They have two CDs on their disc. They just say, "Actually, I can just have this. I can go AB AB back and forth

and and I don't need to have two full partitions. So if I look at my partitions, I don't have the the user A, user B. I do have other partitions because again file systems are fun and we have to put things in different places but I don't have to do a full partition for one and change the boot. The bootloadader is just changing which image it mounts

as loop back and not what partition it boots from. Uh and I know I wrote this Nope. Oh, it's squash. It says I should have a code. I probably messed up all of my notes on this. Uh, let me try one more time. Hello. Set up. Suju. Oh, what am I doing setup? There you go. This shows me where I'm running from. I have a live directory

and that is like my squash fs that it's running that it's mounting. Um, that's what Kyros is doing. It just says instead of partitions, we're going to do it from ISOs so that it's easier to package anything into the ISO. And those ISOs get distributed as container It's just a bunch of tools that they kind of we glue together in certain ways. Um, and again, this is

supposed to protect the right ability. Does anyone know why this CD is illegal in the US now? Not because of Ricky Martin's good looks. This one has a rootkit on it. Um, you all remember Sony back in their infinite wisdom decided that we didn't want people to copy our CDs and so they wrote a rootkit and put it on this CD. So if you load it into

a Windows machine, you couldn't copy the songs off of it. It would actually hide the CD from you. This is exactly the type of stuff that a mutable Linux should protect against. Obviously, this wouldn't run on a Linux distro. Uh, but if I try to do touchy hello, those guys who do it, I can still write to this file system. I still can write to this DRO.

There's nothing that's stopping me. It does the exact same thing that flat car is doing with an overlay. It stores it in a different path under a different name, but it's There's my overlay. I can see my, you know, I have an upper dur. I bet if I look in my Etsy, there's my hello file. And it just merges that into Etsy when I ls Etsy to

say, okay, where is this thing at? So, there are changes I'm still able to make. This is not what we think of as I can't change it in immutable. This is we're going to play some tricks to make it work because Linux was never intended to be fully immutable. There are things that all these distros are making changes for and doing. I don't think I Oh, and

um they also do they they also do ignition user data and system extensions. Exact same thing as fly car. It's exact same technologies to kind of customize it during initialization. Again, it's just taking Iuntu and then how would I modify how would I package Iuntu in an immutable way and then allow people to extend it. um both of these distros if you want to run things like

Nvidia cards in Nvidia GPUs, you have to go get a bunch of system extensions and mount them on the file system or you get a flavor of the DRO that has some of that built in already. These are things that like are really invasive to the file system and the kernel needs them really early in boot to be able to load kernel modules. That's all different things

that you're going to have to do with any of these sorts of distros. The uh Thankfully, they don't have underscores. oh, you know what? That's why I didn't see the file system. This I booted the wrong VM. This is actually running from the live host. I'm gonna hurry up because I know I I'm going to run out of time. Nope. Nope. Uh, most of these dros will

run in a live state from a live CD. And when you install them is actually when they give you all the overlays and things that you might expect. I'm not going to wait for it. Just kill it. Uh, what's our next? Bottle Rocket. Anyone running Bottle Rocket? I'm sorry. Uh, [laughter] I'm not actually going to cover cover bottle rocket because a you can't run it outside of

AWS or technically a version of vSphere. Um, but bottle rocket was intended to be Amazon's immutable container Linux and it only runs inside of AWS. I was at Amazon when we launched it. I had a lot of opinions about how we were doing things. Um, they basically do the the two bucket approach uh for things. Uh, but the thing that they wanted to do was they said,

"Hey, guess what? No SSH and an API for configuration. Those are both lies. [laughter] Uh no config. The the API for configuration is actually just a customized version of cloud init. It literally is just like hey we don't take regular cloud innits. We wrote our own because it's Rust. Yay. Uh and so we have our own configuration in toml that you mount into the file system and

that does your configuration. They don't have like an early ignition stage of doing things. They assume it's always going to run in AWS. And so everything can be done from the metadata endpoints that AWS provides and pull in from their customized version of clouds. No SSH is again it's not on by default because you have to enable an admin container. Uh but they run a proprietary SSH

called session manager. Back in the day I was actually a big fan of session manager. I thought it was cool because it was SSH over HTTP. I'm like that could be neat. Uh but also it's proprietary. [laughter] like you can't do anything with it because they run the backend services for it. So you can't use it anywhere else. If you want to SSM to servers in your

host or like in your data center, you have to connect them to AWS and you go through AWS IM and then back down to your server. It's kind of a pain. So Bottle Rocket in general uh they only have the APIs available that their cloud in it does. It only runs in AWS. They will push it on you as the default container OS you should use everywhere.

FYI, it's it'll come. I don't know if it's default now, but it will come. There's a lot more and more things that are making it the defaults. Um, it it does not have there was promises originally about lower resource utilization and a lot of other things. Most of those things end up being false because when you don't have shared libraries and everything statically compiled in Rust, guess

what? You use more RAM. Um, there's a lot of things that happen uh in a bottle rocket OS that I'm not going to go into all the details, but those are just some things you should be aware of. If anyone tells you run bottle rocket in GCP or something, just just walk away. Like it's not happening. Uh I worked on EKS anywhere and we tried to get

bottle rocket into data centers. We made one version for VMware and we had one customer sign up for it and now we have to support it or they have to support it. I don't know there. Um but that's that's basically it. They only want it for AWS. If you're in AWS and you're running containers, they want you to use bottle rocket. It's only for ECS and EKS.

Nothing else. So just a heads up on Hey, we're to the new core OS. Um, so the old core OS, like I said, uh, the company got bought by Red Hats. Um, Red Hats had a thing called Project Atomic. Did anyone use Project Atomic back in the day? Yes, this is exactly the same time that XDG app was still a thing. Um, I was back in the

day in IRC with a bunch of the Red Hat engineers and they told me they had this really cool thing. It's called OS tree. I'm like, what is that? And they're like, "It's like git for your file system." I'm like, "That sounds terrible. [laughter] You know how good I am at git? Not that great." And so you're going to do that in the whole file system. No

thanks. Um, but then they also did something that made me completely walk away from it where they said, "What if you could still install RPMs?" And I'm like, "You don't get it." Like, that's not at all what I want to do. The exact opposite thing. And they said, "But RPM OS tree gives us this overlay, but we love writing spec files, so we can still distribute RPMs."

I'm like, "I hate spec files. Don't make me do that." Uh, and that's when I stopped paying attention. Now I'm running it. [laughter] Um, so Fedora Coros, again, it went through this whole like weird naming convention thing. When Red Hat bought Core OS, they just said, "Core OS is really popular. we're going to use that name instead of atomic. And so they just rebranded atomic to core

OS and changed almost nothing. They took some pieces like ignition and a few other things that they liked, but the base of the file system, how the file system and how the how the actual OS works is exactly the same as what it was in the atomic days. So if you're used to that, it'll be the same. Oh, and uh also getting to boot C. Anyone heard

about Boot C? It's a build tool. [laughter] It's it's just a build tool. it it doesn't change anything on the OS itself, right? Uh Chyros packages their DRO like I showed you as a as a giant tar file. Um Boots Cy just does the same thing. It's just hey, what if we had our own build packaging system? Um they mostly did that because the RPM or the

the OS tree tools for building a file system really sucked. If you ever had to build a file system and distribute that file system with OS tree, you had to learn an entirely different like language and it was just like what? We're just like putting git repos up on a web server and then this thing's pulling down files that it wants. It was really bad. And so

they said actually all of that nonsense of OS tree sucked. So we're going to redo it all as containers and we call it bootc now. Bootc from the client isn't different. It's it's still OS tree under the hood. you have a bootc command but it's it's the same thing that we had with OS tree before it's just the building and distribution has changed so if anyone tells

you like bootc is better than osree again runs so let's look at it this one fcos I also hate as a name um just because I don't know it seems like it would be something else it. Uh they also have um oh in in Red Hat's infinite wisdom uh ignition is cool but it's JSON and for some reason Red Hat engineers are allergic to JSON. So they

wrote a tool called butane. You can write your ignition as YAML and then you convert it into JSON and then run it the same place. Why does butane exist? I don't know. [laughter] If someone knows, please tell me. But it pains me so much that they don't like you writing the JSON. They're like, "No, you have to compile it from butane first. Compile your butane, make it

ignition, and then boot your server." I'm like, "Stop it. Just just we don't need more tools than this. All right, we're in something that uh OS tree and and Again, I hate calling it for core OS. I don't know why I always call it OS3. I I again I use it on my laptop, but the the thing that they do everything is is here you'll see is

like sim links. Like we're just sim linking all over the place. We're like what if we just had more sim links on the file system because everyone likes reading those and following them. Uh so let's follow one. Uh this OS tree looks interesting. Let's get up top OS tree. Um okay. So, we got boot one, boot, boot one, one, and deploy. Let's go into deploy. Oh, yeah.

Of course, got to have the name in there. Uh, backing. I think that's the one. Nope. Everything is shaw. That's usable. Uh, so this is the first dro that I would say like becomes not for human consumption. This is when you're like, okay, I have to follow I have to know these shaws. is I have to follow these artifacts. Um they literally like recreates an object store

on your file system, right? Just like git, right? We have if you if you've ever tried to like scale S3, you have to like take the first two characters of the SHA and like make a bunch of directories like that so it's evenly spread and they do that on your file system. And I'm like this sounds crazy. Engineers love it, right? Like engineers like oh we get

to do all this cool stuff that we couldn't do in the cloud and now we do it on the file system. Why is that still Oh, it's gonna try. It's trying to update on me. Thanks. Um, please don't cos notes. Of course. See, I called it fedora core OS. Uh, I have to remember where those objects are here. That's that's your file system. Have fun. like that's

that's how we separate it out. That's how we split it out. That's how we make sure it's balanced. And I'm like, you don't need to do that on a hard drive. But I get it. Like engineers love engineering. Uh but we can trace that down. So if I go to bootc status, of course, the thing that's interesting about bootc I'm not limited to two. What if I

wanted 10? It's a bad idea. But if the whole thing is just sim links, I don't care what the partition layout is. Again, again, in and bootc is another one. There's one partition for this that we're just swapping back and forth those boot.1 directories. So your your grub, your SD boot, whatever is like, oh, I'm just going to go one or the other, and then we're going

to follow the tree to make all the files, right? And we're gonna we're going to figure it out, but it's mainly for computers to consume this. Don't don't try to do it yourself. Uh, and so I can see my my commit here. I can see my Shaw. I could follow that all if I wanted to. I'm running out of a little bit of time, so I'm just

going to skip a little bit on this. Um, again, boot C. If it's a container file, you're a cop. Uh, I don't make the rules. Uh, so if I do crane again, crane export. Look, it we we made a container. That's a file system. Isn't that cool? No, because it's just a tar and it looks uglier because we have all those shaws that are all those sim

links that we put everything in here. Again, not for human consumption. They even went as far as you see these files x adder links. That's kind of weird. They separate out the file data from the attributes on the Who does that? That seems crazy. So they actually split like hey what if we had the same file multiple times because we can have 10 boot directories if we

wanted to. We want one attribute file so we make sure that it's always the same partially because of SE Linux but also partially because they're masochists. And so they actually separate out these two things where you say actually when I want to ls a file I have to look at both those things. I have to look at the data and I have to look at the attributes

and they split that out on the file system in OS tree. Does this sound like something you want to be on call for? Looks like I It scares me. I run it on multiple systems, but I don't have to be woken up in the middle of night and try to debug this. Like the the distribution mechanism of it's in a container is neat. The fact that I

can build it in a container file, that's kind of neat. All the other stuff is is a little bit scary to me. And it sounds a little bit like I think my next one. Oh, no. I got to skip one. It it sounds like Nyx. So, we'll get to NYX in just a second. Um, but I'm going to go to micro OS again. I'm gonna go a

little bit faster here. I'm sorry. I'm I'm around two, so we can ask you can ask questions if we uh meet up. Um, let's stop this one. Is anyone using Micros? Hey, sweet. So, I can talk about it. Uh, micro. Wait, I got to start it down Mic. I hate the ooze. Micros start. There we go. Micro OS SSH. It'll come up. So, the interesting thing about

micro OS, as I say, what if all that complexity was just in the file system? We have cool file systems that can do this type of stuff. We don't actually need to make the SIM links and all the stuff everywhere. or we can make it look like a regular file system. That's mostly right. Even if I even if I get a couple sim links in there, that

mostly looks correct. Like there's nothing about this that scares me yet. I can spell snapshots. Snapshots. Uh this is Oh, and I also forgot on on the last on Fedoric Core OS. Yes, you can touch Etsy. It has an overlay. It does the same sort of stuff. Um so it's not immutable. Uh this micro OS does everything in Butterfest. What if we had Butterfest snapshots for all

of this nonsense? All of the sim links, all of that stuff. What if we just made snapshots in Butterfs? And so anytime that I I got a sudo it works. It's still going to work. Where does that go? In this case, Butterfs just says, "I'll make you a new snapshot." That's really cool because Butterf just hides all the details as long as you want to learn Butterfs,

which I do think it's a cool dist file system. I think it's very powerful. I think it can break in certain ways if you're if you're trying to use too many things, but as a as a package component of this, I think it's a great way to implement this stuff. I really like how that works. The thing that I don't like is again I have to remember

how this works. Uh there's my snapper list. So I have snapper list shows me all of those snapshots and then my butterf sub volumes is that it no list of course did I spell it wrong >> sub volume. Thank you snapshots for all those things you'll notice there's like Etsy one. Hey, you probably changed that one. Yeah, I did. That's that's my change. But and again, Butterfest

is really cool. The thing I don't like is how you upgrade it because Micro hased they're like, "Well, actually, we don't have to do any of that nonsense of like hiding upgrades. We're just going to let you do Zapper, which is basically yum, right? We're just like yum update the whole thing and then snapshot it." And I'm like, that's the thing we're trying to not do. That

was the thing I was trying to get away from is not doing yum update on a server. Yeah. >> I have it backwards. Oh, yum was a snapshot and then run the >> right and then when I reboot I I snapshot I I I snapshot snapshot run the updates and then reboot but that that reboot comes from the new snapshot so I have two snapshots but I

can have multiple snapshots again really cool I get all of the really good benefits of what like OS tree because like I could go back in time multiple times but yeah I will do a you know always always back up your database before you do the migration right it's the same thing we're like hey we're going to snapshot this we're going to do our upgrade we're going

to take a snapshot we're going to This also gets me out of some of the patterns that the other operating systems require, which is reboots because I can mutate more of the file system in place and just take snapshots at the point in time that I want it to exist. So that's really really cool. But again, the reason we built all this was because we kept running

into problems of getting in states that we didn't know about. We didn't build the thing and ship it. We just mutated a bunch of stuff in place. And so that confusion and ongoing maintenance and long-term how what state are we in gets confusing. I do get some of the benefits of rollbacks and I do get some of the benefits of uh you know immediate backup sort of

stuff but the ongoing maintenance is isn't the same thing as what I got with the other dros. Was that accurate would you say? >> Yeah snap back. >> Yeah. Yeah. Exactly. It's not it is not a backup. It is a state and time point in time and then it has its own sort of like overlayish thing that it does to let you keep writing copy and write

stuff. Yeah. Yeah. Yeah. You still do need proper backups. Do not do not rely on butterfs. Just like do not rely on raid to be a backup. It's not a backup please. Uh what is our I have like Yes. Okay. Nyx I called it Nyx. Uh, Nyx is literally a PhD thesis that made it to production. It's literally how we got here, right? Like someone wrote this

really cool thing like what if we could always reproduce everything in the world, ship it, and like that's how we got here. And it it transformed into a package manager. It transformed an OS, became package manager, does all this stuff. If you look at it, I think Nyx SSH Everything looks really similar until until like in this one if you go one I think it was like

one directory deeper. There you go. There's the sim links. We hit them at the home at the root, right? No sim links here. Oh, actually they all go into the store. Uh and again it's just like OS3 where you're not supposed to. You don't go in the store. That's not for you. Store. Yeah, sure. Look at those things. Uh, and like I don't like that Nyx is

starting to make more sense to me of why I would want to use it. I don't like that. Uh, especially on something like a laptop or like a desktop that is not a server. That is something that I mutate every day. When I want immediate mutations, I actually want something that is super flexible at letting me put those mutations in place where I'm at and just keep

running. and force me to have the good habits of writing flakes and making sure I document everything I installed. Like that is the power of Nyx is like you have to do it the right way otherwise it won't work. The thing I don't like about Nyx is every command I seem to try to run is experimental. Like everything I try to do is like yeah just enable

experimental and you'll be good. I'm like I don't like that. Usually uh I'm tend to be on the on on the edge of things, but also I need my computer to work and running experimental things sounds a little risky to me. So Nyx is cool and again uh if I sudo touch and see hello, it's still mutable, right? This Nyx is not intended to be immutable. It's

tended to be reproducible. And that's a big difference on how it is treated, how the file system works, how the packaging works. And so it's it's a it's a difference. I'm starting to like it a little more. And I don't like that. Um >> so >> yeah, I know. I I know it's going to happen. And actually, I'm thinking of doing this laptop because I don't use

this laptop very often. Um so the last one is is the one and I only got five minutes for this. So it this is Talos. Um make Talos start. So Talos immutability is not the thing Talos does. Immutability is an artifact in how Talos accomplished its goals. Uh I've I I work at Talos now or at Aerolabs. I had been using Talos for a long time even

back when I was using Core OS because I really liked one thing about it where Talos has an API for management. There's no shell in Talos. You don't you don't have user accounts. Every single one of these Linux distros that we've anytime you want to upgrade them, you start by sshing in. And SSH has a lot of baggage. You need a user account, you need sudo's file,

you need keys, you need all this stuff that comes with that to then run a command to execute something on the file system. That is how all of the dros work. Talos doesn't have that. Yeah, >> micros doesn't require it. >> How do that? How does that work? >> Sure. Fully like you can enable fully automate like flat car core OS. You can enable fully automatic updates.

Yeah, sure. And and that was the default for flat car and core OS [clears throat] as well. The the thing is those are run by default by having an agent that executes the command. >> Yes. Systemd is is on a curon job that says run updates, right? Like that's basically how automatic updates happen. And so yes, I do agree they all do have an automated way of

doing it. These, you know, ways of of making the file system safer do make that maybe a little more assured that's going to happen. Uh but but Talos is like we don't want any of that. We don't want we're just we literally call an API to do all of the stuff against the system. The problem with that is showing you what Talos looks like before before it's

actually doing something. If I actually uh I think I started the wrong one again. I need the installed one because Talis does run in Oh, I didn't build it. We're just going to do this one. We'll show you. Um I don't have a way to lsn in or like to exec in and touch a file, but I wanted to show you how that works. So, I did

something that is really irresponsible. Don't do this. Um, I made a system extension. Talos has a system extensions as well that mounts container images. Uh, the difference for for us with with system extensions is we build them with a version of Talos. So, every system extension only works with one version of Talos and they're signed for that version and they won't work for another version. So, we

assure that this thing you can mount all these system extensions in place. Um, but I made one for SSH. Again, don't do this. But I made I made an SSH system extension that I can exec into uh on a Telos node. So if I make Telos SSH, I am now on a Telos node with a shell. Um which is again not intended to be how it works,

but I want to show you how this thing works. Uh so I can do I think it's on host. Again, it looks similar. Uh this one again isn't installed uh but it's going to have basically the same thing. The thing that Telos again does different is remember that boot sequence that I told you how a system boots Talos stops at the init FS. We don't have a

pivot to another boot thing. The entire thing runs in that stage. So the the thing that everyone else does of like oh here's where your ignition goes. Here's where you customize the system. Here's where you lay files on disk. No, no, no. We only run that piece. So it only runs in memory and we wrote our own innit system for running the API in memory that does

all the things we want to do by making it so that you only go through that API to do anything on disk. I can't clear the screen. Cool. Uh makes it so that even Oh, I don't have sudo. I think I'm rooting here. Yeah. So if I want to touch hello, we finally did it. readonly file system. I cannot mutate that file system because of the way

thanks [laughter] because of the way that we initialize everything up front. So that piece you can change the system but we make assurances that the only changes to the system go through the API or they're shoved into corners that have their own little mutable areas, right? And so that's how that's how Tal is doing it. We do a very similar thing with uh we don't have AB

partitions but there's no buckets. Again, it's an init RMFS. So, we write one RAMFS, we write another RFS, we boot back and forth between them. And that's that's the whole difference. But again, immutability is a Talos feature because of what else we were doing. It's not the goal where most of the other systems that was their goal. Let's make it immutable so we can do upgrades in

a in a standard or a more trusted way. That's it. That's the entire talk. So, we have I think like a couple minutes for questions if there's if there's more. Yeah, right. >> Um I don't know. I thought uh the bit about OS tree is interesting. I know they're trying to pivot to a composer backend for Bootsie. Uh I it looks it looks kind of similar. I

don't really know much about how it works under the hood. I'm pretty sure they also store the the the attributes and the exitures like separately and stuff like that. So, it's it's a little cursed like that, but I think what comes with it is like some like uh verity and like integrity guarantees like with the hashes, which is nice. >> But yeah, I just wanted to know

like your thoughts on that. >> Yeah, I did not I I skipped over some of that just for time. Um DMverity is is a piece of like let's validate all of these things. Um uh Bootsie is doing that. Bottle Rocket does it. Uh and I think that's it right now. Um, but the the composs is like, what if we had this this thing that was an overlay

and what if we made it a three-way getit merge? It's basically what it does. It's like, oh, we don't just like shove the overlay in one place and then just put it on top. It's like we actually take that's exactly how we take the attributes from one file, the file contents from another one, and you could have something else that you're putting in place. Those get merged

into your view. That's the compos fs. Um there are there are limitations around how that works but basically that's we're we're just composing it in a view that from a user's perspective looks like an overlay but functions different in the kernel. Um the DM verarity is a piece that's interesting because it has a performance hit when you're verifying all your reads and writes on a file system.

Uh you actually need to use CPU time and you need use extra memory. But Boot sees pivoting to that and that is something that um they're they're trying to get rid of the layering of O of RPMs. Something I told them was a bad idea 10 years ago. They're like, "Yeah, actually." What's >> So did I. >> So you did too. Yeah, it was a bad idea.

RPM OS3 was a bad idea 10 years ago. They're finally getting rid of it. Thank you for the validation. >> Yeah, great presentation. Um at home, I run flat car Linux. I've fed with the Fedora core OS. I did a quick search for Ciderero. Um, lots of resources on GitHub, but it seemed uh based off of a two-c Google search. I didn't see anything immediately for quick

start at home. Uh, what are resources I could look into? Uh, I mean, docs.live.com has all the like getting started. Uh, Talos only is for Kubernetes. Every other dro that I talked about is either general generalish purpose like Nyx is completely general purpose Linux. Um uh the others are usually like container optimized where it's like hey we have a container runtime and no package manager like that's

that's container optimized um and so you can just run whatever container you want but also like the maintenance is on you to like run docker compose or something like that. Talis is like we only do Kubernetes. It's just like hey we will get you anc uh endpoints and and Kubernetes API and you have fun right and all the system maintenance should just disappear and the reason it

was built was mostly because people were tired of doing I have a user at the OS layer I have SSH keys and then I have Kubernetes and I have arbback and I have the user and I'm like well what if we just get rid of the stuff below. So Talos literally has no like Etsy password file. There's no users defined. You don't have to manage those pieces

of it. Go ahead, Lee. >> Hey, uh, great talk, Justin. I think, um, what I was getting the gist of on the comparison between distros is like, oh, there's different ways of representing file systems and bytes that we can distribute to people, different like bootloadader hacks and stuff. You like, I'm going to point to this disc. I'm going to mount this image. I'm going to overlay this

here. Uh on the Nyx thing, I think something you're making me realize is that things inside of binaries uh and like what the linker points to uh and Nyx's trick of like patch elf, the the content of the binary like pointing to something that's hash that also feels like something that's part of like an addressable file system to me. Um in this comparison, do you have any

comments on patch elf and how nyx does things? Like I noticed uh we were showing overlay, we were showing like uh disks and partitions. Uh could you like uh use like uh L uh LD or something like on a NYX binary and show it's pointing to a hash on the file system? >> I don't have I don't think I have the time. I think we're at time

so I won't be able to demo it. But the thing that's really cool about Nyx is it extends the OS tree stuff to like what if every application had its own file system, right? Like that's the the the thing the the proof of concept of Nyx. It's like, hey, actually, it's not just AB boots or or 10 different boots. We can have every I can have bash

on there 10 times if I wanted and every single one of them works because it knows the tree, but not every Linux binary is compatible with that. And so that's where I think the patch elf comes into play. We're like, hey, actually, if we need to overload LD preload, uh, we don't want to do that for everything. So, we have to do that on the fly for

some of these binaries. That's my my understanding of it. I could absolutely be wrong. I haven't jumped into Nyx yet. Uh but that's like because they want to do it for everything. You have to add a few more hacks in there to make it work as flexibly as possible versus the you know chyros we're just going to download an ISO and mount that as readon and then

you run it right like that is a different approach of let's use a tool to make it look the same and and to the binary itself on Nyx it does look the same but we had to do a little more uh hammering to get it there. know if you can use any binary with the nick store on any dro. >> You can use any binary from the

next store on any dro because well yeah it has all of it compatible. Yeah, it has a whole chain of dependencies. So yeah, that makes absolute sense, >> right? You can put the next door on any dro and use it as a package manager for those binaries. Yeah, >> good. >> I'll also be around I don't think we're out of time, right? There's another talk in here,

so I don't I don't want to take >> Well, actually, we're heading to lunch. >> Oh, we're heading to lunch. Oh, sweet. We can keep going. >> Oh, here we go. Okay. [laughter] >> I forgot it was lunch after this. It's lunch on a Sunday. We got all day. Hey, so I I really liked how you um pointed out uh I don't want to be on call

on that thing because it's not going to be good for human consumption because as a recovering SRE like this talks to me. Haven't touched a pager in 10 years, but who knows? Uh what's the story on Talos there? Because you were like we don't have SSH and I'm like well I think just between us everybody don't stop listening but SSH is probably one of the first tools

I reach out to when I need to troubleshoot something. So what should I do on Talos? The API is the most interesting thing of Talos in my opinion. The immutability the other pieces of it I really don't care about. The API is the thing that I want to exist more places. And what do you do? What are you doing when you SSH? In pretty much every case,

you're going to get some data. You're going to poke at the system to get data to understand the state of the system. We can do that other ways, right? Like like when when I really think about what is SSH doing, right? Like I run SSH and it opens a socket file on my local system, opens a network connection, opens a socket over there, and then like pipes

a shell like my keyboard and mouse over to there out to my display. Like like why isn't that just HTTP? Why can't that be another way to get data? And so Talos, I'm not saying it's perfect. I'm not saying it has all this stuff, but we do have some debugging endpoints, right? Like if you want to go to the API and say, I want a TCP dump

and see what's going on there, we have a TCP dump API endpoint. And so you can just go tell Telis control TCP dump and it does it gets the same data you would get from TCP dump running on that host. It does run a TCP dump implemented and go and just streams the data back to you, right? There's no reason that my interaction point has to be

local socket network local socket sort of thing, right? Like I can just we know how web servers work, right? We secure the internet with web services and we send data back and forth through APIs. And so doing that at an API level, I would love to have more debugging tools and we actually in in the next version we have a debug uh container which is exactly the

same thing that like core OS did back in the day where it was like this is this is your toolbox, right? At some point you do need your own tools to run something and so we have a debug command that will just download an OCI container and put you in the shell, right? Like we can do that. It still goes over the API. It's not SSH, right?

Is it is it is it a is it like session manager like all these proprietary SSH's implementations are kind of just like we just need data from the box and I can't get it from logs so they need to poke at something a little more. So having the API there is just a different way to get the data and I do again I want more ways that

are you know easier for people to consume because LLMs aren't good at shells right like LLMs are great at APIs and specs and I can give them the API spec and say cool get me this data from this spec and it'll do it for me but if I tell it go SSH and then like find something for me I run a bunch of risk of it going

off the deep end of you know running some other command Any other questions? Got some hands up. Yeah, I thought it was interesting about Talos how you were saying that you know in the container environment in Kubernetes that you don't want to deal with that like bloatware of like users and groups and etc. And so I'm really curious like how minimal of an image can you make

in Talos? like what what's the bare minimum and like you can run your apps and still have the API be functional. Like I'm assuming there's some minimum components needed to make that API still work. So >> when I when I started I was like I'm going to start from scratch. When I started Seder I like I'm going to start from scratch and I've never compiled the Linux

kernel this much since college. Um fun times in college. Anyway, uh I I compiled Talos and I was like let me just like dump the file system. Just tell me what's on here. and I looked at uh user bin. I'm like, what's on this? And there's a bunch of similar stuff like wait a minute, there was 11 unique binaries on the entire system. 11. I'm like, my

Chrome runs more unique binaries than that. And I looked at other file systems for other dros and there's, you know, thousands and thousands of executables all over the file system, which is one of the things that like make it hard to upgrade and maintain. Like we don't know what any of this stuff does. We don't know where the CVES come from. It just takes more time to

download it. I I you know remember mini CDs the three and a half inch CDs that you used to use for like um business cards right back when business cards were a thing. Uh they would hold like a 100 megs and I could burn all of Talos to a mini CD and and it was just there was you know there's other stuff in there. We have kernel

modules. We have stuff to run drivers. We designed it to run on bare metal servers. It works in cloud. We do similar things like the OM OEM drive. We have what's called a platform. Like they all kind of do similar things at the end of the day for that. Uh, but there was nothing that I realized you could never strip an Iuntu or a Red Hat or

anything else down enough to say we're good now, right? Like we we took away everything that we don't need. We the the founder took the other approach. We're like, I'm going to take the Linux kernel and I'm going to run the cublets was his original attempt. It didn't work because the cublet's not intended to be run that. He wanted cublet to be pit one and cublet did

everything else. He put the API shim in there just because I do need to debug things. I didn't know when it wasn't working. I didn't know how to fix it after the fact, but that's literally it. We wrote it's about 100 lines of Go for an init system. The systemd is like 10,000 lines of C. Like it's just let's let's we run an API in an init

system to run containerd in a cublet and that's it. And everything else should be in your container. Not everything works. A lot of bad habits over the years of people expecting that binaries exist on the host. They shell out to things. They run stuff that just it's not going to be there. those things. Talis has been around for eight years now, I think. Um, like we've we've

gained some steam on you should package bash if you need it in your container, right? Don't make it don't make it rely on the host. And so that it's come a long way, but I will say like not everything is there, but we we definitely have a lot of users, especially home labers that we get a lot of them that were like K3S is great running K3S

and initializing it fantastic experience. The curl to bash, beautiful, right? But then once you go to upgrade iuntu, you're like, "Oh, shoot. How do I make sure K3S is up to date on the same kernel and all this stuff ends up being not matched, right? Right? And so building one system holistically that says we do one thing and we know what versions we support, we know how

to migrate to and from those versions, it just eliminates a lot of the concern which was in my opinion the original goal of how do we make sure updates are fast and easy and mature and and that like you have to build a a fully stacked system instead of picking and choosing the pieces. >> We're we're like 10 minutes over now so I don't want to take

any other time. So >> thank you all for coming. Yeah, let's give let's give Justin a round of applause. The next talk in this room is going to be starting at 1:45. So, you are released for lunch. Uh remember this is the uh systems and architecture or systems and infrastructure track. So, you'll see everybody here in just a little bit. Thank you so much. Okay. Oh, good.

Testing one, two, three. Test one. Test one, two. Parimonious. Okay, good. I'm not I'm not popping my P's. That's good. Uh yeah, we should be all we should be all set then. Test test. >> Okay. >> Sounds good. >> Okay. Uh, welcome to the system and infrastructure track at scale 23. Uh, if you feel unsafe at any time, please call our safety number posted around the venue.

Our next talk is the tip of the iceberg presented by Alex Rasmmanson. Uh we will take questions at the end of the uh presentation. Please welcome Alex. Good afternoon everybody. Uh thank you Paul for the introduction. Uh my name is Alex Rasmuson. I'm a freelance data engineer. Uh over the course of roughly the next 40 minutes I'll be leading you on a whirlwind tour of the iceberg

table format. Uh so without further ado, let's dive right in. So for the rest of this talk, we'll start by kind of motivating why iceberg exists. Uh we'll talk about what the inside of an iceberg table looks like. Uh we'll talk about some of the benefits that iceberg gives you. And then we'll talk about the hard parts, you know, how do you maintain these tables and keep

them happy in production. Uh and what are the rough edges you're likely to encounter. uh and then we'll conclude by just some thought exercises on how you might decide if iceberg is worth it for your use case. So if you are used to databases, dealing with object stores for table storage if you're dealing with them as raw files is going to be a pretty disappointing Um so

uh if you've done this in the era before table formats like iceberg uh you might have a layout of files that look roughly like this. uh this is what is colloially referred to as hives style partitioning after the system of the same name. Uh and you'll notice that your files are kind of organized into a tree. And if you're in S3, this is a tree of key

prefixes. If you're in HDFS, uh this is a key a tree of directories. Um but regardless, you kind of got this top level that describe your first partition. In this case, we're partitioning by year. uh you've got your kind of child directories that refer to your second level of partitioning, in this case, months. And then at the end in kind of the leaves of your partition tree,

uh you have all of your files. And you can kind of use whatever format you'd like in this in this mode. Uh but more often than not, you're going to see this data stored in a columner format, probably in parquet since it's been sort of the default for a long time. Um, and this layout has a whole lot of problems that you are going to encounter pretty

much immediately. Uh, the first big one is that there's no really well- definfined notion of what a transaction is in this world. Uh, and that causes a whole lot of misery. Uh, the first thing is if you're writing to this table and somebody comes along, uh, can you still read? Can you still hear me? I'm kind of I realize I'm popping into my microphone a little bit

more than I would like. Um, so you're writing to this table, uh, and you crash in the middle of the write. Well, it's possible that you've just left some data scribbled all over your table that a reader is now going to be able to read. Uh, in the database literature, this is what you would call a read uncommitted fault. This happens even if you don't crash if

somebody happens to come around at the wrong time. Um, a lot of engines have tried a bunch of engine specific, let's call them what they are, hacks to get around this problem. everything from multi-art uploads to sentinel files that say I'm done that you have to check before looking at the table uh and all of that uh has just kind of swept the the problem under the

carpet. It hasn't really actually solved anything. So we don't have transactions. That's a shame. These tables are also really hard to change. Uh and there are two kind of ways that you might want to change the structure of your table. The first is by repartitioning it. Changing that partitioning scheme. Um, and since your partitions are so tightly coupled to where those objects are in object storage or

where those files are in your file system, if you want to repartition them, you basically have to rewrite the entire table because you have to change where those files are and what data is contained in each one. Schema evolution is also complicated. Uh, which is a little bit surprising if you think you know file formats like parquet you know they've got schemas attached to every file. you

know, you know what the schema of a parquet file is kind of by looking at its headers. Um, so wouldn't it make sense that if you had a whole bunch of them in one place, you know what the schema of that thing is? Well, yes, kind of, right? Uh, it turns out that uh someone can just come along and write whatever parquet files they want into your

table in this world. Uh, they can have whatever schema they want. So, the only way that you really know what the schema of your table is is by looking at every single file in that table one at a time. Uh you can kind of get around this if you have a data catalog that gives you the kind of schema of the table, but you kind of have

to trust that that data catalog is right um and that all of your writers are manipulating it in the appropriate way. And that's uh maybe not always a safe assumption. So this is bad for operators. It's also really bad for query planners and for database engines of of all kinds uh because the kind of scan planning that you can do in this environment is really primitive. So

in that table that I mentioned before where you've got a directory for year and then every one of those has a directory for months. If you want to select a particular month or a particular range of days, you actually have to structure your query in such a way that you're referring to the year and the month directly. I can't say give me all records on March 8th.

I have to say year equals 2026 and month equals 3 and you know so on. It gets worse though because inside of each of those leaf directories where you've got all your data files, you have no idea what data is in each of those files until you look at them. And so if you're a query planner and you're trying to figure out which files to touch as

part of a query, you really have to look at at least the headers of every single file in the partitions that you're scanning. Uh and when a table gets really big, this gets really really slow uh rather quickly. So all of these things together uh kind of have caused the industry as a whole to shift to the way of thinking that really storing data in this way

for tabular data in object stores is just the wrong abstraction. And a lot of big companies that were early adopters of the kind of big data cloud native way of doing things uh hit this realization at about the same time and one of them was Netflix. So in 2017, a group of engineers at Netflix began development of the iceberg table format. Uh and they described it as

follows. I'm quoting the spec here that iceberg is designed to manage a large slowchanging collection of files in a distributed file system or key value store as a table. So they set out to be like, well, we're going to solve this problem. They're not the only team that had this realization. At about the same time, a group of engineers at Uber were developing Hoodie and a group

of engineers at Data Bricks were developing Delta Lake. Uh we won't really have time to talk about Hoodie and Delta Lake in this talk. Uh suffice to say, they are very similar in some respects and extremely different in others. Uh they're all designed to solve roughly the same set of problems. The reason why I'm focusing on iceberg here is that in terms of just breadth and depth

of industry adoption, iceberg appears to be kind of pulling ahead. It's got it's got the most mind share. Um, every single vendor under the sun either is integrating or has integrated with the iceberg table format at this point. Uh, LinkedIn in particular has uh said publicly that they have more than 10,000 iceberg tables deployed in production. Uh I've personally worked with clients that have hundreds of terabytes

of iceberg data under management. Um so it's here. It's relatively mature. Um and you know a lot of people are excited about it. And these uh table formats together have kind of ushered in the era of what they're calling the lakehouse architecture which is kind of all the nice things about a data warehouse in terms of schemas and transactions and all of that nice stuff. uh with

the scalability and flexibility of a data lake. Uh and we'll talk at the end of the talk about you know the pros and cons of of this approach more broadly. So now we know why iceberg exists. Let's look a little bit about what an iceberg table looks like under the hood. So before we go any further, it's important to note uh iceberg is not like you know

duct db or click house. There's no iceberg binary that you can download and run. Uh, iceberg is a specification of a file format really. It's a specification of a table format and in that way it's very similar to like parquet or jpeg. It describes how stuff is laid out. It describes how you write that stuff and how you mutate it. Um, but there are many different implementations

of the iceberg spec that exists. I would argue that if you're looking for a reference implementation of ice of the iceberg spec, you're going to be looking at the iceberg client for Spark. It's the one that Netflix was working on the most heavily in first and it's by far the most mature in terms of what features it's got implemented. Um there are also several versions of the

iceberg spec. It's a a spec that's continuously evolving. We're going to be talking about version three of the specification here uh just because it's the one that's kind of the mature one as of the time that I'm giving this talk. So what does an iceberg table look like? So the kind of lowest level and most primitive thing that you have in an iceberg table are data files.

As the name might imply, data files are where all of your data lives. It's where all of the rows that comprise your table are stored. Um it's important to note that these data files and in fact all files in an iceberg table are immutable. This means that once you write them, you cannot change them. Uh and a data file is uh storing data in a single partition.

uh but a partition may contain many data files and there may be many partitions in a table. So uh that kind of begs the question well if you can't change data files then how do you update rows in a table? It turns out there are two ways you can do this. The first is what's called copy on write uh where if you want to make a change

to a data file you read that data file you do any changes you need to do and you write out a copy. Second way you can do this is what's called merge on read where you leave that original data file alone but you write these two kind of auxiliary files attached to it. One of which is called a delete file. It's just a very concise way of

saying this data file has the following rows deleted from it. And then you add another data file that describes the rows that you add. So if you want to update a row, you write a delete file deleting the row and you write a new data file that contains the row's new version. These two write methods have their respective trade-offs. Uh, copy on write, it turns out to

be a lot faster for readers because you only have to read one file and you don't have to do any kind of processing to stitch them together. That's just what the data is. Um, on the other hand, uh, if you want to change one record in a data file, you got to copy the whole thing. So, uh, it turns out that writers are a lot slower that

way. On the other hand, merge on read a lot faster for writers because you're only writing those two sidecar files, but it's slower for readers because you have to read all three of those files and then kind of logically stitch them together to figure out what the actual state of that file is at any given time. So, you've got all these data files and delete files sitting

around in your object store. How do you find them? Well, you organize them into manifests. And you store those manifests in these things called manifest files. And a manifest file is really just an AVO file with a row for every data file in the manifest. And each one of those rows has the path to the file, but it also has a bunch of information about the file,

like what partition the file is in and a lot of statistics about what the maximum and minimum values for its columns are and how many rows it contains and all kinds of other stuff like that. And that's really useful if you're a query planner because you can actually look at the manifest and figure out which files in that manifest you need to look at without actually looking

at the individual files themselves. So it speeds that up a whole lot. Um Russell Spitzer gave a really deep dive talk uh at the CMU databases uh seminar series last year about how scan planning works for iceberg and spark. Uh I would encourage you to check that talk out. He goes really, really deep into how this works. Um, and I've put a link up there on screen.

So, you've got these manifest files sitting around on object storage. Well, how do you find those? Well, what you do is you organize those together into a manifest list. And a manifest list looks a lot like a manifest file. It's an AV row file, except every row of this file doesn't refer to a data file. It refers to a manifest file. And it also has metadata, but

that metadata describes all of the data in a given manifest. So it's got minimum and maximum values and counts and all of that other stuff. And this is just also there for the benefit of the planner so that it can look at entire manifests at a time and go, "Oh yeah, I don't actually need to look at anything inside of this inside of this manifest when I'm

writing my query." Um, so you've got your manifest list that points to your manifest files that point to your data files. What we're going to do is we're going to take that all that stuff and we're going to wrap it in a thing called a snapshot. And a snapshot is really just the state of the table at a particular point in time. And all that is concretely

is a bunch of metadata plus a pointer to the manifest list. And that snapshot is kind of the breadandbut thing that represents what you're querying when you query an iceberg tables. You are querying a snapshot of that table. Now snapshot information like everything else is immutable. So every time you write to an iceberg table, you're generating a new snapshot. Well, how do I know where all of

the snapshots are and which snapshot is the right one to be querying right now? Well, we roll that up into the kind of top level thing on the tree of files in iceberg and that's the metadata file. And the metadata file has a lot of information that's useful about the table like its schema, metadata about partitioning, table properties, and that sort of thing. But it also contains

a complete history of snapshots and a pointer to which snapshot is the current one. So, uh, that metadata file represents kind of the current state of the table. But whenever you make a write new metadata file. So, how do I know which metadata file is the right one? Well, that's the catalog's job. So, the catalog has effectively a pointer. And that pointer says, well, my table is

a pointer to this particular metadata file that sits on on object storage right now. And if I want to change the table, I got to write all these new files and then I got to repoint the catalog at the new metadata file. So let's take a look at an example of how that might look. Let's suppose that you have an iceberg table that looks schematically something like

this. You've got some manifest files with a manifest list pointing to all of them. And then you've got your metadata file up at the top with snapshot zero pointing to that manifest list. And you want to do a write. Well, the first thing you do is you figure out which uh manifest files you need to change. You, you know, make your new data files and your new

delete files as the case may be. and you write some new uh manifest files and maybe you throw the old manifest files away and you take all of those new manifest files that make up the table and you make a new manifest list pointing to all of those files and then you call this thing snapshot one. Then you make a new metadata file and that metadata file

says okay I've got two snapshots now. Snapshot zero and snapshot one. Snapshot one is the most recent one. Now, this is where the immutability aspect becomes important because you can see that if let's say like I was going and scribbling on the contents of uh manifest list M0 or M1 for instance, uh that I would have destroyed information about what snapshot zero actually looked like. Uh and

that becomes important for a lot of other kind of more fancy use cases that we're going to see here in a couple of minutes. So, I've got my new metadata file. The last thing I need to do is update the catalog from pointing to the old metadata file to pointing to the new one. And when I do that switch over, I'm done. And my right has now

effectively committed. And this is kind of the the the key hinge upon which all transactional stuff uh will will uh all of that will derive from that one kind of simple repoint of that pointer from one metadata file to the other. So that's roughly how an iceberg table So what benefits do you get from your iceberg table that you didn't have uh with all of your raw

files sitting around on disk? Well, the first thing is that you get transactions. And if we want to be really precise and we were talking about transactions, we got to talk about ACID. So ACID, it's an acronym. Uh it stands for atomicity, consistency, isolation, and durability. Atomicity basically says that all of the changes that you're making as part of a transaction happen all at once in a

single indivisible unit. A right doesn't kind of part way happen. Consistency says that the table moves from one valid state to another. So there's no way that a right can leave the table in a corrupted state. Isolation effectively says that if you have multiple transactions happening at the same time, they result in a final state that's equivalent to if they had run one at a time in

some order, right? So there's going to be no weird stuff happening as a result of interle transactions on this table. Finally, durability just says once a transaction happens, it does not unhapp even if you So how do we guarantee these properties in iceberg? Well, uh, the atomicity property is satisfied by that atomic swap from one metadata file to the other. And it's really the catalog's job to

assume or rather to enforce that that swap is atomic. The uh consistency bit, well, while you're writing that new snapshot, readers don't see the new snapshot until you commit. And if you fail in the middle of that process, you've left some files lying around, but the table's still fine. Like in that situation, if I had crashed in the middle of creating snapshot one, snapshot zero is still

chilling. It's still cool. Um the isolation property you satisfy by applying these snapshots in some total order. The catalog will basically move from one snapshot to the next to the next to the next in some well- definfined order that all readers see. And for durability, I mean, look, you're storing this stuff in S3. It's as durable as S3 is. I'm pretty sure that S3 is going to

outlast all of us. So uh so I think that property is pretty well satisfied. So if we want to get really pedantic about it, uh the kind of isolation guarantee that iceberg gives you is what's called serializable isolation, which means that the transactions appear to have occurred in some total order that all readers see. So uh if I'm, you know, reader A and reader B, reader A

is not going to see snapshot zero appearing before snapshot one. Reader B will see snapshot one appearing before or snapshot zero appearing after snapshot one. They'll all happen in the same order. Uh the way that it handles concurrency is through what's called optimistic concurrency control. This means that if one if multiple writers are trying to write to the table at the same time, one of them will

win and all of the others will lose and they will either have to back off and retry or they'll have to abort. Uh, and as we'll see, uh, this can be kind of tricky in production. Um, we'll talk about that in a minute. So, we have transactions. That's awesome. Schema evolution. Uh, recall that that was very difficult to do well uh, in our previous world. Uh, now

schema evolution is effectively instant. Uh, the way that iceberg does this is really interesting. So, when iceberg defines a schema, it defines schemas referring to fields by a numeric ID. So it doesn't say you know I have columns A B C D and E. It's I have columns with ids 1 2 3 4 and five. And this is really useful because if let's say for example I

drop column A and then later I create a new column whose name is also A but is logically a different thing happens more often than you'd think. then there's no ambiguity about whether I'm talking about past A or present A when I'm referring to a particular column because everything's referring to them by an integer ID and the field's name is just a property. It's not an identifier.

This means that if I want to change the schema, really all I have to do is write a new set of schema metadata to the metadata file and commit the new metadata file. And now my schema has changed and evolution just sort of happens as a side effect of that. Um there's a little bit of metadata that you have to store with your data files so that

if they're referring to these fields by name that you can map the names to ids, but that's a very minor detail. Um, so scheme evolution, think we've got that under control. The table knows what its scheme is and it's very easy to mutate it. Repartitioning, which was a nightmare before, uh, is instantish. So what do I mean by that? So in an iceberg table, what partition a

data file is in has nothing to do with where that data file is in storage. In fact, you can even lay out your data files across a bunch of different randomized prefixes. And iceberg does do this on S3 uh to improve the number of reads that you can do at the same time because S3 rate limits reads based on prefix. So, uh it gets better though. Not

only does the partition that a data file is in have nothing to do with where the data file is, it actually doesn't have to have anything to do with the actual data in the data file itself. at least not directly. Instead, a data files partition is described by a versioned function called a partition spec that's applied to the data files data. So, if you want to figure

out what partition a data file is in, you apply this function to its data and that gives you back a partition which is effectively just a tupole of values. Um I've given a like very truncated example because these things are very difficult to put in a way that will fit on a slide and still be readable. Um but in this sort of snippet we've declared a partition

column called TS day. Uh and it's derived by taking the field with ID 1 and truncating it down to the nearest day. So if you wanted to change the table's partitioning scheme, for instance, if you wanted to partition by month instead of by day because your partitions were getting too numerous, uh the only thing that you would need to do to change the partitioning in the table,

at least you know, lazily speaking, is to make a new version of the partition spec that describes a new function and commit that change to the metadata file. None of your data has to change immediately as the result of that. and any data that you write after this will be written with respect to that new partitioning Now this does mean that you still have the problem that

if you want to sort of optimize your table so that everything is in that new partitioning scheme you have to do a rewrite. But the thing that's nice about this is that you don't have to do the rewrite immediately. You can wait and you can even be selective about what you rewrite. For instance, uh you might decide that for performance purposes, you really want the most recent

year of data in your table to be partitioned in a particular way, but the rest of your data can stay in the legacy partitioning scheme forever because you're hardly ever going to read it. And when you do, you're okay with that being a little bit slower or kind of more suboptimal. And this kind of gives you a lot of power in terms of tuning that sort of

cost performance trade-off, especially once your tables start getting just enormously huge. So these are the core benefits of iceberg, but they are by no means the only things that you can do with iceberg. And in fact, a lot of the kind of cool stuff that you can do with iceberg tables is a consequence of the fact that it consists solely of this sequence of snapshots at a

point in You can do all kinds of stuff with this. The first cool thing you can do with it is time travel. So you can actually query the table as of any point in time or in kind of snapshot history that you still have stored. Um if you uh query as of a particular snapshot, it will just query that particular snapshot ID's metadata file. If you want

to query as of a particular time, the planner is expected to be smart enough to look at the snapshot history, find the one that would have been current as of that time and query that. So that's cool. Gives you a lot of of ability to kind of look back in history and see what changed. You can, you know, build change log tables and stuff like that off

of this too, which is kind of neat. Uh the other thing that you can do with this is disaster recovery. So if somebody uh scribbles all over your table, instead of you know them's the brakes and you have to go and and fix it manually, you can just decide to tell the iceberg table, hey, um the snapshot that you had back in the past, that's the current

snapshot now. And you can do that as of a snapshot ID or as of a time. Um and this does a lot to kind of uh pull you out of the fire if you happen to make a bad write to a table. You can also start to treat this snapshot history kind of sort of a little bit like git. So like in git, snapshots can be tagged.

They can have names and you can define a branch which is really just kind of like a mutable tag that moves forward as writes to the branch uh commit. Uh and that gives you a lot of fun things that you can do. Here's an example. Let's suppose that uh you're on your main branch of your table and you decide that you want to make some changes uh

but you're not sure that you want to commit those changes to prod just yet. So you create a dev branch that starts at the latest snapshot on main. And then you do some writes to it. And while you're doing these writes, notice that anyone that's querying that main branch still sees snapshot three. So you're not doing anything to like mess with them at all. And you can

go in and validate in this case snapshot five and make sure it looks good. And if it does look good, then you can effectively stitch dev and mains histories together with what's called a fast forward. Now the big difference between this and git is that there's really no well-defined notion of a merge here. So if main uh happened to have another commit that was a kind of

a child of snapshot 3, you wouldn't be able to merge dev into it. Um that limits somewhat what you can do with this. But the thing that's really nice is that this kind of enables a pattern that's becoming known as write audit publish where instead of writing directly to the table, you write to a branch, validate the branch, and then effectively commit the branch forward into the

main. Um, which lets you do a whole lot of highlevel semantic checks before you actually make this visible to readers, which can improve just the overall data quality of your of your system by quite a lot. Uh so as I alluded to earlier, not all is sunshine in kittens. Uh there are things that you need to be able to do to keep these tables healthy over time

in prod. And if you don't, uh you will regret it. Uh without maintenance, your performance of your iceberg tables is going to degrade. Uh and the cost of storing these iceberg tables is going to go up. And there are some ways that this happens. The first uh I like to call snapshot bloat. Uh if you keep every single snapshot that you do and you do dozens of

writes to a table a day, uh you will end up with thousands and thousands and thousands of snapshots and every time you make a new metadata file, you will have to read the entire history of the universe, add one snapshot to it, and write the entire history of the universe back out. In some cases, this is what you want. In most cases, it isn't. Uh so what

you want to do is you want to kind of garbage collect old snapshots after a period of time. There's a batch operation that will do this. It's called snapshot expiration. Uh it's you can control uh what snapshots are expired with two different parameters kind of the maximum age of a snapshot and the maximum number of snapshots that you want to keep. Those two things are are important

because you know if you don't write to the table for a long time, you don't want all snapshots to fall off the face of the earth. You really only want the ones that you don't want to keep around anymore. Um those parameters that maximum age and the maximum number to keep can be set on the table level, but they can also be overridden on a branch or

tag basis. And uh this operation effectively it's like any other transactional operation in iceberg. It creates a new metadata file. It doesn't actually create any new manifest lists or manifests or anything like that. It's just an update to the metadata which makes it quite fast. Another problem that you'll find is if you have writes that abort for any reason, they will kind of leave bits and pieces

of partially committed snapshots lying around. Now, as I mentioned, uh that doesn't actually hurt your table at all uh because the table is guaranteed to consistently move from one uh from one valid state to the next, but uh it does start to increase your storage costs over time, especially if you do a really really really big write that fails 95% of the way through. So, every once

in a while, you need to go through and garbage collect all of your orphan files. This process effectively involves enumerating all of the objects in the bucket and comparing that with kind of the closure of all file paths from the metadata file on down um and uh just getting rid of files that aren't on the list of valid files in the table. Then finally uh you are

also victim to a very common problem that is not unique to iceberg called the small files problem. Uh small files can be caused by a bunch of different things. You might have a Spark job with way way way too many tasks. Uh that's writing a bunch of teenytiny files when it really should be writing a few big ones. You could be doing a lot of merge on

read updates in which case you have a lot of delete files and a lot of those tiny little sidecar data files added on. Um you can be doing a lot of small appends or a lot of small deletes. Uh you might also just have partitioned your table poorly, you know, by something that's really high cardality. you've got way too many partitions. Um, and so every time you

write that gets spread across all your partitions and your resulting files are quite small. This can be can be solved though it's solved through a process called compaction. Uh, and compaction just looks one partition at a time, takes all of the data and delete files that exist in that partition and it kind of tries to squeeze them down into as few files as it can. Um, that

has a ton of different parameters that control how aggressive you want to be about the squeezing down and how many partitions you try to do at a time and how often you checkpoint this compaction process by committing a new snapshot with the compressed partitions replacing the uncompressed ones. Um, but that's something that you're definitely going to want to be very mindful of doing uh as your table

continues. So those are your big maintenance challenges with iceberg. There are also a few rough edges with the format uh that you're bound to encounter if you deploy this at any kind of scale. Uh the first that I alluded to earlier is that optimistic concurrency control really does pre present kind of a challenge in production because you kind of have to operate in a world where at

most one thing is writing to your table at any given time most of the time. uh and even if you're really really careful about doing that with the kind of uh the real work that you're trying to do to the table, you also have to worry about the maintenance work and the real work conflicting with each other. So for instance, you might have a compactor that fires

on a regular schedule that takes long enough that it collides with a normal right and that right may have to abort because the compaction gets in its way. Um this is a big problem. It's something that the iceberg folks are very well aware of. Uh there's been a lot of work going on in the context of the iceberg rest API which we don't really have time to

cover in any detail here. Uh around something called server side deconliction which is the idea that the catalog really should be trying its best to merge multiple transactions that have no possibility of conflicting with one another together uh in a way that allows both of them to commit. Um but that's very much like an ongoing So all the maintenance stuff uh that I talked about iceberg is

a table format. Uh you basically have to rely on your query engine or some other thing to do that maintenance for you. There are many many vendors out there in the world that will try to sell you a thing that will do automatic maintenance on your iceberg tables and they will probably charge you an arm and a leg for it. um S3 tables being one of many

many examples of this. Uh if I had to give a piece of very broad advice, I would say if you're just starting out with iceberg, your best bet is probably just to run spark jobs on a cron to do some of these maintenance jobs. Um even at the scale of dozens of tables, it's really not that much maintenance burden. And a lot of these systems don't expose

all of the knobs that you can tune. Uh that if you were to just call a spark procedure yourself, uh that you'd be able to go and dial that in. And sometimes from an operational perspective, that can make a lot of Finally, um you know, iceberg's been around for a little while, but it's only just started to kind of go parabolic in terms of adoption. So engine

support is still a little bit uneven. Uh, Spark has really good iceberg support. Trino has really good iceberg support. A lot of the rest of the stuff, it's a little bit hit or miss. Some engines can only read iceberg tables but not write them. Uh, some draw a weird distinction between what they call internal iceberg tables and external ones. As best as I can tell, like that

really just means that the internal one is their format that's kind of pretending to be an iceberg table from an API perspective, whereas the external ones are written according to the actual spec. Um, there's also like stuff that you're used to from databases like multi-table transactions. Catalog support for that is really really thin. I think like Dreo might be the only one that's really got it fully

fleshed out right now. Although, you know, Polaris is also is also making great strides there. Uh if you want like the most exhaustive comparison of iceberg feature like matrix across uh engines in the history of time, uh a company called Ole publishes this uh this comprehensive breakdown. They go into just incredible detail on what each engine can and can't do and they're keeping this up to date

which is awesome. So in conclusion, um we talked about how table formats ushered in this idea of the data lakehouse, right? All the nice things about the data warehouse and all the nice things about the data lake put together. And based on the last few slides, you might think that the data lakehouse is maybe selling you a little bit of a bill of goods, right? that like

there's a lot here that you have to do yourself, right? Um, and that's kind of true, right? I think just like everything else in our field, uh, it's all about trade-offs. You gain some things and you lose some things, right? So, what do you gain in this kind of lakehouse setup where your storage is completely decoupled from your query engine? You gain a lot of flexibility. You

can have a lot of different kinds of query engines that are operating on the same data stored in the same iceberg tables. And for some places, particularly, you know, people who are dealing with multimodal stuff or people who are dealing with, you know, batch and streaming workloads working together, this has a lot of really nice benefits because you can sort of pick best of breed query engines

for each of your different tasks and as long as they support iceberg, you can go off of a common carry. Um, there are even folks that are crazy enough to do things like replace Kafka's storage tier with iceberg and just have the Kafka API sitting in front of it. Um, but you gain Also, you know, as I alluded to earlier, there there's this multimodal thing here where

like if you want to have video and images and tabular data all sitting in one place, having that all sitting in your S3 bucket and being able to query that is, you know, nice if your query engine can do that sort of thing. There's a sort of like big selling point that the lakehouse vendors give you talking about vendor lock in uh and how iceberg prevents you

from being locked into a vendor or how table formats generally prevent you from being locked into a vendor. You know that's technically true. I think that if you look at the kind of way things are going, a lot of companies are tending to integrate their own solutions vertically and call the resulting thing a lakehouse if the storage engine is or the storage tier is in S3. Um,

looking at you data bricks, you know, Delta Lake is awesome, but you know, you basically have to be in data bicks ecosystem to be able to use it effectively. Um I think really you know and you're also just you're gonna see better integration with a given piece of open source software at the company who's sponsoring all of the people developing that thing you know like uh tabular

got bought and immediately you know everything started to kind of look a lot better for the folks that were acquiring tabular. Um so we talk about what you gain what do you lose? Well, there is a lot to be said about working inside of a system where the query engine, the data catalog, and the storage engine have all been co-designed and co-optimized for one another. You can

get a lot of performance improvements off of that. You can get a lot of just operational simplicity off of that because your system is very vertically integrated and nice and cohesive. Um, is that worth it? Well, you know, you also do have to factor in like if you're in the Snowflake ecosystem, there's a lot of stuff that Snowflake does very well for you that you'd have to

patch together from a bunch of different open source tools if you were doing it the lakehouse way, but you also kind of have to look at that Snowflake bill every month and realize that like it's going to take you an awful lot of work to migrate off of that uh off of that platform. Uh, and really like is it worth it? It really just it depends on

where your business is. It depends on what your goals are. It depends on where you're going. Um, if you want to know more about that, I am available to ask questions too. So, wrapping up, we've kind of established that raw files on object storage is the wrong abstraction for tabular data and that something new was needed. Uh, and iceberg, I think, fits that bill. It makes the

tables that sit in your data lake a whole lot easier to reason about, a whole lot easier to evolve, and a whole lot easier to test. However, you know, it is not without its trade-offs. Uh, you need to do periodic maintenance on these things. Doing operations on it can be tricky at scale. Uh, and for some of these query engines, adoption for iceberg is still in early

days. So, you're kind of going to have to deal with a little bit of rough corners. Overall though, if the situation that you're in right now is you've got a bunch of files scribbled all over a data lake, man, you really should consider upgrading to a data lakehouse. your your uh heartburn will thank you. And with that uh thank you very much for your time and attention.

Uh I'm here to take questions. Uh I'm also I am a freelance data engineer. I help startup companies build, manage, and scale their data infrastructure. If that sounds like something you would be interested in talking more about, uh please feel free to use that QR code or just email me directly. And that's it. I'll take questions. [applause] Okay. Are there any questions? hi. Thank you very much

for the talk. It was brilliant. Um, can you give us some examples of the the types of data that you'd see being stored in in iceberg? >> Totally. Um the biggest one honestly is event data like high volume event data. uh particularly if you're looking for a place to store data archally, right? A use case that I see a lot these days is that folks will keep

the kind of hot tier of their event data for like the most recent 30 or 60 days in an engine like ClickHouse and be able to do kind of real-time OLAP on that. And then the rest of it they'll tear down to iceberg and just append it down as it as it kind of falls off the edge of the world with Clickhouse. And then if they need

to do more archival stuff or they need to do like disaster recovery or backfill, they'll go down to Iceberg for that. Um, I've also seen a lot of folks that are doing CDC off of their OOLTP databases directly into iceberg and then using that to have multiple query engines kind of chew on it depending on uh the size of the problem that they're trying to solve. So,

you know, you might have some things that are using duct DB and then other stuff that's firing up a whole Spark cluster off of that same tabular Okay, there's a question. Nice talk. Uh my question is around the [clears throat] like the offloading data from CDC data from like lot of like the VGM connector go to Kafka and then that's kind of the one of the best

practices. >> One of the things we faced with iceberg uh sync connector like they don't support deletes and updates. So we end up writing to a raw table like everything and then we run a spark job to do the maintenance. So >> yeah. So so that's that's interesting. So the what what you're talking about is kind of the the way that the kind of best practices quote

unquote way that CDC down to iceberg is done. The idea there being that you're taking the kind of raw insert, update, and delete statements off of let's say Dezium and you're writing you're appending those into a file and then you're sort of tailing that list of appends to do the merge and apply. Um that honestly I think is driven more by trying to make sure that you

keep up with the speed of the data coming out of Debzium. Right? So one of the things I didn't really get into this in the talk but one of the things that's interesting about iceberg snapshots is that you can have different kinds right so if you have a snapshot that only appends data to a table and doesn't do any updates or deletes you can actually ask the

the table give me all of the stuff that's been appended to the table since the last time I checked and use that as the input to emerge into operation to like update the actual sort of uh materialized view of your table in in iceberg And the thing that's really nice about that is that especially if you have a table that's being written to at like an absolutely

blistering rate of speed, you really don't want that debezzium topic to get huge. So better to just offload it all into that append only like that that append uh snapshot structure and then come back periodically and materialize it. The other thing that's nice about that is that if you're really really clever, you can create a view on top of that materialized view and the change log stream

that will allow you to either say, well, look, I want this query to be pretty fast, so I'm only going to query the materialized thing. But if I want it to be really up todate, I can query the materialized thing and then like actually just stitch together the rest of the delta at query time and get something that's much more up to date with the current state

of the table that way. And you can kind of choose which one of those you do. And funnily enough, if you look at the internals of like a hoodie or a Delta Lake, it looks very similar to this where you've got this sort of change log of things coming in that periodically gets snapshotted into kind of the the materialized view of the of the universe. >> Yeah.

I I think the idea is like use metal view. Currently what we do when we write, we use the merge and write on the read side. We use the copy so that the spark both both writers and readers have the best performance. But >> right, >> I look into the metalized view solution. >> Cool. Cool. Thank you. >> Okay, a question in the back. Do you have

any suggestions of would a lakehouse be overkill just for back fill or like or would I be better off just using like parquet with duct DB like because right now I'm looking at a use case of we have a a backfill issue that we're trying to fill fill in like lost data on like a daily basis maybe for on the realm of like an hour or maybe

15 minutes of lost data. I see. Um I think it depends. I think if you're the nice thing about storing this data in iceberg is that it's very easy to pinpoint the exact data that needs to be backfilled by querying it and that query is going to be reasonably efficient assuming that you've partitioned the table the right way. I think the problem that I've always seen with,

you know, a bunch of files, just paret files that you query with duct DB, is that you're running into some of the same problems that I was talking about and sort of the the the quote unquote old way of doing things, right? Where if you need to know which parquet files to scan to do that backfill, you have to look at the metadata for every single one

of those parquet files to figure out which ones contain the data that you want. And of course, you can be clever about that if you've laid your files out in a particular way, but sometimes people are and sometimes they aren't. You know, I've particularly seen like folks that really struggle with trying to backfill off of like let's say you have an S3 sync connector from Kafka, right?

That sync connector is just going to write a a file per um topic partition that's just named by the offset of the first record in that file. and it's it's impossible to really tell at a glance which files to scan. So I think in those situations like again though it depends. I think you have to populate that iceberg table somehow if the back fill is a very

occasional operation. it might be better for you to save the cost of doing that query for the few like handful of times that you need to do it and have that common mode write path be really really straightforward and simple. Does that make sense? >> Yeah. I have another follow-up question. Have you seen anybody do anything like crazy like partition down to the hour at the folder

level? Like is that is that like overkill as well? So I have or rather I've seen high cardality partitions in iceberg for sure. Um it can be done. I think really the question how big are your partitions, right? Because you're kind of there's a trade-off between being selective about what you scan and having the individual partitions be large enough that your query engine can actually chew on

them and you're not spending most of your time opening and closing files. So I think could you partition hourly? Sure. If you're getting in tens of gigabytes of data per hour and you're partitioning by hour, that makes a lot of sense. You'll end up creating some pretty big files that way. Your query engine can chew through it. You can probably cluster those partitions with like something that

that speeds that up a little bit. You know, zordering or something along those lines. But like if you're partitioning by hour and you get a couple megabytes of data an hour, then probably not. you'd want to do that by the day. I think the thing that's really nice though I'll say about iceberg's partitioning spec is that you're allowed to get that wrong with very little penalty. Right?

So for instance suppose you you thought initially oh I'll partition this by hour. Later on you're like oh these partitions are numerous and very small and I need to get rid of them. Well, you can change the partitioning spec to partition by day or by month and then just lazily rewrite your partitions one at a time through compaction to make them larger and less numerous. Um, and

you don't have to talk about like, well, I'm going to have to stop the universe to rewrite this entire table because I got my partitioning wrong the first time. >> Okay. Yeah, because uh yeah, right right now we're just focusing on on back fill because we have our sort of live hot data is in in a time series database in in flux and so >> uh >>

we aren't really looking at having a lakehouse for doing all lab operations >> Sure thing. Sure thing. >> Thanks. >> No problem. Any further questions? Well, in that case, thank you, Alex, for your presentation. Great. And, uh, please stick around for the keynote address at 3 PM in the other building at ballroom DE. Thank you. >> Great. Thanks, folks.