How is OS integrity achieved in the Red Hat in-vehicle operating system?

This talk introduces the Red Hat In-Vehicle Operating System, known as RIVAS, and discusses the technological advancements aimed at ensuring OS integrity and tamper-proofing within automotive systems. The speaker highlights the transition within the automotive industry towards fewer, more powerful ECUs and emphasizes the importance of utilizing a secure, open-source Linux-based OS like RIVAS, certified up to ASIL B level. The discussion covers the architecture and update mechanisms of the operating system, detailing how technology like OS tree and composite file systems enhance security and ensure the immutability of the operating system. Additionally, the speaker delves into the use of feature on demand for application installations, drawing parallels with mobile app updates, and explores the relationships between Red Hat products and community-driven projects for automotive applications.

[music] >> Good morning everyone. Thank you for for joining. Thank you for being here. I am the last man standing between you and lunch and I know this is a critical slot in the agenda. So I have two challenges in front of me. A, making this topic interesting and B, trying to make it clear and fast so that I don't stand too much between you and lunch.

Um so the topic of the day is I have a few keywords in there. Uh the one is, you know, a mile long and that's the second sentence in the title here, the Red Hat In-Vehicle Operating System. I'm going to introduce it a little bit as a context but the idea is not to speak too much about this and more to go into the technology that is

enabled in it uh in order to achieve OS integrity and tamper proofing. And we all know that this is a critical piece of uh software, a critical critical feature of every system in automotive where, you know, or in edge in general, where the end device is in the hand of the end user. we've seen that different version of that picture before. Uh so I'm very glad I'm

the only I'm the second speaker of the day because I can still show it. At the end of the day you'll be sick with it. So I'm not going to spend too much time onto it. It's the classic the architecture of of the automotive industry is changing, you know, less ECUs, more powerful, different functions per ECUs and everything. Um the one thing is that this is a

transition that has happened before in other industries, especially in the enterprise world. Uh you know, Red Hat has been leading the OS been part of that transition in enterprise world going from uh dedicated dedicated servers to multi-tenant servers and applications. in that's in that regard, Red Hat is interested in building an operating system that is based on what the the operating system that exists in in the

enterprise world, that's called Red Hat Enterprise Linux. And because our marketing teams like long names, they figure out that Red Hat In-Vehicle Operating System added a lot of letters to to fill in on the slide. Uh the nice thing is that unlike uh Red Hat Enterprise Linux, which as everybody has called RHEL for I don't know, 20 years at this point, uh that only marketing has only

approved the use of RHEL in I think the last year or 3 years, uh we're already allowed to use RIVAS for that title. So, I'm going to be speaking about RIVAS uh in the rest of the the session just a lot nicer. Uh so, RIVAS is meant to be an an Linux-based operating system, fully open source, fully uh available based on on RHEL. It is certified ISO

26262 up to the ASIL B level. And as every other operating system that we want to use, it needs to be secure. Um so, again, one of the uh context-setting slides, what does Linux brings in the automotive industry? Uh I also don't want to go too much in the detail of this. Uh this is not the main focus of my talk, but I'm going to try to

speak long enough so that if you're interested in the topic, you can pick some keywords in the slide while I keep the air busy in the meantime. But if you want me to keep going, I can keep going until you've picked up enough keywords in the slide that, you know, you're satisfied with it. Uh but, you know, uh this one. So, one thing that I do want

to speak about, uh how was the how was the sausage made? Uh I like don't really like that one, but how was the process of building it? Uh in the enterprise world, we are we have the higher picture, and I'm going to start with the higher part and going from the right to the left. On the right side, we have RHEL, Red Hat Red Hat Enterprise Linux.

This is a pure product from Red Hat. It is fully controlled by Red Hat, fully uh like, you know, what goes in there, when it goes in there, updates level, uh customer support, customer will ask will have requests and everything, but it is completely controlled by Red Hat, and it is the product, it is what we sell. Um, before that, we have what we call CentOS Stream,

and CentOS Stream is basically the public in development version of the of the product. So, if you want to know what's going on in Rail, if you want to have a look in the future of Rail, we invite you to go and look at CentOS Stream. And basically, you'll see, you know, what will be Rail in 3 to 6 months, something like that. And then, this is

a shared space. This is shared It's It's built in the CentOS community, and that's what the name CentOS Stream, and it's a shared space between community and Red Hat to collaborate in building the future of Rail. And then, on the top of this, we have a number of SIGs, which are all the way on the left here, which are completely community space. And those are group of

people that want to build on top of CentOS Stream, on top of what will be Rail, in order to add features or complement what is in the product. Uh, so for example, the Hyperscale SIG, all the way at the top there, uh, builds a new systemd, a new kernel, or different different configuration kernel configuration options than what's available in CentOS Stream. Uh, so they they aim to

complement that. For Automotive, we basically build a similar model. We have the Automotive product that is completely controlled by Red Hat, and then we have a a shared space in the automotive uh in uh community umbrella, uh, that's called Automotive SD. That's The full name is the Automotive Stream Distribution, and it's the it's it's basically the same idea as CentOS Stream above. That's why reuse the names,

basically. It is the So, if for everything that I'm going to present to you today, those are features that, you know, Red Hat has been working on for the product. But all of the technology that's in there is available in Automotive SD. If you go today to seattlesantos.com or /automotive, you're going to find all of the documentation that relates to not only the automotive signal the way

at the left that is the and purely community space, but also all the documentation that relates to at OST. How is the how to build images, how to customize images, how to to learn about the technology, the the mixed criticality feature that is part of the product that is certified in rivals. The technology for it is already available in at OST. You can go to for it

today, you can learn about it, you can use it. So, and here today is available available for you to play with in the authority space. without further ado, we want to talk about updates. We want to talk about the way the system is built. What we're used to is, you know, when we look at the the partition of the system, we have the firmware on one side,

we have a boot A partition that starts user space A, and then we have some data partitions on the side. And then, when we want to do an update, we basically download the updates, we install it in slot B both for the boot sequence and for the user space, and then at the firmware level we indicate, "Okay, now I'm we're good. The next time just boot on

slot B, and we'll have the same data space." The way we usually protect this is by using DM Verity. You know, all of these partitions except for the data partition because the data partition needs to be written writes, and DM Verity is not going to allow you Well, the thing is rivals does not have the same model. So, well, before before I go into that, what are

the pros and cons of this approach? The pros it's clearly something we we know, it's something we've been using, it's something that's that's we're familiar to us. We have a clear atomic state, it's either state one or state B. We can do atomic updates, you're either in A or in B, and you can switch easily by going from A to B, and you can roll back. If

you're in B, you can go back to A. The problem is or the limitation of that system is that you basically need for your operating systems two full copy. You know, I'm not going to talk about the the data part, but you basically have you could have two full copy of your your all operating system. So, your storage is be by is basically divided in two. And

every time you want to do you want to to change version, if you want to have new something new available in your operating system in in either of these partitions, you need to do a full reboot. Um and my transition again, what's the what's coming in in in Rivos? Well, Rivos has a has a different model here. the left part is the same. We still have our

firmware partition, and we still have our boot A and boot B partitions, but we only have one user space partition. And that's user space partition needs to be partly read and write. So, the question becomes how do we make this secure? How do we make sure that this partition remains secure, doesn't change, you know, a random bit flip is detected when you have a partition that is

going to need to be partly read and write. And that's the whole question that we're going to debate today. So, pros and cons, let's hold on that a bit more, and let's go and dive into what's the technology that's underneath that. Rivos AutoSD and also of the Red Hat product family are using are relying on the technology called OS tree. And the way to think of this,

it's basically the Git for your operating system. It's the Git of your root FS. Uh that means it relies on a Merkle tree, just like Git does, uh which allows you to do file-based deduplication, uh which just like Git allows you to do atomic updates. You're either in commit A or commit B, you're never somewhere in between. Uh you can have multiple version available. You have multiple

commits, you have multiple version available. You just you no longer need to have only two, you can have more. Uh and just like Git, you can do rollback. If you're not happy about, you know, the last commit you've made, so I'm sure we've all done this, you reset to the previous one. one of the fundamental value of OSTree is that it guarantees that the content, if you

deploy, you know, one specific OSTree commit over a million vehicle, you have a guarantee that this million vehicle is going to have the exact same content. There is no possibility of drifting. Either your vehicle is going to have that commit or it's going to have another one. But at every single time, if you know which commit that vehicle is running, you are able to know which content

is in there. If you have a CVE, if you have an safety issue, you're able to say, "Those are the commits that are impacted." And if you're able to track which vehicle have which commits, you're able to know which vehicle is susceptible to which CVE, to which safety issue. OSTree, one of the challenges that we have about this technology is that OSTree lives in the file system

under a /sysroot/ostree, and we need to be able to write there in order to do updates. So, the OSTree the OSTree commit itself is going to be mounted as read-only uh and part of the OSTree uh you're going to have, you know, /usr, /etc, or /boot, uh but you need to keep, you know, /sysroot read-write in order to be able to do your Git pull, the equivalent

of your Git pull, you know, when you want to download and write the next update version. Uh so, how do we secure that? We have a few options. We have DM integrity, but the problem of DM integrity is that doesn't follow It doesn't support full device authentication. So, it's not going to be useful for us because uh we have uh you know, we can't do partition base,

we can't do file system base. We have dm-verity that we're very familiar with, but dm-verity works at the block device level, so the partition. So, we're able to secure the entire partition, but we just say that we need part of our partition to be read and write. We need We need slash sysroot to be read and write. So, we need something that works, and then we have

fs-verity, and fs-verity works at the file at the file system level. So, fs-verity seems like a good match for us. Uh you know, it's file base instead of a partition base, and it ensures immutability of content. But fs-verity is not the only so is not the only solution. It's not the It's not the solution by itself because it verifies and it's ensures immutability of the content of

the file, but it does not ensure immutability of the directories in which that file is. It does not ensure immutability of the metadata of the file. Your file that was here you know, 777 can become 544 or 644. so, it's possible to change the permission of a file. It's possible to add a file to a directory. It's possible to remove a file from a directory, and fs-verity

is going to be completely fine with it. So, that's of course not acceptable. We need to have a way to seal the file system beyond just the and this is where compound fs enters into the picture. Compound fs basically allows you to to seal the entire file system. Uh and it's it's uses uh Basically, it it creates you have a >> [sighs] >> you have a compound

fs file that is going to be a binary file. It's It's a file system image to begin with. It's Compound fs is a file system like the name says. It's based on EROFS that has been used in the Android world a few years now. Uh and it's basically a allow In the composite FS file you are going to find your entire partition file folders and for every

of them you're going to have, you know, the the location in which they are, the name of the file, the hash of where to find the the the reference to the object in Merkle tree as well as the FS Verity hash that ensure that the content of the So, with this we basically are able to have our OS tree commits that is, you know, again {slash} user

{slash} sfs {slash} boot and then in your OS tree commits you're going to have that composite FS file. And that composite FS file is going to have each file on the system here, you know, the name, the directory, the permissions and again the object name and the object file digest. The object name allows basically the kernel to find where in the Merkle tree is the content of

this file located and the object file digest allows FS Verity to guarantee the file that you're reading on disk is as the exact hash that you expect to have when you build the image. So, from a verified boot perspective, where are we? in our secure enclave on our SOC on our ECU, we have our root of trust key. That root of trust key is then used at

the boot level at the boot level at the firmware level to verify the boot stack. In the boot stack we're going we have a public key in the init ramdisk in the initrd and that public key then is used to to verify the signature of the OS tree commits. That OS tree commit itself has the composite FS file and that composite FS files ensure the guarantee of

the content of the {slash} of the {slash} partition, the root FS itself is in there. And then we have so we have full chain of trust going from our you know, root key at the top there all the way to your user space on the right. And then the partitions at the bottom here are read and write partitions which will need to be secured by other means.

So here everything that happens in the root file system is not going to go unnoticed. If you have a random bit flip in a specific file the kernel is going to detect it because the FSVerity hash is not going to match when you open the file. where do we stand on the Rivals model then? one of the pro it's it's a technology that actually been used in

Red Hat for for a few years. Uh one of the cons is that it's new in automotive and we all know how automotive likes new and shiny things. Uh unlike the enterprise world. Uh but on the pros again we have a clear atomic state. We have atomic updates. We have rollback capabilities. So all of data exists in the traditional AB booting world exists with the software based

AB booting. But we have more benefits. We have these file based duplications which means for the same storage the impact of the operating system on your storage is going to be divided by is going to be a lot lower. Divided by two maybe a little bit optimistic but it's definitely going to be a lot lower than if you have two full copy of your system. And then

on the cons it's going to have the same issue as the traditional AB Uh you know, if you need to if you want to move to a new version of the operating system you need to do a full All right. So this is where challenge is going to happen because this is the second demo of the day. The first one went very smooth. So Murphy's Law should

kick in at some point. So you've been warned. Uh if you're if that goes well and you're presenting another a this afternoon I would check things because that's you know, at some point we are going to run into a the demo effect Uh so, I'm starting here uh virtual machine an OSTree virtual machine that is itself based on uh OSTree and compos FS. Originally, I wanted to

show you uh the impact of compos FS and FS-verity on using three different virtual machine. Uh the problem that I run into is that our integration of compos FS is such that I actually could not build any image without compos FS anymore. Uh so, I went to the next best things. Uh I created a shell script uh that basically is going to allow us to build to

work with an OSTree repo but not a full operating system image, just an OSTree repo. Uh so, what we what we're doing at the top here, we're starting, you know, your git init equivalent, OSTree uh repo init. Uh Uh so, we build we're building an OSTree repo. We're creating some folders, you know, a couple of files in it, uh a subfolder, and a nested file. Uh each

file has, you know, hello from file one, hello from file two, hello from nested file. So, we just put some things in there. And then at bottom here, we're committing that uh you know, that file system, those folders into the OSTree Now, what when we uh >> [snorts] >> I'm going to scroll up a little bit here because otherwise, yep, there we go. we're going to do

just OSTree. We do a git check a git check out. We do an OSTree check out of what we committed before into a new folder, and we're going to check what it is. the first thing we do is we This is nice I like the way this works with, you there, a little bit nicer. Uh so, we're looking at the content of file one, hello file file

one, everything is fine. Uh then we do an echo modify content into this file, and the command returns a zero, so it looks good. And then we cat the file. We can see that the content was modified. We can do a chmod on that file, and the permissions were changed. Uh we can create a new file. Uh you know, this is echo new content into new file,

and we see the new file being created. We can remove a file, and everything it works as I said it would. But, this is, you know, me actually running it and not not just you trusting me. Um so, this is not what we want. Everything is mutable. Everything can change. Uh what about fsverity? Uh so, that's where it gets a little bit more interesting. Uh where are

we? Demo two here. There. Ah, come on. There. Uh so, we're going to do another uh Git checkout in another folder, and we are enabling fsverity for every single file in there. Uh we can do fsverity measure, and that gives us the uh SHA of the the content of that file. And if we, you know, look at the content, it's still hello from file one. And then

when we try to modify that content, then we get an I/O error. Per operation not permitted. Uh so, we can't change the content, we can still chmod it. We can still change its permission. We can still create a new file into that directory, and we can still remove a file, and fsverity is completely fine with it. So, this is still not what we want. So, the third

the third option is, of course, okay, let's look at composite FS with OS tree. uh because now we're doing a a checkout of OS tree, but we're doing a a checkout of that OS tree into a composite FS And that composite FS file is a file system, so we need to mount the file system. So, that's what we're doing at the top here. We're doing an OS

tree checkout {dash} {dash} composite FS. We have the composite FS file, and then we mount that composite FS file. And then when we do an LS of that that file system we see new directories appearing those are added by comforter first automatically at the moment. And then when we look at the file in there, you know, file one is still We're looking also at the mount options.

You can see that they are basically recognized a being comforter first months. That's the first instance there. Why do we have two lines? This is because the virtual machine in which I'm running is itself running OS 3 and so the VAR partition is in fact you know, the I'm running is on the slash root. Slash root is in fact stored in VAR root home. And that VAR

partition is actually coming from sysroot OS 3 deploy default VAR. So everything is in sysroot. We are not OS 3 system. And then if we try to modify the content we get we have a fuel a full read only If we try to add a new file, well again the file system is read only so it's not going to let us do that. And if we try

to remove it again, we can't do that. How does the the comforter first file look? There is the comforter first info command that allows you to dump the binary what's in the binary format and we can find again, you know, we have slash root slash at the top slash root at the team then our file one that txt which has which point the object into the OS

3 repo and then the FS 30 hash associated with it which allows the kernel to verify the content of the file when it tries to access it. So and the one more thing that I wanted to to show you as well is So like I said this VM is itself This VM is in itself a an OS 3 image with comforter first enabled. So you're thinking okay

this is we said compose FS is is mounting the partition as read only, so what if I just remount it as read write? I can do that, right?" Actually not. The kernel is going to prevent you when you work with with compose FS, the kernel is actually going to prevent you to try to remount it. There is no way for you to to tamper with it. So,

with that approach, with the combination of both compose FS and FS read only, we able to very much seal the entire image, but seal just parts of the image that we're interested in. We we're sealing part of the file system, leaving other subsection other part of the file system fully read and write without just being able to do anything. Uh okay, so that's the second demo that

worked today. So, if you're presenting this afternoon, beware. let's go beyond OS tree. Uh one of the limitation of OS tree and of traditional AB booting approach is that when you want to go to the next version and when you want to have applications in that next version, you Uh so, one of the thing that we we went into and this is a a little bit more

of a it's a feature, but what the way we implemented it is a bit more of a prototype at the moment uh that we call feature on demand and it's basically the ability of installing application on the running system in a way that makes the application behave as if it was a native application. So, we talked we presented very conveniently in the previous presentation bootchy and the

way that bootchy manages services through the attraction layer of system D services. Uh and so, the idea here is like, how can I install an application so that the system and system D in this case believe that this application is part of the regular OS image? It's it looks like it's part of OS tree, it behaves as if it was in the OS tree commits, but it's

actually something that was installed, you know, on demand and be made available to the entire image without requiring a reboot. Uh so, we've worked on that. There is a You can find the the POC in the the demo the chain of trust demo here. Uh and we found a way where you basically are able to track you basically are signing and are able to track the making

sure, you know, every file that's installed on the system comes from a trusted source that is not tampered with in such a way that it seems it looks to systemd as if they were native applications. so, when do we use which? When do we use OSTree and when do we use the feature on demand aspect? Well, OSTree guarantees that every single vehicle running a specific commit So,

the the way we're seeing this and would love to discuss this more is that we're thinking OSTree is going to be for everything that is specific to your entire fleet or to category of vehicle and you can define that category however you want. Maybe it's a specific model year. Maybe a specific version of a model year. Maybe it's a specific geo inner versions of a model year.

You can define that category however you want, but it's a group of vehicle and you want all of these groups all of these all of the vehicles in that category to have the exact same version of every software in it. That's going to include your operating system, of course, but also likely your middleware, your ASIL applications, and maybe some of your QM The one thing to keep

in mind here is that while while you can use OSTree to update everything, you don't have to. You can use the same technology to just update, you know, maybe you just want to update your ASIL application without touching your operating system your or your middleware. It's still going to require a full reboot of the system, but it's your your OSTree update doesn't have to include everything. It

can include as much or as little as you want. The technology itself doesn't do things. It's how you use it that's going to matter, basically. And then the feature on demand, that's going to be everything that is user specific. Uh, you know, a user that has subscribed to a specific service, the user that has downloaded a specific application, uh, you know, and at the moment we see

mostly QM apps in there. Uh, we don't think there's going to be ASIL applications that a single person within that fleet of vehicle is going to have access to. We could be wrong. Happy to discuss that. Uh, but I'm the one saying this is more, you know, if you have a specific version of a vehicle, you're going to have a specific set of ASIL applications if you

if you have a different version of that vehicle, you're going to have a different set of ASIL application. And those could be two categories and therefore in your S3 comments. Uh, so if we do the parallel in the Android world, OS3 would be your Android OS update that you get updated either over cell phones. And then the feature on demand would be your Google Play Store kind

of updates. And that's the end of it. >> The auto SD. So, um, I was wondering, is there really a line? Is it two separate streams? Aren't they So, you could say, well, Red and auto SD parts or not the almost stream parts from CentOS are already hardened, so I would derive from there and not take it as a separate stream. So, how much connection or not

connection between the auto SD and the CentOS stream exist? >> Uh, so let me go back to the picture you're referring to. >> I particularly choose this one because you have to go all back all slides. I have a second question for the end of the conference. >> [laughter] >> Exactly this one. So, you see the horizontal separator there and it's like, how thick is the line

of the horizontal separator? >> Uh, so the question is really what's the what's the relation between CentOS stream and auto SD and Rail and Rivos? Uh, that line basically doesn't exist uh, because CentOS auto SD is based on CentOS stream in the same way that RHEL is on is based on RHEL. Uh, the The only difference is we have a few packages in AutoSD that do not

exist in CentOS Stream in the same way that we have packages in RHEL that don't exist in RHEL. One of this package is surprisingly the Linux kernel. AutoSD and RHEL have their own Linux kernel compared to RHEL where we have the real-time patch set that is enabled by default where we have automotive specific configurations or default configurations that are turned on and off. The RHEL kernel defaults

to the XFS In RHEL we default to the ext4 file system. So we have a number of difference in kernel config in kernel options and config. And then in the OS update and the OS image, the way we build the the OS that is a different between RHEL and RHEL, but from a source perspective they are the same. >> And still in the assessment report which you

have for the RHEL >> There's >> There's the assessment report from Exida on the RHEL. There you basically point to the tag which was used for the assessment and then this points to the CentOS directly if I'm correct. So basically means it's also still kept in the same repository infrastructure and so on. Just a different configuration. >> Yeah. >> Right. >> Yeah. The the sources The sources

for the for the RHEL kernel are shared with the RHEL sources. At some point we had How did we do that? At some point we were because the the the workflow changed a little bit. At some point we had two different repositories. One that would just pull from the RHEL and add config on its own. I believe now we just have two different branches. We Then we

moved to having two branches in the same repositories and I believe now we've even merged to everything in the same branch with different folders for the different configurations. If I'm correct, I believe this is the current status of the moment. But there is it is really merged the same sources. >> Thanks. I have another question, but I just ask it when the others have the chance to

ask the questions. >> Uh was there another question? >> Thank you for your nice presentation. So, could you discuss a little bit the boundaries of ACLB? Because in some past discussion with some of your colleagues, for instance, they were mentioning that things like pipes will be covered in ACLD about the networking stack. Because that changes everything for an application that runs. So, I think it would be

interesting about the standard C++ library. So, what are the boundaries that you cover today and what do you plan to cover going forward? Because especially for people that write that opens or closes certain possibility. Yeah. >> Um >> Thank you. >> So, you're you're asking for the the boundaries of the the scope >> covered by ACLB? >> Today and let's say what you plan to because I'm

sure that there are other things you're planning to add. If you can ex- if you can tell. >> I I can answer this. The only question is do do we have anybody in the BU that is in the room on the Red Hat side or can I speak freely? Yeah. So, Philip is saying as an easy process as an easy product that has been certified by Exida,

there is the assessment report that can already give you part of the answer. The short of the answer is our ACL scope today is not enough. We are plainly aware of this. This is something we are working on improving. lib lib C++ scope is in our road map. We're working on that. Uh we are working also on on rest. Uh, we first system surprisingly. Uh, so we

know what kind >> network because that's the part that I care a lot also because you know each time each time you don't have a certified networking stack then you have to add a separation layer that's pipes SLV, right? >> Mhm. >> Uh, but that adds complexity and so any plan to make for specific hardware uh, at least the internet driver and the >> I That's a

very good question I don't have the answer to. >> Okay, thank you anyway. >> Like even if I wanted to I I just couldn't [laughter] answer it. I don't I just simply don't have >> thank you very much. >> Hello. Um, question about this like Merkle tree based file system that you like showed and described with like um, you you compared it very heavily to Git. >>

Like saying it's kind of the same idea, same concept and everything. Uh, like how similar is it really? where is it different and couldn't like actual Git be used for the same purpose? >> Um, so let me rephrase this. How similar is our system to Git and and could I use Git simply? Uh, so it is very similar in principle. It is completely different in the way

it's written and implemented. Uh, it's relies on Merkle tree just like Git. Uh, it is not a collection of shell scripts that Git is which I think is a actually benefits of foreign to motive system to not rely on a bunch of shell scripts. Uh, even though Git from the origin has evolved into something that is a bit more structured than it was originally. >> something >>

Sorry, I couldn't hear you there. >> Um, I mean for implementation there's there are things like Git Oxide or something that's not a bunch of shell scripts anymore, so. >> Things have improved in the Git side. Git is not Git is not optimized for anything that is binary driven. And that is one of the aspects where OSTree actually shines. So, you can do binary diff with OSTree

for doing diff between updates, between commits, that Git is just simply not going to give you. So, they are they are similar in the spirit. They are implementation. And it's also the easiest way to describe to someone who hasn't played with OSTree what OSTree is. So, that's why I keep referring to it and using it as a as a metaphor to be to some extent. One aspect

which I did not mention in technology that is how do you manage your OSTree repository? How do you manage your OSTree commits? How do you build, you know, a commit a diff between two commits? It's very easy to It's something we're familiar with in in Git world, but it's something different when you're talking about OSTree. And for this, there is there is the evolution the next evolution

of of OSTree at Red Hat is all the work that has been done under the tool called Butane. And it basically allows you to put your OSTree commits inside a container image. Which means you're able to use the container infrastructure and tooling to manage all of your OSTree You're basically able to store your OSTree repository into a container into a a container storage into a a container

registry, sorry. And then each of your commits each of your OSTree commits is a And you're able then to leverage all of infrastructure for building container image, but then you're building OSTree commits using it. You're able to audit, you're able to do CI pipelines, you're able to reuse the entire you know, infrastructure or the ecosystem that has been built around OS 3 containers management in general. And

now you're able to do to do that with basically your OS image. So, I don't know I didn't want to the the topic of the day was about, you know, composite fest and tamper proofing. So, I didn't go into that, but that's it's this the the next level is the next question. It's like, "Okay, I have an OS 3 repository, I have OS 3 commits. How do

I manage it? How do I build it? How do I actually use this?" And we'll see for a long time this was actually one of the pain point of OS 3 the there are very few companies that have been able to build that expertise and deliver it at scale. Red Hat is not one of them, I'll be honest there. I believe we have one product that is

that has really embraced OS 3 and that's OpenShift. But the way it's done that is by by basically leaving Red Hat manage the OS 3 for you. So, part of OpenShift, you basically all of your OpenShift cluster all of your OpenShift server runs on top of OS 3. And basically OpenShift itself itself manages the OS 3 the OS layer underneath by by you know, managing at the

OpenShift level. And therefore, you as an OpenShift admin, you don't touch the the underlying OS. The OpenShift cluster updates itself by pulling the next OS 3 layers and everything, which means you don't have to deal with it. And and the question come came for us especially in RHEL was like, "Okay, we want to use OS 3, we want to bring this to edge, we want to bring

this to automotive. How are we going to manage OS 3 repositories?" And Bootsee is the answer to that one. You basically are able to use the entire container infrastructure and being able to manage your OS 3 repository just like you would manage your container images. >> I think I I think I went a little bit sideways for your question, but I think I was able to answer

it. >> we we definitely should talk more after. >> Um earlier you mentioned that you've merged your uh AutoSD with your CentOS projects. how going forward do you plan to manage the differences, especially in uh the kernel as you go forward between these two? Cuz as we all know, automotive is slow to adopt things, and so a lot of automotive is still using like 5.15 kernels, for

example. And I imagine you don't want your CentOS tied to that >> Um so the way they are So how are we going to manage our kernel? The our kernel we keep updating. >> Not not just your kernel, but any the differences between those between the different versions of those operating systems, really. And as the difference grows as you're managing something like long-term. So as AutoSD, you

know, has to maintain a a vehicle lifetime for like 20 years or whatever, and then CentOS is moving constantly all the time, you know. So that um So there are two things there are two things in your question here. Is there is the how do we diverge? How do we support long running kernel? Uh the answer is not here. The answer is not here. Here it keeps

moving. This is a look in the future of RHEL 3 to 6 months down the road, or in the case of CentOS Stream. Um the 15-year support is in the product. You will not get a kernel that is supporting for 15 years in AutoSD. You will get it in in the product. This is the value that the product gives you, the certification, of course, but the 15

years 20 years supports that come that is asked. Uh so this is this is uh you know, product management business unit area, uh and this is where you're going to get these 15 years of support for specific kernel. The idea is that we all are hoping with OSV with that technology that people are amenable or more comfortable in updating regularly. The old STV concept also comes with

the idea that you will need to update more frequently whether you like it or not in some cases. The entire change in AI, the entire, you know, the entre pre-care and I'm forgetting the name of the AI tool. Not cloud, the other one. Metos, thank you. >> The old AI Metos, you know, bundle about being able to find CVs, you know, 80% faster than a human does

means people we will need to update things. There is no way we can go without it. Now there is a there is going to be a question every time. Like it is a CV, yes. It is an important CV, yes. It is impacting enterprise, yes. Does it impact automotive? Like as much as it could be a very important CV in an enterprise deployment and it could be

that this CV doesn't have the same aspect on automotive. Now the the opposite is going to happen as well. It's like something that doesn't matter in enterprise is going to have very big impact in automotive. So we're going to just deal with this as an industry and I don't think there is a way around this. And AI is only going to make it even more fun for

some definition of that word. >> I think that idea with the CRA coming up. Thank you. Thank you. >> Yeah, one more question going in a similar direction about your rolling basically certification of Exida. So how do you handle the the tools, for example, of OSV? So basically tool qualification. Is OSV qualified is the question. >> Yeah, and and is there for example with a product is

there qualified compilers and so on or is Uh so yes, the we have a number of tools qualified. The GCC compiler is qualified uh and it goes back to the scope and what we have on the shelf today is we know insufficient uh but it's also the the discussion that we need to have is so discussing with us as many people as possible to define, you know,

what do you need? Uh so now the first question we have when we are meeting with customers is okay, we know our scope is not good is not good enough today. What do you need? What is what do we need to work on in order to to meet your deadline? Uh >> That's what the customer usually expects from your safety manual. >> That is fair and the

fact is Rivos is not available generally available today. You can't even if you want to buy Rivos today, you can't. You can't deploy it, you can't do anything with it. Everything that you can play with today is in a toy CD. Uh our product is still a product that is in development and that also includes the safety element. We have a version that is set that is

certified. We've proven that we can deliver a product that's certified and mixed criticality. It is nowhere near ready for starting to production today, but it is definitely ready to start to have the discussion about when do you want it and what do you want it when. >> Any other question? Okay, so interesting to see that operating systems still a hot topic actually maybe more now. I have

a quick question. So I know there is a new project called automotive so integration automotive interaction integration for auto SD. >> Can you say something more about that? >> Um Leo, do you want to take this one? You've been leading that project or do you want me to try to answer it? >> Okay. >> What does the project so the integration the automotive integration for auto SD?

So it's a project it's an integration project that we have recently created. The idea is to use auto SD as a foundational platform to run blueprints or other Eclipse SDV projects together. Although auto SD has not yet doesn't provide anything related to safety qualification, but you have the same architecture organization on how to deploy applications between QM and and similar environment which is not really similar environment,

but the idea is to provide this development and deployment and even be able to test your, you know, middleware or other blueprints in a real automotive hardware um as well. >> Yeah, so thanks a lot. So this is going to be an Eclipse project. So it's open source. Everyone can test it. So if you want to check this out, uh it should be available soon. So with

that, I would like to thank you a lot. Uh thank you, Eve. >> [music]