Ballroom C Saturday Mar. 07 - SCaLE 23x

This talk discusses the intricacies of establishing trust with Trusted Platform Modules (TPMs) and the importance of encryption in securing communications with these devices. The speaker highlights vulnerabilities associated with passive and active interposer attacks, explaining how traffic can be secured using encryption-based protocols similar to HTTPS. The talk emphasizes the challenges developers face when implementing TPM sessions and the prevalent security issues arising from improper session setup and key management. Furthermore, the speaker shares insights into the implementation of a new kernel mechanism for safe session handling, detailing the various hierarchies within TPMs and the difficulties in maintaining security without compromising system performance. Concluding with thoughts on the necessity of robust verification methods, the session addresses how to enhance TPM security while balancing performance.

to use it. And so this flipped everybody round from just communicating with the TPM as an ordinary clear text device to assuming that the communication channel of which you talk to the TPM is potentially compromisable. And if you're sending secrets over it like uh systemd has the unlock key for your operating system for instance, you don't really want an an attacker attaching a small device and just

snooping the bus and getting that. So, uh, the only attack that has been demonstrated so far is what's called a passive interposer attack. It's just a little device that snoops and does nothing. These are actually very easy to defeat. All we have to do is encrypt the traffic on the bus and the snooper can't see it. It's very much like using HTTPS. Once you use HTTPS instead

of HTTP, the man-in-the-middle attack is thwarted because the man in the middle can't see your communications. But an active interposer would be something that acts as the endpoint for that traffic. It's sort of like having an HTTPS endpoint sitting where you didn't think you were going, which is why you always verify the service certificate and then retransmitting your commands onto the TPM. And it can do that

with encryption because if it acts as the endpoint for your uh TPM communications, it can pretend to be the TPM you think you were talking to. And we don't really have an infrastructure like serverside certificates in the TPM. We do have other means of identification. Slight problem is nobody uses them, but I'll get on to that later. But in order to get a TPM to encrypt traffic

in the first place, you actually have to set up that encryption. The way you set up that encryption is something called TPM sessions. Um these work effectively by establishing a shared key between the application and the TPM. It's called the session key. And the TPM has a very complicated rolling encryption scheme. Uh the session key is basically a salt from which key derivation functions are applied to

nonsers that are exchanged on the bus. It's one of these really really complicated uh schemes. But the bottom line is if you know the session key, you can decrypt the traffic. And so that the object of all of this uh the complexity of the encryption key scheme actually doesn't matter at all. The object of the attacker is to gain your session key. and the initial session key

for a TPM is something called in TPM terms either bound, which is shared secret. It's sort of like a a WP key for the wireless. something both you and the routter know. And obviously, in order to have a shared secret with the TPM, you've got to have communicated with it before, which often isn't the case for applications that are very ephemeral. Um, or you can use something

called a salted key, which basically means that I have a secret the TPM knows, I encrypt a key to it, or I encrypt a nons to it. That nons becomes a salt for which all the sessions are derived. And that's the way I communicate in secret with the And sessions were actually added to all Linux kernel TPM transactions in the kernel 610, which is now about two

years ago. Uh that was all uh my code. So I actually wrote all of the code that does this. It's uh under a bit of attack at the moment because one of the prices you pay for adding sessions to the TPM is that all TPM operations become slower because you have to encrypt on your end and the bigger problem is the TPM which if it's a discrete

TPM is a very low budget low energy device has to decrypt on its end and the problem is usually it decryptting on its end because it just takes it a long time. But what's not often not understood is that there are actually security problems with the way you set up TPM encryption even with salted sessions because although a salted session is I encrypt to a public key

of the TPM. So all I have to do is ask the TPM for that public key. I encrypt to it and I send my encrypted packet to the TPM. The question is how do I know that the public key it gave me is the one I'm supposed to have? Because if I have an active interposer sitting in the middle, it might have lied about the public key

just give me a public key it knows and therefore it will decrypt my salt. It will intercept all my transactions and pass them on to the uh real TPM as though nothing nothing had ever happened. One of the problems if you actually look at a lot of systems that claim to use TPM sessions um usually you use a library and user space when commu communicating with the

uh TPM. I should add that all of the protections I did in the Linux kernel only work for the Linux kernel. The sessions have to be set up by the application using the TPM. So when the kernel itself uses the TPM, it's the application it sets up the sessions. But when you're using them in user space, like say systemd using the TPM to get the encryption key

for the disk. The systemd itself is the application. The kernel can't add encryption to it. It has to do this all on its own. So this also becomes a scalability problem. The way user space communicates with TPMs is something called TSS which is a trusted secure stack. It's a library for communicating with TPM. And the way that the trusted secure stack in pretty much every implementation and

there are two separate trusted secure stacks out there. There's one written by Intel and one written by IBM. Intel is the one that's most often used. Uh the way they get the public key for encrypting sessions is to ask the TPM. And as we know that's a bit of a problem. So as I said the active interposer can intercept the ask supply its own key and decrypt

the salt decrypt all the sessions. All of your illusion of security is gone even though you think you're using it correctly. And like I said most TSS don't even make it obvious that this problem occurs. Most of them will just quite happily you you use the correct API. It sets up the session. I mean the problem with the TPM is TPM programming is astronomically labyrinthly difficult. So

you use this toolkit to make your life much simpler and this toolkit doesn't do it right and that's the problem. And obviously the real problem is that none provides any verification of the key that you're actually using that public key. How do you know it's the right public key? Obviously there are other problems with the interposer. I mean, even a passive interposer just sitting on the LPC

bus can actually toggle the TPM's reset line. Part of this problem is one of the jobs of a TPM is to store things called uh PCRs, which are registers that extend via a hash and in theory never go backwards. So, you use a PCR to build up logs of what happened on the system. Um, and uh the PCR value is validated against the log. This proves that

all of the stuff that exists from boot to whenever you currently are is valid, happened, and the log hasn't been tampered with. This is the way a TPM predicts a non-reputable log. The property that that relies on is the fact that the PCRs can never be reset. They never go back to zero. Problem. If the interposer toggles the TPM's reset line, all of the PCRs go back

to zero, and you've lost all of the security you had in your logs as well. And even a passive interposer can do this thing. Like I said, they all go back to zero if you toggle the line. Uh, and obviously if you control the operating system in just a small part, uh, you can reset all the P PCRs, rebuild them to where they should have been to

say that you're legitimate, intercept all the logs, and nobody would be any the wiser. So, one of the fundamental questions that we really need to understand is how should you establish trust with a TPM? And there is a provided mechanism. It's so complicated that no one ever does it, which is why the TSS's don't do it. But all discrete TPMs ship with what's called a manufacturer certificate.

That manufacturer certificate validates something called the endorsement key that is resident inside the TPM. And that certificate via X509 chaining goes all the way up to the manufacturer's master CA. So once you have a TPM, if you know what type it is, you go to the manufacturer's website, get their CA certificate, validate the certificate your TPM gives you. So an interposer can't give me a false certificate

because I'm validating it against the manufacturer's CA. And that validation tells me what the endorsement key of the TPM is. And that gives me one public key in the TPM that I know absolutely works. problem here is that the certificates certify the endorsement key. That key can't be used to certify any other objects in TPM. This turns out also to be a significant problem because the usual

uh object you encrypt the salt to is actually the storage key, not the endorsement key. doesn't mean you couldn't encrypt to the endorsement key, but the problem is that most of the time you're playing with the storage hierarchy of the TPM, not the endorsement hierarchy. So the TSS naturally encrypts everything to the storage key, which is even if you validated the TPM certificate, not a key you've

actually certified from the TPM. Uh the TPMs themselves actually generate primary keys from a seed using a key derivation function. This is what allows for crypto agility because if I want an RSA key, I take the seed, generate prime numbers from it. If I want an elliptic curve key, I just feed it into the elliptic curve routines. The point is that seed doesn't change. And so the

keys I derive from it are always the same no matter uh how what algorithm I ask the TPM to use. And one property of a TPM key, a primary key in the TPM is that they can either be signing keys or they can be encryption keys. The endorsement key that the certificate certifies is actually an encryption key. The problem with a TPM is when a TPM certifies

other objects in the TPM. So like I want to encrypt a key to the p I want to encrypt my salt to the public key of the storage route system. I want the TPM to give me a certificate saying this storage root key is valid. It can do that, but it can only do that if I know a signing key. And the only key I know from

the certificate is an encryption key. So, how do I get around this? Um, well, let's go back and ask why we don't actually have a signing key as part of the certification. Turns out it's all the FSF's fault. The trusted computing group actually got badly burned by the FSF backlash against something they called treacherous computing way back in the early 2000s uh because the TPM was muted

as this instrument that would be used for uh you know content certification preventing you from watching videos etc. It's all actually untrue. The TPM is too weak an encryption engine to do this but it was seen as the spy in your computer and the FSF made a lot of fuss about it. They actually coined a phrase called treacherous computing instead of uh what the trusted computing group

was trying to call it which was trusted computing. Uh the fact of the matter is the TPM will work on behalf of anybody who has access to it. You own the operating system on your computer. You're the one guarding the access to the TPM. Nobody else can use it unless you actually give them permission to. And so um treacherous computing was not I mean there were definitely

plans of operating systems that weren't open source to use TPMS for various nefarious purposes but they rapidly got killed by this uh campaign against it. So even Microsoft nowadays is very careful to give you the ownership of the TPM in Windows rather than just taking the ownership itself and controlling your TPM. But obviously as most of you here should be running Linux. That's not even a problem

you need to consider. You own the operating system. You own the interface to the TPM. You own everything that goes on with that device. Like I said, they criticize the TPM as an agent in your PC acting on behalf of others. But it can't communicate with the outside world except through the operating system. So if you control the operating system, you absolutely control it. To counter this,

the TCG, the trusted computing group, tried to introduce privacy preserving features of One of those privacy preserving features was not allowing manufacturers to certify signing keys. The reason for this was because if the manufacturer gave you an easy signing key, that's the key you'd use for all certifications in the TPM. And that would mean that you'd be able to be tracked online by the certification you just

gave, which is tied back to the endorsement key that every manufacturer of the TPM knows. So, this would be a unique tracker in your TPM that allowed anybody online to track you via TPM operations. And therefore, because it's a a single signing key, it would have uniquely identified you. So the trusted computing group said, "We're not going to do this. We'll allow you to create a large

number of possible attestation keys in the TPM." Um, we'll introduce something called a privacy CA, which will link the attistation key in the TPM that you can create to the endorsement key that's only an encryption key. And the privacy CA will do the linking. So only the privacy CA will have this information and the privacy CA would be a trusted third party that's trusted both by you

in the operating system and by whatever on the internet is asking for certification and all whatever on the internet is seen is a signing by an attestation key that you've obtained from the privacy CA that has a certificate saying the privacy CA trusts this so you can trust me and obviously only the privacy CCA then is the thing which can track you and thus could track the

user. And so this should eliminate a lot of the complaints the free software foundation had about uh treacherous computing. Um and obviously they had to introduce a load of special TPM operations. These are operations called make credential, activate credential. You don't need to know anything about how they work. But the point is they are operations you can use to create ephemeral signing keys which can then be

used to certify objects in the TPM which is really what we want because we want to know that the storage key belongs to my TPM. Oops. So problems with this privacy Ca idea has been around for 10 years. None exists today. Nobody ever thought that this scheme would work or at least nobody bought into this scheme enough to actually run one of these things. So this is

now a system that all TPM support that we can't use in practice because we have no privacy CA for my purposes in the kernel. Even if privacy CAS did exist, I can't use them because in early boot the Linux kernel does not talk to the internet. You know, it's a booting kernel on the chip. I have no way of contacting a privacy CA. at start of day

in the Linux kernel. So even if we had them, I can't use them. Um, and one of the things we could do inside the Linux kernel is we do have X509 code that can do certificate validation. If I managed to find all of the master CAS of all of the possible manufacturers of all TPMs, it's theoretically possible that I could validate the endorsement key just using X509

certificate validation. If I could do that, I could actually use the endorsement key to encrypt my salt to I don't need really need to encrypt it to the uh storage key, which is what everybody else does. But the problem with this is there are quite a well when we were first doing this there were a large number of manufacturers of C CA of TPMs nowadays they all

seem to have consolidated to about three or four Israeli and other companies so it may now be possible to do this but when I was first coming up with the kernel mechanisms for doing this it wasn't really possible and the other problem is as I said I've got a firmware TPM firmware TPMs often don't even have endorsement key So if I boot Linux and I find you

don't have a certificate, what the hell do we then do? So given all of these problems, we actually needed to come up with a new scheme. And the kernel solution was to use something called the null primary uh for session salulting. So here's some more TPM trivia. The TPM itself has four separate hierarchies. These are designed for different users of the TPM. There's something called the platform

hierarchy which is only supposed to be used by the bias. After the bias is finished using it, it puts a random password in the platform authority which means nobody else can ever touch that hierarchy. The endorsement hierarchy, which is where the endorsement key lives, but is otherwise not used, is designed for attestations to outside bodies. So you're not really supposed to touch it unless you're actually doing

an attestation to an outside body. the owner or storage hierarchy which is where keys are normally stored is the one that you as the owner of the platform actually get to use. But then there's a fourth hierarchy called the null hierarchy which is normally never used. The interesting property of this null hierarchy is that anybody can use it but it seed changes on every reset which means

that if I encrypt a key to the null hierarchy and reset the TPM I've lost that key because I can't the TPM then can't unencrypt it. So this is often seen as a useless hierarchy for long lived keys. But for operations that only last for the boot of a a a machine which the null seed won't change over, it can actually be used. And like I said,

the interesting property is this seed changes on every reset. So if I salt all of the sessions to the null seed and somebody resets my TPM, that salting will suddenly fail. And so the kernel will actually get wind of the fact that somebody has activated the reset line. And so this is the reason that we actually salt sessions with the null seed in the Linux kernel. But

obviously uh we can do this easily. We ask but but we still ask the TPM what is your null key and it gives me an answer and we still need to know that this answer is correct. And like I I think I've demonstrated there's really no way of doing this at boot. It's the best scheme I can come up with. Um like I said it will reset

will ch we can detect a reset attack. We export the name that we actually saw on boot to a CISFS file. Um names of keys by the way is just a hash of the public area of the key. So it's a unique identifier per key. So once I have the name I and if I had the way of certifying the null object, I would then be able

to completely validate that I boost up with the correct null key. And this is actually what we do. So um we can verify this null name that the kernel exports by more sophisticated means. But I can only really use those sophisticated means after the kernel is booted. So as long as I trust my kernel and user space, I can get conclusive proof that the system booted up

correctly. But at the time the system is booting, the kernel still has no idea that it hasn't been intercepted and it is is booting correctly. Um but we do have enough information to do this verification after the fact. Um what I actually did was uh okay so the OpenSSL TPM2 engine is the toolkit that I did primarily for uh using a TPM through uh OpenSSL. Uh it

provides both an engine and a provider for OpenSSL version one or version three. Um, but it also has a set of cool toolkit stuff for allowing you to man manipulate TPM keys. One of the uh additional uh binaries I added was something called a test TPM2 primary that allows you to do all of these at astation operations. And it actually has three functions. One is minus EK

sign which actually works out if we'd had an assigning EK instead of an encryption EK, what would it look like? And just give me the name of it, which is an easy thing to do. a test which actually runs the make credential activate credential produces a ephemeral key validates the EK signing key and then says that either this key conforms to your certificate or doesn't so now

we have bound the signing EK directly to the certificate that is sitting in your TPM if you run this and obviously this only needs if if you can store the signing EK its name somewhere in an immutable part of your operating system. So it should be installed at time of install of the operating system and it's never modified. You can rely on this ever more. You only

really need to do this at a station once. So we could do this as part of operating system install and then forget it thereafter. And then there's a certify function which is used to certify based on the EK signing the signing EK key all other uh TPM objects. Um, obviously, um, the original problem that the TCG had does not exist here because I'm the person who created

the signing key. I own the laptop. It doesn't really matter if I track myself. So, I didn't really need to do any of the other PCA stuff. So, once I have the EK signing key as the owner of the operating system, that's what I use because I'm using it for my internal purposes. I'm not using it to certify to anybody else. There are no privacy concerns. So

what are the threat models? Um obviously running this tool means I have to trust the operating system and uh the TPM people would tell you the TPM is the only way of trusting your operating system. But that's not true. We do have secure boot people who often don't talk very well to TPM people. But if you're securely booting your laptop, secure boot validates the fact that you

have booted the correct kernel. Um, and it if you're using UKIS, it can be used to validate the initial RAM disk as well. As long as you have full disk encryption, the fact that you had to type a password to decrypt user space validates user space. That gives you sufficient trust actually to validate the TPM after the fact. Even if there's an interposer from an evilmade attack

sitting in your computer, you would now be able to detect this as long as the interoser which is a fairly weak thing can't compromise your kernel sufficiently to get the result of this uh uh the certification that you're going to do. Obviously, this verification is ineffective unless you trust the operating system. So if somebody steals your laptop and turns off secure boot, substitutes their own kernel, defeats

all of this and manages to get your disc encryption key, it's game over. This is one of the reasons why, by the way, although I'll be showing in the demonstration, hopefully I've got time to do it, uh the fact that we can just do a complete unlock with a TPM uh using uh PCR values. You should never do this. You should always have a second factor in

a TPM unlock that ensures if somebody steals your computer, they can't then you know uh through means of intercepting the kernel, damaging secure boot etc. uh unlock it. So always use a pin and make sure that PCR7 which is what certifi attests to secure boot is part of the policy check of systemd. So if I'm lucky and I've got enough time we will try and do a

demo. first of all I'll start a software TPM. So this is the IBM software TPM. It's very uh very useful thing for um checking. I'm actually also going to uh start an interposer here. Um, so what's going to happen is my virtual machine is going to talk to the interposer and the interposer is going to be the thing that talks to my TPM. So I've already set

up an actively interposed TPM system here and then in my VM console I will start the virtual machine. Oops. Do excuse me. Sorry, had my uh virtual machine operating. Okay, here we go. So, this is actually going to boot up. This is a an encrypted disk image, but I've actually set it so that um the disk encryption will release the TPM key based on uh PCR registers.

So, it's already actually released the dis encryption key and it's going to boot me straight into user space. So, there I am and I can log into this now. I'd ignore that. I have so many virtual machines sitting on my laptop that uh security for this one uh system is actually disabled. And so what I'll do is I will demonstrate first of all what kernel am I

running? Uh so I have to apologize that is one week old. Uh it should be RC2 now and it's RC1. So this is the latest Linux I actually have all of this stuff turned on. So, if I go to So, this is where you find the name of my system. So, this is the null seed that it used to boot up with. Now, um, we're going to

pretend that all of this is secure and I've done it. So, one of the first things you should always do is is on first boot I would have got the EK signing name and put it somewhere where I know that can't be tampered with. So it's on the encrypted file system. Unless somebody compromises my encryption, they can't change this file. This is the name that I'm going

to use to validate everything. And obviously um I this is a software TPM right so it has no certificate but Ken Goldman who's IBM's TPM guy uh gave me a mechanism for actually giving this thing a an endorsement key certificate so it actually has a real endorsement key certificate that I can get out of this is the index or so that is a a known place in

the TPM uh NV RAM where the certificate is supposed to be stored. it comes out as deer. Uh I've actually asked uh the TSSNV read to translate this to a PM certificate. And we can actually have a look at this PM certificate. I can just translate it to text. The one I got um as you can see it was issued by IBM Yorktown Heights, which is where

Ken Goldman works. It's uh EKCA. So if I trusted this EKCA, which obviously I created, so I trust it. I would validate the certificate and it would then give me a signature over the TPM endorsement key which is what this is and then all I would do is run an attestation using that certificate and comparing it against the signing EK name that I've already created and it

this is actually going to run a make credential activate credential. You notice there was a slight delay there. It took about 100 milliseconds to return. That's because even for a firmware TPM, this roundtrip operation is quite expensive. But now I validated my signing EK against the endorsement key certificate. So I'm now completely confident that the EK name I have is the correct name. So every time I

d I ask a TPM to derive an EK, I just ask it, what is your public key? Take the hash of that. That hash gives me back the EK name. Actually, I'll show you what they look like. Um all it is is a fourdigit prefix which is a 16bit uh version of the TPM hash and the rest of it is a SH 256 hash of the key.

That's it. That's all the name is. But I can use this name to certify the null key. So I ask the test TPM2 primary to certify the null against the EK sign name. So what it's going to do is ask the TPM for the signing EK, get it to construct it, compare what it gives me against the name that validates it. Then it's going to use the

key that it just constructed to issue a certificate and the certificate is actually signed by the private part of the endorsement key the signing endorsement key. So I know that only the TPM can have act can have actually issued this signature. I will validate that signature in user space. That validation I will then compare on the null name and if the two match everything is correct. actually

that takes a lot less time than the uh at astation did and it gives me a good certification. So now I know that my kernel actually booted up correctly which you wouldn't have expected given the fact that I actually have an interposer in the way but the point is I wrote this interoser so that it wouldn't be detected. So it knows what a kernel TPM transaction looked

like and it stays the hell away. All it's doing is acting as a conduit passing these transactions to the TPM. Now, this means that it can't snoop. It can't steal anything. It can't do anything. But it's waiting for another subtle event where it knows it can steal my credentials. And so, we'll get onto that subtle event now, which is I'm going to take this system I'm going

to tell it to change the TPM based disc key using krypton. And to do I'm going to have to uh my system is actually locked against PCR7. I'm going to lock it against PCR9. Otherwise, systemd would cleverly refuse to enroll this. That is going to be systemd going through the uh also going through the interoser. I need my disk password in order to do the enrolling. And

there it goes. It's wiped and reenrolled the key. Problem here. systemd just asked for the uh it actually asked for the srk and it used what the TPM told it and it didn't validate that against the certificate. So what happened is my interposer here actually did a substitution of the key that systemd asked for. Once it's substituted, it can snoop the transaction. This thing here is actually

the disc key that systemd tried to seal into the TPM. And I can prove that by taking a copy of And if I do, if I just do crypt setup verbose open, ask it to test the passphrase on the disk, and then I'm just going to insert that passphrase. I've done a copy and paste on my keyboard, which you didn't see. It validates that, and I have

successfully unlocked the disk. That proves that I just used an interposer to steal systemd's disc which also proves that TPM security is something that everybody should really take rather seriously. Now as a final thing uh because I am running out of time I will try and do a reset of the TPM because remember I promised you that this system would detect resets. Fortunately, resetting a software TPM

is very good, very easy. I just tell it to power up and start up. If I look in the VM at PCR9, which is what I locked it against, has gone back to zero. Now, if I look at the kernel, the kernel actually hasn't noticed this yet. uh if I have time I will run a certification and the null name has changed. So the ver the certification

is actually failing because I have the wrong null name. And that proves the fact that if I done this I would have been able if it if it had actually done this in the kernel it would have been able to do it. And now the kernel has actually detected here that the integrity of a TPM transaction it tried to do has failed. And the next thing the

colonel is going to do is disable the TPM because we're under attack. So if now I ask it for uh register 9, it won't give me it because I can't talk to the TPM anymore because the kernel is preventing me. So this is a demonstration of at least some of the security procedures we've put in place inside the Linux kernel for all of this. And with that,

I will try and get on to the conclusions and give you time to ask a few questions. I think I did manage to go fast enough that we do have time for questions. So the null seed is an interesting scheme but it does fully protect us from reset attacks as I demonstrated. Um after boot verific the verification after boot proves that the boot you did was fully

secure provided the kernel and user space are trustworthy. So you still are resting that trust in both secure boot and encryption. If you don't have either of those, you probably broken this trust requirement for doing the verification. So it's very important that you do that. Any TPM tool could actually use the null key for salting. And what that means is that if you do use the null

key in user space applications, you can actually compare it against the null key the colonel is exporting and be guaranteed as long as you did the check at start of day, the certification at start of day, that you've actually got the correct null key for salting. And this would mean that everybody who's actually using all applications using the TPM having to create their own sessions could then

be secure both against interosers and against reset attacks. It hasn't happened yet, but I'm hoping one day it might. And obviously you can simply verify the calculated name against SIS vest file for the null name so you always know if it's correct. And thus full safety is always extensible to every TPM tool. with that. Um I this is an impress.js presentation. Um these are the URLs for

both the presentation and the interoser. It's all it's all available as open source. I mean that git repository is a bit thin. It's got about 10 commits in it because it's just what I did for a demo, but you're welcome to look and play with it if you like. And with that, I'll say thank you and call for questions if there are any. Okay, first question. Oh.

I'm curious if >> So the question is uh could we use the null seed in association with systemd boot parameters to you do more key derivation that would actually give us secure keys. The answer to that is actually yes. There's a a guy called Ignat Kagan who's exploring stuff like this. He wants to use it for something he's calling uh oh it's some kind of key that

goes in the kernel. I've forgotten the name he gives to it. But it is perfectly possible. But you have to remember the limits of these keys you've derived. That null seed only survives one boot. You can't use it for any information you want to persist beyond that boot. >> Yep. that fast. >> Right. So the the question is based on the observation that the colonel takes a

while to note that the TPM has been compromised by a reset attack. But the other part of the question was you noticed that I pulled out the key and uh but that wasn't done via a reset attack. That was by a different vector of the interoser. Remember the kernel operated perfectly normally for boot up while the interoser was there. I cleverly programmed my interposer only to steal

the systemd key and not to reveal its presence to the kernel. So it doesn't do anything the kernel is looking for. And it does this by ensuring that all of the kernel transactions are perfectly integrity protected because systemd is the one which has the important transaction that has the key I'm after. Um so to go back to your first question, the delay in detecting is because the

TPM is a pole device. I have no means of getting it to tell me immediately if something is compromised on it. I have to wait until I ask. And right at the moment, the kernel asks the random number generator every 30 seconds to receive the entropy from the TPM. That transaction is protected by sessions because obviously we don't want entropy leaking from the kernel. And when one

of those fails is when I notice, but you're right, in that 30-cond window, like I had when I did the certification and it failed and I showed you the value of PCR9, if you can get your TPM exploit in that 30 second window, yes, you're right. You can do it. Well, you have my deepest sympathy. We all like arm now, though. >> Well, a there. So, so

the question is what if you come from the ARM world because this is fairly x86 centric. Uh the answer is that actually the ARM world does have a lot of systems with TPMs. I have several developer boards that I think the potato board and a few of the other ones they actually do have discrete TPM sitting on the board. Um the other thing is that uh UFI

had it doesn't really have a tight coupling to the TPM. So if I disable my software TPM, my virtual machine will still boot up without it. UFI will detect the fact that I don't have a TPM and just turn turn everything off. All UFI uses the TPM for is the measured boot log because UFI is the first booting section of the kernel. It begins the measurements and

it builds up the measurements into the PCRs, the measurement registers. So if you turn if you disable the TPM or don't have one, all you lose is the fact that you don't have a measured boot trail. But there are a lot of um ARM systems that are UEFI based because UFI is starting to be the default for ARM 64. So a lot of the ARM it's probable

there will be no ARM 32 systems with UFI and universal mechanisms but everybody is starting to adopt ARM 64 as well. I mean my phone has ARM 64 but obviously it doesn't have UFI but a lot of the server class systems that are trying to be sold do have UEFI. they are interested in using measured boot techniques. So they're interested in having the TPM and the ARMbased

UFI actually do all the measurement stuff. So a lot of the server-based ARM systems are starting to look like x86 in this respect. So as long as you have the right ARM system, you can actually do all of this as well. >> Oh, same with risk 5. I mean, I'm not a risk five person, but it's it's basically driven because they need compatibility with all the x86

stuff. The there are lots of people who demand the integrity that measured boot gives you and either ARM and risk five or whatever would go off and reinvent their own system and everybody would complain like Billio because we have one that works or you just follow the least line of resistance and use the one that works which is what ARM 64 and Risk 5 are doing. Okay,

I need to give you at least five minutes to change rooms. So with that I'll say thank you very much and I hope you enjoyed the presentation. Yeah, sure. Let me keep Hello. Yes. >> No, it's okay. >> Thank you. >> Yes. >> Okay. No. Okay. Thank you. >> Uh hello everyone. So I think we can get us started and uh thanks for coming to my session

and I'm very glad to be here and I'm very glad to have this opportunity to share my uh my kernel Linux Linux related work with uh you today. So uh let's get started. So before we get started, so just a quick introduction. So my name is Kang and I'm a Linux uh kernel developer. So uh currently I uh found my company Micronel Technologies. So this is why

I'm here So uh my kernel so I I know this idea maybe not sounds familiar to you. However, before we start, so I the first thing I want to clarify is this idea is definitely not new. So this this idea actually is pretty old in my opinion. So the uh uh at least this idea dated from 2009. So the first paper established this idea and pioneered the

mic kernel operating system work uh is uh uh called barface. So it was in uh 2009. So and it is a completely new operating system and it's definitely not a Linux. is uh the operating system reading from scratch. So this is probably the uh first uh multi kernel operating system implementation at least according to my uh research. However, this work is mostly list day in the academic

research is not uh in production. And the second uh uh work I want to mention is the popcorn Linux. So this one uh deserves attention because this one actually is the first monitor kernel implementation with Linux and this work was in uh 20 uh uh 17 which is also old right and however the main purpose of this uh project is uh targeting the heterogeneous CPUs and based

on my uh entire career I have never seen s hogenous CPU uh in my entire career. So that's why I have you know a very different opinion with uh this the direction of this project. And another project that uh needs to mention is the mega kernel. So mega kernel is a kind of Linux and also kind of not because it's a mix. So my Mac kernel is

actually they build uh another kernel on top of Linux and that kernel is very lightweight and that kernel actually only provide the necessary high performance uh uh layer for the application running on top and so that's why it only provide about uh less than half of the system system course u uh uh uh of the uh Linux operating system. So if the uh system call is not

implemented in the me kernel, it is simply just forward the system call to the actual Linux. So this is how it works and as you can see from you know this very short history. So uh my kernel actually has a very uh uh strong interest at least in the academic research and so what's what's different uh uh with uh uh our approach and why it is different

now because the the approach we choose actually is very different from the uh previous uh uh work and previous project. So first of all the first uh multi kernel implementation I mentioned here they all take the replica based uh approach that means the uh different kernel although they are separated however they share all the replicated uh states so I took we took a a very different approach

we use the mon kernel as isolation so we don't use any uh replica and another significant different with all the previous project is we use uh uh we developed on top of the upstream Linux from day one. So we don't write the operative system from scratch and we don't write you know any uh uh um out of tree code and everything is based on the uh very

latest upstream and uh uh uh more importantly actually we don't uh have any uh compability issue compared with uh me kernel. So that means everything uh you have today running on top of Linux it should continue work on uh our multi kernel Linux. So what is micron Linux? So uh let's get back get back to the idea you know down to the very uh uh minimum uh

uh uh core concept. So monic kernel actually is a very uh simple concept uh uh at the core. So the idea is pretty much just run multiple L kernel uh simultaneously without using virtualization. So this is the entire idea and uh we uh compare with the uh uh uh uh virtualization the advantage is we uh uh all the uh uh multi kernels are running on the real

hardware on the bare hardware. So there is no uh virtualization overhead and uh all the uh hardware resources they are uh participated and dedicated allocated to use of the uh spawn kernel. So each of the kernel gets their own CPU memory region and device resource and ideally each of the application they should run inside the use of the kernel. And the advantage of this actually is the

applica the kernel could be just tailored specifically to uh run the specific application and we could also uh specifically optimize and tuned for that specific application as well and more importantly everything is Linux. So the spine kernel actually could be just the Linux with only the multi kernel uh support and that means we have the uh complete uh uh compatibility. So with this uh idea ideally we

should uh have uh could run the different workloads on the same node and with uh uh my kernel as the isolation. So how does this work? So this this sounds uh uh pretty cool but uh underlying how does this work. So uh uh so uh for the implementation actually it's not uh uh easy to put all the pieces together and the first thing we need to uh

solve is how to start multiple kernels uh simultaneously on the same host right so the infrastructure is there because Linux already supports the key exec uh mechanism for decades and the key exec works and it only uh currently it could only uh start a new kernel and you know replace the the old one. However, for multi kernel, we definitely want to keep the the uh old kernel

still running and you know uh launch multiple uh uh kernels together and also without uh hypervisor that means we need to have some way to partitionship uh to partition all the hardware and you know without using uh uh uh virtualization and uh resource uh uh partitioning is another challenging because it's not just just simply uh uh allocate the resource is also we need to manage for example

the uh conflicts for example when we allocate the memory region we need to avoid memory region conflict between different uh uh kernel instance and this is also a a very big challenge we we have solved for the uh my kernel architecture and the last thing we need to uh uh uh solve is uh dynamic resource isolation because static resource isolation is pretty easy. That's pretty much the

default what we get when we uh start the kernel. As long as we don't change anything during the runtime, we by default get the static uh partitioning that that already works and that's the easy part. The hard part is you know after we start the kernel instance how do we update the uh kernel instances without shutting them down right. So how how do we move the CPUs

from one kernel instance to another? How do we update the memory of one instance and you know maybe you know assign it to another uh kernel instance. So this is another uh big challenge we also have solved with uh uh uh device uh infrastructure provided by uh Linux So uh this is the overview of the entire architecture. So I hope this picture could clarify what I have

mentioned you know how the resource specifically how the CPU cores and memory regions and you know different kernel binary different uh kernel regions how they are uh uh partitioned. So this is a very very uh high level uh overview of the entire multi kernel Linux uh architecture and so why do we need uh uh this multi kernel uh Linux? So it maybe looks cool but why do

we need this? So I do have uh about uh four use cases here. So because like I mentioned you know my kernel is pretty much targeted targeting the isolation. So isolation is pretty much the foundation for the cloud computing today. So for cloud computing today we have two pretty much there are two uh major technologies right. So the first one is a container and the second one

is ving machine. So and also of course we could also combine them together in some way right some you know cuta container actually run containers inside of a virt machine. So you know and people call it a secure container you know whatever terms. So uh and compare with the existing technology uh for cloud computing. So what is what are the advantages of my kernel? So let's uh

uh take a look at them and you know compare with them one by one. So compare with uh container and uh uh the advantage of the mon kernel actually is stronger isolation and more secure. So that's why I believe model kernel is stronger than container. So as you people probably already know you know containers they share the same uh line kernel. So the problem with this of

course is if we the kernel has some security issue that means that security vulnerability will affect all the containers on the same node. That's for sure. And another uh uh issue with container is the resource is the resource isolation. So the reason why container is much lighter than virtual machine is is virtualization is based on the resource uh is based on the software resource isolation. The software

resource is part is provided by the linear kernel. And this the problem with this of course is is uh known as uh a noisy neighbor. So if one container could uh uh uh produce a lot of noise. So that means the uh they would eventually have the uh resource contention on uh in the kernel uh layer. That means the resource contention will definitely slow down other uh

uh containers on the same host. So this is typically called uh noisy neighbor and compared with uh container. The advantage of mic kernel is uh first thing is we don't have any uh uh software u uh resource uh isolation. So it's the kernel level isolation. So because use of the workload has the own uh lin kernel and the kernel is isolated. So that means we don't have

the noisy neighbor issue as long as we have separate kernel and a separate hardware. And the second advantage is because the kernel is also separated. If one kernel has some sec security vulnerability, it definitely won't affect other uh kernel instance if they run if they have uh uh uh uh different kernel versions. And the last advantage is actually uh because we have different kernel and the kernel

could actually be uh optimized tuned and tailored down to the minimum specifically for the uh workload the target workload you want to run. So this is definitely what the container could not uh achieve. And let's take a look at the virt machine. So v machine actually is pretty strong. So in uh my opinion so it's not very easy to uh competitive with v machine to be honest.

However machine also has some limit and the v machine. So the first one actually is because v machine use virtualization. So virtualization has overhead. I know today you know we have the hardware acceleration for the uh virtualization. However, even with hardware acceleration, we still have some overhead maybe already down to 5% or something. However, it is still it is still there. And for some cases, it is

still matter. And another use another uh advantage actually maybe is not obvious. However, it's very real. It's the nest virtualization. So this is real because today we uh many uh uh companies already uh have their uh servers in the cloud. So and when we uh purchase cloud from the cloud providers we already pretty much by default get a virt machine. So if we want to run another

virt machine inside the cloud uh we purchased that means we get a nested virtualization. For example, you know, we have AI agent today. So, and AI agent runs in uh uh uh uh sandbox. So, if that sandbox is a microVM and if you want to run your uh sandbox inside your cloud, that means you have nested virtualization. And this advantage is much more obvious than just you

know compared with the uh fertilization on the bare because the necessary virtualization first the overhead is much higher probably 40% or uh or something and the uh another advantage is actually the support for nest virtualization is not mature so not all the uh uh cloud providers support nest virtualization so even Amazon actually just supported the network virtualization very recently and so this is has been a a

pain point for the uh cloud providers for a long time and uh the last uh uh uh limitation uh of the virtual machine actually is virtual machine is uh complete OS. So that means typically unless you know you use microVM we would go through the entire uh operating system in initialization. So if you use systemd uh good luck on the uh uh booting process and so compared

with virtual machine the advantage of multi kernel is there is no virtualization overhead and especially if you you run my kernel inside your cloud you know which is a virtual machine we you you don't have any necessary virtualation at all and uh my kernel could boot directly into the darker image So that means there is no uh uh operating system initialization at So speaking of docker so

we we we all like docker probably right. So docker is very uh simple. So you know much simpler than system did right. So and the support of docker actually for my kernel is very native from literally from day one. So I we integrated with the docker image. So we could actually boot into the docker image directly. So you can use any existing docker image on docker hub

or uh you know build by your yourself. So uh the the this is a very uh uh strong advantage because you know there are some other uh uh cloud solution uh like uni kernel they could also integrate with docker but you know in a different way. So you would need you know some kind of conversion you know if you want to use uh uni kernel however you

know use multi kernel you don't need to convert anything and so the why this works behind the scenes is actually I uh developed the a new operating system which is dedicated to share the container uh rul image and we just uh have a very minimum uh uh in process to just specifically put into the darker entry point. So there is no operating system in in it at

all. And the workflow would be very simple if you you know just uh already a docker you know just pull your dock image and you know put the uh my kernel with the uh kernel image together with your docker image and you would get you know uh your service running inside the kernel with the existing image. And of course we uh today we have AI agent. So

and every people talks about uh AI agent today. So and AI agent could also uh AI agent also need to run in the uh sandbox right. So and my kernel as isolation actually could also uh provide the sandbox solution for AI agent as well. So and the advantage is actually you could uh get a direct access uh to the uh GPU and uh it could alone could

also offer faster uh snapshot compared with uh uh uh uh microv and v machines you know uh generally they are too heavy and containers they are uh you know a bit leaky. So uh the advantage here actually is you know compared with containers we have a stronger isolation and compared with uh virtual machine especially uh micro virtual machine we actually could have a faster snapshot because you

know everything could stay in the memory compared with uh the uh existing uh uh micro uh uh VM and uh also uh Like I mentioned, you know, if you already have a cloud, you want to run macrovm, good luck with the nest virtualization. And with M kernel, you don't need to worry about that. So, uh let's take a look at other use cases other than the cloud

or the AI agent. So, and uh this use case actually is uh probably more familiar to you. So, if you uh use Linux on a daily basis. So today we when uh when we update the uh Linux kernel we need to reboot the entire machine and you know shutting down all the uh applications. So even with the key exec key exec could speed this up however is

it does not element the uh reboot process. So uh so this is what we currently do. So uh what model kernel could uh do better here is it could completely animate the reboot process uh in theory in theory. So uh what is the theory here? So my theory is because we could actually start multiple kernel in this case two kernels in parallel. So that what does this

mean? This means we already have one old kernel running and we want to update the old kernel to the new one. Right? So we could actually start the uh new kernel uh uh simultaneously you know without shutting down the old kernel. So starting from there we could actually uh gradually migrate everything from you know the application the process and the resource from the old kernel to the

uh new kernel one by one until we could safely retire the old one. So during the entire process there's no report. We don't need any reboot and eventually after the entire process is completed the new kernel could take over everything you know the all the applications you were running before upgrade and all the hardware resources available on on your machine and that's the complete process of the

uh uh uh uh kernel update process with uh uh my kernel. So uh one thing I just want to mention is this is just my theory and this this uh there are a lot of details we need to uh figure out because uh uh uh there are a lot of uh um uh uh uh device driver like you know a hardware related issue we need to deal

with if we want to completely avoid the shutdown especially you know the hardware uh shutdown or reset. So that part is definitely challenging. So that's why I just call this you know theory because in practice we still need to do deal with the hardware which is actually much more complicated than what I described here. So however I do have you know a very uh invention uh goal

here and I do hope you know in the future with mod kernel we could avoid the reboot for the uh kernel updates. And another use case I have is actually a a little bit similar to the uh uh live kernel update is the key K key dump. So for the uh servers for the Linux server today pretty much I think all the servers today you uh uh

use kd dump to capture the kernel crash and uh uh so what we have today is you know when we have the kernel panic or kernel crash because the uh kdam is already uh there. So what does this means? When the kernel panics, the kdump will take over the uh machine and it will uh jump into the uh the kdam kernel and which is already which is

a reboot and during the kdam kernel uh it will save the uh core dump of the uh crashed kernel and after the core dump is saved it will do another full reboot put back to the uh regular kernel. So as as you can see from the it takes two two uh reboots uh from the kernel crash into the uh the service back to the normal. So this

entire process will probably take uh minutes because there is a second reboot which is not with uh uh the fast key. Exact. And the advantage of uh a multi kernel is it offers u uh uh uh no reboot for the entire process like I mentioned. So we could actually start a kdam kernel uh uh along with the main kernel. So this is this part is a little

bit similar to the uh kam kernel except the kdam kernel is running in parallel together with the main kernel. However, it will only take a very minimum resource like probably only one CPU one CPU core and you know very small uh memory just keep it running and monitor the health of the main kernel. So let's say uh and when the main kernel cross and the kernel will

detect this you know for example using heartbeat and starting from there the kdam kernel will recognize the main kernel uh have been crashed. So it will start taking over the entire resource of uh uh uh uh in the machine and you know take uh and this processor will take probably just uh second because all the hardware resources are uh uh shared within the same node. And once

this take over we already have a working kernel. So that means the entire process will probably just take you know seconds instead of minutes and uh uh we would get achieve the same uh result as the uh kdam kernel. And of course if we want to you know save the core dump we could actually also uh copy the uh uh the core dump of the main kernel

before we uh take over the memory. So this could also work and we could also do this uh uh asynchronizedly as well. You know start it over the uh your service meanwhile copying the uh uh kernel code somewhere and uh to do that actually I uh started a project called KO. The KO project actually is is exactly targeting the uh K dump and key exact use case

I mentioned. So in these two use cases the uh uh the they have some overlap. The overlap is they need to transform the kernel right. So they have two kernels to transform whether it's a ced one to uh backup one or you know the old one to the new one you know we have two kernels to transform. So that's why I call it you know kernel transformation.

So and this is why I call this project kov. So we just need to do uh for uh k uh for the kdam uh kdam use case we just do the uh auto healing. So when the kernel crash you know the other kernel just automatically detect the crash of the main kernel and automatically heals the entire machine and uh everything backs uh uh get uh gets back

in uh to normal you know within seconds. And uh for the live kernel update you know this is a little bit challenging. However I do hope you know we could achieve you know the atomic resource hand over and you know if anything uh goes wrong we could also actually go back roll back to the previous one without you know any uh consequences. And so this is pretty

much the uh design uh uh of the uh uh uh this project and this project has just got us started. So and it is still uh a working in progress and uh uh let's uh take a little bit break and take a look at you know uh what we have so far. So I have mentioned you know the mic kernel and compare with you know the existing

solutions you know for many uh uh for uh several use cases and you know in different dimensions. So what is the summary here? So you know this is I hope this uh uh uh chart could also give you a clear uh uh comparison of you know the advantage or maybe some disadvantage you know uh of my kernel compared with the existing solution you know like uh container

and virtual machine. So this is pretty much just a quick summary of what I uh just mentioned previous and of course I uh I I have simplified you know uh a lot of things here and because I just mentioned you know all the uh hardware they are participate that's the uh ideal case however you know in lot of uh uh in practices we do still need to

share something it's not just the hardware sometimes is software maybe the file system we need to share as well and the the reason for example the hardware needs to be shared is for example for uh uh server we typically we already have so many CPU cores today maybe hundreds more than 200 is is uh already kind of common for servers however even with so many CPU cores

the devices the IO devices they are not as many as the CPU cors. So even for the servers they probably only have very kind of maybe just very few uh number of device IO devices on the same server. So in this uh scenario how could we share the device? How how could we participate the hardware device? Definitely we could not uh participate the whole device anymore. So

in this case actually we do have another solution. So instead of you know partition uh partition the entire device we don't we do we partition we partition the sub resource of the hardware for example you know the uh network card today pretty much they have many uh uh uh cues. So uh the idea here is we instead of partitioning uh partitioning the entire device we only partition

partitioning the hardware cues and for the u uh uh network card and storage SSDs they already have so many cues. So that means we have much more uh hardware cues available than just you know the number of devices and in order to uh part partitioning the uh uh hardware cues that means we also need to build some software solution on top of this uh uh uh sharing

model as well. So to do that actually we could leverage the existing uh linear kernel uh solution built for the networking and storage as well. For networking, we already have the XDP uh available and the XDP program is pretty much targeted use of the uh uh network network card cues and for the storage actually we have the IOU ring. We also have the uh UBRK with that

actually we could also bind the IOU ring and UBRK with use of the uh uh SSDQ as well. And with that actually we on the host kernel side we could actually use the existing XTP and UBRK solution and on in the spawn kernel side we just needed to build another software device inside and they they would communicate with the existing shared uh uh memory uh uh together.

So with that actually we would still do achieve the uh uh uh near uh native uh uh performance and uh with near uh uh with the real hardware isolation. And another uh thing we might need to share is the uh fire system. And uh because even if we want to put you know uh multiple containers we might not want to you know build a container for each

of the uh build the docker image for each of the container right and even the some of the containers they share some the darker image as well and they build some you know overlay layer on top of the shared uh docker image. So for this idea actually I built something similar because the existing overlay file system does not work for my kernel and so the the solution

we build is called Dex FS. So Dex FS is very different with all the existing new file system because it must work for different kernel and the different kernel they they only share the memory and uh everything else is not shared. For example, the the uh they should not share the uh kernel data structures like the uh the uh kernel I know the kernel entry this kind

of uh kernel structure and they they should not share the uh any other kernel uh resources like you know the the spin lock. So this is very challenging because what that this uh that means we need to build some autoomic uh uh some atomics within the memory. Fortunately we already have some hardware atomics available in the memory you know like a compare exchange uh for uh uh

provided by CPU. So pretty much we build all the within the shared memory in the uh Dex FS with that actually multiple chronology although they share the same memory they could also safely write the same shared memory uh concurrently without any uh problem because we build the uh atomics on top of that and because of this actually uh the host could also will share the uh file

system. For example, you know, the uh Docker image with different kernel instances and they could also not not only read but also write together and they could also share the page cast within the shared memory region as well. And with that actually you know the uh uh we could achieve the uh real uh writable file system and within the shared memory and there's another advantage actually which

goes beyond my kernel. So uh and this is a uh beautiful part of the design is although the file system is designed for multi kernel. However uh it could also work beyond the mic kernel because there is another use case actually uh we need to deal with for the similar uh scenario which is share the the memory between different host which is CXR. So for CXR uh

3.0 zero. Actually all the infrastructure is there because the CXR memory provide the uh the coherency uh for the used to build the uh atomics we uh we need for the DX FS and with CXR uh memory pooling actually you know multiple host they do need to share the same uh memory region which is actually essentially the same problem. What we are trying to solve for the

micron uh uh use case and the use case for the uh CXR is pretty much the uh uh uh uh AI training because for AI training we could have uh uh different CPUs. Uh we could have the CPU cluster on different host and they could actually share the same uh CXR memory pool. and if their CPU needed to uh coordinate uh within the cluster today we could

could have RDMA right so uh however with CXR we don't need RDMMA we don't need a networking at all just share the memory and also actually let's say if the uh CXR cluster they want to share some uh you know container uh image or uh darker image within the cluster. We could also do it with CX armor without any networking at all. And of course there is

another problem with uh kernel isolation is uh although we isolate kernels but they do still need to communicate with with each other for example when they do the uh resource update. So one kernel need to release the resource and uh and another resource another kernel need to take over the resource right. So this kind of communication is still you know some uh shared communication ch channel we

need to build for the mic kernel. So for that actually we build some uh uh uh uh we use the shared memory again to build the uh uh communication channel on top of uh the existing uh uh infrastructure called virtue socket. So vert is the infrastructure used by the uh v machine today. because they communicate with this uh socket uh on the uh between different voting machines

on the same host. However, uh for mono, we need to change the underlying infrastructure because this one needed to leverage the uh shared memory instead of the vhost v uh uh uh voc communication channel. So that's too heavy and it's not practical for the uh micron use case. And with that actually the existing application if they already use uh vshock actually they pretty much they almost need

nothing to change just probably one line of the uh uh change to uh uh for the uh mic kernel uh ID uh compatibility but other than that there's nothing uh uh in the code they needed to change so it's almost 100% % uh compatible and uh let's take a look at the security model. So the security model of the mic kernel design actually is based on the

kernel enforcer trust. So that means you know the kernel must be trusted. So you need to trust your kernel. So this is the entire security model built for the uh model kernel and this is actually weaker than the virtual machine. The reason why virtual machine is stronger because it uses a hardware virtualization and the hardware virtualization is stronger than the software one and even the kernel enforced

one and uh uh however there's another advantage over volume machines you know uh regarding uh security is the attack surface because we there is no uh uh complete operating system That means you know we don't need uh for systemd right we don't need any other you know initialization you know uh dozens of service we don't need that we just need uh your own service with that that

means the attack service is much smaller than you know the traditional operating system or the virtual machine with the complete operating system and the kernel uh vulnerability is contained with in a single instance. You know this is compared with the container. And what does this mean? And you know this means if you don't if you could trust your kernel actually my kernel is a very good choice

for you. You know you control everything. You don't need to worry about you know uh running some untrust kernel. This is a very good scenario and uh cloud actually is almost perfect today because you know when we when you get a cloud you already get a virt machine you don't need another virtualization right you already have some virtualization already and uh and of course you know uh

you have you if you have your own dedicated server you know they are physically secure for you you don't need a virtualization right there's no reason you need to bother And uh and of course you know uh all what I mentioned is pretty much focus on the kernel space you know how the multi kernel works within the kernel and that's only the only part pretty much only

the half of the kernel. So there is another half as you can see is in the user space. So the euro space is actually uh slightly uh easier actually is slightly easier because in the euros space all what we need is autoation. So and because we don't have a hypervisor that means the autoration is much easier than cumu. So what we have here is I built another

uh project called curve. So this project is dedicated to auto the kernel instances. You could actually put the uh use the existing docker image and you know boot it with the your kernel image and you know uh this is the situation to you would need if you want to try my kernel in the eurospace. So uh curve is the project provide everything here and it could initialize

the resource and uh you don't have to you know assign all the resource to your you know kernel instance for uh isolation and you also it could also create the new instance and load the kernel and docker image into that instance and you know uh eventually run the kernel together with your service and it could also you know do a lot of other orchestration you know like

a shutting down completely delete everything you know this kind of uh uh orchestration layer and uh mostly importantly it could also actually uh uh validate all the resource like I mentioned you know the the resource management is not easy you know if we have a different resources we need to deal with the conflict so this tool actually specifically designed to validate the resource when you uh you

know in the command line to allocate is the resource for your service these two deal with everything behind the scene so you don't need to worry about you know whether the memory regions they have any overlap or anything and uh and another advantage is because you know the hardware they are not just the hardware they have some topologies as well right because they for memory we have

numa so numa have topology you know different CPUs they might be closer to one specific numa node right so this uh tool also actually designed to be the topology aware so that you you don't need to worry about the topology by yourself you don't need to worry about which CPUs are closer to this numa node you are using to launch this kernel this tool will uh do

everything behind for you And the the the situation could get more complicated if uh it gets involved with the devices right you know the PCI topology that this will get more and and uh that's all I pretty much all I have so and I do hope you uh get interested in my kernel so the next questions is if you are interested how do you get started how

do you you know get want to try how do to try this. So this is pretty much the steps pretty much just four steps you uh uh you have to getting started with my kernel. So the first thing is you know money kernel is not yet merged in uh upstream Linux uh and so however everything is open source. So the first thing is you just need to

clone the kernel which supports the uh uh my kernel uh feature and once you clone the uh kernel you just need to build the kernel and turn on the my kernel feature we developed and that's all that's all you have and what you have next is pretty much just like build other kernels you know make make install all the same and after your kernel is Ready. The

next thing one additional thing you need to do is I know this is still a little bit ugly is we need an uh put a parameter for this because when we uh start uh allocated the memory for the uh spawn kernels that means we need to have some uh physically u continuous memory available for allocation. However, you know, Linux kernel has some problem to allocate, you know,

such a huge memory chunk. So, that means we need to do something uh with this boot parameter to reserve the uh memory and put this one into your loader like a grab and once this is ready and once you put into the uh kernel with my kernel support, the next thing is you need to uh download the curve tool. I just mentioned you know for all the

authentication and you download this tool and install this to true and the the last thing you need to do is you know uh just use this tool to create the kernel instance and this this command line is very easy to understand right so you first need to initialize the resource that means the hardware resources you want allocated to the kernel like you know a resource Cool. And

once that is ready, you want to create a different kernel instance, you know, probably with different kernel binaries all the same. So, uh this true does not care. And once you start, you know, all the uh once you create the instance, you need to load the kernel image, you need to load the darker image, you know, uh uh together with the uh kernel uh service as well.

And eventually you could you need to kick it off you know everything will start running. So this is pretty much all you have if you you know just want to try the mic kernel. So I hope it's not that common. So what's next? So I know I have uh we have done a lot you know to make uh my kernel work so far. However you know there

are still a lot of work to do. So there are a lot of work to do actually. So the first thing is you know the uh CPU over commandment uh and you know there is one advantage I didn't mention you know uh of virtual machine is this CPU over commandment. So that means you know although you have one uh uh virtue CPU for let's say two uh

different virtual machine running on the same nodes they could actually share the same physical CPU. So this is you know how the cloud providers make money right. So they actually want you know the CPU overcommenment. So when one virtue CPU is idle they want you know the other busy one to take over the physical uh phys CPU to better utilize the uh physical CPU. So this is

the feature they need for the uh for their uh you know cloud customer. And another uh feature I mentioned you know is for the AI agent right. So the checkpoint and res is definitely a must for the AI agent sandbox. So that means we need to support this as well. However I do have some prototype but not yet ready to demo uh for this feature and the

kernel library update like I mentioned is much more complicated than uh than my theory. So we do also need you know a lot of work there as well and although it already support docker but it does not mean we have the complete integration or support with uh kubernetes. So the next step for uh the existing uh containers uh orchestration is you know to uh get it connected

with Kubernetes and so that you know uh the Kubernetes user could uh use the my kernel with the existing Kubernetes command line. Excuse me. And the last uh uh feature I would like to add is the confidential computing. And this one actually is uh for the uh probably high performance computing. And they probably want you know the uh confidential uh computing support for the isolation. And for

the virtual machine we pretty much have the uh uh uh this feature ready. But for my kernel this is still a very open question and uh and of course if you want to get involved you know like I mentioned everything is open source so and uh we don't like you know although we are open source but you know we don't like many other open source projects you

know many other open source project they open everything after you know the design is complete after you know they have do a lot of development work but I do the opposite so I open everything from day one. So as the reason why I want to do this is I want to invite people to collaborate with uh from the very early stage. So you could have the bigger

impact as a contributor. You could also actually you know uh uh contribute to the design if you have you know any design feedback. So more than just you know some bug fix if you feel like you know bug fix are too trivial too too trivial today right with AI right so we have something more challenging here right and uh and of course if you want to try

this out you know everything is on GitHub on uh our GitHub repo so you know just try and uh if you see any issue you know create a GitHub issue and of course even better you know just submit and pull request that's much better and of course we can build this and this is a quick summaries and I do hope you know everything you will not get

confused for this uh long presentation and and I do hope you get the idea and the uh basic concept of the and if you have any feedback or any question you know you uh this is all the contact information I have and you can send me some email and all the project are open source you know it's in our uh uh GitHub uh directory and also I

do have some video demos but today I don't have time to show so feel free to watch our uh uh video demos on uh uh YouTube you know like you know how to use the uh docker container that's very cool in my also have a discord uh community and this QR code is for the discord. If you are interested, you know, feel free to join the discord

and we also have a meeting list. Yeah, that's all I have. Uh if you have any question, let me Yes. Yes. Oh. Uh the question is when do I finish upstream? I think that will take a long time because you know upstream people they have different opinions and the codebase is very large you know probably more than 50 patches so it's take time yeah yes Yes. Yes.

Yeah. Is this okay? I'm going to change the batteries. Check check check. So what I do is Check. Check. Good. Hello. Oh, good. I think so. Yeah, I think we probably stuck. So people probably still stuck in lunch. And today we will talk about uh uh how wis how we push the wrist to do the cano upstreaming even before the chip comes back from the factory. So

actually and most of the work the upstream work is not done by me but actually the walk actually initial initiation is all by me because I'm the the uh the customer of the SOC riskfire SOC probably the biggest customer. So I had to push that to get it done. Actually this talk is been presented back in uh the the uh NIST uh conference and as well as

in Boston because uh uh I think uh I circle around the world and make sure everybody understand the the importance of especially for the hardware and most of the hardware guy doesn't understand it and and and the software guy understand it but usually really hard for hardware guy to do things. So even on ARM uh they didn't do it perfectly. I think X86 is definitely better and

and I'm going to repeat the the talk but we add a little bit more progress on our chips. So um the talk is all about uh biculant upstreaming for for recy uh for saber and then we set up the the role model and and now set a very high bar for the for the coming SOC. So because one vendor return it and then uh most of other

vendor had to follow anyway. So that is the good stuff even though it's not perfect but it's good stuff. So myself is ying um I'm a software guy and I used to do uh Java VM porting and compiler porting and writing the back end and then I'm a founder of the deep computing. I found it when during COVID after I sold my software stuff and then you

know when we do uh compiler usually we we really arrogant we think we're the best software in the world software engineer in the world and we don't even look up onto the kernel side either so we we think that we're the the most prestigious software engineer very serious. So you know Alan Key says one word if he's serious about software I'm sorry that you make all your

money and now you had to lose it all on the hardware side to improve it. So that's a story of me. So and then I start making uh wisfi laptop and for even for very slow CPU but actually on the hardware side is not very slow. If you look at all the Chromebook of ARM sim similar uh four core and 8 GHz you should be able to

run Chromebook. So why so slow? Yeah, you know why slow? We are software guys, right? So I make the most expensive laptop back in 2023. You can look up everybody saying I'm a scammer. $5,000 each and with uh with NFT. So, so I know Trump is in charge. So, and and that's why I make sure it worth something and and lucky enough and um the British computer

museum have one of them. So, the stage and then 2004 I made a very cheap one and and it's a core much better performance much more RAM and then you very usable one and very cheap all the way from 5,000 to less than 500 and it runs um Fedora and Yuban tools Later on I find it out I should not waste all my money on something that

cannot scale. So I just want to put something on risk buy and then I don't use it. I just realized that 99% of money has nothing to do with risk five and then one of my friends saying that why don't you partner with framework? I said what the hell is framework? That's back in 2005. um uh uh earlier earlier 2004 and then and then I got introduced

to Nina the CEO of framework and then oh my god it's a laptop that can change motherboard man I just find okay I can save 90% of the work I just focus on the motherboard so that's multi multiple architecture and upgrade motherboard because I make every six month have a new SOC coming out from this five. I make motherboard more than anything else. So every six month

have new version. So that that's perfect part. They framework needs me needs us to to promote their open ars uh AMD or whatever you know they open source all the document but no one use it and I'm the one using it and find out all the problem so it's perfect perfect so we we redo the four call because I'm trying to understand how how the logistic partnering

works. So there's nothing to do canas and then and then I last year I do a very powerful one fifth tops AI and 64 Git LDDDR and then a core and the only thing is not good enough without so memory movement is performance is not good and also is by chip that's going by the new so it's a lot performance impact And whoever buy that version from

us for the first uh patch they earn money because it's the ball DDR is the more worth more than any more the whole framework laptop. So that's a joke. So and and now it's coming. So this is proper usable one really usable you can use daily and with uh a log kernel support system 12 systo 18 and even upstream and then I will talk about a story

so this file wears his arm upstream so I used to run Nokia's mobile phone and this is not everything co source. So I see ARMS any chip comes out using ARM IP we scale KK volume we call it immediate they said it doesn't need upstream no right they make a lot of money right mtk rock kit rock chip all winner marvel they make once the chip out

you have a ecosystem you make a lot of money but But wispy is not I don't know whether you guys can see clear wispy is definitely not just now even we see that even the same hardware when in the same ninness similar hardware to ARM four core 1.5 GHz we cannot possibly run Chrome OS or the decent performance in you know why because at that time the

kernel the two chain the stack of software is not so the first version of mother I find it very funny is that if you have a old hardware old hardware meaning newer than ambient is faster than the old versions weird isn't it right usually like Apple they put more new software running the hardware is very slow right so it's weird because ris software is getting better and

better and better so that is why ris future is on the future software is not your current software. So that that that makes the difference. We had to upstream. If you don't up you don't you don't upstream you have no futures, right? Because and and I take a look example one here. So we we have uh the first motherboard JH710 you know and the IP is 2018

the CPIP and the SOC is 2012 or 2022 and then and and the P my make the motherboard is 2024 you can see actually six years but upstreaming the first upstreaming started the thing control is u uh February 2023 uh 20 uh uh 20 and then now the display controller and the ARM mod is still not still stuck. So you can see and so because the capacity

of it this forever to upstream then you start understand why ARM not doing upstreaming right any ARM is not doing upstreaming any of the mobile phone they don't want to do uh Android upgrade right so that is the reason and then the east wing one even even tougher. So, and um the the 50 top AI. So, the the IP is the year uh 2020 and the SOC

is um uh year uh and then the motherboard is the last year. So, and then the CPIP now is not even finished the upstreaming because coherency difference uh from is affecting all the architecture. So, basically I make the motherboard make the product for it. I'm stuck with the old hardware, old software saying we're dead, right? Literally dead. You can't know is the old anything above is oh

unless people backport all the features. So who's doing the backp right? So I'm stuck. So I feel the all the pain of it and then um and back on um actually and then I spoke to grab how on earth back in uh x86 you know in the old days and and how x86 does it says um um um you you can actually you can Get everything up

before your chips up FPGA whatever just buddy do the upstreaming even just make sure you compile otherwise none is going to kill you right and doesn't run doesn't matter because the hardware is not there no one can verify it's not run right isn't it so that that is the the difference so and then I say I need to go back to x86 so that's that's it that's

That's 20 years old slime amazed. So, so basically I think most of the people doing the kernel I think the first generation the kernel guy is x86 the second generation kernel guys arm and most of these five guys so they they forgot all the pay because arm just just make business doesn't need futures right so and that's why I pushed the uh wrist fight is going on

on on this and then Um uh we we managed to do something before the uh after the take out and then start doing all the all the it's not exactly the same but slightly better but still very late. We can't do the the kernel code submission before tape out and and and I don't know why but I'm not breaking CPU so and and it's is still behind

this but it's still better than now so and and because no one had done it before including me right and then I we we do a lot of funny things saying that Okay. And asked the the spaceman, "Do you want to do it?" So I'm my I'm threatening, but actually I don't have any choice. I said, "If you don't do it, I'm not going to buy your

chip and make the motherboard for you, right? I just threatening, but they are my friends." So, and then I say, "No, no, no, no, no, no, no. Okay, I help you to do it." But even though I don't know how to do it, right? I never bodybody do any kind of upstreaming at all but I found a few friends just in case I need and the manager

convinced the founder of it they said let's go let's do it okay and still don't know how to do it right and and funny enough um we because space may have a a few uh maintainer and they are so conscious they say my mutation they never done this before I always wait for the hardware compile runs and upstream. I never lie. So I never never do anything

to before we really going to hurt my reputations right even though grass bug kernel have bugs otherwise we won't have seven zero man the one dog zero will be perfect so but anyway so and and more fun is uh we we we take a look how much work is How do we inherit the previous chip? So and especially I've done two chips. One is K1 and now

the one is K3. Actually you find very good things. We have uh 60 70% of IP is inherited. So that save a lot. Well K1 is already took two year upstream all the way uh beside uh except display controller which is really awful. I had to fix it in two months. and and because uh space I forget out K1 I have no resource for it so I

had to fix it myself and and then um um that is the the convincing thing to do so so then we start I convince somebody to do it I just talking about a story more than technical things I think the story is more interesting than the big a technical code that AI can do. So and and um I I I write a long email with all maintainer

involved and then I send out this architecture. You know what email it comes back and then grab a long email. Who the who the a decide this chip? Anyone spot any problem? So basically the main problem is that they have a symmetric So in risk five have two world one is print RV23. One is RV23. So RVA23 the AI what instruction you you support work this work

this right man now you have two clusters supporting different instructions. Oh man go bananas. the whole thing go bananas you know and and and and plus we have finding out you know arms the way you doing PCIe is very slightly different from from K3 and all these things go pop up all the hardware popups the problem up so and you know and more more it's basically unfixable

so actually 16 core now you will see on my hardware go to this five booth then take a look it becomes a core one so you cannot see the other uh uh a small core. So that small core is completely wasted in in general computing. You can you can only use it for AI and and even more complex is that how I'm going to use AI how

will diver divert all the AI capability or uh uh whatever onto the cluster own that makes the whole software systems compressed. So that we learned that actually Pica upstreaming you can ask the software guy to help you to review the hardware right and then you don't make mistake I think and the reason why they do it that way is back because it says that and and the

the kernel um why you have big core needle core this inherit from ARM but I'm doing very clever same instruction both support no matter it just performance slower But on risk five is not for example they be saying why do I need a small core run hypervisor there's no need right why a small core you know I just put small core small area in order blah blah

blah blah blah you know and and that is the hardware guys mind but it creates the whole software ecosy I haven't think about how do I run the AI uh separate just on on the small core uh in order a with AI instruction customized instructions and with RV different vlan so there's very complex things and and and and more when we start uh upstreaming because the hardware

is not coming out and then all the device tree is funny right it's like what lame we're going to use because this call FPGA and then they say no no no don't ask FPJ because you no one going to have your FPJ you will not beble to build anything for your FPJ then we had to come up with a lame with the first product but we don't

really know what's the first product say and you just pick pickle okay whatever it means so I was saying they pick penguin probably when I do it CPU we do that you know so a lot of a lot of thing never done before we make a lot of mistake but that's purpose of how how we going to improve it right so we improve saying that right in

the design stage we actually on the FBJ side we we can verify all we can upstream all the kernel code and and related to CPU straight away before tape out once you verify on the FBJ you all the hypervisor uh customer instruction blah blah blah blah blah blah all these CPU related feature you can upstream and you separate the CPU feature out of the SOC features. So

for the SOC one user has to make very consistent consistent means that take a look the condo tree take a look ARM SOC IP they use use flipping the same one then you have all the driver someone done for you right then you don't in you don't you are not going to be in pain for rev the driver and upstream right so that we saying the consistent

pick someone already done before and then and the kundle stream upstream can have the driver the support and you don't have to do it yourself. That means that I I made a SOC our software guy have less software guy less maintenance every perfect right so that's why we said a consistent design and also I went back to it just just commit the code make sure compile minimal

right no shame and pick uh that that just need to pay enough so and that's all actually for the experience of it and actually if anyone interested think in helping we still got a lot of work for this SOC to upstreaming right and to make it successful and one big thing is display controller the IM uh GPU driver reverse engineering support because L I think everyone knows

GPU driver they will support upstream right so like uh the most predominant GPU they saying they only support uh uh Lind 5.1 kernel 6.1 kernel that's all they support right that you're going to stuck with it so that's same to us the GPU is exactly the same so you might I had to do research engineer and support so everyone come down and take a look and help

and this is our example now we have the hardware we building the seven do zero kernel and running on it and just have serial USB and network and PCIe. That's it. Just similar very similar to x86. So that's all from me. Any any questions? And feel free to come to the risk booth, take a look all these uh the thing the the pain we going through and

be part of it. Thank you. All right. Hello. >> You guys here in the back? >> Can you hear me? >> Oh, yeah. Sorry. >> we can ask Jordan otherwise Jordan. Uh, when there are questions, can you bring the mic around? Thank you. >> Test. >> Yeah. Um, I'll wait five minutes. >> Yeah, it's supposed to be 45. So, just wait in case people want to Is

this big enough for people in the back? Can you read this? >> Can't see anything. Of course, it it did work last time, so it might work today or it might not. And we'll cut short. I H. Sorry, we just need to wait a bit more. So, it's supposed to be 45 and currently it's 42. So, just in case people want to join, I expect people to

be a bit not late but right on time. So, just to be sure, let's start at 45. I can I can It's tilted. Feels weird. I should not move. I should be completely still. >> Yeah, it does. And I'll have to type at some point. And if I need to hold it, it's going to be a pain. I'm I'm good on this side. Uh but yeah, I've

got small hairs, so maybe it's same issue, you know. All right, it's 45 sharp. We can start. Okay. So, so hi everyone. um you are here today to listen about packet filtering and BPF and a lot of other thing related to network packet filtering. So um I'll introduce you to a tool called BP filter for people who don't know it. Who who has no idea what this

is? Well, no. Who know what this Oh Yeah, sorry. That's nice. Thank Um so we'll dive into it um quickly. before we dive into it. So, a quick introduction. So, my name is Quentin. I work at Meta. I'm fighting with the microphone, right? Okay. I work at Meta in the Linux user space team. I've joined Sorry, a lot of disruption. I've joined roughly uh four and a

half years ago and I've worked in in the Linux user space team for three and a half. We work on open source stuff. systemd BP all right BP filter uh kel whatever like a lot of open source project distros even so I've been working on BP filter for as much as long as I've been in the so the plan we have today is to discuss a bit

about firewall like what kind of firewall do we have today that we could use then we'll discuss about BP filter and what it is. And I guess that's the main reason you're here. Hopefully, otherwise it's the wrong room. And last, we'll have a live demo and I can show you how we use it. And it. Sorry. Hello. Okay. And so finally, we're going to have a live

demo so I can show you how it works and how you could use it. All right. So, let's start with the basics. Who don't know what a firewall you don't know, not trying to single you out. Sorry. So, uh, a firewall is a way to like prevent network traffic from coming to a computer basically or a network. Pretty simple. You define rules. You want to block specific

IP, specific protocol, and anything that match that rule won't go into the network. So what kind of solution do we have right now to do firewall to do packet filtering uh on Linux at least I guess everyone knows about IP tables right who is using IP tables who is using IP tables legacy all right okay so some people still so IP tables so when I mention IP

tables here it's IP tables legacy uh it's the old IP tables it works differently from the new IP tables. Uh, and basically you should not use it I guess according to the net filter maintainers of course. Uh, it's the whole way of doing things. It's still live because like when a firewall works like you don't change it. Um, and on the right hand here you can see

some synthetic benchmark. The benchmark here is just a very very synthetic workload that you would probably not have but it's very hard to benchmark firewall unless you have a specific rule set in which case you can compare different firewalls and that applies to you but having generic benchmark is quite difficult. So this one is to take with a grain of salt, right? Um what this this benchmark

does is that basically we have one rule to drop packets. We drop every traffic and then we had more and more rules before that will be processed but don't match the traffic, right? And we try to find at which point the the throughput starts to drop. So we have a 10 gig uh network link and for IP tables when you start to have like 128 rules filtering

on just the IP of the packet you start having a drop in the throughput. So if it drops you now have increased latency because it takes more time to process the packet and you won't be able to have the same uh amount of traffic coming to the host. So if we don't use IP tables we can use NF tables. Who is using NF table through IP tables

if that makes sense to you? Okay. Who's using NF tables? All right. Some people. Okay. Um NFT tables is a modern replacement for IP tables. Uh when I mentioned who's using NF table through IP tables, there's a front end of IP tables that actually rely on the on the NF tables back end. So those are two different things and NF tables works differently because you send to

the kernel uh net filter by code which is then processed in a net filter VM in the kernel. that benchmark here as I said it's very synthetic and it's not really representative of like what you could have in production. uh in that specific NF tables will be slower than IP tables. So when we have a lot of rules filtering on the same thing it quickly becomes slower

much faster than sorry much faster than IP tables. What kind of options do we have otherwise if I've got like a Linux machine running what could I do to filter packets? There is a third option available. which is BPF. Who knows BPF here? Well, who don't know BPF? All right, quick uh introduction on BPF. Uh BPF is a kernel technology that is used to do all sorts

of things in the kernel, but you can basically insert any bite code into the kernel at runtime and make it run as part of the kernel. Right? So you can do packet filtering, you can do observability, you can do a lot of things like security also. U basically what it looks like is that you have a BPF C program which is some kind of subset of C.

You compile it with clang into BPF health file and you load it in the kernel. when it's in the kernel, you can attach it to somewhere uh let's say um nick for example, a network card and when a packet will cross that nick, your program will be triggered and you'll be able in that case to read the content of the packet with the headers of the packet

and potentially drop or accept the traffic. Um that works for some hooks in the network stack, but you can have hooks on sys calls, you can have hooks basically anywhere. uh and in that specific example I'm mostly focusing on like filtering packets. So having BPF programs attached uh so BPF is great. BPF is like very interesting, very powerful. It's very very efficient. You can filter package before

a socket buffer is allocated in the kernel. So even before the kernel reserve memory to store the packet, you can start processing it and dropping it if you need to. That's great. The downside is that you have to write C code. So I don't know about anyone everyone but like usually the people that write C code are not the same that create like packet filtering policies. And

so if you use BPF, you would have to write that C code to go deep down into the packet and its headers and filter on what you want to filter and you have to maintain it and you have to deploy it and it's much more difficult to maintain than a simple IP table rule which would work everywhere but it's faster and it's way faster. what if we

could have something that is simple to use like IP tables just call it it works and it's filter it filters traffic and something that's as efficient as BPF without all the pain of maintaining C code and writing C code and what if that thing had like a great logo for example just saying and like a great maintainer just an example Of course. Well, you're lucky. So, BP

filter is basically this. It allows you to use like some kind of DSL to define the filtering rules and use BPF in the back to generate BPF programs. Use BP filter in the back to generate BPF programs. Um, if we if we use a synthetic benchmark here, we can see the difference between IP tables, NF tables and BP filter. And we can see that we can have

much more rules in BP filter before we start dropping traffic. Um that is again very synthetic and it might not apply to your use case. The speed improvement is here but it also depend like what you do and how you do it. Of course uh you can see I don't have any benchmark for BPF just a BPF program because it doesn't make sense to do that in

BPF directly. Uh there are multiple ways to do it. No one would do it uh the same way I would do it. It's it's not really relevant for this point, but you would accept something as fast as BP filter. So how do we use it, right? Uh you know now about a great tool called BB filter, but like what do you do with it basically? Well, that

is you. you are a great blue shape and you want to drop traffic there is a malicious like 19216811 IP on the network you don't want that person to come to you right you want to drop every packet coming from them so what you do is that you reach out to BP filter and you're like oh okay I want to drop traffic from that IP and BP

filter will create the BPF program for that so you give it the rules it creates the BPF program and when it has the BPF program it will load it into the kernel and attach it to the right location and you end up with like one command to define a filtering rule in BPF. No more C code, no more like BPF tool to load the program, no more

loader, no more whatever. It's just like one CLI and it works. Well, it should If it doesn't open a ticket if we dive a bit more into it, like what is it composed of, right? Uh if I was giving this talk like two weeks ago, that would be roughly the idea. We have BP filter the demon running somewhere. You have a library and a CLI. The library

is for people that wants to integrate with the demon, right? Uh the CLI is one of the use case. The CLI is linked to the library and it communicates with the demon That's fine, but it was a bit cumbersome as we deploy this at Meta Arena. Now uh we want to avoid having a new demon in production. We want to limit the overhead. So we just moved

to this which is basically no demon just a library a CLI and it works right. So when you use BF CLI now with BPF lib BPF filter you don't have any demon and you give it the command it's going to create the BPF program attach it and make it live on your system. What it does is that basically there are three different steps. The first one is

the DSL which we will see a bit later. Um it's quite similar. If you're used to IP tables or an F table, it's not going to be surprising to you. When we have the DSL, we convert that to an internal representation of the rule set. From that internal representation, internal data, we create the BPF by code. Right? It's very similar to like assembly code. We'll see what

it looks like later, but it's like basically assembly code. Um, and so what we do that we allocate a huge buffer of memory and rewrite the instruction, the assembly instruction one after the other into that buffer. And so every program generated by BP filter looks roughly like this. What we have is that first of all there is a prologue. we set up some kind of internal data

before we process the packet. Um, every program we have will be called once for every packet. So if you attach a program to the nick directly, every packet going through the nick will be called with the program that's attached to the nick. Right? What this mean is that the runtime of the program, the internal data and storage you have will disappear once the program the program terminates.

So it has one run, it does what it has to do and then everything disappears and it has another run with another packets and everything's roughly like temporary. Uh that's how BPF work. uh and so we have that common prologue which uh help us setting up some internal settings for this run of the program. Then because we support different BPF program types, we need to find a

way to have some generic data we can work on. Uh BPF can be attached to various hooks, various location in the kernel and we can attach to XTP which is very early in the network stack. We can attach to TC. We can attach to a croup to filter traffic coming to a specific croup or we can even attach to the IP tables and NF table hooks. But

all these programs are different from BPF point of view. They expect different arguments. They work differently. They return different values. And BP filter must be able to support all of them. So to ensure we do that properly, we have that red part here which is a way for BP filter to like set up the runtime context of the program for that specific packet. And so from there

on we know that every field is properly set. We know that we have a field in the runtime context with like what's the protocol on layer two header? What's the protocol on layer four? What's the packet size? Because depending the program, we don't compute the packet size the same way. And once we have that setup done, we can process the rule the same way for every program

we generate. Uh and the rule is pretty simple, right? We have a way to match the packet and we have an action to take. Uh we have uh for example, we'll have a look if the IPv4 source address 1921 16811. If it's that all right, we can drop the packet. If it's not, it doesn't match, we jump to the next rule. And so when that's done so

we have all the rules one after the other in BPF by code it's like unrolled uh and we if we have no rule that matches the packet when then we'll have a default action which is like a chain policy in IP tables or NF table like by default we're going to accept all the traffic but we can like drop stuff with the rules in the middle and

that's basically it. Um, you can see at the bottom we have custom functions. That's a way for BPF programs to define pieces of code you can recall like you can jump to and come back from. It's basically functions. Uh, and we use that a lot to avoid code duplication by code which would grow the program substantially. Uh, there are pieces of code we reuse a lot like

updating a rule counter for example. Well, we have one cor routine for that which we reuse a Uh and similarly there is some stuff that we don't want to write the b code for. Um I guess some of you already wrote assembly before. It's a pain. It's a pain to debug. We don't want to do that for some features that are not used a lot and that

might not be that might be um very fragile. Uh processing the IPv6 extension headers for example the IPv6 options that's a pain. uh most of the traffic won't have any extension header anyway. So we might as well have a function for that and just reuse it. And this type of function uh we do write that in C. We compile it with clang and we extract the bite

code and we copy paste it basically in the final BPF Um that is very useful to us. We save time and it's more reliable especially for complex assembly. Uh the downside is that you call a function from BPF so you have some overhead. So we do that for non-critical uh non-critical features or the features the the code that we reuse a lot because you have the overhead

of of creating a stack frame and jumping uh later in the program and coming back then. All right. Uh is there any question so far? Am I clear? Is it like a lot to digest at once? I see some. Okay, I can get it. We We'll have a demo right now. So hopefully it's going to make more sense to you. So demo time. It should not fail.

Honestly, we should be fine on this. Yeah, too late now, I guess. Uh, I just have my display here too, so I can actually see what I type. All right, so you should see that. All right, that's great. So, I need to find a way to speak and type at the same time. >> It works. Great. >> Sorry. I Okay, it works. I I'll try not to

move too much. Um, so before we start, that's the documentation for BP filter. Can you see that? >> Okay. I can't turn my head, so I I'll re you on this one. Uh that's the documentation and we can see how you use BFCLI the command line tool and we can see how you define rules and rule sets. So that's what we'll do So before anything let me

be sure I have everything I need. Um the setup is pretty simple. I have a VM on my Mac. Uh I connect to the VM through SSH. Uh and the ping you can see here is my host pinging my guest. So, my host being in my VM, we'll play with that a bit in a minute. Let's ensure I don't have any uh leftover chain here. All right.

Let's have a look at this. Can you see >> All right. So, what you see here is how we define what's called a chain. People using IP tables and F tables roughly know what I mean by chain. is like it it's a set of rule that you apply somewhere in the in the filtering path in the networking path in this case it's a pretty straightforward chain is

just a keyword demo is the name of the chain I called it demo right uh bfoo XTP is the location we will filter packet from so that's XTP it means we attached that chain to your nick basically uh by the time this chain is executed there is no socket buffer allocated yet it's very very early in the networking path of the Uh I give it the interface

to filter on. So where the program should be attached to that's interface 2. That's the main um interface on my machine on my VM. And the default action which is um accepting any packet. No. So when I'm here what I want to I have a lot of trouble like speaking and typing at the same time. So we'll have some quiet times. Um so when I'm here what

I do that I do rule set from a file and that file is 1.bf. All right no error that's fine. Now what I do is I'll try to print the rule set. When I say rule set it means like all the chains all the filtering programs defined on the machine. So printing all the chains we can see I've got just one chain which is the one I've

just defined and we have counters. So by default the chains will have a counter of number of packets and bite that goes through the chain. Uh we can see there is traffic coming. We can see there are no errors. So that's great good to me. And that's uh not much interesting to PNS. It's like it's just something somewhere like watching packets going through. What might be more

interesting is that if you use the usual like the standard BPF tool uh binary, we can see that if I check the network programs, I have one program IDC on interface 2. You can not trust me on this one, but I tell you that's my program. It's not coming from anywhere Let's bring like let's build some confidence, right? What can I do next? I can use Oh

yeah, I should show you um the bite code itself. using this very verbous command from BPF tool, I can dump the actual bite code. So this is BPF by code. It's like assembly. Uh anyone having Anyone who has seen assembly at some point wouldn't be surprised by what happened here. Uh so this is the actual BPF bite code. It's what uh BP filter writes into that memory

buffer loaded into the kernel. Uh that is the very very beginning of the program. We store the arguments in some kind of uh stack uh stack space. Uh we can see the registers being used. uh BPF has 10 register from R0 to R9 and there is R10 which is used to sort the frame pointer. Um I won't go into too much details but like it's basically BPF

assembly right u this is a a specific function call uh BPF has access to some kernel functions uh and this is one of them uh this is part of the kernel sources so we can actually use kernel functions from BPF and this is another function call which doesn't look good it doesn't have like a pretty name this is one of the functions defined inside the program itself.

So it's at the very end and what we do here that we jump into that function somewhere in the program and when that function is done we just come back to um instruction 22 in that case and that will be done for assembly code. Uh we won't watch assembly anymore. You're safe. Uh now let's have a look at the BPF file system. who know what BPF file

system is who don't know then right um BPFS is a virtual file system in the kernel basically when BFCLI creates a BPF program it's going to load it into the kernel when it's loaded what we have is a file descriptor to that program right the issue is file descriptors for BPF object acts like a um a reference counter if we close the as file descriptor the program

is removed and I mean we don't want that right we don't want the program to disappear when we close BFCLI because it's just closed by itself at some point so the solution is to keep a file descriptor open somewhere roughly and to do that what we have is the BPF file system which is a file system where we can pin BPF objects into. So what BP filter

does is that it will create and load the BPF program and then it's going to pin that program into the BPF file system. So when BF CLI returns the program still uh and we can see here what BP filter has defined as BPF objects for that uh simple chain here. Uh BF prog is obviously the program right? So it's the P code we just saw. We can

see a BF link which is a way for BP filter to attach a program to a hook. You can do it without the link but the link allows you to perform like in place automic updates of a program. So BP filter can when you update a chain it's going to create the new W code load it and automatically replace the old one. So that way you don't

have any downtime in in traffic filtering as you would expect from any firewall basically. Uh what do we have? BF context is the um the content in the rule sets in the program but in BP filter format. So we can then reload it and like update it if we need to. And then we have two maps. Um I told you just before the context in the BPF

program at the runtime will disappear when the program returns. Right? So we need a way to store data after the program ends and that that's what BPF maps are for. So a BPF map is like a storage area that you can use in your BPF program that will persist after the program hands. And we have two maps. We have a maps for the debug strings. So when

there is some failure from a kennel function for example we log something. obviously we never log anything because never nothing fails ever, right? Um especially especially not on BP filter side, right? Uh and we have a contest map. The contest map is a way to store how many packets and bytes have been processed by a specific rule. And that's basically it. So now let's do something a

bit more interesting. Uh I have another chain here different name different hook we on TC here TC is a bit after XTP so at this point we have a socket buffer so we have kernel memory allocated it doesn't change much on what BP filter can do it's just that it require more processing on the kernel side at this point there are more resources that have been used

to process this packet uh same option we attached to interface index number two and we can see three different rules. The first one is like uh catching TCP traffic on port 22. We count the traffic with the counter action and we accept it. Same for UDP. And the third one is uh we check the layer 4 protocol in the packet. If it's ICMP, we log the packet.

We actually log the link the link header which is L2 the internet header L3 and the transport header on L4 and then we count the packet and we drop it. So right now I can ping my VM and if everything works as expected when I load this rule set we shouldn't be able to log to ping my VM right so let's do that so this time we

won't use rule set command we'll use chain set the difference here it did not remove the previous chain. So when you do rule set, you apply at the rule set level. So we you you basically override anything that could be running on your system BP filter wise. When you do chain set, you operate at a chain level. So in this case, we define a new chain. If

there was a chain with the chain with the same name defined already, we would have replaced it. Um but there is no chain with the same name. So we just added a new one. And so we have two chains right now. And what we can see here that we don't really care about the TCP and UDP rules. Uh what we care about is the uh ICMP drop.

So we can see that we already matched to 44 packets. So it's been 44 seconds since I've inserted the rule. Uh and 4,000 bytes and growing. And obviously on the other end we can't ping the VM anymore. That's fine. It works. Uh let's have a look at the logs now. So I've got my um my action. The log here is an action. Uh and we would like

to have a look at the logs collect. Oh Jordan, >> I can repeat the question. >> if there was already a rule that said accept, does it I mean it it as soon as you added the chain after the accept, I thought he's going to show us that it's still working because you got to pull the accept out. Does it like does it match a rule and

stop or does it match a rule and continue on until it finds the most restrictive? >> for a single chain uh when it reach a ter what's called a terminal verdict it stops so accept is terminal when there is a rule matching a packet with the accept verdict it's going to stop uh it behaves like IP tables basically if one of the rule match the packet you

apply the action of the rule and then it's done so when we have traffic when we catch traffic on port 22 TCP the First rule here the action is accept. So we accept the packet and the BPF program stops. So we won't do any more There is a way to do more filtering and it's uh with the continue keyword in which case you would jump on the

next Does it answer your question? It doesn't. >> Okay. So the question is >> so this thing says we accept everything right? Okay it does but it accept everything at that hook and then the packets will continue in the network path. If you use IP tables on pre-rooting hook for example you can have uh rules there and you can have actions on the packet. If the rule

accept the packet, the packet will continue its flow in the network path, right? And at some point, it's going to reach the local in or forward hook where you can do more filtering and you can drop the packet at this point. Any other question? >> Then it goes through the monitor chain in the monitor chain. Correct. >> So I'll repeat that. So what happened here is that

the XDP hook is the first hook in the kernel in the network path. So the packet will go through the XTP hook and the chain. At this point we see the packet will accept it and then it will continue its path in the network stack and reach the TC hook. And so again at the TC hook the program will run and filter the packet. At this point

if it if it matches the uh ITMP drop rule when well the packet is dropped and the the packets won't continue in the network stack. It's very similar to the behavior of RP tables and fable. Now let's have a look at the logs. Uh so BFI chain logs the name of the chain which is monitor and we can see traffic. Yeah. Uh unsurprisingly that's just ICMP because

we only log ICMP traffic. We has for L2, L3, L4. So link internet and transport layers. So we can see some details about those protocols. We can see that it's rule number two. How many bytes? And it dropped the That's about it on on this one. Let's do something even more interesting, right? Let's say um I've got u a game server running on my machine. Uh my

guest my host, sorry, my host IP is like the 10 to 11 55.2. So I want to allow myself to connect on my Minecraft server. I want to drop anything else in the network. I don't want anyone to reach out to my precious Minecraft world, right? So I do that and this time we will override everything. So we do set set from a file which is 3.bf.

All right. And it's up and running. I can still access the VM. So it works otherwise I would be blocked which is actually very very common. um when you work on network basically like firewall in that case but something that happen often is that you try the new features right uh and I often try new features and ensure that the new matcher works and I can filter

traffic the thing is on a VM there isn't much traffic coming like there is nothing coming really except you working on the VM so a common case is like to but the thing is usually like when you catch traffic and you play with traffic on port 22, well, you quickly drop traffic on port 22 and you need to reboot the VM, right? So, be careful with that.

And I learned very quickly that the servers we have at Meta are very very slow to reboot when you can't accept them. Not fun having like a a five minute downtime in the middle of the day. So, I've got my traffic here, my rule, sorry. um is great. Uh I can allow my traffic to go through but I might want to like do a land party. Uh

I want to invite a few friends at home. So I would have to let them go I can do that. Of course I can do more So I will assign them an IP and then go with um BP filter to allow those specific rules into my firewall. So I've got uh three friends coming and we can all play together. That's great. Inserting the rule set. Checking out.

Okay, I've got rules for everyone. So my friend can reach out to my Minecraft server. We can play together. That's But the thing is eventually I might have more friends coming, right? Uh and we could play together. Be great to have like a huge land party, right? But there is an issue here. I should not welcome more than a thousand friends to play with me, which might

be an issue for my wife uh if I've got a thousand people joining me at home. But like let's imagine we have a big big land party somewhere and I want like a lot of people to reach out to my Minecraft What happen if I have a thousand rules a thousand people coming or like 2 thousands or 5,000s? The more people I will allow through my firewall,

the slower the throughput will be and the higher the latency will be. And that's going to be an issue for everyone. uh it's not going to be it's not going to be fun for a long time. So what we can do instead is being like a bit smarter with the rule set who know the sets in NF tables or IP sets. So basically that is very simple

concept. It's like you have a bag of IPs and you filter on the bag of IPs on all the IPs at once with one rule instead of having one rule for a thousand different IPs. So that's what we do here. Uh what we define here in BP filter is like we define the key. So we like we're going to filter on the IP4 source address and we

want to check if the address is in one of those address here. So we can do that with just one rule. So instead of having a thousand rules to filter a thousand different IPs, we have one rule that filter on a thousand IPs. U is a bit light. uh we have tested all the way to five millions of IPs in in that set. Uh and it it

filters traffic as fast as like one rule for one IP. That's the huge benefit of it. Uh and so I've defined my key which is IP4s source address and I've defined the the elements in the set. I could as well do uh a composite key. So I would like to filter on the IPv4 source address but also the TCP destination port and I could add oh I

also want to filter on doesn't make sense but like TCPs source port and then I just have to add the missing elements from uh the key and I can go on and like create more complex sets and BP filter will create the bite code that will create that key and look up the key in the set at runtime. time. For now, we'll just filter on one set.

I'll reuse the command and load that rule set. I'll print print the rule set. And we can see that the rule set has been updated. And we filter on a set now. And we can update the set, insert new IPs, and still have one rule. So the processing time is exactly the same if as if I had one um one IP to filter on. It's exactly the

same thing. We can also do something like that. It's slightly different but that is a way to define a set that you can reuse in different rules. Uh so it's a named set. You tell it okay I define a set. It has this name. This is the key. This is the content and then you reuse that set in the rule here. We can see on this line

the benefit of having named set. Did I insert it? Not yet. So it's working now. The benefit the benefit sorry of having um a name set like this is that we can call BFCLI to just update um that set instead of updating the whole chain and modifying the source file and whatever. So I do uh update set uh the name is LAN the set name is my

friends and then I want to add I could also do remove but in that case it's not relevant and I want to have 10 to 11 and we have it in the set and now we will filter on five different IPs and I could have many many many more IPs in that set if if needed. it and that is it for the demonstration and the talk also.

So wait wait wait no no no no no no wait wait wait a minute sorry it's itish right I've got one last slide which is if you want more information about BP filter we have a website which is bpfilter.io IO we have obviously a GitHub which you can have a look at when the GitHub infra is up and running which is sometimes don't say that don't repeat

don't quote me on that please um we have RPM packages which are not really up to date we had a lot of changes lately as we're deploying into production at meta um and I need to update the RPM packages with the latest changes which is no demon uh you can have a look uh at the source code you can open issues if you have issues you can

reach out if you have any question you can reach out if you want like a ticket to work on uh we have tickets for uh issues on GitHub open for beginners uh if you want to have a look feel free to to check it out if you want to have some help into like jumping on the BP filter BP filter code you can reach out I'm always

available uh or if you have any question reach out and this time I'm Um questions. Thank you. Uh so for things like you know Docker and other stuff they obviously already go in and create NF table rules and stuff like that. Can you run BP filter at the same time or and not have issues? OB obviously, you know, have more than one firewall system is not great,

but are you able to use both at the same time? >> Yeah. Uh it's not going to attach to the same thing anyway. You're not going to override the IP table rule sets using BB filter, even if you attach to the same hook. Uh but you might mess up traffic anyway because you would have two ways of filtering traffic. Otherwise, it's Which one would take priority in

that situation then? >> That is a good question. So if you attach to XTP, BP filter would run first. If you attach to TC, same thing. Um, if you attach to one of the IP tables or NF table hooks that is a very good question and having a look at the kos would answer this. Uh, I don't know. I have my memory. Sorry. A little bit of

a coral here. Um, how do you deploy this like in a scale? Like do you just like send these files to to the host and then just reload it or >> how do you deploy this? >> Yeah, exactly. Like like like how do you make this like available in in production? the rule set >> give the rule sets, right? >> Yeah. You send the rule you give

it the >> So you send the files kind of CHF or >> Yeah. Yeah. For example, and and the the rule set can stay on the host and you can have like um system detection at the very uh beginning at very early boot of the system that will call BPFI and apply the rule set. It's roughly like what you would do with IP table save and IP

table whatever restore. Do you have mechanism to check that the configuration doesn't drift that whatever you're running because of the Oh, I thought you you had a question. Okay. So, yeah. So, like like how do you make sure that the things that you're sending to like like the configuration that you're trying to configure doesn't drift from what you want to write? >> So, basically what if someone

modifies the rule sets? >> Yeah. Yeah. Exactly. Like let's modify the rule set in whatever server you have you send it through CHF to the to the host but that thing never never loads right like like do you have way to check those things you can read the rule set and check what's inside um right now what you write in the rule set in the text file

is almost exactly what you read from rule set get when you call BFCLI so you could compare that the only difference is the contra value um but we have plans to like allow allow you to uh specify initial cont value in which case you could just compare it but I think that's a problem that's outside of um BP filter itself it's more like a configuration management problem

>> follow-up question um when deploying at scale obviously I'm assuming that there will be different kernel roles that will have different rule sets applied to them. how do you think like is there a central management system where you configure once and then push push the configuration out? Um >> no I I think that's outside of the scope of the tool itself right it's mostly configuration management issue.

Um as far as baby filter is concerned only the current machine exists. uh and traffic coming obviously but like it it's not aware of like a a whole cluster infra whatever uh that is a different issue that should be resolved by different tools. Um so thank you. Thank you. Uh is there planned for or an existing capability to interface with network uh cues for IDS sensors >>

for IDS sensors or IPS IPF intrusion protection systems and such? >> Uh not yet. Um what we've done the further we've been into like whole system other tools integration is uh for some time I had a patch for IP tables to use BP filter the >> which is not available anymore because it wasn't maintained uh and I think now with like a good AI you could like

translate from IP tables to VP filter if you need to uh and otherwise what we do at meta sometimes like we want to inject latency into some traffic. Um and what BP filter does is that it marks the packet and then we had TCDS catching that mark the traffic with that mark and injecting latency for testing. That's a bit as far as we go so far because

we don't have much use case. >> Hey, two quick questions. First, what were you using for your shell history? >> Uh, it's uh >> is that a toin? >> What is it? No, I >> I don't know. Uh, it's something um so it's my history is piped to FZF FDF. Oh, >> okay. Great. And then >> and it looks quite nice, right? >> Yeah. Yeah. And then

the second one is I think similar to his question. How does this interact with other BPF programs like BPF trace or psyllium? you know, can I can I load these rules before or after those? >> Uh, so with BPF trace, it shouldn't be an issue. I'm pretty sure BPF trace can't link can't um attach to network >> Can it Jordan? >> No, it can't. Okay, Jordan is

working on BPF trace. So, I'm just double checking. I'm not saying dumb things. Um and so if you have other BPF programs running, you're responsible for what could happen, right? Uh there is nothing preventing you from attaching two different TC chain. You can even do that with BP filter. You can have an existing TC program and attach another one with BP filter. No issue. Um how the

can behaves when you do that is that the first program runs first and then the second program. And if the first program drops or accept the packet, the second one won't be executed. Uh XDP, well, you can't because you can only have one program at a time. Uh for Cgroup, it's it's up. Uh sorry, >> is that a technical term? >> Uh for Cgroup, basically what it

does is that if you have two Croup program filtering packets, it's going to run both of them. If one of them return drop the packet, the packet will be dropped. But both for like if the first one drops the packet, the second one will be run anyway and the packet will be dropped. Doesn't make sense. I'm not responsible for this. It's it's weird. Anyway, um but yeah,

you're responsible for like unlink that part because we we're not aware we can't be aware of the other tool using the BPF hooks unfortunately. What kind of uh traffic logging is available? Is it just uh the the human readable or is there a machine? Uh >> what kind of sorry >> traffic logging? >> Um so logging is pretty straightforward. We have a C structure that we log

into a ring buffer and you are free to use the API to get the file descriptor of the ring buffer and you have the binary data. Uh what BFCLI does is that it reads the B binary data which is the basically the packet headers that you request and it prints then and it pretty print them but you could very well like read from the file descriptor and

and print what you want. >> Uh hello. Uh could you remind me where your BP filter places those uh EVPF hooks like at which layer like is that a nick level or is it in the kernel? >> So we we support multiple hooks. So we can attach to XDP which is at the nick before the socket buffer is allocated. We can attach to TC ingress egress. So

at the driver level after SKB allocation on ingress and egress path uh we can have croup socket buffer croup SKB so every packet coming to a croup on ingress or eress uh we have uh the BPF net filter hooks so it's uh pre-rooting postrooting forward local in local out and that's about it and I've got someone working with me working on um so uh croup sock uh

cript socket address which is filtering at the socket level. So instead of having every packet going through you would have the send message or connect or bind request directly uh which contains some network data like an IP, destination IP, source IP, the port that kind of thing and you could filter there directly but it's not there yet the the socket address. >> Got thank you. >> I

have a follow-up question. uh what is the benefit of attaching to later in the networking pipeline if you can do filtering earlier like on the nick like do you have more BP filter features available to you when you have like kernel memory allocated >> you you yeah you have sometimes more features um you can't mark the packet if you don't have a socket buffer this is a

a socket buffer field so XDP can't do that um you can't uh what like BPF uh XDP can do redirects not all of them can do. Uh there is something about like uh sending the traffic to a different nick on egress. We have that recently because we we want to test um lost packet lost traffic and so we use BP filter for that. The issue we had

is that if we drop the packet on to simulate network issue, the kernel will be aware that the packet has been dropped and the kernel won't retransmit the packet. So the kernel is smart. Um and that's not good for us. We want to have the the overhead of resending packets over and over. And so what we do is that we redirect instead of dropping we redirect the

packet to uh the loop back interface and the kernel see the packet as sent but no answer received. So the packet is missed and it returns. So basically the features you can use depends on the hook you attach to. We're good. >> Any last question? >> Going once, going twice. Oh, one more. >> Given that both uh BP filter and NF tables encode their rules as bite

code and run them in a VM in the kernel, why is it that BP filter is so much more performant than NF tables? >> Uh BP filter, BPF is not a VM. BPF sounds like a VM. It looks like a VM. It behaves like a VM. It's not a VM. Well, it's for some definition of VM. Um, so net filter has actual bite code that is run

in a VM in the kernel. Uh, BPF bite code is cheated into machine code at load time and when it runs, it runs as machine code. Uh, at least on x86 and ARM. I don't know about the other architectures but um and so it's like you would write the C code and compile it as part of the kernel and that's the main reason and one other thing

which we don't see here because I I don't have it but like there was an example oh no we can see it actually so we can see that the baseline when there is just one rule between IP tables NF tables and BP filter like we already have that small difference in in in throughput. The reason here is that we filter earlier if we attach to XTP or

TC with PP filter than if we filter on the first net filter hook. So we can have some performance increase at this location too. Uh and so the rest of it is just well we have machine code and they have bite code cut and so it's faster. Thanks. Thanks. Okay. Anybody else? Oh, another one. All >> Yeah, go for it. >> Hey, thanks for the great talk.

Um, can you say a word on how you generate the BPF? uh as in do you generate C code and then compile that with clang or do you generate the the VM or is anything reusable for other tools or what's >> so we use a big buffer of memory we allocate with malo and we write the strict BPF instruction into it one after >> okay so so

you you go from your DSL >> yeah we we we are the compiler >> yeah uh that's that's the idea so it's uh sometimes a pain but it's very fun I suggest you can try Um, and it's like you have full control over what you do. And it's actually very great because sometimes we want to cut corners, which is very hard to do with Clang because you

go from Code and you don't really control what it does. >> Uh, and it's actually very simple. It's not that complex at all. >> What we do is not extremely complex. So, we can do it ourself. Uh there are some cases where we we have BPFC source files into the repo for some features that are hard to write in assembly directly and fragile and not much used.

So we write BPFC which we compile into clang and we just steal the bite code and put it into the final program. >> Got it. Do you think there's scope for a project that um makes generation of BPF code simpler to be used across multiple projects or you think there's no point doing that as in like sharing a compiler between several projects in the future? Do you

think that would make sense? >> It seems very close to what we discussed at some point. Jordan, so Jordan V on BPF trace BPF trace goes from a DSL to a BPF program. um they do it using LLVM and they have much more like it's much wider scope than what we have in BP filter and at some point we actually discussed about like having some like of

common back end uh and so we could create many other tools with a DSL and convert it into BPF bite code. Uh I think that idea stopped quickly when we discussed with some maintainers which was apparently not that great of an ID. Uh so I mean if if there is a use case for it, sure we thought there was apparently no not everyone agrees on this. Um

so >> that's not a great answer but that's as much as I have. Anyone else? >> Okay. Thank you. >> Thanks a lot. Hey, they might be Yeah. Heat. One, two, three. One, two, three. >> Can you hear him in the back there? >> One, two, three. >> Anybody? So, >> hello. Hello. >> Yeah. Okay, that's better. Cool. Thank >> That's perfect, actually. Yeah, that's great. Okay.

>> So like that >> I think it's good. >> Yeah, it's comfortable. >> Yeah. Cool. Hello. Hello. Hi everyone. Thank you for coming to this talk. Uh my name is Andreas Beltran. Um I work for the Linux systems group at Microsoft. Uh my team works on the Azure boost project which is basically a system offered by Microsoft internally uh that offloads um virtualization processes um to dedicated

hardware including SOC's or socks on a chip. Uh we use Linux extensively for for this and obviously we run systemd um as our service manager in this custom dro we we run for Azure boost and that's what the talk is going to be about today. It's um some work I've done with the upstream systemd community to modernize and enhance the capabilities of uh local storage management for

systemd services. So this is going to be the agenda uh for this talk. I'm going to start with a brief systemd overview of what it is. Um, and I'm going to discuss some important concepts that are going to be relevant for this talk including uh the one of dynamic users in systemd and then uh local storage directories. Uh the second part of the presentation is going to

be ID map mounts. Uh some use cases for this um how uh we did the integration of ID mapping for systemd storage directories. uh the internal mechanism for IT band mounts and a quick demo and then I finalized with uh the other part of the presentation which is about these quotas I give I'll give some background about it um again the integration of these quotas for systemd

storage directories uh the mechanism used for this and then another demo uh for this section okay so let's get started so in general terms what is systemd as you may as you may already be familiar with it. It's the init system and service manager uh used by pretty much all major Linux distributions right now including Fedora, CentOS, Red Hat, Ubuntu, you name it, right? Uh system D

runs as the P1 uh which is uh the first process that starts after after boot and then it's in charge of starting and monitoring all other user space services and managing their dependencies as well. It provides very aggressive paralization capabilities upon boot because as you can imagine we want to start everything as soon as possible during early boot. It also provides ondemand activation services socket based activation

services as well. Um it has some capabil capabilities to provide network configuration during boot as well and then it manages resource control for uh for services based on Linux croups or control groups. For all of this, it exposes different configuration files. The most common one is the dotservice one, which is where you define all the uh configurations for for your system service. Uh there are other units

as well. the dots socket to define or to configure your sockets. Timer units which are often associated with the services to determine a cadence in which the service runs. The mount units and targets which basically group together u different services. Uh there are two primary tools in systemd that um will be relevant. Um the one the first one is systemctl which allows to control the start restart

stop of different services. It it also allows you to check the status of your system service and then journal ctl which provides all the logs from the Linux kernel and systemd's uh native units and any other service register with systemd as well. So it's the centralized logging uh provided by by systemd. Okay. So as seen before systemd offers me many capabilities. I'm going to focus just on

a couple ones that will be relevant for for this talk. The first important concept is the one of dynamic users with systemd. So dynamic users can be enabled uh by setting a particular directive called dynamic user and you can set it equal to one or yes to enable it. So when you do that in a service file, um systemd is going to allocate a pair of a

Unix um user and group ID when the service starts and then it's going to release it or release this pair of group user ID when the service terminates. The way systemd goes about this is by picking the first available UID um from a predefined ranged um that is I think hardcoded in systemd. This is from 61184 to 65519. And then um an important part here is that

systemd does not directly touch the ATC password file or the lives in disk. Instead it uses a gip module called NSS systemd uh which writes the new UID and G to the system um user database and group database respectively. The benefit of having a service with dynamic user is that u systemd in this way can run a privilege separate service that leaves no artifact in the system.

So um systemd is going to allocate uh the UID when the service starts and then it's going to release release the UID when it terminates in a safe and secure way uh for later use specifically uh taking care of UID recycling which I'll talk a bit more in the next slide. And the other benefit is that systemd does not to does doesn't need to pre-allocate UIDs um

since they can be allocated dynamically. So this does not drain um the available UID pool any longer than necessary. Okay. So the second important cap capability um will be the one of systemd uh these directories help define where the systemd service uh stores their data. So as you can see on the uh on the table on the right there are four types of storage directories. The first

three are persistent um storage directories including the state directory cache and logs and the last one is for temp files. So the runtime directory that's where you put any data that that is temporary and that will be cleaned up after the Um so the first three are more interesting since as I mentioned they are persistent. So this means that the data will survive from one invocation of

the service to the next and systemd creates namespace bind mounts uh for this directory. So this means that you you will have those paths uh in the middle column in the in the host name space and then systemd is going to create a bind mount when the service starts and it's going to show in the in the units mount namespace um with the same paths. Um now

an important consideration here is the sticky file ownership problem when you combine it with dynamic users as we as we saw before. This basically means that in the scenario where you have a systemd service and this service creates a persistent file. Then in Linux obviously the ownership for that file is sticky meaning that once the file is created unless you do a chain explicitly the the the

owner of the file is going to be the same. Right? So you can imagine this could be a problem when a system service starts creates a file then it terminates and then if the service is using dynamic users then in theory uh the UID could be recycled and a second service uh starting later could grab the same UID and what that's going to do is it's going

to have access to the files and the data that the previous service created which obviously is not what we want and could be a pretty uh bad like security vulnerer. vulnerability. So to take care of that, what systemd does it it creates these boundary directories which are the ones on the last column in the table. Um these are basically host directories that are made inaccessible to any

other user that is not root. So in this case any unprivileged user is not going to have access directly to to the data stored in in those private directories. Effectively this ensures that the access cannot be gained through uh the dynamic UID recycling as I as I explained before. So uh the talk is going to focus on two important features I have added to enhance this um

storage directories. The first one is the integration of ID mapping for for them and the second one is the um application of these quotas for the persistent storage directories. So that's what u I'm going to be talking about next. Okay. So, I'm going to start with ID map So, for a long time, it has usually sufficed to track a single owner and group for files and directories

in Linux. But there's been an increasing number of use cases uh for making the file ownership relative to the environment the the process is running Um so in many of these instances the UIDs and GIS for the files do not necessarily match the ones um for um do not match the current user of of those files. So for example in containers that could be the case where

you have container use running with an unprivileged UID and you might want to have different um UIDs for those files. So in in these cases the ID map mounts provide a solution for for that problem without necessarily changing the files on disk. Um in addition file ID mapping can be useful um to um uh basically for file sharing within um within the containers because you can define

strict ID mappings from one user to to the user in the container. So it becomes easier to access those files as well. And I'll talk a bit more um later in the following slides. So in general terms, ID mapping is a file system feature uh that allows a mount name space to show a different UID and G than what it's actually stored on the file. This feature

was developed by Christian Browner and is available in Linux in in Linux 5.12 plus. So it's been around for a while now. And an important consideration of the ID map mounts is that the future was not added for entire user name spaces. Instead, it's more granular and flexible in the sense that you can define an ID mapping for for each mount in your system which can be

very powerful. Okay. So next I'm going to talk a little about about the different scenarios and use cases for ID mapping. As I quickly mentioned before, the first one would be containers. Uh so usually containers uh need to provide a common root image or base image um uh that needs to be accessed within the container. Now uh in the host system user name spaces are used to

isolate to provide the isolation for the containers but still the containers should be able to access this common or base image uh with root privileges. That's um one of the use cases. So mounting the image with the ID mapping would enable these non-privileged UIDs in the containers to have privileged UIDs and be able to access those common root images within the container. In addition, file sharing also

becomes a bit easier with ID mapping because um you can you just need to define different mappings for the UIDs within each container and you don't necessarily need to change again the ownership of of the user and the group of the files within each container. You just define the mapping that you want for for each mount and it's it's done. You have access to the files you

need and in containers usually to these root images. Okay. Another important use case scenario for ID mapping is related to DAC or discretionary access control permissions when sharing files. So here's an scenario. you you can have a systemd service that um defines the storage directory which could be a state directory as we saw in the previous slides and then we combine this with dynamic users. So the

service is going to be running with an unprivileged UID and then assume that root can create a file on the host name space uh with 770 770 permissions meaning only root can access this file and any other unprivileged UID won't have access. However, this is a shared file. So within the system unit namespace, say that it's running with UAD 60,000. Um the system D service won't be

able to read or write to the file since uh the the last three bits of uh the access permissions are not set, right? Um so one way to solve this problem would be uh change the permissions in the in the host file to 777, right? to give access to all kinds of UIDs in the system and then in this way the system V unit will also have

access to the share file but this is not desirable because obviously any unprivileged UID would be able to read or write to the file so it's it's not the best solution another possibility is to change the ownership of the share file in the host name space um so that the unit the system unit can access it so in the example here we would changed the ownership of

the host file from root to 60,000 which is the UI UID of the system service system D service. So in this way we solved that problem. Um so without before ID mapping was available uh these cases were handling by doing recursive CA Jones one example was systemd homed which basically keeps consistent access of um uh user files uh in the home directory and whenever um the home

directories did not match the the ownership of the files within the home directory wouldn't match the UID of the user logged in uh systemd would have to in some cases ch C ch own the entire uh home directory to match the UID of the user. As you can imagine, this can be a very slow and inefficient operation, especially if the home directory has a lot of files

and and subdirectories. So ID mapping uh comes to the aid for this because you can basically have a fixed UID uh in the host name space mapping to whatever the user UID is um for for the systemd um uh The idea uh here was to extend the same approach of systemd homed to all the other systemd storage directories because those can also have multiple files and directories.

So, CH owning when a system dervice starts or restarts would be very expensive in some scenarios. Okay. And the last use case is related to ephemeral UIDs. Uh this is also related to the concept of dynamic users. When you have a systemd service with dynamic users, systemd can potentially recycle the same UID um after another invocation of the service. So in the example here again you have

um a file in the host name space this time with a 60,000 ownership so that the system the unit can access it and let's say that this uh systemd unit gets restarted and potentially could create it could it could uh be assigned a new UID say 65,000 on the subsequent uh restart. So by changing the UID of the system the unit now is running with 65,000 then

it loses access to the file again because again the the uh the files have sticky ownership so it's still going to be 60,000 and the system unit won't have access anymore to as shown before I mean this can be solved with either uh by either changing the permissions of the file in the host to make it accessible by everyone meaning writable permissions or directly ch owning uh

uh the file in the host name space. But um neither of those options are usually efficient or optimal. So ID mapping is a pretty nice way to resolve that problem as well and would take care of these scenarios where you need to when a systemd unit gets restarted with a different UID. Okay. So this brings me to the actual um implementation of ID mapping for systemd search

directories. So in general terms the feature is when you define a system a systemd service with dynamic user set to yes and then you combine it with any other storage directory and if you have ID mapped um enabled in the kernel then system is going to create an ID mapping from the nobody user which is five uh 65534 to the dynamic UID of the service for each

mounted directory. This feature was implemented in systemd version 257. So it's available starting in in with that version. And then there were some security considerations um that uh were important when implementing this future. The the first one uh was uh the choice of the fixed UID in the host. So uh there were some discussion to either use the nobody user or the root user uh as the

fixed UID in the host name space. We went with a nobody user just to minimize the possibility of any root exploit or any root uh escalation privilege escalation. Um since using root would mean that you know the unprivileged users would get access to the row root own i nodes. So that that was uh why we made the decision of using the nobody user as the fixed UID

in the host. And then obviously we are turning off the set UID for the ID map mount and we're also mounting those with the no exec. So there are no binaries that can be executed within uh the storage Uh the second thing was backward compatibility for uh systemd services that ser systemd services that already created uh storage directories on a previous systemd version. In those cases uh

we continue to do the ch own for those pre-existing directories. um since u I mean the ID mapping would apply to newly created directories. So what's the internal mechanism to apply the ID mappings uh for for these directories? Um setting up ID map mounts basically involved the creation of if you will a dummy username space that would contain the ID mapping tables. So for each directory mount

we would spawn a new user name space and in the UID map and J file and G map files of the user name space that's where we would define the the actual mapping for the mount. So as you can see in the example here um on the UID map table we define a mapping from the nobody user which is the first column of of the of the

of the entry to the UID of the system B service which in this case would be 60,000 for ease of understanding and the third column is just the range of UIDs that you want to map in this case would be just one. The same goes for the G map file. Once we have the ID mapping tables ready within this dummy user namespace, then we need to associate

this name space with the actual mount. And for this we use the mount set atr sys call which um has different fields. Two of them being the attribute set and attribute clear. So to actually enable ID mapping on the mount, we just need to set the mount atr ID oops ID map um attribute uh as part of the attribute set uh field and then we just need

to point the usern FD u entry to the to the file descriptor of the user name space that we have created and in this way we will have the mapping ready for the mount. Cool. So next I'm going to show a pre-recorded demo uh demo that I have um recorded to kind of show how ID mapping can be used with state directories. Here I have an Ubuntu

machine with a systemd version that does not support ID mapping yet. First I will create a sample system D service which sends a ping every minute with dynamic users enabled a sample state directory and I'll start the unit. We can check that the unit got assigned a dynamic user ID. Right now the UID is 63750. Looking at the save directory on the host we can see it

is owned by the If we end into the view of the service, we can see that the same UID has been assigned there as well. Let's create a file from the services view using nenter again and we can confirm that the same dynamic UID shows Now let's modify the service to also use dynamic users. But let's specify a different UID 65,000 which could happen on any unit

we start since dynamic users implies the use of And again let's check the new service UID. Right now it is 65,000 as we defined before. If we list the files under the state directory and the host, all were changed to ownership to the new UID using recursive th. This would be very inefficient if the directory had a lot of files and subdirectories. Now here I have a

cameo instance running a recent systemd version and a Linux kernel that supports ID map mounts. Let's create the same sample unit first with any dynamic UID. Let's check the user ID assigned to the unit. It is 63750. But now if we look at the user ID in the host name space, it's the nobody user or as expected with ID mapping. Now let's send sensor to the units

view to check a few things. We can first see that the bind mounted path has now ID mapped enabled. Let's check the user ID of the directory to make sure it's the services UID. Here we can see the note by a user in the host as seen before was mapped to the services dynamic UID 63750. Let's create a file in the services data directory. It has a

dynamic user ID. Now let's go back to the host name space and check the UID of the new file. It's the nobody user as expected. Finally, let's restart the unit using a different UID 65,000 just as we did We can see that the files still have the nobody user in the host. So, the recursive C own was not needed anymore. Let's confirm that the files within the

services view are owned by the new and they are. This demo shows how ID map mounts are now used for system state directories, but the same applies to other storage directory types. Thank you. Okay, cool. So yeah, that was a quick demo here. We we could see how ID mapping was enabled for the state and it provides a way to again avoid these recursived chones that systemd

was using before this feature was implemented uh with the storage Okay. So the second part of the presentation would be about this quotas. So to give some background about it, as the name suggests, this quota is just a mechanism for limiting the amount of disk space that a user or group can use on a file system. Um, as you could imagine, without these limits, a user could

easily fill up the the entire machine's disk and would that would cause problems for any other user in the system. Uh, currently most modern uh, file systems already support these quotas. This includes you know ext34 ex3 ext3 ext4 xfs and btrfs as well. There are different types of these quotas. The most common ones would be user and group quotas to assign limits to individual users or groups

within a system. Uh the one that we're mostly interested in for this talk would be the one of project quotas which is the third one right there. Project quotas are basically directory tree quotas meaning that the limits are applied to a given directory and all the files and subdirectories below it. Um these type of quotas are only available or supported right now in the xfs and ext4

file systems. Okay. So these quotas for the systemd storage directories u as I mentioned before uh this feature relies on the availability of project quotas. So, uh, this would only work with, uh, systemd services that run on an ext4 or XFS file system. And the idea here is to be able to, um, have these limits enforced by different systemd services and also on a per directory type

basis. So you could potentially define different quotas for different directory types. You can have one limit for storage directories, another one for cache, etc. This feature is available starting in systemd version and the quotas are defined uh for the number of this block these blocks and al optionally the number of iodes. Uh there are two type of limits that can be established for these quotas either hard

or soft limits. Uh with the hard limits obviously the limit cannot be cannot be exceeded with soft limits they can but then warnings are issued when you're going above the limit. Uh there are two main requirements to be able to um start enforcing these quotas for system services. The first one would be to enable the project quota on your file system. This can be accomplished with the

tune tofs utility and the second one would be to actually turn on the quotas on the system again with a quota on utility. Okay. So in system D we've made the distin the distinction between enforcement and accounting directives. Uh the first group of directives would actually enforce the limits. So there are three one for each directory type. We have the state directory quota cache and logs and

runtime here. The runtime directories here is not included because tempest does not currently support project ids. Um you can define either an absolute size limit or a percentage value for these directives. Uh if you pass an absolute limit then only the the number of blocks the limit for the number of blocks is set. If you pass a percentage value then the same percent quota is applied to

both the number of blocks and iodes um for the disk. As I mentioned before only the hard limits are set in this case because we actually want to enforce those limits. The second category of directives is just for the accounting. So the if you establish or if you define these directives in your service then the u limits are not actually enforced. System this is going to track

the units uh disk usage without actually enforcing those limits and the usage will be shown as part of the s if you do a system ctl status of the unit then um the the usage of the unit will will show there for Okay, so the main mechanism to apply quotas uh for systemd service was again using project ids. These are basically unsigned 32-bit integers. Um and systemd

defines a predefined range for this. Um that that has been reserved uh like a pool of project IDs that we take from when defining a new search directory. Uh the project ids are unique per service and per directory type. So each file and directory that belongs to the same systemd service and the same directory type either state cache or logs are going to have the same project

ID. And this is how we actually um aggregate you know all the usage for uh the different directory types and uh keep track of that and enforce the The project IDs are set recursively using one of the extended attributes uh on the file system CXT4 for example which is the FSX project ID and then this is set using the anio. Um and the project ids are only

released if neither enforcement nor accounting is needed anymore uh for for the service. The second part or the second mechanism to enforce this these quotas is using the quota ctlfd ciscoll which allows to set the actual limits um for the for the associated project ID. So as you can see on the right um the quota ctl um sys call depends on this truck which is the dqbk

which um makes avail available different fields. The ones that are more relevant for for this is the first one, the DQBB hard limit, which allows to set the the limit for the number of blocks and then the DQB IAR limit to set the uh the limit for the number of iodes. Okay, so this is the second demo which again I'll be showing how we can define these

quotas for a state directory. I have a chem instance with a systemd version that supports these quotas and a Linux kernel with a quota module installed. I also have a virtual disc partition of about 500 megs with an ext4 I'm going to do some initial setup to enable quotas on the disk. First I will enable project quotas on the ext4 file system. I'll use the tune tofs

utility And we can confirm that project quotas were enabled. Next, I'm going to mount the disk and move /bar lip to it since I'll be demoing using systemd state directory. The last setup step would be to turn on project quotas for the mount point with Now I'm going to create a sample system surface with two state directories called sample jar and sample jar 2. We're going to

enable quota accounting for state directories and set an absolute quota limit of 10 minutes max. Let's check the status of the service. Since we enabled accounting, we will see the current usage and absolute limit in the output. We can see that the current usage is about 8K and the max limit 10 Let's take a look at the extended attributes of the data directories to show the project

IDs assigned to them. We can use the rep quarter utility to see a more detailed description. Here we see that for the assigned project ID we have a hard limit for the number of test blocks which after taking the block size into account is equal to our tagme Now let's create files using an center and see if the storage limits are enforced. First we will create a

5 megabyte file in one of the directories using the d utility. We can see that the file creation went through and that the project ids were applied recursively. Now let's create another file of 20 megabytes to go above the 10meg limit. This time we got an error message saying that the quota was exceeded confirming they were being enforced for the sample The limit can also be defined

in terms of percentages in which case the quot is applied to both number of blocks and iodes. Let's change the limit to 10% and restart the service. If we check the status, we can see that the limit is now around 47 meg, which is about 10% of the total dis space. Checking rep again, we will see that the same project ID was reused after the restart since

the directories already existed. Since we use percentages now, there's also a hard limit for the number of files or items. We could also just keep quota counting but not enforce the limits anymore by commenting out the state directory quota directive. Let's restart the service with this In this case, we can see that the project ID is still active, but there are no hard limits set. If we

do a system ctl status, it will report just the usage without the limit. Finally, we can also remove the account interactive to stop tracking this usage for the unit. In this case, the project ID has been released since there is neither tracking nor enforcement of this OS. This demo showed how these quotas can be used for system date directories but the same applies to cache and log

directories as well. Thank you. Okay, cool. So yeah, that was the demo um using um showing how you can use the different directives that were added uh for system storage directories. Um okay so that pretty much finalizes my talk. Some conclusions about this would be that the first one ID map mounts provide a a safe way for system storage directories uh to avoid doing uh any recursive

th or even changing the ownership or peri permissions of files on disk when uh systemd services starts or restarts which again could be a very slow and inefficient operation um as systemd was doing with homed for example uh in in in the past before ID mapping was enabled in the in the Linux exter kernel. Um for systemd service, this would only apply in units that define and

then the second part obviously is the one of project quotas that are used to enforce and track this limits for uh different systemd services. Uh the different different quotas can be set for different storage directory types as we saw. So you can you could have a pair service per type directory type quota and this is only available if your system is running on an ext4 or xfs

file system which are the ones are pretty common today. So you should be able to create your service and establish the limits that you want uh for for your service. Okay. So yeah that pretty much concludes the talk. These are some references that will be available once the slides are uploaded. So, thank you. Yeah, if there are any questions, I'm happy to to take any. Hey, is

there a security concern if you had a legacy service that is actually running as nobody, it could write the files that are owned by the um systemd managed service? >> Yeah. So the question if is if there were any security concerns when using the nobody user um as the fix UID. So that was one of the design decisions that was discussed with the upstream maintainers. Um nobody

seems safe enough given the fact that systemd already has those uh boundary directories for um systemd services that run with that run with dynamic users. So in that sense service uh the service manager doesn't just randomly create uh files on systemd uh units um and any other UID won't be able to wouldn't be able to access those directories um under private. So in that sense uh the

nobody users seemed like the best option to for for the fixed UID on the host. Now the other one was obviously root which is a little bit more uh we were more skeptical skeptical using root because there could be cases where you could have uh root privilege escalations and things like that. So uh yeah I mean the with the maintainers basically we decided to go with nobody

seemed like the safest option for the fixed UID. Um yeah, I think given the fact that we're also mounting with without set UIDs um set for the mount uh the no exag and all the other um options for the mount. I think it would be it would be pretty safe to to just have it that way uh in the host. >> Any other questions? >> Thank you.

Thank you for for coming. Hello. Hello. Good. Okay. I tell I talk loudly normally, so I feel like I should be okay. Hopefully that won't be me. Cool. Thank you. Say it again. >> Oh, Jordan Ro. See, it's close enough. Oh, really? Do you think Yakov is still watching? It's too late. Oh, yeah. Yeah. Sure, but not yet. All right, almost there. Give people a minute and

we'll start. All right, I think it's about 6:15. I think we can get started. Thank you guys for sticking it out to the last talk of the day at the kernel low-level systems track. Nothing says Saturday night like a talk on BPF. So, try and keep it interesting for you. Um so yeah the talk is u about BPF trace learning to be a language. If you don't

know about BPF or BPF trace that's okay. Um we'll do some primers as we get I guess first things first uh who am I? Uh I'm a software engineer at Meta. I've been there for about eight years. Um but you know in the kindness of their hearts they allow me to work on BPF trace as a maintainer which is an open source project. And um usually for

ice breakers, you know, they do the two truths and a lie. So, you know, at the end of the talk, we'll I'll reveal which one is true. Um but yeah, read that and try and figure out which one is the lie. So, this talk is a journey talk. We're going to talk about the things that we've done this year to sort of get BPF traced to where

it is. Uh I'm also going to be advertising some of the new features and cool things that we've been working on. Um but we're also going to talk about the challenges that BPF trace, you know, has had over the years, what things it was trying to solve and how that's evolved. Uh some of the pitfalls and issues that users have had and how we're trying to really

like move it toward being like a real like you know domain specific language. It's not a general purpose language, but like we're trying to make it more expressive and you know have it like act more like a language. So first of all, what is BPF trace? So it is a CLI tool and kind of a language that allows you to run um Linux ebpf programs on your

Linux systems. Um it promotes or does like live and fast tracing of the Linux and uh kernel and user space. And it's a language that's primarily inspired by like O C and Drace. And as I mentioned, it's open Um I think a lot of you were here for Quinton's talk about uh BP filter where he talked about what BPF is. So I'm going to sort of graze

over this a little bit, but like if you haven't heard of BPF, it is a Linux kernel technology. And as Brendan Greg describes it, it's like putting JavaScript into the Linux kernel. You get to write these little programs, inject them into the Linux kernel at runtime, and then they just do amazing stuff. So like where BP filter was doing networking, BPF trace is more on that third

rung down where it's doing observability. And BPF trace has been around for quite a long time, at least, you know, in tech and sort of BPF years, right? It's been around for about nine years. This was our old BPF trace logo. was a nice pink pony but got rid of it unfortunately. Um, and it was created a long like way before like other BPF or the sort

of standardized BPF um, features were around like uh, BTF uh, libid BPF Kore and you really don't need to know what those things are but you know it's sort of consideration for you know what it's been developing and how it's been developing over years and just sort of like to advertise that it's it's a little bit old and it's had a lot of different active maintainers. Um,

I don't know. I want to say like maybe I was like the 10th or 12th maintainer like through the course of BPF trace's history. Some have come come and left and come back. Um, and a lot of different features have been added to BPF trace over the years. Uh, some with like really narrow scopes for what they were trying to accomplish and I'll talk a little bit

more about that uh later in So, what problem was BPF trace like when it was created? What was it trying to solve? And it really depends on who you So, you know, Brendan might say that it was like the ultimate systems observability tool. You know, Alexi might say it like abstracts the complicated parts of BPF into a simple DSL or like it's, you know, gives kernel engineers

superpowers. And just so you know, none of these are real people. They didn't say any of this stuff. Um, but what problems does BPF trace solve? And the list is like a bit long. I'm not going to read all of it, but I'll go through some of the high levels. And it really tries to answer like tough questions about your system, especially when your system is in

peril. Like it's nice to have a tool that can like get you running up and fast so you can do diagnostics and observability on that system really quickly and have it be flexible. Another problem is that like raw BPF code has a lot of boilerplate. Uh if you've ever written raw libid BPF before, you know there's like a kernel component which is the BPF. C file. There's

a user space component that can be written in like you know C++, Python, um you know, I think Rust, you know, now and on top of that like BPF verifier errors are also very difficult to understand and debug if you've ever written raw BPF programs before. So that's like a couple of things that BPF trace is like currently solving for And just to give a quick video

demo of like what it looks like to run BPF trace Let me just play my little video here. You have to invoke it with pseudo um because you need uh those privileges to run BPF. And here we are running a BPF trace script. The thing is tracking new PIDs per second on my system. So this is just printing out, you know, all those new PIDs per second

and there killing the script. So that's all. It's very easy to get started and running up or get get up and running really quickly. This is what a lot of BPF trace scripts look like. It is this simple. like they are known for oneliners and we'll sort of like go over the anatomy of this script but what this particular script does is it counts page faults and

it groups them by thread name. So again if I were to run this you know BPF tray script on the CLI like this would be the output I would get. It would say attaching a single probe and then when you kill the script it would you would see this aggregate sort of map of thread name and you know the software faults per thread name. So pretty simple

the thing on the far left. So if you've ever worked with a or drace before like this should look familiar but like the thing on the left is the probe. So this is the uh thing that BPF trace is attaching to in the kernel. In this case, it is a software probe. So it's attaching to software page faults and it's doing it at the rate of one.

So for every page fault, it runs this program. If that were to be like software page faults 100, it would be one in every hundred page faults where it would trigger this BPF program to run. And the meat of the program is basically just a map. We are defining a map by that at token there and then the key for that map is comm which is a

built-in in BPF trace that returns the thread name. So we are sort of creating a map with keys of thread names and then we're using this another BPF trace builtin called count where basically every time this function is hit it will like just add one to that counter. So like you can go back and this slide makes a little more sense now that you know like how

that And by the way, I mean it's pretty small group. So if there's like burning questions or stuff that you really don't understand like Oh yeah, go ahead. >> Uh so the question is why isn't there a parenthesis where? >> Oh um it's more like syntactic sugar. So com doesn't take any parameters. You actually can call comm with parentheses in BPF trace. >> Yeah, in BPF trace

builtins like can you can do parenthesis or not like for built-ins that don't have any arguments like just the raw name works in the same way. Yeah. Um, strangely enough though count you can't leave out. You have to do the parenthesis. So I'll get to some of the weirdness in BPF trace later, but yes, good question. U so on top of the things that BBF trace can

solve it can also do a lot right like so it's a general systems observability tool it can do performance deep dive system debugging user space program tracing you can collect stacks you can symbolicate those stacks uh we have all sorts of neat CLI visualizations for BPF trace uh histograms someone recently added even though it's sort of in the unstable feature uh it'll be released Uh, no, I

think it's out now. There's like this cool time series feature where you can see how like values change over time as the CLI goes and prints. It's really neat. Yeah, I should have a demo of that at some point. The funny thing is though, even though BPF trace can do a lot, not a lot is written in BPF trace. So, a lot of problems end up growing

out of it. And even the experts like you know the scripts that we keep in the BPF trace repo itself don't really exceed 200 lines of code. And the largest script we found uh on GitHub was uh somebody implemented Tetris actually in BPF trace which was really cool. And I think there's a version of snake out there also. Um, and on GitHub, we did a little bit

of searching and there's really only like 10,000 lines of unique scripts or 10,000 lines of code for unique scripts of BPF trace on GitHub. So, like what's the problem? Like, why aren't people writing longer BPF trace scripts? Well, if you read Brendan Greg's book, and I'm not sure if you have, it's something that was published many years ago. Like one of the things he sort of advertises

BPF trace for is its oneliners. Like as I showed you before, we have even like a oneliners tutorial, but like that's one of the things that's known for like these tur programs that give you system information. And we probably have like 20 of these or something in the repo. Um so maybe that's one explanation why we don't have like really long BPF trace scripts in problem is

is that it's really just easy to hit a wall in BPF trace. So, you know, users may ask, can I factor out common code that's in multiple probes in BPF trace? Well, not really. Can I use a different type of BPF map? No. Can I access different fields here if the architecture is ARM 64? No. Uh, what about defining my own strct? What about calling this other

Kfunct? It's exactly what I need. and then they just end up leaving. And that's kind of the story. And truthfully, it's easy to hit a wall if you're a developer or you're a contributor to BPF trace. So, let's go through a little story of trying to add a simple feature in BPF trace. So, we had this BPF trace script before where it was aggregating on the thread

name. Let's instead aggregate on this croup ID instead. Okay, so there is a BPF helper that already exists in the kernel called BPF get current croup ID. Like that's all we need. Like how hard can this really be? Well, here's a light diagram of how BPF trace works. So, similar to a compiler, we take a raw string, we lex it, we parse it, we turn it into

an a. There's like a ton of like passes that happen in that little like, you know, error there where I said lots of stuff is happening. We turn it into or there's like an LVM codegen layer where we take our a and the types that we've generated from BPF trace and turn it into LOVM which looks like this. And then we have clang, you know, compile it.

we turn into BPF bite code and that's what gets injected into the verifier. So again somewhat similar to BP filter we're not hand rolling assembly but if you've worked with LVMIR before it's not that much nicer. It's still very complicated. You still have to think about loads and stores. You still have to think about like memory on the stack like alignment things like it's it's also a

pain to work with. So again our feature we are trying to add croup ID to BPF trace. Okay. So just a few simple steps right. Um we need to add the croup ID built-in token to the lexer and parser. We need to add unit test for it. We need to uh add code for how to resolve the type of this new built-in. We need to add unit

test for it. Oh we need to make sure the type interacts properly with other types. Right? We need to make sure it's handled in this thing called the resource analyzer which determines like if we need to save memory in certain parts of BPF trace. Need to add a unit test for that. We need to add possible feature checks. What if the BPF helper doesn't exist in your

kernel version? Well, we should tell the user about it before they try and load it and the kernel verifier rejects it. That would be bad. Add unit test for that. And then we want to make checks to make sure it works for different probe types. Okay, we'll add unit test for that. Then we had to have the code gen for it where we actually call the BPF

helper, add unit test for that, add it to the documentation. Then we add the end to end test for that. It's painful, right? Like this is awful. Like this is just a little snippet of all the different places in BPF trace codebase where you'd have to like incorporate this new feature. And that's the problem, right? Every single feature in BPF trace is a language feature. Like it's

scattered all throughout the codebase. You have to write so many tests for it. It is it is a nightmare if you're a new contributor. And this is the reason why a lot of developers basically just don't contribute to BPF trace. I mean LVM IR being at the end of it is also just a real kicker and people don't want to deal with that. And truthfully, it's not

just a problem with adding new features. Like the BBF trace language itself doesn't really compose all that well. Um, type inference is a little bit sketchy. We have these sort of like internal weird types for some things that don't, you know, work with other things as they you think they should. Uh, there's no way, like I mentioned, to reduce code duplication. There's tons of tech debt all

over the codebase. There's no extensibility. And one of the really sad parts is that BPF trace is trailing behind upstream BPF features, right? So there's new features that get landed in the kernel all the time for BPF like arenas, there's new kunks and unless you go through that awful process which I just, you know, talked about, you can't use it in BPF trace, right? And that's just

sad. And you know I think users want like we're seeing a little bit of trend up here right in the graph of this is time and this is lines of code and I think people want to write more BPF trace maybe because of you know some of those problems I stated lib BPF is hard or raw BPF is hard. So, how does BPF trace go from a

language of oneliners to like being like, you know, like a more proper language, something that doesn't get in your way, that's somewhat expressive and, you know, allows you to do the things that you want to do when you're doing writing observability programs. So, of course, you have to fix everything, right? And that's what we've been doing over the past year. So let's talk about some of those

features that we've added to fix all these things. So first and foremost, we wanted to fix the problem of like a lacking of the ability to reduce code duplication. So the simplest thing we decided to add was macros. So here is a BPF trace macro and it's fairly simple, right? So you define a macro above your BPF trace probes and then you can just call it and

like pass parameters to it and this expands before we actually do the code gen into like you know we just modify the a and inject that macro code into where it's called. Simple right here's an actual BPF trace macro that is actually in the codebase right now. So this is just a little bit more complicated example to give you an idea of the things that you can

do once you have macros. So we have like variables that are checking things and it's a lot more complicated. But one of the nice things about BPF trace and the addition of these macros is that even if you write complex macros, you can inspect what they look like. So if you type out uh your BPF trace script and it may end up something complicated, you're not sure

how macros are getting like injected, you can add this -d command to BPF trace and it'll spit out the whole nicely formatted as if you wrote it yourself. BPF trace a with all the macros expanded. Yeah. Yeah. Yeah. And they're a little >> Sorry. Oh, can you just repeat? >> Oh, yeah. Sorry. So, the question was these are not like uh C macros that just do string

substitution. They're more like list macros and where we're actually modifying the and the answer is yes. Um they're also a little bit smarter about that. So we took this idea of hygienic macros from Rust to prevent accidental mutation of variables in maps. So these need to be explicitly passed in to BPF trace macros in order to modify them in the macro body. So that sort of prevents

users from like shooting themselves in the foot or doing you know other weird stuff. So the next thing we wanted to fix in BPF trace was code organization. So up until recently there was no ability to like separate your BPF trace code if let's say you had and there really wasn't a reason to right like there wasn't uh macros before but now that you have macros like

why not the ability to import other BPF trace scripts that have macros that you can call as if it were a library simple right so little import statement that's the library file you're importing the add one macro easy the other thing we wanted to do is fix our sketchy types. And we did this with BTF. So, if you're not familiar with BTF, that's sort of like the

type language that BPF speaks. It literally stands for BPF type format. And uh I mentioned that BPF trace wasn't around uh or BPF trace was around before BTF really became a big thing, but now we are like pretty much just adopting it like wholeheartedly. Um, and it's we're also using like a constraintbased solving mechanism for BPF trace. I won't get into that too much. I'm not much

of a type theory person myself. This is more you me learning about it, but BPF trace is statically typed, but it is inferred. So, let's look at this uh bit of BPF trace code. So um if you set one to the scratch variable a this is a unsigned um of uh eight bits and then that's same for negative 1 except that's a signed integer and then you

have one minus one that's also a u in 8. So does anyone have a guess what the type of d is? >> or what size? So that's a good guess and maybe that's what it should be. Uh in BPF trace right now that D is an unsigned 64-bit integer and the reason we're doing that is because right now BPF trace is not smart enough to know how

often you call a binary operation. Like if this was inside a loop, right? Like it may easily exceed uh 16 bits, right? Or something like that. So, we're basically just upsizing it to the largest possible integer that BPF trace supports. But the point here is it's it's semi non-intuitive and we were trying to make BPF trace usable without having users like type out the types themselves. So,

we're trying to be a little flexible with how this works. Uh yeah, so as I mentioned, BBFJ is statically typed but inferred. Uh we have our own like bespoke internal type system called size type which I won't get into too much but it's you know basically we're just trying to wrap this whole thing around BTF so it's all much more standardized and yeah as I mentioned uh

getting a proper like type solving algorithm so it all works The other thing we really wanted to do and we've spent a lot of time this year is fixing the lack of extensibility in BPF trace and we're doing that with what we call C interrupt. So as I showed the import statement before where you importing another BPF trace script here you are importing a BPF. C file.

So this is a raw BPF.c file. It gets imported and you can call it from your BPF trace So this one's not too exciting. But here is one that actually lives in the BPF trace codebase now that implements a feature in using BPF.C that gets imported. Right? So we have some like externs of the function just in case these aren't defined in your kernel version. You get

an error message that's handled. We do things like built-in mem copy and BPF task release. These built-ins don't exist in the BPF trace codebase. They are just in this BPF. C file and it works because BPF trace will compile this and link it in and as long as the verifier is somewhat happy with this like it works with BPF trace script. So this fixes extensibility, right? You

no longer have to wait on BPF trace to ship a feature to use something in BPF upstream, right? like there's no more like blockers for like kunks that uh you know you want to use, right? It's an escape hatch for more complex logic that BPF trace doesn't uh support. Oh, >> So when you import a BPF C file, does it just like insert it or are you

or is it compiling it into byte code that is then linked with >> the second thing you said. So the question was uh do we insert the BPF.C code into like maybe the a uh no the answer is we BPF trace compiles it and just links it in at the end. Um and again like as I mentioned like some of the developer pains right like you don't

have to hand roll LVM IR anymore you can just write like familiar semifamiliar BPF.C C code. And this isn't really for the common use case of people like wanting a simple like DSL for writing BPF, but it really like just opens this door for allowing BPF trace to keep up with the BPF ecosystem and make developers who may be familiar with writing BPF. to have, like I

said, an escape hatch for continuing to use BPF trace and what it provides, but not like falling behind or not being able to use a feature that they want. And some compiler things that we've sort of started to adopt this year. Um so we're trying to fix composability with um these two new keywords or well sorry just one keyword comp time and then we're also using uh

type introspection and we'll talk a little bit about that. So like I mentioned we're doing some like new not new but new to BPF trace old compiler techniques like branch pruning and constant folding. So let me go back to um my type example. So the reason uh C variable is a Uint 8 is because these are two integer literals that get folded early in the pipeline. So

if you look at the a like we print out what the a looks like through each one of our like you know compiler passes you'll see it eventually ends up like c equals zero because we're able to determine at compile time that that will always be zero and that's why it's a 8 instead of a64 but so yeah that's an example some compile time folding that bpf

trace has started to implement. Let's look at comp time. Uh this is a keyword that we stole from uh Zigg I believe. Um but you know it's very similar to const expert but it's basically saying that the expression to the right of comp time has to be resolvable um to something at you know a literal basically at compile time. So in this case uh this is a

macro called signal and what this macro does is it sends a signal to uh a particular process uh the ppf trace is uh tracing and so what we're doing is we're inspecting whether or not the probe type where signal is called is a k probe and if it's not we fail so fail is a compile time error. So if we basically realize in the a that we're

going to hit this, we don't even compile it to LVM or try and load the program. We fail immediately and be like, okay, you can't use signal if your probe is on a cape probe. And just to give, you know, again, really straightforward example of how this works. So here's someone trying to signal within within an interval probe. It sort of gets expanded. where interval gets replaced

where probe type was and then you know the compile time folding happens we see that it's true that interval is not a k probe and then you know we prune down to failserts in >> yes exactly yeah so age-old compiler stuff that you know BPF trace is starting to implement you know hopefully to the benefit of our users and uh let's talk about oh let me back

I went forward a little bit to illustrate why else comp time is um important. So here is a comp time branch right and this is basically saying like if I'm on PPC 64 architecture like I want to access uh the user symbol at this register right and if not I want to take that register. So look like this BPF trace script now works on x86 and PPC

64. You don't need two scripts. You can just do a comp time expression and then when you run this script on your command line depending on the architecture it'll just do the right Um let's talk about some of the type introspection functions that we have for BPF trace. So uh one is called type info and this is uh still a little bit experimental but it allows us

to do cool things with comp time and I'll show that later. So here's some pretty simple examples where we have a macro uh that wraps type info and you pass an expression to is string and it basically just checks whether or not uh you know this type info which returns a tupole I believe um is a string type. So this allows like all sorts of type introspection

in ppf trace. We have another one called is integer. Here's the signal macro a little bit expanded. So we had the test for the probe type at the top which we saw. But we want to do different things inside of our signal macro depending on the type of the expression passed in. Right? So here is again another comp time expression where we check if the expression is

a string. If it is we call this other function where we take that string and we turn it into an actual number. So we basically just have like a big map of like you know signal names to numbers that we do the conversion and if it's just a raw integer we'll take that and then if it's something else we got the nice like static assert fail that

it needs to be one or the other and yeah again here's that architecture example for comp time. this also really empowers developers um to do a lot more with the BPF trace language and a lot less with the BPF trace codebase, right? Like as you start to like expose like type introspection functions and things like comp time and the ability to call out to other BPF. programs

like they don't have to touch as much in the BPF trace codebase like you are more like starting to write what looks like a standard library and that's the thing that we're currently developing and there's you'll see if you look in like the the git history for BPF trace there's a lot of things that have been moved out of the BPF trace codebase into standard library functions

that are written in pure BPF trace it's really Um, and you know, users can create their own libraries and start to import them. And again, it really just unlocks folks that want to use BPF trace and the latest BPF features and don't have to like go through the trouble of developing it or waiting for a BPF trace release, right? what problem is BBF trace trying to solve

now? Well, I think we really want to try and stick to like our core principle of being like easy and flexible like system but more so we want to stay out of the developer way. We want to like promote composability extensibility. We want them, you know, to not get confused or just abandon it if they see weird behavior in BPF trace. And you know maybe like BPF

trace is starting to become a bit of a tool chain for BPF programs because currently I think like you know if you write a raw BPF program you need the BPF. C file you need a user space component that needs to be compiled. You need to have something to compile your BPF program right with BPF tool. Like it's it's a long process. So but if you write

it in BPF trace even write your BPF.C C code as long as you know you're okay with just like printing things to the command line or exporting like data as like JSON or something like BPF trace can get you there most of the way and we're actually in talks right now to sort of figure out how we can extend the user space part of BPF trace. So

you can write your own like user space functions that can take your data from your BPF program and do different things with it rather than just like printing it to the command line for example. you know truthfully there are still some like issues with this with like writing having users write their own BPF. C programs like it sort of exposes verifier errors potentially to the user which

is BPF trace is really trying to you know kind of mask that for users. They don't want them to have to deal with it. So that kind of opens them up to that a little bit. Um but we think it's worth it in the long run more or And yeah, we just want BBF trace to like be, you know, work across different kernel versions and architectures without

you having to consider too much about how that all fits together. Um, BPF is about to release uh version 0.25 right now. It's been around for nine years. We've never had a BPF trace 1.0. Um, and I think by, you know, by now from the talk, you can sort of see why, right? Like there was a lot of bespoke features that was sort of like tacked on

to BPF trace for different use cases. Like it still felt like a toolbox of different things rather than like a more cohesive language and unit for you to like write BPF programs or observability programs with. Well, we're sort of biting the bullet this year and we are going to try and launch a BPF trace 1.0 by the end of the year with like these basic principles of

having a simplified core language and internals. We want to make it robust and predictable. We want to move a lot of functionality to a standard library and like I said have it be user extensible and we want to have a language contract that you know we can semi-support forever. There's a lot of breaking changes that go into every BPF trace release whether we like it or not

and that's just because stuff is old and it doesn't work well together and we're trying to change that. So look out for BPF trace 1.0 this Uh we launched a website maybe I think it was earlier last year but we're doing a lot to improve it. Uh I think it's just like bpfrace.org I want to say. Um, yeah, I'll I'll find out that. I think it's at

the end. Um, but it's got a lot of like Um, like our standard libraries in there now, the CLI, the language itself. Um, there's lots of like learning tools of like tutorials. And we recently added a BPF trace playground. This is still experimental, but you can now like works kind of like a ripple, right? Online. You can like type in little BPF trace programs. it runs in

some far away server on the cloud some way and returns you the result. So that's kind of nifty. We're still working on this though. Um yeah, here are the resources for BPF trace. So bpfrace.org uh the main repo is there. Um called that the playground and we switched from um I think matrix IRC to discord recently. Uh so most of the conversation is happening there and that

link is available on the main repo website and that's all I have Um, hey, great talk. Thank you. Um, oh I forgot. No, no. Um, oh yes. Can you talk a little bit about interoperability with other languages? You say like it's on the talks, right? But or like whatever user space you have in your in your mind, right? But it's great, but reading the print statement out

of to to do real work, it's it's a little bit curs. So, >> yeah. >> What what do you guys have in mind? We're not sure yet. Uh I mean it's going to probably have to be something pretty like simple to start because we don't want to have to support a lot of different like user space tool chains. you know, probably something like akin to C or

something or like or maybe it'll be like a BPF trace langu like it'll be a function that's like written in actual BPF trace code itself, but it'll be for like user space output. It's very early stages for this. That's why I don't have a ton of like information for you. Um, but yeah. Yeah, we're working on Thank you very much for the talk and for BPF trace.

I use it a lot. It's it's >> Great deer. >> Um, okay. So, I have a question actually about BPF trace. Not the new stuff you talked about, but some of the kind of existing things. >> Uh, so mostly I use it for with probes for user space stuff. >> And a lot of times I want to access like elements or something something that requires dwarf parsing.

>> And it looks like some of that was in progress. O not quite sure where it is. So uh so I'd like to know about you looking at elements of structures and structures of structures structures and also about using dwarf to do stacken winding like does do any of those work or is there plans? >> Oh yeah yeah yeah we got a few things in progress there.

Okay. So, uh the question was um basically like walking user space um structures with BPF trace, right? Uh possibly with dwarf. So, this is a very very challenging problem. Um we had tried to support basic like container structures for C++ for a little while. I think we were I can't remember what library we were importing. Um, but now we use like libdw for some of our dwarf

stuff. Um, we're hoping that like with the ability to import bpf.c code yourself that you'd be able to like some like walk those structures in C rather than in BPF trace. That's not necessarily like a solution, but it might be a little easier to like do like pointer chasing and some weird stuff that you need to do and see that BPF trace maybe doesn't support in the

syntax. Um, there is somebody currently working uh and for your second question, there's currently somebody working on a uh dwarfbased stack unwinding uh in BPF trace. Um, it's in review right now. Now we're sort of trying to figure out like what to do with it because it's it's a bit complicated. You have to basically tell BPF trace what uh program on your system you want to like

unwind in real time and it has to like go walk the dwarf. It takes a bunch of data from that dwarf and it sticks it into like a BPF map which then BPF trace accesses at runtime to do the stack walking. Um, but I think that we agree that that feature is going to get in soonish. So be on the lookout for that. And is that because

that your programs are compiled without frame pointers? >> Yeah. And that library doesn't have frame pointers. Yes. Yeah. I hear you. >> Yeah. Yeah. Yeah. Yeah. So yeah, uh tunish. Yes. >> Anyone else? >> So I I I thought the the previous question was going to be the question I I was going to be asking, but it's uh it's in a similar vein. Um, and it's uh

it's it's been a little while since I've I've used BPF trace. So, so maybe this is actually a really uh silly question. Um, but I I vaguely remember when working through like when walking trying to walk through a strct in the kernel. >> Um, that was uh that was especially tricky and and maybe I was just missing something. I was maybe using an older version or I

was just maybe unaware of it. But I I I vaguely remember having to either like manually in like include the definition of the strructs or trying to include files and that was really really painful and it was like okay now I need to you know do a lot of work to really build this kind of small script that's going to be working with the strruct in the

kernel and yeah has has that gotten any better or is there any plans to do that or was I just doing something really boneheaded? >> No no you were right that was really not great uh for a long time. So I mentioned we were like leaning into BPF BTF completely. BTF more or less solves that. So as long as like those kernel structures are available in BTF,

you no longer need to import those header files or define your own version of the strct in your BPFJ script. You should just be able to like you know like cast like that argument to like a particular type and we should be able to resolve it in BTF and just know like all the fields and you be able to access and like drill down for that structure.

So hopefully that should just work now and if it doesn't please file an issue but like that's sort of how it should be working. >> Okay. So just make sure that the version of BPF trace that I'm using has been compiled with BTF support and then >> I think I think that should be in BPF trace 24 which is our last release but we're about to do

a new release uh for 25 and that's definitely in there but it should be in 242. >> Okay. Awesome. Thank you. Oh, um just reminded me that you were asking about um dwarf support. We're also going to land a feature in BPF trace that allows you to um specify the source line and uh column for a probe. Uh and you can attach to that as long as

like it's a valid instruction uh using like lib DW for like doing that parsing. If you do perf probe shell, it will renumber the line numbers like starting from the beginning of the function which is kind of weird. >> Wait, what? Sorry. >> Um, so right now you can do that sort of thing with Perf. >> Oh, okay. >> Right. But it it does this weird thing.

So I'm not saying you should do it, but just a comment. It does a weird thing where it will renumber the fun the line numbers starting from the beginning of the function. >> So that that that's how you do that with Perf. But I don't know. >> Oh yeah. No, I don't think I mean we have some like you know uh for the person who's doing the

the PR it has some end to end tests where like the actual program being traced like it has the line number and like it seems to be working like Yeah. Yeah. >> Good. Thank you. No worries. right. Thank you everyone for attending and staying late. Appreciate it.