PyTorch on RISC-V: From Cross-Compilation To Native CI - Ludovic Henry, Meta

This talk explores the integration of PyTorch with RISC-V, highlighting the challenges and collaboration needed to make this possible. The speaker discusses how both PyTorch, an open-source machine learning framework, and RISC-V, an open standard instruction set architecture, present unique opportunities for innovation when combined. Although PyTorch currently does not run natively on RISC-V due to incomplete work across various libraries and tools, the speaker emphasizes that efforts are underway to bridge this gap. Various community collaborations among tech giants and open-source projects are essential for creating the necessary ecosystem. The speaker also notes how the journey involves integrating critical dependencies such as OpenBLAS, SciPy, and Linux, pointing out significant progress made over the past few years. Ultimately, the talk conveys optimism that with continued effort, RISC-V can support AI and ML workloads effectively.

Thank you very much for coming. Um this talk is about PyTorch and Ris 5. Um I believe the two the two communities have have a lot to share. Uh PyTorch is open source. I think it's one of its big big thanks and why there's so much attention adoption about it. Uh Ris 5 is an open standard and open ISA believe that these two really make it um

like like working well very well together. However, PyTorch does not run on risk 5 today. Um it's not so much because someone decided, okay, this is not going to happen, right? But this is mostly because the work hasn't been done entirely yet. Um and that's lot of work, right? Pyon is just the tip of the iceberg as we're going to see after. Uh there's a lot of

things that need to work in order for PyTorch to work like open blast, sci 1DNN, Linux, GCC, and like a lot of libraries. And every single one of these projects um and communities needs to engage and needs to work on risk factor. So um and even if you get that to build then you still need uh CI and you still need to be able to build and

test and distribute and ship and everything PyTorch on CI and this is one of the topic uh of this talk and even if you have CI you still need hardware and that's also a big uh where do you start? How do you how what's the journey that brought us there? So some of the things that we are going to cover is uh the approach we have taken

to fix it like the pytor community as a whole the race five communities hardware vendors and everyone what we learned uh what works what doesn't work spoiler alert unfortunately I'm not here to say everything is everything is shipped right it's not there yet uh there's still some work to be done but let's walk through the step make it compile test it and ship it so hello everyone

I'm ludic I'm based in Paris Um I by day I'm a software engineer at Meta. I joined a few months ago through an acquisition. I'm a co-chair of the technical tech technical streaming committee sorry of rise. But what is rise? Rise sole goal is to enable open source software and risk 5. Um there's a whole bunch of companies like you can you recognize some names here like

Google Canonical Qualcomm Nvidia Meta um and other hardware companies like Andy's reverse that I used to be part of Sci-Fi MediaTek and all of these companies are getting together and made the realization that they can have the best hardware in the world if there's no software to run on it. it's just that's just not going to sell, right? And so all these companies are coming together to

make sure that the software ecosystem as available on R 5 and is working well and that's why we have for example Ubuntu 2404 on R 5. We have Red Hat working on that. We have uh many of the software being enabled. Where does Pych fit in that? um we really identified it as um flagship project in the sense if it runs well on risk 5 then uh

it means that risk 5 is ready for a IML workloads as you probably all know AI is eating the world nowadays and so if a hardware manufacturer today does not make AI something they're not going to sell investors are not going to invest is so so freaking important to that whole ecosystem system. You may ask what is RI five right? Uh so it's a number is by

show of hands anyone knows what risk five is or okay perfect so I'm not going to delve too much into the details but it's a royalty-free open source community anyone can just go contribute participate to the discussions part participate to the spec um the spec in a way is just a document that outlines if you implement this hardware that respect the spec then any software that is

written for this specification is just going to That's basically what uh an um the beauty of it is that anyone can participate to it. It does not belong to a company like X6 like with Intel on MD does not belong to to another company like ARM for example. Um it's you have a foundation called reserve international that takes care of it and that kind of owns that

but it's a foundation and not for profofit and just go to GitHub download the PDF implement a chip no licenses nothing like this. So it makes it uh very uh enticing for hardware renders. There's already billions of uh chips being shipped. For example, you have so many new phones. iPhones have them, Androids have them like sound processors, image processors, a lot of because then you don't have

to pay licenses to ARM. So that's make it cheaper. It really goes from embedded data centers. So you have chips which are like barely a square a millimeter square uh that are running with five ARM as well of course. But it goes all the way to data centers where you have today deployments of resize servers in the clouds like it really ranges like the spec is made

to to answer really any in that. Why should you care? Um I think ris is going to uh eat the world in because it's open so anyone can go and make a chip for that. Um, even you can I've heard people doing tape out for like a few hundred dollars and then they wait a few months and they have it shipped back home and they made it

right. They worked with the fab they worked with all of that. You have uh literally open source CPUs on GitHub. Um, you go there, you have VHDN or Vlog just download that and you make a chip, right? You can test it on FPGA and you wait a few months and you have a chip back, right? That's quite amazing. Um there's also a huge diversity of implementation and

competition thanks to that just because again anyone can do it right so it's pretty easy and it's a natural fit for open source and a IML because if someone identify that clothes would benefit from a special instruction they can do that it's much harder to do with R do x6 in ris 5 that's where the openness nature of the of the spec comes in and any vendor

any individuals can go into it hardware today uh if you if anyone is familiar with scaleway it's like a French cloud service provider uh they have reside machines you could just go to their web interface you say I want a new machine you have a machine SSH into it just Ubuntu or Debian or Federa and it just works so that's available today how do we make it

compile So pytorch VM exeutor is probably what everyone is very happy to talk about today and yesterday and during this conferences right but that's just the tip of the iceberg there's a lot of things under right for you probably familiar with numpy scypi onyx matine a few other python packages right on native libraries open blast leaf 1DNN many others under that of course there's compilers there is

uh os linux distributions the kernel firmware and harder, right? All of that needs to work. If any of these pieces don't work, PyTorch is not going to. So, this is really the journey that we have been going through over the past two years to enable all of that and make sure that it works. So, on the really foundation on the open source and tool chain and firmware

as well. uh we worked with uh Alma Linux and Rocky Linux which are two uh Linux distribution derivative of Red Hat which are the based of many Linux is where every Python packages is kind of built um and so Alma uh announced R 5 in March. Um Rock Linux announced it back in May. So they officially support R 5 and so many Linux was made available in

July of last year based Um, and so that means that since last July, people can actually build Python projects on their own with the upstream tooling for five. That also of course depends on GCC and LVM. There has been years of contributions to to these projects. The support like it supports fairly fast. It's not as good as XA6 and ARM, right? But it's it's getting there. Rice

has invested a lot in that. We worked very closely with Alma and Rocky to make sure that they have the resources, the hardware, the guides, the knowledge and everything. Uh working of course contributing to GCC and LVM to make sure that everything is running. Um AI is all about matrix multiplications. Um so open blast uh one of the main libraries on CPUs. We did contribute RV support.

RV stands for risk five vector extension. That's basically the SIMD the hardware acceleration support for for for math for matric sliff which is a vectorzed math library so basically you want to do a cosine tangent exponential divide whatever that's doing it but vectorized so faster um and when then so open and leaf are working on res today they're available they're shipped they are really when DNN is

not there yet it's working but it's not optimized And when DNN is one of these um improvement over open blast for example on everything at mo they have some approaches which are just more optimized they support more types like FP8 FP16 BF16 that don't are not initially supporting open blast due to API or likes. So one yearn is still in the pipeline it's a performance improvement it's

not a functional requirement and of course python the python packaging ecosystem every packages that pytor depends on uh all of this needs to be available all of them need to ship a res to work or in order for upstream pytor to work without modifications and yeah that's that's been a long journey um so happy to say that some packages are already available. So the the it works

right. So today you can do uh docker run platform Linux ris 5 um on your Mac even on your x6 machine is going to run on ris 5 um and you can do a pip install materine right here and you're going it's going to download the many Linux ris 5 binary right so that's mantine built by upstream distributed by upstream on ris 5 so there is already

some which are like that numpai is not yet we're working with them to to make sure they have uh everything they needs but it's it's on the way right it's coming and that's of course uh not mentioning the hundreds of other projects that is just some logos right uh but new vm rocky unmat that I mentioned uh rust python lama cpp feder like can anything anyone depends

on needs to be ported and that's a lot of work so where are we today um that's for Monday right I kind of got it work on kind of great working it little star up right but uh a binary build native so ris five machine for five uh on GitHub actions uh so you can see that it's around 258 megs uh wheel uh I did download it

verify it works it does work again that's on native right that's not on emulation that's not on cu that's native but the testing yeah don't don't ask questions uh it's still stuck on the kind of early thing so it's not even getting to the mostly a configuration issue but that's for Monday right so I haven't had time with yesterday and today to really dig so how do

we test it like how do we do the real thing so for that no CI means no official support we cannot expect PyTorch upstream community to release something they haven't tested right so rise is providing free runners um that basically just like GitHub runners uh it's running on bare metal it's a firmer and ARM just install the app here and then you can do runs on um

24045 I've been using it for quite a few projects lacbp is using them uh ambian is using them cube is using them like it's basically popping up a bit here and there we're at like three 400 builds a day or something like this so it some adoption what's next for that right now the hardware is not great. Um the next gen is going to be RVA23. Uh

for those familiar RVA23 is a profile which means that it has a certain set of extensions for W 5 among which vector which is kind of the most important one for performance and hypervisor which is al also very overarching goal. We want to match GitHub burner's experience. Uh it should just work. Setup python should work. Setup go should work. Cash should work. The base image should look

and feel the same. uh it should just like you have your pipelines on x6 on ARM you just add Ubuntu 24045 it should just work right in the meantime you're kind of doing white gloves um integration with for flagship project pytor is one of them working with executor which I I met yesterday some of the people just want to make sure everything works communities lacbp and a

few others are if they work we expect that more people are going to adopt it and of sweet um that I expect to to um to um engage with contributors exist any contributor who is interested I want to work with them contractors potentially as well just to make sure that work is happening but yeah overall we just need to work through the test failures figure out what's

bugs fighting issues upstream the patches generally speaking uh rise and the ris ecosystem as a whole is not interested in folks we want everything upstream because we don't want to be maintaining things like on the side. I think that's been a learning from the um ARM ecosystem as a whole of maintaining things on the side just doesn't scale. Ideally, we want the upstreams to just do it

and so we are really targeting that as first target and yeah the goal is green CI on 5. It will start with labels as in you will have to label P request to say okay I want to test that on five. Uh next step so testing every commit on main that will be a pretty big increase in loads and of course the uh northstar is have that

on every pull request. We are not there yet. Uh currently the the build takes around 10 hours on the native machines we have. Not good. We we clearly cannot um and so we basically will need to it will take a few years but we'll get Better hardware is coming. the one that's coming like in a months or two should be two to three times faster. So hopefully

10 hours to like two three hours still not acceptable for requests but it's going to come and then yeah of course to ship it the big question right is upstream shipping it is it shipping from a fork ideally it's upstream right so we I'm working with the LF links foundation and pytor team to understand how we can handle that like what do we need what kind of

performance uh how good the machines needs to be if do They need to be in AWS. Can they be in some other cloud providers? We have to figure that out, right? And again, the goal is upstream, not a fork. Hardware acceleration. Uh I mentioned that before, right? But it's it's just like it's good, but it's not great. It's far from perfect. Um there is an open blast.

Um there's no micro architecture tuning yet. So for example, if you look at open blast, they have kernel's implementation for every generation of Intel CPUs, for every generation of AMD CPUs like AMD goes and contribute that because it's so performance sensitive that the order of instruction how much performance you can extract from the hardware and so they go and just like crazy on that. Race 5 has

uh three implementation, one for vector length 128, 256, 512. There's no other macro architecture tuning. That's not good, right? It's good enough to start, but clearly vendors are going to need to go and optimize cases, right? 1DNN mentioned it before. Uh I think it's really what can get us from like 80 to 90% in terms of performance by having um just jitted kernels for the specific shapes

of the matrix you need to multiply, but also different types. FP32 is great but I mean 16 FP8 FP4 intate info that's not something that open to support. So when DNN needs to to basically come into play and just support all of this uh dealing types and execut um great presentations yesterday about that um that's really the PyTorch for edge and embedded um risk 5 has a

pretty strong presence already in embedded um on the right for example it's it's a Chinese company doing robotics Um it's there's going to be a lot everywhere and so that's where Ris 5 really has a place to shine. Um unfortunately it's not there yet, right? So we need to basically make it work. It's really an important frontier for the for the R5 ecosystem and for the AI

ecosystem as well. That's where I believe there's going to be a lot of compute happening, not only on clouds, but where things are happening. And that's where yeah, the that's what we're trying to make work, right? The virtual cycles. How do you make sure that everyone cares about risk 5? How do you make sure that if you're trying to do use risk 5 just works? And so

we need to port project 5 to have broader ecosystem to have more users using risk hardware vendors making rises to investing more to have better hardware making it easier to build and test and so there's more risk and everything right so that's where rise is trying to step in around here to say well this harder for free just install the stuff it's free and then you can

start doing things right investing money investing time investing energy uh into making that a reality. So, how can you help? If you're a maintainer of a libraries, if you're users of a library, port it like try to enable C on it. Try to enable testing, make sure it works, right? Try to do a release on it. If it shipped, people will know. Feel free to use the

wrist five runners. Uh it's free. Just install the app and you have access to it. And if you can contribute, um go to a library you're using saying, "Hey, I care about W five support. Uh hey here's V five support um tests bug reports anything can do right so to me it's really not question of when uh it is going to happen I hope that 2026 is

going to be a strong year because of the hardware coming in like better hardware coming in and uh distributions being available for example 2604 is going to be only on RV23 that to um keep the momentum going and keep the demand strong. Any questions? >> Yeah, go for it. >> Yeah. Mhm. Yep. So there is work on ML making sure that um so LVM works on res

5 it is able to target res 5 is working quite well there is I don't know that there is everything in pytorch so triton included that works with the integration everything so I I don't know triton specifically the building blocks under are available like lvm is available for example I don't know if triton specifically is able to generate code that is risfy specific or not If there

is x6 specific code so needs to be tested, needs to be figured out, needs to be fixed. Yeah. Yeah, they have I mean 64 uh back end um and it's open source. So I believe they came through the acquisition of uh there was a company in Scotland which was called one API or something and they have been pretty open about open source in that um we did

contribute support to Wendal used by sci so one is working on it today um and we did not get much uh push back right so we are going to try with one DNN um and we're going to see yeah sorry Yes. >> Yes. So that's what we do today. But of course there's no testing and um this is I believe that in order for the platform to

be viable people expect native compilation in a sense uh first of all the python build system as a whole not just pytor right it's not very friendly with cross compilation uh I've tried requires some patches here and there uh setup tools does not really it's like it's it's a bit of a hack right so as soon as you get into numpy dependency and everything it gets funky.

So you can it kind of works but it's not what people expect and I expect it's going to reach the quality that we expect. Um and that's where yeah we'd love to have native compilation if we can as much as we can but yeah that's until we have really performance hardware that can compile in like 20 30 minutes like arm 664 x86 uh we are not going

to be able to enable it on main on p requests that's just not going to happen until then Go ahead. >> Mhm. >> Uh yes, but there's components to Python which are not uh which are native and not Triton. For example, uh all the A10 libraries like the back end to that it's all native, right? Uh it moves to sleeve, open blast, all of this. But even

sliff is a um is it a header library? No, it's not a header library. Uh but to sliff there's for example headers that define okay what is the size of a vector and everything. So pytorch has the lip.so or lip_cp.so So that is name and even if you use torch compa and everything it still depends on that and even the thing that invokes LVM I believe is

is native right and is part of like lip so there is always going to be a bit of binary it's not actually on the hot path that's unfortunate but yes that's where executors for example comes into play right exeutors allows you to you take a graph of operators you do torch to export it's going to export the graph execute torch is able to take that and generate

a native cross compile or not and that you can say yeah generate for 5 and the result of this executor uh execute launch will run on ris 5 yes so you can do all your development training and everything on x86 export the graph export the compared to a native res. That's where the edge kind of comes in >> because uh that's how executor that is what exeutors

enables. It allows you to run on a platform and target another is still targeting the same platform. Even if you do compile, it's for the current machine, right? I I don't know enough. That's I think that's why I also I don't know if any other questions. Thank you very much.