Lightning Talk: ExecuTorch on Microcontrollers: Deploying PyTorch To... RJ Ascani & Matthias Cremon

This talk covers Executive Torch, a framework developed by Meta for deploying PyTorch models on microcontrollers and resource-constrained devices. The speakers explain how Executive Torch facilitates on-device inference without requiring a conversion to other frameworks, enabling users to stay within the PyTorch ecosystem. They highlight the importance of Microcontrollers and TinyML for applications requiring low power, cost efficiency, and local compute capabilities, especially in environments with limited connectivity. Key technologies discussed include ARM Cortex-M processors, DSPs, and custom quantization techniques that reduce model sizes, thereby optimizing performance. The session also touches on runtime execution aspects, memory management, and partnerships with companies like Arm and NXP for effective integration into embedded systems. Overall, the discussion emphasizes the growing relevance of TinyML and opportunities for deploying machine learning on small devices.

Hello everyone. I'm RJ Skanning from Meta's Executive Torch team and this is Matthias Kramm from Reality Labs. Today we're going to talk about Executive Torch on microcontrollers. We're bringing PyTorch models to the most resource-constrained devices you can deploy on. For anyone new to Executive Torch, it is PyTorch's on-device inference framework. The core value proposition is that you stay in Torch all the way down. You train your

model in Torch, you export it with Torch export, you compile, optimize, and serialize it into a file with Executive Torch tooling, and the uses that file to perform inference at the edge. There's no framework conversion step, there's no leaving the ecosystem. We currently support over 12 hardware backends today including XNNPACK, CUDA, Metal, Vulcan, others. That breath matters because the same export and compile workflow can target very

different hardware from GPUs all the way down to microcontrollers with a few hundred kilobytes of RAM. So why microcontrollers? Why TinyML? First, it's worth noting that TinyML is not a new field. People have been running ML on microcontrollers for several years now. But with all the focus on GenAI lately, TinyML hasn't gotten much attention. There are a few things that make ML on microcontrollers pretty compelling. The

three main ones to me are power, cost, and connectivity. Um Microcontrollers enable ambient always-on sensing at milliwatt power budgets. We're talking about devices that can run for months or even years on a small battery. Think wearables or always-on sensor hubs in mobile phones that handle wake word detection while keeping the application processor asleep. I personally So I've been an embedded software engineer for a long time, and

uh last several years it really feels like most of my work uh is dedicated towards keeping Android APs powered off. The next big reason is cost. Uh even low-end application processors cost several times more than a high-performance MCU. Uh simpler MCUs can can be even cheaper. Uh at those prices, you can fit intelligence into products that would never have room for an application processor. The last one

is uh low connectivity. A lot of the world doesn't have reliable cloud access. Microcontrollers let you bring intelligence to remote markets and disconnected environments. The model runs entirely on device, uh no cloud dependency. For many applications, that's a hard requirement. The primary compute platforms fall into two categories, MCUs and DSPs on one side and micro NPUs on the other. For MCUs and DSPs, we're primarily talking about

ARM Cortex-M processors and Cadence HiFi and Vision DSPs. Those are kind of the workhorses of the embedded world. On the MPU side, we're talking about really microcontroller-class accelerators. They're much smaller than uh what you would find on a phone or a laptop. They sit alongside the MCU with a milliwatt power envelope. We currently support ARM Ethos-U and NXP Neutron, but there's a whole host of others as

well. The key constraints across all of this, these devices typically have like less than 10 megabytes of RAM, uh and their compute ranges from single-digit giga ops to a few tops at most. Um and again, uh power is always in milliwatts, and here every kilobyte of memory matters. Um now, let's look at how Executor actually targets this hardware. >> So, one of the main things about Executor

is that it does a lot of things ahead of time. Um what that means for us is that we get uh high level of standardization, which is very important for us, but also full on customizability. So, in terms of the tools that we're using, first tool we use is torch.xport. It's a graph tracer. It's the same graph tracer as torch.compile uses. And that basically allows us to

get a graph from any PyTorch model staying in the same Then we work on edge devices. Quantization is not an optional step. We use tools from torch.ao, also part of the PyTorch ecosystem to quantize our graphs. Important to note, we can quantize with a custom quantization parameter and scheme all the way down to every single node in the graph. And then we have that graph. We also

run a lot of optimization on that graph. Export IR or EXIR is our way to do this. So, it enables us to run any common optimization passes and patterns you might want to run on those graphs. Think about fusion, decomposition, reordering, all of the things that can make your graph better suited for your backend. So, at the end of the day, what you're left with is a

highly customizable ahead-of-time workflow, and that allows you to squeeze the performance of those very constrained devices ahead of time before you send things to the runtime. >> On the runtime side, the the core runtime is OS agnostic and cross-platform. And we have a hard requirement to keep it under 50 kilobytes. It runs on bare metal, RTOSes, Linux, whatever your environment is. There's no system dependencies whatsoever. No

malloc, no file system access, no threads. You provide whatever allocator and data sources that make sense for your platform. Memory planning is hierarchical and done entirely ahead of time, but it has a big impact on runtime performance. These devices typically have different types of memory, tight tightly coupled memory, on-device SRAM, and occasionally external DRAM. And you want to place tensors in the right memory for performance. The

executor memory planner allows you to place activation tensors in the into specific hardware memory segments. Builds can be selective at both the kernel and D-type level. You only link in the kernels that your model actually uses and only the data type specializations that you need. So, if your model only uses int eight, you're not carrying FP32 kernel code. And lastly, constant weight tensors are read directly from

flash with zero copy, no allocating a buffer, or copying weights into RAM at startup. Let me walk through some quick deployment tips for for deploying to MCUs. As Mithi has mentioned, quantization is pretty much non-optional. going to int eight alone reduces the model size by 4x and all of our MCU back end support it. Additionally, using quantized inputs and quantized outputs can be really useful as well.

This avoids float to int conversions at the inference boundary. A lot of MCUs, maybe not a lot, but a lot of the lower end MCUs don't have hardware FPUs and that conversion cost can be non-trivial in both performance and binary size. Uh using the selective ops model flag allows you to point your build system at the model file and it analyzes your model and strips out kernel

code that your model doesn't use. Additionally, you can configure the max kernel number to set the size of the kernel registry. The default supports up to 2,000 kernels and most of these models only need a few. Um so, you can save over 20 KB of RAM. There's a few other size optimization flags as well. Turning off logging saves 50 KB. Um turning off program verification uh will

save another 20 KB, but we really only recommend doing that if you're bundling the model with the the firmware and you're verifying them together. And lastly, enable the D-type selective build option when using the portable kernel library. This prunes unneeded data type support from them. Let's look at how that all adds up in practice. >> So, this slide is summarizing a few of the models we run

for for this talk. This is definitely not meant to be an exhaustive list. This is just to give you give you a little bit of the breadth of things you can run on those small embedded devices. In terms of the the classes of models, there is audio processing, there's CV processing, some classification models. So, quite a lot of things that people run and again, you have to

remember this is running as close as possible to the sensors, the microphones, the cameras. It's at the lowest power you can do and also typically at the lowest clock. So, running all of those models that is usually a huge power win. One more thing I wanted to mention in the flash usage, that's the model size. You have a few things you can expect like tens of kilobytes

for those small devices. But, they can grow to 100 to hundreds of kilobytes and then even around 10 megabytes for ResNet or Conformer. If your backend has access to DRAM or more memory, you can run even a lot more than this and the the executor framework will just scale without any issues on the on the size on the size front. Thank you. So, in terms of partners

and integration, this has definitely be a a major group effort. We've worked directly with Arm including through Alif with NXP and with Cadence among others, but they have been key partners. They've been major contributors and also very active contributors. So, we want to thank them for their contributions. On the meta side, Reality Labs has announced previously publicly that executor is used on a bunch of our devices.

It's used in Ray-Ban Meta glasses for live translation. It's used for Meta Vanguard for some garment related performance tracking. So, we've deployed them for a while and RJ will add a couple things about software support. >> On the software integration side, we recently landed Zephyr our toss support in the executor tree. So now any Zephyr based project can be can pick up executor as an external module

through the West build system. That's really important because Zephyr has become the de facto our toss for the embedded industry. And I want to also thank our partners at arm for their work on that. Very quickly because I'm running out of time. The looking ahead the next things we're focusing on are expanding hardware support and deepening SDK integrations. I think I have to end it there. Sorry.

You can check out the slide. We uploaded that to the site.