Lightning Talk: Flexible Deployment of PyTorch Models on MCU-Class... Robert Kalmar & Martin Pavella

In this session, Robert Calmar discusses the flexible deployment of PyTorch models on microcontroller (MCU) class devices using the Execute Touch framework. He describes the evolution of microcontrollers from simple architectures to complex heterogeneous compute platforms that integrate CPUs, DSPs, and NPUs for efficient AI processing. The speaker showcases portable kernels in C++ that enable the execution of edge operations and how various backends, including Cortex-M and Cadence DSPs, improve performance when running AI models. He emphasizes the importance of using transformation passes and recipes to optimize model deployment across different hardware backends. Calmar also presents benchmarks demonstrating significant speed improvements achieved by leveraging these techniques, ultimately facilitating optimized inference times and power consumption in AI applications on edge devices.

Thank you for joining this session. I'm Robert Calmar, principal machine learning software engineer at NXP and with my colleague Martin Pawel we would like to speak about the flexible deployment of PyTorch model on MCU class devices using the execute touch frame. So this speech will be mainly focusing microcontrollers. So first let's introduce how a typical modern microcontroller look like. So back in time when the micro control

controller consist of single CPU core with memory and peripheral the advance in the semiconductor manufacturing processes allowed us to evolve the microcontroller into still low power relatively constrained but heterogeneous compute platform integrating multiple various on a single chip. So typically on today's modern microcontroller chip you can find CPU cores for general purpose compute then DSP cores with wide send instructions to carry on a processing workload. And

then with the demand for the AI on edge devices neural processing units are also becoming integral part of NPU's. So let's let's have a look what execute touch offers and how the execute touch can support efficient efficient deployment of a AI models on microcontroller class devices. So as a starting point we have portable kernels which is a pure C++ implementation of edge operators. The advantage is they

can compile with they can be compiled with arbitrary C. They have decent performance and that's pretty much. Uh then executor offers also um core specific backends currently for the Cortex-M and uh Cadence uh various Cadence DSPs. Both they are leveraging uh neural highly optimized neural network libraries from either ARM that CMC uh NN leap to accelerate the NN workload on Cortex-M or Cadence And last but not

least, executor also uh offers uh backends for NPUs. As I already mentioned, uh at this moment there are two MCUs with respect to microcontrollers. This uh backend for the Ethos-U uh uh NPU and EIQU NPU. Let's uh take these uh backends uh and portable kernels on a small benchmark. So, we conducted a benchmark on i.MX RT700 running on 325 MHz. Uh the example model was a MobileNetV2

with 64 by 64 and what we saw uh that there is a significant performance improvement uh between co-portable kernels and either Cortex-M or the uh Cadence backend. On the Cortex-M core, it was 30 33 times speed up uh on the Cadence back for DSP, uh it was more than a 100 times speed up going from portable kernels to the core specific uh backend. one more point to

the uh HiFi 4 the Cadence backend. Essentially, the model with portable kernels doesn't even fit Tensilica RT 1700. So, we essentially need to use smaller CNN to get this data. But back to the MobileNet V2, when we moved from the Cortex-M or Cadence backend to Neutron backend, we experienced even bigger speed up. On the Cortex-M backend, it was roughly more than 400 time from between Cortex-M and

the Neutron. On HiFive4, HiFive4 is by nature already a SIMD instruction, so there was a speed up only four times, only four times. It's still significant if you are carrying a not memory, power consumption. So, how we can smartly use also the CPU in executor for working with Neutron or other hardware accelerator. Okay, so now that we have performed the the benchmarks of our model, we know

which backend we want to use, and that's the fastest one, obviously. And usually, when you select the backend and you lower your model using executor, you'll end up with something like this, where at the heart of our model there's a call delegate node. Since the majority of the model, and it calls the NPU directly. In our case, it calls the Neutron NPU. But around this node, we

have some other leftover nodes, which represent operators not supported by by our backend. In our case, it is the quantize and dequantize nodes. And these nodes run run C++ implementation, which is really slow. And from our benchmarks, you can see that the dark blue, the two operators account for more than half of the total inference time. So, we have chosen Neutron because it is the fastest overall.

But, can't we use some other backend also for to speed them up? Well, of course we can. For example, the Cortex-M backend provides transformation passes, which can be applied to this model, and they will just replace these two operators with Cortex-M optimized variants, which are 2.5 times faster. And uh just running these simple passes grants us a 32% increase. that sounds great. Can we turn it into

some general solution? Uh ExecuTorch provides a functionality called recipes. Uh recipes basically allow allow us to specify transformation passes, quantizers, partitioners, and so on, and uh the order in which they are applied based on the preferred We can then use the standard Torch export functionality. We provide our uh PyTorch model. Uh we give it the recipe, and it directly produces the lowered ExecuTorch program according to the

backend requirements. And what's beautiful is that these recipes can be combined together. So, for example, we can select our primary backend and our fallback backend, such as Cortex-M. We can combine the recipes together, and then the export function will first apply the transform passes of both backends. It will then annotate the model sequentially with both backends quantizers. It will run the Neutron partitioner to to replace replace

most of the delegate call, and then it will run the Cortex-M passes to replace the leftover operators with Cortex-M variants. So, that sounds pretty much exactly like what we wanted, right? A generic solution to combine two backends There is one small catch, and that is that uh both of the backends they're they have some operator specific requirements related to, for example, quantization, uh, where they require a

specific scale or or zero point for a given operator. Or, for example, they require some eight-bit operators to be preserved into the Edge dialect. And, uh, when these two require that's when we have a problem with our recipe based approach. So, for example, as I mentioned, in the combined recipe export, uh, the quantizers annotate the model sequentially. So, most of the operators are actually annotated twice, once

by each quantizer. But, then Torch AO these annotations where it's possible, but when they are conflicting, they cannot be merged. So, these these some of these operators will be annotated twice, and they will be quantized twice. So, there will be an extra quantize and dequantize pair of operators inserted, which basically perform unnecessary computation and drastically slow down in So, if we really want to squeeze out the

most out of our model, we first need to identify which operators will be handled by which backend before we do any lowering at all. Then, with this information, we quantize every operator only using one quantizer, uh, and we choose the quantizer based on handling that operator. And, also, we do not preserve, uh, eight-bit operators into the Edge dialect globally, but rather individually. For every single operator, we

know which backend it will end up in, and based on that, we'll decide whether to preserve it into the Edge dialect or to have it change be changed for an Edge Edge or And, as a proof of concept, at NXP, we have a working implementation, uh, which utilizes these principles, and it combines the Neutron and Cortex-M backend. It identifies which nodes are supported by which backend, and

then it lowers them according to the backend's requirements to achieve the optimal result from the perspective of restrictions. So, and as an demonstration how efficient an NPU can be in terms of model processing and with respect to power consumption. So, we were tasked by deploying an audio model and achieve 3 millisecond inference time for single frame within 25 milliwatts. So, first when we deploy the model on

our with our Neutron backend, we still had some CPU operations lefted and also the memory format know for us to channel last was conducted by CPU. We started with 6.5 milliseconds, so not even reaching the the requirements. But once we delegated the memory format change to Neutron, it saved us 2 milliseconds on CPU only 100 microseconds on NPU, so it showed that we are on the right

path. So, we continued, we delegated all the remaining operation operations to see to from CPU to NPU ending up with NPU only inference. We also did some optimization in the Neutron kernel implementation played with the power distribution on the RT 700. Eventually, we reached the inference time 1.4 millisecond what essentially gave gave us quite a nice margin to select the core frequency from the perspective of uh

power consumption which was 192 MHz, uh 2.4 ms inference time, 22 mW power I have one one moment. Okay, thank you. So, uh during this this talk, we mentioned the eIQ Neutron several times uh in in context of uh microcontrollers. Uh just to know the note scalable. So, we have as an NXP the Neutron NPU uh deployed on small microcontrollers like the RT 100 what I mentioned.

Uh and it scales up to i.MX uh application processors uh offering providing uh two tera ops power. And with the latest acquisition of Kinara, we extended our portfolio with discrete NPUs. Uh if you want to uh hear more about Neutron ARA and and uh software enablement related to that, I would like to invite the speech uh every millisecond counts tomorrow uh provided by my colleague Pavel Mazanauer.

Thank you very much, and if you have any questions, feel free to contact us or ask right now. Yep. You want to I mean, you don't have to join as well, but I was thinking So, the question is what was the motivation to use executor? Torch and Relay AI. >> Okay. So, if it was a requirement to use executor or we could have used torch MLIR or

VLM. So, it was for microcontrollers, so the VLM was out of the game. With the torch MLIR, the executor was the requirement from a customer, so yes. Torch MLIR From our perspective and from our experience, it is not as terms of uh pipe Yeah, how to say? Uh the backends are more mature and more and the community is more vigilant in the executors from our perspective. So,

that's essentially was the choice why we are moving with executor.