Lightning Talk: Every Millisecond Counts: The Fine-tuning Journey of an Ultra-Eff... Pavel Macenauer

This talk discusses the integration of the PyTorch ecosystem with NXP chips, which range from microcontrollers to more advanced microprocessors with external accelerators. The speaker outlines the varied devices supported, including small MCUs and higher-end automotive processors equipped with ARM Cortex-A cores and additional accelerators. A key focus is the eIQ Neutron accelerator, designed to enhance performance for machine learning tasks such as anomaly detection and object detection. The speaker elaborates on the use of ONNX as an intermediate format for model deployment, enabling easier conversions from PyTorch and compatibility with systems that utilize MATLAB. Additionally, the speaker highlights the importance of quantization and the development of open-source tools to facilitate seamless transitions between different frameworks. Finally, the advantages of utilizing the NPU are illustrated with performance comparisons, demonstrating significant acceleration over traditional CPU processing.

Hi, everyone. So, welcome to the talk. I would like to speak today more about how we enable the generally the PyTorch ecosystem across uh NXP chips, which varies from all kinds of microcontrollers uh to, let's say, uh higher up uh higher up microprocessors, including discrete NPUs. So, well, oops. Okay. Uh so, to start off, uh I'm going basically describe what kind of devices we support. So, uh

starting, let's say, ranging from microcontrollers, which you can imagine as small uh usually costing something around a dollar or two, uh across a whole uh let's say a range of microprocessors, um to microprocessors, uh to which you plug in external accelerators. So, uh you generally can have something You would probably know something like a Raspberry Pi, and then you would have you would have an external uh

accelerator connected via PCIe or um USB for the huge workloads. So, at NXP, we do have our own accelerator IP. It's called uh eIQ Neutron. So, again, it's something similar to what, for example, Qualcomm has with Hexagons or a lot of other vendors. Uh what kind of use cases we target here? So, uh I'm not going to talk really about the uh smallest smallest MCUs. It's more

the um RT crossovers, which uh typically already have either one or two arm arm cores and then an external accelerator and DSP on top of that and then I'm ranging to uh automotive apps processors uh which already differentiate differentiate by um having typically Linux and already Cortex-A cores from ARM and let's say much more compute, so the MCUs you can imagine as having performance ranging from a

quarter of eight top uh half a top to let's say two tops on the MPUs. the suggested use cases here uh start at anomaly detection, which is in the industrial segment where you detect errors, for example, in a uh on a production line to let's say more advanced video use cases and really ending with uh LLMs, so we are not really targeting a high let's say high

compute LLMs. It would be mostly up to let's say two billion parameters LLMs, which are really uh currently the biggest ones we are able to run. Sorry, it's Outlook. Uh so for uh some details on the Neutron uh so it's a scalable accelerator, which we deploy in most of our um let's say most recent chips. Uh the chips themselves, they even have, for example, ARM Ethos-U on

some of them, yeah, so it's not just the NPU IP we have there, but uh it's a scalable infrastructure, which uh ranges uh let's say from 30 to uh 32 ops per second on, let's say the smallest MCUs, which is already able to support, uh, use cases, uh, such as object detection in real time for the cost of, I don't know, $1 or $2 chip. Yeah, and

then it ranges, the configurations, typically they add up up to, let's say, two tops on the bigger microprocessors, uh, which already supports, I don't know, uh, semantic segmentation and such use cases. So, uh, the way it does this is what I'm showing here is a single core of Neutron with four pipelines. And basically when you add these add these up, you add compute and it's basically up

to the chip how you configure it. Uh, generally we target mostly, uh, we, uh, the use cases for low power and optimize towards latency, low power, and run CNNs, RNNs, and some of the, uh, smaller transformers. Uh, and as for the tooling, uh, on top of it we've got a whole, um, mostly open-sourced, uh, open-source set of libraries tuned mostly for, uh, deployment, debugging deployment, um, optimization

frameworks and so on. And then, uh, driver stack which you can deploy yourself. So, uh, before we dive into that, then, uh, I'm going to introduce ONNX. So, uh, the reason being that ONNX is an intermediate format into which, uh, a lot of frameworks export. So, uh, because we are in an environment where we have, uh, a lot of automotive customers, for uh, those I found out

that they mostly work with MATLAB, for example. So, suddenly, how do you enable MATLAB? Usually not via PyTorch. So, those guys, they export into ONNX. We've got the option of exporting PyTorch into ONNX as well via Dynamo. So, I was quite happy to hear it's being improved quite a bit. That way, it's also boosts our own tooling and so on. So, ONNX is basically an exchange format

into which most of the frameworks can export an ecosystem, and then from which you can go to whatever inference engine you are you are in the end deploying. That also varies because, well, if you've got a Linux system, you can run TFLite, you can run PyTorch, whatever. But then, if you go down into microcontrollers where Executorch has been only very recently available there. So, it's not really

that major. Most of the customers, let's say vendors, are still running TFLite. And eventually, no one knows whether this will change into Light RT or Executorch will be eventually embraced. here is an overall pretty complicated overview of all the paths you can take when when you actually want to deploy your model. So, recently, most of the customers, they already train in PyTorch, and well, almost everybody switched

from TensorFlow to PyTorch nowadays. So, our strategy, and also what we see uh in the market is that yes, there is Executorch, and generally everyone everyone is looking at it, interesting in it, and it seems the way forward. However, it's been announced only last winter. Right? So, before that A for legacy reasons, there are multiple conversion flows, and B because of environments which simply PyTorch itself is

not supporting, which are like automotive segments such as MATLAB or you want to bring in you simply have your legacy neural networks developed in a certain framework which you want to bring in. that way we support let's say basically like the optimized executor workflow. And if you get it running, I mean, fine. I would call it that's uh that's the best of what you can get in

terms of performance power. However, if you want to bring in let's say different environments, you typically need to convert from them. uh for that, well, uh you can export into ONNX, and suddenly when you when you come into the ONNX world, a whole new set of options uh opens up for you. So, for that, we open-sourced two tools. Uh we've got the ONNX to Quant and ONNX

to TFLite. ONNX to TFLite itself is a tool which uh we originally when we started developing this tool, we said, "Okay, well, we can use open-source or like all the conversion tools out there." But, we ended up in a situation, well, uh okay, if you go through MLIR, for example, or some more sophisticated compilers, uh or if you go into TensorFlow and then into TensorFlow Lite, well,

the uh customers typically they came okay, we've got an application it's running on the specific model. We want to deploy it to our device and well it tried to convert it. Why doesn't it work? Well, I don't know just a stack trace. So and the reason was unclear pretty much. Yeah, you could break it in PyTorch to ONNX conversion, but then you would get eventually you would

crash in TFLite inference or light I light RT inference in this case. So the reasons were and so we ended up basically mapping directly from ONNX to TFLite flat buffer schema not really being limited by TFLite itself as an inference engine, but just going into flat buffer serialized format let's say complying to the schema such that the inference engine could eventually run it. But it's basically only

an intermediate representation. Um another issue we found and why we implement this quant thing. Um that was basically when you convert between individual IRs or formats, we had an issue that let's say the quantization methods were not supported eventually in in TFLite. So then basically we converted something but then introduced a lot of unwanted operators such as quant and dequant because and those had to be computed

in floating point and it completely killed the performance. So we said okay, let's first comply to the uh to the uh end point or final quantization schema then we convert it one to one directly map it and eventually it create any or we removed a lot of these floating point unwanted operations. The same applies if you also transform between or transit between different frameworks. You've got the

channel first, channel or last, so that was an issue eventually as well. Uh okay, so this is an overview of the tooling and if I can just have one more minute I would go over let's say some of the let's say results we get in terms of acceleration with our NPU compared to uh compared to the CPU. So uh really on let's say in the low-power segment

you can see that there is a huge difference of of running workloads on the NPU. the increase is starting in the MLPerf Tiny benchmarks with let's say seven seven 17 times increase compared to running it only on a single M33 uh ARM core up to let's say hundreds of uh let's say at a 169 times increase for for the ResNet. that's something really to consider when deploying

to really um cheap, small, tiny chips. if you are more interested let's say into deploying generally on chips or in let's say the NXP ecosystem which is quite interlinked with ARM as well and executor. Well, let me know. We love to open source all our stuff. Most of the tools here I mentioned and showcased are are available freely or welcome contributions. So, it's as easy as raising

a pull request. Okay, thank you.