Lightning Talk: Running ExecuTorch Applications With Silicon Accelera... George Gekov & Aki Makkonen

This talk focuses on running executor applications with silicon acceleration for ultra low power devices, specifically addressing the implementation of a speech-to-text transformer model on battery-powered hardware. The speakers explain how to efficiently deploy a PyTorch trained model, leveraging the Alif Ensemble E8 platform and its Ethos-U 85 neural processing unit (NPU) for high-performance inference. They detail the process of quantizing the model and using the executor runtime without the need for additional version control tools. The presentation also includes insights into the hardware architecture, memory considerations, and the performance metrics achieved during real-world testing, highlighting the efficiency of this setup for extended battery life while performing complex machine learning tasks.

Very good afternoon, everyone. My name is George Geckos and I work as a software engineer in arms executor team and I'm here today with my colleague With Aki Makinen, I work as a staff application engineer at Alif Semi. And we're going to speak to you today about running executor applications with silicon acceleration in ultra low power. So, running speech to text no transformer model on a battery

power device, you know, running it very very fast um it's a very hard problem. So, why is that hard? Well, first, how can we run transformer model with silicon acceleration and in low power? Second of all, how do we develop the end-to-end speech recognition and how do we deploy the application on real silicon? Well, we're here today to tell you that we basically have answers to all

of those questions and we're going to give you the key things that you need to know in order to reproduce this on your own device. So, how can we run transformer model with silicon acceleration with in low The Alif Ensemble E8 platform allows you to do just that. Uh Aki obviously works for Alif. Um how can we develop an end-to-end end-to-end application with PyTorch and executor and

of course running executor provides the benefit that you no longer need to be passing via any kind of version tools. You can stay fully within the PyTorch ecosystem. Um and how do we deploy the application on real silicon? We already have an example application that is available on github that anyone from you can just, you know, clone and play with. So, what we do is that we

take a PyTorch trained model, a transformer model, we deploy it on the Alif E8 SOC and under the hood the Alif E8 SOC is actually relying on the Ethos-U 85 NPU to do all of the heavy lifting. So, at the high level, uh the speech to text is done by the conformer model. This is a transformer neural network. The training code is available on github. It is

just a standard PyTorch training. We haven't had to do anything special in the training code in order to run this on an edge device. We obtain a FP33 FP32 model, we quantize with executor, then we run it with the executor runtime. yeah, so it's a fairly standard pipeline where you have a microphone, you're having a pre-processing, inference, post-processing, and text. And Alif is having a slide where

you can see that from this overall pipeline, in fact, the part that takes the most compute is inference and luckily this is exactly the part that we're accelerating with the Ethos-U 85 and with executor. So, what do you need to know from executor standpoint? The first part point part The first part is obviously that you need to have your PyTorch FP32 weights. You can find such weights

pretty much all over the place online. And then the first step is going to be to export the model. When you export the model, what you're going to obtain is a trace graph. Then you quantize the neural network, you quantize it to 8-bit weights and 8-bit activations or 8-bit weights and 16-bit activations. Of course, you need to compile for the target NPU. In this case, we're going

to compile for the Ethos-U 85 256 MAC configuration because that's what the E8 SOC has on the on the on the in in hardware. And the result of this compilation step is a PTE file. The PTE file is what you take and you then basically place it usually into the external memory of your device, for example, into the external flash and then the NPU is going to

read the command stream and is basically going to perform inference on this PTE So, looking into a little bit more detail at the key stages from executor, the tracing uh sorry, the export is happening with the torch exporter API. Of course, you need to pass the model as well as the example input and you need to pass this in order to deduce the shapes for all the

different layers. That's the first key step and this is the entry point for executor. And the second key step is the quantization where you're going to be using the Ethos-U quantizer that is already available by arm is already available within executor and you also need to use the torch IO prepare PTE and convert PTE uh functions. The actual post-training quantization is fairly standard, I'd say. You have

to pass representative data set in order to know the quantization scale and zero point for each activation for each weight. You need to know how you're going to quantize the different layers. So, you need to you need to show some real data for that. And last but not least, the API where we're doing a lot of the a lot of the intelligence, so to say, is the

torch transformer lower. So, we partition the uh lower the NPU part of the graph into an intermediate representation and finally we compile for the target NPU. Thank you, George. Thank you also for rest of the arm team who has been may making this possible this demonstration. Okay, so here is an overall block diagram of our latest generation SOC. It is It is the flagship Ensemble E8 and

as the bullet point says here, it's a versatile in ML model deployment. We have three different NPUs there and as George put it, uh this is the Ethos-U 85 is now the main main workhorse because uh it has all the all the operators which the conformer model needs. Uh so and and also one thing to note is that the Ethos-U 55s are connected [snorts] processor subsystems so

that the Ethos-U 85 is actually more free to use. It can be directly used from any of of the cores on the chip. So, in that sense, the NPU choice is easy in this this use case. We initially chose the high performance Cortex-M55 as as the CPU core for this for this use case. We also measured M55 high efficiency core. Um the conformer model itself in this

case uses 11 million parameters. So, when it's quantized to integer eights it cannot be put to the SOC's internal MRAM. So, we used external flash memory for that purpose. This is the development kit that we are offering and I wanted to highlight that we have all the necessary bells and whistles here for this kind of demonstration. There's two different kind of microphones on board and the external

OSPI flash is there and you don't even need a debugger probe because that's also available on board and you can you can flash the the on board on board flash memory with that debugger. So, here we have more like development details. I'm not going to go through each and every every bullet point, but I want to highlight here the first one that we ended up into a

lot of different kind of problems when we wanted to deploy a PyTorch model before executor torch was existing and and mature enough and and this demonstration proved that we we can now do it natively without all these steps and and concentrate really on the on the performance and deployment details more. We did an experiment because I mentioned all the already the number of NPUs. We did experiment

also with noise reduction model on HE core running the Ethos-U on on Ethos-U 55 NPU. Uh so, we utilized two NPUs of the uh of the full SOC, but we don't have like power power results here for that one. I just wanted to mention it. I I thought it was cool. So, here we have the inference time which is measured in in real hardware in Ensemble E8

dev kits and we run the model uh with the pre-processing and post-processing steps. So, HP core and HE core, there is difference. HE core runs at 160 MHz and HE core runs at 400 MHz. So, definitely the the pre-processing step is as you can see, it's almost two and a half times uh on on the slower CPU. But, the overall inference time doesn't differ much because because

the NPU is is doing the doing the most of the work. And here, last but not least, slide we have power figures. is measured from from the HP core use case. As I mentioned, there there was also the HE. It's Oh oh yeah, I actually mentioned it there. So, it's slightly less even though it's only a a bit longer time per inference. So, it uses a bit

less energy per inference. There is some some small optimizations probably because we see that the pre-processing step is the the small one bit in the lower lower line. There's all the pre-processing happening. So and and you can see from the spiking where the NPU actually works. So so there there might be still something something to optimize. But again if we have I I put put a question

here that how long would a battery last with such an energy usage. Let's say we take a typical 3-V battery with a 1,500 mA hours then it would means means something like 16,000 J and then you can do the maths from that 104 4 mJ so I got 43 hours and that would be constant inferencing and there's a duty cycle to take into account because it was

something like 700 ms per each 15 seconds of audio. So it would be 36 days of constant audio speech recognition running on this on this platform and then you could still estimate something about the about the use case that probably someone is not pushing the button and speaking all the time with speech recognition on. But anyway as I said there's some optimization possibilities. I would still conclude

that we are quite amazed of these results already. After all we're running a transformer model and just a while ago this kind of device class couldn't have done it at all. So thank you. >> [applause] >> Anyone has a Thank you. It is fundamentally a very brilliant idea. How do you manage the compiler stack for the same? Yeah. Besides PyTorch and Arm side compiler. Yeah. So very

quickly the question is how do we manage the compiler stack here? What is happening is that effectively the compilation happens under the hood. We already support all the operators from this transformer model from the conformer and then under the hood we generate it's called Tosa flat buffer and then under the hood again there is a compiler that is going to take this Tosa flat buffer and is

basically going to schedule all the tensors to run on the Mac Blaze Mac based units of the NPU.