On-Device LLM Inference on Android With ExecuTorch and Qualcomm QNN - Shivay Lamba & Kartikey Rawat

This talk introduces ExecuTorch, an on-device AI framework that enables the running of PyTorch models directly on Android devices and other embedded hardware. The speakers, Karthikeya and Shiva from Qualcomm, explain the developer workflow involved in transitioning PyTorch models to the JNI layer and how various optimizations are applied, including quantization and model size reduction techniques. They discuss the significance of on-device AI for improving privacy and reducing cloud costs, particularly through examples such as CLIP, a multimodal model. The session emphasizes the importance of model quantization and demonstrates practical applications of running AI inference on smartphones, utilizing Qualcomm's hardware capabilities.

Hey folks, my name is Karthikeya. My colleague name is Shiva. We are from Qualcomm and we are senior engineers around there and we try to explain you how you can use exit torch to run on device influencing on Android. Yeah. So, what is our agenda? We will go through what is on device AI in case if nobody is know know about what is on device AI. We

will go to the developer workflow from how to take by torch model to go to the GNI layer and explanation of how the how execution workflow goes. How to use quantization? We will explore some of the more tools apart from like we will obviously share like torch AO, which is the quantization toolkit. But apart from it, we can share our some personal favorite toolkits also for quantization

to make it easy for you in case if you didn't know about quantization. Cool. And last but not the least, we will go through clip model, try to show our demonstration how you can do it and we can explore and give you some more future aspirations in terms of how we can use different different examples to do on device. And it's no more like future. We are

living in that. We you can use SLMs try to on device not only on phones from IoT devices to XR devices to everything. So, let's get started. Now, the era of AI inference. Now, the major focus behind when we talk about inferencing, it's cost. Everyone says like you are running so much cloud bills while running a simple prompt if we take in consideration of chat GPT for

that instant, right? But our take is very different. For simple task, if you want to ask suppose how's the weather in Paris maybe or like what are the prediction of Paris? Why you should go to chat GPT and burn those kind of cloud credits, right? It's a very simple task. It's a very simple question to it. You can ask that using your own using your own phone.

You didn't have to it's very private. So, it's it's not like you're sharing your data anywhere. It's happening on your phone, right? And that's what on device gives gives that opportunity to us, right? Now, the important thing. Data is created at the edge that we know, and these data is goes to cloud, and it will run on training and do on the inference on that part, right?

Now, when we talk about data generating at the edge, data generating related to your phones, sensors, IoT devices, automobile, if if you drive a car, you know, it's like maximum chips are around on Snapdragon and it's built on top of it, right? So, the the prediction that what we are seeing as of now, the global edge computing market from projected from $15.96 billion in 2023 to right

now around $200 billion. Uh and it's going up and up on because even though everyone say privacy is a myth in tech, we know, but we will try to prove it wrong. Privacy is not a myth. Privacy is that what we are going on on that front. But, yeah. Now, why process data? And each points that I listed here, it's linked to each other. So, computational power

of modern smartphones that we are bringing on it, like the upgradation of silicon chips that we are going it. It will focus on the offline processing because everything is happening on offline, which improved on the privacy because your data is not going anywhere. It's just on your device. It can be your laptop or like you can say your compute, your mobile, your XR devices. It can be

Ray-Ban glasses, Android XR glasses, anything, right? And that which will reduce the latency, which means you are getting the faster answer on your on One of some of these are some of the example you can say where data process or get at the edge. Diction, text recognition, machine translation, optimize charging. I think so if you are using your phone, sometime it popped up about the optimize charging.

It's happening on device. Your data is not It basically It is understanding of pattern of charging, how your charging is happening, something like that. Now, what is XZ torch? That is important. This is what what we are here to. Yes. So, yeah. So, exit torch is an on-device AI framework that lets you export any PyTorch model, right? And directly on your Android or mobile devices, XR devices,

and embedded hardwares. And when I'm talking about embedded hardwares, if those who don't know, we recently acquired Arduino and our new products, Arduino Uno Q and and Venture Q, which is very newly product, is actually supporting that actually. You don't you You Your data is not going anywhere. If you If you have any business propositions for IoT devices or you are using IoT devices on it, on-device

AI will work on it actually. Now, this is a basic workflow when we talk about like take a PyTorch model, it will go to exit torch. We have our own Q and N, you can say, engine, which is called Qualcomm AI Engine Direct, which is like a delegate to exit torch. And it will go on three backends as often, all right? CPU, GPU, and NPU, right? As

NPU, we created our own NPU in that way. For GPU and CPU, we have our own CPUs and GPU where we have different like when we talk about CPU, we have excellent pack which supports that. For GPU, we have Vulcan. And for NPU, Q and N supports that actually. the exit torch mental model my my one of my colleague will come here and share that about it.

Yeah. Thanks, Shiva. >> Yeah, thanks for the wonderful introduction to exit torch. So, the idea is as you see that exit torch is like this wrapper or this runtime that allows you to take an original PyTorch based model. So, how many of you, by the way, here have written a PyTorch model from scratch? So, model.nn Has anyone written any PyTorch models? So, of course, you know that

most of your framework and everything infrastructure is on Python. So, what exit torch does for you is that it takes this original PyTorch.net model, then it will run some transformations on top of it. And these are typically ahead-of-time transformations. So, this allows you to save up the compute and make your model much more smaller to be able to run on a device that is really small. And

uh once you've done these ahead-of-time transformations, you get this .pt file. Some of you might have already heard about this .pt file, which is your executable. Then this gets loaded into your device target device. And then it depending on what kind of back end you want to use. When I say back end, it could be the CPU, GPU, or the NPU, depending on how large or how

compute intensive the workload is, it'll automatically handle that. And as a PyTorch developer, you don't have to worry about okay, what's going on behind the scenes? What type of back end should I use? Because the same module will basically run on any type of hardware. It's completely hardware agnostic. Let's jump into more from a diagrammatical angle on what's basically happening, right? So, there are two stages. So,

on the left-hand side, you see the ahead-of-time, right? This is what happens before the final ExecuTorch model is loaded into the device. So, here we'll take a PyTorch model, which will be a torch.net model. We'll have a function which we'll cover later, which basically converts that same model into exported graph. And then the graph lets you run a lot of different types of transformations. So, these transformations

could be things like fusion, quantization. And then once all of those transformations have been done, you can then compile it based on the different kind of target devices on which you want to compile it for. And you finally get this .pt format, which is the ExecuTorch program. And then you go ahead and load this ExecuTorch .pt file into your device target device. So, in our case, today's

examples will show you how you can load it into Android. So, for example, if you're using smartphones and you're are using an Android smartphone and you want to run an on-device AI inference with the help of executables, what that looks like. So, we'll cover that in much more detail. So, that's the ahead-of-time. So, great thing about AOT is that all of this compilation, all of this compute

is happening even before you load the model into the device, which basically means that you save up a ton of on compute. You don't have to worry about all managing all of that on your uh user's target device. And finally then when it comes to the inference, right? That's what's happening on your smartphone. So, this is where there are two things. So, with executables acting as the

runtime, you have loaded the model now onto your target device. There are a couple of things. One is the ops. See, these are operators, which are the machine learning operators, which basically define what are different types of machine learning, you know, because there is convolutional, so there are all these mathematical operations that happen. And second is the delegates. So, these are the backend delegates uh for the

target device. So, these could be the delegates for the CPU CPU and the NPU, which Karthik covered in his uh segment. And selecting the most appropriate target delegate, it'll choose that and you'll basically be able to then run the execution of your code. Now, probably the most important part of this entire slide deck because I know uh not a lot of you might probably already be doing

a lot of Android development, but maybe just for uh sake of getting to know the audience, how many of you have done Android development? Okay, a few. And how many of you are probably willing to get into mobile development at some point in time for on-device AI? That's what you're all here for. So, I assume a lot of you. So, of course, we have already covered some

of this. You start with the pytorch.nn module, and then there's a built-in function in pytorch called torch.export, which will convert your torch.nn module into this intermediate intermediate state, the IR state, which is this uh graph format. And by converting into this graph format, we'll cover that more about that in a bit. It makes it a lot easy for you to apply all of the different transformations that

you want to apply. So, whether it's quantization, fusion, or simplification. Now, once all of these transformations have been done as we have mentioned that ExecuTorch will partition and it will package your entire graph format into the dot PT file. And then, of course, you know, every single type of device architecture is a bit different. So, how does your Android device understand, okay, like what is this dot

PT file? Right? That is where the JNI layer comes in. My colleague Karthik will talk more about that in a bit, but that is what is responsible for being able to load your model into the Android device and then execute it. And now ExecuTorch runtime will come in and it will run the execution and finally it will use the appropriate delegate or backend delegate, which is the

CPU, GPU, and NPU to run the code. And as I mentioned that this works on a platform agnostic way, which means that as the PyTorch developer, you don't have to write custom kernels for the CPU, GPU. Like ExecuTorch will handle all of that for you. It has built-in delegates and support for multiple types of device backends. So, going now one step at each of these steps that

we have covered, right? That entire PyTorch to Android journey, we'll go basically one step at a time. So, the first one which we mentioned was converting the torch.nn into that stable graph. So, this is the static portable graph. And essentially what's happening behind the scenes is that it is using what we call as the eight and ops. So, these are part of the core PyTorch operations. And

this also eliminates the core Python control flow because what we're trying to reduce is the dependencies and the runtime dependencies of Python. So, all of those also get eliminated so that whatever target device you're using, there's no dependency of Python required at all. And of course, the most important aspect is that since we are trying to compile our model to run effectively on device. Quantization is usually

one of the most important things, right? So, what is quantization, right? It's a way for us to take your original model and you would be able to reduce the size of the model and make it more performant to run on more of these edge devices by reducing the precision for the model weights and the attentions, right? So, for example, original model could be a 32-bit 32-bit floating

point and we can reduce it to an 8-bit integer. So, this way you reduce the overall size. Now, of course, sometimes there is also a reduction in the overall accuracy of the model, but that's something we'll explore how you can optimize that for your own workload. So, Torch AO is the open-source quantization technique that's offered as built-in part of PyTorch. So, you can also use this if

you're already using ExecuTorch, you can use AO, but there are also some other open-source frameworks like AIMET, which is from from Qualcomm, that you can also use for And here's an example of converting a PyTorch model into a dot dot into the static graph, and then we're using post-training based quantization. So, this is a simple example of how your quantization using dot AO would look Now, most

important part, what type of quantization should you pick for the edge, right? That's probably the most important decision. So, at the far left corner you see is the best performance where you have W8A8, which basically shows both the weights and activations. Now, what that means is that this model will be the best performing, but it'll also be very large. Now, as you move towards the right, you'll

see that we move towards mixed precision. We are moving towards more, you know, So, yeah, I mean, the entire idea here what you see is that depending on the type of workload that you want to run on, you will choose between different types of quantization. So, the W8 and A8 basically means 8-bit weights and for 8-bit activation, right? So, there are all of these different types of

quantization and depending on your model that you are running it on or device that you're running it on, you will probably choose one of these different types of quantization techniques that are listed. And you can run some benchmarks and try to find what is the most appropriate one for your use case. Now, I'll hand it over back to Karthikeyan. >> Thanks, Shreyas. So, this part is where

is very crucial. Not for you, but just I'm just giving you the interior architecture of how ExecuTorch works. Partitioning and lowering. So, generally when we talk about when we got the final graph from after the runtime, right? That particular graph is scanned by ExecuTorch runtime in a way, right? It will transfer that graph to the backend. Basically, it can be GPU, CPU, or NPU for here. Like

those who are using the engines for NPU, you're using Qualcomm QNN. For GPU, you're using Vulkan. For obviously for CPU, you're using XNNPACK. In case In case all the three backends fall off, it will go to the kernels part of it. And again, I know ExecuTorch provides you the custom ops around it, but in the sense sometimes some kernels are not available or not stabilized to it.

So, what you can do, you can create your own custom kernels also. You can contribute to it to PyTorch for the same, right? And even though those who been active in Hugging Face community, you pretty much know about it. There is a community page called uh kernel hub, where you actually contribute your builds, your architecture builds to them, and community can use it in that way. So,

you can do it. It's just that part of it. And after that, it will go to the lower ops to backend formats, which is actually taken care by ExecuTorch only. packaging basically packaging involves about the five components of after getting from assembly.pt file to to basically package it. One is IR, which is already stated like intermediate representative is a basically a one of the core part of

it. Uh weights, back end sub graphs, meta, and execution schema. Those who didn't know about execution schema, it's just like a final you can say JSON file and you can say in that way where it's contain all the pre-processing part of it and how to structure it just to store in that Now, runtime execution. That's we can say in one of the important part of it, but

yeah. So, uh I will share it in the next slide uh how JNI layer works, but just give you a basic uh surface layer knowledge. Basically, it's a bridge between Java and C++ bindings. So, as we know Kotlin is already before Kotlin, Java is the primary language for mobile development, right? Or like Android development, if I will say. So, it just a bridge between C++ because you

know, when we talk about kernels, all the kernels are written in C++ or maybe right now it started working on Rust also. But, C++ is the priority. To bridge that, uh it uses runtime execution for C++ and Java. It dispatches the sub graphs to correct hardware. Like, what are the specific hardware supports? Sometimes, uh you can obviously you can fix NPU as a back end, but sometimes

that particular op is not supported. So, generally it fall back or it will crash generally. That's why the issue of operators are are uh are very are are in pain actually a lot years back actually. But, XNNPACK basically solved that. And it is obviously hardware agnostic API, so you're not going your data is anywhere. It's happening on your device. So, all the APIs are very hardware agnostic,

so it's related to your CPU, GPU, and NPU. Now, the JNI layer and we can say we are entering to the native AI stack in a way, but yeah. So, what is JNI? The as I already stated, it's a bridge between uh Java and C++. Think in that or Java or Kotlin. So, basically what Java does or what Kotlin layer does, it handles all the Android responsibility.

It can be UI, activities, uh life cycle, permission, sensors, camera, like that. For JNI, uh for C++ JNI layer, it actually very thin in that way, which basically it loads the exit torch model in a C++ binding format, which maps the buffer uh data type, and it execute forward function for it. And it actually focused on about the design goal is basically to as make the JNI

very small so that it will be easy for Java Java layer bindings uh faster processing. And this is a design basically what I'm doing here. I'm just creating a constructor, right? You can say run clip in inference native. And then, after creating a constructor, you just have to pass the embeddings of image data, tokens, image features, and text features for it. Now, let's go on to demo,

but I have to change my laptop for that. Uh uh Shiva, can you take a photo? >> So, what you're seeing so far is uh that what we're going to show you is how we took a CLIP model. A CLIP is a multimodal model from OpenAI, and how we will just run it inside of an Android phone. Uh that's being powered again all on device, and we'll

show you like uh live cosine similarity where the vision model will be able to figure out what's happening on the user's camera. So, what's whatever is being captured by the >> So, basically, this is Microsoft Surface laptop. It have a Accelerate chip, like Qualcomm Accelerate chip. So, I will try to do some computation stuff, but like majorly the demo part to it just to give them an

instant. Yeah. Give me a sec. >> And then, once we are done with the demo, we'll just conclude the talk with some of the things that you should be aware uh when you are building Android apps. uh as machine learning engineers, some of the UI things that you do on Android, those can be a bit tricky to deal with. So, we'll also have some best practices that

we like to just cover quickly. So, we have opened up Android Studio and we'll just quickly walk you through the entire execution flow for our app. Cool. Let me turn on USB debugging so that it will Cool. Let me give me a sec, folks. Let me open Cool. So, I hope so you're able to see the right hand corner. I hope so. Cool. So, let's take a

photo maybe of my my colleague. Cool. Okay, so I take a photo. Okay. I just I'm hoping this is a boy. Okay. Let me do the Okay, so 6% which is bad, but yeah. If I will write This is a man with >> smiling >> smiling face. Now, the point I'm saying the if you say if the clip models have a ratio around if it's less than

20% that means it is false. That means the statement of the prompt and the image encoding is is basically false or negative. And if it's more than 20% that means it's matching and like it's matching in that way. Now, the point Now, the point is if I will be more descriptive in the prompt, it will basically match on that thing. So, at the end of the day

the whole the whole process is how we can make it and again it's very simple. I'm just use the clip model. If you use llama model for that sake, I guess llama model is up on the exit torch library. You can use that for that thing. But but the main thing is apart from the demo Um let me show you the code actually. this two aspect that

I did uh as I already said, this this is the function to basically load the model, okay? Just have to pass the What are the path actually what whatever the path is, This This is the where I'm normalizing just to make it more palatable for for you guys to just uh how clip actually inference on it, okay? Here I'm creating an embedding of the image, okay? How

to be using a clip model how we are taking image embeddings uh and text embeddings simultaneously. So, these are the two aspect to it how I'm taking the image embedding and text embedding. And after that, this is the cosine similarity and uh and you're retaining that thing to it. Again, if you want to very specific for NPU, what you can do you have to uh you can

go to uh Qualcomm AI engine direct. Uh you have to download a file over there. You have to create a directory called JNI lib, and these are the files which actually activate the NPU acceleration on your phones, on your any devices that have a Snapdragon NPU in it, okay? Let's go to uh final >> So, one of the things uh as we'll talk about is that when

you're talking about Android applications, there's something called as the UI thread, right? Similar to how we have background threads. So, one learning is that you should never block the main UI thread. So, you could have some background processes running already as part of your app, but never block the UI thread. And >> Cool. Uh okay, cool. So, these are some of the best practices that that we

follow so that you didn't go into the struggle in between to do that, yeah. But, yeah. Uh first of all, it's very important whatever the image or the data that you're inputting, the size of that input should be equal to the output of it. A lot of folks while building the models, it is very easy to build the model, but when it's come to the uh deployment

part, the always issue is normalization, resizing. These are the always a issue for it. So, try to try try to use resize function to understand what is the input, what are what are you giving, and output should be the same. Otherwise, it will crash, actually. Second is you can flatten once flatten once the data types to it that like data you can say direct byte buffer is

one of the data type of it, okay? And you can map out JVM arrays for that, also, okay? Now, the important thing is performance pattern, zero copy first. That the meaning of is, please do not copy tensors again and again on memory footprint because if the memory footprint will increase, it will take a lot of time to basically inference on it because already you're increasing the memory

mapping on it, okay? Warm up once. First, just load our graph and it will allocate the memory and you didn't have to do nothing, actually. After that, you can focus on preprocessing, inferencing, and postprocessing. It will go to the process of quantization while you're doing the quantization of model. So, you can test on it before preprocessing, during the inference, and during the postprocessing. Last but not the

least that that is shared by my friend by my colleague, threading and life cycle. Never block the UI thread. I know a lot of mobile developers who actually built It is always an issue with it, but what you can do with use Kotlin coroutines. It's like an async for JavaScript. Is there any JavaScript developer here? Okay, cool. I know JavaScript is not that famous compared to Python,

but I'm I'm betting on it, but yeah, cool. And and to release the memory footprint, you can use two functions, on clear function or on destroy function, which basically remove the tensors which are not in use, actually, right? And last but not the least, I guess my friend will take from it, yeah. >> So, we'll just conclude by one final slide, which is we showed you an

example of CLIP, right? For multimodal models. But you could also do something like an on-device retrieval augmentation, which means that if you're running an embedding model, you're running LLMs, all of those can be run completely offline first. You can also use the CLIP as a composable stack, so you could have multiple models running in chain, and you can compose multiple models together to again have fully functional

on-device examples. In fact, like the Google team has shown some authentic examples that you can run so you can run full on agent workflows, agent skills completely offline because of executables because you can target the same build for phones, tablets. We showed you phones, but it will be delegate agnostic and can run across any type of device infrastructure. So, with that we'll conclude and uh yeah, thank

you so much. And these are some of the resources that you can take a look.