Lightning Talk: From Hugging Face To Handheld: Scaling LLM Deployment W... Cormac Brick & Weiyi Wang

This talk focuses on deploying Hugging Face transformer-like language models to edge devices using LART, a tool designed for cross-platform compatibility across various hardware. The speaker demonstrates how to export a 3 billion parameter model to a format called LART IR, which is optimized for running on mobile devices, including Android and iOS. They showcase the process of model conversion using PyTorch and Light RT, highlighting the functionalities such as quantization and runtime performance across different backends. Additionally, the speaker explains the features of the Edge Gallery app, which allows real-time interactions with the models on handheld devices, encouraging developers to build and customize their own applications. With a strong emphasis on ease of use and open-source resources, the session aims to empower developers to leverage cutting-edge AI technology on edge hardware.

Hope everyone have a good lunch and get a good weather. I hope everyone is not going to get asleep, so I decide to move the demo earlier. Okay, but first, what's the topic? Getting the hugging face transformer like language models to your hand-held to the edge devices. With LART, that's the that's the main thing. But let's go through go straight through to demo. So, on this one

like you can download like a LART nightly package and what it command do is like it takes the the Queen 3 0.6 billion parameter models and export it to a format called LART IR. Which we intend that to be run on different platforms say on desktop, on mobile phones including Androids and iOS, IoT, all etc. So, first of all is on PyTorch magic. Run deposition. It's the

it's the building blocks on the AIT ops. Not really back my magic but and after afterwards you see on the MR module some of more you may be familiar with like TFLite. TFLite's converter stack is actually a pure MR stack. So, that's what's it doing and uh Let's see. It's 26 26 seconds now. Yeah, we got a good timer here. Okay, we got model exported. 40 seconds

running quantization. Here 2 seconds and we got the model exported here. Yeah, the temporary directory. So, light T Oh, T LART so Yeah, this is like LART IR is another command we can run the model. Say I will just say hi. What's meaning of life? My favorite smoke check prompt, and it gets down the on running. Also, uh another thing like because we actually run on different

back end. But, if you want to run on CPU, we can add the back end code to the GPU. Yep. Oh. Okay. Wrong spelling. multi-turn. Okay, and what about phones? You can see I actually connect this phone here, and uh let's Let's check whether I dropped my device or not. Cool. And let me push the model to the phone. Uh this is one, right? And in the

meantime, let me do the screen copy. So, if anyone like come to our booth earlier, we have a nice demo app called AI Chat Gallery, where you can run the actually the Gemma 4 models. But, the point is what this mean like this talk is not for Gemma 4. It's for custom model, maybe fine-tuned from anywhere else like your own workflow. So, you can see you can

load the models we just uploaded turn on CPU GPU like we because we can run it on both. Import model, going back. Uh let's just use AI chat. We have got our small here. Because we don't know that on uh let's see what's the back end. Okay, let's go to GPU. Now it's doing just compilation for GPU a couple seconds after State that. Um What's the meaning

of life as well? Yep, that's running on the phone GPUs. Yeah, it's always fun to have like actual device running to them also. It didn't fail, good. But let's go back to the slides. Hopefully you get a little bit wicked up. So, first of all, who we are. Uh like I said, but maybe some of you are familiar more familiar with the the TF Lite brand. On

the old days, uh we had the same team, like actually the same product. Um 2 years ago last year, we actually rebranded our product the TF Lite to remove the TF part, keep the Lite part because we want to have a the optionalities for different frameworks like PyTorch, Keras, said and others. And also we want to have like more ingre- embracing more on like uh models and

also uh more hardware accelerations. So, here we are, Lite RT. And we are actually in on under a bigger bracket on Google AI Edge where we have the different layers of uh offerings from say the auto- MediaPipe which you are familiar with like providing in turnkey solutions to Lite RTLM, which is the orchestration part of light RT building the LM ready features. Light RT itself running the

single model inference on different back ends cross platform. But then we also have like the all the components like the all the toolings like model explorer which you can visualize a large language model on your browser and uh say the Edge Gallery app you see before for the actually the showcasing. But also we have the Edge portal which can help you to uh benchmark and the validate

a single model on a fleet of different uh Android and iOS device because you know fragmentations always hard to debug. Okay, so back to the topic. Uh so as you already see like what's what's happening during the the stack printed in the model export path, but uh in general we come the model coming from Hugging Face uh Transformers model uh safe tensor and it go to the

light RT torch tool chain which have both the PyTorch native authorizations, uh light RT authorizations which mostly is the same as the what we have in TF light authorizations, the whole MLIR And also our new condensation tool chain called Edge Quantizer which provides more flexibility and more uh usability on quantizing a model. Then we on the left two are the on host part, then we go to

on device runtime where we have light RT and light RT runtime which provides a cross platform which means uh Windows, Linux, uh Mac, iOS, Android and all the embedded um and also accelerators including CPU, GPU, and NPUs. Uh and we have like the light RT LM is doing the Edge device optimized LM inference pipeline, but also offering some advanced features like function calling, constraint decoding, things like

that. And to next last step is the deployment where we you can Okay, you got model, you can try it out on yourself on a phone, which you can sideload the model to Edge Gallery. And also you can do a real production deployment with light APIs which have Kotlin bindings and other things you needed for things like macOS and iOS. So, for the first step is the

the conversion lowering step. Uh we have different components where light RT torch has a module called light generative where we provide some pre-built layers on pure PyTorch but optimized for on-device. Advanced users like OctoML and others, they use the those layers to build their own models as well to make sure a smooth development uh flow. After that, we have light RT torch which is MR-based PyTorch to

light RT converter. Uh half and half like we have still used the PyTorch like native FX passes and decompositions on the front end but also we have the pure TFLite MR stack as the organization back end. We have the Edge Quantizer which provides advanced recipes for LLMs so you can choose what's like what's the activation quantization types or like data quantization types for different layers. And coming

soon uh is like for NPU compilation. Right now we what we demo is on CPU and GPU but we all can we are also offer a unified calibration and compilation workflow for different NPUs including Google Tensor which is the NPUs on Pixel phones, uh Qualcomm, MediaTek and for majority of the Android development and Intel Intel for Windows, Linux and Chrome OS and more coming up. The advantage

of us like our stack is like we have a unified stack for the runtime. We're on the company company side we have the structure layer for MPUs say like all the vendor specific implementations are actually hidden the abstraction say like Qualcomm sorry SSI MediaTek pixel you don't need to deal with their SDK directly. We handle that for the for you and that's the same for the runtime

part. Uh we have CPU X and pack GPU ML drift which works on the open CL web GPU metal. That's why we get the cross platform compatibility. But also the compute dispatch part which is similar to the compilation part where you don't need to touch the SDK don't even need to know it's running on MPUs. We handle that for you. Yeah and this is the command I

used in the in terminal which is pretty simple like uh you provide the model name or like even like the directory you save the save tensors and put out output tensor. It gives you a light out TRM file which is a collection of TFLite model or light out TRM model I would say tokenizer and all the necessary metadata. And you can run it directly with the CLI

but all of those you we can we also offer the Python APIs so you can integrate that to your own work stream. Yeah this is what you already see so I will just skip that so Yeah. And for Edge Gallery app so uh we in initially we aimed that to be a demo app but like it actually we found like very useful for uh for people showcasing

the capabilities of the models. So for example like for this time like Gemma 4 we actually introduced the agent skills where you can write your own skills in markdown and the JavaScript and use the model to actually invoke the skills. So, we believe like for for those like we are providing the some like some little ideas for the developers to uh put like more general ideas because

they like everything here is open source. You can just fork the repo and build your your own experience and or maybe productionize to your own uh own app. Yep. And this is the resources like uh first of all, the Go Edge which contains all the uh documentations for all the components, all the products offerings from us including Light RT, Light Torch, uh Light RTOM. Yeah, different parts

for different like use cases. And also feel free to download the the demo app like the carrier carrier app we have. Um in the morning keynotes like the Gemma four models, you can try those uh demo videos are not pure recordings. You can try that on your own phone with those. So, yeah, feel free to download. Any questions or That's all?