On-Device LLM Inference on Android With ExecuTorch and Qualcomm QNN - Shivay Lamba & Kartikey Rawat

This talk covers the execution of large multimodal models like CLIP on Android devices using ExecuTorch and the Qualcomm QNN backend. The speakers, Shivay Lamba and Kartikey Rawat from Qualcomm, demonstrate how to achieve fully on-device CLIP inference, which allows for real-time vision-language understanding while eliminating the need for server connectivity. They provide insights into ExecuTorch's optimizations for QNN, focusing on aspects such as graph lowering, operator fusion, quantization strategies, and memory management that affect latency, memory usage, and power consumption. Additionally, the session discusses model export and compilation workflows, along with real-world benchmarks that illustrate the potential for deploying AI applications that prioritize privacy and operate offline.