Lightning Talk: Accelerating On-Device ML Inference With ExecuTorch and Arm SME2 - Jason Zhu, Arm

This talk explores the challenges of achieving low-latency inference for complex on-device AI workloads within mobile power constraints. The speaker examines how ExecuTorch, in conjunction with Arm's Scalable Matrix Extension 2 (SME2), enhances efficient CPU deployments for production AI tasks. A case study of the SqueezeSAM segmentation model illustrates the use of ExecuTorch with XNNPACK delegation and SME2-optimized kernels to evaluate INT8 and FP16 inference on a flagship smartphone. The discussion includes a detailed operator-level profiling to analyze runtime across various operators, highlighting the significant performance improvements achieved with SME2, which provides up to 3.9 times end-to-end speedup on a single CPU core while shifting the focus to data movement as the main bottleneck. This session offers insights into the practical workflow for deploying, profiling, and optimizing on-device PyTorch models, showcasing how SME2 enhances possibilities for interactive mobile AI.