Lightning Talk: From Hugging Face To Handheld: Scaling LLM Deployment W... Cormac Brick & Weiyi Wang

This session explores the end-to-end journey of deploying custom PyTorch-based Open Source Large Language Models (LLMs) on cross-platform devices using LiteRT. The speakers, Cormac Brick and Weiyi Wang from Google, demonstrate how to convert a Hugging Face Transformers checkpoint for on-device execution, covering essential steps from conversion to deployment. They discuss automated optimization techniques, seamless fine-tuning integration from an Unsloth session to a TorchAO-quantized model, and how they enabled the QWEN0.6 model in just 20 minutes. Additionally, the talk includes interactive validation methods for verifying numerical correctness before device deployment, ensuring a streamlined fine-tune-to-deployment process within the PyTorch and Hugging Face ecosystem.