Lightning Talk: Pluggable PyTorch LLM Inference Architecture With VLL... Yahav Biran & Maen Suleiman

This talk covers the evolution of PyTorch-based large language model (LLM) serving, focusing on the transition from monolithic inference stacks to the integration of diverse hardware accelerators. The speakers, Yahav Biran from Annapurna Labs and Maen Suleiman from Amazon, discuss how modular plugin architectures allow PyTorch models to run efficiently across different backends, specifically showcasing AWS Trainium's integration into vLLM through standardized interfaces. The session includes a technical deep-dive into vLLM's Hardware Plugin architecture, demonstrating automatic platform detection using Python's entry_points, and explores NeuronWorker and NeuronxDistributedModelRunner enhancements. Attendees will learn how this approach maintains compatibility with standard workflow patterns while enabling hardware vendors to contribute to PyTorch inference.