Lightning Talk: Beyond Generic Spans: Distributed Tracing for Actio... Sally O'Malley & Greg Pereira

This talk discusses the critical need for end-to-end observability in production large language models (LLMs) to monitor performance, attribute costs, and validate optimizations. The speakers detail their implementation of tracing for llm-d, a high-performance distributed LLM inference framework, using manual OpenTelemetry instrumentation to create actionable insights. Key topics include how distributed tracing can be utilized to track cache-aware routing, analyze processing choices, profile models across multi-node deployments, and correlate request patterns with workload autoscaling decisions. Attendees will gain an understanding of how LLMOps necessitates a fresh approach to distributed tracing compared to traditional microservices and learn effective strategies for instrumenting inference stacks.