Redefining SLIs for LLM Inference: Managing Hybrid Cloud wit... Christopher Nuland & Hilliary Lipsig

This talk focuses on redefining Service Level Indicators (SLIs) for Large Language Model (LLM) inference and managing hybrid cloud environments through the use of vLLM and LLM-D. The speakers, Christopher Nuland and Hilliary Lipsig from Red Hat, address the challenges SREs face as traditional performance metrics fall short in the context of LLM applications. They discuss the importance of metrics such as Time-to-First-Token, cache hit ratio, and GPU utilization, and explain how vLLM and LLM-D can enhance observability and scalability for LLM inference. Attendees will learn to establish new SLIs, instrument distributed inference systems using tools like Prometheus, OpenTelemetry, and Grafana, and effectively integrate LLM telemetry into Kubernetes SRE practices.