AI Inference at Scale: Reliability, Observability, Cost, and Sustainability - Rohit Bhardwaj

This talk addresses the complexities of AI inference as a production workload that is always on, cost-intensive, and challenging to manage. The speaker presents a vendor-aware playbook for developing reliable, observable, and sustainable inference systems at scale, utilizing insights from the Google Cloud AI/ML Well-Architected Framework, Azure AI Workload Guidance, and Databricks Lakehouse Principles. Attendees will learn practical strategies for managing latency and costs, as well as implementing full-stack observability for metrics related to prompts, vectors, and GPU usage. The session emphasizes the integration of FinOps and GreenOps practices to achieve financial and environmental efficiency, supported by real-world case studies and cross-cloud design patterns.