Lightning Talk: Cross-Region Model Serving: PyTorch Inference, Observability... Suraj Muraleedharan

This talk covers the challenges organizations face in deploying, monitoring, and operating PyTorch models for inference at scale across multiple regions. The speaker presents production-tested architectures for multi-region PyTorch inference and LLMOps workflows, addressing key topics such as latency-based routing, blue-green deployments, and automated failover. The session also explores observability methods using OpenTelemetry for distributed tracing and Prometheus/Grafana dashboards to track critical metrics. Attendees will gain insights into CI/CD pipelines for cross-region model deployment that include automated rollback and drift detection, along with practical serving architectures and dashboards utilizing open-source tools.