Keynote: The Unbearable Lightness of (Agentic) Evaluations - Besmira Nushi

This talk by Besmira Nushi, Senior Manager of AI Research at NVIDIA, explores the evolving discipline of evaluating large language models in the context of agentic tasks, which involve planning and executing autonomous actions in the real world. The speaker addresses the methodological challenges in reproducing agentic evaluations, highlighting issues such as differences in reference implementation, error handling, and tooling definitions. Additionally, the session discusses the infrastructural requirements necessary for conducting these evaluations efficiently at scale. The talk concludes with a discussion on emerging best practices aimed at enhancing consistency in measurement pipelines.