The Science and Practice of Open and Scalable LLM Evaluations - Grzegorz Chlebus, NVIDIA

This talk by Grzegorz Chlebus from NVIDIA delves into the science and practice of open and scalable evaluations for large language models (LLMs). It addresses the challenges of model evaluation in the rapidly evolving AI landscape, emphasizing the importance of rigorous quality assurance and effective testing methods across multiple benchmarks. The speaker presents best practices, practical patterns, and anti-patterns for standardizing evaluation methods, while showcasing Nemo-Evaluator, an open-source tool designed for scalable evaluation. The session highlights how this tool facilitates transparent and reproducible measurement, ultimately contributing to a more open model development process for the Nemotron model family.