Keynote: The Unbearable Lightness of (Agentic) Evaluations - Besmira Nushi

This talk focuses on the evolution of AI evaluation over the last three years, detailing the shift in expectations and challenges in assessing new AI models. The speaker discusses the issues related to emergent capabilities that outpace current evaluation metrics and highlights concerns around benchmark scarcity and data contamination. They introduce a proxy method called 'Kodak' for detecting training contamination, emphasizing improved techniques for understanding model behavior. The talk also addresses the disconnect between a model's recognized intentions and its actions, known as action thought disconnects. Furthermore, the speaker presents the importance of collaborative failure analysis across teams to improve model evaluation, advocating for more reliable and open evaluation methodologies through tools like the Nemo Evaluator SDK.

Hi everyone. It's a pleasure to be here. Um, I'm Bismilla um, and um, I'd like all of us to take the next 10 minutes to reflect about how AI evaluation has changed in the last 3 years and how we can pave a better path forward to make evaluation more approachable for everybody and more reliable. In 2023, the field of AI found itself in a position where the

new AI models could do more than what evaluation could measure. The so-called emergent capabilities were many. Often they were not planned. So much so that the best claim that one could make about a model was that perhaps somebody had seen someone on the internet see a model solve a very hard task, take an anecdotal screenshot about this and share it with the rest of the world. And

it was just like very hard to quantify anything about model behavior and capability. And there was a general sense of lightness and maybe confusion about what frontier models could really do. Now, there were several evaluation challenges at the time. Benchmark scarcity was obvious, sometimes because the tasks were complex, but sometimes also because maybe there was leakage from training from benchmark data to training data. And then um,

models were very sensitive to the way how they were prompted and instructed. And the model updates were very stochastic. You would have one capability in a release and then after a release you'd see that the capability either got worse or disappeared. And then an even higher sense of lightness about even starting to understand what was going on with the data and the models. Lots of excitement about

scaling. Evaluation was considered a scoring function, and data model understanding were left as a problem for the future, eventually. Now, fast forward 3 years after, it is actually very exciting to see that we have made a lot of progress. We have ongoing benchmarks that measure agentic and multi-turn capabilities of The open-source community here has produced several open-source harnesses that help with reproducible evaluation. And then lots of

progress in improving instruction following, maturing data recipes for making model updates uh less However, the expectations have changed. Now, we don't only want token generations, we want models that can take actions in the real world. And there are new challenges related to this and the way how we train um the models. For example, data contamination, while verbatim data contamination is less frequent by now, uh there are

new forms of rephrasing data during SFT or what we call RL exhaustion that make data contamination evaluation much harder. And on top of this, we have other forms of failures like, for example, hallucinations that have reincarnated themselves from factual hallucinations to tool hallucinations. And then spurious correlations of the past, now we see them as action thought disconnects where the model says something but does something else. And

finally, even though the the whole community agrees that uh it is important to scrutinize agentic failures, still like leaderboard first mindset still persists, and um it is um it is just easier to benchmark than to to master the capability. So, let me give you a few examples about each of these challenges. Last year in our group, we developed a simple method, a proxy method for detecting training

contamination. We call this Kodak and it is very simple. Basically, what it does is that it first measures the log probability of a text that is in a benchmark. And then it measures it again after introducing a random example from the same benchmark into context. And then we measure the difference between these two log probabilities to understand how much the in-context example helped the model to better

model this text sequence. What you see here is the cumulative probabilities across tokens. And we see that in the standard case, when the model has not seen this data in training, this the model confidence increases with in-context example. Nothing surprising here. We know that in-context helps. However, in the case when the model has seen the data in training, we see that either this doesn't help the model

with building more confidence or in many cases, the confidence of the model may even shake. Which can be attributed to broken data memorization patterns either on the in-context example or on the text that we're trying to model. Now, we see that this very simple proxy model and then score is a good discriminator between seen and unseen data in open source models that also public training data. And

most recently, we evaluated six models, six open source models. Three were released in 2025 in light green here and three were released in 2026 in dark green. And we see that for math capabilities, we see improvement. There is progress, meaning that these scores are lower. Uh but for agent capabilities and coding measured with tasks like terminal bench or life code bench, we see either very little improvement

or sometimes things get worse with this type of robustness. Meaning that we have not yet uh made the training recipes mature enough to um to generalize to this tests of uh lack of robustness for contamination. And there are other forms of robustness gaps. Uh for example, in in a recent study that we did for evaluating computer use agents, we saw that one of the most major uh

failure modes is a disconnect between what the model says and what the model does. And we call this action thought disconnect. For example, here the user is asking the model to protect their confidential files by assigning a 777 uh permission rights. You can call this a malici- malicious ask, you can call this a confused ask, you can call this a typo. However, what the model does is

that it recognizes that this is not safe. And it spends a reasonable amount of tokens in reflection and reasoning traces, and it articulates this in text. However, it still finds a way to actually do this, and it executes the action without asking the user and with without telling the user this. And um what what this means from evaluation is that instead of just looking at task completion

scores, trajectory evaluation that tags for this type of failure mode would actually be more useful than just reporting uh accuracy rates. Now, moving beyond the leaderboard mindset is hard. It is hard because in a world that rushes to catch the next AI vibe, it is hard to explain nuance. And this needs to be concisely, needs to be done uh carefully. But it is also hard because if

you want to build such of actionable uh pipeline, you need to rely on stages that are very different from each other computationally speaking. So, you need to train. You need to run evaluation at scale, collect the granular logs, conduct error analysis in-depth about what may have gone wrong. Take this and bring it into data synthesis and uh data collection and train again. All All these stages are

very different from each other. They may also sit with teams that sit far away from each other. So, one of the best ways that we have found to make this um actionable is to cut horizontally across the team and organize um failure uh analysis exchange that can inform everybody in model building about what are the most major failure modes that need to be addressed for the model

to get better on a given capability. But what can we do in a world where agentic evaluation uh remains fragile? There are the methodological challenges that I just mentioned, but there are also practical challenges. And this all stem from the fact that evaluating agents means evaluating applications. And applications fail. Applications may have bugs. But these applications and environments are what we rely on today for data generation,

doing our rollouts, and collecting rewards, and for evaluation. Now, dare I say, in a world where these are all one of the same, the real test of time and evaluation for these models is going to be real-world usage and how how users use this in in in their own uh environments. However, we still have to scale in lab evaluation with with the given benchmarks. And for this

to scale, these environments needs to need to be flawless. Flawless in the sense that they need to scale, they need to execute well, but also they have to be reproducible. Meaning that everybody who puts a score out there needs to own the score. And the best way to own this is to make it open source. So, the team that I work with at NVIDIA builds Nemo Evaluator

SDK. This is an open source tool chain that basically puts out the tooling, but it also puts out all the recipes that we use and the configurations that we use for evaluating the Nemo Transformer series of models. My colleague Jakub Szewczyk is going to cover Nemo Evaluator SDK later in the afternoon session. But in spirit, the tooling follows the same mindset as all the Nemo Transformer releases

with the goal of opening up the stack of machine learning from the data that has been used for training to the tooling and also to the evaluation stack. So, with that I will conclude with the message that an open evaluation is a stronger evaluation and it is a more accurate evaluation because it forces all of us to own the scores that are put in every technical report

and in every model card. Thank you.