The Science and Practice of Open and Scalable LLM Evaluations - Grzegorz Chlebus, NVIDIA

This talk focuses on evaluation methodologies at Nvidia, specifically around benchmarking machine learning models. The speaker discusses the limitations in current evaluation practices, such as the lack of comprehensive reporting on metrics, the absence of standard deviations, and methodologies that are often not shared. They emphasize the need for transparency and reproducibility in evaluations, proposing open sourcing not just models but also evaluation recipes and tools. Nvidia is working on their Nemo Evaluator SDK, which aims to simplify the process of running evaluations at scale and ensuring reproducibility. The speaker shares insights into the architectural evolution of evaluation tools, moving towards environments where models can interact with tools over multiple turns, highlighting the importance of logging, tool usage, and managing the scalability of evaluations.

Hello. My name is Grzegorz Flis, I'm a manager at Nvidia and I will be talking today to you about evals and also how we approach evaluations at Nvidia. And I would also like to share some like practical stories that we have internally in the company. Let me start with maybe providing some context because there is still a lot a lot to be done in the eval space

and why why why I'm saying that. So, if we take a look at the recent model releases including our Nemo Trons and also models by OpenAI, Google and other teams which are open-sourcing models, we can see that typically they are running a lot of benchmarks like 20 plus if you take a look at this with one exception of Kwan which managed to run more than 80 evals.

This is typically done across multiple models, you know, because anytime you would like to make a release you typically are looking for a story that you are the best in in some dimension, right? So, either accuracy, so I'm focusing on accuracy evaluations or maybe in some perspective if we are talking about quantized models and so on. So, you cannot really go public without having the story. And

you are saying that we need to we need to run a lot of benchmarks and compare against a lot of models. And it's like leads to hundreds of evaluation runs. And and it's not always published in the model cards how people are doing that. So, we have the numbers, typically only a number. We don't have standard deviations, we don't know the full methodology. Sometimes we'll find something

in the technical report, but typically this is not the full story. So, we typically don't have the prompts that are being used, we don't have the configs, we don't have access to the code. So, it's quite difficult to maybe like within one technical report the numbers are comparable, but if we go to a different paper you have the same benchmark, the same name, but can you really

compare those numbers, right? So, this is the question we are like internally handling and we want to like improve the situation for the for the whole ecosystem. So, to give some examples, for instance, if we take a look at SweetBench, which is quite popular benchmark for coding, and it's also quite challenging to run. Um so, in addition to having the number, we would like to know also

like what kind of coding agent harness was was being used. Um also, whether it was run on the all tasks, because sometimes some of the tasks are quite difficult to run. Uh and people just drop them, and but still report the number. Uh and there was it like reasoning on and off? We also need to know this. If we take a look at ATLE, um which uh

is being often reported with tools, we would like to know which tools were actually used. Um if there was like a search tool, so which search provider? We have different ones. Um and also, if we are allowing the model to make tool calls, so how many? Typically, there is some limit on the how many tool calls the model can do. So, those things are quite important, and

the third example from this slide is uh if we have long context reasoning tasks, typically, there is some limit on the max completion tokens. Also, like what is this limit? This is very often not given. Um or um for instance, if we are truncating uh the responses, so not all samples they are getting a valid generation, what is the logic of handling those cases? Are we excluding

them? Are we like just assigning a zero? This is something that it's important to to know in order to make those numbers comparable. And um what we what we believe is the answer to those problems is just like to completely open source not only the models and the weights, but also the the evaluation recipes. And this is what we are doing at NVIDIA. So, as you know,

like with NeMo Trans we are open sourcing the data, the models, also the training recipe, right? But also, we would like to open source the evaluation recipe, including the configs that we are using for evaluation runs, but also the tooling that we are using for that. So, basically, we can hand hand hand hand hand over the the the code base that you can use to run those

evaluations on your end. So, we did this starting from the NeMo Trans free Nano. Uh and then on the slide you can see uh you can see um the the links to the Nano and the and the Nano Nemotron Super recipes. Uh where you can just take a look at the configs and then run this on your own if you have the time and and the compute

and and and see whether we can reproduce the numbers. But we believe this is the way to go and we would like to uh help to adopt this within the within the whole community. And also like so evals can be quite complex but at the same time we think that it shouldn't be much more complicated than you know like installing a tool and then just like running

a one command with some configuration file that should basically allow you to reproduce the numbers. So this is what we're doing as well. So the team that I'm managing is responsible for the Nemo Evaluator SDK which is the tool that we're building for just running evals at scale and also in the reproducible way. Um then some patterns and anti-patterns that we are seeing when we are like

dealing with different teams that are interested in evals and we have a lot of them internally at Nvidia. So not only the teams which are training models but also teams which are responsible for like quantizing models, pruning models and so on. So typically they are working in the setting where they have a baseline model and they would like to do some optimizations to the model and then

understand whether there is a regression. So this is a very typical setting that we have at the company. Uh and some anti-patterns that I can I can call out is that like it's easy to cherry-pick prompts. So always you will find a prompt that results in a slightly better accuracy for your model. a lot of benchmarks are stochastic so either because they are quite hard and this

is why we have like a lot of variability when it comes to the numbers that we're getting on and some of the evals are just they have a low sample size. So we don't have a lot of examples. So also by nature uh the results and also like by by by the by the fact how like LLMs are operating because they are not really deterministic, we are

getting stochastic results. So also like reporting one number like typically the best that you got across a couple of repeats is something that still happens. So, this is like an anti-pattern, like not knowing the confidence intervals, not knowing the standard errors. It's It's then it's quite difficult to compare the numbers, right? Because if you have don't have this information, you only have like one number, you cannot

really tell if this is like a significant difference or not. And And also like not being able to reproduce, there is no like trace, no configs, no no logs that you can take a look and then reproduce the the run. And And no contamination checks, whether the model was actually trained to some extent on on the data which explains why the model is doing better than others.

so, the direction we would like to go, uh and this is why also we are like open sourcing the tooling that we are that we are building internally at the company, is that we want basically to solve all those problems automatically with just like providing the right set of tools. Which means that like whenever you run with our evaluator, you will get like full reproducibility. We are

versioning the evaluation recipes, the hashes of everything that can be hashed on the way, basically. So, you have like full understanding what happened. Uh we support on repeats, the confidence intervals they can automatically compute for you. So, also not only reporting one metric, but in in in in some cases in in in in some cases of some evals, it makes sense to dig deeper into like some

categories to understand more um on a more granular way what are the capabilities of a given model. Uh and of course like significance testing. So, when you have like this regression testing setting, you would like to understand whether there is a significant change that you should um take a look at. And of course open configs uh that other people from the community can reproduce, and also like

help us to improve it, of course, because we are still probably making some um maybe some assumptions which can be can be can be corrected. Uh and also we are doing some research uh uh on contamination detection. This was um one of the things that Besmira mentioned today in the in the keynote um in the morning. Um so, now I would like to switch gears a little

bit to the architecture that we are um that we are adopting for our evaluator, which is the tool that we are using for making a decision great evaluations at Uh, and I wanted to maybe start with some maybe historical background. So, like when evaluation started to become important, so we typically had a setting when we had like a static prompt and some answers and probably like some

CSV file or JSON file. And we were iterating over that to get the numbers. Then it got a little bit more context, so we had like a little more data sets, different data sets, different scoring mechanisms, different extraction logics. So, it kind of evolved into this generation two, which is about um making it configurable and also having a standardized way to run a set of tasks ideally

using one command line interface. So, one example would be like for instance LM Eval Harness, when we can when we have one command line interface to run different different tasks. And what we are seeing right now is like this movement into environments, right? Because this is also um connected with RL, connected with agentic uh systems, Uh, where we don't only have an LLM, we have an LLM

which is capable of interacting with environment via tools. Uh, it's a multi-turn scenario, right? So, the model can take multiple actions and then decide what to do next. And even we can already like foresee that we are we are we are we will have the fourth generation, which uh which is about like platforms. So, think like agentic operating systems, like one example could be Open Cloze. We

already have a leaderboard or like a benchmark which is called Pinch Bench, which is meant to to evaluate a model in in that setting where we have like an LLM which has access to to to not only tools, but also like to to some to to to memory and it needs to basically remember the stuff that we are talking about. So, we have like a cross cross

session memory and multi-session memory a setting which is basically like a stateful agent. So, so a lot of tooling that is available out there is is built mainly for generation two and we are getting more like frameworks that can also tackle the the third generation. But, actually the current models, they are all about the generation four actually, right? So, we also need to be from the evaluation

perspective, we need to be ready to be able to to run those evils for stateful agents that can do a lot of useful things, right? Including maybe calling someone, right? This is also possible right now. what we believe should such a tooling for evaluation should should provide. So, the way we are we are approaching this like from the architectural level so the main part is is is

the environment, right? Which is basically model with ways to interact with the environment via tools. So, this is like the main the main the main abstraction that we are adopting with an evaluator which which also contains the seeding mechanism which is basically providing the task that should be solved along with the answers. Then, the solver is basically LLM with some tools with memory or with some other

capabilities that is trying to solve solve this solve this problem. And at the end, we have a verify function which should assign a score. So, this is what what we have we have this abstraction inside the evaluator and we are able to we are able to run the environments coming from different providers because frameworks. A lot of them are for RL training that are already implementing different

environments and we want to basically be able to like add a layer on top of them to not re-implement the same environment because it doesn't make sense, but just add this we call it like decision great evaluation which is providing you the the statistical rigorous numbers and so on. So this is basically this this part, we are able to call this verify function And on top of

that we can build all the functionality that you need to basically to to really understand whether your model is better than than than other like some baseline, right? So this is what we are building. And in terms of scalability, which is also important for us because you know, some of the evals are taking quite long. So you need to be able to scale them. And the and

and and right now typically evaluations they require some sandboxing mechanisms, right? because LLMs are outputting some code which cannot be really trusted. So we need to have a way to run them safely. So then you need to go with solutions like Docker or ECS Fargate or like also Kubernetes, which is on our road map to to also support that for for basically running those evals at scale

in a in a in a standardized way. And yeah, this is like to show you that right now there is like maybe this sounds a little bit complex, right? But from the API perspective for customers, it can be as simple as you know, like calling a function that you would like to get environment from different providers or you can even bring your own environment by implementing a

simple Python file. It all boils down to implementing this interface. This interface is like this is the minimum the bare minimum that you would need to implement to to provide a new environment. But this is not very difficult. And as as soon as you do this, you are getting all the functionalities from evaluator for free. Basically the statistical testing, regressions, confidence intervals, and so on. So this

is everything and also like aggregations of results and nice logging, which is also quite important for debuggability. And also like understanding what's what are the failure modes of of of of the So maybe to sum up right now a little bit is So this is what you would like to avoid, I would say. And this is like very often what is getting out of the box from

like some of the frameworks that will find out there. Uh is yeah, like the score is just like a number. It's like an average, but you don't know any any more around that. Uh when it comes to comparison, you can just like compare the averages and then say, "Okay, this is higher, so perhaps this is better." Uh and then typically we don't have like out of the

box regression and reproducibility uh capabilities and what where we should go, we should go towards this kind of reporting of numbers, right? So in the model cards it would be nice to see not only the averages, but also standard errors, maybe some confidence intervals along with information how many repeats were done. If you're saying that like a model is better than the other, maybe it would be

nice to run some statistical tests. Uh and maybe report the pin P value numbers as well. Uh when it comes to some benchmarks where we can we have like a natural way to categorize the scores and subcategories, we should also report them. Um like I said, as the community. And again, like for regression statistical-based approaches. And when it comes to reproducibility, this would be this should be

like a standard that we have that whenever you run an eval, you should be able to tell how you run this how you run that eval. So like someone else, even like from from your own team can can can reproduce number, which is very often not the case. Uh yeah, and then one one example how this could look like. So um in the evaluator functionality which we

call gate, which which you can use in a setting where you have like two runs, baseline and a some like target run. And you have some policy that you would like to apply to understand whether there's there's like a significant problem with let's say model under test. But not only a model. For instance, this can be also used in a setting where um maybe we are like

testing a new PyTorch version or we are testing a new VLM version, right? So also very typically in those scenarios you would like to run some accuracy testing on some core set of models which are important to make sure that the framework is supporting. And then then you at the end of the the day, you need to answer the question whether there is a regression in terms

of accuracy. So, this is also something that can be used in that setting. Uh so, you run um like two experiments, right? And using different models or different uh Uh you name it, basically. Then you have some accuracy numbers and there is a problem or not. So, this is also what what you can do with this with this function. Um and this can be typically used in

a CI setting. So, you get like a go no go decision, basically, out of this out of this function. yes. And now coming again back to the agentic evals to uh go over the things that are challenging a little bit. So, we already talked about sandboxing, which is which is very important uh to support. Uh because we have like untrusted model code uh which uh we which

needs to be executed and we don't really know what what's in there. So, this is an orchestration problem to some extent. then those evals are typically multi-turn, right? So, they can they can run for a while. Um and uh and this is like not like a prompt response per, but this can be longer, so it takes some time. Uh and we have like we can have some

timeouts or we need to manage the context window. We need to compact the context window. What is the strategy to do that? Those are the questions that that make this complicated. Uh then we have tools, right? So, tool execution is very common. Web search, access to file system, some other APIs. So, like providing those tools in a in a standardized way to the models is is also

important. And max tool use cap is also something which is typically the case because the model uh and we'll have an example in a moment can like happily infinitely try to make tool calls in a case where it it it cannot really solve the problem. Uh then scale, again. So, very often we have like more than 1,000 uh examples. And each example in a dataset requires a

sandbox, maybe even like more than one. Uh so scaling this to make uh to be able to provide evaluation signal without any delays is a challenge. Um and then yeah, context compaction I talked about it, which is important for like multi-turn scenarios. So different ways to do that. Um and from the evaluation perspective, there should be an easy way to configure the strategy that you would like

to use Um and also we're going in this like generation forward which adds like memory layer so we can we can have some access to them. We can have some databases and so on. So basically to persist state across the the sessions and then we have like we are entering the world of stateful agents, which means that we need to be able to like seed the state

in an in an efficient way so we can run evals from some let's say memorized state not to repeat all the all the all the sessions from from scratch, so to say. and some examples like from from from uh what we saw what can happen if if we are not not paying like enough attention to to how the evaluation can run. So one example that we had

internally uh surrounding a tool using tools. So I talked a little bit about that. And the problem was that the eval was configured that the model could only run use Python sandbox and that's it. So no web search like dedicated search provider. Uh and we had like two samples out of like this over 2,000 where the model it really needed to like make a Google request. And

and it ended up like just like writing via requests sending request to Google and because we were running this on on the cluster so it was detected as a bot by Google and it was like the the model got blocked but it kept kept going. So like we spent like 100k tokens so I can reasoning so I it can't click on some captcha or I don't really

know what it was but basically it struggled to click and then it tried to install DuckDuckGo, do some scraping with beautiful soup, but like all to no avail, and it kept going and going for in this case for like 7 days. Uh and the lesson from that is okay, so like we wasted some GPU hours in that. like more than 500. Uh and it took a long,

so basically the civil was actually blocked by only two cases. So like also some some mechanism to like limit not only the tool calls, but maybe also the maximum time of of a given sample. It could be could be could be could be implemented, and we did that. and yeah, and also in some of the evals you need to know which tools are are required, or you

need to have some understanding what kind of samples are in or what kind of problems are within the dataset. So you can like do the proper tools. Uh provide the proper tools to the model because otherwise maybe the model will try to re-reimplement them if the model can can can can do Python for instance, right? Another another uh that I have is uh from long context reasoning

uh evaluations where uh and this is like maybe a little bit nuanced, but uh I think this is quite interesting is that uh very often when you are like running stuff on compute clusters, you cannot run like for unlimited time. You typically have some like time limit like 2 hours, 4 hours, whatever. And uh very often 2 hours is not enough to evaluate um uh a given

model on a given dataset. Maybe it's because of the deployment. You can always optimize the deployment, but it's not always possible to like have a large deployment only for one eval. So typically some of the samples they will take just longer than 10 hours. You cannot really avoid that. Um and your tooling on on the orchestration layer needs to be able to cope with that. So typically

what we are doing is so when some samples do not finish, so the finished ones we can cache, so we can remember they are finished, but the ones that didn't finish we need to resume them on the on the second uh allocation, basically. And in this particular case we we the situation where a couple of samples they they did they just didn't finish within the first window,

so they were retried again, right? And then maybe within the seconds and they were started from scratch in this particular case because of how this benchmark is implemented. So, they were started again from scratch and maybe even within the second window they didn't finish. So, this means basically that those samples they got a couple of couple of attempts. So, the model had a couple of chances to

solve a particular problem, which probably is already you can already spot the problem that we don't have we don't want to have an eval where uh only got one chance by the LM and another other ones they get like multiple chances multiple attempts by the model. So, of course we can still take a look at those scores and maybe understand what's happening, but this is basically the

infra that we are using to can contribute to the differences between the scores. And the bugging of such things is quite quite difficult. and you you need to have very good logging in in order to to be able to to basically understand what's happening in there. Um So, I I have 2 minutes more? Okay, I think so. Yeah, okay. Uh so, some lessons. so, so yeah, like

understanding which tools should be provided to the to the to the to the to the model, having some like circuit breakers for instance like limits, it's also a good practice. And then making sure that we have very rich logging it's also very important. Uh so, we can understand what happened after the fact and then improve the tooling. And also what we are doing internally at NVIDIA is

that we have like automated like AI agents basically doing monitoring for us. Because what you would like to have is just to check on the logs of the job even if it's running on the cluster, you would like to take a look take a look at the logs basically to understand whether this job is healthy. And some example from from like last weeks from one of the

agents that we have internally is uh Uh, we were able to like okay like monitor a lot of benchmarks, but the most important thing is that that we can like auto- autonomously recover from some of the of the errors which are not cannot be really avoided because of the nature of the of the of the clusters. So, we can have some out of memory errors because of

like just being unlucky in terms how samples are sent to the model. Maybe we had not the best deployment command and so on. So, we can improve on that by by by having this autonomous agent that can recover autonomously in cases where it makes sense. Uh, and we can save a lot of GPU hours by by doing that because we can like fail abort the run early

and then continue and then continue with a with a fixed run and typically like the current frontier models, they are typically quite good. And knowing when when they can fix, uh, like an evil job automatically. So, so let me just go to this slide maybe because I have one more minute. Um, so so yeah, so what I would like to say is if you would like to

give it a try to deploying this fair building, you can you can go ahead and and and use it. And here I'm I'm I'm I'm I'm showing uh, the preview version that we are we have available as of today actually that is implementing the the most of the functionality that I was talking about including the snail gate. Uh, so you can go ahead and check it out.

It will be published probably like on the release branch by end of this month. Uh, and then yeah, once you can either contribute your own benchmark, bring your own benchmark. We have functionality for that. Uh, or you can run the the divas that we already, um, provide you within the uh, within the tool and and yeah. And check it out. We you can take a look at

the GitHub repo. And if you if you like the tool, if you think that you can use it in your workflows, you can you can maybe give it a star. Uh, and also like you can you can you can reach out to us. Uh, you can reach out to me or or like the rest of my team, uh, via email or LinkedIn. Whatever works for you and

we are happy to to collaborate on the evaluations and then to to to help to improve the situation for all of us. Basically. So, yeah, that's it. Thanks a lot. Do we have some time for questions? One minute, okay. So, maybe the questions if there is someone. Yeah, go ahead. Hi, good great presentation. On some of the last slides you mentioned that you caught some distinct failures

on the left side of that slide you also mentioned logging. So, I would like to ask how did you recognize that those were distinct failures? Did you do some comparison? Did you have agents compare against the list of failures during your evaluation runs? Uh, good question. So, um So, for instance, so we take put a lot of effort into making sure that on the evaluation side we

are emitting logs. So, for instance, we we even contributed to open hands to improve the logging of Swi Bench. So, to make sure that we have all the information. So, for instance, whether there are some issue with getting out of context and so on. So, this is what we care about. And then if you are in this situation, typically just like asking Opus or GPT 5.4 to

take a look at the logs and then see whether everything is okay. It typically enough. Like it doesn't always solve all the issues because still you need to ask the right questions and sometimes just asking to take a look at the logs is not enough because for instance, we are not expecting the models to know about our the infra that we have internally. But if you provide

this context as well, you it's starting to be quite useful. So, one more question I guess maybe we have Okay, so I guess that's that's it. So, thanks a lot again. Cheers, bye.