From Responses To Trajectories: Multi-Turn and Multi-Environ... Kashif Rasul & Sergio Paniego Blanco

This talk, presented by Kashif and Sergio from Hugging Face, focuses on the evolution of reinforcement learning (RL) post-training for large language models (LLMs). The discussion emphasizes a shift from optimizing single responses to optimizing interaction trajectories, which is essential for tasks involving multi-step reasoning. The speaker explains how the Generalized Reinforcement Policy Optimization (GRPO) algorithm integrates with Hugging Face's TRL trainer to support multi-turn and multi-environment training. They highlight the importance of having environments with standardized interfaces and effective runtime orchestration to facilitate the training process. The session includes a demonstration of using environments such as Open Env and a detailed exploration of training setups that support both text-based and multimodal scenarios, including autonomous driving using the Carla simulator.

My name is Kashif, and this is joint work with Sergio here at Hugging Face. Uh so, yeah, this talk is about uh shift in RL post-training for LLMs uh from optimizing single responses to optimizing trajectories now. And if we care about tool use or multi-step reasoning, browsing, coding, simulators, then the unit of training can't uh really just be one prompt, one answer anymore. It has to be

an interaction uh trace uh a trace-based interaction. And we'll look at how the GRPO algorithm in Hugging Face's TRL trainer supports this uh setting, uh including multi-turn and multi-environment training with systems like Open Inf. Uh so, let's dive in. Uh so, post-training, as you maybe saw in the keynote, has been going through a fairly clear progression. First, in SFT, we learn from demonstrations, given a prompt, uh

essentially imitate a target response. And then in algorithms like uh in then in uh paradigms like RLHF and DPO, we still stay in that mostly single-turn setting, but now we're optimizing based on uh you know, preferences over responses uh uh or or rewards. But the basic unit is still one prompt and one completion. And then the next step in that evolution, and really the focus of this

talk, is uh GRPO on trajectories, where the unit of training is no longer a single response, but an interaction trace. You have action, you have observations, action, observations, and at the end you eventually get a reward. And this shift from isolated uh outputs to full rollouts is what forces new training patterns. Uh so, once we move from this uh single response to trajectories, the challenge is no

longer just the GRPO objective itself. We now need a real uh runtime loop, where the model takes an action, and environment returns an observation, and that uh interaction can continue for multiple turns before we compute reward and update the policy. That means we need a few things bunch of things at once. So, we need environments with a clean reset and step interface, correct masking, so we only

optimize model generated tokens, and rollout generation that is fast enough to scale to real training runs. So, in practice, generation can really become the bottleneck so systems like VLLM matter as much as the real RL objective here as well. So, this slide really is the roadmap for the rest of the talk where we'll talk about environments, we'll talk about this runtime orchestration, masking, and then talk about

scalable rollouts as well. Uh so, now that we've set trajectories training needs environment, the next question is what these this environment layer actually looks like, and in we're you going to look at the Open Enve environment from Meta, but obviously there is many other environments including those from Prime RL as well. Uh that can work here as well. So, the main idea is to give agent style

environments a simple standard interface. You reset the episode, the model takes an action with steps, and the environment returns the next observation along with the reward and the state over time. Uh so, this matters because once environments expose a common interface, the training loop no longer needs to be written for every new task. Uh the same pattern can be used for very different settings from games and

coding environments to browsers and simulators. And operationally, these environments can be run locally during the development, packaged in Docker for isolation or and security, or deployed to deployed to Hugging Face Spaces for remote access and and sharing. Architecturally, the Open Env essentially separates training from environment execution. On the training side, the the training libraries like TRL interact with an environment client through a standard API. Uh that

API talks to a environment server, typically over a client-server boundary or a protocol like WebSockets. And each environment runs its own isolated container or service. This isolation is useful not just for safety or flexibility, but because the same training loop can talk to very different backends while keeping a consistent reset step state interface. And for GRPO, that matters because we often need multiple environment instances or concurrent

sessions in parallel to support rollouts generations at training time. So, up till this point, we've talked about why trajectories need environments and how OpenEnv provides uh the execution layer. Now, we can connect that back to GRPO. The important point is that the optimization rule itself does not fundamentally change here. GRPO still gives us the same basic learning recipe. We generate samples, we score them, and then normalize

those rewards into advantages and update the policy. What changes essentially is the trajectory set in the trajectory setting is the unit of training. Instead of one completion, we now train over multiple multi-turn rollouts, so the hard part becomes orchestration. Environments, tool execution, mass batching, and and this the problem of delayed rewards. Uh this is exactly the layer TRL is providing out of the box. So, what TRL

does is essentially expose this environment factory pattern, uh which is a default TRL pattern for based GRPO training. Instead of writing the the whole loop yourself, you define a lightweight environment wrapper class. Here's an example of a class, and this class initial initializes any client or local it you know, provides a local state, implements this reset and and a start of a new episode helpers and exposes

a public method in in that TRL turns into callable loops. And so during training, the GRPO trainer creates one environment instance per generation, calls the resets, then lets the model invoke these tool methods, and then reads the reward back from the environment instance through the reward function. So, the main idea is that you describe the environment interface, and the TRL handles the multi-turn orchestration. So, here's a

very simple minimal concrete version of this pattern. The environment wrapper owns a client to the back-end environment, and a small amount of local state is also defined here, just the reward. the reset essentially starts a fresh episode, and return guess becomes like the tool the model can call during its rollout. When the tool is executed, the the step we step the environment, store the resulting reward on

the instance, and then return the new observation to the trainer. And then the reward function simply reads the environment reward back and each environment for each environment instance. So, this is the essentially the whole contract that you need to define. Reset method, one or more tool methods, and a way to expose the reward back to the trainer. So, once we define such an environment like this, the

next question is what TRL actually does with it uh during its roll So, inside TRL and uh remember you don't need to uh worry about this. So, what's going on is that you have a there's this core run time uh loop that TRL handles for you. At the start of an episode, it resets the environment and you get the initial observation. The model then generates a completion.

TRL parses any tool calls, executes the corresponding environment actions, and appends the resulting observations back into the the context. And then it generates again, and that continues until we have no more tool calls or the episode is finished or done or we have hit a maximum uh you know sequence length. So, with this environment factory pattern, you define the environment interface, but you don't have to you

know write all this uh orchestration loop yourself. So, once you have the loop, um essentially the next uh issue becomes uh you know which tokens should contribute to the loss. And this is where masking comes in. So, once we uh move to this multi-turn training, the sequence uh contains two kind of uh texts, the tokens generated by the model and the tokens coming from the environment or

a tool. And we want the model to contribute on both uh to to be conditioned on both, but we only want to optimize on the tokens the model uh itself generated. So, TRL applies a mask, this mask token, uh uh are applied in the loss, uh where the environment tokens are masked out. Uh so, conceptually you can think of this uh as the environment response becomes essentially

a context for the next uh action, not a target for the model to learn. uh can run uh and mask a multi-turn episode correctly, the next step essentially is that of scaling this uh this uh recipe to beyond a single environment and train across multiple turns in the same run. And this is where Sergio will show you how to do that. >> Okay, so till now we've

seen is this contract that TRL makes with the environment factory and what it's interesting about this pattern that TRL exposes with a GRPU behind the scenes is that it can even scale not only to one environment but to many at the same time. So we can even scale and try to train a model directly in multiple environments as we can see here. So we can think about

having a multiple environments like a water or catch catch at the same time. And what we would need to to expose in this case would be just the the tools for each one of them. So we can see that we would have just the the difference here would be we would see later on how we do it in code but we would need a data set with

for each of for each of them for each environment and tools for each one of them. And then rewards for each one of them as well. Let's see that directly in code since it's easier to understand. What we can see here is exactly the same pattern that we were talking previously. So we would have the the same class but in this case instead of just considering one

environment would have a class that wraps both at the same time. And we can see something that it's interesting here. So we have two methods one for each well in this case since we have only one tool for each environment we have two methods. So one would be guess for for the water tool and then move to the to the other game, but it's still what we

can see below a little bit below is that how do we manage this multi-environment setup? So, what we need is just the creation on a of a data set. For example, with prompts divided for each one of these environments that we will be using during the the training loop. And then if you look above uh to the reset method, we can see that it now it now

now has these active active environments, while probably prior we only had one. propose this way of doing things with the environment factory factory, but in case that we would like to use like let's say to have the whole control of the generation pipeline, we can still use something that it's the rollout function, which is another functionality inside GRPO inside TRL. And this can be interesting, for example,

we when we are training models for another type of frameworks. In this case, we have only introduced OpenM, but we can also think about Nemo Jim by Nvidia. So, in that case, we can directly use the rollout function if we want the full control of the generation pipeline. Okay, so we have we give one option with the environment factory that supports multi-environments and the whole functionality inside

OpenM, but if you want to still have the whole control, you can still use the rollout function and you can generate generation pipeline yourself. Until now, what we've seen is just text text-based scenarios, but why not going to a multimodal environments? For example, there are lots of use cases and super interesting ones like a browser when you just want to have a screenshot of the situation and

based on that, you would like to make the the tool calls of the actions. For example, we can see here what that would look like. So, for example, we have a BLM that wants to click a button and then the environment what it would give us back would be not only the text-based observation, but also the the image. In our case, that would be the the screenshot

of how the browser looks like. And based on that, what the BLM that we are could just scroll down or whatever. Okay, but the idea here is that TRL directly supports this kind of setup as well. And now we're going to connect that with a concrete example of how this would look like in practice. In this case, what we developed was an with an autonomous driving simulator,

which is Carla. Carla is an autonomous driving simulator based on on Unreal Engine 5. You can see on the side how this would look like kind of. Okay, so it's Let's see if you don't If you are not familiar with Carla, it's a game where you just drive through a town with many pedestrians and and vehicles on on the way and the idea is that you just

go along the way without colliding with any vehicle and just following the the the driving rules, of course. In this case, we have simple scenarios. So, this what you see on the side is a Carla, which is supported in Open MS. We have this this wrapper that controls the the the the of generating a training loop using TRL and we have this scenario for example. So we

are heading towards pedestrian and the idea is that we just do a lane change and we stop the the the vehicle so we don't collide with them. Okay, so the model must observe reason and act to minimize harm. In this case if we go directly to the this example we can think about the possible tools so we have three here observe which would give us back the

look of the scene if we are talking about LLM that would be a text based description of how the the scene looks like. But if we are directly consider a BLM that would be for example the camera sensor image retrieve directly. Then we have the emergency stop and then we have the lane change that just moves to the to the other lane. The idea here is that

we provide these tools to give we can all be easily just extend that to have more tools. So we what we present here is just a simplification of what does hope this could look like. We are not considering the assimilation that could be asynchronous so we just move one step then stop the simulation do an inference and then based on that inference we resume the the simulation

for for one step. In this case we are assuming that for a given number of steps but the idea is that we are not considering inference time for these scenarios. And which is interesting is that we are running this example directly remotely. Okay, so we have this Carla simulator heavy in in computational needs and we are running it directly inside HF spaces remotely without even considering infrastructure.

Okay? So, we just deploy that and um give a uh GPU, and that's it. We don't need to uh run that on our own uh Um this is how it looks like uh in code. We have seen already some examples. So, the idea here is just to um go directly and and see um the example for for this uh environment. We have the three tools that we

were considering. So, observe, a emergency stop, and lane change. And we are resetting here the scenario uh for the the one that we were uh talking uh um this is a super simple scenario as we were um as we were talking, and the behavior can be um learned super super quickly with us a rather small um model. So, for this uh use case uh and this experiment

that we run with a QN3 uh a 0.6 billion parameter model with the RPO um in in 50 steps, we are um able to just learn this this um this behavior. Okay? So, just uh what you can see in the in the GIF on the side is how the the model would behave in this in this scenario. So, it just uh moves away to the other lane

and just uh stop. Um if we take a look back uh we took a look uh below the the GIF, what we can see there is how the the observation and the the two calls would look like. So, we have uh four steps here. In the first one, it would just observe the situation, and uh the the simulator, the environment in this case, would give us back

the textual um description of the situation. But if we are all considering a BLM scenario, that would give us back the images from the camera if we have one camera and then based on that with the LLM or BLM in this case decides is that it needs to stop so it just go through to a emergency stop. We apply that then in the third step it moves

away to the to the other lane and it finally stops in the in the fourth step. As you can see here we are only making four steps so we are doing an asynchronous simulation but applying that for a number of steps so we don't do an inference for each one of the steps since that would not make sense. Okay, and this is also supported for when I

was talking about LLM scenario but this is also supported for BLMs. We have released sample scripts so what you have here can be directly run. And we will present the the examples later on and this can be run for both LLMs and also for BLMs since we released this last week for when 3.5 and and Gemma 4. So just this this will be the the full picture

what we started with single turn situations and then we moved to more complex scenarios with rollouts with environments and what we have here is inside TRL we have this full support that we are still working on with support with this environment factory but as I already mentioned we also have the rollout function that can be used as in one run. We have a multi-modality support with BLMs

plus images and as Cassio said, we have many many environments in in Open EM which give us many possibilities with browsing, with with coding environments, etc. Not only toy examples as the ones that we have presented here, but we can scale that to to real world use cases. And we can use these LLMs and BLMs to drive, to play, to to browse or even to to code.

And this is a still the a work in progress. So as I mentioned, for example, this example that I'm presenting here was released the the the BLM version was released and we are still working on this. This is obviously a work in progress. We are working on more environments in Open EM and for example, we released a couple of weeks ago and the async ERPO trainer, the

first version. We have support I haven't commented but we have support for setting and well for generating and setting the the trajectories that we generate during training on the hub. So we can for example train on those afterwards with without the the environment. We are working on richer multimodal and agentic workflows as well. So essentially we are just getting started on on these environments world. Okay? And

here we have both examples that I was mentioning. So the LLM example for Carla and then the BLM example the same for for Carla. Yeah, we have both inside TRL. You you go to the QR code to see directly the code. And we have not only these examples, but a lot of examples. So, if you want to try any of them, um um we just expect that

you do um find them useful. Okay? And that that would be it for from our side. Thank you.