Sandbox Operator: Enabling Session-Aware, Efficient MCP Tool Execution... Mingshan Zhao & Zhen Zhang

This talk explores the development and deployment of AI agent applications on Kubernetes, specifically through a solution called Open Cruise Agent. The speaker, a developer from the Alibaba Cloud Container Service team, addresses challenges that arise particularly with users who are new to Kubernetes, such as AI scientists. Key topics include data security, state management, and cost control within sandbox environments for AI workloads. Open Cruise Agent introduces a custom resource to manage the lifecycle of sandboxes, enabling efficient execution of agent applications while leveraging Kubernetes' architecture. The talk also covers advanced capabilities such as checkpoint and restore functions for better resource management and integration options for AI developers.

Good morning everyone. I'm a developer from the Alibaba cloud container service team. Over the past year, I have been mapping Alibaba cloud user deploy AI agent applications on Kubernetes. Those user usually have the same needs large scale and low latency. But that's a big difference now. Many of them more are new are new to Kubernetes. They are AI scientists who don't know much about Kubernetes while solving

their problems. We will learn a lot and that's exactly what I'm going to share with you today. Next, let's take a look at the outline. First, business challenge of agent sandbox with containers. Second, container based agent sandbox solution called open cruise agent. Third, how it fits into the weather ecosystem. Fourth, the key technologies that make it work. Finally, our future map road map with AI agent. AI

has moved beyond just answer question. Now it can understand what users need, break down problems and actually get things done. is not just for chatting anymore. I can it can do a real work. Let me give you an example. A user ask help me debug the code and select the uh environment. First the agent call RM to think through the problem and break it down into specific

uh steps. Then it use a sandbox to run the code and do dynamic debugging. Based on the result, it calls RM again to try to optimize and fix the code. It runs the code again. And this loop continues continue the code works as expected. Finally, it shows updated code and a summary to the user. This is a typical agent use case. Let's uh uh let's take a

look at the business of challenger this scenario brings. The first thing is the data security. In this example, the agent needs to access the user's code where is running. That code is a assert for the user. So it needs to be strictly as related to print any links. Also the agent needs to debug and run that code as the users requested. But that code could do harmful

things. If we don't isolated it uh properly, attacker could generate and run malicious code like delete or try to break down to the host and do another uh bad stuff. So the computer environment need to be safely isolated. Second, there are massive numbers of agents in the agent error. Every user will have multiple agents and each tool call will run it its own independent sandbox. So the

number of agents will grow exponentially. The third thing is state management and cost control. A sandbox life cycle can be really short, just a few minutes, but it can also last hours, days, or even longer. Keeping our sandbox alive all the time would cost a lot. Agent often spend a lot of time waiting for input from users, tools or art agent. That means agents need to manage

their state well so we can keep cost under control. Based on the needs of state management and cost control, we divided the sandbox life cycle into six states. Three of those states pending, running and complete are consistent with the Kubernetes port. Then we have pause and resume. When sandbox is idle, we release it resource to save cost but we keep its data so it can resume later

with the same context. Also uh for our scenario uh we need to create a checkpoint at a a certain stage. That way we can branch out into different training paths from that P point. Because all because of all this regular Kubernetes port alone cannot handle sandbox uh directly over the past year. While helping user deploy agent application uh on Kubernetes cluster, we will build and refund a

solution and share it with the community with as uh Open Cruise Agent introduced a new custom resource called sandbox to deploy and manage sandbox instance but it doesn't replace Kubernetes P. Uh instead it's a hair level uh abstraction that extend capabilities like a checkpoint and restore and work together with the port life cycle. This let us run run agent uh application as containers on kubernetes. In short,

open course agent act as a bridge between AI agent and kubernetes helping kubernetes become infrastructure uh ready for the for diff for different types of user. Open Cruise Agent offer to integration options for ASentientist. We pride for capability with uh we we provide for compatibility with uh popular community standards like E2B and MCP. Uh you can just use the P SDK to manage the entire sandbox life

cycle. For platform developer, we offer standard uh Kubernetes custom resource uh sandbox claim and the sandbox set. Sandbox claim uh not only claim sandbox from one pole but also support in place update and the dynamic volume mount. Basically a good way to run sandbox on kubernetes. Open cruise is a sin self in contribution uh project that started in 2019. It comes it comes out of Alibaba's large

scale uh coordinative practice and is now widely used in the community. Agent sandbox is our new exploration AI agent scenario and is already open sourced as a sub project. The first opensource version has a three uh core capabilities. First it pro a sandbox manager that's a capability uh that's compatible with the uh E2P and MCP protocols. So AI scientist can get started quickly. It supports uh for

sandbox life cycle management pending running post complete uh to boost the efficiency uh it provider one p to cut down code start time meeting AI agents need for faster resource delivery. We will got more futures coming soon. For example, in the full sandbox life cycle, we're adding checkpoint and restore capabilities for R scenario. Uh this will make sandbox reuse and tax parallelisms uh much more efficient. Right

now the community is still uh figuring out the uh best way to handle agent sandbox life cycle. Uh there are lots of different ways open course agents design is driven by a real world requirement uh from Alibaba cloud user especially large scale uh low latency AI agent workloads. At the same time, we believe that over time as a community grows, a agent sandbox will eventually settle on

a more unified standard. That's why we are always think about how to uh stay compatible with the community and active participating in define sandbox standards. If one day the uh SIG agent sandbox standard uh become stable and its API meet all need open cruise agent will support that too. I will giving you a high level uh or a real of open cruise agent. Now uh let me

look at a a live demo to help you understand the product better. uh as you can see the E2B pass SDK as you manage the entire life cycle of a sandbox uh with post resume It release resource when the sandbox is idle and preserve memory when restored. Uh so the context stay intact. Uh you now have a basic understanding of open cruise agent. Next I will hand

it over to my uh colleague for a deep dive. Uh hello everyone. Yeah, I was I will start with the uh ecosystem integration and because we have to uh think in the uh think in uh in the end because we have two important use cases. One is that is reinforcement learning to train for for computer use. Yeah. We in this uh use case we have used the

VI training framework. Yeah. Uh uh we actually expose uh and in our open cruise man sandboard manager we expose several uh useful uh useful tools such as uh code uh run code and command running and the file operations. Yeah. Uh through the MCP server uh uh the VR has a has a natural integration with uh with MCP. So uh we can just uh enable the MCP search

so that it can discover our our MCP uh tools. Uh so so that when we expose uh submit uh a job in raid to uh to run the vio based jobs drops uh can use the our our tool to run code in isolated and secured environment. Yeah, in this use case uh actually we have uh several key technologies and we haven't mentioned that. First is that we

use a warm pool like like 2,000 of sandboxes and in that is that because in for many of the V uh IO training uh cases for example when we use the SW bench as the training data we have to we as a mostly practice practice practicer will just use the image as a as a source of data for training. So basically uh when you use the warm

warm pro you have to update the data with uh with with the corresponding image. So we have a in place update with the image so that it can different samples can run different uh the different data to train. Yeah. Another is that the sample sample set based warm pool will just refill the warm pool uh for every uh clean samples. So so that you can just keep

key the warm pool as as poor and possible so that when we run multi round of bouch training when one batch is finished basically the sample pool is already refilled. So in the next batch basically you can just get a very quick sandbox provisions. Yeah. Another case is very uh is for open core. Yeah. Uh open open crew sandbox agent sandbox can just accelerate open core deployment.

Yeah. Uh actually open CL is quite popular in China and many of them has a personal coupon installation. Yeah. Uh in these cases our users will have a very tight uh schedule to to go production and they also very like to have a the resource uh to keep a reasonable uh level. Another is that uh they are not very tolerant to to uh the code starter latency.

Yeah. So we uh we enable the user to use our E2B based integration so that they can just integrate with E2 SDK very easily. And the third is that that they can when the one problem of open crow is that it runs agent just inside the sandboxes. So the agent will need to uh to to inter interact with the RM but the the API key to RM

must to be kept as safe as possible. So we just use the file system API of our samples runtime which we'll mention later which will just uh inject the the personal API key um to the to the sample boxes. Another ca another uh useful features that we use here is that uh when the sample when the open claw uh is is idle for example uh the user

has doesn't have any any instruct uh instru instruction for the open core yeah it can just sit idle and just eat all your CPU CPU resources so in this case when the open core is is is idle we can just automatically suspend the symbol boxes and when does uh when this does uh when the user just uh come back Yeah. And he can just uh when he

chat some prompt in the uh in the in the chat in the chat application then the the the sandboxes will get resume. Yeah. And with the previous uh saved file system. Yeah. Yeah. Lindon let's give a deep dive of all the technology we used in the uh open cruise agents. Uh first is that actually we provide uh two kind of APIs. One is is MCP uh which

is quite useful for uh free uh agent related framework integration. Another is E2B uh which is also quite useful for the a for the sandboxes. Yeah. uh for MCP we just manage because we just provide the uh several welldefined uh tools such as uh code running and the command running. So uh it doesn't in introduce uh the idea of a sandboxes instead we introduce the idea of

a session. So every new user will just get a new session initiated and when the session expires we will uh automatically clean clean up underlying uh sandboxes. Well in E2B we have intro uh we will have exposed the ID of a sandbox. So so the user can just use a samples ID to operate the sandbox like just manually resume uh the pause and resume the sandbox or

just create a snapshot of the sandboxes. Uh here we also introduce another uh uh component called sandbox gateway. Uh because we remember that we have run uh unsecure code in our sandboxes which is may be generated by the malicious prompts from the LM. So we have to we have used uh secure uh container runtime to ensure the runtime is secured but we also use uh technology like

uh traffic uh network policy to ensure that the network is well protected. So we ensure that all all incoming uh traffic is coming from the sandbox gateway. The sample gateway is a just efficient uh gateway uh powered by uh envoy filters technology. Uh it is aware of the sample state like paused or pause the state. It can just use the it can just translate the sandbox it

directly to IP uh without any overhead introduced by by ser community service. uh for sample one poor uh basically it's just a a a set of the sandbox that is up and running uh so that we don't have any trouble with the code start uh in actually uh why we introduce one point is that not every uh cloud vendor or just installation of kubernetes have very high

uh port uh port provision speed. Yeah, especially in the for for the for the secure container runtime there for for example in ka container because every uh sandbox has its own uh kernel is so so that it needs some time to start start the OS of the corresponding sandbox. Yeah. So it maybe starts uh a little bit uh slow and and actually even the most uh efficient

micro VM such as uh firecracker just doesn't doesn't have enough speed to cross the boundary of like 200 milliseconds of this uh port generation. Uh so with the sandbox uh samples warm pool actuallying our our load test uh we can reach for uh like average latency of 20 milliseconds uh which is which is very crucial for our for for the agent serving uh use cases. Yeah. Actually

in our uh load test we have a generate sample clean operation like 25 20 50 kil uh curies per second that creates around like one uh 10,000 of uh the use of pooling also introduce some problem because uh actually the user request for sandbox is quite contigious. They have maybe have a different images uh they may they want to introduce uh mount some uh shared storage object

storage or file based storage uh to to do some backup or to share some datas also they want they may have different kind of uh resource requirements. uh the user of pooling may introduce problem because we have may have only limited number of worm pool. So to overcome this problem we have introduced technology uh like in uh in place image update uh which will just when after

playing the port from the warm pool we can just change uh change the image of the container. Yeah. So that we can just have very few kind few warm pool to in to to meet the request of multiple uh image. Another is that we have a dynamic storage mounting capability. uh in which case we can just uh during the sample claim uh when when we when when

we get the the sandbox uh we can just dynamically mount any uh shared storage like just not any NAS or object storage like yeah another another technology we plan is that we will do some implant resizing of the container uh so that the sandbox can just uh can scale up. So when when the resource requirement is not the same as as as the one in the warm

pool and below is an example of of using E2B SDK to just create a sandbox and just changed uh changed the the container image well as as well as mounting uh another storage. Uh another technology we use is the pause resume capability when when the sandbox is idle. Uh we can just pause the pause the sandbox and when the port is uh is when sample is paused

we actually just delete we will delete the port from the kubernetes and we we also ensure that the uh desired state is persist uh in some uh persist storage. uh we have currently support uh the memory file system and also the plan is that the IP and the GPU memory is also uh future directions when the port is resumed uh we'll just uh just copy uh copy

or just use the the what the save the states in the persistent share storage and just create them from uh from from the snapshot. Yeah. Uh here just we have to be be honest that currently there's not no vendor neutral implementation for the position capability and we are working with the uh the checkpoint and the restore working group to ensure that the vendor neutral uh such kind

of capability the API is exist. Uh another another useful feature is that is the checkpoint capability. Uh the check point is that when when a port uh for example the risk cases that for example in the open claw installation uh one may for example when open claw just initiate uh have a a new uh new version we may want to update it. So so before update we

have to ensure that we have a we have a we have a safe way back. So we can just use a checkpoint to uh to give to to back up all the states including the root fs and and the memory. When we have finished the checkpoint, we can safely delete delete port uh and and do some upgrade. And if the upgrade upgrade fail we can just create

a sandbox and point it to to the previous checkpoint so that we can come back with the previous state. Uh the checkpoint we have a uh a corresponding customer resources. uh we can you can just uh in in uh instruct the controller to via via CR to to uh the what what kind uh what what is the target sandbox and whether or not to just keep the

sandbox running after the checkpoint. Uh actually we have a just kiosk in the project pavilion uh these days. So if you have any question we can um meet up in the kiosk and also have a another talk uh named operationalized AI workloads on kubernetes with open cruise which will introduce uh day2 operations that useful for training and inference also the agents. Yeah. Uh we also have some

road map that they planned. The first is that we have to uh the the checkpoint and restored capability for for for Kubernet the cloud native is not standardized yet. So every vendor have its own implementations. Uh yeah so we are working with the working group to uh to come up with a new uh standard API and maybe when the API is ready yeah we can integrate the

new API in the in our operators. Next is that we have a a plan to do a universal uh samples runtime uh because the sandbox needs some uh capability that like a code running and command running and then file system operation which is not which is not uh it is supported not by the API server instead it is in empowered by the demon inside the port. Yeah.

So we plan to just extract the demon outside the user app containers so that it can be run uh without any user modifications. The third direction is that we plan to uh to make the Woo more cost efficient uh so that the worm pool can be autoscaling autoscaled uh or scheduled to to to scale the W pool up and down or use in place up in place

resizing to ensure that uh the the worm pool will kept the minimum resource requirement during idle time. Yeah, that's all our our talk.