Lightning Talk: Full-Stack PyTorch Robotics VLA: From Data To... Samet Akcay & Dmitriy Pastushenkov

This talk introduces a new full stack robotics vision language action model-based framework developed by Intel, which simplifies the robotic development process from data collection to edge deployment. The speaker explains how traditional robots required hardcoding for specific tasks, but advancements in AI allow for learning capabilities through teleoperation. The session outlines the framework's lifecycle, highlighting data collection from robots equipped with cameras, followed by policy training, optimization, and deployment stages. Various user interfaces, including a graphical user interface (GUI), command-line interface (CLI), and application programming interface (API), are presented for different levels of user expertise. The talk also delves into model deployment using Intel's OpenVINO, which optimizes and runs models across different Intel hardware architectures. Lastly, the coordination between training and inference utilizing Executor Torch as a backend is discussed, providing a seamless workflow for developers.

I'm Summit and together with my colleague Dimma, we're from Intel. Uh we're going to be talking about our new framework uh full stack uh robotics vision language action modelbased uh framework um all the way from data to the edge deployment. So we're going to be having some uh journey. So what's what's this framework? Uh if we think about robot development, it's not new, right? It's been here

for quite some time. But traditionally speaking, uh robots were hardcoded uh for certain tasks. And if you want to carry out some uh another task, you need to um redesign it, hardcode it again and again uh which takes uh weeks if not months. But with the current progress of uh AI uh with the vision models, visual language models or or vision language action models, it's possible to

add the learning capabilities to to robots um basically via this tele operation as you can see on the right hand side of the image. And this is what we're doing. If you talk about the the overall life cycle of physical AI development uh and robotic learning um it basically starts with a robot and camera as you can imagine. So you have a robot, it has cameras and

other sensor modalities where we use that and we collect the data from the robot but via teleoperation and the next step is um once we have the enough data we can train a policy similar to um other uh model training and other modalities and um after the training we have this optimization and export stage. Then finally once we have uh an optimized model or a policy uh

we can just do the deployment for the inference stage. So this is kind of like the overall life cycle um in in general and what we do here is we provide this framework um that gives you the endto-end robot learning capabilities um all the way from uh a guey a CLI or API and u I'll just talk uh more in detail shortly. So let's have a look

at this guey experience. So you can uh basically uh use this guey. You don't need to know anything about robotics. Um you can just uh utilize um by calibrating the robot, uploading the data uh like this as you can see. So this is done via tail operation. So you can visualize your training data and once you have the data, you can just fine-tune your VA model. Uh

it takes a while. It's not this fast. Uh obviously it's sped up. Uh and then finally once you have a train policy you can just do the the inference bit so deployment so it's all the end to end life cycle you saw in the previous slide um you can just uh do that with this UI um so this is a um a quick video from uh embedded

world uh from last month for those who who are not familiar with embedded world is kind of like the champions league of the edge um so it's so we were showing this that we can use this UI to collect the data and also do the deployment. So that video was showing you that how you can do the deployment uh for a simple uh pick and sort task.

So that kind of shows you how we can do this uh with the the GUI application. But if you would like to dive deeper into uh the codebase uh it is also possible that we provide a CLI and a API experience as well. So let's say you want to use a CLI. So you can have a config file um so provide the data config um like what

data you would like to use what are the configuration like arguments and so on you can provide the the model configuration like pi05 which is uh kind of the state-of-the-art uh right now uh in the literature and then you can just call oneliner uh CLI command um physical AI fit pass the config file and you'll be able to train uh a policy if you want to use

uh an API I um more like a Python development. You can do the same thing pretty much the same thing uh onetoone mapping. So like import all these uh classes all the way from data module to model and the trainer and instantiate all these classes and then you can just do trainer.fit um pass the policy pass the data and you'll be able to train a a VA

policy. So let's talk a little bit about the the high level architecture. So obviously we start with PyTorch uh on the uh lowest level uh layer. Uh we add PyTorch Lightning to um hide away some uh abstraction and then um everything is on top of uh these four main components. So we have the CLI layer for uh fit, validation, test and prediction entry points and we have

the trainer um that gives you all this um boilerplate components like multiGPU distributor training, checkpointing um adding logging and all these custom callbacks and um the data module is training to to validation including the gym environment and the policy will give you um uh the API components. So you can use our first party API or if you would like to use uh the amazing Lurobot framework that's

out there and everyone's using so we can just use uh the Lurobot policies of the shelf uh and use it within our um within our pipeline. Um so this kind of has um all the uh best practices. So traditionally this is kind of like a um a conventional way of uh doing the training right uh but we also need to handle all these uh distributed training and

mixed precision and all these uh extra bottleneck right so we um basically using the lightning abstraction uh it's uh kind of hides all this complexity so I can just create the class uh using the policy uh add the training step validation step uh and so on and we can basically add the accelerator whether we want an XPU, Intel GPU or Nvidia or AMD whatever we want um

um number of devices how uh the precision is the mix precision or full precision and so on. So like all these details you can just pass uh and then lightning kind of um abstract uh hides away all these boilerplate stuff. So this would mean that we can focus on the research and the product development but not the uh boilerplate infrastructure. So we give you all this So

I showed you how we can do the training. So um we have the the training done and now what? So the next step would be um we want to be able to just um optimize and export these so that we can use it in inference and my colleague Dimma will talk about how we can do this export and dimma. So just you want to talk about it.

>> Thank you summit. uh uh and uh hey everyone I'm uh I will presenting about the inference part of the story and how at Intel we deploy the uh models and how we optimize uh the models and uh our main AI framework for the model deployment uh is called uh open uh uh veno as you can see uh here open uh veno is able to work with

uh different uh model uh uh format And of course with uh PyTorch uh this is like the most uh uh form uh the most popular uh uh uh model format right now for the um uh GI models especially but also for other uh models uh uh types and uh uh uh getting model as input. Open veno is able to optimize the uh model. First of all, it's

uh uh uh done uh using the uh tool which is called insafe uh by quantization or weight compression uh techniques to reduce the memory and in increase the inference uh uh uh speed and also uh for with optimization we utilize the uh all uh uh acceleration capabilities of uh Intel uh hardware and uh then uh model is able to run on CPU, not only x86 from Intel,

but also on ARM CPUs for example and we are able to deploy the models on GPU, integrated GPU, discrete GPU, on npu and even on uh if IPJ and always done with uh standard uh API on uh operating systems like Windows, Linux and uh Mac OS But uh open veno can be used not only as a standalone tool it only it also can be used as a

a back end and uh the great example of using open vo as a back end is execute torch. uh uh at executor was presented several times uh already and tintel we support exeutor as well and there are two uh phases u uh if you want to deploy uh the model using execute torch uh first phase is compilation phase and uh uh basically uh uh the uh model

which is represented in pytorch is exported to executor intermediate representation using torch uh dynamo. Uh then uh there is a open winner quantizer which is b based on ncf tool and using open winner quantizer we can uh quantize the model or perform weight uh compression. Uh after uh that uh uh uh open veno is able to analyze the graph and decide which operations uh operators are supported

by open veno and which are not and uh then we uh get the uh final final uh load lower graph and uh uh put it in the uh so-called PTE uh uh uh file and with this uh uh file Well, developers are able to start uh inference uh with the application. Uh so like we have some inputs image uh for example and the uh model. Uh so

we run inference in executor runtime and we uh basically uh can also uh uh run the model on the CPU, GPU and npu with open vo back end of executor torch. uh but if some operator is not yet supported in open vo so we are safe as well because when it's uh uh executed on the uh CPU in a native execute uh torch so this is our

architecture and uh uh just quickly about the key uh features the uh uh whole end to end solution with exeutor is optimized for the client and for the age so it has a small memory footprint so it provides maximum maximum performance on Intel uh hardware. So you can deploy with uh confidence. So it means if operator is not supported on open vin you can still run the

model by uh uh uh by fall back of this operator to uh the uh CPU and you uh uh you uh you stay in the PyTorch ecosystem pytor workflow from from training to inference. Uh so as I mentioned like it supports multiple Intel uh uh uh hardware uh it supports quantization and uh uh weight uh weight compression. So we uh constantly support new models. For example, recently

we supported uh YOLO 26 uh model and uh also we support LLM's model. Recently we supported Gwen 2.5 uh models and uh uh you basically uh uh can uh uh use your application with execute torch on uh different uh uh uh platforms with open vo back end or uh if uh with other back ends if you uh need. So uh back to summit. >> All right, thank

you. Um yeah just the concluded slides. So we have the fit. So um Dimma talked about how we can export. Now uh I can show you how we can do this. So um I can load the weights um from the train model. And we have a couple of ways of do do the export. So I the first one is basically I can do policy.export or policy to

open veno and I can export it to open vino model or alternatively as dimma mentioned so we have the executor support. So like the open can be the back end in execute torch. Uh so we can just do um execute torch export as well. So this would mean that we have the model ready uh to be deployed. And finally um what we can do is uh we

can utilize um this uh CLI command uh and then um load the the model and the robot and then we can do the the inference uh on the robot itself. That's about it. Thank you very much. Um for those who are uh interested more uh that we can uh you can read the blog or visit the poster down just just across right. Um yeah that's about it.

Thank