Edge-Cloud architecture for scalable multi-simulator robotics validation

This talk focuses on Edge Cloud architecture for scalable robotics validation, presented by Massimiliano Vanini and Leonardo Montella from Adaptive Reply. They discuss their involvement in the IPCEI project, aimed at creating a comprehensive EU-wide cloud edge infrastructure for robotics simulations. The speakers detail challenges faced by robotics developers, emphasizing the importance of simulation for scalable validation in environments like logistics and manufacturing. They address pain points such as fragmented workflows and lack of standardized evaluation metrics. The proposed solution advocates for a structured, open-source validation framework that standardizes scenario definitions, execution, and evaluations, ultimately enabling reproducibility and scalability in robotics development.

Hi everyone. Thank you to be here. Today we are here to talking about Edge Cloud architecture for scalable multi-simulator robotics validation. I'm Massimiliano Vanini and he's Leonardo Montella. We are from Adaptive Reply. So today we will introduce a project where we are working for. We are building some software open source and we are talking about robotics specially in simulation environment. So we are working in a very

ambitious project. Actually is a called IPCEI. This is the world's first European project to build a an EU wide multi-provide cloud edge infrastructure that actually is aiming on a very ambitious goal to actually detach from external EU provider, for example big data centers that have all data sovereignty inside. So Europe has bring much much effort to build this kind of large infrastructure to allow European user companies

and many startups and whatever to allow to be interoperable, security and sustainable but also sovereignty with the data. Just to give us some number we have involved more than 12 EU member states more than 220 ecosystem participant from startups, center of research and many more and 19 strategic strategic project leaders and we are one of this. So Reply is a big consultants company. We are more than

60,000 employee. We are present more in 50 country all around the world and also we have 2. almost 2.5 billion revenue. Our expertise domain is a much wider, so we are across different kinds of expertise such robotics artificial intelligence and whatever. So, Reply is a holding group. We are one of the company inside of the group is a called Adept Reply that its mission is ambitious goal

is participant to the in this project and developing an end-to-end Telco cloud orchestrator to manage all distributed data resource across multiple domains. The second point is to create an autonomous robotic framework to integrate robotic and sensor system intelligent edge platform and we are here to taking a small piece of that. And third point, very important for us and the reason why we are here today, we are

making all of this open source. So, we are contributing to communities such a Neon Ifos from Linux Foundation open source base. So, we are here to share what we are building for. So, let's go make an introduction the topic of today. So, today we are here to talks about robotics. I don't know if in the audience there is anyone that work in robotics development, maybe raise your

hands if anyone maybe because some of the concept we are sharing maybe you are familiar with. And the problem is that robotics developer today for scaling robotics we believe that simulation simulation is the actually a key enable to the make that. Why this? Because when you work with a lots of robots, for example, talk is about a logistic manufacturing plant with thousand of robots, you know? So,

it's very hard to actually to develop algorithms such kind of such kind of place, you know? You have to validate a lots of things. You have to large scale scenario exploration that without simulation is not possible because it's very costly actually. And the second reason is that failure are hard to reproduce and also unsafety. So, the second way and advantage for to use simulation inside this kind

of environment is to use simulation to have a pre-safe validation of the algorithms to after the development the deployment on real robots. And the third point is that validation also require many environments and condition and to only way to achieve that theoretically is possible only in simulation controllable and repeatable way. The problem is that simulation stock that are used today in robotics that we will see soon

are not reliable by default. And what does it mean? We identify three major pain points that we are trying in our ambitious goal to solve. The first point is that process workflow is fragmented. What does it mean? Is that every time any developer that works in in robotics specially in the simulation environment have to set up this kind of tools manually every time every kind of project

change is different. You have to change set up configuration. You have to set up the execution of this and debugging logs and make all these kind of extract matrix such evaluation stuff. But all of this is made in a ad hoc way. So, manual set up every time in robotics community this is one of the most biggest pain actually and this is like this almost from 10

years actually. So, we are tired about that. We are try to solving stuff and make this kind of runs in a reproducible way. The second major problem is that we we still have a lot of tools simulation tools open source very cool very interesting. The problem is that actually to use this tool and to compare result across this tool is very hard because there is no common

KPI layer. So, there is no an objective way to compare results between each tools and there is also a way is hard to differentiate, for example, when you go in real world deployment, why you failed before in simulation because there is always a real sim gap that is not extinguishable, but actually we are try to find something to compare this kind of matrix. And the third point

is the fact that the this all of this is not scalable. Because by default you you need a lots of infrastructure resource, so not only computational resource that is the the the base in some sense, but also an infrastructure to the manage all this kind of simulation because if you don't have that, you are limited in coverage on what you can test actually. And also, if you

want, for example, explore more scenario, more multiple configuration, you have too many to execute without a standardization layer. So, many critical failure remains untested. our day today we are here to propose a mindset shift that we need to require to make robotics validation scalable in the future. So, actually what we are proposing here is a mindset shift from an ad hoc validation, so actually every time put

a lots of effort. Basically, most of the time you put more effort on set up things than actually innovate and developing real software produce a real value, for example, for your customers. So, and we need to shift in a structured validation, so where you are able to become, for example, the financial area, repeatable execution, standardized evaluation. So, to enable consistency, comparability, and scale. We identify four pillars

actually on what we believe that are enabled to standardize our robots are tested in simulation. The first one is scenario standardization. So, actually a way in a consistent to the user make able to define this kind of scenario in a way that every time you execute this kind of scenario, you can across multiple simulator in the same way without changing every time the kinds of configuration. The

second major pillars is execute standardization. Actually, have an environment to allow repeatable and controlled run. The third point is actually evaluation standardization. So, today is not present in the open source community something that made for example collect metric for you is very reliant to the developer and this is painful actually. So, we believe and a standardization of this evaluation metrics is will be one of the major

pillars to invest on this project. And the third point of course is scalability. So, to have to allow the full potential of simulation environment parallel exploration of many scenarios. This is point is not neglectable. So, this is one of the stocks that today in robotics are mostly used. So, I don't know if many are familiar with that they base on robotics today. For example, Ross 2 has

a middleware, Zenoh as protocol layer, communication layer, simulation engine, and for example, Kubernetes for execution layer. The problem is that in open source community, this tool are very strong. But actually, there is no way to connect by default this system together and this very painful to actually do that. So, there is no standard way to define and execute this system together without So, the core of today,

actually, and what we are try to building in ambitious manner, is to find and structure a unified development layer where its core is based on standardized validation layer. So, a layer that allow the user to have scenario definition, execution definition, and evaluation KPI in a standardized manners where an orchestration engine manage all this kind of resource. So, we have a run manager, a resource scheduler, and a

data collection in a standardized way. All All of that is based on the external ecosystem we showed before. And the advantage of to that is also that the user will be able to on top to build additional modules. So, for example, if you want make AI training and optimization, or maybe you want to create your own asset and use pipeline to make generation, or maybe you want

to make logs and failure analysis that are common phases in robotics development. So, and all of this, this is why we are here today, is open by design. So, we believe a lot in open source. And because not only from the ecosystem integration we are using that all all open source based, of course, but we believe in extensibility that we think that user could trust this the

platform only if we are able to extend the adapters, for example, scenario and KPI. But, the major important fact is that also to be transparent. So, because it's such an ambitious project, validation layer has to be finally inspectable by the user. So, this is was just an introduction. Now, I want to pass the step to my colleagues that Leonardo Montella is the software architect of the platform

and will go deep dive into our what we are building. Thank you. Thank you, Massimiliano. Okay, so now let's take a deep dive into a more technical part. So, let's start by looking at how robotic simulation is typically developed today in most organization. If we look to a traditional setup approach, we see that um robotic simulation are strongly tied to specific simulators and often run in local

environments. This basically means that engineers works directly inside tools like Gazebo or Isaac Sim that are all simulator GUI softwares. And they must perform each step uh separately and manually. This typically means starting by importing the robot model, which is an URDF file, setting up all the sensors one by one, writing a lot of code, a lot of Python scripts, connecting ROS 2 nodes, tuning all the

parameters, all things side a simulation, all inside a simulator software also. So, uh robotic simulation is too related to specific simulator versions, sometimes even to specific developers. So, the result of all of this is fragmentation because um if we want to rebuild the same scenario, we must rebuild it from scratch over and over again, passing through all the steps every time, so losing a lot of time

and a lot of effort. So, this makes results difficult to compare and also changes are very difficult to track. And this makes the simulation still today a manual activity instead of a robot an engineering process. the all approach not scalable, not automated at all by design. So, that's why a platform-based approach introduces a fundamental shift. With a platform-based approach, we move on from the simulator-centric workflow to

a scenario-centric development model. So, instead of setting up each step separately, manually, the user will interact directly with a unique platform that is an interface which will lead the user to use a standardized configuration. How we do this? We create a JSON file configuration which describes all the wire elements that are required in a simulation starting from the environment, the robots, the physical sensors, everything we need

in that So, this JSON configuration acts at a standard interface between the user and the simulation. It also is useful to share this configuration maybe across teams, maybe you reuse it multiple time to study robustness, also use it into different simulator backends. even if all the simulator are different in internal architecture, we the user experience with our platform remains consistent. So, also during the simulation when the

simulation is running, the user can monitor all the simulation with some ROS 2 camera topics that are placed in the environment and also from the robot point of view. And the platform will automatically elaborate some results, some KPI that the user can then compare with other simulation and other stuff. So, all of this makes an this approach more automated, more standardized, and very scalable also by design.

Another key aspect of our platform is a general central asset repository management. How we deal this? We basically use a lot of cloud service uh operators. So, we used to store every single data, every single physical file inside an a cloud object storage. And all the related metadata in an SQL database. This metadata layer acts actually is very useful for the platform to reason automatically about the

file formats, also compatibility of the asset because we use different simulator. So, for example, when we want to use a specific asset, the platform will reason uh if it is compatible with a simulator or whether a conversion step is needed. So, we with this approach, we move on from a shared folder, directory, whatever, to a very specific and standardized version of an asset management repository. So, when

all the configurations and all the asset are standardized, we can execute the same scenario in two different simulators back end. So, we told before that simulators are very different. Their architecture is very very different. Um but we standardize the process by sending the configuration from the platform. Each simulator will run independently inside its own containerized version. they basically behind the scene they will download all the asset

from the general repository, only the asset that are needed for that configuration. Then will run the simulation independently and then will they they will generated some metrics. The advantage is that if we have the same configuration standard, we can see all the difference between the simulator that now becomes measurable. So, all this all of this turns the simulator diversity from a problem into a real validation step.

So, another thing that we we developed is a KPI based metric layer. We believe that metrics general metrics offers passive monitoring and they often are also context agnostic. So, we believe that a KPI's metric layer with some defined success criteria is the perfect option to have a more clear overview of the entire simulation behavior. How we do this? We do this by making the user select by

our platform all the targets for each KPI. So, for example, a navigation task will be no longer evaluated subjectively, but it has to respect a certain threshold that the user can customize directly with the platform. So, for example, it needs to respect a cross track error or maybe a specific number of collision that are allowed in the simulation. So, the the reason we introduced this is also

the fact that we use the same configuration and the same metrics, so we can compare, aggregate all the um for actually um have a clear overview of what is happening, and also to make robotics teams doing their decision based on all the results they have retrained with this metrics. Also, another key aspect of our platform is distribution. The distributed execution in our platform is not only about

scaling all the simulation, but is also about managing how the simulation requests are handled. So, uh robotic simulation are often long-running and are also resources intensive. So, if you send a lot of parallel requests for robotic simulation without any type of coordination, this will lead to us certainly um race condition and non-deterministic behavior. So, we designed an architecture that is based on FastAPI, Celery, and Redis. All

the All of these run into a cluster Kubernetes, which is on the Edge Cloud Continuum, and is a cloud service. So, the all the architecture is distributed in pod. Each simulator got its own pod that is running internally. All the requests are managed by a FastAPI control service. So, the platform will send the JSON configuration file. The FastAPI will elaborate it, also will normalize it, and will

send it in a queue instead of executing it immediately. So, then Redis acts as the message broker, while Celery manage all the task execution. So, this basically means that we create a queue that is ordered by the number of requests we received. And each pod can run simultaneously, and also is very independent from the others. So, why we decide to use a cluster architecture? Because the Kubernetes

environment adds also resilience through pod isolation, and also guarantee automatic restarts in a safe way. So, if a pod or a worker fails, it will automatically restart by itself. So, this will generate, of course, a consistent workflow that will never stop and will be always non-blocking. So, this all of this architecture will guarantee, as Massimiliano was saying before, a very, very automaticable, and scalable architecture and workflow.

So, now let's see a concrete example of how a user can set up all the simulation. It can firstly choose an environment, then it will scroll for all the studied asset he wants to spawn inside the simulation. He can pick a camera and decide where to spawn it in order to monitor what is happening inside the simulation. He can choose one wheeled robot, and selecting easily the

way points on the map. The robot will follow all the way points until reaching the goal post. Obviously, a multi-robot simulation is also allowed in our platform. The robot will not collide with themself. And we can see how the user can select all the KPI we were talking about before with a certain a certain target threshold. He can save the template of this configuration in order to

reuse it later. And then we can see how the user can monitor the simulation while it's running. So, everything is live. The user we we can check of course the cameras he positioned with a custom position. And also this camera are teleoperated. And they can they they they will offer a clear overview of what is happening inside the simulation. And it's a faster way respect to use

all the software GUI that is very difficult to approach. Then we can see how easily is obtaining all the KPI with the success criteria if they respected the threshold. And of course, as we saw before, we can reuse the same configuration over and over again. Even we we can use it for study the robustness of the same configuration. Or the developer can make a little changes in

order to gain some performance metric and improve the quality of their work. I will pass my word to Massimiliano. So, thank you so much Leonardo for this technical dive. I want just to conclude the presentation of today. So, what we are show is that we believe that simulation is essential tool for scaling robotics development today for any team around the world, but doesn't not work without a

structure. So, to allow comparability, reproducibility, and scalability, we require a standardized development layer. And we strongly believe that such a layer must be open, also open source, to be adopted, extended, and trusted across the ecosystem. Thank you so much. So, question and answer time. Can you walk us through the observability and telemetry part again? Basically, I see there's Kubernetes, so there will be some kind of infrastructure

telemetry. And there also robotic systems, so there will be some kind of like raw stack and something. What is your telemetry stack look like? One, two. I'll never So, we are actually now developing an observability model about all this platform that is obviously regarding the raw stack you were saying. We actually are implementing another model that is running outside all these layers analyze the raw stack. If

you work in robotic simulation, you know that raw stack are always huge and difficult also sometimes to interpret. So, we are actually in the development developing an LLM that will reason and will also do some details. It will give some details to the platform about the simulation, and so you can even more understand what happened, what what went wrong, and maybe you can use also this data

to to also using it on a real one, a real machine. So, you can compare also the raw stack of the simulation with the real world one. But, we are currently developing The model introduced by Leonardo is the one we're working for because I strongly believe that without logs, for example, debugging tools, user, especially developer, cannot trust this kind of we are trying to innovate also in

this part. Actually, we are finding a way to manage because the problem in robotics is that all of these robots generate thousand and gigabytes of data and because there are so many, for example, uh camera topics, sensor topics. There are lots of different kinds of data types and also a lots of them. So, it's not easy to process all this data. So, we are trying to find

an easy way so to make the user upload this rosbag, okay? So, these kinds of data have a clear graphical interface to, for example, plot these kinds of data, for example, using classical graphs to compare it. And also, as introduced Leonardo, we are working on using a specialized AI agent because by default, if you throw to an LLM this kind of bags is unreliable actually because it's

too big, this file. So, you have to find a way to slice every kinds of topics and also to be to train your LLM especially to be verticalized to the kinds of analysis. And this is very we find we are very interesting discovery because uh uh they actually when you submit this kind of problem, the LLM actually extract all the potential root cause failure. So, it's very

cool actually to debugging so fast. And that this we believe this is a very strong accelerator in this sense. So, problem you are mentioning is you need some kind of collector to collect all the logs, these thousands of logs. So, I've worked with OTEL setup and in which you have all this data pouring in and then you project it with say LGTM stack in Grafana, so that

you can see visually. I do believe there will be some alternatives for the robotic simulations because the problem I am solving is for infrastructure. And we also like get thousands of log. So, yeah. Thank you. >> Oh, nice. Thank you. Thank you. Any other question? Um it's a viewer's German Aerospace Center. So, thank you for the talk. I have one question um concerning dynamic environments. So, you

mentioned that you can have several robots, but what what is about interaction with other moving objects or persons even? Uh how's this possible? Thank you for the question. I can reply. So, basically uh in our platform we are developing maybe in the video you we show just fast. We didn't have a a zoom on that. So, basically we have a define a section when you have to

define, for example, this kind of dynamic actors. So, for example, walkers, operational workers, any other machines that could be automated. So, you could define the same path you see before for robots, you could define this path also for dynamic object to interact inside the platform. So, you tested if the robot when you find and sense this kind of obstacle could actually avoid inside simulation. Yes, also if

I can add something, we are using uh the navigation stack map two with all tuning parameters and it will follow the local cost map. So, even if you gave them some waypoints so a path to follow, it will, of course, be aware of the obstacle if there are some moving also some trajectories, so it will detect them and try to avoid them. >> Yes, the the first

step that actually is not showing in this presentation is actors. So, it's basically a step where you could define actually this kind of kind of a dynamic obstacle, of course. But there's always also for the other objects there's always fixed trajectories, so there's no interaction in the same No, no, you can you can for this kind of object customize the path for every object dynamic object that

you are putting on the simulation, of Is there any other question? Then thank you. >> And have a nice day. Bye.