Fifty shades of SDV: A blueprint-driven roadmap for orchestration adoption

This talk explores the complexities of software orchestration in the context of Software-Defined Vehicles (SDV). The speaker discusses the fragmentation of orchestration solutions and the development of a blueprint to integrate these technologies effectively. They highlight several key orchestration projects, including Eclipse Symphony, Ankaios, and Flurry, each addressing different aspects of vehicle software management. The session emphasizes the importance of managing mixed criticality workloads and the implementation of over-the-air updates. A demo illustrates how these orchestrators work in tandem to handle telemetry data and deployment strategies such as canary deployment. The speaker advocates for collaboration in developing these frameworks to enhance automotive software lifecycle management.

[music] >> Um So, uh the motivation, as I said, uh we have lots of different software, uh and STV's quote first uh is like lot of LEGO blocks, and especially for a while it looked like everybody came with their own orchestration software into STV. And um so, we were landscape of different orchestration bits. Uh there are some common themes, like they all have a model-based approach, declarative,

and there's some kind of a an agent that sits on the device, and there's some kind of a server that, you know, some digital twins managing the state and the state-seeking capabilities. So, And I think that was the not because it was fragmented, it's because the problem has so many different aspects to it, then uh people were trying to to answer those questions. So, uh you probably

have seen a a variant of the data from many different sources. So, automotive is get getting basically more complex. This is from a report by um uh McKinsey. Uh very long timelines, and uh I think the first CEO has a very explicit driven sort of [laughter] a a comment about this uh that uh you have hundreds of ECUs, and each ECU has uh many different components on

it, and these uh are managed by different vendors, and they all have their own solutions of how to update, how to manage the life cycle of these software. So, it's a really complex problem. So, it's going to remain complex, but uh I think the orchestration is not about making something complex simpler, but it is about the tools to manage it. And um And I think we do

have a good foundation in STV that addresses some of these problems. So, that is the motivation behind trying to come up with this blueprint. So, let's see what we have and let's see how we can put them together and provide some examples that can be the beginning points of a collaboration. So, that was the idea of the 50 Shades. Uh it's not 50 Shades of STV, it's

50 Shades of the STV orchestration technologies. So, um uh you have probably heard some of these um and these are all great pieces of software Onos, Bluewater, uh Pull Theory, Symphony, and there is many more that are coming in like Timpani uh that um uh are open source and some are uh from maybe commercial offerings that are integrating with these software and we've been trying to put

together um uh some of these into blueprints that come with an example. Uh one thing that I find it hard, these are great software, it's great technologies, there's code there, but when you go read the documentation at the GitHub and you attend the talks, it's kind of hard to go from that piece of software to a working practical example. So, that is uh the bridge that we

want to to form with this blueprint. So, the percep- as as I said, the perception is it looks very fragmented, but the reality is there is not fragmented, they're all addressing different problems. It's a complex problem. Uh so, we have to create some glue code and where can we create this glue code? Uh that is the purpose of the Eclipse uh STV blueprints. I don't know if

you have visited that project. We already have a fleet management blueprint, for example, that shows how telemetry can happen from the devices, from the vehicles to the cloud. We do have a service blueprint, an orchestration a new end-to-end a blueprint that goes from devices, different network architectures, commercial vehicles. So, it's constantly growing ecosystem of different STD technologies working with each other. can we do this for orchestration?

That's how we came up with um currently I had the project leads from Eclipse Oniro project and from Eclipse Symphony and with some guidance with Eclipse Bluetie project, Mutua project working in some kind of an executable reference example with some glue coordinates. So, this is hopefully I'll be able to show with the demo here. And the use case we want to pick was over-the-air update. that's a

relevant use case. It's also very hard use case because typical updates involve a whole stack of different components on mixed criticality. Software, some of them are on ECUs, some of them are on infotainment systems, MCUs. So, how do you manage all that? And how do you, for example, do a canary deployment? If something fails, how do you roll it back? How can you manage the cloud part

of it? How can you manage the distribution of the software to different parts? So, that is problem that we will try to address. Before I go ahead, I just want to give you a very quick map of uh at least the ones that are involved in this uh demonstration. Um Eclipse Symphony um is our meta-orchestrator. It is a state-seeking system. Uh it is very agnostic to to

payloads or solutions and the devices, the targets that you have. Uh basically, you decide I want these components to be enabled on those target environments. These can be vehicles, MCUs, ECUs, and uh Symphony constantly tries to seek that state. Uh so, in that way, it can work with other orchestrators and um and make sure that they can deliver those workloads onto those devices. So, I think it's

a really critical piece of software that if you are going to use the SDV technology stack, you have to look at how Symphony can be integrated in that whole um area. Then, you have basically uh different domains. Uh uh Eclipse Flurry and Eclipse Muto are good examples. Eclipse Muto targets runtimes that are ROS-based. So, you have an environment that runs on robotic operating systems like some of

the autonomous driving software and components and bits, they will use ROS. And ROS has a has its own uh model for nodes, software stacks, and this can be also declarative. So, you need somebody who understands that model to enable workloads and and manage the life cycle there. Similarly, Eclipse Flurry uh understands um uh state transitions and workflows. And for example, I want these things to happen when

the car is parked or uh I want them to, you know, the cars to go into a battery saving load mode if my battery is less than 10%. So, it can manage these things. And finally, when you have uh uh these conditions decided, you may have to bring up these workloads in very different uh uh again runtime structures. Eclipse Ankaios can uh work with containerized workloads and

can manage these uh uh in nodes or networks of uh devices on the vehicle. Uh it also has a little agentic mode that there's an agent and there's also an Ankaios server that can talk to these different devices. Similarly, Eclipse Blue Chi is a more deterministic one uh which uses a different model uh at system level services with TCP and D-Bus which is probably more useful for

resource constrained environments. And uh more recently, uh I think there is a project uh that's coming in uh Timpani that's related to Eclipse Paho Theory. Uh timing is very important when you're deploying these workloads. So, uh these are all different bits. They're all the orchestration problem. They have similar approaches. They want to have some kind of a model of the thing that they are going to manage,

uh but they have to work together. They don't all uh all the same problem. So, I think I have went too far >> [laughter] >> and talked about this already. So, as I said, just very brief terms, um Eclipse Ankaios is an automotive native container orchestrator. Uh it has its models and manifests. It's basically it almost works like a Kubernetes but in an automotive environment. Uh Blue

Chi and Timpani, uh they are more deterministic multi-node service controllers. Uh they can work with native workloads. Uh, they use a different type of a system uh communication model and very strict execution logic and state logic um is managed by these. Uh, Eclipse Bluetie is um done by Red Hat companies, LG. and uh Pool Party is looking at sort of managing policies and uh specific node configurations

in safety-critical systems. Um Um uh they are uh positioning this for highly regulated safety-critical automotive environments. Uh, as I said, Eclipse Muto is a different one. It can work with all of this. Um but in a robotics sys- uh environment where ROS is managing and reconciling the nodes and the uh desired state in an robotic system. So, and finally, as I said, the meta orchestrator uh Eclipse

Symphony can manage uh these uh definitions on the cloud uh and is able to uh talk to many of these different orchestrators and uh it can also manage them manage these hybrid environments, many different orchestrators with orchestrators at the same time, too. Um using a provider-based model. So, it allows it to talk to uh the edge devices using very different protocols. Uh, and it does not care.

It's very agnostic to the payloads and the state models or the the models of the workloads that it manages. So, it's it's able to do these very uh very easily. uh [snorts] that's that's pretty much sort of gives you a very high at very high level the uh a picture of all the different orchestration software that we have in SDV. I recommend that you actually go to

the GitHub pages and documents for each one of these different uh they really have depth and they talk about very different aspects of this orchestration problem. And if you have to do a mixed criticality deployment to a vehicle, you have to actually have combinations of these technologies working in tandem. and then you need someone to orchestrate them. Then you have things like Eclipse Symphony. And you have

to know if things are happening, you need things like digital twin. So you have to really go into more depth with these things. So we wanted to start somewhere, so we had with a simple example, simple enough that we don't have to deal with the details of the workload itself, but the mechanisms of how these orchestrations work with each other. So this mixed criticality over-the-air update with

mixed criticality what we mean is we'd like to do maybe some container-based workloads and some critical native workloads all at the using the same orchestration maybe workflow. And then do this in a canary deployment, do some kind of a deployment, see if it works, and then put it out to the whole fleet. And if it doesn't work, roll it back. that was the whole demo. It kind

of builds on the fleet management blueprint. It's basically a telemetry workload wants to push some data to the That is the the package that we'd like to deliver. And in this case, this is a much simplified version of the the fleet management uh picture. If you want to have more detail and go into more depth, you have Please go ahead and look at the the fleet management

blueprint, but this one is a simple one. It's it basically simulates just as a sort of a sorted uh uh telemetry data being pushed out of the vehicle. It's it's kind of a a dummy workload. So, we want to push that dummy workload to a fleet and we want to simulate some uh failures if we can. Uh you can also with Symphony uh and all these middleware

you can use different uh protocols and communication layers. Uh here we use MQTT because it's very easy to set up, but you can use U protocol or Zeno or something else. So, that that is again um beside the point. Um So, this is how it will go. Uh we will start uh I will try to build it very slowly. Uh I will start with a a cloud

command. I will uh that starts a campaign. The campaign is uh Symphony terminology for a workflow. Basically, it it has um stages, and these stages uh uh can fork into different parallel stages or with, you know, failures I can take a different path. Um and on each stage I can ask uh for maybe uh a part of the fleet targeted with a certain workload and wait for

telemetry. So, that whole thing is managed by Symphony. Uh for Symphony everything is state, and it tries to seek that state in these targets. Uh we will use uh Ankaios on the the vehicles uh to deliver this containerized workload, and Ankaios will do whatever is necessary to bring that workload up and running. And uh just to sort of emulate a safety-critical system uh I did not have

time to to talk to uh Eclipse Volt Orion and uh Tim Penny guys to do theirs, but we will use the Blue Chi to to deploy uh a telemetry workload using the system D. So, that's all there is. And here is a very sort of a high-level diagram of what you're going to see. Active Symphony will run in a container and will have a middleware broker running

with an up man MQTT. And then we will have containerized vehicles. Um each will have an agent on it. Uh these are Symphony agents. Uh and they can talk to on cars or Blue Chi or whatever. So, that that's is what you will see. And then finally, uh what I will try to do is uh a more complicated scenario where uh we will start a campaign uh

running on a car, we'll push some kind of a workload, and we'll try to see if it's running properly. If it does pass this uh a test, it will go to the whole fleet. If it doesn't, it will roll back. So. There we go. So, let me try to give you a demo. >> And uh hopefully things will work. I seem to have enough time. Um by

the way, this is currently uh in a common repository that um we started. It's a personal one, but I will start the process to make it a blueprint um uh under Eclipse SDV blueprints. Um and um as I said, Eclipse SDV blueprints is um is a whole repo of different um blueprints that are happening right now. Uh and um you should really go have a look at

it. And if you have um a proposal to put together different SDV technologies together, uh this is the place that I'd recommend to meet us. So, let's look at the code. what is here is basically um there's a little cloud setup which brings up an InfluxDB and MQTT broker. It's all containerized, so I will just bring that up and running. And um uh what is interesting here

is uh we try to simulate all the vehicles as containers. So, there you'll see a sort of replica of many different vehicles. Uh some are containerized, some are not. Uh so, we try to simulate these with Uh we want some people to just clone this repository, bring it up, and see all these pieces running together without much of a trouble. So, you can bring it up and

running. So, you can see all my vehicles and let me just do a uh little demo here. Uh I don't know if you can read that one, but uh uh you can see that the Symphony app is running on the cloud That's the cloud part. And uh InfluxDB uh is uh um is consuming all the telemetry data through MQTT. There's a Telegram Telegraph plugin for that one.

And um at the bottom you see the vehicles. There's a bunch of vehicles uh that are running these either containerized system or a basic uh low uh footprint uh end device ex- that is represented by this blue cheating. Okay. So, once this is done, we put together a a small dashboard app, so this is a bit easier to demonstrate and and show. Um, so this is the

web app that comes with it. Uh, basically it goes through different levels of the scenario. in the first one, basically it's uh, I will skip this one. Uh, but all it shows is how this telemetry workload works and how you can use Unikernel OS uh, to manage this workload in a containerized environment. So, uh, since it's more sort of interesting from our perspective, uh, I will start

with the uh, uh, um, example. In this one, uh, basically um, uh, Symphony will send uh, all these messages to the agents uh, on the vehicles. And this agent talks to the Unikernel OS to deploy the workload. And then that workload starts publishing Just to show you again, there's there's a little bit of this uh, uh, diagram here. Um, how these things will work. I will push

that um, solution which is a definition of a state that is expected on a vehicle. And when this is published, Symphony agent vehicle starts trying to seek the state. It will receive the state and that agent does what is necessary. In this case, bringing up that workload in a containerized environment, will talk to Eclipse Unikernel OS uh, through a and that will bring up the workload. So,

that's the whole thing. And um, we have the descriptions of how we can run this on a uh, command line, but I will just use this little web API start with uh, some of the my apologies for being so technical here. So, I will start with listing the targets. Um, uh, I just used the Symphony APIs to get my targets. I can see that bunch of vehicles

that are already expecting HPC1, HPC2, Blue TV call one and Blue TV call two. So, this these are targets that are expecting some state to be delivered to them. At the simplest at the simplest form, basically I can define a solution which is a telemetry application to Symphony as a state. Then basically I can create instances of the state on many different devices. So, by doing this

Symphony will try to do this. This is the basic model. uh, let me show you what a a basic Symphony solution and um, this is a very simple JSON model which describes what a solution is. It has some versioning, it has some metadata, and then it has a bit which is which Symphony doesn't really care about this portion. this is very Onkyo specific. Basically it's telling Onkyo

to seek this state. It it wants it to run, you know, certain components with properties on that device. So, this is the solution that we're talking about. I create that solution. I can see my solutions on the device. And then uh I can ask this is the sort of tricky part. Uh it's not a tricky part. That's the nice part. Uh basically I'm asking Symphony to deploy

that solution, that version onto its a specific target. This is basically pointing a single solution to a single car. Um you can do this maybe for lab environments. Um But let's give it a try. So, then I send us send this message. Uh let me see. Okay, so it seemed to be running. Now, let's go look at um my little dashboards here on Grafana. Uh one of

them is the Symphony overview. Uh this is again using the similar APIs. I can see uh different uh vehicles available to me. And there's also a solution available there and I can see that there is an instance running. and if it's pushing telemetry data, sorry. That should be happening. Yeah. Okay. That vehicle started pushing some telemetry data already uh onto the Uh so, this is the basic

example. Uh but let's build on this one. Um the second uh phase that we will do in this demo is um basically go back to Uh let me reset this whole thing so there's nothing deployed. Okay, it has deleted basically one instance and one one solution. The second thing we want to do is uh do this in a campaign. And com- campaigns are workflows. Um And uh

for a canary deployment I will instead of targeting a single vehicle we will create a fleet basically those targets that come up they have different properties they are parts of a fleet it's a property on that target vehicle so I will create catalogs of solutions instead of creating single solutions that is targeting a single state I can create catalogs of these with Symphony. one of them is

the same application but this time we have created a catalog and I also create a catalog for a group of vehicles let's call this fleet one. So now I have two catalogs and uh the interesting thing is this workflow and this workflow let me just give you a very quick overview of what a typical workflow looks like. Basically a workflow is again is a model it has

different stages has inputs and outputs and it can also have basically logical statements in them deciding what stage to select after certain conditions. So it looks at the state in these vehicles to move them to stage. So in this category of course we basically deploy one package wait for approval or some kind of a test to pass and then push it to the whole fleet. Otherwise it

will just roll back. >> So, create the campaign. Once the workflow is created, it's like a process um that's waiting to be started. Activation something that you do with some input para- parameters. You can trigger that process to start. So, here I gave my activation a name. I can put some different parameters. So, I will run this activation. And then, you can see what is happening. I

can see different stages of these passing. basically, this should have deployed the same workload this time not to a single car, but multiples of them. Uh let's do a refresh on here. Okay. Now, I do have uh multiple vehicles sending telemetry. I have deployed a similar solution to a fleet. And uh let's do this with a a canary deployment, which is actually not going to do something

very different, but um basically, it's a multi-process. I can simulate a failure or a success. So, I again start with cleaning everything else. This time, basically, deleted all my catalogs and whatnot, but also the instances that happened. And the canary deployment is a bit more complicated one uh in terms of flow, uh but all the same concepts. Basically, it will start with check the catalogs. It will

deploy to the canary, observe the canary, verify it, and if it succe- succeeds, it will deploy to the whole fleet, otherwise it will roll back. So, let's follow the same thing. create the my catalogs. W- will that will get deployed to and finally I basically choose what solutions what catalogs and how this should happen. Next one uh basically I can either um try a um an activation

uh with a success meaning uh it's a good workload, otherwise I will ask for a failure. Uh this is sort of a observes it for 30 seconds. If everything passes goes. Um let's activate it. I can check the status. I should see that it it should be waiting uh at the observe canary. I can keep running this for a little bit and um after about 30 seconds,

it will if it passes that test, everything is successful, it will go and it deploy the Maybe I should have given 10 seconds. Okay, so it says deployment uh fully succeeded. Um and I should be able to see well, similar picture, but again deploy to the whole fleet. Uh I think I'm out of time. So, I will do the I will skip the failure scenario. Uh but

um in short, this will be an open blueprint. I would invite all who's interested in expanding this um scenario and also showing different types of workloads for different types of targets, uh different type of criticality and timing problems to come and and show how we can put these things together for a real automotive use case for over-the-air update. Um if you have any questions, I'll be more

than happy to answer, but um thank you very much. >> [applause] >> Thank you very much. Very nice presentation and a very good demo. So, Angelo Corsaro, Eclipse Zenoh project lead. One clarification. Um Mutuo, Eclipse Mutuo. You mentioned ROS. Do you mean ROS or ROS 2? Because you Okay, that's Okay, because also on on the F1/10th demo that you have on a clip on uh it's written

ROS, but it's not clear. Okay, great. Okay, perfect. Thank you very much for the clarification. >> Thank you. Any other question? Here you go. Thanks. Yeah, also thanks for the presentation. So, is this only used for software simulations or also for production real world fleet management? Uh Well, I I think these are uh um software at different levels of maturity now. Um say they are mostly experimental

at this stage, but there's nothing that would stop us from taking that step to trying these on real vehicles for real workloads because they I mean, we can we can see that they can be used for those purposes. Yeah. Okay, Follow up question. And if it's used for real use case, for example, if you have different vehicles from different vendors that use sensors, telemetry that is stored

at different addresses, then how does the API look like for the symphony on ECU? Okay. Well, how she is here, maybe he's the one better to address that, but symphony API is pretty standardized and there's a very low footprint agent that that you have to run on those vehicles and it can talk through multiple protocols. So, but yeah, he's the expert for for symphony there. Yeah. Thank

you. So, symphony is designed to be like platform and protocol agnostic. So, in theory, by provider model as as you mentioned, you can hook in different communication protocols or package formats. Symphony really doesn't care because it's doing high level state seeking. They just say there's an old state, there's a new state. If they're different, they will trigger provider to do something, right? That's that's how it does.

And seeing of the material of the the project, actually I'm from Microsoft. Microsoft has released a service called I think some strange name, configuration management based on the same symphony code base. So, there's a managed version available from Microsoft offering. Yeah. Okay, any other question? So, I I cannot stress enough how important is this SDV blueprint project as well as what you touched upon which is having

multiple overlapping in apparent projects. So, you heard today in the keynote from Mike that the STV working group started as a collection of building blocks. This is the first attempt of putting these building blocks together. So, uh to assess to to address the question regarding uh the the level of readiness or the TRL level of this project. So, some of them are quite advanced like Ankaios, uh

Pulpito. So, they could be in principle close to being certifiable, but others are pretty much uh proof of concept. But, um I think that having those interface one another is an essential step and also clarifies that having project that cover the same task is important. And we should never be king's maker in deciding only to have one project that is an orchestrator and only one project. So,

having an heterogeneous landscape is essential particularly in open source and also around their readiness level. So, I think you touched upon two very important topics. Thanks a lot, Nagaty.