Automated software safety – The key success factor for functional safety in open source

This talk focuses on automated software safety within the context of open source projects, especially in the automotive sector. The speaker, Ulrich Huber from BMW AG, discusses the challenges of maintaining functional safety processes when transitioning from proprietary software to open source, including the lack of trained personnel and the need for scalable solutions. He emphasizes the importance of ISO 26262 as a framework for software safety and introduces key concepts such as the three C's: correctness, consistency, and completeness. Huber also presents the Safety Sentinel project, aimed at automating the safety verification process to reduce manual effort and enhance collaboration among contributors. He outlines the integration of various tools and methodologies, including the Bazel build system, for optimizing safety analysis and validation.

Hello everybody. My name is Ulrich Huber. I'm working with the BMW AG. And today I want to talk about automated software safety. When we decided to open source our communication middleware to S-Core, we kind of started a problem. And the problem is the safety process that we internally do in our projects, it's based on, well, trained personnel, manual steps, and a ton of proprietary software. And as

soon as you go into open source, well, you don't have trained personnel anymore. At least you can't ensure that everybody is trained. Um manual work, yeah, it kind of scales as long as you don't have too many contributors, but you want to have a lot of contributors because you're open source now. well, that proprietary software is kind of the biggest blocker. You're just You don't have access

to that. You need something that is at least freeware or best case open source. And out of that, um we in essence then decided, "Okay, we need a new process. We just can't use the one that we're currently using, that we are um familiar with. We have to do something new." But let's take first a step back. Um why do we actually do functional safety in our

middleware? And well, imagine you're sitting in a car, you're approaching a traffic jam, you want to push the brake pedal, and it doesn't work. Yeah, that is why we do functional there are different ways to actually demonstrate functional safety in your software development process and well, I guess the most popular one, the one everybody uses is the ISO 26262. the way that the ISO 26262 actually approaches

safety of your of your software is that it says, okay, you need to have a process that actually cares about Um that process must do some steps to actually achieve safety, some quality steps. And in the end, and that is the important thing, you will never achieve a software without any bugs. You can only reach a certain level of well, bug freeness, but you will still have

some left. And that is something you have to keep in mind because that also means this process doesn't search for the perfect solution that fixes all the problems. It just wants to give you a certainty that you at least can say, okay, I found the big problems. Of course, you can still try to get the best solution. That's what everybody wants. And the ISO 26262 um has

quite a lot of parts. Today we'll only focus on part six, um which is about the software development. Um just that you don't confuse um the scope we are taking here. Then one more thing um that you need um for the rest of the talk. So, there are some key demands that you kind of have to fulfill. Um we call them the three C's. which is in

essence correctness, consistency, and completeness. And whenever you do some verification on your software process and also on the software itself, then you normally can fit that in one of those three categories or there is an overlap between the categories and your test verifies two or three of them. Um, and whatever tooling we actually want to create um, that helps us with goes into one of those directions.

if you look at open source, well, we as a big company going open source is like one part of the problem. The problem also goes the other way round. So, in automotive, well, we also don't really enjoy reinventing the wheel. There is this um, famous funny example of the JSON library in A-core. Why does A-core need its own JSON library? Yeah, the problem lies in this V-model

here again. So, if you look at open source projects out there like the traditional open source projects, you will mostly find that they actually take care only of well, the small blue part here. So, the software unit design and implementation. And if you are lucky, then they might also have some software unit verification, but it's most often not 100% in most projects it's even frowned upon to

have 100% code coverage. you might have like 80% if it's really, really good. But then you also have like all this stuff here and all that stuff up here, um, which normally is just not there. Sometimes you find yeah, some architecture documentation, likely outdated, probably not really in a standardized format, likely some markdown file that just describes a little bit. It's not really usable in automotive because

in the end for the strict software quality process that you have internally in the big project, you just can't use that. You would have to redo all this qualification um and in essence take care of all these grayed out steps here yourself. And if you look a little bit at the effort what those steps mean, it's actually 80% of the total effort of creating this software. So,

we are really talking about you get 20% from out there, then you have to understand all that stuff that isn't counted in those 80% that are missing, and then you still have to do 80% of the work to actually qualify to level to be able to use it. I don't want to blame any of the maintainers out there because in the end it's also quite clear why

they don't do The first reason is, well, our community that needs all that like process coverage, it's quite small. If you look at all the people out there that are interested in open source, not a lot of them actually need that for safety relevant functionality. The second thing, and that is what I described with those 80% is this stuff is really time intensive. at least so far,

there is no real shortcut for that. And then the third one, yeah, that is there are not a lot of maintainers out there. And in the end, you can't really force people to take care of safety in their pull request if they are not willing to do that. And as maintainer, you have to do that on top or you also have to fix quality problems in existing

code when somebody contributes Well, what does that mean for us in automotive where we can't say, "Nah, safety, I I don't care. It's It's fine. Probably nobody will be killed by that." Um well, we can't really increase the community. just not in our hands. We can also not really increase the capacity of the maintainers, at least not to a meaningful level because that would mean everybody of

us now goes out there, takes some open source project, and just asks to be also a maintainer in his free time probably because your boss won't pay you for that. What we can do is we can make this stuff less time intensive. And that is where our idea comes in about the project safety sentinel. And the key ID idea is currently safety processes are a lot of

manual work. With checklists, you have analysis that you have to do, you have some kind of KPI dashboards lying around on some website that you look at if you're good on daily basis, and most projects, I guess, you look at that at the end of the project and then you look, "Oh gods, I have like 5 weeks left and 6 weeks of fixing problems." that's something that

just doesn't work in open source. So, what we want to have there is a process that is really fast, that has additional step for maintainers that they don't have to do. So, everything that is actually automatable should be automated. And everything that is not automatable should at least be guided that the person that does all that safety work can actually focus on finding problems and not on

just satisfying some checklist where he says, "Okay, half of that stuff, come on, guys. That's not something I should do. That's something that some code should do." For this project, we Well, we defined some I would say guide words, um also some of them not really being single words, but the idea is that with those guide words, we actually understand in what direction our solution should go.

And the first one is kind of the key to actually lift all the potential. So, treat this code. So, whatever tool you have, whatever piece of safety process you try to tackle, it must start from code and it must end in code. What happens in between doesn't really matter that much. Can be human time, it can be some machine code, it can be AI. But in the

end, you need to start from code, but because that's something that is checked in in the project that you can actually take. And it also needs to end in code because the next step of the process needs that result. The second one I already went into, automation, of course. Whatever you can do automated, we want to do that automated. And the third one, connect. the more you

connect in your safety process, certainty your safety argumentation actually gets. Because if you have a problem in your architecture, that problem actually surfaces in some test that you're running, some verification test, well, then you should still find that also in the architecture. You should see that not at the final step of the build saying, "Ah, there is some problem in there. I don't know. You should find

it where that problem actually is in that component in that unit." Below those guide words, I also brought some focus areas. Um safety has a ton of I can't go through all of them today for obvious reasons. So, I'll focus on five of them. Um that doesn't mean that in the end we also have to tackle others in addition. Um but those five I picked out to

actually go into a little bit more detail and explain to you. Before we can do that, we need a little bit of basics. if you automate something, you always have to consider how do you scale that? Because what we don't want is a tool that takes half an hour after every pull request. And then as a maintainer, you review that pull request, then you go for a

coffee break. Would actually be nice, but still you come back half an hour later, you get the result from this check and you're like, "Hm okay, I have to review again." That's something we don't want to have. Um for that reason, we need to be smart about when do we apply what process step? don't get me wrong, all this safety work, it is still a lot of

work. Even if you automate that, it will take time. It won't be a 3-second and boom, it's there. What can we do? Well, the first thing is we couple this whole process with the build system. Because the build system is already aware how the different pieces of the project go together. And it allows you to optimize when to do the different steps. The first piece we can

do is Oops, that was the wrong button. This one here. The first piece is that you actually do the tests as soon as you can do them. So, as soon as you have all the input artifacts that you require to verify you perform that verification at that stage. That allows you to optimize your memory footprint. That allows you to optimize your run time. Of course, in the

end, you have to have one single big report that tells you about all the problems. And you also have tests that are not local to a single build artifact, that actually have to look at multiple build to find out whether this is fine or not. So, what we also need is some kind of linking. So, we need to be able to combine artifacts of um build targets

that we have and do verifications on that. And if we perform verifications on a sub set of the dependencies, we also need to take this output of this verification and transfer it along the build graph to actually generate this one big final report in the end. the third one, and that is in essence where you can really gain a lot of speed and limit the impact on

people, is that you use caching. Because in the end, we don't have to redo every verification step if we know that only five lines of code changed, and those five lines only have impact on a specific subset of all our functional safety-relevant code. And the build system is really a a nifty trick there, because the build system it knows, at least if you choose the right build

system, what part changed from the last time you ran that? We chose for that Basil. Um now a few people will say like, "Oh God, why Basil?" yes, I know Basil is a little bit different than other build systems in the way it approaches the problems. But one of the killer features of of Basil, why we really want that and why we also need that, why there

are there are not really alternatives out there, is that it enables polyglot builds. Okay, now some people might look up what polyglot builds means. Well, polyglot builds means in one single build, you can actually handle multiple languages at the same time and condense them to one single artifact in the end. if we look at our or also at our use case for the project Safety Sentinel, then

we do have a lot of programming languages or languages that the build system needs to understand. It's not just the languages that the project is written in, which for us would already mean we have C++, we have Rust, and we have Python. It also means all the tools that we use for the safety process, they generate artifacts. And those artifacts might be written in JSON, they might

be written in YAML, they might be some SVG file, and so on and so forth. And all that, it's not really a programming language, but it is also a language that the build system must understand. it needs to connect those pieces. It take the output of one tool and feed it into another tool. historically, traditionally, tools don't like to speak with each other. And that is really

what Project Safety Sentinel tries to do, to make those things talk to each other, to actually allow an automation across multiple different tools. Bazel comes with a few more things. Um so, we also get a lot of caching there. Um we actually get a good project structure, um where we can also do partial builds, where we can reproduce builds easily. it's also open source, which is great.

Now, with that, we can actually go into those focused areas that I picked out. The first one I picked out is safety analysis. well, I kind of did a meta safety analysis for a safety analysis. I just said, "Okay, um when you actually do a safety process, you can also do mistakes. And that is also a risk of the safety of your product in the end. And

manual steps in the safety process, they always infer the biggest risk, just because of human error. You can do your analysis wrong. You can do a mistake there, and you can say this is safe while it actually isn't safe. And in the spirit of a safety analysis, well, we have this big risk here, that your safety analysis is actually wrong, and we break that down to root

causes, and then on those root causes, we define control measures. So, ways to mitigate the risk. It's not really to completely get rid of that risk. It's just to mitigate. Because in the end, the safety analysis is not something we can automate completely. The safety analysis something where some person needs to sit down and actually think about the stuff and find the problems could happen that impact

the safety and also break them down. And there, well, his focus should be on actually doing exactly that. It shouldn't be about, "Oh, I need to draw a diagram." And "Oh, I kind of also need to make sure that I have like a traceability in the end." All that stuff is not something that he should focus on. Because in the end, that is something that is just

an outcome. But it actually it diverges his concentration from from the actual problem. our control measures that we actually find for the topic of safety analysis are four tools that we're using. We're trying to not reinvent the wheel here. Um therefore, we chose tools that are already out there, that are also used And based on those, um actually just put in the project safety sentinel this glue

to make them work together. The first piece that we're using is TLC. TLC is a way to describe requirements in The second piece, um is PlantUML. PlantUML, well, it's a way to create diagrams. when you do a safety analysis, it helps tremendously to actually visually see this analysis while you create it. Then, as third tool, we have Lobster. Lobster is in essence a traceability tool that works

on TLC and is very extensible to also include inputs from other tools. We use that to actually make sure that each risk has at least one control measure attached um and that this control measure then is also in some way transported to the right places. And as the final piece, well, we have Basil, which it doesn't just invoke those tools. No, what it also does, it actually

checks that Lobster doesn't find a traceability error. And if it finds one, then actually reports that as part of your test execution. So, if you run Basil test with that target, you will actually get an error back if there is some traceability And with those tools, what we then end up with is that as a developer or as the person that needs to do the safety analysis,

you actually focus on doing the analysis itself. And in the end, all this surrounding stuff comes for free. And you can have a certainty, at least a higher certainty, that you didn't miss something big. Because at least you know, okay, I have per risk that I need to mitigate. On to the next example. This one is also interesting. Manual analysis. Well, as the title already says, this

is also not something we can fully automate. Um those are the things where an automation will fail because there is at least one step in this analysis that needs to be manually done. from where we are coming from, from this process, um well, I don't know whether there are even better tools out there. Potentially, there are even worse tools But, we faced four big problems that we

wanted to solve. The first one is this tool didn't really make clear what the context of the analysis was. Yes, you could attach that manual analysis to a requirement, but that manual manual analysis sat like five websites away from your project. And whenever you changed some code in your project, believe me, nobody thought to actually look at the manual analysis, whether he needs to update that, whether

it's still correct. People did that at hoping that they don't find a problem. luck was not always on our side. The second one, well, I just explained. The third one, well, they believed a good way to actually retain a manual analysis and structure it is a single free text field. It's quite good because you can do everything you want in there, but it also means it really

depends on the person that did the manual analysis whether you can follow the different steps that that person took to either redo that analysis or actually verify that it's correct. And while this might still work internally when you have trained personnel doing those analysis, as soon as those people are no longer trained and outside that is the case, this will fall apart. And the fourth part, well,

this kind of plays into the safety sentinel itself. Because it is a free text field, well, somebody wrote at the end, "Analysis okay." The other one just didn't write anything. Well, from the the it was kind of clear that it was Um the third one forgot to update from analysis not okay to analysis is okay. It was really really hard to actually take the result of that

analysis and do something with that because you would have at least needed AI to kind of understand what that person did. So this lack of structuring um the actual content the outcome of this analysis meant also that nobody could use that to improve the process around it. What we did in the project Safety Sentinel was well first we actually thought okay how do we structure a manual

analysis. Are there certain steps that you normally do? Is there some kind of repetition that you should consider? How can you actually make this something that a well not just a human can understand but also a machine can understand? And out of that we kind of got a model um of simple steps simple types of steps manual steps automated steps loops decisions and assertions. And with those

you can essence like build together block by block your The benefit of doing it this way means you now also have a very good way to actually structure that in a document. And with that document it also means now you're part of the source code. So you don't have an additional system next to your source code you're actually inside the source code. being part of the source

code has then the benefit you also part of the build system and the build system allows you to actually solve this upper point. You can now clearly define the context that you're working in. It also allows you to verify the actuality of your analysis. So, meaning you can define that context and by defining that context, you can also find out did that context change? And then the

build system can flag that and tell you, "Hey, now you have to redo that manual analysis." At least to update it even if it stays or maybe something changed now. So, it actually tells you when you have to do this step of the safety process. And as I said for the safety analysis, it's the same for the manual analysis. If you structure it, then the people actually

focus on the task. So, by guiding them interactively through the different steps of the you can actually make them focus on that single task that they currently should look at and not at what they did 5 minutes before and have to do in 5 minutes. And of course, in the end, it's machine readable. So, you can actually act on the result. Then, the next topic um is

about scoping. I guess some people of you worked on bigger projects that took care of some function that had safety impact. I would say quite a typical approach to actually limit the amount of time you have to spend in your in your safety work is that you decompose your big project into smaller pieces where you say, "Okay, this one is safety relevant and this one is not

safety relevant." But that comes with a problem because now you suddenly have to check that this piece that is safety relevant doesn't depend on something that you deemed as non-safety relevant. And doing that by hand is really error-prone, because at some point somebody will add a dependency in the and you will not think to actually check whether that dependency in your process or not. Bazel already has

something that we can use for that. Um it's called aspects. And what aspects do is that you in essence can introspect the dependency graph of your project. So you can with Bazel look at what dependencies each target has, and you can also attach additional information to those targets. And we use that to attach to each target a specific integrity level. So ASIL B or also QM. And

in the future if score decides that we want to have more than those two integrity levels, we can also put others. And then based on that, you can also perform checks. So you can in essence say, "Okay, I have this ASIL B target down here, um target X, and it depends on T, it depends on Y, and it depends on Z." we found out, "Okay, QM, oh

that one is QM. That one's not fine. Z would be fine. This W up here would also be fine. But we need to get rid of this Y, or we need to make Y also safety relevant." the nice thing with Bazel now is we can bake that into the build rules themselves. So it's not something that a developer has to think, "Oh, I need to do this

additionally. No, he can actually just run his normal builds and Basil does that in the background automatically. Architecture validation. That is also a tricky one because architecture architecture validation requires lots of different artifacts to actually create meaningful tests. Architecture is kind of the blueprint for what you do in all your other steps. And if you remember back to those three C's that I explained, you can for

each of these C's, you can define a test. For completeness, for example, well, there is a problem if your architecture diagrams do not mention a component that you implemented. It's also the other way around when you implement when you miss to implement something, but your architecture describes that, that can also be a problem because it might be that health check of the system. what those checks at

least need to know about is that there are diagrams and that there is implementation and how to match those two. For consistency, well, you might have five diagrams describing the same thing and one of those diagrams calls the function so magic and the other one calls it request magic. Which do you now implement? So, also this is something where you at least need to understand, okay, there

might be multiple diagrams that talk about the same thing. How do I correlate them? And correctness, well, this unicorn is really cute, but it's not UML. So, what do people understand under And that is also something that you need to consider and also verify. What we did is in essence we approach this um with well, all the tools that I explained in the basics. So, compilation step,

linking step, and caching step. And with that we actually um are able to do lots of different checks in very short time. And this is also very extensible because in the end it runs on a data model um that we generate out of all these this model can be extended and you can also write your own checks on that. And then finally I brought an AI case

because everybody needs to do a little bit of AI at least. Um quality of requirements. This is at least for me personally something where I was always unhappy with the tools that we had. what we always had was kind of these strict rule-based algorithms that say there must be a should in your requirement or you must use this name for whatever. And that doesn't really work well.

On the one hand, yes, you should have some structure to your requirements, but on the other hand like there are always a lot of different ways to write requirements and they are there for a reason. AI is actually a game-changer here. Because with AI you can those requirements in a more flexible approach. What we did is we gave a well, starting prompt to the AI um to

give it a role saying that it is an analyst for requirement quality. And what it should do is in the project we have a checked-in guide how to write requirements with certain categories of problems, um keywords that you should have or at least um go along that line. Um rules in a loose sense, like for example, a requirement should be atomic. Well, there are a lot of

ways to make a requirement in atomic. Um and AI can help with understanding And to actually then give the person that wrote those requirements a little bit of help, we said, "Okay, please score those requirements on a scale from 0 to 10." And to not just blame somebody that he writes bad requirements, also give that person clear ways to improve this requirement to actually get a higher

score. We implemented that um based on a framework called LangChain, um which allows you to use different um large language models underneath. And with Bazel, we then combine this with TRCL. So, in essence, what you have is a single Bazel target that you call, so Bazel run some label. And that then automatically runs this AI check on the requirements. And that simplifies um this approach significantly. Um

given that you actually get a stability in the um behavior of the AI, you can also run that based in a CI. Works partially currently, to be honest. but in essence, this really makes writing requirements easier, and it also improves the quality of those requirements significantly. With that, I'm through with the topics that I brought with me. Um, I have two QR links to our QR codes

here. Um, one of them, the left one, goes um, to the actual implementation of the safety sentinel. be aware, it's work in progress. There are still a lot of big question marks um, that we need to solve. Um, but it's definitely worth a look. And the right one, um, that one, it's, well, it's upcoming. This is the place where we actually use the safety sentinel. Um, we're

starting on that and it's definitely worth that you take a look from time to time. Um, because that actually shows you how the safety sentinel behaves, how to use it, how to integrate it in your project.