Applying and scaling Trustable Software Framework

This talk introduces the Trustable Software Framework (TSF) and its application in software quality assurance and safety engineering. The speaker, who comes from a background in complex software applications, shares his experiences in bridging the gap between traditional safety engineering practices and modern software development, particularly in open source. He discusses the TSF method, which aims to define and measure quality goals in a generic yet applicable way. A practical example involving a template project demonstrates how to integrate TSF into continuous integration workflows, automate the scoring of quality metrics, and manage documentation effectively. Ultimately, the speaker emphasizes the potential for collaborative development within the open source community by sharing resources and refining quality measurements based on the TSF method.

[music] >> Let's welcome everybody. I wasn't expecting more than five people at this point of the conference. Um thank you for being here still. Question, who of you has like either this week or earlier heard read anything about the Trustable Software Framework method? Uh Edmond, you don't count. Okay, most of you Who has no clue what that is? You are okay like raise your hands because then

I know kind of where I need to start. Okay, one or two people. Okay. We'll if you're missing anything like if there's a gap or a step that I should be talking about you will shout, please. We can also can send the presentation to someone that has a working thing. All right. Um So intro. You can see a picture in a couple of minutes. I have been

following seeing like you know talking about with the guys that TS F method when it came up one and a half years ago plus or minus. Um I have been I'm a software guy meaning I'm coming from a world of like real computers like always Linux plus not embedded which means complex applications, complex dependency chains. Um So that entire embedded world is something that when I you

know started in the industry was new to me. And I since then essentially until today I've been having conversations with safety people coming from that embedded space talking about things like ISO and I've since then been trying to kind of overlap those two worlds because methodically speaking there is not a lot of overlap between, you know, what these methods want you to do and a complex application

that pulls in 120 rust crates or something like that. That That simply does not compute. >> [laughter] >> So, based on that based on that context um first time I heard about TSF, I paid attention because that was the first method that I saw that was trying to address this gap, you know, to be formalized and formatted in a way that could actually be applied to a

context like a complex real software project with complex supply chains and even open source development models. I looked into the method a bit. I talked with the TSF guys a little bit. Um essentially where I'm coming from never having been part of an operational project that really does safety engineering. That's just not my professional expertise so far. Um I can understand the goals of all of that.

I could understand I can understand the method, but how to really actually apply that like how to get going. What is like, you know, target granularity of your statements, etc. How to really actually put it into a real GitHub repo and so on. How to automate this, I had no clue. And the reason I built a little project that I'm going to show you today soon. This

is like doctors trying to save the patient. >> I can send you an email with the presentation if that helps. A bit of got network, right? So, that works. So, I should I send it to you? Should I send it to you or to you? Let's see. I thank you for bearing with us for a minute. We shall be It's picky. We almost made it to the

end of the conference without any hiccups. So, thank you, Daniel. Uh Do I have you here? Thank you, Daniel, for providing us some uh some relief. All right. Um so, essentially, I decided to build a little Hello World style template example project where I can try all of that. Apply the method in a very, very shallow way, but apply the method integrated with the CI, automate as

much as possible, and um essentially, create something that could become or is supposed to be an extended part of the TSF method documentation onboarding kit, if you want. And that's what I've what what I've built, and that's what I wanted to talk about today. Is that the right one? Let me attach the file. Some excitement at the end. Yes. So, maybe while we wait, since that sounded

a quite informed audience. So, you've heard about the Trustable Software Framework. If anyone has anyone tried to use it, actually? Okay, that is Well, I kind of thought as much, but >> [clears throat] >> Okay, has any one of you been sitting in the workshop that we had? [laughter] Uh yeah, okay. I see a couple of people. I'm trying to remember my slides. Okay, so that's I

talked about what I was building here and I will I I will soon show you what I've built before I have the way I've been thinking about this, let's say, workflow or the application of a method like this. And before I Sorry, before I go there, I'll have a quick um 5 minutes or something about TSF, the method, actually. Um anyone who knows better, shout. Anyone who

doesn't get anything, shout as well. TSF is a method that's supposed to be uh supposed to enable you as a project um to define and measure quality goals. Very generic, the method is uh supposed to be generic, but also informed by the context of quality goals that are related to safety engineering. You know, things like your feasibility and, you know, how to define what exactly your software

is supposed to do, how do you measure what it's doing, etc., etc. Um but the method as such is quite generic, so it can be used to measure any kind of quality goals. I'll get to a couple of examples. Um and the way it works is um formally speaking, at the end of the day, it's uh the statements that you make, the goals that you want to

reach, and how you break them down to simpler statements that you actually can prove. That's all done in little text files, markdown snippets, which live in your in a in your GitHub repo. So, this is part of the code. There's no separation uh from what you're doing compared to what your goals uh goal description is. Um part one. It uses an standard open source uh graphing framework

library dot graph to kind of link these statements up into a tree, which essentially means, you know, you have a goal, and that goal is broken down into sub goals, and if necessary, detailed out into sub goals to a level of granularity where you're making a statement, a goal statement that you can ideally prove to be true or false. I'll have an example essentially saying a project

should have documentation. An Eclipse project should have a good project readme. My project has a good project readme. Yes no, I can prove that. Whether it's good, okay, that's a second stage beyond that, but you know what I mean. And essentially, first part of TSF, the method is having that conceptual framework for defining that tree graph containing your goals, and the idea to as you work through

what you're doing, you're detailing out the graph, you're adding statements wherever you want to kind of, you know, increase the coverage of your quality goal and measurement. That's that's the idea. Um that's the first part. The second part of the method, or like of the project, is a little tool, also built by the TSF folk, which essentially is used to manipulate that graph, add an item, link

an item to that parent, um remove an item, etc. Also, um that tool is used to compute the score, the quality score based on the evidence that you're providing for items. So, you know, I have I should have a readme. Here's my readme. Okay, you get a 1.0, and then you get that kind of computation across your graph. You You sum sum up your sub scores, and

you know, as you go up to the top, and you end up with a top level score for that thing that you're doing. Essentially, this can be considered a metric that, for instance, you can use in a simple, you know, dashboard. I'm improving over here, but this got worse. Or I can't use that metric to kind of reject or not reject a pull request, you know, because

it did improve or not improve that metric, um like, you know, code coverage test, for instance. That's the kind of um operational thing you can do with that number. Hey, thank you, Sarah. Now? Almost there. >> Let's try to catch up with what I've been talking about. What did you do? Okay, let's recap. Context, why I told you the story, why I became interested in TSF, that

is the slide for that. Commercial versus open source development models. Open source projects are understaffed. People don't have time or patience for meta things that they don't see a value in. That's a bit Let's say there's different motivations and different resource situations in commercial projects because, you know, having a high heavyweight and a complex process over there, well, you charge your customer for it, right? That's fine.

That's how we do uh safety in the industry for like a long time. Essentially, the gap that I was talking about Then uh also what I spoke about TSF, uh when I saw that thing, I thought I recognized that there's design goals here that help bridge that gap, that make that kind of method um applicable to a project that needs to automate everything, where people do not

have a lot of time and patience for meta So, that's the recap for that. Now, I started to speak about the TSF method. This is an an example for the core TSF part of that graph, a little snippet. But, essentially, you know, what is that? There's high-level goals that are broken down subsequently into more specific goals, and you as a project applying the method, you will detail

out that tree down to a level of granularity where you can actually make two-force statements based on what you're doing. That's the idea. There's different areas, measurement areas uh proposed by the core method. You're free to extend them, but, you know, things like expectations, aka the bad word requirements, what is this supposed to do, how is it supposed to do how is that thing building, can I

reproduce the build, all those kinds of things are part of this core core framework. quick uh my last information regarding is that thing actually, you know, worth a damn? You guys can update me if I get this wrong. The latest statement in that context that I have that I'm aware of is the method as such has been essentially accepted or um approved {quote} {unquote} by a um

validation authority to be sufficient if applied correctly, sufficient to achieve, you know, quality assessment goals like ASIL for automotive. So, the method as such, again, if you apply it correctly, etc. etc. should get you there. That's my latest. Okay, now that was the intro. Now we're synced up again. Any questions so far? No? Okay, keep them for later. All right. Now applying the method to something tangible,

to something, you know, that you're actually doing. That's what I wanted to figure out for myself. And also, I wanted the result of that. Let's see what happens. But I was hoping that the result of this me figuring this out can actually be maybe useful to other projects in, you know, maybe this community. That's why I called the talk scaling TSF. I'm not uh right now I'm

not thinking about scaling this to a huge project, but scaling this into a community. So, that's the idea behind what I've built. And the conceptual um approach or like my mental model for how I was thinking about what I've been building that applying a method like this essentially breaks down into two phases or two things that you do in my mind. Again, people who know better, please

feel free to correct me uh later. Um the first um intro stage, if you will, is selecting the method and doing an initial adoption, tailoring, contextualization of whatever the method is to your context. Right? So, you select whatever you want to do and then you set it up and maybe you can do a few initial kind of tailorings of that graph to make it fit your context

and maybe kind of, you know, hide, disable certain parts of that branch because tree because they don't apply, etc. That kind of customization uh, stage one. That's what you do to get started. You do that once, ideally, or only occasionally, right? But, that's set up. Um, stage two is the day-to-day. Like, as you're developing your thing, building your thing, uh, you write code, you write documentation for

the code, maybe tests, uh, specifications, etc. That stays the same. The idea now, if you're applying a method like that, is you have to do two extra things. Um, extra thing number one, you will have to continue to expand and detail out your TSF goals graph, right? You're doing something new and that new thing, ideally, will support or aim for supporting a quality goal, um, or, you

know, some aspect of the graph. So, you have to maybe create a new part of the graph, a new node, you know, that you then work towards. >> [snorts] >> Like, for instance, I now need a readme for my project, so I can expand that part of the quality graph to make a statement, I have a readme with those kinds of attributes, for example. That's part one

of applying TSF on top of your regular workflow. part two is you need to provide evidence that you actually are doing something for the goal you just defined. So, if my goal is have a readme, well, then I should think about how, for instance, for instance, doing a release build that I do, how can I now formally, ideally, prove, create a link between I have a readme,

this is the goal. You can score this now. Like, how do I document link evidence to support my quality goals? So, that's the day-to-day thing that you do while you build your project, while you build your thing. based on these uh steps so far, the initial setup and the day-to-day, you now have a lot of information, a lot of structure that you can use for automated generation

of dashboards, of reports, of the scoring of your project documentation, etc., etc. So, like, build pipelines, release um merge PR, merge pipelines, etc., can now use all of that to do interesting and hopefully helpful things with that data. So, that's the three stages. Um stage number one is that contextualization tailoring of the method of your graph. I have not been doing a lot of that in my

project. However, this is where I think um TSF has a lot of applicability and uh value add for an open-source community and development model because Here we go. I was talking about uh tailoring your project to a context. What is a context? Context can, for instance, be a technology context. Every Rust project, for instance, comes with a bunch of features, capabilities from the language definition, from the

Rust compiler. There's a lot of Rust ecosystem tools that help reach quality goals. The Cargo uh the the crates infrastructure is amazing and again gives you a lot of tooling, a lot of input for free to reach some of those quality goals. So, if you're a Rust project, for example, there is a lot of graph detail nodes that you can reach almost for free. But every Rust

project can reach those quality goals almost for free because it all use the same tech stack. Not every Rust project, however, needs to reinvent and re define the Rust specific quality goals, right? We can share those. Once we have a Rust statement library with Rust quality goals, then the next project does not have to redo that work. We can evolve that together, and every project doing Rust

can now pull into their quality graph the Rust derived goals and can instantly essentially have a starting point for, you know, getting a certain part of your tree scored. >> Second example is something like uh the Eclipse Foundation. The Eclipse Foundation defines um the Eclipse project handbook and the Eclipse process. That's, you know, formal development process like uh definitions. Uh for instance, they require that a project

has a good readme. We all share that context. So, from those documents, you can derive one or two dozens of of TSF statements, you know, that formalize the TSF that the Eclipse requirements into your TSF And once we have that kind of TSF Eclipse context, we don't need to reinvent that every time. We can share it and evolve it together. Formalization of the Eclipse project handbook essentially.

So, that's examples for what these tailorings might look like, and that is one of the huge things where I'm thinking the method being open and more than one project applying them in an open source style, we can share those contexts. Um I have created a Rust and an Eclipse library to get started. Uh you will see or you can see um those in the project I I've

put together here later. conversation is ongoing with the TSF folks because to make this work, right? That approach needs to move into the TSF project where we have a country folder and, you know, a little community process to manage that kind of library. But that's the takeaway for that stage one. Where's the value at? Why is this cool in the context of an open source community? Stage

two, the doing the work part. I was talking about that. You need to define your graph, and you need to link evidence of what you've been doing to support the statements, the definitions that you made in your graph. This isn't really something that can be automated because, you know, that's where you have to have a brain, ideally a human brain, because something with intention and a real

world model. But as I also said, CSF statements are little markdown files, ideally short and brief and simple ones, so you know, we can do that. What I've been doing in this project, you'll see that later hands-on. built something to automate only do once the linking part between evidence and the supported statements. So that, you know, if I create a readme to support my quality goal for

the readme, I can now once in my CI in that case, document that connection, yeah, for this note in the CSF quality graph, this, for example, file in my repo is the evidence, and I only do that one time, and automatically going forward, yeah, in in release builds, for instance, that link is used to score my graph, get the report, and so on. Again, you'll see examples

later. At that stage three, again, I'm not reinventing any wheels. I'm not doing any rocket science. I am combining all the tools that, for instance, the TFS guys use anyways, some other obvious things like MKDocs or GitHub Pages for working with all the data we've now created. I've got the quality graph, I can do the scoring, I can create the report, I can embed this with my

other nice project documentation in a good project website, and so on and so forth. Again, you'll see examples later, but this is essentially um the automation part, the the core of the automation part. All right, to get a grip a bit a little idea at least of what that actually looks like, um we will now follow one example item through that process that I've built. and as

I said, this is shallow, right? We're not talking about deep safety engineering. I'm talking about do I have a read me or not because it's supposed to illustrate the concept and not give you a 2,000 graph node at this point. Um so essentially, but as I said, and as we just saw on that stage one slide earlier, you start by setting up your graph, meaning with the

help of the TSF tool, you pull in the TSF core uh starting point graph. My idea, as I talked about earlier for making this worthwhile in an open source context, you pull in an Eclipse specific context library, which, you know, uses later uses this generic statement to make a more specific one, like that supports the a statement above, and it this might not be all the breakdown

that you do for that top level graph, but it's certainly one possible breakdown. and then final stage, you do your project level things, um the first part of your workflow, you add your own project specific mapping of this context. The I project have a read me. Like Eclipse told me to do, like TSF wants a project to do, right? That's that's what we're doing. Essentially, as I

said, creating little markdown snippets. the There's tool support. You can just create a new file, put that sentence in fine, whatever. Question or later? What I just told you, there's a repo, it's public. You get the link later. Unsurprising, that's what it looks like. Right, this is markdown sentence, um three different files. Not rocket science. But that's what what what this looks like. One thing that is

maybe worth pointing pointing out, is um >> in the header of my project specific one, there's something that's that what you will not find right now in that core TSF documentation, this little placeholder thing here where it says evidence ref. So, um footnote. The TSF tooling essentially um expects or is able to pro- uh process evidence, any uh different kinds of evidence references in the header of

certain items. So, you can put a reference to a file or something in there, a certain uh syntax that's defined by TSF. And that then, when you score your tree and create the report, that evidence will be looked at and, you know, inform the score. And that's the place where this kind of evidence uh should go. Note, mental note for you guys for right now, I've put

a placeholder here. That's That's the reason for that. We'll get to reason for that, this is the second part of the workflow of the day-to-day workflow that I was talking about, linking the evidence to items. Again, I want this to be done only once. And I want this to then be automated, you know, in release workflows and so on, so that I don't have to repeat So,

what I've done um to kind of uh do that uh to to enable that is I've built a little GitHub action that I'm calling from my CI release workflow. GitHub action is creatively called TSF transfer TSFer. Um and it doesn't do much. The only thing that it does, it records a relationship between a TSF node ID, that's the going to be the project readme thing that you

just saw. That's the ID of that node. reference evidence reference information for that node. That's what that thing does. In my example here, the evidence that I'm linking is a download URL to my readme file, which I have earlier uploaded to my release artifacts in that release workflow. So, if you look at the GitHub page of that project at that release, you will see among the release

artifacts my readme file, the download URL for that file for that release I'm using as evidence for um this node here. That's how I record the link. Again, please let's not discuss that this might be a stupid example. That's not the point. the point is, however, and you can have breakout workshops, whatever discussions going forward, you can imagine that if you get a big tree in a

big project, you will have to spend some time designing how you partition and construct your CI workflows because obviously this will be um getting big as that was it for the, you know, day-to-day workflow essentially. You do that a lot if you're applying the method, but that's what you do. Um once you've done that, this is now like towards the end of my release As I said,

that's the automation that you use all the data that you put in. what I'm doing in my example project is stupid and simple because the heavy lifting is done by other open source projects, the TSF tool for instance and others. Um what I'm doing here, I'm calling another GitHub action that I wrote um called TSF link and what that thing does, it collects all these evidence references

that we that you saw on this page before, collects all of these from that entire workflow run context. It will then go through the TSF tree and everywhere where this evidence ref thing is in the header, it will insert at that place, you know, the relevant evidence for that item so that my graph is now complete with all the available evidence. It will then run the TSF

tool to score to create the report and then I can use the report and all that data for again nice project web page um dashboard something that for instance might be handed over to an assessor and so on and >> Again, now that my uh computer seems to work, I might even be able to show you live. And what that does at the end of all of

this, once you have your release run you get what I just was talking about, you get your project website with in this case embedded trustable compliance support um highlighting our beloved read me note, you know, the score for that and Let's let's try out whether I can show you a click around live a bit, but that's essentially um the brief short version of it is not very

complex, but it is quite a a lot of tools that I'm integrating here. So, the repo looks quite busy even though content-wise it's just that stupid read me example, but anyways. that was the principle and now what I've added is one little feature. It's a little feature, but it's my feature and I'll show you um to wrap this up. Um because that was the first time for

me personally where it clicked when I understood the value of requirements tracing. Because in the past people have been talking at me and talking me talking at me about having the need of for having requirements and then people showed me requirements documents, 500 pages of things that nobody ever reads, much less understands, much less is able to even like you make to hear in sense of. Um

so what I've what I've done here to make this make sense for myself and I do like the result. Um I use requirements tracing in this project as well. There's a little open source tool called Open FastTrace, but you can use any tool, Sphinx, Read the whatever. Open FastTrace is very simple and we're using that for another protocol project as well, so what it boils down to

is you specify Oh, I'm sorry. You specify a requirement. This is stupid, but but this requirement says I have to have a readme. Um that requirement gets an requirement identifier and essentially a little statement that says, "Okay, this requirement needs an implementation to be complete." That's requirements tracing 101. You guys know how that works normally. Um I'm implementing the requirement. So this is the header, the top

of my readme file. This little snippet here essentially just is the Open FastTrace syntax which says, "Okay, this file, this location is an implementation of this requirement identifier." Um and then you know the rest of my readme follows. Um and the Open FastTrace tool will go through your code base and will pull out all of these little references and create a coverage graph, right? What these tools

tend to do. So in my release workflow, I'm I'm doing the Open fast-paced analysis and here I'm using a requirements identifier like this one. Requirements identifier, I'm using that identifier as a type of evidence for my TSF graph for the project uh readme And by combining these tools and again not doing anything intelligent on my part, just putting these things together, the end result is in that

TSF for this node for the evidence section of this node, I can now render out that little um representation of my requirements. So like for this node, I'm using this requirement as evidence. I get the requirement text. I get the requirement specification location. I get the status of the requirement and I get clickable links to wherever things are implemented, tested, and so on. And this again is

almost for free, but this means now actually you have requirements accessible in the context of the actual bigger quality goal and you know, the thing that you actually want to do. And again, this might be simple and stupid, but for me that was the first real useful application of that concept because now you know, it's where you expect it to be. Anyways, um Sarah, should I Oh.

>> Okay. No worries. Um you can No, let's see. The link is here. The link is also Ah. Maybe status of closing thoughts. Um what I've built here and you are very much invited to have a look. Um This is right now living in my own GitHub um account. Essentially, this is not an Eclipse project or anything. I would like this to be a bit more visible

slash a useful to people which means um I we started uh with the TDF project started conversation with the Eclipse folks and the Eclipse Dash project um to kind of maybe later this year move these pieces over to where they make sense. So, I have two GitHub actions. I have my example template hello world thing and um I do have a little render for my you know,

for some of these evidence types. All of these would belong or make a lot of sense in these locations. The conversation about that has started. Um so, right now if you look at that link on the next page, um don't be surprised. This is my repo. it's going to move somewhere, I hope. Um if you want to read about that uh you know, TSF in general, there's

one pointer here that you can use as a starting point. Um and anyways again, link links all over the place. Let me know, you'll get the slides. Now, we just need to get it on screen. >> Yeah. All right. You guys rock. This is the project website that I've what I showed you earlier that I'm generating uh with my release CI pipeline and that I'm essentially uploading

to GitHub pages. This is an MKDocs materials for MKDocs webpage. You can see, you know, this is the version or like the the generated page for release one 016 of what I've been doing over your page. I've got my you know, all the project read me all the everything that I told you a lot more structured and clearer. I've tried to write down here. Can have a

look. little things that are for free because the data is there. I can just add a little badge giving the last scoring that I you know, got for that release. Like this is cheap. This is a one line link web web API call. Um that's all all of that is getting very easy once you get your things all in that one place. And you know, you can

I talked about requirements tracing. So um the open fast trace report for this release is available. Can be published. Can be um looked at. Can follow uh the requirements things that you're interested in. I've got the trustable report. There's a dashboard. Again, don't get hung up by the score. It's horrible, but this is about the read me not about the score. And uh the that little feature

I was talking about and we are talking. This is the um uh the detailed report that you get just generated by TSF by the tooling. There's a bunch of notes that I'm not doing anything with, right? That's the majority of my tree, but I do have something for my read me. Can click on that. Now you get the detailed uh breakdown of why this thing is getting

the score that it's getting down here. That's the evidences that I've been providing. So you know, my download URL for read me. That's uh that's the first one. That's unexciting. I can now click here and you know, I'll be taken to that file. And again, the little feature that I like is this one where you essentially now get your breakdown of this requirement. You can look at

where it's defined. Um you can look at the implementation The repo is also online but I don't want to jinx my luck now. Thank you for listening. Thank you for sticking in this out until the end. >> [music]