The Developer’s Nightmare: How To Survive Compliance Che... Alexandra Hou Aldershaab & Thomas Vitale

This talk addresses the challenges developers face during compliance checks when trying to release software efficiently. The speakers, Alexandra and Thomas, who are experienced in DevOps and software architecture, discuss strategies to navigate the obstacles presented by compliance checklists, focusing on tools like SBOMs (Software Bills of Materials) and Dependency-Track. They emphasize the importance of automating compliance processes, such as license checks and vulnerability assessments, to improve efficiency and reduce manual workloads. By integrating these tools into the development pipeline, developers can streamline the release process while maintaining compliance requirements. The session promotes a collaborative approach between developers and compliance teams, highlighting how automation can serve as a foundation for a smoother software delivery experience.

Yeah, hi everyone and welcome to the developers nightmare. How to survive compliance checklists and still ship fast. So um Hey Thomas, I've got this really cool pull request. I made a really cool feature and I would love for you to just, you know, approve it. >> Yeah. They're waiting. Yeah, but I want to show them the new feature though. >> Okay. So Let me see. Oh, I

have to open the book. Uh that one. Yeah. Oh, you implemented a new feature. >> Yeah, I did a new feature and I did a pull request. It's very It's a very awesome new feature. Oh, I like the title. Quack quack, okay. Well, um yeah, it looks good. Let me merge it. Perfect. All right. So, I merged your pull request, but who are you? Oh, oh yeah.

We should probably uh should probably introduce myself. I'm Alexandra. I work as a DevOps consultant for Efficode. I'm part of the like uh the friends of Dorothy community group and uh and I would love to tell you more about that after this talk. If you want to know, it's part of Merge Forward. I'll still be here for the developer experience tag. Very interested in developer experience. And

uh probably should introduce the cool guy who improved my pull request. I'm Thomas Vitale. I work as a senior software architect at Systematic. It's a Danish software company. I'm really passionate about anything Java and cloud native related. Among the other things I created a project called Arconia for Java applications. Um okay, I approved your pull request. And now I even know who you are. Yeah, so obviously

oh Let's go to production. Cuz like I want to show all these cool people what I've built. I think that's a cool idea, but one does not simply release to Have you met the guardians of compliance? I mean, you have so many things to consider. You You just release to production. I'm sorry. I I approved the pull request. Maybe the build pipeline is even green, Okay. Change

approval board? Have you reached out to them? How about licenses, tickets, meetings, waivers, vulnerabilities, checklist, more meetings? I mean, it's not that easy. But But it was so cool. Okay. I I think we can find a way. Let's Let's break it down because it's a lot. It's very overwhelming. Let's start with licenses. Let's try to get approval for Did you get it? No. So, you don't have

any formal request for approvals. No. Ah, I'm afraid we can't release to Okay. I I think we can fix it. First of all, you have to send an email. And there is a company-approved license approval request template you have to use. Send the email to legal, add some spreadsheets with some numbers. Doesn't matter what numbers as long as they're numbers because somehow that helps. It makes it

look serious. Then you have to list all the components in your application that you changed. For each of them, add a link. That's a document tracking system where they can approve formally the license based on the process. You know, it's something like this. So, they can keep track of all the approvals over all the releases. And clearly it's all manual, so you have to generate a unique

ID for the approval request. They go in there. So, super easy, right? That doesn't sound like a lot of fun. I feel like I feel like I built something really cool, and I kind of want to show it to people. I don't I don't want to sit and do a lot of manual and I think we can do better than that, >> Yeah, I hope so. So,

uh first of all, we need to get like an overview of what what our components we're actually using, right? So, let's mention SBOMs, right? Cuz we want to we want to have a software bill of material. We want to make sure that we know what is actually in the application that I've built. Um And the cool thing is that uh because of the application how it's built

in this one, it's uh it's using Gradle, so Java application, and we can use the built-in plugin to generate an SBOM using in the CycloneDX format, which is one of the standard formats for how to do SBOMs. And so, this works for any language, right? Yes, so >> Most language like languages have plugins. This is the one that we'll be using today, but as soon as you

get a CycloneDX format, you can use it anywhere. Oh, that's awesome. Yeah. And what we'll be using is we're going to be using Dependency-Track to keep uh to keep an overview of what is actually in our SBOM for the application. Let me show you that. here we have Dependency-Track. And the cool thing we're doing here is that instead of me sending an email to legal with all

the components and all the numbers in the spreadsheet, uh instead, I want legal to go in here and create a license audit to basically state what is allowed, what is not allowed. That way, they get to maintain it once. Um and all teams, all developers get to take advantage of this, so we already know what's doable and what's not. And we can take a look at our

project in here, and we have a policy violations tab, and we can see that there are some licenses that are not compliant with what legal has set up. There could be multiple reasons for this. It could be that uh the license is uh well, legal forgot to, you know, approve that one. But then, if we can we probably still have to reach out to legal and get

that approved, but then they approve it once, and then it trickles down to all the projects. It will be available everywhere instantly. Um there's also the possibility that it is simply a license we don't want to use, because if we use it, we have to uh adhere to some demands that we don't want to do due to our company policy or something like that. In that case,

it's really nice that we already found it, right? Cuz we don't want to use that dependency. there's also the possibility of not being able to find the license. In that case, we if there is no license, we definitely don't want to use it because then it's all rights reserved and we are not allowed. It might also just be that it's hiding a bit and we can find

it if we dig enough and then we can add it in here so that everybody will know that this is the license that we have. And we'll be able to keep track of all the licenses like this and make sure that we're compliant. We can even set up so that if there is a non-compliant license, we will uh create a GitHub issue so that developers know instantly

that they are working with a license that's non-compliant. Yeah, that's really interesting because of course we would like ideally to work in the same platform in dependency dependency track, have legal approved the license least once, right? But if they really want to have an email or maybe an issue being created, then we can get dependency track to automate that creation, right? Here we built a custom integration

using the dependency track API so that automatically we create GitHub issues. Yeah, which is nice because we might not be able to turn the ship around in one day, right? So if legal does actually want an email with the spreadsheet, we can send them an email with the spreadsheet, but we won't have to build the spreadsheet or write the email. We can let the automation do that

for us. Oh, I love that. Yeah, so at this point I'm feeling like uh I got uh we got the licenses covered and we're ready to go to production. Well, you think that, but no. Uh So, next thing I I have to ask because they're going to ask. They are really, you know, they're really uh careful about this. Is there any CVs in your application? Uh yeah,

there might be a couple. Uh no, we can't go to production. We have to fix this. How many CVs do you have? Well, I think I think I think we got about three. Three? So, yeah. But But they're not really relevant cuz they're like we couldn't really we couldn't really fix them. They're not really They're not really important, but obviously I have to write I forgot that

I have to obviously make this waiver request to kind of explain what these what these CVEs are and how they're not really a problem in the application the way we're using them. Like I go through the checklists of uh all the things that we have to do to get this waiver. And obviously then I'll have to sit and wait until someone approves that this CVE is not

actually a problem. Yeah, there is a big problem with CVEs. Of course, we don't want to deliver to production software that is vulnerable. But the fact that is a known CVEs attached to one of the libraries we we use in our software it doesn't mean that our application is vulnerable. We have to contextualize and sometimes we have really bad badly reported CVEs. Like take for example the

curl project. Curl is everywhere. We have this goal in mind to reach zero CVEs, but I would say that's too risky. Of course, it's really good from a metrics perspective. You can have a dashboard centrally in your company if you reach zero CVEs, good. Your software is super high quality and then perhaps you used AI to generate everything from scratch. You're not using any library, so it's

zero CVEs. Doesn't mean it's secure? I don't think so. But I really like the use case from a curl. Now, Daniel Stenberg, the creator and maintainer of curl, published several blog posts about all the AI slope CVE reports that the project is getting. And what I like about this work that he's been doing, you know, making people aware of this problem is also that he's being maintaining

a list of all the worst AI slope reported CVEs affecting the curl project. And my favorite one is this. Someone reported, you know, it was responsible disclosure that because the source code is open source, then it's vulnerable because the attackers can see the source code. And that is like that's not a CV. It's reported as critical. So, we cannot fully trust CVs. They are an indicator that

something might be wrong, but we have to contextualize. And it's good because we already have dependency track in place. We generate S-bombs automatically using plugins available from the OWASP project for any kind of language. So, no matter your tech stack, there is a plugin that can generate the S-bomb for you. So, as a developer, you don't have to do it manually. And on top of checking the

inventory for the software components, we can also check for software vulnerabilities. We can check for licenses, as we just saw. We can also check for outdated components and for the integrity, whether any dependency we're using in the application fails the integrity check. So, let's have a look. We go back to Dependency Track. And we have here policies also about vulnerabilities. So, in this case, the security team

has configured a policy that we don't allow shipping to production if there's any critical vulnerability associated with the software. But if there is, what are we going to do? As I said, just the fact that a CV is reported doesn't mean that it applies to our software. So, we always need to go and check the description of the vulnerabilities. We need to be sure that that kind

of vulnerability is exploitable. If it's not, then we can document our assessment. So, we can say, "Hey, we are not affected." Or maybe it's a false positive. Sometimes, of course, the vulnerability is exploitable. At that point, we have to make a decision, but we can all document this through Dependency Track. And under the hood, it's using some standard formats to document this information. So, if the guardians

of compliance want to get all of these documented. We can keep the audit trail of all these assessments done by the development teams. Now, I would like to tell you that there is a magic solution to skip this assessment part, but I'm not aware of any of that solution. So, if you know of one, please reach out to us afterwards. There are some tools also in the

OWASP trying to find if certain vulnerability are exploitable in a more automated way, find the reachability path in the software, but for now there's not a solution that works 100% of the time. So, at least we have to be aware of that. So, in this case, I have a vulnerability. This is a Jackson library. It's for parsing JSON in Java, and I know that actually this vulnerability

is exploitable in my application. What am I Of course, I can create like I showed earlier GitHub issue. So, as a developer, I can get the issue assigned and I go and fix it, but even better we can automate all of that. So, in this case, I am using Renovate. That's an open source tool that I integrated in my GitHub project, and you can run it anywhere

also on premises. And what it does is continuously check all the dependencies in my repository, and if there's any new version detected, then it will create a pull request for me to review. So, now I can go in here. I know that the updated version of the JSON library is part of this plugin. So, I can go ahead, approve it, and merge the pull request. So, there's

not much work in vault in handling vulnerabilities like this because this is all part of the same pipeline, right? So, if something fails, I know that the new version has some incompatibilities. So, I have to spend more time on it, but if it's green, then I should be able to trust the pipeline. And it might be worth mentioning, which I forgot earlier, that obviously getting our our

SBOM into dependency track requires a little step because it doesn't magically appear in there. That would have been cool. Uh if we have to kind of tell dependency track. So, we do have in our CSD workflow, which is just a GitHub action, we do have uh after compiling and obviously building the thing and and and creating the S-bomb, we also have a publish S-bomb uh action that

will just send it to dependency track. Yeah, and there are some out-of-the-box integrations built tools, but since dependency track has this nice and convenient REST API, you can build your own integration if it's not supported out-of-the-box by the platform that you're using. And since we're here, just a little hint, uh a lot of the dependencies used for developing and building this project are defined as code. So,

we don't have the situation where I have my development environment defined locally, maybe following a wiki page, and then I have a completely different configuration on the build pipeline, but I'm using a tool called Flux. It's based on the Nix ecosystem. Uh so, the entire development environment is defined as code. So, I can activate my development environment, and in here I will have a list of different

tools, and they will be the exact same tools and versions that are also used by the pipeline. So, that means that if something fails in the pipeline, I can easily reproduce it locally. I'm not depending on something that is a vendor specific. Flux works across uh different operating systems and architecture, and really uh provides this nice developer experience when the build fails because again, we want to

focus on the business logic, right? Not debugging problems in the build pipeline uh or dealing with all this uh paperwork. All right. Yeah, so so now we got we got we got the licenses under control, and we got we got the CVEs fixed. So, um Yeah, I know what you're going to say. Production. I would like to say, yes, let's go, but I'm afraid we're not ready

yet. I mean, we need They're going to ask They're going to ask to document the entire process of building our application starting from the source code, how we build it, how we package it. Did you document that? But but the source code that's simple, right? Cuz since we're using Git, we already have like an audit trail and we know who did what and who made what and

what code we have, right? They they won't allow that. Oh. Yep, we all use Git and Git has this night nice property. You can use it as an audit trail. You can see all the commits and who made them. So, for example, I have here a repository. The first thing I do with the repository is configuring my username. I can configure my email for GitHub and now

I can start making changes very creatively. Then we All right. Now I'm going to define a commit message that is very telling of what I updated. So, my teammates will be happy about it. And then I'm going to push. Now, if I go >> No problems with that. Yeah. It never happens, right? >> Perfect. Okay, now on GitHub I can see there is a commit from me

committed now. It was me. I updated something. We don't know what because I write terrible commit messages. But probably if you do something like same process, right? Yeah, we can see who did that what who did what. So, you would also go there and define your Yeah. Alexandra, right? Definitely. Yeah. You probably also configure some email. Yeah, just my email. Yeah. Lowercase? Probably. Something like this, right?

And then you would also make very creative changes. Yes. We share the same creativity style, yes. >> Yes. And probably also uh, commit using update cuz, you know, I updated the thing, right? >> uh, more update. Just Oh, yeah. So, it doesn't look like you copied me. Yeah, yeah, exactly. A very original. Like >> And then I can push. What happens here? You're not me. Oh, I

just hacked your Git account. You were not expecting that, right? That's not okay. So, you are all witnesses here, right? So, it was me making the changes. What happened? Git doesn't have the concept of authentication. You provide this metadata as configuration in the CLI, and that's it. So, you can be anyone you want to be, really, in Git. So, clearly, something doesn't work. Git by itself doesn't

provide the audit trail we expect, because it's not validated. You just saw now, it looks like Alexandra made a change, and GitHub even shows a nice link, so I can check your profile, but yeah, it wasn't you. So, the problem is uh, we need to introduce authentication somehow. So, a solution to it is enforcing that all commits are signed. In my case, I have configured an SSH

key, uh, with GitHub, so I can prove that it's me making some changes. So, if I, uh, update this again with uh, Git, and commit again, this time I'm going to add this, uh, dash, uh, uppercase S flag. That's for signing. Not to be confused with lowercase S, which is short for sign off, but it's not for signing. That's interesting. Now, I'm going to update and push.

So, I signed the commit with my private key. And what happens? Let's see. Now, GitHub says, "Oh, this is unverified." We can see some metadata information in the commit done by Alexandra, but I'm using my private key. So, if I now switch to my uh, uh username and email, I can use my private key, and then there's a match, and GitHub will say, "Okay, this is verified."

So, we can enforce this in Git, in uh GitHub, or GitLab, or other platforms, so that if the commit is not signed, we're not going to uh let it through. >> that's that's a good part we can do there, right? Yeah, so now we have an audit trail of who who actually wrote the code. Yes. But, you have to document and update the documentation for how you

build it from I I didn't do that. That That also, like, it's just updating yet another Confluence page. Like, it's it's where knowledge goes to die. Nobody wants to >> keeps documentation up-to-date. So, you didn't do that? No? Nah, nah. Uh Okay, but then, just like we just saw, we can we can sign our commits, so why not why not take that even further? So, what we

just saw is we can we can sign the commits, and we can make sure that the commits are actually made by the people that say they did the It looks like who did the commits. We can also sign our images. So, if we use uh Cosign by Sigstore, very, very simply sign our image so that we know so that everybody can verify that this is actually the

image that I wanted to put into production, and not some random image. Yeah, but okay. Okay, so now you're signing an image. I know it was you, but I don't know what's inside. Maybe you want to get back at me for hacking your Git account. There's some malware in it. I I don't trust it. I don't put it in production. There's also there's also the very, very

unlikely and hypothetical scenario of me building an image before I pulled the latest version of the main branch. Which obviously, like, never happens. Like, never happened before. Um but yeah, so we kind of want to make sure that we're doing this not just locally on my machine introducing another type of human error. So, we want to do this in our pipeline. So, if we go look at

our Yeah. This one. And not only do we So, we can see that this is where we publish the S-bomb. But, if we move even further down in our action, we will be doing Here you go. We will sign the image as part of the pipeline. So, when the pipeline is building the newest version, which is the one where we generate the S-bomb, which is the one

that we're getting all the information in, we can also sign that image. So, we know that the one created in our is actually that image that we're also running in production. And we can even take this further by adding attestations to the image. So, this is a very small little command, but what is really cool is that if we then go to our actions tab, we'll be

able to see down here the attestations. So, not only did we sign the image, but we'll also automatically have an up-to-date thing about when did we create things, what commit was it, what is the build summary, what's the workflow file. And we have all of this information readily available and automatically generated, which means that we are not we're not relying on someone updating a Confluence page manually,

potentially copy-pasting something wrong, or just, simply forgetting. Cuz that never happens. We always keep our documentation up-to-date even if we build it manually, right? Okay, so here I can validate whatever you put into the package, right? Trace it back to the Git pipeline, all the code, all the commit. Okay, I trust you again now. And yeah, so this is basically uh this is basically Salsa attestations. So,

this is a really cool project framework where we're kind of looking at, you know, what are the different aspects of the supply chain that we want to secure, how do we do this, and how do we build these attestations so that we make sure that we have all the information needed to have a secure pipeline for our software development. production? There's one tiny thing. I think we

need a meeting with the change approval board. Yeah, but so we don't really, do we? Cuz we've automated absolutely everything and we can take it even one step further. So, we can say that our pipeline will never, you know, run it through if the licenses are not working. It will never if there are any CVEs found that we haven't dealt with. It will just simply block it

right there. We can, since we are deploying this into a Kubernetes cluster, we can run Caverno policies where we say, "Well, if the salsa attestations can't be verified, then we won't actually run the image in production." So, we can make sure that all of the things the change approval board would normally be, uh, crossing off their little checklist in the meeting, we can make sure that all

is built in. So, it would never reach production if we haven't already thought of all of that. So, >> Okay. Okay. Don't tell them that we skipped the step, but yeah, let's go to production. And let's finally get the audience to try out your awesome new feature. Yeah, >> please scan the QR code. Check out the application and, uh, we >> what's the feature about? Because I

just, uh, blindly approved it, but I didn't look at the code. Yeah, it happens all the time, right? >> Yeah. So, yeah, we built It's a little thing so you can kind of let us know, what you have met of these, uh, very, um, hypothetical sort of scenarios of how to do compliance, which ones is hindering you in your day-to-day life. Um, cuz we're probably not alone.

Like, we work for different companies and we've seen a this in various versions throughout our careers and we'd like to know which one is your favorite or at least That's very interesting. Yes. So, we're leading with platform operations, team bottlenecks, uh slow manual approval processes. Okay. it's kind of uh reassuring that we're not the only ones with these problems, Yeah, it definitely looks like uh they are

not as hypothetical as we would have loved them to be these scenarios. Wow. Okay. But at this point though maybe we should ask ourselves a question, right? We have all these things we have to jump through as developers if we want to get something to production. But do we really need Do we really need compliance? Is it really needed? Our answer is that yes, it's really needed

and we need it now more than ever because we have all these urgent applications now. We are getting more and more code being generated, pull requests being approved without even being reviewed. So, we really need those guardrails in place because we don't want to put in production something that is unsafe, something that is vulnerable, something that will hurt or damage our users and customers. So, we really

need compliance. The problem though is that usually we implement it wrong. The way that we check for compliance uh creates this friction, creates this developers nightmare and we would like to fix that. And we saw that we can actually automate lots of those checks and uh maybe push all these concerns down to a platform so we can that we can get compliance as a service, right? Yeah,

and you've already seen us uh look into uh we'll show you dependency track, how we use that to manage uh licenses and vulnerabilities. Uh briefly mentioned Kivana to uh to verify self attestations and make sure images only reach production if they are made the right way. But, this is this is quite a big big mouthful and and how do we how do we kind of avoid moving

just the toil of writing a bunch of emails and creating spreadsheets and stuff into having to learn all of these systems? Yeah, we don't want to replace it with all this cognitive load. But, that's the role of Crossplane up there because with Crossplane, the platform team can define a nice developer-friendly API like this one where as a developer I only need to provide the information specific to

the application like what's the name, what's the image I want to publish, what are the services I want to integrate with, and then everything else is supported by the platform out of the box. So, all the S-bomb generations, license checks, all the signing, that can be all automated and made available by the platform. And the platform team can interact with the security expert, with the legal team,

with the compliance people. So, they can all contribute their skills and expertise into building compliance into probably you might not want to use YAML in all situations, but you can make this the foundation of a nice golden path. So, if you have a developer portal, for example, with Backstage, you can create these templates. So, whenever you bootstrap a new project, you get all of these out of

the box. So, so are you telling me that next time I want to create a new project, I just go there and then I have everything already ready and I don't have to actually jump through all these hoops and do all these things myself? >> Yes, and you can just start building the new awesome feature. Yes, cuz features are the best. >> Cool. They're happy now. Yes,

and that really ties it all together, right? Cuz not only do I get to focus on building features and delivering value to to the users, to the to my customers, I get to do what I do best, and I get to do it while the guardians of compliance are actually happy, not feeling the toil, and then we work together cuz we do want the same goal. We

have the same vision. We want to develop something that we can give to our users that they will be happy with and honestly they're probably not happy if we leak all their data or if everything is hacked all the time or if we don't have our licenses figured out. Exactly. We let's be friends. We are part of the same company. We want to contribute to the same

end goal. So >> I'm happy. I'm so happy too now as a developer. So now that we're all happy I have another pull request, right? >> Again? Yeah, I built this other feature. >> That was fast. Did you use AI? Like why why was it so fast? Yeah, it works in the background. It's really cool. Just have to find it. Yes? Yes, I also I did a

very awesome other feature. So do you make any features that are not awesome? Like what's the criteria that >> I'm awesome so all my features are obviously awesome. And look, dependency track is even happy with it, too. Okay. Well, then I'm just going to go ahead and merge it. Perfect. >> Cool. All right. So there was like >> Well, um so I kind of want to put

it in production, but there was was this something that I like I lost track. What do I need to do? Oh, it's already to production, right? We already made it everything. Yes? Right. Awesome. >> Enjoy. Yeah. Well, thank you for coming to our talk. Thank you. Feel free to scan the QR codes to rate our talk. Any feedback would be highly appreciated. Thank you.