Engineering for the Long Haul: Operating Open Source at Scale | Artem Karasev (EN)

In this talk, Artem Karasev, product marketing manager at TuxCare, discusses the challenges organizations face in managing open source security at scale. He highlights the alarming increase in open source vulnerabilities, with 107% growth reported in commercial code bases. Many organizations struggle with the patching process, often knowing they are vulnerable but unable to apply security patches due to various factors such as downtime and dependency complexities. TuxCare offers solutions like rebootless patching and endless life cycle support for open source software, ensuring organizations can maintain security without significant disruptions. By monitoring vulnerabilities and providing timely patches, TuxCare helps organizations secure their systems and manage their dependencies effectively.

[music] >> All right, I guess we can start. Hey everyone. Thank you for coming. I'm actually surprised seeing people here cuz I was thinking like last day of the conference, like sponsored talk that may seem unrelated to what you do. But I'll try to keep it interesting for you. Anyway, thanks for thanks for being here. My name is Artem Karasev. I'm product marketing manager at TuxCare. And

we secure open source. So I'm going to talk a little bit about it during today's presentation. But just to start a quick show of hands like how many of you manages or like contributes to more than one application in production. Okay. Quite a few. So I guess then it is quite possible that you're running some component that is end of life or is going to be end

of life in the next 12 months without even knowing about it. And chances are that there is a security patch sitting somewhere waiting to be applied, waiting for the maintenance window that actually never comes. And this is actually what today's talk is about. So I'm going to walk you through the based based on our research, the challenges that organizations face in when they try to keep open

source secure and reliable at at scale. And some of it may like you may already know about and some of it might seem surprising. And along the way I'll show you how TuxCare with its uh product portfolio can help to uh solve those issues. So, uh let's start. And to frame the conversation, there are like three numbers. Uh first of all, there is a 107% increase in

open source vulnerabilities in commercial code bases, and uh this number is coming from the 2026 Black Duck report. Uh and I guess that correlates really well with uh what we all do. Uh so, adding more dependencies, we're adding them faster, AI assistance uh suggest popular libraries uh with deep dependency trees. and uh 48% of open source using organizations uh face cybersecurity incidents in the last 12 months,

and they it actually scales with size. So, among organizations uh who have more than 1,000 employees, uh this number was uh above 60%. So, 61% of organizations of the same organizations faced incidents because uh uh there was a patch, but uh it hasn't been applied. uh organizations uh like patch was readily available, but for for the whatever reason, there was there was some issues that prevented organizations

to apply it. And uh there are some factors that uh explain uh what went wrong. So, when I first saw that stat, it uh made me stop like for a second. Like 93% of organizations knew they were vulnerable uh before the incident occurred. So, 93% uh it means that teams knew they were exposed, but for whatever reason, they couldn't apply apply the patch in time and close

this uh exposure. And when we ask organizations uh whether the patching speed has improved, if it uh if it like uh became faster or like if there any changes, 49% told us uh there was no change whatsoever. So, not slower, uh not faster, just stuck. And I'm curious like how many of you think that uh the organization the organization's patching process uh has improved like has become

faster in the the last 12 months? Okay. Okay, so one person. Well, that that tracks uh really with the results of the survey cuz only 1% of organiza- of organizations uh like side at mentioned that uh they they've seen an improvement in their patching speed. And uh there are several factors that uh influence that affects that. So, uh one of the major ones was uh downtime. So,

43% of companies were unable to patch systems because they couldn't take those systems offline. So, there are critical systems that uh need patching the most, but you can't uh patch them because you're afraid to touch them. And uh there was several others. So, 32% uh said that human error uh in manual handoff. So, someone needs to uh schedule a maintenance window, coordinate with uh different teams, uh

execute the changes, and when the uh jobs like the handoffs are done manually, like this obviously uh quite often quite frequently lead leads to errors. Uh 29% can't keep up with the vulnerability volume, and uh it correlates really well. So, remember that 107% uh number at the very beginning? So, it means that the pile of uh is growing uh significantly faster than the teams can uh work

through it. 24% can't track whether vulnerabilities are remediated, so they patch something somewhere, but are not sure if it uh took effect across the entire uh stack across all systems. And there is a major one uh at the bottom. So, nearly half uh told us that uh dependency tree complexity uh was the number one barrier to applying patches. So, this means that uh you patch one library

and it breaks uh like five others. So, what uh was meant to be like a a simple sprint uh turned out to be like a multi-month month uh refactoring or write project. And uh I guess like many of you like working with Gradle and Maven dependency trees uh face the same thing. So, uh you patch one component and suddenly your build turns red because of uh the

reasons like several layers deep. >> [snorts] >> Uh but let's switch for a moment from patching to the life cycle because uh these things are actually interconnected. So, nearly third of organizations actually uh third of organizations discover the component they are using is end of life uh only after a feature breaks. So, not through a like notification from a vendor, not uh from regular reviews, but when

when something stops working. And when that happens, uh you are already in in a reactive mode. So, this is why organizations, like 50% of organizations, tend to uh turn to extended life cycle support services because they need more breathing room uh to be able to uh plan their upgrade properly. And uh what what is on the right is quite interesting. So, based on our research, Angular JS,

uh like a framework which is end of life since 2021, is still used by 14% of organizations. It was actually the second most popular framework uh cited by our survey respondents. CentOS, like a very popular operating system rail derivative with the last version CentOS 7, which went end of life in 2024, still running in environments surveyed. So, this means that end of life is not quite an

edge case and many organizations uh have end of life systems like a a significant portion of their stack. And the migrations are deferred or upgrades are deferred not because the teams don't know, but because they just can't move fast enough and align their upgrades with the speed of new version releases. So, downtime prevents patching. Dependency trees slow like complexity of the dependency trees slows things down. End

of life systems stays in your environments for a longer than you expect. And these these are the kind of challenges that TuxCare can help with with its portfolio of products. So, we deliver rebootless patching for the Linux systems. So, we have two products KernelCare, LibCare that help organizations to avoid downtime and server reboots for their critical systems. So, we they deliver patches in memory uh for systems

that are went end of life and you can't upgrade for whatever reasons. We continue delivering security patches for those systems like whatever technology you may be using open source if it's open source. We will continue delivering security patches. Uh we deliver commercial support for Linux distributions, community ones, AlmaLinux, Rocky Linux. And we also have specific like Linux first patch validation tool to make sure clearly see what's

been patched on your systems and what really requires attention cuz like leading or popular security scanners often got things wrong because they don't account for Linux specifics like certain distros deliver security fixes to the same versions and security scanners just take the versions that are on disk to base their decision on the if if the vulnerability is there or not. So we that kind of specifics and

help to really get a clear picture and prioritize better. So I'm going to stop at the first two for a bit and actually this section is not about application code but about infrastructure. I'd like you to stick with me because this directly relates to how often you hear something like we need to take your service down for maintenance. when there is a Linux kernel vulnerability, someone needs

to schedule your team to take the service down for the maintenance hours, needs to patch the system, and needs to make sure that like to bring the system up cleanly. And clearly like anything can go wrong during this process. And actually the kernel is not the only that contributes to the downtime issue. Of course applications, middleware, they they have all their own update cycles. But kernel probably

is the hardest component to patch because it it always in in all cases requires systems to be rebooted in order to patch it. And on the right there is rebootless patching. So, what we do is we deliver that do not require systems to be rebooted. So, it all happens in the background. There is no disruption and no downtime. And I understand that you may not like interact

with it directly in your everyday job. But, actually you'll feel the difference. So, it means that there will be fewer pay we need to take your service down to to we need to take your service down on Saturday messages. There will be fewer deployment freezes around patching windows. And also there will be fewer kernel related incidents because of the delayed I want to stop for a brief

moment on how it works cuz the mechanism is generally interesting. So, what KernelCare does is that it allocates kernel memory near the vulnerable function. It uploads it loads the new secure code into it. Then it freezes all processes momentarily like for nanoseconds, which is not noticeable at all. And then it redirects the entry points of the vulnerable function to the secure code. Making that the old function

the old code will never be executed. Then it unfreezes all processes and that's it. So, the system is patched within like a seconds. It all happens in the background automatically. No downtime and no reboots. And this is about like kernel and the systems that stay in your components that stay in your environment for longer uh are also like not always uh easy to patch. So, uh to

avoid forced upgrades, we deliver endless life cycle support for open So, what actually happens when uh the component goes end of life? They're actually like each component, each version has uh a life cycle. So, uh an active maintenance support maintenance and uh end of life. So, after end of life, uh several things happen. So, security patching stops. Uh there are no more patches for from the community.

And cybercriminals start to uh like look for end of life uh versions and start to exploit newly discovered vulnerabilities. It's kind of a lay low-hanging fruit for uh for them. systems start falling out of compliance with uh internal policies and with uh external regulations. And when that happens organizations and when uh it's uh the end of life is found like reactively organizations are forced to uh rush

upgrades. And rush upgrades uh means disruption, technical debt build up, and uh unplanned costs. What makes the problem even uh more difficult is that the uh modern compliance standards like widely uh adopted uh standards like PCI DSS, HIPAA, FedRAMP uh and others, they all mandate uh strict security patching timelines. So, high and critical security vulnerabilities need to be patched within 14 uh to 30 uh days uh

time window. And when you are managing an actively supported systems that is tough already. But when you're managing [snorts] an end of life system, that becomes mission impossible. And this is this what makes organizations to force upgrades like in many cases. And when upgrades are rushed, there are several things started to happen. So, development and support teams are thrown off of their schedules because they have to

deal with bugs that start popping up. Uh workflows are disrupted. Uh short cuts that are taken to do the upgrade quicker uh like long-term maintenance headache. And uh unplanned costs, so overtime, uh downtime, infrastructure spikes, so all that inflates budget significantly. Is anyone here have been through through a rushed upgrades like in in your career? Great. How was it? Not good, right? What was the most painful

thing? I'm sorry? Oh, really? Okay, so you you are still running the old version? Okay. Okay, so you know how painful it can be. And based on the responses in the survey like most customers say that all four uh at like a different the different scale, but all four is painful. Uh and this is why we deliver endless life cycle support for open source software. So, we

support more than a thousand of uh technologies already. Different packages, and uh we can easily support whatever technology uh you may be using. uh based on a request, we add uh technology to to our support within like a of weeks usually. And we we continue delivering security patches for whatever version you might be using. We have a on using Fortran actually. Uh still which is kind of

sad but uh uh we're glad that we can help with even with such cases. Uh and what is tricky with end of life is that based on our observations which are actually validated by third-party uh 65% of end of life exposure hides in transitive dependencies. fixing the direct and transitive dependencies are the ones that uh got pulled in uh like without you choosing them. So they automatically

pulled in by your direct dependencies. So looking for the uh and fixing vulnerabilities in direct dependencies is good but it's not enough. Uh even a single vulnerability like in a in your dependency tree uh can expose the entire stack. Uh and this is uh why uh we cover both direct dependencies and transitive dependencies out of the box. So uh we monitor the entire dependency tree and whether

whenever there is a vulnerability in one of the dependencies we fix it and we rebuild the entire project providing you with the fully secured uh version. So what else endless life cycle support includes? Uh all our patches are SLA backed so we have uh different SLA levels for different uh CV uh different uh vulnerability severities. preserve compatibility with your existing code so all our patches are extensively

tested to work uh the same way as the original package. Uh they're drop-in replacements so uh don't have to change your workflows, you don't have to uh uh change your tools. You need just to uh add our dependencies in your configuration, and that's it. Uh our patches are delivered through uh repositories, private ones. Uh all packages are signed, tamper-proof. And our support is 24/7. So, it can

help you whenever there is a question with uh CV triage, patch request, or any troubleshooting you might need. For organizations that need verifiable assurance, uh we provide several layers supply chain transparency. So, we deliver uh software bill of materials that lists all the components inside uh inside like all all the parts uh that are used inside the package. So, you always know what what you're running. We

deliver vulnerability uh exploitability exchange contexts, so uh you know what vulnerabilities have been patched, uh what don't apply. So, this helps to uh prioritize your efforts accordingly. And we're working towards uh SLSA level 3 compliance. Just for verifiable source thing building and uh validation, and we are going to uh release it like in the next uh month or so or so. Uh how many of you actually

consume or uh produce S-bombs? Okay. Okay, good. So, then you most likely know that it's important for your providers to like deliver such artifacts because the compliance standards also get stricter uh whenever you install uh an open-source component, it's like your responsibility to make it secure uh regardless of who manages it. So, it's uh always important to know Uh all of this information is available at uh,

tuxcare.com or like there is a simple uh, simple simpler URL. So, cv.tuxcare.com. So, it's our CV tracker. Uh, it provides uh, it's single it it's a single source of uh, end-of-life vulnerability intelligence. So, you can see the uh, vulnerabilities, uh, the available fixes across all open source technologies that we and versions that we currently support uh, including their dependency chains. So, you can find vulnerabilities in direct

You can see all the direct vulner- vulnerabilities. You can see all the all the transitive ones. You can filter by severity, product, component, and uh, patch status. So, you can track that to uh, be always aware of uh, your exposure and whether patches are available and or not. And you can subscribe for updates when uh, new CVs pop up or uh, the status for the specific CV

changes. So, I encourage all of you just to check the portal after the presentation. Again, it's cv.tuxcare.com. Pretty simple to remember. a few words about how endless life cycle support works. So, we continuously monitor for vulnerabilities across different trusted sources. Like uh, it's automatic. It's 24/7. We monitor NVD. We monitor uh, GitHub security advisories, vendor advisories, like Debian security tracker, and and other sources. Uh, once CVs

are detected, we check if the version that we support is affected by the CV or not. And then, if it does, uh, then if if it is, then we backport the security patch from the newer version to the older one, to the one that we currently support. Uh, all patches are compiled in in binary format. Uh, so, they're tested for integrity, for regression, for for for functionality.

they are signed and deployed to our repositories from where like your package managers, your build tools, your orchestrators can pull them automatically and and install them. So pretty pretty easy like there is like the process is complex but the main stages are these ones. We deliver security patches backed by different SLA tiers. So for high and critical security issues, we deliver fixes within 14 days and it

covers the entire Endless lifecycle support portfolio. For medium vulnerabilities, we deliver fixes within 60 days and for low severity CVEs within 90 days. And here's the main question like why TuxCare? So we deliver the broadest coverage on the market. So whatever technology you might be using probably we either already supported on or we are ready to take it in into our support. It's pretty quickly and we

provide support across the entire open source We cover all direct and transitive dependencies out of the box. So you don't need to request a patch for a vulnerability somewhere in the dependency tree. They're all included without additional cost. Like we deliver like otherwise the solution is incomplete. So we deliver you with a complete solution. So we cover that out We provide end-to-end and we have a proven

source open source security patching experience which is based on 15 years of the company existence. So we've patched to date more than 5,000 CVEs which is which which can be easily checked on our CVE tracker. And by the numbers like 2300 companies are our customers, including Fortune 100 customers. we patched more than 10,000 key packages. We've delivered uh more than 200,000 patches. Uh fixed more than 5,000

vulnerabilities. 10K Linux kernels continually continuously supported and like almost all Linux distributions and their versions are supported as well. So, this is in a nutshell about TuxCare. Uh I'm happy to take any questions, but uh I encourage you just to come by our booth if you haven't already. Uh we have like cool swag. We're ready to answer any questions. And uh we're also going to have a

raffle at uh 3 45, I believe. So, you need to come by and grab your ticket. That's it for me. If uh there any questions, uh I'll be glad to answer them right now or at the booth.