About this talk
This talk focuses on the integration of risk management principles into IT security practices, specifically how to quantitatively assess and prioritize vulnerabilities. The speaker, Mike, draws from his extensive experience in risk management to address a common challenge faced by software developers and security teams: deciding which vulnerabilities to fix first amidst a vast array of threats. He critiques traditional qualitative methods, such as the CVSS score, which can lead to ambiguous prioritization, and introduces more effective quantitative approaches using tools like Monte Carlo simulations. The discussion emphasizes the importance of clear communication between IT security experts and decision-makers to better allocate resources based on the actual business impact of risks. Throughout the session, Mike provides actionable insights on leveraging quantitative methods to foster informed decision-making and ultimately enhance organizational resilience against threats.
Full transcript
[Music] hi everyone so uh we just had a little hiccup there but I'm introducing my friend Mike uh to present today uh Mike has over 20 years of experience talking on this specific area of uh organizational risk uh on this topic of a quantitative approach to measuring risk and prioritizing fixes um without any further Ado look Mike take the stage and introduce yourself thank you very much
nisel for introduction um is it possible that we change uh to to the screen the presentation yeah if this works better oh no it's the other presentation guess um which other presentation sorry which was oh sorry yeah this one yeah thank you so my animation work better um so yeah that's because I'm a risk manager I always have a plan B so I shared both the presentation
and the screen which uh uh leads to this confusion so sorry for that um yeah as I said I'm a risk expert and uh not an IT security expert but this talk is about it security and how to use risk management um talking about risk um about 20 years ago I founded uh a company a software company and back then we developed a software for newspaper production
uh the company name is uh Bob systems and uh it's still uh in business and we are still creating uh these uh software systems the special thing about newspaper is if there's a only a short downtime uh of a few minutes can lead to high cost high damage um because uh logistic chains are disrupted and uh products are not delivered and advertisers uh would demand funds or
free reprints and so on so every time the software is down uh high damage can happen and uh this early experience taught me the importance of thinking through event change and the costs of not having a contingency plan um this of course made me a better system designer and also a better system architect but it also trained me to see the chances in high Financial risks and
also communicate risk through uh yeah in terms of uh Financial losses so now with uh my other company which I founded in uh 2016 um I'm a consultant and uh I train leaders across Industries on how to leverage risk uh risk management for business agility okay so much for that uh now let's talk about why do I give this talk uh in the first place so as
a consultant I'm attending conferen attending conferences is part of my job and uh as a software developer interested in Risk Management I very frequently uh attend it security talks and uh I observed that one question uh rais is raised at the end of the talk when the audience learned about uh the multitude of uh vulnerabilities and threats and exploits and whatever uh can harm uh the smooth
uh operation uh of software and uh can harm uh the protection of data um one frequent question is uh how to decide what to fix first because it's so many things that can go wrong uh which one uh should we put our efforts on to to fix the typical answer from uh uh when when this question is raised uh involves uh terms like high critical medium or
low and uh these qualitative terms are based on uh the CVSs score um which you probably we know but I explain it little bit later but the action then when it comes to the actions what should we do with with critical uh and high should they should they be fixed immediately would be best of course but uh who should do that and uh the medium uh severity
should they be fixed as soon as possible and low Whenever there is time okay those of you who are in software development and know the pressure of software development creating new feature and so on know that if the advice is whenever there is time it means it will never happen uh the same is with as soon as possible immediately well maybe uh in the next few months
um so we have uh kind of uh ambiguous uh advice here to uh do some action so today I'll uh offer a solution to this problem and your feedback is highly appreciated uh I will share um my uh LinkedIn link through uh at the end of the slide deck and uh if you connect and uh have further questions I I think we have time for questions for
for five minutes at the end um I will happy to answer your questions so let's start with uh some matrics from a completely different field um I want to introduce you to the upar score the upar score is used in new newborn Health assessments before its inventions uh many babies died because there was no protocol uh which action to apply if um the baby is not in
a in a good um health uh situation and uh it sometimes was just decided uh the baby is not healthy enough to live so uh many babies died because of that uh after introducing the upar score uh which consists of uh five different attributes the activity the pulse the Grimace the appearance and respiration activity of the baby um the situation changed drastically because when we have the
situation the baby uh has respiration uh problems action immediate action is required to uh really uh mitigate the risk of dying here so lives are at stake uh when using the score and um there's a process a protocol depending on the score that is found uh that has to be applied and uh that is okay 7 to 10 no action is required the baby appears normal four
to six intervention until baby's UPA score is seven and 0 to3 um immediate uh attention is required to save the baby's life so and I think that's that's the thing uh that's the part where we have a difference to to the uh CVSs or I introduce on the next slide um here's an action so let's see how it is handled in security I already mentioned the CVSs
score uh it's very common it's open source uh it's well known among uh it Specialists and um yeah it's a little bit more complex than the APPA score so we have many more uh attributes including is there user interaction uh required to exploit a vulnerability uh do we need privileges uh what's what's the uh attack complexity can uh unexperienced experienced hackers exploit uh the vulnerability and so
on and so on so all these aspects lead to a score between 0.0 or 0.1 to uh 10.0 and is mapped on a qualitative scale uh like it is displayed on the right side so and there we found our low medium high critical uh severities for the um vulnerabilities but what should we do we still have the open question action is required uh what we are going
to do so can we fix all the vulnerabilities on this slide um you can see that the number of known vulnerabilities per year increased dramatically so we are now I think at around about uh 25,000 uh known vulnerabilities per year is it possible to fix them all it's a question here some companies may say yes we have that uh resources uh others um probably not but uh
anyway focusing only on critical and high vulnerability scores might be a small fraction of all of them let's say 0.5% but if you have 0.5% of 25,000 we end up with 125 vulnerabilities which are all critical and high and the problem here is is everything is critical nothing is so we can't prioritize uh 125 uh critical and high uh vulnerabilities very easily but we should because even
among them some are more important than others so how do we prioritize uh these things we have limited time that's reality we we may wish we had uh uh this would not be the case but we always have limited time and resources so how to use our time and resources best prioritizing uh is a good way um to do so and by the way we can't ignore
medium scores which may contain severe risks in the context of our project and uh this is very likely because the majority of all vulnerabilities have a medium score most of them so there might be some uh which we would miss if we just them priority prioritization um is not the only thing we need to care about because uh lists of prioritized uh vulnerabilities do not guarantee that
the uh that they are fixes that uh applied Risk Managers um sometimes report that they create all this information for the development department and pass it to the and nothing happens um they are just ignored uh which is the worst case but um the excuse always is we need to create new features because features bring the money they have a business value so in my talk title
I say okay what's the business value of uh fixing and um if we know that communication will be much easier between it Security Experts who understand CVSs scores and decision makers whoever that may be uh one person multiple persons uh simple simply speaking the simplest model is the boss you approach the boss with the list of vulnerabilities and say I need uh time from the developers to
fix all this and the boss says no we need to we need to create new features because we have contracts to fulfill and uh to earn money um here we have aetric knowledge between decision makers and it security expert and risk management can Bridge the knowledge gap between it Security Experts and decision makers so that's that's another application of risk here so where where does this risk
come from from uh we have the vulnerabilities you can also have other things like uh known exploits uh application security issues in applications and so on and so on everything in it security that uh will very likely cause a risk and um you can as an example uh we use the vulnerability and say okay a vulnerability is causing a risk if it does Nota cause a risk
it's probably not vulnerability at least I do not have to it and um the first thing I need to do is to describe what is the risk all about so it's just writing down uh uh what can happen and there's a nice template to do that uh which uh describes which aspects we should describe and the first one is the consequence what can happen if the uh
uh event uh maybe a hacker uh gets into the system and steals data what's the consequence maybe high costs High fees uh or uh if you have rent somewhere you need to pay the ransom or have much many costs uh uh have to pay much money for for recovering after the attack that's the consequence the asset is what is in danger our sens sensitive data an asset
can also be uh the um how how the company or the organization is seen by the user so if they trust them trust can also be an asset uh so write down what whatever is in danger here uh the source of the attack maybe an external hacker or internal users um has to be uh part of the risk description and the event that happens um when the
risk is uh applied so when the risk the risk is based on the event so something has to happen like the hacker gets into the system um to cause the damage so if you create these case description you already have a Bas a basis to communicate with other people because it's generally understandable you do not need to understand a vulnerability score everybody can understand uh a case
description so you can easy discuss the risk moreover risks can be broken down into smaller more understandable Parts risk are context dependent uh so not every vulnerability has uh the same risk in different projects or products um and Al severity might be different some might be critical in one product but not in another for example the impact of data loss in uh in a product handling sensitive
data is of greater and bigger than uh if you your data is just uh something that is not of value documenting risks with case makes them easier to understand and communicate but we will still need a value to compare with new features so that's not enough so remember risk ATS context we can see the context um in our own projects uh to prioritize which one to fix
and which one not applying risk management to it security is not a new idea there are some Frameworks around and some of these framework uh some of these Frameworks I think most of the Frameworks use risk matrices to assess probab probability impact uh you see on the x-axis uh qualitative uh terms like very unlikely to very likely to express the probability uh of the occurrence of the
event and on the Y AIS you see uh the same for impact from negligible to catastrophic so these qualitative terms and scales are ambiguous and nonlinear so they can lead to misunderstanding and uh the risk Matrix itself introduces another uh risk um that uh for example qualitative ratings lead to the assignment uh of uh higher uh severities to quantitatively smaller risks so if you you're interested in
the details and why this weakness occurs in Risk matrices and maybe also how to work around it you can refer to the article what's wrong with risk matrices from Tony Cox um if you Google it you probably find it and uh if not uh send me a message I sent you the link okay another weakness of Rick risk matrices is that uh when risk and impact is
negatively correlated Al so the uh probability and impact are negatively correlated uh then the decisions based on this information can be worse than useless worse than useless refers to you will better off if you flip a coin so it's worse than random and that's not what we want we want to make good decisions here uh when we prioritize uh fixes and uh deploy our resources to fix
um so for me the most important part not to matrices is that you need to discuss each and every risk separately and the person you are communicating with needs to understand risk and that is a real cumbersome task and many people reported that they dropped uh risk matrices uh in communication because they had no additional value uh it would be better to uh talk about opinions and
uh convince people to uh resources u in just by negotiating about it but what's the alternative to risk mattresses so let's imagine let's Dream a Little Bit let's imagine we have a graph showing the impact of all risk xaxis and the probability of losing money on the y-axis so here's a big difference we don't have a qualitative scale we have a linear and absolute scale with money
on the x-axis and uh probability in percent uh for the Y AIS so what we can do now here is we can go to the b or the decision maker and okay we have a probability of 20% uh losing € 150,000 or any other currency um in the next year are you okay with that so probably the boss would say okay I can live with that it's
only every 5 years € 150,000 Euro no extra required but um sometimes they say okay that's not acceptable we need to do something about that and that that's the point where the action comes in now the Security Experts propose a mitigation of this risk by the way um this green curve is called the loss exceedence curve it's uh very common in Financial Risk Management so now we
create a proposal for a mitigated risk curve so which um vulnerabilities have to be fixed is decided by the experts the decision makers just see the mitigated curves you they it is shown to the risk uh decision makers say do you accept that should we do that and uh they can decide to deploy uh the required um resources to mitigate the risk in that way another nice
thing you can add to the graph is a risk tolerance curve uh which you get by asking the decision makers about their uh expected expectations okay so that would be easy negotiation uh about all the different risk is required the people who decide about the resources decide just which level of security is acceptable for us use that one so to create the graph we need quantitative methods
and assess probability and impact maybe you asked yourself already so where do I get the data from that okay let's have a closer look to uh how we approach probability and imp imp quantitatively probability can be estimated the problem with estimating uh probability in percent is that uh many know the frequentist probability which expresses the relative frequency of the occurrence of a specific event for example if
you roll a fair die uh the probability is 16% uh that each of the sides are shown for estimating the probability of a specific event in the next year this definition is misleading because uh people respond we do not have enough data we can't estimate the frequency here so um that's where the basian definition of uh probability is uh coming into play which is defined as the
degree of belief and that's basically the same that what we do when we use qualitative SK scales we express our degree of of belief but with the basian probability in percent we do that more accurately and we have a numbers and numbers uh are a great thing you can calculate things with it for example create loss curves you may say this is not very accurate and you
are right um so it's not the end of the story you know we are not just estimating around and leave it like it like it is so that's just opinion what we do is we check our estimations regularly after one year we know which uh event happened and which not and then we can apply a score system again uh that's the Brier score Glen W Brier uh
proposed the score for um uh checking weather forecast the accuracy of weather forecasts but we can apply it to our risk forecast as well and uh when we do so P of T is the probability of our forecast and T is if the event occurred or not so one it happened and uh zero it did not happen so that's pretty simple we put it into the formula
and we get a score uh between zero and one where uh zero is very good and uh one is the opposite and uh if you're in the middle 0.5 or something you are in the mayb zone and uh yeah that's a little bit like uh when you ask something uh what's the probability that it will rain and say oh it's 50/50 uh uh then you probably take
your uh umbrella with you because uh you do not trust uh that uh you are the lucky person has has the 50% chance chance of okay that's how we we can do that we can create this Brier score uh continuously and see Tendencies so and uh how you get better that's a question uh which I do not answer in this talk but there are techniques uh to
improve forecasts very much by uh many different uh simple ideas okay let's have a look at impact we want to estimate the impact of a risk in money but that's not possible because uh we do not know an exact sum uh we can apply at uh just not possible but what we can do is we we can very good we can say okay when this happens it
will cost at least this amount of money so we have a lower bound and when we do it on the other side uh it will not cost more than another amount of money and then we have an upper Bond and if we are 90% confident that the real value is within this interval then we have a 90% confidence interval if we are not sure if you do
not trust our intuition about uh the 90% uh confidence here we can check ourselves by asking ourselves um would we bet uh, of our own money that uh the value the real value is within the uh confidence interval or would we um like to bet on a Fortune Wheel where we have a 90% chance to win if your gut is crying Take the Wheel Take the Wheel
then reconsider your estimations because you're probably wrong and you are not consistent with yourself um well that's a that's a mental trick you can apply so a little bit more about the impact know we have this interval now but we still don't know what will be the real value that's because it's a uncertain value an uncertain number uh we are dealing with here a random number and
random numbers or uncertain numbers can be easily uh graphically displayed as a histogram of uh probability distributions this one is the normal distribution and if you don't know the dis the real distribution of uh your um impact then you can just apply the normal distribution it's a good guess distribution uh that is for some uh risks much more realistic um when you have a very low costs
or low costs or low impact um most of the time but in rare cases uh when you don't have luck uh then it will become very very expensive so the impact is very very high uh then you can use this distribution that's the log normal and uh it's a so-called longtail distribution and may reflect the reality a little bit better so why do we need that of
course uh we want to create our loss exceedence curve so we need to find um a little bit more data for it so that's what these distributions are for now we have everything collected and we can write it down into a Excel sheet or whatever tool you want to uh use a database whatever and um some fields are the event on the on the left side the
probability of the event in the next year our estimations for lower bound and upper bound so this data we can use to apply a Monte Carlo simulation Monti Carlo simulation when you heard of it comes from physics uh was invented during the Manhattan Project uh because uh statistic uh St statistical uh methods were two cumbersome to apply to very very complex situations um in in this project
and uh for us it's just okay for the normal people uh statistical mathematics is cumbersome uh in all projects um that's why we just try and we can do that because we have comp computers and um computers are great calculators they are really great in calculating things and not only typewriters and we can use that uh without any special software uh we can use Excel directly and
dupit a notebook so sometimes I hear I need an expensive tool to do do mono simulation you you don't you don't need that you can do this with Excel or with pyth uh with no additional cost and you can run 1,000 uh 100,000 and more experiments per seconds that's great isn't it so let's put things together how do we calculate this stuff we have the confidence interval
we apply a distribution to the confidence interval and use for example python with the um formula np. random normal average Sigma and uh can draw a value for the impact um here and we have event will it happen in the next year here's the formula in Excel IF random uh read it yourself uh and uh it can be one or zero and this is Multiplied with the
impact if the event does not happen of course uh the impact is zero with that we generate the data for the next 10,000 years and to uh find the probability of a specific loss we just count the data rows and uh calculate the probability out of it that's a very thing okay now we have the curve now we have all the steps in place place um I
speed up a little bit um identify and document risk in this case estimate impact and probability provide information to estimate risk mitigation select fixes based on business value gather data about incidents and update estimations yeah that's it you can do this continuously and you are done you have a integrated risk management process uh that is also uh in uh based on uh EO standards okay but finally
I want to uh share a story uh from from Amsterdam and make a suggestion what you can do with this numeric data uh as well it's an example from Donella Meadows from uh uh uh thinking in systems and uh it's about uh a research uh when households in Dam during the energy crisis in the' 70s uh used uh one3 more energy than others but they were not
different and uh they asked the question why why are using why they are using more energy the different they found was that the meter for the electricity was mounted in the hallway uh in the uh households that did not use so much energy and in the other households the meter was in the cellar where nobody body could see it so my advice and learning from uh this
story is automate Monte Carlo simulations don't do them only uh manual manually from time to time automate them share security Matrix with others show them uh the progress uh of your efforts to fix things change Behavior Uh by using this feedback and then you maybe don't need to communicate so much about uh the requirement to build uh to to fix things key takeaways quantitative risk data Fosters
informed decisions use Monte Carlo simulations no expens expensive tools requires share Matrix to change Behavior use feedback to get better and a call to action start communicating the costs of not fixing security issues and the gains of fixing them thank you very much uh I think we are on time right Nigel sorry I UNM myself just there um yes we are absolutely with time um the one
thing I'll say is we don't have any Q&A coming in right now at this moment in time uh okay those who are in attendance if you would like like to ask a question and is anything you thought was interesting about this uh talk then feel free to ask your questions right now in the Q&A section in the pinea and um I'm sure Mike should be able to
answer those questions for you um unless there's Mike if there's anything else you would like to add in the meantime while we wait on that Q&A just a last note to recap um you can see by my uh LinkedIn QR code there uh would be happy if you connect with me I can answer questions on on the chat on on LinkedIn later if you like sometimes uh
questions are very personal stuff you don't want to share with others um but feel free to contact me uh I can also share the uh slides with you if you like uh I don't know if it is done in the conference itself because uh there's one more slide I have here that is uh the reading list uh where you can deeper in the different aspects of uh
risk management and uh cyber security is one book I want to mention here is how to measure anything in cyber security risk where the methods uh I explained here uh are partly originated and uh some other I I think very interesting uh sources of information how to apply mathematical and quantitative methods uh in it and it security now this is awesome and then there was also corra
me if I'm wrong what's wrong with risk matrices by La Cox I think that was also a a recommended read on your part yeah it should be in here no it's not it's okay you did it's also on the slide but if you contact me I can send you send you the link uh directly it's a it's a scientific paper uh where Tony Cox uh really uh
uh DET goes really detailed in uh the flaws of risk man uh risk matrices and the errors they introduce into risk management and uh yeah that's a little bit uh what I'm trying to do and people deal with risk and in any aspect of uh professional work project management uh uh development software development and so on to use quantitative risks uh methods because they are not so
complicated if you spend a little time to dig into it uh you get get a real big gain out of it uh in terms of uh communication so I never have problems in project uh if I ask for resources when I tell okay you have a 20% risk to lose €150,000 yeah so um if you have a risk matrix it's difficult for people to understand and uh
decisions are made that uh are probably not leading into the right direction and success is the result not of only one right or best or great decision uh success is the result of many qualitative good decisions so if you increase if you're able to increase the quality of your small decisions in everyday life spread it over a whole team then success becomes more likely and just one
question this actually from myself but you know that when we're talking qualitative is based purely on the data and as you say there's sometimes ambiguity about what we're actually looking on how accurate it is so it's best for us to do something based on quantitative um you mentioned those Monte Carlo experiments correct me if I'm saying them wrong and you mention hundreds of thousands of tests you
can perform in a short amount of time and this can be done for free using python or whatever but for those attending today's session who are looking at it and saying well I've never written a python script is there H is there any guidance on how they could start implementing these type of experiments in their organization with no heavy background in programming or is it just an
answer of no you you really need to get knowledge of these kind of languages and Frameworks in order to perform these experiments okay if you if you want to write Python scripts you need to learn to write python there's no uh way around that uh but some people are very familiar with Excel can can work great with Excel and Excel has uh a feature that's called the
data table uh and the data table uh together with a random function can be used to uh do to Carlo simulations and Exel without writing any macro I think that's that's a good way here um there are also many Tools around uh that enhance Excel um look for open source tools or some tools uh that are free of charge they are also around and they can do
the job um it's also much easier so doing these things requires a little bit of effort or finding someone who can write Python and write the script by the way such a Monte Carlo simulation is typically um about 20 to 100 lines of code uh so it's it's not really a big deal um I also have a uh python uh simulator where um Monte Carlo simulator where
you can just put in the Excel sheet um which is based on Python and uh streamlit it creates a small app that is running in a browser you can start it you can upload your excit sheet and you get the curve if you are interested contact me I I show again the oh wait the Q code contact me tell me I will send you uh the code
for that so so you have an example how to do it yeah no look all of this is fantastic I know we're just coming up to the last minute now um so I might leave it to our moderator Erica to leave the closing note but all I can say Mike thank you so much for the session uh it was genuinely informative very interesting and uh again to
anyone who's listening uh feel free to scan that QR code and contact mik directly I always think uh having these conversations is the best way to to learn more sure thank you nitel uh see you in vus see you in vus thanks
