Marcin Niemiec: Threat Modeling Using LLMs

This talk focuses on the topic of threat modeling and how it can be enhanced through the use of large language models (LLMs). The speaker explains that threat modeling is a technique used to identify security issues before implementation occurs, advocating for its application early in the development process rather than relying solely on post-implementation penetration testing. He discusses various concepts of threat modeling, including components like authorization boundaries and how to categorize threats using the STRIDE framework. The speaker also provides examples of how LLMs, specifically models like ChatGPT, can assist in generating threat models by processing architectural descriptions and identifying potential threats. Finally, the talk emphasizes the importance of clear prompt design when using LLMs to ensure useful and relevant results in threat modeling efforts.

[Music] ladies and Gentlemen please welcome our next speaker marcini Meek presenting the topic threat modeling using llms hey hello can you hear me yeah great um happy to have you all here it's maybe not the full room as it was before but I see that you are passionate about security and AI so I hope you will uh walk out from the talk with something uh and yeah

let's start so my name is Marchin as you already know I have a blog post and I'm sometimes put something on the X so if you want to check go ahead I'm from the form free we are real time payment processing platform but let's not talk about that right now okay thread modeling so how many many of you know that how many of you did that what

is your experience with Fred modeling what can I see there one person two three four okay good that was like it was really hard to put the the hand up I I in this I think development Community um okay so what I want to give you today is kind of a bit provocative if you struggle with the trade modeling yeah maybe you are as a developer Maybe

you don't like this practice or maybe you are like security person and you know you also have a lot of them to do and you you cannot scale it or something like that so AI makes thread modeling easy for you yeah let's see let's find out during that presentation okay let's start with the Fred modeling oneon-one because the most of you don't know this technique what it

is about so yeah um so the threed modeling is a technique to find security problems before the implementation happen yeah so you know when you have this pen testing you get back the list of the vulnerabilities that you need to fix and it's already late and you know you don't want to fix them you don't understand them so you can do something else you can introduce security

at the beginning of the process not at the end of it uh where you have implementation but only when you have ideas Concepts design yeah on that picture you can do the thread modeling so someone call it evil brainstorming I I like that term very much it's like when you think about tread modeling sometimes you know you you think it is the highest yeah we are planning

the highest on our application we are not really doing anything anything specific and yeah if you want to refer to the to the pentest thing it's like pentesting on the paper because we are not pent testing application that is running but rather we are just taking that that design design okay and how we can do it so I I took this picture from The Martin fer blog

just because I knew that this is development community and maybe you know Martin fer so I would say if you if you read something about it hear something about him maybe you will you will want to to check that okay where is my mouse my mouse is here sadly okay okay that won't be easy uh uh is there here laser or something or not no okay no

worries so I will be looking at it and watching okay so this is very simple tread modeling so no not to install you okay this is normal tread modeling yeah okay so here oh my mouse is crazy so here's the user yeah and those blocks they are just some components of the system for example like react up in the front end then this is dotted line that's

quite interesting one it's called authorization boundary or the trust boundary then we have some backend components like the server nodejs where is my friend from the node.js is he here on the S I don't see him oh no he left uh payment service also nodejs and then the database like that and now we start thinking okay let's first put the author authorization bandar so we know that

the react app will uh work in the browser so basically anyone can do anything in the browser so we we say that this is unrusted we say okay those components are a bit more trusted together so they are they are not having any boundary between them because they're on the same level of trust the same as the database so this right side of the picture is the

same trust level and the left side of the picture is the same trust level but those trust levels are the different that's why those boundary and then we can f okay what bad can happen here so first to ask to answer that question oh he's my man from the j nodejs i I keep it for you so what are the things that we need to protect here's

the money because we have the payment service so we are processing uh payment so the money money is here this is how not only where we are processing someone else money but also where we are making money and here are all Pi data maybe some credit cards information and so on and this is where how we going into that yeah is the same like with home if

we would do trade modeling on the home we would say okay what is important in my home yeah maybe I just show the flat and I have only one room and I have nothing in that room because I'm from the Z generation and then I'm just just not about that or maybe I'm just you know like senior uh someone and I'm having like the lot of art

or the crowd or something there so I need to protect that so I need to know what I want to protect and then I'm asking myself okay what can go wrong yeah what can go wrong here so uh on the Martin Fuller Brock they said unencrypted data hm that could be a threat yeah reasonable that could be a threat that could be a threat like the real

attacker could get to our our server room bring the disc and grab the dis and and go home yeah but that could be also a thread like the compliance thread because we told our customers that we are storing encrypted data and now we are not storing encrypted data there could be a Doos yeah that's that's that's one of the thing let's say we have a a competitor

that is also selling something and they're selling the same thing and they go to this hacking their web forum and they they paid for the Dos attack on our service especially in the Christmas where we were wanted to uh make a lot of money yeah sadly oh no that's I hope that this is not insulting you when I saw this is thread modeling version hard so this

is something that you can get as the reference architecture uh of the thread modeler for the Azu and this is like uh looking really crazy and to be honest if you ask never tried thread model application is the very shitty application uh created by the Microsoft some time ago but it still stands and some company still using them okay so yeah I love thread modeling and watching

P pain dry is also fun for me so okay so we now know what is this thread modeling okay what are the Lang large language models yeah so how many of you used L large language models oh a lot yeah a lot more than the yeah okay so the llms yeah sometimes they they feel a bit like the magic but in the end they are uh parameters

yeah and when you train those parameters you just do optimalization of the thing and uh why it's so crazy from last two years because it turn out when we are training them uh in unsupervised way on the next token so we are asking them to generate next token and the same doing back propagation so training for exactly this token it turn out that those llms can encode

um meaning of the words meaning of the tokens yeah so for example what is the the concrete example the researchers find out that the Germany Hitler Italy and musolini they are connected in the meaningful way in the llms so as the way in the wakes yeah so is the LM by the training of the next World find out that we are describing in the internet in the

books Hitler molini German and Italy in the same way somehow yeah so they they're related to each other that's that's quite a fun so and that's why the next picture exactly chat GPT olm uh and this is how looks like our our knowledge and knowhow um yeah that that's only it and that's uh and that's and that's it yeah let's let's let's move on the house is

burning let's move let's run okay so the thread modeling so if you already I bet yeah I bought you and when you get back to your companies you will start doing thread moding so I will tell you what ingredients you need to do it manually so description of the thing we need to know what we are doing maybe we can have architecture of the project we need

developers because developers know how to create uh the thing they know what they create or what would they will would like to create we need Security Experts so yeah you need me to do thread yeah but how what would it be if we would like to do this thread modeling with the LM what then so we still need description of the thing we still still need developers

because they need to run something and write something they don't you don't need me anymore so maybe sad for me good for you but instead you need the prompt so the question and the llm to host it okay and S general idea is very simple we take our documents you put it to the prompts and then we get our shiny okay and my friend Daniel mler so

you can uh follow him because he is putting all some interesting things about the uh the thread modeling uh so he said that the AI isn't competing with the Einstein to story and house AI is competing with the task not being done at all it's being done poorly inconsistently slowly costing too much so he thinks everything but when you remember those few people that RIS at their

hands you know I think that we are sticking with the task not being done at all so basically we are not doing thread modeling so that could be a good good idea for us to use llm for that to democratize usage of those techniques and security okay short demo because uh yeah so let me run that um that's from 4 GTP GPT 4 so I just wanted

to try I wanted to know if the chat GPT is really able to do Trad modeling or not because maybe it this is not able to do that let's see is giving us some answer of course yeah you cannot read that but it's generating generating and it started gting the threat table threat table so as you see it is putting something there yeah we'll go to the

to details okay so what are the results the results let's see those results so uh we have like thread ID component name thread name stride category and the explanation so I asked uh the the CH GPT to create a thread model based of the architecture that you don't know what it is but I will show you in the moment but that I just wanted to to give

you the the thing that CH GPT was able to to do that yeah we'll go go back to the details here some more but we'll we'll just keep that so first of all first of all CH GPT knows thread modeling okay ch ch GPT can correctly find the components that's good that was was not not easy thing can write meaningful threads but can we get even better

than that okay so here is a St but I know that you started reading so ah you can't okay so let's let's think only about the first one because the stride is the technique but if you I don't want to bring too much on you so if you uh know if you know thread modeling The Stride is technique of doing thread modeling is the is is the

ponic that each letter represents different category of the threads uh we need that for certain reason I will tell you later so but I just want you to grasp it like the first S letter is from the spoofing and what is important for us is that the when we talk about the spooling we have like things like authentication yeah so like MFA or of open ID JWT

and so on so we want to check if the party is is authentic here okay stri per component no I will I will skip that for you okay so uh what ingredients do we have okay as as before we will starting with the descript description of the thing yeah and then then we'll put other ingredients description of the thing do you know what kind of diagram is

that who knows what kind of diagram is that anything and anyone was creating design diagrams architecture diagrams in his life and the name container tell something here those are C4 type diagrams this is the container type diagram from the C4 methodology uh and let's let's check that where is my mouse okay oh no no no go back okay uh okay so the first thing at the left

you see that so this is the m plan the me planner is said that this is external uh external what external system external system the me planner is like the external system used by di dietitians so dietitians are the people that are preparing diet if you need diet if you want to get uh feed or maybe you have some health problems you go to them and then

they are preparing you the the diet but they're not doing that on the paper andal but they are putting it on the program that is calculating all the stuff and my idea is that we will uh have a company that will have a product product is named AI nutrition Pro this is the S product that will work with those meal planner applications and it will provide some

AI capabilities yeah so that the diet is a bit um more AI something yeah to just get more clients for it so what we see here is that from this me uh planner application we are we are calling the the API Gateway and then from this API Gateway we are calling API application and the API application is calling API database yeah so this is the main flow

and also API application API application is calling chpt and also there is a second flow with the control plane because you we need to somehow administrate this whole thing uh so in nutshell this is it so this is how it's working so it's very simple uh few components I added some uh description here like the goong a WS elastic container services what it is doing and so

on just to have more information that the that the AI will because AI will be consuming that so it will have more to consume and what is interesting I coded it in the mermind so I code put that in the mermind uh as you see C4 container and those are all those those uh those items and the llm chat GPT and the GPT from the API knows

uh very well the marind and those diagrams so it can understand it and I added something about the security yeah because when we are preparing the design document if you feel that you earn the budget of the uh the architect new compy you know that you need to pay attention to the nonfunctional requirements so this is one of it and I just want for you to remember

this first one yeah so what it is is like authentication with the meal planner applications each has individual API key so on the one hand I think I could have a better sentence key because this sentence feels like English from the primary school but still it's uh and remember that so the meal planner applications will be authentication using API key the simple possible oh one and so

maybe maybe more posses like there will be no authentication at all but yeah we we are doing this is talk about security so pay attention to that okay that's and uh short side uh side track so no more excuses for no writing documentation I know that you may have this book on your book bookshelf but you should say instead you should have those two books yeah so

why the first book if you if you think oh that's boring then you have this first book yeah so there is like 100 Pages why to write documentation why that's important and this is really good and then you have the second part which is saying how to do that with those C4 diagrams So based on your rise hands I think that you should definitely R read one

of those two books oh better both okay uh so we have description of the thing you will read those two books we have developer at least one we need this one developer to write that and now let's go to the prompt so let's go to the meet that you were waiting for uh oh you are vegan then then not me uh so uh yeah this is what

we are starting we will be Inquisitor okay so there are this key components of the prompt so this is also if you are interested in the prompting I think you should uh pay attention here to those slides if you were on the previous uh presentation the guy was talking about warming up the the chat GPT or warming up llm so this is the same idea yeah so

this is this is kind of the same idea here I'm setting up Persona I don't know if that's effective or not but I'm still using it uh I'm doing stepbystep thinking that's that's that's doing the stuff this is enforcing this is somehow they find out that this is enforcing logical reasoning on the llm so step asking for the stepbystep thinking uh then I using this thread modeling

uh with stride per component so very security term very very very interesting thing but the what is the most what is the important part that this is well defined technique so this technique is well defined is not perfect but it is at least well defined and it can tackle major security problems that's why I'm using it okay next components thread explanation this is this warming up so

I'm asking llm to explain me why this thread is important for the component in context of architecture description if we will be doing tread modeling session if you will be doing third modeling session with me and you will ask me why I think this threat is important for the component I would do the the the bad look on you and saying you are talking to me that

I'm I don't know why this component why this threat is important come on but we need to ask the LNM and also the yeah my let's say my children of my beloved Children Of My Mind explain if this threat is already mitigated in the architecture description or not so those two things are really warming up uh and is really squeezing and pushing llm into the right direction

yeah so we are not only asking okay give me the threat no we are trying to push it for this explanation if you want to know why it's important you can ask me afterwards because we will not have that much time for it oh yeah and this is the zero short prompt if you are thinking about it so I'm not providing any good example of the work

to the AI uh first thing why uh the when I started that the the context length was very limited so I couldn't provide any good evidence for the ey because I would just have no space for my prompt and my data and later on I I just realized yeah that's good enough without without even an any additional uh examples okay so uh what you can do here

what you can use so we can use Char PT as I show you but that's kind of not consist and it's easy for the pro for the experimenting but the most important things is to put your thread models where is your documentation yeah this is what you would like to do you don't want to or maybe if you are in the big Enterprise you have already like

10 or 20 systems where you do different stuff so maybe one more will not uh kill you but in case of the startups we want to have everything in the one place so if we are having the code in the GitHub if we are having documentation in the GitHub we want to have our trade models in the GitHub and not somewhere else in Confluence um and yeah

so days before Ai and day days after right so a lot of debugging very least of coding uh if you are suffer from that because maybe you already started developing something you can chat with me afterwards I can uh tell you about uh the lsmith and the the L chain Library uh the lsmith is the tool I can I can tell you a bit how it help

Okay so I wrote this AI trade modeling action that I published for 15 minutes okay I will I will be wrapping up uh so I wrote this action so you can go to the GitHub so the whole slides and everything will be shared so don't don't worry about it you will get an email from the organizators with the link to the Dropbox where all the slides and

everything so we don't need to to do anything about that this is experimental project uh what you can you can use it on your own GitHub uh GitHub repository you can forit unit prompt for your need it is free license you don't need to do anything and uh you can also use it with the python strips that are underneath uh yeah it looks like that that it

takes the input documents then it is running and it can push to the branch it can create the pool request and most interesting for me it can comment on the issue even so if you are uh having your issues about let's say something then it can comment with the thread model on that issue the features there is high level security and privacy requirements so this is the

prompt basically This Thread modeling of the architecture security acceptance criteria for the user story this is something I'm let's say maybe looking the most as the most inter interesting for the future and review of architecture Des description but let's focus on that thread modeling so how to how to use it yeah very quickly just to to to enjoy this little run here with the code so we

are checking up the oh no what I did uh we are checking out the repo then we're configuring it as you see we can because we are using API we can pick up the temperature if you don't know what is the temperature in the LMS this is kind of the thing that if higher temperature makes the output more random and the lower uh more uh less random

yeah and you can pick up different models from the open Ai and the open router and then you can commit it to the some some Branch okay and finally the master P yeah the example that you all waited so if that's really satisfying you yeah so at the end I will show that and I will ask you to rise hand who is satisfied with the output and

um who is not then then no no matter okay so first of all we see here that there are some data flow data flow one me planner application to API Gateway so I ask it to identify the data flows yeah because there are different data flows so I ask ask for it and it did it correctly yeah it can count numbers component name that's correct API Gateway

thread name spoofing attack by using stolen or guested API Keys you remember that I told you about those API Keys yeah so you see right now that it already it read that that we are using those API keys and it is saying that someone can stall it or guest okay that seems reasonable strike category spoofing yeah we need that because this is this technique that we explanation

an attacker could use stolen or guet API key to impersonate a legit legitimate M planner application gaining an authorized access to the API Gateway I think that's pretty cool explanation yeah I would be proud if I would write something like that myself on the thread uh how the threat is already mitigated in the architecture so is there let's check if if there is a logic reasoning here

partially mitigated by individual API keys for authentication as stated in architecture description wow yeah that's good mitigations Implement multiactor authentication uh regular rotate API keys and monitor for unusual activity quite good and then the risk severity High okay who is satisfied with this output okay I bought few people then my work is done here I was you know with this kind of technical presentations you only want

to have like this one person that you that is really paying attention I I saw that there are Le few so so I I yeah I really enjoy that uh if you are not satisfied because you think that that's you can come to me after the meeting and you can tell me why and maybe I will do something better I want to show you one one particular

thing maybe focus a bit on this risk severity you know so the the risk severity in the high level it says how urgent the problem is how urgent we need to fix that problem and um it only give us one word hi why we don't know why because we didn't ask for the explanation we didn't ask for the reasons and if we would leave if we would

not add those additional columns with the explanation and how thread is mitigated and the mitigations uh it will do the same thing to the thread name so it will give as a thread but without any explanation and that would make it uh worst the performance the overall performance of the so the the quality of the threads will be worse uh because we didn't ask for the explanation

so it just it would just made up it yeah so it would give us anything and the same happened here so we could try to uh make this more let's say with the Lo logic reasoning and we can push LM more asking why this is high yeah so how do you think about the probability of the exposure how do you think about severity of the exposure here

so that you are thinking that this is high yeah so remember about that that pushing L for for logical reasoning okay so now we are sitting you are sitting here and you are thinking if I will if I will lose my job yeah and then then then what yeah so I think that if you can grade those results yeah there are the developer um a average developer

no one is average developer everyone are like exceptional I know but there are some average developer uh there is like security Champion so this is like a person that was trained in the security but still doing the development and there is like Security Experts so like like me like yeah you know kind of stuff um and then there is this llm that I presented you so I

think that this is on the level of the security Champion yeah so if it's not like uh bad or anything it's not like exceptionally good maybe it's not like for the banking or the military but it is still better as you as you have seen that you are not doing thread modeling which is very simple technique um and the llm will democratize this technique and other techniques

other things yeah so you will not need to wait for the security expert to do something uh but you could run it by yourself on maybe on your feature on the whole architecture on anything yeah so this is to to remember that this is a tool that you can that you can use okay and the summary just to wrap up so elements can perform thread modeling and

the logic reasoning uh good description is the key of the useful results yeah the same as uh um you know being a good husband is is uh is good and kind of stuff uh gp4 and cloudy cloudy 3 perform well in the thread modeling and GitHub action give us a easy way to understand existing documents with AI feature flows okay and if you are not um if

you want to get even more knowledge I just leave those links of course you cannot see those links but when you will get the Dropbox version you will be able to click them and check my experiments there are also other people working in the same space like Matthew Adam uh he's also doing um uh kind of things with the stride and GPT so you can look on

your on his work clean gber uh created like um summary of the current state of cyber security and Ai and there some few of my blog post uh the one about reviewing your architecture using LMS that could be interesting for you as the developers because maybe you would like to you know this is what even I did yeah I wrote the architecture and I asked myself is

it good is it bad five minutes okay and but then I ask llm yeah what do you think about this architecture is it something missing could I add something there or that's that's the good way of using it on your design documents by the way uh if they are not leaking of course of course they are leaking uh okay so thank you very much for your attention

I hope I I bought at least few people to try it and yeah the questions okay to do the Trad moding with llm it requires to prompt business sensitive data basically leading info to the third party are you sure LM can change Security Experts so right now if you will of course if you will take any business data any business sensitive data and you will send it

to the open AI cloudy J any anywhere you are at the risk yeah of course because there are storing it processing it and we don't really know how and why they are promising something that they will be not uh learning with that but that may be true that may be not true this is like we are like uh and so the ear adopters will go there yeah

your company it's not like you are making that decision your company is making that decision is it like your company need to take two things if I docu if our documents will be leaked is it worse than not using a AI yeah so they are they need to have those those things you you shouldn't take that uh decision your company should make that okay uh how could

you compare Trad modeling withs to pay tools like Trad um thread modeler yeah so basically I was uh typically not using paid tools to do trade modeling because I think that the trade modeling is uh the team team or developers time with the security yeah so that we can have some kind of the relationship we can talk with each other we can know what the development I'm

doing and we can help that we can accelerate that uh with that kind of discussion and I'm lucky lucky person because I'm working in the financial in company and we are having like the big team of the security so we don't need to think okay we don't have enough talents we don't have enough time because we have that and that's that's why we are lucky but for

many companies there is like one security person for the 100 developers yeah so that's that's not that that that's hard and then you need to do something I didn't personally use the thread modeler if that's pay tool or maybe do you uh do you think about the Microsoft modeler or just trade modeler some tool who wrote that okay no one okay so the Microsoft thread modeler if

you rem that to will give you exactly the same result as the llm exactly the same almost exactly the same why because it is based on the rules but what is the difference the difference is that uh it will give you result that is more high level it will give you result in the way that we saw from the chat GPT at the beginning without any any

logical reasoning any additional context it will but it will give you the same yeah that's that that's kind of F that kind of uh stuff but with the llm you don't need to uh use different tool yeah you don't need to put your diagram in the different tool do all those stuff you can use like the GitHub uh nevermind or even you can use some um image

to to marind for example stuff like that LM that will transfer your image from your know uh Dr diagram to the to the m mind uh so yeah so we are use you with LM you can use thread modeling in the place where you are doing your documentation so you don't need to use any other tool okay any from the audience okay so thank you very much

[Applause]