About this talk
This talk addresses the security challenges associated with AI-enabled systems, particularly in the context of critical infrastructure such as power systems. The speaker, an AI consultant and cybersecurity lecturer, emphasizes the dual role of AI as both a potential attacker and defender in cybersecurity. He discusses the growing demand for professionals skilled in AI and cybersecurity, highlighting the emerging trends like AI-driven security tools and automation for incident response. The presentation outlines various methods for utilizing AI in threat detection and response, as well as strategies for securing applications against adversarial attacks. The speaker also warns about the risks of misinformation in AI and the importance of continuous model retraining for improved security.
Full transcript
Hi, my name is Sugar and I'm going to present you about securing K power system threat challenges and best practices. Very first about me, I'm an AI consultant with over 30 years of experience with a lot of different projects, web mobile, blockchain. I've worked a little bit with robotics as well, Sierra MP systems and lat machine learning and cyber security. also co-founder of digitalixing agency focused on
net and cloud projects lecturer at the cyber security department of technical university sophia PhD candidate there focused on application of AI and cyber security for the defense of critical infrastructure those are things like nuclear power plants and so on and also with a team of authors we call out to ring a couple of books focused on cloud security and here is my contacts uh so what is
a presentation about It's about cyber security, security risk of air powered systems, adversarial AI or how AI could be used for attacks, securing enhanced applications and future trends in AI. So let's start. Why should you care about AI at all? Here are some statistical information. There is a large growth of jobs that require AI skills like AI architects, data science specialists, machine learning specialists and more. and
also a huge uh rise of uh jobs that requires cyber security skills. Also AIdriven cyber attacks would rise you know that defects around increasing also some other false information and so on and AI could be used both as an attacker and a defender. So not only can AI use be be used to attack but also to defend our enterprise environments and also big tech companies are employing
a lot of AI experience professionals and PhDs and for example Cisco can send a lot of uh its consultants to Ukraine to defend uh its government against uh hybrid uh cyber attacks. So uh this kind of specialists are in high demand today. Also AIdriven uh tools would be a standard by 2030. By AIdriven security tools, I mean tools for example that could perform some kind of automation
for example automatically create incidents in service now or tasks in Jira. And of course more complex tools that uh perhaps scan and analyze your environment. Uh give you some recommendations on how to improve it or maybe even do the improvements themselves. and uh other statistical information there reduces CCS response time by a lot due to this automation and some researchers say that uh there is an improvement
in the operations of u business efficiency uh that's based on research that say that uh uh AI improves developer performance by up to 35% uh which means that we still cannot replace developers but u we could in a team of several developers we could perhaps lay off one or two of them. Just like with SRP systems, you could lay off sales professionals and so on. And a
lot of organizations believe that they enhances security, something that is not on this slide. Also, a lot of organizations believe that when using AI tools, they actually increase their security risks. So that's why this presentation is relevant. And a lot of AI governance walls have come into place like a act this fabric and others. Uh so uh let's talk about a cyber security. How can we use
it? Uh we can use it for anomaly detection, API traffic, behavior analysis, schedule authentication, automated thread detection response, rate limiting and API uh abuse detection, data protection, API encryption, monitoring. Let's go through them quickly. Uh so let's say we have some sources of information like application works, network walks and so on. We use that data to train our machine learning model. Uh and we train it so
that it can detect anomalies in the application works in the networks and once it detects an anomaly it could create an alert maybe incident service generate some reports for PowerBI do so or sorts of things that we want it to do. Uh what is uh behavior abnormality? Let's say we have a senior uh software developer that uh does everything that a software developer should do. I mean
he joins scrum meetings, perform code reviews, things like that. But what is abnormal behavior? That's when in 3 in the morning he starts to download like 50 GB of data from sales marketing folders and so on. That's usually abnormal behavior. And a lot of companies actually record teams calls uh and other meetings in order to then analyze these meetings with artificial intelligence to understand how the employees
feel whether they are putting something against the company or so or things like that. And a lot of HR teams actually employ AI tools to analyze uh interview recordings in order to understand whether the candidate is a good fit uh his personality type and so on. So basically it was used throughout all the departments of a company and they're becoming a norm a little bit about uh
threat detection and response automated. So basically again we have an enterprise environment again we scan it analyze it find vulnerabilities and threats but not only that we can also automatically respond to threats. Basically let's say we have a container instance that has been uh damaged in some way. We could basically uh remove that container instance and raise a new one or perhaps we could uh box some
APIs that from which we receive some harmful network traffic or do some kind other kind of action that actually improves the security of our environment. There are lots of options and again we could uh walk uh incidents in service now generate reports and so on and uh of course raised limiting and API abuse detection basically uh we could limit the amount of request to certain APIs based
on certain policies and rules and about data protection that's a topic of itself but what we should do is basically encrypt the data at rest in transit access uh control the access to the data and uh make sure that everybody access the data with the least privilege. uh nobody should have administrative uh rights if uh he doesn't have to and uh things like that and maybe you
know but uh actually Snowden managed to steal so much data exactly because he had a lot of privileges that he shouldn't have also use data centers cloud storage rotate your keys periodically secure data disposal following policies like GDPR and so on and secure monitoring can response language security with penetration testing So a little bit about adversary AI and model exploitation. How can we use AI for attacks?
First of all, what is AI? That's not something that thinks. Those are machine learning algorithms. There are a lot of them here uh grouped in different types like for regression, classification, clustering. They could be combined to create interesting solutions. And I suggest to anybody who wants to deal with artificial intelligence. Don't wor just the language models. Uh go to data science, go to machine learning and so
on. so that you can actually get into the topic. So uh let's go to a so-called convolutional error network that's a machine algorithm that uses depression convolution. I don't have the time to go into details but basically that's the neo network with out of layers and each layer extracts more and more specific features from the previous layers. So let's say the first layer extracts let's say the
contours or the form of the image. Uh then each subsequent layer extract a group of pixels perform some operations on them until at the end we could um perform some kind of prediction on the end date. Basically we could predict whether that's an image and whether that image is on the number seven. So how can we use that in practice? Let's say we have a uh deep
learning botnet based detector that is a narrow network that uh analyzes network traffic and tries to make a prediction whether it's her or not. Uh the problem is that u machine models could be easily fooled. For example, John McCaffy back in the day managed to uh feso car model by putting black tape on the speed limit sign of 35. And the car cameras uh figured that this
is not a speed limit of 35 but a speed limit of 85 which usually every anyone who is has good eyesight, is not drinking anything could easily recognize that it's not the number eight. But the machine learning algorithm didn't do that. that could lead to a speed ticket and in worst case fatalities. So a question for all of you who is legally responsible for AI induced fatalities.
How many of you think that only business owners uh owners of the software responsible? Please raise your hand. Uh how many people think that company owners responsible for AI? Nobody. Okay, that's good because that's true. Everybody is responsible for AI. That's developers, testers, uh those who train the models and everything. And of course, the first party that is viable here are the business owners, but everybody along
the chain is also responsible. And here is a gun-based attack engine. Uh basically we could use a generative adversarial network which is a combination of two narrow networks. I'll try to explain it as simple as possible. Basically, the first network generates artificial examples or in our case artificial network traffic that tries to fool our uh botnet uh network defender and we have a discriminator that tries to
uh figure out whether that traffic is artificially generated. And uh those examples who actually uh get passed by the discriminator could be sent uh to uh botnet detector network that we test and we can try to see whether we could uh fool our defender or not and uh that's a way to generate a lot of attacks very fast and uh eventually be able to fool a lot
of different defenses that use AI. So how do we actually prevent that? Well, basically we introduce the same gun algorithm into our defense. Basically all the data that is uh uh wrongly labeled as safe traffic should be passed to the training data of the defender model of the deep learning based bot detector and uh that uh network should be retrained along with all the newly acquired data
and then we update the weights of the victim model and perhaps next on the next iteration our uh deep learning our network would correctly qualify that uh we have uh attacker network traffic and as many of you have probably guessed uh that is not a one-time process. Basically every time we have new data we have zero exploits and so on we have to retrain our models again
and again. So that's a process uh which we have to do over and over and over to improve our defenses. So what are the security risk of air power systems? Uh most of you probably have heard about OASP top 10 for large language models. They are listed here along with where they apply. Uh we can have prompt injection where users um enter prompts that could uh expose
sensitive information or some uh information about the AM models and so on. We can have supply chain vulnerabilities where uh some kind of updates uh to some third party services used to exploit our systems. uh the data in the models could be poisoned and so on but there is something that is heavily underestimated that's misinformation because uh language models could hallucinate so we should never overrust a
application that uses AI uh we should uh basically uh I mean there are design patterns that could um that we have a lot of different English models for example the second and third one model verifies everything that's generated by the first large change which model. So we can have different design patterns where our language models check each other but we should always monitor and everything have humans
in the pool and so on and of course there are other frameworks like bit framework that also lists a lot of attack vectors and mitigation strategies and so on. So securing and hest applications what is a typical and hest application let's say we have some kind of front end net application mobile whatever I won't go into the details how it's implemented cqas or things like that but
it usually communicates with things like cognitive services our own custom machine learning models third party APIs maybe some kind of automations which models and so on for example let's say we have an insurance company business and the user takes pictures of his damaged car also takes pictures of some insurance documents and to write some chat message to the uh software basically maybe some angry message or maybe
some fear of insensitive message or whatever. So that application basically uh takes these pictures, analyze them, analyzes what estimates the damage to the car, then uh analyze the text from the assurance documents. It can pass all that data. It can perform also semantic analysis on the text and analyze how the user is feeling. It can pass all that data to the large language model which can perform
things like retrieval augmented generation to fetch some additional information and then it can in the unified resource aggregator. It could aggregate all that data and pass it back to the user and it could be a lot more complex. Uh we can have a lot of multiple model agents functioning in such an architecture. So how do we defend it? We can have secure design patterns. I won't go
into the so-called agent design patterns. I'll go into some more basic things. But uh uh anyone who is developer probably knows the classic API gateway pattern or the so-called facet pattern where we have front end at the front and we have an API gateway that serves as a facet through which all the requests are processed. uh there is a lot of logging, monitoring, authentication, load balancing and
things like that done at that level and all the requests are then passed to the back end services. Well, we should have something similar when we have generative AI uh solutions. Basically, all the requests that are done to our AI services should again uh be rate limited. they should be stored, monitored and locked because we could be under some kind of prompt injection attack or something else
and we should know that and uh basically everything should then be reported to the uh proper teams and a little bit about zero test architecture that's very simplified diagram uh we should basically not only care about our applications but the entire infrastructure we should strongly authenticate all the users and uh third party services that access our We should have device complies for our end points. We should
comply with the policies that are specific for the organization like GDPR N and others. We should uh use threat protection and risk assessment using tools like Asia defender for cloud to Asia Sentinel that could be extended with custom machine learning solutions. We should also perform traffic filtering and segmentation. All the data should be classified, tabled, encrypted, the things that I showed in the start. And uh that's
it about security very fast. So a little bit about future tense in AI cyber security. I've already talked about the power thread detection response. I've talked about the augmented cyber threats. Something I haven't talked about is quantum resistance securities. Some of you may know quantum computing is becoming a norm. And a lot of cloud providers are starting to provide services that involve cloud computing like Amazon uh
and uh uh Europe the European Union is basically developing more complex encryption algorithms that quantum resistant because with quantum computing you can basically break the existing encryption algorithms quite fast. Also there is the so-called cyber security as a service paradigm where just like infrastructure as a service and some of the other uh as a service paradigm. So basically we could have a single solution that scan our
environment gives us recommendations and maybe automatically uh do modifications to our environment to improve the security and big tech companies are currently taking that par to heart and they're trying to develop such solutions. I mentioned Sentinel before. So things like that and of course air powered compliance where such tools are basically uh helping us uh automatically be compliant with uh policies like GDPR, NIST and others by
against scanning our environment and giving us recommendations how to be more compliant. So a question for you, how many of you have seen these movies? I'm certain that most of you have seen the right one, but how many of you have seen the left one? Let's see. Okay, great. Because the left one, I won't spoil it. It's about free pilots that get a third partner that is
a a powered fighter jet with nuclear war that tries to destroy human humanity, but the free power stopped. Well, the thing is that the first uh the left movie is already real. We already have AI powered fighter jets using deep link network and the only real and the only reason that scenario isn't happening is because uh the developers haven't let the fighter jets take anonymous decisions. All
these uh uh decisions are taken by the pilot them himself. So the question is when would the second movie become real? And another question would you trust an AI pilot? How many of you would trust an AI pilot? Well, I'm glad that not so many people because uh uh you should never trust AI with anything that could lead to fatalities or loss of human life. And how
many of you have seen these movies? Because uh let's see how many of you Okay, a lot of people because in the next 5 to 10 years these movies would also become real. We already have uh uh human looking uh robots that are empowered by different machine learning algorithms. There are startups developing which model enable robots and so on. And also we all we already have some
form of predictive policing. Let's say in China they use social scoring and other data to uh predict uh how people behave. And for example in South Korea they use cameras to analyze the uh body language of people to analyze their behavior and could predict potential robberies with more than 40% accuracy. So predictive policing and u uh large language model enabled robots would soon become unable and some
useful resources uh some YouTube channels that I follow like OAS foundation Microsoft security as academy IBM technology Cisco and others free books that I highly recommend uh cy book or cyber security book of knowledge it's a very large book that is regularly updated but it uh has a lot of uh things about cyber security everything new is added there. I recommend it to all my students another
huge book machine learning deep learning and AI for cyber security that has a lot of use cases for different machine learning algorithms in different cyber security cases. And a third book recommended by Bill Gates himself how to statistics. It's a book that explains how statistics could be manipulated and how we should defend against that. At the start of my presentation, I had some statistical information but that
all the data behind it is checked. So anytime you see statistical data always check the data behind it. So thank you for your attention and that's my contact information. Thanks AI. Uh we have a few minutes for questions. Any questions from the audience? All right we have one. Um, Nikolai from web pros. Um, maybe I missed it in the start of the talk, but do you have
any examples of AI actually except from adversarial gun networks and stuff uh actually used in the wild for attacks like as a component of attacking system? Well, there could be when one case scenario that is very harmful is using K you could basically create almost identical um images of u uh how can I explain it? Let's say have uh cancer diagnosis or medical uh um based medical
images of a medical research performed on a uh on a patient. You can create almost identical images that could fool you on doctors. So basically you can introduce uh uh such images and uh that would force people that are healthy to go uh have an operation and of course people that are in critical condition to look healthy and that kind of attacks uh have already been performed
or some kind of hospitals and uh things like that uh and uh yeah that's a example. So defakes of course you know about them fake information all sorts of things like that. All right other questions. >> Thank you. Uh since you mentioned deep fakes and attacks using like uh AI generated voice AI generated videos can AI be used to detect such AI threats? Well, yes, using the
uh generative adversarial networks that I showed in one of the slides. But uh basically we have to train our models with those types of uh attacks. Uh basically we have to provide him with this type of data. We should label it as artificial and uh fake and we should train our models to in order to recognize that data otherwise it would recognize that data is actually real
not fake. Basically, we have to train our model with a lot of fake data in order for it to run to recognize fake data. No other way possible. There's no other way possible to defend against it. And of course, models would generate more sophisticated fake data and we have to retrain our models and that's a continuous process. Unfortunately, that's the only way. >> All right. Thanks. >>
What? >> Absolutely. the result machineops is basically focused on things like that specifically. >> All right, thank you. Our time is up so you can catch up with outside in the speakers corners. Thanks again. Thank
