Ron Ekins: Protecting Database Backups From Ransomware and Malicious Intent

This session focuses on protecting Oracle databases and backups from ransomware and malicious attacks, presented by Ron Atkins, an expert in database security. He defines malware and ransomware, emphasizing their threats to data security and operational continuity. The speaker discusses the increasing sophistication and prevalence of ransomware attacks, highlighting their potential impact on both production environments and backups. He introduces a defense-in-depth strategy that includes securing policies, physical and network security, host hardening, and application security. Atkins also covers practical tools and methods for safeguarding data, including snapshots and automating recovery processes. Lastly, he stresses the importance of preparedness and training in responding to potential cyber threats and data breaches.

I will remove from stage okay run and then I will okay thank you [Music] welcome to the first session of the data track uh we are in the devops Pro uh event online this is a two days of online talks and uh we will continue with two days of live sessions at vus Lithuania Lithuania sorry I probably pronounce it even even bad for a Lithuanian person uh

my name is Francisco bolini I work for open neula systems and I will be your host for today data session track uh so here we are the first session will be um develop um delivered by Ron Atkins uh Ron is the principal architect and database practice lead for databases at Pure Storage uh he has more than 30 years of experience but probably he doesn't like to say

that he's a little bit old no it's not old he's you will see him on stage he's a lot of experience and expertise uh he has five us patents file around data security in governments so welcome run stage uh welcome uh we will be having your session today about uh security so I will leave you with um the slides and the floor is yours can you hear

me run hello run I think that we lost you hello hello good morning I'm not sure if you can hear me okay yes we can hear you we can hear you hello go ahead good morning welcome to the session protecting oral databases and backups and Ransom where and malicious intent so that's me uh ronins director of field solution architecture em latan for p storage as Francisco introduced

uh easy email address if you want to get a hold of me ronp storage.com uh I use Twitter a lot so ronans if you want to follow me there I have a a technology theme blog ring.com it's one of the top 50 all blogs globally so uh whenever I do a presentation whenever I go to a conference I like to uh write about it and share code

examples on my blog and all the code examples that I use throughout my demonstration throughout my uh presentations I upload at my gab repository below that picture of me there is my orle a director uh logo so I'm one of only 70 orle Ace directors globally so the Ace Program is a community program recognition program by Oracle 600 Aces globally uh three tiers associate professional and Ace

director and I I say there's only uh 70 Ace directors so I'm very proud of that so looking forward to speaking to you so today's agenda uh what is malware and ranw let's get some terminology out there uh said as Franchesco said I've been working in defense and security for the whole of my career uh working in Aerospace and defense Industries and finance so what we've done

and what we always continue to do is think defense in depth and we'll just talk about that briefly uh and then the meat of the presentation how we can protect our data from ransomware militias and tent and then finally maybe what else we can do so what are malware and ransomware so malware and this is a definition by the national cyber security Center in the UK uh

malware is militia software which if able to run can cause harm in many ways including and there's a list there and the ones we're going to be talking about today are the ones in Orange stealing deleting or encrypting data but it's not just limited to that so when we talk of malware it could be using your devices against you it could be consuming Services which you're paying

for those so maybe doing Bitcoin mining on your platform or using your machines to attack another Organization for industrial Espionage so that's malware but what ransomware so ransomware is a form of malware so it's about blocking access to your computer and the data stored on it so this could be stealing the data deleting the data encrypting the data it's holding your data as a hostage so in

the same way we have ransom for people we can have ransome for data so you pay us to get access to your data or if you don't pay us we we will share your data with other people it's becoming more and more welln you know and for a long time W to cry was an example of a a ransomware malware attack but there are many others now

there is a confusion that it's actually very complex well it's not many many people can establish a ransomware as a service there a ransomware as a service kits available you only need minimal it knowledge to how to use those so you go to a a site where the ransom kit provider gives you the tools to create your payload so they pay you pay for this service typically

they take 20% as the offer and then you take 80% of the money but you take 100% of the risk so depending on the complexity and the sophistication of solution there would be different pay pay Rises but as you can see on the right hand side the online Builder there you have a number of prompts and then you can create your payload and then distribute it all

organizations now are are targets for for ransomware attacks you know we can see this in public forums like uh deutan train tables there being attacked or in the the news so we like to think it's about big organizations maybe Airlines here Bangkok Airlines or a school you think why would anyone attack a school or every company every organization whether that be commercial public has data all data

is available attack all Industries are vulnerable hotels here we have a an example of MGM Studios uh MGM casinos being attacked last year I was in Vegas at the time they was unable to check anyone into hotels during this period they ended up paying the money because they were losing business the trouble is if you pay a a ransom wear fine a fee you end up paying

again and again so that's really attacking the production server but hackers are no longer just limiting themselves to the production data they're also hunting for your backups and they spend many days on your network looking for the production database and test databases backups before looking at encrypting the if you know that people's recovery mechanism is to go for the backups in event of a disaster then the

first thing the hackers do is destroy your backups and they're getting better at it so back in 200 11 this is the first time it was recorded uh the term dwell time how long uh hackers on your platform before they they uh they attack it was a long time it took a long time to navigate the organization navigate all directory Shar uh file systems where backups are

to map that organization roll forward 10 years down to 24 days a lot more automation 22 21 days and last year you know the the research is saying that they're getting this done in a day less than a day the volume of attacks are increasing and the sophistication of the attacks are increasing every time we think we fixed it the ransom worked guys find a way around

it a lot of innovation there because it's a very financially rewarding piece of work so we need to make sure our backups are there to protect us if your data is encrypted your backups are your last line of defense or it's the pay and pray and pay again and again and again so what can we do so we need to think defense in depth data Security application

security and I like to think of security as an onion so we start on the outside with our policies and procedures then we look at our physical security network security host Security application security and at the heart of the uh solution has to be our data when we start on the outside think of awareness so our policies start with our employees and our contractors make sure they're

trained make sure you know who you're hiring physical security you know what uh cameras do you have there what Gates physical security uh controlling your site and that was good in a physical world and that's when we was only on one site but with more and more people working remotely we need to think about network security and not just working from home working in a coffee shop

working on a plane I'm not sure if it's just me but if I'm sitting on a plane I have no interest in the person's work they're working on but my eyes will always cast over to the spreadsheet that they're working on or the word uh document that they may be typing it's human nature we just look same as way we may be want interested what film they're

watching on the plane so we need to think about where we're working and how we're looking after the data what we're presenting then we look at uh our host security inside our company Harden our servers so when we make that Harden servers only use required Services if you don't need the service disable it don't advertise more information than you need to so if you have a banner

message you know you may have a security Banner message there that you need to include but maybe you don't want to broadcast the version of the operating system or the the uh and the release version because that gives someone that's trying to hack your server information they can use application security it's great protecting the server but if your application hasn't been fully patched and if you're not

maintaining those then the easiest way to your data is through the application so we need to make sure that we're working with our application providers that to ensure the that they're secure and they're protecting the data and then when you get to data security know where your data is not just your production data but when you take that production data and you create development clones are they

masked or they the same set of data when we do a backup do we know where that backup goes is it on another set of discs another file system on a tape in the cloud who has access to it so we need to understand the full life cycle about data use software tools Solutions like s to uh to capture logs and analyze it look for unusual patterns

so this is part of that picture so when we think about our database you know what can we do how can we protect our database from human error malware malicious intent so from an oral database point of view we could actually use orle to help us so orle ASM works on block storage so it's harder for malware and in true to discover block storage if you're not

using a file system it's harder to most malware Solutions look to encrypt a file system so if you're using a raw system where's a little bit harder not impossible it's just maybe if you're only onto a platform for a day before you get detected you're going to go for the easy solution so maybe you're you realize it's an allo database running on or ASM and move on

or give us another option using Oracle ASM we can use the filter driver so this gives us another level of control it limits access to just the all to the diss to just the oral database processes so a root user so if your platform's been compromised they can't get to the data either they can't look at these volumes so we can use some of the software and

some of the tools that database lenders provide here's an example of the ASM filter driver so I'm listing the devices I have ASM filter enabled I try to write to those blocks to encrypt it and it gets denied so this is functionality we have today that you can use to protect your oral database and there I can see that the block is fine another line of defense

is using Data Guard when we think of dat uh databases we think of a data center we think of Data Guard maintaining a standby copy typically in our Dr site so maybe our primary data center gets compromised but our secondary data center hasn't been compromised our Dr site so in this respect because we're not using file based replication we're using a redo stream maybe our standby database

is okay because we're not replicating encrypted files but it may be fine it may not be fine we wouldn't want to guarantee that that's not going to be compromised because if our primary day site has been compromised maybe the virus maybe the malware maybe the Intruder has also got to our Dr site so it's another line of defense but it's not going to give us that peace

of mind so thinking about our primary database we want to really use the features of our storage platforms so use primary storage snapshots wherever possible uh so most modern storage platforms have the ability to take storage snapshots these provide instant point in time recoveries they're readon immutable snapshots so we can use these to protect our database and application and give us that point in time recovery what

I would say here is don't just take a snapshot of your data include the application binaries and the environment it lives in so we can roll back not just the data but the application what you don't want to do is find that you have have some libraries that we compromised and as soon as you roll your database back the libraries kick in and then compromise the data

again so protect your environment with snapshots nothing's far F than a metadata operation a pure metadata operation we're not actually having to restore any data at this point when we're using snapshots we want to whatever possible automate take hands off keyboard remove the human element of uh of data protection use rest interfaces programming language whatever skills you have whether that be python Powershell Java bash most modern

applications data databases and storage platform support a rest interface so automate these automate the management test it in development test production and have the same configuration the same way of working have that consistency of operations gives us better protection and allows us to automate and know the outcomes of any deployments any changes here's an example of our I've used to take a snapshot using psql so this

is from inside oral database I can automate my storage platform to take a snapshot so the codes available in my get up repository uh I am creating a session on my storage platform from inside my database and then taking a snapshot so I can do this before I do a database release do an upgrade maybe before I do a a batch job or do anything which significant

I want to take a snapshot to protect it so it could be at 9:00 in the morning 9:00 at night every maybe every hour so we can schedule snapshots or we can have these triggered by the completion of a job all from inside the database and here's my plsql code my SQL Plus Code execute fa snapshot and then I'll give it a URL and a token and

if I go to my storage platform I can see that's taken place so that's looking at using snapshots to protect our production database what about our backups I said we need to protect our backups as well so this could be from accidental deletion from uh an oral Arman policy or backup policy depending on the Backup Tool you're using which says expire my backups after a week what

you got that policy wrong what rather than saying 10 days you said one day and you come in the next morning you find all your backups have been deleted because that's what you told it to do it was it wasn't a deliberate act it's just a human error we can protect ourselves against that as well so when I look at data security data protection I like to

use the the sto the original concept 3 to but extend it with a plus so 3 2 1 so finger out your database living on primary storage your database service so you need to be backing up to a different platform so you can use NFS S3 SB depending on the backup software and the database you're using so you need a storage platform which can support those so

free create one primary backup and two copies of your data so we have our primary copy and then we have our backup and then a secondary backup so we have that approach two save backups on two different media sites so I here I have my primary backup on my my primary data on my primary storage platform so this could be block and my first backup on file

on an NFS server so I've got two different media types block and file could be Cloud it could be S3 and then one off site so once I got it onto my backup device I want to replicate it to cloud or or to a another backup platform give me that off-site copy and maybe what has changed over the last 15 years is the plus we've always been

able to do snapshot for a long time we've been able to take backups and store those but we haven't really tied those together so now we think we want to do a production database snapshot there backups take snapshots there they were mutable but what happens if our storage platform gets compromised and someone has the credentials to delete snapshots we now need to protect our snapshots as well

so use your storage platform to take snapshots and to replicate it but also think what will happen if those storage platforms get compromised and someone can delete your snapshots and that's the plus we need to be thinking about how we protect that now one thing I always said to talk about the hardening the service we want to make it as hard as possible want to put block

in place want to delay The Intruders if it's automated software that's scanning our system it won't be able to discover a hidden NFS Mount point so install auto FS create an auto file and then start the service so what this will do is it will only Mount the file system when we're doing our backup once the backup's finished the file system will unmount so a casual Intruder

onto the platform wouldn't see the backups someone that has a bit of knowledge May oh look there is a directory called Mount or a backup maybe my backups are there and they will see D there they do an LS minus L see nothing and then move on this may give us a level of protection but when the backup runs because it's Tred to write to that directory

and it knows that there was a subd directory called demo in this screenshot here it will Mount the file system and suddenly my backup files appear and I can use those to wipe my up once I've done backup I want to take that snapshot so what I will do is automatically create a snapshot of my backups so that if there's a about policy error they won't get

deleted if there's a human error or a malicious intent someone CD so direct you where the backups have been created I have my Snapshot so make sure you always have a readon copy of your snap your backups in the event of a disaster again we can take a snapshot on whatever language you're using so if you're confident and Power shell Java python you can do that if

you use plsql we can also take snapshots here so here I'm taking a file based snapshot so again signing on to my storage platform and then taking a snapshot once I taken the snapshot I can see this in the operating system I can see the snapshot directory but I can't delete those files so they're protected from that human error that malware that encryption so that was a

plsql but also in Python so now I have my do snapshot try what can I do with it I need to do a recovery well I could copy the files out the do snap into my working directory but for orle we don't need to do that we can actually increase our catalog tell the orle our man catalog to use the snapshot directory so catalog start with snapshot

directory and that will import that into the catalog once it's imported into the catalog we can validate the database so for before we ever do a restore we should always maybe protect production system CU that may be better than what we're going back to so take a snapshot on your primary site and then also validate the Integrity of your backup before you do the restore just in

case there was some corruption there or it was incomplete or someone killed the backup during that uh backup you needed to use so validate the back up and then once it's validated you can do a restore directly at the snapshot directory but here ex this example taken out of a NFS bile system directly onto my or ASM but also could be restoring from an S3 bucket so

protecting our data for backups so talk to your storage platforms uh admin see if they have a way of protecting snapshots so we want to have our NFS snapshots and we want our block storage snapshots protected these could be from accidental or policy based deletion so many storage vendors now have the ability to lock the snapshots so even if if the storage platform gots compromised you could

not delete the snapshots they will be retained and you'd have to engage with the the storage vendor to be able to get access to those to remove those and this is that end to end process so primary storage backups appearing on a storage platform and replicate to another platform and then safe mode snapshots enabled to make them I love this expression uh quote that I heard at

a Polish loal User Group schroers backups the condition of any backup is unknown to restore is tempted play on the schroers cat experiment so just because we take backups doesn't know doesn't mean that we know that they're good we don't know that they're going to work until we use them so it's good practice to periodically test your backups do restore so you make you so you understand

how long they take to restore do a restore to make sure that they are complete and they are fit for purpose and also do a restore to make sure that you have good good knowledge of being able to repeat this what we don't want to do is do this for the first time when we have to so whether this is using backups to restore for development test

environments or part of operational testing make sure that it's documented and you have the skills to do so I guess this is the takeaway are you ready for a ransomware attack what will happen if you got attacked this afternoon do you know how to support that conversation do you know the steps required do you know if you have to take your storage platform offline and give that

to a government agency for them or the police to control access to that platform while the investigations going on what happens to your business during this time do you have relationship with a where you can ship in another storage platform while this is going on how you going to get your data back what if your data is encrypted there are lots of questions that have to be

had within the it and the business to decide what your strategy should be but we need to be having those conversations so that we are prepared in the event of an attack so it's not more if you are going to be attacked it's more when and when you attacked what will you do do service level agreements with the business if not what should they be what acceptable

if your system was down for a day a week would you still have a business how long does the restore take I've seen databases now 500 terabytes the biggest one I've been working on recently was 700 terabytes a restore can take many weeks with traditional methods so we need to think about how a business would survive that are they into system depend dependencies if you restored your

CRM system would you also have to restore your Erp and your HR System what about cloud-based Solutions SAS Services what inter system dependencies do they have if you have a system which says I I've raised purchase orders and you roll that one back and you have a separate system which does billing and it's says I paid the the the order or the order's already gone out and

been sent to a supplier maybe you've lost that order you don't know it's been sent out and you're going to get an invoice because you've rolled your system back so we that when we look at our operations do we have run books are the procedures maintained when we do operational changes at the infrastructure level do we roll those into the playbooks so peer review give your run

books to someone that hasn't written the Run book a clean pair of I see different things we are really good as human when we have a a a document we read what we want to read we put the steps in that are missing because we are familiar with those if we wrote that run book clean pair of I seeings so PE review all your procedures and documents

your run books to make sure they're complete finally get someone else to process if they are following the runbook line by line they will discover issue you're doing it yourself maybe your your do a step because you're familiar with that step other person isn't so wherever possible peer review and get someone test the best analogy I can think of when we think of our backups and data

protection is that of a pilot so a pilot doesn't work in isolation there were a number of people in the cockpit so the pilots and his uh second pilot but also he has a cockpit companion this is a paper document which is maintained and provided by the airline this lives in the plane in the cockpit hopefully he'll never use this but he has that Lun book available

to him but also he has the control tower he has advice that he can seek for from the airline and from the plane manufacturer so make sure you have those lines of communication you know you have that support contract with your infrastructure and storage vendors you have the relationships inside your business who's going to give you the authority to do that database restore how long will it

take to to get that decision made so all of this needs to be understood and then finally you need to have this companion guide of available to you so document simulate practice Pilots are really good at going into a flight simulator and experiencing things that hopefully they'll never see in their career as data professionals how often do we simulate disasters and try and recover from them not

very often when we think of the ransomware protection and the malware attacks we need to be simulating these disaster so we can protect ourselves from them data protection needs a team we need to think defense in depth we need to harden from the outside in engaging with our network teams our physical security teams our application teams our teams secure the data we need to constantly be reviewing

and challenging ourselves on operational ways of working when we change infrastructure does that change an error R books new malware gives us new threats so we need to keep up to date adopt that multi-layer approach snapshots backups immutable snapshots multi-layer gives our best chance of succeeding and don't work in isolation when you look at automation you know Unix administrator storage administrator Network administrators been really good at

automation but what we haven't been good at is collaborating with teams so develop a solution develop automation develop run books which span multiple teams application developers SES dbas we all need to work together to make sure safe and we can do this if we P our resources and that's me finished so I hope you found that uh challenging and enjoyable this morning so uh uh I will

uh leave the uh presentation now and start taking questions cool thanks Ron for the presentation I think it was super useful I have I have can't see any questions coming up yet yeah I have many questions there are not many questions on the chat but let's if we can welcome back something yeah welcome um you mentioned that uh about the ram somewere all the attacks and all

that this is something that we cannot just got any have you seen any chat uh any oh sorry we can't hear you ah you can hear me no hello hello hello can you hear me because I see no no sound on my end can you hear me Ron hello hello no you cannot hear me that's weird let's see how we can solve this okay in the meantime

if you have any questions uh feel free to ask on the um let's let me see if we can solve this issue with run okay well since uh we have this situation so I understand there's no questions in the uh in the chat so uh uh thank you Francisco for monitoring the uh I believe there is a a closing message from the sponsors for y the event

so I will I put myself on mute and hand the the Reigns back over to Francisco thank you for having us and have a wonderful day across the whole of Europe and enjoy your conference whether that be virtually or physically in sight on site over the next few days thank you again hope to uh maybe come out and see you in person year thanks Ron thanks actually

pretty nice to to have him speaking well he doesn't hear me so it's a nice thing anyways uh yeah he we I I can hear myself on the stream so that's uh something that we have an issue with Ron so apologies for the technical um situation right now um stay with us uh we'll be returning back at sorry I'm saying the time of uh the the conference

the time of the conference is 11:55 which is three hours U plus three uh UTC uh so if you're are in CST meaning that uh you are in the something between Poland no Poland is gm3 uh between Germany and Spain uh it's 10:55 um so stay with us uh we be returned shortly for the next session uh with Alper e kogu I'm probably going to pronounce it

so bad today the the surnames I'm I'm not so good on that so thanks everyone and see you in a few moments for