Falko Banaszak: Why a Layered Storage Architecture is Critical for Cyber Resilience

In this talk, Falko Wennerschark, a principal field solutions architect at Everpure, discusses the importance of a layered storage architecture for enhancing cyber resilience. He defines cyber resilience as the ability of organizations to withstand and recover from cyber attacks while ensuring continuous business operations. The speaker emphasizes that storage plays a critical role in this architecture by providing various layers of protection, including immutable snapshots, secure isolated recovery environments (SIRE), and leveraging cloud solutions. He explains the differences between block, file, and object storage, highlighting their respective advantages and challenges. Furthermore, he discusses the need for integrated security across all storage layers rather than treating it as an add-on feature. The goal is to achieve a robust recovery process, minimizing the risk of re-infection during data restoration.

All right. Thank you for being here. Um, my topic is called why a layered storage architecture is critical for cyber resilience. We will have to define some words, but before that first question, raise your hands, who deals with storage in their day-to-day job? One, two. Oh, many people, great, great. Good. All right, um, let's start with a little bit about me. Um, my name is Falko Wennerschark.

I'm a principal field solutions architect at Everpure. I'm from Germany. I do ramble in the field of data protection, cyber security, and I also consult on, yeah, business continuity management, which is not talking tech. Interesting, right? Um, but still it gives a lot of information about how customers run their business processes and what they need to achieve their recovery outcomes. Um, first of all, I want to

define cyber resilience because there's many buzzwords out there, many, I would say, declarations and terms which can be, or can have a different meaning. How I define personally cyber resilience the ability of an organization to anticipate, withstand, recover from, and adapt, so you learn, to cyber attacks, data breaches, and systems failures while maintaining continuous uninterrupted business operations. It's a long sentence, I know. And it also means

there's much to it in terms of cyber resilience, right? It also means there's many, many things to consider. It also means there's many metrics to consider. There's, yeah, basically it's a lot of stuff. And also, if you look at key metrics, for example, how it differs from cyber security, the cyber security key metric is more of the mean time to detect, whereas the cyber resilience key metric

is the mean time to recover, in my definition, MTTCR, mean time to clean recover. Because that's way more important than just recovering things. It's more important to really know that the things you recover are actually clean, or actually usable. So, why do we care about cyber resilience at all? I teased that a bit, but in general, we want to reduce one metric, which is called the work

recovery time. So, the work recovery time is the amount of time it takes to verify that a system, after a failure, is operational again. And there can be many things to it. It doesn't necessarily have to be an IT issue. It can also be an operations issue. It could also be, I don't know, a truck which doesn't arrive at your facilities to give you the material you

need to produce something. That could also be Um and that's why I say cyber resilience, that's also the big sentence, has some really many meanings to it. That's how I define it. why [clears throat] do we talk about storage? Because storage is old and boring, right? We just consume it, and it's there, and yeah, why not? Um I want to try a little example with the audience,

depending on how you guys want to do this. But, let's think of explaining storage to me like I was five. Who can tell me what this kind of storage is? Nope. It's block storage, obviously. Block storage means you have a fixed block. This is a parking lot. The car is your data, your block you're writing into. It's fixed, and that's why you can see you can have

many, many blocks in a volume, block storage, which gives you the ability to purchase the data there. What is this? Any guesses? That's file storage. The parking lot is your folder, and you can choose wherever you want to park in that folder in that file system. Very interesting. And now the premium, which also costing the most, is object storage. Probably most of the people here deal with

object storage. How does it work? You take your car, you drive to the valet guy, get your ticket, metadata, your secret access key, and place your storage anywhere, your data anywhere. This is This is the definition I use when I try to explain how storage works to a 5-year-old. But, let's go away from that. Uh let's look at it how it really is. Like I said, block

storage is structured data broken into fixed-size blocks, which means it can be used in high-performance infrastructures, mainly used for SAN, iSCSI, Fibre Channel, and most applications like databases or virtual machines benefit from it. Many people have block storage deployed out there, very popular and very important. File storage is hierarchical data organized in files and folders, and also it's shared across different probably departments. It's shared across different

different protocols. You can use NFS, SMB, but also multi-access if you want to. And mostly used for collaborative unstructured data or for unstructured data in general, like dumping files in And then we have object storage, which rise a bit from the last couple of years, where you have flat address space with rich metadata, and you can access it via API. That means uh you're very flexible with

that, and most of the uh I would say customers, but also applications and vendors out there adopt object storage in their offering, and also adapt in terms of which applications you can use to address and use object storage. That's important for the next couple of things. But, this is also where the problems begin, right? We still think and deploy in silos. Oh, there's the block storage department.

Oh, there's the file guys. Oh, there's the object storage guys. That's That's a thing that is common and it's very hard to combine all these types of storage to make it a platform. Platform is also a word which is probably a bad word, but still it means you have silos, you have different people, different technologies, different access methods, and different caveats to it and really makes it

hard to get everything under one single umbrella. So, if you look at the caveats uh per silo, let's go for production storage first. Obviously, you don't have any visibility into the content. It's just a block and a volume. There's no possible uh data loss prevention because you always have to mount the block volume. You don't know what's inside. Furthermore, um you don't have any IAM. There's no

real like a secret key or access key like with an object storage where you can really define who has access and which kind of access to it. When you can host a or you can mount a volume to a host, you can have pretty much fun with it because you can can just delete it, And there's so much more, but just a bit of that. File storage,

complex ACLs. Whoever had Did anyone have to deal with ACLs with NFS and SMB before? How do you like it? Boo. >> [laughter] >> Yeah, fun's done. Perfect. That's exactly what it is. It's complex um and furthermore, when someone just copies policies, you inherit permissions. And that's obviously a a security risk, right? Because when you inherit permissions, some people could access it and you don't even know

about it. And also, there's many, many applications out there with legacy protocols like SMB version one. Hopefully never no one uses it anymore, it is something to really care about. And also, file due to the fact that you can place millions and billions of files on file system is very slow when it comes to scanning or when it comes to actually using the data. So, there are

some gaps there. Object storage is not the holy grail, obviously. It also has some gaps and caveats. First off, um API and misconfigurations. That means you can expose an API, for example. Um who of you thinks that it's an uncommon thing to see some markdown files in a GitHub repository with external secret keys? Guess what? It's everywhere. Every GitHub repository No, not every, but we've seen that

example before. People committing stuff, people doing their GitOps and forgetting to basically strip down or remove external secret keys. If you can have those information, you can actually um get access to the bucket. And also shadow data because of the rise of adoption of object storage um was really really really fast. People just began to use it because it was so easy, right? So, you have to

manage multiple silos again, which you don't know of. That's That's some some of the gaps. The next problem is traditional thinking. You know the terms primary storage, secondary storage. Oh, primary is the fast one for my application. Secondary is the cheap one for backups. Yeah, guess what? That's not the new normal anymore. That's something we used to do. And either we tiered it, so you had like

a primary system or an application which accesses a short-term repository or short-term storage tier to a second one, or you materialize it. There's many different iterations to this problem. But, what I really want to say is that the key problem here is that security is often treated as an add-on to secondary storage, rather than integrated to the whole stack. And that's the whole talk about today. Many

people think about security in the backup space or in the secondary space. That's not enough anymore nowadays. That's what I'm trying to achieve and to explain today. And if you look at the history, um yeah, we we went through a change because we went from high availability back in 1990s where I was born um until now for cyber resilience which combines the security and the recovery part

of it. So actually, you want to know what's inside the data. You want to restore it clean. You don't want to just provide an HA solution. You really want to have the knowledge what's going on there. And the approach we use here, and that's what I'm explaining today, is air-gapping isolated recovery environments, many more, and that's also what we try to achieve here. So, why can we

talk about storage and cyber resilience then? Storage is important because obviously of first data resilience. That means threat actors will actively try to compromise your backups or seek to search for them and delete them. Second is for security performance. So, threat detection systems, SIEM systems, UEBA systems, they really really have a good advantage when they perform on good fast storage, right? But also in terms of layering,

that's what the whole topic is about today, uh it's not enough anymore to just have a backup layer. You need more layers. You need more layers of protection. And last but not least, obviously recovery performance. Performance always matters when you want and this is important as well. So, what I want to say with this, storage is now an active part of cyber resilience, not a necessity, not

something where you dump data to, but storage should really take place in your whole stack, in your whole architecture. What about the layers now? How does my example function in terms of a layered resilience architecture? we start with something which is pretty common to every one of you. So, by the way, this is not a technical session. Um there's no demo involved. Boo, I know. But still,

hopefully, it's interesting for you because I'm trying to break down the architecture components to just explain how it works. Everybody knows this. You have a data center, you have applications, you have environment, you have virtual machines running on a storage box. So, the first layer is a storage snapshot. The first layer of protection is a snapshot you can use, which is hopefully, depending on the storage vendor

of choice, immutable or even indelible. Indelibility is a great word for a German to pronounce because it's way harder than immutability. So, but still, indelibility means you cannot get rid of the data. That means no root access to the storage box, that means no policy changing, that means no whatever you want to do in terms of getting rid of it just with physical access or a baseball

bat, depending on what you want to do. That's the first layer. Everyone knows this, everyone has this. And then we replicate the data. We go, create a second data center, we create synchronous or asynchronous replication to a separate storage box, and make that HA available. So, that means the data you have in your first data center and the second is actually the same, but you protected yourself

from a geo-redundant failure, for example. Of course, this is not enough. And that's what I mean with now we start with the third layer. And the third layer is where most people think about cyber resilience, data protection. Your backup target, your archive target. That's not enough anymore. What I want to show is or to say is basically the backup target or the archive target, that's fine. It

has to be immutable, it has to be indelible, it has to be resilient, but start from the very first bit ever written on the first data center on the first storage box. Start there with features and functionalities which give you the protection layer. >> [sighs] >> And then we extend this SIRE, a secure isolated recovery environment. What that means is basically a complete separate environment locked down

from the primary environment with a one one-way ingest, for example, and with data which sits there waiting to be recovered or to be validated or to be audited when something breaks through data center one and two. You could say this is a disaster recovery environment. Yes, it was back in 1990. Nowadays, your data protection tools, your cybersecurity tools should have features and functionalities to actually get insights

in the data if it's clean, if you can recover it, if it's sensible to recover it. That's what the whole SIRE is about. >> Also, the cloud, the public cloud is another layer. You can always, for example, on small and storage platforms trigger snapshots which go through a cloud volume. You can use it there to recover back on premises if you want to. It's it's a very

valid layer. Not everyone uses it, but you can definitely do this. And what it gives you is a really quick another geolocation. Fifth layer, sixth layer, ultimate layer is the complete orchestrated recovery with a tool of choice. There's many data protection vendors out there who can control all those layers. That means who can control the data flow between one, two, three, four, five, and six or until

five, sorry. What that means is there's tools out there that can help you create backups, create snapshots, orchestrate them, replicate them. And that tool of choice you're also using when something really bad goes wrong, you can take that tool and take any layer you have and restore the data. So, why is this important? Why so many layers? Like I said, backups alone are not And I always

like to show this slide here because what it gives you is a glimpse of when you should use a backup actually. You can see RPO and RTO. And obviously when you have two data centers with two storage arrays and it's a synchronous replication with a transparent failover, RPO and RTO is basically zero. One service system goes down, second takes over, we don't see anything happen, right? But

if you go down that list, oh wonder, streaming backup and restore is the last level of choice. Because it can and I'm not talking a single virtual machine here or a single file, I'm talking about a mass restore at scale, right? It can take hours and days. So what I mean with that is all the other layers we just saw can be a very valid recovery point

to get your data back before you have to think about backup and restore or back uh restoring >> [gasps] >> That means having as much layers as possible is pretty interesting. Immutability versus indelibility, again pronunciation is a thing. I just want to make people aware of immutability is not that word for for the holy grail. Many people probably know that term immutability. But what it really means

is that you cannot modify data, right? Indelibility is much more important because indelibility means there's no modification and deletion possible before any policy or mechanism basically applies. Think of an XFS file system, there's sell backup appliances with an a hardened XFS Change attribute {dash} I, it's gone, right? So it's it's not indelible, it's immutable. And that's why I just want to make people aware of really think

and really ask questions, what does it actually do? So what do the the locking mechanisms do? Is it just protecting you from modification? Or can't you really not delete it? Is it like a indelible thing? So So normally how this works is when you destroy data or a storage snapshot, it goes into some sort of eradication bin. And in that eradication bin, that's probably how most vendors

out there work nowadays, there's a policy which gets applied there which only allows you to get rid of the data until a specific policy basically applies. So when you say 14 days of indelibility, you cannot get rid of the data until that a policy runs It's also important because um that timer obviously should not be dependent on the NTP or system time or any other times you

could modify, for example. Cuz that's mostly the first question. Very interesting topic now. Are snapshots backups? >> Who thinks snapshots are backups? Yes. Perfect. Andrew, in your session it was the same. Um no, obviously they are not backups, that said, everyone has an opinion on this, right? But it mostly comes down to the locality, if you can index it, if it's a application or crash consistent But

when you do it right and when you use the correct data protection software, for example, it can be a very well recovery point. It can be application consistent. And it can complement your backup infrastructure. It can complement your backups. then there's a very important discussion because nowadays there are storage vendors out there, for example, they say, "Let's go for snapshots all the way." Which helps you protect

your data and do it does backups. That's not really the case, right? It is an additional layer of protection. And also you don't have to move the data across the wire, which is a huge benefit for some environments. So yeah. Let's go through all the layers we just and talk about the benefits. Obviously, in this environment here, which is the basic first and two layers, the benefits

are you have two copies of data, you have immutability or indelibility from the first bit ever Remember, we're still not talking about backup here. You have a high availability option, and you can fail over or fail back if you want to do this because a transparent uh failover with a synchronous replication can help you to fail back if you want to do this. It's also feasible for

asynchronous replication because the small storage systems on the market know what they can expect when it comes when it comes to failover. Next layer is the backup and archive layer. You have three copies of data in this case. That means you not only have the high availability and failover possibility, you also have a backup and archive layer which is introduced. And what I recommend most of the

time here is introduce a new protocol. Most data protection vendors out there nowadays use object storage or S3 as their protocol of choice for backup storage. Do that because you then have block storage and file in the front end, you have object in the back end in the backup storage, which gives you three um kinds of um layers and also basically three different protocols, so it's it's

not that easy to hack everything when everything's the same. But still, it means that you have to manage it. So, really make sure that the platform you choose has some good management uh opportunities there. And now we come to the sire. That's the biggest part for today. Oh, the wrong spot. What is the sire? The sire is the most important cyber resilience piece in my eyes because

it's not dependent on your first environment. That means separate IAM, separate network, separate users and policies, separate access. There's not the same user, not the same workflow as you have on your primary And you could basically say you build up a secondary customer, a secondary complete environment. But what it has to do with the primary environment is that you ingest data from So that means you ingest

data over snapshots or over backups. Most vendors out there can do that and can connect to two And then you cut it off. You cut it off with networking, you cut it off with You really make it like a black box. There's features, functions, and also best practices available for many protection vendors out there already how to implement this and how to really get the benefits out

of that. Um secondly, uh of course, different firewalls, different switches, and networking. And you could also choose another vendor if you want to because that gives you also some more uh flexibility. But one more thing is important. In that environment, the sire environment, you can run forensic tools. So data you get there, which you need for restoring, can be analyzed, can be used for forensic stuff basically

to see if the data coming in the sire is actually recoverable, it's actually clean, and really gives you the confidence when you test it, which is a very important thing, um to save and clean recover. yeah, that being said, that's that's the most important thing. the benefits of the sire is clearly you have an air-gapped environment from the network. You have extreme restricted access controls that mean

one administrator could not have the um credentials for environment A, but also for environment B. And it's a dedicated and uncompromised hardware and software. That is very important because when really something and it affects your production environment and forensic people or some some police people I'm just missing the word in English right now that are coming to your environment and say, "Hey, you cannot use that data

anymore or that storage anymore." You have a separate environment which has never been affected. They will not let you not use that environment, right? That's the most important thing here. And also when you use the forensic tools, you prevent yourself from being re-infected during the recovery. That's what the sire is about. The forensic tools, the data protection cyber security tools you use in that sire environment helps

you to prevent the re-infection during that recovery. That also means you can use it monthly, weekly, depending on your SLA to actually prove the data, create reports, make someone else happy. But that also means you have the to really be sure that you can restore So, the purpose of this is obviously clear, right? We want to have a safe restoration of all the systems and data. And

we also make sure that we can detect the patient zero, the root cause. How does that function? Most and modern storage environments have some sort of anomaly detection. That means you can use that anomaly Sorry. You can use the anomaly detection to see if anything in a snapshot, in a block volume, in a file system, or an S3 bucket has some some some weird behaviors. And then

it gives you the information which kind of data point or snapshot or data backup you should use in terms of restoring. >> [sighs and gasps] >> And of course testing, testing, testing, testing, simulated. Try to really shut down the environment. try to really bring up the sire and see if everything works. Um and before anyone asks, you don't have to put everything in your production environment in

the sire. The sire should only reflect your, let's say, top 10 systems, top 10 applications. If it's 15, it's 15, right? But you get it. The thing is, if you have, for example, speaking of virtual machines, if you have 500 virtual machines, but only 25 are important for recovering your minimum viable business operations, use the 25 machines. That also means that the sire environment doesn't have to

be necessarily as big as the production environment, because then it's about expenses again, it's about, "Oh, why do I need all those memory and CPU?" Nowadays, it's pretty expensive. But it's a best practice. And also, like I guess, said again, regular testing and updates is important. Patch it, test it, test it, test it, and document The guy responsible for the sire when he leaves the company, do

you have a run book? Do you have a guide which helps you to clean or basically to help helps you to do that restore process, that whole recovery process? >> One more information, um there's a difference between some people call it clean room or sire. The clean room actually supports investigations. That means the clean room is also a very good thing, but it also conducts the forensic

analysis and it handles all the sensitive data. But it doesn't necessarily mean, like the meaning of the sire, that you can recover it. So, the sire really has to focus on the and on ensuring the backups and the data can be quickly recovered and cleanly recovered. The clean room is something many people build up with their data protection vendors to see, "Hey, the data I'm having there

is it clean?" Okay, yes, it's clean, check mark. But can you actually recover it? That's the >> Um what are the impacts now of a sire of a sire in terms of cyber resilience? Obviously, faster recovery times because and when you're confident that it works, you can simply hit a button. Very well said, right? But you have a process, you have a runbook ready to spin up

a whole complete environment without having to think about, "Ah, what virtual machine do I have to start? What application here? What database there?" No, it's all in your sire already. It's all there. One thing. Um you can test it. The sire the separate environment gives you all the information, all the networking, all the lab, everything you need basically to test everything to make sure you're confident when

really something bad happens. And it also helps forensic people. That means when something really goes wrong and forensic people from your cyber security insurance, for example, go to your environment and say, "Hey, we have to check it." You can use that sire efficiently to help search for that patient zero. And also it produces one very important thing, the reduced risk of a re-infection uh when it comes

to uh restoring data. Some people without a sire restore snapshots, restore backups, and then they see, "Ah, no, that was not the right one. We have to go one back further or one week further." So, that's something you hopefully can avoid when using a sire. What are the benefits uh with layer five and six now when it comes to the recovery in the cloud? Obviously, leveraging the

cloud can be another environment separate from production, which you don't have to build up yourself because obviously a sire is not something which you can simply buy somewhere or swipe a credit card through and say, "Hey, let's go." Um and also cloud mostly has built-in HA as well as geographic separation redundancy. So, you don't have to care about the availability, you can care about the data. That's

why the cloud is a very valid tool to choose for disaster recovery or for another layer. And also most data protection tools out there have an orchestration possibility to orchestrate storage data, backups, snapshot into the cloud and back and forth. This is the last slide and this is also something I want to really give to you, which is the key takeaways from this session before we go

into questions. Storage is and should be nowadays an active part of every cyber resilience architecture. It's not about the backup software only, it's not about the cyber security software only, it's about the whole stack and storage is a part of that. Storage helps you. Furthermore, a layer within the architecture ensures data availability through multiple isolation points. Immutability is great, like, but indelibility is is greater, it's it's

much more important, it's what you want. And the sire really helps you to adapt and facilitate recovery and forensics. When you set up a sire, you already made the investment into testing, testing, testing, restoring, and getting the workflow That is really, really important. And also snapshots are no backups, but when you do it right and correctly, they serve as a very valuable and speedy recovery option. That's

the key takeaways I want to give you. And uh thank you for listening. I'm open for questions now. >> [applause]