Rook: Intro and Deep Dive With Ceph Storage - Artem Torubarov, Deepika Upadhyay & Niels de Vos

This talk provides an in-depth overview of Rook, an open-source cloud-native storage orchestrator for Kubernetes. The speakers discuss the architecture of Rook and its integration with Ceph, highlighting how Rook automates storage deployment, configuration, and updates using custom resource definitions (CRDs). They explore the benefits of Rook in terms of scalability, data replication, and customization. A case study is presented, detailing a migration project in which Rook and Ceph supported a storage architecture managing 120 petabytes of data. Additionally, the session delves into recent enhancements within Rook and Ceph, including the introduction of erasure coding, which optimizes storage efficiency by reducing the amount of needed storage. The speakers emphasize the importance of community feedback and the open-source nature of both Rook and Ceph.

So hello everyone uh good afternoon uh we are gathered here for uh Rook storage for Kubernetes talk. I'm Dipika. I work with Clyo. I have been a Rook contributor quite for quite some time consulting around Rook as well. Uh excited to be here. Uh Artum. >> Yeah. Hello. My name is Artum. Also working for Clya. I'm an active uh Rook contributor for the last couple of years.

My name is Neil Devos. I work for IBM at the SEF team and I'm one of the SECI maintainers. I contribute occasionally to Rook. >> So coming to the agenda for the day, we'll have a short introduction around what Rook and SE is. Then we'll dive into uh how you can uh configure the cluster topology and do maintenance for your rrook clusters. Then we'll have a very

interesting production user study that's pabyte scale. So hopefully uh you'll find it interesting and then we'll uh dive a bit into more technical side with uh understanding how se CSI driver is and what are the features around it how it works. Lastly, we'll have a recent enhancement uh in uh SE which is erasia coding and with that improvement SE uh will become more faster and more efficient

in storing your data. So we'll also highlight more about that. So coming to the introduction to Rook. Uh so what is Rook? It essentially brings self storage to your Kubernetes cluster. uh Rook is uh an operator uh that will help you install self storage and uh with the help of operator and CRDs so completely Kubernetes patterns it will automate the deployment configuration and updates of SE storage

for you again that has been a pain point for SE uh so Rook solves that for you and uh it uh will give you an easy to consume storage uh for your Kubernetes cluster. It can be in form of storage classes, PV PVCs. So, uh you can just plug it uh in your apps and get started with uh using uh Rook and SE. Uh it's completely open

source. Rook is open source. SE is open source. So, aim to get completely open source solution with Apache 2.0 license. And we are celebrating the 10th birthday. So, yeah, you can drop by and uh say hi to our setup booth as well. So coming into how uh the architecture layer looks for Rook. Uh firstly we have the Rook operator that will deploy and manage SE for you.

Once SE has been deployed by uh Rook the CSI layer will help you provision and mount storage for you. Your applications uh now will have uh uh uh provision or mounts with uh SE storage. Once that is established, your data will get uh stored in the data layer which is SE itself. Once uh that connection is made uh there is a direct uh uh direct communication between

the application and data storage uh for the your application. So looking into what SE is, it's completely open-source distributed enterprisegrade storage platform. It provides you with all-in-one storage solution for your Kubernetes. Uh uh it provides you with uh readride once with block uh using SER RBD rewrite many with uh file using SEFS and S3 like object storage with SER rgw. So uh now we looked into uh

what the storage uh is uh how it installs. So the short answer to that is wherever you have Kubernetes you can install your rrookf clusters. What it uh means is you can install it in um cloud environment using EBS or persistent dish virtual machines for your flexibility or just getting started or you can have it a bare metal or on-prem installation as well with your customization your

SSDs NVME HDDs anything for performance and control over your data or you can uh mix uh and bring best of both worlds with hybrid or multicloud. solutions we using any vendor or no vendor at all as well uh for resiliency and full customization. So looking into why Rook so uh some of the challenges we face with other storage providers uh one big one is fear of data

loss. If I centralize my data storage, there is a chance that something happens to that uh storage unit and your just data gets lost. We solve this with Rook and SE by replicating and distributing your data across availability zones. So you have complete control over how many replicas you want to store and in which availability zones you want to store. You can have four copies, three copies,

six copies uh spread across wherever you want and that's completely customizable according to your needs. Then we have some problems around limited PVs per nodes or poor performance when we have a lot of small PVS. Uh with SE uh it is designed in such a way that it is storing data essentially to pool level. So once the communication has been established between your application and SE uh

it just communicates directly to store data and uh completely um uh kind of uh dynamically and optimizes uh the data placement for you uh from its end. So having small PVs is never a problem for SE and you don't have any vendor lockin uh and you have uh uh generally if you uh go with other providers you might uh have vendor lock in come in to picture

later on but with it's completely open source uh provides you with all three in one solution so you can get started with just using it as a unified storage solution across multiple cloud out. So YF uh it's very uh customizable and very scalable. You can add capacity without any downtime. You can uh uh scale it uh up or out. That means you can add more nodes to

it. You can add more disc to it and that is without any downtime. So uh as your need grows uh your SE cluster uh as well grows and adapts and it's thin provision that means if you are creating a lot of PVCs it's not hogging your data space it's going to efficiently store data uh according to how much storage has actually been used. So your storage capacity

is independent of PVCs. Now looking into the recent features uh that has been added to Rook. Uh we have a lot of improvements coming in the SEC CSI operator. It's now enabled by default in Rook version 1.18. And uh we we can customize a lot of things now uh using CRDs in SEC CSI driver itself that uh that involves RBD, SEFS, NFS, NVMe. Neil will cover more

about it in later on. And uh in the coming release that's 1.20 in May 2026, we want to um establish everything to be customized using custom resources for SEC CSI and completely remove uh the Rook operators uh uh settings reliance uh for SEC CSI operator. So uh everything's exciting there. And then we have a lot of improvements uh around adding support for NVME OF gateway which is

in experimental phase right now. Uh now we have support for NVME over fabrics protocol which is a NVME TCP protocol that supports and works with SE block devices. And now your external clients can have access to SEB block devices using the standard protocol using NVMEF initiators. This will help uh deliver a low latency high throughput block storage access for you with your existing network infrastructure. So this

is an experimental feature in Rook 1.19 with SE tentacle or version 20. So uh do give it a try and we are looking for feedback there. So yeah uh now my colleague would talk about cluster maintenance. >> Um yes hello let's see how maintenance implemented uh in rook. So uh how can I take down part of the cluster for maintenance for upgrade or other maintenance operation without

affecting the users? Uh the answer is that we should understand cluster topology. Uh suppose we have a safe cluster with three availability zones A, B and C and have replication size three. In this case SE make sure that each of the zones has exactly one copy of the data. Now uh one of the zones go down. Uh don't worry uh in this case cluster is fully operational

and online for users. It's continued to accepting reads and rights and if zone will go back online the data will be rebalanced and the cluster will heal itself. Uh now second zone also goes down. What happens now? So in this case uh data is still safe. Uh the last copy is preserved on the last note uh lason sorry but um in this case seph uh stop serving

any IO to the users to make sure that uh the last uh uh copy of the data is preserved. Um the most common maintenance is upgrade uh in rook it's implemented in uh with the port disruption budgets. Uh it makes kubernetes aware of uh safe cluster topology. Um uh in a simple words um it makes sure that during upgrade we have only one zone down at a

time. Uh Rook also has a keep plugin. It helps to uh have a shortcuts for some uh safe operations like checking status uh removing OSD uh restoring monitor quorum. So please uh let us know which other uh SE operations are you doing and we looking what else could be automated. Uh now I want to present a short uh case study from the project which I was involved

for the last couple of years. It's a team effort uh of many brilliant people from SAP Clyo and of course uh Rook open source community and the project maintainers. Uh on the slide is a map of SAP cloud infrastructure. Uh it's responsible for storing 120 pabytes of data across 15 regions and uh 26 availability zones. Uh the old storage architecture uh was using uh proprietary appliance and

was using also open stack swift uh as a object storage. Uh, OpenStack Swift is uh eventually consistent storage without uh read after rights guarantee and as you may know uh a few years ago uh AWS made S3 strictly consistent and sometime later they also added conditional rights. So these two abstractions are extremely useful and nice for application developers. So today uh any modern application expects uh strict

consistency and uh read after rights from any object storage. Uh and the goal of this project was to migrate um storage architecture into modern solution. Uh this project uh is using a reference architecture. Uh it's a project governed by Neon Foundation part of the Linux Foundation Europe and funded by European Union. The goal of the project is to develop enterprise ready fully open-source cloud stack uh for

European businesses and the Rook together with the SE was selected to serve um storage layer. Uh the migration started two years ago uh with the first region went live in May 2024. Today we have 10 regions live uh with 37 uh pabytes of row capacity. Uh right now it's only object storage but the goal is to also migrate VM block devices into safe airbd and have uh

at least 30 clusters with 120 pabytes uh capacity. So I think uh the smallest cluster today is uh uh 15 nodes. The biggest one is 60 and it's growing. Uh all clusters had the same uh nodes u hardware architecture with the 16 NVME drives and the 100 GBs network. Uh before uh making uh region available to the customers, it's always a good idea to test its limits.

Uh what we did here is created uh a bucket uh in in NVMe cluster with 2 million objects and used 20 K6 uh instances to uh gradually ramping up load for 30 minutes until we started seeing back pressure errors and latency spikes. Uh I hope you can see it on the slide but somewhere until um 90k requests the latency is absolutely flat. uh then saturation begins and

uh around 170k requests we start seeing spike in uh back pressure errors and um also big latency spike. So for this cluster we selected 90k as a safe baseline for operation. Um the whole apa stack from bare metal to platform layer provisioned as operators and using infrastructure as a code. So Rook was a natural fit for the project. Uh uh we have a base blueprint as a

helm chart with all uh cluster defaults uh device filters and we have a per region overlay with the pool definition and GW placement targets. So today we can actually provision uh the whole region with hardware in one click. Uh yeah as uh I explained before uh maintenance and upgrade is routine operation with Rook. Uh in this environment uh we have monthly OS upgrade. Uh we also upgrade

Kubernetes version uh once per quarter. Uh the same for Rook. Uh sometimes we upgrade Rook uh even more often if you need some particular features from upstream. Uh we also did one uh major safe version upgrade last year from reef to squit and so far uh there is no um uh customerf facing incidents or downtime. Uh yeah we published this case study in r blog post in

text you can scan cure code or just search for rrooklo it will be in the first link in search engine. Uh yeah if you have also question and want more details we will be in solution showcase in rot table and uh other team members also so we'll be happy to uh explain more details uh in the case study. So now uh welcome to speak about CSI. >>

Yes. So SECI is one of the layers that Deepika mentioned in her slides. Um it's below rrook um above the se cluster and still somewhere in between your application pots. Um sefi itself is eventually a container image. This container image provides four different backends on the storage. um RBD and SEFS are the default drivers that we would recommend in Kubernetes clusters, but there's also NFS that users

can use um for um operating systems that may not support native SEFS um or uh live outside of the Kubernetes cluster and want to connect into uh the SE storage from outside. The same for NVMe. NVME is something that we recently added. It's still a bit experimental. There are some corner cases on kernel versions that are required and what not and we're still trying to figure out

the best combination and the actual requirements before making it in general available. Um, however, users can try it out and and test it for standard workloads. Um, SEFS is the readr many file system. Uh, very similar to what NFS does. And you can store files, read files from multiple nodes at the same time on a single PVC. Um, this has some performance issues. If you use a

lot of small files, um, metadata operations are slow. So for example, a whole tree of millions of files and walking through that with just an ls is not very efficient. Um for that we would recommend RBD with read write once uh mode which is an network block device formatted with for example ext4 or xfs. Um this is way more efficient for small files, databases and and similar

kind of workloads. RBD does offer readrite many um a local file system like ext4 should not be attached to multiple nodes at the same time because reading and writing data will cause uh data inconsistencies and corruption. Readrite many can be used in block mode. So you have a raw block image attached to multiple nodes and projects like cube use this to enable live migration of virtual machines

between different nodes. So at a certain time when the virtual machine is going to get live migrated the block device is attached to a second node the destination the virtual machine is migrated um all its memory and whatn not is copied over whatever KVM QMU protocol to the other node. Um the vual machine is started on the second node and the first node is unmounted again the

block device is detached. We also have NFS and NVME UF. Um both are layered on top of um in a case of NFS sefs and in the case of NVME UF um on RBD SEC CSI uh as I mentioned is a single container image. So whatever backend driver you want to use, you just pull the container image once and if you use different storage back ends on

the same node, you just have the single image, but it's used in different parts. The functionality of SECSI is controlled by different by different process parameters. Um you tell it if it's an uh controller or provisioner um or a note plugin which is doing the mounting. um you also pass it whatever back end it should use. So depending on how many backends you have on your worker

node if you have seested RBD you have two instances of SEC CSI running on that particular work worker node. SECSI uses Radus OMAP a lot. The OMAP is a key value store and we use it to manage the unique ID of a PV um to map to the backend um storage volume. So this can be an LBD image, a CFS sub volume um and whatnot. Uh the

key value map is a rather important part and we'll come back to that uh at a later point in the presentation. SEC CSI mostly is using go for any SE operations. Um initially segi used to execute se comment parameters and um executing a command from within go works well but a command is shortliving and every time you execute a command it needs to connect to the se

cluster. This is not very performant because you have to connect to the se cluster for every little bit that you do. Um with the go Seth bindings we actually use the uh C libraries the shared libraries uh that seph provides lip rad slip fs and so on and with that we can keep a connection open it's way more efficient and performance um for providing provisioning volumes um

maintaining volumes is way way better um it also gives us a lot of options on how to handle things internally CSI I offers a lot of features but it's still pretty basic um on operations on storage operations. Um one of the features is uh extending volumes. You can't re reduce the size of a volume because data might be allocated on the volume that you can't just chop

off. So only expansion is available. You can do cloning and snapshotting um to duplicate your volumes uh potentially to create a staging environment or um make backups. It is possible to use uh topology awareness so you can read from the local OSD your storage back end um instead of a remote one. This is important if you have multiple zones in certain cloud providers where interzone communication costs

extra money and local communication to in the local zone might be for free. we have ephemeral volumes. These are short living volumes and just um get cleaned up once uh the pot dies. SECI offers a fancy feature um for uh PVC encryption. Uh on RBD this user can use block encryption with looks. Uh for file systems also on RBD you can use uh FS script. SEFS also

supports FScript the keys are stored in the different KMSs that are available in uh various cloud environments. Um it's a little bit chopped off. Uh the last line says IBM HPCS and KIP. KIP is a standard protocol for uh key exchanges and and so on. New features include volume group snapshots. Uh volume group snapshots can be seen as something called consistency groups. In traditional storage, you have

an application that uses multiple volumes and you want to create a snapshot of this application. You don't create a snapshot of each volume separately because you want to have the volumes to be consistent together. So your database and your uploaded files or whatnot um need to be at the same state in order to make sense when you restore it. So you can label PVCs and you can

create a group snapshot and um these snapshots that are generated as part of this group all have the same state. Before creating this group snapshot all IO is paused. The snapshot is created of all of the volumes and after that the snapshots uh are created the volume uh IO is resumed again. We added uh block metrics usually on file systems so can get metrics of the PVCs

but these are usually file system metrics um or metrics that don't need credentials um with SE we do need credentials to connect to the SE cluster and we couldn't get the metrics of block devices for example for VMs and recently we added a way to actually get this metrics which is very useful for project like cube foot again um they tend to use RBD relatively a lot

um and come with uh requests for features. One of those features is change block tracking. Change block tracking is used to create differential backups. So you have an backup application like custom IO that uh suggest or that requests the difference between two snapshots and um the change block tracking returns a list of blocks that have been modified between those two snapshots. So the backup application they can

actually efficiently back up things quite nicely. CSI add-ons is a project that extends the CSI specification. It adds uh a controller and a sidecar. The sidecar is part of every CSI driver deployment um and talks to the CSI driver to execute the final operations. This makes it possible to do extra complex or advanced storage operations like uh reclaiming space, network fencing and uh key encryption if you

have encrypted volumes. So this is quite nice. It's not a standard Kubernetes feature but it works on uh CD. So very familiar for for Kubernetes um users. For example, you have jobs to uh reclaim space and so on. We have an additional feature for disaster recovery. Uh volume replication, volume group replication. This can be used to replicate the state of your volumes to a secondary SE cluster.

uh if for whatever reason your SE cluster goes down, your whole data center is powerless or internetless or what not um you can resume the state on your secondary SE cluster and have your disaster recovery uh from there. This ties in with a project called Ramen DR and that has abilities to mark applications to uh fail over as a primary and fall back um once the the

cluster comes back online. For example, one of the features that comes up now is erasia coding. Erasia coding makes it possible for the storage to be used less. So usually SE volumes are replicated or most of the SE volumes are replicated and they are three-way replicated meaning that if you store your data on the volume it's actually stored three times on the back end. This is very

inefficient and cost like three times the hard disks that you would expected to use. um eraser coding uh depending on the the eraser coding profile that you choose um can uh reduce the required storage really a lot. So instead of like three times um you might just need one and a half times the storage but the availability and the um uh failure toleration is still the same.

So this is very beneficial. Uh there are a few things that that need to be done for this. Um these one of the standard profiles is a four plus two profile and that means that you have four data blocks you have two encoded chunks and either one either chunk or either data block you can lose two of these and you still have enough uh data or parity

chunks to calculate everything else. This is similar to like a rate five configuration. Um, and this can save you really a lot of of With Rook, it's pretty simple to set up. Uh, with Seph on the left side, you would see that there's a lot of things that are required. Um with Rook you just create the pool and you configure um some of the values and Rook

will configure the whole STE including the failure domains and whatnot in such a way that um it's easy and safe. If you do it manually it's rather tricky. There are some trade-offs and that means that um erasia coding should be used in environments that have sufficient CPU because calculating the parity takes um some CPU. It is a little bit slower than replication. Um so you need to

be careful what kind of workloads you put on top of it. There is a dependency on how many zones availability zones you have and that depends on which K and M values you calculate. Um so how many bits and pieces of a data piece may be missing before um it becomes unavailable and um currently we mostly recommend it for um archived workloads and so on. So large

files this is uh the best way uh currently the next version of EC the next version of SE introduces a bit more um currently we're in septentacle and we already have many enhancements in uh AC encoding. Um let's move on a little bit. uh the green line is what the baseline is the replication. The orange line is what we want to see. So ideally the lines are

flat. That means that there's very little uh latency and you see in the curve going towards the right that um latency goes up. The right side is the random right and that's where the green and orange are pretty similar and that means that the EC implementation is almost the same as or performs almost the same as what the replication does. The QR code on the left bottom

is for a talk on fast EC from Sepholocon in October. So that's pretty recent and contains all of the technical details on what's what's been improved. We'll skip it a little bit because we're running out of time. Um, erasia coding is usually configured in a data pool. Um, in the future we have more features in uh SE that makes it possible to just have um one a

single pool for all your data. they don't need the metadata pool anymore and you can migrate um from your old erasia coded environment into the new configurations. Um so this looks really good and it's coming up in the next release this summer. That's it for our talk. Thank you very much for joining. I don't think we have time for questions. That's very unfortunate. tomorrow morning um at

the project stand. Um we're there again and you can ask your questions there or catch us um once we leave the room. Thank you so much.