The Symbiosis of Storage and Workloads: Longhorn for Hyperconverged Bl... Jinhong Kim & Jangseon Ryu

This talk focuses on the challenges faced by Neighbor, a leading internet company in South Korea, in operating large-scale storage systems. The speaker discusses the inefficiencies and performance issues caused by traditional centralized storage models when handling distributed applications like Kafka. To address these issues, the presentation introduces a hyper-converged storage model that integrates local disks for improved flexibility and reduced replication overhead. The speaker highlights the importance of locality in storage placement, introduces Longhorn for managing storage in Kubernetes, and explains the role of CSI storage capacity in optimizing pod scheduling based on storage availability. The session concludes with insights on designing effective hyper-converged environments and the need for storage-aware scheduling.

Hi everyone, good afternoon. Thank you for joining us today. Uh today we'd like to share our experience operating large scale stories and neighbor and this scale we faced the several challenges and cost efficiency performance interference and operational complexity. So in this talk we'll talk about uh how we apply a hyper composite model using longhorn and how it helped us improve our system. Before we dive in let

us briefly introduce ourselves. My name is Jang newu and I'm a senior technical lead on container storage team and neighbor. I'm joined today by my coreer Gino Kim from the same team. Today we'll walk you through the background and challenges and Chino will explain how we solve them in detail. So who is neighbor? Neighbor is one of leading internet company in South Korea providing a wide range

digital services. Our services include such content, e-commerce, fintech and cloud services. Million of users use neighbor every day and generate massive traffic and data. neighbor scale we operate kubernetes across roughly 60,000 node and running around 90 900,000 path in production. So but storage we operate a large stepbased platform with about 5,000 node and roughly 50,000 disk in total this provide around 100 pabytes lo capacities and today

we served around 250,000 persistent volumes. Our infrastructure is built in on a centralized SE storage platform and SE works very well but however the centralized model doesn't fit every type of workload. So especially distribute application like capka open source sheet and betest. So those application already handled replication at the application layer and when you place them on top of distributed storage some inefficiency and side effect began

to appear. That's when we start to listen what kind of storage model would better fit these workload. So this led us to hyper converged approaching. So let's first look at the problem we faced and then we were trying to solve. So first major issue we faced was cost inefficiency. This inefficiency was caused by duplicate replication se by design use three-way replication or irre coding with roughly one

and a half times redundancy. So at at the same time many distributed application also replicate their data usually two copies sometimes three copies. So when you stack these layer together, replication quickly adds up. In the worst case, the same data stored up to nine time. It also affect performance through IO amplification. So I mean more copies means more IO. The second major issue we encountered was performance

intererience. Many distributed application assume that each volume behaves like a independent disk. But in reality all volumes share the same underlying storage system. Because of this application become sensitive to storage level event like hard failure rebuilding operation or heavy from other workloads increase can increase latency. So from other from the application perspective this appear as higher latency or unstable performance. To address these challenges we were already

using local disk in some workload. Local disk have several advantage. They avoid duplicate replication and reduce performance interference. But they introduce a different problem node rocking. So when a volume is created on local disk it becomes tightly bound that to that specific node and if that node pay the application can access its data. So we pace trade off set provide flexibility but can introduce duplication and interference.

local disc reduce interference but creates node rocking and operational complexity. So we needed a model that combine the strength of centralized storage and local that model was hyper composited. So we didn't want to replace SE because I already told you the SE works very well but for many workloads but we wanted was to combine the strengths of both model while avoiding their weakness. So and that and

from that we define five key goals. The first goal was flexible replication but different workload. Different workload require different replication strategies. So many distributed application already replicated their own data at the same at the uh application layer. So but this workload the single storage replica is opening up but workload built in uh without built-in the application require storage level redundancy. So I mean replication must be flexible

depending on workload needs. The second goal was better isolation workload isolations in centralized storage. Uh like said many workload share the same stories in PR that mean performance intererience can affect many volumes across the cluster with the hyper converged model solid resources are distributed across node and disk. So this improved workload isolation and reduced the impact noise neighbors. The third goal was locality with service continuity. So

we try to place storage on the same node as the workloads to achieve low latency. But if local capacity local dis capacity is not available the volume can be connect remotely. So latency may increase slightly but the service continues strong. The procore was coordinated scheduling to ensure locality. So in that in in the department model pass scheduling and storage scheduling are independent. So Kubernetes schedule part based

on compute resource and why volume placed replica based on storage capacities. So this mismatch can break locality and trigger unnecessary rebuilding. So we align part placement and storage placement to maintain locality. The fifth goal is uh was storage mobility. So storage mobility enable much more flexible operation. For example, node may run out of capacity or sometimes no require maintenance or no noise neighbor may impact performance. In

this situation, the ability to move storage becomes extremely valuable. To address this challenge, we start looking into storage project uh in the CNCF ecosystem and we evaluate several options that could could support the hyper convergence model. So after comparing different trade, we decide to adapt long hon. So what is longhorn? Longhorn is cloud native distributed block storage system built per kubernetes. It use local disk attached to

kubernetes node and expose storage through through the standard CSI interface. From a user point of view, it behaves like any other Kubernetes storage class but internally it manage replica rebuild and volume life cycle within the clusters. So today we'll focus on portal and how we address it. The protocol is coordinated scheduling for stable locality. So we'll look at why locality is difficult to guarantee with the default

coordinate scheduleuler and what problem this cause. So now Chinhong will walk you through how we solve this. Uh hi guys. Uh let me start with a quick overview of the lumen architectures. To use storage in Kubernetes, a user first create a PBC. Uh once the PBC is created, Kubernetes and the long controller work together to provision the storage. Uh as part of this process, a PB process

volume is created and the PB map directly to a single long one volume. That long one volumes is made up of two key components. The engine and the replicas. The engine is the components that handles application IO. It sits in the data pass processes read and write request from the applications and communicate with the replicas to replicate data and maintain consistency. The replicas on the other hand

are the components that actually store the data. Because multiple replicas can be placed across different node, longhorn can provide redundancy and improve resilience. In a short, a PBC triggers PB provisioning. The Pway is map to a long horn volumes and that volumes consists of one engines and multiple replicas. If you if you'd like to learn more about Longhorn in details, I'd also recommend joining the maintainer session

tomorrow at 11:45 in room E103. So, the next question was simple. Where should the replica be placed? Originally p scheduling and replica scheduling were handled independently. Pots were placed based on compute side resource by coupululer while replicas were placed based on storage side resource by longhorn. That means even if related parts are intentionally spread across different node their storage replicas may still be concentrate on the same

node. To solve that we consider two approaches. One is to place replicas on the pass node. The other is to place the parts on the replicas node. But the second approach can conflict with the user's own scheduling policy such as pod affinity or anti-affffinity. So we decide to make storage follows path placement. That means we place replicas on the same node as the path so that storage

placement stay aligned with the workload placement intent. Uh here longhan provide a data locality options in the storage class. This option defines how closely storage placement follows workload placement. There is three options disabled, best effort and strict worker. If it is set to disable replicas, replicas are scheduled independently from the workload node. If it is set to best effort, Longhorn tries to place one workload whenever possible.

If the workload later moves to another node, Longhorn may try to rebuild a replicas on the workload node to keep that locality. And if it's set to six local, longhorn requires the only one replicas to be placed on the workload node. If that requirement cannot be satisfied, the volume creation fails. And now sticker is essentially similar to using local disk. However, as Jangzan mentioned earlier, our goal

was slightly different from pure local storage. Uh during normal operation, we wanted the replica to stay on the same node as the But during node maintenance or other operational events, we also wanted the storage to be able to move to another So for our use case, best effort was a better fit. It give us locality in normal situations while still keeping some flexibility during operations. So with

wafer first consumer and best effort this is what can happen with with first consumer means the pod is scheduled first and storage storage is provisioned afterward. Let's assume we have two parts each requesting a 40 gigabyte PBC and I refer to them as SCS0ero and SCS1 because there are part of a stateful set. So anti-affffinity applies between them. SCS0ero with a 40 GB PC is first scheduled

to node A by Kubernetes. Once the path lands on node A, long creates its replicas on the same node. Then SCS1 also with a 40 GB PBC is scheduled because of stateful set anti-affffinity. Kubernetes avoid node A and place it node B. Longhorn then tries to place the replicas on node B following the best effort policy. But since node B does not have enough storage capacity that

attempts fails. So instead the replica is create on node A and SCS1 use it remotely. Uh this is the key limitations. Coupuler can make decision based on compute side constraint but it does not fully understand sense storage side capacity constraint. As a result, the parts may be distributed, but the replicas may still be concentrated and that can undermine the availability we wanted from the stateful set design.

What we really want is for node A to be excluded by anti-affffinity, node B to be excluded by insufficient storage capacity and for SCS1 to be placed on node C where both requirements are satisfied. So in the ideal case, node B should have been filtered out before the pod was scheduled and that is exactly the role of CSI storage capacity. Um it allows coupuler to consider storage

capacity during the filter phase. To support this, Kubernetes provide a resource called CSI storage capacity to track storage availability by topology such as node or zone. This was introduced by KP472 so that coupe scheduler can be aware can become aware of storage capacity during part scheduling for a given storage class. CSI storage capacity reports available capacity for a specific topology scope. Storage class name indicates which storage

class the record belongs to and node topology indicates where the capacity applies. For example, node or zone and capacity is the available storage reported by the CSI driver. Let me walk through how this work in order of Kublet external provisioner and coupuler. First Kublet talks to the CSI driver on each node. It send node get info request to the CSI node server to learn information about that

node including the topology information exposed by the driver. Once kublet get that response it creates or updates a CSI node object in the API server. In other words, this is how Kubernetes record which CSI driver is available on each node. Next, the external provisioner comes into the pictures. Uh, it starts by reading the CSI node object from the API server because it needs to know which topology

segment exists and where capacity information should be reported. Then it send a get capacity request to the CSI controller server. I will come back to the get get capacity implementations on the next slide. Uh so at that point the storage provider such as Longhorn returns the available capacity for a given topology and Using that result the external provisioner then creates or updates CSI storage capacity object in

Kubernetes. These object are basically the published storage capacity information that Kubernetes can later use during scheduling. And finally coupuler read those CSI storage capacity object. When it evaluates candidate node for a part that use persistent storage it can check whether the required storage capacity is available in that topology. So the key idea is that Kubernetes can make a scheduling decision with storage capacity in mind instead of

placing the pod first and only finding out later that the storage cannot actually be provisioned there. So overall the flow is kublet publishes CSI node information external provisioner publish CSI storage capacity information and coupuler use that information to filter node during Now let's take a closer look at get capacity. Longhorn only needs to implement get capacity for CSI search capacity. Inside get capacity. It first validate whether

the node can accept new scheduling. If the node is not ready, not schedulable under evictions or does not match the selectors, it returns zero. Then it checked each disk on that node. Only healthy and schedulable disk are considered for each eligible disk. Lon long calculates schedule schedulable and importantly it returns the maximum among the disk not the sum. Uh this is intentional because a single PBC cannot

be cannot be split across multiple disk. So using maximum value prevents Kubernetes from overestimating capacity and scheduling a PBC that is larger than any single disc can actually support. Next is next this show how path scheduling works with GSI storage capacity. CSIS stands for CSI storage Here coupuler is aware of the available storage capacity of each node through CSI storage capacity object. In this example it is

100 on node A, 20 on node B and 100 100 GB on node C. So when two state preset part SCS0 and SCS1 each requesting a 40 GB PBC are scheduled, node B is filtered out first because it does not have enough capacity. As a result, the pods are scheduled onto node A and node C just we intended. Then after the volumes are provisioned, the remaining capacity

is updated and published again through CSI storage In other word, CSI storage capacity allows coupuler to make scheduling before placement happen. When we first prepared this presentations, our original plan was to end the scheduling story here and then move to another issue we worked on excessive node resource consumption during replica relocations and how we address it. Uh however while working on CSI storage capacity, we came across

more fundamental limitation. We thought this was actually more important to share because it shows the limitation of relying on CSI storage capacity alone for storage aware scheduling. So instead of ending the scheduling section here, I'd like to briefly explain that limitation and share how we currently addressing it. Now let me show you the limitation we found. CSI storage capacity is considered when Kubernetes schedule a pod for

the first time before the borium is provisioned and bound. But after the PBC and PE are already bound later part rescheduling does not go through the same storage capacity check anymore. In other words, CSI search capacity is helpful for the initial placement but not for later rescheduling events. This may be acceptable for many local storage scenario where storage does not need to move once it is provisioned.

But our case is different. In our case, this becomes a problem because we want replica placement to follow pot placement. So when a pot moved to another node, the destination node may also need enough storage capacity for replica relocation. But Kubernetes may reschedu the pod without rechecking CSI storage capacity simply because the volume is already For example, in a node range scenario, the pod is moved to

another node, but that relocation happens without rechecking storage capacity. And in a part restart scenario, coupul select a node again from all eligible nodes without considering whether the current replica already exist. As a result, a pod may lands on a node without a local replica which can So this is the core limitation. CSI storage capacity helps with initial provisioning, but it does not fully address storage aware

scheduling after the volume is already bound. In particular, for rescheduling events such as pod restart, it is not enough to consider only available capacity. The scheduler also needs to understand where the existing replicas are so that it can prefer a node that already has a data. In other words, we what we needed was not just capacity aware scheduling but longhorn aware or more broadly storage So at

this point we realized that storage capacity alone was not enough. P scheduling also needed to consider lowman state especially where replicas already exist. To address that we choose theuler framework. This approach let us bring lman state directly into the pot scheduling process. And because the framework provide multiple extension points, we can intervene at several stage of scheduling including filters, score, reserve and bind. Of course, there are

trade-off. We would need to version specific scheduler builds to support many production cluster running different kubernetes version. It also increase operational overhead because we would need to sa safely operate and validate not only longhorn itself but also a customuler that directly affect part But it return we get much more direct control over scheduling behavior and that became the foundation foundation for the solution I will explain next.

This slide show the core idea of scheduling policy. First we check whether the pod volumes already has the ex existing replica. If it does, we try the replica node first. If that node is not scheduulable, we fall back to other nodes. in the forback case or if this is the first placement and no replica exists yet we filter out node with insufficient storage capacity. So the policy

is simple prefer existing replica locality when possible and otherwise use storage capacity as a forback condition. Right now this is still a work in progress and we are testing this customuler in our internal environment. Once the service is stable, we'd like we we we also like to share more detailed implementation work including how we design and build the scheduleuler. With that, let me move to the conclusion.

To wrap up, I'd like to leave you with two points. The first is key design consideration for hypercon converged storage. From our experience, it is important to consider not only storage performance and capacity but also how storage and workload are placed together in cluster. The second is storage aware scheduling for hypercon converged environment. Through this work, we learned that capacity alone is not always enough. Scheduling may

also need to consider actual storage state such as where What we what we wanted to share today is not that there is only one right answer but that these are important point to consider when designing hyper converged I'd also like to sincerely thanks my teammates and neighbor who work together to build this system and the long one teams for their thoughtful review and support. Thank you and

I will hand over to Jangzon. we don't have time that's all time we we have already used so uh we will lab here uh so if you have any questions please feel free there's our email address so uh reach out by email and then we around here after session so so feel free to come and talk to us. Okay, thank you. Bye.