Cloud Native Theater | Data on Kubernetes Day: From PVC to Mount Point: Dissecting a... Vivek Singh

In this talk, Vivek, a software engineer at MongoDB, explores the dynamic volume provisioning process in Kubernetes, particularly focusing on persistent volume claims (PVCs). He explains how the PVC API facilitates the automatic creation of storage volumes for stateful workloads. Vivek highlights the workflow from PVC creation to volume attachment, emphasizing the importance of understanding these mechanisms for effective management of stateful applications. He details the roles of the persistent volume controller and the Container Storage Interface (CSI) plugins. The session concludes with an overview of the processes involved in volume creation and attachment, showcasing practical implementations and best practices for debugging issues related to dynamic volume provisioning.

All right. Hey, hello everyone. My name is Vivek. I am a software engineer at MongoDB and in this particular session, we are going to look into how exactly dynamic volume provisioning happens in case of Kubernetes. So, in case of Kubernetes, when you try to let's say if you have a use case where where you want to run a stateful workload workload on top of Kubernetes, you would

want to have a volume attached to that workload and in that case Kubernetes gives us a powerful API. It's It's PVC API and the problem with PVC API is as soon as you create a PVC resource, a volume magically appears and gets attached to your Kubernetes node that can later be attached to the pod so that your pod can consume that volume. And that magic word that

I used, that's the problem in in today's world, I would say. So, I was I was talking to some some guys day before yesterday while dinner and it looks like a lot of platform teams are still not very confident in running their stateful workloads on Kubernetes. So, that's exactly what this session is going to focus on. This particular session In talk about the entire workflow, entire journey

of a of a PVC resource from when you when you create the PVC resource, how exactly it's provisioned, how exactly it's attached to the to the node and then later to your pod. And understanding that behind-the-scene mechanism is at least according to me very very crucial in running your stateful workloads on on Kubernetes. If you don't really understand how exactly that storage layer works, you will not

really be confident in your running stateful stateful applications on on top of Kubernetes. And after after this session, I think we are going to going to go away with the kind of enough understanding about about the dynamic volume provisioning so that if something goes wrong, we would be able to let's say debug that problem. We would be able to understand the issues. So, like I said, in

case of Kubernetes, we we have a very nice API very nice resource called PVC persistent volume claim. And as soon as you create a persistent volume claim resource with a specific requirement, let's say you want maybe 50 GB of of a volume. In that case, you create the PVC resource. And as soon as the PVC resource is created, the persistent volume controller that is shipped with Kubernetes

controller manager, that looks into that PVC resource and then it basically figures out if there are there are any already available volumes in the cluster that satisfy the requirements that are that are configured in the PVC And if there aren't any any already available volumes that satisfy the resource requirements that are configured in the PVC resource, in that case, PV controller actually looks into the provisioner for

that particular PVC request. So, we have a PVC. PVC refers to a storage class resource where the provisioner is actually specified. And the provisioner name for a for a particular volume can either have kubernetes.io in in the name or not. It may or may not have kubernetes.io in the in the provisioner name. And if the provisioner name has kubernetes.io, PV controller gets to the conclusion that this

particular PVC can be provisioned using entry provisioner. Entry provisioner is something that I'm going to maybe expand a little on in just a bit. But, we are not going to focus mainly on entry provisioners. They are kind of deprecated. And the reason is they were being shipped with the Kubernetes code base itself. It was not scalable. There were There were quite a few problems because of that

entry provisioners are actually deprecated. So, coming back to PV controller, PV controller looks into the provisioner for the PVC. And if it doesn't start with kubernetes.io, PV persistent persistent volume controller gets to the basically conclusion, determines that this particular PVC needs to be provisioned by an external provisioner. And I'm going to use this term external provisioner and CSI plugin kind of interchangeably, but we are going to

get into the definition of CSI plugin in just a bit. So, so PV controller looked into the PV series And based on the name of the provisioner, it figured out whether it should be provisioned using entry provisioner or or external provisioner. Now, that it figured out that the PVC should be provisioned using a CSI plugin, an external provisioner, all it does is it annotates that PVC with

a specific annotation that is that is mentioned here, storage hyphen And if I if I try to if I speak a little loosely here, that's pretty much it that Kubernetes does when it comes to provisioning volume for your PVC resource. That's pretty much it. And once the once the PV controller has annotated the PVC resource with this annotation called called storage provisioner, then the PVC resource is

kind of just lying there in your Kubernetes cluster. If we don't have any other resource, any other not resource, but any other component, any other application deployed to take care of that PVC resource, nothing would really happen. And that is that is where this this concept of CSI spec and CSI plugin come into picture. Like I said, uh the PVC was annotated, there should be an application,

a program, a component installed in your Kubernetes cluster. That component should be able to look into that PVC with a specific annotation called storage provisioner. And if the PVC has that annotation, in that case, this component should be able to talk to the storage provider to provision the volumes and and attach that volume to the node so that it can later be made available to your to

your pod. And this component that I'm talking about, this is the component that we call CSI plugin. And to to implement So, if the CSI plugin is going to talk to your Kubernetes API server to to let's say, uh look into the PVC resources, and then it's also going to talk to the to the storage provider APIs, this CSI plugin needs to kind of adhere to some

rules, I would say. It it it needs to follow some some rules, and those rules are documented in the CSI spec. So, this is what exactly CSI spec and CSI plugin are. CSI spec is just, let's say, a rule book that has a set of rules that can be followed by by storage providers to write CSI plugins. I I hope this at least definition uh makes sense.

And the other point that I would like to highlight is even though we have the we even though we have the CSI plugin, Kubernetes doesn't really talk to CSI plugin. Like I said, Kubernete- Kubernetes looks into the PVC resource, and Kubernetes, uh after looking into the provisioner name, it just annotated the PVC resource. It doesn't really talk to the CSI plugin. It's CSI plugin's responsibility to look

into the cluster, look for the PVC resources that it can that it can provision. And so, the the CSI spec that I talked about, uh like I said, it's a rulebook. It mentions, it outlines some of the things that CSI plugin should be followed so that they can be called a CSI compliant CSI plugin, let's say. And those rules are, for example, if you talk about any

workflow, let's say snapshotting a volume. In that case, CSI plugin is actually going to specify that to snapshot a volume, these are the functions, these are the RPCs that a particular CSI plugin should implement. Out of those RPCs, uh which particular RPCs are going to be item potent, which particular RPCs are going to be not, all of these things, how exactly the error handling is going to

work for those RPCs, all of those things are actually mentioned in this in this CSI specification. Uh the the other thing that is mentioned, obviously, is how exactly the communication is going to happen between the between the Kubernetes and the CSI plugin. It doesn't talk about how exactly this CSI plugin is going to going to be packaged or anything like that. So, yeah, I mean, there are

some things that are talked about in CSI spec and and that are that are left out intentionally. Now, let's actually go ahead and and really dig deep into into one of the volume-related workflows, and that is creating a volume. You created a PVC resource. How exactly CSI plugin knows about how exactly Yeah, how exactly PVC plugin knows about that PVC, sorry, CSI plugin knows about that PVC,

and how exactly it gets to work to to provision the volume and then everything like that. If you look into the diagram on the right side this is exactly This is is copy-pasted from the CSI plugin. And if you see here, it specifies that if you create a create a PVC resource to get that to get that PVC available on the node so that pod can consume

that volume, these are all the steps that are involved. First one is create volume. It creates the volume, controller publish volume, and then we have node stage volume, and things like that. We are going to actually look into each of these in in just a second. So, now that we understand what exactly CSI plugin is, and we understand there is a rule book that has a set

of, let's very loosely, a set of a set of RPCs that should be defined by CSI plugin. All those RPCs are, let's say, grouped together in in three services, and those three services are identity service, controller service, and node service. So, all these services are implemented by two components. So, CSI plugin itself is, let's say, built using two components, controller plugin and node plugin. Node plugin needs

to get installed into every node of your Kubernetes cluster, and we are going to look into why exactly that is the case, why why exactly that is required. And controller plugin can actually run anywhere on on your cluster, basically. Yeah, even even yeah, any any node anywhere. So, yeah, CSI plugin has two components, controller plugin, node plugin, and these two plugins collectively all the all the RPCs

that are defined in the in services that we just talked about. Now, let's yeah, this is let's look into this diagram. So, we have a PVC resource with external provisional annotation, and we have controller plugin and node plugin deployed. We have storage provider, but nothing is no one is actually doing anything because, like I said, our plugin uh implements the create volume RPC, but it doesn't really

look into the PVC resource. So, what happens is we deploy the controller plugin with a sidecar called CSI provisioner sidecar. And CSI provisioner sidecar looks into the PVC resource and its annotation and it gets to know that, "Okay, this annotation has my provisioner name. So, I am going to work on this particular PVC." And then, basically, we have right now in this diagram two containers. One is

controller plugin, the other one is CSI provisioner. And controller plugin has all the all the kind of uh gRPC uh functions services running on a on a Linux socket. And that socket is shared to the to the CSI provisioner using using a shared empty dir volume. So, that CSI provisioner can actually call the respective RPC from the con- controller plugin. So, in this diagram, CSI provisioner looks

into the PVC resource. It calls the create volume It calls the create volume RPC. And I'm going to show you how exactly create volume RPC uh is implemented very briefly. I'm going to show you uh the the code for it. So, if you see here, I'm not sure is it Is it visible to you all? Okay. So, if you see here, uh what's happening is we are

Yeah, I mean, you can do all the validations here. Create volume. So, this is the RPC that gets called from your from your sidecar. You have CSI provisioner sidecar that looked into the PVC resource and it called create volume RPC for that particular PVC. And if you see we are validating we can validate everything here. For is the volume mode that is that is requested even if

we support that volume mode or not. And eventually, if you see we are just calling create volume for the for the storage provider. And this particular plugin, this is a test plugin that I wrote for this talk. This is for Digital Ocean, and it actually goes ahead and creates a volume in the in the in the Digital Ocean. so this is this is basically pretty much that

create volume RPC does. So now now that create volume RPC was called create volume called the storage provider to actually provision the volume. Now we have a volume. So we have a volume in in storage provider, and CSI provisioner sidecar if you if you look into that arrow, that actually goes ahead and creates a PV resource now. And that PV resource just denotes that there is a

respective volume created in the in the storage provider. Again, that's pretty much it that that Kubernetes uh not even Kubernetes. That's pretty much it that that provisioner side external provisioner sidecar does. And once we have the PV resource that denotes the storage that denotes the volume that was provisioned in in in storage provider, what happens is like we have provisioner sidecar, we have another sidecar called attacher

sidecar. And this attacher sidecar looks into looks into the volume attachment resources. But even before volume attachment resource got created, I think I think we should discuss who created that that volume attachment resource. So if you go to the last slide, uh the PV resource got created, and then scheduler actually schedules the workload on a particular node. And once that happens, a volume attachment resource is actually

created by by the attached detached controller. When again, if I try to rephrase it, when the workload got scheduled to a particular node, when spec.nodeName was set for a for a for a particular workload, attached detached controller created the volume attachment resource. And after the volume attachment resource was created, CSI attacher sidecar got to know about because CSI attacher sidecar looks for the volume attachment resource, and

it goes ahead and it calls the controller publish volume for the controller plugin again. And controller publish volume, again if you go back to this diagram here, after create volume, controller publish volume is called. So, controller publish volume RPC was called, and this particular RPC actually attaches the volume that was created to a particular node. And volume attachment resource is very helpful here because volume attachment resource

is the resource that kind of associates a particular volume with a particular node ID. And controller publish volume just goes ahead and attaches that volume to a So, once the attachment is completed, the the attached field of the volume attachment resource is set to true. And as soon as that happens, kubelet actually gets to action, and kubelet kubelet tries to talk to talk to the node plugin,

and then it tries to mount that attached device file to a particular mount path so that pod can uh pod can actually use that use that file system. Uh and this is the reason I would say this is the reason when initially we started we discussed that node plugin needs to be installed on all the nodes. This is the reason because once the volume is attached to

a particular node, node plugin is the kind of component that is responsible to mount that mount that volume to staging location and then to to the actual target location. And then from this target location, this particular volume can be mounted to a particular container, a particular pod using using container runtime interface CRI implementations, I would say. So, this the entire kind of goal of the CSI plugin

ends when the volume is available on the on the on the target location. Uh yeah, the key takeaways are directly to the to the to the CSI plugin. CSI plugin reaches out components of CSI plugin reaches out to the to the API server to know about PVCs, volume attachments, and things Uh every step in that in that life cycle volume life cycle diagram kind of is is

yeah, related to a particular component. And if something fails, you can instead of kind of guessing, you can go ahead and look into the logs of of that particular component to figure out what's happening. And yeah, I think this is pretty much it, I would say. I have some links here if you want to go through and and understand things better. Uh this is this QR code

is for for my GitHub. I would love to take some questions if you have any questions. Uh yeah.