Hey there and welcome back to the channel. I hope you're having an excellent day. In today's video, I'm going to show you how to expand the capabilities of a Kubernetes cluster by adding shared storage.
For the demo, I'll be upgrading my home cluster by connecting a Synology NAS as an NFS server that can serve read-write many persistent volumes. But before I get into the demo, it's probably a good idea to explain why you'd want to do this and what are the upsides and the downsides of sticking with the default storage that comes with a Kubernetes cluster. In the last video I released, we set up a two-node Kubernetes cluster using the two Mac Minis you see on the screen. But no matter if this is just a small home setup or a full-scale enterprise compute cluster, Kubernetes running on nodes such as these are really designed to provide only compute resources. And when I say compute resources, what I mean is essentially the logic needed to do the computational task at hand.
Whether that be a web server or a database or background process or whatever you can dream up. It's really only designed to handle crunching the numbers and not a whole lot else. Since Kubernetes is running Docker containers, compute resources on a Kubernetes cluster are designed to be started, stopped, and kind of shuffled around at will. For you, this means that if your compute workload needs to store things to a disk for anything other than temporary usage, it'll need some place else to store that data. On Kubernetes, that means declaring persistent volumes in the definition of your app.
So when your workload starts up, It looks to mount a disk of some sort for storing things. Then, if the workload restarts or gets moved around, it can just reattach to the same storage volume and continue right where it left off. The issue, though, that we start to run into immediately is the location and the availability of those requested storage volumes.
Just like I mentioned a minute ago, if you have a single node Kubernetes cluster running, you can just create your volume on the same machine that's running Kubernetes. This is called a host path volume. In the case of a single node cluster, everything is pretty simple.
You know that if a workload restarts, it's guaranteed to start on your single node. When it starts, it'll be able to find the volume because it's on the same exact physical machine. When you move to two or more nodes, you almost immediately have issues with these simple host-based volumes. Let's use my little cluster here for an example. Let's just say that I have a workload running on node 1, and it has a host path volume attached.
At this point, everything works just fine. But... What if one node becomes overwhelmed and this workload needs to be moved to node number two? Now, all of a sudden, you have a workload on node two that is looking for a volume that only exists on node one. In this case, it won't be able to start up because the volume isn't available.
A similar but slightly different problem occurs if we need to have two instances of the workload running. In the case of scaling workload, Kubernetes will try to start one instance of the workload on each node and this actually makes two different problems altogether. First off, you have the same unavailable volume issue as the first example.
But even if you did somehow make this disk available to both workloads, the second issue is that the hostpath volume, or what's called read-write-once. In Kubernetes lingo, this means that while there are two workloads can mount this volume, only one of them has permission to write to it. So that's a different problem altogether.
Luckily, there's a pretty easy way to solve all of this. It's called the Kubernetes NFS provisioner, or NFS container storage interface. NFS servers have been around for forever and have provided a simple way for multiple workloads to connect to one disk. It even allows both workloads to read and write simultaneously. The NFS CSI allows a multi-node Kubernetes cluster to create and mount volumes that are backed by NFS.
Installation and configuration of the CSI is a one-time thing, and after that, it's completely transparent to the user of the cluster. So let's get into the steps, and I'll show you how to set up a Kubernetes cluster to use NFS for provisioning volumes. The very first thing we need to do is actually have an NFS server in the first place. There's a bunch of different ways to do this, and I'm going to show you how to do two of them.
The first is by adding NFS to one of the Kubernetes nodes. While this method will solve the problems and all the issues I listed earlier, it's not really the preferred way of going about doing it. Since it's installing NFS on the node, it's not really separating concerns between compute and storage. For example, if the node with NFS went down, the other node would obviously not be able to mount the volumes it needs because everything's on the one node. Installing NFS on one of your nodes is actually pretty easy.
All the instructions are on the MicroK8's website, and all you really need to do is SSH into the server and then just copy and paste the commands. Once it's done, you're all set. The second method, which is the more preferred and technically correct way to do it, is to have a completely different device for running NFS. In my case, I've had this Synology DS120 for a couple years, and I've historically used it as a target for time machine backups and things.
It has plenty of space available, so I'm going to go ahead and just... enable NFS on it. Very quickly, enabling NFS on the Synology NAS is pretty straightforward.
If you already have one or you plan on buying one, here's how to do it. If not, feel free to skip to the next After and save yourself 30 seconds. First off, log into the NAS manager.
Next, create a shared folder on an available disk. Next, open the properties on the folder and set up NFS permissions. Since this is a home server with no outside access, I've made the permissions super simple. It essentially allows any IP in my home to connect and it squashes all the users down to root. Next, back from the control panel, open up file services, go to the NFS tab and check the box to enable NFS.
Save all that and you're done. Okay, now that we have an NFS server available, we need to install the NFS CSI driver on the Kubernetes cluster. One thing to note here is that Kubernetes supports a bunch of different storage interfaces. In our case, we're going to install the NFS one today.
But I'll put a link in the description if you're at all interested in figuring out all the other supported ones. Back in the MicroK8 site, go ahead and click down to step two. As you can see, this kind of gives instructions on how to install this using a Helm chart. Even something like this is just another workload installed on the Kubernetes cluster, which makes it all...
super relatable. SSH into your cluster and run the four commands to install the NFS CSI driver and wait for it to finish. Once it's done, you're all set with any of this installation stuff.
Okay, so now that we have this installed, let's go over how to use it. The first thing we need to do is make what's called a storage class. In Kubernetes, these are essentially a set of instructions for how the system will go about provisioning a volume when you ask for one.
In our case, we're going to create a storage class called NFS CSI, and just FYI, this is just a name I made up. You can call it whatever you want. On the very next line down, you'll see the provisioner.
This is where you're telling Kubernetes that anytime you request a volume using this storage class, it should attempt to get the volume from the NFS provisioner that we just installed earlier. Finally, the storage class parameters tell Kubernetes how to provision the volume you requested. In our case, we provide the NFS server's IP address and the name of the share. All this information should be available to you from the previous steps that we just went through.
The rest of the stuff you can set exactly as I have them on the screen. But one item you might want to change is the reclaim policy. If you set this to retain instead of delete, Kubernetes will never delete the underlying data in the NFS server when you delete the volume on the cluster.
If you have critical data on your cluster, that might be a really good idea. Or you could make a second storage class, one that retains and one that deletes, then you can choose based on your needs at the time. So let's just recap what we did here because it's super important.
The storage class object is essentially a set of instructions for Kubernetes. It tells the system that you have a class of storage called NFS CSI, and when a user of the cluster requests a volume by this storage class name, it should send the request to the NFS CSI provisioner, and along with the IP address and the path, and the NFS CSI will then reach out to the NFS server and create a space on the server for the storage you requested. Behind the scenes, it's just a folder on the NFS server, but to the workload, it looks like a new shared persistent. volume.
So now save that file and install it on your cluster using kubectl. Use the command kubectl apply dash f and then the file name of whatever you just named this file. Now finally we're ready to go ahead and actually use this. Just to test it out go ahead and make a persistent volume claim. Under the specs section set the access modes to read write many and the storage class name to nfs csi or whatever you named your storage class earlier.
Like I mentioned just a second ago by putting the storage class name on the persistent volume. claim, you're telling Kubernetes to go and provision this claim from the NFS CSI with the instructions you put in the storage class object. Install the persistent volume claim the same way you did with the storage class by using kubectl apply dash f and then the file name of whatever you just named that file. After a few minutes, Kubernetes should show that the PVC is bound. That means it worked.
Now if you want to use this shared PVC as volume in your deployment, just add it to your deployment in the volume section. Specify the type of volume as persistent volume claim and then provide the claim name just like you would any other. Okay, now you have a pretty close replica to a Kubernetes cluster you would see in a production environment.
Storage is separated from compute and all is right in the world. Hey, if you liked this video, I am really glad you liked it. The like and subscribe buttons are right down there if you feel so inclined. And if you haven't watched the first part of this video where I set up this cluster in the first place, the video will be up here in the corner. I truly appreciate you watching this video to the end.
Thank you, and until next time, happy coding.