Kubernetes

I've been migrating from having lots of VMs doing one or many functions (like a VM for home assistant and node red, a VM as a reverse proxy, a VM to host Unifi controller in etc) to a docker based infrastructure. Moved to docker compose for a few things a while ago and loved it. But I always wanted to take the next step, have a more resilient system for hosting docker images, that can recover and move the workload to another machine if a machine becomes unavailable.

Well, I knew that's what Kubernetes and Docker Swarm did. I watched some videos on Pluralsight about both, and asked Nigel Poulton which I should play with, and bit the bullet and a few months ago decided to learn Kubernetes, and implement it at home. I'm so glad I did.

I initially had a very rocky road getting things working though, but eventually figured it out and over the past few months have become very familiar with it, and have created Ansible scripts to enable easy management of the cluster. Now I run all my home applications on it, like Home Assistant, Emby, Nodered etc, is it overkill? Hell yes! But where's the fun in not going overkill?

Kubernetes is a system to manage containers. You form a Kubernetes cluster, consisting of Nodes, which are either physical or virtual computers (in my case a mixture of both, with some powerful x86 PCs, and even some ARM Raspberry Pis in the cluster, you can mix and match) and then containers, or in the Kubernetes world, Pods, run inside the cluster. The Pods can be scheduled to run on any of the nodes, depending on the current state of the cluster. You can also do High Availability if the application you are running inside the pod supports it, so you can have replicas on various nodes, load balancing traffic and keeping the application running even if the other pod dies for some reason. A feature I really like about Kubernetes is that the tool to manage it, kubectl, runs on all operating systems and can talk to the cluster and connect to individual pods to look inside them, or read their output. Kubernetes also has a REST API which you can use to manipulate the cluster, which I also use (I might do a post about this in the future).

In my case, none of the applications I run support high availability, obviously the application has to be able to work with a replica, and most of the applications I run aren't designed to work like this. That being said, Kubernetes is still pretty nice, because if a Node goes down, it will then spin up that Pod on another Node, assuming it has capacity for it, so the application will be up and running in a couple of minutes. It would be much nicer to have a proper high availability configuration though, but the applications I'm running are really designed for the home user, and not an enterprise level solution like Kubernetes :)

Then you set up an ingress, this is basically a reverse proxy that allows one endpoint to route traffic to pods within the cluster using host routing, so you can have two domains pointed at your ingress going to different services. The ingress also usually handles terminating the SSL tunnel.

The one big flaw though is that out of the box Kubernetes is really aimed at cloud providers, not so much bare metal self hosting. For example, the only way, out of the box to get traffic into that ingress pod above is via Hostport, which is a port that all the nodes will use to forward traffic to a pod. This is fine for when you're getting started, but it isn't the best. Which node do you direct the traffic to? If that node is offline, your ingress is then offline as your traffic is going to a dead server, even when the pod itself may well be fine.

Cloud providers have the concept of a load balancer service, where Kubernetes gives an IP to a pod, then can dynamically direct traffic to that pod/pods.

The way to do this on bare metal is to use MetalLB which enables this behaviour. Using BGP routing, MetalLB communicates with my router, pfsense, and tells it which IPs it's using, and how to direct that traffic to the nodes in the cluster, so now we can have virtual IPs on the network that can be connected to pods, as if they were virtual machines. Brilliant. This means now that I can give the ingress a load balanced IP, and don't have to worry about a node going down breaking the path to the pods.

All the pods mount NFS and iSCSI shares on my TrueNAS box for persistence that works amongst the nodes, so no matter where the pods end up being scheduled, they can access their config.

This all works perfectly. It took a bit of time to set up, but now that I have, my system is as stable as ever. I've deleted tons of VMs and saved a lot of space, both from the HDD itself and the backups, and I've got the entire Kubernetes configuration in Git and Ansible, so if my cluster was to die, I could get it up and running again within an hour as the files in Git are declarative, so they will just give me the same state after I've run them.

I don't back up my nodes, because the cluster and the nodes within it, should be treated as cattle, not pets. If the cluster goes south, I just reset the nodes and create a new cluster. Nothing is stored on the nodes, so there's nothing worthwhile to back up.

In summary it is a great way to run containers, and this, along with serverless is in my opinion the future of hosting applications. This is why I love running a home lab, I can run real world applications that are actually in use by my family, but use whatever technology I want, to learn about it and actually play with it in a semi-production environment :)