Problem

As some of you may know I’ve been running my in-home on premises server for the last 6 years. Originally that was accomplished through a single-node micro k8s kubernetes cluster. However due to increased workloads I had to expand to a multi node kubernetes cluster. It became clear to me that my existing storage solution microk8s-hostpath would not be sufficient since true production storage had the following requirements.

Storage requirements

In order of importance:

Resilient to node failure: Given the need to perform regular security and operating system level node upgrades as well as hardware failures, the data had to be resilient to the temporary or complete catastrophic failure of a machine.
Diverse workload support: I run a plethora of different applications each with their own unique storage needs. The storage solution had to be versatile, providing support for block file and object storage
Ease of management: To maintain a lean operation and minimise the risk of human error, I needed a solution that was easy to install, configure, and manage. Moreover, an ideal storage solution would offer automatic healing of damaged data nodes and provide a straightforward path to disaster recovery.

Evaluation

OpenEBS Mayastor - https://mayastor.gitbook.io/introduction/

Pros

Mayastor caught my eye with its high performance and replication features for data protection, which are critical in a production environment.

Cons

However, its limited support for diverse applications and excessive manual configuration needs were major drawbacks. Additionally, the solution's immaturity and specific huge_pages requirement, which caused compatibility issues with PostgreSQL, made it less appealing.

Longhorn - https://longhorn.io/

Pros

Longhorn stood out due to its ease of management and handy features such as volume snapshots, backup, and restore, which are key to maintaining data integrity.

Cons

Nevertheless, Longhorn's limitation of only supporting read-write operations on a single node at a time posed a significant challenge to my requirement of resilience to node failure.

Rook-Ceph - https://rook.io/

Pros

Rook-Ceph impressed me with its comprehensive support for diverse applications, scalability, and resilience. Its maturity and the backing of a robust community and wide adoption provided further confidence.

Cons

One hiccup I encountered with Rook-Ceph was its requirement for full drives for provisioning, which presented challenges with my partitioned drives.

Conclusion

After careful evaluation, Rook-Ceph emerged as the optimal choice for my multi-node on-premises MicroK8s cluster.