
In my previous blog, I’ve showed the installation of Kubernetes cluster using Raspberry Pi’s. It was working very well until recently, when one of the SD cards failed. This is due to heavy R/W’s on the disks thanks to etcd and logging.
In this blog, you’ll see how to setup Rancher K3S Kubernetes Cluster with external database MySQL/PostgreSQL
My Setup
- Raspberry Pi 4 8GB X 6
- Raspberry Pi POE HAT X 6
- Ubuntu Server 21.04
- POE Switch (Gigabit preferred) X 1
- External MySQL server running on QNAP TS431K System
- SSD Card Class 10 128 GB (32 GB is enough) X 6
- Total Cores = 24 Total Memory = 48 GB Total Disk = 768 GB
- Persistence Storage NFS Share = 2 TB on QNAP TS431K System
- Management host with Ansible and Python3
To setup the cluster use Pyratlabs Ansible Role for installing and configuring HA Cluster with MySQL. You can use any external host to run the MySQL server on it. I’m using QNAP for many things which comes with MySQL server pre-installed. Running a MySQL on a highly available RAID-1 configured QNAP NAS System is the best choice I’ve got!
https://github.com/PyratLabs/ansible-role-k3s
Two files which needs to be redited are shown as under.
ha.yaml
---
- name: Build a cluster with HA control plane
hosts: k3s_cluster
vars:
k3s_become_for_all: true
k3s_etcd_datastore: true
k3s_use_experimental: true # Note this is required for k3s < v1.19.5+k3s1
k3s_server:
advertise-address: "{{ ansible_eth0.ipv4.address }}"
bind-address: 0.0.0.0
datastore-endpoint: "mysql://USERNAME:PASSWORD@tcp(192.168.15.25:3306)/k3s"
write-kubeconfig-mode: 644
node-external-ip: "{{ ansible_eth0.ipv4.address }}"
# cluster-cidr: 172.20.0.0/16
# flannel-backend: 'none' # This needs to be in quotes
disable:
- traefik
- servicelb
k3s_agent:
node-ip: "{{ ansible_eth0.ipv4.address }}"
node-external-ip: "{{ ansible_eth0.ipv4.address }}"
roles:
- role: xanmanning.k3s
In the above file, database-endpoint is pointing to the MySQL server running on QNAP with IP address 192.168.15.25. Replace Username and Password according to your setup. I’ve disabled traefik and servicelb replacing the latter with nginx and metallb
inventory.yaml
k3s_cluster:
hosts:
kube-0:
ansible_user: ubuntu
ansible_host: 192.168.15.24
ansible_python_interpreter: /usr/bin/python3
k3s_control_node: true
kube-1:
ansible_user: ubuntu
ansible_host: 192.168.15.23
ansible_python_interpreter: /usr/bin/python3
k3s_control_node: true
kube-2:
ansible_user: ubuntu
ansible_host: 192.168.15.22
ansible_python_interpreter: /usr/bin/python3
k3s_control_node: true
kube-3:
ansible_user: ubuntu
ansible_host: 192.168.15.21
ansible_python_interpreter: /usr/bin/python3
kube-4:
ansible_user: ubuntu
ansible_host: 192.168.15.20
ansible_python_interpreter: /usr/bin/python3
kube-5:
ansible_user: ubuntu
ansible_host: 192.168.15.19
ansible_python_interpreter: /usr/bin/python3
First 3 nodes are master nodes while the remaining nodes are worker nodes.
To install the K3S cluster and configure as per above configuration run the following command.
/home/naanu/ansible-role-k3s# ansible-playbook -i inventory.yml ha.yaml

Install kubectl on the management host. I’m using Oh My ZSH https://github.com/ohmyzsh/ohmyzsh with kubectl alias enabled. Below are the things to do after initial cluster configuration.
- Metallb – Load balancer
Manifest
kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.10.2/manifests/namespace.yaml
kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.10.2/manifests/metallb.yaml
- Nginx – Ingress Controller
Steps
https://www.youtube.com/watch?v=UvwtALIb2U8
helm show values ingress-nginx/ingress-nginx > ingress-nginx.yaml
edit hostNetwork = True
hostPort = True
Kind = Deployment
kubectl create ns ingres-nginx
helm install ingress ingress-nginx/ingress-nginx -n ingress-nginx –values ingress-nginx.yaml
kgp -A
- Cert-Manager – Certificate Manager using Letsencrypt
Manifest
kubectl apply -f https://github.com/jetstack/cert-manager/releases/latest/download/cert-manager.yaml
Add Letsencrypt Cluster Issuer
k apply -f letsencrypt.yml
If it throws an error like “Error from server (InternalError): error when creating “letsencrypt.yml”: Internal error occurred: failed calling webhook “webhook.cert-manager.io”: Post “https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s”: context deadline exceeded”
Restart the nodes
ansible -i inventory.yml k3s_cluster -b -u ubuntu -a “systemctl restart k3s”
k apply -f letsencrypt.yml
- Install NFS Client Provisioner and make it default
Clone https://github.com/justmeandopensource/kubernetes
go to kubernetes/yamls/nfs-provisioner
k apply -f rbac.yaml
k apply -f default-sc.yaml
k apply -f deployment.yaml
If deployment throws an error like ErrImgPull change the image of nfs-provisioner in deployment.yaml to
k8s.gcr.io/sig-storage/nfs-subdir-external-provisioner:v4.0.2
- Install Nextcloud
helm repo add nextcloud https://nextcloud.github.io/helm/
helm repo update
k create ns nextcloud
helm install nextcloud nextcloud/nextcloud –namespace nextcloud –values nextcloud.values.yml
k apply -f ingress.yml
If it throws an error, disable Liveness and other probes
** Use below commands for migration **
Copy all the files to the new PVC
Login to the container, install sudo if not available
Run command as user www-data
sudo -u www-data php -d memory_limit=-1 occ files:scan –all –verbose
- Docker Private Registry
helm repo add twuni https://helm.twun.io
helm repo update
k apply -f 4-pvc-nfs.yaml
helm install docker-registry twuni/docker-registry -f config-values.yaml
6.a Registry Credentials
kubectl create secret docker-registry regcred –docker-server=http://registry.vinaybabu.in/v2/ –docker-username=babvin –docker-password=Rundeck21
k patch serviceaccount default -p ‘{“imagePullSecrets”: [{“name”: “regcred”}]}’
- Mariadb
k apply -f mariadb_deploy.yml -f secret.yml -f mariadb_svc_internal.yml
After completing all the above steps, you have a self hosted cluster with load balancer, ingress controller, NFS controller as default storage class for persistent data storage, Nextcloud storage for sharing files, and Docker Private registry.

If you enjoyed this post, I’d be very grateful if you’d help it spread by emailing it to a friend, or sharing it on Twitter or Facebook. Thank you!
What am I missing here? Let me know in the comments and I’ll add it in! OR tweet it to me @babvin
Very nice blog, I love it.
I hope you have thought of how to avoid hardcoding username/password in ha.yaml file.
Absolutely! you are spot on in highlighting security improvement which can be made by using Ansible Vault