Add or remove nodes

Or how to scale horizontally. To replace the same node with a clean OS, remove it and add it again.

Add new nodes

Tip

You can add multiple nodes at the same time

Add its details to the inventory:

 {
  "homelab-1": {"ip": "192.168.1.110", "mac_address": "02:00:00:00:00:01", "disk": "/dev/sda"},
  "homelab-2": {"ip": "192.168.1.111", "mac_address": "02:00:00:00:00:02", "disk": "/dev/sda"},
  "homelab-3": {"ip": "192.168.1.112", "mac_address": "02:00:00:00:00:03", "disk": "/dev/sda"},
+ "homelab-4": {"ip": "192.168.1.113", "mac_address": "02:00:00:00:00:04", "disk": "/dev/sda"}
 }

Install the OS and join the cluster:

just deploy production

Then verify the new node:

export KUBECONFIG="$PWD/infra/production/kubeconfig.yaml"
kubectl get nodes

That’s it!

Remove a node

Warning

It is recommended to remove nodes one at a time

Use the environment’s kubeconfig:

export KUBECONFIG="$PWD/infra/production/kubeconfig.yaml"

Drain the node:

kubectl drain ${NODE_NAME} --delete-emptydir-data --ignore-daemonsets --force

Remove the node from the cluster

kubectl delete node ${NODE_NAME}

Shutdown the node:

ssh root@${NODE_IP} poweroff

Remove it from the inventory:

 {
  "homelab-1": {"ip": "192.168.1.110", "mac_address": "02:00:00:00:00:01", "disk": "/dev/sda"},
  "homelab-2": {"ip": "192.168.1.111", "mac_address": "02:00:00:00:00:02", "disk": "/dev/sda"},
  "homelab-3": {"ip": "192.168.1.112", "mac_address": "02:00:00:00:00:03", "disk": "/dev/sda"},
- "homelab-4": {"ip": "192.168.1.113", "mac_address": "02:00:00:00:00:04", "disk": "/dev/sda"}
 }

Reapply the environment with just deploy production. Don’t replace the initializer with a fresh server in an existing cluster.

Optionally, after Ceph finishes recovering, clean up the removed node’s OSD records using the Rook guide.