Prerequisites

Fork this repository

Because this project applies GitOps practices, it’s the source of truth for my homelab, so you’ll need to fork it to make it yours:

Fork khuedoan/homelab

By using this project you agree to the license.

Important

License TL;DR

• This project is free to use for any purpose, but it comes with no warranty
• You must use the same GPLv3 license in docs/reference/license.md
• You must keep the copy right notice and/or include an acknowledgement
• Your project must remain open-source

Hardware requirements

Initial controller

Note

The initial controller is the machine used to bootstrap the cluster, we only need it once, you can use your laptop or desktop

• A Linux machine with Nix and sudo access (for privileged ports when PXE booting with Nixie).
• An SSH agent, preferably with a TPM-backed key to keep the private key in hardware.
• Add your public SSH key to infra/src/modules/metal/flake/authorized_keys.

Servers

Any modern x86_64 computer(s) should work, you can use old PCs, laptops or servers.

Note

This is the requirements for each node

Additional capabilities:

• Ability to boot from the network (PXE boot)
• Wake-on-LAN capability, used to wake the machines up automatically without physically touching the power button

Network setup

• All servers must be connected to the same wired network with the initial controller
• Access to your domain’s DNS settings in Cloudflare

External resources

A domain managed by Cloudflare is required.

Although I try to keep the amount of external resources to the minimum, there’s still need for a few of them. Below is a list of external resources and why we need them (also see some alternatives below).

Create credentials

Cloudflare

• Add your domain to Cloudflare if you haven’t already
• Log in from the development shell:

cf auth login

Grant the following permissions:

• Account API Tokens Write
• R2 bucket and object access
• DNS
• Tunnel

If your existing login lacks these permissions, run cf auth login --force and approve them.

Alternatives

To avoid vendor lock-in, each external provider must have an equivalent alternative that is easy to replace:

• Cloudflare R2:
  • Update the backend and bootstrap hook in infra/root.hcl for another state backend
• Cloudflare Tunnel:
  • Use port forwarding if it’s available
  • Create a small VPS in the cloud and use WireGuard to route traffic via it
  • Access everything via VPN
  • See also awesome tunneling

BIOS setup

Note

You need to do it once per machine if the default config is not sufficient, usually for consumer hardware this can not be automated (it requires something like IPMI to automate).

Common settings:

• Enable Wake-on-LAN (WoL) and network boot
• Use UEFI mode and disable CSM (legacy) mode
• Disable secure boot

Boot order options (select one, each has their pros and cons):

1. Only boot from the network if no operating system found: works on most hardware but you need to manually wipe your hard drive or delete the existing boot record for the current OS
2. Prefer booting from the network if turned on via WoL: more convenience but your BIOS must support it, and you must test it thoroughly to ensure you don’t accidentally wipe your servers

Note

Example

Below is my BIOS setup for reference. Your motherboard may have a different name for the options, so you’ll need to adapt it to your hardware.

Devices:
  NetworkSetup:
    PXEIPv4: true
    PXEIPv6: false
Advanced:
  CPUSetup:
    VT-d: true
Power:
  AutomaticPowerOn:
    WoL: Automatic  # Use network boot if Wake-on-LAN
Security:
  SecureBoot: false
Startup:
  CSM: false

Gather information

• MAC address for each machine
• OS disk name (for example /dev/sda)
• Choose a static IP address for each machine (just the desired address, we don’t set anything up yet)
• Choose an unused IP range for LoadBalancer Services (e.g. 192.168.1.224/27) and a control-plane VIP (e.g. 192.168.1.100)
• Subnet prefix length and gateway