Skip to content

Prerequisites

Fork this repository

Because this project applies GitOps practices, it's the source of truth for my homelab, so you'll need to fork it to make it yours:

Fork east4ming/homelab2

By using this project you agree to the license.

License TL;DR

  • This project is free to use for any purpose, but it comes with no warranty
  • You must use the same GPLv3 license in LICENSE.md
  • You must keep the copy right notice and/or include an acknowledgement
  • Your project must remain open-source

Hardware requirements

Router HardWare

Router that support configuration of static routes, because Cilium is performance tuned to use native routing feature, this requires a router that supports static routing. (Of course, you can also modify Cilium's configuration to not use native routing feature.)

(Optional) It would be better if the router supports DHCP/DNS -> PXE/TFTP function, which can easily work with netboot.xyz for PXE based on dhcp proxy. Like below:

OpenWrt -> DHCP/DNS -> PXE/TFTP

Initial controller

Info

The initial controller is the machine used to bootstrap the cluster, we only need it once, you can use your laptop or desktop

  • A Linux machine that can run Docker (because the host networking driver used for PXE boot only supports Linux, you can use a Linux virtual machine with bridged networking if you're on macOS or Windows).

Servers

Any modern x86_64 computer(s) should work, you can use old PCs, laptops or servers.

Info

This is the requirements for each node

Component Minimum Recommended
CPU 2 cores 4 cores
RAM 8 GB 16 GB
Hard drive 128 GB 512 GB (depending on your storage usage, the base installation will not use more than 128GB)
Node count 1 (checkout the single node cluster adjustments tutorial) 3 or more for high availability

Additional capabilities:

  • Ability to boot from the network (PXE boot)
  • In fact, there are alternatives to a PXE boot, which is to automate the installation using a USB as a boot disk (which is what my Cheshi does), such as using the Ventoy Autoinstall Plugin
  • Wake-on-LAN capability, used to wake the machines up automatically without physically touching the power button

Network setup

  • Smooth access to the Internet, including but not limited to: docker.io and GitHub...
  • All servers must be connected to the same wired network with the initial controller
  • You have the access to change DNS config (on your router or at your domain registrar)

Router DHCP PXE Proxy Config

For OpenWrt, You can configure the following:

vi /etc/config/dhcp

Add the following:

config boot
    option filename 'netboot.xyz.efi'
    option servername 'netbootxyz'
    option serveraddress '192.168.3.225'

✍Notes:

192.168.3.225 is the IP address of your initial controller

Then reboot service:

/etc/init.d/dnsmasq restart

Or, you can use the following command(for openwrt):

uci set dhcp.linux="boot"
uci set dhcp.linux.filename="netboot.xyz.efi"
uci set dhcp.linux.serveraddress="192.168.3.225"
uci set dhcp.linux.servername="netbootxyz"
uci commit dhcp
/etc/init.d/dnsmasq restart

Domain

We use Tailscale MagicDNS directly to provide domains/certificates/tunnel, all you need to do is to enable this, as follows:

BIOS setup

Info

You need to do it once per machine if the default config is not sufficent, usually for consumer hardware this can not be automated (it requires something like IPMI to automate).

Common settings:

  • Enable Wake-on-LAN (WoL) and network boot
  • Use UEFI mode and disable CSM (legacy) mode
  • Disable secure boot

Boot order options (select one, each has their pros and cons):

  1. Only boot from the network if no operating system found: works on most hardware but you need to manually wipe your hard drive or delete the existing boot record for the current OS
  2. Prefer booting from the network if turned on via WoL: more convenience but your BIOS must support it, and you must test it throughly to ensure you don't accidentally wipe your servers

Example

Below is my BIOS setup for reference. Your motherboard may have a different name for the options, so you'll need to adapt it to your hardware.

Devices:
  NetworkSetup:
    PXEIPv4: true
    PXEIPv6: false
Advanced:
  CPUSetup:
    VT-d: true
Power:
  AutomaticPowerOn:
    WoL: Automatic  # Use network boot if Wake-on-LAN
Security:
  SecureBoot: false
Startup:
  CSM: false

Tailscale (requires third-party account)

For Node

Get an auth key from Tailscale admin console, which is used for tailscale installation on node:

  • Description: homelab
  • Reusable: optionally set this to true

You can now connect to your homelab via Tailscale and invite user to your Tailscale network.

For Kubernetes Operator

📚️Reference:

Kubernetes operator · Tailscale Docs

  1. In your tailnet policy file, create the tags tag:k8s-operator and tag:k8s, and make tag:k8s-operator an owner of tag:k8s. If you want your Services to be exposed with tags other than the default tag:k8s, create those as well and make tag:k8s-operator an owner.
"tagOwners": {
   "tag:k8s-operator": [],
   "tag:k8s": ["tag:k8s-operator"],
}
  1. Create an OAuth client in the OAuth clients page of the admin console. Create the client with Devices Core and Auth Keys write scopes, and the tag tag:k8s-operator.

Gather information

  • Decide a control_plane_endpoint
  • Adapt the configuration to your situation: registries_config_yaml. Minimum configuration is like below. This enables the latest features of k3s: Embedded Registry Mirror | K3s
  • MAC address for each machine
  • OS disk name (for example /dev/sda)
  • Network interface name (for example eth0)
  • Choose a static IP address for each machine (just the desired address, we don't set anything up yet)
registries_config_yaml: |
  mirrors:
    "*":

△ Minimum configuration for registries_config_yaml