My homelab k3s cluster runs on a single little Ubuntu box, and for a long time my “disaster recovery plan” was hoping the disk didn’t die. When I started planning a hardware upgrade, I realized the two problems are the same problem: I need to be able to take everything that machine is — the cluster, the volumes, the IP address, even its SSH identity — and drop it onto a different machine without reconfiguring anything else on my network. So I built k3s-backup, a small set of bash scripts that does exactly that.
The goal: a true drop-in replacement
Plenty of guides cover backing up the k3s datastore, but the datastore is only half the machine. My node is also a fixed point in the network: other machines SSH into it by IP, k3s agents and kubeconfigs point at that IP, and NFS clients mount shares from it. If the replacement box comes up with a new IP and new SSH host keys, I’d spend the evening chasing down every client that broke.
So the restore target here is stricter than “the cluster comes back”: the new machine takes over the old machine’s hostname, static IP (via netplan), SSH host keys, and authorized_keys. After the restore reboots, everything on the network talks to the new box exactly like it was the old one — ssh doesn’t even warn, because the host identity really is the same.
What gets backed up
One tar.gz archive captures:
- The k3s datastore — the full
/var/lib/rancher/k3s/serverdirectory: cluster state (SQLite or embedded etcd — auto-detected), TLS CAs, service-account keys, tokens, and manifests. Kubernetes Secrets live in the datastore, so they come along too; nothing needs re-creating after a restore. - Persistent volume data — everything under
/var/lib/rancher/k3s/storage, which is where the default local-path provisioner keeps volume data. - k3s config —
/etc/rancher(config.yaml,registries.yaml, the node password). - The exact k3s binary plus its systemd unit and the install script, so the restore installs the same version and works fully offline.
- Machine identity — hostname,
/etc/hosts, the netplan config, SSH host keys, and per-userauthorized_keys. - NFS exports (opt-in) — with
--nfs, every folder listed in/etc/exportsplus the exports file itself, so the new machine serves the same shares.--nfs-dir <path>grabs any arbitrary extra folder.
Container images are deliberately not included — they keep archives small, and the node just re-pulls them from their registries on first boot.
Taking a backup
On the node itself:
sudo ./backup.sh -o /mnt/nas/k3s --keep 7
Or from another machine — the wrapper uploads the script over SSH, runs it with sudo, pulls the archive back, and cleans up after itself:
./remote-backup.sh [email protected] ./backups -- --nfs
The wrapper scripts are plain bash and only need ssh and scp, so if your workstation is a Windows machine, run them from WSL — that’s how I use them. Just remember WSL has its own ~/.ssh, so the key that reaches your node needs to live there (and be chmod 600), not in your Windows profile.
The script is careful about consistency without taking the cluster down. It stops the k3s service only for the few seconds it takes to archive the datastore — running pods aren’t touched; the API just blips. Volume data is copied while pods keep running, which is fine for most workloads, but if you run databases on local-path volumes there’s a --cold flag that stops all pods (k3s-killall.sh), takes a fully consistent copy, and starts everything back up.
Since --keep N prunes old archives, a root cron entry is all the automation you need:
0 3 * * * /opt/k3s-backup/backup.sh -o /mnt/nas/k3s --keep 14 >> /var/log/k3s-backup.log 2>&1
Restoring onto new hardware
The restore flow is where the drop-in-replacement idea pays off:
- Install a fresh Ubuntu on the new machine — any temporary IP and hostname are fine.
- Update packages and reboot (
sudo apt update && sudo apt full-upgrade -y && sudo reboot), so the restore lands on the new kernel and unattended-upgrades isn’t fighting you mid-restore. - Power off the old machine. Two boxes must never be up on the same IP.
- Push the archive over and run the restore:
./remote-restore.sh [email protected] ./backups/k3s-backup-node1-20260721.tar.gz
The script restores the machine identity, installs the exact k3s version from the archive, lays the datastore and volume data back down, and reboots. The machine comes back up on the old machine’s IP with the old SSH keys and a running cluster. Verification is one command:
ssh tim@<old-ip> 'sudo k3s kubectl get nodes && sudo k3s kubectl get pods -A'
One implementation detail I like: if the datastore is embedded etcd rather than SQLite, you can’t just start k3s on restored data — etcd needs a --cluster-reset to rebuild its membership, and it should happen after the machine has its final IP. The restore script handles this with a one-shot systemd unit that runs the cluster reset on first boot, starts k3s, and then deletes itself.
Treat the archive like a private key
Worth saying plainly: because the backup contains the full datastore and the SSH host keys, the archive holds every secret in your cluster in an extractable form. The script writes it with root-only permissions, but wherever you copy it, keep it that way — and if it leaves your network for offsite storage, encrypt it first:
age -p k3s-backup-node1-20260721.tar.gz > backup.tar.gz.age
Limitations
This is built for the common homelab shape: a single server node, possibly with agents. Agents aren’t backed up, but because the restored server keeps the same IP and token, existing agents reconnect on their own. External datastores (MySQL/Postgres) aren’t supported, and if your new hardware names its network interface differently (eth0 vs enp3s0), you may need one console-side netplan edit after the first boot — the old config is kept at /etc/netplan.pre-restore just in case.
The scripts and a more detailed README are on GitHub: timothydodd/k3s-backup. If it saves your cluster (or your Saturday), let me know.