Fresh Installing NixOS on a new system

So greencloudVPS just had its 12th birthday sale, and with greed, i bought their server bundle... 220 USD/3 years for a 4 core, 12GB ram server with 4 TB storage was a server deal I could not give up on... So with 4TB, I wanted it as my offsite backup server + another node for my kubernetes cluster. And now with 2 public IPs, I could do fun stuff like floating IP for more reliability.

Another thing I want to do is slowly migrate away from my 1 $5/month digital ocean server, which mainly runs headscale, to one of these servers instead.

So the question is, do I really need headscale, or can I just use plain wireguard and with with nix, is it much easier to push my changes to a cluster now with tools like colmena? My homelab is using colmena right now, but I'm slowly migrating to this repo for all my nix needs.

Another thing to look into is completely restructuring my current infra setup, now that I have more than 1 cloud node.

But I digress, this article is on installing nixOS on this system.

The tools

• nixos anywhere NixOS anywhere, as its name suggests, is a tool to install nixOS on any linux system supporting kexec over ssh.
• disko Nix tool for declarative disk partitioning

Figuring out partitioning with disko

The nut-gc2 server has 2 drives, vda at 60GB, and vdb, at 4TB. I want the vda to be the primary partition, storing the root file system, and VDB just being used for longhorn.

So lets get that out in disko. I initially used UEFI but i was having issues with it booting up, so I'm switching to BIOS.

{
  disko.devices = {
    disk = {
      main = {
        type = "disk";
        device = "/dev/vda";
        content = {
          type = "gpt";
          partitions = {
            boot = {
              size = "1M";
              type = "EF02";
            };
            ESP = {
              size = "512M";
              type = "EF00";
              content = {
                type = "filesystem";
                format = "vfat";
                mountpoint = "/boot";
                mountOptions = [ "umask=0077" ];
              };
            };
            root = {
              end = "-4G";
              content = {
                type = "filesystem";
                format = "ext4";
                mountpoint = "/";
                mountOptions = [ "noatime" ];
              };
            };
            swap = {
              size = "100%";
              content = {
                type = "swap";
                discardPolicy = "both";
                resumeDevice = false;
              };
            };
          };
        };
      };
      vdb = {
        type = "disk";
        device = "/dev/vdb";
        content = {
          type = "gpt";
          partitions = {
            root = {
              size = "100%";
              content = {
                type = "filesystem";
                format = "ext4";
                mountpoint = "/srv";
                mountOptions = [ "noatime" ];
              };
            };
          };
        };
      };
    };
  };
}

A simple configuration creating 3 partitions on vda, boot, root and swap, and 1 partition for data mounted to srv. I'll be serving longhorn from the /srv/longhorn directory and garage for s3 from the /srv/garage directory.

Next would be the general system configuration. Since this is long and very custom, I'm just gonna link it here, and you can view this in the repo.

Now that the configuration is done, I have secrets(such as my k3s token) that I want to transfer over to the new system. To cleanly do this, I can generate the host keys on this system, use that to rekey the secret, transfer over the host keys and then erase them from my system when complete.

tmp_dir=$(mktemp -d)
mkdir -p "$tmp_dir/etc/ssh"
MACHINE_NAME="nut-gc2"

ssh-keygen -A -f "$tmp_dir"
for key in "$tmp_dir"/etc/ssh/ssh_host_*_key; do
    [ -f "$key" ] || continue
    echo "Updating comment on $key ..."
    ssh-keygen -q -c -C "root@$MACHINE_NAME" -f "$key"
done
echo "ssh key: $(cat $tmp_dir/etc/ssh/ssh_host_ed25519_key.pub)"

Now with the ssh key generated, we can copy that over into our secrets.nix, and use age to rekey the secret

agenix --rekey

With this rekeyed, we can get to infecting the system

nix run github:nix-community/nixos-anywhere -- --generate-hardware-config nixos-generate-config ./systems/x86_64-linux/nut-gc2/hardware-configuration.nix --flake ".#${MACHINE_NAME}" --extra-files "${tmp_dir}" --print-build-logs --build-on remote "root@${MACHINE_IP}"

Now with the server infected, you'll need to update your ~/.ssh/known_hosts and you can then ssh into your machine