Skip to content

Design Spec: NixOS VMs on Proxmox

Implementation Status

Component / Feature Status Details
proxmox-vm Role Fully Implemented QEMU guest profile, VirtIO drivers, guest agent, systemd-boot.
Host Structure Pattern Fully Implemented Logical/physical split with per-host Disko layout.
Baseline Disk Layout Fully Implemented GPT, 512M ESP, 100% ext4 root partition on /dev/sda.
ISO Artifact Delivery Fully Implemented Installer ISO staged on the node via Terraform.
Terraform VM Pattern Fully Implemented Reference VM resource shape in vms-pve1.tf.
Host Integration Tests Fully Implemented Automatically generated by the flake for host configs.
Deployed VM Hosts (hl02) Fully Implemented hl02 on pve1 follows this framework end-to-end.

1. Goal

Provide a reusable framework for fully declarative NixOS virtual machines on the Proxmox nodes: a shared proxmox-vm role for guest hardware settings, a standard host directory structure with Disko partitioning, automatic provisioning via Terraform and nixos-anywhere, and auto-generated integration tests.

The framework covers NixOS VMs only. Non-NixOS VMs (such as the Debian hl01 VM, which is provisioned via cloud-init) are out of scope.

2. Rationale

This spec is the VM counterpart of the NixOS LXC Containers on Proxmox framework. VMs carry a full kernel and bootloader, so — unlike LXC hosts — they need hardware guest settings, a partition layout (Disko), and an installer path. Those concerns are the same for every NixOS VM in the lab, so they are defined once here; each host only declares its logical parameters and its virtual hardware sizing.

It builds upon the global Home Lab Bootstrapping Spec for installation and the Declarative Integration Testing Spec for CI/CD validation.

3. proxmox-vm Role (config/nix/roles/proxmox-vm/default.nix)

The role generalizes everything a NixOS guest needs to run well under Proxmox/QEMU, so host directories do not carry a hardware configuration file:

  • Imports the built-in NixOS qemu-guest profile.
  • Loads the standard VirtIO initrd modules (virtio_pci, virtio_scsi, sd_mod, ...) so the installer and the installed system can see the virtual disks.
  • Enables the QEMU Guest Agent (services.qemuGuest), which Proxmox and Terraform rely on for IP reporting and clean shutdowns.
  • Enables boot.growPartition so the root partition expands when the virtual disk is enlarged.
  • Optimizes the console for the Proxmox serial console (console=ttyS0).
  • Configures systemd-boot for UEFI (OVMF) VMs, capping retained generations (configurationLimit) so the small ESP cannot fill up.

All settings use lib.mkDefault where a host may plausibly need to override them (e.g. a legacy-BIOS VM switching to GRUB).

4. Host Structure

Following the standard logical/physical split (see Testing Spec - Split Design), each VM host is described by a small set of files under its host directory:

config/nix/hosts/<host>/
├── default.nix          # Physical entrypoint: configuration.nix + disko.nix
├── configuration.nix    # Logical config (roles + host parameters)
├── disko.nix            # Filesystem and partition layout
└── test-override.nix    # Optional test-only overrides
  • default.nix is what the flake discovers and builds as nixosConfigurations.<host>; its presence also registers the host for auto-generated integration tests. For a VM host it imports configuration.nix and disko.nix.
  • configuration.nix is the logical configuration: it imports roles/common, roles/proxmox-vm, and roles/comin (pull-based GitOps), and defines the per-host logical parameters:
    • networking.hostName = "<host>"
    • networking.hostId = "<hostid>"
    • networking.useDHCP = true
    • system.stateVersion = "<release>"
  • disko.nix accepts { inputs, ... } (provided globally via specialArgs), imports the core Disko module (inputs.disko.nixosModules.disko), and declares the host's physical disk layout (see the baseline layout).
  • test-override.nix (optional) supplies host-specific test assertions or configuration overrides; it is automatically merged by the flake, so a dedicated test.nix entrypoint is not required.

No hardware.nix file is needed: standard VM hardware configuration (VirtIO drivers, systemd-boot) is generalized in the proxmox-vm role.

5. Baseline Disk Layout (Disko)

The baseline disko.nix declares a GPT partition table on the primary SCSI virtual disk (/dev/sda):

  • EFI Partition (ESP): 512MB, formatted as vfat, mounted at /boot with standard secure mount options (umask=0077).
  • Root Partition: 100% of the remaining space, formatted as ext4, mounted at /.

This layout is the default for new VM hosts. A host with different storage needs overrides it in its own disko.nix; the layout is per-host by design.

6. ISO Artifact Delivery

To bridge the local Nix build pipeline with the hypervisor environment, the compiled installer ISO must be staged on the Proxmox target node prior to VM orchestration.

  • Mechanism: Handled declaratively via the proxmox_virtual_environment_file resource within the Terraform workspace.
  • Source Target: The local Nix flake build output directory (./result/iso/\*.iso).
  • Destination Target: Stored in the cluster's default structural storage (local:iso/).
  • Staging: Build the proxmox-images aggregate package before running Terraform, so the ISO coexists behind result with the other image artifacts Terraform reads.
  • Node Coverage: The ISO is uploaded to every node (pve1 and pve2) on each apply, mirroring the LXC template upload pattern. Any node is therefore ready to receive its first NixOS VM without additional staging work.

7. Infrastructure Provisioning (Terraform)

VM hardware is declared per host in config/terraform/220-proxmox-workloads/vms-pve<N>.tf using the bpg/proxmox provider. Each node's file is created when the node receives its first VM (currently only vms-pve1.tf exists); a resource uses that node's provider alias (e.g. proxmox.pve2) and references that node's installer ISO resource. Every NixOS VM resource follows the same reference shape; only the sizing values and identifiers change per host (see the deployed hosts table).

  • Hardware: Host-type CPU cores and dedicated RAM per the host's sizing, with a virtio-scsi-pci SCSI controller.
  • EFI Disk & Storage: OVMF BIOS with an EFI disk of type 4m, stored on the local-zfs pool.
  • Disk Interface (scsi0): Presents the virtual disk as /dev/sda inside the VM, matching the disko.nix expectations.
  • Storage Optimization: Enables discard = "on" (TRIM) to reclaim unused blocks dynamically on the underlying ZFS pool.
  • Installer CDROM (ide2): Mounts the custom installer ISO (uploaded automatically as described in Bootstrapping Spec - Custom ISO).
  • MAC Pinning: Pins a static MAC address to the virtio network interface, enabling stable DHCP reservations.
  • Automated Boot Order: Configures boot_order = ["scsi0", "ide2"].
    • First Boot: The VM disk /dev/sda is empty, so EFI falls back to booting from the installer ISO on ide2.
    • Subsequent Boots: Boots directly from the production system on /dev/sda (scsi0), bypassing the installer.

8. Integration Testing

Integration tests are automatically generated and registered for all hosts containing a configuration.nix file. The flake dynamically discovers each config/nix/hosts/<host>/configuration.nix and creates the host-<host>-test check using the global test generator config/nix/tests/make-test.nix.

Because VM host configurations import roles/common (enabling SSH) and roles/proxmox-vm (enabling the QEMU Guest Agent), the auto-generated test automatically verifies:

  1. Successful boot (multi-user.target reached).
  2. SSH port 22 availability.
  3. QEMU Guest Agent service status.

Host-specific assertions go in the optional test-override.nix (see Host Structure). Tests run in CI (see Testing Spec - Dynamic Discovery).

9. Deployed VM Hosts

Host Node VM ID Cores RAM Disk MAC Address hostId State Version
hl02 pve1 101 2 4 GB 20 GB BC:24:11:D4:F6:65 92bbb1e6 25.11

Bringing up a new NixOS VM host consists of:

  1. Create config/nix/hosts/<host>/ per the host structure, with fresh hostName, hostId, and the current stateVersion.
  2. Add the VM resource to the node's vms-pve<N>.tf following the Terraform reference pattern, with a fresh VM ID and pinned MAC address.
  3. Add the host to the home_lab_nix_vms group in the Ansible inventory (config/ansible/inventory/hosts.yml), which is the operational record of which hosts run NixOS.
  4. Map the pinned MAC address to the designated IP for the host in the router's DHCP reservations.
  5. Stage the image artifacts and apply Terraform (see ISO Artifact Delivery).

  6. Install the system:

    nix develop .#operations -c nixos-anywhere --flake .#<host> root@<host>.<fqdn>

After installation, the comin role keeps the host converged with this repository through the pull-based GitOps model.

10. Assumptions and Constraints

10.1 Disk Device Configuration (/dev/sda)

The physical Disko configuration explicitly targets /dev/sda. This assumes the Proxmox VM is provisioned using a VirtIO SCSI (virtio-scsi-pci) controller. If the VM is provisioned using a VirtIO Block controller, the disk will present as /dev/vda and partitioning will fail.

  • Constraint: The VM must use a VirtIO SCSI disk interface (scsi0) in Terraform to maintain support for TRIM/Discard on thin-provisioned ZFS pools.

10.2 Networking (DHCP)

VM hosts use DHCP for simplicity in this phase, with stable addressing provided by MAC-pinned router DHCP reservations. Transitioning to static IPs can be done in a future specification.

11. Future Work

Future work items are tracked centrally in the specs index.