Back to Home

Homelab Rebuild Part 23: microVM Sandboxes on Proxmox

Technology Homelab Proxmox
| October 15, 2026 | 8 min to read

Intro

I wanted throwaway sandboxes next to the cluster: something I can reach from my laptop, break freely, and delete. Containers share the host kernel, and a full VM takes a template, cloud-init and half a minute. Then I found pve-microvm , which adds QEMU’s microvm machine type to Proxmox: VMs with their own kernel that boot in seconds and are managed with the normal qm tools.

The catch is in its README: highly experimental, patches qemu-server internals. This post covers how I checked that it couldn’t break the k0s VMs on the same host, the apt command that almost removed Proxmox, and the two fixes the stock template needed.

TL;DR: pve-microvm 0.3.29 applied cleanly on qemu-server 9.2.11, one minor version past what upstream tested. The proof that ordinary VMs were untouched: their qm showcmd output was byte-identical before and after. The stock Debian template never got an IP (no udev, so systemd-networkd waits forever) and its cloud-init hung. A golden template with udev, baked-in SSH keys and cloud-init disabled gives me a sandbox in 19 seconds, with a guest boot of 5.6 seconds.

The Situation

  • Three Proxmox hosts on PVE 9.2.21 (qemu-server 9.2.11, pve-qemu-kvm 11.0.3, kernel 7.0.14). Upstream tested 9.1.x with QEMU 10.1 and kernel 6.17.
  • pve-01 and pve-02: Ryzen 5 2400GE, 31 GiB RAM, 12.8 and 20 GiB free. pve-03 (the laptop) had 1.2 GiB free, so it’s out.
  • Storage is local LVM-thin only, so no HA or migration for these VMs. microvm can’t live-migrate anyway.
  • pve-01 also runs a k0s controller and two workers. Breaking VM starts on that host is the real risk.
  • pve-02 is still on the bad power adapter (Part 21), so pve-01 goes first.

What pve-microvm actually is: a Debian package that patches three Perl files in qemu-server (QemuServer.pm, Machine.pm, BlockJob.pm) so that machine: microvm builds a QEMU command with direct kernel boot (a shipped 6.12 guest kernel), virtio devices on a minimal PCIe bus, and no firmware or display. It isn’t Firecracker, but Firecracker rootfs images boot on it.

Setup

A baseline that proves nothing changed

Before installing, I recorded what “unchanged” looks like:

dpkg -V qemu-server       # prints nothing: files match the package
sha256sum /usr/share/perl5/PVE/QemuServer.pm \
  /usr/share/perl5/PVE/QemuServer/{Machine,BlockJob}.pm > /root/qemu-server-sha256.pre-microvm
for id in 101 102 200; do qm showcmd $id --pretty > /root/microvm-baseline/showcmd-$id.txt; done
tar czf /root/etc-backup-pre-microvm-$(date +%F).tgz /etc

qm showcmd prints the full QEMU command line Proxmox would run for a VM, without starting it. If the patch changes anything for ordinary VMs, it shows up there first.

Issue 1: The Dependency That Would Remove Proxmox

Symptom

The docs list skopeo, umoci and qemu-utils as dependencies, so I installed all three:

W: (pve-apt-hook) touch '/please-remove-proxmox-ve'
W: (pve-apt-hook) run apt purge proxmox-ve to remove the meta-package
E: Sub-process /usr/share/proxmox-ve/pve-apt-hook returned an error code (1)

Cause

pve-qemu-kvm already provides qemu-utils and conflicts with Debian’s real package:

Provides: qemu-system-arm, qemu-system-x86, qemu-utils
Conflicts: ... qemu-system-x86, qemu-utils

Asking for qemu-utils by name told apt to install Debian’s package, which meant removing pve-qemu-kvm, which meant removing proxmox-ve. Proxmox’s apt hook refused, and nothing was removed. The .deb’s own dependency on qemu-utils is already met by the virtual package.

Fix

Install only umoci (skopeo was already in), then the .deb, after checking it against the release’s sha256:

apt-get install -y umoci
dpkg -i pve-microvm_0.3.29-1_all.deb
/usr/share/pve-microvm/pve-microvm-patch status
# applied (2026-10-15T19:08:59+07:00)

The gate

101 identical
102 identical
200 identical
??5??????   /usr/share/perl5/PVE/QemuServer/BlockJob.pm
??5??????   /usr/share/perl5/PVE/QemuServer/Machine.pm
??5??????   /usr/share/perl5/PVE/QemuServer.pm

The command lines of all three k0s VMs were identical, and only the three expected files changed. A throwaway ordinary VM started and stopped normally. Rollback is apt remove pve-microvm, which restores the originals from backup.

Issue 2: A Sandbox That Never Got an IP

Symptom

The template tool built a Debian trixie template in under a minute. A clone booted, and the guest agent answered after 4.5 seconds, but the only address was 127.0.0.1:

IDX LINK TYPE     OPERATIONAL SETUP
  2 eth0 ether    off         pending

Cause

The image has no udev (udevadm: not found). systemd-networkd only configures a link after udev has initialized it, so every link stays pending forever. The template script installs systemd with --no-install-recommends, which leaves udev out. Its RPM path adds a dhclient fallback unit, but the Debian path doesn’t. The first-boot setup service waits for network-online.target, so it only ran after the wait-online timeout.

Fix

Proven on the clone first: bring the link up by hand with dhclient, install udev, reboot. After the reboot the link came up as enp0s4, networkd reported it routable configured, and DHCP worked without help.

Issue 3: cloud-init That Never Finished

Symptom

qm set --sshkeys did nothing, because the clone had no cloud-init drive. Adding one to the running VM failed (hotplug problem - error on hotplug device 'scsihw0'), since microVMs have no hotplug. With the VM stopped it attached, cloud-init found the drive, and then it sat at status: running:

cloud-config.service   activating  start  Cloud-init: Config Stage
cloud-init-network.service  inactive  dead  Cloud-init: Network Stage
└─ nc -Uu -W1 /run/cloud-init/share/config.sock

Cause

cloud-init 25 runs its stages in one process, and each systemd unit signals that process over a socket. The network stage never got a start job in this image, so the config stage waited on the socket forever.

Fix

I didn’t need cloud-init at all: the only things it would deliver are SSH keys and a hostname. So I disabled it and built a golden template instead.

The Golden Template

build-template.sh builds the stock template into a scratch VMID, boots one full clone, lets its first-boot setup finish (guest agent, apt sources), and runs a customize script inside it:

apt-get install -y --no-install-recommends udev gnupg
curl -fsSL https://tailscale.com/install.sh | sh      # installed, not logged in
touch /etc/cloud/cloud-init.disabled
printf '%s\n' "$KEYS" > /root/.ssh/authorized_keys
systemctl enable ssh-hostkeys.service                 # ssh-keygen -A on first boot
rm -f /etc/ssh/ssh_host_*
truncate -s 0 /etc/machine-id

Then shut down and qm template 9110. Wiping the host keys and machine-id matters, because every clone would otherwise share the same SSH identity. A second script clones a sandbox, sizes it, sets the hostname and prints the IP:

proxmox/microvm/new-sandbox.sh pve-01 9110 810 sandbox-01 1024 2 10
# sandbox-01 (810) on pve-01: 192.168.101.232
# 18.8 s total

Inside the sandbox:

6.12.22
Mem:   973 MiB
/dev/vda  9.8G  419M  8.9G   5% /
Startup finished in 545ms (kernel) + 5.095s (userspace) = 5.640s

The disk grows online with resize2fs /dev/vda, because the root filesystem sits directly on the disk with no partition table. SSH from the Mac goes straight to the LAN IP. For the tailnet, tailscale up --ssh inside the sandbox is the only step.

Result

Value
Packagepve-microvm 0.3.29 on pve-01
Template9110, Debian 13, udev, Tailscale, cloud-init off
New sandbox18.8 s from qm clone to IP
Guest boot545 ms kernel, 5.6 s total
k0s VMs on the same hostQEMU command lines unchanged

pve-02 follows after its new adapter arrives. Sandboxes stay outside Terraform, because the Proxmox provider has no microvm machine type, and outside Kubernetes. The nightly power schedule treats them like any other VM.

Lessons

  1. Define “unchanged” before you install. dpkg -V, checksums and qm showcmd output turned “it seems fine” into a diff that came back empty.
  2. Never name a virtual package on Proxmox. pve-qemu-kvm provides qemu-utils. Asking for it by name would have removed Proxmox, and only the PVE apt hook stopped it.
  3. Minimal images drop things you assume. Without udev, systemd-networkd doesn’t fail; it waits, which is harder to spot.
  4. Remove a component instead of debugging it when you don’t need it. cloud-init would only have delivered a hostname and SSH keys. A golden template does that with less to break.
  5. Experimental fits a sandbox host, not a quorum member. pve-03 stays out for RAM reasons, and pve-02 waits for stable power.

Series

This post is part of Homelab Rebuild:

  1. Part 1: Upgrading Proxmox VE 8 to 9
  2. Part 2: Starting Over When the IaC Is Broken
  3. Part 3: When Tailscale Breaks Your Cilium Gateway
  4. Part 4: Living on the Tailnet
  5. Part 5: Secrets Without Secrets in Git
  6. Part 6: Certificates, DNS and a Private CA
  7. Part 7: Self-Hosted CI and Getting Back to IaC
  8. Part 8: Stateful Workloads on Unequal Hardware
  9. Part 9: Going Public Safely with Cloudflare Tunnel
  10. Part 10: When the Hypervisor Went Quiet
  11. Part 11: Hardening the Cluster in Five Layers
  12. Part 12: Observability with Coroot, and Losing pve-01
  13. Part 13: The Host That Kept Losing Power
  14. Part 14: Observability, Take Two with OpenTelemetry
  15. Part 15: Requests and Limits for Everything
  16. Part 16: Handing the Cluster to FluxCD
  17. Part 17: Making the Guardrails Earn Their Keep
  18. Part 18: When the API Server Outgrew Its VM
  19. Part 19: Logging In to OpenBao with Cloudflare Access and Google
  20. Part 20: Headlamp SSO, or Teaching kube-apiserver to Trust Cloudflare Access
  21. Part 21: pve-02 Loses Power, and Pi-hole Has Nowhere to Go
  22. Part 22: Turning Four Days of Fixes into AI Runbooks
  23. Part 23: microVM Sandboxes on Proxmox (this post)

References