Homelab Rebuild Part 23: microVM Sandboxes on Proxmox

Intro

I wanted throwaway sandboxes next to the cluster: machines I can reach from my laptop, break freely, and delete. Containers share the host kernel, and a full VM takes a template, cloud-init and half a minute. Then I found pve-microvm, which adds QEMU’s microvm machine type to Proxmox. These are VMs with their own kernel. They boot in seconds, and you manage them with the normal qm tools.

The catch is in its README: “highly experimental, patches qemu-server internals”. This post covers how I made sure that it did not break the k0s VMs on the same host. It also covers the apt command that almost removed Proxmox, and the two fixes the stock template needed.

TL;DR: pve-microvm 0.3.29 applied cleanly on qemu-server 9.2.11, one minor version past what upstream tested. The proof that ordinary VMs did not change: their qm showcmd output was byte-identical before and after. The stock Debian template never got an IP (no udev, so systemd-networkd waits forever), and its cloud-init hung. A golden template with udev, baked-in SSH keys and cloud-init disabled gives me a sandbox in 19 seconds, with a guest boot of 5.6 seconds.

The Figure

The sealed chamber is the microVM, and the bench under it is pve-01, with the three k0s VMs standing at the back. A sandbox starts as a linked clone of golden template 9110 on the shelf, which passes into the chamber through the airlock. The glove ports are the narrow way in: virtio devices, no firmware, no display. The bench timer counts from qm start to an open SSH port. A mark shows where a standard VM gets there. Click or press N for a new sandbox. Press D to delete it. I generated the figure with the anatomy agent skill.

The Situation

• Three Proxmox hosts on PVE 9.2.21 (qemu-server 9.2.11, pve-qemu-kvm 11.0.3, kernel 7.0.14). Upstream tested 9.1.x with QEMU 10.1 and kernel 6.17.
• pve-01 and pve-02: Ryzen 5 2400GE, 31 GiB RAM, 12.8 and 20 GiB free. pve-03 (the laptop) had 1.2 GiB free, so it is out.
• Storage is local LVM-thin only, so no HA or migration for these VMs. microvm cannot live-migrate anyway.
• pve-01 also runs a k0s controller and two workers. Breaking VM starts on that host is the real risk.
• pve-02 is still on the bad power adapter (Part 21), so pve-01 goes first.

What pve-microvm actually is: a Debian package that patches three Perl files in qemu-server (QemuServer.pm, Machine.pm, BlockJob.pm). With the patch, machine: microvm builds a QEMU command with direct kernel boot (a shipped 6.12 guest kernel), virtio devices on a minimal PCIe bus, and no firmware or display. It is not Firecracker, but Firecracker rootfs images boot on it.

Setup

A baseline that proves nothing changed

Before the install, I recorded what “unchanged” looks like:

dpkg -V qemu-server       # prints nothing: files match the package
sha256sum /usr/share/perl5/PVE/QemuServer.pm \
  /usr/share/perl5/PVE/QemuServer/{Machine,BlockJob}.pm > /root/qemu-server-sha256.pre-microvm
for id in 101 102 200; do qm showcmd $id --pretty > /root/microvm-baseline/showcmd-$id.txt; done
tar czf /root/etc-backup-pre-microvm-$(date +%F).tgz /etc

qm showcmd prints the full QEMU command line that Proxmox runs for a VM, but it does not start the VM. If the patch changes anything for ordinary VMs, it shows up there first.

Issue 1: The Dependency That Can Remove Proxmox

Symptom

The docs list skopeo, umoci and qemu-utils as dependencies, so I installed all three:

W: (pve-apt-hook) touch '/please-remove-proxmox-ve'
W: (pve-apt-hook) run apt purge proxmox-ve to remove the meta-package
E: Sub-process /usr/share/proxmox-ve/pve-apt-hook returned an error code (1)

Cause

pve-qemu-kvm already provides qemu-utils and conflicts with Debian’s real package:

Provides: qemu-system-arm, qemu-system-x86, qemu-utils
Conflicts: ... qemu-system-x86, qemu-utils

When I asked for qemu-utils by name, apt tried to install Debian’s package. That meant it had to remove pve-qemu-kvm, and then proxmox-ve. Proxmox’s apt hook refused, and apt removed nothing. The virtual package (a name that another package provides) already meets the .deb’s own dependency on qemu-utils.

Fix

Install only umoci (skopeo was already in). Then compare the .deb with the release’s sha256, and install it:

apt-get install -y umoci
dpkg -i pve-microvm_0.3.29-1_all.deb
/usr/share/pve-microvm/pve-microvm-patch status
# applied (2026-10-15T19:08:59+07:00)

The gate

101 identical
102 identical
200 identical
??5??????   /usr/share/perl5/PVE/QemuServer/BlockJob.pm
??5??????   /usr/share/perl5/PVE/QemuServer/Machine.pm
??5??????   /usr/share/perl5/PVE/QemuServer.pm

The command lines of all three k0s VMs were identical, and only the three expected files changed. A throwaway ordinary VM started and stopped normally. Rollback is apt remove pve-microvm. It restores the originals from backup.

Issue 2: A Sandbox That Never Got an IP

Symptom

The template tool built a Debian trixie template in under a minute. A clone booted, and the guest agent answered after 4.5 seconds. But the only address was 127.0.0.1:

IDX LINK TYPE     OPERATIONAL SETUP
  2 eth0 ether    off         pending

Cause

The image has no udev (udevadm: not found). systemd-networkd configures a link only after udev initializes it, so every link stays pending forever. The template script installs systemd with --no-install-recommends, and that leaves udev out. Its RPM path adds a dhclient fallback unit, but the Debian path does not. The first-boot setup service waits for network-online.target, so it ran only after the wait-online timeout.

Fix

I proved this on the clone first: bring the link up by hand with dhclient, install udev, reboot. After the reboot, the link came up as enp0s4. networkd reported it routable configured, and DHCP worked without help.

Issue 3: cloud-init That Never Finished

Symptom

qm set --sshkeys did nothing, because the clone had no cloud-init drive. I tried to add one to the running VM, and it failed (hotplug problem - error on hotplug device 'scsihw0'), because microVMs have no hotplug. With the VM stopped, it attached and cloud-init found the drive. Then cloud-init sat at status: running:

cloud-config.service   activating  start  Cloud-init: Config Stage
cloud-init-network.service  inactive  dead  Cloud-init: Network Stage
└─ nc -Uu -W1 /run/cloud-init/share/config.sock

Cause

cloud-init 25 runs its stages in one process, and each systemd unit signals that process over a socket. The network stage never got a start job in this image, so the config stage waited on the socket forever.

Fix

I did not need cloud-init at all. It delivers only SSH keys and a hostname here. So I disabled it and built a golden template (a ready-made image that clones copy) instead.

The Golden Template

build-template.sh builds the stock template into a scratch VMID and boots one full clone. It lets the first-boot setup of the clone finish (guest agent, apt sources). Then it runs a customize script inside it:

apt-get install -y --no-install-recommends udev gnupg
curl -fsSL https://tailscale.com/install.sh | sh      # installed, not logged in
touch /etc/cloud/cloud-init.disabled
printf '%s\n' "$KEYS" > /root/.ssh/authorized_keys
systemctl enable ssh-hostkeys.service                 # ssh-keygen -A on first boot
rm -f /etc/ssh/ssh_host_*
truncate -s 0 /etc/machine-id

Then it shuts down and runs qm template 9110. The wipe of the host keys and machine-id is important. Without it, every clone shares the same SSH identity. A second script clones a sandbox, sizes it, sets the hostname and prints the IP:

proxmox/microvm/new-sandbox.sh pve-01 9110 810 sandbox-01 1024 2 10
# sandbox-01 (810) on pve-01: 192.168.101.232
# 18.8 s total

Inside the sandbox:

6.12.22
Mem:   973 MiB
/dev/vda  9.8G  419M  8.9G   5% /
Startup finished in 545ms (kernel) + 5.095s (userspace) = 5.640s

The disk grows online with resize2fs /dev/vda, because the root filesystem is directly on the disk with no partition table. SSH from the Mac goes directly to the LAN IP. For the tailnet, tailscale up --ssh inside the sandbox is the only step.

Result

pve-02 follows after its new adapter arrives. Sandboxes stay outside Terraform, because the Proxmox provider has no microvm machine type, and outside Kubernetes. The nightly power schedule treats them like any other VM.

Lessons

1. Define “unchanged” before you install. dpkg -V, checksums and qm showcmd output changed “it seems fine” into a diff that came back empty.
2. Never name a virtual package on Proxmox. pve-qemu-kvm provides qemu-utils. A request for it by name tried to remove Proxmox, and only the PVE apt hook stopped it.
3. Minimal images drop things you assume. Without udev, systemd-networkd does not fail. It waits, and that is harder to spot.
4. Remove a component instead of debugging it when you do not need it. cloud-init can deliver only a hostname and SSH keys. A golden template does that with less to break.
5. Experimental fits a sandbox host, not a quorum member. pve-03 stays out for RAM reasons, and pve-02 waits for stable power.

Update: a real comparison (2026-10-16)

The numbers above had no baseline. I never booted a standard VM the same way. So I did that on pve-01, with a full clone of each template at the same size (2 vCPU, 1 GiB). I did 3 warm runs of each, in alternating order, with the k0s VMs running alongside. Medians:

The standard VM is Debian 13 genericcloud on q35 with SeaBIOS and GRUB. The microVM boots its kernel directly, with no firmware or bootloader.

What the table does not say on its own:

• Part of the userspace gap is my template, not the machine type. The standard VM runs cloud-init on every boot. The microVM template has it disabled.
• qm start is now half of the microVM’s start time. About 2.6 s goes to Proxmox’s own Perl tooling before the guest does anything.
• Isolation is the same as any VM. Same KVM, same QEMU process confinement as every Proxmox guest. The microVM only exposes fewer devices (no PCI bus, no firmware). There is no Firecracker-style jailer here.
• The earlier 5.6 s guest boot was a single first boot on 1 vCPU. Warm, with 2 vCPU, it is 3.3 s.

Three runs on one busy host are indicative, not a lab benchmark. They are enough to say that a sandbox answers SSH in about 5 seconds instead of 15. It also uses 85 MiB less host memory.

Update: pve-02 joins (2026-10-16, evening)

pve-02 got its new power adapter (see Part 21), and its SATA SSD logged no errors after the swap. So I did the second install the same way as the first, by hand, with the same gate:

dpkg -V qemu-server                                  # prints nothing
for id in $(qm list | awk 'NR>1{print $1}'); do
  qm showcmd $id --pretty > /root/microvm-baseline/showcmd-$id.txt
done
apt-get install -y skopeo umoci                      # never qemu-utils
dpkg -i pve-microvm_0.3.29-1_all.deb                 # digest checked against the release
/usr/share/pve-microvm/pve-microvm-patch status
# applied (2026-10-16T22:37:04+07:00)
# then the same showcmd loop, diffed against the baseline:
# 201 identical
# 202 identical
# 300 identical
# 8000 identical
# 9200 identical

The golden template script needed no changes on the second host:

Both Ryzen hosts now run sandboxes: VMIDs 810 to 819 on pve-01 from template 9110, and 820 to 829 on pve-02 from 9210. pve-03 stays out, for RAM.

Series

This post is part of Homelab Rebuild:

1. Part 1: Upgrading Proxmox VE 8 to 9
2. Part 2: Starting Over When the IaC Is Broken
3. Part 3: When Tailscale Breaks Your Cilium Gateway
4. Part 4: Living on the Tailnet
5. Part 5: Secrets Without Secrets in Git
6. Part 6: Certificates, DNS and a Private CA
7. Part 7: Self-Hosted CI and Getting Back to IaC
8. Part 8: Stateful Workloads on Unequal Hardware
9. Part 9: Going Public Safely with Cloudflare Tunnel
10. Part 10: When the Hypervisor Went Quiet
11. Part 11: Hardening the Cluster in Five Layers
12. Part 12: Observability with Coroot, and Losing pve-01
13. Part 13: The Host That Kept Losing Power
14. Part 14: Observability, Take Two with OpenTelemetry
15. Part 15: Requests and Limits for Everything
16. Part 16: Handing the Cluster to FluxCD
17. Part 17: Making the Guardrails Earn Their Keep
18. Part 18: When the API Server Outgrew Its VM
19. Part 19: Logging In to OpenBao with Cloudflare Access and Google
20. Part 20: Headlamp SSO, or Teaching kube-apiserver to Trust Cloudflare Access
21. Part 21: pve-02 Loses Power, and Pi-hole Has Nowhere to Go
22. Part 22: Turning Four Days of Fixes into AI Runbooks
23. Part 23: microVM Sandboxes on Proxmox (this post)
24. Part 24: Pi-hole Without a Home Host
25. Part 25: It Was the Router
26. Part 26: TLS on the Front, Plaintext Behind It
27. Part 27: One Bad Power Adapter, Two Different Deaths
28. Part 28: Observability, Take Three: Making It Worth Looking At
29. Part 29: One Meter for Every Model, LiteLLM on Workers AI
30. Part 30: Proxmox Joins the SSO, and a Plan Token Loses Admin
31. Part 31: First Backups, Postgres, OpenBao and etcd to R2
32. Part 32: Alerts That Reach Me, Even When the Lab Is Dead
33. Part 33: The Firewall Held, the Tailnet Did Not
34. Part 34: From Watching to Blocking, and a Record of Who Did What
35. Part 35: Wazuh Without the 8 GB, Endpoint Security for One Mac and Three Hosts
36. Part 36: The Fans Were Loud, the CPUs Were Idle
37. Part 37: A Power Trip, a Search Domain and a Wildcard
38. Part 38: An Autonomous AI Pentester in a Sealed Box
39. Part 39: Wasm Pods on k0s, and the Label That Kept Them Packed

References

• GitHub: wheresryan22/anatomy
• GitHub: rcarmo/pve-microvm
• QEMU: microvm machine type
• systemd: systemd-networkd
• cloud-init: Boot stages
• Proxmox: qm manual