Homelab Rebuild Part 36: The Fans Were Loud, the CPUs Were Idle

| October 19, 2026 | 10 min to read

Intro

The three mini PCs under the desk got loud. My first guess was that something in the cluster was pinning the CPUs, so that is how I asked the question: “what is maxing out the CPU?” Nothing was. The CPUs were mostly idle and running at full clock anyway, because that is what Proxmox tells them to do.

TL;DR: Proxmox VE sets the performance CPU frequency governor (the kernel policy that picks the clock speed). It keeps cores near their top clock even at idle. pve-03 sat at 2700 MHz and 75°C on 5% load. A small boot-time unit (cpu-power.service) now sets powersave with EPP balance_power on the Intel host and schedutil on the two Ryzens. pve-03 dropped to 800 MHz and 56°C, and pve-01 dropped to about 1600 MHz. I then checked three suspects for the rest of the load (the descheduler, Kyverno’s reports controller, KVM overhead) and ruled out all three. pve-02 is louder because it runs about twice the guest work.

The Figure

Flyballs: they fly out as the load risesStop finger: the lowest clock the mode allowsThe load spins the spindle, the balls fly out and lift the sleeve, and the lever opens the throttle: the clock on the dial. The knob sets a stop the throttle cannot close past. In performance (the Proxmox default) it holds the i7-8550U in pve-03 at 2700 MHz and 75 °C at 5% load; in powersave with EPP balance_power the same chip idles at 800 MHz and 56 °C, and schedutil lets a Ryzen 2400GE fall to its 1.6 GHz floor (pve-01, 57 °C) or sit near 2800 MHz at about 30% load (pve-02, 65 °C). Push the load up and every mode still boosts to the chip's maximum.

The figure is a centrifugal flyball governor, the steam-engine part the Linux feature is named after. Load spins the spindle, the balls swing out, the sleeve rises and the lever opens the throttle. Switch it to performance, and the throttle stays pinned open no matter what the balls do. That was my lab this morning. The idle numbers in the readout are the ones I measured. The full-load numbers are modelled from the maximum clock of each chip. I did not measure them.

The Situation

Host	CPU	cpufreq driver	Guests
pve-01	Ryzen 5 2400GE	acpi-cpufreq	controller-01, two workers
pve-02	Ryzen 5 2400GE	acpi-cpufreq	controller-02, two workers
pve-03	i7-8550U	intel_pstate	controller-03, quorum-only worker

These are low-power chips (35 W and 15 W) in small cases with small fans. They get noisy as soon as they get warm, and “warm” here is a lot closer to “idle” than I expected.

Issue 1: The Load Numbers Lied, a Little

Symptom. top -bn1 on pve-02 showed the KVM processes at several hundred percent together, then 184% a few seconds later. Inside the guests, the same VMs looked nearly idle.

Cause. The first sample of top -bn1 has no earlier sample to compare against, so its CPU numbers swing wildly. It is not a reading you can act on. The second sample of top -bn2 is a reading you can act on. So is the Proxmox RRD (its round-robin metrics store), which keeps an average per minute.

top -bn2 -d10        # read the second block
pvesh get /nodes/pve-02/rrddata --timeframe day --output-format json

Fix. Read the trend, not the snapshot. Over the day, no host went above 58%, and that peak was the 06:00 boot of all eight VMs. Steady state was about 12% on pve-01, 30% on pve-02 and 5% on pve-03. Nothing was maxing out. So the question became: why are the fans loud at 5%?

Issue 2: The Governor Held the Clock Up

Symptom. At 5% load, pve-03 averaged 2700 MHz across its cores and read 75°C. The Ryzens sat near 2900 to 3000 MHz at 12 to 30%.

cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_driver     # intel_pstate / acpi-cpufreq
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor   # performance
grep MHz /proc/cpuinfo
sensors

Cause. Proxmox VE sets the performance governor on purpose: a hypervisor that never waits for a clock ramp gives guests the lowest latency. On a rack server with real airflow, that trade is cheap. On a mini PC, idle cores stay near full clock and the package stays warm. The fan curve does the rest.

Fix. Pick the governor per driver, because the two drivers offer different ones:

intel_pstate (pve-03): powersave plus the energy-performance preference (EPP) balance_power. Despite the name, powersave under intel_pstate is not “always slow”. The hardware still boosts under load. The EPP hint only tells it how eagerly to boost.
acpi-cpufreq (pve-01, pve-02): schedutil, which sets the clock from the scheduler’s own utilisation numbers.

I wrote both by hand first and watched the result. pve-03 dropped 12°C within a minute.

Issue 3: Making It Survive a Reboot

Symptom. Writes to /sys are gone after a reboot, and these hosts follow a daily power schedule.

Fix. I added a oneshot systemd unit, which runs once at boot and exits. I installed it the same way on all three hosts from the infra repo:

#!/bin/sh
# /usr/local/sbin/cpu-power
for c in /sys/devices/system/cpu/cpu*/cpufreq; do
  if [ "$(cat "$c/scaling_driver")" = intel_pstate ]; then
    echo powersave > "$c/scaling_governor"
    echo balance_power > "$c/energy_performance_preference"
  else
    echo schedutil > "$c/scaling_governor"
  fi
done

# /etc/systemd/system/cpu-power.service
[Unit]
Description=Power-saving CPU governor

[Service]
Type=oneshot
ExecStart=/usr/local/sbin/cpu-power

[Install]
WantedBy=multi-user.target

for h in pve-01 pve-02 pve-03; do ssh root@$h bash -s < proxmox/cpu-power/install.sh; done
ssh root@pve-03 'cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor; systemctl is-enabled cpu-power'

One script handles both vendors because it reads scaling_driver. There is no per-host configuration to keep in sync. To revert, run systemctl disable --now cpu-power and reboot.

What I Ruled Out

pve-02 stayed warmer than the others after the fix, so I looked for a software cause before blaming the hardware. I had three suspects, and none of them held up.

The descheduler

The descheduler evicts pods so the scheduler can place them better. It only helps if the placement is bad. Here it is not:

CPU requests per worker: 16% and 13% on pve-01, 23% and 15% on pve-02, 5% on the quorum-only pve-03 worker.
The heavy pods (Postgres, OpenBao, VictoriaMetrics) are pinned to their node by local-path volumes. The descheduler can evict them, but they can only come back to the same node.
Every VM boots at 06:00, so the scheduler places everything fresh once a day anyway.

Kyverno’s reports controller

kyverno-reports-controller sat exactly at its 200m CPU limit. That looks like a pod that is starved of CPU. When I measured it, it used 0.12 to 0.21 of a core. That covered 113 to 214 policy evaluations a minute at about 16 ms each. It ran background scans of Pods against about 19 Audit policies and kept 409 PolicyReports up to date. Its restarts were clean exits (Completed, exit code 0), not OOM kills. If I move the background scan from every hour to every six hours, I save less than a tenth of a core. That is not worth the loss of fresher reports.

KVM overhead

Was the hypervisor itself burning cycles on behalf of the guests?

Halt polling. KVM can spin briefly on an idle vCPU instead of sleeping. halt_poll_ns is 200000, but halt_poll_ns_grow is 0, so polling never starts. KVM attempted zero polls.
Guest interrupts. Each worker takes about 4,000 local timer and 2,500 function-call interrupts a second, on both Ryzen hosts alike. These come from the Go runtime timers in kubelet, containerd and every controller. No host setting changes them.
AVIC (AMD interrupt virtualisation, which makes those interrupts cheaper) is off. It stays off, because it has known errata on Zen 1 parts like the 2400GE.

Both hosts showed the same exits and the same interrupt rates per guest. The difference is the amount of guest work. pve-02 carries controller-02, which runs at 25% of its 2 vCPUs. controller-01 runs at 11%. The workers on pve-02 hold the reports controller, Wazuh, the cluster OpenTelemetry collector, openbao-0 and pihole-2. It does about twice the work, so it uses about twice the CPU.

Results

Host	Before	After
pve-01	2990 MHz, 65°C	about 1600 MHz, 57°C
pve-02	2887 MHz, 62°C	2600 to 3000 MHz, 65°C
pve-03	2700 MHz, 75°C	800 MHz, 56°C

pve-03, the one that was loudest, is now quiet. pve-01 dropped by half its clock. pve-02 still clocks up because it has real work most of the time. That is the correct behavior for schedutil. The next step for pve-02 is physical: dust out the fan and check the BIOS fan curve.

Lessons

Ask what is wrong. “The CPU is maxing out” was really “the fans are loud”. The two have different causes.
Do not act on a first top sample. Use a second sample or the RRD.
Server defaults are server defaults. performance is a sane choice for a rack box and the wrong one for a mini PC on a desk.
powersave is not “slow” under intel_pstate. It still boosts. EPP sets how eagerly.
Measure before tuning the suspects. A pod at its CPU limit, a busy hypervisor and a lopsided host all had plain explanations, and none of them needed a fix.

Series

This post is part of Homelab Rebuild: