Back to Home
post-thumb

Homelab Rebuild Part 42: A Detonation Sandbox Behind a Telegram Bot

Technology Homelab Security Proxmox
| November 09, 2026 | 12 min to read

Intro

People send me links and files on Telegram and WhatsApp. Some of them are phishing. I wanted one simple habit for all of them: forward the thing to a bot, and get an answer back. The answer must come from a machine that can break without hurting anything, and that machine must reset itself after every check.

This post covers the detonation sandbox (an isolated VM where an untrusted link or file can run). The parts are two microVMs, the jail inside them, the evidence the hosts keep, the n8n workflow that drives them, and the Telegram relay in front. It also covers the errors on the way.

TL;DR: Two microVMs, 830 on pve-01 and 840 on pve-02, have internet access only. A Proxmox firewall group blocks the LAN, the tailnet and the cluster. Inside, every job runs as an unprivileged user behind three layers: a systemd-run seccomp jail, an extra filter for io_uring, and nono 0.81.0 (Landlock plus a logging proxy). The host records a pcap and the serial console from outside the VM and uploads them to R2. After each job the VM rolls back to the snapshot clean in about 3 s. A relay pod polls the Telegram bot. A web link opens in headless Chromium in the VM and comes back as a screenshot, a verdict and the hosts it contacted. A file, or a link that serves a file, is hashed and looked up on VirusTotal. If VirusTotal does not know it, the VM pulls it from R2 into the jail, and the bot reports what nono saw. The script that runs the sample is not in the VM image yet.

830840
The sample, and a kiln window while its VM runs a jobDials: the pcap and serial capture on the hostClick the hopper (or press L) to send a web link: it goes to the first idle kiln, 830 before 840, and the bot replies busy when both run. Click the card index (K) to send a file whose hash VirusTotal knows: the verdict comes back and nothing runs. Click a kiln (U) to send a file VirusTotal does not know: it goes through R2 into the jail of a kiln. The script that runs the sample is not in the VM image yet, so nothing executes it. While a kiln fires, its dial climbs: that is the host recording from outside the VM. Then the kiln rolls back to the snapshot clean. One second here is five seconds.

The figure is a twin test kiln. Each kiln is one VM, and its door window lights while a job runs. After the job, the kiln rolls back to the snapshot clean. The hopper with the antenna is the relay that polls Telegram. A web link goes straight to a kiln. A file goes to the card index first, which is the VirusTotal lookup, and a known hash drops into the tray with its verdict. The dial in front of each kiln is the capture that the host keeps from outside the VM. Click the hopper, the card index or a kiln to send a link, a known file or an unknown file. I built it with the anatomy skill (see References).

The Situation

  • Proxmox VE 9.2 on pve-01, pve-02 and pve-03, pve-microvm 0.4.1 (QEMU microvm machine type) on pve-01 and pve-02
  • Guest: Debian 13, kernel 6.12.113, systemd 257.13, Chromium 156.0.8078.79, Node 20.19.2
  • n8n 2.45.7 in the k0s cluster, behind oauth2-proxy on the tailnet only (see part 41)
  • Cloudflare R2 for evidence and samples, LiteLLM for the summary line
  • Telegram bot that already sends my Grafana alerts

The flow for one message:

flowchart LR
  T[Telegram bot] -->|getUpdates| R[tg-inbox relay]
  R -->|web page| F[n8n form: Detonation URL check]
  R -->|file| V[VirusTotal hash lookup]
  V -->|unknown| S[R2 samples/]
  S --> F
  F --> VM[VM 830 or 840: jail + nono]
  VM --> F
  F -->|screenshot or report| T
  VM -.->|pcap, serial| E[R2 evidence/]

Setup

Two VMs that see only the internet

VMs 830 (192.168.101.83) and 840 (192.168.101.84) are clones of the microVM templates: 2 GiB, 2 vCPU, 8 GB disk, tag no-autostart. A firewall group detonation in cluster.fw drops everything to private ranges and allows DNS to 1.1.1.2 and the internet. Each VM also has ipfilter and macfilter on, so a sample cannot spoof another address. A check script confirmed this from inside both VMs. Every LAN, tailnet, cluster, SMTP and other DNS target is closed. 1.1.1.2:53 and 1.1.1.1:443 are open, and a spoofed .99 source gets nothing back.

The jail

The job runs as user detonate through /opt/detonate/jail, a wrapper around a transient systemd unit:

systemd-run --quiet --pipe --wait --collect --service-type=exec \
  -p User=detonate -p NoNewPrivileges=yes -p CapabilityBoundingSet= \
  -p SystemCallFilter=@system-service landlock_create_ruleset landlock_add_rule landlock_restrict_self \
  -p SystemCallFilter=~@privileged @mount @module @reboot @swap @raw-io @clock @debug @obsolete @keyring \
  -p RestrictAddressFamilies="AF_UNIX AF_INET AF_INET6" -p RestrictNamespaces=yes \
  -p ProtectSystem=strict -p ProtectHome=yes -p ReadWritePaths="/work /var/lib/detonate/.local/state/nono" \
  -p ProtectProc=invisible -p ProcSubset=pid -p MemoryMax=1G -p TasksMax=256 -p RuntimeMaxSec=300 \
  -- /opt/detonate/enosys nono run --allow /work --read /opt/detonate --deny-domain blocked.invalid -- <job>

nono adds Landlock (a kernel feature that limits which paths a process can open) and a proxy for all traffic. --deny-domain blocked.invalid turns the proxy on but allows every real host, so every connection lands in nono’s session.json with its decision, target and port. The sample cannot write to that file. Only nono can.

A blocked syscall kills the process instead of returning EPERM. The kernel then logs a line like audit: type=1326 ... syscall=425 to the serial console, and the host keeps that line.

Evidence on the host

A Proxmox hookscript starts a tcpdump on the VM’s tap device and a reader on its serial socket. After the VM stops, it uploads both to evidence/<vmid>/<timestamp>/ in the R2 bucket detonation-reports with curl --aws-sigv4 aws:amz:auto:s3. The guest cannot touch any of it. A lifecycle rule deletes samples/ after 1 day and everything after 30 days.

A token that can only drive two VMs

n8n uses the API token detonator@pve!n8n, with a role on /vms/830 and /vms/840 only: VM.Audit, VM.PowerMgmt, VM.Snapshot.Rollback, VM.GuestAgent.Audit and VM.GuestAgent.Unrestricted. agent/exec needs the last one. With this token, cluster/resources lists only the two VMs. Requests for the status of another VM, a clone, a new snapshot or a config change all return 403.

The n8n workflow

“Detonation URL check” takes a form submission:

  1. Pick 830 if it is stopped, else 840, else reply “busy”.
  2. Start the VM and wait for the guest agent.
  3. agent/exec a bash script. The URL goes in as $1, never inside the shell string.
  4. Poll exec-status, then file-read of /work/result.json: exit code, nono’s events, a base64 screenshot and the log tail.
  5. Stop the VM and roll back to clean, also on every error path.
  6. Drop the calls that Chromium makes to Google hosts on its own.
  7. Apply the verdict rules. An exit code other than 0 gives “needs a look”. Any nono denial gives “suspicious”. Anything else gives “no red flags”, which does not prove that the target is safe.
  8. Ask LiteLLM for a 3-line summary of the facts only.
  9. Send the screenshot with the verdict to Telegram.

A run takes 25 to 35 s. In file mode, the script first pulls the sample from a presigned R2 link with curl into /work/sample. Then it starts /opt/detonate/run-file.sh /work/sample in the same jail. That script is not in the snapshot clean yet, so the jail runs true in its place, and true does not execute the sample. The report lists nono’s session data: the command, the tracked paths, the hosts and the denials.

File mode on VM 830: random-test-2.bin goes into the jail, but run-file.sh is not in the image yet, so the command is true and the sample does not run

The Telegram relay

n8n is not on the internet, so a Telegram webhook cannot reach it. The relay tg-inbox is about 120 lines of Node in one pod. It long-polls getUpdates (outbound only) and answers only my chat:

  • A link to a web page goes to the n8n form.
  • A file, or a link whose HEAD says it is not HTML, is read into memory, hashed with SHA-256 and looked up on VirusTotal.
  • If VirusTotal returns 404, the relay puts the file in R2 under samples/<sha256> with a presigned PUT. Then it sends a presigned GET to the form in file mode.

Its egress is the internet minus 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16 and 100.64.0.0/10, so a link cannot point it at the LAN.

eicar-word-macro-write-file.doc
MALICIOUS: 45 of 75 engines flag it
https://www.virustotal.com/gui/file/c2e0751687570b457f56ae600aaab2107c166369055d2cffcf9b45fb0425d76a

Issue 1: Node Killed at Start

Symptom

node --version inside the jail died with a seccomp kill, syscall=425.

Cause

libuv 1.50 calls io_uring_setup (425). UV_USE_IO_URING=0 does not stop it, because libuv also uses a ring for epoll. A deny rule SystemCallFilter=~io_uring_setup:ENOSYS works alone, but next to the allow list systemd fails with Failed to apply system call filters: Permission denied (status 228/SECCOMP).

Fix

Allow the io_uring calls in systemd, then stack a second filter. /opt/detonate/enosys uses python3-seccomp with default ALLOW and ERRNO(ENOSYS) for io_uring_*, then execs the job. With stacked filters, the stricter result wins, so libuv gets -1 38 and falls back to epoll. V8 then needed pkey_alloc, pkey_free and pkey_mprotect (syscall 330), and the Chromium zygote needed capset (126). The empty bounding set keeps capset harmless.

Issue 2: Chromium Refused to Start

Symptom

# lacks support for the sse3 instruction set

Cause

Debian’s chromium wrapper greps /proc/cpuinfo, and ProcSubset=pid hides it.

Fix

Call /usr/lib/chromium/chromium directly with --no-sandbox, because the jail blocks the namespaces that the Chromium sandbox needs. Take --proxy-server=127.0.0.1:<port> from the HTTPS_PROXY value of nono. The nono proxy accepts loopback without auth. --disable-background-networking and --disable-component-update cut most of Chromium’s own calls home.

Issue 3: The post-start Hook Never Ran

Symptom

No pcap and no serial log after the first runs.

Cause

On qemu-server 9.2.14 with pve-microvm 0.4.1, Proxmox calls pre-start, pre-stop and post-stop for a microvm, but never post-start. Then tcpdump failed with That device is not up, because the tap device’s operstate stays unknown.

Fix

Start the captures from pre-start with a background waiter that checks the IFF_UP bit in /sys/class/net/tap<vmid>i0/flags. $! after setsid is not the capture’s PID, so post-stop uses pkill -f "$d/" on the evidence directory. A side effect: an SSH command that names that directory gets killed too, so I run qm stop in its own SSH call.

Issue 4: Samples Do Not Fit Through the Guest Agent

Symptom

agent/file-write failed with larger content.

Cause

The limit sits between 60000 bytes (works) and 80000 bytes (fails). file-read stops at 16777216 bytes and returns raw bytes.

Fix

Samples go in through a presigned R2 GET that the guest fetches before the jail starts. Reports come out through file-read, with the screenshot as base64 inside the JSON.

Issue 5: n8n Could Not Sign Requests to R2

Symptom

The S3 node rejected region auto with Unsupported AWS region, then got 403 because it makes bucket-level calls. The HTTP node with an AWS credential failed with Missing x-amz-content-sha256, then with Credential service should be s3, not <account id>.

Cause

n8n takes the service name from the first label of a custom host, which for R2 is the account ID.

Fix

n8n does not talk to R2. The hosts upload the evidence with curl, and the relay presigns sample uploads itself with about 20 lines of SigV4. A Code node can also presign, but only with environment access turned on for every workflow, which exposes all n8n secrets. I left that off.

Symptom

A link to a .doc file showed “Opening…” and then nothing for 5 minutes.

Cause

Headless Chromium with --screenshot waits for a page that never comes when the URL is a download. The jail’s RuntimeMaxSec=300 ended it.

Fix

The relay sends a HEAD first. Anything that is not text/html goes to the hash lookup. The same EICAR test link then came back in seconds.

Symptom

The relay’s presigned PUT and GET both returned 403.

Cause

The R2 secret in OpenBao ends with a newline. The External Secret for the relay copies the value as it is, so R2_SECRET_ACCESS_KEY keeps the newline, and the relay signs with a key that R2 does not know.

Fix

The relay trims every environment value at start.

Issue 8: Telegram Rejected the File Report

Symptom

Bad request - please check your parameters from the Telegram node.

Cause

The node sends legacy Markdown, and nono’s key tracked_paths has a bare underscore.

Fix

Escape _, *, ` and [ in the report and in the LLM summary.

Lessons

  1. Kill on a blocked syscall. A kill shows up on the serial console, which the host keeps. EPERM is quiet and leaves no trace.
  2. Stack seccomp filters when one rule set cannot say it. The stricter result wins, so a small second filter can return ENOSYS where systemd cannot.
  3. Keep the evidence outside the guest. pcap and serial come from the host, so a sample that owns the VM still cannot clean up after itself.
  4. Poll when you do not want a public endpoint. getUpdates needs only outbound HTTPS, so n8n stays on the tailnet.
  5. Look before you open. A HEAD request decides between the browser and the hash lookup, and it saves a 5-minute hang.
  6. Know what the VM can run. The VMs are Linux. A Windows .exe or an Office macro does not run there, so for those files the VirusTotal answer is the useful one.

References