Persistent VMs on pfSense with vm-bhyve
Bottom Line: Configure your VMs to start and run automatically on pfSense with vm-bhyve.
If you’re trying to get a Linux VM running under bhyve on pfSense, I strongly recommend that you start with my first post on the topic. Once you have things running interactively, it’s time to try to get the VM to start and run automatically.
vm-bhyve is designed to help make the process a little easier, and it’s available in the default pfSense repo, which is highly convenient.
First off, it’s important to realize that pfSense apparently does not use
the standard FreeBSD rc init
I would have saved a lot of time had I realized this earlier, as it means that
the default FreeBSD instructions on this
topic, which advise adding a number of settings to
/etc/rc.conf, won’t work.
Further, at boot time, you can automatically run scripts in
/usr/local/etc/rc.d/, but they
must end in
.sh and be executable.
With these two facts in mind, the rest wasn’t too difficult. As a reminder, I’m
bash as my shell, and all commands below are being run as root (I’m
$ instead of
# in codeblocks below for better markdown syntax
You’ll probably want to keep vm-bhyve’s GitHub page open to references its documentation.
pkg install vm-bhyve
- Using the web interface, configure pfSense to load the necessary kernel
modules on boot by adding the following to
/boot/loader.conf(following the official FreeBSD instructions, though I didn’t need
vmm_load="YES" if_bridge_load="YES" if_tap_load="YES"
- Configure pfSense to bring up your TAP interface on boot:
Open TAP on boot for vm-bhyve
- This is probably a good time to reboot, which should load / activate the above settings and make sure they are working
vm-bhyve’s rc script to something pfSense will run:
$ ln -s /usr/local/etc/rc.d/vm /usr/local/etc/rc.d/vm.sh
vm-bhyve’s config. The contents of mine are the following lines, yours may vary:
vm_enable="YES" vm_dir="zfs:pfSense/vm" vm_list="nixos0" vm_delay="5"
- You may need to remove the remnants of the zfs virtual drive created in the
$ zfs destroy pfSense/vm/nixos0
- Create and populate vm-bhyve’s directory structure:
$ vm init
- Copy the UEFI file we downloaded in the last post to a place that
vm-bhyvewill look for it:
$ cp BHYVE_UEFI.fd /pfSense/vm/.config/
- Next you need to configure the VM
- Start by looking through a few of the samples:
- Next, I downloaded and then edited the default config:
$ curl 'https://raw.githubusercontent.com/churchers/vm-bhyve/master/sample-templates/default.conf' > /pfSense/vm/.templates/nixos.conf $ vim /pfSense/vm/.templates/nixos.conf
My config ultimately ended up looking like this:
loader="uefi-custom" cpu=1 memory=1024M network0_type="virtio-net" network0_switch="public" disk0_type="virtio-blk" disk0_dev="sparse-zvol" disk0_name="nixos0" disk0_size="16G" graphics="no"
- I had trouble with the vm-bhyve console until I configured it to run in
tmux; if you’re not a tmux user maybe skip this:
$ vm set console="tmux"
- Create a “manually” managed switch (since we’ve configured it in pfSense in
the prior post, and pfSense will manage it)
$ vm switch create -t manual -b bridge0 public
- Create a VM named
nixos0based on your customized
$ vm create -t nixos nixos0
vm-bhyveto download the installer ISO:
$ vm iso https://releases.nixos.org/nixos/22.11/nixos-22.11.2979.47c00341629/nixos-minimal-22.11.2979.47c00341629-x86_64-linux.iso
- Install in the foreground using your currently active terminal session. Note
that immediately after running this command I had to hold down the down
arrow key and keep tapping it for 15 seconds or so, during which the
entire SSH session seemed be frozen. Afterwards, it comes up with the option
to go into
Accessibilityand redirect its output to the serial console, just like in the last post.
$ vm install -f nixos0 nixos-minimal-22.11.2979.47c00341629-x86_64-linux.iso
- At this point, you should be able to get a shell,
sudo suto elevate,
systemctl start sshd,
passwdto set a root password,
ip addrto get your IP address, and you’re off to the races! (Don’t forget to add
boot.kernelParams = [ "console=ttyS0" ];if using NixOS, and you’ll need SSH access configured so you can access the machine once it’s booting in the background)
- Once you’ve completed your installation, see if your VM comes up (make sure
to give it a minute):
$ vm poweroff nixos0 $ vm start -f nixos0
- If that works, try SSH access. If that also works, try rebooting pfSense – your VM should automatically start in the background a few seconds after bootup is complete, at which point you should be able to connect via SSH (check your pfSense logs for the IP address, for which you might want to add a DHCP reservation at this point).
- If you’ve made it to this point, everything seems to be working.
Congratulations! All that’s left is to make a snapshot of your working VM,
which I suppose you should be able to
zfs sendto another machine, or to which you can roll back if something goes wrong in the future:
$ vm stop nixos0 $ vm snapshot [email protected] $ vm info nixos0 vm info nixos0 ------------------------ Virtual Machine: nixos0 ------------------------ state: stopped datastore: default loader: uefi-custom uuid: 0b49b710-e2ce-11ed-b922-00e0672a504a uefi: default cpu: 1 memory: 1024M network-interface number: 0 emulation: virtio-net virtual-switch: public fixed-mac-address: 58:9c:fc:03:dc:eb fixed-device: - virtual-disk number: 0 device-type: sparse-zvol emulation: virtio-blk options: - system-path: /dev/zvol/pfSense/vm/nixos0/nixos0 bytes-size: 17179869184 (16.000G) bytes-used: 1650454528 (1.537G) snapshots pfSense/vm/[email protected] 0 Mon Apr 24 15:25 2023 pfSense/vm/nixos0/[email protected] 0 Mon Apr 24 15:25 2023
I hope you’ve found this useful – I still have a lot to learn, so if you see any major missteps or recommendations for improvement please let me know in the comments.