Part 2 of experiments in FreeBSD and Kubernetes: Creating your first guest
bhyve (pronounced as “beehive”), FreeBSD’s type 2 (software) virtualization hypervisor, is pretty cool. It supports a number of operating systems as guest virtual machines (VMs), including not only the *BSD family, but also Linux, Illumos, and even Windows NT.
TIL MacOS's HyperKit is built on xhyve, a port of FreeBSD bhyve. Which doesn't surprise me, but cool.— Karen Bruner (@fuzzyKB) October 26, 2020
You can run Docker on your Mac thanks to FreeBSD. Of course, you can run your Mac thanks to FreeBSD, too.
Apropos reminder of Mac OS X’s FreeBSD roots, which are still tangled with its parent’s
CBSD will simplify managing your bhyve VMs, but first I want to create some manually, more or less, to get a taste of what goes on under the hood.
My First Guest
First, I’m going to create a simple FreeBSD 11.4 guest. I’ve downloaded the 11.4 ISO image. The standard FreeBSD 13.0-CURRENT
userworld installation has the
bhyve command-line interface (CLI) in
/usr/sbin and also ships with a wrapper script,
At this point, it should boot the FreeBSD installer/LiveCD image. If you let it go, it starts the installer or it may ask for console type first. (FYI,
cons25w works best when I ssh to the FreeBSD host from my Chromebook Linux terminal.)
nucklehead gets its network configuration from my router’s DHCP service.
bridge0 will also need an IP address. I ran
dhclient bridge0 to get another IP address on my LAN from router via DHCP.
em0 is the NUC’s ethernet interface.
Once the installation finishes, you can use the
Shell option to exit the VM so you can boot off the disk image.
vmm kernel driver keeps a device file for each running VM:
If I shut down the VM, the
vmm driver should remove it. If the device file persists after you think you have shut down the VM or
bhyve exited with an error, you can remove the running VM with
bhyvectl --destroy --vm=MYVMNAME. If no VMs remain,
vmm.ko should also remove the directory
bhyve also supports Linux for guest VMs.
Note that Linux VMs require the
grub-bhyve utility from the
sysutils/grub2-bhyve port. If you just installed FreeBSD on your hypervisor and have not installed any ports yet, you will probably need to wait awhile (all night) while the dependencies compile. Plan accordingly.
High-Maintenance FreeBSD Guest
While we wait for the
grub-bhyve port and its dependencies to compile, let’s see what it takes to run a FreeBSD guest without the
vmrun.sh helper script.
If we haven’t rebooted since our first experiment and don’t have any other VMs running, we shouldn’t need to (re)create the network interfaces. We will create a new
img file, though.
I tried to create a FreeBSD VM building off the
bhyve(8) man page but I kept getting errors like
virtual machine cannot be booted. So I figured the
vmrun.sh wrapper script was responsible for more magic than just the
bhyve command line. I made a copy, added
set -x to capture the actual commands it was running, and then spent 15 minutes trying to figure out why my file handle redirects weren’t working as intended. Oh. I forgot
root defaults to
/bin/csh on FreeBSD.
Oh, since when did root's shell default to csh on FreeBSD?— Karen Bruner (@fuzzyKB) October 29, 2020
(For all I know, it's been "forever," but I forgot. But wondering why my patently-Bourne shell multi-file handle redirection wasn't working was making me nuts, until I finally thought to check...)
Anyway, yes, the script was doing more. There’s this thing called a boot loader, see…
The loader menu appears…
And if all goes well, it should boot the installer/live cd image and off you go.
A couple tips:
- When creating FreeBSD guests, you will always need to run the
- Make sure the RAM size in the
-marguments match, or you will get the error
Unable to setup memory (22). (Ask me how I know.)
- When you’re done with the VM, you may have to clean up manually with
bhyvectl --destroy --vm=highmaintenance.
-s #,... options are my favorite. They create virtual PCI slots. (Be sure to check out the
bhyve man page.) I am now having flashbacks to setting up CD burners on my home FreeBSD desktop a bazillion years ago.
The next post in this series will cover creating your Linux guest manually with
bhyve. By then, with luck, your
/usr/ports tree should have finished compiling.