Part 2 of experiments in FreeBSD and Kubernetes: Creating your first guest

See all posts in this series

bhyve Background

bhyve (pronounced as “beehive”), FreeBSD’s type 2 (software) virtualization hypervisor, is pretty cool. It supports a number of operating systems as guest virtual machines (VMs), including not only the *BSD family, but also Linux, Illumos, and even Windows NT.

Apropos reminder of Mac OS X’s FreeBSD roots, which are still tangled with its parent’s

CBSD will simplify managing your bhyve VMs, but first I want to create some manually, more or less, to get a taste of what goes on under the hood.

My First Guest

First, I’m going to create a simple FreeBSD 11.4 guest. I’ve downloaded the 11.4 ISO image. The standard FreeBSD 13.0-CURRENT userworld installation has the bhyve command-line interface (CLI) in /usr/sbin and also ships with a wrapper script, /usr/share/examples/bhyve/vmrun.sh.

At this point, it should boot the FreeBSD installer/LiveCD image. If you let it go, it starts the installer or it may ask for console type first. (FYI, cons25w works best when I ssh to the FreeBSD host from my Chromebook Linux terminal.)

nucklehead gets its network configuration from my router’s DHCP service. bridge0 will also need an IP address. I ran dhclient bridge0 to get another IP address on my LAN from router via DHCP. em0 is the NUC’s ethernet interface.

Screen shot of FreeBSD installer progress meter


Once the installation finishes, you can use the Shell option to exit the VM so you can boot off the disk image.

…

The vmm kernel driver keeps a device file for each running VM:

If I shut down the VM, the vmm driver should remove it. If the device file persists after you think you have shut down the VM or bhyve exited with an error, you can remove the running VM with bhyvectl --destroy --vm=MYVMNAME. If no VMs remain, vmm.ko should also remove the directory /dev/vmm.

Linux Guest

bhyve also supports Linux for guest VMs.

Note that Linux VMs require the grub-bhyve utility from the sysutils/grub2-bhyve port. If you just installed FreeBSD on your hypervisor and have not installed any ports yet, you will probably need to wait awhile (all night) while the dependencies compile. Plan accordingly.

High-Maintenance FreeBSD Guest

While we wait for the grub-bhyve port and its dependencies to compile, let’s see what it takes to run a FreeBSD guest without the vmrun.sh helper script.

If we haven’t rebooted since our first experiment and don’t have any other VMs running, we shouldn’t need to (re)create the network interfaces. We will create a new img file, though.

I tried to create a FreeBSD VM building off the bhyve(8) man page but I kept getting errors like virtual machine cannot be booted. So I figured the vmrun.sh wrapper script was responsible for more magic than just the bhyve command line. I made a copy, added set -x to capture the actual commands it was running, and then spent 15 minutes trying to figure out why my file handle redirects weren’t working as intended. Oh. I forgot root defaults to /bin/csh on FreeBSD.

Anyway, yes, the script was doing more. There’s this thing called a boot loader, see…

The loader menu appears…

And if all goes well, it should boot the installer/live cd image and off you go.

A couple tips:

  • When creating FreeBSD guests, you will always need to run the bhyveload command first.
  • Make sure the RAM size in the bhyveload and bhyve -m arguments match, or you will get the error Unable to setup memory (22). (Ask me how I know.)
  • When you’re done with the VM, you may have to clean up manually with bhyvectl --destroy --vm=highmaintenance.

The -s #,... options are my favorite. They create virtual PCI slots. (Be sure to check out the bhyve man page.) I am now having flashbacks to setting up CD burners on my home FreeBSD desktop a bazillion years ago.

The next post in this series will cover creating your Linux guest manually with bhyve. By then, with luck, your /usr/ports tree should have finished compiling.

Sources / References