Adventures in Freebernetes: bhyve My Guest

Part 2 of experiments in FreeBSD and Kubernetes: Creating your first guest

See all posts in this series

bhyve Background

bhyve (pronounced as “beehive”), FreeBSD’s type 2 (software) virtualization hypervisor, is pretty cool. It supports a number of operating systems as guest virtual machines (VMs), including not only the *BSD family, but also Linux, Illumos, and even Windows NT.

Apropos reminder of Mac OS X’s FreeBSD roots, which are still tangled with its parent’s

CBSD will simplify managing your bhyve VMs, but first I want to create some manually, more or less, to get a taste of what goes on under the hood.

My First Guest

First, I’m going to create a simple FreeBSD 11.4 guest. I’ve downloaded the 11.4 ISO image. The standard FreeBSD 13.0-CURRENT userworld installation has the bhyve command-line interface (CLI) in /usr/sbin and also ships with a wrapper script, /usr/share/examples/bhyve/vmrun.sh.

root@nucklehead:/vm # kldload vmm
root@nucklehead:/vm # ifconfig bridge0 create
root@nucklehead:/vm # ifconfig tap0 create
root@nucklehead:/vm # sysctl net.link.tap.up_on_open=1
net.link.tap.up_on_open: 0 -> 1
root@nucklehead:/vm # ifconfig bridge0 addm em0 addm tap0
root@nucklehead:/vm # ifconfig bridge0 up
root@nucklehead:/vm # dhclient bridge0
DHCPREQUEST on bridge0 to 255.255.255.255 port 67
DHCPACK from 192.168.0.1
bound to 192.168.0.11 — renewal in 43200 seconds.
root@nucklehead:/vm # truncate -s 2G guest.img
root@nucklehead:/vm # sh /usr/share/examples/bhyve/vmrun.sh -i \
-c 1 -m 1024M \
-t tap0 -d guest.img \
-I images/FreeBSD-11.4-RELEASE-amd64-bootonly.iso \
threedayoldfish
view raw gistfile1.txt hosted with ❤ by GitHub

At this point, it should boot the FreeBSD installer/LiveCD image. If you let it go, it starts the installer or it may ask for console type first. (FYI, cons25w works best when I ssh to the FreeBSD host from my Chromebook Linux terminal.)

nucklehead gets its network configuration from my router’s DHCP service. bridge0 will also need an IP address. I ran dhclient bridge0 to get another IP address on my LAN from router via DHCP. em0 is the NUC’s ethernet interface.

Screen shot of FreeBSD installer progress meter

Once the installation finishes, you can use the Shell option to exit the VM so you can boot off the disk image.

When finished, type 'exit' to return to the installer.
# halt -p
Oct 29 00:09:01 halt: halted by root
Oct 29 00:09:01 syslogd: exiting on signal 15
Waiting (max 60 seconds) for system process `vnlru' to stop… done
Waiting (max 60 seconds) for system process `bufdaemon' to stop… done
Waiting (max 60 seconds) for system process `syncer' to stop…
Syncing disks, vnodes remaining… 0 0 done
All buffers synced.
Uptime: 42s
acpi0: Powering system off
root@nucklehead:/vm # sh /usr/share/examples/bhyve/vmrun.sh \
-c 1 -m 1024M \
-t tap0 -d guest.img \
threedayoldfish
view raw gistfile1.txt hosted with ❤ by GitHub

โ€ฆ

Starting sshd.
Starting sendmail_submit.
Oct 29 00:33:11 threedayoldfish sm-mta[649]: My unqualified host name (threedayoldfish) unknown; sleeping for retry
Oct 29 00:34:11 threedayoldfish sm-mta[649]: unable to qualify my own domain name (threedayoldfish) — using short name
Starting sendmail_msp_queue.
Oct 29 00:34:11 threedayoldfish sm-msp-queue[652]: My unqualified host name (threedayoldfish) unknown; sleeping for retry
Oct 29 00:35:11 threedayoldfish sm-msp-queue[652]: unable to qualify my own domain name (threedayoldfish) — using short name
Starting cron.
Starting background file system checks in 60 seconds.
Thu Oct 29 00:35:11 UTC 2020
FreeBSD/amd64 (threedayoldfish) (ttyu0)
login:
view raw gistfile1.txt hosted with ❤ by GitHub

The vmm kernel driver keeps a device file for each running VM:

root@nucklehead:/vm # ls -l /dev/vmm
total 0
crw——- 1 root wheel 0x5a Oct 29 01:23 threedayoldfish
root@nucklehead:/vm #
view raw gistfile1.txt hosted with ❤ by GitHub

If I shut down the VM, the vmm driver should remove it. If the device file persists after you think you have shut down the VM or bhyve exited with an error, you can remove the running VM with bhyvectl --destroy --vm=MYVMNAME. If no VMs remain, vmm.ko should also remove the directory /dev/vmm.

root@threedayoldfish:~ # halt -p
Oct 29 01:26:13 threedayoldfish halt: halted by root
Oct 29 01:26:13 threedayoldfish syslogd: exiting on signal 15
Waiting (max 60 seconds) for system process `vnlru' to stop… done
Waiting (max 60 seconds) for system process `bufdaemon' to stop… done
Waiting (max 60 seconds) for system process `syncer' to stop…
Syncing disks, vnodes remaining… 1 0 done
All buffers synced.
Uptime: 2m27s
acpi0: Powering system off
root@nucklehead:/vm # ls -l /dev/vmm
ls: /dev/vmm: No such file or directory
root@nucklehead:/vm #
view raw gistfile1.txt hosted with ❤ by GitHub

Linux Guest

bhyve also supports Linux for guest VMs.

Note that Linux VMs require the grub-bhyve utility from the sysutils/grub2-bhyve port. If you just installed FreeBSD on your hypervisor and have not installed any ports yet, you will probably need to wait awhile (all night) while the dependencies compile. Plan accordingly.

High-Maintenance FreeBSD Guest

While we wait for the grub-bhyve port and its dependencies to compile, let’s see what it takes to run a FreeBSD guest without the vmrun.sh helper script.

If we haven’t rebooted since our first experiment and don’t have any other VMs running, we shouldn’t need to (re)create the network interfaces. We will create a new img file, though.

I tried to create a FreeBSD VM building off the bhyve(8) man page but I kept getting errors like virtual machine cannot be booted. So I figured the vmrun.sh wrapper script was responsible for more magic than just the bhyve command line. I made a copy, added set -x to capture the actual commands it was running, and then spent 15 minutes trying to figure out why my file handle redirects weren’t working as intended. Oh. I forgot root defaults to /bin/csh on FreeBSD.

Anyway, yes, the script was doing more. There’s this thing called a boot loader, seeโ€ฆ

root@nucklehead:/vm # truncate -s 2G manual.img
root@nucklehead:/vm # bhyveload -c stdio -m 1024M \
-d images/FreeBSD-11.4-RELEASE-amd64-bootonly.iso \
highmaintenance
view raw gistfile1.txt hosted with ❤ by GitHub

The loader menu appearsโ€ฆ

______ ____ _____ _____
| ____| | _ \ / ____| __ \
| |___ _ __ ___ ___ | |_) | (___ | | | |
| ___| '__/ _ \/ _ \| _ < \___ \| | | |
| | | | | __/ __/| |_) |____) | |__| |
| | | | | | || | | |
|_| |_| \___|\___||____/|_____/|_____/ “` `
s` `…..—…….–.“` -/
+============Welcome to FreeBSD===========+ +o .–` /y:` +.
| | yo`:. ๐Ÿ˜ฎ `+-
| 1. Boot Multi User [Enter] | y/ -/` -o/
| 2. Boot [S]ingle User | .- ::/sy+:.
| 3. [Esc]ape to loader prompt | / `– /
| 4. Reboot | `: :`
| | `: :`
| Options: | / /
| 5. [K]ernel: kernel (1 of 2) | .- -.
| 6. Configure Boot [O]ptions… | — -.
| | `:` `:`
| | .– `–.
| | .—…..—-.
+=========================================+
/boot/kernel/kernel text=0x1553188 data=0x146038+0x4cdeb0 syms=[0x8+0x16e2d0+0x8+0x187067]
Booting…
root@nucklehead:/vm # bhyve -H -A -P \
-c 1 -m 1024M \
-s 0,hostbridge \
-s 1,lpc \
-s 2:0,virtio-net,tap0 \
-s 3,virtio-blk,/vm/manual.img \
-s 31,ahci-cd,/vm/images/FreeBSD-11.4-RELEASE-amd64-bootonly.iso \
-l com1,stdio \
highmaintenance
view raw gistfile1.txt hosted with ❤ by GitHub

And if all goes well, it should boot the installer/live cd image and off you go.

A couple tips:

  • When creating FreeBSD guests, you will always need to run the bhyveload command first.
  • Make sure the RAM size in the bhyveload and bhyve -m arguments match, or you will get the error Unable to setup memory (22). (Ask me how I know.)
  • When you’re done with the VM, you may have to clean up manually with bhyvectl --destroy --vm=highmaintenance.

The -s #,... options are my favorite. They create virtual PCI slots. (Be sure to check out the bhyve man page.) I am now having flashbacks to setting up CD burners on my home FreeBSD desktop a bazillion years ago.


The next post in this series will cover creating your Linux guest manually with bhyve. By then, with luck, your /usr/ports tree should have finished compiling.

Sources / References

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

Blog at WordPress.com.

Up ↑

%d bloggers like this: