Adventures in Freebernetes: More Linux bhyve-iour Plus CBSD

Part 4 of experiments in FreeBSD and Kubernetes: UEFI Booting + Installing CBSD

See all posts in this series

At the end of the previous post, I hit an issue while trying to create a Debian guest backed by a ZFS volume on the FreeBSD hypervisor. Installation from a virtual CD ISO onto the ZFS volume went fine, but when trying to boot from the ZFS virtual disk using grub-bhyve, the FreeBSD kernel panicked.

Booting with UEFI

After spending half a day waiting for the sysutils/bhyve-firmware port and its missing dependencies, which include the past-its-expiration-date Python 2.7, to compile, I try to boot the Debian guest using UEFI, but after a few failed attempts and more reading, it looks like converting an existing BIOS disk is iffy, especially as UEFI requires disk space of its own, which I didn’t preserve.

So I boot from the installer image again, but adding UEFI bootrom. (Note that UEFI boot does not require running bhyveload or grub-bhyve first.)

root@nucklehead:~ # bhyve -H -A -P \
-s 0,hostbridge \
-s 1,lpc \
-s 2,virtio-net,tap0 \
-s 3,virtio-blk,/dev/zvol/zroot/debianguest0 \
-l com1,stdio \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
-c 2 -m 1024M \
-s 31,ahci-cd,/vm/images/debian-10.6.0-amd64-netinst.iso \
view raw gistfile1.txt hosted with ❤ by GitHub

And install. And then run the same command, but without the ISO virtual device. And success!

Loading Linux 4.19.0-12-amd64 …
Loading initial ramdisk …
error: no suitable video mode found.
Booting in blind mode
rdmsr to register 0x140 on vcpu 0
rdmsr to register 0x140 on vcpu 1
rdmsr to register 0x64e on vcpu 0
rdmsr to register 0x34 on vcpu 0
Unhandled ps2 mouse command 0xe1
Unhandled ps2 mouse command 0x88
Debian GNU/Linux 10 debian ttyS0
debian login:
view raw gistfile1.txt hosted with ❤ by GitHub

Ok, the terminal is messed up because UEFI assumes a graphical output, so I effectively have a 3-line display. (Not joking.) So I get the IP address with ip addr show | grep inet (after cursing the lack of ifconfig) and ssh in.

So, yes, using a ZFS volume as the root disk for a Linux VM works (and should provide some disk I/O performance benefits, which I am not benchmarking), but you have to use UEFI instead of BIOS.

Grub Booting ZFS Redux

Meanwhile, on FreeBSD Twitter, I was connected with Chuck Tuffli, who gave me access to an update development version of grub-hyve to test with the grub-based Debian ZFS volume. Before I had overwritten my original Debian ZFS volume for UEFI booting, I had created a clone:

root@nucklehead:~ # zfs snapshot zvol/zroot/debianguest0@yesterday
root@nucklehead:~ # zfs clone zvol/zroot/debianguest0@yesterday zvol/zroot/debianguest1
root@nucklehead:~ # fdisk /dev/zvol/zroot/debianguest1
******* Working on device /dev/zvol/zroot/debianguest1 *******
parameters extracted from in-core disklabel are:
cylinders=261 heads=255 sectors/track=63 (16065 blks/cyl)
parameters to be used for BIOS calculations are:
cylinders=261 heads=255 sectors/track=63 (16065 blks/cyl)
Media sector size is 512
Warning: BIOS sector numbering starts with sector 1
Information from DOS bootblock is:
The data for partition 1 is:
sysid 131 (0x83),(Linux native)
start 2048, size 4190208 (2046 Meg), flag 80 (active)
beg: cyl 0/ head 32/ sector 33;
end: cyl 260/ head 243/ sector 47
The data for partition 2 is:
The data for partition 3 is:
The data for partition 4 is:
root@nucklehead:~ #
view raw gistfile1.txt hosted with ❤ by GitHub
root@nucklehead:/vm # ~/grub-bhyve
–root=/dev/zvol/zroot/debianguest1 \
–vm=debian1 \
–device-map=/vm/ \
view raw gistfile1.txt hosted with ❤ by GitHub
GNU GRUB version 2.05
Minimal BASH-like line editing is supported. For the first word, TAB
lists possible command completions. Anywhere else TAB lists possible
device or file completions.
grub> ls
(hd0,1) (proc)
grub> ls (hd0,1)/boot
config-4.19.0-11-amd64 vmlinuz-4.19.0-11-amd64 config-4.19.0-12-amd64 grub/ Sys initrd.img-4.19.0-11-amd64 vmlinuz-4.19.0-12-amd64 Syst initrd.img-4.19.0-12-amd64
grub> linux (hd0,1)/boot/vmlinuz-4.19.0-12-amd64 root=/dev/vda1
grub> [ vmlinuz-4.19.0-12-am 5.03MiB 100% 132.48MiB/s ]
grub> initrd (hd0,1)/boot/initrd.img-4.19.0-12-amd64
grub> [ initrd.img-4.19.0-12 24.65MiB 100% 138.44MiB/s ]
grub> boot
root@nucklehead:/vm # bhyve -H -A -P \
-s 0,hostbridge \
-s 1,lpc \
-s 2,virtio-net,tap0 \
-s 3,virtio-blk,/dev/zvol/zroot/debianguest1 \
-l com1,stdio \
-c 1 -m 1024M \
view raw gistfile1.txt hosted with ❤ by GitHub

This time grub-bhyve read the ZFS volume without making FreeBSD panic, and I was able to load and boot the kernel for Debian.

Hopefully this updated grub-bhyve version, which also adds XFS support, will get pushed to the ports tree soon.

Getting Ready for CBSD

The last two posts and change have demonstrated some of the permutations of possible bhyve guests and the amount of effort even simple proofs-of-concept involve.

The CBSD project exists to help manage that complexity. While it also supports FreeBSD jails and Xen VMs, I’m going to focus on bhyve. I’m building it from the sysutil/cbsd port, so now, among other packages, I now have sudo installed.

After the port finishes compiling, it reminds the user to finish installation by running env workdir="/usr/cbsd" /usr/local/cbsd/sudoexec/initenv and interactively fed it the following values:

—cut here —
# cbsd initenv preseed file for nucklehead host
# refer to the /usr/local/cbsd/share/initenv.conf
# for description.
—end of cut—
view raw initenv.conf hosted with ❤ by GitHub

Running CBSD

With that installed, I reboot the host to clear the network interfaces and VMs I had created and then try to start bhyve.

root@nucklehead:~ # cbsd bconstruct-tui
No kldloaded module: vmm
Please add vmm_load="YES" to /boot/loader.conf and
put kld_list="vmm if_tuntap if_bridge nmdm" into your /etc/rc.conf then reboot the host.
Press any key...

Given then the CBSD had added entries to both files, I’m assuming it skipped these because they had been loaded in the running kernel when I ran initenv, but it did not or could not test for cross-boot persistence.

After adding the entries and rebooting, I try again.

root@nucklehead:~ # cbsd bconstruct-tui
The current version requires tmux
Please run pkg install tmux  or make -C /usr/ports/sysutils/tmux install it.
===> tmux-3.1b has known vulnerabilities:
tmux-3.1b is vulnerable:
tmux — stack overflow in CSI parsing
1 problem(s) in 1 installed package(s) found.
=> Please update your ports tree and try again.
=> Note: Vulnerable ports are marked as such even if there is no update available.
=> If you wish to ignore this vulnerability rebuild with 'make DISABLE_VULNERABILITIES=yes'
*** Error code 1
make[1]: stopped in /usr/ports/sysutils/tmux
*** Error code 1
make: stopped in /usr/ports/sysutils/tmux
root@nucklehead:/usr/ports/sysutils/tmux # make DISABLE_VULNERABILITIES=yes inst
view raw gistfile1.txt hosted with ❤ by GitHub

That did not require installing half the ports tree this time, so when I ran cbsd bconstruct-tui again, it opened the text UI. Third time’s a charm.

Screen shot of CBSD's text-based user interface showing menu options for creating a new bhyve virtual machine
Yes, I need to find a better terminal type for these menus in my Chrome OS Linux terminal. Suggestions welcome.

The next post will focus on experiments with CBSD.

Sources / References

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

Blog at

Up ↑

%d bloggers like this: