Part 4 of experiments in FreeBSD and Kubernetes: UEFI Booting + Installing CBSD
At the end of the previous post, I hit an issue while trying to create a Debian guest backed by a ZFS volume on the FreeBSD hypervisor. Installation from a virtual CD ISO onto the ZFS volume went fine, but when trying to boot from the ZFS virtual disk using grub-bhyve, the FreeBSD kernel panicked.
I'm trying to create a bhyve Debian guest using a ZFS drive and FreeBSD keeps crashing... Oops.
— Karen Bruner (@fuzzyKB) October 31, 2020
How's your Saturday?
Booting with UEFI
After spending half a day waiting for the sysutils/bhyve-firmware port and its missing dependencies, which include the past-its-expiration-date Python 2.7, to compile, I try to boot the Debian guest using UEFI, but after a few failed attempts and more reading, it looks like converting an existing BIOS disk is iffy, especially as UEFI requires disk space of its own, which I didn’t preserve.
I'm still not wrong in thinking UEFI was just a plot fueled by Microsoft to increase the pain of installing OSS operating systems on PCs, right?
— Karen Bruner (@fuzzyKB) November 2, 2020
So I boot from the installer image again, but adding UEFI bootrom. (Note that UEFI boot does not require running bhyveload or grub-bhyve first.)
And install. And then run the same command, but without the ISO virtual device. And success!
Ok, the terminal is messed up because UEFI assumes a graphical output, so I effectively have a 3-line display. (Not joking.) So I get the IP address with ip addr show | grep inet (after cursing the lack of ifconfig) and ssh in.
So, yes, using a ZFS volume as the root disk for a Linux VM works (and should provide some disk I/O performance benefits, which I am not benchmarking), but you have to use UEFI instead of BIOS.
Grub Booting ZFS Redux
Meanwhile, on FreeBSD Twitter, I was connected with Chuck Tuffli, who gave me access to an update development version of grub-hyve to test with the grub-based Debian ZFS volume. Before I had overwritten my original Debian ZFS volume for UEFI booting, I had created a clone:
This time grub-bhyve read the ZFS volume without making FreeBSD panic, and I was able to load and boot the kernel for Debian.
Hopefully this updated grub-bhyve version, which also adds XFS support, will get pushed to the ports tree soon.
Getting Ready for CBSD
The last two posts and change have demonstrated some of the permutations of possible bhyve guests and the amount of effort even simple proofs-of-concept involve.
The CBSD project exists to help manage that complexity. While it also supports FreeBSD jails and Xen VMs, I’m going to focus on bhyve. I’m building it from the sysutil/cbsd port, so now, among other packages, I now have sudo installed.
After the port finishes compiling, it reminds the user to finish installation by running env workdir="/usr/cbsd" /usr/local/cbsd/sudoexec/initenv and interactively feed it the following values:
Running CBSD
With that installed, I reboot the host to clear the network interfaces and VMs I had created and then try to start bhyve.
root@nucklehead:~ # cbsd bconstruct-tui
No kldloaded module: vmm
Please add vmm_load="YES" to /boot/loader.conf and
put kld_list="vmm if_tuntap if_bridge nmdm" into your /etc/rc.conf then reboot the host.
Press any key...
Given then the CBSD had added entries to both files, I’m assuming it skipped these because they had been loaded in the running kernel when I ran initenv, but it did not or could not test for cross-boot persistence.
After adding the entries and rebooting, I try again.
root@nucklehead:~ # cbsd bconstruct-tui
The current version requires tmux
Please run pkg install tmux or make -C /usr/ports/sysutils/tmux install it.
That did not require installing half the ports tree this time, so when I ran cbsd bconstruct-tui again, it opened the text UI. Third time’s a charm.
Yes, I need to find a better terminal type for these menus in my Chrome OS Linux terminal. Suggestions welcome.
The next post will focus on experiments with CBSD.