Part 3 of experiments in FreeBSD and Kubernetes: Linux guests
In the previous post, we started compiling the
sysutils/grub2-bhyve port, required for running Linux guests with bhyve. We also need the ISO 9660 image for a Linux installer (I’m using Arch Linux).
grub-bhyve is the bhyve boot loader for Linux images. Just like we realized in the previous post that we needed to run
bhyveload before we could run a FreeBSD guest, we need
grub-bhyve so we can boot Linux guests.
We’re assuming you’ve already created the virtual network interfaces (see the previous post) and no other VMs currently exist.
Booting Linux… Maybe
This should bring up the GRand Unified Bootloader (grub) menu. Now we need to figure out where Arch Linux hides the
vmlinuz kernel and load that.
And now we can boot the loaded kernel.
Hmmm, that didn’t work. Some searching turns up the fix: add the ISO image’s label to the kernel arguments. I also made grub-bhyve barf at least once before I got it all working. (Arch names its ISOs
ARCH_YYYYMM with the datestamp of the release version.)
I MISSED YOU SOOOO MUCH, FREEBSD CORE FILES!!!! ❤️🔥 pic.twitter.com/ORmzIRqrLi— Karen Bruner (@fuzzyKB) October 31, 2020
And again run the
bhyve command to try to boot. Victory!
I just gave the whole disk to the root partition because lazy.
Next we format and mount the root partition, and because the network interface was configured via DHCP at boot, we can start installing.
Then set up time zone, locale, network configuration, etc., and power off the VM.
I take the
cd0 entry out of the
device.map file and run
grub-bhyve again, giving it
hd0 as the root device:
grub-bhyve -m device.map -M 1024M -r hd0 arch
A Guest on ZFS
Now that I have reacquainted myself with exactly how minimal Arch Linux actually is (
-bash: which: command not found – really???), I’ll use Debian to test giving a VM its own ZFS volume instead of using a
img file for its virtual disk.
The Debian image already has grub installed and pre-configured, so all we have to do is choose “Install.”
That option only loads the kernel with the installer arguments. We still have to boot it, which should drop us immediately into the interactive installer.
Nobody Panic, But…
When the Debian installation finished, I exited and ran
grub-bhyve -m debian-device.map -M 1024M -r hd0 debian and then at the grub menu, run
ls … FreeBSD panicked. I rebooted tried again, same outcome.
dmesg was capturing the error.
Some searching turned up this FAQ, which explains that the
witness(4) lock diagnostic watches for potential deadlocks in the kernel. (After reading The Design and Implementation of the FreeBSD Operating System about 10 years ago, the word “mutex” immediately makes me think of the FreeBSD kernel.)
witness(4) is enabled by default in
CURRENT kernels for development and debugging purposes, and as I’m running a build of FreeBSD 13.0-CURRENT from last week, that fits.
Anyway, I am now down the FreeBSD maintainers rabbit hole, so I will throw this post out there for now. The next post should have resolution.
The next post in this series, as we work toward, yes, someday actually running Kubernetes on FreeBSD, will hopefully show a working Linux-with-ZFS-disk VM and then look at CBSD, which helps manage your bhyve VMs.