Part 3 of experiments in FreeBSD and Kubernetes: Linux guests

See all posts in this series

Prep Work

In the previous post, we started compiling the sysutils/grub2-bhyve port, required for running Linux guests with bhyve. We also need the ISO 9660 image for a Linux installer (I’m using Arch Linux).

grub-bhyve is the bhyve boot loader for Linux images. Just like we realized in the previous post that we needed to run bhyveload before we could run a FreeBSD guest, we need grub-bhyve so we can boot Linux guests.

We’re assuming you’ve already created the virtual network interfaces (see the previous post) and no other VMs currently exist.

Booting Linux… Maybe

This should bring up the GRand Unified Bootloader (grub) menu. Now we need to figure out where Arch Linux hides the vmlinuz kernel and load that.

And now we can boot the loaded kernel.

Hmmm, that didn’t work. Some searching turns up the fix: add the ISO image’s label to the kernel arguments. I also made grub-bhyve barf at least once before I got it all working. (Arch names its ISOs ARCH_YYYYMM with the datestamp of the release version.)

And again run the bhyve command to try to boot. Victory!

https://gist.github.com/kbruner/1a9bac1d62d250d88eec4829f0c71524

I just gave the whole disk to the root partition because lazy.

Next we format and mount the root partition, and because the network interface was configured via DHCP at boot, we can start installing.

Ohai perl.

Then set up time zone, locale, network configuration, etc., and power off the VM.

I take the cd0 entry out of the device.map file and run grub-bhyve again, giving it hd0 as the root device: grub-bhyve -m device.map -M 1024M -r hd0 arch

Tada!

A Guest on ZFS

Now that I have reacquainted myself with exactly how minimal Arch Linux actually is (-bash: which: command not found – really???), I’ll use Debian to test giving a VM its own ZFS volume instead of using a img file for its virtual disk.

The Debian image already has grub installed and pre-configured, so all we have to do is choose “Install.”

That option only loads the kernel with the installer arguments. We still have to boot it, which should drop us immediately into the interactive installer.

Screen shot of the text-based Debian installer


Nobody Panic, But…

When the Debian installation finished, I exited and ran grub-bhyve -m debian-device.map -M 1024M -r hd0 debian and then at the grub menu, run ls … FreeBSD panicked. I rebooted tried again, same outcome.

Fortunately dmesg was capturing the error.

Some searching turned up this FAQ, which explains that the witness(4) lock diagnostic watches for potential deadlocks in the kernel. (After reading The Design and Implementation of the FreeBSD Operating System about 10 years ago, the word “mutex” immediately makes me think of the FreeBSD kernel.)

witness(4) is enabled by default in CURRENT kernels for development and debugging purposes, and as I’m running a build of FreeBSD 13.0-CURRENT from last week, that fits.

Anyway, I am now down the FreeBSD maintainers rabbit hole, so I will throw this post out there for now. The next post should have resolution.

The next post in this series, as we work toward, yes, someday actually running Kubernetes on FreeBSD, will hopefully show a working Linux-with-ZFS-disk VM and then look at CBSD, which helps manage your bhyve VMs.

Sources / References