Adventures in Freebernetes: Installing FreeBSD

Part 1 of experiments in FreeBSD and Kubernetes

See all posts in this series

Read the introduction to this series for background. This post documents getting a working, bootable installation of FreeBSD 13.0-CURRENT on an older NUC PC.

This post is roughly divided into two parts: the rather-chatty story version and then the actual step-by-step instructions for how I finally got a successful installation.

Skip straight to the step-by-step instructions for a custom, manual FreeBSD 13.0-CURRENT ZFS-on-root + EFI boot install

Day 1: The Adventure Begins

The adventure begins on Twitter. And on my floor.

CBSD strongly recommends using a ZFS filesystem. My experience with FreeBSD also mostly predates widespread production use of ZFS. And of course, modern UEFI booting is always super-fun when you’re not Windows. Brad had made the NUC dual-boot Linux along with the original Windows 10 installation. Even though I avoid Windows whenever possible, for some weird, OS-pack-rat reason, I decided I would try to preserve the Windows installation if possible.

Picture of the FreeBSD installer's text-based UI with blue background, asking if you want to install, run a shell, or launch the Live CD
The FreeBSD installer: The happiest blue screen

The only option in the FreeBSD installer for a custom ZFS installation is via shell and use gpart. (The stalwart FreeBSD fdisk is available, but none of the guides I saw recommended using it. I’m not sure if it has to do with UEFI support or what, but I didn’t dig.)

Image of screen with an interactive shell running gpart commands to create and view disk partitions
Fun with gpart, said no one ever

I went through one guide and the installation seemingly completed successfully but… it wouldn’t boot. I messed with the efi stuff a bit; nothing. Trying to bypass UEFI and use the boot loader from the USB stick; even though the boot loader could read the ZFS pool that I had installed just fine, I couldn’t get it to boot. It failed with “unknown file system.” Not sure if I missed some settings to enable ZFS-on-root that would normally be passed in /boot/loader.conf.

At this point, based on bits of what I’d seen in search results and the age of the NUC’s BIOS (4+ years), it seemed like a good idea to upgrade that, so I did. Still, no dice. I decided to sleep on it and if I still couldn’t boot, try a fresh install in the morning.

Even though I didn’t have a successful boot, I had gotten reacquainted with gpart and I had messed around with ZFS enough to get familiar with some of the basic commands.

Day 1, The Sequel: The Adventure Begins Again

The BIOS upgrade didn’t cure the problem, so I decided to start from scratch with a different guide. I ended up working from this how-to, even though it uses a ZFS pool on mirrored disks with geli encryption.. I only have one disk (which is partitioned), and I’m not interested in geli for the time being, so I just stripped out the mirroring- and encryption-specific steps.

I restarted the installer, used the shell option for disk partition, deleted the partitions I had previously created, and ended up with this partition table:

> 34 1000215149 nvd0 GPT (477G)
34 4096 nvd0p5 efi (2.0M)
4130 1046494 – free – (511M)
1050624 32768 nvd0p2 ms-reserved (16M)
1083392 499972894 nvd0p3 ms-basic-data (238G)
501056286 512 nvd0p1 freebsd-boot (256K)
501056798 738 – free – (369K)
501057536 1751040 nvd0p4 ms-recovery (855M)
502808576 16777216 nvd0p6 freebsd-swap (8.0G)
519585792 419430400 nvd0p7 freebsd-zfs (200G)
939016192 61198991 – free – (29G)
view raw gistfile1.txt hosted with ❤ by GitHub

One important change: I did have to increase the source example’s sizes for the efi and freebsd-boot partitions, presumably because they were too small for the files I had to copy in. No big deal. The freebsd-boot partition went in there in case I kept having issues with UEFI and tried to make legacy boot work instead.

Fortunately, the how-to also warned about this little potential issue with /dev/null:

But this time around, success!

I still didn’t figure out how to boot from ZFS-on-root from the boot loader (I tried that before letting UEFI go) so I’m sure I’m missing something. It’s been years since I’ve had to spend any time with the FreeBSD loader, and the ZFS and UEFI stuff in it is all new to me. I may go back to figure it out later, or not.

Anyway, my FreeBSD 13.0-CURRENT PC is all humming. Next I’ll work on getting bhyve (probably via CBSD) up and running. I’ve never touched bhyve before, so I’m curious to see what that looks like.

Step-by-Step Manual ZFS-on-root FreeBSD Install

USE AT YOUR OWN RISK. Seriously, I have no idea what I’m doing.

Derived from this how-to, which also includes geli disk encryption and mirrored disks.

My system and use case (if yours differs you may have to make alterations or use a different tutorial completely):

  • a single drive with existing partitions I didn’t want to delete (no second disk so no mirroring)
  • no geli encryption
  • disk already uses GPT partitioning. If it uses Master Boot Record and you want to keep the existing partitions, you’ll need to find another tutorial!
  • x64 CPU

For reference, my PC’s disk device is nvd0 and I named my ZFS pool zroot.

  1. Boot the FreeBSD installer as usual
  2. When you get to the disk partition menu, choose the Shell option
  3. Using gpart, carefully delete any partitions you no longer want
  4. Create the new FreeBSD partitions and ZFS file system
# Remember to replace nvd0 with your disk's device name
# Legacy boot partition
gpart add -t freebsd-boot -s 256k -l boot nvd0
# UEFI
gpart add -t efi -s 2m -l efi nvd0
newfs_msdos /dev/gpt/efi
# Swap. Adjust size (-s) as desired.
gpart add -t freebsd-swap -l swap -a 1m -s 8G nvd0
# ZFS. Adjust size as needed or omit -s flag and arg to use remaining disk
gpart add -t freebsd-zfs -l zroot -a 1m -s 200G nvd0
# ZFS pool
zpool create -R /mnt -o cachefile=/tmp/zpool.cache -O mountpoint=/ \
-O atime=off -O canmount=off -O compression=on \
zroot nvd0
zfs create -o canmount=off -o mountpoint=none zroot/ROOT
zfs create -o mountpoint=/ zroot/ROOT/default
zpool set bootfs=zroot/ROOT/default zroot
zfs create zroot/home
zfs create -o canmount=off zroot/usr
zfs create zroot/usr/local
zfs create zroot/usr/obj
zfs create zroot/usr/src
zfs create zroot/usr/ports
zfs create zroot/usr/ports/distfiles
zfs create -o canmount=off zroot/var
zfs create zroot/var/jail
zfs create zroot/var/log
zfs create zroot/var/tmp
zfs create zroot/tmp
# Copy the zpool cache file to /boot
mkdir -p /mnt/boot/zfs
cp /tmp/zpool.cache /mnt/boot/zfs
exit
view raw gistfile1.txt hosted with ❤ by GitHub
  1. Proceed as normal. At the end of the installation, when the installer asks if you want to reboot or exit to a shell, choose Shell to finish the efi configuration.
cat >> /boot/loader.conf <<END
zfs_load="YES"
zpool_cache_load="YES"
zpool_cache_name="/boot/zfs/zpool.cache"
zpool_cache_type="/boot/zfs/zpool.cache"
vfs.root.mountfrom="zfs:zroot/ROOT/default"
END
cat >> /etc/rc.conf <<END
zfs_enable="YES"
zfsd_enable="YES"
END
cat >> /etc/fstab <<END
/dev/gpt/efi /boot/efi msdosfs rw,late 0 0
/dev/nvd0p6 none swap sw 0 0
END
# For the /dev/null issue mentioned above
rm /dev/null
mount -t devfs devfs /dev
# UEFI
mkdir /boot/efi
mount -t msdosfs /dev/gpt/efi /boot/efi
mkdir -p /boot/efi/efi/boot
cp /boot/loader.efi /boot/efi/efi/boot/bootx64.efi
# efirt was already compiled into 13.0-CURRENT installer kernel
# but do this anyway
kldload efirt
mkdir -p /boot/efi/efi/freebsd
cp /boot/loader.efi /boot/efi/efi/freebsd/
# Make sure you use your efi partition's device below
efibootmgr -c -L freebsd -l nvd0p5:/efi/freebsd/loader.efi
# Assuming your freebsd EFI partition is numbered 0001:
efibootmgr -a -b 1
# Add any other boot partitions with -a, then add to the
# -o option 1 to set the boot order
efibootmgr -o 1,0
# Legacy boot partition. Check the partition index (-1)
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 nvd0
exit
# And good luck
view raw gistfile1.txt hosted with ❤ by GitHub

Let the system reboot. Remove the installer media or manually select the disk for the boot device. With luck, it will boot on its own, through to your login prompt!


Part 2 covers creating my first FreeBSD bhyve guest.

Sources / References

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

Blog at WordPress.com.

Up ↑

%d bloggers like this: