Adventures in Freebernetes: VM Management with CBSD

Part 5 of experiments in FreeBSD and Kubernetes: Getting started with CBSD

See all posts in this series

At the end of the previous post, I had finally finished installing CBSD and its dependencies and configuration.

Doing Stuff with CBSD

There are a bunch of video tutorials for managing bhyve with CBSD. I’m going to start by trying to creating a basic FreeBSD VM.

Screen shot of CBSD's text-based user interface showing menu options for creating a new bhyve virtual machine
Yes, I (still) need to find a better terminal type for these menus in my Chrome OS Linux terminal. Suggestions welcome.

Other than choosing a jname of freebsd1, I keep all the defaults, and tell it to create the VM immediately.

Screenshot of shell output after telling cbsd to create my VM immediately, with commands for interacting with the VM
Global VM ZFS guid: 4974603855691936582
To edit VM properties use: cbsd bconfig jname=freebsd1
To start VM use: cbsd bstart freebsd1
To stop VM use: cbsd bstop freebsd1
To remove VM use: cbsd bremove freebsd1
For attach VM console use: cbsd blogin freebsd1
Creating freebsd1 complete: Enjoy!
root@nucklehead:~ # cbsd bstart freebsd1
Looks like /usr/cbsd/vm/freebsd1/dsk1.vhd is empty.
May be you want to boot from CD?
[yes(1) or no(0)]
yes
Temporary boot device: cd
vm_iso_path: iso-FreeBSD-x64-12.1-RELEASE-amd64-disc1
media found: iso-FreeBSD-x64-12.1-RELEASE-amd64-disc1 –> /usr/cbsd/src/iso/cbsd-iso-FreeBSD-12.1-RELEASE-amd64-disc1.iso
VRDP is enabled. VNC bind/port: 127.0.0.1:5900
For attach VM console, use: vncviewer 127.0.0.1:5900
Resolution: 1024×768.
bhyve renice: 1
Waiting for PID.
PID: 29014
root@nucklehead:~ #
view raw gistfile1.txt hosted with ❤ by GitHub

Oh, wait, I could use my Chromebook’s VNC app if the VNC port was bound to a routeable IP address.

root@nucklehead:~ # cbsd bget jname=freebsd1 | grep vnc
vm_vnc_port: 0
bhyve_vnc_tcp_bind: 127.0.0.1
bhyve_vnc_resolution: 1024×768
cd_vnc_wait: 1
vnc_password: 0
bhyve_vnc_vgaconf: io
vnc_port: 5900
root@nucklehead:~ # cbsd bset jname=freebsd1 bhyve_vnc_tcp_bind=192.168.0.10
bhyve_vnc_tcp_bind: changed
root@nucklehead:~ # cbsd brestart jname=freebsd1
Send SIGTERM to freebsd1. Soft timeout is 30 sec. 16 seconds left [………….Send SIGTERM to freebsd1. Soft timeout is 30 sec. 15 seconds left [………….Send SIGTERM to freebsd1. Soft timeout is 30 sec. 14 seconds left [………….Send SIGTERM to freebsd1. Soft timeout is 30 sec. 13 seconds left [………….Send SIGTERM to freebsd1. Soft timeout is 30 sec. 12 seconds left [………….Send SIGTERM to freebsd1. Soft timeout is 30 sec. 11 seconds left [………….Send SIGTERM to freebsd1. Soft timeout is 30 sec. 10 seconds left [………….Send SIGTERM to freebsd1. Soft timeout is 30 sec. 9 seconds left […………..Send SIGTERM to freebsd1. Soft timeout is 30 sec. 8 seconds left […………..Send SIGTERM to freebsd1. Soft timeout is 30 sec. 7 seconds left […………..Send SIGTERM to freebsd1. Soft timeout is 30 sec. 6 seconds left […………..Send SIGTERM to freebsd1. Soft timeout is 30 sec. 5 seconds left […………..Send SIGTERM to freebsd1. Soft timeout is 30 sec. 4 seconds left […………..Send SIGTERM to freebsd1. Soft timeout is 30 sec. 3 seconds left […………..Send SIGTERM to freebsd1. Soft timeout is 30 sec. 2 seconds left […………..Send SIGTERM to freebsd1. Soft timeout is 30 sec. 1 seconds left […………..Send SIGTERM to freebsd1. Soft timeout is 30 sec. 0 seconds left […………………………]
bstop done in 1 seconds
Looks like /usr/cbsd/vm/freebsd1/dsk1.vhd is empty.
Temporary boot device: cd
vm_iso_path: iso-FreeBSD-x64-12.1-RELEASE-amd64-disc1
media found: iso-FreeBSD-x64-12.1-RELEASE-amd64-disc1 –> /usr/cbsd/src/iso/cbsd-iso-FreeBSD-12.1-RELEASE-amd64-disc1.iso
VRDP is enabled. VNC bind/port: 192.168.0.10:5900
For attach VM console, use: vncviewer 192.168.0.10:5900
Resolution: 1024×768.
em0
bhyve renice: 1
Waiting for PID.
PID: 39774
Multiple bstart: done
root@nucklehead:~ #
view raw gistfile1.txt hosted with ❤ by GitHub
VNC desktop image of FreeBSD LiveCD menu
Screenshot of VNC desktop view of FreeBSD CBSD guest

I select “Install” and let it go. After rebooting, all is copacetic.

Screenshot of VNC app showing FreeBSD VM console after successful boot

With CBSD, we get a FreeBSD bhyve guest with a ZFS-backed virtual disk and VNC desktop with just a few commands. Compare that to my first experiment with manual bhyve VM creation, when I had to create the virtual network interfaces manually, create my disk file, download the FreeBSD ISO disk image, keep track of the virtual device files, and execute separate load and boot commands each time to bring up the VM.

Doing Linux Stuff with CBSD

Ok, a FreeBSD guest was pretty simple. What about Linux? Arch Linux was the first distribution I tried when manually creating bhyve Linux VMs, so I’ll try that first with CBSD, which supports a lot of Linux distros out of the box. (You can also add others for your own use, which I will try later.)

Screenshot of CBSD installer OS selector menu
Screenshot of CBSD installer supported Linux distributions

I selected ArchLinux, set the jname, set the VNC IP address so I could connect to the console from my Chromebook.

root@nucklehead:~ # cbsd bstart arch1
Looks like /usr/cbsd/vm/arch1/dsk1.vhd is empty.
May be you want to boot from CD?
[yes(1) or no(0)]
[yes(1) or no(0)]
yes
Temporary boot device: cd
vm_iso_path: iso-ArchLinux-x86-2020
No such media: /usr/cbsd/src/iso/cbsd-iso-archlinux-2020.08.01-x86_64.iso in /usr/cbsd/src/iso
Shall i download it from: http://artfiles.org/archlinux.org/iso/2020.08.01/ https://archlinux.surlyjake.com/archlinux/iso/2020.08.01/ https://mirror.aarnet.edu.au/pub/archlinux/iso/2020.08.01/ http://br.mirror.archlinux-br.org/iso/2020.08.01/ http://archlinux.de-labrusse.fr/iso/2020.08.01/ https://arch-mirror.wtako.net/iso/2020.08.01/ http://archlinux.prometeolibero.eu/archlinux/iso/2020.08.01/ https://mirror.yandex.ru/archlinux/iso/2020.08.01/ http://mirror.bytemark.co.uk/archlinux/iso/2020.08.01/ http://mirrors.acm.wpi.edu/archlinux/iso/2020.08.01/ https://mirror.us.leaseweb.net/archlinux/iso/2020.08.01/ ?
[yes(1) or no(0)]
yes
Download to: /usr/cbsd/src/iso/cbsd-iso-archlinux-2020.08.01-x86_64.iso
Scanning for fastest mirror…
Mirror source: Bytes per 3sec:
* [ 1/17 ] http://archlinux.de-labrusse.fr/iso/2020.08.01/: failed
* [ 2/17 ] http://archlinux.prometeolibero.eu/archlinux/iso/2020.08.01/: failed
* [ 3/17 ] http://artfiles.org/archlinux.org/iso/2020.08.01/: failed
* [ 4/17 ] http://br.mirror.archlinux-br.org/iso/2020.08.01/: failed
* [ 5/17 ] http://cbsd.lifec0re.net/iso/: 380928
* [ 6/17 ] http://electro.bsdstore.ru/iso/: 0
* [ 7/17 ] http://mirror.bytemark.co.uk/archlinux/iso/2020.08.01/: failed
* [ 8/17 ] http://mirrors.acm.wpi.edu/archlinux/iso/2020.08.01/: failed
* [ 9/17 ] https://arch-mirror.wtako.net/iso/2020.08.01/: failed
* [ 10/17 ] https://archlinux.surlyjake.com/archlinux/iso/2020.08.01/: failed
* [ 11/17 ] https://clonos.ca.ircdriven.net/iso/: 0
* [ 12/17 ] https://clonos.us.ircdriven.net/iso/: 0
* [ 13/17 ] https://electrode.bsdstore.ru/iso/: 0
* [ 14/17 ] https://mirror.aarnet.edu.au/pub/archlinux/iso/2020.08.01/: failed
* [ 15/17 ] https://mirror.bsdstore.ru/iso/: 0
* [ 16/17 ] https://mirror.us.leaseweb.net/archlinux/iso/2020.08.01/: failed
* [ 17/17 ] https://mirror.yandex.ru/archlinux/iso/2020.08.01/: failed
Winner: http://cbsd.lifec0re.net/iso/
Processing: http://cbsd.lifec0re.net/iso/archlinux-2020.08.01-x86_64.iso
retrieve archlinux-2020.08.01-x86_64.iso from cbsd.lifec0re.net, size: 671m
/usr/cbsd/src/iso/cbsd-iso-archlinux-2020.08.0 1% of 671 MB 247 kBps 01h00m
view raw gistfile1.txt hosted with ❤ by GitHub

Because I don’t have the ISO image stored locally, it checks its configured mirrors and starts the download.

Automatically register iso as: cbsd-iso-archlinux-2020.08.01-x86_64.iso
Path already exist for: iso-ArchLinux-x86-2020
VRDP is enabled. VNC bind/port: 192.168.0.10:5901
For attach VM console, use: vncviewer 192.168.0.10:5901
Resolution: 1024×768.
em0
bhyve renice: 1
Waiting for PID……….
PID: 0
Sun Nov 8 14:16:06 PST 2020
Boot from CD
cmd: env LIB9P_LOGGING=/usr/cbsd/jails-system/arch1/cbsd_lib9p.log /usr/bin/nice -n 1 /usr/sbin/bhyve 6 bhyve_flags -c 1 -m 1073741824 -H -A -U e2184453-2207-11eb-bd78-b8aeede991dd -s 0,hostbridge -s 4,ahci-hd,/usr/cbsd/vm/arch1/dsk1.vhd,sectorsize=512/4096 -s 3,ahci-cd,/usr/cbsd/src/iso/cbsd-iso-archlinux-2020.08.01-x86_64.iso,ro -s 5,virtio-net,tap3,mtu=1500,mac=00:a0:98:d5:12:cc -s 6,virtio-rnd -s 7,fbuf,tcp=192.168.0.10:5901,w=1024,h=768,wait -s 30,xhci,tablet -s 31,lpc -l com1,stdio -l bootrom,/usr/local/cbsd/upgrade/patch/efi.fd arch1
—–
Usage: bhyve [-abehuwxACDHPSWY]
[-c [[cpus=]numcpus][,sockets=n][,cores=n][,threads=n]]
[-g <gdb port>] [-l <lpc>]
[-m mem] [-p vcpu:hostcpu] [-s <pci>] [-U uuid] <vm>
-a: local apic is in xAPIC mode (deprecated)
-A: create ACPI tables
-c: number of cpus and/or topology specification
-C: include guest memory in core file
-D: destroy on power-off
-e: exit on unhandled I/O access
-g: gdb port
-h: help
-H: vmexit from the guest on hlt
-l: LPC device configuration
-m: memory size in MB
-p: pin 'vcpu' to 'hostcpu'
-P: vmexit from the guest on pause
-s: <slot,driver,configinfo> PCI slot config
-S: guest memory cannot be swapped
-u: RTC keeps UTC time
-U: uuid
-w: ignore unimplemented MSRs
-W: force virtio to use single-vector MSI
-x: local apic is in x2APIC mode
-Y: disable MPtable generation
Please use for debug: /usr/local/cbsd/share/bhyverun.sh -c /usr/cbsd/jails-system/arch1/bhyve.conf
view raw gistfile1.txt hosted with ❤ by GitHub

Hmmm. I hadn’t changed any other values. But that 6 bhyve_flags bit in the bhyve command it’s trying to run look suspicious. I look in the configuration file /usr/cbsd/jails-system/arch1/bhyve.conf and see the line bhyve_flags='6 bhyve_flags' which looks suspicious. I had descended into the bhyve flags options dialog in the UI, but hadn’t changed anything. There may have been some default text there which got entered? Either way, I edit the conf file to set bhyve_flags='' and run cbsd bstart arch1 again.

And got the same error. Even though I had edited the conf file, it had reverted. Apparently it’s getting generated. This time I run cbsd bset jname=arch1 bhyve_flags='' and try again. This time, success!

root@nucklehead:~ # cbsd bset jname=arch1 bhyve_flags=''
bhyve_flags: changed
root@nucklehead:~ # cbsd bstart arch1
Looks like /usr/cbsd/vm/arch1/dsk1.vhd is empty.
May be you want to boot from CD?
[yes(1) or no(0)]
yes
Temporary boot device: cd
vm_iso_path: iso-ArchLinux-x86-2020
media found: iso-ArchLinux-x86-2020 –> /usr/cbsd/src/iso/cbsd-iso-archlinux-2020.08.01-x86_64.iso
VRDP is enabled. VNC bind/port: 192.168.0.10:5901
For attach VM console, use: vncviewer 192.168.0.10:5901
Resolution: 1024×768.
em0
bhyve renice: 1
Waiting for PID.
PID: 65921
root@nucklehead:~ #
view raw gistfile1.txt hosted with ❤ by GitHub
Screen shot of VNC client showing ArchLinux installer shell

I follow the same installation steps as earlier. (One difference: the ZFS disk shows up as /dev/sda). I reboot and voila.

Screen shot of VNC client showing rebooted ArchLinux VM console

Ok, granted, that didn’t simplify the manual installation steps for Arch Linux, but it still reduced all the steps require for configuring the virtual network interface and disk, in addition to handling UEFI.

VM Reproduction

Using ZFS volumes for the VM virtual disk opens up the possibility of easily making copies of the VM image in the same ZFS storage pool. CBSD already has some tooling to support creating and using ZFS volume snapshots.

Since I’ve just installed a fresh Arch Linux VM, it would make a good source for a snapshot. I shut down the VM. (I don’t know if this is strictly necessary, but I don’t trust snapshots of unquiesced file systems. Yes, I know what AWS documentation says about EBS snapshots made on live systems being viable backups. No, they’re not.)

root@nucklehead:~ # cbsd bclone arch1 arch2
Please set bclone: old
root@nucklehead:~ # cbsd bclone old=arch1 new=arch2
bhyve is online. Please stop them: arch1
root@nucklehead:~ # cbsd bstop arch1
Send SIGTERM to arch1. Soft timeout is 30 sec. 13 seconds left […………….Send SIGTERM to arch1. Soft timeout is 30 sec. 12 seconds left […………….Send SIGTERM to arch1. Soft timeout is 30 sec. 11 seconds left […………….Send SIGTERM to arch1. Soft timeout is 30 sec. 10 seconds left […………….Send SIGTERM to arch1. Soft timeout is 30 sec. 9 seconds left [……………..Send SIGTERM to arch1. Soft timeout is 30 sec. 8 seconds left [……………..Send SIGTERM to arch1. Soft timeout is 30 sec. 7 seconds left [……………..Send SIGTERM to arch1. Soft timeout is 30 sec. 6 seconds left [……………..Send SIGTERM to arch1. Soft timeout is 30 sec. 5 seconds left [……………..Send SIGTERM to arch1. Soft timeout is 30 sec. 4 seconds left [……………..Send SIGTERM to arch1. Soft timeout is 30 sec. 3 seconds left [……………..Send SIGTERM to arch1. Soft timeout is 30 sec. 2 seconds left [……………..Send SIGTERM to arch1. Soft timeout is 30 sec. 1 seconds left [……………..Send SIGTERM to arch1. Soft timeout is 30 sec. 0 seconds left […………………………]
bstop done in 2 seconds
root@nucklehead:~ # cbsd bclone old=arch1 new=arch2
Cloning..ok
Global VM ZFS guid: 11878493802073682028
bclone done in 1 seconds
root@nucklehead:~ # zfs list | grep arch
zroot/ROOT/default/arch1 1.07G 178G 55.5K /usr/cbsd/vm/arch1
zroot/ROOT/default/arch1/dsk1.vhd 1.07G 178G 1.07G –
zroot/ROOT/default/arch2 18K 178G 55.5K /usr/cbsd/jails-data/arch2-data
zroot/ROOT/default/arch2/dsk1.vhd 0B 178G 1.07G –
root@nucklehead:~ #
view raw gistfile1.txt hosted with ❤ by GitHub

Oh, wait, I halted the VM from the shell, but I didn’t shut it down in bhyve. CBSD wants me to shut it down before cloning it, so there you go. Because ZFS clones use copy-on-write, they initially take up no additional disk space. Only when blocks get written or diverge from the original snapshot do they allocate actual disk space. CBSD also supports making an actual copy of the volume, which means it will no longer require the source snapshot. A full copy can take much longer to create, depending on the size of the volume and the storage performance.

root@nucklehead:~ # cbsd bhyve-dsk-list
JNAME DSK_CONTROLLER DSK_PATH DSK_SIZE DSK_SECTORSIZE BOOTABLE
freebsd1 virtio-blk dsk1.vhd 10g 512/4096 true
arch1 ahci-hd dsk1.vhd 6g 512/4096 true
arch2 ahci-hd dsk1.vhd 6g 512/4096 true
root@nucklehead:~ # cbsd bstart arch2
All CD/ISO ejected: arch2
VRDP is enabled. VNC bind/port: 192.168.0.10:5902
For attach VM console, use: vncviewer 192.168.0.10:5902
Resolution: 1024×768.
em0
Waiting for PID.
PID: 82244
root@nucklehead:~ #
view raw gistfile1.txt hosted with ❤ by GitHub

And my new VM boots right up!


In the next part of this series, we will look at more options with CBSD, including configuring a custom VM profile.

Sources / References

3 thoughts on “Adventures in Freebernetes: VM Management with CBSD

Add yours

  1. Great review, thanks! By the way, the most interesting are cloud images, for example:

    # cbsd bcreate jname=test vm_os_profile=”cloud-CentOS-7-x86_64″ vm_os_type=”linux” ci_gw4=10.0.0.1 ci_ip4_addr=”10.0.0.88″ vm_ram=1g imgsize=10g vm_cpus=2 astart=0 runasap=1
    # cbsd blogin

    see: `ls /usr/local/cbsd/etc/defaults/ | egrep \*-cloud-`

    Liked by 1 person

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

Blog at WordPress.com.

Up ↑

%d bloggers like this: