Note: this post is an updated version of this original k3s tutorial

See all posts in the FreeBSD Virtualization Series

1. Overview

• Intended Audience
• Technical Specs
  • Kubernetes Cluster Specs
  • Host Requirements
  • My Test System
• Part 1: Required Tools
• Part 2: Building k3sup
• Part 3: Configure Networking
• Part 3: Creating VMs
• Part 5: Create the Cluster
• Part 6: Test the Cluster
• Part 7: Clean Up
• Sources / References

Overview

This tutorial will build a Kubernetes cluster on FreeBSD’s bhyve virtualization platform, by installing a lightweight k3s control plane using the k3sup tool, which automates much of the process.

Topics covered:

• Compiling k3sup on FreeBSD
• Configuring CBSD to create and manage bhyve VMs
• Running k3sup to bring up a k3s cluster on bhyve VMs
• Configuring the FreeBSD firewall, DNS, and routing for cluster networking

While this tutorial builds a cluster with a redundant control plane, all the VMs are on a single hypervisor, making it suitable for testing and experimentation, but it is not production grade.

This tutorial only covers creating a cluster. For Kubernetes basics and terminology, you should start with the official Kubernetes documentation.

Important: only run this tutorial on a non-production host. It will make some changes to your network. Most of these are temporary, unless you want to make them persistent, but they may overwrite existing configurations.

Intended Audience

For this tutorial, you don’t need to know anything about Kubernetes. You do need to have a host with FreeBSD installed; an understanding of basic FreeBSD system administration tasks, such as installing software from FreeBSD ports or packages, and loading kernel modules; and familiarity with csh or sh. Experience with FreeBSD bhyve virtual machines and the CBSD interface is useful but not required.

Technical Specs

Kubernetes Cluster Specs

We will use these targets for the Kubernetes cluster we’re building. Most of these settings can be adjusted.

• Control plane (“K3s servers”)
  • 3 VMs: 2 CPU cores, 2Gb RAM, 20Gb virtual disk each
  • If you plan on creating more than 10 worker nodes, increase the control plane VM sizes
• Worker nodes (“K3s agents”)
  • 3 VMs: 2 CPU cores, 2Gb RAM, 10Gb virtual disk each
• VM OS: Ubuntu Server 22.04
• Kubernetes version: 124.
• Control plane backing database: embedded etcd
  • K3s also supports using an external datastore, such as MySQL

Host (FreeBSD Hypervisor) Requirements

• Hardware, physical or virtual
  • CPU
    • CPUs must support FreeBSD bhyve virtualization (see the FreeBSD Handbook page on bhyve for compatible CPUs)
    • CPU core allocations for bhyve VMs are completely virtual, so you can have VMs running with a total virtual core count greater than your host system’s. You should use the suggested core count for VMs, as Kubernetes will use those for scheduling. You can increase the cores if you have a larger host.

  • RAM: Minimum 2Gb per VM. You can use less for the agent VMs if necessary.

  • Free disk space: 100Gb
• Operating system
  • FreeBSD: tested on 13.1-CURRENT
  • File system: ZFS

Part 1: Required Tools

We’re going to install the tools needed on the FreeBSD hypervisor. You can compile from theportstree or install from packages, whichever you prefer. The tutorial assumes you have already installed all these ports. When additional post-installation configuration is required, we will walk through it at the appropriate point in the tutorial.

We need to build k3sup locally, so this list includes the build tools for compiling from source.

Here is the full list of required packages, including the versions, with the ports section in parentheses:

• Build tools
  • git (devel)
  • go (lang)
• K8s tools
  • kubectl (sysutils)
• FreeBSD system tools
  • CBSD (sysutils)
  • nsd (dns)
• Misc tools
  • wget (ftp)

If you want to use the pkg system, you can run pkg install git go kubectl cbsd nsd wget

Part 2: Build k3sup

• 2.1 Setup Your Go Environment
• 2.2 Check Out and Build k3sup

We’ll need to build k3sup for our FreeBSD hypervisor.

2.1 Setup Your Go Environment

If you already have a working golang build environment on your FreeBSD hypervisor, you can skip this section.

First, create your go workspace. This tutorial will assume you are using the path ${HOME}/go.

export GOPATH="${HOME}/go"
mkdir -p $GOPATH ${GOPATH}/bin ${GOPATH}/pkg ${GOPATH}/src ${GOPATH}/src/k8s.io ${GOPATH}/src/github.com

2.2 Check Out and Build k3sup

cd $GOPATH/src/github.com
git clone https://github.com/alexellis/k3sup/
cd k3sup
git checkout 0.12.8
go build -ldflags="-X github.com/alexellis/k3sup/cmd.Version=0.12.8"

You should copy this k3sup binary somewhere in your PATH.

Part 3: Configure Networking

• 3.1 Choose Your Network Layout
• 3.2 Load Kernel Modules
• 3.3 Add Bridge Gateways
• 3.4 Configure NAT Gateway
• 3.5 Configure Local DNS

3.1 Choose Your Network Layout

3.1.1 Select Subnets

I’m going to use a VLAN in 10.0.0.0/8 for the cluster and its pods and services. You can use another block, but you will have to adjust commands throughout the tutorial.

• 10.0.0.1/32 - VLAN gateway on bridge interface
• 10.0.0.2/32 - VIP (round-robin virtual IP) for the Kubernetes API endpoint
• 10.0.10.0/24 - VM block
  • 10.0.10.1[1-3] - K3s servers
  • 10.0.10.2[1-3] - K3s agents (nodes)
• 10.1.0.0/16 - pod network
• 10.2.0.0/16 - service network

3.1.2 Pick a .local Zone for DNS

This zone just needs to resolve locally on the FreeBSD host. I’m going with k3s.local because I’m too lazy to think of a clever pun right now. You cannot connect to the new VMs yet. CBSD creates a bridge interface the first time you create a VM. We need to add gateways for our cluster VLANs to that interface so we can route from the hypervisor to the VMs and vice versa. In most cases, CBSD will use the bridge1 interface.

3.2 Load kernel modules

We need to load the virtualization and networking kernel modules.

kldload vmm if_tuntap if_bridge nmdm

For persistence across reboots:

sysrc kld_list+="vmm if_tuntap if_bridge nmdm"

3.3 Add Bridge Gateway

CBSD creates and uses the bridge1 network interface for connecting to the VM network. We will pre-create it and configure it to route our selected subnets.

You’ll also need to change wlan0 to your network interface

ifconfig bridge1 create addm wlan0 up
ifconfig bridge1 alias 10.0.0.1/32
ifconfig bridge1 alias 10.0.10.1/24
ifconfig bridge1 alias 10.1.0.1/16
ifconfig bridge1 alias 10.2.0.1/16

For persistence across reboots:

sysrc cloned_interfaces="bridge1"
sysrc ifconfig_bridge1="up"
sysrc ifconfig_bridge1_alias0="inet 10.0.0.1/32"
sysrc ifconfig_bridge1_alias1="inet 10.0.10.1/24"
sysrc ifconfig_bridge1_alias2="inet 10.1.0.1/16"
sysrc ifconfig_bridge1_alias3="inet 10.2.0.1/16"

3.4 Configure NAT Gateway

With bridge1 configured, we can connect to the VMs, but the VMs can’t talk to the Internet because only the FreeBSD host can route to this 10.0.0.0/8 block. We will use ipfw as a NAT (Network Address Translation) gateway service. These steps will enable ipfw with open firewall rules and then configure the NAT. These changes will take effect immediately.

# Set internet-facing interface
net_if=wlan0
kenv net.inet.ip.fw.default_to_accept=1
sysrc firewall_type="open"
sysctl net.inet.tcp.tso=0
sysctl net.inet.ip.fw.enable=1
sysctl net.inet.ip.forwarding=1
sysctl net.inet6.ip6.forwarding=1
sysctl net.inet.tcp.tso=0
kldload ipfw_nat ipdivert
service ipfw onestart
ipfw disable one_pass
ipfw -q nat 1 config if "$net_if" same_ports unreg_only reset
ipfw add 1 allow ip from any to any via lo0 
ipfw add 200 reass all from any to any in
ipfw add 201 check-state
ipfw add 205 nat 1 ip from 10.0.0.0/8 to any out via "$net_if" 
ipfw add 210 nat 1 ip from any to any in via "$net_if"

For persistence:

Download this ipfw rules script and place it in /etc. Note: this file also contains rules we will add below, which is why we won’t load it now.

Then run the following:

echo net.inet.ip.fw.default_to_accept=1 >> /boot/loader.conf
echo net.inet.tcp.tso=0 >> /etc/sysctl.conf
echo net.inet.ip.fw.enable=1 >> /etc/sysctl.conf
echo net.inet.ip.forwarding=1 >> /etc/sysctl.conf
echo net.inet6.ip6.forwarding=1 >> /etc/sysctl.conf
echo net.inet.tcp.tso=0 >> /etc/sysctl.conf
sysrc kld_list+="ipfw_nat ipdivert"
sysrc firewall_enable="YES"
sysrc firewall_nat_enable="YES"
sysrc gateway_enable="YES"
sysrc firewall_script="/etc/ipfw.bridge.rules"

3.5 Configure Local DNS

We need a way to resolve our VM host names. We need to pick a private .local DNS domain, configure an authoritative server for the domain, and then set up a local caching server that knows about our domain but can also still resolve external addresses for us. We will follow this nsd/unbound tutorial closely.

3.5.1 Enable unbound for recursive/caching DNS

FreeBSD has a caching (lookup-only) DNS service called unbound in the base system. It will use the configured nameservers for external address lookups and the local nsd service (configured next) for lookups to our private zone. Copy unbound.conf and make any edits as necessary to IP addresses or your local zone name.

You will also want to update the FreeBSD host’s /etc/resolv.conf to add your local domain to the search list and add an entry for nameserver 127.0.0.1.

wget https://raw.githubusercontent.com/kbruner/freebernetes/main/k3s/dns/unbound/unbound.conf -O /etc/unbound/unbound.conf
service local_unbound onestart

For persistence:

sysrc local_unbound_enable="YES"

/etc/resolv.conf:

search k3s.local
nameserver 127.0.0.1

3.5.2 Configure the Authoritative DNS Service

We will use nsd, a lightweight, authoritative-only service, for our local zone. After copying the files, you can edit/rename the copied files before proceeding to make changes as necessary to match your local domain or IP addresses.

mkdir -p /var/nsd/var/db/nsd /var/nsd/var/run /var/nsd/var/log /var/nsd/tmp
chown -R nsd:nsd /var/nsd
cd /var/nsd
wget -q https://raw.githubusercontent.com/kbruner/freebernetes/main/k3s/dns/nsd/nsd.conf
wget -q https://raw.githubusercontent.com/kbruner/freebernetes/main/k3s/dns/nsd/zone.10
wget -q https://raw.githubusercontent.com/kbruner/freebernetes/main/k3s/dns/nsd/zone.k3s.local
nsd-control-setup -d /var/nsd
nsd-control -c /var/nsd/nsd.conf start

To start the service at boot:

sysrc nsd_enable="YES"
sysrc nsd_config="/var/nsd/nsd.conf"

Part 4: Create VMs

4.1 Initialize CBSD

If you haven’t run CBSD on your FreeBSD host before, you will need to set it up. You can use this seed file. Edit it first to set node_name to your FreeBSD host’s name and to change jnameserver and nodeippool if you are using a private network other than 10.0.0.0/8.

sysrc cbsd_workdir="/usr/cbsd"
wget https://raw.githubusercontent.com/kbruner/freebernetes/main/k3s/cbsd/initenv.conf
vi initenv.conf
/usr/local/cbsd/sudoexec/initenv inter=0 `pwd`/initenv.conf
service cbsdrsyncd stop
sysrc -x cbsdrsyncd_enable
sysrc -x cbsdrsyncd_flags

Download this Ubuntu VM configuration file and copy it to /usr/cbsd/etc/defaults

4.2 Create VMs

Copy this instance.jconf VM template file and update ci_gw4, ci_nameserver_search, and ci_nameserver_address fields as needed. If you want to set a password for the ubuntu user in case you want to log in on the console via VNC, you can assign it to cw_user_pw_user, but note this is a plain-text field.

When you run cbsd bcreate, if CBSD does not have a copy of the installation ISO image, it will prompt you asking to download it. After the first time, it will re-use the local image.

# create server VMs
for i in 0 1 2; do
  jconf="/tmp/server-${i}.jconf"
  cp instance.jconf "$jconf"
  cbsd bcreate jconf="$jconf" jname="server-$i" \
  ci_ip4_addr="10.0.10.1${i}/24" ci_jname="server-$i" \
  ci_fqdn="server-${i}.k3s.local" ip_addr="10.0.10.1${i}" \
  imgsize="20g" vm_cpus="2" vm_ram="2g"
done
# start server VMs
for i in 0 1 2; do cbsd bstart jname="server-$i"; done
# create agent VMs
for i in 0 1 2; do
  jconf="/tmp/agent-${i}.jconf"
  cp instance.jconf "$jconf"
  cbsd bcreate jconf="$jconf" jname="agent-$i" \
  ci_ip4_addr="10.0.10.2${i}/24" ci_jname="agent-$i" \
  ci_fqdn="agent-${i}.k3s.local" ip_addr="10.0.10.2${i}" \
  imgsize="10g" vm_cpus="2" vm_ram="2g"
done
# start agent VMs
for i in 0 1 2; do cbsd bstart jname="agent-$i"; done

Part 5: Create the Cluster

• 5.1 Install Servers
• 5.2 Install Agents
• 5.3 Set Up Service Load Balancing

5.1 Install Servers

We’ll create the control plane by creating the cluster on server-0, then adding server-1 and server-2 to the cluster.

We want to load balance requests to the Kubernetes API endpoint across the three server VMs. For true high-availability, we would want to use a load balancer with liveness health checks. For this tutorial, though, we will just use a simple round-robin DNS entry for kubernetes.k3s.local.

Note: This assumes Ubuntu configured each VM’s network interface as enp0s6. You may need to change the arguments in the ssh commands if that interface does not exist.

# Add our VIP route
ipfw add 300 fwd 10.0.10.10 ip from any to 10.0.0.2 keep-state
# Create VIP on server-0
ssh -o StrictHostKeyChecking=no -i ~cbsd/.ssh/id_rsa ubuntu@10.0.10.10 sudo ip address add 10.0.0.2/32 dev enp0s6:1
k3sup install \
  --host server-0 \
  --user ubuntu \
  --cluster \
  --k3s-channel stable \
  --ssh-key ~cbsd/.ssh/id_rsa \
  --tls-san 10.0.0.2 \
  --k3s-extra-args '--cluster-cidr 10.1.0.0/16 --service-cidr 10.2.0.0/16 --cluster-dns 10.2.0.10'
k3sup join \
  --host server-1 \
  --user ubuntu \
  --server \
  --server-host kubernetes.k3s.local \
  --server-user ubuntu \
  --k3s-channel stable \
  --ssh-key ~cbsd/.ssh/id_rsa \
  --k3s-extra-args '--cluster-cidr 10.1.0.0/16 --service-cidr 10.2.0.0/16 --cluster-dns 10.2.0.10'
k3sup join \
  --host server-2 \
  --user ubuntu \
  --server \
  --server-host kubernetes.k3s.local \
  --server-user ubuntu \
  --k3s-channel stable \
  --ssh-key ~cbsd/.ssh/id_rsa \
  --k3s-extra-args '--cluster-cidr 10.1.0.0/16 --service-cidr 10.2.0.0/16 --cluster-dns 10.2.0.10'
# Create VIPs on server-1 and server-2
ssh -o StrictHostKeyChecking=no -i ~cbsd/.ssh/id_rsa ubuntu@10.0.10.11 sudo ip address add 10.0.0.2/32 dev enp0s6:1
ssh -o StrictHostKeyChecking=no -i ~cbsd/.ssh/id_rsa ubuntu@10.0.10.12 sudo ip address add 10.0.0.2/32 dev enp0s6:1
export KUBECONFIG=/root/kubeconfig
kubectl config set-context default
# In case the server endpoint is set to localhost, we'll change it to our VIP
sed -I "" -e 's/server: .*$/server: https:\/\/10.0.0.2:6443/' $KUBECONFIG
kubectl get nodes -o wide

5.2 Install Agents

for i in 0 1 2; do
k3sup join \
  --host agent-$i \
  --user ubuntu \
  --server-host kubernetes.k3s.local \
  --server-user ubuntu \
  --k3s-channel stable \
  --ssh-key ~cbsd/.ssh/id_rsa
done
# Remove temporary firewall rule
ipfw delete 300
# Create round-robin firewall rules
ipfw add 300 prob 0.33 fwd 10.0.10.10 ip from any to 10.0.0.2 keep-state
ipfw add 301 prob 0.5 fwd 10.0.10.11 ip from any to 10.0.0.2 keep-state
ipfw add 302 fwd 10.0.10.12 ip from any to 10.0.0.2 keep-state
kubectl get nodes -o wide

5.3 Set Up Service Load Balancing

Generally, if you want to expose a Kubernetes application endpoint on an IP address outside the cluster’s network, you would create a Service object of type LoadBalancer. However, because load balancer options and implementations are unique for each cloud provider and self-hosted environment, Kubernetes expects you to have a controller running in your cluster to manage service load balancers. We have no such controller for our FreeBSD hypervisor, but we have a couple basic alternatives.

5.3.1 Routing to NodePort Services

For Services of type NodePort, we can route directly to the Service’s virtual IP, which will be in our 10.2.0.0/16 service network block. Each service VIP is routeable by every node, so if we set up round-robin forwarding rules on the hypervisor’s firewall, we should be able to reach NodePort endpoints.

ipfw add 350 prob 0.333 fwd 10.0.10.20 ip from any to 10.2.0.0/16 keep-state
ipfw add 351 prob 0.5 fwd 10.0.10.21 ip from any to 10.2.0.0/16 keep-state
ipfw add 352 fwd 10.0.10.22 ip from any to 10.2.0.0/16 keep-state

5.3.2 K3s Service Load Balancer

K3s has its own option for load balancer services. You can read the documentation for details. The service IP address will share the IP address of a node in the cluster. We will see a demonstration in the next section, when we test our cluster.

Note that with the K3s service load balancer, you run the risk of being unable to create a LoadBalancer-type service because of a high risk of port collisions, which are not usually a problem with most Kubernetes LoadBalancer implementations.

Part 6: Test the Cluster

We’ll run the following tests:

1. Create nginx deployment
2. Port-forward to nginx pod
3. Check nginx pod’s logs
4. Expose nginx NodePort Service
5. Expose nginx LoadBalancer Service on port 8080
6. Test pod-to-pod connectivity

# Create deployment
kubectl create deployment nginx --image=nginx
kubectl get pods 

# Port-forward to pod
POD=$(kubectl get pods -l app=nginx -ojsonpath="{.items[0].metadata.name}")
PID="$(kubectl port-forward $POD 8080:80 >/dev/null 2>&1 & echo $!)"

curl http://localhost:8080/
kill "$PID"

# Create NodePort service
kubectl expose deployment nginx --port 80 --type NodePort
kubectl get svc 
CLUSTERIP="$(kubectl get svc nginx -ojsonpath='{.spec.clusterIP}')"
curl -I http://${CLUSTERIP}/
kubectl delete svc nginx

# Create LoadBalancer Service
kubectl expose deployment nginx --port 8080 --target-port 80 --type LoadBalancer
kubectl get svc
LBIP="$(kubectl get svc nginx -ojsonpath='{.status.loadBalancer.ingress[0].ip}')"
curl -I http://${LBIP}:8080/
kubectl delete svc nginx

# Test pod-to-pod connectivity
PODIP="$(kubectl get pod "$POD" -ojsonpath='{.status.podIP}')"
kubectl run -it busybox --image busybox --rm=true --restart=Never -- wget -q -S -O /dev/null "http://${PODIP}"

Part 7: Clean Up

To clean up:

• Stop and remove the VMs
  • cbsd bstop agent-0 # etc
  • cbsd bremove agent-0 # etc
• Edit /etc/rc.conf to remove any settings that were added
• Edit /etc/sysctl.conf to remove any settings that were added
• Reboot

Please let me know if you have questions or suggestions either in the comments or on Twitter.

Sources and References

• https://blog.socruel.nu/freebsd/how-to-implement-unbound-and-nsd-on-freebsd.html
• https://cbsd.io/
• https://github.com/alexellis/k3sup/
• https://k3s.io/
• https://kubernetes.io/
• https://kubernetes.io/docs/concepts/services-networking/service/
• https://www.freebsd.org/doc/handbook/firewalls-ipfw.html
• https://www.freebsd.org/doc/handbook/network-bridging.html
• https://www.freebsd.org/doc/handbook/ports.html

Intro

This how-to will show you the basics of running a Linux container directly on FreeBSD, no virtual machine required!

Thanks to FreeBSD’s Linux emulation and lots of hard work by many people, including the contributions of everyone involved in this containerd pull request and especially Samuel Karp’s runj, an Open Container Initiative (OCI)-compliant way to launch FreeBSD jails, we can now run Linux containers via containerd.

This compatibility is all pretty early-stage and is not guaranteed to work for all containerd use cases (yet!), but all of this work is a huge step toward using FreeBSD hosts natively as Kubernetes nodes

Please note that currently networking doesn’t seem to be implemented!

Prerequisitites

Audience

This post assumes a basic working knowing of Linux containers, compiling source code, and basic FreeBSD system administration skills. (You can find a cheat sheet here)

Platform

I’m running on FreeBSD 13.1-RELEASE. I haven’t tested on other versions.

You also need a working ZFS pool.

Setup

All these commands need to be run as root.

Packages/ports

Install the following packages or ports:

• bash (shells)
• git (devel)
• gmake (devel)
• go (lang)

Set up Go

# Check /usr/local to find the name of the installed go dir and make a        
# symbolic link to /usr/local/go. My version of `go` is 1.19                  
ln -s /usr/local/go119 /usr/local/go                                          
export GOPATH=/usr/local/go

Build containerd

# Check out containerd source
git clone https://github.com/containerd/containerd.git
cd containerd
# Minimum required containerd version is v1.6.7
# Find the latest with `git tag -l | tail`
git checkout v1.6.8
# Use gnu make!
gmake install
# Start containerd
service containerd onestart
# For persistence across reboots, add containerd to /etc/rc.conf run at boot
# echo 'containerd_enable="YES"' >> /etc/rc.conf
#
# This also seems to be necessary
mkdir /var/lib/containerd/io.containerd.snapshotter.v1.zfs

Build runj

git clone https://github.com/samuelkarp/runj.git
cd runj
gmake install

Enable Linux emulation

kldload linux
# To load the Linux module at boot, run
# echo 'linux_load="YES"' >> /boot/loader.conf
service linux onestart
# To enable Linux emulation at boot, run
# echo 'linux_enable="YES"' >> /etc/rc.conf

Run a Linux container!

# Pull the image
ctr image pull --platform=linux docker.io/library/alpine:latest
#
# And ....
ctr run --rm --tty --runtime wtf.sbk.runj.v1 --snapshotter zfs --platform linux docker.io/library/alpine:latest mylinuxcontainer uname -a

[root@nucklehead ~]# ctr image pull --platform=linux docker.io/library/alpine:latest
docker.io/library/alpine:latest:                                                  resolved       |++++++++++++++++++++++++++++++++++++++| 
index-sha256:bc41182d7ef5ffc53a40b044e725193bc10142a1243f395ee852a8d9730fc2ad:    done           |++++++++++++++++++++++++++++++++++++++| 
manifest-sha256:1304f174557314a7ed9eddb4eab12fed12cb0cd9809e4c28f29af86979a3c870: done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:213ec9aee27d8be045c6a92b7eac22c9a64b44558193775a1a7f626352392b49:    done           |++++++++++++++++++++++++++++++++++++++| 
config-sha256:9c6f0724472873bb50a2ae67a9e7adcb57673a183cea8b06eb778dca859181b5:   done           |++++++++++++++++++++++++++++++++++++++| 
elapsed: 2.5 s                                                                    total:  2.7 Mi (1.1 MiB/s)                                       
unpacking linux/amd64 sha256:bc41182d7ef5ffc53a40b044e725193bc10142a1243f395ee852a8d9730fc2ad...
done: 227.895869ms
[root@nucklehead ~]# ctr run --rm --tty --runtime wtf.sbk.runj.v1 --snapshotter zfs --platform linux \
> docker.io/library/alpine:latest mylinuxcontainer uname -a
 
Linux 3.17.0 FreeBSD 13.1-RELEASE releng/13.1-n250148-fc952ac2212 GENERIC x86_64 Linux
[root@nucklehead ~]# uname -a
FreeBSD nucklehead 13.1-RELEASE FreeBSD 13.1-RELEASE releng/13.1-n250148-fc952ac2212 GENERIC amd64
[root@nucklehead ~]# 

The command line

Breaking down the command ctr run --rm --tty --runtime wtf.sbk.runj.v1 --snapshotter zfs --platform linux docker.io/library/alpine:latest mylinuxcontainer uname -a

• ctr run – containerd client and its subcommand to run a container
• --rm – remove the container upon completion
• --tty – attach terminal for STDIN/STDOUT to your terminal
• --runtime wtf.sbk.runj.v1 – use the runj runtime for FreeBSD
• --snapshotter zfs – use ZFS as the storage backend
• --platform linux – use a container image built for Linux
• docker.io/library/alpine:latest – the container image name, in this case the one we pulled earlier
• mylinuxcontainer – the name I gave my container. You can use any string
• uname -a – the command to run in the container

And that’s it! Wow!

If you have questions or comments, you can @ me on Twitter.

References

• https://docs.freebsd.org/en/books/handbook/jails/
• https://docs.freebsd.org/en/books/handbook/linuxemu/
• https://docs.freebsd.org/en/books/handbook/zfs/
• https://github.com/containerd/containerd
• https://github.com/containerd/containerd/pull/7000
• https://github.com/samuelkarp/runj
• https://iximiuz.com/en/posts/containerd-command-line-clients/
• https://kubernetes.io/docs/concepts/architecture/nodes/
• https://linuxcontainers.org/
• https://opencontainers.org/
• https://productionwithscissors.run/2022/09/02/fun-with-freebsd-first-linux-guest/#freebsd-tips-for-linux-people
• https://samuel.karp.dev/blog/
• https://www.freebsd.org/releases/13.1R/installation/

Intro

The FreeBSD operating system contains innumerable powerful features. One of these features is bhyve, its native type 2 (OS-level) hypervisor, which can host virtual machines running multiple different OSes, including Linux.

This post will walk you through creating a Linux virtual machine on FreeBSD using the CBSD tool, which greatly simplifies creating and managing bhyve VMs.

Note: This tutorial is meant for a learning experiment only. It is not meant for use setting up production systems. It skips most security features and sets up no fault tolerance.

Audience

This tutorial assumes you have basic FreeBSD or Linux command-line or system administration skills and have gone through the process of installing FreeBSD.

FreeBSD Tips for Linux People

Software installation

FreeBSD has a very mature and reliable software package management system and a library of third-party packages similar to most Linux distributions. You can install pre-compiled binary packages using pkg or you can compile from source code using the ports collection.

I generally compile from ports, which allows custom compile-time options and can sometimes be more up-to-date, but pkg should be fine and definitely much faster.

Text-based User Interfaces

FreeBSD uses a very distinctive style of interactive menus. If you compile from ports, you will see it a lot. CBSD also uses it.

The distinctive FreeBSD interactive interface with its shadow box menu and its light blue background

• To navigate the menu: use the <up/down> arrow keys
• To change an option: navigate to that line, press the spacebar.
  • If the item is a checkbox, it will toggle the selection
  • A pop-up will appear if the option is a multi-choice or a text box entry

When you have finished the configuration, use the <left>/<right> arrow keys to navigate the options below the menu. Select <Ok> to confirm and hit enter.

Tools

• Your favorite Linux command-line tools like grep, sed, awk, and find are all here but they sometimes behave a little differently than on GNU/Linux.
• If you absolutely need the GNU versions, you can usually install them via packages or ports, with g or gnu prefixed to the name (gnugrep, gsed, etc.)
• bash is also not installed by default. You can install it via pkg or ports

Root access

sudo access is not set up automatically, unlike many Linux distributions. In the default installation, FreeBSD expects users needing root access to be added to the wheel group (in /etc/group) and to su to root, using the root password to authenticate. If this is an important or shared host, you can configure sudo

The root user also defaults to csh as its login shell. You probably want to install bash and use that instead.

Setup and configuration

Host (FreeBSD Hypervisor) Requirements

• CPUs must support FreeBSD bhyve virtualization (see the FreeBSD Handbook page on bhyve for compatible CPUs)
• FreeBSD: At least 13.0-CURRENT. I tested this on 13.1-RELEASE
• File system for VM storage: ZFS
  • If you don’t want to install root on ZFS, that’s fine. You can create a ZFS pool on another disk or disk partition.
  • I installed 13.1-RELEASE as ZFS-on-root and let the installer use its default partitions. Previous FreeBSD installers may require additional ZFS configuration
    • Static IP. If your host’s IP address changes, you may have to reconfigure CBSD later.

VNC Client

To connect to the console of the new VM during the installation phase, you will need a VNC (Virtual Network Computing) client on a graphics-based host (for example, Windows or a Linux desktop). When connecting from a Linux host, I like the TigerVNC client, which is available as a pre-built package for many Linux distros.

ZFS Storage

These instructions assume you have a ZFS pool of reasonable size and it’s named zpool. “Reasonable size” depends on how many VMs you want to create, but let’s assume a bare minimum of 10-20Gb per VM.

Install and Configure Tools

Tools

You will need to install the following via the pkg command or by compiling the port. The port group is in parentheses.

• git (devel)
• CBSD (sysutils)
• nsd (dns)
• tmux (sysutils)

Initialize CBSD

Before we run CBSD the first time, we need to initialize its configuration. You can find the default values in /usr/local/cbsd/share/initenv.conf but we’re going to change a few of those.

In addition to populating /usr/cbsd (the “workdir”) and writing configuration files, the initenv subcommand given the above values will also append boot-time CBSD options to /etc/rc.conf and /boot/loader.conf.

You can run the initenv subcommand one of two ways:

1. Interactively. This is long and rather confusing if you don’t know what you’re doing.
2. Using a seed file with the values pre-set.

Important: all CBSD commands need to be run as root

Notes:

• My host has the hostname nucklehead and the IP address 192.168.1.76. You should use your own host’s values for those options.
• For the RSYNC and RACCT questions, because this is a non-production system, you probably want to answer disable them to save resources
• CBSD version 13.1.13 seems to ignore the racct=0 option (don’t enable accounting). After initialization is done, you can remove the line kern.racct.enable=1 from /boot/loader.conf
• initenv may offer different configuration questions depending on answers to previous questions

Initialize CBSD the Quick Way

Initialize CBSD the Long Way

Enable pf Networking

The pf firewall needs some additional setup.

# Create the configuration file
cp /usr/local/examples/pf.conf /etc/pf.conf
# Enable the NAT gateway
echo 'gateway_enable="YES"' >> /etc/rc.conf
# Start
service start

Load kernel modules

We need to set up several kernel modules. We load them now then add them to /boot/loader.conf so they will load automatically at boot time.

for module in vmm if_tuntap if_bridge nmdm; do
    kldload "$module"
    echo "${module}_load=\"YES\"" >> /boot/loader.conf
done

Create our Linux Virtual Machine

We’re finally ready to create our VM.

We’re going to select the latest available supported version of Ubuntu. Note that this may not be the current Ubuntu release. We’ll cover how to configure releases with no CBSD support in a later post.

Notes

• In this example, we are going to use Ubuntu Server 22.04.
• Your menu options may differ if you’re using a different version of CBSD.
• We will configure the VM manually through the UI. A later post will show how to configure from a template file instead.

Configure the VM

Run cbsd bconstruct-tui to start the VM configuration.

This menu shows my configuration. You may want to tweak `jname` (the VM's identifying name), `host_hostname`, or network options.

We also want to set up our VNC endpoint so we can connect to the VM after it boots.

Navigate to the bhyve_vnc_options option and press enter to bring up the sub-menu.

VNC configuration sub-menu

VNC options:

• If you connect to your hypervisor via SSH, you want to change the bind IP (bhyve_vnc_tcp_bind) to that host’s IP address. You can also set it to 0.0.0.0 to bind it to all the network interfaces on the host hypervisor.
• If the vm_vnc_port is set to 0, a random port number will be assigned. We’ll set it here to 5901. Be careful of port collisions if you have more than one VM on the host. They each need a unique VNC port.
• Be sure to set the password!
• Select Save when done to go back to the main menu.

Once you’ve finished the configuration, select Ok in the main menu. You will then be asked if you want to create the vm immediately. We’ll select yes here.

Start the VM

Our VM still hasn’t started running, but CBSD has created its configuration. We can start the VM by running one more command.

cbsd bstart mylinuxvm

The first time you start an image with a specific operating system, you will need to wait as the image is pulled from the internet, which can take take several minutes or more, depending on the speed of your internet connection. CBSD will try to find the fastest mirror for you. Once the image for a specific Linux version has been downloaded, though, you can re-use it for more VMs with the same version, which will speed up the process greatly.

[root@nucklehead ~]# cbsd bstart mylinuxvm
init_systap: waiting for link: em0
Looks like /usr/cbsd/vm/mylinuxvm/dsk1.vhd is empty.
May be you want to boot from CD?
[yes(1) or no(0)]
1
Temporary boot device: cd
vm_iso_path: iso-Ubuntu-Server-22.04-amd64
No such media: /usr/cbsd/src/iso/cbsd-iso-ubuntu-22.04-live-server-amd64.iso in /usr/cbsd/src/iso
Shall i download it from: http://mirror.truenetwork.ru/ubuntu-releases/22.04/ http://ubnt-releases.xfree.com.ar/ubuntu-releases/22.04/ http://mirror.pop-sc.rnp.br/mirror/ubuntu-releases/22.04/ http://mirror.easyspeedy.com/ubuntu-iso/22.04/ http://ubuntu.mirrors.ovh.net/ubuntu-releases/22.04/ http://ftp.halifax.rwth-aachen.de/ubuntu-releases/22.04/ http://ubuntu.connesi.it/22.04/ http://mirror.nl.leaseweb.net/ubuntu-releases/22.04/ http://releases.ubuntu.com/22.04/ http://mirror.waia.asn.au/ubuntu-releases/22.04/ ?
[yes(1) or no(0)]
1
Download to: /usr/cbsd/src/iso/cbsd-iso-ubuntu-22.04-live-server-amd64.iso
Scanning for fastest mirror...
             Mirror source:                       bytes/sec:
 * [ 1/17  ] http://electro.bsdstore.ru:          0
 * [ 2/17  ] http://ftp.halifax.rwth-aachen.de:   failed (http://ftp.halifax.rwth-aachen.de/ubuntu-releases/22.04/ubuntu-22.04-live-server-amd64.iso)
 * [ 3/17  ] http://mirror.easyspeedy.com:        failed (http://mirror.easyspeedy.com/ubuntu-iso/22.04/ubuntu-22.04-live-server-amd64.iso)
 * [ 4/17  ] http://mirror.nl.leaseweb.net:       failed (http://mirror.nl.leaseweb.net/ubuntu-releases/22.04/ubuntu-22.04-live-server-amd64.iso)
 * [ 5/17  ] http://mirror.pop-sc.rnp.br:         failed (http://mirror.pop-sc.rnp.br/mirror/ubuntu-releases/22.04/ubuntu-22.04-live-server-amd64.iso)
 * [ 6/17  ] http://mirror.truenetwork.ru:        failed (http://mirror.truenetwork.ru/ubuntu-releases/22.04/ubuntu-22.04-live-server-amd64.iso)
 * [ 7/17  ] http://mirror.waia.asn.au:           failed (http://mirror.waia.asn.au/ubuntu-releases/22.04/ubuntu-22.04-live-server-amd64.iso)
 * [ 8/17  ] http://releases.ubuntu.com:          failed (http://releases.ubuntu.com/22.04/ubuntu-22.04-live-server-amd64.iso)
 * [ 9/17  ] http://ubnt-releases.xfree.com.ar:   failed (http://ubnt-releases.xfree.com.ar/ubuntu-releases/22.04/ubuntu-22.04-live-server-amd64.iso)
 * [ 10/17 ] http://ubuntu.connesi.it:            failed (http://ubuntu.connesi.it/22.04/ubuntu-22.04-live-server-amd64.iso)
 * [ 11/17 ] http://ubuntu.mirrors.ovh.net:       failed (http://ubuntu.mirrors.ovh.net/ubuntu-releases/22.04/ubuntu-22.04-live-server-amd64.iso)
 * [ 12/17 ] https://clonos.ca.ircdriven.net:     593920
 * [ 13/17 ] https://clonos.us.ircdriven.net:     643072
 * [ 14/17 ] https://electrode.bsdstore.ru:       0
 * [ 15/17 ] https://mirror.bsdstore.ru:          0
 * [ 16/17 ] https://mirror2.bsdstore.ru:         0
 * [ 17/17 ] https://plug-mirror.rcac.purdue.edu: 8125098
 Winner: https://plug-mirror.rcac.purdue.edu/cbsd/iso/
Processing: https://plug-mirror.rcac.purdue.edu/cbsd/iso/ubuntu-22.04-live-server-amd64.iso
retrieve ubuntu-22.04-live-server-amd64.iso from plug-mirror.rcac.purdue.edu, size: 1g
/usr/cbsd/src/iso/cbsd-iso-ubuntu-22.04-live-s        1398 MB   10 MBps 02m08s
Checking CRC sum: 84aeaf7823c8c61baa0ae862d0a06b03409394800000b3235854a6b38eb4856f...Passed
Automatically register iso as: cbsd-iso-ubuntu-22.04-live-server-amd64.iso
Path already exist for: iso-Ubuntu-Server-22.04-amd64
VRDP is enabled. VNC bind/port: 192.168.1.76:5901
For attach VM console, use: vncviewer 192.168.1.76:5901
Resolution: 1024x768.
VNC pass: vncpass

Warning!!! You are running a system with open VNC port to the world wich is not secure
Please use IP filter or balancer with password to restrict VNC port access
Or change vnc_bind params to 127.0.0.1 and reboot VM after maintenance work

bhyve renice: 1
Waiting for PID.
PID: 75699
bstart done in 4 minutes and 16 seconds
[root@nucklehead ~]#

We can check the status of our vm:

cbsd bls mylinuxvm

[root@nucklehead ~]# cbsd bls mylinuxvm
JNAME      JID    VM_RAM  VM_CURMEM  VM_CPUS  PCPU  VM_OS_TYPE  IP4_ADDR  STATUS  VNC
mylinuxvm  75699  2048    0          2        0     linux       DHCP      On      0.0.0.0:5901

Once it starts, we can open our VNC client. For the VNC server, you need to enter the IP address of your FreeBSD host and the port that you assigned during configuration, e.g. 192.168.1.76:5901

Install Ubuntu

A window should open showing the the Linux boot. Depending on how long it takes you to connect, you may see the grub boot menu or the Ubuntu installer may have already started. Go through the installation process.

Success! We're booting!

And installing!

Rebooting will usually close the VNC client, forcing reconnection. The host should boot to console, allowing you to log in with the username and password you chose during installation.

Voila!

Clean up

Once you’re done with the VM, you can run the following to stop and free the resources.

# Stop the VM
cbsd bstop jname=mylinuxvm
# Delete the VM configuration and ZFS volume
cbsd bdestroy jname=mylinuxvm

• CBSD added some configuration options to the files to /etc/rc.conf and /boot/loader.conf that you will want to remove.

You can reboot your system to clear the following two changes, or manually revert them. Leaving them in place until the next reboot is probably harmless, though.

• Unload the kernel modules you added via kldload above
• Remove the VM’s network interfaces, probably named bridge1 and tap1, depending how many VMs you created. You can remove these with the command for each interface. (DO NOT try to destroy any other interfaces! Leave lo0 and em0 alone!)

  ifconfig tap1 down
  ifconfig tap1 destroy
  

And that’s it! Now you know how to create a basic Linux VM on FreeBSD bhyve!

If you have questions or comments, you can @ me on Twitter.

References

• https://cbsd.io/
• https://docs.freebsd.org/en/books/handbook/bsdinstall/
• https://docs.freebsd.org/en/books/handbook/firewalls/#firewalls-pf
• https://docs.freebsd.org/en/books/handbook/ports/#pkgng-intro
• https://docs.freebsd.org/en/books/handbook/ports/#ports-using
• https://docs.freebsd.org/en/books/handbook/security/#security-sudo
• https://docs.freebsd.org/en/books/handbook/zfs/
• https://tigervnc.org/
• https://www.freebsd.org/
• https://www.freebsd.org/doc/handbook/virtualization-host-bhyve.html

The mermaid markdown (mmd) for this graph:

gitGraph
   commit id: "Need a project"
   commit id: "Play with FreeBSD again"
   branch write-blog-post-on-freebsd
   checkout write-blog-post-on-freebsd
   commit id: "Start writing markdown"
   branch reinstall-freebsd
   checkout reinstall-freebsd
   commit id: "Get FreeBSD reinstalled"
   checkout write-blog-post-on-freebsd
   merge reinstall-freebsd
   commit id: "Write more markdown"
   branch install-nvim-markdown-plugin
   checkout install-nvim-markdown-plugin
   commit id: "Spend 4 hours trying to get it working"
   branch write-blog-post-about-nvim-plugins
   checkout write-blog-post-about-nvim-plugins
   commit id: "Write a post to save others from my fate"
   checkout install-nvim-markdown-plugin
   merge write-blog-post-about-nvim-plugins
   checkout write-blog-post-on-freebsd
   merge install-nvim-markdown-plugin
   branch tweak-blog-settings
   checkout tweak-blog-settings
   commit id: "Fix formatting and other shit"
   checkout write-blog-post-on-freebsd
   merge tweak-blog-settings
   branch play-with-mermaid
   checkout play-with-mermaid
   commit id: "Here we are"

To embed in an .md file, you can wrap it like this:

Rendering to a file

You can use the cli to write directly to a file. You can install the cli as an npm package locally, but I got dependency errors and I’m not a node person, so forget that.

Fortunately, they distribute a docker container. If you already have docker installed, you just need to pull the image, and you’re good to go.

The cli supports svg, png, md, and pdf output formats. I want an image file, so first I tried png but the file wasn’t loadable in one image viewer I tried. Instead I output to svg and then used gimp to convert it to png. (Plenty of other tools exist for that image conversion.)

k@slaapmatje:~$ mkdir mermaid
k@slaapmatje:~$ mv my-brain.mmd mermaid
k@slaapmatje:~$ cd mermaid
k@slaapmatje:~/mermaid$ chmod 777 .
k@slaapmatje:~/mermaid$ sudo docker pull minlag/mermaid-cli
Using default tag: latest
latest: Pulling from minlag/mermaid-cli
Digest: sha256:bf130e7ce53fa2269d4d7784c4d7d3edea63185f2fb72d337148224ffd22f1ec
Status: Image is up to date for minlag/mermaid-cli:latest
docker.io/minlag/mermaid-cli:latest
k@slaapmatje:~/mermaid$ sudo docker run -it -v `pwd`:/data minlag/mermaid-cli -i /data/my-brain.mmd -o /data/my-brain.svg
Generating single mermaid chart
k@slaapmatje:~/mermaid$ ls
my-brain.mmd  my-brain.svg
k@slaapmatje:~/mermaid$

(We give the mermaid directory global write permissions so the docker user can write to it.)

Mermaid supports a number of graph and diagram formats, and it’s fun to write!

This assumes you’re on a Linux/BSD-ish system and you know basic vi/vim/nvim commands such as how to quit out of the editor.

1. Install neovim. Even if your OS package manager has an neovim package, it may not be a recent version that works with newer plugins. I suggest using the correct package for the latest stable version
2. Install the packer plugin manager

    git clone --depth 1 https://github.com/wbthomason/packer.nvim \
     ~/.local/share/nvim/site/pack/packer/start/packer.nvim
   

3. Create the neovim configuration directory

    mkdir -p "${HOME}/.config/nvim/lua"
   

4. Create the plugins.lua file to load Packer

    cat <<EOF >>"${HOME}/.config/nvim/lua/plugins.lua"
    local vim = vim
    local execute = vim.api.nvim_command
    local fn = vim.fn
       
    -- ensure that packer is installed
    local install_path = fn.stdpath('data')..'/site/pack/packer/opt/packer.nvim'
       
    if fn.empty(fn.glob(install_path)) > 0 then
        execute('!git clone https://github.com/wbthomason/packer.nvim '..install_path)
        execute 'packadd packer.nvim'
    end
       
    vim.cmd('packadd packer.nvim')
    local packer = require'packer'
    local util = require'packer.util'
       
    packer.init({
      package_root = util.join_paths(vim.fn.stdpath('data'), 'site', 'pack')
    })
       
    return require('packer').startup(function()
      use 'wbthomason/packer.nvim'
       
      -- Add plugins here!
       
    end)
    EOF
   

5. Configure nvim to load the plugins.lua

    cat <<EOF >>"${HOME}/.config/nvim/init.vim"
    lua require('plugins')
    EOF
   

6. Install/update Packer and your plugins
   1. Run nvim (no file necessary)
   2. Type :PackerSync
   3. Reply y
   4. Exit nvim

You should now be able to add Packer-compatible plugins around the -- Add plugins here! comment.

If you try to execute :PackerSync and get an error, I don’t really have any debugging tips. Try to clean out your ~/.config/nvim directory and try again?

A couple months ago, Dan and Ana introduced me (i.e., got me addicted) to Olipop soft drinks. Olipop is a “new kind of soda,” one of those newfangled “functional drinks,” that are supposed to be both tasty and have some health benefits. In Olipop’s case, each can says that it supports digestive health. I can’t speak to that, but they are damn tasty.

There are 10 flavors available now, with Banana Cream being the newest addition. Even though we had tried most or all the flavors, Ana, Dan, and I decided to do a systematic review of each flavor. (Note: we later learned that Orange Cream is a “seasonal” flavor, and a previous seasonal flavor, Blackberry Vanilla, is currently not available.)

These tastings were split over several days, because that’s a lot of soda to consume at once. Comments have been reconstituted from hastily taken notes.

Day One: Banana Cream, Tropical Punch, Classic Grape

We tried these three flavors together in person when I visited Dan and Ana.

Banana Cream

Someone didn't swirl the can enough before pouring

Banana Cream comes in a very yellow can emblazoned with, of all things, a Minion. This co-branding seems to be a first for Olipop, and Ana asked how this came to be, aside from the story on the side of the can.

Bouquet

Me: It smells like banana candy
Dan: Like banana candy or banana bread
Ana: Like banana bread with a hint of vanilla

Color

Ana: It looks milkier with a slight thickness
Me: It looks like lemonade
Dan: I dunno, yellow?

Taste

Ana: It tastes less intense than it smells. It has light carbonation
Dan: Smooth mouthfeel, slight tingle
Me: What Ana said. It tastes sweet but not as chemical as it smells. It’s more like a tonic.

Overall

Me: Potable but not my first choice.
Dan: It’s ok I guess. An amusing novelty.
Ana: I think it’s nicely refreshing. Also not my first choice.
After I admit that I’d never seen of the Minion movies, Dan asks if I’m more likely to watch The Rise of Gru after drinking the Banana Cream. (No, I am not.)

Tropical Punch

Thankfully it's not dyed Hawaiian Punch red

Dan had previously noted that the name Tropical Punch reminded him of Hawaiian Punch, that faux-tropical and vert faux-red soft drink purveying to suburban kids. (I can remember the color, although I can’t remember how it tastes. If I did taste it again, though, I would know.)

Color

Me: Not dyed red like Hawaiian Punch
Ana notes this flavor isn’t as cloudy as the Banana Cream.
Dan: Looks like pee

Bouquet

Ana: Pineapple, much less intense than the banana
Dan: Smells like pineapple, although it doesn’t smell artificial like the banana
Me: Pineapple

Taste

Ana: Pineapple juice without the acidity, more sweetness
Me: Slight afternotes of passion fruit?
Dan: Not acidic. I can’t taste anything except the pineapple

Overall

Ana: It tastes like pineapple and very sweet. It’s not as carbonated as the banana.
Me: Very sweet
Dan: It makes me want to try real Hawaiian Punch again

Classic Grape

Ana really liked the ruby red color

Color

Ana: I can see bubbles. Prettiest color so far, like a deep red ruby
Dan and I agree it is very clear

Bouquet

Ana: The smell is not that strong. It smells like candy.
Dan: It smells like Welch’s Grape Juice
Me: Yeah, like the can of concentrate you’d get in the frozen food aisle

Taste

Ana: It feels less sweet than the others.
Dan: It has a sweet aftertaste that lingers
Ana: Tastes more artificial.
Me: I would drink it but probably only if it was the last can in the fridge

Day One summary

We all agree that the Tropical Punch is probably our favorite so far.

Day Two: Classic Root Beer & Strawberry Vanilla

We performed this and the following tastings virtually over Zoom.

Classic Root Beer

Ana, a naturalized American, starts by noting that she thinks of root beer as “one of those American things only Americans like.” Dan says that when they send her to American reprogramming camp, they will have A&W Root Beer there. I assume there will also be Hawaiian Punch and Welch’s grape juice.

Appearance

Clear and slightly carbonated

Bouquet

Me: No real smell
Ana: Smells like medicine
Me: Smells like root beer

Taste

Ana: Makes me think of something I get at the dentist, like mouthwash or toothpaste or something. Vague mint aftertaste
Dan: Like root beer but with some olipopness with it. No bite, more like being gently gummed.
Me: Definitely tastes weaker than A&W
Ana: I would definitely drink this if I was lost on a desert island and the others were gone. It’s less bad than I thought it would be.
Me: Tastes a little watered down.
Dan: It’s still one of my favorites
Dan (reading the label): “organic root beer flavor extract”

At this point I launch into my story about living in Pennsylvania as a kid, where the local pizza parlor had both a player piano and birch beer. We google to figure out the difference (the kind of tree roots used, unsurprisingly.) I can’t remember the taste difference after this many decades, though.

Strawberry Vanilla

Color/Bouquet

Dan: Some solids
Me: It does smell a bit like candy strawberry
Ana: Yes, less intense
Dan: Smells like strawberry ice cream
Me: Maybe a little like strawberry medicine?

Taste

Me: Not a strong taste
Ana: Kind of pleasant, not very obviously strawberry. Mild compared to the root beer

Impressions

Dan, reading the can: “Himalayan pink salt”
(A discussion about the pros, cons, and general questions around Himalayan pink salt ensued.)
Ana: I quite like this but closer to flavored sparkling water. Like it has been in the same room as a strawberry. Tastes less sweet than it smells, less sweet than the banana cream. (To Dan: Where’s the cat?)
Me: Pleasant unlike the powerwash of Cherry Vanilla (full review to follow)
Ana: tastes very one-note
Me: Less like Olipop pop
Dan: I’m still surprised by the Himalayan pink salt

Day Two summary

Current stack rankings after 2 days and 5 pops:

Ana: Strawberry Vanilla, Tropical Punch, and Banana Cream are close together, a bit higher than Classic Grape. Classic Root Beer is decisively at the bottom.

Dan: Classic Root Beer is top, following by Tropical Punch, Banana Cream, Classic Grape, and Strawberry Vanilla

Me: Strawberry Vanilla is barely on top, then Tropical Punch, Classic Root Beer, Banana Cream, and Classic Grape

Day Three: Battle of the Oranges

Olipop has two orange-themed flavors, Orange Squeeze and Orange Cream. We decided to try these two side-by-side as the best way to compare their orange-iness.

Orange Squeeze

Color/Bouquet

Me: Smells like orangeade
Dan: More pulp than orangeade. It looks like a Tropicana orange product that has been puréed then added back.
Ana: Smells more like orange juice than soda.
Me: It smells a little like Tang.

Taste

Me: Sedate taste
Ana: Like a more pleasant Tang
Dan: I like it
Ana: Quite good, my favorite so far
Me: Not too sweet
Ana: It actually has more sugar than some of the others
Dan: Light Olipop light carbonations
Ana: Like more diluted Orange Fanta but in a good way

Overall

Ana: It’s just nice

Orange Cream

Color/Bouquet

Me: Does it live up to the hype?
Ana: Slightly cloudier than Orange Squeeze
Me: Smells more vanilla than Orange Squeeze

Taste

Dan: They really have nailed that creamsicle flavor
Me: I taste the orange first and then the “cream”
(Notes both Orange Squeeze and Orange Cream have 12% juice)
Me: it’s not like there’s milkfat or some solid to give it that cream feel

Impressions

Ana: When I’m thirsty I’d just go for the Orange Squeeze. Orange Cream isn’t really chuggable. Orange Cream is sweeter.
Dan: I feel like these would make good ice cream floats.
Me: Orange Cream would make a better after-dinner faux-liqueur

Day Three summary

New stack ranks:

Ana: Orange Squeeze and Orange Cream are at the top, depends on which I need

Dan: Orange Cream, Classic Root Beer, Orange Squeeze

Me: Orange Cream, Orange Squeeze, then Strawberry Vanilla

Day Four: Vintage Cola, Cherry Vanilla, Ginger Lemon

Vintage Cola

Note: Vintage Cola seems to be the only flavor with caffeine in it, which comes from some green tea.

Color/Bouquet

Dan: It smells like coke on an airplane
Ana: Yeah, it absolutely does smell like that, why is that?
We agree it has the traditional coke/cola/pop color, although it’s not as fizzy
I try to describe how I think maybe it smells a little like RC Cola, but only because I can’t remember how that smells (as opposed to Coke or Pepsi)
Ana: It smells of cola in a way Coke doesn’t
Mostly I think it smells flat

Taste

I also think it tastes flat
Dan: It feels even flatter than their (Olipop’s) root beer
Ana: I enjoy it because it is more lightweight
We talk a lot about how we can’t help but compare it to the mainstream colas, good or bad.
Dan: I still like it
Me: I just think it tastes like flat off-brand cola

Cherry Vanilla

Dan always calls this flavor Cherry Pie, because he likened drinking one to the experience of having a McDonalds cherry pie shoved into one’s mouth.

Ana, looking at the color: Garnet
Dan: Black cherry
Both: very pretty color
Ana: I barely notice a scent, it’s almost herbal
Me: Light medicinal overtone
Dan: We’re starting to get closer to its true horrible nature
Dan: It smells like maraschino cherry
Me: No, not nearly as sweet as maraschino, and this also tastes like it was near an actual cherry
Dan (reads label): “White cherry flavor”

At this point we come to realize that while we’re definitely trying the same flavor, the cans’ labels and lists of ingredients differ slightly. Apparently my deliveries of some flavors are from more recent stock. We decide any differences are probably negligible.

Back to the Cherry Vanilla:
Ana: I like the flavor
Dan: I want to put a shot glass of this on top of an ice cream sundae.
Me: Yeah, it does seem one step away from a hyper-concentrated syrup. Vintage Cola is too diluted and this one isn’t diluted enough.

Here Dan wonders what goes in actual cherry pie filling, leading the conversation into a rabbit hole involving black birds and other avian pies, down to a poisoned pigeon pie in a Game of Thrones episode.

Ginger Lemon

This is our final flavor of the ten available!

Color/Bouquet

Ana: It looks like lemonade
Dan: Nice smelling. I smell some ginger and some lemon
Ana: A little cloudy
Me: I smell more ginger

Taste

Dan: I’m not a big fan of ginger
Ana: I really like this. It’s one of my favorites. It’s not very sweet at all.
Me: I really like strong ginger. I wish this had more bite. It’s very sedate. The lemon is more just an aftertaste, like a hint of acid.
Ana: It tastes less like a soda because it’s less sweet. It tastes more sophisticated.

Impressions

Dan: I’m not sure I could tell when this goes flat. Very little carbonation.
We agree it’s probably the most “mature” of the flavors.

Final Rankings

After some discussion and contemplation, we came up with our personal stack ranks. Ana notes that she doesn’t really have a favorite as much as a drink she’s more likely to select for different needs. We all agree that none of the flavors are completely bad.

Me: There’s nothing I would absolutely refuse to drink again.
Ana: I think the root beer is there for me.

Well, that’s our take on the existing Olipop flavors we can get our hands on! Maybe as we get to try more, we’ll add them here! Or not, we have short attention spans.

Solving blindfolded can look really hard at first, but like most skills, you need to learn the basics and then practice a lot.

Why Learn to Solve Blindfolded

Learning how to solve a cube blindfolded is a good challenge, and once you learn how to do it, it’s a good way to exercise your short-term memory and your concentration. It may also be a cool party trick if people want to watch you for 5-10 minutes.

Basic Description

I use the Old Pochmann method. It’s not the fastest method, but it’s relatively straightforward, and I’m not trying to set any records.

Outline

1. Each tile of each piece is assigned a letter, A-X. Each letter will be used twice, once for an edge piece and once for a corner.
2. Edges have 2 letters
3. Corners have 3 letters
4. All algorithms are PLL moves. They only move pieces on the top layer.
5. To get the next target place into the top layer, you also need to perform a few intuitive moves.
6. We solve all the edges first, then the corners.
7. Inspecting the cube and memorizing the sequence.

Tutorial Conventions

• Cube orientation
  • Up – yellow
  • Front – red
• WCA Notation
• Algorithms
  • T perm – R U R’ U’ R’ F R2 U’ R’ U’ R U R’ F’
  • Ja perm – L U’ R’ U L’ U2 R U’ R’ U2 R
  • Jb perm – L’ U R U’ L U2 R’ U R U2 R’
• Practice scramble – R’ F’ Rw Uw’ Rw’ F2 R’ U’ Rw F R2 Fw’ Rw’ F’ R
  • 
• Terminology

  • Buffer piece – the next piece to move into the correct, target position

  • Target – the correct position for the current piece in the buffer

Lettering the Tiles

Each face of each piece, except the center pieces, get a unique letter. Each letter will be used twice, once for a corner and once for an edge.

I use this convention illustrated below. You can pick any lettering you like as long as you can remember it easily with some practice.

Tips

• Pick one lettering scheme that makes sense to you
• You can use a cheap cube and put letter stickers on each piece to help the learning phase

Algorithms

Blindfolded solving uses only PLL algorithms, specifically those which swap two edges and two corners at the same time.

You can solve the whole cube using just one algorithm, but I started by using T and Ja perms. I use both when solving the edges, and just Ja when solving corners.

Why these algorithms?

Because they swap a pair of edges and a pair of corners back and forth, and because we can move in targets without disturbing the pieces each algorithm moves. By just swapping known pairs back and forth, we don’t have to memorize different cube states.

• Ja perm
  • Corners: UBR, UFR
  • Edges: UB, UR
• T perm
  • Corners: UBR, UFR
  • Edges: UL, UR

These two algorithms also make solving “parity” easier, because you don’t have to learn a special algorithm. (See below.)

Tips

• Start with one or two PLL algorithms. Ja perm is a good choice!
• Use algorithms which swap one pair of edges and one pair of corners together.
• Practice the Ja and T permutation algorithms until you can do them with your eyes closed!

Moving the Target

While the algorithms are all PLL algos, you will always have to move the target piece to the top layer. You could memorize the moves for each piece to get it to the target position, but it will probably be easier to work intuitively. If you do want to make a list to memorize, be sure to work it against the algorithm(s) you will use.

The below images show the steps required to move the UR edge to FL. Remember, Up is yellow and Front is red.

Starting position: UR ('H' edge piece) -> FL ('H' edge target)

Use L' to move H face into staging 'D' position for T algorithm

Start to move ‘R’ from UR to FL

Work T algorithm

Use L to move 'H' edge piece into target position

Finish moving ‘H/R’ edge to target position

Corners work similarly. If you use the Ja permutation for corners, your target goes in the ‘C’ (UFR) position.

Tips

• Practice moving the edge tiles and corner tiles to their correct positions!

Inspection and Memorization

The sequence of moves

The sequence will always start with the buffer position. As you move the buffer piece into its correct target, the old target piece moves to the buffer. You can think of it like a queue with each tile moving through in turn.

Special case: if the ‘B/I’ edge moves into the buffer spot before you’ve solved all the edges, you need to find a new edge to complete the sequence.

Using the example scramble and the orientation (Up is yellow, Front is red) above, you would get this edge sequence:

K L E P Q O N X H A U .

In this example, we have all the edge pieces without a break in sequence. Count the number of letters: 11. Because we have an odd number of edges to move, we need to perform another Ja algorithm after we finish the edges but before we start the corners. This step is usually (inaccurately) called “parity.”

Now we sequence the corners. The cube has fewer corners than edges (8 vs 12), so we will probably have a shorter sequence to memorize.

We have a special case here: the buffer piece is already in the correct position. Let’s pick the ‘C’ (UFR) position to start with, because we don’t have to move a target in place.

C L R X Q F .

The ‘C’ and ‘F’ tiles are on the same corner, so we need to stop and find a new starting tile that has not been solved.

U W S .

Now we should have all the corners in the correct order.

Counting moves

• If you have an odd number of edge moves, you will have an odd number of corner moves.
• If you have an even number of edge moves, you will have an even number of corner moves.
• If one set is even and the other is odd, you are missing a move somewhere. Redo your inspection.

Tips

• Practice inspection first, writing down the sequence.
  • Work off the paper first
  • Then practice memorization
  • Then skip the paper – WCA rules don’t allow written notes
• Use a scramble generator so you can practice the same scramble several times to understand mistakes
• During inspection, use your fingers to keep track of which pieces you’ve added to the sequence.

Working the sequences

Holding the cube

• ALWAYS KEEP UP FACE YELLOW AND FRONT FACE RED
• One of the main ways to mess up is to change the orientation. Practice keeping the faces in place in your hands.
• You want to keep a tighter grip on the cube than usual to keep the orientation correct and to keep from dropping it.

“Parity”

• If your edge move count is odd, you will have to do an extra move after the edge sequence to get the buffer and target corners back to their original positions.
• If your buffer and target corners are at UBR and UFR (for Ja and T perms), you can just do an extra Ja before starting your corners.

Flipped pieces

• If you have all the pieces in the correct placement, but the tiles are flipped, you probably wrote down/memorized the wrong tile on the correct piece.
• Edges will always be flipped in pairs (2/4/etc.).
• 2 or more corners may be rotated incorrectly.

Next steps

This post already has a lot of information to digest. For memorization, you can develop mnemonic phrases to help. For learning the target set-up moves, you can practice to see which moves for each tile feel most natural to you. If requested, I can write out an exact set of moves and algorithms to solve the example.

If you don’t understand how some part of the method works, try to find another tutorial which makes more sense to you. Or you can ask here in the comments.

Most importantly, practice. Steps which may not be clear might make more sense after you’ve tinkered with them over time.

References

• https://github.com/x4Cx58x54/rubik-image – Code used to generate illustrations
• https://jperm.net/bld – One description of Old Pochmann
• https://ruwix.com/the-rubiks-cube/how-to-solve-the-rubiks-cube-blindfolded-tutorial/ – Another Old Pochmann method
• https://www.speedcubereview.com/blind-solving-algorithms.html – List of methods and algorithms that can be used for blindfolded solving

Previously I wrote about how I figured out how to solve the 3x3 pyramorphix (note: I wrote this post before I learned the CFOPmethod for cubes; you can use either!) and the 5x5 gigamorphix. You can use that 5x5 guide to solve a most parity issues for a megamorphix, but with the 4x4 puzzle and other morphix puzzles with an even number of layers, you may encounter one parity not found in the puzzles with an odd number of layers.

On these even layer puzzles, the center block on the last layer can be rotated 90° with respect to its edges and corners. We don’t see this issue with standard 4x4x4 speed cubes because those centers don’t have any visible difference if a center gets rotated.

In the megamorphix, this parity often isn’t obvious until you have tried to orient and permutate the last layer. If the corners and edges are in place, the center will be rotated. If the edges along with the center block, two corners will be swapped.

1. Solve the first 3 layers

1. Group the centers and the edge groups. You can solve edge groups (“dedges”) using the standard 4x4x4 speed cube methods. When solving the center groups, be sure to orient each center’s pieces. You can use the method from my 5x5x5 post.

You can use the CFOP method or whatever method you want to solve the first 3 layers.

2. Start solving the last layer

To orient and permute the edges and corners on the last layer, you can adopt the CFOP method. Correct a single flipped edge using a 4x4x4 dedge algorithm.

Next, pick the solution that matches your puzzle’s current configuration.

If you have solved the puzzle without a parity, you can stop now!

This megamorphix puzzle appears to be solved except for two corners on the last layer which are swapped

3. Re-align edges

If the edges and corners of your puzzle’s top layer are solved, you can skip to Step 4.

If your puzzle looks like this, although any two corners can be swapped on the last layer (instead of three possible corners, which we can fix using a U algorithm), use these algorithms to move the edge pieces around the center into their final configuration.

Note: U algorithms will rotate some centers by 180 °. Don’t worry, we will fix these at the end using an OLL algorithm. You can also cancel out the rotated centers by using two Ua permutations instead of one Ub permutation, or two Ub permutations instead of one Ua.

This megamorphix puzzle appears to be solved except for two corners on the last layer which are swapped

Use a U permutation to move the edges in a quarter .

Two of your edges should be flipped out like bird wings (the red edges in the example).

First we rotate the edges around the last layer center block by 90 degrees

Then we flip the pair of edges across from the other two, properly oriented edges around the last layer center block

Holding one of the second set of edges in UF and the other in UR, use the following algorithm to flip them into wings:

R B (M’ U’ M’ U’ M’ U M U’ M U’ M U2) B’ R’

The top layer of your puzzle should now look something like this, although the corners may be in different positions.

4. Rotate last center 90°

Now we need to rotate the center blocks so they fit the edges around them.

The easiest way to do this is to pick one of the solid center pieces and use commutators to rotate the whole block in 3 steps. The example here assumes the block needs to be rotated +90°.

Permuting the rotated center to the correct orientation on a 4x4 megamorphix puzzle

Note: Lower case letter in this notation mean move the inner slice only instead of the outer slice.

Always hold the “last” layer face as F. Always use the same color piece for each commatation. (The pictured example uses the red piece.) You will need to repeat the commutator algorithm 3 times. Be careful not to break up edge groups. Try to rotate the F and U faces back after each commutation to keep the first 3 layers solved.

To move the center clockwise

Match the solid pieces in positions UBR and UFR.

Algorithm: r U’ l’ U r’ U’ l U

To move the center counterclockwise

Match solid pieces in UBL and UFL.

Algorithm: l’ U r U’ l U r’ U’

In this example, the center needed to be rotated clockwise. I matched the red piece in the centers each tim. The picture shows the progress after the first commutation.

5. Solve corners on last layer

If you didn’t solve the corners with the edges before rotation the center, do that now.

After that, your puzzle should be solved!

In this example, we just need to use the A* permutations twice to position the corners. We may also need to flip one or two of the 3-color corners.

References

• https://www.reddit.com/r/Cubers/comments/3y7q36/megamorphix_help_please/
• https://www.speedsolving.com/wiki/index.php/4x4x4_parity_algorithms
• https://www.speedsolving.com/wiki/index.php/CFOP_method
• https://www.speedsolving.com/wiki/index.php/Commutator
• https://www.speedsolving.com/wiki/index.php/NxNxN_Notation
• https://www.speedsolving.com/wiki/index.php/Reduction_Method
• https://youtu.be/8KzCzqmHk88

Anyone who has watched Star Wars: A New Hope will, of course, remember that the Rebel Alliance was able to destroy the humongous Death Star. Blueprints of the space station, smuggled out of the Empire, revealed that the Death Star could be destroyed by just one well-aimed torpedo.

However, the failure of the Death Star was just one Imperial engineering disaster of many, albeit a very critical failure. Like many projects that don’t live up to their original ideal, a lack of focus on reliability and resiliency in the design phase, poor or missing design review, useless or nonexistent operational procedures, all tracing back to abysmal product management and arrogant leadership, doomed the Death Star and, ultimately, the Empire itself.

So, let’s break down Imperial engineering project failures in the films Episodes IV-VI. We’ll skip Episodes I-III, which predated the Empire, and VII-IX, because I don’t even want to try to pretend that I can explain what’s going on there. As a bonus, we will also throw in Rogue One, the direct prequel to Episode IV.

Episode IV: A New Hope

Let’s start with the obvious issue in the original installment: the Death Star comprised many systems with their own single points of failure (SPoFs; we’re going to use this acronym a lot). Of course, the most serious SPoF allowed a proton torpedo fired into an exterior exhaust vent to destroy the entire moon-sized space station. (We’ll get deeper into that design flaw when we break down Rogue One below.)

Design issues

1. The Death Star tractor beam used to pull the Millennium Falcon onto the battle station had not one but multiple single points of failure (SPoFs; we’re going to use this acronym even more): the entire tractor beam system could be disabled at any one of seven power couplings.
   Lesson: Redundancy of critical components over redundancy of engineering failure points.
2. That whole “you just need to shoot one proton torpedo into this exhaust vent and the whole Death Star will go boom!” issue has been well-covered.
   Lesson: You need better threat modeling.

Operational issues

While design and engineering issues built major vulnerabilities into the Death Star, the poor quality of the Empire’s operational practices and the limited ability of their personnel to carry out those procedures also created serious problems.

1. The gunners on the star destroyer in orbit above Tatooine were picking off escape pods leaving the captured Rebel ship. However, when they found no life signs in one of the pods, they let it go. Were laser blasts in such short supply?
   Lesson: Apply your security enforcement practices consistently.
2. Luke Skywalker, Han Solo, and Obi-Wan Kenobi leverage social engineering on multiple occasions to infiltrate and then escape from the Death Star. Luring Imperial stormtroopers onto the Millennium Falcon and stealing their armor; feigning com issues to trick the deck commander into opening their blast door; using Chewbacca as a “prisoner” to gain access to the detention block. While no single measure or procedure could have prevented all of these breaches, social engineering exploits start here and will continue to pose an ongoing problem for the Empire.
   Lesson: Don’t ignore non-technical security vulnerabilities.
3. At one point, we see two stormtroopers asking each other if they knew what was going on. “Maybe it’s another drill.” While it’s important to practice for incident readiness, the Death Star command and crew have apparently been missing critical attack vectors in their drill plans.
   Lesson: Practice only makes perfect if you’re practicing the right things the right way.

One last question: when the Death Star jumped into orbit around the planet Yavin IV, why didn’t they jump closer to the moon where the Rebellion was headquartered?

Episode V: Empire Strikes Back

This installment doesn’t really expose major Imperial engineering failures, although the bit about the Millennium Falcon hiding by clamping onto the back of a star destroyer’s command tower also seems a little inexplicable, even taking into account the Empire’s questionable engineering prowess.

The most relevant takeaway from this film: The expressions of people frozen in carbonite bear an uncanny resemblance to DMV photos in our universe.

Episode VI: Return of the Jedi

The Empire’s brilliant new plan: build a new Death Star! Could they pull out a win this time? Let’s find out!

• Win: While the main reactor in the original design could be destroyed by a torpedo fired into an external exhaust port, the main reactor in this new design could only be destroyed by direct blaster fire.
  Score: +1
• Loss: Conveniently for the Rebellion, the design also included navigable shafts that ran from the surface to the reactor core.
  Score: -2
• Win: The Death Star construction project did include a force field to prevent unauthorized access to these access shafts.
  Score: +5
• Loss: During construction, the Death Star’s shield had only a single generator, situated on the forest moon of Endor. Unfortunately for the Empire, any firewall, when built or operated with SPoFs at critical load-bearing points, can provide only severely limited and unreliable protection (yup, still seeing those single points of failure in action).
  Score: -10
• Win: The force field generator was well-guarded and housed in a unit with multiple blast doors, in an area heavily patrolled by Imperial scout transports.
  Score: +1 (limited points because it’s still unnecessarily a SPoF)
• Loss: Those safeguards were doomed to failure with the Empire’s typical lack of attention to social engineering exploits and non-standard combat styles.
  Score: -5

Hubris coming from the top down ultimately doomed the second Death Star and the entire Empire. Large projects can be more prone to failure if no one believes they can fail. This arrogance fuels the neglect to exploring or adopting measures that could increase the chance of success and mitigate clear risk. The Emperor had tied his success to this new Death Star. When that project failed, it cost him his job, among other losses.

Rogue One

While this film, “the prequel we never knew we needed,” takes place immediately before A New Hope, we’re covering it last on this list, and not just because of the chronological release order. This whole movie functions as the consummate how-to guide to ruining your most critical engineering projects.

With our step-by-step instructions, you will learn how to

• Motivate by threats!
• Alienate your workforce and key talent!
• Skip design reviews!
• Promote the people least capable of doing the job!
• Miss all your deadlines!
• Do whatever it takes to get your promotion and vanity title!
• Deliver a flawed but shiny product!
• Destroy the Empire!

All with pictures! (There are no pictures.)

Get your vanity title while trying to cover your ass!

Orson Krennic, who apparently holds the vanity title of “Director of Advanced Weapons Research,” is clearly an asshole, but he’s also reporting to even more tyrannical executives. Risk of being “managed out” or even missing out on regular job title inflation motivates many product managers. Krennic’s superiors constantly remind him that failure of the Death Star project would be met with a very tall enforcer who can force-choke Krennic just by making some squeezy hand gestures.

Lesson: Promote bad managers and motivate them with threats to get the worst out of them!

Steal credit when a project looks successful!

Krennic seems to expend as much energy trying to secure a promotion as he does overseeing the actual Death Star project. While politics generally do win out over actual job competence in too many organizations, failing to deliver a critical project still doesn’t generally work in one’s favor. When Krennic proceeds with a successful, but unsanctioned, test of the Death Star’s weapon, he not only fails to gain favor with his superiors, he also gets kicked off the project because someone else wants to take credit. Ouch.

Lesson: Be sure to leverage an apparently successful project for a promotion, but watch out for someone else who wants to steal credit!

Adopt involuntary recruitment and other methods to alienate your design engineers!

Krennic forces the brilliant researcher Galen Erso to create the Death Star only after murdering the scientist’s wife. That sort of antisocial behavior generally leads to disgruntled employees who might want to sabotage the project they’re working on, which is unsurprisingly the case here. Krennic also “motivates” Erso by threatening to kill his co-workers, who have as little freedom as Erso himself. The scientist feels he must continue to work on the Death Star, but he also hides a dangerous flaw in its implementation.

Lesson: Forcibly hire and torture your workforce for the most disastrous results!

Skip design reviews and quality assurance to ensure failure!

The Death Star project also proceeds again a backdrop of poor engineering design and brittle operational practices. Incompetent and dictatorial management has no idea how to design robust engineering infrastructure and does not understand the necessity of investing into these “overhead” areas. Such deficiencies often arise from an organizational focus limited to feature delivery, without recognizing the results of operational failures and the necessity of trying to prevent outages and limit the blast radius when they do occur, sometimes literally. They are also endemic to a culture that does not respect the expertise of its engineers.

Lesson: Leave out design reviews because quality and reliability assurances have no place in your product!

_ Now you should have all the skills you need to create an unsuccessful product and destabilize your entire organization! _

If you’re part of the Rebel Alliance, you may also want to check out our new book!

Learn how to

• Reprogram Imperial droids!
• Make those droids snarky!
• Steal Imperial transports and landing codes!
• Retrieve disks from Imperial archives!
• Run data cables to transmit stolen blueprints!
• Reposition satellite dishes!
• Take out planetary force fields!
• Disable star destroyers with ion blasts!
• And more!

In closing, we have a good idea of why the Empire failed. Don’t repeat their mistakes, unless, of course, you’re working from within to sabotage and destabilize your whole organization, too!

With the release of The Book of Boba Fett and the Disney+ slate of upcoming series, apparently one each for just about every character in the Star Wars universe, I’ve updated the list with a few more characters.

A few notes

• This list contains episodes which I personally think are important or interesting. Other people who have also sat through a bajillion hours of these shows and movies are entitled to make their own lists.
• For table entries with multiple episodes, the episodes comprise a story arc.
• I use the season.episode number convention. (Example: episode 3 of season 1 would be 1.3. An arc of episodes 12-14 from season 2 would be 2.12-14)
• Main characters/entities for an arc are in bold.
• Currently supported characters/entities
  • Ahsoka Tano
  • Mandalore
    • Bo-Katan Kryze
    • Darksaber
  • Boba Fett
  • Asajj Ventress (Clone Wars series villain)
  • Maul (oops, spoiler)
• Disney+ lists The Clone Wars in their original release order, which does not correspond at all with the actual story chronology until the final seasons. The list below conforms to the proper order.
• Bad Batch season 1 features many supporting characters we’ve seen in other series. I haven’t added any of them here, although that may change.
• Rebels has a more linear, self-contained storyline. You can watch the whole thing or not. Warning: Ezra Bridger is just so ducking annoying.

The Clone Wars

Rebels

Viewing chronology

1. Theatrical Episodes I & II
2. A few stray episodes from The Clone Wars
3. 2.16
4. 1.16
5. The Clone Wars animated movie
6. Lots of Clone Wars episodes totally out of order
7. Final 4 episodes (Season 7, episodes 9-12) of The Clone Wars. This arc overlaps with the film Episode III. (Arguments can be made for a different viewing order, but this is the one I like.)
8. CW episode 7.9
9. Episode III: Revenge of the Sith
10. Remaining CW episodes: 7.10-12
11. Star Wars Bad Batch
12. Star Wars Rebels
13. Films: Rogue One, Episodes IV-VI
14. The Mandalorian
15. The Book of Boba Fett jumps around in time but seems to take place immediately after season 2 of Mando
16. Star Wars Resistance: animated series leading up to Episode VII (I couldn’t watch past the first season)
17. Episodes VII-IX

References

• Chronological list of Clone Wars episodes
• Wookieepedia