Adventures in Freebernetes Tutorial: Build Your Own Bare-VM k3s Cluster

Step-by-step tutorial for deploying a Kubernetes cluster with k3s on FreeBSD bhyve VMs

Overview

In the last tutorial in this series, we went through the steps of creating a Kubernetes cluster on FreeBSD bhyve VMs, adapting the completely manual process in Kelsey Hightower’s Kubernetes the Hard Way. This tutorial will also build a Kubernetes cluster on bhyve, but it will install a lightweight k3s control plane using the k3sup tool, which automates much of the process.

Topics covered:

Compiling k3sup on FreeBSD
Configuring CBSD to create and manage bhyve VMs
Running k3sup to bring up a k3s cluster on bhyve VMs
Configuring the FreeBSD firewall, DNS, and routing for cluster networking

While this tutorial builds a cluster with a redundant control plane, all the VMs are on a single hypervisor, making it suitable for testing and experimentation, but it is not production grade.

This tutorial only covers creating a cluster. For Kubernetes basics and terminology, you should start with the official Kubernetes documentation.

Intended Audience

For this tutorial, you don’t need to know anything about Kubernetes. You do need to have a host with FreeBSD installed; an understanding of basic FreeBSD system administration tasks, such as installing software from FreeBSD ports or packages, and loading kernel modules; and familiarity with csh or sh. Experience with FreeBSD bhyve virtual machines and the CBSD interface is useful but not required.

Technical Specs

Kubernetes Cluster Specs

We will use these targets for the Kubernetes cluster we’re building. Most of these settings can be adjusted.

Control plane (“K3s servers”)
- 3 VMs: 2 CPU cores, 2Gb RAM, 20Gb virtual disk each
- If you plan on creating more than 10 worker nodes, increase the control plane VM sizes
Worker nodes (“K3s agents”)
- 3 VMs: 2 CPU cores, 2Gb RAM, 10Gb virtual disk each
VM OS: Ubuntu Server 20.04
Kubernetes version: 1.19
Control plane backing database: embedded etcd
- K3s also supports using an external datastore, such as MySQL

Host (FreeBSD Hypervisor) Requirements

Hardware, physical or virtual
- CPU
  - CPUs must support FreeBSD bhyve virtualization (see the FreeBSD Handbook page on bhyve for compatible CPUs)
  - CPU core allocations for bhyve VMs are completely virtual, so you can have VMs running with a total virtual core count greater than your host system’s. You should use the suggested core count for VMs, as Kubernetes will use those for scheduling. You can increase the cores if you have a larger host.
- RAM: Minimum 2Gb per VM. You can use less for the agent VMs if necessary.
- Free disk space: 100Gb
Operating system
- FreeBSD: 13.0-CURRENT. It may work with 12.0-RELEASE, but it has not been tested
- File system: ZFS

My Test System

Hardware
- CPUs: Intel(R) Core(TM) i5-6260U: 4 CPUs, 2 cores each
- RAM: 32Gb
Operating system
- FreeBSD 13.0-HEAD-f659bf1d31c