Adventures in Freebernetes Tutorial: Build Your Own Bare-VM k3s Cluster

Part 3: Create VMs

  • 3.1 Choose Your Network Layout
  • 3.2 Create the Linux VMs
  • 3.1 Choose Your Network Layout

    3.1.1 Select Subnets

    My FreeBSD hypervisor has a 192.168.0.0/24 address on its physical network interface. I’m going to use a VLAN in 10.0.0.0/8 for the cluster and its pods and services. You can use another block, but you will have to adjust commands throughout the tutorial.

    • 10.0.0.1/32 – VLAN gateway on bridge interface
    • 10.0.0.2/32 – Virtual IP for Kubernetes API endpoint
    • 10.0.10.0/24 – VM block
      • 10.0.10.1[1-3] – K3s servers
      • 10.0.10.2[1-3] – K3s agents (nodes)
    • 10.1.0.0/16 – pod network
    • 10.2.0.0/16 – service network

    3.1.2 Pick a .local Zone for DNS

    This zone just needs to resolve locally on the FreeBSD host. I’m going with k3s.local because I’m too lazy to think of a clever pun right now.

    3.2 Create the Linux VMs

    3.2.1 Initialize CBSD

    If you haven’t run CBSD on your FreeBSD host before, you will need to set it up. You can use this seed file. Edit it first to set node_name to your FreeBSD host’s name and to change jnameserver and nodeippool if you are using a private network other than 10.0.0.0/8.

    ~ # sysrc cbsd_workdir="/usr/cbsd"
    cbsd_workdir: -> /usr/cbsd
    ~ # wget https://raw.githubusercontent.com/kbruner/freebernetes/main/k3s/cbsd/initenv.conf
    [ edit initenv.conf ]
    ~ # /usr/local/cbsd/sudoexec/initenv inter=0 `pwd`/initenv.conf # need full path for initenv.conf ¯\_(ツ)_/¯
    [ lots of output ]
    ~ # grep cbsd /etc/rc.conf
    cbsd_workdir="/usr/cbsd"
    cbsdrsyncd_enable="YES"
    cbsdrsyncd_flags="–config=/usr/cbsd/etc/rsyncd.conf"
    cbsdd_enable="YES"
    ~ # service cbsdrsyncd stop
    ~ # sysrc -x cbsdrsyncd_enable
    ~ # sysrc -x cbsdrsyncd_flags
    ~ # grep cbsd /etc/rc.conf
    cbsd_workdir="/usr/cbsd"
    cbsdd_enable="YES"
    sysrc cbsd_workdir="/usr/cbsd"
    wget https://raw.githubusercontent.com/kbruner/freebernetes/main/k3s/cbsd/initenv.conf
    vi initenv.conf
    /usr/local/cbsd/sudoexec/initenv inter=0 `pwd`/initenv.conf
    service cbsdrsyncd stop
    sysrc -x cbsdrsyncd_enable
    sysrc -x cbsdrsyncd_flags
    view raw commands.sh hosted with ❤ by GitHub

    3.2.2 Create VMs

    Copy this instance.jconf VM template file and update ci_gw4, ci_nameserver_search, and ci_nameserver_address fields as needed. If you want to set a password for the ubuntu user in case you want to log in on the console via VNC, you can assign it to cw_user_pw_user, but note this is a plain-text field.

    When you run cbsd bcreate, if CBSD does not have a copy of the installation ISO image, it will prompt you asking to download it. After the first time, it will re-use the local image.

    ~ # for i in 0 1 2; do
    cbsd bcreate jconf=/root/instance.jconf jname="server-$i" \
    ci_ip4_addr="10.0.10.1${i}/24" ci_jname="server-$i" \
    ci_fqdn="server-${i}.k3s.local" ip_addr="10.0.10.1${i}" \
    imgsize="20g" vm_cpus="2" vm_ram="2g"
    done
    Global VM ZFS guid: 414491023960324541
    To edit VM properties use: cbsd bconfig jname=server-0
    To start VM use: cbsd bstart server-0
    To stop VM use: cbsd bstop server-0
    To remove VM use: cbsd bremove server-0
    For attach VM console use: cbsd blogin server-0
    Creating server-0 complete: Enjoy!
    auto-generate cloud-init settings: /usr/cbsd/jails-system/server-0/cloud-init
    [ similar output for server-1 and server-2 ]
    ~ # for i in 0 1 2; do cbsd bstart jname="server-$i"; done
    cloud-init: enabled
    vm_iso_path: cloud-ubuntu-x86-20.04.1
    cloud init image initialization..
    Clone cloud image into first/system vm disk (zfs clone method)
    /sbin/zfs get -Ht snapshot userrefs zroot/ROOT/default/cbsd-cloud-cloud-Ubuntu-x86-20.04.1.raw@boot-server-0
    Eject cloud source: media mode=detach name=cloud-ubuntu-x86-20.04.1 path=/usr/cbsd/src/iso/cbsd-cloud-cloud-Ubuntu-x86-20.04.1.raw type=iso jname=server-0
    DELETE FROM media WHERE name="cloud-ubuntu-x86-20.04.1" AND path="/usr/cbsd/src/iso/cbsd-cloud-cloud-Ubuntu-x86-20.04.1.raw" AND jname="server-0"
    vm_iso_path: changed
    Detach to: server-0
    All CD/ISO ejected: server-0
    VRDP is enabled. VNC bind/port: 127.0.0.1:5901
    For attach VM console, use: vncviewer 127.0.0.1:5901
    Resolution: 1024×768.
    bhyve renice: 1
    Execute master script: cloud_init_set_netname.sh
    :: /usr/cbsd/jails-system/server-0/master_prestart.d/cloud_init_set_netname.sh
    Waiting for PID.
    PID: 15858
    CBSD setup: bhyve ipfw counters num: 99/100
    [ similar output for server-1 and server-2 ]
    [ repeat for agent VMs ]
    # create server VMs
    for i in 0 1 2; do
    cbsd bcreate jconf=/root/instance.jconf jname="server-$i" \
    ci_ip4_addr="10.0.10.1${i}/24" ci_jname="server-$i" \
    ci_fqdn="server-${i}.k3s.local" ip_addr="10.0.10.1${i}" \
    imgsize="20g" vm_cpus="2" vm_ram="2g"
    done
    # start server VMs
    for i in 0 1 2; do cbsd bstart jname="server-$i"; done
    # create agent VMs
    for i in 0 1 2; do
    cbsd bcreate jconf=/root/instance.jconf jname="agent-$i" \
    ci_ip4_addr="10.0.10.2${i}/24" ci_jname="agent-$i" \
    ci_fqdn="agent-${i}.k3s.local" ip_addr="10.0.10.2${i}" \
    imgsize="10g" vm_cpus="2" vm_ram="2g"
    done
    # start agent VMs
    for i in 0 1 2; do cbsd bstart jname="agent-$i"; done
    view raw commands.sh hosted with ❤ by GitHub

    Pages: 1 2 3 4 5 6 7 8

    Leave a Reply

    Fill in your details below or click an icon to log in:

    WordPress.com Logo

    You are commenting using your WordPress.com account. Log Out /  Change )

    Google photo

    You are commenting using your Google account. Log Out /  Change )

    Twitter picture

    You are commenting using your Twitter account. Log Out /  Change )

    Facebook photo

    You are commenting using your Facebook account. Log Out /  Change )

    Connecting to %s

    Blog at WordPress.com.

    Up ↑

    %d bloggers like this: