Adventures in Freebernetes Tutorial: Build Your Own Bare-VM k3s Cluster

Part 4: Configure Networking

  • 4.1 Add Bridge Gateways
  • 4.2 Configure NAT
  • 4.3 Configure Local DNS
  • You cannot connect to the new VMs yet. CBSD creates a bridge interface the first time you create a VM. We need to add gateways for our cluster VLANs to that interface so we can route from the hypervisor to the VMs and vice versa. In most cases, CBSD will use the bridge1 interface.

    4.1 Add Bridge Gateways

    Note that these changes will not survive across reboots. I have not tested if adding a persistent entry for bridge1 in /etc/rc.conf would work as expected with CBSD, as it manages the bridge1 interface.

    ~ # ifconfig bridge1 alias 10.0.0.1/32
    ~ # ifconfig bridge1 alias 10.0.10.1/24 # VM network
    ~ # ifconfig bridge1 alias 10.1.0.1/16 # pod network
    ~ # ifconfig bridge1 alias 10.2.0.1/16 # service network
    ~ # ifconfig bridge1
    bridge1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: em0
    ether 58:9c:fc:10:ff:b8
    inet 10.0.0.1 netmask 0xffffffff broadcast 10.0.0.1
    inet 10.0.10.1 netmask 0xffffff00 broadcast 10.0.10.255
    inet 10.1.0.1 netmask 0xffff0000 broadcast 10.1.255.255
    inet 10.2.0.1 netmask 0xffff0000 broadcast 10.2.255.255
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    [ … ]
    view raw commands + output hosted with ❤ by GitHub
    ifconfig bridge1 alias 10.0.0.1/32
    ifconfig bridge1 alias 10.0.10.1/24
    ifconfig bridge1 alias 10.1.0.1/16
    ifconfig bridge1 alias 10.2.0.1/16
    view raw commands.sh hosted with ❤ by GitHub

    4.2 Configure NAT

    We can reach our VM just fine from the host, but the VMs can’t talk to the Internet because only the FreeBSD host can route to this 10.0.0.0/8 block. We will use ipfw as a NAT (Network Address Translation) service. These steps will enable ipfw with open firewall rules and then configure the NAT. These changes will take effect immediately. The service and kernel settings will persist across reboots, but the ipfw firewall rules will not. See the ipfw chapter about how to create and enable a firewall script.

    Note that my host’s physical interface is named em0. You may have to alter some commands if yours has a different name.

    ~ # kenv net.inet.ip.fw.default_to_accept=1
    net.inet.ip.fw.default_to_accept="1"
    ~ # echo net.inet.ip.fw.default_to_accept=1 >> /boot/loader.conf
    ~ # sysrc firewall_enable="YES"
    firewall_enable: NO -> YES
    ~ # sysrc gateway_enable="YES"
    gateway_enable: NO -> YES
    ~ # sysrc firewall_nat_enable="YES"
    firewall_nat_enable: NO -> YES
    ~ # sysctl net.inet.tcp.tso=0
    net.inet.tcp.tso: 0 -> 0
    ~ # echo net.inet.tcp.tso="0" >> /etc/sysctl.conf
    ~ # service ipfw start
    Firewall logging enabled.
    ~ # ipfw disable one_pass
    ~ # ipfw -q nat 1 config if em0 same_ports unreg_only reset
    ~ # sysctl net.inet.ip.fw.enable=1
    net.inet.ip.fw.enable: 0 -> 1
    ~ # sysctl net.inet.ip.forwarding=1
    net.inet.ip.forwarding: 0 -> 1
    ~ # sysctl net.inet6.ip6.forwarding=1
    net.inet6.ip6.forwarding: 0 -> 1
    ~ # ipfw add 1 allow ip from any to any via lo0
    00001 allow ip from any to any via lo0
    ~ # ipfw add 200 reass all from any to any in
    00200 reass ip from any to any in
    ~ # ipfw add 201 check-state
    00201 check-state :default
    ~ # ipfw add 205 nat 1 ip from 10.0.0.0/8 to any out via em0
    00205 nat 1 ip from 10.0.0.0/8 to any out via em0
    ~ # ipfw add 210 nat 1 ip from any to any in via em0
    00210 nat 1 ip from any to any in via em0
    ~ # ipfw show
    00001 0 0 allow ip from any to any via lo0
    00200 2689 197170 reass ip from any to any in
    00201 0 0 check-state :default
    00205 0 0 nat 1 ip from 10.0.0.0/8 to any out via em0
    00210 46 3188 nat 1 ip from any to any in via em0
    65535 106815 10861896 allow ip from any to any
    view raw commands + output hosted with ❤ by GitHub
    kenv net.inet.ip.fw.default_to_accept=1
    echo net.inet.ip.fw.default_to_accept=1 >> /boot/loader.conf
    sysrc firewall_enable="YES"
    sysrc gateway_enable="YES"
    sysrc firewall_nat_enable="YES"
    sysctl net.inet.tcp.tso=0
    echo net.inet.tcp.tso="0" >> /etc/sysctl.conf
    service ipfw start
    ipfw disable one_pass
    ipfw -q nat 1 config if em0 same_ports unreg_only reset
    sysctl net.inet.ip.fw.enable=1
    sysctl net.inet.ip.forwarding=1
    sysctl net.inet6.ip6.forwarding=1
    ipfw add 1 allow ip from any to any via lo0
    ipfw add 200 reass all from any to any in
    ipfw add 201 check-state
    ipfw add 205 nat 1 ip from 10.0.0.0/8 to any out via em0
    ipfw add 210 nat 1 ip from any to any in via em0
    view raw commands.sh hosted with ❤ by GitHub

    4.3 Configure Local DNS

    We need a way to resolve our VM host names. We need to pick a private .local DNS domain, configure an authoritative server for the domain, and then set up a local caching server that knows about our domain but can also still resolve external addresses for us. We will follow this nsd/unbound tutorial closely.

    4.3.1 Enable unbound for recursive/caching DNS

    FreeBSD has a caching (lookup-only) DNS service called unbound in the base system. It will use the configured nameservers for external address lookups and the local nsd service (configured next) for lookups to our private zone. Copy unbound.conf and make any edits as necessary to IP addresses or your local zone name.

    You will also want to update the FreeBSD host’s /etc/resolv.conf to add your local domain to the search list and add an entry for nameserver 127.0.0.1.

    wget https://raw.githubusercontent.com/kbruner/freebernetes/main/k3s/dns/unbound/unbound.conf -O /etc/unbound/unbound.conf
    sysrc local_unbound_enable="YES"
    service local_unbound start
    view raw commands.sh hosted with ❤ by GitHub

    4.3.2 Configure the Authoritative DNS Service

    We will use nsd, a lightweight, authoritative-only service, for our local zone. After copying the files, you can edit/rename the copied files before proceeding to make changes as necessary to match your local domain or IP addresses.

    ~ # mkdir -p /var/nsd/var/db/nsd /var/nsd/var/run /var/nsd/var/log /var/nsd/tmp
    ~ # chown -R nsd:nsd /var/nsd
    ~ # sysrc nsd_enable="YES"
    nsd_enable: -> YES
    ~ # sysrc nsd_config="/var/nsd/nsd.conf"
    nsd_config: -> /var/nsd/nsd.conf
    ~ # cd /var/nsd
    /var/nsd # wget -q https://raw.githubusercontent.com/kbruner/freebernetes/main/k3s/dns/nsd/nsd.conf
    /var/nsd # wget -q https://raw.githubusercontent.com/kbruner/freebernetes/main/k3s/dns/nsd/zone.10
    /var/nsd # wget -q https://raw.githubusercontent.com/kbruner/freebernetes/main/k3s/dns/nsd/zone.k3s.local
    /var/nsd # nsd-control-setup -d /var/nsd
    setup in directory /var/nsd
    Generating RSA private key, 3072 bit long modulus (2 primes)
    ……………..++++
    …………………………………………………………………………………++++
    e is 65537 (0x010001)
    Generating RSA private key, 3072 bit long modulus (2 primes)
    .++++
    ……………….++++
    e is 65537 (0x010001)
    Signature ok
    subject=CN = nsd-control
    Getting CA Private Key
    removing artifacts
    Setup success. Certificates created. Enable in nsd.conf file to use
    /var/nsd # nsd-control -c /var/nsd/nsd.conf start
    [2020-12-24 23:28:08.976] nsd[35621]: notice: nsd starting (NSD 4.3.4)
    [2020-12-24 23:28:08.976] nsd[35621]: notice: listen on ip-address 127.0.0.1@53530 (udp) with server(s): *
    [2020-12-24 23:28:08.976] nsd[35621]: notice: listen on ip-address 127.0.0.1@53530 (tcp) with server(s): *
    view raw commands + output hosted with ❤ by GitHub
    mkdir -p /var/nsd/var/db/nsd /var/nsd/var/run /var/nsd/var/log /var/nsd/tmp
    chown -R nsd:nsd /var/nsd
    sysrc nsd_enable="YES"
    sysrc nsd_config="/var/nsd/nsd.conf"
    cd /var/nsd
    wget -q https://raw.githubusercontent.com/kbruner/freebernetes/main/k3s/dns/nsd/nsd.conf
    wget -q https://raw.githubusercontent.com/kbruner/freebernetes/main/k3s/dns/nsd/zone.10
    wget -q https://raw.githubusercontent.com/kbruner/freebernetes/main/k3s/dns/nsd/zone.k3s.local
    nsd-control-setup -d /var/nsd
    nsd-control -c /var/nsd/nsd.conf start
    view raw commands.sh hosted with ❤ by GitHub

    Pages: 1 2 3 4 5 6 7 8

    Leave a Reply

    Fill in your details below or click an icon to log in:

    WordPress.com Logo

    You are commenting using your WordPress.com account. Log Out /  Change )

    Google photo

    You are commenting using your Google account. Log Out /  Change )

    Twitter picture

    You are commenting using your Twitter account. Log Out /  Change )

    Facebook photo

    You are commenting using your Facebook account. Log Out /  Change )

    Connecting to %s

    Blog at WordPress.com.

    Up ↑

    %d bloggers like this: