Adventures in Freebernetes: A Bridge Not Far Enough

Part 6 of experiments in FreeBSD and Kubernetes: Automated VM installation with CBSD and falling down the network rabbit hole

See all posts in this series

Automated CBSD VMs with Cloud-Init

In the previous post, we cloned a VM by using a snapshot of the ZFS volume of an Arch Linux volume we had installed from (virtual) CD. You can also automate OS bootstrapping in CBSD by using cloud-init.

I speed through cbsd bconstruct-tui, use all the defaults (except changing the default VNC address to the NUC’s network interface card (NIC)), and hit go. Except then it tells me I need to set cloud_options so I go back and do that, using all the defaults except user password. (I took the screenshot before changing that.)

Screenshot of the cloud-init options menu in cbsd
cloud-init options menu
Main cbsd bhyve menu with everything filled out
Ok, ready to go
Global VM ZFS guid: 8726167484183186502
To edit VM properties use: cbsd bconfig jname=debian1
To start VM use: cbsd bstart debian1
To stop VM use: cbsd bstop debian1
To remove VM use: cbsd bremove debian1
For attach VM console use: cbsd blogin debian1
Creating debian1 complete: Enjoy!
auto-generate cloud-init settings: /usr/cbsd/jails-system/debian1/cloud-init
root@nucklehead:~ # cbsd bstart debian1
cloud-init: enabled
vm_iso_path: cloud-debian-x86-10.4.0
cloud init image initialization..
Clone cloud image into first/system vm disk (zfs clone method)
/sbin/zfs get -Ht snapshot userrefs zroot/ROOT/default/cbsd-cloud-cloud-Debian-x86-10.4.0.raw@boot-debian1
Eject cloud source: media mode=detach name=cloud-debian-x86-10.4.0 path=/usr/cbsd/src/iso/cbsd-cloud-cloud-Debian-x86-10.4.0.raw type=iso jname=debian1
UPDATE media SET jname='-' WHERE jname="debian1" AND name="cloud-debian-x86-10.4.0" AND path="/usr/cbsd/src/iso/cbsd-cloud-cloud-Debian-x86-10.4.0.raw"
vm_iso_path: changed
Detach to: debian1
All CD/ISO ejected: debian1
VRDP is enabled. VNC bind/port: 192.168.0.11:5903
For attach VM console, use: vncviewer 192.168.0.11:5903
Resolution: 1024×768.
em0
bhyve renice: 1
Execute master script: cloud_init_set_netname.sh
:: /usr/cbsd/jails-system/debian1/master_prestart.d/cloud_init_set_netname.sh
Waiting for PID.
PID: 23914
root@nucklehead:~ #
view raw gistfile1.txt hosted with ❤ by GitHub

It boots up and everything on the console (viewed via VNC) looks ok.

Ok, great, except… the new VM’s IP defaulted to 10.0.0.1. And my NUC and LAN subnet are in the 192.168.0.0/24 space, sooooo…. that’s not good. Or is it?

I try ssh debian@10.0.0.1 and it fails to connect. Ok, so now we enter…

The Rabbit Hole

So, ok, the VM virtual NICs are all members of the same bridge, so shouldn’t I be able to route to the VM’s virtual NIC?

root@nucklehead:~ # ssh debian@10.0.0.1
ssh: connect to host 10.0.0.1 port 22: Operation timed out
root@nucklehead:~ # ifconfig bridge1
bridge1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: em0
ether 58:9c:fc:10:ff:b8
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto stp-rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: tap5 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 8 priority 128 path cost 2000000
member: tap4 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 7 priority 128 path cost 2000000
member: tap3 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 6 priority 128 path cost 2000000
member: tap2 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 5 priority 128 path cost 2000000
member: tap1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 4 priority 128 path cost 2000000
member: em0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 1 priority 128 path cost 20000
groups: bridge
nd6 options=1<PERFORMNUD>
root@nucklehead:~ # netstat -r4
Routing tables
Internet:
Destination Gateway Flags Netif Expire
default Broadcom UGS em0
localhost link#2 UH lo0
192.168.0.0/24 link#1 U em0
nucklehead link#1 UHS lo0
view raw gistfile1.txt hosted with ❤ by GitHub

Oh, hey, there’s no entry for 10.0.0.0/8 or specifically 10.0.0.1, so traffic to it goes through the default gateway, which is the physical NIC (em0) connected to my router. My router doesn’t know about 10.0.0.1 because it’s not configured to.

So, I could try adding a static route in my router, or I could try adding a local route on the NUC, right?

Well, CBSD didn’t give bridge1 an IP address. FreeBSD’s route(8) command requires an IP address rather than an interface for a route’s gateway. Hrm, ok. First I’ll give it the IP address for em0 since that’s a member of bridge1 along with tap5.

root@nucklehead:~ # route add -net 10.0.0.0/8 192.168.0.11
add net 10.0.0.0: gateway 192.168.0.11
root@nucklehead:~ # netstat -r4
Routing tables
Internet:
Destination Gateway Flags Netif Expire
default Broadcom UGS em0
10.0.0.0/8 nucklehead UGS em0
localhost link#2 UH lo0
192.168.0.0/24 link#1 U em0
nucklehead link#1 UHS lo0
root@nucklehead:~ # ssh debian@10.0.0.1
ssh: connect to host 10.0.0.1 port 22: No route to host
view raw gistfile1.txt hosted with ❤ by GitHub

Ok, that didn’t work. What if we give bridge1 an IP address?

root@nucklehead:~ # dhclient bridge1
DHCPDISCOVER on bridge1 to 255.255.255.255 port 67 interval 7
DHCPOFFER from 192.168.0.1
DHCPREQUEST on bridge1 to 255.255.255.255 port 67
DHCPACK from 192.168.0.1
bound to 192.168.0.14 — renewal in 43200 seconds.
root@nucklehead:~ # route change -net 10.0.0.0/8 192.168.0.14
change net 10.0.0.0: gateway 192.168.0.14
root@nucklehead:~ # netstat -r4
Routing tables
Internet:
Destination Gateway Flags Netif Expire
default Broadcom UGS em0
10.0.0.0/8 nucklehead UGS em0
localhost link#2 UH lo0
192.168.0.0/24 link#1 U em0
nucklehead link#1 UHS lo0
nucklehead link#3 UHS lo0
root@nucklehead:~ # ssh debian@10.0.0.1
ssh: connect to host 10.0.0.1 port 22: No route to host
view raw gistfile1.txt hosted with ❤ by GitHub

That didn’t help.

To try to trace whether the connection is even trying to send packets through bridge1, I fire up tcpdump -i bridge1 'host 10.0.0.1' and get nothing. I try watching on the em0 interface, nothing. I remove the 10.0.0.0/8 rule and try again. Ah, yes, without a route entry for 10.0.0.0/8, it does send the packets out on em0. (Of course, we never get a reply.)

root@nucklehead:~ # tcpdump -i bridge1 –immediate-mode 'host 10.0.0.1'
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on bridge1, link-type EN10MB (Ethernet), capture size 262144 bytes
^C
0 packets captured
22 packets received by filter
0 packets dropped by kernel
root@nucklehead:~ # tcpdump -i em0 –immediate-mode 'host 10.0.0.1'
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on em0, link-type EN10MB (Ethernet), capture size 262144 bytes
^C
0 packets captured
3175 packets received by filter
0 packets dropped by kernel
root@nucklehead:~ # tcpdump -i em0 –immediate-mode 'host 10.0.0.1'
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on em0, link-type EN10MB (Ethernet), capture size 262144 bytes
14:36:08.694852 IP nucklehead.49297 > 10.0.0.1.ssh: Flags [S], seq 3203534683, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 338465326 ecr 0], length 0
14:36:09.730896 IP nucklehead.49297 > 10.0.0.1.ssh: Flags [S], seq 3203534683, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 338466362 ecr 0], length 0
14:36:11.944779 IP nucklehead.49297 > 10.0.0.1.ssh: Flags [S], seq 3203534683, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 338468576 ecr 0], length 0
14:36:16.144386 IP nucklehead.49297 > 10.0.0.1.ssh: Flags [S], seq 3203534683, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 338472776 ecr 0], length 0
view raw gistfile1.txt hosted with ❤ by GitHub

By now I’ve read everything about FreeBSD bridging and my google-fu isn’t picking up anything which seems like an obvious answer to my issue. However, I did not RTFM very well.

Bridges bridge networks. While dhclient had assigned a IP address to bridge1, it was effectively treated as a /32 subnet. I had this a-ha moment re-reading the handbook page on bridging, looking at the examples like ifconfig bridge0 inet 192.168.0.1/24. It’s not assigning a single IP address; it’s assigning the class C network 192.168.0.0/24 (not a typo; 192.168.0.0/24 and 192.168.0.1/24 are equivalent) to the bridge interface.

And at this point I start to wonder how much IPv4 knowledge my brain has lost and if I should re-read Stevens Vol 1 instead of using it as a stepping stool when I can’t reach the top shelf of the bookcase.

Ok. so I kill the dhclient process for bridge1, remove the 192.168.0.14/32 IP, add 10.0.0.0/8, and…

root@nucklehead:~ # ifconfig bridge1 -alias
root@nucklehead:~ # ifconfig bridge1
bridge1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: em0
ether 58:9c:fc:10:ff:b8
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto stp-rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: tap5 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 8 priority 128 path cost 2000000
member: tap4 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 7 priority 128 path cost 2000000
member: tap3 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 6 priority 128 path cost 2000000
member: tap2 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 5 priority 128 path cost 2000000
member: tap1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 4 priority 128 path cost 2000000
member: em0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 1 priority 128 path cost 20000
groups: bridge
nd6 options=1<PERFORMNUD>
root@nucklehead:~ # ifconfig bridge1 inet 10.0.0.0/8
root@nucklehead:~ # ifconfig bridge1
bridge1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: em0
ether 58:9c:fc:10:ff:b8
inet 10.0.0.0 netmask 0xff000000 broadcast 10.255.255.255
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto stp-rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: tap5 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 8 priority 128 path cost 2000000
member: tap4 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 7 priority 128 path cost 2000000
member: tap3 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 6 priority 128 path cost 2000000
member: tap2 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 5 priority 128 path cost 2000000
member: tap1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 4 priority 128 path cost 2000000
member: em0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 1 priority 128 path cost 20000
groups: bridge
nd6 options=1<PERFORMNUD>
root@nucklehead:~ # ssh debian@10.0.0.1
The authenticity of host '10.0.0.1 (10.0.0.1)' can't be established.
ECDSA key fingerprint is SHA256:QE9Jkvn6Ctto1063XnjbtCFnR9hKmxoNx7/rBjyOMfI.
No matching host key fingerprint found in DNS.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '10.0.0.1' (ECDSA) to the list of known hosts.
debian@10.0.0.1's password:
Linux debian1 4.19.0-9-amd64 #1 SMP Debian 4.19.118-2 (2020-04-29) x86_64
The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.
Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Fri Nov 13 14:30:55 2020
——————: System Data :——————————-
Hostname: debian1 (10.0.0.1 )
Kernel: 4.19.0-9-amd64 (Debian GNU/Linux 10 (buster))
Uptime: 15:00:05 up 57 min, 2 users, load average: 0.00, 0.00, 0.00
CPU: Intel(R) Core(TM) i5-6260U @ 1.80GHz (1 cores)
Memory(Mb): 985 total / 842 free
Env info:
————————: Logged as: [debian] ——————————
This image was created for ClonOS/CBSD Project.
ClonOS/CBSD user group: https://web.telegram.org/#/im?p=%40cbsdofficial
CBSD issues page: https://github.com/cbsd/cbsd/issues
Please Support Us: https://clonos.tekroutine.com/contact.html
debian@debian1:~$
view raw gistfile1.txt hosted with ❤ by GitHub

Yay! Ok, we can now exit that rabbit hole.

Still in the Rabbit Hole

Ok, that’s great, except even though I can make a successful bi-directional connection to my 10.0.0.1 VM, the VM itself has no idea how to get out.

debian@debian1:~$ ssh napalm@192.168.0.11
ssh: connect to host 192.168.0.11 port 22: Network is unreachable
debian@debian1:~$ netstat -r
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
10.0.0.0 0.0.0.0 255.255.255.0 U 0 0 0 enp0s5
debian@debian1:~$
view raw gistfile1.txt hosted with ❤ by GitHub

Ok, so I need to add a default route. I assume it should use the VM’s interface as the gateway, as it’s a member of the NUC’s bridge and, well, nothing else has a 10.0.0.0/8 address to act as a gateway.

debian@debian1:~$ sudo /sbin/route add default gw 10.0.0.1
debian@debian1:~$ netstat -r
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
^C
debian@debian1:~$ /sbin/route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
^C
debian@debian1:~$ !ssh
ssh napalm@192.168.0.11
The authenticity of host '192.168.0.11 (192.168.0.11)' can't be established.
ECDSA key fingerprint is SHA256:kUuDGjs7t6D9c5hVPfKkbiP6HAwkr5G7xCLgNNizasA.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.168.0.11' (ECDSA) to the list of known hosts.
Password for napalm@nucklehead:
view raw gistfile1.txt hosted with ❤ by GitHub

Success! I’m not sure why the attempts to read the routing table seemed to hang (I only gave them a few seconds before interrupting, but normally they return quickly), but enough of this networking rabbit hole.


Ok, that was very exciting, We booted and installed a VM without additional human intervention, I remembered that I used to know more about IPv4 without necessarily remembering what I used to know, and we can now put our VMs on arbitrary subnets and still route back and forth. Ideally, though, we’d want to add to the default gateway the cloud-init setup.

In the next part of this series, I may or may not look at configuring a Linux distro installation from scratch in CBSD, depending on whether I fall into another rabbit hole along the way

Sources / References

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

Blog at WordPress.com.

Up ↑

%d bloggers like this: