Boilerplates/Proxmox/ThunderboltNetwork.md

How to get Thunderbolt Network up and running

Main idea is from here, but I like to be able to copy paste a bit more comfortable.

On all Nodes

Optional package to track which node can see which other one.

apt install -y lldpd

Mandatory packages.

apt install -y frr

Add kernel modules.

# remove empty lines
sed '/^$/d' /etc/modules > temp.txt && mv temp.txt /etc/modules

# add modules
tee -a /etc/modules <<EOF
thunderbolt
thunderbolt-net
EOF

Add the thunderbolt links.

tee -a /etc/systemd/network/00-thunderbolt0.link <<EOF
[Match]
Path=pci-0000:00:0d.2
Driver=thunderbolt-net
[Link]
MACAddressPolicy=none
Name=en05
EOF

tee -a /etc/systemd/network/00-thunderbolt1.link <<EOF
[Match]
Path=pci-0000:00:0d.3
Driver=thunderbolt-net
[Link]
MACAddressPolicy=none
Name=en06
EOF

Automatic setup of interface to be up after reboot or cable insertion.

tee -a /etc/udev/rules.d/10-tb-en.rules <<EOF
ACTION=="move", SUBSYSTEM=="net", KERNEL=="en05", RUN+="/usr/local/bin/pve-en05.sh"
ACTION=="move", SUBSYSTEM=="net", KERNEL=="en06", RUN+="/usr/local/bin/pve-en06.sh"
EOF

tee -a /usr/local/bin/pve-en05.sh <<EOF
#!/bin/bash
/usr/sbin/ifup en05
EOF

tee -a /usr/local/bin/pve-en06.sh <<EOF
#!/bin/bash
/usr/sbin/ifup en06
EOF

chmod +x /usr/local/bin/pve-en05.sh
chmod +x /usr/local/bin/pve-en06.sh

Enable IPv4 forwarding.

sed -i "s/\#net.ipv4.ip_forward\=1/net.ipv4.ip_forward\=1/" /etc/sysctl.conf

Presetup and configuration of FRR.

sed -i "s/fabricd=no/fabricd=yes/" /etc/frr/daemons
systemctl restart frr

Make sure interface is coming up!

tee -a /etc/network/if-up.d/restart-frr <<EOF
#!/bin/sh

# Check if the interface is either en05 or en06
if [ "$IFACE" = "en05" ] || [ "$IFACE" = "en06" ]; then
    # Restart the frr service
    /usr/bin/systemctl restart frr.service
fi
EOF
chmod +x /etc/network/if-up.d/restart-frr

Different settings per Node!

Adjust the /etc/network/interfaces. Remove any section that belongs to any auto added thunderbolt0 or thunderbolt1 interface.

Node 1

sed -i '/iface lo inet loopback/a\
\
auto lo:0\niface lo:0 inet static\n        address 10.0.0.81/32' /etc/network/interfaces
sed -i '/^source \/etc\/network\/interfaces\.d\/\*$/d' /etc/network/interfaces
sed '${/^$/d;}' /etc/network/interfaces > temp.txt && mv temp.txt /etc/network/interfaces

tee -a /etc/network/interfaces <<EOF
allow-hotplug en05
iface en05 inet manual
       mtu 65520

allow-hotplug en06
iface en06 inet manual
       mtu 65520
EOF

Node 2

sed -i '/iface lo inet loopback/a\
\
auto lo:0\niface lo:0 inet static\n        address 10.0.0.82/32' /etc/network/interfaces
sed -i '/^source \/etc\/network\/interfaces\.d\/\*$/d' /etc/network/interfaces
sed '${/^$/d;}' /etc/network/interfaces > temp.txt && mv temp.txt /etc/network/interfaces

tee -a /etc/network/interfaces <<EOF
allow-hotplug en05
iface en05 inet manual
       mtu 65520

allow-hotplug en06
iface en06 inet manual
       mtu 65520
EOF

Node 3

sed -i '/iface lo inet loopback/a\
\
auto lo:0\niface lo:0 inet static\n        address 10.0.0.83/32' /etc/network/interfaces
sed -i '/^source \/etc\/network\/interfaces\.d\/\*$/d' /etc/network/interfaces
sed '${/^$/d;}' /etc/network/interfaces > temp.txt && mv temp.txt /etc/network/interfaces

tee -a /etc/network/interfaces <<EOF
allow-hotplug en05
iface en05 inet manual
       mtu 65520

allow-hotplug en06
iface en06 inet manual
       mtu 65520
EOF

Open VTYSH CLI.

vtysh

Enter config mode.

configure

Node1 Config.

ip forwarding
!
interface en05
ip router openfabric 1
exit
!
interface en06
ip router openfabric 1
exit
!
interface lo
ip router openfabric 1
openfabric passive
exit
!
router openfabric 1
net 49.0000.0000.0001.00
exit
!

end
write memory
exit

Node2 Config.

ip forwarding
!
interface en05
ip router openfabric 1
exit
!
interface en06
ip router openfabric 1
exit
!
interface lo
ip router openfabric 1
openfabric passive
exit
!
router openfabric 1
net 49.0000.0000.0002.00
exit
!

end
write memory
exit

Node3 Config.

ip forwarding
!
interface en05
ip router openfabric 1
exit
!
interface en06
ip router openfabric 1
exit
!
interface lo
ip router openfabric 1
openfabric passive
exit
!
router openfabric 1
net 49.0000.0000.0003.00
exit
!

end
write memory
exit

vtysh -c "show running-config"

Time for the reboot.

/sbin/reboot

Debugging

# shows the actual configuration
vtysh -c "show running-config"

# shows all links
vtysh -c "show openfabric topology"