Boilerplates/Proxmox/ThunderboltNetwork.md

How to get Thunderbolt Network up and running

Main idea is from here, but I like to be able to copy paste a bit more comfortable.

on all Nodes

Optional package to track which node can see which other one.

apt install -y lldpd

Mandatory packages.

apt install -y frr

Add kernel modules.

# remove empty lines
sed '/^$/d' /etc/modules > temp.txt && mv temp.txt /etc/modules

# add modules
tee -a /etc/modules <<EOF
thunderbolt
thunderbolt-net
EOF

Adjust the /etc/network/interfaces. Remove any section that belongs to any auto added thunderbolt0 or thunderbolt1 interface.

sed -i '/^source \/etc\/network\/interfaces\.d\/\*$/d' /etc/network/interfaces
sed '${/^$/d;}' /etc/network/interfaces > temp.txt && mv temp.txt /etc/network/interfaces
nano /etc/network/interfaces

Node 1 at the beginning below default auto lo.

auto lo:0
iface lo:0 inet static
        address 10.0.0.81/32

Node 2 at the beginning below default auto lo.

auto lo:0
iface lo:0 inet static
        address 10.0.0.82/32

Node 3 at the beginning below default auto lo.

auto lo:0
iface lo:0 inet static
        address 10.0.0.83/32

On each node at the end.

allow-hotplug en05
iface en05 inet manual
       mtu 65520

allow-hotplug en06
iface en06 inet manual
       mtu 65520

Add the thunderbolt links.

tee -a /etc/systemd/network/00-thunderbolt0.link <<EOF
[Match]
Path=pci-0000:00:0d.2
Driver=thunderbolt-net
[Link]
MACAddressPolicy=none
Name=en05
EOF

tee -a /etc/systemd/network/00-thunderbolt1.link <<EOF
[Match]
Path=pci-0000:00:0d.3
Driver=thunderbolt-net
[Link]
MACAddressPolicy=none
Name=en06
EOF

Automatic setup of interface to be up after reboot or cable insertion.

tee -a /etc/udev/rules.d/10-tb-en.rules <<EOF
ACTION=="move", SUBSYSTEM=="net", KERNEL=="en05", RUN+="/usr/local/bin/pve-en05.sh"
ACTION=="move", SUBSYSTEM=="net", KERNEL=="en06", RUN+="/usr/local/bin/pve-en06.sh"
EOF

tee -a /usr/local/bin/pve-en05.sh <<EOF
#!/bin/bash
/usr/sbin/ifup en05
EOF

tee -a /usr/local/bin/pve-en06.sh <<EOF
#!/bin/bash
/usr/sbin/ifup en06
EOF

chmod +x /usr/local/bin/pve-en05.sh
chmod +x /usr/local/bin/pve-en06.sh

Enable IPv4 forwarding.

sed -i "s/\#net.ipv4.ip_forward\=1/net.ipv4.ip_forward\=1/" /etc/sysctl.conf

Presetup and configuration of FRR.

sed -i "s/fabricd=no/fabricd=yes/" /etc/frr/daemons
systemctl restart frr

Open VTYSH CLI.

vtysh

Enter config mode.

configure

Node1 Config.

ip forwarding
!
interface en05
ip router openfabric 1
exit
!
interface en06
ip router openfabric 1
exit
!
interface lo
ip router openfabric 1
openfabric passive
exit
!
router openfabric 1
net 49.0000.0000.0001.00
exit
!

end
write memory
exit

vtysh -c "show running-config"

Time for the reboot.

/sbin/reboot