Collection of scripts help setup of rdif with FM10K kernel module with UIO for Silicom cards.
deps | ||
platform | ||
.gitignore | ||
.gitmodules | ||
mask_as_nic.sh | ||
mask_as_switch.sh | ||
README.md | ||
setup.sh |
FM10K + rdif setup
These colection of scripts help setup of rdif with FM10K kernel module with UIO for Silicom cards.
Cards tested:
- Silicom PE3100G2DQIRM-QX4
Linux Kernels tested:
- Proxmox
- 5.4.x-pve
- 5.11.x-pve
- Ubuntu
- 5.4.x-ubuntu
- 5.8.x-ubuntu
- 5.11.x-ubuntu
Installation and setup
- Install build tools and git
# apt install build-essential git
- Install Linux kernel headers
- Debian:
# apt install linux-headers
or# apt install linux-headers-amd64
- Ubuntu:
# apt install linux-headers-generic
- Proxmox:
# apt install pve-headers
- Debian:
- (Optional) Install DKMS for automated module building
# apt install dkms
. - Clone the repository
$ git clone https://git.gammaspectra.live/Sillycom/fm10k-rdif-setup && cd fm10k-rdif-setup
. - Execute setup.sh (as root)
# ./setup.sh
- This will compile fm10k kernel module with UIO support for running rdif switch manager.
- Build rdifctl and copy rdif binaries.
- rdif will be stopped and kernel module fm10k reloaded
- Default fm_platform_attributes.cfg will be placed under
/etc/rdi/fm_platform_attributes.cfg
if folder doesn't exist.
Mapping PCIe devices to fm10k driver
- You need to enable IOMMU and SR-IOV on your BIOS settings.
- This is not necessary under most kernels. Only do this if not kernel is not loading drivers.
- Set up IOMMU in your kernel cmdline GRUB_CMDLINE_LINUX_DEFAULT (for example
intel_iommu=on iommu=pt
under /etc/default/grub). - (optional) You may want 1GB hugepages! DPDK and similar want them badly.
- Use
# lshw -c network -businfo
to find the PCIe bus of the given devices. (Ethernet Switch FM10000 Host Interface) - Two NIC devices should appear per card. Make sure the slot they are connected to have 16x PCIe lanes and your BIOS is setup to do 8x8 PCIe bifurcation.
- You can flash the card via fm10k-dump and rrcSmall to unlock the Management PEP6 on all PCIe interfaces. After this you don't need bifurcation to manage the card (any interface will suffice).
- Only the main card is needed directly to be able to control the switch. Secondary card is used to have full 200G bandwidth on host.
- Map module override permanently using for example
# driverctl set-override 0000:01:00.0 fm10k
for listed devicepci@0000:01:00.0
. - If needed,
# rmmod fm10k
and# modprobe fm10k
to reload the module. - A restart might also be needed to properly load UIO (
/dev/uio0
). Make sure module uio is not blacklisted.
Running rdif
- NOTE: The
bprd_ctl
module is not necessary for working switch manager, and most cards do not have a bypass mode. - To start:
# rdif start
- To stop:
# rdif stop
- Control it via
rdifctl
. Seesrc/rdif/readme.txt
for more information. - An example exists under
./mask_as_switch.sh
that connects all ports under the same bridge. - Another example under
./mask_as_nic.sh
connects odd internal ports to external port 1 (QSFP 0) and even internal ports to external port 2 (QSFP 1) - Edit running configuration on
/etc/rdi/fm_platform_attributes.cfg
. Specifically you might want to setuioDevName
orethernetMode
of ports.
Troubleshooting
- Only x86_64 platforms are supported at this time.
- Check the device platform README for notes.
- Make sure you don't have another IES (Intel(R) Ethernet Switch (IES) Software API) instance running elsewhere.
- Check that FM_LIBERTY_TRAIL_CONFIG_FILE is not set, or if set, that is where platform configuration files will get searched at.