Collection of scripts help setup of rdif with FM10K kernel module with UIO for Silicom cards.

Go to file

DataHoarder 6f54952a42 Update 'README.md'		2021-11-27 10:59:09 +00:00
deps	Update fm10k deps for DKMS	2021-10-25 20:10:43 +02:00
platform	Fix typo on card platform README	2020-10-31 14:02:42 +00:00
.gitignore	Project now uses submodules	2020-10-20 20:59:55 +02:00
.gitmodules	Updated submodule location	2020-12-12 16:13:31 +00:00
mask_as_nic.sh	Added mask_as_nic.sh and updated information on mask_as_switch.sh	2020-10-20 16:46:03 +02:00
mask_as_switch.sh	Added mask_as_nic.sh and updated information on mask_as_switch.sh	2020-10-20 16:46:03 +02:00
README.md	Update 'README.md'	2021-11-27 10:59:09 +00:00
setup.sh	Fix non-existent DKMS	2021-10-25 20:38:58 +02:00

FM10K + rdif setup

These colection of scripts help setup of rdif with FM10K kernel module with UIO for Silicom cards.

Cards tested:

Linux Kernels tested:

Installation and setup

Install build tools and git # apt install build-essential git
Install Linux kernel headers
- Debian: # apt install linux-headers or # apt install linux-headers-amd64
- Ubuntu: # apt install linux-headers-generic
- Proxmox: # apt install pve-headers
(Optional) Install DKMS for automated module building # apt install dkms.
Clone the repository $ git clone https://git.gammaspectra.live/Sillycom/fm10k-rdif-setup && cd fm10k-rdif-setup.
Execute setup.sh (as root) # ./setup.sh
- This will compile fm10k kernel module with UIO support for running rdif switch manager.
- Build rdifctl and copy rdif binaries.
- rdif will be stopped and kernel module fm10k reloaded
- Default fm_platform_attributes.cfg will be placed under /etc/rdi/fm_platform_attributes.cfg if folder doesn't exist.

You need to enable IOMMU and SR-IOV on your BIOS settings.
This is not necessary under most kernels. Only do this if not kernel is not loading drivers.
Set up IOMMU in your kernel cmdline GRUB_CMDLINE_LINUX_DEFAULT (for example intel_iommu=on iommu=pt under /etc/default/grub).
(optional) You may want 1GB hugepages! DPDK and similar want them badly.
Use # lshw -c network -businfo to find the PCIe bus of the given devices. (Ethernet Switch FM10000 Host Interface)
Two NIC devices should appear per card. Make sure the slot they are connected to have 16x PCIe lanes and your BIOS is setup to do 8x8 PCIe bifurcation.
- You can flash the card via fm10k-dump and rrcSmall to unlock the Management PEP6 on all PCIe interfaces. After this you don't need bifurcation to manage the card (any interface will suffice).
Only the main card is needed directly to be able to control the switch. Secondary card is used to have full 200G bandwidth on host.
Map module override permanently using for example # driverctl set-override 0000:01:00.0 fm10k for listed device pci@0000:01:00.0.
If needed, # rmmod fm10k and # modprobe fm10k to reload the module.
A restart might also be needed to properly load UIO (/dev/uio0). Make sure module uio is not blacklisted.

NOTE: The bprd_ctl module is not necessary for working switch manager, and most cards do not have a bypass mode.
To start: # rdif start
To stop: # rdif stop
Control it via rdifctl. See src/rdif/readme.txt for more information.
An example exists under ./mask_as_switch.sh that connects all ports under the same bridge.
Another example under ./mask_as_nic.sh connects odd internal ports to external port 1 (QSFP 0) and even internal ports to external port 2 (QSFP 1)
Edit running configuration on /etc/rdi/fm_platform_attributes.cfg. Specifically you might want to set uioDevName or ethernetMode of ports.

Only x86_64 platforms are supported at this time.
Check the device platform README for notes.
Make sure you don't have another IES (Intel(R) Ethernet Switch (IES) Software API) instance running elsewhere.
Check that FM_LIBERTY_TRAIL_CONFIG_FILE is not set, or if set, that is where platform configuration files will get searched at.