Collection of scripts help setup of rdif with FM10K kernel module with UIO for Silicom cards.
Go to file
DataHoarder 6f54952a42
Update 'README.md'
2021-11-27 10:59:09 +00:00
deps Update fm10k deps for DKMS 2021-10-25 20:10:43 +02:00
platform Fix typo on card platform README 2020-10-31 14:02:42 +00:00
.gitignore Project now uses submodules 2020-10-20 20:59:55 +02:00
.gitmodules Updated submodule location 2020-12-12 16:13:31 +00:00
README.md Update 'README.md' 2021-11-27 10:59:09 +00:00
mask_as_nic.sh Added mask_as_nic.sh and updated information on mask_as_switch.sh 2020-10-20 16:46:03 +02:00
mask_as_switch.sh Added mask_as_nic.sh and updated information on mask_as_switch.sh 2020-10-20 16:46:03 +02:00
setup.sh Fix non-existent DKMS 2021-10-25 20:38:58 +02:00

README.md

FM10K + rdif setup

These colection of scripts help setup of rdif with FM10K kernel module with UIO for Silicom cards.

Cards tested:

  • Silicom PE3100G2DQIRM-QX4

Linux Kernels tested:

  • Proxmox
    • 5.4.x-pve
    • 5.11.x-pve
  • Ubuntu
    • 5.4.x-ubuntu
    • 5.8.x-ubuntu
    • 5.11.x-ubuntu

Installation and setup

  • Install build tools and git # apt install build-essential git
  • Install Linux kernel headers
    • Debian: # apt install linux-headers or # apt install linux-headers-amd64
    • Ubuntu: # apt install linux-headers-generic
    • Proxmox: # apt install pve-headers
  • (Optional) Install DKMS for automated module building # apt install dkms.
  • Clone the repository $ git clone https://git.gammaspectra.live/Sillycom/fm10k-rdif-setup && cd fm10k-rdif-setup.
  • Execute setup.sh (as root) # ./setup.sh
    • This will compile fm10k kernel module with UIO support for running rdif switch manager.
    • Build rdifctl and copy rdif binaries.
    • rdif will be stopped and kernel module fm10k reloaded
    • Default fm_platform_attributes.cfg will be placed under /etc/rdi/fm_platform_attributes.cfg if folder doesn't exist.

Mapping PCIe devices to fm10k driver

  • You need to enable IOMMU and SR-IOV on your BIOS settings.
  • This is not necessary under most kernels. Only do this if not kernel is not loading drivers.
  • Set up IOMMU in your kernel cmdline GRUB_CMDLINE_LINUX_DEFAULT (for example intel_iommu=on iommu=pt under /etc/default/grub).
  • (optional) You may want 1GB hugepages! DPDK and similar want them badly.
  • Use # lshw -c network -businfo to find the PCIe bus of the given devices. (Ethernet Switch FM10000 Host Interface)
  • Two NIC devices should appear per card. Make sure the slot they are connected to have 16x PCIe lanes and your BIOS is setup to do 8x8 PCIe bifurcation.
    • You can flash the card via fm10k-dump and rrcSmall to unlock the Management PEP6 on all PCIe interfaces. After this you don't need bifurcation to manage the card (any interface will suffice).
  • Only the main card is needed directly to be able to control the switch. Secondary card is used to have full 200G bandwidth on host.
  • Map module override permanently using for example # driverctl set-override 0000:01:00.0 fm10k for listed device pci@0000:01:00.0.
  • If needed, # rmmod fm10k and # modprobe fm10k to reload the module.
  • A restart might also be needed to properly load UIO (/dev/uio0). Make sure module uio is not blacklisted.

Running rdif

  • NOTE: The bprd_ctl module is not necessary for working switch manager, and most cards do not have a bypass mode.
  • To start: # rdif start
  • To stop: # rdif stop
  • Control it via rdifctl. See src/rdif/readme.txt for more information.
  • An example exists under ./mask_as_switch.sh that connects all ports under the same bridge.
  • Another example under ./mask_as_nic.sh connects odd internal ports to external port 1 (QSFP 0) and even internal ports to external port 2 (QSFP 1)
  • Edit running configuration on /etc/rdi/fm_platform_attributes.cfg. Specifically you might want to set uioDevName or ethernetMode of ports.

Troubleshooting

  • Only x86_64 platforms are supported at this time.
  • Check the device platform README for notes.
  • Make sure you don't have another IES (Intel(R) Ethernet Switch (IES) Software API) instance running elsewhere.
  • Check that FM_LIBERTY_TRAIL_CONFIG_FILE is not set, or if set, that is where platform configuration files will get searched at.