Some FM10K and PE3100G2DQiRM-QX4 notes and similar.

DataHoarder commented

2020-12-12 14:51:00 +00:00

Owner

# PE3100G2DQiRM-QX4 notes + DPDK patches:
# * On Silicom PE3100G2DQiRM-QX4 you need to connect all lanes of the first card at least (2x 8xPCIe), so you need bifurcation support on a 16x slot.
# * In addition, daughter card also seems needed, but you may be able to edit the config / fm10k_switch.h as well.
# * dpdk_fm10k.conf defaults to 100G speeds. You can set it to 100G, 40G, 25G (splits into 4x25G), 10G (splits into 4x10G). Such setting applies to all external ports.
# * After testing, this is how internal ports are mapped: PF 0 is daughter card PCIe #2, PF 1 is daughter card PCIe #1, PF 2 is main card PCIe #1, PF 3 is main card PCIe #2, 
# * dpdk_fm10k.conf has ability to set flows. This has not been tested.
# * Current working config doesn't have 2xPF merged onto one for 100G speeds per port. Further research is needed.
#

# Silicom PE3100G2DQIRM-QX4 (with IES / rdif / match)

## Notes
* This card requires at least a x16 PCIe slot bifurcated into two x8 slots. The management port is located on PEP 6 and having a single x8 makes it not accessible.
* The daughter card is not required to function properly. In fact, you might even be able to throw it onto another host!

## Mapping PCIe devices to fm10k driver
* **You need to enable IOMMU and SR-IOV on your BIOS settings.**
* Set up IOMMU in your kernel cmdline GRUB_CMDLINE_LINUX_DEFAULT (for example `intel_iommu=on iommu=pt` under _/etc/default/grub_).
* (optional) You may want 1GB hugepages! DPDK and similar want them badly.
* Use `# lshw -c network -businfo` to find the PCIe bus of the given devices. (_Ethernet Switch FM10000 Host Interface_)
* Two NIC devices should appear per card. **Make sure the slot they are connected to have 16x PCIe lanes and your BIOS is setup to do 8x8 PCIe bifurcation.**
* Only the main card is needed directly to be able to control the switch. Secondary card is used to have full 200G bandwidth on host.
* Map module override permanently using for example `# driverctl set-override 0000:01:00.0 fm10k` for listed device `pci@0000:01:00.0`.
* If needed, `# rmmod fm10k` and `# modprobe fm10k` to reload the module.
* A restart might also be needed to properly load UIO (`/dev/uio0`). Make sure module _uio_ is not blacklisted.

``` # PE3100G2DQiRM-QX4 notes + DPDK patches: # * On Silicom PE3100G2DQiRM-QX4 you need to connect all lanes of the first card at least (2x 8xPCIe), so you need bifurcation support on a 16x slot. # * In addition, daughter card also seems needed, but you may be able to edit the config / fm10k_switch.h as well. # * dpdk_fm10k.conf defaults to 100G speeds. You can set it to 100G, 40G, 25G (splits into 4x25G), 10G (splits into 4x10G). Such setting applies to all external ports. # * After testing, this is how internal ports are mapped: PF 0 is daughter card PCIe #2, PF 1 is daughter card PCIe #1, PF 2 is main card PCIe #1, PF 3 is main card PCIe #2, # * dpdk_fm10k.conf has ability to set flows. This has not been tested. # * Current working config doesn't have 2xPF merged onto one for 100G speeds per port. Further research is needed. # ``` ``` # Silicom PE3100G2DQIRM-QX4 (with IES / rdif / match) ## Notes * This card requires at least a x16 PCIe slot bifurcated into two x8 slots. The management port is located on PEP 6 and having a single x8 makes it not accessible. * The daughter card is not required to function properly. In fact, you might even be able to throw it onto another host! ``` ``` ## Mapping PCIe devices to fm10k driver * **You need to enable IOMMU and SR-IOV on your BIOS settings.** * Set up IOMMU in your kernel cmdline GRUB_CMDLINE_LINUX_DEFAULT (for example `intel_iommu=on iommu=pt` under _/etc/default/grub_). * (optional) You may want 1GB hugepages! DPDK and similar want them badly. * Use `# lshw -c network -businfo` to find the PCIe bus of the given devices. (_Ethernet Switch FM10000 Host Interface_) * Two NIC devices should appear per card. **Make sure the slot they are connected to have 16x PCIe lanes and your BIOS is setup to do 8x8 PCIe bifurcation.** * Only the main card is needed directly to be able to control the switch. Secondary card is used to have full 200G bandwidth on host. * Map module override permanently using for example `# driverctl set-override 0000:01:00.0 fm10k` for listed device `pci@0000:01:00.0`. * If needed, `# rmmod fm10k` and `# modprobe fm10k` to reload the module. * A restart might also be needed to properly load UIO (`/dev/uio0`). Make sure module _uio_ is not blacklisted. ```

q3k was assigned by DataHoarder

2020-12-12 14:51:00 +00:00

DataHoarder commented

2020-12-12 15:47:55 +00:00

Author

Owner

Probably a good idea to put all of this device documentation under https://git.gammaspectra.live/FM10K/Documentation/src/branch/master/devices/Silicom/PE3100G2DQiRM-QX4

DataHoarder commented

2020-12-13 14:06:06 +00:00

Author

Owner

While investigating board pictures, these interesting chips were found on the main board.

The area above the main chip is mostly composed of buck converters and linear voltage regulators, for power delivery to all components. Also next to external power connector, on both sides.
- TI LMZ31710: 10-A Power Module With 2.95-V to 17-V Input and Current Sharing
  in QFN Package, U1, Product page, Datasheet
- TI TPS56121: 4.5V to 14V Input, 15A Synchronous Step-Down SWIFT™ Converter, U2, Product page, [Datasheet]
- TI LM96163: Remote Diode Digital Temperature Sensor with Integrated Fan Control and TruTherm ® BJT Transistor Beta Compensation Technology, U13, Product page, Datasheet
- 3x IR 3555M: 60A Integrated PowIRstage IR3555 ®, U3, U4, U5, Datasheet
- TI TPS74401: 3-A, low-VIN (0.8-V), low-noise, high-PSRR, adjustable ultra-low-dropout voltage regulator, U12, Product page, Datasheet
- IR 3584 (back side): Dual Loop, 4+1 multiphase Intel VR12 and VR12.5 PWM controller for high efficiency, highly accurate VR solutions, U21, Product page, Datasheet
- 2x 020N03LS (back side): MOSFET, Q2, Q3, Datasheet
- Other unidentified SMD chips
2x MAX6971: 16-Port, 36V Constant-Current LED Driver
- U8, U18: Located next to each LED status assembly, front side.
- Product page, Datasheet
LCMXO2-1200HC-4TG100C: Flexible Interface Bridging FPGA
- U6: Located to the right of SW1 dip switch, front side.
- Has SPI FLASH CON I/O port below.
- Has unknown port J2 above sideways, but might not be related.
- Product page, Datasheet
adesto "45DB321E" (AT45DB321E-SHF-B): 32-Mbit DataFlash (with Extra 1-Mbits), 2.3V Minimum, SPI Serial Flash Memory
- Datasheet
2x TLV3257: Quad 1-of-2 multiplexer/demultiplexer
- U19: Located opposite of unknown port J2 above, on the back side.
- U29: Another one is located next to power area, below IR 3584, on the back side.
- Product page, Datasheet
Si 53154-A01A: PCIe Quad Clock Buffers
- U17:
- Product page, Datasheet

While investigating [board pictures](https://git.gammaspectra.live/FM10K/Documentation/src/branch/master/devices/Silicom/PE3100G2DQiRM-QX4/pictures), these interesting chips were found on the main board. * The area above the main chip is mostly composed of buck converters and linear voltage regulators, for power delivery to all components. Also next to external power connector, on both sides. - TI **LMZ31710**: _10-A Power Module With 2.95-V to 17-V Input and Current Sharing in QFN Package_, U1, [Product page](https://www.ti.com/product/LMZ31710), [Datasheet](https://www.ti.com/lit/gpn/lmz31710) - TI **TPS56121**: _4.5V to 14V Input, 15A Synchronous Step-Down SWIFT™ Converter_, U2, [Product page](https://www.ti.com/product/TPS56121), [Datasheet] - TI **LM96163**: _Remote Diode Digital Temperature Sensor with Integrated Fan Control and TruTherm ® BJT Transistor Beta Compensation Technology_, U13, [Product page](https://www.ti.com/product/LM96163), [Datasheet](https://www.ti.com/lit/gpn/lm96163) - 3x **IR 3555M**: _60A Integrated PowIRstage IR3555 ®_, U3, U4, U5, [Datasheet](https://www.infineon.com/dgdl/pb-ir3555.pdf?fileId=5546d462533600a40153567fd3ac28d3) - TI **TPS74401**: _3-A, low-VIN (0.8-V), low-noise, high-PSRR, adjustable ultra-low-dropout voltage regulator_, U12, [Product page](https://www.ti.com/product/TPS74401), [Datasheet](https://www.ti.com/lit/gpn/tps74401) - **IR 3584** (back side): _Dual Loop, 4+1 multiphase Intel VR12 and VR12.5 PWM controller for high efficiency, highly accurate VR solutions_, U21, [Product page](https://www.infineon.com/cms/en/product/power/dc-dc-converters/digital-multiphase-controllers/ir3584/), [Datasheet](https://www.infineon.com/dgdl/pb-ir3584.pdf?fileId=5546d462533600a40153568068f628fb) - 2x **020N03LS** (back side): _MOSFET_, Q2, Q3, [Datasheet](https://www.infineon.com/dgdl/Infineon-BSC020N03LS%20G-DataSheet-v02_00-EN.pdf?fileId=db3a304412b407950112b427641b3c1f) - Other unidentified SMD chips * 2x **MAX6971**: _16-Port, 36V Constant-Current LED Driver_ - U8, U18: Located next to each LED status assembly, front side. - [Product page](https://www.maximintegrated.com/en/products/power/display-power-control/MAX6971.html), [Datasheet](https://datasheets.maximintegrated.com/en/ds/MAX6971.pdf) * **LCMXO2-1200HC-4TG100C**: _Flexible Interface Bridging FPGA_ - U6: Located to the right of SW1 dip switch, front side. - Has SPI FLASH CON I/O port below. - Has _unknown port J2_ above sideways, but might not be related. - [Product page](https://www.latticesemi.com/en/Products/FPGAandCPLD/MachXO2), [Datasheet](https://www.latticesemi.com/view_document?document_id=38834) * adesto "**45DB321E**" (AT45DB321E-SHF-B): _32-Mbit DataFlash (with Extra 1-Mbits), 2.3V Minimum, SPI Serial Flash Memory_ - [Datasheet](https://eu.mouser.com/datasheet/2/590/doc8784-1385823.pdf) * 2x **TLV3257**: _Quad 1-of-2 multiplexer/demultiplexer_ - U19: Located opposite of _unknown port J2_ above, on the back side. - U29: Another one is located next to power area, below IR 3584, on the back side. - [Product page](https://www.nexperia.com/products/analog-logic-ics/i-o-expansion-logic/bus-switches/74CBTLV3257PW.html), [Datasheet](https://assets.nexperia.com/documents/data-sheet/74CBTLV3257.pdf) * Si **53154-A01A**: _PCIe Quad Clock Buffers_ - U17: - [Product page](https://www.silabs.com/timing/buffers/pci-express-clock-buffers/device.si53154), [Datasheet](https://www.silabs.com/documents/public/data-sheets/Si53154.pdf)

DataHoarder self-assigned this 2020-12-13 16:21:46 +00:00

DataHoarder commented

2020-12-13 19:57:35 +00:00

Author

Owner

Here are a few close-ups of a few of the parts that were small, though freehand macro maybe is not the best idea. Better ones should be made if we want to put them under pictures.

8.9 MiB

8.8 MiB

8.8 MiB

9.1 MiB

9.1 MiB

8.8 MiB