470 lines
17 KiB
Plaintext
470 lines
17 KiB
Plaintext
README for Intel(R) Ethernet Switch Host Interface Driver
|
||
===============================================================================
|
||
|
||
===============================================================================
|
||
|
||
November 26, 2018
|
||
|
||
===============================================================================
|
||
|
||
Contents
|
||
--------
|
||
|
||
- Overview
|
||
- Building and Installation
|
||
- Command Line Parameters
|
||
- Additional Features & Configurations
|
||
- Known Issues
|
||
|
||
|
||
Important Notes
|
||
===============
|
||
|
||
Configuring SR-IOV for improved network security
|
||
------------------------------------------------
|
||
In a virtualized environment, on Intel(R) Ethernet Network Adapters that
|
||
support SR-IOV, the virtual function (VF) may be subject to malicious behavior.
|
||
Software-generated layer two frames, like IEEE 802.3x (link flow control), IEEE
|
||
802.1Qbb (priority based flow-control), and others of this type, are not
|
||
expected and can throttle traffic between the host and the virtual switch,
|
||
reducing performance. To resolve this issue, and to ensure isolation from
|
||
unintended traffic streams, configure all SR-IOV enabled ports for VLAN tagging
|
||
from the administrative interface on the PF. This configuration allows
|
||
unexpected, and potentially malicious, frames to be dropped. See "Configuring
|
||
VLAN Tagging on SR-IOV Enabled Adapter Ports" in this README for configuration
|
||
instructions.
|
||
|
||
|
||
Overview
|
||
========
|
||
This driver supports kernel versions 2.6.32 and newer.
|
||
|
||
Driver information can be obtained using ethtool, lspci, and iproute2 ip.
|
||
Instructions on updating ethtool can be found in the section Additional
|
||
Configurations later in this document.
|
||
|
||
|
||
Intel(R) Ethernet SDI Adapter FM10420-100GbE-QDA2 Notes
|
||
-------------------------------------------------------
|
||
|
||
The Intel(R) Ethernet SDI Adapter FM10420-100GbE-QDA2 supports the following
|
||
cables:
|
||
Supplier Part Number
|
||
FCI 10121178-2010LF
|
||
FCI 10121178-4020LF
|
||
FCI 10121178-4030LF
|
||
FCI 10121178-4040LF
|
||
FCI 10121178-4050LF
|
||
|
||
Note: The Intel(R) Ethernet SDI Adapter FM10420-100GbE-QDA2 is a Gen3 x16 card
|
||
that is electrically bifurcated into two x8 ports. Without a BIOS that supports
|
||
bifurcation, only 1 port is available.
|
||
|
||
|
||
Building and Installation
|
||
=========================
|
||
To build a binary RPM package of this driver
|
||
--------------------------------------------
|
||
Note: RPM functionality has only been tested in Red Hat distributions.
|
||
|
||
1. Run the following command, where <x.x.x> is the version number for the
|
||
driver tar file.
|
||
|
||
# rpmbuild -tb fm10k-<x.x.x>.tar.gz
|
||
|
||
NOTE: For the build to work properly, the currently running kernel MUST
|
||
match the version and configuration of the installed kernel sources. If
|
||
you have just recompiled the kernel, reboot the system before building.
|
||
|
||
2. After building the RPM, the last few lines of the tool output contain the
|
||
location of the RPM file that was built. Install the RPM with one of the
|
||
following commands, where <RPM> is the location of the RPM file:
|
||
|
||
# rpm -Uvh <RPM>
|
||
or
|
||
# dnf/yum localinstall <RPM>
|
||
|
||
NOTES:
|
||
- To compile the driver on some kernel/arch combinations, you may need to
|
||
install a package with the development version of libelf (e.g. libelf-dev,
|
||
libelf-devel, elfutilsl-libelf-devel).
|
||
- When compiling an out-of-tree driver, details will vary by distribution.
|
||
However, you will usually need a kernel-devel RPM or some RPM that provides the
|
||
kernel headers at a minimum. To find the kernel-devel header sources for a
|
||
particular kernel, you will usually fill in the link at /lib/modules/'uname
|
||
-r'/build.
|
||
|
||
|
||
To manually build the driver
|
||
----------------------------
|
||
1. Move the base driver tar file to the directory of your choice.
|
||
For example, use '/home/username/fm10k' or '/usr/local/src/fm10k'.
|
||
|
||
2. Untar/unzip the archive, where <x.x.x> is the version number for the
|
||
driver tar file:
|
||
|
||
# tar zxf fm10k-<x.x.x>.tar.gz
|
||
|
||
3. Change to the driver src directory, where <x.x.x> is the version number
|
||
for the driver tar:
|
||
|
||
# cd fm10k-<x.x.x>/src/
|
||
|
||
4. Compile the driver module:
|
||
|
||
# make install
|
||
|
||
The binary will be installed as:
|
||
/lib/modules/<KERNEL
|
||
VERSION>/updates/drivers/net/ethernet/intel/fm10k/fm10k.ko
|
||
|
||
The install location listed above is the default location. This may differ
|
||
for various Linux distributions.
|
||
|
||
5. Load the module using the modprobe command.
|
||
|
||
To check the version of the driver and then load it:
|
||
|
||
# modinfo fm10k
|
||
# modprobe fm10k [parameter=port1_value,port2_value]
|
||
|
||
Alternately, make sure that any older fm10k drivers are removed from the
|
||
kernel before loading the new module:
|
||
|
||
# rmmod fm10k; modprobe fm10k
|
||
|
||
6. Assign an IP address to the interface by entering the following,
|
||
where ethX is the interface name that was shown in dmesg after modprobe:
|
||
|
||
# ip address add <IP_address>/<netmask bits> dev ethX
|
||
|
||
NOTE: Before proceeding, ensure that netdev is enabled and that a
|
||
switch manager is running. To enable netdev, use one of the following
|
||
commands:
|
||
# ifconfig <netdev> up
|
||
or
|
||
# ip link set <netdev> up
|
||
|
||
7. Verify that the interface works. Enter the following, where IP_address
|
||
is the IP address for another machine on the same subnet as the interface
|
||
that is being tested:
|
||
|
||
# ping <IP_address>
|
||
|
||
Note: For certain distributions like (but not limited to) RedHat Enterprise
|
||
Linux 7 and Ubuntu, once the driver is installed, the initrd/initramfs file may
|
||
need to be updated to prevent the OS loading old versions of the fm10k driver.
|
||
The dracut utility may be used on RedHat distributions:
|
||
# dracut --force
|
||
|
||
For Ubuntu:
|
||
# update-initramfs -u
|
||
|
||
|
||
Command Line Parameters
|
||
=======================
|
||
If the driver is built as a module, the following optional parameters are used
|
||
by entering them on the command line with the modprobe command using this
|
||
syntax:
|
||
|
||
# modprobe fm10k [<option>=<VAL1>]
|
||
|
||
For example:
|
||
|
||
# modprobe fm10k max_vfs=7
|
||
|
||
The default value for each parameter is generally the recommended setting,
|
||
unless otherwise noted.
|
||
|
||
|
||
RSS
|
||
---
|
||
Valid Range: 0-128
|
||
0 = Assign up to the lesser value of the number of CPUs or the number of queues
|
||
X = Assign X queues, where X is less than or equal to the maximum number of
|
||
queues (128 queues).
|
||
|
||
|
||
max_vfs
|
||
-------
|
||
This parameter adds support for SR-IOV. It causes the driver to spawn up to
|
||
max_vfs worth of virtual functions.
|
||
Valid Range: 0-64
|
||
|
||
NOTE: This parameter is only used on kernel 3.7.x and below. On kernel 3.8.x
|
||
and above, use sysfs to enable VFs. Also, for Red Hat distributions, this
|
||
parameter is only used on version 6.6 and older. For version 6.7 and newer, use
|
||
sysfs.
|
||
|
||
For example, you can create 4 VFs as follows:
|
||
|
||
# echo 4 > /sys/class/net/<dev>/device/sriov_numvfs
|
||
|
||
To disable VFs, write 0 to the same file:
|
||
|
||
# echo 0 > /sys/class/net/<dev>/device/sriov_numvfs
|
||
|
||
The parameters for the driver are referenced by position. Thus, if you have a
|
||
dual port adapter, or more than one adapter in your system, and want N virtual
|
||
functions per port, you must specify a number for each port with each parameter
|
||
separated by a comma. For example:
|
||
|
||
# modprobe fm10k max_vfs=4
|
||
|
||
This will spawn 4 VFs on the first port.
|
||
|
||
# modprobe fm10k max_vfs=2,4
|
||
|
||
This will spawn 2 VFs on the first port and 4 VFs on the second port.
|
||
|
||
NOTE: Caution must be used in loading the driver with these parameters.
|
||
Depending on your system configuration, number of slots, etc., it is impossible
|
||
to predict in all cases where the positions would be on the command line.
|
||
|
||
NOTE: Neither the device nor the driver control how VFs are mapped into config
|
||
space. Bus layout will vary by operating system. On operating systems that
|
||
support it, you can check sysfs to find the mapping.
|
||
|
||
NOTE: When SR-IOV mode is enabled, hardware VLAN filtering and VLAN tag
|
||
stripping/insertion will remain enabled. Please remove the old VLAN filter
|
||
before the new VLAN filter is added. For example:
|
||
|
||
# ip link set eth0 vf 0 vlan 100 // set vlan 100 for VF 0
|
||
# ip link set eth0 vf 0 vlan 0 // Delete vlan 100
|
||
# ip link set eth0 vf 0 vlan 200 // set a new vlan 200 for VF 0
|
||
|
||
|
||
Configuring SR-IOV for improved network security
|
||
------------------------------------------------
|
||
In a virtualized environment, on Intel(R) Ethernet Network Adapters that
|
||
support SR-IOV, the virtual function (VF) may be subject to malicious behavior.
|
||
Software-generated layer two frames, like IEEE 802.3x (link flow control), IEEE
|
||
802.1Qbb (priority based flow-control), and others of this type, are not
|
||
expected and can throttle traffic between the host and the virtual switch,
|
||
reducing performance. To resolve this issue, and to ensure isolation from
|
||
unintended traffic streams, configure all SR-IOV enabled ports for VLAN tagging
|
||
from the administrative interface on the PF. This configuration allows
|
||
unexpected, and potentially malicious, frames to be dropped. See "Configuring
|
||
VLAN Tagging on SR-IOV Enabled Adapter Ports" in this README for configuration
|
||
instructions.
|
||
|
||
|
||
Configuring VLAN Tagging on SR-IOV Enabled Adapter Ports
|
||
--------------------------------------------------------
|
||
To configure VLAN tagging for the ports on an SR-IOV enabled adapter, use the
|
||
following command. The VLAN configuration should be done before the VF driver
|
||
is loaded or the VM is booted. The VF is not aware of the VLAN tag being
|
||
inserted on transmit and removed on received frames (sometimes called "port
|
||
VLAN" mode).
|
||
|
||
# ip link set dev <PF netdev id> vf <id> vlan <vlan id>
|
||
|
||
For example, the following will configure PF eth0 and the first VF on VLAN 10:
|
||
|
||
# ip link set dev eth0 vf 0 vlan 10
|
||
|
||
|
||
Additional Features and Configurations
|
||
======================================
|
||
|
||
Configuring the Driver on Different Distributions
|
||
-------------------------------------------------
|
||
Configuring a network driver to load properly when the system is started is
|
||
distribution dependent. Typically, the configuration process involves adding an
|
||
alias line to /etc/modules.conf or /etc/modprobe.conf as well as editing other
|
||
system startup scripts and/or configuration files. Many popular Linux
|
||
distributions ship with tools to make these changes for you. To learn the
|
||
proper way to configure a network device for your system, refer to your
|
||
distribution documentation. If during this process you are asked for the driver
|
||
or module name, the name for the Base Driver is fm10k.
|
||
|
||
|
||
Viewing Link Messages
|
||
---------------------
|
||
Link messages will not be displayed to the console if the distribution is
|
||
restricting system messages. In order to see network driver link messages on
|
||
your console, set dmesg to eight by entering the following:
|
||
|
||
# dmesg -n 8
|
||
|
||
NOTE: This setting is not saved across reboots.
|
||
|
||
|
||
Jumbo Frames
|
||
------------
|
||
Jumbo Frames support is enabled by changing the Maximum Transmission Unit (MTU)
|
||
to a value larger than the default value of 1500.
|
||
|
||
Use the ifconfig command to increase the MTU size. For example, enter the
|
||
following where X is the interface number:
|
||
|
||
# ifconfig ethX mtu 9000 up
|
||
|
||
Alternatively, you can use the ip command as follows:
|
||
|
||
# ip link set mtu 9000 dev ethX
|
||
# ip link set up dev ethX
|
||
|
||
This setting is not saved across reboots. The setting change can be made
|
||
permanent by adding 'MTU=9000' to the following file:
|
||
/etc/sysconfig/network-scripts/ifcfg-ethX for RHEL
|
||
or
|
||
/etc/sysconfig/network/<config_file> for SLES
|
||
|
||
NOTE: The maximum MTU setting for Jumbo Frames is 15342. This value coincides
|
||
with the maximum Jumbo Frames size of 15364 bytes.
|
||
|
||
NOTE: This driver will attempt to use multiple page sized buffers to receive
|
||
each jumbo packet. This should help to avoid buffer starvation issues when
|
||
allocating receive packets.
|
||
|
||
NOTE: Packet loss may have a greater impact on throughput when you use jumbo
|
||
frames. If you observe a drop in performance after enabling jumbo frames,
|
||
enabling flow control may mitigate the issue.
|
||
|
||
|
||
ethtool
|
||
-------
|
||
The driver utilizes the ethtool interface for driver configuration and
|
||
diagnostics, as well as displaying statistical information. The latest ethtool
|
||
version is required for this functionality. Download it at:
|
||
https://kernel.org/pub/software/network/ethtool/
|
||
|
||
Supported ethtool Commands and Options for Filtering
|
||
----------------------------------------------------
|
||
-n --show-nfc
|
||
Retrieves the receive network flow classification configurations.
|
||
|
||
rx-flow-hash tcp4|udp4|ah4|esp4|sctp4|tcp6|udp6|ah6|esp6|sctp6
|
||
Retrieves the hash options for the specified network traffic type.
|
||
|
||
-N --config-nfc
|
||
Configures the receive network flow classification.
|
||
|
||
rx-flow-hash tcp4|udp4|ah4|esp4|sctp4|tcp6|udp6|ah6|esp6|sctp6
|
||
m|v|t|s|d|f|n|r...
|
||
Configures the hash options for the specified network traffic type.
|
||
|
||
udp4 UDP over IPv4
|
||
udp6 UDP over IPv6
|
||
|
||
f Hash on bytes 0 and 1 of the Layer 4 header of the rx packet.
|
||
n Hash on bytes 2 and 3 of the Layer 4 header of the rx packet.
|
||
|
||
|
||
NAPI
|
||
----
|
||
NAPI (Rx polling mode) is supported in the fm10k driver.
|
||
For more information on NAPI, see
|
||
https://www.linuxfoundation.org/collaborate/workgroups/networking/napi
|
||
|
||
|
||
Flow Control
|
||
------------
|
||
The Intel(R) Ethernet Switch Host Interface Driver does not support Flow
|
||
Control. It will not send pause frames. This may result in dropped frames.
|
||
|
||
|
||
Tunnel/Overlay Stateless Offloads
|
||
---------------------------------
|
||
Supported tunnels and overlays include VXLAN, GENEVE, and others depending on
|
||
hardware and software configuration. Stateless offloads are enabled by default.
|
||
|
||
To view the current state of all offloads:
|
||
|
||
# ethtool -k ethX
|
||
|
||
|
||
Known Issues/Troubleshooting
|
||
============================
|
||
|
||
FUM_BAD_VF_QACCESS error on port reset
|
||
--------------------------------------
|
||
A FUM_BAD_VF_QACCESS error may be written to the message buffer when a command
|
||
or application triggers a reset on the port's physical function (PF). When the
|
||
PF is reset, any bound virtual functions (VFs) can no longer access their
|
||
queues. This behavior is expected. No user intervention is required. After the
|
||
PF reset is complete, the VFs will be able to access their queues normally.
|
||
|
||
|
||
Traffic and pings fail to pass after switch reset command issued
|
||
----------------------------------------------------------------
|
||
Resetting the switch may cause a failure to pass traffic and pings, with a
|
||
"init_hw failed: -4" message. To fix this, reload the VF driver or unbind
|
||
and rebind the VF device.
|
||
|
||
|
||
Packets dropped when issuing ping with very large payload
|
||
---------------------------------------------------------
|
||
|
||
If you issue a ping with a very large payload, you may see packets dropped.
|
||
Very large frames can fragment into many descriptors with a long enough delay
|
||
that the CPU will go to sleep. If the CPU is in a deep C-state, it may not
|
||
wake fast enough to copy those packets out of the internal descriptor cache
|
||
and into host memory. If some descriptors are dropped, then the whole
|
||
fragmented frame will be discarded because the ping is not able to located
|
||
all required fragments.
|
||
|
||
To fix this, ensure the CPU does not enter a deep C-state by using one of
|
||
the following methods:
|
||
- Change your BIOS to disable the lowest C-states
|
||
- Change the CPU governor
|
||
- Use /dev/cpu_dma_latency (see the Linux Kernel's Documentation folder)
|
||
|
||
|
||
Driver cannot bind to PF when VM assigned to PF is started
|
||
----------------------------------------------------------
|
||
In a virtualization setup, if the fm10k driver is not installed on the host OS
|
||
and a VM that has been assigned one of the PCI PF host interface devices is
|
||
started, the fm10k driver can no longer bind to the PCI PF host interface
|
||
devices on the host OS.
|
||
|
||
To correct this, manually unbind the bus driver from the PF host interface
|
||
device (or stop the VM) and then manually bind the fm10k driver to the PF host
|
||
interface device. Alternatively, you can add "fm10k" to the device's
|
||
driver_override entry in the /sys filesystem to prevent the bus driver from
|
||
binding to the PF host interface device in the first place.
|
||
|
||
|
||
Support
|
||
=======
|
||
For general information, go to the Intel support website at:
|
||
http://www.intel.com/support/
|
||
|
||
or the Intel Wired Networking project hosted by Sourceforge at:
|
||
http://sourceforge.net/projects/e1000
|
||
|
||
If an issue is identified with the released source code on a supported kernel
|
||
with a supported adapter, email the specific information related to the issue
|
||
to e1000-devel@lists.sf.net.
|
||
|
||
|
||
License
|
||
=======
|
||
This program is free software; you can redistribute it and/or modify it under
|
||
the terms and conditions of the GNU General Public License, version 2, as
|
||
published by the Free Software Foundation.
|
||
|
||
This program is distributed in the hope it will be useful, but WITHOUT ANY
|
||
WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A
|
||
PARTICULAR PURPOSE. See the GNU General Public License for more details.
|
||
|
||
You should have received a copy of the GNU General Public License along with
|
||
this program; if not, write to the Free Software Foundation, Inc., 51 Franklin
|
||
St - Fifth Floor, Boston, MA 02110-1301 USA.
|
||
|
||
The full GNU General Public License is included in this distribution in the
|
||
file called "COPYING".
|
||
|
||
Copyright(c) 2015-2018 Intel Corporation.
|
||
|
||
|
||
Trademarks
|
||
==========
|
||
Intel is a trademark or registered trademark of Intel Corporation or its
|
||
subsidiaries in the United States and/or other countries.
|
||
|
||
* Other names and brands may be claimed as the property of others.
|
||
|
||
|