454 lines
17 KiB
Plaintext
454 lines
17 KiB
Plaintext
README for Intel(R) Ethernet Switch Host Interface Driver
|
|
===============================================================================
|
|
|
|
===============================================================================
|
|
|
|
February 23, 2017
|
|
|
|
===============================================================================
|
|
|
|
Contents
|
|
--------
|
|
|
|
- Overview
|
|
- Building and Installation
|
|
- Command Line Parameters
|
|
- Additional Features & Configurations
|
|
- Known Issues
|
|
|
|
|
|
================================================================================
|
|
|
|
|
|
Important Notes
|
|
---------------
|
|
|
|
Configuring SR-IOV for improved network security
|
|
------------------------------------------------
|
|
In a virtualized environment, on Intel(R) Ethernet Server Adapters that support
|
|
SR-IOV, the virtual function (VF) may be subject to malicious behavior.
|
|
Software-generated layer two frames, like IEEE 802.3x (link flow control), IEEE
|
|
802.1Qbb (priority based flow-control), and others of this type, are not
|
|
expected and can throttle traffic between the host and the virtual switch,
|
|
reducing performance. To resolve this issue, configure all SR-IOV enabled ports
|
|
for VLAN tagging. This configuration allows unexpected, and potentially
|
|
malicious, frames to be dropped.
|
|
|
|
|
|
Overview
|
|
--------
|
|
This driver supports kernel versions 2.6.32 and newer.
|
|
|
|
Driver information can be obtained using ethtool, lspci, and iproute2 ip.
|
|
Instructions on updating ethtool can be found in the section Additional
|
|
Configurations later in this document.
|
|
|
|
|
|
Intel(R) Ethernet SDI Adapter FM10420-100GbE-QDA2 Notes
|
|
-------------------------------------------------------
|
|
|
|
The Intel(R) Ethernet SDI Adapter FM10420-100GbE-QDA2 supports the following
|
|
cables:
|
|
Supplier Part Number
|
|
FCI 10121178-2010LF
|
|
FCI 10121178-4020LF
|
|
FCI 10121178-4030LF
|
|
FCI 10121178-4040LF
|
|
FCI 10121178-4050LF
|
|
|
|
Note: The Intel(R) Ethernet SDI Adapter FM10420-100GbE-QDA2 is a Gen3 x16 card
|
|
that is electrically bifurcated into two x8 ports. Without a BIOS that supports
|
|
bifurcation, only 1 port is available.
|
|
|
|
|
|
================================================================================
|
|
|
|
|
|
Building and Installation
|
|
-------------------------
|
|
To build a binary RPM* package of this driver, run 'rpmbuild -tb
|
|
fm10k-<x.x.x>.tar.gz', where <x.x.x> is the version number for the driver tar
|
|
file.
|
|
|
|
Note: For the build to work properly, the currently running kernel MUST match
|
|
the version and configuration of the installed kernel sources. If you have just
|
|
recompiled the kernel reboot the system before building.
|
|
|
|
Note: RPM functionality has only been tested in Red Hat distributions.
|
|
_lbank_line_
|
|
|
|
1. Move the base driver tar file to the directory of your choice. For
|
|
example, use '/home/username/fm10k' or '/usr/local/src/fm10k'.
|
|
|
|
2. Untar/unzip the archive, where <x.x.x> is the version number for the
|
|
driver tar file:
|
|
tar zxf fm10k-<x.x.x>.tar.gz
|
|
|
|
3. Change to the driver src directory, where <x.x.x> is the version number
|
|
for the driver tar:
|
|
cd fm10k-<x.x.x>/src/
|
|
|
|
4. Compile the driver module:
|
|
# make install
|
|
The binary will be installed as:
|
|
/lib/modules/<KERNEL
|
|
VERSION>/updates/drivers/net/ethernet/intel/fm10k/fm10k.ko
|
|
|
|
The install location listed above is the default location. This may differ
|
|
for various Linux distributions.
|
|
|
|
5. Load the module using the modprobe command:
|
|
modprobe <fm10k> [parameter=port1_value,port2_value]
|
|
|
|
Make sure that any older fm10k drivers are removed from the kernel before
|
|
loading the new module:
|
|
rmmod fm10k; modprobe fm10k
|
|
|
|
6. Assign an IP address to the interface by entering the following,
|
|
where ethX is the interface name that was shown in dmesg after modprobe:
|
|
|
|
ip address add <IP_address>/<netmask bits> dev ethX
|
|
|
|
NOTE: Before proceeding, ensure that netdev is enabled and that a
|
|
switch manager is running. To enable netdev, use one of the following
|
|
commands:
|
|
#ifconfig <netdev> up
|
|
or
|
|
#ip link set <netdev> up
|
|
|
|
7. Verify that the interface works. Enter the following, where IP_address
|
|
is the IP address for another machine on the same subnet as the interface
|
|
that is being tested:
|
|
ping <IP_address>
|
|
|
|
Note: For certain distributions like (but not limited to) RedHat Enterprise
|
|
Linux 7 and Ubuntu, once the driver is installed the initrd/initramfs file may
|
|
need to be updated to prevent the OS loading old versions of the fm10k driver.
|
|
The dracut utility may be used on RedHat distributions:
|
|
# dracut --force
|
|
For Ubuntu:
|
|
# update-initramfs -u
|
|
|
|
|
|
================================================================================
|
|
|
|
|
|
Command Line Parameters
|
|
-----------------------
|
|
If the driver is built as a module, the following optional parameters are used
|
|
by entering them on the command line with the modprobe command using this
|
|
syntax:
|
|
modprobe fm10k [<option>=<VAL1>]
|
|
|
|
For example:
|
|
modprobe fm10k max_vfs=7
|
|
|
|
The default value for each parameter is generally the recommended setting,
|
|
unless otherwise noted.
|
|
|
|
|
|
RSS
|
|
---
|
|
Valid Range: 0-128
|
|
0 = Assign up to the lesser value of the number of CPUs or the number of queues
|
|
X = Assign X queues, where X is less than or equal to the maximum number of
|
|
queues (128 queues).
|
|
|
|
|
|
max_vfs
|
|
-------
|
|
This parameter adds support for SR-IOV. It causes the driver to spawn up to
|
|
max_vfs worth of virtual functions.
|
|
Valid Range:0-64
|
|
|
|
NOTE: This parameter is only used on kernel 3.7.x and below. On kernel 3.8.x
|
|
and above, use sysfs to enable VFs. Also, for Red Hat distributions, this
|
|
parameter is only used on version 6.6 and older. For version 6.7 and newer, use
|
|
sysfs. For example:
|
|
#echo $num_vf_enabled > /sys/class/net/$dev/device/sriov_numvfs //enable
|
|
VFs
|
|
#echo 0 > /sys/class/net/$dev/device/sriov_numvfs //disable VFs
|
|
|
|
The parameters for the driver are referenced by position. Thus, if you have a
|
|
dual port adapter, or more than one adapter in your system, and want N virtual
|
|
functions per port, you must specify a number for each port with each parameter
|
|
separated by a comma. For example:
|
|
|
|
modprobe fm10k max_vfs=4
|
|
|
|
This will spawn 4 VFs on the first port.
|
|
|
|
modprobe fm10k max_vfs=2,4
|
|
|
|
This will spawn 2 VFs on the first port and 4 VFs on the second port.
|
|
|
|
NOTE: Caution must be used in loading the driver with these parameters.
|
|
Depending on your system configuration, number of slots, etc., it is impossible
|
|
to predict in all cases where the positions would be on the command line.
|
|
|
|
NOTE: Neither the device nor the driver control how VFs are mapped into config
|
|
space. Bus layout will vary by operating system. On operating systems that
|
|
support it, you can check sysfs to find the mapping.
|
|
|
|
NOTE: When SR-IOV mode is enabled, hardware VLAN filtering and VLAN tag
|
|
stripping/insertion will remain enabled. Please remove the old VLAN filter
|
|
before the new VLAN filter is added. For example,
|
|
ip link set eth0 vf 0 vlan 100 // set vlan 100 for VF 0
|
|
ip link set eth0 vf 0 vlan 0 // Delete vlan 100
|
|
ip link set eth0 vf 0 vlan 200 // set a new vlan 200 for VF 0
|
|
|
|
|
|
Configuring SR-IOV for improved network security
|
|
------------------------------------------------
|
|
In a virtualized environment, on Intel(R) Ethernet Server Adapters that support
|
|
SR-IOV, the virtual function (VF) may be subject to malicious behavior.
|
|
Software-generated layer two frames, like IEEE 802.3x (link flow control), IEEE
|
|
802.1Qbb (priority based flow-control), and others of this type, are not
|
|
expected and can throttle traffic between the host and the virtual switch,
|
|
reducing performance. To resolve this issue, configure all SR-IOV enabled ports
|
|
for VLAN tagging. This configuration allows unexpected, and potentially
|
|
malicious, frames to be dropped.
|
|
|
|
|
|
Configuring VLAN tagging on SR-IOV enabled adapter ports
|
|
--------------------------------------------------------
|
|
To configure VLAN tagging for the ports on an SR-IOV enabled adapter, use the
|
|
following command. The VLAN configuration should be done before the VF driver
|
|
is loaded or the VM is booted.
|
|
|
|
$ ip link set dev <PF netdev id> vf <id> vlan <vlan id>
|
|
|
|
For example, the following instructions will configure PF eth0 and the first VF
|
|
on VLAN 10.
|
|
$ ip link set dev eth0 vf 0 vlan 10
|
|
|
|
|
|
================================================================================
|
|
|
|
|
|
Additional Features and Configurations
|
|
-------------------------------------------
|
|
|
|
Configuring the Driver on Different Distributions
|
|
-------------------------------------------------
|
|
Configuring a network driver to load properly when the system is started is
|
|
distribution dependent. Typically, the configuration process involves adding an
|
|
alias line to /etc/modules.conf or /etc/modprobe.conf as well as editing other
|
|
system startup scripts and/or configuration files. Many popular Linux
|
|
distributions ship with tools to make these changes for you. To learn the
|
|
proper way to configure a network device for your system, refer to your
|
|
distribution documentation. If during this process you are asked for the driver
|
|
or module name, the name for the Base Driver is fm10k.
|
|
|
|
|
|
Viewing Link Messages
|
|
---------------------
|
|
Link messages will not be displayed to the console if the distribution is
|
|
restricting system messages. In order to see network driver link messages on
|
|
your console, set dmesg to eight by entering the following:
|
|
dmesg -n 8
|
|
|
|
NOTE: This setting is not saved across reboots.
|
|
|
|
|
|
Jumbo Frames
|
|
------------
|
|
Jumbo Frames support is enabled by changing the Maximum Transmission Unit (MTU)
|
|
to a value larger than the default value of 1500.
|
|
|
|
Use the ifconfig command to increase the MTU size. For example, enter the
|
|
following where <x> is the interface number:
|
|
|
|
ifconfig eth<x> mtu 9000 up
|
|
Alternatively, you can use the ip command as follows:
|
|
ip link set mtu 9000 dev eth<x>
|
|
ip link set up dev eth<x>
|
|
|
|
This setting is not saved across reboots. The setting change can be made
|
|
permanent by adding 'MTU=9000' to the file:
|
|
/etc/sysconfig/network-scripts/ifcfg-eth<x> for RHEL or to the file
|
|
/etc/sysconfig/network/<config_file> for SLES.
|
|
|
|
NOTE: The maximum MTU setting for Jumbo Frames is 15342. This value coincides
|
|
with the maximum Jumbo Frames size of 15364 bytes.
|
|
|
|
NOTE: This driver will attempt to use multiple page sized buffers to receive
|
|
each jumbo packet. This should help to avoid buffer starvation issues when
|
|
allocating receive packets.
|
|
|
|
|
|
ethtool
|
|
-------
|
|
The driver utilizes the ethtool interface for driver configuration and
|
|
diagnostics, as well as displaying statistical information. The latest ethtool
|
|
version is required for this functionality. Download it at:
|
|
http://ftp.kernel.org/pub/software/network/ethtool/
|
|
|
|
Supported ethtool Commands and Options for Filtering
|
|
----------------------------------------------------
|
|
-n --show-nfc
|
|
Retrieves the receive network flow classification configurations.
|
|
|
|
rx-flow-hash tcp4|udp4|ah4|esp4|sctp4|tcp6|udp6|ah6|esp6|sctp6
|
|
Retrieves the hash options for the specified network traffic type.
|
|
|
|
-N --config-nfc
|
|
Configures the receive network flow classification.
|
|
|
|
rx-flow-hash tcp4|udp4|ah4|esp4|sctp4|tcp6|udp6|ah6|esp6|sctp6
|
|
m|v|t|s|d|f|n|r...
|
|
Configures the hash options for the specified network traffic type.
|
|
|
|
udp4 UDP over IPv4
|
|
udp6 UDP over IPv6
|
|
|
|
f Hash on bytes 0 and 1 of the Layer 4 header of the rx packet.
|
|
n Hash on bytes 2 and 3 of the Layer 4 header of the rx packet.
|
|
|
|
|
|
NAPI
|
|
----
|
|
NAPI (Rx polling mode) is supported in the fm10k driver.
|
|
For more information on NAPI, see
|
|
https://www.linuxfoundation.org/collaborate/workgroups/networking/napi
|
|
|
|
|
|
Flow Control
|
|
------------
|
|
The Intel(R) Ethernet Switch Host Interface Driver does not support Flow
|
|
Control. It will not send pause frames. This may result in dropped frames.
|
|
|
|
|
|
VXLAN Overlay HW Offloading
|
|
---------------------------
|
|
|
|
Virtual Extensible LAN (VXLAN) allows you to extend an L2 network over an L3
|
|
network, which may be useful in a virtualized or cloud environment. Some
|
|
Intel(R) Ethernet Network devices perform VXLAN processing, offloading it from
|
|
the operating system. This reduces CPU utilization.
|
|
|
|
VXLAN offloading is controlled by the tx and rx checksum offload options
|
|
provided by ethtool. That is, if tx checksum offload is enabled, and the
|
|
adapter has the capability, VXLAN offloading is also enabled.
|
|
|
|
If rx checksum offload is enabled, then the VXLAN packets rx checksum will be
|
|
offloaded, unless the command #ethtool -K $INTERFACE_NAME rx off was used to
|
|
specifically disable the VXLAN rx offload.
|
|
|
|
VXLAN Overlay HW Offloading is enabled by default. To view and configure VXLAN
|
|
offload on a VXLAN-overlay offload enabled device, use the following command:
|
|
|
|
# ethtool -k ethX
|
|
(This command displays the offloads and their current state.)
|
|
|
|
For more information on configuring your network for overlay HW offloading
|
|
support, refer to the Intel Technical Brief, "Creating Overlay Networks Using
|
|
Intel Ethernet Converged Network Adapters" (Intel Networking Division, August
|
|
2013):
|
|
|
|
http://www.intel.com/content/dam/www/public/us/en/documents/technology-briefs/
|
|
overlay-networks-using-converged-network-adapters-brief.pdf
|
|
|
|
|
|
================================================================================
|
|
|
|
|
|
Known Issues/Troubleshooting
|
|
----------------------------
|
|
|
|
FUM_BAD_VF_QACCESS error on port reset
|
|
--------------------------------------
|
|
A FUM_BAD_VF_QACCESS error may be written to the message buffer when a command
|
|
or application triggers a reset on the port's physical function (PF). When the
|
|
PF is reset, any bound virtual functions (VFs) can no longer access their
|
|
queues. This behavior is expected. No user intervention is required. After the
|
|
PF reset is complete, the VFs will be able to access their queues normally.
|
|
|
|
|
|
Traffic and pings fail to pass after switch reset command issued
|
|
----------------------------------------------------------------
|
|
Resetting the switch may cause a failure to pass traffic and pings, with a
|
|
"init_hw failed: -4" message. To fix this, reload the VF driver or unbind
|
|
and rebind the VF device.
|
|
|
|
|
|
Packets dropped when issuing ping with very large payload
|
|
---------------------------------------------------------
|
|
|
|
If you issue a ping with a very large payload, you may see packets dropped.
|
|
Very large frames can fragment into many descriptors with a long enough delay
|
|
that the CPU will go to sleep. If the CPU is in a deep C-state, it may not
|
|
wake fast enough to copy those packets out of the internal descriptor cache
|
|
and into host memory. If some descriptors are dropped, then the whole
|
|
fragmented frame will be discarded because the ping is not able to located
|
|
all required fragments.
|
|
|
|
To fix this, ensure the CPU does not enter a deep C-state by using one of
|
|
the following methods:
|
|
- Change your BIOS to disable the lowest C-states
|
|
- Change the CPU governor
|
|
- Use /dev/cpu_dma_latency (see the Linux Kernel's Documentation folder)
|
|
|
|
|
|
Driver cannot bind to PF when VM assigned to PF is started
|
|
----------------------------------------------------------
|
|
In a virtualization setup, if the fm10k driver is not installed on the host OS
|
|
and a VM that has been assigned one of the PCI PF host interface devices is
|
|
started, the fm10k driver can no longer bind to the PCI PF host interface
|
|
devices on the host OS.
|
|
|
|
To correct this, manually unbind the bus driver from the PF host interface
|
|
device (or stop the VM) and then manually bind the fm10k driver to the PF host
|
|
interface device. Alternatively, you can add "fm10k" to the device's
|
|
driver_override entry in the /sys filesystem to prevent the bus driver from
|
|
binding to the PF host interface device in the first place.
|
|
|
|
|
|
================================================================================
|
|
|
|
|
|
Support
|
|
-------
|
|
For general information, go to the Intel support website at:
|
|
http://www.intel.com/support/
|
|
|
|
or the Intel Wired Networking project hosted by Sourceforge at:
|
|
http://sourceforge.net/projects/e1000
|
|
If an issue is identified with the released source code on a supported kernel
|
|
with a supported adapter, email the specific information related to the issue
|
|
to e1000-devel@lists.sf.net.
|
|
|
|
|
|
================================================================================
|
|
|
|
|
|
License
|
|
-------
|
|
This program is free software; you can redistribute it and/or modify it under
|
|
the terms and conditions of the GNU General Public License, version 2, as
|
|
published by the Free Software Foundation.
|
|
|
|
This program is distributed in the hope it will be useful, but WITHOUT ANY
|
|
WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A
|
|
PARTICULAR PURPOSE. See the GNU General Public License for more details.
|
|
|
|
You should have received a copy of the GNU General Public License along with
|
|
this program; if not, write to the Free Software Foundation, Inc., 51 Franklin
|
|
St - Fifth Floor, Boston, MA 02110-1301 USA.
|
|
|
|
The full GNU General Public License is included in this distribution in the
|
|
file called "COPYING".
|
|
|
|
Copyright(c) 2015-2017 Intel Corporation.
|
|
================================================================================
|
|
|
|
|
|
Trademarks
|
|
----------
|
|
Intel and Itanium are trademarks or registered trademarks of Intel Corporation
|
|
or its subsidiaries in the United States and/or other countries.
|
|
|
|
* Other names and brands may be claimed as the property of others.
|
|
|
|
|