.. Copyright (c) <2010-2017>, Intel Corporation All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: - Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. - Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. - Neither the name of Intel Corporation nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. ================================ Kernel NIC Interface (KNI) Tests ================================ Description =========== This document provides the plan for testing the Kernel NIC Interface application with support of rte_kni kernel module. Kernel NIC Interface is a DPDK alternative solution to the existing linux tun-tap interface for the exception path. Kernel NIC Interface allows the standard Linux net tools(ethtool/ifconfig/tcpdump) to facilitate managing the DPDK port. At the same time, it add an interface with the kernel net stack. The test supports Multi-Thread KNI. All kni model parameter detail info on user guides:http://dpdk.org/doc/guides/sample_app_ug/kernel_nic_interface.html The ``rte_kni`` kernel module can be installed by a ``lo_mode`` parameter. loopback disabled:: insmod rte_kni.ko insmod rte_kni.ko "lo_mode=lo_mode_none" insmod rte_kni.ko "lo_mode=unsupported string" loopback mode=lo_mode_ring enabled:: insmod rte_kni.ko "lo_mode=lo_mode_ring" loopback mode=lo_mode_ring_skb enabled:: insmod rte_kni.ko "lo_mode=lo_mode_ring_skb" The ``rte_kni`` kernel module can also be installed by a ``kthread_mode`` parameter. This parameter is ``single`` by default. kthread single:: insmod rte_kni.ko insmod rte_kni.ko "kthread_mode=single" kthread multiple:: insmod rte_kni.ko insmod rte_kni.ko "kthread_mode=multiple" The ``kni`` application is run with EAL parameters and parameters for the application itself. For details about the EAL parameters, see the relevant DPDK **Getting Started Guide**. This application supports two parameters for itself. - ``--config="(port id, rx lcore, tx lcore, kthread lcore, kthread lcore, ...)"``: Port and cores selection. Kernel threads are ignored if ``kthread_mode`` is not ``multiple``. ports cores:: e.g.: --config="(0,1,2),(1,3,4)" No kernel thread specified. --config="(0,1,2,21),(1,3,4,23)" One kernel thread in use. --config="(0,1,2,21,22),(1,3,4,23,25) Two kernel threads in use. - ``-P``: Promiscuous mode. This is off by default. Prerequisites ============= If using vfio the kernel must be >= 3.6+ and VT-d must be enabled in bios.When using vfio, use the following commands to to load the vfio driver and bind it to the device under test:: modprobe vfio modprobe vfio-pci usertools/dpdk-devbind.py --bind=vfio-pci device_bus_id The DUT has at least 2 DPDK supported IXGBE NIC ports. The DUT has to be able to install rte_kni kernel module and launch kni application with a default configuration (This configuration may change form a system to another):: rmmod rte_kni rmmod igb_uio insmod ./x86_64-default-linuxapp-gcc/kmod/igb_uio.ko insmod ./x86_64-default-linuxapp-gcc/kmod/rte_kni.ko ./examples/kni/build/app/kni -c 0xa0001e -n 4 -- -P -p 0x3 --config="(0,1,2,21),(1,3,4,23)" & Test Case: ifconfig testing =========================== Launch the KNI application. Assume that ``port 2 and 3`` are used to this application. Cores 1 and 2 are used to read from NIC, cores 2 and 4 are used to write to NIC, threads 21 and 23 are used by the kernel. As the kernel module is installed using ``"kthread_mode=single"`` the core affinity is set using ``taskset``:: ./build/app/kni -c 0xa0001e -n 4 -- -P -p 0xc --config="(2,1,2,21),(3,3,4,23)" Verify whether the interface has been added:: ifconfig -a If the application is launched successfully, it will add two interfaces in kernel net stack named ``vEth2_0``, ``vEth3_0``. Interface name start with ``vEth`` followed by the port number and an additional incremental number depending on the number of kernel threads:: vEth2_0: flags=4098 mtu 1500 ether 00:00:00:00:00:00 txqueuelen 1000 (Ethernet) RX packets 14 bytes 2098 (2.0 KiB) RX errors 0 dropped 10 overruns 0 frame 0 TX packets 0 bytes 0 (0.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 vEth3_0: flags=4098 mtu 1500 ether 00:00:00:00:00:00 txqueuelen 1000 (Ethernet) RX packets 13 bytes 1756 (1.7 KiB) RX errors 0 dropped 10 overruns 0 frame 0 TX packets 0 bytes 0 (0.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 Verify whether ifconfig can set Kernel NIC Interface up:: ifconfig vEth2_0 up Now ``vEth2_0`` is up and has IPv6 address:: vEth2_0: flags=4163 mtu 1500 inet6 fe80::92e2:baff:fe37:92f8 prefixlen 64 scopeid 0x20 ether 90:e2:ba:37:92:f8 txqueuelen 1000 (Ethernet) RX packets 30 bytes 4611 (4.5 KiB) RX errors 0 dropped 21 overruns 0 frame 0 TX packets 6 bytes 468 (468.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 Verify whether ifconfig can add an ipv6 address:: ifconfig vEth2_0 add fe80::1 ``vEth2_0`` has added ipv6 address:: 29: vEth2_0: mtu 1500 qlen 1000 inet6 fe80::1/128 scope link valid_lft forever preferred_lft forever inet6 fe80::92e2:baff:fe37:92f8/64 scope link valid_lft forever preferred_lft forever Delete the IPv6 address:: ifconfig vEth2_0 del fe80::1 The port deletes it:: 29: vEth2_0: mtu 1500 qlen 1000 inet6 fe80::92e2:baff:fe37:92f8/64 scope link valid_lft forever preferred_lft forever Set MTU parameter:: ifconfig vEth2_0 mtu 1300 ``vEth2_0`` changes the mtu parameter:: 29: vEth2_0: mtu 1300 qdisc pfifo_fast state UNKNOWN mode DEFAULT qlen 1000 link/ether 90:e2:ba:37:92:f8 brd ff:ff:ff:ff:ff:ff Verify whether ifconfig can set ip address:: ifconfig vEth2_0 192.168.2.1 netmask 255.255.255.192 ip -family inet address show dev vEth2_0 ``vEth2_0`` has IP address and netmask now:: 29: vEth2_0: mtu 1300 qdisc pfifo_fast state UNKNOWN qlen 1000 inet 192.168.2.1/26 brd 192.168.2.63 scope global vEth2_0 Verify whether ifconfig can set ``vEth2_0`` down:: ifconfig vEth2_0 down ifconfig vEth2_0 ``vEth2_0`` is down and no ipv6 address:: vEth2_0: flags=4098 mtu 1300 inet 192.168.2.1 netmask 255.255.255.192 broadcast 192.168.2.63 ether 90:e2:ba:37:92:f8 txqueuelen 1000 (Ethernet) RX packets 70 bytes 12373 (12.0 KiB) RX errors 0 dropped 43 overruns 0 frame 0 TX packets 25 bytes 4132 (4.0 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 Repeat all the steps for interface ``vEth3_0`` Test Case: Ping and Ping6 testing ================================= If the application is launched successfully, it will add two interfaces in kernel net stack named ``vEth2_0``, ``vEth3_0``. Assume the link status of ``vEth2_0`` is up and set ip address is ``192.168.2.1`` and ``vEth3_0`` is up and set ip address is ``192.168.3.1``. Verify the command ping:: ping -w 1 -I vEth2_0 192.168.2.1 it can receive all packets and no packet loss:: PING 192.168.2.1 (192.168.2.1) from 192.168.2.1 vEth2_0: 56(84) bytes of data. 64 bytes from 192.168.2.1: icmp_req=1 ttl=64 time=0.040 ms --- 192.168.2.1 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 0.040/0.040/0.040/0.000 ms Assume ``port A`` on tester is linked with ``port 2`` on DUT. Verify the command ping from tester:: ping -w 1 -I "port A" 192.168.2.1 it can receive all packets and no packet loss. Verify a wrong address:: ping -w 1 -I vEth2_0 192.168.0.123 no packets is received:: PING 192.168.0.123 (192.168.0.123) from 192.168.0.1 vEth2_0: 56(84) bytes of data. --- 192.168.0.123 ping statistics --- 1 packets transmitted, 0 received, 100% packet loss, time 0ms Verify the command ping6:: ping6 -w 1 -I vEth2_0 "Eth2_0's ipv6 address" it can receive all packets and no packet loss:: PING fe80::92e2:baff:fe08:d6f0(fe80::92e2:baff:fe08:d6f0) from fe80::92e2:baff:fe08:d6f0 vEth2_0: 56 data bytes 64 bytes from fe80::92e2:baff:fe08:d6f0: icmp_seq=1 ttl=64 time=0.070 ms --- fe80::92e2:baff:fe08:d6f0 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 0.070/0.070/0.070/0.000 ms Verify the command ping6 from tester:: ping6 -w 1 -I "port A" "Eth2_0's ipv6 address" it can receive all packets and no packet loss. Verify a wrong ipv6 address:: ping6 -w 1 -I vEth2_0 "random ipv6 address" no packets is received:: PING fe80::92e2:baff:fe08:d6f1(fe80::92e2:baff:fe08:d6f1) from fe80::92e2:baff:fe08:d6f0 vEth2_0: 56 data bytes --- fe80::92e2:baff:fe08:d6f1 ping statistics --- 1 packets transmitted, 0 received, 100% packet loss, time 0ms Repeat all the steps for interface ``vEth3_0`` Test Case: Tcpdump testing ========================== Assume ``port A and B`` on packet generator connects to NIC ``port 2 and 3``. Trigger the packet generator of bursting packets from ``port A and B`, then check if tcpdump can capture all packets. The packets should include ``tcp`` packets, ``udp`` packets, ``icmp`` packets, ``ip`` packets, ``ether+vlan tag+ip`` packets, ``ether`` packets. Verify whether tcpdump can capture packets:: tcpdump -i vEth2_0 tcpdump -i vEth3_0 Test Case: Ethtool testing ========================== In this time, KNI can only support ethtool commands which is to get information. So all below commands are to show information commands. Verify whether ethtool can show Kernel NIC Interface's standard information:: ethtool vEth2_0 Verify whether ethtool can show Kernel NIC Interface's driver information:: ethtool -i vEth2_0 Verify whether ethtool can show Kernel NIC Interface's statistics:: ethtool -S vEth2_0 Verify whether ethtool can show Kernel NIC Interface's pause parameters:: ethtool -a vEth2_0 Verify whether ethtool can show Kernel NIC Interface's offload parameters:: ethtool -k vEth2_0 Verify whether ethtool can show Kernel NIC Interface's RX/TX ring parameters:: ethtool -g vEth2_0 Verify whether ethtool can show Kernel NIC Interface's Coalesce parameters. It is not currently supported:: ethtool -c vEth2_0 Verify whether ethtool can show Kernel NIC Interface's MAC registers:: ethtool -d vEth2_0 Verify whether ethtool can show Kernel NIC Interface's EEPROM dump:: ethtool -e vEth2_0 Repeat all the steps for interface ``vEth3_0`` Test Case: Packets statistics testing ===================================== Install the kernel module with loopback parameter ``lo_mode=lo_mode_ring_skb`` and launch the KNI application. Assume that ``port 2 and 3`` are used by this application:: rmmod kni insmod ./kmod/rte_kni.ko "lo_mode=lo_mode_ring_skb" ./build/app/kni -c 0xff -n 3 -- -p 0xf -i 0xf -o 0xf0 Assume ``port A and B`` on tester connects to NIC ``port 2 and 3``. Get the RX packets count and TX packets count:: ifconfig vEth2_0 Send 5 packets from tester. And check whether both RX and TX packets of ``vEth2_0`` have increased 5. Repeat for interface ``vEth3_0`` Test Case: Stress testing ========================= Insert the rte_kni kernel module 50 times while changing the parameters. Iterate through lo_mode and kthread_mode values sequentially, include wrong values. After each insertion check whether kni application can be launched successfully. Insert the kernel module 50 times while changing randomly the parameters. Iterate through lo_mode and kthread_mode values randomly, include wrong values. After each insertion check whether kni application can be launched successfully:: rmmod rte_kni insmod ./kmod/rte_kni.ko ./build/app/kni -c 0xa0001e -n 4 -- -P -p 0xc --config="(2,1,2,21),(3,3,4,23)" Using ``dmesg`` to check whether kernel module is loaded with the specified parameters. Some permutations, those with wrong values, must fail to success. For permutations with valid parameter values, verify the application can be successfully launched and then close the application using CTRL+C. Test Case: loopback mode performance testing ============================================ Compare performance results for loopback mode using: - lo_mode: lo_mode_fifo and lo_mode_fifo_skb. - kthread_mode: single and multiple. - Number of ports: 1 and 2. - Number of virtual interfaces per port: 1 and 2 - Frame sizes: 64 and 256. - Cores combinations: - Different cores for Rx, Tx and Kernel. - Shared core between Rx and Kernel. - Shared cores between Rx and Tx. - Shared cores between Rx, Tx and Kernel. - Multiple cores for Kernel, implies multiple virtual interfaces per port. :: insmod ./x86_64-default-linuxapp-gcc/kmod/igb_uio.ko insmod ./x86_64-default-linuxapp-gcc/kmod/rte_kni.ko ./examples/kni/build/app/kni -c -n 4 -- -P -p --config="" & At this point, the throughput is measured and recorded for the different frame sizes. After this, the application is closed using CTRL+C. The measurements are presented in a table format. +------------------+--------------+-------+-----------------+--------+--------+ | lo_mode | kthread_mode | Ports | Config | 64 | 256 | +==================+==============+=======+=================+========+========+ | | | | | | | +------------------+--------------+-------+-----------------+--------+--------+ Test Case: bridge mode performance testing ========================================== Compare performance results for bridge mode using: - kthread_mode: single and multiple. - Number of ports: 2 - Number of ports: 1 and 2. - Number of flows per port: 1 and 2 - Number of virtual interfaces per port: 1 and 2 - Frame size: 64. - Cores combinations: - Different cores for Rx, Tx and Kernel. - Shared core between Rx and Kernel. - Shared cores between Rx and Tx. - Shared cores between Rx, Tx and Kernel. - Multiple cores for Kernel, implies multiple virtual interfaces per port. The application is launched and the bridge is setup using the commands below:: insmod ./x86_64-default-linuxapp-gcc/kmod/rte_kni.ko ./build/app/kni -c -n 4 -- -P -p --config="" & ifconfig vEth2_0 up ifconfig vEth3_0 up brctl addbr "br_kni" brctl addif br_kni vEth2_0 brctl addif br_kni vEth3_0 ifconfig br_kni up At this point, the throughput is measured and recorded. After this, the application is closed using CTRL+C and the bridge deleted:: ifconfig br_kni down brctl delbr br_kni The measurements are presented in a table format. +--------------+-------+-----------------------------+-------+ | kthread_mode | Flows | Config | 64 | +==============+=======+=============================+=======+ | | | | | +--------------+-------+-----------------------------+-------+ Test Case: bridge mode without KNI performance testing ====================================================== Compare performance results for bridge mode using only Kernel bridge, no DPDK support. Use: - Number of ports: 2 - Number of flows per port: 1 and 2 - Frame size: 64. Set up the interfaces and the bridge:: rmmod rte_kni ifconfig vEth2_0 up ifconfig vEth3_0 up brctl addbr "br1" brctl addif br1 vEth2_0 brctl addif br1 vEth3_0 ifconfig br1 up At this point, the throughput is measured and recorded. After this, the application is closed using CTRL+C and the bridge deleted:: ifconfig br1 down brctl delbr br1 The measurements are presented in a table format. +-------+-------+ | Flows | 64 | +=======+=======+ | 1 | | +-------+-------+ | 2 | | +-------+-------+ Test Case: routing mode performance testing =========================================== Compare performance results for routing mode using: - kthread_mode: single and multiple. - Number of ports: 2 - Number of ports: 1 and 2. - Number of virtual interfaces per port: 1 and 2 - Frame size: 64 and 256. - Cores combinations: - Different cores for Rx, Tx and Kernel. - Shared core between Rx and Kernel. - Shared cores between Rx and Tx. - Shared cores between Rx, Tx and Kernel. - Multiple cores for Kernel, implies multiple virtual interfaces per port. The application is launched and the bridge is setup using the commands below:: echo 1 > /proc/sys/net/ipv4/ip_forward insmod ./x86_64-default-linuxapp-gcc/kmod/rte_kni.ko ./build/app/kni -c -n 4 -- -P -p --config="" & ifconfig vEth2_0 192.170.2.1 ifconfig vEth3_0 192.170.3.1 route add -net 192.170.2.0 netmask 255.255.255.0 gw 192.170.2.1 route add -net 192.170.3.0 netmask 255.255.255.0 gw 192.170.3.1 arp -s 192.170.2.2 vEth2_0 arp -s 192.170.3.2 vEth3_0 At this point, the throughput is measured and recorded. After this, the application is closed using CTRL+C. The measurements are presented in a table format. +--------------+-------+-----------------------------+-------+-------+ | kthread_mode | Ports | Config | 64 | 256 | +==============+=======+=============================+=======+=======+ | | | | | | +--------------+-------+-----------------------------+-------+-------+ Test Case: routing mode without KNI performance testing ======================================================= Compare performance results for routing mode using only Kernel, no DPDK support. Use: - Number of ports: 2 - Frame size: 64 and 256 Set up the interfaces and the bridge:: echo 1 > /proc/sys/net/ipv4/ip_forward rmmod rte_kni ifconfig vEth2_0 192.170.2.1 ifconfig vEth3_0 192.170.3.1 route add -net 192.170.2.0 netmask 255.255.255.0 gw 192.170.2.1 route add -net 192.170.3.0 netmask 255.255.255.0 gw 192.170.3.1 arp -s 192.170.2.2 vEth2_0 arp -s 192.170.3.2 vEth3_0 At this point, the throughput is measured and recorded. After this, the application is closed using CTRL+C. The measurements are presented in a table format. +-------+-------+-------+ | Ports | 64 | 256 | +=======+=======+=======+ | 1 | | | +-------+-------+-------+ | 2 | | | +-------+-------+-------+