INTERNET-DRAFT BMWG S. Kommu Internet-Draft VMware Intended status: Informational J. Rapp Expires: Sep 2019 VMware Mar 11, 2019 Considerations for Benchmarking Network Virtualization Platforms draft-skommu-bmwg-nvp-03.txt Status of this Memo This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet- Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html This Internet-Draft will expire on September 11, 2019. Copyright Notice Copyright (c) 2019 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License. Kommu & Rapp Expires Sep 11, 2019 [Page 1] Internet-Draft NVP Benchmarking Considerations March 2019 Abstract Current network benchmarking methodologies are focused on physical networking components and do not consider the actual application layer traffic patterns and hence do not reflect the traffic that virtual networking components work with when using network virtualization overlays (NVO3). The purpose of this document is to distinguish and highlight benchmarking considerations when testing and evaluating virtual networking components in the data center. Table of Contents 1. Introduction .................................................. 3 2. Conventions used in this document ............................. 4 3. Definitions ................................................... 4 3.1. System Under Test (SUT) ................................. 4 3.2. Network Virtualization Platform ......................... 5 3.3. Microservices ........................................... 6 4. Scope ......................................................... 7 4.1.1. Scenario 1 ........................................ 7 4.1.2. Scenario 2 ........................................ 7 4.1.3. Learning .......................................... 7 4.1.4. Flow Optimization ................................. 7 4.1.5. Out of scope ...................................... 7 4.2. Virtual Networking for Datacenter Applications .......... 8 4.3. Interaction with Physical Devices ....................... 8 5. NVP Benchmarking Considerations ............................... 9 5.1. Learning ............................................... 12 5.2. Traffic Flow Optimizations ............................. 12 5.2.1. Fast Path ........................................ 12 5.2.2. Dedicated cores / Co-processors .................. 12 5.2.3. Prioritizing and de-prioritizing active flows .... 13 5.3. Server Architecture Considerations ..................... 13 5.3.1. NVE Component considerations ..................... 13 5.3.2. Frame format/sizes within the Hypervisor ......... 17 5.3.3. Baseline testing with Logical Switch ............. 17 5.3.4. Repeatability .................................... 17 5.3.5. Tunnel encap/decap outside the Hypervisor ........ 17 5.3.6. SUT Hypervisor Profile ........................... 18 5.4. Benchmarking Tools Considerations ...................... 20 5.4.1. Considerations for NVE ........................... 20 5.4.2. Considerations for Split-NVE ..................... 20 6. Control Plane Scale Considerations ........................... 20 6.1.1. VM Events ........................................ 21 6.1.2. Scale ............................................ 22 6.1.3. Control Plane Performance at Scale ............... 22 Kommu & Rapp Expires Sep 11, 2019 [Page 2] Internet-Draft NVP Benchmarking Considerations March 2019 7. Security Considerations ...................................... 23 8. IANA Considerations .......................................... 23 9. Conclusions .................................................. 23 10. References .................................................. 24 10.1. Normative References .................................. 24 10.2. Informative References ................................ 24 11. Acknowledgments ............................................. 24 Appendix A. Partial List of Parameters to Document .............. 25 A.1. CPU .................................................... 25 A.2. Memory ................................................. 25 A.3. NIC .................................................... 26 A.4. Hypervisor ............................................. 26 A.5. Guest VM ............................................... 27 A.6. Overlay Network Physical Fabric ........................ 27 A.7. Gateway Network Physical Fabric ........................ 28 A.8. Metrics ................................................ 28 1. Introduction Datacenter virtualization that includes both compute and network virtualization is growing rapidly as the industry continues to look for ways to improve productivity, flexibility and at the same time cut costs. Network virtualization is comparatively new and expected to grow tremendously similar to compute virtualization. There are multiple vendors and solutions out in the market. Each vendor often has their own recommendations on how to benchmark their solutions thus making it difficult to perform a apples-to-apples comparision between different solutions. Hence, the need for a vendor, product and cloud agnostic way to benchmark network virtualization solutions to help with comparison and make informed decisions when it comes to selecting the right network virtualization solution. Applications traditionally have been segmented using VLANs and ACLs between the VLANs. This model does not scale because of the 4K scale limitations of VLANs. Overlays such as VXLAN were designed to address the limitations of VLANs. With VXLAN, applications are segmented based on VXLAN encapsulation (specifically the VNI field in the VXLAN header), which is similar to VLAN ID in the 802.1Q VLAN tag, however without the 4K scale limitations of VLANs. For a more detailed discussion on this subject please refer RFC 7364 'Problem Statement: Overlays for Network Virtualization'. VXLAN is just one of several Network Virtualization Overlays (NVO). Some of the others include STT, Geneve and NVGRE. STT and Geneve have expanded on the capabilities of VXLAN. Please refer IETF's nvo3 Kommu & Rapp Expires Sep 11, 2019 [Page 3] Internet-Draft NVP Benchmarking Considerations March 2019 working group < https://datatracker.ietf.org/wg/nvo3/documents/> for more information. Modern application architectures, such as Micro-services, because of IP based connectivity within the app, place high demands on the networking and security when compared to the traditional three tier app models such as web, app and db. Benchmarks MUST consider whether the proposed solution is able to scale up to the demands of such applications and not just a three-tier architecture. The benchmarks will be utilizing the various terminology and definitions from the NVO3 working group including RFC 8014 and RFC 8394. 2. Conventions used in this document The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here. 3. Definitions 3.1. System Under Test (SUT) Traditional hardware based networking devices generally use the device under test (DUT) model of testing. In this model, apart from any allowed configuration, the DUT is a black box from a testing perspective. This method works for hardware based networking devices since the device itself is not influenced by any other components outside the DUT. Virtual networking components cannot leverage DUT model of testing as the DUT is not just the virtual device but includes the hardware components that were used to host the virtual device Hence System Under Test (SUT) model MUST be used instead of the traditional device under test With SUT model, the virtual networking component along with all software and hardware components that host the virtual networking component MUST be considered as part of the SUT. Kommu & Rapp Expires Sep 11, 2019 [Page 4] Internet-Draft NVP Benchmarking Considerations March 2019 Virtual networking components, because of their dependency on the underlying hardware and other software components, may end up leveraging NIC offload benefits, such as TCP Segmentation Offload (TSO), Large Receive Offload (LRO) and Rx / Tx Filters. Such underlying hardware and software level features, even though they may not be part of virtual networking stack itself, MUST be considered and documented. Note: Physical switches and routers, including the ones that act as initiators for NVOs, work with L2/L3 packets and may not be able to leverage TCP enhancements such as TSO. Please refer to section 5 Figure 1 for a visual representation of System Under Test in the case of Intra-Host testing and section 5 Figure 2 for System Under Test in the case of Inter-Host testing. 3.2. Network Virtualization Platform This document focuses on the Network Virtualization Overlay platform as outlined in RFC 8014 and use cases from RFC 8394. Network Virtualization platforms, function closer to the application layer and are able to work with not only L2/L3 packets but also segments that leverage TCP optimizations such as Large Segment Offload (LSO). NVPs leverage TCP stack optimizations such as TCP Segmentation Offload (TSO) and Large Receive Offload (LRO) that enables NVPs to work with much larger payloads of up to 64K unlike their counterparts such as NFVs. Because of the difference in the payload, which translates into one operation per 64K of payload in NVP verses ~40 operations for the same amount of payload in NFV because of having to divide it to MTU sized packets, results in considerable difference in performance between NFV and NVP. Please refer to figure 1 for a pictorial representation of this primary difference between NPV and NFV for a 64K payload segment/packet running on network set to 1500 bytes MTU. Note: Payload sizes in figure 1 are approximates. Kommu & Rapp Expires Sep 11, 2019 [Page 5] Internet-Draft NVP Benchmarking Considerations March 2019 NVP (1 segment) NFV (40 packets) Segment 1 Packet 1 +-------------------------+ +-------------------------+ | Headers | | Headers | | +---------------------+ | | +---------------------+ | | | Payload - upto 64K | | | | Payload < 1500 | | | +---------------------+ | | +---------------------+ | +-------------------------+ +-------------------------+ Packet 2 +-------------------------+ | Headers | | +---------------------+ | | | Payload < 1500 | | | +---------------------+ | +-------------------------+ . . . . Packet 40 +-------------------------+ | Headers | | +---------------------+ | | | Payload < 1500 | | | +---------------------+ | +-------------------------+ Figure 1! Payload NPV vs NFV Hence, normal benchmarking methods are not relevant to the NVPs. Instead, newer methods that leverage TCP optimizations MUST be used for testing Network Virtualization Platforms. 3.3. Microservices Moving from traditional monolithic application architectures such as the three tier web, app and db architectures to microservices model open up networking and security stacks to new scale and performance Kommu & Rapp Expires Sep 11, 2019 [Page 6] Internet-Draft NVP Benchmarking Considerations March 2019 related challenges. At a high level, in a microservices model, a traditional monolithic app that may use few IPs is broken down into 100s of individual one-responsibility-only applications where each application has connectivity and security related requirements. These 100s of small one-responsibility-only micro-services need their own IP and also secured into their own segments, hence pushing the scale boundaries of the overlay from both simple segmentation perspective and also from a security perspective. For more details regarding microservices, please refer to wiki on microservices: https://en.wikipedia.org/wiki/Microservices 4. Scope Focus of this document is the Network Virtualization Platform in two separate scenarios as outlined in RFC 8014 section 4, Network Virtualization Edge (NVE) and RFC 8394 section 1.1 Split-NVE and the associated learning phase: 4.1.1. Scenario 1 RFC 8014 Section 4.1 "NVE Co-located with server hypervisor": Where the entire NVE functionality will typically be implemented as part of the hypervisor and/or virtual switch on the server. 4.1.2. Scenario 2 RFC 8394 Section 1.1 "Split-NVE: A type of NVE (Network Virtualization Edge) where the functionalities are split across an end device supporting virtualization and an external network device." 4.1.3. Learning Address learning rate is a key contributor to the overall performance of SUT specially in microservices type of use cases where a large amount of end-points are created and destroyed on demand. 4.1.4. Flow Optimization There are several flow optimization algorithms that are designed to help improve latency or throughput. These optimizations MUST be documented. 4.1.5. Out of scope This document does not address Network Function Virtualization which has been covered already by previous IETF documents Kommu & Rapp Expires Sep 11, 2019 [Page 7] Internet-Draft NVP Benchmarking Considerations March 2019 (https://datatracker.ietf.org/doc/draft-ietf-bmwg-virtual- net/?include_text=1). Network Function Virtualization (NFV) focuses on being independent of networking hardware while providing the same functionality. In the case of NFV, traditional benchmarking methodologies recommended by IETF may be used. Considerations for Benchmarking Virtual Network Functions and Their Infrastructure IETF document addresses benchmarking NFVs. Typical NFV implementations emulate in software, the characteristics and features of physical switches. They are similar to any physical L2/L3 switch from the perspective of the packet size, which is typically enforced based on the maximum transmission unit used. 4.2. Virtual Networking for Datacenter Applications This document focuses on the virtual networking for east-west traffic within on-prem datacenter and/or cloud. For example, in a three tier app such web, app and db, this document focuses on the east-west traffic between web and app. This document addresses scale requirements for modern application architectures such as Micro-services to consider whether the proposed solution is able to scale up to the demands of micro- services application models that basically have 100s of small services communicating on some standard ports such as http/https using protocols such as REST. 4.3. Interaction with Physical Devices Virtual network components MUST NOT be tested independent of other components within the system. Example, unlike a physical router or a firewall, where the tests can be focused solely on the device, when testing a virtual router or firewall, multiple other devices may become part of the SUT. Hence the characteristics of these other traditional networking switches and routers, LB, FW etc. MUST be considered. o Hashing method used o Over-subscription rate o Throughput available o Latency characteristics Kommu & Rapp Expires Sep 11, 2019 [Page 8] Internet-Draft NVP Benchmarking Considerations March 2019 5. NVP Benchmarking Considerations In virtual environments, the SUT may often share resources and reside on the same physical hardware with other components involved in the tests. Hence SUT MUST be clearly documented. In these tests, a single hypervisor may host multiple servers, switches, routers, firewalls etc. Intra host testing: Intra host testing helps in reducing the number of components involved in a test. For example, intra host testing would help focus on the System Under Test, logical switch and the hardware that is running the hypervisor that hosts the logical switch, and eliminate other components. Because of the nature of virtual infrastructures and multiple elements being hosted on the same physical infrastructure, influence from other components cannot be completely ruled out. For example, unlike in physical infrastructures, logical routing or distributed firewall MUST NOT be benchmarked independent of logical switching. System Under Test definition MUST include all components involved with that particular test. +---------------------------------------------------+ | System Under Test | | +-----------------------------------------------+ | | | Hyper-Visor | | | | | | | | +-------------+ | | | | | NVP | | | | | +-----+ | Switch/ | +-----+ | | | | | VM1 |<------>| Router/ |<------>| VM2 | | | | | +-----+ VW | Fire Wall/ | VW +-----+ | | | | | etc., | | | | | +-------------+ | | | | Legend | | | | VM: Virtual Machine | | | | VW: Virtual Wire | | | +-----------------------------------------------+ | +---------------------------------------------------+ Figure 2! Intra-Host System Under Test In the above figure, we only address the NVE co-located with the hypervisor. Inter-host testing: Inter-host testing helps in profiling the underlying network interconnect performance. For example, when testing Logical Switching, inter host testing would not only test the Kommu & Rapp Expires Sep 11, 2019 [Page 9] Internet-Draft NVP Benchmarking Considerations March 2019 logical switch component but also any other devices that are part of the physical data center fabric that connects the two hypervisors. System Under Test MUST be well defined to help with repeatability of tests. System Under Test definition in the case of inter host testing, MUST include all components, including the underlying network fabric. Figure 2 is a visual representation of system under test for inter- host testing. Kommu & Rapp Expires Sep 11, 2019 [Page 10] Internet-Draft NVP Benchmarking Considerations March 2019 +---------------------------------------------------+ | System Under Test | | +-----------------------------------------------+ | | | Hyper-Visor | | | | +-------------+ | | | | | NVP | | | | | +-----+ | Switch/ | +-----+ | | | | | VM1 |<------>| Router/ |<------>| VM2 | | | | | +-----+ VW | Firewall/ | VW +-----+ | | | | | etc., | | | | | +-------------+ | | | +------------------------_----------------------+ | | ^ | | | Network Cabling | | v | | +-----------------------------------------------+ | | | Physical Networking Components | | | | switches, routers, firewalls etc., | | | +-----------------------------------------------+ | | ^ | | | Network Cabling | | v | | +-----------------------------------------------+ | | | Hyper-Visor | | | | +-------------+ | | | | | NVP | | | | | +-----+ | Switch/ | +-----+ | | | | | VM1 |<------>| Router/ |<------>| VM2 | | | | | +-----+ VW | Firewall/ | VW +-----+ | | | | | etc., | | | | | +-------------+ | | | +------------------------_----------------------+ | +---------------------------------------------------+ Legend VM: Virtual Machine VW: Virtual Wire Figure 3! Inter-Host System Under Test Virtual components have a direct dependency on the physical infrastructure that is hosting these resources. Hardware characteristics of the physical host impact the performance of the virtual components. The components that are being tested and the impact of the other hardware components within the hypervisor on the performance of the SUT MUST be documented. Virtual component performance is influenced by the physical hardware components within Kommu & Rapp Expires Sep 11, 2019 [Page 11] Internet-Draft NVP Benchmarking Considerations March 2019 the hypervisor. Access to various offloads such as TCP segmentation offload, may have significant impact on performance. Firmware and driver differences may also significantly impact results based on whether the specific driver leverages any hardware level offloads offered. Packet processing could be executed on shared or dedicated cores on the main processor or via a dedicated co-processor or embedded processor on NIC. Hence, all physical components of the physical server running the hypervisor that hosts the virtual components MUST be documented along with the firmware and driver versions of all the components used to help ensure repeatability of test results. For example, BIOS configuration of the server MUST be documented as some of those changes are designed to improve performance. Please refer to Appendix A for a partial list of parameters to document. 5.1. Learning SUT needs to learn all the addresses before running any tests. Address learning rate MUST be considered in the overall performance metrics because address learning rate has a high impact in microservices based use cases where there is huge churn of end points as they are created and destroyed on demand. In these cases, both the throughput at stable state, and the time taken to get to stable state MUST be tested and documented. 5.2. Traffic Flow Optimizations Several mechanisms are employed to optimize traffic flows. Following are some examples: 5.2.1. Fast Path A single flow may go through various switching, routing and firewalling decisions. While in the standard model, every single packet has to go through the entire process/pipeline, some optimizations help make this decision for the first packet, store the final state for that packet, and leverage it to skip the process for rest of the packets that are part of the same flow. 5.2.2. Dedicated cores / Co-processors Packet processing is a CPU intensive workload. Some NVE's may use dedicated cores or a co-processor primarily for packet processing instead of sharing the cores used for the actual workloads. Such cases MUST be documented. Tests MUST be performed with both shared Kommu & Rapp Expires Sep 11, 2019 [Page 12] Internet-Draft NVP Benchmarking Considerations March 2019 and dedicated cores. Results and differences in results MUST be documented. 5.2.3. Prioritizing and de-prioritizing active flows Certain algorithms may prioritize or de-prioritize traffic flows based on purely their network characteristics such as the length of the flow. For example, de-prioritize a long-lived flow. This could result in changing the performance of a flow over a period of time. Such optimizations MUST be documented, and tests MUST consist of long-lived flows to help capture the change in performance for such flows. Tests MUST note the point at which performance changes. 5.3. Server Architecture Considerations When testing physical networking components, the approach taken is to consider the device as a black-box. With virtual infrastructure, this approach would no longer help as the virtual networking components are an intrinsic part of the hypervisor they are running on and are directly impacted by the server architecture used. Server hardware components define the capabilities of the virtual networking components. Hence, server architecture MUST be documented in detail to help with repeatability of tests. And the entire hardware and software components become the SUT. 5.3.1. NVE Component considerations 5.3.1.1. NVE co-located Components of NVE co-located may be hypervisor based or offloaded entirely to the NIC card or a hybrid model. In the case of hypervisor-based model, they may be running in user space or kernel space. Further, they may use dedicated cores, shared cores or in some cases dedicated co-processors. All the components and the process used MUST be documented. 5.3.1.2. NVE split NVE split scenario generally has three primary components as documented per RFC 8394. "tNVE: Terminal-side NVE. The portion of Split-NVE functionalities located on the end device supporting virtualization. The tNVE interacts with a Tenant System through an internal interface in the end device." tNVE may be made of either hypervisor controlled components such as hypervisor provided switches or NVE controlled Kommu & Rapp Expires Sep 11, 2019 [Page 13] Internet-Draft NVP Benchmarking Considerations March 2019 components where the network functionality is not provided by the hypervisor. In either case, the components used MUST be documented. "nNVE: Network-side NVE. The portion of Split-NVE functionalities located on the network device that is directly or indirectly connected to the end device that contains the corresponding NVE. The nNVE normally performs encapsulation to and decapsulation from the overlay network." All the functionality provided by the nNVE MUST be documented. "External NVE: The physical network device that contains the nNVE." Networking device hardware specs MUST be documented. Please use Apendix A for an example of the specs that MUST be documented. In either case, NVE co-located or NVE split all the components MUST be documented. Where possible, individual components MUST be tested independent of the entire system. For example, where possible, hypervisor provided switching functionality MUST be tested independent of the NVE. Per RFC 8014, "For the split-NVE case, protocols will be needed that allow the hypervisor and NVE to negotiate and set up the necessary state so that traffic sent across the access link between a server and the NVE can be associated with the correct virtual network instance." Supported VM lifecycle events, from RFC 8394 section 2, MUST be documented as part of the benchmark process. This process MUST also include how the hypervisor and the external NVE have signaled each other to reach an agreement. Example, see section 2.1 of RFC 8394 "VM creation event". The process used to update agreement status MUST also be documented. Kommu & Rapp Expires Sep 11, 2019 [Page 14] Internet-Draft NVP Benchmarking Considerations March 2019 +---------------------------------------------------+ | System Under Test | | +-----------------------------------------------+ | | | Hyper-Visor | | | | +-----+ +-------------+ | | | | | VM1 |<------>| tNVE | | | | | +-----+ VW +-------------+ | | | | ^ | | | | | TSI | | | | v Switch | | | | +--------------------------+ | | | | | External NVE | | | | | | Router/Firewall/etc., | | | | | +--------------------------+ | | | | ^ | | | | | TSI | | | | v | | | | +-------------+ +-----+ | | | | | tNVE |<------>| VM2 | | | | | +-------------+ VW +-----+ | | | +-----------------------------------------------+ | +---------------------------------------------------+ Legend VM: Virtual Machine VW: Virtual Wire TSI: Tenant System Interface tNVE: Terminal Side NVE Figure 4 NVE Split collocated - System Under Test Kommu & Rapp Expires Sep 11, 2019 [Page 15] Internet-Draft NVP Benchmarking Considerations March 2019 +---------------------------------------------------+ | System Under Test | | +-----------------------------------------------+ | | | Hyper-Visor | | | | +-------------+ | | | | +-----+ | NVP | | | | | | VM1 |<------>| Interface | | | | | +-----+ VW +-------------+ | | | +-----------------------------------------------+ | | ^ | | | Network Cabling | | v | | +-----------------------------------------------+ | | | Physical switches, routers, firewalls etc., | | | +-----------------------------------------------+ | | ^ | | | Network Cabling | | v | | +-----------------------------------------------+ | | | Hyper-Visor/ +--------------------------+ | | | | ToR Switch/ | NVP Split | | | | | NIC etc., | Router/Firewall/etc., | | | | | +--------------------------+ | | | +-----------------------------------------------+ | | ^ | | | Network Cabling | | v | | +-----------------------------------------------+ | | | Physical switches, routers, firewalls etc., | | | +-----------------------------------------------+ | | ^ | | | Network Cabling | | v | | +-----------------------------------------------+ | | | Hyper-Visor +-------------+ | | | | | NVP | +-----+ | | | | | Interface |<------>| VM2 | | | | | +-------------+ VW +-----+ | | | +-----------------------------------------------+ | +---------------------------------------------------+ Legend VM: Virtual Machine VW: Virtual Wire Figure 5 NVE Split not collocated - System Under Test Kommu & Rapp Expires Sep 11, 2019 [Page 16] Internet-Draft NVP Benchmarking Considerations March 2019 5.3.2. Frame format/sizes within the Hypervisor Maximum Transmission Unit (MTU) limits physical network component's frame sizes. The most common max supported MTU for physical devices is 9000. However, 1500 MTU is the standard. Physical network testing and NFV uses these MTU sizes for testing. However, the virtual networking components that live inside a hypervisor, may work with much larger segments because of the availability of hardware and software based offloads. Hence, the normal smaller packets based testing is not relevant for performance testing of virtual networking components. All the TCP related configuration such as TSO size, number of RSS queues MUST be documented along with any other physical NIC related configuration. NVE co-located may have a different performance profile when compared with NVE split because, the NVE co-located may have access to offloads that may not be available when the packet has to traverse the physical link. Such differences MUST be documented. 5.3.3. Baseline testing with Logical Switch Logical switch is often an intrinsic component of the test system along with any other hardware and software components used for testing. Also, other logical components cannot be tested independent of the Logical Switch. 5.3.4. Repeatability To ensure repeatability of the results, in the physical network component testing, much care is taken to ensure the tests are conducted with exactly the same parameters. Example parameters such as MAC addresses used. When testing NVP components with an application layer test tool, there may be a number of components within the system that may not be available to tune or to ensure they maintain a desired state. Example: housekeeping functions of the underlying Operating System. Hence, tests MUST be repeated a number of times and each test case MUST be run for at least 2 minutes if test tool provides such an option. Results SHOULD be derived from multiple test runs. Variance between the tests SHOULD be documented. 5.3.5. Tunnel encap/decap outside the Hypervisor Logical network components may also have performance impact based on the functionality available within the physical fabric. Physical Kommu & Rapp Expires Sep 11, 2019 [Page 17] Internet-Draft NVP Benchmarking Considerations March 2019 fabric that supports NVO encap/decap is one such case that may have a different performance profile. Any such functionality that exists on the physical fabric MUST be part of the test result documentation to ensure repeatability of tests. In this case SUT MUST include the physical fabric if its being used for encap/decap operations. 5.3.6. SUT Hypervisor Profile Physical networking equipment has well defined physical resource characteristics such as type and number of ASICs/SoCs used, amount of memory, type and number of processors etc., Virtual networking components performance is dependent on the physical hardware that hosts the hypervisor. Hence the physical hardware usage, which is part of SUT, for a given test MUST be documented, for example, CPU usage when running logical router. CPU usage, changes based on the type of hardware available within the physical server. For example, TCP Segmentation Offload greatly reduces CPU usage by offloading the segmentation process to the NIC card on the sender side. Receive side scaling offers similar benefit on the receive side. Hence, availability and status of such hardware MUST be documented along with actual CPU/Memory usage when the virtual networking components have access to such offload capable hardware. Following is a partial list of components that MUST be documented both in terms of what is available and also what is used by the SUT o CPU - type, speed, available instruction sets (e.g. AES-NI) o Memory - type, amount o Storage - type, amount o NIC Cards - * Type * number of ports * offloads available/used - following is a partial list of possible features o TCP Segmentation Offload o Large Receive Offload Kommu & Rapp Expires Sep 11, 2019 [Page 18] Internet-Draft NVP Benchmarking Considerations March 2019 o Checksum Offloads o Receive Side Scaling o Other Queuing Mechanisms * drivers, firmware (if applicable) * HW revision o Libraries such as DPDK if available and used o Number and type of VMs used for testing and * vCPUs * RAM * Storage * Network Driver * Any prioritization of VM resources * Operating System type, version and kernel if applicable * TCP Configuration Changes - if any * MTU o Test tool * Workload type * Protocol being tested * Number of threads * Version of tool o For inter-hypervisor tests, * Physical network devices that are part of the test o Note: For inter-hypervisor tests, system under test is no longer only the virtual component that is being Kommu & Rapp Expires Sep 11, 2019 [Page 19] Internet-Draft NVP Benchmarking Considerations March 2019 tested but the entire fabric that connects the virtual components become part of the system under test. 5.4. Benchmarking Tools Considerations 5.4.1. Considerations for NVE Virtual network components in NVE work closer to the application layer then the physical networking components, which enables the virtual network components to take advantage of TCP optimizations such as TCP Segmentation Offload (TSO) and Large Receive Offload (LRO). Because of this optimizations, virtual network components work with type and size of segments that are often not the same type and size that the physical network works with. Hence, testing virtual network components MUST be done with application layer segments instead of the physical network layer packets. Testing MUST be done with application layer testing tools such as iperf, netperf etc., 5.4.2. Considerations for Split-NVE In the case of Split-NVE, since they may not leverage any TCP related optimizations, typical network test tools focused on packet processing MUST be used. However, the tools used MUST be able to leverage Receive Side Scaling (RSS). 6. Control Plane Scale Considerations For a holistic approach to performance testing, control plane performance must also be considered. While the previous sections focused on performance tests after the SUT has come to a steady state, the following section focusses on tests to measure the time taken to bring the SUT to steady state. In a physical network infrastructure world view, this could be various stages such as boot up time, time taken to apply configuration, BGP convergence time etc., In a virtual infrastructure world, this involves lot more components which may also be distributed across multiple hosts. Some of the components are: o VM Creation Event o VM Migration Event i How many total VMs can the SUT support Kommu & Rapp Expires Sep 11, 2019 [Page 20] Internet-Draft NVP Benchmarking Considerations March 2019 o What is the rate at which the SUT would allow creation of VMs Please refer to section 2 of RFC 8394 for various VM events and their definitions. In the following section we further clarify some of the terms used in the above RFC. VM Creation For the purposes of NVP control plane testing, VM Creation event is when a VM starts participating for the first time on a NVP provided network. This involves various actions on the tNVE and NVP. Please refer to 2.1 "VM Creation Event" of RFC 8394 for more details. In order to rule out any Hypervisor imposed limitations, System Under Test must first be profiled and baselined with-out the use of NVP components. For the purposes of baselining control plane, the VM used may have very small footprint such as DSL Linux which runs in 16MB RAM. Once a baseline has been established for a single HV, a similar exercise MUST be done on multiple HVs to establish a baseline for the entire hypervisor domain. However, it may not be practical to have physical hosts and hence nested hosts may be used for this purpose 6.1.1. VM Events Performance of various control plane activities which are associated with the System Under Test, MUST BE documented. o VM Creation: Time taken to join the VMs to the SUT provided network o Policy Realization: Time taken for policy realization on the VM o VM Migration: Time taken to migrate a VM from one SUT provided network to another SUT provided network For the test itself, the following process could be use: 1 API to call to join VM on the SUT provided network 2 Loop while incrementing a timer - till the VM comes online on the SUT provided network Similarly, policy realization and VM migration may also be tested with a check on whether the VM is available or not available based on the type of policy that is applied. Kommu & Rapp Expires Sep 11, 2019 [Page 21] Internet-Draft NVP Benchmarking Considerations March 2019 6.1.2. Scale SUT must also be tested to determine the maximum scale supported. Scale can be multi-faceted such as the following: o Total # of VMs per Host o Total # of VMs per one SUT Domain o Total # of Hosts per one SUT Domain o Total # of Logical Switches per one SUT Domain * Total # of VMs per one SUT provided Logical Switch o Per Host o Per SUT Domain o Total # of Logical Routers per one SUT Domain * Total # of Logical Switches per one Logical Router * Total # of VMs on a single Logical Router o Total # of Firewall Sections o Total # of Firewall Rules per Section o Total # of Firewall Rules applied per VM o Total # of Firewall Rules applied per Host o Total # of Firewall Rules per SUT 6.1.3. Control Plane Performance at Scale Benchmarking MUST also test and document the control performance at scale. That is, o Total # VMs that can be created in parallel * How long does the action take o Total # of VMs that can be migrated in parallel * How long does the action take Kommu & Rapp Expires Sep 11, 2019 [Page 22] Internet-Draft NVP Benchmarking Considerations March 2019 o Total amount of time taken to apply 1 firewall across the entire VMs under a SUT o Time taken to apply 1000s rules on a SUT 7. Security Considerations Benchmarking activities as described in this memo are limited to technology characterization of a Device Under Test/System Under Test (DUT/SUT) using controlled stimuli in a laboratory environment, with dedicated address space and the constraints specified in the sections above. The benchmarking network topology will be an independent test setup and MUST NOT be connected to devices that may forward the test traffic into a production network, or misroute traffic to the test management network. Further, benchmarking is performed on a 'black-box' basis, relying solely on measurements observable external to the DUT/SUT. Special capabilities SHOULD NOT exist in the DUT/SUT specifically for benchmarking purposes. Any implications for network security arising from the DUT/SUT SHOULD be identical in the lab and in production networks. 8. IANA Considerations No IANA Action is requested at this time. 9. Conclusions Network Virtualization Platforms, because of their proximity to the application layer and since they can take advantage of TCP stack optimizations, do not function on packets/sec basis. Hence, traditional benchmarking methods, while still relevant for Network Function Virtualization, are not designed to test Network Virtualization Platforms. Also, advances in application architectures such as micro-services, bring new challenges and need benchmarking not just around throughput and latency but also around scale. New benchmarking methods that are designed to take advantage of the TCP optimizations or needed to accurately benchmark performance of the Network Virtualization Platforms Kommu & Rapp Expires Sep 11, 2019 [Page 23] Internet-Draft NVP Benchmarking Considerations March 2019 10. References 10.1. Normative References [RFC7364] T. Narten, E. Gray, D. Black, L. Fang, L. Kreeger, M. Napierala, 'Problem Statement: Overlays for Network Virtualization', RFC 7364, October 2014, https://datatracker.ietf.org/doc/rfc7364/ [RFC8014] D. Black, J. Hudson, L. Kreeger, M. Lasserre, T. Narten 'An Architecture for Data-Center Network Virtualization over Layer 3 (NVO3)', RFC 8014, December 2016, https://tools.ietf.org/html/rfc8014 [RFC8394] Y. Li, D. Eastlake 3rd, L. Kreeger, T. Narten, D. Black 'Split Network Virtualization Edge (Split-NVE) Control- Plane Requirements', RFC 8394, May 2018, https://tools.ietf.org/html/rfc8394 [nv03] IETF, WG, Network Virtualization Overlays, 10.2. Informative References [RFC8172] A. Morton 'Considerations for Benchmarking Virtual Network Functions and Their Infrastructure', RFC 8172, July 2017, https://tools.ietf.org/html/rfc8172 11. Acknowledgments This document was prepared using 2-Word-v2.0.template.dot. Kommu & Rapp Expires Sep 11, 2019 [Page 24] Internet-Draft NVP Benchmarking Considerations March 2019 Appendix A.!Partial List of Parameters to Document A.1. CPU CPU Vendor CPU Number CPU Architecture # of Sockets (CPUs) # of Cores Clock Speed (GHz) Max Turbo Freq. (GHz) Cache per CPU (MB) # of Memory Channels Chipset Hyperthreading (BIOS Setting) Power Management (BIOS Setting) VT-d Shared vs Dedicated packet processing User space vs Kernel space packet processing A.2. Memory Memory Speed (MHz) DIMM Capacity (GB) # of DIMMs DIMM configuration Total DRAM (GB) Kommu & Rapp Expires Sep 11, 2019 [Page 25] Internet-Draft NVP Benchmarking Considerations March 2019 A.3. NIC Vendor Model Port Speed (Gbps) Ports PCIe Version PCIe Lanes Bonded Bonding Driver Kernel Module Name Driver Version VXLAN TSO Capable VXLAN RSS Capable Ring Buffer Size RX Ring Buffer Size TX A.4. Hypervisor Hypervisor Name Version/Build Based on Hotfixes/Patches OVS Version/Build IRQ balancing vCPUs per VM Modifications to HV Kommu & Rapp Expires Sep 11, 2019 [Page 26] Internet-Draft NVP Benchmarking Considerations March 2019 Modifications to HV TCP stack Number of VMs IP MTU Flow control TX (send pause) Flow control RX (honor pause) Encapsulation Type A.5. Guest VM Guest OS & Version Modifications to VM IP MTU Guest VM (Bytes) Test tool used Number of NetPerf Instances Total Number of Streams Guest RAM (GB) A.6. Overlay Network Physical Fabric Vendor Model # and Type of Ports Software Release Interface Configuration Interface/Ethernet MTU (Bytes) Flow control TX (send pause) Flow control RX (honor pause) Kommu & Rapp Expires Sep 11, 2019 [Page 27] Internet-Draft NVP Benchmarking Considerations March 2019 A.7. Gateway Network Physical Fabric Vendor Model # and Type of Ports Software Release Interface Configuration Interface/Ethernet MTU (Bytes) Flow control TX (send pause) Flow control RX (honor pause) A.8. Metrics Drops on the virtual infrastructure Drops on the physical underlay infrastructure Authors' Addresses Samuel Kommu VMware 3401 Hillview Ave Palo Alto, CA, 94304 Email: skommu@vmware.com Jacob Rapp VMware 3401 Hillview Ave Palo Alto, CA, 94304 Email: jrapp@vmware.com Kommu & Rapp Expires Sep 11, 2019 [Page 28]