Benchmarking Methodology Working Group G. Lencse Internet-Draft Széchenyi István University Intended status: Informational K. Shima Expires: 22 April 2024 SoftBank Corp. 20 October 2023 Recommendations for using Multiple IP Addresses in Benchmarking Tests draft-lencse-bmwg-multiple-ip-addresses-00 Abstract RFC 2544 has defined a benchmarking methodology for network interconnect devices. Its test frame format contained fixed IP addresses and fixed port numbers. RFC 4814 introduced pseudorandom port numbers, but it kept the usage of a single source and destination IP address pair when a single destination network is used. This limitation may cause an issue when the device under test uses Receive-Side Scaling (RSS) mechanism in the packet processing flow. RSS has two types of implementations: the first one only includes the IP addresses, whereas the second one also includes the port numbers into the tuple used for hashing. Benchmarking tests that use a single IP address pair and RFC 4814 pseudorandom port numbers are biased against the first type of RSS implementation, because in this case, the traffic is not distributed among the processing elements. This document recommends the usage of pseudorandom IP addresses in a similar manner as RFC 4814 did it with the port numbers. If accepted, this document updates all affected RFCs, including RFC 2544, RFC 4814, RFC 5180, RFC 8219. Status of This Memo This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet- Drafts is at https://datatracker.ietf.org/drafts/current/. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." This Internet-Draft will expire on 22 April 2024. Lencse & Shima Expires 22 April 2024 [Page 1] Internet-Draft Multiple IP Addresses October 2023 Copyright Notice Copyright (c) 2023 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/ license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Revised BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Revised BSD License. Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 1.1. Requirements Language . . . . . . . . . . . . . . . . . . 3 2. Recommendation for Multiple IP Addresses . . . . . . . . . . 3 2.1. Potential Ranges for IPv4 Addresses . . . . . . . . . . . 3 2.2. Potential Ranges for IPv6 Addresses . . . . . . . . . . . 5 2.3. Considerations for the IP Address Ranges to be Used . . . 5 3. Implementation of the Recommended Solution . . . . . . . . . 6 4. Considerations for Stateful Tests . . . . . . . . . . . . . . 6 5. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 6 6. Security Considerations . . . . . . . . . . . . . . . . . . . 6 7. References . . . . . . . . . . . . . . . . . . . . . . . . . 6 7.1. Normative References . . . . . . . . . . . . . . . . . . 6 7.2. Informative References . . . . . . . . . . . . . . . . . 7 Appendix A. Change Log . . . . . . . . . . . . . . . . . . . . . 7 A.1. 00 . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 7 1. Introduction [RFC2544] has defined a comprehensive benchmarking methodology for network interconnect devices, which is still in use. It was amended by several RFCs which, however, did not formally update it. [RFC4814] introduced pseudorandom port numbers (instead of fixed ones). [RFC5180] addressed IPv6 specificities and also added technology updates, but declared IPv6 transition technologies out of its scope. [RFC8219] addressed the IPv6 transition technologies. Receive-Side Scaling (RSS) aims to distribute the workload caused by packet arrivals among the CPU cores evenly to achieve high performance. It has two types of implementations: the first one only includes the IP addresses, whereas the second one also includes the port numbers into the tuple used for hashing. [RFC4814] compliant Lencse & Shima Expires 22 April 2024 [Page 2] Internet-Draft Multiple IP Addresses October 2023 testers work properly in the second case, however, pseudorandom port numbers cannot provide entropy if the Device Under Test (DUT) follows the first type of RSS implementation, therefore, these devices produce poor benchmarking results in [RFC4814] compliant laboratory tests, whereas they can exhibit high performance in production environments where the usage of multiple IP addresses ensures the entropy for the proper operation of their RSS implementation. Therefore, the conditions of laboratory tests should be improved to ensure unbiased performance testing. To that end, this document examines how the usage of multiple IP addresses can be introduced in the performance testing of network interconnect devices using IPv4 or IPv6 addresses observing the limitations of the ranges of special purpose IPv4 and IPv6 addresses reserved for benchmarking measurements. Practical recommendations are given for the usage of pseudorandom source and destination IP addresses in the case of both IPv4 and IPv6 following the approach of RFC 4814 regarding the port numbers and also considering the effect of the growing number of ARP or NDP table entries. 1.1. Requirements Language The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here. 2. Recommendation for Multiple IP Addresses 2.1. Potential Ranges for IPv4 Addresses The 198.18.0.0/15 IPv4 address range was reserved for benchmarking tests. It is divided into two halves: 198.18.0.0/16 and 198.19.0.0/16 are to be used on the two sides of the test setup. Considering the requirement of [RFC2544] regarding the IP addresses, the test suite SHOULD be first run with a single source and destination address pair. Then the destination networks should be random using the 16-23 bits of the above mentioned network addresses. The Device Under Test (DUT) is assigned the first address of each network (198.18.R.1/24 and 198.19.R.1/24, where R is pseudorandom in the 0-255 interval) and the Tester can be assigned e.g., the second address from each networks (198.18.R.2/24 and 198.19.R.2/24). The above framework imposes serious limitations to the design of how multiple IP addresses can be used. It means that when e.g., the very first networks (198.18.0.0/24 and 198.19.0.0/24) are used at each side of the test setup, the maximum range of the IP addresses assigned to the Tester can be 198.18.0.2/24-198.18.0.254/24 and Lencse & Shima Expires 22 April 2024 [Page 3] Internet-Draft Multiple IP Addresses October 2023 198.19.0.2/24-198.19.0.254/24 as shown in Figure 1. When 256 destination networks are used, then the 16-23 bits identifying the destination networks also contribute to the entropy provided to the hash function. When only a single destination network is used, then the 16-23 bits can also be leveraged for generating a higher number of IP addresses, thus their ranges can be: 198.18.0.2/16-198.18.255.254/16 and 198.19.0.2/16-198.19.255.254/16 as shown in Figure 2. 198.18.0.2/24-198.18.0.254/24 198.19.0.2/24-198.19.0.254/24 \ +----------------------------------+ / \ | | / +-------------| Tester |<------------+ | | | | | +----------------------------------+ | | | | +----------------------------------+ | | | | | +------------>| DUT: IPv4 router |-------------+ 198.18.0.1/24 | | 198.19.0.1/24 +----------------------------------+ Figure 1: Test setup for benchmarking IPv4 routers when using multiple destination networks 198.18.0.2/16-198.18.255.254/16 198.19.0.2/16-198.19.255.254/16 \ +----------------------------------+ / \ | | / +-------------| Tester |<------------+ | | | | | +----------------------------------+ | | | | +----------------------------------+ | | | | | +------------>| DUT: IPv4 router |-------------+ 198.18.0.1/16 | | 198.19.0.1/16 +----------------------------------+ Figure 2: Test setup for benchmarking IPv4 routers when using a single destination network Lencse & Shima Expires 22 April 2024 [Page 4] Internet-Draft Multiple IP Addresses October 2023 2.2. Potential Ranges for IPv6 Addresses The 2001:2::/48 IPv6 address range, which was reserved for benchmarking tests, is large enough. If it is split into two halves to be used on the two sides of the test setup as 2001:2::/49 and 2001:2:8000::/49, the ranges are abundant. Even if their first /56 subnets (2001:2::/56 and 2001:2:8000::/56) are enough to ensure 256 networks on each side of the test setup. As these networks are of /64 size, their host ID parts are vastly abundant. For convenience considerations, we recommend the usage of their 96-111 bits to generate potentially 65536 different IP addresses as shown in Figure 3 in the case when a single destination network is used. (And the 256 destination networks can be described by the 56-63 bits as mentioned before.) 2001:2::[0000-ffff]:2/64 2001:2:0:8000::[0000-ffff]:2/64 \ +----------------------------------+ / \ | | / +-------------| Tester |<------------+ | | | | | +----------------------------------+ | | | | +----------------------------------+ | | | | | +------------>| DUT: IPv6 router |-------------+ / | | \ / +----------------------------------+ \ 2001:2::1/64 2001:2:0:8000::1/64 Figure 3: Test setup for benchmarking IPv6 routers 2.3. Considerations for the IP Address Ranges to be Used On the one hand, the more IP addresses are used, the more entropy is ensured and thus the most even distribution of the load over the processing elements can be expected. However, one the other hand, the usage of multiple IP addresses has its costs: multiple Address Resolution Protocol (ARP for IPv4) or Neighbor Discovery Protocol (NDP, for IPv6) table entries are used. Increasing them over a few thousands may have a deteriorating effect on the performance of the DUT. It is noted that under the typical operating conditions, a router is not connected directly to a high number of devices. If it is a backbone router, then it is connected directly to several other routers. If it is a local router, then it is connected directly to a Lencse & Shima Expires 22 April 2024 [Page 5] Internet-Draft Multiple IP Addresses October 2023 single upstream router (or at most a few of them) and (through a switch) to the local hosts, the number of which is unlikely to be higher than a few thousands. In both cases, a high number of different IP addresses may provide entropy for hashing without causing pressure to the ARP or NDP tables of the router. The optimal number of different IP addresses to be used in laboratory tests is still a research topic. 3. Implementation of the Recommended Solution As a proof of concept, the recommended solution has been implemented in siitperf [SIITPERF]. Multiple IPv4 and IPv6 addresses are supported from commit number 165cb7f on September 6, 2023. 4. Considerations for Stateful Tests Stateful technologies like stateful NAT44 or stateful NAT64 are out of scope of this document. They are covered by [I-D.ietf-bmwg-benchmarking-stateful]. 5. IANA Considerations This document does not make any request to IANA. 6. Security Considerations TBD. 7. References 7.1. Normative References [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997, . [RFC2544] Bradner, S. and J. McQuaid, "Benchmarking Methodology for Network Interconnect Devices", RFC 2544, DOI 10.17487/RFC2544, March 1999, . [RFC4814] Newman, D. and T. Player, "Hash and Stuffing: Overlooked Factors in Network Device Benchmarking", RFC 4814, DOI 10.17487/RFC4814, March 2007, . Lencse & Shima Expires 22 April 2024 [Page 6] Internet-Draft Multiple IP Addresses October 2023 [RFC5180] Popoviciu, C., Hamza, A., Van de Velde, G., and D. Dugatkin, "IPv6 Benchmarking Methodology for Network Interconnect Devices", RFC 5180, DOI 10.17487/RFC5180, May 2008, . [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, May 2017, . [RFC8219] Georgescu, M., Pislaru, L., and G. Lencse, "Benchmarking Methodology for IPv6 Transition Technologies", RFC 8219, DOI 10.17487/RFC8219, August 2017, . 7.2. Informative References [I-D.ietf-bmwg-benchmarking-stateful] Lencse, G. and K. Shima, "Benchmarking Methodology for Stateful NATxy Gateways using RFC 4814 Pseudorandom Port Numbers", Work in Progress, Internet-Draft, draft-ietf- bmwg-benchmarking-stateful-04, 12 September 2023, . [SIITPERF] Lencse, G., "Siitperf: An RFC 8219 compliant SIIT and stateful NAT64/NAT44 tester written in C++ using DPDK", source code, available from GitHub, 2019-2023, . Appendix A. Change Log A.1. 00 Initial version. Authors' Addresses Gábor Lencse Széchenyi István University Győr Egyetem tér 1. H-9026 Hungary Email: lencse@sze.hu Lencse & Shima Expires 22 April 2024 [Page 7] Internet-Draft Multiple IP Addresses October 2023 Keiichi Shima SoftBank Corp. 1-7-1 Kaigan, Tokyo 105-7529 Japan Email: shima@wide.ad.jp URI: https://softbank.co.jp/ Lencse & Shima Expires 22 April 2024 [Page 8]