Network Working Group Gabor Feher, BUTE INTERNET-DRAFT Istvan Cselenyi, TRAB Expiration Date: January 2001 Krisztian Nemeth, BUTE July 2000 Benchmarking Terminology and Methodology for Routers Supporting Resource Reservation 1. Status of this Memo This document is an Internet-Draft and is in full conformance with all provisions of Section 10 of RFC2026. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet- Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt The list of Internet-Draft shadow directories can be accessed at http://www.ietf.org/shadow.html This memo provides information for the Internet community. This memo does not specify an Internet standard of any kind. Distribution of this memo is unlimited. 2. Table of content 1. Status of this Memo.............................................1 2. Table of content................................................1 3. Abstract........................................................2 4. Introduction....................................................2 5. Existing definitions............................................3 6. Definitions of Terms............................................3 6.1 Resource Reservation Protocol...............................3 6.1.1 Resource Reservation Session...........................4 6.1.2 Multicast Resource Reservation Session.................4 6.1.3 Reservation Capable Router.............................4 6.1.4 Signaling End-point....................................5 6.1.5 Reservation Initiator..................................5 6.1.6 Signaling Path.........................................6 6.1.7 Signaling Message Sequence.............................6 6.1.8 Multicast aware Resource Reservation Protocol..........7 6.1.9 Scalability Limit......................................7 Feher, Cselenyi, Nemeth Expires January 2001 [Page 1] INTERNET-DRAFT July 2000 6.2 Traffic Types...............................................8 6.2.1 Premium Traffic........................................8 6.2.2 Best-Effort Traffic....................................8 6.3 Router Load Types...........................................8 6.3.1 Signaling Load.........................................8 6.3.2 Signaling Burst........................................9 6.3.3 Reservation Load.......................................9 6.4 Performance Metrics........................................10 6.4.1 Signaling Message Handling Time.......................10 6.4.2 Premium Traffic Extra Delay...........................11 6.4.3 Best-effort Traffic Extra Delay.......................12 6.4.4 Signaling Message Loss................................12 7. Methodology....................................................13 7.1 Evaluating the Results.....................................13 7.2 Test Set up................................................13 7.2.1 One Tester............................................13 7.2.2 Two Testers...........................................14 7.2.3 Test Set up for Unicast Resource Reservation..........14 7.2.4 Test Set up for Multicast Resource Reservation........14 7.2.5 Signaling Message Verification........................15 7.3 Scalability Tests..........................................15 7.3.1 Maximum Signaling Message Burst Size -- Unicast Case..16 7.3.2 Maximum Signaling Message Burst Size -- Multicast Case16 7.3.3 Maximum Signaling Intensity -- Unicast Case...........17 7.3.4 Maximum Signaling Intensity -- Multicast Case.........18 7.3.5 Maximum Number of Reservation Sessions -- Unicast Case18 7.3.6 Maximum Number of Reservation Sessions -- Multicast Case ............................................................19 7.4 Benchmarking Tests.........................................19 7.4.1 Signaling Message Handling Time -- Unicast Case.......20 7.4.2 Signaling Message Handling Time -- Multicast Case.....22 7.4.3 Premium Traffic Extra Delay -- Unicast Case...........22 7.4.4 Premium Traffic Extra Delay -- Multicast Case.........23 7.4.5 Best-effort Traffic Extra Delay.......................24 8. Acknowledgement................................................24 9. References.....................................................24 10. Authors' Addresses:...........................................25 3. Abstract The purpose of this document is to define terminology and methodology specific to the performance benchmarking the resource reservation signaling capability of IP routers. In addition to defining the performance metrics and the tests, this document also describes specific formats for reporting the results of the tests. 4. Introduction The IntServ over DiffServ framework [3] outlines a heterogeneous Quality of Service (QoS) architecture for multi domain Internet services. Signaling based resource reservation (e.g. by RSVP [5]) is Feher, Cselenyi, Nemeth Expires January 2001 [Page 2] INTERNET-DRAFT July 2000 an integral part of that model. While this way scalability constraints are released for most of the core routers, the performance of border routers that handle signaling is still crucial. Therefore network operators, who are planning to deploy this model, shall scrutinize the scalability limitations in reservation capable routers and the impact of signaling on the routers’ forwarding performance. An objective way for quantifying the scalability constraints of QoS signaling is to perform measurements on routers that are capable of resource reservation. This document defines a specific set of tests that vendors or network operators can use to measure and report the signaling performance characteristics of router devices that support resource reservation protocols. The results of these tests will provide comparable data for different products supporting the decision process of purchase. Moreover, these measurements provide input characteristics for the dimensioning of a network in which resources are provisioned dynamically by signaling. Finally, these test are applicable for characterizing the impact of control plane signaling on the forwarding performance of routers. The first part of the document, "Terminology", defines the terms that are used later on in the document. The second part is the "Methodology" that defines measurement methods to find the scalability limits and characterize the signaling performance of the tested router. This benchmarking terminology and methodology is based on the knowledge gained by examination of four very different resource reservation protocols: RSVP [5], Boomerang [6], YESSIR [7] and ST2+ [8]. Nevertheless, this document aspires to compose terms and methods that are valid in general and not restricted to these four protocols. 5. Existing definitions RFC 1242 [1] "Benchmarking Terminology for Network Interconnect Devices" and RFC 2285 [2] "Benchmarking Terminology for LAN Switching Devices" contains discussions of a number of terms relevant to the benchmarking of signaling performance of reservation capable routers and should be consulted before attempting to make use of this document. For the sake of clarity and continuity this document adopts the template for definitions set out in Section 2 of RFC 1242 [1]. Definitions are indexed and grouped together in sections for ease of reference. 6. Definitions of Terms 6.1 Resource Reservation Protocol This group of definitions applies to various resource reservation protocols implemented in router devices. Feher, Cselenyi, Nemeth Expires January 2001 [Page 3] INTERNET-DRAFT July 2000 6.1.1 Resource Reservation Session Definition: A resource reservation session (or shortly reservation) expresses that certain QoS treatment is assigned to the packets of a traffic flow along the path between two hosts. Discussion: The QoS treatment can be specified explicitly by giving the amount of networking resources (e.g. forwarding capacity or buffer space) or it can be specified implicitly by referring to a forwarding behavior (e.g. EF PHB in case of Differentiated Services [9]). The flow can be specified either by the five-tuple (i.e. as a micro- flow in [9]) or as traffic aggregate. See Also: Signaling Path 6.1.2 Multicast Resource Reservation Session Definition: Multicast resource reservation session (or shortly multicast reservation) denotes that certain QoS treatment is assigned to the packets of a traffic flow related to a multicast address. Discussion: There are resource reservation protocols that allow different amount of resource dedication in reservation capable routers for one multicast reservation session. These protocols distinguish different traffic sources and different traffic destinations and can have more than one reservation models to express the resource needs within the reservation. (e.g. RSVP SE/WF/FF [5]) Issues: - Reservation aggregation See Also: Signaling Path 6.1.3 Reservation Capable Router Definition: A router is reservation capable if it understands a resource reservation protocol that signals the set up, tear down or modification of a reservation in the router. Discussion: Resource reservation protocols can be divided into two groups, the soft state and the hard state protocols. Both kinds of protocol tear a resource reservation entry upon request but in case of soft state protocols the router also tears an entry if a refresh message does not renew the resource reservation entry within a Feher, Cselenyi, Nemeth Expires January 2001 [Page 4] INTERNET-DRAFT July 2000 refresh period. Thus in case of soft state protocols the number of refresh messages may cause considerable signaling load, while on the other hand it is easier to build a robust protocol that is soft state. 6.1.4 Signaling End-point Definition: A Signaling End-point can be a host or a network node, which is capable to initiate and/or terminate or proxy signaling sessions. Discussion: Usually, signaling end-points have a protocol stack that is capable to generate and understand signaling messages. However in some cases this is not necessary. When the messages of the resource reservation protocol are payloads of other protocols that the host understands and the resource reservation protocol does not require any special action from signaling end-points then the host becomes a signaling end-point without knowing the resource reservation protocol (e.g. Boomerang [6] encapsulates the reservation requests to an ICMP Echo message). Reservation proxies are protocol translators that translate the signaling messages of one resource reservation protocol into messages of other resource reservation protocols bridging different networks with different resource reservation protocols. Thus the reservation proxy is two signaling end-points in one, as it is both a signaling terminator and a signaling initiator. 6.1.5 Reservation Initiator Definition: The reservation initiator is the signaling end-point, which initiates the reservation set up. Discussions: Resource reservation protocols can be divided into different groups according to the initiator of the reservation request, what can be the traffic source(s), the traffic destination(s), or both of them. Typical example for the later is RSVP, where the data traffic destination(s) requests the resource reservation. This scheme is called receiver-oriented where "receiver" refers to the data traffic receiver. A receiver-oriented protocol must assure that the data flow(s) coming from the traffic source(s) to the traffic destination(s) takes the same path as the reservation request, which travels in the opposite direction. IP routing does not guarantee the same path for the different directions between end-points, so the resource reservation protocol has to take care of it. Therefore RSVP has a signaling message primitive, called the PATH message, which establishes the signaling path(s) among traffic sources and traffic destinations before the reservation can happen and Feher, Cselenyi, Nemeth Expires January 2001 [Page 5] INTERNET-DRAFT July 2000 afterwards the signaling messages are forwarded on this path backward. See Also: Signaling end-point Signaling path 6.1.6 Signaling Path Definition: A signaling path is a sequence of network nodes and links on which signaling messages travel from one signaling end-point to the other. Discussion: In case of reservation session where the data traffic is unicast and goes point-to-point, the signaling path is mostly the same as the data path. However in case of multicast reservations there are more than one signaling path belonging to one reservation session according to the definition. There is one signaling path between each traffic source and traffic destination. Both in the unicast and multicast case, it might happen that the path of the reservation is not the same as the path of the data stream, as recent reservation capable routers do not take effort to guarantee it. Still, the current practice is that the signaling messages and the data packets are addressed with the same IP address, trusting that in this case they are forwarded on the same path. Issues: - Multicast signaling path graph - Route change 6.1.7 Signaling Message Sequence Definition: Signaling message sequence is defined as a series of signaling messages, which are related to the same reservation session and whose order follows the specification of the reservation protocol. Discussion: A typical signaling message sequence is a reservation set up and reservation tear down pair, where the first signaling message sets up a reservation in the router and the second one tears that down. Signaling message sequences are used in measurements where the changes in the states of the router affect the measurement. Using signaling message sequences the router states are restored after the last message in the sequence. (e.g.: RSVP's PATH and PATHTEAR message pair) Feher, Cselenyi, Nemeth Expires January 2001 [Page 6] INTERNET-DRAFT July 2000 6.1.8 Multicast aware Resource Reservation Protocol Definition: A resource reservation protocol is multicast aware when it supports resource reservation for multicast sessions. Discussion: There are two types of multicast support, one is the many-to-many multicast and the other is the one-to-many multicast. The former allows reservations for many traffic sources in the multicast group, while the latter allows for only one traffic source in the group. When there are more than one data traffic sources in a multicast reservation session then the protocol might aggregate the reservation towards the data traffic destinations. Certainly a multicast aware protocol is more complex than a non- multicast aware. Currently perhaps the most well-known IP resource reservation protocol, RSVP [5], does support many-to-many multicast, while another protocol, Boomerang [6], supports one-to- many multicast. Issues: - Multicast traffic - IP multicast 6.1.9 Scalability Limit Definition: Scalability limit is the border between the steady and the overloaded state of the Device Under Test (DUT). Discussion: All existing routers have finite memory buffer range and finite processing power (CPU). In the steady state of the router the memory buffers are not fully utilized and the processing power is enough to cope with the different tasks of the router. However when the load rises in the router there is a certain point where no more memory space is available or the processing power is not enough to satisfy the needs of all the tasks and thus the router becomes overloaded. The critical load condition, when the router is still in the steady state but the smallest amount of load increase would drive it to the overloaded state is the scalability limit of the router. Usually the overloaded state of the router can be recognized by some kinds of data loss. A resource reservation capable router may drop signaling messages, data packets or reservation entries, as it does not have enough capacity to process or maintain them. Smart task scheduling is able to overcome on this for short periods when the load is above the scalability limit, thus the router remains in the steady state. Therefore the scalability limit should be determined under a long lasting constant load. Feher, Cselenyi, Nemeth Expires January 2001 [Page 7] INTERNET-DRAFT July 2000 Issues: - Scalability tests 6.2 Traffic Types This group of definitions defines traffic types forwarded by resource reservation capable routers. 6.2.1 Premium Traffic Definition: Premium traffic is a traffic type that the router distinguishes from the best-effort traffic and forwards its packets according to a QoS agreement. Discussion: Each premium traffic flow has a resource reservation entry in the router set up by signaling messages. The router might define more than one premium traffic type (e.g. delay sensitive traffic, loss sensitive traffic) and different premium traffic types have different QoS treatments. 6.2.2 Best-Effort Traffic Definition: Best-effort traffic is the traffic that has no reservation entry in the router. Discussion: Traffic flows that do not require QoS guaranties along their path are considered as best-effort traffic. Best effort means that the router takes its best effort to forward every data packets, but it cannot give any guarantee. This type of traffic is the standard in today’s Internet. 6.3 Router Load Types This group of definitions defines different load components, which are used for testing the resource reservation performance of the reservation capable router. 6.3.1 Signaling Load Definition: The signaling load is characterized by the signaling intensity, which expresses, how many signaling messages arrive to the reservation capable router within a time unit. Discussion: As the processing of signaling messages consumes router power, the signaling intensity is correlated with the load. This load is called signaling load because it is generated by signaling. The Feher, Cselenyi, Nemeth Expires January 2001 [Page 8] INTERNET-DRAFT July 2000 higher the signaling intensity is, the more signaling messages hit the router within a time unit. Therefore the processing capacity spent on signaling handling increases, resulting higher load in the router. Most of the resource reservation protocols have several protocol primitives realized by different signaling message types. Each of these message types requires different processing capacity from the router. Issues: - Different signaling message types load the router differently - Message generation burstiness Measurement unit: Number of signaling messages per second [1/s] See Also: Signaling burst 6.3.2 Signaling Burst Definition: The signaling burst denotes a certain number of signaling messages, which arrive to the input port(s) of the router continuously causing persistent load for the signaling message handler. The single parameter of the signaling burst is the number of signaling messages in the burst. Discussion: Back-to-back signaling messages on one port of the router form a typical signaling burst. But other cases can be imagined also where signaling messages arrive on different ports simultaneously or with an overlap in time (i.e. when the tail of one signaling message is behind the head of another one arriving on another port). Certainly, as in case of signaling load, the signaling message type is an issue here, too. Issues: - Different types of signaling messages Measurement unit: The signaling burst size is the number of messages arrived during the burst See Also: Signaling intensity 6.3.3 Reservation Load Definition: Feher, Cselenyi, Nemeth Expires January 2001 [Page 9] INTERNET-DRAFT July 2000 The reservation load is characterized by the number of resource reservation sessions set up in the router. Discussion: In case of micro-flow classification the packet classifier shall distinguish the flows, which have reservations in the router from the others, which do not. Therefore the more reservation session is set up in the router the more complex the traffic classification is. This is one factor that contributes to the reservation load. Moreover, most signaling based resource reservation protocols require that the routers maintain some kind of state for each flow they keep reservation for. The growth of this state space is a scalability issue as on one hand it requires memory capacity and on the other hand searching through a larger state space may increase the signaling handling time. Note that some protocols (e.g. RSVP, Boomerang) allow making reservation for the aggregation of several micro-flows. In this case the number of reserved sessions can be smaller than the number of micro-flows. In case of soft state protocols the refresh messages required to maintain the reservations cause a considerable signaling load as well beside the reservation load. Although, some soft state protocols might be capable to aggregate refresh messages that decreases significantly the signaling load. Issues: - Micro-flow aggregation - Refresh message aggregation Measurement unit: Number of reservation sessions in the router 6.4 Performance Metrics This group of definitions is the collection of the measurable effects of the impact of the resource reservation on a router device. 6.4.1 Signaling Message Handling Time Definition: The signaling message handling time (or shortly signal handling time) is the time that a signaling message spends inside the router before it is forwarded to the next hop. Discussion: Usually signaling messages are issued by a signaling end-point and forwarded via the signaling path by the routers. However, beside the message forwarding the router interprets the content of the messages and acts according to it. Thus the message handling time is longer than forwarding time of data packets of the same size. Moreover, there are signaling message primitives that are altered Feher, Cselenyi, Nemeth Expires January 2001 [Page 10] INTERNET-DRAFT July 2000 during the processing and there might be messages also that are drained in the router or generated by the router. Thus, the signal message handling time is the time difference of a signaling message entering time and the leaving time of the corresponding processed signaling message. When a message is drained by the router then the signal handling time is immeasurable therefore it is not defined. In case of signaling messages that carry instructions for multicast flows, the router might issue multiple signaling messages after processing. In this case, by definition, the signal handling time is the time interval elapsed between the arrival of the incoming signaling message to the router and the departure of the last, related signaling message. Signal handling time is an important measure as it directly affects the setup time of a session. On the other hand, it is also an indication of the signal processing capacity of the measured router as it is correlated to the number of signaling messages that can be processed within a time unit. This metric is also dependent on the type of the signaling message. For example, it takes generally smaller time to tear down a session within a router than to set it up. Issues: - Refresh message aggregation - The type of the signaling message Measurement unit: seconds 6.4.2 Premium Traffic Extra Delay Definition: Premium traffic extra delay is the delay that a data packet of a reserved flow suffers because of the resource reservation protocol running on the router. Discussion: This metric shows the effect of the classification, policing and shaping along with the cross-effect of the control plane on the data plane. Classification [9] is the mechanism for filtering out premium traffic. All data packet belonging to premium traffic must be classified in order to find the right resource bounds for the flow where the packet belongs. In systems using only one processor for both the processing of signaling messages and for data forwarding, this cross influence is expected to be important. Issues: - Smart classification, policing, shaping routines Feher, Cselenyi, Nemeth Expires January 2001 [Page 11] INTERNET-DRAFT July 2000 - Core and border routers Measurement unit: seconds See Also: Best-effort traffic extra delay 6.4.3 Best-effort Traffic Extra Delay Definition: Best-effort traffic extra delay is the delay that a non- prioritized data packet transfer because the resource reservation protocol is running on the router. Discussion: It is obvious that classification or policing algorithms do not address the best-effort traffic. However the cross-effect between the control and data plane influences the traffic and adds an extra delay to the forward procedure of each packet. Measurement unit: seconds See Also: Premium traffic extra delay 6.4.4 Signaling Message Loss Definition: Signaling message loss is the ratio of the signaling messages that has actually left the router and signaling messages that are expected to leave the router as a response to the corresponding signaling message entering to the router. Discussion: As in case of signaling message handling time there is a problem here as well with the signaling messages that drained or generated inside the router. Those signaling messages are immeasurable and therefore the definition does not apply to them. Signaling loss considers signaling messages only that leave the router as a consequence of processing an entering signaling message. Note, that signaling messages in a multicast reservation session might trigger several signaling messages. Losing a signaling message is almost always an indication of saturation of the router. This measure is therefore suitable for sounding out the scalability limits of the router. Issues: - Multicast signaling - Refresh message aggregation Feher, Cselenyi, Nemeth Expires January 2001 [Page 12] INTERNET-DRAFT July 2000 Measurement unit: percentage value, but in many cases the existence of signaling loss is enough 7. Methodology 7.1 Evaluating the Results RFC2544 [4] describes considerations regarding the implementation and evaluation of tests, which are certainly valid for this test suite too. Namely, we shall emphasize that we intended to create a system from commercially available measurement instruments and devices for the sake of easy implementation of the described tests. Simple test scripts and test software for Linux are available from the Boomerang homepage [10]. Secondly, care should be taken for selecting the proper tests for a specific router, since not all of the tests apply to all types of Devices Under Test (DUTs). Finally, the selection of the relevant measurement data and their evaluation requires experience and it must be done with an understanding of generally accepted testing practices regarding repeatability, variance and statistical significance of small numbers of trials. 7.2 Test Set up 7.2.1 One Tester The ideal way to perform the measurements is to connect a tester device to both the incoming and outgoing network interfaces of the DUT. The tester sends signaling messages and data traffic on its outgoing port to one of the incoming ports of the reservation capable router device, while the outgoing network port of the router, where the processed signaling messages and the forwarded packets appear, is connected to an incoming port of the tester device. Thus the tester device is capable to measure the signaling message handling time, the various traffic forwarding times and the signaling loss. This scenario can be seen in Figure 1 [4]. In this case the tester is both the signaling and data traffic source and the destination device in one. +------------+ | | +------------| tester |<-------------+ | | | | | +------------+ | | | | +------------+ | | | | | +----------->| DUT |--------------+ Feher, Cselenyi, Nemeth Expires January 2001 [Page 13] INTERNET-DRAFT July 2000 | | +------------+ Figure 1 7.2.2 Two Testers The measurements can be performed with two tester devices as well, separating the sender and receiver functionality into two pieces of equipment. In this case the sender test device is the driver of the input network interface of the DUT and the second one, the receiver test device, is connected to the output network interface of the DUT and collects the messages and traffic packets leaving from the DUT. This scenario can be seen in Figure 2. Using this test scenario the synchronization of the clocks in the tester devices must be assured for certain tests. Nevertheless, the scalability tests do not depend on time synchronization. +--------+ +------------+ +----------+ | | | | | | | sender |-------->| DUT |--------->| receiver | | | | | | | +--------+ +------------+ +----------+ Figure 2 The main benefit of using the single tester device scenario described above is that the time measurement happens within a single device and does not require any time synchronization. Using one tester device is recommended during all of the tests. The description of the benchmarking methodology uses the expressions of sender and receiver devices but they do not have to be two physically separate appliances. When the DUT runs a multicast aware resource reservation protocol then the test must be performed in the multicast resource reservation test scenario as well as in the standard unicast scenario. 7.2.3 Test Set up for Unicast Resource Reservation The test set up in the unicast test scenario is the same as described generally in the test set up section. During the test the tester device must use unicast addresses for the data traffic and must send resource reservation requests for host-to-host (i.e. unicast) reservation. 7.2.4 Test Set up for Multicast Resource Reservation The test set up in the multicast test scenario is similar to the one described in the beginning of this chapter (see sections 7.2.1 and 7.2.2), but all the outgoing ports (where the router emits signaling messages or data packets as a response to one signaling message or data packet on the incoming port) must be connected to the tester device or tester devices. Furthermore the data traffic from the Feher, Cselenyi, Nemeth Expires January 2001 [Page 14] INTERNET-DRAFT July 2000 tester must be sent to a multicast address and the tester device must ask resource reservation requests for multiple hosts. There are resource reservation protocols, like RSVP, which has more than one reservation scheme for multicast flows. In this case all reservation schemes must be tested. There are many of ways to possibly define a multicast reservation session. The possibly large number of end-points and the distribution of traffic sources and traffic destinations within the group results in a very large number of combinations. The benchmarking procedure does not require measuring all possible multicast sessions, just one scenario is required, while others are still recommended. The proposed multicast scenario consists of a multicast group with 4 end- points where there are one traffic originator and three traffic drainers. The traffic originator should be placed to the sender side of the DUT, while the drainers should be placed to the receiver side. The benchmarking reports carried on multicast scenarios always have to include the full description of the scenario itself. 7.2.5 Signaling Message Verification However the conformance testing of the resource reservation is beyond the scope of this document, defective signaling message processing can be expected in an overloaded router. Thus, during the tests, when signaling messages are processed in the DUT, the receiver device must verify the messages whether they fully conform to the message format of the protocol specification and they are the expected signaling messages at the given situation. When any of the messages do not conform to the protocol specification the report must indicate implementation errors. However, further analysis of the conformance of a malfunctioning protocol implementation is beyond the scope of this test suit that targets performance analysis. Verifying data traffic packets are not required, since the signaling performance benchmarking of reservation capable routers should not deal with data traffic. For this purpose there are other benchmarking methodologies that verify data traffic during the measurements, like the one described in RFC 2544 [4]. 7.3 Scalability Tests Unicast scalability tests are defined to explore the scalability limits of a reservation capable router when it deals with unicast resource reservation requests. According to our definition, a router reaches its scalability limit, when it steps out of the steady state operation and steps into a phase where it cannot process all of the signaling messages and begins to drop some of them. Multicast scalability tests are applied for multicast-aware resource reservation protocols only. They are similar to unicast scalability tests, but all the resource reservations should ask resources for Feher, Cselenyi, Nemeth Expires January 2001 [Page 15] INTERNET-DRAFT July 2000 multicast reservation sessions and the multicast test scenario must be used for the measurements. 7.3.1 Maximum Signaling Message Burst Size -- Unicast Case Objective: To determine the maximum number of the signaling messages in a burst that the router is still able to handle without signaling loss. Procedure: 1. Select one signaling message or a signaling message sequence from the specification of the resource reservation protocol. 2. A number of signaling messages or signaling message sequences must be sent back-to-back in order to form a burst. All the signaling messages in the burst must be valid messages and the sender must check if the router is able to process them without any error. The signaling messages must be addressed in a way that they go through the DUT and the receiver should be able to catch them all. 3. Send the burst towards the DUT and count the signaling messages received by the receiver. 4. When the number of sent messages equals to the number of received messages, the number of messages in a burst must be increased and the test is to be repeated. When the receiver receives less signaling messages than the sender has sent, the DUT is over its scalability limit. The scalability limit for the maximum signaling message burst size is found when the sender is able to send a signaling burst without signaling loss but one more signaling message in the burst would result signaling loss. During the test the DUT must not receive other data packets than the signaling messages that the sender sends. The test should be repeated at least 30 times for each signaling message or signaling message sequence. The DUT should be reset to its initial state before each step of the test. Reporting format: The report should indicate the type of the signaling message or signaling message sequence and the median of the measured maximum signaling burst size. Note: There are resource reservation protocols, like RSVP, where certain kind of signaling messages (e.g. RESV) are valid only when some other signaling messages (e.g. PATH) have already established corresponding states in the router. In this situation the sender must ensure the establishment of such states in the router before the test begins. 7.3.2 Maximum Signaling Message Burst Size -- Multicast Case Objective: Feher, Cselenyi, Nemeth Expires January 2001 [Page 16] INTERNET-DRAFT July 2000 To determine the maximum number of the signaling messages in a burst that the router is still able to handle without signaling loss. Procedure: The procedure is the same as it is defined for the unicast case, except all the signaling messages should refer to multicast sessions. The test stops when there is a signaling loss on any of the outgoing interfaces of the DUT. Reporting format: The reporting format is the same as in the unicast case. 7.3.3 Maximum Signaling Intensity -- Unicast Case Objective: To determine the maximum number of signaling messages that can be sent to the router within a time unit. Procedure: 1. Select one signaling message or a signaling message sequence from the specification of resource reservation protocol. 2. Construct a flow of valid signaling messages or signaling message sequences where the individual signaling messages are equally spaced and there are a specified number of messages in one second according to the signaling intensity. The signaling messages must be addressed in a way that they must go through the DUT and the receiver should be able to catch them all. 3. Send the flow towards the DUT for at least one minute. Count the signaling messages received by the receiver. 4. When the number of sent messages equals to the number of received messages then the signaling intensity of the flow must be increased and the test must be repeated. When the receiver receives less signaling messages than the sender has sent, the DUT is over its scalability limit. The scalability limit for the maximum signaling intensity is found, when the signaling flow arrives to the receiver without any signaling loss but the DUT would lose signaling messages, if the signaling intensity would be larger. During the test the DUT must not forward other data packets than the signaling messages that the sender sends. The test should be repeated at least 30 times for each signaling message or signaling message sequence. The DUT should be reset to its initial state before each step of the test. Reporting format: The report should indicate the type of signaling message or signaling message sequence and the median of the measured maximum signaling intensity. Note: Feher, Cselenyi, Nemeth Expires January 2001 [Page 17] INTERNET-DRAFT July 2000 There are resource reservation protocols, like RSVP, where certain kind of signaling messages are valid only when some other signaling messages have already established corresponding states in the router. In this situation the sender must ensure the establishment of such states in the router before the test begins. 7.3.4 Maximum Signaling Intensity -- Multicast Case Objective: To determine the maximum number of signaling messages that can be sent to the router within a time unit. Procedure: The procedure is the same as it is defined for the unicast case, except that each signaling messages should refer to multicast sessions. The test stops when there is a signaling loss on any of the outgoing interfaces of the DUT. Reporting format: The reporting format is the same as in the unicast case. 7.3.5 Maximum Number of Reservation Sessions -- Unicast Case Objective: To determine the maximum number of reservation sessions that can exist simultaneously in a reservation capable router. Procedure: 1. Set up a reservation session in the reservation capable router by sending the appropriate signaling message sequence over the DUT. 2. After a successful reservation setup, wait for a specified amount of time (T) while still maintaining the established reservations. Care should be taken of the maintenance of established reservation sessions. If the DUT uses soft state resource reservation protocol, then the waiting time, T must be at least as long as the protocol specification requires for reservation time out. This waiting is necessary to assure that DUT is able to refresh the reservations. In case of hard state resource reservation protocols it is not necessary to wait thus T can be zero. . 3. Repeat the previous steps until the router is not able to set up any new reservation or it is not able to maintain the existing reservations. That is, if you have set up N reservations and the DUT starts dropping reservations during the waiting time or after it, then the maximum number of reserved sessions is passed over and the actual number of maximum reservation sessions is N-1. The test should be repeated at least 5 times and the DUT should be reset between the tests cycles. Reporting format: Feher, Cselenyi, Nemeth Expires January 2001 [Page 18] INTERNET-DRAFT July 2000 The report must indicate the median of the measured maximum number of reservation sessions. When the number of reserved sessions grow over a number that is surely enough in the given technology conditions, then the test can be canceled and the report can state that the resource reservation protocol implementation performs the maximum number of reservation sessions over that limit. (e.g. "Over 10.000 sessions") Checking the reservation sessions in a reservation capable router might be difficult if the router does not support any interface to monitor its interior states. Lack of such support the signaling-end point might try to send overrated data traffic across all the reservation sessions and dropping the right amount of data traffic means that the reservation sessions are alive. Of course other smart methods can be used also. 7.3.6 Maximum Number of Reservation Sessions -- Multicast Case Objective: To determine the maximum number of reservation sessions that can exist simultaneously in a reservation capable router. Procedure: The procedure is the same as it is defined for the unicast case, except all the signaling messages should refer to multicast reservation sessions. Reporting format: The reporting format is the same as in the unicast case. 7.4 Benchmarking Tests Benchmarking tests are defined to measure the QoS signaling related performance metrics on the router device. During the tests the DUT must stay in steady state. This means that the router must not drop any signaling message or data packet, i.e. it should operate below its scalability limits. In case of signaling message or data traffic loss, the test must be stopped, and the parameters of the test must be adjusted to prevent the DUT to leave its steady state operating range. During all of the benchmarking tests the sender tester loads the DUT by sending a specified amount of signaling and data traffic to the receiver device across the DUT. Moreover, the signaling end-points must also assure that the DUT maintains a certain number of reservation sessions. All of the performance metrics are measured under different load conditions, where the load is a combination of: a. Signaling load b. Premium traffic load c. Best-effort traffic load d. Reservation load Feher, Cselenyi, Nemeth Expires January 2001 [Page 19] INTERNET-DRAFT July 2000 Signaling load must be generated by sending signaling messages periodically to the DUT. This way the signaling load is measured by the signaling intensity and expressed with the same signaling measurement unit as it is in case of signaling intensity. The signaling messages sent by the sender device should be equally distributed on the time scale. Most of the resource reservation protocols have more than one signaling message type and the benchmarking is complete when the test has been carried over for each signaling message type. Premium traffic load must be generated by sending a data flow from the sender to the receiver across the DUT where the DUT must have a valid reservation for that flow. The traffic must consist of equally spaced and equally sized data packets. The premium traffic must be reported by its traffic parameters: data packet size in octets and the calculated bandwidth of the stream in kbps unit. The data packet size should include both the payload and the header of the IP packet. Signaling messages maintaining the flow do not belong to the data traffic flow and must be ignored when calculating the data traffic parameters. The best-effort traffic load must be generated by sending a data flow from the sender tester to the receiver tester across the DUT where the DUT must not have any reservation for that flow. The traffic must consist of equally spaced and equally sized data packets. The best- effort traffic must be reported by traffic parameters as it is described in case of the premium traffic. The reservation load must be generated by reserving resources for traffic flows via signaling in the DUT. During the test the sender device must maintain the reservation sessions with signaling messages if the resource reservation protocol requires it. The reservation sessions should not need to be loaded with data traffic. The reservations must differ at least in one of the IP addresses or port number of the endpoints in the reservation entry. The four load types have influence on each other from their nature, but during the tests these cross-effects must be minimized. The signaling load must establish as few temporary resource reservations in the DUT as possible. When a new resource reservation is set up in the DUT the signaling end-points must arrange to restore the number of reservations in the router as soon as possible. Signaling messages are datagrams in the real word, but during the measurements they are not treated as premium or best-effort traffic. 7.4.1 Signaling Message Handling Time -- Unicast Case Objective: To determine how fast the resource reservation protocol responds to one type of signaling message or signaling message sequence in various load conditions. Feher, Cselenyi, Nemeth Expires January 2001 [Page 20] INTERNET-DRAFT July 2000 Procedure: 1. Select one signaling message or a signaling message sequence from the resource reservation protocol specification. 2. Load the DUT with a certain kind of load except signaling load. The DUT should preserve its steady state operation during the load. 3. Form a signaling message flow from the selected signaling message type or signaling message sequence. The signaling messages should be distributed equally in time, this way the signaling flow has a constant signaling intensity. 4. Send the signaling flow to the receiver across the DUT and measure the time intervals that signaling messages spend inside the router as the difference of absolute time (e.g. time stamp) when a signaling message enters to the DUT and the time when the corresponding signaling message leaves it. The signaling flow must be up for at least one minute and the measurement should begin 30 seconds after the first signaling message is processed in the DUT and should last at least 30 seconds. The result of the measurement is the average of the individually measured signaling message handling times grouped by signaling message types. 5. Repeat the test 30 times for each desired signaling intensity value for the signaling flow. Always reset the DUT between two test cycles. 6. Alter the signaling intensity and the load conditions and repeat the whole test. The signaling message handling time must be measured in function of many different load conditions. Measuring all possible load condition however is almost impossible so it is not required, instead the parameters of the load generation is given in a range and a step. The data traffic parameters have to be selected from common traffic parameters. It is recommended to choose a packet size of: 54, 64, 128, 256, 1024, 1518, 2048 and 4472. The same values used in RFC 2544 that gives methodology for benchmarking network interconnect devices. The packet rate is recommended to be 1, 10, 100 and 1000 kbit/s. At least 2 different data traffic parameter set should be chosen both for the premium traffic and the best-effort traffic. It is recommended to use UDP packets constructing data traffic but any other transfer protocol can be used. The number of reserved sessions in the DUT should be in the range of 1 and the maximum number of reservation sessions for the tested resource reservation protocol implementation. At least 3 different values must be chosen. The signaling intensity of the signaling message flow should be in the range of 1 and the maximum signaling intensity for the tested resource reservation protocol implementation. At least 3 different signaling message intensity values must be chosen. Feher, Cselenyi, Nemeth Expires January 2001 [Page 21] INTERNET-DRAFT July 2000 Additionally, the signaling message handling time must be measured in unloaded conditions as well. In this case there is no data traffic on the DUT, no reserved resources and the signaling intensity should take the 1 signaling message per second value. Reporting format: The results should be reported in a table. The columns of the table are the values of the four load components and the fifth column is the measured signaling message handling time in seconds. The table entries for the traffic parameters must include the packet size, the packet rate and the transfer protocol type. There should be one table for each signaling message type or signaling message sequence. Note: The number of the tests is the product of the number of different load parameters. The minimum is 37 which includes: 2 parameter sets for the premium traffic, 2 parameter sets for best- effort traffic, 3 different reservation session number and 3 different signaling intensity values for the signaling sequence plus one measurement where the signaling load is the only load on the DUT. 7.4.2 Signaling Message Handling Time -- Multicast Case Objective: To determine how fast the resource reservation protocol responds to one type of signaling messages or signaling message sequence in various load conditions. Procedure: The procedure is similar to the unicast case but the signaling messages should refer to multicast sessions, the reserved sessions in the DUT must be entries for multicast reservation sessions and the premium traffic must be addressed to a multicast group. Reporting format: The reporting format is the same as in the unicast case. 7.4.3 Premium Traffic Extra Delay -- Unicast Case Objective: To determine how the resource reservation protocol affects the premium traffic forwarding performance of the router when it handles resource reservations for unicast flows. Procedure: 1. Load the DUT with a certain kind of load except premium traffic load. The DUT should preserve its steady state operation during the load. 2. Construct a premium traffic flow from data packets of the same size. The flow should have a constant bitrate. Feher, Cselenyi, Nemeth Expires January 2001 [Page 22] INTERNET-DRAFT July 2000 3. Send the premium flow to the receiver across the DUT with different traffic parameters and measure the time intervals that data packets spend inside the router as they are being forwarded. The signaling must set up a reservation session for this flow in the DUT before the test starts and it should not be torn down during the test. The traffic flow must be active for at least one minute. The measurement should begin 30 seconds after the first data packet belonging to the measured flow is forwarded in the DUT, and should last at least 30 seconds. It is not necessary to measure all premium packets, but at least 20 samples are required in a second and the samples should be distributed equally in time scale. The result of the measurement is the average of the individually measured packet forwarding time values. 4. Repeat the test 30 times for each desired parameter set of the premium traffic. Always reset the DUT between two test cycles. 5. Alter the parameters of the premium traffic and the load conditions and repeat the whole test. The premium traffic extra delay must be measured in function of many different load conditions. Choose the load parameters according to the test description of the signaling message handling time, but here at least 3 traffic parameter sets are required to be tested in case of the premium flow. Additionally, the premium traffic extra delay must be measured in unloaded conditions as well. In this case there is neither best- effort traffic and nor signaling load on the DUT and there is only one resource reservation entry, which one reserves the resources for the premium traffic flow used in the test. Reporting format: The results should be reported in a table. The columns of the table are the values of the four load components, the fifth column is the premium traffic delay in seconds and the sixth column is the difference of the premium traffic delay with the given load conditions and the premium traffic delay measured in the unloaded router. The table entries for the traffic parameters must include the packet size, the packet rate and the transfer protocol type. The column of the signaling load must contain the signaling message type or the signaling message sequence along with the signaling intensity. Note: The number of the tests is at least as large as in case of the signaling message handling time test. Using the same parameters as in case of the signaling message handling time test is highly recommended. 7.4.4 Premium Traffic Extra Delay -- Multicast Case Objective: Feher, Cselenyi, Nemeth Expires January 2001 [Page 23] INTERNET-DRAFT July 2000 To determine how the resource reservation protocol affects the premium traffic forwarding performance of the router when it handles resource reservations for multicast flows. Procedure: The procedure is similar to the unicast case but the signaling messages should refer to multicast sessions, the reservation sessions in the DUT must be multicast reservations sessions and the premium traffic must be addressed to a multicast group. Reporting format: The reporting format is the same as in the unicast case. 7.4.5 Best-effort Traffic Extra Delay Objective: To determine how the resource reservation protocol affects the best- effort traffic forwarding performance of the router when it handles resource reservations for unicast flows. Procedure: The procedure is almost the same as in case of the premium traffic extra delay test, but the best-effort traffic must be measured instead of the premium traffic and it should be measured for at least 3 different traffic parameter sets. Of course the best-effort traffic does not require any reservation session entry. Reporting format: The reporting format is the same as in case of the premium traffic delay test, except that the fifth column is for the best-effort traffic instead of the premium traffic. Note: Measuring the best-effort traffic delay for multicast flows is not necessary as the vast amount of the best-effort traffic is unicast and considered to remain unicast in the future as multicast flows usually requires QoS services. 8. Acknowledgement We would like to thank the following individuals for their help in designing and implementing the benchmarking framework presented in this document: Joakim Bergkvist and Norbert Vegh from Telia Research AB, Sweden, Gabor Kovacs, Anrdas Korn, Balazs Szabo from High Speed Netwroks Laboratory of BUTE. 9. References [1] S. Bradner, "Benchmarking Terminology for Network Interconnection Devices", RFC 1242, July 1991 [2] R. Mandeville, "Benchmarking Terminology for LAN Switching Devices", RFC 2285, February 1998 Feher, Cselenyi, Nemeth Expires January 2001 [Page 24] INTERNET-DRAFT July 2000 [3] Y. Bernet, et. al., "A Framework For Integrated Services Operation Over Diffserv Networks", Internet Draft, May 2000, [4] S. Bradner, J. McQuaid, "Benchmarking Methodology for Network Interconnect Devices", RFC 2544, March 1999 [5] B. Braden, Ed., et. al., "Resource Reservation Protocol (RSVP) - Version 1 Functional Specification", RFC 2205, September 1997. [6] J. Bergkvist, I. Cselenyi, "Boomerang Protocol Specification", Internet Draft, June 1999, [7] P. Pan, H. Schulzrinne, "YESSIR: A Simple Reservation Mechanism for the Internet", Computer Communication Review, on-line version, volume 29, number 2, April 1999 [8] L. Delgrossi, L. Berger, "Internet Stream Protocol Version 2 (ST2) Protocol Specification - Version ST2+", RFC 1819, August 1995 [9] D. Black, S. Blake, M. Carlson, E. Davies, Z. Wang and W. Weiss, "An Architecture for Differentiated Services", RFC 2475, December 1998 [10] Boomerang Team, "Boomerang homepage - Benchmarking Tools", http://boomerang.ttt.bme.hu 10. Authors' Addresses: Gabor Feher Budapest University of Technology and Economics (BUTE) Department of Telecommunications and Telematics Pazmany Peter Setany 1/D, H-1117, Budapest, Phone: +36 1 463-3110 Email: feher@ttt-atm.ttt.bme.hu Istvan Cselenyi Telia Research AB Vitsandsgatan 9B SE 12386, Farsta SWEDEN, Phone: +46 8 713-8173 Email: istvan.i.cselenyi@telia.se Krisztian Nemeth Budapest University of Technology and Economics (BUTE) Department of Telecommunications and Telematics Pazmany Peter Setany 1/D, H-1117, Budapest, Phone: +36 1 463-2225 Email: nemeth_k@ttt-atm.ttt.bme.hu Feher, Cselenyi, Nemeth Expires January 2001 [Page 25]