Network Working Group Z. Sarker
Internet-Draft Ericsson AB
Intended status: Informational V. Singh
Expires: December 23, 2018 callstats.io
X. Zhu
M. Ramalho
Cisco Systems
June 21, 2018

Test Cases for Evaluating RMCAT Proposals
draft-ietf-rmcat-eval-test-06

Abstract

The Real-time Transport Protocol (RTP) is used to transmit media in multimedia telephony applications, these applications are typically required to implement congestion control. This document describes the test cases to be used in the performance evaluation of such congestion control algorithms.

Status of This Memo

This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.

Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at https://datatracker.ietf.org/drafts/current/.

Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."

This Internet-Draft will expire on December 23, 2018.

Copyright Notice

Copyright (c) 2018 IETF Trust and the persons identified as the document authors. All rights reserved.

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License.

Table of Contents

1. Introduction

This memo describes a set of test cases for evaluating congestion control algorithm proposals for real-time interactive media. It is based on the guidelines enumerated in [I-D.ietf-rmcat-eval-criteria] and the requirements discussed in [I-D.ietf-rmcat-cc-requirements]. The test cases cover basic usage scenarios and are described using a common structure, which allows for additional test cases to be added to those described herein to accommodate other topologies and/or the modeling of different path characteristics. The described test cases in this memo SHOULD be used to evaluate any proposed congestion control algorithm for real-time interactive media.

2. Terminology

The terminology defined in RTP, RTP Profile for Audio and Video Conferences with Minimal Control, RTCP Extended Report (XR), Extended RTP Profile for RTCP-based Feedback (RTP/AVPF), and Support for Reduced-Size RTCP apply.

3. Structure of Test cases

All the test cases in this document follow a basic structure allowing implementers to describe a new test scenario without repeatedly explaining common attributes. The structure includes a general description section that describes the test case and its motivation. Additionally the test case defines a set of attributes that characterize the testbed, for example, the network path between communicating peers and the diverse traffic sources.

+---+                                                           +---+
|S1 |====== \                 Forward -->              / =======|R1 |
+---+       \\                                        //        +---+
             \\                                      //
+---+       +-----+                               +-----+       +---+
|S2 |=======|  A  |------------------------------>|  B  |=======|R2 |
+---+       |     |<------------------------------|     |       +---+
            +-----+                               +-----+
(...)         //                                     \\         (...)
             //          <-- Backward                 \\
+---+       //                                         \\       +---+
|Sn |====== /                                           \ ======|Rn |
+---+                                                           +---+

Figure 1: Example of A Testbed Topology

Any attribute can have a set of values (enclosed within "[]"). Each member value of such a set MUST be treated as different value for the same attribute. It is desired to run separate tests for each such attribute value.

The test cases described in this document follow the above structure.

4. Recommended Evaluation Settings

This section describes recommended test case settings and could be overwritten by the respective test cases.

4.1. Evaluation metrics

To evaluate the performance of the candidate algorithms the implementers MUST log enough information to visualize the following metrics at a fine enough time granularity:

  1. Flow level:
    1. End-to-end delay for the congestion controlled media flow.
    2. Variation in sending bit rate and goodput. Mainly observing the frequency and magnitude of oscillations.
    3. Packet losses observed at the receiving endpoint.
    4. Feedback message overhead.
    5. Convergence time - time to reach steady state for the congestion controlled media flow(s).
  2. Transport level:
    1. Bandwidth utilization.
    2. Queue length (milliseconds at specified path capacity):
      • average over the length of the session.
      • 5 and 95 percentile.
      • median, maximum, minimum.

4.2. Path characteristics

Each path between a sender and receiver as described in Figure 1 have the following characteristics unless otherwise specified in the test case.

Examples of additional network parameters are discussed in [I-D.ietf-rmcat-eval-criteria].

For test cases involving time-varying bottleneck capacity, all capacity values are specified as a ratio with respect to a reference capacity value, so as to allow flexible scaling of capacity values along with media source rate range. There exist two different mechanisms for inducing path capacity variation: a) by explicitly modifying the value of physical link capacity; or b) by introducing background non-adaptive UDP traffic with time-varying traffic rate. Implementers are encouraged to run the experiments with both mechanisms for test cases specified in Section 5.1, Section 5.2, and Section 5.3.

4.3. Media source

Unless otherwise specified, each test case will include one or more media sources as described below.

5. Basic Test Cases

5.1. Variable Available Capacity with a Single Flow

In this test case the bottleneck-link capacity between the two endpoints varies over time. This test is designed to measure the responsiveness of the candidate algorithm. This test tries to address the requirements in [I-D.ietf-rmcat-cc-requirements], which requires the algorithm to adapt the flow(s) and provide lower end-to-end latency when there exists:

It should be noted that the exact variation in available capacity due to any of the above depends on the underlying technologies. Hence, we describe a set of known factors, which may be extended to devise a more specific test case targeting certain behaviors in a certain network environment.

Expected behavior: the candidate algorithm is expected to detect the path capacity constraint, converges to the bottleneck link's capacity and adapt the flow to avoid unwanted oscillation when the sending bit rate is approaching the bottleneck link's capacity. The oscillations occur when the media flow(s) attempts to reach its maximum bit rate but overshoots the usage of the available bottleneck capacity then to rectify, it reduces the bit rate and starts to ramp up again.

Evaluation metrics : as described in Section 4.1.


                             Forward -->
+---+       +-----+                               +-----+       +---+
|S1 |=======|  A  |------------------------------>|  B  |=======|R1 |
+---+       |     |<------------------------------|     |       +---+
            +-----+                               +-----+
                          <-- Backward

Figure 2: Testbed Topology for Limited Link Capacity

Testbed topology: One media source S1 is connected to the corresponding R1. The media traffic is transported over the forward path and corresponding feedback/control traffic is transported over the backward path.

Testbed attributes:

Path capacity variation pattern for forward direction
Variation pattern index Path direction Start time Path capacity ratio
One Forward 0s 1.0
Two Forward 40s 2.5
Three Forward 60s 0.6
Four Forward 80s 1.0

5.2. Variable Available Capacity with Multiple Flows

This test case is similar to Section 5.1. However in addition this test will also consider persistent network load due to competing traffic.

Expected behavior: the candidate algorithm is expected to detect the variation in available capacity and adapt the media stream(s) accordingly. The flows stabilize around their maximum bit rate as the maximum link capacity is large enough to accommodate the flows. When the available capacity drops, the flows adapt by decreasing their sending bit rate, and when congestion disappears, the flows are again expected to ramp up.

Evaluation metrics : as described in Section 4.1.

+---+                                                         +---+
|S1 |===== \                                         / =======|R1 |
+---+      \\             Forward -->               //        +---+
            \\                                     //
            +-----+                               +-----+
            |  A  |------------------------------>|  B  |
            |     |<------------------------------|     |
            +-----+                               +-----+
              //                                    \\
             //          <-- Backward                \\
+---+       //                                        \\       +---+
|S2 |====== /                                          \ ======|R2 |
+---+                                                          +---+

Figure 3: Testbed Topology for Variable Available Capacity

Testbed Topology: Two (2) media sources S1 and S2 are connected to their corresponding destinations R1 and R2. The media traffic is transported over the forward path and corresponding feedback/control traffic is transported over the backward path.

Testbed attributes:

Testbed attributes are similar as described in Section 5.1 except the test specific capacity variation setup.

Test Specific Information: This test uses path capacity variation as listed in Table 2 with a corresponding end time of 125 seconds. The reference bottleneck capacity is 2Mbps. When using background non-adaptive UDP traffic to induce time-varying bottleneck for congestion controlled media flows, the physical path capacity is 4Mbps and the UDP traffic source rate changes over time as (4-x)Mbps, where x is the bottleneck capacity specified in Table 2.

Path capacity variation pattern for forward direction
Variation pattern index Path direction Start time Path capacity ratio
One Forward 0s 2.0
Two Forward 25s 1.0
Three Forward 50s 1.75
Four Forward 75s 0.5
Five Forward 100s 1.0

5.3. Congested Feedback Link with Bi-directional Media Flows

Real-time interactive media uses RTP hence it is assumed that RTCP, RTP header extension or such would be used by the congestion control algorithm in the backchannel. Due to asymmetric nature of the link between communicating peers it is possible for a participating peer to not receive such feedback information due to an impaired or congested backchannel (even when the forward channel might not be impaired). This test case is designed to observe the candidate congestion control behavior in such an event.

It is expected that the candidate algorithms are able to cope with the lack of feedback information and adapt to minimize the performance degradation of media flows in the forward channel.

It should be noted that for this test case: logs are compared with the reference case, i.e, when the backward channel has no impairments.

Evaluation metrics : as described in Section 4.1.

Testbed topology: One (1) media source S1 is connected to corresponding R1, but both endpoints are additionally receiving and sending data, respectively. The media traffic (S1->R1) is transported over the forward path and corresponding feedback/control traffic is transported over the backward path. Likewise media traffic (S2->R2) is transported over the backward path and corresponding feedback/control traffic is transported over the forward path.

	 +---+                                                         +---+
	 |S1 |===== \                Forward -->              / =======|R1 |
	 +---+      \\                                       //        +---+
	             \\                                     //
       		  +-----+                               +-----+
	          |  A  |------------------------------>|  B  |
	          |     |<------------------------------|     |
	          +-----+                               +-----+
	             //                                     \\
	            //            <-- Backward               \\
	+---+      //                                         \\       +---+
	|R2 |===== /                                           \ ======|S2 |
	+---+                                                          +---+

Figure 4: Testbed Topology for Congested Feedback Link

Testbed attributes:

Path capacity variation pattern for forward direction
Variation pattern index Path direction Start time Path capacity ratio
One Forward 0s 2.0
Two Forward 20s 1.0
Three Forward 40s 0.5
Four Forward 60s 2.0

Path capacity variation pattern for backward direction
Variation pattern index Path direction Start time Path capacity ratio
One Backward 0s 2.0
Two Backward 35s 0.8
Three Backward 70s 2.0

5.4. Competing Media Flows with same Congestion Control Algorithm

In this test case, more than one media flows share the bottleneck link and each of them uses the same congestion control algorithm. This is a typical scenario where a real-time interactive application sends more than one media flow to the same destination and these flows are multiplexed over the same port. In such a scenario it is likely that the flows will be routed via the same path and need to share the available bandwidth amongst themselves. For the sake of simplicity it is assumed that there are no other competing traffic sources in the bottleneck link and that there is sufficient capacity to accommodate all the flows individually. While this appears to be a variant of the test case defined in Section 5.2, it focuses on the capacity sharing aspect of the candidate algorithm. The previous test case, on the other hand, measures adaptability, stability, and responsiveness of the candidate algorithm.

Expected behavior: It is expected that the competing flows will converge to an optimum bit rate to accommodate all the flows with minimum possible latency and loss. Specifically, the test introduces three media flows at different time instances, when the second flow appears there should still be room to accommodate another flow on the bottleneck link. Lastly, when the third flow appears the bottleneck link should be saturated.

Evaluation metrics : as described in Section 4.1.

+---+                                                         +---+
|S1 |===== \                Forward -->              / =======|R1 |
+---+      \\                                       //        +---+
            \\                                     //
+---+       +-----+                               +-----+       +---+
|S2 |=======|  A  |------------------------------>|  B  |=======|R2 |
+---+       |     |<------------------------------|     |       +---+
            +-----+                               +-----+
              //                                     \\
             //          <-- Backward                 \\
+---+       //                                         \\       +---+
|S3 |====== /                                           \ ======|R3 |
+---+                                                           +---+

Figure 5: Testbed Topology for Multiple congestion controlled media Flows

Testbed topology: Three media sources S1, S2, S3 are connected to R1, R2, R3 respectively. The media traffic is transported over the forward path and corresponding feedback/control traffic is transported over the backward path.

Testbed attributes:

Media Timeline for Video and Audio media sources
Flow ID Media type Start time End time
1 Video 0s 119s
2 Video 20s 119s
3 Video 40s 119s
4 Audio 0s 119s
5 Audio 20s 119s
6 Audio 40s 119s

5.5. Round Trip Time Fairness

In this test case, multiple media flows share the bottleneck link, but the end-to-end path latency for each flow is different. For the sake of simplicity it is assumed that there are no other competing traffic sources in the bottleneck link and that there is sufficient capacity to accommodate all the flows. While this appears to be a variant of test case 5.2, it focuses on the capacity sharing aspect of the candidate algorithm under different RTTs.

It is expected that the competing flows will converge to bit rates to accommodate all the flows with minimum possible latency and loss. Specifically, the test introduces five media flows at the same time instance.

Evaluation metrics : as described in Section 4.1.

Testbed Topology: Five (5) media sources S1,S2,..,S5 are connected to their corresponding media sinks R1,R2,..,R5. The media traffic is transported over the forward path and corresponding feedback/control traffic is transported over the backward path. The topology is the same as in Section 5.4. The end-to-end path delays are: 10ms for S1-R1, 25ms for S2-R2, 50ms for S3-R3, 100ms for S4-R4, and 150ms S5-R5, respectively.

Testbed attributes:

Media Timeline for Video and Audio media sources
Flow IF Media type Start time End time
1 Video 0s 299s
2 Video 10s 299s
3 Video 20s 299s
4 Video 30s 299s
5 Video 40s 299s
6 Audio 0 299s
7 Audio 10s 299s
8 Audio 20s 299s
9 Audio 30s 299s
10 Audio 40s 299s

5.6. Media Flow Competing with a Long TCP Flow

In this test case, one or more media flows share the bottleneck link with at least one long lived TCP flow. Long lived TCP flows download data throughout the session and are expected to have infinite amount of data to send and receive. This is a scenario where a multimedia application co-exists with a large file download. The test case measures the adaptivity of the candidate algorithm to competing traffic. It addresses the requirement 3 in [I-D.ietf-rmcat-cc-requirements].

Expected behavior: depending on the convergence observed in test case 5.1 and 5.2, the candidate algorithm may be able to avoid congestion collapse. In the worst case, the media stream will fall to the minimum media bit rate.

Evaluation metrics : following metrics in addition to as described in Section 4.1.

  1. Flow level:
    1. TCP throughput.
    2. Loss for the TCP flow

Testbed topology: One (1) media source S1 is connected to the corresponding media sink, R1. In addition, there is a long-live TCP flow sharing the same bottleneck link. The media traffic is transported over the forward path and corresponding feedback/control traffic is transported over the backward path. The TCP traffic goes over the forward path from, S_tcp with acknowledgment packets go over the backward path from, R_tcp.

	 +--+                                                       +--+
	 |S1|===== \              Forward -->              / =======|R1|
	 +--+      \\                                     //        +--+
	            \\                                   //
            +-----+                             +-----+
            |  A  |---------------------------->|  B  |
            |     |<----------------------------|     |
            +-----+                             +-----+
	             //                                   \\
	            //        <-- Backward                 \\
	+-----+    //                                       \\    +-----+
	|S_tcp|=== /                                         \ ===|R_tcp|
	+-----+                                                   +-----+

Figure 6: Testbed Topology for TCP vs congestion controlled media Flows

Testbed attributes:

5.7. Media Flow Competing with Short TCP Flows

In this test case, one or more congestion controlled media flow shares the bottleneck link with multiple short-lived TCP flows. Short-lived TCP flows resemble the on/off pattern observed in the web traffic, wherein clients (browsers) connect to a server and download a resource (typically a web page, few images, text files, etc.) using several TCP connections (up to 4). This scenario shows the performance of a multimedia application when several browser windows are active. The test case measures the adaptivity of the candidate algorithm to competing web traffic, it addresses the requirements 1.E in [I-D.ietf-rmcat-cc-requirements].

Depending on the number of short TCP flows, the cross-traffic either appears as a short burst flow or resembles a long TCP flow. The intention of this test is to observe the impact of short-term burst on the behavior of the candidate algorithm.

Evaluation metrics : following metrics in addition to as described in Section 4.1.

  1. Flow level:
    1. Variation in the sending rate of the TCP flow.
    2. TCP throughput.

Testbed topology: The topology described here is same as the one described in Figure 6.

Testbed attributes:

5.8. Media Pause and Resume

In this test case, more than one real-time interactive media flows share the link bandwidth and all flows reach to a steady state by utilizing the link capacity in an optimum way. At this stage one of the media flows is paused for a moment. This event will result in more available bandwidth for the rest of the flows as they are on a shared link. When the paused media flow resumes it would no longer have the same bandwidth share on the link. It has to make it's way through the other existing flows in the link to achieve a fair share of the link capacity. This test case is important specially for real-time interactive media which consists of more than one media flows and can pause/resume media flows at any point of time during the session. This test case directly addresses the requirement number 5 in [I-D.ietf-rmcat-cc-requirements]. One can think it as a variation of test case defined in Section 5.4. However, it is different as the candidate algorithms can use different strategies to increase its efficiency, for example in terms of fairness, convergence time, reduce oscillation etc, by capitalizing the fact that they have previous information of the link.

Evaluation metrics : following metrics in addition to as described in Section 4.1.

  1. Flow level:
    1. Variation in sending bit rate and goodput. Mainly observing the frequency and magnitude of oscillations.

Testbed Topology: Same as test case defined in Section 5.4

Testbed attributes: The general description of the testbed parameters are same as Section 5.4 with changes in the test specific setup as below-

6. Other potential test cases

It has been noticed that there are other interesting test cases besides the basic test cases listed above. In many aspects, these additional test cases can help further evaluation of the candidate algorithm. They are listed as below.

6.1. Media Flows with Priority

In this test case media flows will have different priority levels. This will be an extension of Section 5.4 where the same test will be run with different priority levels imposed on each of the media flows. For example, the first flow (S1) is assigned a priority of 2 whereas the remaining two flows (S2 and S3) are assigned a priority of 1. The candidate algorithm MUST reflect the relative priorities assigned to each media flow. In the previous example, the first flow (S1) MUST arrive at a steady-state rate approximately twice of that of the other two flows (S2 and S3).

The candidate algorithm can use a coupled congestion control mechanism for the bandwidth distribution according to the respective media flow priority.

6.2. Explicit Congestion Notification Usage

This test case requires to run all the basic test cases with the availability of Explicit Congestion Notification (ECN) [RFC6679] feature enabled. The goal of this test is to exhibit that the candidate algorithms do not fail when ECN signals are available. With ECN signals enabled the algorithms are expected to perform better than their delay based variants.

6.3. Multiple Bottlenecks

In this test case one congestion controlled media flow, S1->R2, traverses a path with multiple bottlenecks. As illustrated in Figure 7, the first flow (S1->R1) competes with the second congestion controlled media flow (S2->R2) over the link between A and B which is close to the sender side; again, that flow (S1->R1) competes with the third congestion controlled media flow (S3->R3) over the link between C and D which is close to the receiver side. The goal of this test is to ensure that the candidate algorithms work properly in the presence of multiple bottleneck links on the end to end path.

Expected behavior: the candidate algorithm is expected to achieve full utilization at both bottleneck links without starving any of the three congestion controlled media flows.



                             Forward ---->


             +---+          +---+        +---+      +---+
             |S2 |          |R2 |        |S3 |      |R3 |
             +---+          +---+        +---+      +---+
               |              |            |          |
	       |              |            |          |
+---+       +-----+       +-----+      +-----+     +-----+       +---+
|S1 |=======|  A  |------>|  B  |----->|  C  |---->|  D  |=======|R1 |
+---+       |     |<------|     |<-----|     |<----|     |       +---+
            +-----+       +-----+      +-----+     +-----+

                     1st                       2nd
              Bottleneck (A->B)          Bottleneck (C->D)

                           <------ Backward

Figure 7: Testbed Topology for Multiple Bottlenecks

Testbed topology: Three media sources S1, S2, and S3 are connected to respective destinations R1, R2, and R3. For all three flows the media traffic is transported over the forward path and corresponding feedback/control traffic is transported over the backward path.

Testbed attributes:

7. Wireless Access Links

Additional wireless network (both cellular network and WiFi network) specific test cases are defined in [I-D.ietf-rmcat-wireless-tests].

8. Security Considerations

Security issues have not been discussed in this memo.

9. IANA Considerations

There are no IANA impacts in this memo.

10. Acknowledgements

Much of this document is derived from previous work on congestion control at the IETF.

The content and concepts within this document are a product of the discussion carried out in the Design Team.

11. References

11.1. Normative References

[I-D.ietf-rmcat-eval-criteria] Singh, V., Ott, J. and S. Holmer, "Evaluating Congestion Control for Interactive Real-time Media", Internet-Draft draft-ietf-rmcat-eval-criteria-07, May 2018.
[I-D.ietf-rmcat-video-traffic-model] Zhu, X., Cruz, S. and Z. Sarker, "Modeling Video Traffic Sources for RMCAT Evaluations", Internet-Draft draft-ietf-rmcat-video-traffic-model-04, January 2018.
[I-D.ietf-rmcat-wireless-tests] Sarker, Z., Johansson, I., Zhu, X., Fu, J., Tan, W. and M. Ramalho, "Evaluation Test Cases for Interactive Real-Time Media over Wireless Networks", Internet-Draft draft-ietf-rmcat-wireless-tests-04, May 2017.
[RFC3550] Schulzrinne, H., Casner, S., Frederick, R. and V. Jacobson, "RTP: A Transport Protocol for Real-Time Applications", STD 64, RFC 3550, DOI 10.17487/RFC3550, July 2003.
[RFC3551] Schulzrinne, H. and S. Casner, "RTP Profile for Audio and Video Conferences with Minimal Control", STD 65, RFC 3551, DOI 10.17487/RFC3551, July 2003.
[RFC3611] Friedman, T., Caceres, R. and A. Clark, "RTP Control Protocol Extended Reports (RTCP XR)", RFC 3611, DOI 10.17487/RFC3611, November 2003.
[RFC4585] Ott, J., Wenger, S., Sato, N., Burmeister, C. and J. Rey, "Extended RTP Profile for Real-time Transport Control Protocol (RTCP)-Based Feedback (RTP/AVPF)", RFC 4585, DOI 10.17487/RFC4585, July 2006.
[RFC5506] Johansson, I. and M. Westerlund, "Support for Reduced-Size Real-Time Transport Control Protocol (RTCP): Opportunities and Consequences", RFC 5506, DOI 10.17487/RFC5506, April 2009.
[RFC6679] Westerlund, M., Johansson, I., Perkins, C., O'Hanlon, P. and K. Carlberg, "Explicit Congestion Notification (ECN) for RTP over UDP", RFC 6679, DOI 10.17487/RFC6679, August 2012.

11.2. Informative References

[HEVC-seq] HEVC, "Test Sequences", http://www.netlab.tkk.fi/~varun/test_sequences/
[I-D.ietf-rmcat-cc-requirements] Jesup, R. and Z. Sarker, "Congestion Control Requirements for Interactive Real-Time Media", Internet-Draft draft-ietf-rmcat-cc-requirements-09, December 2014.
[RFC5681] Allman, M., Paxson, V. and E. Blanton, "TCP Congestion Control", RFC 5681, DOI 10.17487/RFC5681, September 2009.
[xiph-seq] Xiph.org, "Video Test Media", http://media.xiph.org/video/derf/

Authors' Addresses

Zaheduzzaman Sarker Ericsson AB Luleå, SE 977 53 Sweden Phone: +46 10 717 37 43 EMail: zaheduzzaman.sarker@ericsson.com
Varun Singh Nemu Dialogue Systems Oy Runeberginkatu 4c A 4 Helsinki, 00100 Finland EMail: varun.singh@iki.fi URI: http://www.callstats.io/
Xiaoqing Zhu Cisco Systems 12515 Research Blvd Austing, TX 78759 USA EMail: xiaoqzhu@cisco.com
Michael A. Ramalho Cisco Systems, Inc. 6310 Watercrest Way Unit 203 Lakewood Ranch, FL 34202-5211 USA Phone: +1 919 476 2038 EMail: mramalho@cisco.com