TSVWG Y. Li INTERNET-DRAFT X. Zhou Intended Status: Informational Huawei Expires: March 30, 2019 September 26, 2018 Overlayed Path Segment Forwarding (OPSF) Problem Statement draft-li-overlayed-path-segment-forwarding-ps-00 Abstract Various overlays are used in networks including WAN, enterprise campus and others. End to end path are divided into multiple segments some of which are overlay encapsulated to achieve better path selection, lower latency and so on. Traditional end-to-end transport layer is not very responding to microburst and non-congestive packet loss caused by the different characteristics of the path segments. With the potential transport enhancement for the existing or purposely created overlayed path segment, end to end throughput can be improved. This document illustrates the problems in some use cases and tries to inspire more about whether and how to solve them by introducing a reliable, efficient and non-intrusive transport forwarding over the overlayed path segment(s). Status of this Memo This Internet-Draft is submitted to IETF in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet-Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/1id-abstracts.html The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html Li, et al [Page 1] INTERNET DRAFT OPSF Problem Statement September 2018 Copyright and License Notice Copyright (c) 2018 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License. Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . . 3 3. Use Cases and Problems . . . . . . . . . . . . . . . . . . . . 3 3.1 Microburst in Long Haul Network . . . . . . . . . . . . . . 4 3.2 Non-congestive Loss in WiFi Accessed Campus Overlay . . . . 6 3.3 Higher Reliability and Low Latency for Interactive Application . . . . . . . . . . . . . . . . . . . . . . . . 8 4. Features to be Considered for OPSF (Overlayed Path Segment Forwarding) . . . . . . . . . . . . . . . . . . . . . . . . . . 8 5. Security Considerations . . . . . . . . . . . . . . . . . . . . 10 6. IANA Considerations . . . . . . . . . . . . . . . . . . . . . . 10 7. References . . . . . . . . . . . . . . . . . . . . . . . . . . 10 7.1 Normative References . . . . . . . . . . . . . . . . . . . 10 7.2 Informative References . . . . . . . . . . . . . . . . . . 10 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 11 Li, et al [Page 2] INTERNET DRAFT OPSF Problem Statement September 2018 1. Introduction Overlay tunnels are widely deployed for various networks, including long haul WAN interconnection, enterprise wireless access networks, etc. End to end connection are normally broken into multiple path segments for different purposes, for instance, selecting a better overlay path over the WAN or deliver the packets over the heterogenous networks like enterprise access and core networks. TCP-like transport layer provides end to end flow control and congestion control for path reliability and high throughput. Such an approach has the problems of slow congestion responding and non- congestive loss misinterpretation at the sender and does not achieve the optimal performance in certain cases. Some of the problems have been well known over years. With new technologies are emerging like NFV (Network Function Virtualization) and various flexible overlay protocols, forwarding over the specific overlayed path segment(s) can be considered to be enhanced by providing a reliable and non-intrusive transport to improve the throughput to solve those problems. This document illustrates the problems in some use cases and tries to inspire more about whether and how to solve them by introducing a reliable, efficient and non-intrusive transport forwarding for the overlayed path segment(s). 2. Terminology The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here. OPSF: Overlayed Path Segment Forwarding 3. Use Cases and Problems The following subsections presents use cases from different scenarios using overlay tunnels with a common need of higher performance and reliable overlayed path segment in best effort networks. Li, et al [Page 3] INTERNET DRAFT OPSF Problem Statement September 2018 3.1 Microburst in Long Haul Network Internet is a huge sized network of networks. The interconnections of end devices using this global network are normally provided by ISPs (Internet Service Provider). This ISP provided huge network is considered as traditional Internet. CSPs (Cloud Service Provider) are connecting their data centers using Internet or self-constructed networks/links. This expands Internet's infrastructure and together with the original ISP's infrastructure, forms the Internet underlay. NFV further makes it easier to dynamically provision a new virtual node as a work load in a cloud for CPU/storage intensive functions. With the aid of various mechanisms such as kernel bypassing and Virtual IO, forwarding based on virtual node is becoming more and more effective. The interconnections among the purposely positioned virtual nodes and/or the existing nodes with vitalization functions potentially form "the Second Plane" or overlay of Internet. It is called the Cloud-Internet Overlay Network (CION) in this document. CION makes use of overlay technologies to direct the traffic going to the specific path regardless the underlying physical topology to achieve better service delivery. Figure 1 shows an emerging multi- segment overlay over large geographic distances. It purposely creates or selects overlay nodes (ON) from clouds/Internet. Segment here is the virtual hop between two ONs. By directing the traffic to be forwarded along those virtual nodes rather than the default path, better delivery in terms of throughput and delay can be achieved. When a large number of potential virtual nodes are available, there is a high chance that a better path could be found [CRONets]. Li, et al [Page 4] INTERNET DRAFT OPSF Problem Statement September 2018 _____________ / domain 1 \ / \ ___/ -------------\ / \ PoP1 ->--ON1 \ | | ON4------>-- PoP2 | | ON2 ___|__/ \__|_ |->| _____ / | | \|__|__ / \ / | | | | \____/ \__/ | \|/ | | _____ | | | | ___/ \ | | | \|/ / \_____ | | | | / domain 2 \ /|\ | | | | ON3 | | | | | \ |->| | | | | | \_____|__|_______/ | | /|\ | | \|/ | | | | | | | | | | /|\ | | +--------------------------------------------------+ | | | | | | | Internet | | o--o o---o->---o o---o->--o--o underlay | +--------------------------------------------------+ Figure 1. Cloud-Internet Overlay Network (CION) Microburst is an unexpected data bursts within a very small time window (probably in micro-seconds). Some research shows microbursts happen even for underutilized link [BurstyAna]. The short spikes caused by microburst result in higher jitter and sometimes packet loss in a network. Such loss may trigger the congestion control like reducing the sending rate at the TCP sender as it exhibits the normal pattern of congestion loss in terms of duplicate acknowledgements and/or RTT increases. As microburst is extremely short, the packet loss caused by it is non-persistent and rather random. Therefore it does not necessarily require the sender to reduce its sending rate. Invoking the congestion control at the sender may unnecessarily make the average sending rate low and degrades the throughput in long haul CION. In addition, long haul transmission may take hundreds of milliseconds. The packet loss response at the sender to microburst over the long haul transmission is not timely. Sender's reaction does not really respond to the current instantaneous path situation. Overlay nodes in the middle can potentially offer new possibilities, Li, et al [Page 5] INTERNET DRAFT OPSF Problem Statement September 2018 e.g. retransmission over ONs, to better response to microbursts. Such enhancement can be enabled based on the individual overlayed path segment rather than on the entire end to end path to improve the response time and performance from the packet loss/re-order caused by microburst. Such enhancement should avoid racing with higher layer transport protocols. 3.2 Non-congestive Loss in WiFi Accessed Campus Overlay Different path segments have different characteristics. The probabilities of packet loss over every and each segments have a large variance. The non-congestive packet loss usually occurs in some specific overlayed path segments. End to end TCP-like transport protocols do not take this factor into careful account. It assumes that packet loss for any reason is almost evenly distributed across the entire path, and adjusts the sender to accommodate the packet loss of the bottleneck segment. This results in non-optimal sending rate in some cases. Figure 2 shows the WiFi accessed enterprise campus. AP connects to its edge switch normally using Cat5/5e twisted-pair cable which typically provides less than 10G bandwidth. The data packets are tunneled using various overlay mechanisms, like VXLAN [RFC7348], LISP [RFC6830] or CAPWAP [RFC5415]. Two edge switches use another overlay segment over campus core network to deliver the packets which provides more functions like policy enforcement and mobility enhancement. This overlay is usually over fiber which provides higher bandwidth. Li, et al [Page 6] INTERNET DRAFT OPSF Problem Statement September 2018 _____________________________ +-----+ /| |\ +-----+ |edge1| | | | | |edge2| +-----+ \|____________________________|/ +-----+ _ _ /_\ fiber /_\ | | | | | | | | | | Cat5/5e | | Cat5/5e | | cable | | cable |_| |_| \_/ \_/ +-----+ +-----+ | AP1 | | AP2 | +-----+ +-----+ | | | WiFi access | WiFi access | | +----+ +----+ |STA1| |STA2| +----+ +----+ Figure 2. WiFi accessed Campus Overlay Network Cat5/5e cables, especially UTP (Unshielded twisted pair), are susceptible to distance, interference, and bundling. The environment and the way they are deployed cause drastic changes in random loss rate. The overlay tunnel running over it will have more transmission unreliability than the overlay running on the fiber. Current transport layer is not able to identify such specific problematic segment and simply leaves it for the end to end congestion control to handle it so that the sender may be kept at a lower sending rate and the throughput is not optimal. In addition to the uplink of the AP, the non-congestive packet loss generated by the wireless access link itself accounts for the largest proportion in the end-to-end path. Wifi access is affected by fades, interference, attenuation and corruption. Non-congestive loss is common. Its link layer has mechanisms to do the packet recovery. However the number of local link layer retransmission is usually based on the empirical value or the static configuration. When the value is not properly chosen, the TCP sender can be unnecessarily exposed by the random packet loss and reduce the sending rate. It is hard to make the link layer frame recovery work in concert with the current end to end transport layer. Li, et al [Page 7] INTERNET DRAFT OPSF Problem Statement September 2018 3.3 Higher Reliability and Low Latency for Interactive Application Mobile gaming and VoIP like application normally can not tolerate a retransmission even over a path segment. When two divergent overlay segments are available like shown in figure 3 for path from ON1 to ON2, purposely duplicating packets over two segments provides more reliability. Two disjoint segments can usually be obtained by measuring to find segments with very low mathematical correlation in latency change. When the number of overlay nodes is large, it is easy to find such disjoint segments. Random node or memory failure may also benefit from the duplicating packets over disjoint segments. ON-A +----------o------------------+ | | | | A -----o ON1 ON2o----- B | | +-----------------------o-----+ ON-B Figure 3. Multiple Overlayed Path Segments for Higher Reliability 4. Features to be Considered for OPSF (Overlayed Path Segment Forwarding) The diagram shown in Figure 4 illustrates a typical scenario with an overlayed path segment. Transport layer provide the end to end flow control between two end host. When an overlayed path segment exists or is purposely created between two overlay nodes, an enhanced forwarding over that segment can potentially solve some problems of end to end transport performance issues and at the same time provides more reliability and flexibility to traffic path. Li, et al [Page 8] INTERNET DRAFT OPSF Problem Statement September 2018 ON=overlay node UN=underlay node +-----------+ +-----------+ |Application|<-------------- end-to-end -------------> |Application| +-----------+ +-----------+ | Transport |<-------------- end-to-end -------------> | Transport | | | | | +-----------+ overlayed +-----------+ | | +----+ path segment+----+ | | | | | ON |<----------->| ON | | | | Network | +----+ +----+ +----+ +----+ | Network | | |<->| UN |<->| UN |<->| UN |<-->| UN |<--->| | +-----------+ +----+ +----+ +----+ +----+ +-----------+ End Host End Host Figure 4. A Simple Overlayed Path Segment Forwarding Usage Scenario Features need more investigations include, - Enhancement for the overlayed path segment, like retransmission, FEC(forward error correction), duplicating packet over the segments, lightweighted congestion control, etc. When the segment is a small portion of the whole end to end path, the retransmission over it has more benefit. Retransmission over the path segment has to be carefully designed to avoid the racing condition with the upper layer. The segment enabled retransmission may measure the segment RTT by itself to determine the appropriate retransmission attempts. On the other hand, the upper layers including the applications can indicate the credit as the safe band time that allows for the overlayed path segment to do the retransmission. At the same time, the persistent congestion caused packet loss should be exposed to the upper transport layer, so that the sender's congestion control can work properly. The timing of activation of the enhancement scheme, parameters such as the threshold setting of retransmission are worthy of further determination. - Measurement based path selection for better performance, backup or load balancing. Overlay nodes have to be continuously monitored in order to find one or more appropriate overlayed paths. Such measurement can be in-band or out of band of data packets. When more than one overlayed segment with the same ingress and egress are used, Li, et al [Page 9] INTERNET DRAFT OPSF Problem Statement September 2018 it has to be determined how the traffic are split and merged. 5. Security Considerations TBD 6. IANA Considerations No IANA action is required. 7. References 7.1 Normative References [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997. 7.2 Informative References [RFC4122] Leach, P., Mealling, M., and R. Salz, "A Universally Unique IDentifier (UUID) URN Namespace", RFC 4122, July 2005. [RFC5415] Calhoun, P., Ed., Montemurro, M., Ed., and D. Stanley, Ed., "Control And Provisioning of Wireless Access Points (CAPWAP) Protocol Specification", RFC 5415, March 2009. [BurstyAna] Chung S., Agrawal D., Kim M., Hong J., and Park K. "Analysis of bursty packet loss characteristics on underutilized links using SNMP", IEEE/IFIP E2EMON, 2004. [CRONets] Cai, C. X., Le, F., Sun, X., Xie, G. G., Jamjoom, H., and Campbell, R. H. CRONets: Cloud-Routed Overlay Networks. In 36th International Conference on Distributed Computing Systems (ICDCS) (2016), IEEE, pp. 67--77. [RFC6830] Farinacci, D., Fuller, V., Meyer, D., and D. Lewis, "The Locator/ID Separation Protocol (LISP)", RFC 6830, January 2013. [RFC7348] Mahalingam, M., Dutt, D., Duda, K., Agarwal, P., Kreeger, L., Sridhar, T., Bursell, M., and C. Wright, "Virtual eXtensible Local Area Network (VXLAN): A Framework for Li, et al [Page 10] INTERNET DRAFT OPSF Problem Statement September 2018 Overlaying Virtualized Layer 2 Networks over Layer 3 Networks", RFC 7348, August 2014. Authors' Addresses Yizhou Li Huawei Technologies 101 Software Avenue, Nanjing 210012 China Phone: +86-25-56624584 EMail: liyizhou@huawei.com Xingwang Zhou Huawei Technologies Huawei Technologies 101 Software Avenue, Nanjing 210012 China EMail: zhouxingwang@huawei.com Li, et al [Page 11]