Network Working Group L. Dunbar Internet Draft H. Song J. Kaippallimalil Intended status: Standard Futurewei Expires: May 2, 2021 November 2, 2020 IP Layer Metrics for 5G Edge Computing Service draft-dunbar-ippm-5g-edge-compute-ip-layer-metrics-01 Abstract This draft describes the IP Layer metrics and methods to measure the Edge Computing Servers running status and environment for IP networks to select the optimal Edge Computing server location in 5G Edge Computing (EC) environment. Those measurements are for IP network to dynamically optimize the forwarding of 5G edge computing service without any knowledge above IP layer. Status of this Memo This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79. This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79. This document may not be modified, and derivative works of it may not be created, except to publish it as an RFC and to translate it into languages other than English. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet-Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." xxx, et al. Expires May 2, 2021 [Page 1] IP Layer Metrics for 5G Edge Computing Services The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html This Internet-Draft will expire on April 7, 2021. Copyright Notice Copyright (c) 2020 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License. Table of Contents 1. Introduction..............................................3 1.1. 5G Edge Computing Background.........................3 1.2. Problem 1: Selecting 5G Edge Application Location....4 1.3. Problem 2: UE mobility creates unbalanced anycast distribution..............................................5 2. Conventions used in this document.........................7 3. IP-Layer Metrics Definitions for 5G EC services...........9 3.1. IP-Layer Application ID..............................9 3.2. IP-Layer metric for App Server Load Measurement......9 3.3. Capacity Index in the overall cost..................12 3.4. Site Preference Index in the overall cost...........12 3.5. RTT to an ANYCAST Address in 5G EC..................13 4. Algorithm in Selecting the optimal Target Location.......14 Dunbar, et al. Expires May 2, 2021 [Page 2] IP Layer Metrics for 5G Edge Computing Services 5. Scope of IP Layer Metrics Advertisement..................14 6. Manageability Considerations.............................15 7. Security Considerations..................................16 8. IANA Considerations......................................16 9. References...............................................16 9.1. Normative References................................16 9.2. Informative References..............................16 10. Acknowledgments.........................................17 1. Introduction 1.1. 5G Edge Computing Background In 5G Edge Computing environment [3GPP-EdgeComputing], one Application can have multiple Application Servers hosted in different Edge Computing data centers that are close in proximity. Those Edge Computing (mini) data centers are usually very close to, or co-located with, 5G base stations, with the goal to minimize latency and optimize the user experience. When a UE (User Equipment) initiates application packets using the destination address from a DNS reply or from its own cache, the packets from the UE are carried in a PDU session through 5G Core [5GC] to the 5G UPF-PSA (User Plan Function - PDU Session Anchor). The UPF-PSA decapsulate the 5G GTP outer header and forwards the packets from the UEs to the Ingress router of the Edge Computing (EC) Local Data Network (LDN). The LDN for 5G EC, which is the IP Networks from 5GC perspective, is responsible for forwarding the packets to the intended destinations. Routers in the local IP network should be able to select the "best" or "closest" application server location out of many. However, simply using distance alone as a metric may not be sufficient as there may be many locations in close proximity. Moreover, one of the main aims of locating application servers so close to the user is to provide lower latency. When a user moves and attaches from another access router (UPF), the local IP network should be able to continue routing to the established application server. As a user keeps moving further away, a closer application server maybe able to serve the user better. Network measurements, Dunbar, et al. Expires May 2, 2021 [Page 3] IP Layer Metrics for 5G Edge Computing Services including latency of various paths are provided to the application domain to assist in reselection. These problems are outlined in sections 1.2, and 1.3. 1.2. Problem 1: Selecting 5G Edge Application Location Having multiple locations closer to UEs to host one Application server can greatly improve the user experience. But selecting an optimal location for the application traffic from a UE may not be that simple. Using DNS to reply with the address of the Application Server location closest to the requesting UE can encounter issues like: - UE can cache results indefinitely, when the UE moves to a 5G cell site very far away, the cached address may still be used, which can incur large network delay. - The Application Server at a specific location whose address replied by the DNS might be heavily loaded causing slow or no response, when there are available low utilized Application Server, for the same application, at different locations very close in proximity. - No inherent leverage of proximity information present in the network (routing) layer, resulting in loss of performance - Local DNS resolver become the unit of traffic management Increasingly, Anycast is used extensively by various application providers and CDNs because ANYCAST makes it possible to dynamically load balance across locations that host the Application server based on network conditions. Application server location selection using Anycast address leverages the proximity information present in the network (routing) layer and eliminates the single point of failure and bottleneck at the DNS resolvers and application layer load balancers. Another benefit of using ANYCAST address is removing the dependency on UEs that use their cached destination IP addresses for extended period. Dunbar, et al. Expires May 2, 2021 [Page 4] IP Layer Metrics for 5G Edge Computing Services But selection of an ANYCAST location purely based on the network condition can encounter issue of the location selected by network routing information being overutilized while there are available underutilized locations close by. 1.3. Problem 2: UE mobility creates unbalanced anycast distribution Another problem of using ANYCAST address for multiple locations of an Application Server in 5G environment is that UEs' frequent moving from one 5G site to another. The frequent move of UEs can make it difficult to plan where Application server should be hosted. When a large number of UEs using a particular Application congregate together unpredictably, the ANYCAST location selected based on routing distance can be heavily utilized, while the same Application Server at other locations close-by are underutilized. Dunbar, et al. Expires May 2, 2021 [Page 5] IP Layer Metrics for 5G Edge Computing Services +--+ |UE|---\+---------+ +------------------+ +--+ | 5G | +-----------+ | S1: aa08::4450 | +--+ | Site A +----+ +----+ | |UE|----| | Ra | | R1 | S2: aa08::4460 | +--+ | +----+ +----+ | +---+ | | | | | S3: aa08::4470 | |UE1|---/+---------+ | | +------------------+ +---+ |IP Network | L-DN1 |(3GPP N6) | | | | +------------------+ | | | | S1: aa08::4450 | | | +----+ | | | | R3 | S2: aa08::4460 | v | +----+ | | | | S3: aa08::4470 | | | +------------------+ | | L-DN3 +--+ | | |UE|---\+---------+ | | +------------------+ +--+ | 5G | | | | S1: aa08::4450 | +--+ | Site B +----+ +----+ | |UE|----| | Rb | | R2 | S2: aa08::4460 | +--+ | +----+ +----+ | +--+ | | +-----------+ | S3: aa08::4470 | |UE|---/+---------+ +------------------+ +--+ L-DN2 Figure 1: multiple ANYCAST instances in different edge DCs This document describes the measurements at IP Layer that can reflect the application server running status and environment at the specific locations. This document also describes the method of incorporating those measurements with IP routing cost to come up with a more optimal criteria in selecting ANYCAST locations. Note: for the ease of description, the Edge Application Server and Application Server are used interchangeably throughout this document. Dunbar, et al. Expires May 2, 2021 [Page 6] IP Layer Metrics for 5G Edge Computing Services 2. Conventions used in this document A-ER: Egress Router to an Application Server Instance, [A-ER] is used to describe the last router that the application instance is attached. For 5G EC environment, the A-ER can be the gateway router to the Edge Computing Data Center. ANYCAST Instance: refer to the Application Server Gateway at a specific location which is reachable by the ANYCAST address. Application Server: An application server is a physical or virtual server that host the software system for the application. Application Server Location: Represent a cluster of servers at one location serving the same Application. One application may have a Layer 7 Load balancer, whose address(es) are reachable from external IP network, in front of a set of application servers. From IP network perspective, this whole group of servers are considered as the Application server at the location. EC: Edge Computing Edge Application Server: used interchangeably with Application Server throughout this document. Edge Computing Hosting Environment: An environment, such as psychical or virtual machines, providing support required for Edge Application Server's execution. NOTE: The above terminologies are the same as those used in 3GPP TR 23.758 Dunbar, et al. Expires May 2, 2021 [Page 7] IP Layer Metrics for 5G Edge Computing Services Edge DC: Edge Data Center, which provides the Edge Hosting Environment. It might be co-located with 5G Base Station and not only host 5G core functions, but also host frequently used Edge server instances. gNB next generation Node B Instance: the term "Instance" if used in the context of Application Server, is referring to one location of an application server; if used in the ANYCAST context, is referring to one location of the Application server with the same ANYCAST address. L-DN: Local Data Network PSA: PDU Session Anchor (UPF) RTT: Round Trip Time RTT-ANYCAST: A list of Round trip times to a group of routers that have the ANYCAST instances directly attached. SSC: Session and Service Continuity UE: User Equipment UPF: User Plane Function The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here. Dunbar, et al. Expires May 2, 2021 [Page 8] IP Layer Metrics for 5G Edge Computing Services 3. IP-Layer Metrics Definitions for 5G EC services 3.1. IP-Layer Application ID From network perspective, an application server has an Identifier, or IP Layer Application Server ID, which is usually an ANYCAST address that can represent multiple locations that host the application server. 3.2. IP-Layer metric for App Server Load Measurement There are many network techniques and protocols to optimize forwarding and ensuring QoS for applications, such as DSCP/DiffServ, Traffic Engineered (TE) solutions, Segment Routing, etc. But the reality is that most application servers don't expose their internal logics to network operators. Their communications are generally encrypted. Most of them do not even respond to PING or ICMP messages initiated by routers or network gears. The proposed IP Layer Metrics and algorithms enable the IP networks to dynamically optimize the forwarding of 5G edge computing service without any knowledge above IP layer. In a way, the proposed IP Layer Metrics and algorithm enable the IP networks to be more aware of Application behavior without dependency on getting information from Applications themselves. Without knowledge of application internal logics, network layer or IP Layer can monitor the traffic patterns to/from the Application Server at each location to gauge the running status of the application server at the location. First, the network needs to discover which router(s) has the application server attached. Those routers are called Application Server Egress Router, or A-ER for short. A-ER is usually the Gateway Router to an Edge Computing Data Center. To discover if a (Gateway) router is the A-ER for a specific Edge Application Server, the (Gateway) router can periodically send reverse ARP (IPv4) or Neighbor Discovery scan with the address of the Application Server to discover if the Application Servers are hosted in its edge computing data center. If yes, the router or routers are identified as the A-ER for the Application Server. For one Application Server, there can be many A-ERs at different EC Data Centers. Dunbar, et al. Expires May 2, 2021 [Page 9] IP Layer Metrics for 5G Edge Computing Services For an Application Server at a specific location, which is identified by the address of the application server at the IP layer, the A-ER can measure the amount of traffic destined towards the address & the amount of the traffic from the specific address, such as: - Total number of packets to the attached App Server (ToPackets); - Total number of packets from the attached App Server (FromPackets); - Total number of Bytes to the attached App Server (ToBytes); - Total number of bytes from the attached App Server (FromBytes); The actual load measurement to the App Server attached to an A-ER can be based on one of the metrics above or including all four metrics with different weights applied to each, such as: LoadIndex = w1*ToPackets+w2*FromPackes+w3*ToBytes+w4*FromBytes Where 0<= wi <=1 and w1+ w2+ w3+ w4 = 1. The weights of each metric contributing to the load index of the App Server attached to an A-ER can be configured or learned by self-adjusting based on user feedbacks. The raw measurement is useful when the A-ER routers cannot be configured with a consistent algorithm to compute the aggregated load index and the raw measurements are needed by a central analytic system. The A-ER can advertise either the aggregated Load Index or the raw measurements periodically, by BGP UPDATE messages or OSPF/ISIS Link statement Advertisements, to a group of routers that have traffic destined towards the ANYCAST addresses of those application servers. Dunbar, et al. Expires May 2, 2021 [Page 10] IP Layer Metrics for 5G Edge Computing Services Even though it would be better to have applications or their controllers directly reporting their own workload running status to network, it is not easy to have third party applications to provide information to network operators in addition to that applications servers can be running anywhere and. The IP-Layer Load Measurements provides an intelligent estimate of the application server running status at a specific location without requiring cooperation from third party Applications or Application controllers. Dunbar, et al. Expires May 2, 2021 [Page 11] IP Layer Metrics for 5G Edge Computing Services 3.3. Capacity Index in the overall cost Given that different Edge Computing DCs may have different capacity, the following metrics need to be included to gauge an application Server's running status at a specific location: - Capacity Index: Capacity Index is used to differentiate the running environment of the application server. Some data centers can have hundreds, or thousands, of servers behind an Application Server's App Layer Load Balancer that is reachable from external world. Other data centers can have very small number of servers for the application server. "Capacity Index", which is a numeric number, is used to represent the capacity of the application server in a specific location. "Capacity Index" can be a configured value indicating the capacity of a specific Application Server at a specific location, e.g. an Edge Computing DC. The Capacity Index is Application Server specific, meaning at one location, one Application Server can have the Capacity Index to be 10 and another server can be 2. If the Application Server capacity configuration is not available, a network analytics tool can use the historic measurements as the basis to estimate the site capacity. If an Application Server at a specific site has high volume for extended period historically, the site capacity can be considered as higher than the other site with historic low volume. This is under the assumption that application controllers monitor utilization of the application servers at different locations. If an application server has prolonged over-utilized servers at some locations, the application controller will trigger manual intervention to increase the computing powers at those locations. However, the manual intervention cycles can be in weeks/months. That is why the IP layer metrics and algorithms that can change flows direction in minutes become very essential. 3.4. Site Preference Index in the overall cost Dunbar, et al. Expires May 2, 2021 [Page 12] IP Layer Metrics for 5G Edge Computing Services As described in [IPv6-StickyService] and [ISPF-EXT-EC], an EC sticky service needs to connect a UE to the application server that has been serving the UE before the UE moves to a new 5G Site, unless there is failure to that location. To achieve the goal of sticking a flow from one specific UE to a specific site, a "site Preference Index" is created. The value of the Site Preference Index can be manipulated for packets of some flows to be steered towards an application server location farther away in routing distance. The "Site Preference Index" enables some sites to be more preferred for handling the UE traffic to a application server than others. 3.5. RTT to an ANYCAST Address in 5G EC ANYCAST used in 5G Edge computing environment is slightly different from the typical ANYCAST address being deployed. Typical ANYCAST address is used to represent instances in vast different geographical locations, such as different continents. ANCAST address for "app.net" for Asia lead packets to a server instance of "app.net" hosted in Asia. Therefore, the RTT for "app.net" in Asia, is a single value that represent the round time trip to the server in Asia that host the "app.net". 5G Edge Computing environment can have one Application server hosted in multiple Edge Computing DCs close in proximity. Routers, i.e. the ingress router to 5G LDN (Local Data Network), can forward packets for the ANYCAST address of "app.net" to different egress routers that have "app.net" servers attached. If "app.net" is hosted in four different 5G Edge Computing Data Centers. All those DCs have the same ANYCAST address for the "app.net". The RTT to "app.net" ANYCAST address need to be a group of values (instead of one RTT value to a unicast address). The RTT group value should include the A- ER router's specific unicast address (e.g. the loopback address) to which the Application Server is attached. RTT to "app.net" ANYCAST Address is represented as: List of {Egress Router address, RTT value} This list is called "RTT-ANYCAST". Dunbar, et al. Expires May 2, 2021 [Page 13] IP Layer Metrics for 5G Edge Computing Services In order to better optimize the ANYCAST traffic, each router adjacent to 5G PSA needs to periodically measure RTT to a list of A-ER routers that advertise the ANYCAST address. The RTT to egress router at Site-i is considered as the RTT to the ANYCAST instance at the Site-i. 4. Algorithm in Selecting the optimal Target Location The goal of the algorithm is to equalize the traffic among multiple locations of the same ANYCAST address. The main benefit of using ANYCAST is to leverage the IP- layer information to equalize the traffic among multiple locations of the same Application Server, usually identified by one or a group of ANYCAST addresses. For 5G Edge Computing environment, the ingress router to each LDN needs to be notified of the Load Index and Capacity Index of the Application Servers at different EC site to make the intelligent decision on where to forward the traffic from UEs for an application. The Algorithm needs to take the following attributes into consideration: - Load Measurement Index [Section 3.2], - capacity index [Section 3.3], - Preference Index [Section 3.4], and - network delay [Section 3.5]. Here is an algorithm for a router, e.g. the router directly attached to the 5G PSA, to compare the cost to reach the Application Server between the Site-i or Site-j: Load-i * CP-j Pref-j * Delay-i Cost-i=min(w *(----------------) + (1-w) *(------------------)) Load-j * CP-i Pref-i * Delay-j Load-i: Load Index at Site-i, it is the weighted combination of the total packets and bytes sent to and received from the Application Server at Site-i during a fixed time period. Dunbar, et al. Expires May 2, 2021 [Page 14] IP Layer Metrics for 5G Edge Computing Services CP-i (Capacity-i) (lower value means higher capacity): capacity index at the site i. Delay-i: Network latency measurement (RTT) to the A-ER that has the Application Server attached at the site-i. Pref-i (Preference Index: lower value means higher preference): Network Preference index for the site-I. w: Weight for load and site information, which is a value between 0 and 1. If smaller than 0.5, Network latency and the site Preference have more influence; otherwise, Server load and its capacity have more influence. 5. Scope of IP Layer Metrics Advertisement Each Application Server might be used by a small group of UEs. Therefore, it is not necessary for A-ER router to advertise the IP layer metrics to all other routers in the 5G LDN. Likewise, each EC Data Center may only host a small number of application servers. "Application Bound Group Routers" is used to refer a group of routers that are interested in a group of specific ANYCAST addresses. The IP Layer Metrics for an Application Server should be advertised among the routers in the "Application bound Group Routers". BGP RT Constrained Distribution [RFC4684] can be used to form the "Application Bound Group Routers". Since there are much more Application Servers than the number of routers in 5G LDN, a more practical way to form the "Application Bound Group of Routers" is for each ingress router to query a network controller upon receiving the first packet to a specific ANYCAST address to be included in the "Application Bound Group Routers". There should be a timer associated with Ingress router, as the UE that uses the Application Server might move away. Upon timer expires, the Ingress Router is removed from the "Application Bound Group of Routers". 6. Manageability Considerations To be added. Dunbar, et al. Expires May 2, 2021 [Page 15] IP Layer Metrics for 5G Edge Computing Services 7. Security Considerations To be added. 8. IANA Considerations To be added. 9. References 9.1. Normative References [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997. [RFC4364] E. rosen, Y. Rekhter, "BGP/MPLS IP Virtual Private networks (VPNs)", Feb 2006. [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, May 2017, . [RFC8200] s. Deering R. Hinden, "Internet Protocol, Version 6 (IPv6) Specification", July 2017 9.2. Informative References [3GPP-EdgeComputing] 3GPP TR 23.748, "3rd Generation Partnership Project; Technical Specification Group Services and System Aspects; Study on enhancement of support for Edge Computing in 5G Core network (5GC)", Release 17 work in progress, Aug 2020. [RFC5521] P. Mohapatra, E. Rosen, "The BGP Encapsulation Subsequent Address Family Identifier (SAFI) and the BGP Tunnel Encapsulation Attribute", April 2009. Dunbar, et al. Expires May 2, 2021 [Page 16] IP Layer Metrics for 5G Edge Computing Services [BGP-SDWAN-Port] L. Dunbar, H. Wang, W. Hao, "BGP Extension for SDWAN Overlay Networks", draft-dunbar-idr-bgp- sdwan-overlay-ext-03, work-in-progress, Nov 2018. [SDWAN-EDGE-Discovery] L. Dunbar, S. Hares, R. Raszuk, K. Majumdar, "BGP UPDATE for SDWAN Edge Discovery", draft-dunbar-idr-sdwan-edge-discovery-00, work-in- progress, July 2020. [Tunnel-Encap] E. Rosen, et al "The BGP Tunnel Encapsulation Attribute", draft-ietf-idr-tunnel-encaps-10, Aug 2018. 10. Acknowledgments Acknowledgements to Donald Eastlake for their review and contributions. This document was prepared using 2-Word-v2.0.template.dot. Dunbar, et al. Expires May 2, 2021 [Page 17] IP Layer Metrics for 5G Edge Computing Services Authors' Addresses Linda Dunbar Futurewei Email: ldunbar@futurewei.com HaoYu Song Futurewei Email: haoyu.song@futurewei.com John Kaippallimalil Futurewei Email: john.kaippallimalil@futurewei.com Dunbar, et al. Expires May 2, 2021 [Page 18]