Routing Area Working Group J. Dong Internet-Draft S. Bryant Intended status: Informational Huawei Expires: January 3, 2019 Z. Li China Mobile T. Miyasaka KDDI Corporation July 2, 2018 Enhanced Virtual Private Networks (VPN+) draft-dong-teas-enhanced-vpn-00 Abstract This draft describes a number of enhancements that need to be made to virtual private networks (VPNs) to support the needs of new applications, particularly applications that are associated with 5G services. A network enhanced with these properties may form the underpin of network slicing, but will also be of use in its own right. Editor's Note: This is draft-bryant-rtgwg-enhanced-vpn moved to the TEAS WG. Status of This Memo This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet- Drafts is at https://datatracker.ietf.org/drafts/current/. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." This Internet-Draft will expire on January 3, 2019. Copyright Notice Copyright (c) 2018 IETF Trust and the persons identified as the document authors. All rights reserved. Dong, et al. Expires January 3, 2019 [Page 1] Internet-Draft VPN+ July 2018 This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License. Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 2. Requirements Language . . . . . . . . . . . . . . . . . . . . 4 3. Overview of the Requirements . . . . . . . . . . . . . . . . 4 3.1. Isolation between Virtual Networks . . . . . . . . . . . 4 3.2. Diverse Performance Guarantees . . . . . . . . . . . . . 6 3.3. A Pragmatic Approach to Isolation . . . . . . . . . . . . 7 3.4. Integration . . . . . . . . . . . . . . . . . . . . . . . 8 3.5. Dynamic Configuration . . . . . . . . . . . . . . . . . . 9 3.6. Customized Control Plane . . . . . . . . . . . . . . . . 9 4. Applicability . . . . . . . . . . . . . . . . . . . . . . . . 10 5. Architecture and Components of Enhanced VPN . . . . . . . . . 10 5.1. Communications Layering . . . . . . . . . . . . . . . . . 10 5.2. Multi-Point to Multi-point . . . . . . . . . . . . . . . 13 5.3. Candidate Underlay Technologies . . . . . . . . . . . . . 13 5.3.1. FlexE . . . . . . . . . . . . . . . . . . . . . . . . 14 5.3.2. Dedicated Queues . . . . . . . . . . . . . . . . . . 14 5.3.3. Time Sensitive Networking . . . . . . . . . . . . . . 15 5.3.4. Deterministic Networking . . . . . . . . . . . . . . 15 5.3.5. MPLS Traffic Engineering (MPLS-TE) . . . . . . . . . 15 5.3.6. Segment Routing . . . . . . . . . . . . . . . . . . . 16 5.4. Control Plane Considerations . . . . . . . . . . . . . . 19 5.5. Application Specific Network Types . . . . . . . . . . . 19 5.6. Integration with Service Functions . . . . . . . . . . . 20 6. Scalability Considerations . . . . . . . . . . . . . . . . . 20 6.1. Maximum Stack Depth . . . . . . . . . . . . . . . . . . . 21 6.2. RSVP Scalability . . . . . . . . . . . . . . . . . . . . 21 7. OAM and Instrumentation . . . . . . . . . . . . . . . . . . . 21 8. Enhanced Resiliency . . . . . . . . . . . . . . . . . . . . . 22 9. Security Considerations . . . . . . . . . . . . . . . . . . . 23 10. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 23 11. References . . . . . . . . . . . . . . . . . . . . . . . . . 23 11.1. Normative References . . . . . . . . . . . . . . . . . . 23 11.2. Informative References . . . . . . . . . . . . . . . . . 23 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 25 Dong, et al. Expires January 3, 2019 [Page 2] Internet-Draft VPN+ July 2018 1. Introduction Virtual networks, often referred to as virtual private networks (VPNs) have served the industry well as a means of providing different groups of users with logically isolated access to a common network. The common or base network that is used to provide the VPNs is often referred to as the underlay, and the VPN is often called an overlay. Driven largely by needs surfacing from 5G, the concept of network slicing has gained traction. There is a need to create a VPN with enhanced characteristics. Specifically there is a need for a transport network supporting a set of virtual networks each of which provides the client with dedicated (private) networking, computing and storage resources drawn from a shared pool. The tenant of such a network can require a degree of isolation and performance that previously could only be satisfied by dedicated networks. Additionally the tenant may ask for some level of control of their virtual network e.g. to customize the service paths in the network slice. These properties cannot be met with pure overlay networks, as they require tighter coordination and integration between the underlay and the overlay network. This document introduces a new network service called enhanced VPN (VPN+). VPN+ refers to a virtual network which has dedicated network resources allocated from the underlay network. Unlike traditional VPN, an enhanced VPN can achieve greater isolation and guaranteed performance. These new network layer properties, which have general applicability, may also be of interest as part of a network slicing solution. This document specifies a framework for using the existing, modified and potential new networking technologies as components to provide an enhanced VPN (VPN+) service. Specifically we are concerned with: o The design of the enhanced VPN data-plane o The necessary protocols in both underlay and the overlay of enhanced VPN, and o The mechanisms to achieve integration between overlay and underlay o The necessary method of monitoring an enhanced VPN o The methods of instrumenting an enhanced VPN to ensure that the required tenant Service Level Agreement (SLA) is maintained Dong, et al. Expires January 3, 2019 [Page 3] Internet-Draft VPN+ July 2018 The required layer structure necessary to achieve this is shown in Section 5.1. One use for enhanced VPNs is to create network slices with different isolation requirements. Such slices may be used to provide different tenants of vertical industrial markets with their own virtual network with the explicit characteristics required. These slices may be "hard" slices providing a high degree of confidence that the VPN+ characteristics will be maintained over the slice life cycle, of they may be "soft" slices in which case some degree of interaction may be experienced. 2. Requirements Language The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC2119]. 3. Overview of the Requirements In this section we provide an overview of the requirements of an enhanced VPN. 3.1. Isolation between Virtual Networks The requirement is to provide both hard and soft isolation between the tenants/applications using one enhanced VPN and the tenants/ applications using another enhanced VPN. Hard isolation is needed so that applications with exacting requirements can function correctly despite a flash demand being created on another VPN competing for the underlying resources. An example might be a network supporting both emergency services and public broadband multi-media services. During a major incident the VPNs supporting these services would both be expected to experience high data volumes, and it is important that both make progress in the transmission of their data. In these circumstances the VPNs would require an appropriate degree of isolation to be able to continue to operate acceptably. We introduce the terms hard (static) and soft (dynamic) isolation to cover cases such as the above. A VPN has soft isolation if the traffic of one VPN cannot be inspected by the traffic of another. Both IP and MPLS VPNs are examples of soft isolated VPNs because the network delivers the traffic only to the required VPN endpoints. However the traffic from one or more VPNs and regular network traffic may congest the network resulting in delays for other VPNs operating normally. The ability for a VPN to be sheltered from this effect is Dong, et al. Expires January 3, 2019 [Page 4] Internet-Draft VPN+ July 2018 called hard isolation, and this property is required by some critical applications. Although these isolation requirements are triggered by the needs of 5G networks, they have general utility. In the remainder of this section we explore how isolation may be achieved in packet networks. It is of course possible to achieve high degrees of isolation in the optical layer. However this is done at the cost of allocating resources on a long term basis and end-to-end basis. Such an arrangement means that the full cost of the resources must be borne by the service that is allocated the resources. On the other hand, isolation at the packet layer allows the resources to be shared amongst many services and only dedicated to a service on a temporary basis. This allows greater statistical multiplexing of network resources and amortizes the cost over many services, leading to better economy. However, the degree of isolation required by network slicing cannot easily be met with MPLS-TE packet LSPs as they guarantee long-term bandwidth, but not latency. Thus some trade-off between the two approaches needs to be considered to provide the required isolation between virtual networks while still allows reasonable sharing inside each VPN. The work of the IEEE project on Time Sensitive Networking is introducing the concept of packet scheduling where a high priority packet stream may be given a scheduled time slot thereby guaranteeing that it experiences no queuing delay and hence a reduced latency. However where no scheduled packet arrives its reserved time-slot is handed over to best effort traffic, thereby improving the economics of the network. Such a scheduling mechanism may be usable directly, or with extension to achieve isolation between multiple VPNs. One of the key areas in which isolation needs to be provided is at the interfaces. If nothing is done the system falls back to the router queuing system in which the ingress places it on a selected output queue. Modern routers have quite sophisticated output queuing systems, traditionally these have not provided the type of scheduling system needed to support the levels of isolation required by the applications that are the target of VPN+ networks. However some of the more modern approaches to queuing allow the construction of virtual channelized sub-interfaces (VCSI). With VCSIs there is only one physical interface, but the queuing system is used to provide virtual interfaces with dedicated resources. Sophisticated queuing systems of this type may be used to provide end-to-end virtual isolation between tenant's traffic in an otherwise homogeneous network. Dong, et al. Expires January 3, 2019 [Page 5] Internet-Draft VPN+ July 2018 [FLEXE] provides the ability to multiplex multiple channels over an Ethernet link in a way that provides hard isolation. However it is a only a link technology. When packets are received by the downstream node they need to be processed in a way that preserves that isolation. This in turn requires a queuing and forwarding implementation that preserves the isolation end-to-end. 3.2. Diverse Performance Guarantees There are several aspects to guaranteed performance: guaranteed maximum packet loss, guaranteed maximum delay and guaranteed delay variation. Guaranteed maximum packet loss is a common parameter, and is usually addressed by setting the packet priorities, queue size and discard policy. However this becomes more difficult when the requirement is combine with the latency requirement. The limiting case is zero congestion loss, and than is the goal of the Deterministic Networking work that the IETF and IEEE are pursuing. In modern optical networks loss due to transmission errors is already asymptotic to zero due, but there is always the possibility of failure of the interface and the fiber itself. This can only be addressed by some form of packet duplication and transmission over diverse paths. Guaranteed maximum latency is required in a number of applications particularly real-time control applications and some types of virtual reality applications. The work of the IETF Deterministic Networking (DetNet) Working Group is relevant, however the scope needs to be extended to methods of enhancing the underlay to better support the delay guarantee, and to integrate these enhancements with the overall service provision. Guaranteed maximum delay variation is a service that may also be needed. Time transfer is one example of a service that needs this, although the fungible nature of time means that it might be delivered by the underlay as a shared service and not provided through different virtual networks. Alternatively a dedicated virtual network may be used to provide this as a shared service. The need for guaranteed maximum delay variation as a general requirement is for further study. This leads to the concept that there is a spectrum of grades of service guarantee that need to be considered when deploying an enhanced VPN. As a guide to understanding the design requirements we can consider four types: o Guaranteed latency Dong, et al. Expires January 3, 2019 [Page 6] Internet-Draft VPN+ July 2018 o Enhanced delivery o Assured bandwidth o Best effort Best effort is the service that current VPNs provide. Providing assured bandwidth to VPNs, for example by using an RSVP-TE is not widely deployed at least partially due to scalability concerns. Guaranteed latency and enhanced delivery are not yet integrated with VPNs. It is these later two design requirements that enhanced VPNs provide. In Section 3.1 we considered the work of the IEEE Time Sensitive Networking (TSN) project and the work of the IETF DetNet Working group in the context of isolation. However this work is of greater relevance in assuring end-to-end packet latency. It is also of importance in considering enhanced delivery. A service that is guaranteed latency has a latency upper bound provided by the network. It is important to note that assuring the upper bound is more important than achieving the minimum latency. A service that is offered enhanced delivery is one in which the network (at layer 3) attempts to deliver the packet through multiple paths in the hope of avoiding transient congestion [I-D.ietf-detnet-dp-sol]. A useful mechanism to provide these guarantees is to use Flex Ethernet [FLEXE] as the underlay. This is a method of bonding Ethernets together and of providing time-slot based channelization over an Ethernet bearer. Such channels are fully isolated from other channels running over the same Ethernet bearer. As noted elsewhere this produces hard isolation but at the cost of making the reclamation of unused bandwidth harder. These approaches can usefully be used in tandem. For example, It is possible to use FlexE to provide tenant isolation, and then to use the TSN/Detnet approach over FlexE to provide service performance guarantee inside the a slice/tenant VPN. 3.3. A Pragmatic Approach to Isolation A key question to consider is whether it is possible to achieve hard isolation in packet networks. Packet networks were never designed to support hard isolation, just the opposite, they were designed to provide a high degree of statistical multiplexing and hence a significant economic advantage when compared to a dedicated, or a Dong, et al. Expires January 3, 2019 [Page 7] Internet-Draft VPN+ July 2018 Time Division Multiplexing (TDM) network. However the key thing to bear in mind is that the concept of hard isolation needs to be viewed from the perspective of the application, and there is no need to provide any harder isolation than is required by the application. From a historical perspective it is good to think about pseudowires [RFC3985] which emulate services that in many would have had hard isolation in their native form. However experience has shown that in most cases an approximation to this requirement is sufficient for most uses. Thus, for example, using FlexE or channelized sub-interface,together with packet scheduling as interface slicing, and optionally, also together with the slicing of node resources (Network Processor Unit (NPU), etc.), it may be possible to provide a type of hard isolation that is adequate for many applications. Other applications may be satisfied with a classical VPN with or without reserved bandwidth, but yet others may require dedicated point to point fiber. The requirement is thus to qualify the needs of each application and provide an economic solution that satisfies those needs without over- engineering. This spectrum of isolation is shown below: O=============================O===================O | | | Statistical Pragmatic Absolute Multiplexing Isolation Isolation (Traditional (Enhanced VPN) (Dedicated VPNs) Network) At one end of the above figure, we have traditional statistical multiplexing technologies that support VPNs. This is a service type that has served the industry well and will continue to do so. At the opposite end of the spectrum we have the absolute isolation provided by traditional networks. The goal of enhanced VPN is pragmatic isolation. This is isolation that is better than is obtainable from pure statistical multiplexing, more cost effective and flexible than a dedicated network, but which is a practical solution that is good enough for the majority of applications. 3.4. Integration A solution to the enhanced VPN problem will need to provide seamless integration of both overlay VPN and the underlay network resources. This needs be done in a flexible and scalable way so that it can be widely deployed in operator networks. Given the targeting of both this technology and service function chaining at mobile networks and Dong, et al. Expires January 3, 2019 [Page 8] Internet-Draft VPN+ July 2018 in particular 5G the co-integration of service functions is a likely requirement. 3.5. Dynamic Configuration It is necessary that new enhanced VPNs can be introduced to the network, modified, and removed from the network according to service demand. In doing so due regard must be given to the impact of other enhanced VPNs that are operational. An enhanced VPN that requires hard isolation must not be disrupted by the installation or modification of another enhanced VPN. Whether modification of an enhanced VPN can be disruptive to that VPN, and in particular the traffic in flight is to be determined, but is likely to be a difficult problem to address. The data-plane aspect of this are discussed further in Section 5.3. The control-plane and management-plane aspects of this, particularly the garbage collection are likely to be challenging and are for further study. As well as managing dynamic changes to the VPN in a seamless way, dynamic changes to the underlay and its transport network need to be managed in order to avoid disruption to sensitive services. In addition to non-disruptively managing the network as a result of gross change such as the inclusion of a new VPN endpoint or a change to a link, consideration has to be given to the need to move VPN traffic as a result of traffic volume changes. 3.6. Customized Control Plane In some cases it is desirable that an enhanced VPN has a custom control-plane, so that the tenant of the enhanced VPN can have some control to the resources and functions partitioned for this VPN. Each enhanced VPN may have its own dedicated controller, it may be provided with an interface to a control-plane that is shared with a set of other tenants, or it may be provided with an interface to the control-plane of the underlay provided by the underlay network operator. Further detail on this requirement will be provided in a future version of the draft. Dong, et al. Expires January 3, 2019 [Page 9] Internet-Draft VPN+ July 2018 4. Applicability The technologies described in this document is applicable to a number types of VPN technology such as: o Layer 2 point to point services such as pseudowires [RFC3985] o Layer 2 VPNs [RFC4664] o Ethernet VPNs [RFC7209] o Layer 3 VPNs [RFC4364], [RFC2764] Where such VPN types need enhanced isolation and delivery characteristics the technology described here can be used to provide an underlay with the required enhanced performance. 5. Architecture and Components of Enhanced VPN Normally a number of enhanced VPN services will be provided by a common network infrastructure. Each enhanced VPN consists of both the overlay and a specific set of dedicated network resources and functions allocated in the underlay to satisfy the needs of the VPN tenant. The integration between overlay and underlay ensures the isolation between different enhanced VPNs, and facilitates the guaranteed performance for different services. An enhanced VPN needs to be designed with consideration given to: o Isolation of enhanced VPN data plane. o A scalable control plane to match the data plane isolation. o The amount of state in the packet vs the amount of state in the control plane. o Mechanism for diverse performance guarantee within an enhanced VPN o Support of the required integration between network functions and service functions. 5.1. Communications Layering The communications layering model use to build an enhanced VPN is shown in Figure 1. Dong, et al. Expires January 3, 2019 [Page 10] Internet-Draft VPN+ July 2018 Tenant Tenant connection Tenant CE1 ----------------------------------------CE2 \ / AC \ OP Provider VPN OP /AC +- PE1------------------------------PE1 -+ Enhanced Paths ============================== Underlay ++++++++++++++++++++++++++++++ Figure 1: Communication Layering The network operator is required to provide a tenant connection between the tenant's Customer Equipment (CE) (CE1 and CE2). These CEs attach to the Operator's Provider Edge Equipments (PE) (PE1 and PE2 respectively). The attachment circuits (AC) are outside the scope of this document other than to note that they obviously need to provide a connection of sufficient quality in terms of isolation, latency etc. so as to satisfy the needs of the user. The subtlety to be aware of is that the ACs are often provided by a network rather than a fixed point to point connection and thus the considerations in this document may apply to the network that provides the AC. A provider VPN is constructed between PE1 and PE2 to carry tenant traffic. This is a normal VPN, and provides one stage of isolation between tenants. An enhanced path is constructed to carry the provider VPN using dedicated resources drawn from the underlay. This layered architecture is shown in more detail in Figure 2. Dong, et al. Expires January 3, 2019 [Page 11] Internet-Draft VPN+ July 2018 Network Controller ] Network ======================= } Control . . . . . . . . . . . N----N----N . } . / / . } N-----N-----N----N-----N } N----N } / / \ } Virtual N-----N----N----N-----N } Networks N----N } / / } N-----N-----N----N-----N } +----+ ===== +----+ ===== +----+ ===== +----+ } Physical +----+ ===== +----+ ===== +----+ ===== +----+ } Network +----+ ===== +----+ ===== +----+ ===== +----+ } +----+ +----+ +----+ +----+ N L N L N L N N = Partitioned node L = Partitioned link +----+ = Partition within a node +----+ ====== = Partition within a link Figure 2: The Layers Architecture Underpinning everything is the physical layer consisting of partitioned links and nodes which provide the underlying resources used to provision the logical networks. Various components and techniques as discussed in Section 5.3 are used to provide these resources, such as FlexE links, Time Sensitive Networking, Deterministic Networking etc. These partitions may be physical, or virtual so long as the SLA required by the higher layers is met. These resources provision the virtual networks with dedicated resources that they need. To get the required functionality there needs to be integration between these overlays and the underlay providing the physical resources. The network controller is used to create the virtual networks, to allocate the resources to each virtual network and to control and manage these networks. Dong, et al. Expires January 3, 2019 [Page 12] Internet-Draft VPN+ July 2018 The creation and allocation process needs to take a holistic view of the needs of all of its tenants, and to partition the resources accordingly. However within a virtual network these resources can if required be managed via a dynamic control plane. This provides the required scalability and isolation. 5.2. Multi-Point to Multi-point At a VPN level connections are frequently multi-point-to-multi-point (MP2MP). As far as such services are concerned the underlay is an abstract MP2MP medium. However when service guarantees are provided, such as with an enhanced VPN, each point to point path through the underlay needs to be specifically engineered to meet the required performance guarantees. 5.3. Candidate Underlay Technologies A VPN is a network created by applying a multiplexing technique to the underlying network (the underlay) in order to distinguish the traffic of one VPN from that of another. A VPN path that travels by other than the shortest path through the underlay normally requires state in the underlay to specify that path. State is normally applied to the underlay through the use of the RSVP Signaling protocol, or directly through the use of an SDN controller, although other techniques may emerge as this problem is studied. This state gets harder to manage as the number of VPN paths increases. Furthermore, as we increase the coupling between the underlay and the overlay to support the enhanced VPN service, this state will increase further. In an enhanced VPN different subsets of the underlay resources are dedicated to different VPNs. Any enhanced VPN solution thus needs tighter coupling with underlay than is the case with classical VPNs. We cannot for example share the tunnel between enhanced VPNs which require hard isolation. In the following sections we consider a number of candidate underlay solutions for proving the required VPN separation. o FlexE o Time Sensitive Networking o Deterministic Networking o Dedicated Queues Dong, et al. Expires January 3, 2019 [Page 13] Internet-Draft VPN+ July 2018 We then consider the problem of slice differentiation and resource representation. Candidate technologies are: o MPLS o MPLS-SR o Segment Routing over IPv6 (SRv6) 5.3.1. FlexE FlexE [FLEXE] is a method of creating a point-to-point Ethernet with a specific fixed bandwidth. FlexE supports the bonding of multiple links, which supports creating larger links out of multiple slower links in a more efficient way that traditional link aggregation. FlexE also supports the sub-rating of links, which allows an operator to only use a portion of a link. FlexE also supports the channelization of links, which allows one link to carry several lower-speed or sub-rated links from different sources. If different FlexE channels are used for different services, then no sharing is possible between the services. This in turn means that it is not possible to dynamically re-distribute unused bandwidth to lower priority services increasing the cost of operation of the network. FlexE can on the other hand be used to provide hard isolation between different tenants by providing hard isolation on an interface. The tenant can then use other methods to manage the relative priority of their own traffic. Methods of dynamically re-sizing FlexE channels and the implication for enhanced VPN are under study. 5.3.2. Dedicated Queues In an enhanced VPN providing multiple isolated virtual networks the conventional Diff-Serv based queuing system is insufficient for our purposes due to the limited number of queues which cannot differentiate between traffic of different VPNs and the range of service classes that each need to provide their tenants. This problem is particularly acute with an MPLS underlay due to the small number of traffic class services available. In order to address this problem and thus reduce the interference between VPNs, it is likely to be necessary to steer traffic of VPNs to dedicated input and output queues. Dong, et al. Expires January 3, 2019 [Page 14] Internet-Draft VPN+ July 2018 5.3.3. Time Sensitive Networking Time Sensitive Networking (TSN) is an IEEE project that is designing a method of carrying time sensitive information over Ethernet. As Ethernet this can obviously be tunneled over a Layer 3 network in a pseudowire. However the TSN payload would be opaque to the underlay and thus not treated specifically as time sensitive data. The preferred method of carrying TSN over a layer 3 network is through the use of deterministic networking as explained in the following section of this document. The mechanisms defined in TSN can be used to meet the requirements of time sensitive services of an enhanced VPN. 5.3.4. Deterministic Networking Deterministic Networking (DetNet) [I-D.ietf-detnet-architecture] is a technique being developed in the IETF to enhance the ability of layer 3 networks to deliver packets more reliably and with greater control over the delay. The design cannot use classical re-transmission techniques such as TCP since can add delay that is above the maximum tolerated by the applications. Even the delay improvements that are achieved with SCTP-PR are outside the bounds set by application demands. The approach is to pre-emptively send copies of the packet over various paths in the expectation that this minimizes the chance of all packets being lost, but to trim duplicate packets to prevent excessive flooding of the network and to prevent multiple packets being delivered to the destination. It also seeks to set an upper bound on latency. Note that it is not the goal to minimize latency, and the optimum upper bound paths may not be the minimum latency paths. DetNet is based on flows. It currently makes no comment on the underlay, and so at this stage must be assumed to use the base topology. To be of use in this application DetNet there needs to be a description of how to deal with the concept of flows within an enhanced VPN. How we use DetNet in a multi-tenant (VPN) network, and how to improve the scalability of DetNet in a multi-tenant (VPN) network is for further study. 5.3.5. MPLS Traffic Engineering (MPLS-TE) Normal MPLS runs on the base topology and has the concepts of reserving end to end bandwidth for an LSP, and of creating VPNs. VPN traffic can be run over dedicated RSVP-TE tunnels to provide reserved Dong, et al. Expires January 3, 2019 [Page 15] Internet-Draft VPN+ July 2018 bandwidth for a specific VPN connection. This is rarely deployed in practice due to scaling and management overhead concerns. 5.3.6. Segment Routing Segment Routing [I-D.ietf-spring-segment-routing] is a method that prepends instructions to packets at entry and sometimes at various points as it passes though the network. These instructions allow packets to be routed on paths other than the shortest path for various traffic engineering reasons. These paths can be strict or loose paths, depending on the compactness required of the instruction list and the degree of autonomy granted to the network (for example to support ECMP). With SR, a path needs to be dynamically created through a set of segments by simply specifying the Segment Identifiers (SIDs), i.e. instructions rooted at a particular point in the network. Thus if a path is to be provisioned from some ingress point A to some egress point B in the underlay, A is provided with the A..B SID list and instructions on how to identify the packets to which the SID list is to be prepended. By encoding the state in the packet, as is done in Segment Routing, per-path state is transitioned out of the network. A-------B-----E | | | | | | C-------D-----+ Figure 3: An SR Network Fragment Consider the network fragment shown in Figure 3. To send a packet from A to E via B, D & E: Node A prepends the ordered list of SIDs (B, D, E) to the packet and pushes the packet to B. SID list {B, D, E} can be used as a VPN path. Thus, to create a VPN, a set of SID Lists is created and provided to each ingress node of the VPN together with packet selection criteria. In this way it is possible to create a VPN with no state in the core. However this is at the expense of creating a larger packet with possible MTU and hardware restriction limits that need to be overcome. Note in the above if A and E support multiple VPN an additional VPN identifier will need to be added to the packet, but this is omitted from this text for simplicity. Dong, et al. Expires January 3, 2019 [Page 16] Internet-Draft VPN+ July 2018 A---P---B---S---E | | | | Q | | | | C---R---D-------+ Figure 4: Another SR Network Fragment Consider a further network fragment shown in Figure 4, and further consider VPN A+D+E. A has lists: {P, B, Q, D}, {P, B, S, E} D has lists: {Q, B, P, A}, {E} E has lists: {S, B, P, A}, {D} To create a new VPN C+D+B the following list are introduced: C lists: {R, D}, {A, P, B} D lists: {R, C}, {Q, B} B lists: {Q, D}, {P, A, C} Thus VPN C+D+B was created without touching the settings of the core routers, indeed it is possible to add endpoints to the VPNs, and move the paths around simply by providing new lists to the affected endpoints. There are a number of limitations in SR as it is currently defined that limit its applicability to enhanced VPNs: o Segments are shared between different VPNs, o There is no reservation of bandwidth, o There is limited differentiation in the data plane. Thus some extensions to SR are needed to provide isolation between different enhanced VPNs. This can be achieved by including a finer granularity of state in the core in anticipation of its future use by authorized services. We therefore need to evaluate the balance between this additional state and the performance delivered by the network. Both MPLS Segment Routing and SRv6 Segment Routing are candidate technologies for enhanced VPN. With current segment routing, the instructions are used to specify the nodes and links to be traversed. However, in order to achieve the required isolation between different services, new instructions Dong, et al. Expires January 3, 2019 [Page 17] Internet-Draft VPN+ July 2018 can be created which can be prepended to a packet to steer it through specific dedicated network resources and functions, e.g. links, queues, processors, services etc. Clearly we can use traditional constructs to create a VPN, but there are advantages to the use of other constructs such as Segment Routing (SR) in the creation of virtual networks with enhanced properties. Traditionally a traffic engineered path operates with a granularity of a link with hints about priority provided through the use of the traffic class field in the header. However to achieve the latency and isolation characteristics that are sought by the enhanced VPN users, steering packets through specific queues and resources will likely be required. The extent to which these needs can be satisfied through existing QoS mechanisms is to be determined. What is clear is that a fine control of which services wait for which, with a fine granularity of queue management policy is needed. Note that the concept of a queue is a useful abstraction for many types of underlay mechanism that may be used to provide enhanced isolation and latency support. From the perspective of the control plane and from the perspective of the segment routing the method of steering a packet to a queue that provides the required properties is a universal construct. How the queue satisfies the requirement is implementation specific and is transparent to the control plane and data plane mechanisms used. Thus for example a FlexE channel, or time sensitive networking packet scheduling slot are abstracted to the same concept and bound to the data plane in a common manner. We can introduce the specification of finer, deterministic, granularity to path selection through extensions to traditional path construction techniques such as RSVP-TE and MPLS-TP. We can also introduce it by specifying the queues through an SR instruction list. Thus new SR instructions may be created to specify not only which resources are traversed, but in some cases how they are traversed. For example, it may be possible to specify not only the queue to be used but the policy to be applied when enqueuing and dequeuing. This concept can be further generalized, since as well as queuing to the output port of a router, it is possible to queue to any resource, for example: o A network processor unit (NPU) o A Central Processing Unit (CPU) Core o A Look-up engine such as TCAMs Dong, et al. Expires January 3, 2019 [Page 18] Internet-Draft VPN+ July 2018 5.4. Control Plane Considerations It is expected that enhanced VPN would be based on a hybrid control mechanism, which takes advantage of the logically centralized controller for on-demand provisioning and global optimization, whilst still relies on distributed control plane to provide scalability, high reliability, fast reaction, automatic failure recovery etc. Extension and optimization to the distributed control plane is needed to support the enhanced properties of VPN+. Where SR is used as a the data-plane construct it needs to be noted that it does not have the capability of reserving resources along the path nor do its currently specified distributed control plane (the link state routing protocols). An SDN controller can clearly do this, from the controllers point of view, and no resource reservation is done on the device. Thus if a distributed control plane is needed either in place of an SDN controller or as an assistant to it, the design of the control system needs to ensure that resources are uniquely allocated to the correct service, and no allocated to multiple services causing unintended resource conflict. This needs further study. On the other hand an advantage of using an SR approach is that it provides a way of efficiently binding the network underlay and the enhanced VPN overlay. With a technology such as RSVP-TE LSPs, each virtual path in the VPN is bound to the underlay with a dedicated TE- LSP. RSVP-TE could be enhanced to bind the VPN to specific resources within the underlay, but as noted elsewhere in this document there are concerns as to the scalability of this approach. With an SR- based approach to resource reservation (per-slice reservation), it is straightforward to create dedicated SR network slices, and the VPN can be bound to a particular SR network slice. 5.5. Application Specific Network Types Although a lot of the traffic that will be carried over the enhanced VPN will likely be IPv4 or IPv6, the design has to be capable of carrying other traffic types. In particular the design SHOULD be capable of carrying Ethernet traffic. This is easily accomplished through the various pseudowire (PW) techniques [RFC3985]. Where the underlay is MPLS Ethernet can be carried over the enhanced VPN encapsulated according to the method specified in [RFC4448]. Where the underlay is IP Layer Two Tunneling Protocol - Version 3 (L2TPv3) [RFC3931] can be used with Ethernet traffic carried according to [RFC4719]. Encapsulations have been defined for most of the common layer two type for both PW over MPLS and for L2TPv3. Dong, et al. Expires January 3, 2019 [Page 19] Internet-Draft VPN+ July 2018 5.6. Integration with Service Functions There is a significant overlap between the problem of routing a packet though a set of network resources and the problem of routing a packet through a set of compute resources. Service Function Chain technology is designed to forward a packet through a set of compute resources. A future version of this document will discuss this further. 6. Scalability Considerations For a packet to transit a network, other than on a best effort, shortest path basis, it is necessary to introduce additional state, either in the packet, or in the network of some combination of both. There are at least three ways of doing this: o Introduce the complete state into the packet. That is how SR does this, and this allows the controller to specify the precise series of forwarding and processing instructions that will happen to the packet as it transits the network. The cost of this is an increase in the packet header size. The cost is also that systems will have capabilities enabled in case they are called upon by a service. This is a type of latent state, and increases as we more precisely specify the path and resources that need to be exclusively available to a VPN. o Introduce the state to the network. This is normally done by creating a path using RSVP-TE, which can be extended to introduce any element that needs to be specified along the path, for example explicitly specifying queuing policy. It is of course possible to use other methods to introduce path state, such as via a Software Defined Network (SDN) controller, or possibly by modifying a routing protocol. With this approach there is state per path per path characteristic that needs to be maintained over its life- cycle. This is more state than is needed using SR, but the packet are shorter. o Provide a hybrid approach based on using binding SIDs to create path fragments, and bind them together with SR. Dynamic creation of a VPN path using SR requires less state maintenance in the network core at the expense of larger VPN headers on the packet. The scaling properties will reduce roughly from a function of (N/2)^2 to a function of N, where N is the VPN path length in intervention points (hops plus network functions). Reducing the state in the network is important to VPN+, as VPN+ Dong, et al. Expires January 3, 2019 [Page 20] Internet-Draft VPN+ July 2018 requires the overlay to be more closely integrated with the underlay than with traditional VPNs. This tighter coupling would normally mean that significant state needed to be created and maintained in the core. However, a segment routed approach allows much of this state to be spread amongst the network ingress nodes, and transiently carried in the packets as SIDs. These approaches are for further study. 6.1. Maximum Stack Depth One of the challenges with SR is the stack depth that nodes are able to impose on packets. This leads to a difficult balance between adding state to the network and minimizing stack depth, or minimizing state and increasing the stack depth. 6.2. RSVP Scalability The traditional method of creating a resource allocated path through an MPLS network is to use the RSVP protocol. However there have been concerns that this requires significant continuous state maintenance in the network. There are ongoing works to improve the scalability of RSVP-TE LSPs in the control plane [I-D.ietf-teas-rsvp-te-scaling-rec]. This will be considered further in a future version of this document. There is also concern at the scalability of the forwarder footprint of RSVP as the number of paths through an LSR grows [I-D.sitaraman-mpls-rsvp-shared-labels] proposes to address this by employing SR within a tunnel established by RSVP-TE. This work will be considered in a future version of this document. 7. OAM and Instrumentation A study of OAM in SR networks has been documented in [I-D.ietf-spring-oam-usecase]. The enhanced VPN OAM design needs to consider the following requirements: o Instrumentation of the underlay so that the network operator can be sure that the resources committed to a tenant are operating correctly and delivering the required performance. o Instrumentation of the overlay by the tenant. This is likely to be transparent to the network operator and to use existing methods. Particular consideration needs to be given to the need Dong, et al. Expires January 3, 2019 [Page 21] Internet-Draft VPN+ July 2018 to verify the isolation and the various committed performance characteristics. o Instrumentation of the overlay by the network provider to proactively demonstrate that the committed performance is being delivered. This needs to be done in a non-intrusive manner, particularly when the tenant is deploying a performance sensitive application o Verification of the conformity of the path to the service requirement. This may need to be done as part of a commissioning test. These issues will be discussed in a future version of this document. 8. Enhanced Resiliency Each enhanced VPN, of necessity, has a life-cycle, and needs modification during deployment as the needs of its user change. Additionally as the network as a whole evolves there will need to be garbage collection performed to consolidate resources into usable quanta. Systems in which the path is imposed such as SR, or some form of explicit routing tend to do well in these applications because it is possible to perform an atomic transition from one path to another. However implementations and the monitoring protocols need to make sure that the new path is up before traffic is transitioned to it. There are however two manifestations of the latency problem that are for further study in any of these approaches: o The problem of packets overtaking one and other if a path latency reduces during a transition. o The problem of the latency transient in either direction as a path migrates. There is also the matter of what happens during failure in the underlay infrastructure. Fast reroute is one approach, but that still produces a transient loss with a normal goal of rectifying this within 50ms. An alternative is some form of N+1 delivery such as has been used for many years to support protection from service disruption. This may be taken to a different level using the techniques proposed by the IETF deterministic network work with multiple in-network replication and the culling of later packets. Dong, et al. Expires January 3, 2019 [Page 22] Internet-Draft VPN+ July 2018 In addition to the approach used to protect high priority packets, consideration has to be given to the impact of best effort traffic on the high priority packets during a transient. Specifically if a conventional re-convergence process is used there will inevitably be micro-loops and whilst some form of explicit routing will protect the high priority traffic, lower priority traffic on best effort shortest paths will micro-loop without the use of a loop prevention technology. To provide the highest quality of service to high priority traffic, either this traffic must be shielded from the micro-loops, or micro-loops must be prevented. 9. Security Considerations All types of virtual network require special consideration to be given to the isolation between the tenants. However in an enhanced virtual network service hard isolation needs to be considered. If a service requires a specific latency then it can be damaged by simply delaying the packet through the activities of another tenant. In a network with virtual functions, depriving a function used by another tenant of compute resources can be just as damaging as delaying transmission of a packet in the network. 10. IANA Considerations There are no requested IANA actions. 11. References 11.1. Normative References [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997, . 11.2. Informative References [FLEXE] "Flex Ethernet Implementation Agreement", March 2016, . [I-D.ietf-detnet-architecture] Finn, N., Thubert, P., Varga, B., and J. Farkas, "Deterministic Networking Architecture", draft-ietf- detnet-architecture-05 (work in progress), May 2018. Dong, et al. Expires January 3, 2019 [Page 23] Internet-Draft VPN+ July 2018 [I-D.ietf-detnet-dp-sol] Korhonen, J., Andersson, L., Jiang, Y., Finn, N., Varga, B., Farkas, J., Bernardos, C., Mizrahi, T., and L. Berger, "DetNet Data Plane Encapsulation", draft-ietf-detnet-dp- sol-04 (work in progress), March 2018. [I-D.ietf-spring-oam-usecase] Geib, R., Filsfils, C., Pignataro, C., and N. Kumar, "A Scalable and Topology-Aware MPLS Dataplane Monitoring System", draft-ietf-spring-oam-usecase-10 (work in progress), December 2017. [I-D.ietf-spring-segment-routing] Filsfils, C., Previdi, S., Ginsberg, L., Decraene, B., Litkowski, S., and R. Shakir, "Segment Routing Architecture", draft-ietf-spring-segment-routing-15 (work in progress), January 2018. [I-D.ietf-teas-rsvp-te-scaling-rec] Beeram, V., Minei, I., Shakir, R., Pacella, D., and T. Saad, "Techniques to Improve the Scalability of RSVP Traffic Engineering Deployments", draft-ietf-teas-rsvp-te- scaling-rec-09 (work in progress), February 2018. [I-D.sitaraman-mpls-rsvp-shared-labels] Sitaraman, H., Beeram, V., Parikh, T., and T. Saad, "Signaling RSVP-TE tunnels on a shared MPLS forwarding plane", draft-sitaraman-mpls-rsvp-shared-labels-03 (work in progress), December 2017. [RFC2764] Gleeson, B., Lin, A., Heinanen, J., Armitage, G., and A. Malis, "A Framework for IP Based Virtual Private Networks", RFC 2764, DOI 10.17487/RFC2764, February 2000, . [RFC3931] Lau, J., Ed., Townsley, M., Ed., and I. Goyret, Ed., "Layer Two Tunneling Protocol - Version 3 (L2TPv3)", RFC 3931, DOI 10.17487/RFC3931, March 2005, . [RFC3985] Bryant, S., Ed. and P. Pate, Ed., "Pseudo Wire Emulation Edge-to-Edge (PWE3) Architecture", RFC 3985, DOI 10.17487/RFC3985, March 2005, . [RFC4364] Rosen, E. and Y. Rekhter, "BGP/MPLS IP Virtual Private Networks (VPNs)", RFC 4364, DOI 10.17487/RFC4364, February 2006, . Dong, et al. Expires January 3, 2019 [Page 24] Internet-Draft VPN+ July 2018 [RFC4448] Martini, L., Ed., Rosen, E., El-Aawar, N., and G. Heron, "Encapsulation Methods for Transport of Ethernet over MPLS Networks", RFC 4448, DOI 10.17487/RFC4448, April 2006, . [RFC4664] Andersson, L., Ed. and E. Rosen, Ed., "Framework for Layer 2 Virtual Private Networks (L2VPNs)", RFC 4664, DOI 10.17487/RFC4664, September 2006, . [RFC4719] Aggarwal, R., Ed., Townsley, M., Ed., and M. Dos Santos, Ed., "Transport of Ethernet Frames over Layer 2 Tunneling Protocol Version 3 (L2TPv3)", RFC 4719, DOI 10.17487/RFC4719, November 2006, . [RFC7209] Sajassi, A., Aggarwal, R., Uttaro, J., Bitar, N., Henderickx, W., and A. Isaac, "Requirements for Ethernet VPN (EVPN)", RFC 7209, DOI 10.17487/RFC7209, May 2014, . Authors' Addresses Jie Dong Huawei Email: jie.dong@huawei.com Stewart Bryant Huawei Email: stewart.bryant@gmail.com Zhenqiang Li China Mobile Email: lizhenqiang@chinamobile.com Takuya Miyasaka KDDI Corporation Email: ta-miyasaka@kddi.com Dong, et al. Expires January 3, 2019 [Page 25]