IDR Working Group S. Hares Internet-Draft Hickory Hill Consulting Intended status: Informational July 14, 2020 Expires: January 15, 2021 BGP IPSEC VPNs - A Solution Analysis draft-hares-idr-bgp-ipsec-analysis-00 Abstract This draft describes problems with IGP convergence time in some IPRAN networks that use a physical topology of grid backbones that connect rings of routers. Part of these IPRAN network topologies exist in data centers with sufficient power and interconnections, but some network equipment sits in remote sites impacted by power loss. In some geographic areas these remote sites may be subject to rolling blackouts. These rolling power blackouts could cause multiple simultaneous node and link failures. In these remote networks with blackouts, it is often critical that the IPRAN phone network re- converge quickly. The IGP running in these networks may run in a single level of the IGP. This document seeks to briefly describe these problems to determine if the emerging IGP technologies (flexible algorithms, dynamic flooding, layers of hierarchy in IGPs) can be applied to help reduce convergence times. It also seeks to determine if the improvements of these algorithms or the IP-Fast re-route algorithms are thwarted by the failure of multiple link and nodes. Status of This Memo This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet- Drafts is at https://datatracker.ietf.org/drafts/current/. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." This Internet-Draft will expire on January 15, 2021. Hares Expires January 15, 2021 [Page 1] Internet-Draft BGP-IPSEC-ANALYSIS July 2020 Copyright Notice Copyright (c) 2020 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License. Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 1.1. Requirements Language . . . . . . . . . . . . . . . . . . 3 1.2. IPSEC deployments . . . . . . . . . . . . . . . . . . . . 3 1.3. History of BGP passing Tunnel Endpoints . . . . . . . . . 4 1.4. Overview of proposals . . . . . . . . . . . . . . . . . . 5 1.5. Method of analysis . . . . . . . . . . . . . . . . . . . 7 2. Security and Privacy . . . . . . . . . . . . . . . . . . . . 8 2.1. Option 1 - Configuration Plus BGP Routes with Tunnel SA IDs . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 2.2. Option 2- BGP passes client routes with SA-ID plus NLRI passes underlay SAs . . . . . . . . . . . . . . . . . . . 13 2.3. Option 3: Secure EVPN (client routes + SA information) . 14 2.4. comparison of security issues . . . . . . . . . . . . . . 16 3. Manageability and Scaling . . . . . . . . . . . . . . . . . . 16 3.1. Configuration sizes - used for theoretical comparison . . 17 3.2. BGP Route sizes for theoretical comparison . . . . . . . 18 3.2.1. Size of Tunnel encapsulation attribute with 1 SA per tunnel endpoint . . . . . . . . . . . . . . . . . . . 18 3.2.2. Size of Tunnel encapsulation attribute with 10 SAs per tunnel . . . . . . . . . . . . . . . . . . . . . 19 3.3. Network Scenario 1 . . . . . . . . . . . . . . . . . . . 19 3.4. Network scenario 2 . . . . . . . . . . . . . . . . . . . 19 3.5. Scaling Memory sizes . . . . . . . . . . . . . . . . . . 19 4. Key differences between the options . . . . . . . . . . . . . 19 5. Processing of BGP routes . . . . . . . . . . . . . . . . . . 19 6. Future Issues - SBGP and Secure IPSEC VPNs . . . . . . . . . 19 7. Security Considerations . . . . . . . . . . . . . . . . . . . 19 8. IANA considerations . . . . . . . . . . . . . . . . . . . . . 20 9. References . . . . . . . . . . . . . . . . . . . . . . . . . 20 9.1. Normative References . . . . . . . . . . . . . . . . . . 20 9.2. Informative References . . . . . . . . . . . . . . . . . 20 Hares Expires January 15, 2021 [Page 2] Internet-Draft BGP-IPSEC-ANALYSIS July 2020 Author's Address . . . . . . . . . . . . . . . . . . . . . . . . 21 1. Introduction This document analyzes three solutions for using a BGP based approach on SDWAN edge nodes to establish secure IPSec tunnels for the overlay routes distribution. The solutions are: o [I-D.hujun-idr-bgp-ipsec] o [I-D.dunbar-idr-sdwan-edge-discovery] o [I-D.sajassi-bess-secure-evpn] These three drafts propose an IPsec related tunnel type for an augmentation of [I-D.ietf-idr-tunnel-encaps] to support IPsec tunnels. At IETF 105, IDR and BESS WG held a session to discuss the security issues in these emerging drafts with security directorate people. The security people provided excellent feedback on on how to approach security, privacy, and scaling. The IDR/BESS working members provided provided feedback on the scaling and concepts. As a result of this session, it became evident that each proposal has started with a unique user scenario. Therefore, this draft simply reviews the technical qualities in terms of: 1) the security and privacy of each technology, and 2) how the technology is managed (manageability) and how the technology scales. The purpose of this draft to grow our joint understanding of each proposed IPSec tunnel type so that IDR and BESS can make informed decisions. It is non-goal to determine which pf these 3 solutions fits a particular use case using VPN using BGP to pass IPsec tunnel end points. 1.1. Requirements Language The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 [RFC2119]. 1.2. IPSEC deployments Secure VPNs based on IPsec tunnels began appearing around 2000. IPsec tunnels were used for secure link transport. The IPSEC VPNs utilized IPsec tunnels over physical links or underlay networks to virtual links for these VPNs. These IPsec tunnels can be created by configuring routers with tunnel endpoints and setting up security associations for these tunnels. Automated processes can use IETF Hares Expires January 15, 2021 [Page 3] Internet-Draft BGP-IPSEC-ANALYSIS July 2020 network management protocols (NETCONF and RESTCONF) to configure Yang modules in the routers to set up these tunnels. Enterprise VPNs were created from these secure tunnels. EVPNs and SDWANs have also deployed VPNs using IPSEC. The BGP client routes which use the tunnel as a pathway also distribute pathway information (endpoint, inner encapsulation, outer encapsulation) via BGP with tunnel attribute [I-D.ietf-idr-tunnel-encaps] For IPsec tunnels, there are three approaches to what security association information is included with the tunnel attribute. (See [I-D.hujun-idr-bgp-ipsec], [I-D.dunbar-idr-sdwan-edge-discovery] and [I-D.sajassi-bess-secure-evpn]. One of the newly defined user scenarios for the secure VPN is the SDWAN. [ [I-D.ietf-rtgwg-net2cloud-problem-statement] describes the problems faced by SDWAN. [I-D.ietf-rtgwg-net2cloud-gap-analysis] describes the gaps in IETF technology for this use case. [I-D.dunbar-bess-bgp-sdwan-usage] describes how BGP is used as control plane to setup the SDWAN networks for various SDWAN use cases. SDWAN overlay networks can run over physical forwarding by a wide variety of underlay networks. SDWAN is one of the more recent developments in IPsec based VPNS created by an SDN controller. The author welcomes additional information on other use cases. 1.3. History of BGP passing Tunnel Endpoints [RFC5512] defined SAFI to pass tunnel endpoint encapsulation information. However, many operators and vendors preferred to pass this information in a BGP attribute. [I-D.ietf-idr-tunnel-encaps] defines a BGP attribute for tunnels to replace [RFC5512] functionality, but does not address how to use RFC5566 without the encapsulation SAFI. EVPN [RFC8365] also defined tunnel types for encapsulation. The tunnel types registered with IANA (www.IANA.org) list the following tunnel types from [RFC5512], [RFC5566], and [RFC8365]: o L2TPv3 over IP [RFC5512] [value 1], o GRE [RFC5512] [value 2] o Transmit tunnel endpoint [RFC5566][value 3] o IPsec in tunnel mode [RFC5566] [value 4] o IP in IP tunnel with IPsec Transport mode [RFC5566][value 5] Hares Expires January 15, 2021 [Page 4] Internet-Draft BGP-IPSEC-ANALYSIS July 2020 o MPLS-in-IP tunnel with IPsec Transport mode [RFC5566][value 6] o IP in IP [RFC5566] [value 7] o VXLAN encapsulation [RFC8365][value 8] o NVGRE encapsulation [RFC8365][value 9] o MPLS Encapsulation [RFC8365][value 10] o MPLS in GPE encapsulation [RFC8365] [value 11] o VXLAN GPE encapsulation [RFC8365] [value 12] [I-D.ietf-idr-tunnel-encaps] has been created to address deficiencies in RFC5512 [RFC5512]. These deficiencies include: operational costs of using SAFI for tunnel identifiers, inability to specify egress endpoint of tunnel, unclear prefix-tunnel association, and inability to specify inner/outer encapsulation. [I-D.ietf-idr-tunnel-encaps] defines new Sub-TLVs to support inner and outer encapsulation for these encapsulation types, and will become the main reference for these tunnel types. RFC5566 [RFC5566] defined the IP Tunnel Authenticator Sub-TLV for use in the SAFI, but these recent proposals have suggested different alternatives for replacing the Tunnel Authenticator function. 1.4. Overview of proposals This section provides a technical overview of the 3 proposals [I-D.hujun-idr-bgp-ipsec], [I-D.dunbar-idr-sdwan-edge-discovery], and [I-D.sajassi-bess-secure-evpn]. [I-D.hujun-idr-bgp-ipsec] proposes 3 new Sub-TLVs: local/remote Prefix Sub-TLV, Public Routing Instance Sub-TLV, and IPSec Configuration Sub-TLV (IPsec-Config). The local/remote prefix Sub- TLV will not be discussed here as it does not clearly align to [I-D.ietf-idr-tunnel-encaps]. The optional Public Routing Instance (PRI) is used instead of a route target community so that local policy can filter routes for a specific community. This feature provides the same feature as a Route target for a pre-configured set of PRIs. The IPsec Configuration Sub-TLV contains 4 octet opaque value to link the tunnel to the Tunnel Authentication entry found in a security association table on the local node. This table will need to include which tunnel endpoints this security association is valid for. This Hares Expires January 15, 2021 [Page 5] Internet-Draft BGP-IPSEC-ANALYSIS July 2020 analysis assumes the IETF protocols NETCONF RESTCONF configure a YANG module that has these security associations. [I-D.dunbar-idr-sdwan-edge-discovery] proposes UPDATEs from client routers to include the IPsec SA identifiers (ID) to reference the IPsec SA attributes being advertised by separate Underlay Property BGP UPDATE messages. The security association table is built dynamically from the information passed in these Underlay Property BGP Updates plus some local configuration. If a client route can be encrypted by multiple IPsec SAs, then multiple IPsec SA IDs are included in the Tunnel-Encaps Path attribute for the client route. This draft proposes two new Sub-TLVs: IPsec-SA-ID and IPsec-SA-Group. The IPsec-SA-ID is similar to [I-D.hujun-idr-bgp-ipsec] IPSec Config Sub-TLV passing a 2 octet pointer to into a security association table. IPsec-SA-Group Sub-TLV optimizes passing the same information when multiple IPsec SAs with the same inner encapsulation header. [I-D.dunbar-idr-sdwan-edge-discovery] proposes underlay tunnel topology information for SDWAN in BGP UPDATEs. The information is passed in a combination of NLRI with an SAFI=74 (SDWAN SAFI) and a Tunnel Encapsulation attribute with tunnel type being SDWAN-Underlay. Security association information for the tunnels in this underlay will be passed in the Tunnel Attribute using in the SDWAN Underlay tunnel type. This new tunnel type which will support the current tunnel Sub-TLV plus the newly proposed IPSec SA Sub-TLV(s). There are two types of IPsec SA Sub-TLVs proposed by [I-D.dunbar-idr-sdwan-edge-discovery], one is for general purpose deployment which requires a full-set of Security Association, including Nonce, Public Key, Proposal and Transform Sub-TLVs in the SDWAN SAFI NLRI (SA-TYPE =2). Another type is for simple deployment which only needs one simple IPsec SA Sub-TLV included (SA-TYPE=1). In addition, it can also include other optional Sub-TLVs like NAT, WAN Port, Geo-location with the SDWAN SAFI route. [I-D.sajassi-bess-secure-evpn] proposes defines 2 new tunnel types (ESP-Transport and ESP-in-UDP-Transport) and 3 new Sub-TLVS (DIM Sub- TLV, Key-Exchange Sub-TLV, and proposal Sub-TLV) for these new tunnel types. The new Sub-TLVs pass information regarding security associations. The DIM Sub-TLV is required to be supported for the two new tunnel types. As noted above, the SDWAN-WAN-Underlay tunnel type from [I-D.dunbar-idr-sdwan-edge-discovery] supports equivalent features to IPsec-SA, Public-key, and SA-Transforms. [I-D.dunbar-idr-sdwan-edge-discovery] and [I-D.sajassi-bess-secure-evpn] differ in the information included in the client routes. [I-D.sajassi-bess-secure-evpn] attaches all the IPsec SA information to the actual client routes, whereas the Hares Expires January 15, 2021 [Page 6] Internet-Draft BGP-IPSEC-ANALYSIS July 2020 [I-D.dunbar-idr-sdwan-edge-discovery] only includes the IPsec SA IDs for the client routes. The IPsec SA IDs used by [I-D.dunbar-idr-sdwan-edge-discovery] reference (point) to the SA- Information which is advertised separately. All the SA-Information are attached to routes which describe the SDWAN underlay, such as WAN Ports or Node address. [I-D.sajassi-bess-secure-evpn] supports tunnel types of ESP-Transport and ESP-in-UDP transport, but not traditional IPsec tunnel types (IPsec in tunnel mode, IP in IP tunnel with IPsec transport, MPLS-in- IP tunnel with transport mode). The use of the new tunnel type could be used in a similar fashion to [I-D.dunbar-idr-sdwan-edge-discovery] to pass SA-information regarding the underlay. [I-D.sajassi-bess-secure-evpn] seems to point to passing client routes upon a rekeying request. This method will increase the amount of BGP traffic passed in the crash or initial start-up in the tunnel encapsulation attribute. Since [I-D.sajassi-bess-secure-evpn] draft has not recently been updated, it is not clear if the recent changes to [I-D.ietf-idr-tunnel-encaps] are reflected in this draft. [I-D.sajassi-bess-secure-evpn] depends on [I-D.carrel-ipsecme-controller-ike] which received many security comments at IETF 105. Therefore, the author has analyzed [I-D.sajassi-bess-secure-evpn] solutions based on the following assumptions: o ESP-Transport and ESP-in-UDP would have been aligned with the latest version of the [I-D.ietf-idr-tunnel-encaps], o Only the DIM Sub-TLV is required to be sent during initialization, PE rekey requests, routing periodic updates, and node restarts (crash/load) for shared security controller policies. o The multiple policy environments may increase the size of Tunnel Encapsulation attribute as transforms and transform attributes are sent. 1.5. Method of analysis The things matter to the network operator of IP-SEC VPN in SDWAN: security, manageability, scaling, and privacy. Each deployment of an IPSec VPN may combine different underlay networks with different challenges to security, manageability, scaling and privacy. This analysis compares the basic technologies of these proposals in terms of two groups of features: 1) security and privacy, and 2) manageability and scaling. This analysis drafts looks at each solution based on the strengths are weaknesses of each type. Hares Expires January 15, 2021 [Page 7] Internet-Draft BGP-IPSEC-ANALYSIS July 2020 Analyzing scaling can either be done at the 50,000 foot level or in excruciating detail. This analysis will be at the 50,000 foot level using two example scenarios (small and very large) Scenario 1: The 3 Route Reflectors (RR) each have 5 client routers per router reflector. The client routers have a potential of 5 tunnels with 1 security association per tunnel. Each client router has 200 routes. The total number of configured tunnels is 20 tunnels per RR cluster and the total number of client routes is 3000. Diagram 1 in section 1.3 showed this simple topology for these route reflectors. Scenario 2: The 3 Route Reflectors (RR) each have 10,000 client routers. Each client router supports 100 tunnels, 10K routes, and 10 security associations per tunnel. Each Route Reflectors will receive from its client routers a total of 100 million client routes with 1 million tunnels client tunnels (100*10K client routers), and 10 million security associations. The totals for all 3 RR may be up to 3 times this level (300 Million client routes, 3 million tunnels, and 10 million security associations), but it is likely the RR will contain some redundancy. Our scenario focuses on the challenges within a single RR clustr. The BGP scaling in these two scenarios contrast small IPsec VPNs and very large IPsec VPNs. BGP routing products handle route distribution of over 100 Million routes so this scaling is well within the range of the BGP products. 2. Security and Privacy During an initial security review of this information, Ben Kaduk made the following comments: "First off, when we start to get IPsec configuration via BGP, it's helpful to think of what other information we get in the same way, and to analyze the effects of misconfiguration or malicious configuration both on IPsec and the broader system. For example, if we are getting NLRI from BGP, then a misconfiguration that gives us parameters that are incompatible with a peer's is not causing particularly new harm, since we could just as easily be told that peer is unreachable and we wouldn't try to talk to them anyway. On the other hand, we could be given configuration to use computationally expensive algorithms which would increase the DOS risk in a way that may not (or may!) be already possible. " (email to IDR and BESS WGs after IETF 105)." The security analysis in this draft assumes that the route distribution for BGP routes is done via a mesh of Route Reflectors Hares Expires January 15, 2021 [Page 8] Internet-Draft BGP-IPSEC-ANALYSIS July 2020 which have route reflector clients associated. The IBGP mesh of route reflectors within a domain is assumed to run over secure transport links (such as TLS). If privacy is an issue for BGP route distribution, the TLS encrypts/decrypts the data in the IBGP mesh. If a single AS IBGP mesh of Route Reflectors connects to another AS, the EBGP connection is also over a secure transport (such as TLS). [Full mesh topology within a IBGP cloud LAN used to simply drawing) TLS =============================== | | | RR1---------- RR2 ------------ RR3 TLS / | \ / | \ / | \ C1 C2 Cn C1 C2 Cn C1 C2 Cn Route Reflector Topology This security topology has the same transport link secure topology as the NETCONF/RESTCONF security of set of NETCONF/RESTCONF servers. The example NETCONF topology is below. Back-end configuration database TLS =============================== | | | Netconf Netconf Netconf Client1 -------Client2 ------- Client 3 TLS / | \ / | \ / | \ NS1 NS2 NSn NS1 NS2 NSn NS1 NS2 NSn NETCONF Topology The security aspects of using network management protocols NETCONF or RESTCONF servers to control IPsec SA distribution has been considered as part of SDN-based IPSEC flow consideration (see [RFC8192]). The user data traffic runs over secure IP tunnels whether the configuration is via NETCONF or BGP RR mesh. Figure 3 below shows a Route Reflector topology with BGP sessions in a RR mesh and client traffic over IPSEC tunnels. Figure 3 - pending [Editor's note: Large scale topology needs svg drawings] Figure 4 shows an equivalent topology can be used with NETCONF client-servers. Notice that the NETCONF topology requires a common database behind the network clients to provide the correct Hares Expires January 15, 2021 [Page 9] Internet-Draft BGP-IPSEC-ANALYSIS July 2020 configurations. If the NETCONF servers work across administrative domains, a shared database must be developed to provide the appropriate information given the correct policy filters and access (NETCONF NACM). Diagram 4 - pending [Editor's note: Large scale BGP topologies need .svg ] There are two parts of the security for control plane traffic: link security and data security. Link security entails making sure the data is secure as the data is transmitted across the link. Link security in the NETCONF configuration cloud shown in diagram 2 entails making sure the configuration data passes across each of the links. The links from the configuration database (DB in diagram) to each client server must be secured via TLS. The links from the netconf client to the netconf server on the node must be secured via TLS. Data security in the NETCONF configuration cloud entails making sure the data from the configuration data base (DB) travels through the netconf clients (e.g. netconf client1) to the node's netconf server (e.g. NS1) without change. Data privacy for configuration pathways traffic entails making sure no other party snoops on the data while it travels from configuration database (DB) to the netconf client to the server. NETCONF client/servers are designed to operate in a single administrative domain. NETCONF client/servers require additional policy filters and checks to run between multiple administrative domains. The Database to NETCONF client link is not standardized by IETF. Correspondingly, the link security in a BGP RR mesh requires that the data is secure across any link in the BGP RR mesh (RR to/from any RR client or within the RR mesh). Data security for control plane traffic entails ensuring that the data placed into the BGP mesh (from RR clients or RR) arrives at the appropriate destination without change. BGP does not provide data security on control plane traffic as the data may be modified via policy at each node. SBGP does provide data security. For most networks, physical security of each node and link security is considered sufficient. The data written into a node using configuration data writes (NETCONF edit-config or RESTCONF PUT/POST) uses the NETCONF client to write to the network server on the client router. The data which is sent from the route-reflector to the RR client routers is sent via BGP, saved in the BGP RIBs, and installed in the router. Hares Expires January 15, 2021 [Page 10] Internet-Draft BGP-IPSEC-ANALYSIS July 2020 The network management protocols (NETCONF/RESTCONF) and BGP both have access policy that controls the data is written into the router. The error handling for incorrect data is different between these two network management protocols (NETCONF/RESTCONF) and BGP. If netconf tries save data with the wrong format, it will provide an error information in the response (rpc-error). The BGP error handling of a malformed Tunnel Attribute in the TLV simply ignores the tunnel attribute while accepting the route. The common resolution is that NETCONF, RESTCONF, and BGP write error information to a local log. Error reports can be tracked in a Yang module which can be automatically streamed to central controller via an alternate channel NETCONF/RESTCONF logging channel. [Editorial note: Should I give the details of the NETCONF/RESTCONF logging channel?] The SDWAN environment or any VPN that uses BGP to transfer tunnel configuration and security association information SHOULD consider augmenting the base BGP Yang model with BGP tunnel encapsulation Yang model for all tunnel types including IPSEC. The logging features or the reporting of the BGP errors can be combined with any error reporting on NETCONF/RESTCONF configuration or any operational state from the tunnel interface. The NETCONF/RESTCONF logging feature providing throttling so any type of error reporting can be configured to be manageable within a large network. This implies the SDWAN environment should design a BGP Yang model augmenting the BGP base model for the BGP tunnel encapsulation functionality for all tunnel types including IPSEC AND provides logging features the reporting can be the same as NETCONF/RESTCONF. Given Ben's rule of thumb, the transmission of the routes, the tunnel end points, and the link to the security association information via the BGP protocol does not cause extra security risks. The next 3 sections summarize the security and privacy of each technology in terms of: o what is distributed via netconf, o what is distributed via BGP, o link security provided, o data security provided, o suggested Yang models that will augment error handling, Hares Expires January 15, 2021 [Page 11] Internet-Draft BGP-IPSEC-ANALYSIS July 2020 o privacy issues. 2.1. Option 1 - Configuration Plus BGP Routes with Tunnel SA IDs Document: draft-hujun-idr-bgp-ipsec-02.txt [I-D.hujun-idr-bgp-ipsec] What is distributed via NETCONF: Tunnel Configuration, Security associations, and the mapping of the security association to a tunnel end point (identified y IPsec tunnel identifier), and SA (security association) for the each IPSec tunnel. What is distributed via BGP: Client Routes with IPsec Sub-TLV per tunnel attribute with IP-SEC tunnel. Optionally, the Public Instance Sub-TLV may augment the BGP tunnel attributes Sub-TLV for tunnel endpoint. Sub-TLVs added: IPsec SUB-TLV in IP-Sec Tunnel Attribute (proposed): 4 octet opaque tag. Public Instance Sub-TLV: identifies the remote instance the Sub- TLV for Tunnel End-Point Identifier takes its address from. NETCONF Link Security: Distribution is secured by Client-server TLS Configuration Data security: Configuration clients SHOULD have host and data security. This is beyond NETCONF/RESTCONF security. Client synchronization of data with other clients must have security links and security mechanisms. Suggested Yang Models for Configuration and Reporting o Tunnels configuration and operational state o SA configuration and operational state, o BGP Tunnel Attribute Yang model augmenting base BGP model with tunnel attributes data and error log. (Tunnel attribute information includes the tunnel attributes plus the mapping of routes to tuples of [tunnel endpoint, security association, and encryption].) Privacy: Link privacy assumes the ability to encrypt the link data to provide privacy. Node Privacy requires software secure containers within the NETCONF/RESTCONF clients/servers and BGP modules for each of these models. Hares Expires January 15, 2021 [Page 12] Internet-Draft BGP-IPSEC-ANALYSIS July 2020 2.2. Option 2- BGP passes client routes with SA-ID plus NLRI passes underlay SAs Document: [I-D.dunbar-idr-sdwan-edge-discovery] What is distributed via NETCONF/RESTCONF or locally configured: Policy and/or templates so that automation may use NLRI with SDWAN SAFI to configure tunnels. This policy may be expressed in as little as 1 line of local configuration. What is distributed via BGP: Client Routes with tunnel attribute with IPsec Tunnel type, IPSec SA ID(s) which reference Security Association attributes being advertised by SDWAN-Underlay UPDATE. The Sub-TLVs added include: IPSec-SA-ID SUB-TLV (proposed): 2 octet pointer into SA table IPsec-SA-Group SUB-TLV: list of pointers into SA table grouped by inner encapsulation type The Underlay property is NLRI attached to port addresses or node address with SDWAN SAFI: Includes Site Type, IPSEC-SA-Type, Port- Local-ID, SDWAN-Site-ID, SDWAN Node ID. Depending on the SITE-Type and IPSec-SA-type, this SAFI carries either template references for pre-configured security association (SAs) or full SA information. Note: Since the Security association information is carried in a different AFI/SAFI pair, this information may be transmitted in a different BGP update than the client routes with the Tunnel attribute. Link Security: Distribution is secured between RR to RR clients and between RR in the RR mesh is secured with Transport layer security. If the RR mesh with underlay information is compromised, it does not mean the route with tunnel attributes will be compromised. Data Security: Data distribution security of tunnel endpoints, SA (security association), routes and mapping (tunnel endpoint, SA, routes) SHOULD have RR and RR Client security on modules processing the data. Full data security (with certificates that the data originated is what arrives at the final destination) for the BGP routes and attributes is beyond the normal mechanism of BGP. These features may be available in SBGP. SBGP signature processing is computationally expensive and requires additional memory space. Synchronization of the routing information on RR (routes, tunnel endpoints, SA-links) and underlay security association information (from AFI/SAFI SDWAN) may be impacted policy that distributes the data. Hares Expires January 15, 2021 [Page 13] Internet-Draft BGP-IPSEC-ANALYSIS July 2020 Technical Note: Many ISPs have chosen to only validate the route origin attribute of the BGP route to insure reduction of "human errors" and some classes of attacks. Suggested Yang Models for Reporting Errors o Tunnels configuration and operational state of tunnels, o SA configuration and operational state of SA information, o BGP Tunnel Attribute Yang model augmenting base BGP model with tunnel attributes data and error log. (Tunnel attribute information includes the tunnel attributes plus the mapping of routes to tuples of [tunnel endpoint, security association, and encryption]. o Augmentation to base BGP Model to display information passed in NLRI with SDWAN SAFI Privacy: (Same as option 1) o Link privacy assumes the ability to encrypt the link data to provide privacy. o Node Privacy requires secure containers within the netconf/ restconf clients/servers and BGP modules for the BGP control plane data. 2.3. Option 3: Secure EVPN (client routes + SA information) Document: draft-ietf-sajassi-bess-secure-evpn [I-D.sajassi-bess-secure-evpn] What is distributed: Client routes with tunnel attribute with ESP- Transport and ESP-in-UDP-Transport tunnel types. SUB-TLVs in ESP-Transport and ESP-in-UDP-Transport Attribute (proposed): BASE DIM, Key-Exchange, ESP SA, and Transform Sub- Structure. This solution would require the Tunnel TLV for the IPsec to contain: Tunnel Endpoint TLV and the DIM TLV. The DIM SUB-TLV has the following fields: * ID-length * Nonce length, Hares Expires January 15, 2021 [Page 14] Internet-Draft BGP-IPSEC-ANALYSIS July 2020 * I-flag * Flags * Re-key counter * Originator ID + (Tenant ID) + (Subnet ID) + (Tenant Address * Nonce data Technical Note: The data rate for retransmitting the client routes with the DIM Sub-TLV must be done at the rekeying rate. This automatic re-key counter is distributed with the data. Link Security: Distribution is secured between RR to RR clients and the RR mesh is secured with transport link security. The regular data distribution of the SA nonce and the rekeying counter provides a potential attack vector for man-in-the middle attacks if the link security is compromised. Data Security: Data distribution security of tunnel endpoints, SA (security association), routes and mapping (tunnel endpoint, SA, routes) SHOULD have RR and RR Client security on modules processing the data. In addition the processes handling SA information [I-D.carrel-ipsecme-controller-ike] should exist in a protected process. Full data security for the BGP routes and attributes is beyond the normal mechanism of BGP, but may be available in SBGP. SBGP signature processing is computationally expensive and requires additional memory space. Synchronization of the routing information on RR (routes, tunnel endpoints, SA-links) and underlay security association information (from AFI/SAFI SDWAN) may be impacted policy that distributes the data. Yang Models for Reporting Errors o Tunnels configuration and operational state, o configuration and operational state, o BGP Tunnel Attribute Yang model augmenting base BGP model with tunnel attributes data and error log. (Tunnel attribute information includes the tunnel attributes plus the mapping of routes to tuples of [tunnel endpoint, security association, and encryption].) Hares Expires January 15, 2021 [Page 15] Internet-Draft BGP-IPSEC-ANALYSIS July 2020 o Yang models for the operational state in [I-D.carrel-ipsecme-controller-ike]. Privacy: Same as option 1 and 2. 2.4. comparison of security issues The security of each of these solutions utilizes similar distribution and error reporting. Man in the Middle attacks based on snooping, would need to break the TLS security and encryption for privacy. The [I-D.sajassi-bess-secure-evpn] provides more data directly linked to the routes which could allow an attack vector. [I-D.hujun-idr-bgp-ipsec] and [I-D.dunbar-idr-sdwan-edge-discovery] provide the route, tunnel information, and link to the SA information. This indirect access to SA information could lessen the attack vector for the tunnel. [I-D.dunbar-idr-sdwan-edge-discovery] and [I-D.sajassi-bess-secure-evpn] have options send the SA information on unique tunnel types. [I-D.dunbar-idr-sdwan-edge-discovery] placement of the SA data in a NLRI can allow a separate encryption between the SA data and the route/tunnel information. While all 3 solutions can be used with automated tools (SDN based on simply configuration based), the each solution has benefits and deficits. 3. Manageability and Scaling Manageability involves how much manual effort is involved to set up IPSec tunnels using each of the three options. The manageability must handle the following: initial set-up of nodes, reporting of status or errors, and rekeying efforts. BGP data distribution and processing of routes to set-up forwarding is stressed during: initial start-up, crash of a RR, and start-up of RR. The scaling of the system should handle the data distribution and the manageability should handle both the network scenario 1 and network scenario 2. Scenario 1 and scenario 2 both consider one security association per tunnel and 10 security associations per 10. This comparison is given to help understand the impact of rekeying the security associations. [I-D.hujun-idr-bgp-ipsec] would need to send rekeying via NETCONF/RESTCONF, but the rekeying that causes a tunnel to switch security associations can be sent via BGP. [I-D.hujun-idr-bgp-ipsec] use of the NETCONF/RETCONF to send a configuration becomes a bottleneck if the network sizes reach scenario 2. [I-D.dunbar-idr-sdwan-edge-discovery] uses two parallel Hares Expires January 15, 2021 [Page 16] Internet-Draft BGP-IPSEC-ANALYSIS July 2020 BGP NLRI processes where one passes routes and security association identifiers, and the second process sends rekeying based on topology information. Rekeying information is transmitted prior to BGP passes the rekeying of the tunnel. [I-D.sajassi-bess-secure-evpn] passes the rekeying information with the client routes. During initial start-up or RR crash, this rekeying data substantially increases the memory footprint. A continual rekeying process in [I-D.sajassi-bess-secure-evpn] could also cause periodic BGP updates to continue to use bandwidth in the network. One alternative to the periodic rekeying is to allow the association of more than 1 security association (SA) per tunnel, and allow a local mechanism to switch security associations are a particular time. This analysis looks at the scaling issues of 10 SAs per tunnel allows this analysis to look at the scaling in terms of memory required for this mechanism of rekeying. The estimate of 10 security associations is admittedly imperfect, but it may help to start the discussion on the memory usage during rekeying. NETCONF/RESTCONF data distribution scales when the client to netconf/ restconf server ratio is low. 1 client per server is best, but 5 servers per client is a low level. 1 client configuring 10K servers on network nodes is beyond most NETCONF/RESTCONF servers. Pushing multiple types of data may also cause stress on the client ability to pull data from the back-end configuration database. The difference between NETCONF/RESTCONF and BGP mechanisms matter in for SDWAN deployments. NETCONF/RESTCONF is optimized for a single administrative domain and BGP is optimized for inter-domain policy. In SDWAN the nodes are distributed across multiple administrative domains. BGP implementations have many levels of policy. Using BGP each node can be under different RR. Each node can have default SA attributes such as supported encryption algorithms, the nonce, and the public key. The SA ID is only locally significant to the node (or to the port), which is less prone to misconfiguration. BGP also has policy at the route level. Using BGP built-in RT constraint distribution, BGP implementations distribute the SA information to the nodes specified as authorized peers. 3.1. Configuration sizes - used for theoretical comparison To provide a simple estimate, it is assumed that 100 items needed to be configured in the Yang modules prior to starting the IPsec VPN. o BGP peer items per node: 20 Hares Expires January 15, 2021 [Page 17] Internet-Draft BGP-IPSEC-ANALYSIS July 2020 o Tunnel configuration items node: 20 o SA Configuration items per node: 40 o Monitoring configuration per node: 20 3.2. BGP Route sizes for theoretical comparison The following estimates are for route and the tunnel Attribute are used for this comparison: o average of 4 bytes for IPv4 prefix o average of 8 bytes for IPv6 prefix. 3.2.1. Size of Tunnel encapsulation attribute with 1 SA per tunnel endpoint The space required in the BGP packet for the tunnel attribute per 1 tunnel with 1 Security association (SA) for each of the options is as follows: o Tunnel TLV header [4 bytes] o Sub-TLV for tunnel endpoint for IPv4 [12 bytes] o IP-Sec Sub-TLVs (required) with 1 SA per tunnel endpoint: * Option 1 [I-D.hujun-idr-bgp-ipsec]: 6 octets * Option 2 [I-D.dunbar-idr-sdwan-edge-discovery] 4 octets * Option 3 [I-D.sajassi-bess-secure-evpn] : 35 octets + Sub-TLV header: 3 octets + Dim: 32 octets (header (4), rekey (8), address (8), Nonce (12)) The total space in the tunnel attribute for each type per tunnel endpoint with one security association is the following: Option 1 [I-D.hujun-idr-bgp-ipsec]: 22 octets Option 2 [I-D.dunbar-idr-sdwan-edge-discovery]: 20 octets Option 3 [I-D.sajassi-bess-secure-evpn] : 52 octets Hares Expires January 15, 2021 [Page 18] Internet-Draft BGP-IPSEC-ANALYSIS July 2020 Encapsulation mechanisms such as GRE and VXLAN may add 6-16 octets per tunnel to the Tunnel attribute per tunnel. This addition is due to adding encodings for inner mechanisms (4-12), and outer encodings (2-4) The total space with encapsulations would then be: Option 1 [I-D.hujun-idr-bgp-ipsec]: 28-38 octets Option 2: [I-D.dunbar-idr-sdwan-edge-discovery]:: 26-36 octets Option 3 [I-D.sajassi-bess-secure-evpn] : 56-68 octets 3.2.2. Size of Tunnel encapsulation attribute with 10 SAs per tunnel This will be completed in version -01 3.3. Network Scenario 1 This will be completed in version -01 3.4. Network scenario 2 This will be completed in version -01.txt 3.5. Scaling Memory sizes This section includes scaling for network scenario 1 and 2. 4. Key differences between the options (to be completed in version -01) 5. Processing of BGP routes (to be completed in version -01) 6. Future Issues - SBGP and Secure IPSEC VPNs (to be completed in version -01) 7. Security Considerations This draft is analysis that includes security and privacy. The draft does not cause any further security issues, but hopes to enhance the security considerations in other drafts. Hares Expires January 15, 2021 [Page 19] Internet-Draft BGP-IPSEC-ANALYSIS July 2020 8. IANA considerations This draft does not make any requests to IANA for allocations. It is an analysis for review of future allocations in the BGP registry. 9. References 9.1. Normative References [I-D.carrel-ipsecme-controller-ike] Carrel, D. and B. Weis, "IPsec Key Exchange using a Controller", draft-carrel-ipsecme-controller-ike-01 (work in progress), March 2019. [I-D.dunbar-idr-sdwan-edge-discovery] Dunbar, L., Hares, S., Raszuk, R., and K. Majumdar, "BGP UPDATE for SDWAN Edge Discovery", draft-dunbar-idr-sdwan- edge-discovery-00 (work in progress), July 2020. [I-D.hujun-idr-bgp-ipsec] Hu, J., "BGP Provisioned IPsec Tunnel Configuration", draft-hujun-idr-bgp-ipsec-02 (work in progress), March 2020. [I-D.ietf-idr-tunnel-encaps] Patel, K., Velde, G., Ramachandra, S., and J. Scudder, "The BGP Tunnel Encapsulation Attribute", draft-ietf-idr- tunnel-encaps-16 (work in progress), July 2020. [I-D.sajassi-bess-secure-evpn] Sajassi, A., Banerjee, A., Thoria, S., Carrel, D., Weis, B., and J. Drake, "Secure EVPN", draft-sajassi-bess- secure-evpn-03 (work in progress), July 2020. [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997, . 9.2. Informative References [I-D.dunbar-bess-bgp-sdwan-usage] Dunbar, L., Guichard, J., Sajassi, A., Drake, J., Najem, B., and D. Carrel, "BGP Usage for SDWAN Overlay Networks", draft-dunbar-bess-bgp-sdwan-usage-08 (work in progress), July 2020. Hares Expires January 15, 2021 [Page 20] Internet-Draft BGP-IPSEC-ANALYSIS July 2020 [I-D.ietf-rtgwg-net2cloud-gap-analysis] Dunbar, L., Malis, A., and C. Jacquenet, "Networks Connecting to Hybrid Cloud DCs: Gap Analysis", draft-ietf- rtgwg-net2cloud-gap-analysis-06 (work in progress), May 2020. [I-D.ietf-rtgwg-net2cloud-problem-statement] Dunbar, L., Jacquenet, C., and M. Toy, "Dynamic Networks to Hybrid Cloud DCs Problem Statement", draft-ietf-rtgwg- net2cloud-problem-statement-10 (work in progress), May 2020. [RFC5512] Mohapatra, P. and E. Rosen, "The BGP Encapsulation Subsequent Address Family Identifier (SAFI) and the BGP Tunnel Encapsulation Attribute", RFC 5512, DOI 10.17487/RFC5512, April 2009, . [RFC5566] Berger, L., White, R., and E. Rosen, "BGP IPsec Tunnel Encapsulation Attribute", RFC 5566, DOI 10.17487/RFC5566, June 2009, . [RFC8192] Hares, S., Lopez, D., Zarny, M., Jacquenet, C., Kumar, R., and J. Jeong, "Interface to Network Security Functions (I2NSF): Problem Statement and Use Cases", RFC 8192, DOI 10.17487/RFC8192, July 2017, . [RFC8365] Sajassi, A., Ed., Drake, J., Ed., Bitar, N., Shekhar, R., Uttaro, J., and W. Henderickx, "A Network Virtualization Overlay Solution Using Ethernet VPN (EVPN)", RFC 8365, DOI 10.17487/RFC8365, March 2018, . Author's Address Susan Hares Hickory Hill Consulting US Email: shares@ndzh.com Hares Expires January 15, 2021 [Page 21]