BESS WG Y. Wang Internet-Draft Z. Zhang Intended status: Standards Track ZTE Corporation Expires: 10 February 2022 9 August 2021 ARP/ND Synching And IP Aliasing without IRB draft-wang-bess-evpn-arp-nd-synch-without-irb-07 Abstract This draft discusses the Service Interfaces of EVPN Signalled L3VPNs. EVPN Signalled L3VPNs are used to improve L3VPNs for some new use cases. Then it discusses how EVPN control plane procedures will be different among these service interfaces. Status of This Memo This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet- Drafts is at https://datatracker.ietf.org/drafts/current/. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." This Internet-Draft will expire on 10 February 2022. Copyright Notice Copyright (c) 2021 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/ license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License. Wang & Zhang Expires 10 February 2022 [Page 1] Internet-Draft EVPN ARP/ND Synch no IRB August 2021 Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 1.1. Terminology and Acronyms . . . . . . . . . . . . . . . . 3 2. Service Interfaces of L3 EVIs . . . . . . . . . . . . . . . . 4 2.1. Mono VLAN-based Service Interface . . . . . . . . . . . . 4 2.2. Multiple VLAN-based Service Interface . . . . . . . . . . 5 2.3. VLAN-bundle Service Interface . . . . . . . . . . . . . . 5 2.4. Shared Risk VLAN-bundle Service Interface . . . . . . . . 6 2.5. Integrated Routing and Cross-connecting Service Interface . . . . . . . . . . . . . . . . . . . . . . . . 7 3. ARP/ND Synching and IP Aliasing . . . . . . . . . . . . . . . 7 4. Forwarding Unicast Packets . . . . . . . . . . . . . . . . . 7 5. RT-5 Routes in EVPN signalled L3VPN . . . . . . . . . . . . . 7 5.1. RT-5E Advertisement on Distributed L3 GW . . . . . . . . 8 5.2. Distributed RT-5G Advertisement . . . . . . . . . . . . . 8 5.3. Centerlized RT-5G Advertisement for Distributed L3 Forwarding . . . . . . . . . . . . . . . . . . . . . . . 8 5.3.1. Centerlized CE-BGP . . . . . . . . . . . . . . . . . 9 5.3.2. RT-2E Advertisement from PE1/PE2 to PE3 . . . . . . . 10 5.3.3. RT-5G Advertisement from PE3 to PE1/PE2 . . . . . . . 10 5.3.4. RT-2E Advertisement between PE1 and PE2 . . . . . . . 11 5.3.5. Egress ESI Link Protection between PE1 and PE2 . . . 11 5.3.6. Mass-Withdraw by EAD/ES Route . . . . . . . . . . . . 11 5.3.7. On the Failure of PE3 Node . . . . . . . . . . . . . 11 5.3.8. Floating GW-IP between R1 and R2 . . . . . . . . . . 12 5.4. RT-5L Advertisement . . . . . . . . . . . . . . . . . . . 12 6. Load Balancing of Unicast Packets . . . . . . . . . . . . . . 13 7. Special Considerations for Single-Active ESIs . . . . . . . . 13 8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 13 9. Security Considerations . . . . . . . . . . . . . . . . . . . 13 10. References . . . . . . . . . . . . . . . . . . . . . . . . . 13 10.1. Normative References . . . . . . . . . . . . . . . . . . 13 10.2. Informative References . . . . . . . . . . . . . . . . . 15 Appendix A. Explanation for Physical Links of the Use-cases . . 15 A.1. Failure Detections for P1.2 (or P2.1) . . . . . . . . . . 16 A.2. Protection Approaches for N1 (or N2) . . . . . . . . . . 16 A.2.1. CCC-Approaches . . . . . . . . . . . . . . . . . . . 16 A.2.1.1. CCC Active-Active Protection . . . . . . . . . . 17 A.2.1.2. CCC Active-Standby Protection . . . . . . . . . . 17 A.2.2. VSI-Approaches . . . . . . . . . . . . . . . . . . . 17 Appendix B. GW-IP Compatibility Considerations . . . . . . . . . 17 B.1. Section 3.2 of I-D.ietf-bess-evpn-prefix-advertisement . 17 B.2. Resolve GW-IP Overlay Index to Another RT-5 . . . . . . . 18 B.3. Specail RT2 route for GW-IP Address . . . . . . . . . . . 19 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 19 Wang & Zhang Expires 10 February 2022 [Page 2] Internet-Draft EVPN ARP/ND Synch no IRB August 2021 1. Introduction Service interface describes how an ES is attached to a L3EVI. This draft discusses the following service interfaces: * Mono VLAN-based Service Interface. * Multiple VLAN-based Service Interface. * VLAN-bundle Service Interface. * Shared Risk VLAN-bundle Service Interface. * IRC Service Interface. Different service interface will require different control-plane procedures, then this draft discusses the behavior of RT5 routes advertisement per each service interface, especially when they are RT5 routes with ESI as overlay index or GW-IP as overlay index. Note that an ES may be attached to different L3EVIs via different VLANs, and mutiple ESes can be attached to the same L3EVI Instance. So service interface is ESI-specific and EVI-specific. When ES1 is of VLAN-bundle Service Interface to EVI1, it may be of Mono VLAN- based Service Interface for EVI2. Thus service interfaces of L3EVIs are -specific in this draft. 1.1. Terminology and Acronyms Most of the acronyms and terms used in this documents comes from [RFC7432] and [I-D.sajassi-bess-evpn-ip-aliasing] except for the following: * VRF AC: An Attachment Circuit (AC) that attaches a CE to an IP-VRF but is not an IRB interface. * VRF Interface: An IRB interface or a VRF-AC or an IRC interface. Note that a VRF interface will be bound to the routing space of an IP-VRF. * L3 EVI: An EVPN instance spanning the Provider Edge (PE) devices participating in that EVPN which contains VRF ACs and maybe contains IRB interfaces or IRC interfaces. * IP-AD/EVI: Ethernet Auto-Discovery route per EVI, and the EVI here is an IP-VRF. * IP-AD/ES: Ethernet Auto-Discovery route per ES, and the EVI for one of its route targets is an IP-VRF. * RMAC: Router's MAC, which is signaled in the Router's MAC extended community. * ESI Overlay Index: ESI as overlay index. * ET-ID: Ethernet Tag ID, it is also called ETI for short in this document. * RT-2E: A MAC/IP Advertisement Route with a non-reserved ESI. Wang & Zhang Expires 10 February 2022 [Page 3] Internet-Draft EVPN ARP/ND Synch no IRB August 2021 * RT-5E: An EVPN Prefix Advertisement Route with a non-reserved ESI as its overlay index. * IRC: Integrated Routing and Cross-connecting, thus a IRC interface is the virtual interface connecting an IP-VRF and an EVPN VPWS. * CE-BGP: The BGP session between PE and CE. Note that CE-BGP route doesn't have a RD or Route-Target. * RT-5G: An EVPN Prefix Advertisement Route with a zero ESI and a non-zero GW-IP. * RT-5L: An EVPN Prefix Advertisement Route with both zero ESI and zero GW-IP. 2. Service Interfaces of L3 EVIs The detailed explanation of this network's physical links are described in Figure 6 and Appendix A. But this network is illustrated briefly in the following sections per each Service Interface. Ethernet segment ES1 is the ethernet segment of P1 and P2 of Figure 6, and its ESI is ESI21. 2.1. Mono VLAN-based Service Interface +-------------------+ PNEC1 PE1 | | +---------+ +----------+--------+ | | | | __(P1.1)__(VPNx) | | | | | P1 | / | | | #==============< | | PE3 | | | ESI21 | \__ __ | +----+----+ | N1+--+ | + | (P1.2) (VPNy) | | | | | | | +-----------+-------+ | (VPNx)---+N3 | | | | | | | | | | | | | | | | | | PE2 | | | | | | | +-----------+-------+ | (VPNy)---+N5 | N2+--+ | + | __(P2.2)__(VPNy) | | | | | | ESI21 | / | +----+----+ | #==============< | | | | | P2 | \__ __ | | | | | (P2.1) (VPNx) | | +---------+ +----------+--------+ | | | +-------------------+ Figure 1: Mono VLAN-Based SI Wang & Zhang Expires 10 February 2022 [Page 4] Internet-Draft EVPN ARP/ND Synch no IRB August 2021 In this service interface, each ESI can have no more than one of its VLANs attached to a specified EVPN Instance. Take above figure for example, P1.1 and P1.2 are two subinterfaces of the same ESI, and is Mono VLAN-based service interface, thus P1.1 and P1.2 can't be attached to the same EVPN Instance. Actually, P1.1 are attached to VPNx, while P1.2 are attached to VPNy. 2.2. Multiple VLAN-based Service Interface +-----------------------+ PNEC1 PE1 | | +---------+ +-------+------+ | | | | __(P1.1) | | | | | | / \ | | | #=============< (VPN1) | | PE3 | | | ESI21 | \__ / | +----+----+ | N1+--+ | + | (P1.2) | | | | | | | +--------+-----+ | | | | | | | | | | | | | | | (VPN1)----+N3 | | | | PE2 | | | | | | | +--------+-----+ | | | N2+--+ | + | __(P2.2) | | | | | | ESI21 | / \ | +----+----+ | #=============< (VPN1) | | | | | | \__ / | | | | | (P2.1) | | +---------+ +-------+------+ | | | +-----------------------+ Figure 2: Multiple VLAN-based SI This network is similar to Figure 1 with a few notable exceptions as below: The L3EVIs VPNx and VPNy there are the same L3EVI (VPN1) here. So two of PE1's VPN1's ACs are both subinterfaces (P1.1 and P1.2) of the same ESI (ESI23). Note that P1.1 is the gateway of N1, while P1.2 is the gateway of N2. N1 and N2 are just not in the same subnets. 2.3. VLAN-bundle Service Interface When two VLANs of the same ES shares the same Gateway IP address of the same EVPN, These two VLANs can be configured into the same subinterface of that ES. This is VLAN-bundle service interface. Wang & Zhang Expires 10 February 2022 [Page 5] Internet-Draft EVPN ARP/ND Synch no IRB August 2021 +-----------------------+ PNEC1 PE1 | | +---------+ +-------+-------------+ | | | | | | | | | | (P1.1) AC1 | | | #=============< >======(VPN1) | | PE3 | | | ESI21 | (P1.2) | +----+----+ | N1+--+ | + | | | | | | | | +--------+------------+ | | | | | | | | | | | | | | | (VPN1)----+N3 | | | | PE2 | | | | | | | +--------+------------+ | | | N2+--+ | + | | | | | | | ESI21 | (P2.1) AC2 | +----+----+ | #=============< >======(VPN1) | | | | | | (P2.2) | | | | | | | +---------+ +-------+-------------+ | | | +-----------------------+ Figure 3: Separate Risk VLAN-Bundle SI This network is similar to Figure 2 with a few notable exceptions as below. P1.1 and P1.2 are aggregated into the same subinterface (that is AC1). It is AC1 that is attached to VPN1. Note that although P1.1 and P1.2 are aggregated into the same subinterface AC1, when P1.1 fails, P1.2 may not fail. Thus we say that AC1 are configured with Separated Risk VLAN-Bundle. Note that VLAN-bundle Service Interface is actually Separated Risk VLAN-Bundle Service Inerface. 2.4. Shared Risk VLAN-bundle Service Interface There may be other network which is similar to Section 2.3, except for that the VLAN-bundle of AC1 is Shared Risk VLAN-bundle, not Separated Risk VLAN-bundle. When we say subinterface AC1 is of Shared Risk VLAN-bundle, we are saying that when an event result in P1.1's failure, that event will also result in P1.2's failure. When AC1 is of Shared Risk VLAN-bundle, we say that is Shared Risk VLAN-bundle service interface. Wang & Zhang Expires 10 February 2022 [Page 6] Internet-Draft EVPN ARP/ND Synch no IRB August 2021 2.5. Integrated Routing and Cross-connecting Service Interface The service interface in [I-D.wz-bess-evpn-vpws-as-vrf-ac] can be called as IRC (Ingegrated Routing and Cross-connecting) Service Interface. 3. ARP/ND Synching and IP Aliasing * For Mono VLAN-based Service Interface, The route format can be the same as [I-D.sajassi-bess-evpn-ip-aliasing]. * For Multiple VLAN-based Service Interface, it follows [I-D.wang-bess-evpn-ether-tag-id-usage]. * For VLAN-bundle Service Interface, it follows [I-D.wang-bess-evpn-ether-tag-id-usage]. * For Shared Risk VLAN-bundle Service Interface, The route format can be the same as [I-D.sajassi-bess-evpn-ip-aliasing]. * For IRC Service Interface, It follows [I-D.wz-bess-evpn-vpws-as-vrf-ac]. 4. Forwarding Unicast Packets Note that in [I-D.sajassi-bess-evpn-ip-aliasing] the IP-AD per EVI route carries a "Router's MAC" extended community in case the RMAC is not the same among different PEs. In these cases, the inner destination MAC of the corresponding data packets from PE3 to PE1/PE2 must use the RMAC in IP-AD/EVI route instead, even if there is a RMAC in RT-2E route. Note that this is a data-plane update of [I-D.ietf-bess-evpn-prefix-advertisement] for both EVPN signalled L3VPN and [I-D.sajassi-bess-evpn-ip-aliasing]. According to [I-D.ietf-bess-evpn-prefix-advertisement] section 4.3 or [I-D.ietf-bess-evpn-inter-subnet-forwarding] section 5.4, the inner destination MAC will follow the RMAC of RT-5E Route or RT-2E Route. 5. RT-5 Routes in EVPN signalled L3VPN EVPN signalled L3VPN can be deployed without EVPN IRB like what MPLS/ BGP VPNs have done for a long time, but it can be combined with EVPN IRB. The EVPN siganlled L3VPN without EVPN IRB is not well defined yet, so we take the non-IRB usecase as an example. But the following routes and procedures can be used in EVPN IRB usecase too. Note that in EVPN IRB usecase, the IRB interfaces are VRF-interface too. Wang & Zhang Expires 10 February 2022 [Page 7] Internet-Draft EVPN ARP/ND Synch no IRB August 2021 5.1. RT-5E Advertisement on Distributed L3 GW Given that PE1/PE2 can install a synced ARP entry to its proper VRF- interface benefitting from the RT-2 route of Section 3. So it is not necessary for PE1/PE2 to advertise per-host IP prefixes to remote PEs (e.g. PE3) by RT-2 routes. It is recommended that PE1/PE2 advertise an RT-5 route per subnet to PE3 instead. The ESI of these RT-5E routes can be set to the ESI of the corresponding VRF interface. If the VRF interface fails, these subnets will achieve more faster convergency on PE3 by the withdraw of the corresponding IP-AD/EVI route. Note that N1/N2 may be a host or a router, when it is a router, those subnets (which are advertised by RT-5E routes) will be the subnets behind it. When N1 and N2 are hosts, those subnets will be the subnets of N1 and N2 whether they are different subnets or not. The detailed procedures can be found in the following drafts: * For Mono VLAN-based Service Interface, the route format can be the same as [I-D.sajassi-bess-evpn-ip-aliasing]. * For Multiple VLAN-based Service Interface, it follows [I-D.wang-bess-evpn-ether-tag-id-usage]. * For VLAN-bundle Service Interface, it follows [I-D.wang-bess-evpn-ether-tag-id-usage]. * For Shared Risk VLAN-bundle Service Interface, the route format can be the same as [I-D.sajassi-bess-evpn-ip-aliasing]. * For IRC Service Interface, It follows [I-D.wang-bess-evpn-ether-tag-id-usage]. Note that the Ethernet Tag ID (of RT-5E route of this case) is the VPWS Service Identifier of the corresponding IRC interface. 5.2. Distributed RT-5G Advertisement It follows [I-D.wz-bess-evpn-vpws-as-vrf-ac] Section 2.3. Note that these procedures can be used in every L3EVI Service Interface. 5.3. Centerlized RT-5G Advertisement for Distributed L3 Forwarding When N1/N2/N3 is a router, it is called R1/R2/R3 in the following figure. Note that figure 1 only illustrates the physical ethernet links, but figure 2 illustrates the logical L3 adjacencies between PE and CE as the following. Wang & Zhang Expires 10 February 2022 [Page 8] Internet-Draft EVPN ARP/ND Synch no IRB August 2021 PE2 +----+ +---------------+ | | 20.2 | 20.1 +------+ | ------> | R2 |===+------------| | | RT-2E | | | | |IPVRF1| | 20.2 PE3 +----+ | +---------| | | ESI1 +---------------+ Prefix2 | | | 10.1 +------+ | | | | | +---------------+ | +-----------+ | | | ^ | | IPVRF1 | | | | | RT-2E <-------- | | |----R3 | | ESI1 | 10.2 RT-5G | | 3.3.3.3 | | | | | ESI1 Prefix1 | +-----------+ | | | | 10.2 | ^ | | | +---------------+ | | | Prefix1 | | | 20.1 +------+ | +---|-----------+ +----+ +--|---------| | | | | | | | |IPVRF1| | | | R1 |======+---------| | | ------> | | | 10.2 | 10.1 +------+ | RT-2E | +----+ +---------------+ 10.2 | CE-BGP | PE1 ESI1 | Prefix1 | | NH=10.2 | CE-BGP | +------------------------>------------------------+ Figure 4: Centerlized RT-5G Advertisement Note that R1/R2 should establish CE-BGP session with both PE1 and PE2 in case of one of them fails, PE1 and PE2 will advertise RT-5E route to PE3 for their prefixes learned from CE-BGP independently. If R1/ R2 prefers to establish a single CE-BGP session, it can establish the CE-BGP session with PE3 instead. This CE-BGP session can be called the centerlized CE-BGP session. But when we use centerlized CE-BGP session, we should use RT-5G route instead. Note that we just use centerlized CE-BGP session to do route advertisement, but we still expect a distributed Layer 3 forwarding framework. 5.3.1. Centerlized CE-BGP The CE-BGP session between R1 and PE3 is established between 10.2 and 3.3.3.3. The CE-BGP session between R2 and PE3 is established between 20.2 and 3.3.3.3. The IP address 10.2/20.2 is called the uplink interface address of R1/R2 in this document. The IP address 3.3.3.3 is called the centerlized loopback address of IPVRF1 in this document. The IP address 10.1/20.1 is called the downlink VRF- interface address of PE1/PE2 in this document. Wang & Zhang Expires 10 February 2022 [Page 9] Internet-Draft EVPN ARP/ND Synch no IRB August 2021 Note that the downlink VRF-interface is a Layer 3 link and it needn't attach an BD. R1 advertises a BGP route for a prefix (say "Prefix1") behind it to PE3 via that CE-BGP session. The nexthop for Prefix1 is R1's uplink interface address (say 10.2). The route advertisement of R2 is similar to the above advertisement. Note that the packets from R1/R2 to the centerlized loopback address may be routed following the default route on R1/R2. 5.3.2. RT-2E Advertisement from PE1/PE2 to PE3 When PE1 learns the ARP entry of 10.2, it advertises a RT-2E route to PE3. The ESI value of the RT-2E route is ESI1, which is the ESI of PE1's downlink VRF-interface for R1. The RT-2E route is constructed following section 2.1. Note that in [RFC7432], when the ESI is single-active, the MAC forwarding only use the label and the MPLS nexthop of the RT-2E route as long as they are valid for forwarding status. But in RT-5 routes we assume that the ESI is always preferred even if the ESI is single- active. This is similar to [I-D.ietf-bess-evpn-prefix-advertisement] section 3.2 Table 1. The ESI usage in IP forwarding is out of the [RFC7432]'s scope. The RT-2E route advertisement of PE2 is similar to the above advertisement. 5.3.3. RT-5G Advertisement from PE3 to PE1/PE2 When PE3 receives the prefix1 from the CE-BGP session. The nexthop for Prefix1 is 10.2, and the ESI for 10.2 is ESI1. So PE3 advertises a RT-5G route to PE1/PE2 for Prefix1. The GW-IP value of the RT-5G route for Prefix1 is 10.2. Note that PE3 can load-balance packets for Prefix1 via the IP-AD/EVI routes from PE1/PE2. Because ESI1 is the ESI for Prefix1's GW-IP. The RT-5 route advertisement and packet forwarding for Prefix2 is similar to the above. Note that the centerlized loopback address is advertised by PE3 via RT-5L route. The nexthop of the RT-5L route is PE3, and the GW-IP value of the RT-5L route is zero. The label of the RT-5L route is IPVRF1's label on PE3. The RMAC of the RT-5L route is PE3's MAC when the encapsulation is VXLAN. Wang & Zhang Expires 10 February 2022 [Page 10] Internet-Draft EVPN ARP/ND Synch no IRB August 2021 5.3.4. RT-2E Advertisement between PE1 and PE2 The RT-2E routes advertisement between PE1 and PE2 is used to sync these ARP entries to each other in order to avoid ARP missing. The ESI Value of these two RT-2E routes is ESI1. Note that we assume that the ARP entry for 10.2 will be learned on PE1 only, and 20.2 will be learned on PE2 only. Note that the two downlink VRF-interfaces for R1/R2 on PE1/PE2 are sub-interfaces of the same physical interface. So they have the same ESI. 5.3.5. Egress ESI Link Protection between PE1 and PE2 The IP-AD/EVI routes between PE1 and PE2 is used to do egress link protection. The egress link protection follows the second approach of the [RFC8679] section 6. Note that although the ARP entry for 10.2 on PE2 is synced from PE1 via RT-2E route. The ARP entry on PE2 is installed to forward packets directly to the corresponding downlink VRF-interface primarily. The bypass tunnel following the IP-AD/EVI route is only activated when the downlink VRF-interface fails. 5.3.6. Mass-Withdraw by EAD/ES Route We can assume that R1 and R2 are attached to different IP-VRFs(say IPVRF1 and IPVRF2 respectively), and the physical interface of the downlink VRF-interfaces on PE1 fails, PE1 will withdraw the IP-AD/ES route of ESI1, so PE3 will re-route 10.2 for Prefix1 in IPVRF1 and 20.2 for Prefix2 in IPVRF2 at the same time. Then data packets for Prefix1 and Prefix2 will be sent to PE2 instead. 5.3.7. On the Failure of PE3 Node On the failure of PE3, PE1/PE2 should delay the deletion of the RT-5G route from PE3. PE3 can use a new BGP attribute to indicate the delayed-deletion requirement to PE1/PE2. Otherwise the L3 traffic between R1 and R2 will be interrupted. Fortunately, PE3 will typically have a redundant node (PE3' in Figure 3), and PE3' can be used to take PE3's place when PE3 fails. Note that from the viewpoint of R1 and R2, the total of PE1, PE2, PE3, PE3' and the underlay network between them is regarded as the following logical router: Wang & Zhang Expires 10 February 2022 [Page 11] Internet-Draft EVPN ARP/ND Synch no IRB August 2021 +---------------------------------+ | | | +----------------------+ | | | RPU1 (PE3) | | | +----------------------+ | | | | +----------------------+ | | | RPU2 (PE3') | | | +----------------------+ | | | | +----------------------+ | R1-----------| Line Card 1 (PE1) | | | +----------------------+ | | | | +----------------------+ | R2-----------| Line Card 2 (PE2) | | | +----------------------+ | | | +---------------------------------+ Figure 5: The Logical Router Framework R1 and R2 connect to the line-cards of the logical router. and the data packets between R1 and R2 just pass through the line-cards, not through the RPUs(Routing Processing Units). But R1/R2 establish the BGP session with the RPUs, not the line-cards. When the RPU1(or actually PE3) fails, the line-cards(or actually PE1/PE2) will keep the forwarding state unchanged untill the RPU1 or RPU2 comes up. So the delayed deletion on PE1/PE2 for PE3's sake is apprehensible for the same reason. 5.3.8. Floating GW-IP between R1 and R2 It is similar to [I-D.ietf-bess-evpn-prefix-advertisement] section 4.2 except for a few notable differences as described in the following. There may be no BD in PE1/PE2/PE3. There is no need for a PE node that don't have an IP-VRF instance to advertise the RT-5G routes here. 5.4. RT-5L Advertisement When R1/R2 establish CE-BGP sessions with both PE1 and PE2, it is enough for PE1/PE2 to advertise RT-5L routes to PE3. There is no need for RT-5G or RT-5E advertisement on PE1/PE2 in that usecase. Note that when R1/R2 establish CE-BGP sessions with both PE1 and PE2, the downlink VRF-interface addresses on PE1 and PE2 may be different IP addresses of the same subnet. Wang & Zhang Expires 10 February 2022 [Page 12] Internet-Draft EVPN ARP/ND Synch no IRB August 2021 Note that when centerlized CE-BGP session is used, the prefixes from R3 and the local loopback addresses on PE3 are advertised to PE1/PE2 using RT-5L too. 6. Load Balancing of Unicast Packets It is similar to [I-D.sajassi-bess-evpn-ip-aliasing] except for a few notable exceptions as explained in section 6.2.3 and the following. Note that when the encapsulation is VXLAN, PE3 will encapsulate the RMAC of the RT-2E route for corresponding GW-IP address. And the RMAC of PE1 MAY have the same value with the RMAC of PE2. This can be achieved by configuration. When a IP packet is encapsulated with a VNI label according to an IP-AD/EVI route, the packet SHOULD be encapsulated with a Destination-MAC according to the RMAC of the same IP-AD/EVI route, if and only if the IP-AD/EVI route have a RMAC of its own. Note that PE1/PE2 just do egress link protection following IP-AD/EVI and EAD/ES route. Even if ESI1 is configured as all-active ESI, PE1/ PE2 will not load-balance between local downlink VRF-interface and the bypass tunnel. The downlink VRF-interfaces will always have more higher priority than the bypass tunnel. 7. Special Considerations for Single-Active ESIs When the R1 is an Ethernet Segment of MHD type, and the uplink interfaces of R1 operates in linux network-bonding mode type 1. So the Primary flag according to DF election may cause packet-drop on R1 because of the nature of linux bond1. In the linux bond1 use case, we propose that the Layer 2 extended community should not be included. and on PE3 the single-active ESI have lower priority than the MAC/IP route's own MPLS nexthop, but at the same time the downlink VRF-interface on PE1/PE2 may still have higher priority than the bypass tunnel to make convergency faster. 8. IANA Considerations no IANA Considerations. 9. Security Considerations TBD. 10. References 10.1. Normative References Wang & Zhang Expires 10 February 2022 [Page 13] Internet-Draft EVPN ARP/ND Synch no IRB August 2021 [I-D.wang-bess-evpn-ether-tag-id-usage] Wang, Y., "Ethernet Tag ID Usage Update for Ethernet A-D per EVI Route", Work in Progress, Internet-Draft, draft- wang-bess-evpn-ether-tag-id-usage-00, 6 August 2021, . [I-D.sajassi-bess-evpn-ip-aliasing] Sajassi, A., Badoni, G., Warade, P., Pasupula, S., Drake, J., and J. Rabadan, "EVPN Support for L3 Fast Convergence and Aliasing/Backup Path", Work in Progress, Internet- Draft, draft-sajassi-bess-evpn-ip-aliasing-02, 8 June 2021, . [I-D.ietf-bess-evpn-prefix-advertisement] Rabadan, J., Henderickx, W., Drake, J., Lin, W., and A. Sajassi, "IP Prefix Advertisement in EVPN", Work in Progress, Internet-Draft, draft-ietf-bess-evpn-prefix- advertisement-11, 18 May 2018, . [I-D.ietf-bess-evpn-inter-subnet-forwarding] Sajassi, A., Salam, S., Thoria, S., Drake, J., and J. Rabadan, "Integrated Routing and Bridging in EVPN", Work in Progress, Internet-Draft, draft-ietf-bess-evpn-inter- subnet-forwarding-15, 26 July 2021, . [RFC7432] Sajassi, A., Ed., Aggarwal, R., Bitar, N., Isaac, A., Uttaro, J., Drake, J., and W. Henderickx, "BGP MPLS-Based Ethernet VPN", RFC 7432, DOI 10.17487/RFC7432, February 2015, . [RFC8679] Shen, Y., Jeganathan, M., Decraene, B., Gredler, H., Michel, C., and H. Chen, "MPLS Egress Protection Framework", RFC 8679, DOI 10.17487/RFC8679, December 2019, . [I-D.wz-bess-evpn-vpws-as-vrf-ac] Wang, Y. and Z. Zhang, "EVPN VPWS as VRF Attachment Circuit", Work in Progress, Internet-Draft, draft-wz-bess- evpn-vpws-as-vrf-ac-01, 28 July 2021, . Wang & Zhang Expires 10 February 2022 [Page 14] Internet-Draft EVPN ARP/ND Synch no IRB August 2021 10.2. Informative References [I-D.ietf-idr-tunnel-encaps] Patel, K., Velde, G., and S. Ramachandra, "The BGP Tunnel Encapsulation Attribute", Work in Progress, Internet- Draft, draft-ietf-idr-tunnel-encaps-15, 1 December 2019, . Appendix A. Explanation for Physical Links of the Use-cases +------------------+ PE1 | P6 | L2NE1 +----------+---------+ | +----------+ | __(P1.1)__(VPNx) | | +---+ P4 | | P1 | / \ | | |N1 |-----O==------=======< (NIz) | P6 | PE3 +---+ | \ / | | \__ __ / | +----+-------+ | | | | | (P1.2) (VPNy) | | | +----|--|--+ +-----------+--------+ | (VPNx)--+N3 | | | | / | P3.1 | | P3.2 | P7 | (NIz)--------+N4 | | PE2 | | \ | +----|--|--+ +-----------+--------+ | (VPNy)--+N5 | \/ | | __(P2.2)__(VPNy) | | | +---+ | /\ | | / \ | +----+-------+ |N2 |-----O====--=========< (NIz) | P8 | +---+ P5 | | P2 | \__ __ / | | +----------+ | (P2.1) (VPNx) | | L2NE2 +----------+---------+ | | P8 | +------------------+ Figure 6: Physical Links Illustrated There are three PEs, two L2NEs (Layer 2 Network Elements) and five L3NEs (Layer 3 Network Elements) in abobe network. The PEs are PE1, PE2 and PE3. The L2NEs are L2NE1 and L2NE2. The L3NEs are N1/N2/N3/N4/N5. They are all illustrated in Figure 6. There are 9 physical links among these 10 physical devices as illustrated in Figure 6. These physical links are called as PLi (i=1,2...8). The two physical ports of the same physical link PLi are both called as Pi (i=1,2...8). Wang & Zhang Expires 10 February 2022 [Page 15] Internet-Draft EVPN ARP/ND Synch no IRB August 2021 As illustrated in Figure 6, some of these physical ports may have subinterfaces. When a subinterface's VLAN ID is j and it is physical port Pi's subinterface, that subinterface is called as Pi.j. For example, P1.2 is a subinterface of physical port P1 and its VLAN ID is 2. There are three NIs (Network Instances) among PE1, PE2 and PE3. They are VPNx, VPNy and NIz. Two subinterfaces are attached to VPNx, they are P1.1 and P2.1. Other two subinterfaces are attached to VPNy, they are P1.2 and P2.2. N3 is also attched to VPNx, while N5 is also attached to VPNy. There are two EVCs (Ethernet Virtual Connections) between L2NE1 and L2NE2, they are EVC1 and EVC2. The L2NE1's EVC1 instance (which is illustrated as the "O" on L2NE1) have three member interfaces, they are P4, P1.1 and P3.1, where P3.1 and P1.1 are of the same protection-group. The L2NE2's EVC1 instance have two member interfaces, they are P3.1 and P2.1. The L2NE2's EVC2 instance (which is illustrated as the "O" on L2NE2) have three member interfaces, they are P5, P2.2 and P3.2, where P3.1 and P1.1 are of the same protection-group. The L2NE1's EVC2 instance have two member interfaces, they are P3.2 and P1.2. The L2NE2's EVC1 instance and L2NE1's EVC2 instance are both CCC (Circuit Cross Connection) local connections. VPNx and VPNy are associated to NIz on each PE. A.1. Failure Detections for P1.2 (or P2.1) There is a CFM session CFM1 between P1.2 of PE1 and L2NE2's P3.2, when physical port P3 fails, the CFM session CFM1 will go down. There is a CFM session CFM2 between P2.1 of PE2 and L2NE1's P3.1, when physical port P3 fails, the CFM session CFM2 will go down. A.2. Protection Approaches for N1 (or N2) A.2.1. CCC-Approaches The L2NE1's EVC2 instance and L2NE2's EVC1 instance are both CCC local connections too. In L2NE1's EVC1 instance, P1.1 and P3.1 are of the same protection-group PG1. In L2NE2's EVC2 instance, P2.2 and P3.2 are of the same protection-group PG2. In PG1, both P1.1 and P3.1 will receive data packets. In PG2, both P2.2 and P3.2 will receive data packets. Wang & Zhang Expires 10 February 2022 [Page 16] Internet-Draft EVPN ARP/ND Synch no IRB August 2021 A.2.1.1. CCC Active-Active Protection L2NE1 (or L2NE2) will load-balance N1's (N2's) data packets between P1.1 and P3.1 (or P2.2 and P3.2). A.2.1.2. CCC Active-Standby Protection In PG1, P1.1 is the active path, P3.1 is the backup path. In PG2, P2.2 is the active path, P3.2 is the backup path. That's saying that L2NE1 (or L2NE2) will not send N1's (or N2's) data packets over P3.1 (or P3.2), unless P1.1 (or P2.2) or P1 (or P2) has been in failure before that data forwarding. A.2.2. VSI-Approaches L2NE1's EVC2 instance and L2NE2's EVC1 instance are both VSI instances in this case. P1.1, P3.1, P2.2 and P3.2 are all individual ACs in these VSIs. Note that L2NE2's EVC1 instance and L2NE1's EVC2 instance are still both CCC local connections in this case, and there is no PG1 or PG2 in this case, and there are no PWs in this case. Appendix B. GW-IP Compatibility Considerations B.1. Section 3.2 of I-D.ietf-bess-evpn-prefix-advertisement The following bullets in Section 3.2 of [I-D.ietf-bess-evpn-prefix-advertisement]: o ... It is important to note that recursive resolution of the Overlay Index applies upon installation into an IP-VRF, and not upon BGP propagation (for instance, on an ASBR). ... o ... o In order to enable the recursive lookup resolution at the ingress NVE, an NVE that is a possible egress NVE for a given Overlay Index must originate a route advertising itself as the BGP next hop on the path to the system denoted by the Overlay Index. For instance: . If an NVE receives an RT-5 that specifies an Overlay Index, the NVE cannot use the RT-5 in its IP-VRF unless (or until) it can recursively resolve the Overlay Index. . If the RT-5 specifies an ESI as the Overlay Index, recursive Wang & Zhang Expires 10 February 2022 [Page 17] Internet-Draft EVPN ARP/ND Synch no IRB August 2021 resolution can only be done if the NVE has received and installed an RT-1 (Auto-Discovery per-EVI) route specifying that ESI. . If the RT-5 specifies a GW IP address as the Overlay Index, recursive resolution can only be done if the NVE has received and installed an RT-2 (MAC/IP route) specifying that IP address in the IP address field of its NLRI. . If the RT-5 specifies a MAC address as the Overlay Index, recursive resolution can only be done if the NVE has received and installed an RT-2 (MAC/IP route) specifying that MAC address in the MAC address field of its NLRI. Note that the RT-1 or RT-2 routes needed for the recursive resolution may arrive before or after the given RT-5 route. B.2. Resolve GW-IP Overlay Index to Another RT-5 The following paragraph in above section: "If the RT-5 specifies a GW IP address as the Overlay Index, recursive resolution can only be done if the NVE has received and installed an RT-2 (MAC/IP route) specifying that IP address in the IP address field of its NLRI." is not saying that if the recursive resolution can't find out a RT-2, that RT-5 should not be installed. It is just saying that we have to check that such a RT-2 is already there before the recursive resolution. But before the recursive resolution, we have to iterate every RT-2 route to find out that RT-2. Because that the key of RT-2 is , not just an IP address. This is too inefficient to be done. On the contrary, we just find out that RT-2 route relaying on that recursive resolution. It may be weird for us to rely the recursive resolution itself to decide whether the recursive resolution should be done or not. We should also note that above section was written based on the following principles: * The following paragraph sepecifies how the recursive lookup resolution will be done: Wang & Zhang Expires 10 February 2022 [Page 18] Internet-Draft EVPN ARP/ND Synch no IRB August 2021 "In order to enable the recursive lookup resolution at the ingress NVE, an NVE that is a possible egress NVE for a given Overlay Index must originate a route advertising itself as the BGP next hop on the path to the system denoted by the Overlay Index." * The examples that is constrained by the phrase "For instance:" described some use-cases that followed above paragraph, with the understanding that new use-cases were possible in the future with new documents, as long as the rules in section 3 were respected. B.3. Specail RT2 route for GW-IP Address If there are devices that have interpreted above section as the following: "if the recursive resolution can't find out a RT-2 for that RT-5's GW-IP (say IP_0), that RT-5 should not be installed." The PE that learns IP_0 from CE can advertise a RT-2 (R2_IP_0) for that IP_0. This advertisement should be triggered by policy. R2_IP_0 should be advertised for its IP-VRF instance, and it's MPLS Label1 field should be set to a pre-configured label-value, it's MAC Address in NLRI should be set to a pre-configured MAC address, and it should be advertised along with the lowest preferrence/weight/metric. Authors' Addresses Yubao Wang ZTE Corporation No.68 of Zijinghua Road, Yuhuatai Distinct Nanjing China Email: wang.yubao2@zte.com.cn Zheng(Sandy) Zhang ZTE Corporation No. 50 Software Ave, Yuhuatai Distinct Nanjing China Email: zhang.zheng@zte.com.cn Wang & Zhang Expires 10 February 2022 [Page 19]