ICMP Error Handling for VPNs in SRv6 Networks

Troubleshooting is an essential part of all IP networks. IP VPN service of transport networks always represented a special challenge to provide ICMP error handling, as the hosts in a VPN are not part of the transport network. For VPNs the main challenge to use ICMP for connectivity check and fault localization is that network internal nodes (a.k.a. P routers) are not service (i.e., VPN) aware. Therefore, P routers cannot route VPN-specific ICMP error messages back to the original source of the packet, that triggered the generation of the ICMP error message. Furthermore, in SRv6 networks the P routers may be IPv6-only (i.e., may not support the protocol (e.g. IPv4) or address space used by the VPN clients). The Uniform Model defined in allows visibility of traversed nodes outside the provider network. This is acheived by allowing the time-to-live (TTL) or hop-limit (HL) fields to propagate during the SRv6 encapsulation of the customer packet. This document proposes solutions that are capable to allow (1) ping or trace for VPN endpoints with transport network visibility; and (2) find broken link or node within the transport network (e.g., SRv6 domain).

This document uses the Segment Routing terminology established in and in . The reader is assumed to be familiar with those documents and their terminology. The following terms used within this document are defined in : Segment Routing, SR domain, Segment ID (SID), SRv6, SRv6 SID, Active Segment, and SR Policy. The following terms used within this document are defined in : Segment Routing Header (SRH). The following terms used within this document are defined in : NH (next header), SL (the Segments Left field of the SRH), FIB (Forwarding Information Base), SA (Source Address), DA (Destination Address), and SRv6 Endpoint Behavior.

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here.

describes how the IPv6 OAM mechanisms can be used to diagnose problems within the SRv6 networks. For example, explains how the existing traceroute mechanisms for PE-to-PE traceroute. However, does not describe a mechanism for a CE-to-CE traceroute when the provider network deploys Uniform Model . shows the reference topology used to describe the ICMP error handling.

| | | +-----+ +----+ +----+ +-----+ CE1----+ PE1 +------+ P1 |------| P2 +------+ PE2 +------CE2 +-----+ +----+ +----+ +-----+ | | |<--------------- VPN -------------->| ]]> In the reference topology: CE1 and CE2 are Customer Edge devices of IPv4 or IPv6 data plane capability. CE1::1 is an IPv6 address assigned to CE1. CE2::1 is an IPv6 address assigned to CE2. CE1.1.1.1 is an IPv4 address assigned to CE1. CE2.2.2.2 is an IPv4 address assigned to CE2. PE1 and PE2 are provider edge nodes. Node PEj has an IPv6 loopback address PEj::1/128. P1 and P2 are provider core nodes. Node Pj has an IPv6 loopback address Pj::1/128. Node Pj might have or might not have IPv4 address; if IPv4 address is assigned, it has an IPv4 loopback address Pj.j.j.j/32. Note: The IPv6 notations used in this document are not syntactically valid. Letters such as 'P', 'j', 'src', and 'dst' are not hexadecimal IPv6 components, but used for better human readability and refer to symbolic placeholders. (SA1,DA1,HL=x,NH=IPv6)(SA2,DA2,HL=y,NH=UDP)(UDP payload) represents an IPv6 packet with: Outer IPv6 header with source address SA1, destination address DA1, and IPv6 as the next header. The hop limit of the packet is x. IPv6 follows the inner IPv6 header with source address SA2 and destination address DA2 with hop limit of y and NH = UDP. (UDP payload) represents the UDP payload of the packet. (SA1,DA1,HL=x,NH=IPv4)(SA2,DA2,TTL=y)(UDP payload) represents a similar IPv6 packet with an IPv4 inner header with TTL of y. Note: The CE nodes can be IPv6 or IPv4 nodes. Please note that traceroute to a SID is exemplified using UDP probes. However, the procedure is equally applicable to other implementations of the traceroute mechanism. The UDP-encoded message to traceroute a SID uses the UDP ports assigned by IANA for "traceroute use". Without loss of generality, the rest of the document focuses on ce-to-ce traceroute in SRv6-VPN. However, the procedure applies to all ICMPv6 error types handling in SRv6-based VPN.

The issue addressed by this draft is illustrated using a traceroute scenario using the reference topology. CE1 and CE2 are IPv6-capable routers. Consider processing the following traceroute packet at P1:

Please note, in PE1 HL propagation is enabled, thus the value of HL in the outer header is propagated from the inner header, i.e.,:

When the packet comes to P1, the hop limit expires at P1. P1 implements the standard procedure from Section 3.3, Section 8, and sends an ICMPv6 erro packet towards PE1 as follows:

When the ICMP error arrives to PE1, PE1 needs to generate an ICMPv6 (or ICMPv4) error message towards the originating CE1. However, the ICMP error message does not contains any local information about the VPN, where CE1 resides in. Therefore, despite the fact that CE1 is directly connected, PE1 cannot route directly the VPN-specific ICMP error messages back to the original source of the packet (i.e., CE1), that triggered the generation of the ICMP error message.

In Method-1 procedure the SRv6 capable P node sends (tunneles) the error towards the end of SRv6 cloud using the VPN SID of the egress PE. The VPN SID of the egress PE is obtained from the invoking packet. This is similar to handling such cases in MPLS network where message is sent (tunneled) to end of the LSP. Once the packet reaches the egress PE, ICMP packet is decapsulated. Subsequent forwarding table lookup on ICMP packet DA results in the ICMPv6 packet being forwarded to the ingress PE using the VPN SID of the Ingress PE. The ingress PE decapsulates the packet and send it to the intended CE.

Method-1 proposes an alternative procedure in which an SRv6-capable P node can generate the ICMP error message towards the originating CE. Method-1 approach is different compared to the procedure documented in , Section 3.3, and , Section 8 - in SRv6 based networks, for sending ICMP Error messages, triggered in response to received IPv6-encapsulated IPv4 or IPv6 packets are trapped due to HL expiration. This alternate procedure of Method-1 does not require support from the PE1 node. Additionally, an implementation might provide means to administratively limit the application of the alternate procedure for sending ICMP Error messages to only IPv6 packets where the destination address of the upper IPv6 header (provider header) is in the SRv6 Locator block, such as 5f00::/16 reserved address block (), or any other block used for SRv6 locators in particular deployment. If such administrative limits are imposed, the alternative procedure on sending ICMP Error messages is not applied to packets with a destination address outside the specified SRv6 locator block. CE-to-CE traceroute from IPv6 customer edge when the customer enables HL propagation, as well as ICMP tunneling, works in a way similar to its MPLS counterpart. The P node where HL expires generates an ICMPv6 Time Exceeded message for the CE that initiated the traceroute request. The ICMP error is sent (tunneled) towards the end of the SRv6 cloud using the VPN SID of the egress PE (similar to MPLS, where it is sent to the end of the LSP). The VPN SID of the egress PE is obtained from the packet that the ICMP error invoking packet. Once the packet reaches egress PE, the ICMPv6 packet is decapsulated. Subsequent forwarding table lookup on the ICMPv6 packet destination address results in the ICMPv6 packet being forwarded to the CE that initiated the traceroute request. The CE-to-CE traceroute from the IPv4 customer edge works in the same fashion as the CE-to-CE traceroute from the IPv6 customer edge. The exception is that it is assumed the P node, where TTL expiry occurs, is capable of generating an ICMPv4 Time Exceeded message. That ensures that the IPv4 CE can consume the ICMP message properly. For cases when the provider router is not capable of generating an ICMPv4 packet, we recommend not enabling TTL propagation.

ICMP Error Handling of Method-1 has the following characteristics: It assumes that there are no failures between ingress-PE and egress-PE. It defines new functions only P nodes. It might provide means to administratively limit the application of the alternate ICMP procedure. It does not result in additional complexity on PE nodes. It involves Egress PE nodes in the forwarding of the ICMP error messages. Note: further details (with all pros and cons) to be added in a latter version of the document with similar structure for all three methods.

While a solution for diagnostics in MPLS VPNs has been created, the solution designed for MPLS based VPN ping or traceroute has many inherited drawbacks. MPLS technology has its special encapsulation, i.e., the MPLS header is a label stack. In case of MPLS, P routers have no options to identify the ingress of the MPLS tunnel, as labels in the header point towards the network egress point. This characteristic restricts the possible solutions to provide VPN-specific ICMP handling in MPLS networks and resulted in involving of egress-PE nodes in the forwarding of the ICMP error messages. IPv6 encapsulation used by SRv6 has an IP SA field referring to the originator of the IP packet, i.e., the ingress endpoint of the SRv6 tunnel. Therefore the MPLS restriction does not have to apply for SRv6 networks. The solution described here takes advantages from this presence of the ingress endpoint information to provide an optimal method and not to change procedures for P nodes. Node functions in the described method are as follows: Ingress PE (ingress node of the SRv6 tunnel): VPN packet encapsulation follows Uniform model. The node adds VPN-specific information to the encapsulated packet (i.e., IP SA=VPN-specific-SID of the Ingress PE) and forwards it over the SRv6 network. P node = Originator of the ICMP error (within the SRv6 domain): it does standard operation, so an ICMP error message is sent to the originator (i.e., the ingress PE) of the SRv6 encapsulated packet, that caused the ICMP message generation (e.g., when the Hop Limit of the packet expired). Ingress PE: it processes the ICMP error message and forwards it to the original source of the (payload) packet, what is located within the VPN context. This processing is done by a VPN-associated-ICMP-process-function and is described in detail in .

shows the reference topology used to describe the ICMP error handling. Packet processing as per Method-2 works as follows: CE1 sends a packet to CE2. PE1 encapsulates the packet in an SRv6 tunnel (using the Uniform model). The IP SA of the encapsulation is a VPN-specific SID of PE1. Encapsulated packet reaches P2 where Hop Limit expires. P2 generates an ICMP Error Message and sends it to PE1, using the VPN-specific SID as an IP DA. PE1 processes the ICMP Error Message according to its VPN-associated-ICMP-process-function and identifies the related VPN instance. PE1 sends the processed ICMP Error Message to CE1. CE1 is informed about the Hop Limit expire event and its network location (i.e., P2). The VPN-specific SID of PE1 refers to the VPN instance where the prefixes of the VPN can be looked up. The VPN-specific SID is allocated by the PE. SIDs processing already defines the upper-layer header steps (as per , section 4.1.1). The upper-layer = ICMPv6, therefore there is no need for extra parsing rules. There might be no need for extra SID allocation for a VPN. The solution uses a SID per VPN, what is allocated for the VPN service. More specifically for a VPN service the PE node can allocate SID(s) per-prefix (e.g., End.DX6) or per-vrf (e.g., End.DT6). The solution uses a per-vrf SID (e.g., End.DT6) in the IP SA of the SRv6 encapsulated packets. For more sophisticated VPN configurations (e.g., Hub-and-Spoke VPN) where multiple VRFs (and SIDs) are configured for a given VPN, the VPN specific SID of PE1 always refers to the VRF instance (and its per-vrf SID) where the prefixes of the connected customer site(s) can be looked up. As the locator part of the VPN-specific SID is routable within the SRv6 domain other PE and P nodes of the SRv6 domain can send/route packets to it. The SRv6 encapsulation process on the ingress PE node needs several input information to construct the outer SRv6 header. One group of information is related to the IP DA and the SRH part of the SRv6 encapsulation. They are derived from the remote service information (e.g., VPN SID on the egress PE) and the SR policy (if exists). The SR policy defines the path to which an ingress PE node steers a packet flow. Applying a SR policy means to select the path (e.g., defined by a SID list) and placing the path descriptors into the IP DA and the SRH fields of the outer SRv6 encapsulation. Another group of information is needed as well for the SRv6 encapsulation, like IP SA, Traffic Class, FlowLabel, HopLimit, NextHeader. They are derived by various local functionalities. The here described solution (Method-2) impacts only the selection of the IP SA. As per the source IP MUST resolve to a unique node in the SRv6 domain, what is fulfilled by the above described VPN-specific SID. All other fields are defined by related RFCs. For example, Traffic Class might be copied from the inner packet, FlowLabel might be locally generated, etc.

The VPN-associated-ICMP-process-function may translate the IP address of the Originator-of-the-ICMPv6-error-message (e.g., a P node) to limit the VPN-specific visibility characteristics. For example, if the SRv6 domain operator does not want to export the real NodeIP or SID values used by the SRv6 domain nodes.

ICMP Error Handling of Method-2 has the following characteristics: It works even in case of failures between ingress-PE and egress-PE and it supports direct localization of failures. It defines new functions only for Ingress PE nodes. It uses a VPN specifc SID as a source address on ingress PE nodes. It does not result in additional complexity on P nodes. It is compliant to existing standards on P nodes, like . It makes P nodes service agnostic and allows building IPv6-only core networks. It does not involve Egress PE nodes in the forwarding of the ICMP error messages. It can hide the SIDs used inside the SRv6 domain and can provide different visibility for served VPNs if needed. Note: further details (with all pros and cons) to be added in a latter version of the document with similar structure for all three methods.

Rules for processing the received ICMP error message at PE1 is a local decision at PE1. PE1 needs to generate an ICMPv6 or ICMPv4 error message towards the originating CE. If the source address of the original probe packet is a local address, PE1 consumes the packet (e.g., this may be an ICMP error associated with a probe generated by PE1 itself). If the source address of the probe packet is not a local address at PE1 (e.g., CE1::1 address is not local), PE1 encapsulate the ICMPv6 or ICMPv4 message with an outer IPv6 header with destination set to the matching VPN SID (PE2:DT6::) and forwards it towards the egress PE. The generated packet loops back from the egress PE towards the invoking CE using the SRv6 data plane. The advantage of this approach is that the node generating ICMP error (P1 in this example) can be a node without SRv6 capability. Node functions in the described method are as follows: Ingress PE (ingress node of the SRv6 tunnel): VPN packet encapsulation follows Uniform model and forwards it over the SRv6 network. It uses a generic source IPv6 (e.g. loopback IPv6 address) when doing the SRv6 encapsulation. P node = Originator of the ICMP error (within the SRv6 domain): it does standard operation, so an ICMP error message is sent to the originator (i.e., the ingress PE) of the SRv6 encapsulated packet, that caused the ICMP message generation (e.g., when the Hop Limit of the packet expired). Ingress PE: it processes/manipulates the ICMP error message and forwards it over the network towards the egress-PE. The manipulated ICMP error message is encapsulated based on the outer header(s) of the invoking packet (found in the payload of the received ICMP error packet). Egress PE: it makes a vrf lookup to identify the location of the original source and sends the packet towards it over the network. Note: further details to be added in a latter version of the document. For example, at step 3 in the above list the ingress PE first process the packet in the global routing table to find out what further processing is needed.

This section illustares for Method-1 the CE-to-CE Traceroute from IPv6 customer edge in an SRv6-VPN network where the provider network deploys Uniform Model . Using the reference topology, consider a scenario where CE1 initiates a traceroute request to CE2. CE1 and CE2 are both IPv6 routers. The tracerout packet with hop limit set to 2 as it leaves CE1 is encoded as follows:

On PE1, HL propagation is enabled, thus HL value from inner packet is propagated to outer header.

Note: In some scenarios, a packet might have multiple transport outer IPv6 headers preceding the customer's inner IPv6 header. TI LFA is an example of such a scenario. Specifically, when SRv6 encapsulation mode is used to construct backup TI LFA next-hop, an additional IPv6 header is pushed on the packet. The proposed procedure handles such scenarios. The traceroute displays the TI-LFA backup path when it is active. However, the illustration does not cover such scenarios. When the packet comes to P1, the hop limit expires at P1. Based on a local policy at P1, P1 doesn't follow the standard procedure described in RFC9259 and sends an ICMPv6 packet towards CE1 as follows:

Essentially, the encapsulation of the generated ICMPv6 packet is constructed as follows: Copy of the original inner IPv6 header and packet (original Traceroute probe). ICMPv6 packet with: SA = IPv6 address of local node DA = IPv6 source address of the original inner IPv6 header HL = 64 IPv6 header with: SA = IPv6 source address of the original IPv6 outer header DA = IPv6 destination address of the original IPv6 outer header HL = 64 When original packet might have multiple "transport" outer IPv6 headers, or Segment Routing Header (SRH). In such case, the procedure is as follows (from inner to outer): Copy of the original most inner IPv6 header and packet (original Traceroute probe). ICMPv6 packet with: SA = IPv6 address of local node DA = IPv6 source address of the original most inner IPv6 header HL = 64 Copy of all "transport" outer IPv6 and Segment Routing headers from the original packet, except for the most outer IPv6 header. IPv6 header with: SA = IPv6 source address of the original IPv6 outer header DA = IPv6 destination address of the original IPv6 most outer header HL = 64 The encapsulated packet as it leaves P2 is as follows: When a packet arrives at PE2, it performs the local END.DT6 (or END.DT46) processing, decapsulating the packet and processing the inner IPv6 header. As the destination in the inner packet is CE1::1, PE2 encapsulates it so it can be delivered to CE1::1. Assuming HL/TTL propagation is enabled to PE2, PE2 generates the following packet: The packet is IPv6 routed back to PE1. When this packet arrives at PE1, PE1 performs the local END.DT6 (or END.DT46) processing, decapsulates the packet, and processes the inner IPv6 header. Assuming HL/TTL propagation is enabled on PE1, the inner packet is IPv6 routed back to the CE1 node as follows: CE1 processes the hop limit exceeded packet sourced at P1. CE1 generates a new Traceroute probe with incremented HL. This time, the packet expires at P2. The processing of the packet at P2 is similar to the packet processing at P1.

In case of IPv6-VPN service the The VPN-associated-ICMP-process-function operation contains the following steps: It processes the received ICMPv6 error message (originated e.g., from a P node within the SRv6 domain). It identifies the related VPN, based on the VPN-specific IP DA value in the received ICMPv6 error message. It modifies the ICMP error message: It removes the SRv6 domain specific encapsulation/header(s) of the received ICMPv6 error message. It identifies the VPN-specific source of the original packet that caused the ICMPv6 error message, based on the invoking packet header part of the ICMPv6 error message payload. It removes the SRv6 domain specific header(s) from the invoking packet header part of the ICMPv6 error message payload. It creates a new header for the ICMP error message, where the IP SA refers to the Originator-of-the-ICMPv6-error-message and the IP DA=SourceIP-of-the-invoking-packet. Forwards the modified ICMP error message according to the local VPN routing table (VRF). From encapsulation perspective the SRv6 VPN service is an IPv6 Tunnel. The described operation of the VPN-associated-ICMP-process-function is inline with the sense of Section 8. of and can be treated as extending it to use the VPN-specific information in the IP SA for finding the the source of the original IPv6 packet. The ingress PE node is the tunnel entry-point node that receives the ICMP error message generated by a node inside the tunnel, i.e., the P node. The VPN-associated-ICMP-process-function differs from Section 8. of by keeping the SA of the original ICMP error message and using it in the relayed ICMP error message for the here described VPN described. This section illustares a VPNv6 Traceroute from a customer host. HostA is behind CE1 and HostB is behind CE2.

Note: further details to be added in a latter version of the document.

This section outlines CE-to-CE traceroute for IPv4 customer edge. Consider the following traceroute packet received at P1 and its processing there: On PE1, TTL/HL propagation is enabled; thus, the TTL value from the inner packet is propagated to the outer header: When the packet reaches P1, its hop limit expires. With ICMP tunneling enabled on P1, it does not follow the standard procedure described in RFC9259 but instead generates and sends an ICMPv4 packet destined to CE1 tunneled via PE2. P1 is assumed to be a node capable of generating ICMPv4. P1 will inspect the payload of the invoking packet and decide to generate an ICMPv4 Time Exceeded message. However, P1 might or might not have a local IPv4 address that could be used as the source address of the ICMPv4 Time Exceeded message. If P1 has a local IPv4 address, it generates and sends an ICMPv4 packet as follows: If P1 has no local IPv4 address, it sends: P1 uses 192.0.0.8 as the dummy address for the IPv4 source of the ICMPv4 message, in accordance with Section 4.8 of RFC 7600. It also attaches an NIO (Node Identification Object) to the ICMPv4 message, as specified in Section 3 of [I-D.ietf-intarea-extended-icmp-nodeid]. The NIO provides information about P1's real IPv6 address. Encapsulation of the ICMPv4 packet generated at P1 is as follows (from inner to outer): Copy of the original inner IPv4 header and packet (original Traceroute probe). ICMPv4 packet with: SA = IPv4 address of local node (if exists), or 198.0.0.8 (if local IPv4 address doesn't exist) DA = IPv4 source address of the original inner IPv4 header TTL = 64 NIO = IPv6 address of local node (only if IPv4 address doesn't exist) IPv6 header with: SA = IPv6 source address of the original IPv6 outer header DA = IPv6 destination address of the original IPv6 outer header HL = 64 When the original packet has multiple "transport" outer IPv6 headers, the procedure is: Copy of the original most inner IPv4 header and packet (original Traceroute probe). ICMPv4 packet with: SA = IPv4 address of local node (if exists), or 198.0.0.8 (if local IPv4 address doesn't exist) DA = IPv4 source address of the original inner IPv4 header TTL = 64 NIO = IPv6 address of local node (only if IPv4 address doesn't exist) Copy of all "transport" outer IPv6 headers from the original packet. IPv6 header with: SA = IPv6 source address of the original IPv6 outer header DA = IPv6 destination address of the original IPv6 most outer header HL = 64 Examples: When a packet arrives at PE2, it performs the local END.DT4 (or END.DT46) processing, decapsulates the packet, and processes the inner IPv4 header. As the destination is CE1.1.1.1, PE2 re-encapsulates the packet for delivery to CE1. Assuming TTL propagation is enabled, PE2 generates the following: Once received at PE1, it performs END.DT4 (or END.DT46), decapsulates the packet, and routes it to CE1: CE1 processes the ICMPv4 Time Exceeded message sourced at P1. If the packet has IPv4 source address 192.0.0.8 with an NIO, CE1 does not use the IPv4 address but instead uses the NIO for operator display. If CE1 does not support NIO, it displays the IPv4 address 192.0.0.8. CE1 then generates a new traceroute probe with incremented TTL. This time, the packet expires at P2. P2's processing of the packet is similar to that of P1.

In case of IPv4-VPN service the VPN-associated-ICMP-process-function operates as follows (v4/v6 are noted for clarity): It processes the received ICMPv6 error message (originated e.g., from a P node within the SRv6 domain). It identifies the related VPN, based on the VPN-specific IP DA value in the received ICMPv6 error message. It synthesizes an ICMPv4 error message based on the received ICMPv6 error message: It identifies the VPNv4 specific source of the original IPv4 packet that caused the ICMPv6 error message, based on the invoking packet header parts of the ICMPv6 error message payload. It creates the header for the ICMPv4 error message, in accordance with (Section 4.8) and (Section 3), i.e., IPv4 SA=192.0.0.8, Node Identification Object containing the IPv6 SA of the ICMPv6 error message and IPv4 DA=IPv4-SA-of-the-original-packet. Forwards the modified ICMPv4 error message according to the local VPNv4 routing table (VRF). When PE node is aware of the IPv4 address of the SRv6 node that generated the ICMPv6 error message, then the PE node may use it as the IPv4 SA of the synthesized ICMPv4 message. How the PE node is aware of that information is out-of-scope in this document. This section illustares a VPNv4 Traceroute from a customer host.

Note: further details to be added in a latter version of the document.

Some network scenarios result in a packet having multiple transport outer IPv6 headers preceding the customer's inner IP header. Note: further details to be added in a latter version of the document.

Network scenarios with daisy-chain of multiple technologies are left for further analysis. Note: further details to be added in a latter version of the document.

To be added in a latter version of the document.

This document does not impose any additional security challenges to be considered beyond the security threats described in .

This document makes no IANA requests.

Contributors Verizon Inc.

gyan.s.mishra@verizon.com

Cisco Systems, Inc.

sijoy@cisco.com

Authors extend their appreciation to Janos Farkas, Ferenc Fejes, Xiao Min, Liu Yao, Greg Mirsky, Syed Kamran Raza, and Mustapha Aissaoui for their insightful comments and productive discussion that helped to improve the document.