Internet Engineering Task Force N. Akiya
Internet-Draft Big Switch Networks
Updates: 4379,6424 (if approved) G. Swallow
Intended status: Standards Track Cisco Systems
Expires: January 25, 2016 S. Litkowski
B. Decraene
Orange
J. Drake
Juniper Networks
July 24, 2015

Label Switched Path (LSP) Ping/Trace Multipath Support for Link Aggregation Group (LAG) Interfaces
draft-ietf-mpls-lsp-ping-lag-multipath-01

Abstract

This document defines an extension to the MPLS Label Switched Path (LSP) Ping and Traceroute as specified in RFC 4379. The extension allows the MPLS LSP Ping and Traceroute to discover and exercise specific paths of Layer 2 (L2) Equal-Cost Multipath (ECMP) over Link Aggregation Group (LAG) interfaces.

This document updates RFC4379 and RFC6424.

Requirements Language

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 [RFC2119].

Status of This Memo

This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.

Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at http://datatracker.ietf.org/drafts/current/.

Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."

This Internet-Draft will expire on January 25, 2016.

Copyright Notice

Copyright (c) 2015 IETF Trust and the persons identified as the document authors. All rights reserved.

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License.


Table of Contents

1. Introduction

1.1. Terminology

The following acronyms/terms are used in this document:

1.2. Background

The MPLS Label Switched Path (LSP) Ping and Traceroute as specified in [RFC4379] are powerful tools designed to diagnose all available layer 3 (L3) paths of LSPs, i.e., provides diagnostic coverage of L3 Equal-Cost Multipath (ECMP). In many MPLS networks, Link Aggregation Group (LAG) as defined in [IEEE802.1AX], which provide Layer 2 (L2) ECMP, are often used for various reasons. MPLS LSP Ping and Traceroute tools were not designed to discover and exercise specific paths of L2 ECMP. The result raises a limitation for following scenario when LSP X traverses over LAG Y:

With the above scenario, MPLS LSP Ping and Traceroute will not be able to detect the label switching failure of problematic member link(s) of the LAG. In other words, lack of L2 ECMP diagnostic coverage can produce an outcome where MPLS LSP Ping and Traceroute can be blind to label switching failures over problematic LAG interface. It is, thus, desirable to extend the MPLS LSP Ping and Traceroute to have deterministic diagnostic coverage of LAG interfaces.

Creation of this document was motivated by issues encountered in live networks.

2. Overview

This document defines an extension to the MPLS LSP Ping and Traceroute to describe Multipath Information for LAG member links separately, thus allowing MPLS LSP Ping and Traceroute to discover and exercise specific paths of L2 ECMP over LAG interfaces. Reader is expected to be familiar with mechanics of the MPLS LSP Ping and Traceroute described in Section 3.3 of [RFC4379] and Downstream Detailed Mapping TLV (DDMAP) described in Section 3.3 of [RFC6424].

MPLS echo request carries a DDMAP and an optional TLV to indicate that separate load balancing information for each L2 nexthop over LAG is desired in MPLS echo reply. Responder LSR places the same optional TLV in the MPLS echo reply to provide acknowledgement back to the initiator. It also adds, for each downstream LAG member, a load balance information (i.e. multipath information and interface index). The following figure and the texts provides an example using an LDP network. However the problem and the mechanism is applicable to all types of LSPs which can traverse over LAG interfaces.

  <----- LDP Network ----->

          +-------+
          |       |
  A-------B=======C-------E
          |               |
          +-------D-------+

  ---- Non-LAG
  ==== LAG comprising of two member links

      Figure 1: Example LDP Network
	
  1. Downstream C over Non-LAG (upper path).
  2. First Downstream C over LAG (middle path).
  3. Second Downstream C over LAG (middle path).
  4. Downstream D over Non-LAG (lower path).

This document defines:

Note that the mechanism described in this document does not impose any changes to scenarios where an LSP is pinned down to a particular LAG member (i.e. the LAG is not treated as one logical interface by the LSP).

Also note that many LAGs are built from p2p links, and thus router X and router X+1 have the same number of LAG members. It is possible to build LAGs asymmetrically by using Ethernet switches in the middle. Appendix A lists some cases which this document does not address; if an operator deploys LAGs in a manner similar to what's shown in Appendix A, the mechanisms in this document may not suit them.

3. LSR Capability Discovery

The MPLS Ping operates by an initiator LSR sending an MPLS echo request message and receiving back a corresponding MPLS echo reply message from a responder LSR. The MPLS Traceroute operates in a similar way except the initiator LSR potentially sends multiple MPLS echo request messages with incrementing TTL values.

There has been many extensions to the MPLS Ping and Traceroute mechanism over the years. Thus it is often useful, and sometimes necessary, for the initiator LSR to deterministically disambiguate the difference between: Section 6). When the initiator LSR wishes to discover the capabilities of the responder LSR, the initiator LSR includes the LSR Capability TLV in the MPLS echo request message. When the responder LSR receives an MPLS echo reply message with the LSR Capability TLV included, then the responder LSR MUST include the LSR Capability TLV in the MPLS echo reply message with the LSR Capability TLV describing features and extensions supported by the local LSR.

To allow the initiator LSR to disambiguate the above differences, this document defines the LSR Capability TLV (described in

It is RECOMMENDED that implementations supporting the LAG Multipath extensions defined in this document include the LSR Capability TLV in MPLS echo request messages.

4. Mechanism to Discover L2 ECMP Multipath

4.1. Initiator LSR Procedures

The MPLS echo request carries a DDMAP with the "LAG Description Indicator flag" (G) set in the DS Flags to indicate that separate load balancing information for each L2 nexthop over LAG is desired in MPLS echo reply. The new "LAG Description Indicator flag" is described in Section 7.

4.2. Responder LSR Procedures

This section describes the handling of the new TLVs by nodes which understand the "LAG Description Indicator flag". There are two cases - nodes which understand the "LAG Description Indicator flag" but which for some reason cannot describe LAG members separately, and nodes which both understand the "LAG Description Indicator flag" and are able to describe LAG members separately. Note that Section 6, Section 8 and Section 9 describe the new TLVs referenced by this section , and looking over the definition of the new TLVs first may make it easier to read this section.

A responder LSR that understand the "LAG Description Indicator flag" but is not capable of describing outgoing LAG member links separately uses the following procedures:

A responder LSR that understands the "LAG Description Indicator flag" and is capable of describing outgoing LAG member links separately uses the follow procedures, regardless of whether or not outgoing interfaces include LAG interfaces:

Based on the procedures described above, every LAG member link will have a Local Interface Index Sub-TLV and a Multipath Data Sub-TLV entries in the DDMAP. The order of the Sub-TLVs in the DDMAP for a LAG member link MUST be Local Interface Index Sub-TLV immediately followed by Multipath Data Sub-TLV. A LAG member link may also have a corresponding Remote Interface Index Sub-TLV. When a Local Interface Index Sub-TLV, a Remote Interface Index-Sub-TLV and a Multipath Data Sub-TLV are placed in the DDMAP to describe a LAG member link, they MUST be placed in the order of Local Interface Index Sub-TLV, Remote Interface Index-Sub-TLV and Multipath Data Sub-TLV.

A responder LSR possessing a LAG interface with two member links would send the following DDMAP for this LAG interface:

   0                   1                   2                   3
   0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  ~   DDMAP fields describing LAG interface with DS Flags G set   ~
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |[MANDATORY] Local Interface Index Sub-TLV of LAG member link #1|
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |[OPTIONAL] Remote Interface Index Sub-TLV of LAG member link #1|
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |[MANDATORY] Multipath Data Sub-TLV LAG member link #1          |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |[MANDATORY] Local Interface Index Sub-TLV of LAG member link #2|
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |[OPTIONAL] Remote Interface Index Sub-TLV of LAG member link #2|
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |[MANDATORY] Multipath Data Sub-TLV LAG member link #2          |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |                       Label Stack Sub-TLV                     |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

      Figure 2: Example of DDMAP in MPLS Echo Reply

4.3. Additional Initiator LSR Procedures

The procedures above allow an initiator LSR to:

When an initiator LSR receives a DDMAP containing LAG member information from a downstream LSR with TTL=n, then the subsequent DDMAP sent by the initiator LSR to the downstream LSR with TTL=n+1 through a particular LAG member link MUST be updated with following procedures:

Using the DDMAP example described in the Figure 2, the DDMAP being sent by the initiator LSR through LAG member link #1 to the next downstream LSR should be:

   0                   1                   2                   3
   0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  ~   DDMAP fields describing LAG interface with DS Flags G set   ~
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |[OPTIONAL] Remote Interface Index Sub-TLV of LAG member link #1|
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |             Multipath Data Sub-TLV LAG member link #1         |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |                       Label Stack Sub-TLV                     |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

      Figure 3: Example of DDMAP in MPLS Echo Request

5. Mechanism to Validate L2 ECMP Traversal

Section 4 defines the responder LSR procedures to constructs a DDMAP for a downstream LAG, and also defines that inclusion of the Remote Interface Index Sub-TLVs describing the incoming LAG member links of the downstream LSR is optional. The reason why it is optional for the responder LSR to include the Remote Interface Index Sub-TLVs is that this information from the downstream LSR is often not available on the responder LSR. In such case, the traversal of LAG member links can be validated with procedures described in Section 5.1. If LSRs can provide the Remote Interface Index Sub-TLVs in DDMAP objects, then the validation procedures described in Section 5.2 can be used.

5.1. Incoming LAG Member Links Verification

Without downstream LSRs returning remote Interface Index Sub-TLVs in the DDMAP, validation of the LAG member link traversal requires that initiator LSR traverses all available LAG member links and taking the results through a logic. This section provides the mechanism for the initiator LSR to obtain additional information from the downstream LSRs and describes the additional logic in the initiator LSR to validate the L2 ECMP traversal.

5.1.1. Initiator LSR Procedures

The MPLS echo request is sent with a DDMAP with the "Interface and Label Stack Object Request flag" and "LAG Description Indicator flag" set in the DS Flags to indicate the request for Detailed Interface and Label Stack TLV with additional LAG member link information (i.e. interface index) in the MPLS echo reply.

5.1.2. Responder LSR Procedures

A responder LSR that understands the "LAG Description Indicator flag" but is not capable of describing incoming LAG member link is to use following procedures:

A responder LSR that understands the "LAG Description Indicator flag" and is capable of describing incoming LAG member link MUST use the following procedures, regardless of whether or not incoming interface was a LAG interface:

These procedures allow initiator LSR to:

5.1.3. Additional Initiator LSR Procedures

Along with procedures described in Section 4, the procedures described in this section will allow an initiator LSR to know:

Expectation is that there's a relationship between the interface index of the outgoing LAG member link at TTL=n and the interface index of the incoming LAG member link at TTL=n+1 for all discovered entropies. In other words, set of entropies that load balances to outgoing LAG member link X at TTL=n should all reach the nexthop on same incoming LAG member link Y at TTL=n+1.

With additional logics, the initiator LSR can perform following checks in a scenario where the initiator knows that there is a LAG, with two LAG members, between TTL=n and TTL=n+1, and has the multipath information to traverse the two LAG members.

The initiator LSR sends two MPLS echo request messages to traverse the two LAG members at TTL=1: Appendix A. LAG provisioning models in operated network should be considered when analyzing the output of LSP Traceroute exercising L2 ECMPs.

Note that defined procedures will provide a deterministic result for LAG interfaces that are back-to-back connected between routers (i.e. no L2 switch in between). If there is a L2 switch between LSR at TTL=n and LSR at TTL=n+1, there is no guarantee that traversal of every LAG member link at TTL=n will result in reaching different interface index at TTL=n+1. Issues resulting from LAG with L2 switch in between are further described in

5.2. Individual End-to-End Path Verification

When the Remote Interface Index Sub-TLVs are available from an LSR with TTL=n, then the validation of LAG member link traversal can be performed by the downstream LSR of TTL=n+1. The initiator LSR follows the procedures described in Section 4.3.

The DDMAP validation procedures by the downstream responder LSR are then updated to include the comparison of the incoming LAG member link (which MPLS echo request was received on) to the interface index described in the Remote Interface Index Sub-TLV in the DDMAP. Failure of this comparison results in the return code being set to "Downstream Mapping Mismatch (5)".

A responder LSR that is not able to perform the above additional DDMAP validation procedures is considered to lack the upstream LAG capability. Thus, if the received MPLS echo request contained the LSR Capability TLV, then the responder LSR MUST include the LSR Capability TLV in the MPLS echo reply and the LSR Capability TLV MUST have the "Upstream LAG Info Accomodation flag" cleared.

6. LSR Capability TLV

The LSR Capability object is a new TLV that MAY be included in the MPLS echo request message and the MPLS echo reply message. An MPLS echo request message and an MPLS echo reply message MUST NOT include more than one LSR Capability object. Presence of an LSR Capability object in an MPLS echo request message is a request that a responder LSR includes an LSR Capability object in the MPLS echo reply message, with the LSR Capability object describing features and extensions supported. When the received MPLS echo request message contains an LSR Capability object, an responder LSR MUST include the LSR Capability object in the MPLS echo reply message.

LSR Capability TLV Type is TBD1. Length is 4. The value field of the LSR Capability TLV has following format:

   0                   1                   2                   3
   0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |             Type              |            Length             |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |                      LSR Capability Flags                     |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

      Figure 4: LSR Capability TLV

   0                   1                   2                   3
   0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |                  Must Be Zero (Reserved)                  |U|D|
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

   Flag  Name and Meaning
   ----  ----------------

      U  Upstream LAG Info Accommodation

         An LSR sets this flag when the node is capable of
         describing a LAG member link in the Incoming Interface
         Index Sub-TLV in the in the Detailed Interface and
         Label Stack TLV.

      D  Downstream LAG Info Accommodation

         An LSR sets this flag when the node is capable of
         describing LAG member links in the Local Interface
         Index Sub-TLV and the Multipath Data Sub-TLV in the
         Downstream Detailed Mapping TLV.

7. LAG Description Indicator Flag: G

One flag, G, is added in DS Flags field of the DDMAP TLV. The G flag of the DS Flags field in the MPLS echo request message indicates the request for detailed LAG information from the responder LSR. In the MPLS echo reply message, the G flag MUST be set if the DDMAP TLV describes a LAG interface. It MUST be cleared otherwise.

DS Flags

    0 1 2 3 4 5 6 7
   +-+-+-+-+-+-+-+-+
   | MBZ |G|MBZ|I|N|
   +-+-+-+-+-+-+-+-+

  Flag  Name and Meaning
  ----  ----------------

     G  LAG Description Indicator

        When this flag is set in the MPLS echo request, responder is
        requested to respond with detailed LAG information. When this
        flag is set in the MPLS echo reply, the corresponding DDMAP
        describes a LAG interface.

8. Local Interface Index Sub-TLV

The Local Interface Index object is a Sub-TLV that MAY be included in a DDMAP TLV. Zero or more Local Interface Index object MAY appear in a DDMAP TLV. The Local Interface Index Sub-TLV describes the index assigned by the local LSR to the egress interface.

The Local Interface Index Sub-TLV Type is TBD2. Length is 8, and the Value field has following format:

   0                   1                   2                   3
   0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |             Type              |            Length             |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |    Interface Index Flags      |         Must Be Zero          |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |                     Local Interface Index                     |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

      Figure 5: Local Interface Index Sub-TLV

   0                   1
   0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |   Must Be Zero (Reserved)   |M|
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

  Flag  Name and Meaning
  ----  ----------------

     M  LAG Member Link Indicator

        When this flag is set, interface index described in
        this sub-TLV is a member of a LAG.

Local Interface Index

9. Remote Interface Index Sub-TLV

The Remote Interface Index object is a Sub-TLV that MAY be included in a DDMAP TLV. Zero or more Remote Interface Index object MAY appear in a DDMAP TLV. The Remote Interface Index Sub-TLV describes the index assigned by the downstream LSR to the ingress interface.

The Remote Interface Index Sub-TLV Type is TBD3. Length is 8, and the Value field has following format:

   0                   1                   2                   3
   0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |             Type              |            Length             |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |    Interface Index Flags      |         Must Be Zero          |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |                    Remote Interface Index                     |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

      Figure 6: Remote Interface Index Sub-TLV

   0                   1
   0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |   Must Be Zero (Reserved)   |M|
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

  Flag  Name and Meaning
  ----  ----------------

     M  LAG Member Link Indicator

        When this flag is set, interface index described in
        this sub-TLV is a member of a LAG.

Remote Interface Index

10. Detailed Interface and Label Stack TLV

The "Detailed Interface and Label Stack" object is a TLV that MAY be included in a MPLS echo reply message to report the interface on which the MPLS echo request message was received and the label stack that was on the packet when it was received. A responder LSR MUST NOT insert more than one instance of this TLV. This TLV allows the initiator LSR to obtain the exact interface and label stack information as it appears at the responder LSR.

Detailed Interface and Label Stack TLV Type is TBD4. Length is K + Sub-TLV Length (sum of Sub-TLVs). K is the sum of all fields of this TLV prior to Sub-TLVs, but the length of K depends on the Address Type. Details of this information is described below. The Value field has following format:

   0                   1                   2                   3
   0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |             Type              |            Length             |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  | Address Type  |             Must Be Zero                      |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |                   IP Address (4 or 16 octets)                 |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |                   Interface (4 or 16 octets)                  |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |          Must Be Zero         |        Sub-TLV Length         |   
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  .                                                               .   
  .                      List of Sub-TLVs                         .   
  .                                                               .   
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

      Figure 7: Detailed Interface and Label Stack TLV

         Type #        Address Type           K Octets
         ------        ------------           --------
              1        IPv4 Numbered                16
              2        IPv4 Unnumbered              16
              3        IPv6 Numbered                40
              4        IPv6 Unnumbered              28

10.1. Sub-TLVs

This section defines the sub-TLVs that MAY be included as part of the Detailed Interface and Label Stack TLV.

        Sub-Type    Value Field
        ---------   ------------
          1         Incoming Label stack
          2         Incoming Interface Index

10.1.1. Incoming Label Stack Sub-TLV

The Incoming Label Stack sub-TLV contains the label stack as received by the LSR. If any TTL values have been changed by this LSR, they SHOULD be restored.

Incoming Label Stack Sub-TLV Type is 1. Length is variable, and the Value field has following format:

   0                   1                   2                   3
   0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |             Type              |            Length             |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |                 Label                 | TC  |S|      TTL      |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  .                                                               .
  .                                                               .
  .                                                               .
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |                 Label                 | TC  |S|      TTL      |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

      Figure 8: Incoming Label Stack Sub-TLV

10.1.2. Incoming Interface Index Sub-TLV

The Incoming Interface Index object is a Sub-TLV that MAY be included in a Detailed Interface and Label Stack TLV. The Incoming Interface Index Sub-TLV describes the index assigned by this LSR to the interface which received the MPLS echo request message.

Incoming Interface Index Sub-TLV Type is 2. Length is 8, and the Value field has the same format as the Local Interface Index Sub-TLV described in Section 8, and has following format:

   0                   1                   2                   3
   0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |             Type              |            Length             |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |    Interface Index Flags      |         Must Be Zero          |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |                   Incoming Interface Index                    |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

      Figure 9: Incoming Interface Index Sub-TLV

   0                   1
   0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |   Must Be Zero (Reserved)   |M|
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

  Flag  Name and Meaning
  ----  ----------------

     M  LAG Member Link Indicator

        When this flag is set, interface index described in
        this sub-TLV is a member of a LAG.

Incoming Interface Index

11. Security Considerations

This document extends LSP Traceroute mechanism to discover and exercise L2 ECMP paths. As a result of supporting the code points and procedures described in this document, additional processing are required by initiator LSRs and responder LSRs, especially to compute and handle increasing number of multipath information. Due to additional processing, it is critical that proper security measures described in [RFC4379] and [RFC6424] are followed.

The LSP Traceroute allows an initiator LSR to discover the paths of tested LSPs, providing deep knowledge of the MPLS network. Exposing such information to a malicious user is considered dangerous. To prevent leakage of vital information to untrusted users, a responder LSR MUST only accept MPLS echo request messages from trusted sources via filtering source IP address field of received MPLS echo request messages.

12. IANA Considerations

12.1. LSR Capability TLV

The IANA is requested to assign new value TBD1 for LSR Capability TLV from the "Multiprotocol Label Switching Architecture (MPLS) Label Switched Paths (LSPs) Ping Parameters - TLVs" registry.

  Value   Meaning                                      Reference
  -----   -------                                      ---------
  TBD1    LSR Capability TLV                           this document

12.1.1. LSR Capability Flags

The IANA is requested to create and maintain a registry entitled "LSR Capability Flags" with following registration procedures:

 Registry Name: LAG Interface Info Flags

 Bit number Name                                        Reference
 ---------- ----------------------------------------    ---------
         31 D: Downstream LAG Info Accommodation        this document
         30 U: Upstream LAG Info Accommodation          this document
       0-29 Unassigned

12.2. Local Interface Index Sub-TLV

The IANA is requested to assign new value TBD2 (from the range 4-31743) for the Local Interface Index Sub-TLV from the "Multiprotocol Label Switching Architecture (MPLS) Label Switched Paths (LSPs) Ping Parameters - TLVs" registry, "Sub-TLVs for TLV Types 20" sub-registry.

  Value   Meaning                                      Reference
  -----   -------                                      ---------
  TBD2    Local Interface Index Sub-TLV                this document

12.2.1. Interface Index Flags

The IANA is requested to create and maintain a registry entitled "Interface Index Flags" with following registration procedures:

 Registry Name: Interface Index Flags

 Bit number Name                                        Reference
 ---------- ----------------------------------------    ---------
         15 M: LAG Member Link Indicator                this document
       0-14 Unassigned

Note that this registry is used by the Interface Index Flags field of following Sub-TLVs:

12.3. Remote Interface Index Sub-TLV

The IANA is requested to assign new value TBD3 (from the range 32768-49161) for the Remote Interface Index Sub-TLV from the "Multiprotocol Label Switching Architecture (MPLS) Label Switched Paths (LSPs) Ping Parameters - TLVs" registry, "Sub-TLVs for TLV Types 20" sub-registry.

  Value   Meaning                                      Reference
  -----   -------                                      ---------
  TBD3    Remote Interface Index Sub-TLV               this document

12.4. Detailed Interface and Label Stack TLV

The IANA is requested to assign new value TBD4 for Detailed Interface and Label Stack TLV from the "Multiprotocol Label Switching Architecture (MPLS) Label Switched Paths (LSPs) Ping Parameters - TLVs" registry ([IANA-MPLS-LSP-PING]).

  Value   Meaning                                      Reference
  -----   -------                                      ---------
  TBD4    Detailed Interface and Label Stack TLV       this document

12.4.1. Sub-TLVs for TLV Type TBD4

The IANA is requested to create and maintain a sub-registry entitled "Sub-TLVs for TLV Type TBD4" under "Multiprotocol Label Switching Architecture (MPLS) Label Switched Paths (LSPs) Ping Parameters - TLVs" registry.

Initial values for this sub-registry, "Sub-TLVs for TLV Types TBD4", are described below.

  Sub-Type     Name                                    Reference
  -----------  --------------------------------------  ---------
  1            Incoming Label Stack                    this document
  2            Incoming Interface Index                this document
  3-16383      Unassigned (mandatory TLVs)
  16384-31743  Experimental
  32768-49161  Unassigned (optional TLVs)
  49162-64511  Experimental

12.5. DS Flags

The IANA is requested to assign a new bit number from the "DS flags" sub-registry from the "Multi-Protocol Label Switching (MPLS) Label Switched Paths (LSPs) Ping Parameters - TLVs" registry ([IANA-MPLS-LSP-PING]).

Note: the "DS flags" sub-registry is created by [RFC7537].

 Bit number Name                                        Reference
 ---------- ----------------------------------------    ---------
       TBD5 G: LAG Description Indicator                this document

13. Acknowledgements

The authors would like to thank Nagendra Kumar and Sam Aldrin for providing useful comments and suggestions. The authors would like to thank Loa Andersson for performing a detailed review and providing number of comments.

The authors also would like to extend sincere thanks to the MPLS RT review members who took time to review and provide comments. The members are Eric Osborne, Mach Chen and Yimin Shen. The suggestion by Mach Chen to generalize and create the LSR Capability TLV was tremendously helpful for this document and likely for future documents extending the MPLS LSP Ping and Traceroute mechanism. The suggestion by Yimin Shen to create two separate validation procedures had a big impact to the contents of this document.

14. References

14.1. Normative References

[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997.
[RFC4379] Kompella, K. and G. Swallow, "Detecting Multi-Protocol Label Switched (MPLS) Data Plane Failures", RFC 4379, DOI 10.17487/RFC4379, February 2006.
[RFC6424] Bahadur, N., Kompella, K. and G. Swallow, "Mechanism for Performing Label Switched Path Ping (LSP Ping) over MPLS Tunnels", RFC 6424, DOI 10.17487/RFC6424, November 2011.
[RFC7537] Decraene, B., Akiya, N., Pignataro, C., Andersson, L. and S. Aldrin, "IANA Registries for LSP Ping Code Points", RFC 7537, DOI 10.17487/RFC7537, May 2015.

14.2. Informative References

[I-D.ietf-mpls-ipv6-only-gap] George, W. and C. Pignataro, "Gap Analysis for Operating IPv6-only MPLS Networks", Internet-Draft draft-ietf-mpls-ipv6-only-gap-04, November 2014.
[IANA-MPLS-LSP-PING] IANA, "Multi-Protocol Label Switching (MPLS) Label Switched Paths (LSPs) Ping Parameters"
[IEEE802.1AX] IEEE Std. 802.1AX, "IEEE Standard for Local and metropolitan area networks - Link Aggregation", November 2008.
[RFC5226] Narten, T. and H. Alvestrand, "Guidelines for Writing an IANA Considerations Section in RFCs", BCP 26, RFC 5226, DOI 10.17487/RFC5226, May 2008.

Appendix A. LAG with L2 Switch Issues

Several flavors of "LAG with L2 switch" provisioning models are described in this section, with MPLS data plane ECMP traversal validation issues with each.

A.1. Equal Numbers of LAG Members

R1 ==== S1 ==== R2

A.2. Deviating Numbers of LAG Members

           ____
R1 ==== S1 ==== R2

A.3. LAG Only on Right

R1 ---- S1 ==== R2

A.4. LAG Only on Left

R1 ==== S1 ---- R2

Authors' Addresses

Nobo Akiya Big Switch Networks EMail: nobo.akiya.dev@gmail.com
George Swallow Cisco Systems EMail: swallow@cisco.com
Stephane Litkowski Orange EMail: stephane.litkowski@orange.com
Bruno Decraene Orange EMail: bruno.decraene@orange.com
John E. Drake Juniper Networks EMail: jdrake@juniper.net