MPLS Working Group N. Akiya
Internet-Draft Big Switch Networks
Updates: 6790 (if approved) G. Swallow
Intended status: Standards Track C. Pignataro
Expires: March 9, 2017 Cisco
A. Malis
Huawei Technologies
S. Aldrin
September 5, 2016

Label Switched Path (LSP) and Pseudowire (PW) Ping/Trace over MPLS Network using Entropy Labels (EL)


Multiprotocol Label Switching (MPLS) Label Switched Path (LSP) Ping and Traceroute are methods used to test Equal-Cost Multipath (ECMP) paths. Ping is known as a connectivity verification method and Traceroute as a fault isolation method, as described in RFC 4379. When an LSP is signaled using the Entropy Label (EL) described in RFC 6790, the ability for LSP Ping and Traceroute operations to discover and exercise ECMP paths is lost for scenarios where Label Switching Routers (LSRs) apply different load balancing techniques. One such scenario is when some LSRs apply EL-based load balancing while other LSRs apply non-EL-based load balancing (e.g., IP). Another scenario is when an EL-based LSP is stitched with another LSP which can be EL-based or non-EL-based.

This document extends the MPLS LSP Ping and Traceroute multipath mechanisms in RFC 6424 to allow the ability of exercising LSPs which make use of the EL. This document updates RFC 6790.

Requirements Language

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 [RFC2119].

Status of This Memo

This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.

Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at

Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."

This Internet-Draft will expire on March 9, 2017.

Copyright Notice

Copyright (c) 2016 IETF Trust and the persons identified as the document authors. All rights reserved.

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents ( in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License.

Table of Contents

1. Introduction

1.1. Terminology

The following acronyms and terms are used in this document:

1.2. Background

MPLS implementations employ a wide variety of load balancing techniques in terms of fields used for hash "keys". The mechanisms in [RFC4379] and updated by [RFC6424] are designed to provide multipath support for a subset of techniques. The intent of this document is to provide multipath support for the supported techniques which are compromised by the use of ELs [RFC6790]. Section 10 describes supported and unsupported cases, and it may be useful for the reader to first review this section.

The Downstream Detailed Mapping (DDMAP) TLV [RFC6424] provides Multipath Information which can be used by an LSP Ping initiator to trace and validate ECMP paths between an ingress and egress. The Multipath Information encodings defined by [RFC6424] are sufficient when all the LSRs along the path(s), between ingress and egress, consider the same set of "keys" as input for load balancing algorithms, e.g., either all IP-based or all label-based.

With the introduction of [RFC6790], some LSRs may perform load balancing based on labels while others may be IP-based. This results in an LSP Ping initiator that is unable to trace and validate all the ECMP paths in the following scenarios: [RFC6790] typically have a mixture of nodes that support ELI/EL and nodes that do not. There will also typically be a mixture of areas that support ELI/EL and areas that do not.

These scenarios can be quite common because deployments of

As pointed out in [RFC6790], the procedures of [RFC4379] (and consequently of [RFC6424]) with respect to Multipath Information Type {9} are incomplete. However, [RFC6790] does not actually update [RFC4379]. Further, the specific EL location is not clearly defined, particularly in the case of Flow Aware Pseudowires [RFC6391]. This document defines a new FEC Stack sub-TLV for the entropy label. Section 3 of this document updates the procedures for Multipath Information Type {9} described in [RFC4379] and applicable to [RFC6424]. The rest of this document describes extensions required to restore ECMP discovery and tracing capabilities for the scenarios described.

[RFC4379], [RFC6424], and this document will support IP-based load balancers and label-based load balancers which limit their hash to the first (top-most) or only entropy label in the label stack. Other use cases (refer to Section 10) are out of scope.

2. Overview

[RFC4379] describes LSP traceroute as an operation where the initiating LSR sends a series of MPLS echo requests towards the same destination. The first packet in the series has the TTL set to 1. When the echo reply is received from the LSR one hop away, the second echo request in the series is sent with the TTL set to 2. For each additional echo request the TLL is incremented by one until a response is received from the intended destination. The initiating LSR discovers and exercises ECMP by obtaining Multipath Information from each transit LSR and using a specific destination IP address or specific entropy label.

From here on, the notation {x, y, z} refers to Multipath Information Types x, y, or z. Multipath Information Types are defined in Section 3.3 of [RFC4379].

The LSR initiating LSP Ping sends an MPLS echo request with Multipath Information. This Multipath Information is described in the echo request's DDMAP TLV, and may contain a set of IP addresses or a set of labels. Multipath Information Types {2, 4, 8} carry a set of IP addresses, and Multipath Information Type {9} carries a set of labels. The responder LSR (the receiver of the MPLS echo request) will determine the subset of initiator-specified Multipath Information which load balances to each downstream (outgoing) interface. The responder LSR sends an MPLS echo reply with resulting Multipath Information per downstream (outgoing interface) back to the initiating LSR. The initiating LSR is then able to use a specific IP destination address or a specific label to exercise a specific ECMP path on the responder LSR.

The current behavior is problematic in the following scenarios: [RFC6790]. This document defines a new Multipath Information Type to be used in the DDMAP of MPLS echo request/reply packets for [RFC6790] LSPs.

The above scenarios demonstrate the existing Multipath Information is insufficient when LSP traceroute is used on an LSP with entropy labels

The responder LSR can reply with empty Multipath Information if no IP address is set or label set is received with the Multipath Information. An empty return is also possible if an initiating LSR sends Multipath Information of one type, IP Address or Label, but the responder LSR load balances on the other type. To disambiguate between the two results, this document introduces new flags in the DDMAP TLV to allow the responder LSR to describe the load balancing technique being used.

To use this enhanced method end-to-end, all LSRs along the LSP need to be able to understand the new flags and the new Multipath Information Type. Mechanisms to verify this condition are outside of the scope of this document. The rest of the requirements are detailed in the initiating LSR and responder LSR procedures. Two additional DS Flags are defined for the DDMAP TLV in Section 6. These two flags are used by the responder LSR to describe its load balance behavior on a received MPLS echo request.

Note that the terms "IP-Based Load Balancer" and "Label-Based Load Balancer" are in context of how a received MPLS echo request is handled by the responder LSR.

3. Multipath Type 9

[RFC4379] defined Multipath Type {9} for tracing of LSPs where label-based load balancing is used. However, as pointed out in [RFC6790], the procedures for using this type are incomplete as the specific location of the label was not defined. It was assumed that the presence of Multipath Type {9} implied the value of the bottom-of-stack label should be varied by the values indicated by multipath to determine the respective outgoing interfaces.

Section 5 defines a new FEC-Stack sub-TLV to indicate an entropy label. These labels MAY appear anywhere in a label stack.

Multipath Type {9} applies to the first label in the label stack that corresponds to an EL-FEC. If no such label is found, it applies to the label at the bottom of the label stack.

4. Pseudowire Tracing

This section defines procedures for tracing pseudowires. These procedures pertain to the use of Multipath Information Type {9} as well as Type {TBD4}. In all cases below, when a control word is in use, the N-flag in the DDMAP MUST be set. Note that when a control word is not in use, the returned DDMAPs may not be accurate.

In order to trace a non-flow-aware Pseudowire, the initiator includes an EL-FEC instead of the appropriate PW FEC at the bottom of the FEC stack. Tracing in this way will cause compliant routers to return the proper outgoing interface. Note that this procedure only traces to the end of the MPLS LSP that is under test and will not verify the PW FEC. To actually verify the PW FEC or in the case of a MS-PW, to determine the next pseudowire label value, the initiator MUST repeat that step of the trace (i.e., repeating the TTL value used) but with the FEC Stack modified to contain the appropriate PW FEC. Note that these procedures are applicable to scenarios where an initiator is able to vary the bottom label (i.e., Pseudowire label). Possible scenarios are tracing multiple non-flow-aware Pseudowires on the same endpoints or tracing a non-flow-aware Pseudowire provisioned with multiple Pseudowire labels.

In order to trace a flow-aware Pseudowire [RFC6391], the initiator includes an EL FEC at the bottom of the FEC Stack and pushes the appropriate PW FEC onto the FEC Stack.

In order to trace through non-compliant routers, the initiator forms an MPLS echo request message and includes a DDMAP with Multipath Type {9}. For a non-flow-aware Pseudowire it includes the appropriate PW FEC in the FEC Stack. For a flow-aware Pseudowire, the initiator includes a Nil FEC at the bottom of the FEC Stack and pushes the appropriate PW FEC onto the FEC Stack.

5. Entropy Label FEC

The entropy label indicator (ELI) is a reserved label that has no explicit FEC associated, and has label value 7 assigned from the reserved range. Use the Nil FEC as the Target FEC Stack sub-TLV to account for ELI in a Target FEC Stack TLV.

The entropy label (EL) is a special purpose label with the label value being discretionary (i.e., the label value is not from the reserved range). For LSP verification mechanics to perform its purpose, it is necessary for a Target FEC Stack sub-TLV to clearly describe the EL, particularly in the scenario where the label stack does not carry ELI (e.g., flow-aware Pseudowire [RFC6391]). Therefore, this document defines an EL FEC sub-TLV (TBD1, see Section 12.1) to allow a Target FEC Stack sub-TLV to be added to the Target FEC Stack to account for EL.

The Length is 4. Labels are 20-bit values treated as numbers.

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
|                 Label                 |          MBZ          |

        Figure 1: Entropy Label FEC

6. DS Flags: L and E

Two flags, L and E, are added to the DS Flags field of the DDMAP TLV. Both flags MUST NOT be set in echo request packets when sending, and SHOULD be ignored when received. Zero, one or both new flags MUST be set in echo reply packets.

 DS Flags

     0 1 2 3 4 5 6 7
    |  MBZ  |L|E|I|N|

 Flag  Name and Meaning
 ----  ----------------
    L  Label-based load balance indicator
       This flag MUST be cleared in the echo request. An LSR
       which performs load balancing on a label MUST set this
       flag in the echo reply. An LSR which performs load
       balancing on IP MUST NOT set this flag in the echo

    E  ELI/EL push indicator
       This flag MUST be cleared in the echo request. An LSR
       which pushes ELI/EL MUST set this flag in the echo
       reply. An LSR which does not push ELI/EL MUST NOT set
       this flag in the echo reply.

7. New Multipath Information Type: TBD4

One new Multipath Information Type is added to be used in DDMAP TLV. This new Multipath Type has the value of TBD4.

  Key   Type                  Multipath Information
  ---   ----------------      ---------------------
 TBD4   IP and Label set      IP addresses and label prefixes

Multipath Type TBD4 is comprised of three sections. The first section describes the IP address set. The second section describes the label set. The third section describes another label set which associates to either the IP address set or the label set specified in the other sections.

Multipath Information Type TBD4 has following format:

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
|IPMultipathType|     IP Multipath Length       | Reserved(MBZ) |
~                                                               ~
|                  (IP Multipath Information)                   |
~                                                               ~
|LbMultipathType|    Label Multipath Length     | Reserved(MBZ) |
~                                                               ~
|                 (Label Multipath Information)                 |
~                                                               ~
|  Assoc Label Multipath Length |         Reserved(MBZ)         |
~                                                               ~
|            (Associated Label Multipath Information)           |
~                                                               ~

        Figure 2: Multipath Information Type TBD4

When a section is omitted, the length for that section MUST be set to zero.

8. Initiating LSR Procedures

The following procedure is described in terms of an EL_LSP boolean maintained by the initiating LSR. This value controls the Multipath Information Type to be used in the transmitted echo request packets. When the initiating LSR is transmitting an echo request packet with DDMAP with a non-zero Multipath Information Type, then the EL_LSP boolean MUST be consulted to determine the Multipath Information Type to use.

In addition to procedures described in [RFC4379], as updated by Section 3 and [RFC6424], the initiating LSR MUST operate with the following procedures:

In the following conditions, the initiating LSR may have lost the ability to exercise specific ECMP paths. The initiating LSR MAY continue with "best effort" in the following cases:

9. Responder LSR Procedures

Common Procedures:

The following subsections describe expected responder LSR procedures when the echo reply is to include DDMAP TLVs, based on the local load balance technique being employed. In case the responder LSR performs deviating load balance techniques on a per downstream basis, appropriate procedures matched to each downstream load balance technique MUST be followed.

9.1. IP-based Load Balancer & Not Pushing ELI/EL

9.2. IP Based Load Balancer & Pushes ELI/EL

9.3. Label-based Load Balancer & Not Pushing ELI/EL

9.4. Label-based Load Balancer & Pushes ELI/EL

9.5. Flow-Aware MS-PW Stitching LSR

A stitching LSR that cross-connects flow-aware Pseudowires behaves in one of two ways:

10. Supported and Unsupported Cases

The MPLS architecture does not define strict rules on how implementations are to identify hash "keys" for load balancing purposes. As a result, implementations may be of the following load balancer types:

  1. IP-based load balancer.
  2. Label-based load balancer.
  3. Label- and IP-based load balancer.

For cases (2) and (3), an implementation can include different sets of labels from the label stack for load balancing purpose. Thus the following sub-cases are possible:

  1. Entire label stack.
  2. Top N labels from label stack where the number of labels in label stack is > N.
  3. Bottom N labels from label stack where the number of labels in label stack is > N.

In a scenario where there is one flow label or entropy label present in the label stack, the following further cases are possible for (2b), (2c), (3b) and (3c):

  1. N labels from label stack include flow label or entropy label.
  2. N labels from label stack do not include flow label or entropy label.

Also in a scenario where there are multiple entropy labels present in the label stack, it is possible for implementations to employ deviating techniques:

Furthermore, handling of reserved (i.e., special) labels varies among implementations:

It is important to point this out since the presence of GAL will affect those implementations which include reserved labels for load balancing purposes.

As can be seen from the above, there are many types of potential load balancing implementations. Attempting for any OAM tools to support ECMP discovery and traversal over all types would require fairly complex procedures. Complexities in OAM tools have minimal benefit if the majority of implementations are expected to employ only a small subset of the cases described above. [RFC4379], [RFC6424], and this document support cases (1) and (2a1), where only the first (top-most) entropy label is included when there are multiple entropy labels in the label stack.


11. Security Considerations

While [RFC4379] and [RFC6424] already allow for the discovery and exercise of ECMP paths, this document extends the LSP Ping and Traceroute mechanisms to more precisely discover and exercise ECMP paths when an LSP uses ELI/EL in the label stack. Sourcing or inspecting LSP Ping packets can be used for network reconnaissance.

The extended capability defined in this document requires small additional processing for the responder and initiator nodes. The responder node that pushes ELI/EL will need to compute and return multipath data including associated EL. The initiator node will need to store and handle both IP Multipath and Label Multipath Information, and include destination IP addresses and/or ELs in MPLS echo request packets as well as in Multipath Information sent to downstream nodes. The security considerations of [RFC4379] already cover Denial-of-Service attacks by regulating LSP Ping traffic going to the control plane.

Finally, the security measures described in [RFC4379], [RFC6424], and [RFC6790] are applicable. [RFC6424] provides guidelines if a network operator wants to prevent tracing or does not want to expose details of the tunnel and [RFC6790] provides guidance on the use of the EL.

12. IANA Considerations

12.1. Entropy Label FEC

The IANA is requested to assign a new sub-TLV from the "Sub-TLVs for TLV Types 1, 16, and 21" section from the "Multi-Protocol Label Switching (MPLS) Label Switched Paths (LSPs) Ping Parameters - TLVs" registry ([IANA-MPLS-LSP-PING]).

 Sub-Type Sub-TLV Name          Reference
 -------- ------------          ---------
 TBD1     Entropy label FEC     this document

12.2. DS Flags

The IANA is requested to assign new bit numbers from the "DS flags" sub-registry from the "Multi-Protocol Label Switching (MPLS) Label Switched Paths (LSPs) Ping Parameters - TLVs" registry ([IANA-MPLS-LSP-PING]).

Note: the "DS flags" sub-registry is created by [RFC7537].

 Bit number Name                                        Reference
 ---------- ----------------------------------------    ---------
 TBD2       E: ELI/EL push indicator                    this document
 TBD3       L: Label-based load balance indicator       this document

12.3. Multipath Type

The IANA is requested to assign a new value from the "Multipath Type" sub-registry from the "Multi-Protocol Label Switching (MPLS) Label Switched Paths (LSPs) Ping Parameters - TLVs" registry ([IANA-MPLS-LSP-PING]).

Note: The "Multipath Type" sub-registry is created by [RFC7537].

 Value      Meaning                                  Reference
 ---------- ---------------------------------------- ---------
  TBD4      IP and label set                         this document

13. Acknowledgements

The authors would like to thank Loa Andersson, Curtis Villamizar, Daniel King, Sriganesh Kini, Victor Ji, Acee Lindem, Deborah Brungard, Shawn M Emery, Scott O. Bradner, and Peter Yee for performing thorough reviews and providing most valuable comments.

Carlos Pignataro would like to acknowledge his lifetime friend Martin Rigueiro, with deep gratutide and esteem, for sharing his contagious passion for engineering and sciences, and for selflessly teaching so many lessons.

14. Contributing Authors

Nagendra Kumar
Cisco Systems, Inc.


15. References

15.1. Normative References

[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997.
[RFC4379] Kompella, K. and G. Swallow, "Detecting Multi-Protocol Label Switched (MPLS) Data Plane Failures", RFC 4379, DOI 10.17487/RFC4379, February 2006.
[RFC6424] Bahadur, N., Kompella, K. and G. Swallow, "Mechanism for Performing Label Switched Path Ping (LSP Ping) over MPLS Tunnels", RFC 6424, DOI 10.17487/RFC6424, November 2011.
[RFC6790] Kompella, K., Drake, J., Amante, S., Henderickx, W. and L. Yong, "The Use of Entropy Labels in MPLS Forwarding", RFC 6790, DOI 10.17487/RFC6790, November 2012.
[RFC7537] Decraene, B., Akiya, N., Pignataro, C., Andersson, L. and S. Aldrin, "IANA Registries for LSP Ping Code Points", RFC 7537, DOI 10.17487/RFC7537, May 2015.

15.2. Informative References

[IANA-MPLS-LSP-PING] IANA, "Multi-Protocol Label Switching (MPLS) Label Switched Paths (LSPs) Ping Parameters"
[RFC6391] Bryant, S., Filsfils, C., Drafz, U., Kompella, V., Regan, J. and S. Amante, "Flow-Aware Transport of Pseudowires over an MPLS Packet Switched Network", RFC 6391, DOI 10.17487/RFC6391, November 2011.
[RFC7325] Villamizar, C., Kompella, K., Amante, S., Malis, A. and C. Pignataro, "MPLS Forwarding Compliance and Performance Requirements", RFC 7325, DOI 10.17487/RFC7325, August 2014.

Authors' Addresses

Nobo Akiya Big Switch Networks EMail:
George Swallow Cisco Systems, Inc. EMail:
Carlos Pignataro Cisco Systems, Inc. EMail:
Andrew G. Malis Huawei Technologies EMail:
Sam Aldrin Google EMail: