Export of On-Path Delay
in IPFIXSwisscomBinzring 17Zurich8045Switzerlandthomas.graf@swisscom.comHuaweibenoit.claise@huawei.comINSA-LyonLyonFrancealex.huang-feng@insa-lyon.frThis document introduces new IP Flow Information Export (IPFIX)
information elements to expose the On-Path Telemetry measured delay on
the IOAM transit and decapsulation nodes.Network operators want a statistical delay view of their networks.
They want to understand where in the network, for which customer
traffic, how much and why delay is being accummlated. In order to answer
why and where, delay needs to be reported into device and control-plane
context. In order to understand which customer traffic is affected,
delay needs to be reported into customer data-plane context. That
enables network operators to quickly identify when the control-plane
updates the current path with a different next-hop and therefore the
forwarding path changes to different nodes and interfaces, how the path
delay changes for which customer traffic.With On-Path Telemetry, described in the Network Telemetry Framework and applied in In-situ OAM, Path Tracing and In-situ Flow Information
Telemetry, the path delay between two endpoints is measured by
inserting a timestamp in the packet.On-Path Telemetry can be distinguished between two modes. Passport
mode, , where only the last hop in the
forwarding path of the On-Path Telemetry domain exposes all the metrics,
and postcard mode, , where the metrics are
also exposed in the transit nodes. In both modes the forwarding path
exposes performance metrics allowing to determine how much delay has
been accumulated on which hop.This document defines eight new IPFIX Information Elements (IEs),
exposing the On-Path delay on IOAM transit and decapsulation nodes.
Since these IPFIX IEs are performance metrics ,
they must be registered as registered performance metrics in the "IANA
Performance Metric Registry.The delay is measured by calculating the difference between the
timestamp imposed with On-Path Telemetry in the packet at the IOAM
encapsulation node and the timestamp exported in the IPFIX flow record
from the IOAM transit and decapsulation nodes. Depending on the IE, the
lowest, highest or the sum of measured path delay is being exported.On the usecase showed in using IOAM to
export the delay metrics, the node R2 exports the delay D1, the node R3
exports the delay D2 and the decapsulation node R4 exports the total
delay D3 using IPFIX.The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
"OPTIONAL" in this document are to be interpreted as described in BCP 14
when, and only when,
they appear in all capitals, as shown here.This document makes use of the terms defined in and .The following terms are used as defined in .IPFIXIPFIX Information ElementsTemplateTemplate RecordOptions TemplateOptions Template RecordData RecordData SetThe following terms are used as defined in .IOAM encapsulation nodeIOAM transit nodeIOAM deacsulation nodeThis section defines and describes the new performance metrics by
applying the template defined in .This section specifies four Registry Entries for the Hybrid Type I
Passive assessment of IP One-Way Delay.All column entries besides the ID, Name, Description, and Output
Reference Method categories are the same; thus, this section defines
four closely related Registry Entries. As a result, IANA has assigned
corresponding URLs to each of the four Named Metrics.This category includes multiple indexes to the Registry Entry:
the element ID and Metric Name.<insert a numeric Identifier, an integer, TBD>IANA has allocated the numeric Identifiers TBD1-4 for the four
Named Metric Entries in this sectionTBD1: OWDelay_HybridType1_Passive_IP_RFCTBD_Seconds_MeanTBD2: OWDelay_HybridType1_Passive_IP_RFCTBD_Seconds_MinTBD3: OWDelay_HybridType1_Passive_IP_RFCTBD_Seconds_MaxTBD4: OWDelay_HybridType1_Passive_IP_RFCTBD_Seconds_SumURL: URL: URL: URL: This metric assesses the one-way delay of IP packets constituting
a single connection between two hosts. We consider the measurement
of one-way delay based on a single Observation Point (OP) somewhere in the network. The output is the
one-way delay for all successfully forwarded packets expressed as
the <statistic> of their conditional delay distribution, where
<statistic> is one of:MeanMinMaxSumIETF1.0This category includes columns to prompt the entry of all necessary
details related to the metric definition, including the immutable
document reference and values of input factors, called "Fixed
Parameters".Almes, G., Kalidindi, S., Zekauskas, M., and A. Morton, Ed., "A
One-Way Delay Metric for IP Performance Metrics (IPPM)", STD 81, RFC
7679, DOI 10.17487/RFC7679, January 2016,
<https://www.rfc-editor.org/info/rfc7679>. Morton, A. and E. Stephan, "Spatial Composition of Metrics", RFC
6049, DOI 10.17487/RFC6049, January 2011,
<https://www.rfc-editor.org/info/rfc6049>. Section 3.4 of provides the reference
definition of the singleton (single value) one-way delay metric.
Section 4.4 of provides the reference
definition expanded to cover a multi-value sample. Note that terms
such as "singleton" and "sample" are defined in section 2 of .With the OP typically located between
the hosts participating in the IP connection, the one-way delay
metric requires one individual measurement between the OP and
sourcing host, such that the Spatial Composition of the measurements yields a one-way delay
singleton.This category includes columns for references to relevant sections
of the RFC(s) and any supplemental information needed to ensure an
unambiguous method for implementations.The foundational methodology for this metric is defined in
section 4 of using the Timestamps option
with modifications that allow application at a mid-path OP .N/AThe Fixed Parameters above give a portion of the Traffic Filter.
Other aspects will be supplied as Runtime Parameters (below).This metric requires a partial sample of all packets that qualify
according to the Traffic Filter criteria.Runtime Parameters are input factors that must be determined,
configured into the measurement system, and reported with the
results for the context to be complete.The IP address of the host in the host A Role
(format ipv4&nbhy;address-no-zone value for IPv4 or
ipv6-address-no-zone value for IPv6; see section 4 of .The IP address of the host in the host B Role
(format ipv4&nbhy;address-no-zone value for IPv4 or
ipv6-address-no-zone value for IPv6; see section 4 of .Set at desired value.Set at desired value.Set at desired value.The timestamp when the packet is being
received at IOAM encapsulation node. Format depends on On-Path
Telemetry implementation. For IOAM, Section 4.4.1 of describes what kind of timestamps are
supported. Section 4.4.2.3 and 4.4.2.4 describe where the
timestamp is being inserted. For Path Tracing, Section 4.1 of
describes what
kind of timestamps are supported. Section 9.2 describe the SRH
path tracing TLV where the timestamp is being inserted.Launches the IP packet to open the
connection. The Role of "host A" is synonymous with the IP
address used at host A.Receives the IP packet to open the
connection. The Role of "host B" is synonymous with the IP
address used at host B.Receives the IP packet to open
the connection and encapsulates the timestamp into the packet.
The Role of "Encapsulation Node" is synonymous with the
timestamp inserted in the packet.Receives the IP packet to open the
connection and measures the delay between the timestamp in the
packet and the timestamp when the packet was received.Receives the IP packet to open
the connection and measures the delay between the timestamp in
the packet and the timestamp when the packet was received and
removes the IOAM header from the packet.This category specifies all details of the output of measurements
using the metric.OWDelay Types are discussed in the subsections below.For all output types:The one-trip delay
of one IP packet is a SingletonFor each <statistic>, Singleton one of the following
subsections applies.The mean SHALL be calculated using the conditional distribution
of all packets with a finite value of one-way delay (undefined
delays are excluded) -- a single value, as follows:See section 4.1 of for details on the
conditional distribution to exclude undefined values of delay, and
see section 5 of for background on this
analysis choice.See section 4.2.2 of for details on
calculating this statistic; see also section 4.2.3 of .The time value of the result is expressed
in units of seconds, as a positive value of type decimal64
with fraction digits = 9 (see section 9.3 of ) with a resolution of
0.000000001 seconds (1.0 ns), and with lossless
conversion to/from the 64-bit NTP timestamp as per section 6
of .The minimum SHALL be calculated using the conditional
distribution of all packets with a finite value of one-way delay
(undefined delays are excluded) -- a single value, as follows:See section 4.1 of for details on the
conditional distribution to exclude undefined values of delay, and
see section 5 of for background on this
analysis choice.See section 4.3.2 of for details on
calculating this statistic; see also section 4.3.3 of .The time value of the result is expressed
in units of seconds, as a positive value of type decimal64
with fraction digits = 9 (see section 9.3 of ) with a resolution of
0.000000001 seconds (1.0 ns), and with lossless
conversion to/from the 64-bit NTP timestamp as per section 6
of .The maximum SHALL be calculated using the conditional
distribution of all packets with a finite value of one-way delay
(undefined delays are excluded) -- a single value, as follows:See section 4.1 of for details on the
conditional distribution to exclude undefined values of delay, and
see section 5 of for background on this
analysis choice.See section 4.3.2 of for a closely
related method for calculating this statistic; see also section
4.3.3 of . The formula is as follows:The time value of the result is expressed
in units of seconds, as a positive value of type decimal64
with fraction digits = 9 (see section 9.3 of ) with a resolution of
0.000000001 seconds (1.0 ns), and with lossless
conversion to/from the 64-bit NTP timestamp as per section 6
of .The sum SHALL be calculated using the conditional distribution
of all packets with a finite value of one-way delay (undefined
delays are excluded) -- a single value, as follows:See section 4.1 of for details on the
conditional distribution to exclude undefined values of delay, and
see section 5 of for background on this
analysis choice.See section 4.3.5 of for details on
calculating this statistic. However in this case FiniteDelay or
MaxDelay MAY be used.The time value of the result is expressed
in units of seconds, as a positive value of type decimal64
with fraction digits = 9 (see section 9.3 of ) with a resolution of
0.000000001 seconds (1.0 ns), and with lossless
conversion to/from the 64-bit NTP timestamp as per section 6
of .The <statistic> of one-way delay is expressed in seconds,
where <statistic> is one of:MeanMinMaxSumThe one-way delay of the IP connection singleton is expressed
in seconds.Passive Measurements at an OP could be calibrated against an
Active Measurement at host A where the Active Measurement
represents the ground truth.CurrentThis RFC1.0RFC DateNoneThis section defines and describes the new IPFIX IEs. 16-bit unsigned integer that identifies the mean
path delay in microseconds, between the IOAM encapsulation node and
the local node with the IOAM domain (either an IOAM transit node or
an IOAM decapsulation node).
32-bit unsigned integer that identifies the mean path delay in
nanoseconds, between the IOAM encapsulation node and the local node
with the IOAM domain (either an IOAM transit node or an IOAM
decapsulation node).
16-bit unsigned integer that identifies the lowest path delay in
microseconds, between the IOAM encapsulation node and the local node
with the IOAM domain (either an IOAM transit node or an IOAM
decapsulation node).
32-bit unsigned integer that identifies the lowest path delay in
nanoseconds, between the IOAM encapsulation node and the local node
with the IOAM domain (either an IOAM transit node or an IOAM
decapsulation node).
16-bit unsigned integer that identifies the highest path delay in
microseconds, between the IOAM encapsulation node and the local node
with the IOAM domain (either an IOAM transit node or an IOAM
decapsulation node).
32-bit unsigned integer that identifies the highest path delay in
nanoseconds, between the IOAM encapsulation node and the local node
with the IOAM domain (either an IOAM transit node or an IOAM
decapsulation node).
32-bit unsigned integer that identifies the sum of the path delay in
microseconds, between the IOAM encapsulation node and the local node
with the IOAM domain (either an IOAM transit node or an IOAM
decapsulation node).
64-bit unsigned integer that identifies the sum of the path delay in
nanoseconds, between the IOAM encapsulation node and the local node
with the IOAM domain (either an IOAM transit node or an IOAM
decapsulation node).The measured On-Path delay can be aggregated with Flow Aggregation as
defined in to the following device and
control-plane dimensions to determine:With node id and egressInterface(IE14), on which node which
logical egress interfaces have been contributing to how much
delay.With node id and egressPhysicalInterface(253), on which node
which physical egress interfaces have been contributing to how much
delay.With ipNextHopIPv4Address(IE15) or ipNextHopIPv6Address(IE62),
the forwarding path to which next-hop IP contributed to how much
delay.With mplsTopLabelIPv4Address(IE47) or srhActiveSegmentIPv6 from
, the forwarding
path to which MPLS top label IPv4 address or SRv6 active segment
contributed to how much delay.BGP communities are often used for setting a path priority or
service selection. With bgpDestinationExtendedCommunityList(488) or
bgpDestinationCommunityList(485) or
bgpDestinationLargeCommunityList(491) which group of prefixes
accumulated at which node how much delay.With destinationIPv4Address(13), destinationTransportPort(11),
protocolIdentifier (4) and sourceIPv4Address(IE8), the forwarding
path delay on each node from each IPv4 source address to a specific
application in the network.Taking figure 1 from section 1 as topology example. Below example
table shows the aggregated delay per each node, egressInterface and
srhActiveSegmentIPv6.This document requests IANA to create new performance metrics under
the "Performance Metrics" registry with the
values defined in section 2.This document requests IANA to create new IEs (see table 3) under
the "IPFIX Information Elements" registry
available at "IANA Performance Metric
Registry and assign the following initial code points.Note to the RFC-Editor:Please replace TBD5 - TBD12 with the values allocated by
IANAPlease replace the [RFC-to-be] with the RFC number assigned to
this document
Name:
PathDelayMeanDeltaMicroseconds
ElementID:
TBD5
Description:
This Information Element identifies the mean path delay
between the IOAM encapsulation node and the local node with the
IOAM domain (either an IOAM transit node or an IOAM decapsulation
node) in microseconds.
Abstract Data Type:
unsigned16
Data Type Semantics:
OctedDelta
Reference:
[RFC-to-be], xxx
Name:
PathDelayMeanDeltaNanoseconds
ElementID:
TBD6
Description:
This Information Element identifies the mean path delaybetween
the IOAM encapsulation node and the local node with the IOAM
domain (either an IOAM transit node or an IOAM decapsulation node)
in nanoseconds.
Abstract Data Type:
unsigned32
Data Type Semantics:
OctedDelta
Reference:
[RFC-to-be], xxx
Name:
PathDelayMinDeltaMicroseconds
ElementID:
TBD7
Description:
This Information Element identifies the lowest path delay
between the IOAM encapsulation node and the local node with the
IOAM domain (either an IOAM transit node or an IOAM decapsulation
node) in microseconds.
Abstract Data Type:
unsigned16
Data Type Semantics:
OctedDelta
Reference:
[RFC-to-be], xxx
Name:
PathDelayMinDeltaNanoseconds
ElementID:
TBD8
Description:
This Information Element identifies the lowest path delay
between the IOAM encapsulation node and the local node with the
IOAM domain (either an IOAM transit node or an IOAM decapsulation
node) in nanoseconds.
Abstract Data Type:
unsigned32
Data Type Semantics:
OctedDelta
Reference:
[RFC-to-be], xxx
Name:
PathDelayMaxDeltaMicroseconds
ElementID:
TBD9
Description:
This Information Element identifies the highest path delay
between the IOAM encapsulation node and the local node with the
IOAM domain (either an IOAM transit node or an IOAM decapsulation
node) in microseconds.
Abstract Data Type:
unsigned16
Data Type Semantics:
OctedDelta
Reference:
[RFC-to-be], xxx
Name:
PathDelayMaxDeltaNanoseconds
ElementID:
TBD10
Description:
This Information Element identifies the highest path delay
between the IOAM encapsulation node and the local node with the
IOAM domain (either an IOAM transit node or an IOAM decapsulation
node) in nanoseconds.
Abstract Data Type:
unsigned32
Data Type Semantics:
OctedDelta
Reference:
[RFC-to-be], xxx
Name:
PathDelaySumDeltaMicroseconds
ElementID:
TBD11
Description:
This Information Element identifies the sum of the path delay
between the IOAM encapsulation node and the local node with the
IOAM domain (either an IOAM transit node or an IOAM decapsulation
node) in microseconds.
Abstract Data Type:
unsigned32
Data Type Semantics:
OctedDelta
Reference:
[RFC-to-be], xxx
Name:
PathDelaySumDeltaNanoseconds
ElementID:
TBD12
Description:
This Information Element identifies the sum of the path delay
between the IOAM encapsulation node and the local node with the
IOAM domain (either an IOAM transit node or an IOAM decapsulation
node) in nanoseconds.
Abstract Data Type:
unsigned64
Data Type Semantics:
OctedDelta
Reference:
[RFC-to-be], xxx
The same recommendation as defined in section 4.5 of for IPFIX applies in terms of clock precision to
this document as well.The mean (average) path delay can be calculated by dividing the
PathDelaySumDeltaMicroseconds(TBD5) or
PathDelaySumDeltaNanoseconds(TBD6) by the packetDeltaCount(2) at the
IPFIX data collection.For IOAM, Section 4.4.1 of describes what
kind of timestamps are supported. Section 4.4.2.3 and 4.4.2.4 describe
where the timestamp is being inserted.For Path Tracing, Section 4.1 of describes what kind of
timestamps are supported. Section 9.2 describe the SRH path tracing
TLV where the timestamp is being inserted.There are no significant extra security considerations regarding the
allocation of these new IPFIX IEs compared to .The authors would like to thank Al Morton and Greg Mirsky for their
review and valuable comments.IANA Performance Metric Registry