Metrics and Methods for IP
CapacityAT&T Labs200 Laurel Avenue SouthMiddletown,NJ07748USA+1 732 420 1571+1 732 368 1192acm@research.att.comDeutsche TelekomHeinrich Hertz Str. 3-7Darmstadt64295Germany+49 6151 5812747Ruediger.Geib@telekom.deAT&T Labs200 Laurel Avenue SouthMiddletown,NJ07748USAlencia@att.comThis memo revisits the problem of Network Capacity metrics first
examined in RFC 5136. The memo specifies a more practical Maximum
IP-layer Capacity metric definition catering for measurement purposes,
and outlines the corresponding methods of measurement.The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
"OPTIONAL" in this document are to be interpreted as described in BCP
14 when, and only when,
they appear in all capitals, as shown here.The IETF's efforts to define Network and Bulk Transport Capacity have
been chartered and progressed for over twenty years. Over that time, the
performance community has seen development of Informative definitions in
RFC 3148 for Framework for Bulk Transport Capacity (BTC), RFC 5136 for
Network Capacity and Maximum IP-layer Capacity, and the Experimental
metric definitions and methods in , Model-Based
Metrics for BTC.This memo recognizes the importance of a definition of a Maximum
IP-layer Capacity Metric at this time, a definition that is both
practical and effective for the performance community's needs, including
Internet users. The metric definition is intended for Active Methods of
Measurement , and a method of measurement is
included here.The most direct active measurement of IP-layer Capacity would use IP
packets, but in practice a transport header is needed to traverse
address and port translators. UDP offeres the most direct assessment
possibility, and in the [copycat] measurement
study to investigate whether UDP is viable as a general Internet
transport protocol, the authors found that a high percentage of paths
tested support UDP transport. A number of liaisons have been exchanged
on this topic [ refs to ITU-T SG12, ETSI STQ, BBF liaisons ], discussing
the laboratory and field tests that support the UDP-based approach to
IP-layer Capacity measurement.This memo also recognizes the many updates to the IP Performance
Metrics Framework published over twenty years,
and makes use of for Advanced Stream and
Sampling Framework, and RFC 8468 IPv4, IPv6, and
IPv4-IPv6 Coexistence Updates.NOTE: The text contains Author comments, in brackets [RG: , acm:
]The scope of this memo is to define a metric and corresponding method
to unambiguously perform Active measurements of Maximum IP-Layer
Capacity.The main goal is to harmonize the specified metric and method across
the industry, and this memo is the vehicle through which working group
(and eventually, IETF) consensus will be captured and communicated to
achieve broad agreement, and possibly changes in other Standards
Development Organizations (SDO).A local goal is provide efficient test procedures where possible, and
to recommend reporting with additional interpretation of the results.
Also, to foster the development of protocol support for this metric and
method of measurement.As with any problem that has been worked for many years in various
SDOs without any special attempts at coordination, various solutions for
metrics and methods have emerged.There are five factors that have changed (or begun to change) in the
last 5 years, and the presence of any one of them on the path requires
features in the measurement design to account for the changes:Internet access is no longer the bottleneck for many usersBoth speed and latency are important to user's satisfactionUDP's growing role in Transport, in areas where TCP once
dominated.Content and applications moving closer to users.Possibly less traffic crossing ISP gateways in future.This section sets requirements for the following components to
support the Maximum IP-layer Capacity Metric.Type-P-Max-IP-Capacity, or informally called Maximum IP-layer
Capacity.Note that Type-P depends on the chosen method.This section lists the REQUIRED input factors to specify a Route
metric.Src, the address of a host (such as the globally routable IP
address).Dst, the address of a host (such as the globally routable IP
address).i, the limit on the number of Hops a specific packet may visit
as it traverses from the host at Src to the host at Dst (such as
the TTL or Hop Limit).MaxHops, the maximum value of i used, (i=1,2,3,...MaxHops).T, the time at the start of measurement intervalI, the duration of measurement intervaldt, the duration of N equal sub-intervals in ITs, the host time of a transmitted test packet as measured at
MP(Src), meaning Measurement Point at the Source.Ta, the host time of a test packet's *arrival* as measured at
MP(Dst), assigned to packets that arrive within a "reasonable"
time (see parameter below).Tmax, a maximum waiting time for test packets to arrive at the
destination, set sufficiently long to disambiguate packets with
long delays from packets that are discarded (lost), such that the
distribution of one-way delay is not truncated.F, the number of different flows synthesized by the method.flow, the stream of packets with the same n-tuple of designated
header fields that (when held constant) result in identical
treatment in a multi-path decision (such as the decision taken in
load balancing). Note: The IPv6 flow label MAY be included in the
flow definition when routers have complied with guidelines at the Tunnel End Points (TEP), and
the source of the measurement is a TEP.Type-P, the complete description of the packets for which this
assessment applies (including the flow-defining fields). Note that
the UDP transport layer is one requirement specified below.PM, a list of fundamental metrics, such as loss, delay, and
reordering, and corresponding Target performance threshold. At
least one fundamental metric and Target performance threshold MUST
be supplied (such as One-way IP Packet Loss [RFC7680] equal to
zero).This section defines the REQUIRED aspects of the measureable
Maximum IP-layer Capacity metric (unless otherwise indicated) for
measurements between specified Source and Destination hosts:Define the IP-layer capacity, Maximum_C(T,I,PM), to be the maximum
number of IP-layer bits (including header and data fields) that can be
transmitted from the Src host and correctly received by the Dst host
during one contiguous sub-interval, dt in length, during the interval
[T, T+I], and where the packet count of that single sub-interval dtn
in [T, T+I] indicates the maximum number of IP-layer bits
n0[dtn-1,dtn] which was captured as part of all packet counts n for
all dt in [T, T+I]. The interval dt SHOULD be set to a natural number
m so that T+I = T + m*dt with dtn - dtn-1 = dt and with 0 < n <=
m. Parameter PM represents the other performance metrics [see section
Related Round-Trip Delay and Loss Definitions below] and their
measurement results for the maximum IP-layer capacity. At least one
target performance threshold MUST be defined. If more than one target
performance threshold is defined, then the sub-interval with maximum
number of bits transmitted MUST meet all the target performance
thresholds.[RG: this requires more explanation. Do you mean that all results
must hit the target performance level? Or is a one / two / k times hit
out of x trials [T, T+I] a criterium indicating that a target has been
reached? Or do you look for the maximum capacity without packet loss
and queuing or added RTD latency, respectively? acm: I think I've
clarified this in the text above... ]Mathematically, this definition can be represented as:and:n0 is the total number of IP-layer header and payload bits that
can be transmitted in Standard Formed packets from the Src host
and correctly received by the Dst host during one contiguous
sub-interval, dt in length, during the interval [T, T+I],Maximum _C(T,I,PM) corresponds to the maximum value of n0
measured in any sub-interval ending at dtn (meaning T + n*dt),
divided by the constant length of all sub-intervals, dt.all sub-intervals SHOULD be of equal duration. Choosing dt as
non-overlapping consecutive time intervals allows for a simple
implementation. [RG: a sliding window of dt has its charme too,
why exclude it? acm: seems less practical for real-time
feedback.][acm: I think this is a discussion point, not essential to the
definition] If traffic conditioning applies along a path for which
Maximum _C(T,I,PM) is to be determined, different values for dt
SHOULD be picked and measurements be executed during multiple
intervals [T, T+I]. Any single interval dt SHOULD be chosen so
that is an integer multiple of increasing values k times
serialisation delay of a path MTU at the physical interface speed
where traffic conditioning is expected. This should avoid taking
configured burst tolerance singletons as a valid Maximum
_C(T,I,PM) result.The bandwidth of the physical interface of the measurement
device must be higher than that of the interface whose Maximum
_C(T,I,PM) is to be measured.In this definition, the m sub-intervals can be viewed as trials
when the Src host varies the transmitted packet rate, searching for
the maximum n0 that meets the PM criteria measured at the Dst host in
a test of duration, I. When the transmitted packet rate is held
constant at the Src host, the m sub-intervals may also be viewed as
trials to evaluate the stability of n0 and metric(s) in the PM list
over all dt-length intervals in I.Measurements according to these definitions SHALL use UDP transport
layer. [RG: don't we need loss free transmission without added latency
as criteria and add that UDP without closed loop flow control needs to
be applied ? acm: I don't think we should require loss-free
transmission, because most networks allow a small loss ratio which
would likely appear to be zero loss in most trials. Methods may use
feedback, let's talk about how to diffentiate from flow-control]RTD[dtn-1,dtn] is defined as a sample of the Round-trip Delay between the Src host and the Dst
host over the interval [T,T+I]. The statistics used to to summarize
RTD[dtn-1,dtn] MAY include the minimum, maximum, and mean.RTL[dtn-1,dtn] is defined as a sample of the Round-trip Loss between the Src host and the Dst
host over the interval [T,T+I]. The statistics used to to summarize
RTL[dtn-1,dtn] MAY include the lost packet count and the lost packet
ratio.Depending on the@@@@ not yet addressed,This section needs development, based on Annex A of Y.1540.The duration of a test, I, MUST be constrained in a production
network, since this is an active test method and it will likely cause
congestion on the Src to Dst host path during a test.Additional Test methods and configurations may be provided in this
section, after review.A table is pre-built defining all the offered load rates that will
be supported (R1 – Rn, in ascending order). Each rate is defined
as datagrams of size S, sent as a burst of count C, every time
interval T. While it is advantageous to use datagrams of as large a
size as possible, it may be prudent to use a slightly smaller maximum
that allows for secondary protocol headers and/or tunneling without
resulting in IP-layer fragmentation.At the beginning of a test, the sender begins sending at rate R1
and the receiver starts a feedback timer at interval F (while awaiting
inbound datagrams). As datagrams are received they are checked for
sequence number anomalies (loss, out-of-order, duplication, etc.) and
the delay variation is measured (one-way or round-trip). This
information is accumulated until the feedback timer F expires and a
status feedback message is sent from the receiver back to the sender,
to communicate this information. The accumulated statistics are then
reset by the receiver for the next feedback interval. As feedback
messages are received back at the sender, they are evaluated to
determine how to adjust the current offered load rate (Rx).If the feedback indicates that there were no sequence number
anomalies AND the delay variation was below the lower threshold, the
offered load rate is increased. If congestion has not been confirmed
up to this point, the offered load rate is increased by more than one
rate (e.g., Rx+10). This allows the offered load to quickly reach a
near-maximum rate. Conversely, if congestion has been previously
confirmed, the offered load rate is only increased by one (Rx+1).If the feedback indicates that sequence number anomalies were
detected OR the delay variation was above the upper threshold, the
offered load rate is decreased. If congestion has not been confirmed
up to this point, the offered load rate is decreased by more than one
rate (e.g., Rx-30). This allows the offered load to back off enough to
compensate for the fast initial ramp-up. Conversely, if congestion has
been previously confirmed, the offered load rate is only decreased by
one (Rx-1).If the feedback indicates that there were no sequence number
anomalies AND the delay variation was above the lower threshold, but
below the upper threshold, the offered load rate is not changed. This
allows time for recent changes in the offered load rate to stabilize,
and the feedback to represent current conditions more accurately.Lastly, the method for confirming congestion is that there were
sequence number anomalies OR the delay variation was above the upper
threshold for two consecutive feedback intervals.This section needs development...The following text TO BE REPLACED !!!!=======================================The results SHOULD be reported in the format of a table with a row
for each of the tested frame sizes and Number of Flows. There SHOULD be
columns for the frame size with number of flows, and for the resultant
average frame count (or time) for each type of data stream tested.The number of tests Averaged for the Benchmark, N, MUST be
reported.The Minimum, Maximum, and Standard Deviation across all complete
tests SHOULD also be reported.The Corrected DUT Restoration Time SHOULD also be reported, as
applicable.Frame Size, octets + # FlowsMax IP-Layer Capacity, bpsMin,Ave,StdDevTime dt, Sec64,1002600025500,27000,200.00004Static and configuration parameters:Number of test repetitions, NMinimum Step Size (during searches), in frames.Active metrics and measurements have a long history of security
considerations [add references to LMAP Framework, etc.].<There are certainly some new ones for Capacity testing>This memo makes no requests of IANA.Thanks tocopycat: Testing Differential Treatment of New Transport
Protocols in the Wild (ANRW '17)Back2Back Testing Time Series (from CI)Evolution of Repeatability in Benchmarking: Fraser Plugfest
(Summary for IETF BMWG)AT&T LabsSpirent CommunicationsETSI GS NFV-TST 009 V3.1.1 (2018-10), "Network Functions
Virtualisation (NFV) Release 3; Testing; Specification of Networking
Benchmarks and Measurement Methods for NFVI"ETSI Network Function Virtualization
ISG