]>

dnygate@outlook.com

Operations and Management IP Performance Measurement latency conversational RTP endpointing voice This document defines mouth-to-ear response latency (MRL), a performance metric for conversational voice systems, together with an active method for measuring it at the RTP reference point of the calling endpoint. MRL is the interval between the transmission of the final speech sample of a caller's utterance and the arrival of the first sample of the system's response audio. Two variants are defined, one taken at packet arrival and one taken behind a de-jitter buffer of stated target depth. The method is specified so that both timestamps are drawn from a single clock on a single host, so that the metric requires no synchronisation between the measuring endpoint and the system under test. Requirements for stimulus material, capture content, quality control, calibration and reporting are given. About This Document Status information for this document may be found at . Discussion of this document takes place on the IP Performance Measurement Working Group mailing list (), which is archived at . Subscribe at . Source for this draft and an issue tracker can be found at .

Introduction Conversational voice systems built from speech recognition, language modelling and speech synthesis components are now widely deployed over SIP and RTP. Response latency is a primary determinant of whether such a system is usable, and it is measured today by several parties who do not agree on what is being measured. There is no common definition of the quantity, no agreed reference point at which to observe it, and no convention for reporting its uncertainty. Figures produced under different assumptions are routinely compared as though they were commensurable, which is the practical problem this document exists to address. sets out the terms that separate one figure from another, and describes the axes along which current measurement practice divides. This document takes no position on the magnitude of any term, on the accuracy of any published figure, or on the merits of any existing measurement effort. It defines a quantity and a method of observing it, so that figures produced by different parties are comparable.

Motivation and decomposition For a caller connected to a conversational voice system over SIP and RTP, the interval between the end of the caller's utterance and the arrival of response audio comprises, in order of occurrence:

Term
de-jitter buffer depth on the inbound leg
voice activity detection and endpointing decision
speech recognition finalisation after the endpoint decision
orchestration, retrieval and any tool invocation
language model time to first token
speech synthesis time to first audio
encoding, packetisation and media relay
de-jitter buffer depth on the outbound leg

A figure covering one or two of these terms does not predict the interval a caller experiences, and the terms are not separable by observation at the caller. MRL is defined here as the aggregate, observed at a single reference point, precisely because the aggregate is what an external observer can measure without instrumenting the system under test.

Existing practice Two kinds of figure are published today. Component-level figures are reported by the operators of individual components, most commonly the interval from a request reaching a speech synthesis endpoint to the first audio byte returned, and less commonly the interval to a language model's first token. These are observed at an API boundary internal to the system and cover a subset of the terms above. Caller-side figures are produced by placing a call and observing the response. At least one such effort publishes both its results and its tooling openly , reporting the interval between the caller's speech end and the onset of the response as observed in a recording of the call rather than in any timestamp reported by the system. Caller-side efforts differ from one another along axes that render their outputs incomparable, and those axes are the reason this document exists:

• the reference point at which the response is observed, which may be an RTP endpoint, a recording made by a carrier or a conferencing bridge, or an endpoint's audio device, each placing a different and frequently uncharacterised quantity of transport and buffering inside the measured interval. enumerates these so that a party observing anywhere on that list can produce a conforming report which declares the difference, and gives a way to measure it rather than disclose it;
• the determination of the caller's speech end, which may be derived from a voice activity detector applied to the transmitted audio, or from prior annotation of known stimulus material;
• the treatment of buffering, since an interval taken at packet arrival and an interval taken after a de-jitter buffer differ by the depth of that buffer;
• the definition of response onset, which for audio that ramps in gradually is a choice of threshold rather than an observation;
• whether the measuring instrument has itself been calibrated against a known delay, and therefore whether a reported figure is a point estimate or an upper bound.

An effort that states its choices along these axes produces figures a reader can interpret, whereas two efforts that have chosen differently produce figures which cannot be placed side by side even when each is internally correct. This document specifies one set of choices and requires that they be reported alongside any figure derived under them.

Scope This document specifies:

• the definition of the MRL metric and its two required variants;
• the method of measurement, including the reference point, the determination of the interval endpoints, and the required contents of a capture;
• the conditions under which a measurement MUST be treated as invalid;
• the fields that MUST accompany a reported figure.

This document does not specify endpointing strategy, barge-in handling, speech recognition or synthesis behaviour, or any property of the system under test. It does not specify a signalling protocol; the method assumes an established RTP session and is independent of how that session was established.

Relationship to existing work The metric defined here is an application-layer performance metric in the sense of , and this document follows the template in Section 5.4 of that document. defines end-to-end performance metrics for telephony sessions at the SIP layer and is the closest existing IETF work in subject matter; the metric defined here concerns the media plane rather than signalling, and is complementary. No existing IETF document defines the quantity described in or specifies a reference point at which to observe it. The framework of and the delay metric of inform the treatment of error and uncertainty. The method described here is an active method in the taxonomy of , since it generates the stimulus whose response it measures. and its extensions define a reporting mechanism by which endpoints convey media quality metrics in band. Conveying MRL in that manner is out of scope for this document and is noted in as possible follow-on work.

• [[EDITOR'S NOTE, on venue. The IPPM charter bounds the group's work to "metrics and methodologies which are applicable over transport-layer protocols over IP", while also covering "applications running over transport layer protocols". Whether a metric whose dominant terms are speech recognition, inference and synthesis falls inside that boundary is a fair question and should be settled before effort is spent on -01. The precedent in favour is draft-ietf-ippm-responsiveness, an adopted IPPM work item targeting Proposed Standard, which measures at the application layer over HTTP/2 and HTTP/3 and justifies itself explicitly on user experience. The argument against is that responsiveness remains a property of the network under load, whereas most of MRL is not attributable to the network at all. If IPPM declines, the alternatives are to take the question to DISPATCH, which exists for work with no obvious home, or to pursue publication through the Independent Submission stream. The document is written to stand in any of the three. Separately, decide whether to pursue registration in the Performance Metrics Registry (, initially populated by ); holds a draft entry. The registry has to date been populated with IP-layer path metrics, so eligibility should be confirmed before the appendix is presented as a proposal.]]

Conventions and Definitions The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here. The following terms are used throughout.

Calling endpoint:

The endpoint that generates the stimulus utterance and observes the response. All measurement is performed here.

System under test (SUT):

The conversational voice system that receives the stimulus and generates a response. The method treats the SUT as opaque.

Reference point:

The point at which timestamps are taken. See .

Stimulus:

A prerecorded utterance transmitted by the calling endpoint to elicit a response.

Speech end:

The final sample of speech in the stimulus, determined offline. See .

Response onset:

The first sample of the SUT's response audio, determined offline. See .

Metric Definition

Metric name Mouth-to-Ear Response Latency (MRL), reported in two variants named Ingress MRL and Playout MRL.

Metric description MRL is the interval between the instant at which the calling endpoint transmits the final speech sample of a stimulus utterance and the instant at which the first sample of the SUT's response reaches the calling endpoint. MRL is defined as: where t0 and t1 are as defined in and . MRL is signed. A negative value indicates that response audio reached the calling endpoint before the stimulus had finished being transmitted, which occurs with aggressive endpointing and with backchannel responses. Implementations MUST report negative values as measured and MUST NOT clamp them to zero.

Interval start: t0 t0 is the instant at which the final speech sample of the stimulus is transmitted by the calling endpoint at the reference point. t0 MUST be determined by:

1. annotating the stimulus offline to sample precision to locate the speech end;
2. mapping that sample onto the RTP packet that carried it, using RTP timestamps as defined in ;
3. interpolating within that packet at the sample rate to obtain the sample's offset from the packet's transmission instant.

t0 is therefore the transmission instant of the carrying packet plus the target sample's offset within that packet. The sign of the offset term is significant; see . t0 MUST NOT be derived from voice activity detection performed at run time. A run-time detector has a decision lag of its own, that lag is a term within the quantity being measured, and using it to define t0 would conceal the term. Annotation of the stimulus MUST NOT extend the speech end by a fixed constant. A fixed extension displaces t0 later by that constant and, since MRL is t1 minus t0, reduces every reported figure by the same constant. See . An implementation MUST NOT be required to use any particular annotation algorithm, and MUST be able to demonstrate that its annotation agrees with a published reference. The reference implementation uses decay-following hysteresis with a short sliding RMS refinement, and publishes a frozen stimulus signal whose speech end is exact by construction, together with the boundary its own annotator locates . An implementation claiming conformance SHOULD reproduce that boundary and MUST state the deviation where it does not. Specifying the conformance test rather than the algorithm is deliberate. The quantity that matters to a reader is whether two parties locate the same boundary in the same audio, and an algorithm mandated in prose can be implemented differently by two careful people while a boundary in a published waveform cannot be argued about.

• [[EDITOR'S NOTE: the reference signal is frozen bytes rather than a generator seed, because a signal rebuilt by its generator changes when the generator does. Whether the signal itself should be carried in an IANA registry, an appendix, or by reference to the archived implementation is a question for the working group; reference is assumed here.]]

Interval end: t1 t1 is the instant at which the first sample of the SUT's response reaches the calling endpoint, taken at the reference point. t1 MUST be determined by reassembling the received stream in RTP timestamp order, locating the response onset as specified in , and mapping the onset sample back through the packet that carried it. Two variants of t1 are defined, and both MUST be reported.

Ingress MRL Ingress MRL takes t1 at the arrival instant of the onset sample, that is, at the reference point with no buffering applied. Ingress MRL isolates the contribution of the SUT and of the network path from the buffering policy of the calling endpoint. Its variance under a jittered path tracks the jitter of that path, because the metric reports the path as it finds it, and an implementation showing less variance than the path exhibits is smoothing a quantity it was asked to observe.

Playout MRL Playout MRL takes t1 at the instant at which the onset sample would be released from a de-jitter buffer of stated target depth. The target depth MUST be reported with the figure. The de-jitter model used MUST anchor on the minimum transit delay observed over a stated initial window, which is the behaviour adaptive buffers converge toward, rather than on the arrival instant of the first packet received. Playout MRL corresponds to the interval a caller waits and is the variant to compare against conversational turn-taking norms. Playout MRL is distinct from the one-way transmission delay addressed by , and the two are not interchangeable. One-way transmission delay concerns the time taken to carry audio across a path and applies to a conversation between two people, whereas MRL concerns the time a system takes to begin responding and includes transmission delay as one term among several. A system may satisfy the transmission delay guidance in on both legs and still exhibit an MRL an order of magnitude larger.

Both variants required An implementation MUST report both variants. Ingress MRL on its own understates the interval a caller experiences, while a bare Playout MRL leaves the SUT confounded with whatever buffering policy the calling endpoint happened to apply, so neither figure can be interpreted without the other.

Response onset The instant at which audio begins has no unique definition, because synthesised speech commonly ramps in over tens of milliseconds rather than beginning at full level. Onset is therefore computed under three named variants, and the dispersion across them is a component of the reported uncertainty.

Variant	Above noise floor	Absolute floor	Sustained for
sensitive	6 dB	-55 dBov	10 ms
headline	10 dB	-50 dBov	20 ms
strict	12 dB	-45 dBov	30 ms

Levels are expressed in dBov, referenced to full-scale RMS. The headline variant is the reported figure. The spread of MRL across all three variants is the onset-definition uncertainty of the measurement and MUST be published alongside any headline figure. On an abrupt onset the spread is small; on a gradual ramp it grows with the ramp duration and becomes the dominant uncertainty term. Sub-frame refinement of the onset instant MUST use a short sliding RMS. Instantaneous sample magnitude crosses any fixed threshold on isolated noise peaks at a rate high enough to displace the boundary materially; see .

Unprompted audio A conversational voice system commonly speaks before the caller does, opening with a greeting that arrives within a few hundred milliseconds of the session being established. That audio responds to nothing, because the caller has not yet spoken. Response-onset detection MUST NOT begin before the end of any such unprompted audio. The point at which it ended MUST be recorded in the capture, and the noise-floor estimate required by MUST be taken from audio following that point. Detection across the whole received stream locates the greeting instead of the response. Since the greeting precedes t0, the resulting MRL is large and negative, and because this document licenses negative values as genuine behaviour there is no bound against which such an error announces itself. In the reference implementation, a capture carrying an 800 ms greeting with a true MRL of 900 ms yielded -2085 ms and satisfied every condition in . A greeting falling inside the noise-floor window is speech rather than channel noise, so it raises the estimate and displaces every onset threshold derived from it. On the same capture the floor moved by 1.35 dB. A calling endpoint SHOULD NOT begin transmitting its stimulus while unprompted audio is still in progress. A system that implements barge-in detection will stop speaking when it hears the caller, which truncates the greeting and alters the interaction under measurement. The interval from session establishment to the onset of unprompted audio is a distinct and useful quantity, since a caller who hears nothing for several seconds after the line opens is poorly served whatever the system's MRL turns out to be. It shares no terms with MRL and is not defined here; see .

Filler audio and response continuity t1 as defined in is the onset of the system's first response audio, whatever that audio happens to contain. A system that emits an earcon, a breath or a filled pause while its response is still being generated therefore records a low MRL while conveying nothing during that interval, and would rank above a system that stayed quiet and then answered. This document does not resolve that by identifying which audio carries meaning. A metric incorporating a judgement about meaning cannot be re-derived from a published capture by an independent reviewer, and that reproducibility is the property which makes the rest of this specification worth having. The discriminator is structural instead: filler is followed by silence before the substantive response begins, and continuous speech is not. An implementation MUST, within a window of 2000 ms following t1, measure the longest interval whose level lies below the onset threshold of the headline variant. Intervals carried by no packet MUST be excluded from that measurement, because a lost or late-discarded frame leaves a gap indistinguishable from a deliberate pause and would otherwise allow a degraded path to manufacture filler. Where that interval exceeds 150 ms:

• the response MUST be reported as discontiguous;
• a second onset MUST be reported, at the start of the final contiguous segment, together with the MRL derived from it.

MRL itself is unchanged by this section, so no figure measured under an earlier revision becomes invalid. A reader receives both onsets and can see whether they differ and by how much. The 2000 ms window and the 150 ms threshold are fixed by this document rather than left to the implementation, for the reason given in : figures derived under different parameters cannot be compared even when each is internally correct. Both MUST be reported alongside any figure derived under them.

Method of Measurement

Reference point All timestamps MUST be taken at the RTP egress and ingress reference point of the calling endpoint, that is, immediately before a packet is passed to the operating system for transmission and immediately after a packet is received from it. The reference point is chosen so that the measurement includes every term a caller experiences downstream of the calling endpoint's own send path, and excludes the calling endpoint's own playout hardware, which is a property of the observer rather than of the SUT.

Observing elsewhere Not every party measuring this quantity controls an RTP endpoint. A figure derived from a carrier's call recording is a real measurement of something a caller experienced, and declaring it as such is more useful than being unable to describe it at all. An implementation MAY therefore observe at a point other than the one in , provided it declares which, and provided it discloses what that choice places inside the measured interval. Such a report conforms to this document. What it does not do is produce a figure comparable with one taken at the RTP reference point, because the two intervals contain different terms.

reference_point	observed at	additional terms inside the interval
rtp-endpoint	immediately before the packet is passed to the operating system, and immediately after it is received from it	none; this is
host-packet-capture	a packet capture facility on the calling host	the host's own network stack, and asymmetrically: on transmission the capture is taken after the send path, on reception before the application reads
bridge-recording	a session border controller or conferencing bridge in the path	the leg between caller and bridge in both directions, and the bridge's own media handling
carrier-recording	a recording made by a carrier or communications platform	the leg to the carrier in both directions, any transcoding it performs, its buffering, and its recording pipeline
endpoint-audio-device	the caller's audio device, or a loopback capture of it	the endpoint's de-jitter buffer, its audio stack and the device's own latency

Where a value other than rtp-endpoint is declared, the additional terms MUST be disclosed and the figure MUST be reported as an upper bound rather than a point estimate, unless those terms have been characterised as below. Figures taken at different reference points MUST NOT be pooled into one distribution.

Characterising the additional terms An upper bound is a weaker result than it needs to be, and the additional terms are measurable rather than merely acknowledgeable. Where the path to the system under test traverses infrastructure the measuring party does not control, an implementation SHOULD place a second call over the same route to a responder replying at a delay it has programmed, and subtract that measurement from the first. The infrastructure's contribution is common to both and cancels, leaving the system's response. Two routes over the same carrier are not identical, so this bounds the additional terms rather than eliminating them exactly, and the bound is what should be reported. This is the same differential construction the calibration in uses, applied to a term outside the instrument instead of inside it. It turns a disclosed caveat into a measured quantity, and it is available to any party that can dial a number it controls.

Stimulus requirements Stimulus material MUST be prerecorded and MUST be transmitted at the nominal frame rate of the codec in use. The stimulus MUST be hashed and the hash MUST be recorded with the capture, so that a changed stimulus invalidates a comparison loudly rather than silently. The method is independent of the codec, and the codec in use MUST be reported with any figure derived under it. Where a payload format from is used, the sample rate and frame period follow that profile, and the companding of applies to the PCMU and PCMA formats.

Transmission pacing The calling endpoint MUST measure the deviation of its own transmission instants from the nominal frame grid and MUST record the worst deviation observed during the call. An endpoint that cannot pace its own transmission has an unreliable t0 and therefore an unreliable MRL. See .

Capture contents A capture MUST contain raw payloads and raw timestamps only. A capture MUST NOT contain any derived quantity, including any latency figure. This requirement exists so that a revised onset definition, or a definition proposed by a reviewer, can be applied to existing data without repeating a collection. The uncertainty analysis in is not possible otherwise.

Clock requirements Both t0 and t1 MUST be taken from a single monotonic clock on the calling endpoint. Because the interval is the difference of two timestamps drawn from one clock on one host, the metric requires no synchronisation between the calling endpoint and the SUT, and is insensitive to offset between them. This is a deliberate property of the definition and distinguishes the method from one-way delay measurement, where clock synchronisation between the two hosts dominates the error budget as described in Section 3.7.1 of . A wall-clock timestamp MAY be recorded in parallel for the sole purpose of correlating captures with traces obtained from the SUT. Any offset or skew in that clock affects such correlation only and MUST NOT affect the reported MRL.

Sources of error and calibration An implementation MUST state its calibrated accuracy, and MUST state the conditions under which that calibration was obtained. Calibration is performed by replacing the SUT with a reference responder that replies at a programmed delay, so that ground truth is known exactly. Under each channel condition the offset from ground truth that the physics of the channel requires is predictable in advance: for Ingress MRL it is the base transit delay plus the mean jitter excess, and for Playout MRL it is the base transit delay plus the buffer target depth. The calibration criterion is therefore the residual after subtracting that predicted offset, rather than the raw difference from ground truth. Calibration MUST NOT be performed exclusively at programmed delays that are integer multiples of the frame period. A responder that evaluates its emission deadline once per received frame quantises its own output to the frame period, and that error is invisible at frame-commensurate delays because the deadline then falls on a frame boundary. The following error mechanisms are known to produce plausible-looking but incorrect figures, and an implementation is advised to test for each:

Fixed annotation extension:

Extending the stimulus speech end by a constant biases every figure low by that constant.

Instantaneous-magnitude thresholding, at either boundary:

Sub-frame refinement on sample magnitude rather than a short sliding RMS lets isolated noise peaks cross the threshold. At the stimulus end boundary this displaces t0 late; at the response onset it displaces t1 early, biasing MRL low by up to one analysis window. The second was masked in the reference implementation for as long as its calibration responder placed responses on the analysis grid, and surfaced only once a media-clock responder placed them where they began: it accounted for the whole of a -0.40 ms headline bias and a 0.99 ms spread between onset variants on a hard onset. A fix applied at one boundary has to be checked against its mirror.

Frame-quantised reference responder:

Adds a uniform error between zero and one frame period, invisible at frame-commensurate calibration delays.

Frame-offset sign error:

Using the wrong sign for the within-frame offset term of t0 produces a residual of twice the offset with the opposite sign. A large bias accompanied by a tight spread is the signature of a definitional or arithmetic error rather than of host timing noise, and implementations are advised to report that discrimination automatically.

Symmetric jitter modelling:

A simulated channel that models network jitter as zero-mean Gaussian permits a minimum-tracking de-jitter anchor to sit earlier than the minimum transit delay allows, which appears as a negative bias in Playout MRL. Network jitter is one-sided and has a hard floor at the minimum transit delay. Sender pacing deviation is a different mechanism and is symmetric, because a timer-driven sender can fire either early or late.

Calibration source that is not an honest RTP sender:

A reference responder whose RTP timestamps count frames rather than follow a media clock, or whose idle-stream pacing depends on whether it is receiving anything, produces playout figures with errors that ingress cannot see, since ingress is derived from arrival and playout through the timestamp. Observed as a 44.7 ms transit slip that flagged late discard on jitter-free calls, and 9.54 ms of playout spread that no buffer target reduced. Both surfaced only from kept captures over a real path.

• [[EDITOR'S NOTE: the reference implementation's calibrated figures are published in and are deliberately not reproduced here. A metric specification that carries one implementation's results becomes stale and invites the reader to treat those figures as a conformance target. Confirm this is the right call in review.]]

Units of Measurement MRL is reported in milliseconds, signed, to a resolution of not coarser than 0.1 ms. Uncertainty terms accompanying the figure are reported in the same units.

Measurement Points and Measurement Domain The single measurement point is the calling endpoint's RTP reference point as defined in . No observation of the SUT's internal state is required, and none is assumed to be available. The measurement domain is the path between the calling endpoint and the SUT together with the SUT itself. The method does not separate the two, and reported figures are therefore properties of the pairing rather than of the SUT alone. Where separation is required, the path contribution has to be characterised independently and reported alongside.

Measurement Timing Each measurement corresponds to one stimulus and one response. A reported distribution MUST state the number of measurements attempted and the number discarded, as required by .

Turns within a session Several exchanges within one session are permitted and are the realistic case, since a caller does not hang up after one question. They are not interchangeable, though, and a report that pools them without saying so is not comparable with one that does not. A system's first response in a session carries terms that later responses do not: session establishment, whatever a model does on a cold start, caches not yet populated, and connections not yet open. Later responses may in turn benefit from conversational state the system has accumulated. Both effects are real properties of the system rather than measurement artefacts, and both are invisible in an aggregate that does not record which turn each measurement came from. Therefore:

• each measurement MUST record its turn index within the session, counting from one;
• a reported distribution MUST state which turn indices it covers;
• first-turn measurements MUST NOT be pooled with later ones unless the report states that they are pooled and gives the counts of each.

This specifies what must be reported rather than how many turns to measure. A study of cold-start behaviour measuring only first turns, and a study of steady-state behaviour discarding them, are both valid and are answering different questions; what makes them comparable to each other is that both said which they did. A calling endpoint conducting a multi-turn session MUST NOT transmit its next stimulus while the system is still speaking, for the reason given in : a system implementing barge-in detection will stop, which alters the interaction under measurement. Locating the end of the system's turn is the same problem as locating the end of a greeting and admits the same solution.

Quality Control and Result Validity Two classes of condition are defined.

Blocking conditions A measurement exhibiting any of the following MUST be discarded and MUST NOT be reported as a figure:

• no speech end could be located in the stimulus;
• no packets were received from the SUT;
• no response onset was found;
• the response onset fell within the first received frame, so that the true onset may precede the observation window;
• the worst transmission pacing deviation exceeded a stated threshold.

The pacing threshold in the reference implementation is 5 ms. A calling endpoint that cannot pace its own transmission within that bound has an unreliable t0. Discarding a measurement under these conditions is correct behaviour. An implementation MUST NOT relax a blocking condition in order to retain a measurement.

Advisory conditions The following conditions do not invalidate a measurement but MUST be reported with it:

• packet loss above a stated threshold;
• late discard above a stated threshold.

A call over a lossy path remains a valid measurement of a lossy path. Where the frame carrying the response onset is itself lost, onset detection is deferred by whole frames and the resulting figure is an upper bound rather than a point estimate, which is why the flag has to travel with the number.

Reporting of discards Every reported distribution MUST be accompanied by the count of measurements discarded. A run that discards a large fraction of its measurements is not comparable with one that discards none, irrespective of the percentiles of the survivors.

Reporting

Required fields A reported MRL figure MUST be accompanied by:

• Ingress MRL and Playout MRL, both signed;
• the playout target depth;
• the onset variant used for the headline figure, and the spread across all three variants;
• the codec and frame period;
• the number of measurements attempted and the number discarded;
• any advisory flags raised;
• the calibrated accuracy of the measuring implementation and the conditions of that calibration;
• an identifier for the SUT configuration sufficient to establish that two figures refer to the same configuration, without necessarily disclosing that configuration.

Reporting format A reported result MUST be expressible as a JSON object carrying the members defined below. The format exists so that a reader can establish, without contacting the party who produced a figure, whether two figures were derived under the same choices. The schema member MUST be present and MUST be the string mrl-report/1 for reports conforming to this document.

instrument Describes the measuring implementation and its calibration. All members are REQUIRED.

name, version:

Identify the implementation that produced the report.

calibration:

An object recording the outcome of the procedure in . It MUST carry conditions, a human-readable statement of the channel and host conditions under which calibration was performed, and for each of ingress and playout a bias_ms and a p95_abs_error_ms. It SHOULD carry reference, a URI or DOI at which the calibration evidence can be inspected. An implementation that has not been calibrated MUST set calibration to null rather than omitting it, and every figure in such a report is an upper bound rather than a point estimate.

measurement Describes the choices along the axes in . All members are REQUIRED.

reference_point:

Where t1 was observed, taking one of the values enumerated in . Any value other than rtp-endpoint MUST be accompanied by reference_point_notes stating what that choice places inside the measured interval, and by reference_point_bound_ms where the additional terms have been characterised as in , or null where they have not and the figures are therefore upper bounds.

codec, frame_period_ms, sample_rate_hz:

The media parameters in force.

playout_target_ms:

The de-jitter buffer target depth used to derive Playout MRL.

onset_variants:

An array of objects, each carrying name, margin_db, absolute_dbov and sustain_ms. The parameters MUST be stated rather than referenced by name alone, so that a report remains interpretable if the defaults in are ever revised.

headline_variant:

The name of the variant whose figures are quoted as the headline.

continuity_window_ms, continuity_gap_threshold_ms:

The parameters of , stated rather than assumed so that a report remains interpretable if the defaults are ever revised.

subject Identifies what was measured. stimulus_id and stimulus_sha256 are REQUIRED. sut_identifier is REQUIRED and sut_config_sha256 is RECOMMENDED, the latter allowing two reports to be shown to concern the same configuration without that configuration being disclosed.

results Aggregate figures. All members are REQUIRED.

n_attempted, n_reported, n_discarded:

Counts of measurements. n_attempted MUST equal n_reported plus n_discarded.

turn_indices:

The turn indices this distribution covers, and the count of measurements from each. REQUIRED by , which forbids pooling a first turn with later ones without saying so. A single-turn study reports one entry, which is the honest way to say that the figures describe cold starts.

discard_reasons:

An object mapping each blocking condition in to the number of measurements it discarded. The counts MUST sum to n_discarded.

ingress, playout:

Objects carrying at least mean_ms, p50_ms, p95_ms and max_ms, computed over the reported measurements under the headline variant. Values are signed.

onset_definition_uncertainty_ms:

The spread of MRL across all variants in onset_variants, carrying at least p50 and max. This member MUST be present, since a headline figure quoted without it is incomplete under .

continuity:

An object carrying gap_p50_ms and gap_max_ms over the reported measurements, n_discontiguous, and a contiguous object of the same shape as ingress giving the MRL to the start of uninterrupted speech. Where n_discontiguous is zero the contiguous figures equal the ingress figures, which is the expected case and is reported rather than omitted so that its absence never has to be inferred.

advisory_flags:

An object mapping each advisory condition in to the number of reported measurements carrying it.

measurements and captures measurements SHOULD carry one object per individual measurement, each with the measurement's identifier, its session identifier and turn index as required by , its t0 and t1 in nanoseconds on the instrument's monotonic clock, its ingress and playout MRL under every variant, the continuity gap and contiguous onset from , the end of any unprompted audio as required by , and any flags raised. captures SHOULD carry one object per capture with the measurement identifier, a sha256 of the capture file, and a URI at which it can be obtained. Both members are optional because a party may be unable to publish raw material. Omitting them removes the reader's ability to re-derive the figures under a different onset definition, which identifies as the dominant uncertainty term, so a report omitting them is weaker evidence than one that includes them.

Example The following is a report with the per-measurement and capture arrays elided.

• [[EDITOR'S NOTE: this structure is deliberately close to what the reference implementation already emits, which keeps at least one producer honest, and it is the part of the document most likely to change on review. Two questions in particular. Whether the format should be registered as a media type. And whether reference_point should be an enumeration with values for carrier-side and bridge-side recording, so that efforts observing elsewhere can produce conforming reports that declare the difference, rather than being unable to conform at all. The second would widen adoption considerably and is probably worth doing.]]

Use and Applications The metric supports:

• comparison of conversational voice systems from the position of a caller;
• characterisation of a single system across configurations or load levels;
• regression testing of a deployed system over time.

MRL measures an interval and carries no information about what the response contained. Response quality, recognition accuracy and the usefulness of the reply are separate quantities needing separate instruments, and a system that answers quickly and wrongly will score well here. Cases in which the metric does not apply, or applies only with care, are set out in .

Applicability and Limitations The metric as defined applies to systems that respond to a completed caller utterance with a discrete response. The following cases are outside its current scope or require care:

Filler and non-lexical audio:

Addressed by the continuity measurement in , which separates first audio from first uninterrupted audio without interpreting content. A caller comparing systems should read both figures, since MRL alone rewards a system for making a noise.

Unprompted audio:

Handled by , which requires detection to begin after any greeting. Without that constraint the measurement locates the greeting, and the figure it produces describes a different event.

Barge-in:

Where the caller interrupts the system, the interval defined here is not the quantity of interest and the method does not address it.

Incremental and streaming responses:

Where a system emits partial audio that it subsequently revises, first-audio onset does not characterise the response.

Path contribution:

As noted in the measurement domain, the figure is a property of the pairing of path and SUT. Where the path includes infrastructure the measuring party does not control, bounds its contribution rather than leaving it as a caveat.

Conditioning across turns:

See the open item under Measurement Timing.

Security Considerations The method described here generates traffic directed at a system under test and elicits responses from it. Measurement of a system operated by another party without that party's authorisation may constitute unauthorised use, may breach terms of service, and at sufficient volume is indistinguishable from a denial of service attempt. Parties performing measurement:

• SHOULD obtain authorisation from the operator of the system under test;
• SHOULD limit the measurement rate to a level agreed with that operator;
• SHOULD identify their measurement traffic where a mechanism to do so exists.

Captures produced by this method contain audio. Where stimulus material contains recorded human speech, that material is personal data in many jurisdictions and may carry biometric significance, and captures also hold the SUT's response audio, which can reflect the content of the stimulus. Implementers:

• SHOULD prefer synthetic or consented stimulus material;
• SHOULD apply access control to stored captures;
• SHOULD set a retention limit rather than keeping captures indefinitely.

Publication of captures as measurement evidence, which the method otherwise encourages, has to be weighed against these considerations. Reported figures may be commercially sensitive to the operator of the system under test. The configuration identifier described in is specified as an identifier rather than as the configuration itself so that reproducibility and confidentiality can coexist.

IANA Considerations This document has no IANA actions. Registration of this metric in the Performance Metrics Registry established by would be a reasonable later step, and shows what such an entry would contain. It is deliberately not requested here. The registry has to date been populated with IP-layer path metrics , so eligibility for a metric of this kind is a question for the working group rather than an entitlement to assert in an individual submission, and a request made before the venue question of is settled would be moot if this work moves elsewhere.

Future Work Conveying MRL between endpoints in band, by means of an RTCP Extended Report block in the manner of , would allow a system to report the metric about itself rather than requiring an external caller to measure it. That is a separate document and a different working group. Time to greeting, the interval from session establishment to the onset of the unprompted audio described in , is a second quantity the method already observes without defining. It is at least as visible to a caller as MRL, and it appears to be published by nobody. Defining it here would widen this document beyond a single metric, so it is noted rather than specified, and it would sit naturally in whatever document this one becomes part of.

References Normative References This memorandum describes RTP, the real-time transport protocol. RTP provides end-to-end network transport functions suitable for applications transmitting real-time data, such as audio, video or simulation data, over multicast or unicast network services. RTP does not address resource reservation and does not guarantee quality-of- service for real-time services. The data transport is augmented by a control protocol (RTCP) to allow monitoring of the data delivery in a manner scalable to large multicast networks, and to provide minimal control and identification functionality. RTP and RTCP are designed to be independent of the underlying transport and network layers. The protocol supports the use of RTP-level translators and mixers. Most of the text in this memorandum is identical to RFC 1889 which it obsoletes. There are no changes in the packet formats on the wire, only changes to the rules and algorithms governing how the protocol is used. The biggest change is an enhancement to the scalable timer algorithm for calculating when to send RTCP packets in order to minimize transmission in excess of the intended rate when many participants join a session simultaneously. [STANDARDS-TRACK] This document describes a profile called "RTP/AVP" for the use of the real-time transport protocol (RTP), version 2, and the associated control protocol, RTCP, within audio and video multiparticipant conferences with minimal control. It provides interpretations of generic fields within the RTP specification suitable for audio and video conferences. In particular, this document defines a set of default mappings from payload type numbers to encodings. This document also describes how audio and video data may be carried within RTP. It defines a set of standard encodings and their names when used within RTP. The descriptions provide pointers to reference implementations and the detailed standards. This document is meant as an aid for implementors of audio, video and other real-time multimedia applications. This memorandum obsoletes RFC 1890. It is mostly backwards-compatible except for functions removed because two interoperable implementations were not found. The additions to RFC 1890 codify existing practice in the use of payload formats under this profile and include new payload formats defined since RFC 1890 was published. [STANDARDS-TRACK] JavaScript Object Notation (JSON) is a lightweight, text-based, language-independent data interchange format. It was derived from the ECMAScript Programming Language Standard. JSON defines a small set of formatting rules for the portable representation of structured data. This document removes inconsistencies with other specifications of JSON, repairs specification errors, and offers experience-based interoperability guidance. In many standards track documents several words are used to signify the requirements in the specification. These words are often capitalized. This document defines these words as they should be interpreted in IETF documents. This document specifies an Internet Best Current Practices for the Internet Community, and requests discussion and suggestions for improvements. RFC 2119 specifies common key words that may be used in protocol specifications. This document aims to reduce the ambiguity by clarifying that only UPPERCASE usage of the key words have the defined special meanings. Informative References The purpose of this memo is to define a general framework for particular metrics to be developed by the IETF's IP Performance Metrics effort. This memo provides information for the Internet community. It does not specify an Internet standard of any kind. This document defines a set of metrics and their usage to evaluate the performance of end-to-end Session Initiation Protocol (SIP) for telephony services in both production and testing environments. The purpose of this document is to combine a standard set of common metrics, allowing interoperable performance measurements, easing the comparison of industry implementations. [STANDARDS-TRACK] This document describes a framework and a process for developing Performance Metrics of protocols and applications transported over IETF-specified protocols. These metrics can be used to characterize traffic on live networks and services. This memo documents an Internet Best Current Practice. This memo defines a metric for one-way delay of packets across Internet paths. It builds on notions introduced and discussed in the IP Performance Metrics (IPPM) Framework document, RFC 2330; the reader is assumed to be familiar with that document. This memo makes RFC 2679 obsolete. This memo provides clear definitions for Active and Passive performance assessment. The construction of Metrics and Methods can be described as either "Active" or "Passive". Some methods may use a subset of both Active and Passive attributes, and we refer to these as "Hybrid Methods". This memo also describes multiple dimensions to help evaluate new methods as they emerge. This document defines the format for the IANA Registry of Performance Metrics. This document also gives a set of guidelines for Registered Performance Metric requesters and reviewers. This memo defines the set of initial entries for the IANA Registry of Performance Metrics. The set includes UDP Round-Trip Latency and Loss, Packet Delay Variation, DNS Response Latency and Loss, UDP Poisson One-Way Delay and Loss, UDP Periodic One-Way Delay and Loss, ICMP Round-Trip Latency and Loss, and TCP Round-Trip Delay and Loss. This document defines the Extended Report (XR) packet type for the RTP Control Protocol (RTCP), and defines how the use of XR packets can be signaled by an application if it employs the Session Description Protocol (SDP). XR packets are composed of report blocks, and seven block types are defined here. The purpose of the extended reporting format is to convey information that supplements the six statistics that are contained in the report blocks used by RTCP's Sender Report (SR) and Receiver Report (RR) packets. Some applications, such as multicast inference of network characteristics (MINC) or voice over IP (VoIP) monitoring, require other and more detailed statistics. In addition to the block types defined here, additional block types may be defined in the future by adhering to the framework that this document provides. ITU-T ITU-T OpenBenchmarks Labs

Draft Performance Metrics Registry Entry

• [[EDITOR'S NOTE: filled in against the column structure defined in Section 7 of , using the blank template in Section 11 of that document, as far as the current text supports. Incomplete deliberately; the gaps are the same gaps flagged in the body. Note that Section 7.1.2 imposes a structured naming convention on registered metrics, which the Name entry below has yet to satisfy.]]

Identifier:

TBD by IANA

Name:

TBD, following the naming convention of Section 7.1.2 of

URI:

TBD

Description:

The interval between transmission of the final speech sample of a caller's utterance and arrival of the first sample of a conversational voice system's response, observed at the calling endpoint's RTP reference point.

Change Controller:

IETF

Version:

1

Reference Definition:

This document, through .

Fixed Parameters:

Onset variant thresholds as tabulated in ; de-jitter anchor rule as specified in .

Reference Method:

This document, Method of Measurement.

Packet Stream Generation:

Prerecorded stimulus transmitted at the nominal frame rate of the codec in use.

Traffic Filter:

The RTP stream of the session under measurement.

Sampling Distribution:

TBD; see the open item under Measurement Timing.

Runtime Parameters:

Codec, frame period, playout target depth, stimulus identifier and hash, SUT configuration identifier.

Roles:

Calling endpoint; system under test.

Output Type:

Signed scalar, reported in two variants, with an uncertainty term.

Metric Units:

Milliseconds.

Calibration:

As specified in . An implementation reports its own calibrated accuracy and the conditions of calibration.

Reference Implementation An open-source implementation of this method exists and is archived at . It implements the metric definition, the onset variants, the quality control conditions and the calibration procedure described here, and it publishes per-host calibration results. This appendix is informative. Conformance is to this document, and any disagreement between this document and that implementation is a defect in the implementation.

Open Questions for Review This appendix is retained deliberately rather than removed before submission. An individual submission exists to attract review, and the author's own list of doubts is a more efficient way to obtain useful review than letting each reader rediscover them.

1. Filler and non-lexical audio is now specified in by a structural discriminator rather than by content recognition, and implemented. What remains open is whether 150 ms and 2000 ms are the right constants. They were chosen to sit well above the pauses inside ordinary speech, measured at around 50 ms on the reference implementation's material, and no corpus of real filler behaviour informed them yet. Evidence welcome.
2. The stimulus annotation rule is now a conformance test against a published reference signal rather than a mandated algorithm, and the signal exists. What remains open is where the signal should live: carried in this document, registered, or referenced in the archived implementation as it is here.
3. The interchange record in , specifically whether it warrants a registered media type. The reference_point enumeration of answers the other half of what this question used to ask; what remains open there is whether the list is the right list, since it was drawn from observed practice rather than from a survey.
4. Whether Informational is the right category, or whether the conformance language argues for Standards Track.
5. Venue. Whether IPPM's charter admits a metric of this kind, and if not whether DISPATCH or the Independent Submission stream is the better route. See the editor's note in the Introduction.
6. Whether registry registration is appropriate for an application-layer metric of this kind.
7. Whether draws the line in the right place. It requires the turn index to be recorded and forbids silent pooling, and it deliberately does not prescribe how many turns to measure. An argument that a fixed number belongs in the specification, for comparability, is one the author would want to hear.