Live Agent Interaction over MoQ

Live Agent Interaction over MoQ Alibaba Inc.

miaoji.lym@alibaba-inc.com

Alibaba Cloud

max.ldp@alibaba-inc.com

Web and Internet Transport Media Over QUIC Internet-Draft This document defines a protocol for real-time interactive communication between users and AI agents over Media over QUIC Transport (MOQT). It specifies how streaming inference outputs (ASR transcripts, LLM tokens, TTS audio) map to the MOQT object model, defines a turn-taking control protocol with barge-in support for voice interactions, and establishes track structure conventions for live agent sessions. The protocol operates as an application-layer profile on top of MOQT without modifying transport semantics.

Introduction Large Language Models (LLMs) and multimodal AI systems have enabled a new class of interactive applications where users communicate with AI agents in real-time through voice and text. These "live agent" interactions share characteristics with both traditional media streaming and conversational protocols, but fit neatly into neither category.

Motivation and Use Cases The following application scenarios motivate the design of a dedicated protocol profile for live agent interaction over MOQT:

Voice AI Assistants: A user speaks naturally to an AI agent and receives spoken responses in real-time. The agent performs streaming ASR on user audio, generates a response via LLM, and synthesizes speech (TTS) delivered with sub-second latency. The user may interrupt the agent mid-response (barge-in), requiring immediate cessation of agent output. This demands continuous bidirectional audio streaming, low time-to-first-audio latency, graceful interruption handling, and the ability to deliver partial text results ahead of audio for perceived responsiveness.
Real-Time Customer Service Agents: In live commerce or customer support deployments, an AI agent handles simultaneous voice or text interactions with customers, accessing external tools (inventory lookup, order status, payment processing) and relaying structured results alongside natural language responses. This demands reliable delivery of tool results alongside best-effort audio delivery, relay-based fan-out for scaling to thousands of concurrent sessions, and per-session isolation with independent priority and timeout policies.
Multimodal Scene-Aware Agents: A user points their device camera at a real-world scene (e.g., a landmark, exhibit, or street sign) while speaking to an AI agent that acts as a digital tour guide. The agent subscribes to the user's audio and video input tracks, performs visual understanding and speech recognition jointly, and publishes spoken narration, text annotations, and contextual information about the scene. This demands concurrent processing of multiple input modalities (audio + video), low-latency multimodal fusion at the agent backend, multiple independent output tracks with heterogeneous delivery requirements (reliable text vs. best-effort audio), and partial reliability where stale video frames or audio segments may be dropped without retransmission.

Why Existing Approaches Are Insufficient Live agent interactions impose strict latency budgets: users expect sub-second time-to-first-token and time-to-first-audio for the interaction to feel comparable to natural conversational turn-taking. HTTP-based streaming approaches (SSE, WebSocket) operate over TCP, where head-of-line blocking, connection-level flow control, and lack of stream multiplexing make it difficult to meet these latency targets — particularly when multiple output modalities (text, audio, tool results) must be delivered concurrently with independent priority and reliability requirements. Furthermore, these approaches cannot express per-object delivery timeouts or relay-assisted fan-out at the transport level, forcing application-layer workarounds that add complexity and latency. Purpose-built AI inference APIs operate in request-response or unidirectional streaming modes without support for concurrent input processing, turn management, or barge-in. Media over QUIC Transport addresses these limitations at the transport layer: QUIC's stream multiplexing eliminates head-of-line blocking between modalities, MOQT's priority system ensures latency-critical signals (barge-in, user audio) are scheduled first, and delivery timeouts allow stale data to be discarded without blocking fresh output. The relay infrastructure provides scalability without per-connection state at the agent backend. However, MOQT lacks application-layer conventions for mapping AI inference semantics onto these primitives. This document fills that gap.

Distinguishing Properties A live agent interaction has the following distinguishing properties:

Asymmetric streaming: User input is continuous (audio stream), while agent output is incremental and multi-modal (text tokens, synthesized audio, tool results).
Turn-based with interruption: Unlike media broadcast, the interaction follows a dialogue structure where either party can take or yield the floor.
Latency-critical incremental delivery: Users perceive agent responsiveness through time-to-first-token and time-to-first-audio, requiring sub-second delivery of partial results.
Heterogeneous reliability requirements: Within a single turn, interim ASR transcripts are ephemeral, final transcripts are authoritative, TTS audio is time-bounded, and tool results must be delivered reliably.

Media over QUIC Transport provides a publish/subscribe protocol with features well-suited to these requirements: prioritized delivery, partial reliability through delivery timeouts, group-based object organization, and relay infrastructure for scalability. However, MOQT defines no application-layer semantics for mapping inference streams to its object model, nor for managing conversational turn-taking. This document specifies:

A mapping of streaming inference outputs to the MOQT object data model ().
A turn control protocol for managing dialogue state and handling barge-in interruptions ().
Track structure conventions and naming for live agent sessions ().
Delivery policies appropriate for each stream type ().

Architecture Overview

Protocol Scope and Layering This document defines an application-layer profile that operates on top of MOQT without modifying its transport semantics. The relationship to the MoQ protocol suite is illustrated below:

Protocol Layering The following table summarizes how live agent domain concepts map to MOQT primitives: Domain Concept to MOQT Mapping

Domain Concept	MOQT Primitive	Semantics
Conversation Turn	Group	Atomic dialogue unit; GROUP_ORDER descending prioritizes latest turn
Inference Step	Subgroup	A sentence, audio segment, or tool call within a turn
Token Batch / Audio Frame	Object	Minimum delivery unit; subject to OBJECT_DELIVERY_TIMEOUT
Stream Modality	Track	Independent subscribe, priority, and reliability per modality
Barge-in Signal	Datagram	Highest priority (0x00); bypasses head-of-line blocking
Turn Control	Control Track	Reliable delivery of state-machine transitions

This document:

USES the MOQT object model (Track, Group, Subgroup, Object) to represent conversational structure.
USES LOC as the container format for audio payloads.
USES MOQT native mechanisms (SUBSCRIBE, priority, delivery timeouts, GROUP_ORDER) for QoS enforcement.
MAY USE Secure Objects for end-to-end encryption of agent output through untrusted relays.
DOES NOT define new transport-layer framing or modify MOQT wire format.

Design Principles The protocol is guided by the following architectural principles:

Native MOQT Integration:: Map application semantics directly to the MOQT object hierarchy rather than introducing intermediate framing layers. This ensures MOQT relays can perform correct scheduling, timeout-based discard, and caching without understanding application-layer payload formats.
Relay Transparency:: All protocol operations MUST work through unmodified MOQT relays. The relay sees standard Tracks, Groups, Subgroups, and Objects with associated priorities and timeouts. No relay-side payload inspection is required.
Asymmetric by Design:: The protocol explicitly models the user-to-agent asymmetry: user input is continuous and latency-critical for the agent; agent output is incremental, multi-modal, and interruptible. This asymmetry is reflected in priority assignment, timeout configuration, and track structure.
Latency Budget Awareness:: Every protocol mechanism is evaluated against its contribution to end-to-end latency. Zero additional round-trips for session setup (reuse MOQT session). Datagram delivery for time-critical signals. Batching strategies that bound flush latency.
Partial Reliability as a Feature:: Not all data within a turn has equal value. The protocol assigns per-track and per-subgroup delivery timeouts that allow the transport to discard stale data (old audio frames, obsolete interim transcript hypotheses) while guaranteeing delivery of authoritative results (final text, tool outputs).
Modality Agnostic:: The protocol does not mandate specific codecs, model architectures, or inference pipelines. It defines structural conventions (Group=Turn, Subgroup=inference step) that apply regardless of whether the agent produces text, audio, video, or structured data.

Protocol Components This document comprises four logical components:

Inference Stream Delivery (): Defines how streaming outputs from ASR, LLM, and TTS pipelines map to the MOQT object data model. Covers text token batching, audio segmentation, tool result framing, and cross-track synchronization.
Turn Control Protocol (): Defines the conversational state machine, control signal format, barge-in handling, VAD integration, and priority assignment for managing dialogue flow.
Track Structure and Naming (): Defines namespace conventions, standard track names, and catalog integration for live agent sessions.
Delivery Policies (): Specifies per-track timeout configurations, transport selection guidelines (Datagram vs Stream), and relay caching behavior.

Conventions and Definitions The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here. The following terms are used in this document:

Live Agent Session:: A stateful interaction between a user endpoint and an AI agent backend, conducted over one or more MOQT sessions.
Turn:: A contiguous period during which one party (user or agent) holds the conversational floor. Mapped to a MOQT Group.
Inference Stream:: A sequence of incremental outputs from an AI model (e.g., LLM tokens, ASR transcripts, TTS audio chunks).
Barge-in:: An event where the user begins speaking while the agent is still producing output, causing the agent to yield the floor.
Partial Result:: An intermediate inference output for which the final result has not yet been signaled. A Partial Result can be an append-only delta, a snapshot, or a revision only when the corresponding track semantics explicitly define that behavior.
Final Result:: A definitive inference output that will not be further modified.

Deployment Examples This protocol is compatible with multiple deployment topologies. Two examples are illustrated below.

Example 1: With MoQ Relay

Deployment with MoQ Relay | | | | |--- Audio Track ----->|-------- fwd --------->| |--- Video Track ----->|-------- fwd --------->| | | | |<-- Audio Track ------|<------ publish -------| |<-- Text Track -------|<------ publish -------| |<-- Tool Results -----|<------ publish -------| | | | |--- Control Signals ->|-------- fwd --------->| |<-- Control Signals --|<------ publish -------| ]]> The relay forwards user input to the agent and fans out agent output to subscribers. This topology is suited for scenarios requiring:

Multiple subscribers to a single agent session (monitoring, recording, accessibility overlays).
Geographic distribution where relays are placed close to users.
Caching of agent output for late-joining clients.

Example 2: Without Relay

Deployment without Relay | | | |--- Audio Track ----------------------------->| |--- Video Track ----------------------------->| | | |<-- Audio Track ------------------------------| |<-- Text Track -------------------------------| |<-- Tool Results -----------------------------| | | |--- Control Signals ------------------------->| |<-- Control Signals --------------------------| ]]> The client connects directly to the agent backend. This topology is suited for scenarios requiring:

Minimal latency (no intermediate hop).
Simpler deployment without relay infrastructure.
Single-subscriber sessions (1:1 user-to-agent interactions).

Protocol Compatibility This specification operates correctly under both topologies. The application-layer semantics (track structure, turn control, object model mapping) are identical regardless of whether a MoQ relay is present:

With relay: the relay handles subscription management, priority-based scheduling, and delivery timeout enforcement transparently.
Without relay: the agent backend itself implements MOQT session handling. Priority and timeout semantics still apply to the QUIC streams between client and agent.

Deployments MAY combine both topologies, for example using direct connections for latency-sensitive single-user sessions while routing multi-subscriber sessions through relays.

Object Model Mapping for Inference Streams This section defines how streaming inference outputs map to the MOQT object data model defined in .

Mapping Principles The MOQT object hierarchy consists of Track > Group > Subgroup > Object. This document assigns conversational semantics to each level: Object Model Semantic Mapping

MOQT Level	Live Agent Semantic	Rationale
Track	Stream type (audio, text, control)	Independent subscription unit
Group	Conversation turn	Atomic unit of dialogue; enables turn-level operations
Subgroup	Inference step within a turn	Logical segment: a sentence, an audio segment, a tool call
Object	Atomic delivery unit	Smallest independently decodable/renderable item

Unless a track definition explicitly states otherwise, all media, text, tool, and control Objects that belong to the same application turn MUST use the same turn_id. The value of turn_id is the MOQT Group ID on tracks that publish Objects for that turn. Subgroup identifiers are scoped to a track; matching Subgroup identifiers across tracks indicate the same logical segment only where this document defines such alignment. This mapping enables:

Subscribing to a specific turn onwards (Group-based filtering).
Dropping an entire stale turn when interrupted (Group-level discard).
Prioritizing recent turns over old ones (GROUP_ORDER = descending).
Independent reliability per inference step (Subgroup-level timeouts).

Agent Text Output Track The agent text output track carries streaming LLM token output. This track uses append-only delta semantics: each Object contributes new text that follows the text carried by earlier Objects in the same Subgroup.

Object Structure Each Object in the text output track carries a token batch: one or more sequential tokens generated within a single flush interval.

Text Output Object Format

Batching Strategy Implementations SHOULD batch tokens to amortize per-object overhead. The following strategies are RECOMMENDED:

Time-based: Flush every 50ms, collecting all tokens generated in that interval into a single Object.
Size-based: Flush when accumulated token text reaches 128 bytes.
Semantic-based: Flush at sentence boundaries or punctuation marks.

An implementation MUST flush immediately when:

The inference step completes (flags = 0x02, final).
A barge-in interrupt is received (flags = 0x04, cancelled).
The subgroup ends (last object in subgroup).

Partial and Final Semantics Within a Subgroup (inference step), Objects are delivered incrementally:

Objects with flags=0x01 (partial) carry append-only token deltas. A subscriber MAY append them to the displayed text immediately for real-time display. Earlier delta Objects in this text track are not superseded by later delta Objects.
An Object with flags=0x02 (final) indicates the inference step is complete. The subscriber SHOULD treat the concatenation, in seq order, of all non-cancelled delta Objects in the Subgroup as the definitive output for that Subgroup.
An Object with flags=0x04 (cancelled) indicates the inference step was interrupted (e.g., by barge-in). The subscriber SHOULD discard or visually mark the incomplete output.

Tracks or future extensions that need ASR interim transcript snapshots or patch-style text revision semantics MUST define that replacement or revision behavior explicitly. Such streams MUST NOT rely on the append-only output/text semantics above.

Group Lifecycle A new Group is created when:

The agent begins responding to a new user turn.
The turn counter increments (see ).

The Group is closed (LARGEST_OBJECT property set) when:

The agent completes its full response for this turn.
The agent is interrupted by barge-in (final Object has cancelled flag).

Agent Audio Output Track The agent audio output track carries TTS-synthesized audio. The encoded audio segment is carried as a complete LOC payload inside the Live Agent audio envelope defined below. This document does not extend the LOC header or append Live Agent metadata to a raw LOC payload.

Object Structure Each Object carries one audio segment (typically 20-60ms of audio). The envelope provides explicit payload boundaries and optional alignment metadata. A profile-aware receiver parses the envelope, then passes the loc_payload bytes unchanged to its LOC decoder.

Audio Output Object Format The optional alignment fields are present only when the alignment_present flag is set. They belong to this Live Agent envelope, not to LOC. A receiver that only understands LOC will not be able to decode this envelope directly; an endpoint or gateway can recover the standard LOC payload by extracting loc_payload.

Subgroup Semantics for Audio Each Subgroup in the audio track corresponds to one utterance or sentence boundary in the agent's response. This enables:

Dropping a complete sentence if delivery is too late (SUBGROUP_DELIVERY_TIMEOUT).
Rendering audio sentence-by-sentence with natural pauses.
Aligning with text Subgroups at sentence granularity.

Cross-Track Synchronization The agent text track and agent audio track use the same Group ID for the same conversational turn. Within a turn:

Text Subgroup N corresponds to Audio Subgroup N (same sentence).
The align_seq field in audio Objects references the text Object sequence number being spoken at that audio moment.

This enables a subscriber receiving both tracks to:

Display text as it arrives (lower latency than audio).
Highlight the currently-spoken text segment during audio playback.
Fall back to text-only if audio delivery times out.

User Audio Input Track The user publishes a continuous audio input track.

Object Structure Each Object carries a fixed-duration audio frame (typically 20ms) using the LOC container format.

Group Semantics Groups in the user audio track are segmented by voice activity:

A new Group begins when the user starts speaking (VAD trigger).
The Group ends when the user stops speaking (silence detection).

This enables the agent backend to:

Subscribe starting from the latest Group (skip silence gaps).
Process each utterance as a unit.
Implement endpoint detection without additional signaling.

Tool Output Track The agent MAY publish a tool output track for structured results from tool/function calls.

Object Structure

Tool Output Object Format The metadata field is an extension point for action-level metadata. If metadata_length is zero, no metadata is present. When the authorization_ref_present bit is set, metadata MUST include an authorization reference that identifies the authorization artifact for the action identified by tool_id and call_id.

Tool Action Authorization Tool results can be purely informational, or they can correspond to consequential actions such as payments, permission changes, or bookings. When a tool action requires per-action authorization, the agent MUST keep the authorization artifact transport-agnostic and use this profile only to define its MOQT carriage and correlation. If authorization is required before completing a tool action, the agent MUST publish a reliable AUTH_REQUIRED control signal on control/agent and SHOULD publish a Tool Output Object with the authorization_required metadata flag set. The corresponding control payload identifies the pending action:

Authorization Control Payload The user endpoint or authorizing component can answer with an AUTH_RECEIPT control signal on control/user using the same payload format. The artifact field is opaque to this profile; it can carry an offline-verifiable receipt, a challenge reference, or another application-defined authorization artifact. Subscribers correlate the authorization artifact with the tool action using tool_id and call_id; no separate authorization track is required by this profile.

Delivery Requirements Tool outputs MUST be delivered reliably (no delivery timeout). Tool invocation and result Objects are complete status records. The subscriber MUST NOT discard tool results due to lateness. AUTH_REQUIRED and AUTH_RECEIPT control signals MUST be delivered reliably.

Turn Control Protocol This section defines the control protocol for managing conversational turns between the user and agent.

Turn State Machine A live agent session maintains the following turn states:

IDLE → USER_SPEAKING: User audio VAD detects speech onset.
USER_SPEAKING → AGENT_PROCESSING: User speech ends (silence timeout or explicit end-of-turn signal).
AGENT_PROCESSING → AGENT_SPEAKING: Agent produces first output Object in any output track.
AGENT_SPEAKING → IDLE: Agent completes response (closes Group in all output tracks).
AGENT_SPEAKING → USER_SPEAKING: Barge-in event (user starts speaking while agent is outputting).
AGENT_PROCESSING → USER_SPEAKING: Barge-in event before the agent has produced its first output Object.

The AGENT_SPEAKING state means that the agent is publishing response output in any output modality; the state name does not require audio to be present. TURN_STARTED signals that the agent has accepted the user turn and is entering the agent side of the turn. It MUST be sent no later than the first output Object for the turn and SHOULD be sent when the agent enters AGENT_PROCESSING if processing is expected to be visible to the user. THINKING is an optional progress signal while the session is in AGENT_PROCESSING. If the user starts speaking while the session is in AGENT_PROCESSING or AGENT_SPEAKING, the event is a barge-in for the active agent turn. If the relevant turn has already reached TURN_COMPLETE and all output Groups for that turn are closed, the speech start begins a new user turn instead of interrupting the completed turn. Implementations that detect a false speech start SHOULD either avoid sending SPEECH_START until the signal is stable or follow it with SPEECH_END carrying no media Objects for that turn.

Control Track Turn control signals are exchanged on a dedicated bidirectional control track pair (one per direction). Datagram delivery of BARGE_IN uses the same Control Object Payload format as the control track; it is a fast path for the same logical event, not a separate signal format. Control Objects use the following format:

Control Signal Format

Barge-in Handling Barge-in is the critical interaction where a user interrupts the agent's ongoing output. The protocol defines the following sequence:

Barge-in Sequence | | (via Datagram, highest priority) | | | | [agent stops TTS, notes position] | | | |<- Control: INTERRUPT_ACK (turn_id=N) ----- | | payload: {interrupted_group: N, | | interrupted_subgroup: M, | | interrupted_object: K} | | | | [agent closes Group N with cancelled flag]| | | |<- Text Object: flags=cancelled ----------- | |<- Audio: subgroup FIN -------------------- | | | | [agent begins processing new user input] | | | ]]>

Barge-in Signal Delivery The BARGE_IN signal has the following delivery requirements:

MUST be sent via MOQT Datagram for minimum latency. The Datagram payload MUST be a complete Control Object Payload with signal=BARGE_IN.
MUST be assigned the highest publisher priority (0x00).
SHOULD be sent immediately upon local VAD detection, without waiting for speech_end.
SHOULD also be published on the user-to-agent control track as a reliable mirror of the same event unless an INTERRUPT_ACK for that event has already been received.
The agent MUST process BARGE_IN within one processing cycle (target: < 50ms from receipt to output cessation).

The turn_id in a BARGE_IN Control Object identifies the interrupted agent turn. The BARGE_IN payload MUST contain an event_id that is unique within the MOQT session and a new_turn_id for the user speech that caused the interruption:

BARGE_IN Payload An agent MUST deduplicate BARGE_IN events by (sender, event_id). If a Datagram copy and a reliable control-track copy of the same event are both received, only the first copy that is processed changes the state machine. Later copies are acknowledgements of delivery and MUST NOT trigger a second interrupt. If BARGE_IN races with TURN_COMPLETE for the same turn_id, BARGE_IN takes precedence while the agent turn is still active. Once the agent has committed TURN_COMPLETE and closed all output Groups for that turn, a later BARGE_IN for that turn_id is stale and MUST be ignored as an interrupt; the associated speech can still start new_turn_id.

Agent Interrupt Behavior Upon receiving BARGE_IN, the agent MUST:

Cease generating new output Objects for the current turn.
Close the current output Group with a cancelled Object (flags=0x04 in text track, stream FIN in audio track).
Send INTERRUPT_ACK with the position where output stopped.
Transition to processing the new user input.

The agent SHOULD NOT:

Abruptly truncate mid-audio-frame (finish current audio Object).
Discard context from the interrupted response (the agent has it in its context window for the next turn).

Client Interrupt Behavior Upon sending BARGE_IN, the client SHOULD:

Immediately stop audio playback of the agent's output.
Visually indicate the response was interrupted (e.g., fade text).
Begin capturing and publishing user audio for the new turn.

VAD Integration Speech activity detection events drive the turn state machine. This document does not mandate a specific detection algorithm (traditional energy-based VAD, neural VAD, or other approaches) but defines the signaling semantics:

SPEECH_START: Published when the implementation determines that the user has begun speaking.
SPEECH_END: Published when the implementation determines that the user has finished speaking.
BARGE_IN: Published when SPEECH_START occurs during AGENT_PROCESSING or AGENT_SPEAKING state. This is a composite signal (implies SPEECH_START + interrupt request).

VAD signals are sent on the user→agent control track. Implementations MAY perform VAD on the client, on the relay, or on the agent backend. When VAD is performed on the client, it SHOULD be sent as Datagram for lowest latency.

Priority Assignment The following priority assignments are RECOMMENDED for live agent sessions (lower numeric value = higher priority): Recommended Priority Assignment

Track/Signal	Priority	Rationale
Control signals (BARGE_IN)	0x00	Must preempt all other traffic
Control signals (other)	0x01	Turn management is time-critical
User audio input	0x02	Agent cannot process without input
Agent audio output	0x03	Primary user-perceived output
Agent text output	0x04	Secondary output (lower bandwidth)
Tool results	0x05	Non-time-critical structured data

Within agent output tracks, GROUP_ORDER SHOULD be set to descending (deliver newest group first) so that relay congestion drops stale turns rather than current ones.

Track Structure and Naming

Namespace Convention A live agent session uses the following namespace structure: Where:

{authority} is the domain of the agent service.
{session-id} is a unique session identifier (RECOMMENDED: UUIDv7).

Track Names The following track names are defined within a session namespace: Standard Track Names

Track Name	Direction	Content
input/audio	User → Agent	User microphone audio (LOC)
input/text	User → Agent	User text messages
output/audio	Agent → User	TTS synthesized audio (LOC)
output/text	Agent → User	Streaming LLM text tokens
output/tool	Agent → User	Tool invocations and results
control/user	User → Agent	User control signals
control/agent	Agent → User	Agent control signals

Additional tracks MAY be defined for:

input/video: User camera input.
output/video: Agent avatar or visual output.
meta/catalog: Session catalog in MSF format .

Catalog Integration A live agent session SHOULD publish a catalog track conforming to the MOQT Streaming Format . The catalog declares:

Available tracks and their codec parameters.
Agent capabilities (supported input modalities, languages).
Session metadata (model identifier, context window size).

The catalog enables late-joining subscribers and relay-assisted discovery of session characteristics.

Delivery Policies

Datagram vs Stream Selection Transport Selection Guidelines

Track	Default Transport	Fallback	Condition
control/* (BARGE_IN)	Datagram	Stream mirror	Fast path plus reliable recovery
control/* (other)	Stream	—	Reliable delivery needed
input/audio	Stream	Datagram	If partial reliability desired
output/audio	Stream	Datagram	For loss-tolerant low-latency
output/text	Stream	—	Must be reliable
output/tool	Stream	—	Must be reliable

Relay Considerations

Relay Transparency This protocol is designed to operate through standard MOQT relays without relay modification. Relays treat live agent traffic as normal MOQT objects with the following beneficial behaviors:

Priority-based scheduling: Relays respect publisher priority, ensuring control signals and user audio are forwarded first under congestion.
Timeout-based expiry: Relays discard Objects that exceed their delivery timeout, preventing stale audio from consuming bandwidth.
Group-order delivery: With descending group order, relays under congestion naturally shed older turns.

Caching Behavior Relays MAY cache agent output Objects for the duration specified by the MAX_CACHE_DURATION track property. This enables:

Late-joining clients to receive the current turn's output.
Reconnecting clients to resume from where they left off.

Relays SHOULD NOT cache:

Control track Objects (they are ephemeral state transitions).
User audio input (privacy-sensitive, single-consumer).

Multi-Subscriber Scenarios A single agent session MAY have multiple subscribers to output tracks (e.g., accessibility tools, monitoring, recording). The relay naturally fans out agent output to all subscribers without additional agent-side overhead.

Security Considerations

Authentication and Authorization Live agent sessions MUST authenticate both the user and agent endpoints. The MOQT AUTHORIZATION_TOKEN parameter () SHOULD be used for per-track authorization. User audio input tracks contain sensitive biometric data and MUST be restricted to the intended agent subscriber. Relays MUST enforce subscription authorization for input tracks.

End-to-End Encryption For deployments where relay operators are not fully trusted, agent output tracks MAY use end-to-end encryption as defined in . Control tracks SHOULD NOT be E2E encrypted as relay-level inspection may be needed for priority enforcement.

Privacy Considerations

User audio MUST NOT be cached by relays beyond the immediate delivery requirement.
Session IDs MUST be cryptographically random (UUIDv7 with random component) to prevent session correlation attacks.
Control signals (VAD events, barge-in) leak interaction timing metadata. Implementations MAY add padding to control track Objects to mitigate traffic analysis.

Denial of Service

Barge-in signals are high-priority and processed immediately. Implementations MUST rate-limit barge-in signals per session (RECOMMENDED: maximum 10 per second) to prevent priority inversion attacks.
Relays SHOULD enforce per-session bandwidth quotas to prevent a single agent session from starving other traffic.

IANA Considerations

MOQT Track Property Registrations This document registers the following track properties in the "MOQT Track Properties" registry: Track Property Registrations

Property Name	Property ID	Type	Description
AGENT_SESSION_ROLE	TBD	varint	0=user, 1=agent
TURN_GROUP_ORDER	TBD	varint	Confirms Group=Turn mapping

Control Signal Type Registry IANA is requested to create a "Live Agent Control Signal Types" registry under the "Media over QUIC (MoQ)" group. The registration procedure is Specification Required. Initial registrations: Control Signal Type Registry

Value	Signal Name	Reference
0x01	SPEECH_START
0x02	SPEECH_END
0x03	BARGE_IN
0x04	TURN_STARTED
0x05	TURN_COMPLETE
0x06	INTERRUPT_ACK
0x07	THINKING
0x08	AUTH_REQUIRED
0x09	AUTH_RECEIPT

Values 0x0A-0xFF are available for assignment.

Object Payload Flags Registry IANA is requested to create a "Live Agent Object Flags" registry. Initial registrations: Object Flags Registry

Bit	Flag Name	Description	Reference
0	PARTIAL	Object is intermediate; final not yet signaled	This document
1	FINAL	Object is definitive	This document
2	CANCELLED	Object indicates interruption	This document

References Normative References Media over QUIC Transport Cisco Google Google Meta This document defines Media over QUIC Transport (MOQT), a publish/ subscribe protocol that runs over QUIC and WebTransport. MOQT leverages the features of these transports, such as streams, datagrams, priorities, and partial reliability. MOQT operates both point-to-point and through intermediate relays, enabling scalable low-latency delivery. Despite its name, MOQT is media agnostic and can be used for a wide range of use cases. Low Overhead Media Container Cisco Cisco Microsoft This specification describes a Low Overhead Media Container (LOC) format for encoded and encrypted audio and video media data to be used primarily for interactive Media over QUIC Transport (MOQT). It may be used in the MOQT Streaming Format (MSF) specification, which defines a catalog format for publishers to declare and describe their LOC tracks and for subscribers to consume them. Examples are also provided for building media applications using LOC and MOQT. QUIC: A UDP-Based Multiplexed and Secure Transport This document defines the core of the QUIC transport protocol. QUIC provides applications with flow-controlled streams for structured communication, low-latency connection establishment, and network path migration. QUIC includes security measures that ensure confidentiality, integrity, and availability in a range of deployment circumstances. Accompanying documents describe the integration of TLS for key negotiation, loss detection, and an exemplary congestion control algorithm. Key words for use in RFCs to Indicate Requirement Levels In many standards track documents several words are used to signify the requirements in the specification. These words are often capitalized. This document defines these words as they should be interpreted in IETF documents. This document specifies an Internet Best Current Practices for the Internet Community, and requests discussion and suggestions for improvements. Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words RFC 2119 specifies common key words that may be used in protocol specifications. This document aims to reduce the ambiguity by clarifying that only UPPERCASE usage of the key words have the defined special meanings. Informative References MOQT Streaming Format Akamai Cisco This document specifies the MOQT Streaming Format, designed to operate on Media Over QUIC Transport. End-to-End Secure Objects for Media over QUIC Transport Cisco Cisco Cisco This document specifies an end-to-end authenticated encryption scheme for application objects transmitted via Media over QUIC (MoQ) Transport. The scheme enables original publishers that share a symmetric key with end subscribers, to ensuring that MoQ relays are unable to decrypt object contents. Additionally, subscribers can verify the integrity and authenticity of received objects, confirming that the content has not been modified in transit. Additionally it allows MoQ parameters to be protected so the publisher can select if they are readable and/or modifiable by relays. Agent Protocol over MoQ Alibaba Cloud Cisco This document specifies a Agent-to-Agent communication framework enabling structured, low-latency, and semantically rich communication between autonomous agents over the Media over QUIC (MoQ) protocol. It leverages MoQ's efficient media transport capabilities while introducing a new application-layer framing mechanism to support control signaling, session management, and large data fragmentation. The design supports both intra-domain and inter-domain deployment, with an emphasis on interoperability, extensibility, and minimal overhead.

Interaction Examples

Basic Voice Conversation Turn

Basic Voice Turn Example -----> ASR processes t0 Control: SPEECH_START ----------->------> | t1 [User stops speaking] t1 Control: SPEECH_END ------------->------> | LLM generates response t2 <------<------- Control: TURN_STARTED t2 <------<------- PUBLISH output/text | Group=1, Subgroup=0 | Object 0: "The weather" | Object 1: " in Hangzhou" | Object 2: " is sunny," t3 <------<------- PUBLISH output/audio | Group=1, Subgroup=0 | [TTS: "The weather in | Hangzhou is sunny,"] | t4 <------<------- Object (text, final): | " 28°C today." t4 <------<------- Control: TURN_COMPLETE ]]>

Barge-in During Agent Response

Barge-in Example ------> received | t1 [stops TTS generation] t1 <------<------- Control: INTERRUPT_ACK | {interrupted: G=1,SG=2,O=5} t1 <------<------- Text Object(cancelled) t1 <------<------- Audio subgroup FIN | t2 PUBLISH input/audio Group=2 ---->------> ASR: "just tell me temp" t2 Control: SPEECH_START ---------->------> | t3 Control: SPEECH_END ------------>------> | LLM: context includes | interrupted response t4 <------<------- Control: TURN_STARTED t4 <------<------- Text Group=2: "It's 28°C." t4 <------<------- Audio Group=2: [TTS] t5 <------<------- Control: TURN_COMPLETE ]]>

Concurrent Text and Audio Delivery

Cross-Track Synchronization Example

Design Rationale

Why Not a Custom Frame Layer This document maps directly to the native MOQT object model rather than introducing a custom frame layer because:

MOQT Groups/Subgroups already provide the sequencing and boundaries needed for turns and inference steps.
MOQT delivery timeouts and priorities operate at the Object level, which is the right granularity for inference delivery.
Standard MOQT relays can handle live agent traffic without modification or frame parsing.
Reusing the object model means existing MOQT tooling (monitoring, debugging, relay management) works unchanged.

Why Group = Turn Alternatives considered:

Group = entire session: Loses the ability to discard stale turns and prevents Group-level priority ordering.
Group = single inference step: Too fine-grained; creates excessive Group metadata overhead and prevents turn-level operations.
Group = time window (e.g., 1 second): Arbitrary boundary that doesn't align with application semantics; complicates barge-in.

Group = Turn provides the natural boundary for: - What to discard when interrupted (the current turn). - What to prioritize (the latest turn). - What to cache for late-joiners (the most recent complete turn).

Why Separate Control Track Embedding control signals in-band with media or text Objects was considered but rejected because:

Control signals require different delivery characteristics. BARGE_IN uses a Datagram fast path with a reliable control-track mirror, while other turn-management signals use the reliable control track directly.
Relays can apply priority to entire tracks but not to individual Objects within a track.
Subscribers may want control-only subscription (e.g., turn status for UI state management without receiving media).

Acknowledgements The authors would like to thank the participants of the MoQ working group for their contributions to the underlying transport protocol that makes this work possible.