Network Working Group D. Liu Internet-Draft Y. He Intended status: Standards Track X. Yu Expires: 4 September 2023 X. Kai S. Li Alibaba Group March 2023 Protocol for interactive low-latency media transmission system draft-liu-protocol-interactive-media-transmission-00 Abstract This document introduces a protocol used for interactive low-latency media transmission network. Requirements Language The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 [1]. Status of This Memo This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet- Drafts is at https://datatracker.ietf.org/drafts/current/. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." This Internet-Draft will expire on 2 September 2023. Copyright Notice Copyright (c) 2023 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/ license-info) in effect on the date of publication of this document. Liu, et al. Expires 4 September 2023 [Page 1] Internet-Draft Protocol for interactive low-latency med March 2023 Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Revised BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Revised BSD License. Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 2. System Architecture . . . . . . . . . . . . . . . . . . . . . 3 3. Signaling procedure . . . . . . . . . . . . . . . . . . . . . 5 4. Signaling Specification . . . . . . . . . . . . . . . . . . . 7 4.1. Merging signaling message . . . . . . . . . . . . . . . . 8 4.2. Switching signaling message . . . . . . . . . . . . . . . 8 4.3. Grabbing signaling message . . . . . . . . . . . . . . . 9 4.4. Pulling signaling message . . . . . . . . . . . . . . . . 10 4.5. Pushing signaling message . . . . . . . . . . . . . . . . 11 5. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 11 6. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 11 7. Security Considerations . . . . . . . . . . . . . . . . . . . 11 8. Normative References . . . . . . . . . . . . . . . . . . . . 11 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 12 1. Introduction Emerging real-time interactive video/audio communication applications bring new challenges for existing protocols. This documents introduces the use cases, requirements and protocol for interactive low-latency multimedia transmission network over the Internet. Interactive real-time media communication is getting popular with the growth of short video, on-line education, on-line gaming and other similar applications. Some application providers build their own interactive real-time media communication network to support their applications yet facing high cost and technical issues. For example, interactive communication between users is unpredictable, which results in high cost when dedicated entity for interaction is used and the wastage of reserved resources for interaction. To avoid the aforementioned issues and challenges, some other application providers attempt to use third party's interactive real- time media communication network provided by giant cloud operator. However, there are several challenges of existing protocol to support the above mentioned scenarios. 1. Interactive Online broadcasting service is flexible and much more complicated compared with traditional media broadcasting service. For interactive Online broadcasting applications, audiences may Liu, et al. Expires 4 September 2023 [Page 2] Internet-Draft Protocol for interactive low-latency med March 2023 occasionally request to setup bidirectional real-time communication with the broadcaster and all the other audiences should be able to receive the merged interactive media traffic containing the broadcaster and connected audience. To meet this end, there is a need for standardized signaling protocol which can support media stream merging，switching and pulling to support those complicated scenarios. 2. Applications such as interactive online broadcasting, short video, on-line education, on-line gaming are very delay sensitive. Thus, the protocols for media stream merging，switching and pulling should be able to meet the latency requirement for those applications. 3. There are many different media transmission protocols (e.g. QUIC, WebRTC, etc) across different layers, which are widely used in the ecosystem, the protocols for media stream merging，switching and pulling should be able to compatible with different transmission protocols. This document specifies a protocol for media stream merging, pulling and switching that used for Interactive real-time media communication system. 2. System Architecture This section specifies the system architecture of the Interactive real-time media communication system. Liu, et al. Expires 4 September 2023 [Page 3] Internet-Draft Protocol for interactive low-latency med March 2023 Sever for media streaming control +-----------+ | | | | | | | | | | +-----+-----+ +-----+ +-----+ | | | | | | | | | | | | | | |<------+ | +------->| | | | | +-----------------v------------------+ | | | | | | | | | +-----+ +-----+ | | | | Audience Broadcaster | | +----+ +-----+ +--------------------->| | | | |Interactive real-time communication | | | | network +------------>| | +-----+ +--------------------->| | | | | | | | | | | | | | | +----+ +-----+ | |<------+ | | | Audience | | | | | +-----+ | | +-------------^-----+----------------+ | | | +-----+ | | | | | Audience | | +------->| | connected with | | | | the broadcaster +-+-----v---+ | | | | +-----+ | | Audience | | | | | | +-----------+ Server for media stream merging Figure 1: Architecture Liu, et al. Expires 4 September 2023 [Page 4] Internet-Draft Protocol for interactive low-latency med March 2023 The interactive real-time communication network can be provided by giant cloud provider. The communication network can provide fundamental capabilities of media stream, including media pulling, media pushing. In addition, the network can also support capabilities such as media merging and media switching. The capabilities can be triggered by control server and server for media streaming merging, which are provided by 3rd party. Based on those capabilities, the audience can receive corresponding media from broadcaster or merged media between broadcaster and requested audience for interaction seamlessly. 3. Signaling procedure This section defines the signaling procedure of Interactive real-time media communication network. Figure 2 shows the signaling procedure of Interactive real-time media communication among broadcaster, requested audience for interaction and other audience. The broadcaster and audience firstly push their media streams to the interactive real-time media communications network. A audience wishes to interact with the broadcaster and thus sends a request to the control server for interaction. The control server processes the request and sends comment for media merging to the server for media stream merging. Upon the receipt of merging request from control server, the server for media stream merging pulls the corresponding streams from both the broadcaster and the requested audience for interaction and processes with the media merging. After the completion of media merging, the server for media stream merging pushes the merged media to the Interactive real-time media communication network which then sends the merged media to corresponding edge media distribution servers which connects the audiences who watch the media. After the distribution, the control server sends the command for media switching to the Interactive real- time media communication network. The network then forwards the switching signaling message to the edge node. Up the receipt of the signaling message, the edge node performs the media switching by pushing the merged media to the audiences. Audience Interactive real-time connected with Server for media media communication Broadcaster the broadcaster Control Server stream merging network Audience +--------------+ +--------------+ +--------------+ +--------------+ +--------------+ +--------------+ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Liu, et al. Expires 4 September 2023 [Page 5] Internet-Draft Protocol for interactive low-latency med March 2023 +------+-------+ +-------+------+ +-------+------+ +------+-------+ +-------+------+ +------+-------+ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Push media stream | | | | +----------------------------+---------------------------+-------------------------+---------------------------->| | | | | | | | | | | | | | | | | | | | | | | | | Pull media stream | | | | | |<------------------------+ | | | | | | | | | | | | | | | Push media stream | | | | +---------------------------+----------------------------+------------------------->| | | | | | | | | | | | | | | | | | | | | | | | | | | Pull media stream | | | | | +----------------------------+---------------------------+----------------------------+------------------------->| | | | | | | | | | | | | | | | | | | | | | | | | | | | Pull media stream | | | | | +---------------------------+----------------------------+------------------------->| | | | | | | | | | |Command for stream merging | | | | | +--------------------------->| Pull stream for merging | | | | | +------------------------->| | | | | | | | | | | | Push merged stream | | | | | +------------------------->| | | | |Command for stream switching| | | | | +----------------------------+------------------------->| | | | | | +--+ | | | | | | | | | | | | |<-+ | | | | | |Perform stream switch | | | | | | | Figure 2: Procedure Liu, et al. Expires 4 September 2023 [Page 6] Internet-Draft Protocol for interactive low-latency med March 2023 4. Signaling Specification This section defines the signaling specification for the interactive real time media communication. In order to achieve the merging and switching functionalities for different media source, signaling messages need to be delivered to the corresponding entities (e.g. control server, edge node, etc) in order to perform the proper operations. The signaling message of interactive media control protocol is shown as follows: Interactive Media Control Message { Message Type (i), Message Length (i), Message Payload (..), } Figure 3: Interactive media signaling message To process with the signaling message, the corresponding entities need to identify the type of signaling message. This can be achieved via using message type which can be carried by the message header. The message types of Interactive media control protocol can be described as follows: +=====+===========+ | ID | Messages | +=====+===========+ | 0x0 | Merging | +-----+-----------+ | 0x1 | Switching | +-----+-----------+ | 0x2 | Grabbing | +-----+-----------+ | 0x3 | Pulling | +-----+-----------+ | 0x4 | Pushing | +-----+-----------+ Table 1: Message types of Interactive media control protocol The message length indicates the total length of the message payload filed in bytes. Message payload contains the information for controlling media merging and media switching. The subsequent sub- section describes these two message types and related payload in detail. Liu, et al. Expires 4 September 2023 [Page 7] Internet-Draft Protocol for interactive low-latency med March 2023 4.1. Merging signaling message Merging signaling message is used to request the server for media stream merging to perform media merging between a broadcaster and an audience. The merging signaling message is shown as follows: Merging Message {, Payload Type (i), first media info { 1st media ID (i), 1st media URL (b), } 2nd media info { 2nd media ID (i), 2nd media URL (b), } } Figure 4: Merging signaling message The payload type field in the header indicates the merging signaling message. First media ID and second media ID represent the IDs of the media from broadcaster and the requested audience for interaction, respectively. The ID is comprised of a string which represents the unique ID of an media source. Each media info contains the media ID, media URL. The media URL represents the address of edge node which interacts with the audience. 4.2. Switching signaling message Switching signaling message is used to instruct the Interactive real- time media communication system to perform media switching upon the receipt of the request from the control server. The switching signaling message is shown as follows: Switching Message { Payload Type (i), Source media info { Src media ID (i), Src media URL (b), }, Destination media info { Dst media ID (i), Dst media URL (b), } } Figure 5: Switching signaling message Liu, et al. Expires 4 September 2023 [Page 8] Internet-Draft Protocol for interactive low-latency med March 2023 The payload type field in the header indicates the switching signaling message. Source media info contains the information regarding source media from the broadcaster. Destination media info contains the information regarding destination media which is the merged media between the broadcaster and the requested audience for interaction. Each media info contains the media ID, media URL. The switch signaling message is sent to the edge node which manages the media delivery for the audience. If the edge node acknowledges the media switching, it re-directs the media content with the destination media using media transmission protocols (e.g. QUIC, WebRTC, etc). Upon the receipt of the switching signaling message, the media transmission protocol decides time-stamp, information regarding I-frame, and optionally the sequence number to achieve the re-direction of the new merged media. This is to make sure that the audience can smoothly switch to the merged media without the negative impact on user experience. 4.3. Grabbing signaling message Grabbing signaling message is used to instruct the Interactive real- time media communication system to switch edge node for audience, for example, in mobility scenario. In the mobility case, the Interactive real-time media communication system may decide to switch a more suitable edge node for media pushing for an audience according the location information. The grabbing signaling message is shown as follows: Grabbing Message { Payload Type (i), new media info { new media ID (i), new media URL (b), }, error_code, } } Figure 6: Grabbing signaling message Liu, et al. Expires 4 September 2023 [Page 9] Internet-Draft Protocol for interactive low-latency med March 2023 The grabbing signaling message is sent from Interactive real-time media communication system to the edge node. A new edge node firstly start pushing media to the audience. Meanwhile, it registers the service to the Interactive real-time media communication system. The system detects that the media pushing service already exists and thus sends the grabbing signaling message to the old edge node. For the old edge node, the grabbing signaling message is used to instruct the node to drop the media pushing to the audience. The error code indicates the reason for dropping. The reasons are shown below: +========+=====================+ | Reason | Code | +========+=====================+ | 0x0 | Dropped by Mobility | +--------+---------------------+ | 0x1 | Proactive dropping | +--------+---------------------+ | 0x2 | Passive dropping | +--------+---------------------+ Table 2: Reason code for grabbing signaling message Dropped by Mobility indicates the case where a new edge node has taken place and pushes the media to the audience instead of the old edge node. Proactive dropping indicates the case where an edge node gets issues on the media pushing and the audience can request for re- connection for the delivery of the media. Passive dropping indicates the case where the corresponding media has been banned and thus can not be pushed anymore. 4.4. Pulling signaling message Pulling signaling message is sent from audience to the edge node. Once the pulling signaling message is acknowledged, the edge node sends the corresponding media to the audience. The pulling signaling message is shown below: Pulling Message { Payload Type (i), Media info { Media URL (b), } } Figure 7: Pulling signaling message Liu, et al. Expires 4 September 2023 [Page 10] Internet-Draft Protocol for interactive low-latency med March 2023 The payload type field in the header indicates the pulling signaling message. The media URL indicates the address of the target media which can be obtained from the edge node. The edge node allocates a media ID for the broadcaster or the requested audience for interaction so that the media can be uniquely identified in the communication system. Upon the receipt of the pulling signaling message, the edge node acknowledges the signaling message with the media ID which uniquely identifies the target media. 4.5. Pushing signaling message Pushing signaling message is sent from broadcaster or the requested audience for interaction to the edge node in order to start pushing media to the edge node. The pulling signaling message is shown below: Pushing Message { Payload Type (i), Media info { Media URL (b), } } Figure 8: Pushing signaling message The payload type field in the header indicates the pushing signaling message. The media ID indicates the media that is about to sent to the edge node. The URL represents the address of the broadcaster or the requested audience for interaction which push media to the edge node. Upon the receipt of the pulling signaling message, the edge node acknowledges the signaling message with a media ID which uniquely identifies the pushed media. 5. Acknowledgements 6. IANA Considerations TBD. 7. Security Considerations The signaling messages defined in this document should be protected by security mechanism. 8. Normative References Liu, et al. Expires 4 September 2023 [Page 11] Internet-Draft Protocol for interactive low-latency med March 2023 [1] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", March 1997, . [2] Rose, M., "Writing I-Ds and RFCs using XML", RFC 2629, DOI 10.17487/RFC2629, June 1999, . Authors' Addresses Dapeng(Max) Liu Alibaba Group Email: max.ldp@alibaba-inc.com Yaming He Alibaba Group Email: heyaming.hym@alibaba-inc.com Xiaobo Yu Alibaba Group Email: shibo.yxb@alibaba-inc.com Xiao Kai Alibaba Group Email: xiaokaikai.xk@alibaba-inc.com Songlin Li Alibaba Group Email: songlin.lsl@alibaba-inc.com Liu, et al. Expires 4 September 2023 [Page 12]