Network Working Group O. Boyaci Internet-Draft H. Schulzrinne Intended status: Standards Track Columbia U. Expires: April 30, 2009 October 27, 2008 RTP Payload format for Application and Desktop Sharing draft-boyaci-avt-app-sharing-00 Status of this Memo By submitting this Internet-Draft, each author represents that any applicable patent or other IPR claims of which he or she is aware have been or will be disclosed, and any of which he or she becomes aware will be disclosed, in accordance with Section 6 of BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet- Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt. The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. This Internet-Draft will expire on April 30, 2009. Boyaci & Schulzrinne Expires April 30, 2009 [Page 1] Internet-Draft RTP Payload Format for App Sharing October 2008 Abstract This document defines a protocol to support accessing general graphical user interface (GUI) desktops and applications remotely, either by a single remote user or embedded into a multiparty conference. The protocol is designed to allow sharing of, and access to general windowing system applications that have not been expressly written to be accessed remotely. Table of Contents 1. Definitions . . . . . . . . . . . . . . . . . . . . . . . . . 4 2. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 5 3. Requirements Notation . . . . . . . . . . . . . . . . . . . . 8 4. Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 4.1. Coordinate System . . . . . . . . . . . . . . . . . . . . 9 4.2. Operation . . . . . . . . . . . . . . . . . . . . . . . . 11 4.3. Participants using UDP . . . . . . . . . . . . . . . . . . 13 4.4. Participants using TCP . . . . . . . . . . . . . . . . . . 13 4.5. Protocol Overview . . . . . . . . . . . . . . . . . . . . 13 4.5.1. Remoting Protocol Overview . . . . . . . . . . . . . . 14 4.5.2. Human Interface Protocol (HIP) Overview . . . . . . . 15 5. Remoting Protocol . . . . . . . . . . . . . . . . . . . . . . 16 5.1. Payload Format . . . . . . . . . . . . . . . . . . . . . . 16 5.1.1. RTP Header Usage . . . . . . . . . . . . . . . . . . . 16 5.1.2. Common remoting/HIP header . . . . . . . . . . . . . . 16 5.2. AH-to-participant messages . . . . . . . . . . . . . . . . 17 5.2.1. WindowManagerInfo . . . . . . . . . . . . . . . . . . 17 5.2.2. RegionUpdate . . . . . . . . . . . . . . . . . . . . . 19 5.2.3. MoveRectangle . . . . . . . . . . . . . . . . . . . . 21 5.2.4. MousePointerInfo . . . . . . . . . . . . . . . . . . . 22 5.3. Participant-to-AH Messages . . . . . . . . . . . . . . . . 22 5.3.1. Picture Loss Indication (PLI) . . . . . . . . . . . . 22 5.3.2. NACK Request . . . . . . . . . . . . . . . . . . . . . 22 6. Human Interface Protocol (HIP) . . . . . . . . . . . . . . . . 23 6.1. Payload Format . . . . . . . . . . . . . . . . . . . . . . 23 6.1.1. RTP Header Usage . . . . . . . . . . . . . . . . . . . 23 6.1.2. Common remoting/HIP header . . . . . . . . . . . . . . 23 6.2. MousePressed . . . . . . . . . . . . . . . . . . . . . . . 24 6.3. MouseReleased . . . . . . . . . . . . . . . . . . . . . . 24 6.4. MouseMoved . . . . . . . . . . . . . . . . . . . . . . . . 24 6.5. MouseWheelMoved . . . . . . . . . . . . . . . . . . . . . 25 6.6. KeyPressed . . . . . . . . . . . . . . . . . . . . . . . . 25 6.7. KeyReleased . . . . . . . . . . . . . . . . . . . . . . . 26 6.8. KeyTyped . . . . . . . . . . . . . . . . . . . . . . . . . 26 7. Implementation Notes . . . . . . . . . . . . . . . . . . . . . 27 8. Security Considerations . . . . . . . . . . . . . . . . . . . 28 Boyaci & Schulzrinne Expires April 30, 2009 [Page 2] Internet-Draft RTP Payload Format for App Sharing October 2008 9. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 29 9.1. Remoting Message Types Subregistry . . . . . . . . . . . . 29 9.2. HIP Message Types Subregistry . . . . . . . . . . . . . . 29 9.3. Media Type Registrations . . . . . . . . . . . . . . . . . 30 9.3.1. Registrations of Media Type application/remoting . . . 30 9.3.2. Registrations of Media Type application/hip . . . . . 31 10. Mapping to the Session Description Protocol (SDP) . . . . . . 33 10.1. Mapping remoting Media Type Parameters into SDP . . . . . 33 10.2. Mapping hip Media Type Parameters into SDP . . . . . . . . 33 10.3. SDP Example . . . . . . . . . . . . . . . . . . . . . . . 33 Appendix A. Using BFCP for Application and Desktop Sharing . . . 35 11. References . . . . . . . . . . . . . . . . . . . . . . . . . . 36 11.1. Normative References . . . . . . . . . . . . . . . . . . . 36 11.2. Informative References . . . . . . . . . . . . . . . . . . 37 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 38 Intellectual Property and Copyright Statements . . . . . . . . . . 39 Boyaci & Schulzrinne Expires April 30, 2009 [Page 3] Internet-Draft RTP Payload Format for App Sharing October 2008 1. Definitions Application Host (AH): An application host (AH) is the computer which runs the shared application, distributes the screen updates to the participants, and regenerates human interface events received from participants. Participant: Participant is the computer which receives screen updates from AH and sends human interface events back to the AH. Participants do not need to store or run the shared application. More than one participant may connect to a single AH. Remoting Protocol: Remoting protocol messages allows the AH to distribute windowing information and screen updates to participants. Human Interface Protocol (HIP): Human Interface Protocol (HIP) allows participants to send human interface device (HID) events to AH. HIDs generates mouse or keyboard events such as a key press, key release, mouse move, and mouse click. Boyaci & Schulzrinne Expires April 30, 2009 [Page 4] Internet-Draft RTP Payload Format for App Sharing October 2008 2. Introduction Application and desktop sharing allows sharing of any software application with one or more participants over the Internet. The application host (AH) is the computer which runs the shared application. The participants receive the screen-view of the shared application from the AH. They do not need to run or store the shared application. Their mouse and keyboard events are delivered and regenerated at the AH. An Application and desktop sharing system adheres to a client-server architecture [Figure 1]. +-------------+ | | | participant | | | +-------------+ ^ | | | Screen updates | | HIP | | | | | v +-------------+ Screen updates +---------+ | |<-----------------| | | participant | | AH | | |----------------->| | +-------------+ HIP +---------+ Figure 1: Application sharing system architecture Application and desktop sharing enables collaborative work, software tutoring, and e-learning over the Internet. While two-party and multi-party conferencing using standards-based protocols is now common and well-developed, protocols for sharing applications are largely proprietary or based on the aging T.120 [T120] suite of protocols. In this document, we define an RTP payload format for application and desktop sharing. Application sharing differs from desktop sharing. In desktop sharing, a computer distributes all screen updates. In application sharing, the AH distributes screen updates if and only if they belong to the shared application's windows. Applications often consist of a changing set of related windows which serve the same task and are usually associated with the same process on the AH. Considering only the boundary of the shared windows is not enough. Other non-shared windows may cover the shared window or shared application may open new child windows such as those for selecting options or fonts. A Boyaci & Schulzrinne Expires April 30, 2009 [Page 5] Internet-Draft RTP Payload Format for App Sharing October 2008 true application sharing system must blank all the nonshared windows and must transfer all the child windows of the shared application. We note that remote access to an application ("remote desktop") and multiple users sharing an application within a collaboration setting such as a multimedia call or multiparty conference are very similar. The protocols defined in this document therefore support both. Remote access differs from video transmission of the sort for which most video encodings have been designed. In particular, screen encoding may need to be lossless and typically operates on artificial rather than natural (photographic) video input. The video input is characterized by large areas of the screen that remain unchanged for long periods of time, while others change rapidly. (However, rendering the output of a modern computer-generated animation application such as video games blurs the distinction between traditional motion video output and screen sharing.) Application sharing can be integrated into the existing IETF session model, encompassing session descriptions using the Session Description Protocol (SDP) [RFC2327] or successors and the Session Initiation Protocol (SIP) [RFC3261]. Application sharing needs many of the same control functions as other multimedia sessions, such as address binding, session feature and media negotiation. The session model is also beneficial for the remote desktop case, as it allows to re-use many of the well-developed session components and easily supports hybrid multimedia models, such as the delivery of desktop audio to the remote user. Remote access to graphical applications and desktops, as defined in this document, has two important characteristics. First, the access protocol is unaware of any semantic characteristics of the applications being shared; it only transmits the visual characteristics of the windows. This is different, therefore, from shared-drawing or shared-editing tools that allow distributed modification of documents. Secondly, the protocol is designed to work with applications which were not written to be used remotely, by intercepting or simulating their connections to their native window systems. That distinguishes it from systems such as the X Window System [X] which allow natively-written applications to be displayed on remote viewers. We distinguish between the AH user and participants. The AH user interacts with the application using normal operating system mechanisms. Participants interact via the delivery protocols described here. The application sharing problem can be divided into four components: Boyaci & Schulzrinne Expires April 30, 2009 [Page 6] Internet-Draft RTP Payload Format for App Sharing October 2008 (1) setting up a session to the AH, (2) transporting human interface events from the participants to the AH, (3) delivering screen output from the AH to the participants, (4) moderating access to shared human interface devices such as pointing devices (e.g., mouse, joystick, trackball) and text input (keyboard). We refer to component (2) as the "human interface protocol (HIP)" and component (3) as the "remoting protocol". Remote user input access is moderated by the Binary Floor Control Protocol (BFCP) [RFC4582]. Session negotiation and description can be provided by existing session setup protocols. Thus, they are beyond the scope of this document. The rest of this document is laid out as follows. Section 3 defines the common terminology for normative references. Section 4 gives an overview of the protocol's architecture and components. The remoting protocol and HIP are described in Section 5 and 6, respectively. Section 8 gives implementation notes, and Section 9 discusses security considerations. Boyaci & Schulzrinne Expires April 30, 2009 [Page 7] Internet-Draft RTP Payload Format for App Sharing October 2008 3. Requirements Notation The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC2119]. Boyaci & Schulzrinne Expires April 30, 2009 [Page 8] Internet-Draft RTP Payload Format for App Sharing October 2008 4. Overview 4.1. Coordinate System The origin (0,0) of the coordinate system is the upper left corner. All coordinates carried in the protocol messages are absolute and measured in pixels. 0,0 <- x axis -> +---------------------------------------------------+ | | | 220,150 | | +------ A -------+ | | | | 850,320 | | | | +-- C ---+ | | | 450,400 | | | | | | +--------- B ----+ | | | | | | | +---------+ | | | | | | | +---------| | | | | | | | +-----------------+ | | 800,700 | | | | | +---------------------------------------------------+ AH 1280,1024 Figure 2: An AH shares three windows Boyaci & Schulzrinne Expires April 30, 2009 [Page 9] Internet-Draft RTP Payload Format for App Sharing October 2008 0,0 +------------------------------------------------+ | | | 220,150 | | +------ A -------+ | | | | 850,320 | | | | +-- C ---+| | | 450,400 | | || | | +--------- B ----+ | || | | | | +---------+| | | | | | | +---------| | | | | | | | +-----------------+ | | 800,700 | +------------------------------------------------+ Participant-1 1024,768 Figure 3: Participant 1 displays the shared windows in their original coordinates 0,0 +------ A -------+---------------------------------+ | | 630,170 | | | +-- C ---+ | | 230,250 | | | | | +--------- B ----+ | | | | | | +---------+ | | | | | +---------| | | | | | | | +-----------------+ | | 580,550 | | | | | | | | | +---------------------------------------------------+ Participant-2 1280,1024 Figure 4: Participant 2 displays the shared windows in shifted coordinates Boyaci & Schulzrinne Expires April 30, 2009 [Page 10] Internet-Draft RTP Payload Format for App Sharing October 2008 0,0 +------ A -------+----------+ | | | | 60,180 +-- C ---+ | | +--------- B ----+ | | | | | | | | | |-+ | | | | | +----| | | +----+-----------------+-----+ Participant-3 640,480 Figure 5: Participant 3 displays the shared windows in completely different coordinates Figure 2 shows an example scenario where three windows are shared. All coordinates are absolute. A participant can display the windows in their original coordinates or it can display them in different coordinates. In Figure 3, participant 1 displays the windows in their original coordinates. Participant 2 shifts all the windows 220 pixels left and 150 pixels up Figure 4. Participant 2 preserves the relations between windows, while participant 3 combines all the windows in order to fit them to its small screen Figure 5. In this example scenario, all participants preserve the z-order of windows. The AH informs the participants about windows' positions and sizes, z-order, and their groupings. The AH MAY assign same group identifier to the windows which belongs to the same process. Grouping information MAY be used by the participant while relocating the windows or enforcing the z-order. A participant MAY allow changing the z-order (i.e., stacking order) of windows locally, without changing the z-order in the AH. Several remoting/HIP message types carries left, top, width and height fields. The units of these fields are in pixels and they are unsigned. The AH MUST only accept legitimate HIP events by checking whether the requested coordinates are inside the shared windows. 4.2. Operation Detecting a change in the GUI of the shared application, the AH prepares an RTP [RFC3550] packet containing an encoded image of the updated region. RTP allows the participants to re-order the packets, recognize missing packets and synchronize application sharing with other media types like audio and video. The screen updates can be encoded with PNG [I-D.boyaci-avt-png], JPEG [RFC2435], JPEG 2000 [RFC5371], Theora [theora] or other media types like H.264 [RFC3984], according to their characteristics. PNG is an open image format which uses a lossless compression algorithm and more suitable for Boyaci & Schulzrinne Expires April 30, 2009 [Page 11] Internet-Draft RTP Payload Format for App Sharing October 2008 computer generated images. JPEG is lossy, but more suitable for photographic images. JPEG 2000 supports both lossless and lossy compression, therefore suitable for both computer generated images and movies. Theora is an open source video codec comparable to H.264 and suitable for movie encoding. Although multiple users could receive the screen updates simultaneously, clearly only one of them can manipulate the application via keyboard and mouse events. The Binary Floor Control Protocol (BFCP) [RFC4582] MAY be used to restrict the control of the application to a single user. BFCP receives floor request and floor release messages from participants; and then it grants the floor to the appropriate participant for a period of time while keeping the requests from other participants in a FIFO queue. The details of utilizing BFCP in the context of application and desktop sharing are given in Appendix A. The protocol supports two different mouse pointer models. Mouse pointer images can be transmitted as RegionUpdate messages or they may be transmitted seperately as MousePointerInfo messages. The AH decides which mouse model to use. The participants MUST support both mouse models. HIP supports both UTF-8 encoded unicode characters and other keyboard keys which are not defined in unicode such as function and control keys. For keyboard events publicly available Java virtual key codes [keycodes] are used. The application and desktop sharing models defined in this document can be integrated into the IETF conferencing model. The Session Initiation Protocol (SIP) [9] can be used to intiate and control remote access. This allows the use of existing SIP mechanisms for confidentiality, authentication and authorization, user location, conferencing. Additional, optional mechanisms can enhance application and desktop sharing. Audio streams can be associated with a desktop or application; participant-side scaling can be used to optimize transmission of data to participants with a small screen; and it is often useful to allow copy-and-paste between applications running on a participant and those running on an AH. This document does not define any such extensions. The AH can support both multicast and unicast transmissions. For unicast connections, either UDP or TCP can be used. The AH can share an application to TCP participants, UDP participants, and several multicast addresses in the same sharing session. Boyaci & Schulzrinne Expires April 30, 2009 [Page 12] Internet-Draft RTP Payload Format for App Sharing October 2008 4.3. Participants using UDP The AH controls the transmission rate for participants using UDP, because UDP itself does not provide flow and congestion control. Several simultaneous multicast sessions with different transmission rates can be created at the AH. Participants can join a sharing session anytime, and they need the shared windows' information and full screen buffer before receiving partial updates. Therefore, participants using UDP send an RCTP- based feedback message, Picture Loss Indication (PLI) [RFC4585], after joining the session. The AH prepares and transmits the windows' state information and image of the whole shared region after receiving a PLI message. 4.4. Participants using TCP Since TCP provides reliable communication and flow control, it is more suitable for unicast sessions. TCP participants may have different bandwidths, so an algorithm which sends the updates at the link speed of each participant is needed. Neither TCP nor RTP declares the length of an RTP packet. Therefore, RTP framing [RFC4571] is used to split RTP packets within the TCP byte stream. The AH prepares and transmits the windows' state information and image of the whole shared region to the new participant, right after the TCP connection establishment. 4.5. Protocol Overview Application and desktop sharing protocol consists of two subprotocols: remoting and human interface protocol (HIP). Remoting messages transmit screen updates from AH to participants. HIP messages transmit mouse and keyboard events from the participant to the AH. Remoting and HIP messages are RTP messages. They consist of an RTP header, common remoting/HIP header, message-type specific header, and message payload [Figure 6]. The HIP messages have a different payload type than the remoting messages. Boyaci & Schulzrinne Expires April 30, 2009 [Page 13] Internet-Draft RTP Payload Format for App Sharing October 2008 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ . . . RTP header . . . +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Common remoting/HIP header | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ . . . Message-type specific header . . . +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ . . . . . Message Specific Payload . . +-+-+-+-+-+-+-+-+ . | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 6: Application sharing protocol message structure 4.5.1. Remoting Protocol Overview The remoting protocol consists of four messages from the AH to the participant and two control messages from participant to AH. The AH- to-participant messages are WindowStateInfo, RegionUpdate, MoveRectangle, and MousePointerInfo. The RTCP messages from UDP- based participant to AH are "Picture loss indication (PLI)" and "NACK request". Participants MUST implement all protocol messages described in this document. The WindowManagerInfo message informs the participants about the windowIDs of the windows, their positions and sizes, z-order, and their groupings. All remoting messages carry the windowID to identify the target of message. For TCP participants, the AH transmits WindowManagerInfo message right after establishing a connection. UDP participants send a "Picture loss indication (PLI)" to the AH as soon as they join the session. Receiving this PLI message, the AH transmits WindowManagerInfo message. The AH transmits RegionUpdate messages for updated regions. Whenever the shared window resizes or relocates, the AH sends a WindowManagerInfo message. Similarly, if the z-order of windows changes, the AH send a WindowManagerInfo message. MoveRectangle instructs the participant to move a region from one place to another, which is efficient for some drawing operations like scrolls. The MousePointerInfo message transmits the position and icon of the mouse pointer. Some AHs may transmit pointer images inside the RegionUpdate messages, so they may Boyaci & Schulzrinne Expires April 30, 2009 [Page 14] Internet-Draft RTP Payload Format for App Sharing October 2008 not need MousePointerInfo message. "Picture loss indication (PLI)" and "NACK request" are control messages and they are transmitted as RTCP messages. The "NACK request" is used only by UDP participants to request retransmission of missing packets from the AH. AHs MAY support retransmissions. PLI can be used by both UDP and TCP participants to request a full screen refresh. 4.5.2. Human Interface Protocol (HIP) Overview HIP consist of seven messages: namely, Mouse Pressed, Mouse Released, Mouse Moved, Mouse Wheel Moved, Key Pressed, Key Released and Key Typed. These messages are all from AH to participant and carried as RTP messages. However, these HIP messages have different payload type than the remoting messages. Boyaci & Schulzrinne Expires April 30, 2009 [Page 15] Internet-Draft RTP Payload Format for App Sharing October 2008 5. Remoting Protocol 5.1. Payload Format 5.1.1. RTP Header Usage Marker bit: The marker bit indicates the last packet of a multi- packet RegionUpdate (Section 5.2.2 message. The marker bit allows the receiver to finish decoding the picture, without waiting for the next packet with a new timestamp. Unless defined otherwise, all other message types MUST set this bit to zero Timestamp: The RTP timestamp indicates the time instance the remoting message has been created at the AH. The RTP timestamp is based on a 90-kHz clock. If a RegionUpdate message occupies more than one packet, the timestamp SHALL be the same for all of those packets. Furthermore, the initial value of the timestamp MUST be random (unpredictable) to make known-plaintext attacks more difficult [RFC3550]. The remaining RTP header fields are used as specified in RFC 3550. 5.1.2. Common remoting/HIP header All remoting protocol messages carry a common remoting/HIP header [Figure 7] which follows the RTP header. Message type and parameter fields are 8 bit identifiers, whereas the windowID is a 16-bit identifier. The windowID field is unsigned and has a range of 0-65535. 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Msg Type | Parameter | WindowID | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 7: Common remoting/HIP header Table 1 enumerates the message types defined in this document. Participants MUST implement all of them. Section 9 describes how additional message types can be registered with IANA. Participants MAY ignore such additional message types. Boyaci & Schulzrinne Expires April 30, 2009 [Page 16] Internet-Draft RTP Payload Format for App Sharing October 2008 +-------+-------------------+ | Value | Message Type | +-------+-------------------+ | 1 | WindowManagerInfo | | | | | 2 | RegionUpdate | | | | | 3 | MoveRectangle | | | | | 4 | MousePointerInfo | +-------+-------------------+ Table 1: Remoting protocol message types 5.2. AH-to-participant messages 5.2.1. WindowManagerInfo The WindowManagerInfo message informs the participants about windows, their positions and sizes, z-order, and their groupings. This message transfers the complete window manager state to the participants. Each shared window resize and relocation in any coordinate triggers a WindowmangerInfo message. Parameter and WindowID fields of common remoting/HIP header MUST be ignored. This message carries a message specific payload. One or more window records [Figure 8] follow the common remoting/HIP header. 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | WindowID | GroupID | Reserved | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Left | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Top | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Width | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Height | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 8: A Window Record Each window record is 20-bytes. The z-order information is given implicitly to the participants. The first record describes the window at the bottom of the stacking order, the last record the one on top. The "left" and "Top" fields carries the upper-left coordinate of the window. The "Width" and "Height" fields carries Boyaci & Schulzrinne Expires April 30, 2009 [Page 17] Internet-Draft RTP Payload Format for App Sharing October 2008 the width and the height of the window, respectively. Each window is assigned a WindowID. The participant MUST create a window for each new WindowID and MUST close this window after receiving a WindowManagerInfo message which does not contain this WindowID. GroupID fields informs the participant about grouping of windows. The AH MAY assign same GroupID to the windows which belongs to the same process. Grouping information MAY be used by the participant while relocating the windows or enforcing the z-order. The value of "0" for GroupID field is reserved and represents no grouping for given window. Figure 9 is an example WindowManagerInfo message for the three shared windows in Figure 2. The participant MUST keep the existing window image after a resize and relocation. Boyaci & Schulzrinne Expires April 30, 2009 [Page 18] Internet-Draft RTP Payload Format for App Sharing October 2008 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Msg Type = 1 | Parameter = 0 | WindowID = 0 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | WindowID = 1 | GroupID = 1 | Reserved = 0 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Left = 220 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Top = 150 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Width = 350 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Height = 450 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | WindowID = 2 | GroupID = 2 | Reserved = 0 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Left = 850 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Top = 320 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Width = 160 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Height = 150 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | WindowID = 3 | GroupID = 1 | Reserved = 0 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Left = 450 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Top = 400 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Width = 350 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Height = 300 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 9: An example WindowManagerInfo message with three Window Records 5.2.2. RegionUpdate The RegionUpdate message instructs the participant to update the specified region of a window with new content. This message carries a message-type specific header and payload. This protocol supports all media types which have an RTP payload specification. It is possible that AH or participant may support only some media types. Therefore, they should negotiate supported media types during the session establishment. The 8 bit "parameter" field of the common Boyaci & Schulzrinne Expires April 30, 2009 [Page 19] Internet-Draft RTP Payload Format for App Sharing October 2008 remoting/HIP header will carry both the FirstPacket bit and the actual payload type of the content. The 7 bit PT field carries the actual payload type of the content which can be PNG, JPEG, Theora, or any other media type which has an RTP payload specification. All AH and participant software implementations MUST support PNG images [I-D.boyaci-avt-png]. The message-type specific header follows common remoting/HIP header. Message-type specific header consists of two 32 bit parameters, left and top. These two parameters informs the participants about the left-top coordinate of the RegionUpdate. The width and height of the RegionUpdate is not transmitted explicitly by this protocol. 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | RegionUpdate |F| PT | WindowID | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 10: Common remoting/HIP header for RegionUpdate messages If the content of the update does not fit into a single RTP message, it will be carried in several RTP payloads. All the payloads will carry the 32 bit common remoting/HIP header, while left and top fields are carried only in the first RTP payload. The marker bit and FirstPacket bit informs the particapants about the fragmentation. +------------+-----------------+-----------------------+ | Marker bit | FirstPacket bit | Fragment Type | +------------+-----------------+-----------------------+ | 1 | 1 | Not Fragmented | | | | | | 0 | 1 | Start Fragment | | | | | | 0 | 0 | Continuation Fragment | | | | | | 1 | 0 | End Fragment | +------------+-----------------+-----------------------+ Table 2: Marker and FirstPacket bits carry fragmentation info Figure 11 displays an example RegionUpdate message. The RTP header is omitted in Figure 11. The RegionUpdate is non fragmented, therefore both the marker bit in the RTP header and FirstPacket bit in the payload header is set to 1. Boyaci & Schulzrinne Expires April 30, 2009 [Page 20] Internet-Draft RTP Payload Format for App Sharing October 2008 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Msg Type = 2 |1| PT | WindowID = 1 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Left | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Top | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ . . . . . Payload . . +-+-+-+-+-+-+-+-+ . | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 11: An example non fragmented RegionUpdate message 5.2.3. MoveRectangle The MoveRectangle message instructs the participant to move the specified region of a window to a new position. This message carries a message-type specific header. The AH informs the participants about the source rectangle via source left, source top, width and height parameters. Participants learns the destination coordinates from destination left and destination top parameters. Source and destination rectangles may overlap. 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Source Left | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Source Top | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Width | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Heigth | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Destination Left | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Destination Top | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 12: Message specific payload for move rectangle Boyaci & Schulzrinne Expires April 30, 2009 [Page 21] Internet-Draft RTP Payload Format for App Sharing October 2008 5.2.4. MousePointerInfo Some AHs MAY include the mouse pointer image inside the RegionUpdate messages. However, some AHs MAY choose to inform the participant about the mouse position and icon explicitly. If the RegionUpdate messages contain the mouse pointer icon, then the MousePointerInfo message is unnecessary. When receiving this message, the participant should draw the mouse pointer to the given position. This message carries a message specific payload. The format of this message is same as RegionUpdate message Section 5.2.2 except they have different message types. The payload of MousePointerInfo message can be only the left and top coordinates. In this case, the participant MUST move the existing pointer image to the given coordinates. Payload MAY carry both the left and top coordinates and the new image of the mouse pointer. The participant MUST store and use this image until a new image arrives from the AH. If the AH uses MousePointerInfo messages, it MUST inform the late joiners about the current position and image of mouse pointer. 5.3. Participant-to-AH Messages Participants using UDP can send two RTCP messages to the AH. Late- joiners MAY inform the AH using the "Picture loss indication (PLI)" message in order to receive a full screen update. For the missing packets, UDP participants MAY send a "NACK Request". 5.3.1. Picture Loss Indication (PLI) "Picture Loss Indication (PLI)" message instructs the AH to generate a full screen update of the shared region. Before the full screen update, the AH will send a WindowManagerInfo message to inform the new participant about windows. Both TCP and UDP participants MAY transmit this message. The message format conforms to the "Picture Loss Indication (PLI)" section 6.3.1 of [RFC4585]. 5.3.2. NACK Request "NACK Request" message informs the AH about missing RTP packets. The message format conforms to the "Generic NACK" section 6.2.1 of [RFC4585]. Multicast participants and AHs MAY take necessary precautions to prevent NACK storms such as waiting random amount of time before sending a "NACK Request" message. Boyaci & Schulzrinne Expires April 30, 2009 [Page 22] Internet-Draft RTP Payload Format for App Sharing October 2008 6. Human Interface Protocol (HIP) Participants MAY send human interface events to the AH in order to interact with the shared application. 6.1. Payload Format 6.1.1. RTP Header Usage Marker Bit: The marker bit MUST be set to zero by the participant and ignored by the AH. Timestamp: The RTP timestamp indicates when the keyboard or mouse event occurred at the participant. The RTP timestamp of HIP messages is based on a 90-kHz clock. The initial value of the timestamp MUST be random (unpredictable) to make known-plaintext attacks on encryption more difficult; see RTP [RFC3550]. The remaining RTP header fields are used as specified in RFC 3550. 6.1.2. Common remoting/HIP header All HIP messages carry the same common remoting/HIP header shown in Figure 7 and discussed in Section 5.1.2. The WindowID parameter indicates the window where the keyboard or mouse event took place, i.e., the window that had keyboard or mouse focus. The following HIP message types are defined: +-------+-----------------+ | Value | Message Type | +-------+-----------------+ | 121 | MousePressed | | | | | 122 | MouseReleased | | | | | 123 | MouseMoved | | | | | 124 | MouseWheelMoved | | | | | 125 | KeyPressed | | | | | 126 | KeyReleased | | | | | 127 | KeyTyped | +-------+-----------------+ Table 3: HIP Message Types Boyaci & Schulzrinne Expires April 30, 2009 [Page 23] Internet-Draft RTP Payload Format for App Sharing October 2008 6.2. MousePressed The MousePressed message instructs the AH to generate a mouse pressed event at the given coordinates of the screen. This message carries a message-type specific header. The "parameter" field of the common remoting/HIP header carries the mouse button information. The values of 1, 2 and 3 are defined for left, right, and middle button, respectively. The AH and participant MAY negotiate additional values for other mouse buttons. The AH MAY ignore unrecognized values. 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Left | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Top | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 13: Message specific payload for mouse pressed 6.3. MouseReleased The MouseReleased message instructs the AH to generate a mouse released event at the given coordinates of the screen. This message carries a message-type specific header. The "parameter" field of the common remoting/HIP header carries the mouse button information. The values of 1, 2 and 3 are defined for left, right, and middle button, respectively. Other values MAY be defined for other mouse buttons. The AH MAY ignore unrecognized values. 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Left | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Top | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 14: Message specific payload for mouse released 6.4. MouseMoved The MouseMoved message instructs the AH to move the mouse pointer to the coordinates provided. This message carries a message-type specific header. Boyaci & Schulzrinne Expires April 30, 2009 [Page 24] Internet-Draft RTP Payload Format for App Sharing October 2008 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Left | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Top | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 15: Message specific payload for mouse moved 6.5. MouseWheelMoved The MouseWheelMoved message instructs the AH to generate a mouse wheel moved event at given coordinates of the screen. This message carries a message-type specific header. The "distance" field carries the wheel rotation amount as "120 * (number of notches)". A mouse wheel has discrete, evenly spaced notches. When user rotates the wheel, a wheel message is sent to OS as each notch is encountered. The "distance" field does not carry number of notches in order to support a smooth-scrolling wheel. Instead, "distance" field carries each notch as 120. Some mice MAY only send multiples of 120, while a smooth-scrolling mouse MAY send any values. A positive value indicates that the wheel was rotated forward, away from the user; a negative value indicates that the wheel was rotated backward, toward the user. The negative values are transmitted using 2's complement method. 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Left | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Top | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Distance | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 16: Message specific payload for mouse wheel moved 6.6. KeyPressed The KeyPressed message instructs the AH to generate a "key pressed" event. This message carries a message-type specific header which consists of a 32 bit KeyCode. Java virtual keycodes are used and they are publicly available on the openJDK website [keycodes]. The actual values are inside the KeyEvent.java file. For example, F1 key is defined as "int VK_F1 = 0x70;" in KeyEvent.java. Boyaci & Schulzrinne Expires April 30, 2009 [Page 25] Internet-Draft RTP Payload Format for App Sharing October 2008 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | KeyCode | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 17: Message specific payload for key pressed 6.7. KeyReleased The KeyReleased message instructs the AH to generate a "key released" event. This message carries a message-type specific header which consists of a 32 bit KeyCode. Java keycodes are used and they are publicly available at openJDK website [keycodes]. The actual values are inside the KeyEvent.java file. For example, F1 key is defined as "int VK_F1 = 0x70;" in KeyEvent.java. A KeyReleased event for a key without a prior KeyPressed event for this key is acceptable. 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | KeyCode | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 18: Message specific payload for key released 6.8. KeyTyped KeyTyped message instructs the AH to inject some number of UTF-8 encoded characters into the operating systems input queue. This message carries a message specific payload. There is no padding for the UTF-8 string. The participant MUST send more than one KeyTyped message if the string does not fit into a single KeyTyped packet. 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ . . . . . UTF-8 String . . +-+-+-+-+-+-+-+-+ . | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 19: Message specific payload for key released Boyaci & Schulzrinne Expires April 30, 2009 [Page 26] Internet-Draft RTP Payload Format for App Sharing October 2008 7. Implementation Notes Application hosts shouldn't blindly send every screen update they observed to the participants. Instead, they should monitor the state of their TCP transmission buffers (through mechanisms such as the select() command) and only send the most recent screen data when there is no backlog. This will prevent screen latency for rapidly- changing images, when a viewer usually only needs to see the final state of the image. Boyaci & Schulzrinne Expires April 30, 2009 [Page 27] Internet-Draft RTP Payload Format for App Sharing October 2008 8. Security Considerations RTP packets using the payload format defined in this specification are subject to the security considerations discussed in the RTP specification [RFC3550]. Application sharing inherently exposes the shared applications to risks by malicious participants. They may, for example, access resources beyond the application itself, e.g., by installing or running scripts. It may be difficult to constrain access to specific user data, e.g., a specific set of slides, unless the user application can be sandboxed or run in some kind of "jail", with the sandbox control outside the view of the remoting protocol. Boyaci & Schulzrinne Expires April 30, 2009 [Page 28] Internet-Draft RTP Payload Format for App Sharing October 2008 9. IANA Considerations The IANA has created a new registry for Application and Desktop Sharing Parameters called "Application and Desktop Sharing parameters". This new registry has a number of subregistries which are described in the following sections. 9.1. Remoting Message Types Subregistry This section establishes the Remoting Message Types subregistry under the Application and Desktop Sharing Parameters registry. As per the terminology in RFC 2434 [RFC2434], the registration policy for Remoting Message Types shall be "Specification Required". For the purposes of this subregistry, the Remoting Message Types for which IANA registration is requested MUST be defined by a standards-track RFC. Such an RFC MUST specify the Remoting Message Type's value, name, format, and semantics. For each Remoting Message Type, the IANA registers its value, its name, and the reference to the RFC where the Remoting Message Type is defined. The following table contains the initial values of this subregistry. +-------+-------------------+------------+ | Value | Message Type | Reference | +-------+-------------------+------------+ | 1 | WindowManagerInfo | [RFC nnnn] | | | | | | 2 | RegionUpdate | [RFC nnnn] | | | | | | 3 | MoveRectangle | [RFC nnnn] | | | | | | 4 | MousePointerInfo | [RFC nnnn] | +-------+-------------------+------------+ Table 4: Initial values of the Remoting Message Type subregistry 9.2. HIP Message Types Subregistry This section establishes the HIP Message Types subregistry under the Application and Desktop Sharing Parameters registry. As per the terminology in RFC 2434 [RFC2434], the registration policy for HIP Message Types shall be "Specification Required". For the purposes of this subregistry, the HIP Message Types for which IANA registration is requested MUST be defined by a standards-track RFC. Such an RFC MUST specify the HIP Message Type's value, name, format, and semantics. Boyaci & Schulzrinne Expires April 30, 2009 [Page 29] Internet-Draft RTP Payload Format for App Sharing October 2008 For each HIP Message Type, the IANA registers its value, its name, and the reference to the RFC where the HIP Message Type is defined. The following table contains the initial values of this subregistry. +-------+-----------------+------------+ | Value | Message Type | Reference | +-------+-----------------+------------+ | 121 | MousePressed | [RFC nnnn] | | | | | | 122 | MouseReleased | [RFC nnnn] | | | | | | 123 | MouseMoved | [RFC nnnn] | | | | | | 124 | MouseWheelMoved | [RFC nnnn] | | | | | | 125 | KeyPressed | [RFC nnnn] | | | | | | 126 | KeyReleased | [RFC nnnn] | | | | | | 127 | KeyTyped | [RFC nnnn] | +-------+-----------------+------------+ Table 5: Initial values of the HIP Message Type subregistry 9.3. Media Type Registrations Following the guidelines in RFC 4855 [RFC4855] and RFC 4288 [RFC4288], this section registers new 'application' media subtypes for remoting and hip. 9.3.1. Registrations of Media Type application/remoting Type name: application Subtype name: remoting Required parameters: rate: RTP timestamp clock rate, which is equal to the sampling rate. The typical rate is 90000; other rates may be specified. retransmissions: Informs the participants whether the AH supports UDP retransmissions. The possible values are yes and no. Boyaci & Schulzrinne Expires April 30, 2009 [Page 30] Internet-Draft RTP Payload Format for App Sharing October 2008 Optional parameters: none Encoding considerations: This media type is framed binary data (see RFC 4288, Section 4.8). Security considerations: See Section Section 8 of RFC nnnn Interoperability considerations: none Published specification: RFC nnnn Additional information: none Person & email address to contact for further information: Omer Boyaci and Henning Schulzrinne Intended usage: COMMON Restrictions on usage: This media type depends on RTP framing, and hence is only defined for transfer via RTP (RFC 3550). Transfer within other framing protocols is not defined at this time. Applications that use this media type: Application and Desktop sharing tools. Remote tutoring tools. Author: Omer Boyaci and Henning Schulzrinne Change controller: IETF Audio/Video Transport working group delegated from the IESG. 9.3.2. Registrations of Media Type application/hip Type name: application Subtype name: hip Required parameters: rate: RTP timestamp clock rate, which is equal to the sampling rate. The typical rate is 90000; other rates may be specified. Optional parameters: none Encoding considerations: This media type is framed binary data (see RFC 4288, Section 4.8). Boyaci & Schulzrinne Expires April 30, 2009 [Page 31] Internet-Draft RTP Payload Format for App Sharing October 2008 Security considerations: See Section Section 8 of RFC nnnn Interoperability considerations: none Published specification: RFC nnnn Additional information: none Person & email address to contact for further information: Omer Boyaci and Henning Schulzrinne Intended usage: COMMON Restrictions on usage: This media type depends on RTP framing, and hence is only defined for transfer via RTP (RFC 3550). Transfer within other framing protocols is not defined at this time. Applications that use this media type: Application and Desktop sharing tools. Author: Omer Boyaci and Henning Schulzrinne Change controller: IETF Audio/Video Transport working group delegated from the IESG. Boyaci & Schulzrinne Expires April 30, 2009 [Page 32] Internet-Draft RTP Payload Format for App Sharing October 2008 10. Mapping to the Session Description Protocol (SDP) The information carried in this payload format has a specific mapping to fields in the Session Description Protocol (SDP) [RFC4566], which is commonly used to describe RTP sessions. When SDP is used to specify sessions, the mappings are as follows: 10.1. Mapping remoting Media Type Parameters into SDP The media type ("application") is carried in the SDP m= attribuate as the media name. The media subtype ("remoting") is carried in the SDP a=rtpmap as the encoding name. The parameter "rate" is carried in the SDP a=rtpmap attribuate as the clock rate. The mandated parameter "retransmissions" MUST be included in the SDP a=fmtp attribute. 10.2. Mapping hip Media Type Parameters into SDP The media type ("application") is carried in the SDP m= attribuate as the media name. The media subtype ("hip") is carried in the SDP a=rtpmap attribuate as the encoding name. The parameter "rate" is carried in the SDP a=rtpmap attribuate as the clock rate. 10.3. SDP Example The following example shows an example SDP usage. This SDP message is from AH to participant. The HIP stream and BFCP session are associated with each other via "label" and "m-stream" attributes according to SDP Format for BFCP Streams [RFC4583]. This SDP message informs the participant that AH supports both TCP and UDP for remoting. The AH supports UDP retransmissions, so participants MAY send NACK requests for missing packets. The port numbers MUST be same if AH is remoting the same content over both TCP and UDP. In this example, AH is sending the same content from port 6000. It is possible that AH may have more than one remoting session, in this case each session MUST use different port numbers. Boyaci & Schulzrinne Expires April 30, 2009 [Page 33] Internet-Draft RTP Payload Format for App Sharing October 2008 m=application 50000 TCP/BFCP * a=floorid:0 m-stream:10 m=application 6000 RTP/AVP 99 a=rtpmap:99 remoting/90000 a=fmtp: retransmissions=yes m=application 6000 TCP/RTP/AVP 99 a=rtpmap:99 remoting/90000 m=application 6006 TCP/RTP/AVP 100 a=rtpmap:99 hip/90000 a=label:10 Boyaci & Schulzrinne Expires April 30, 2009 [Page 34] Internet-Draft RTP Payload Format for App Sharing October 2008 Appendix A. Using BFCP for Application and Desktop Sharing Application and desktop sharing tools MAY utilize Binary Floor Control Protocol (BFCP) [RFC4582] for managing the ownership of AH's human interface devices (HID). BFCP defines several messages, but only five of them is a MUST for Application and Desktop Sharing, namely "Floor Request", "Floor Release", "Floor Granted", "Floor Released" and "Floor Request Queued". In Application and Desktop Sharing context, the floor is the AH's HIDs. In this context, it is possible that the AH MAY temporarily block HID events without revoking the floor control. For example, the AH MAY temporarily block HID events if the shared application loses the focus or is covered by a non-shared application. The AH informs the current floor holder about the status of HIDs via STATUS- INFO attribute of "Floor Granted" messages. The participant MAY receive several "Floor Granted" messages with different "HID Status" values. Participant applications MAY inform the user about current "HID Status". HID Status values are 16-bit unisgned values and are defined as: +-------+------------------------+ | Value | Status | +-------+------------------------+ | 0 | STATE_NOT_ALLOWED | | 1 | STATE_KEYBOARD_ALLOWED | | 2 | STATE_MOUSE_ALLOWED | | 3 | STATE_ALL_ALLOWED | +-------+------------------------+ Figure 20: HID Status Values Boyaci & Schulzrinne Expires April 30, 2009 [Page 35] Internet-Draft RTP Payload Format for App Sharing October 2008 11. References 11.1. Normative References [I-D.boyaci-avt-png] Boyaci, O. and H. Schulzrinne, "RTP Payload Format for Portable Network Graphics (PNG) image", draft-boyaci-avt-png-00 (work in progress), September 2008. [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997. [RFC2327] Handley, M. and V. Jacobson, "SDP: Session Description Protocol", RFC 2327, April 1998. [RFC2435] Berc, L., Fenner, W., Frederick, R., McCanne, S., and P. Stewart, "RTP Payload Format for JPEG-compressed Video", RFC 2435, October 1998. [RFC3261] Rosenberg, J., Schulzrinne, H., Camarillo, G., Johnston, A., Peterson, J., Sparks, R., Handley, M., and E. Schooler, "SIP: Session Initiation Protocol", RFC 3261, June 2002. [RFC3550] Schulzrinne, H., Casner, S., Frederick, R., and V. Jacobson, "RTP: A Transport Protocol for Real-Time Applications", STD 64, RFC 3550, July 2003. [RFC3984] Wenger, S., Hannuksela, M., Stockhammer, T., Westerlund, M., and D. Singer, "RTP Payload Format for H.264 Video", RFC 3984, February 2005. [RFC4571] Lazzaro, J., "Framing Real-time Transport Protocol (RTP) and RTP Control Protocol (RTCP) Packets over Connection- Oriented Transport", RFC 4571, July 2006. [RFC4582] Camarillo, G., Ott, J., and K. Drage, "The Binary Floor Control Protocol (BFCP)", RFC 4582, November 2006. [RFC4583] Camarillo, G., "Session Description Protocol (SDP) Format for Binary Floor Control Protocol (BFCP) Streams", RFC 4583, November 2006. [RFC4585] Ott, J., Wenger, S., Sato, N., Burmeister, C., and J. Rey, "Extended RTP Profile for Real-time Transport Control Protocol (RTCP)-Based Feedback (RTP/AVPF)", RFC 4585, July 2006. Boyaci & Schulzrinne Expires April 30, 2009 [Page 36] Internet-Draft RTP Payload Format for App Sharing October 2008 [RFC5371] Futemma, S., Itakura, E., and A. Leung, "RTP Payload Format for JPEG 2000 Video Streams", RFC 5371, October 2008. 11.2. Informative References [RFC2434] Narten, T. and H. Alvestrand, "Guidelines for Writing an IANA Considerations Section in RFCs", BCP 26, RFC 2434, October 1998. [RFC4288] Freed, N. and J. Klensin, "Media Type Specifications and Registration Procedures", BCP 13, RFC 4288, December 2005. [RFC4855] Casner, S., "Media Type Registration of RTP Payload Formats", RFC 4855, February 2007. [T120] International Telecommunication Union, "Data Protocols for Multimedia Conferencing", X ITU-T Recommendation T.120. [X] Scheifler, R., "X Window System Protocol", X Consortium Standard X Version 11, November 2004. [keycodes] OpenJDK Community, "Java key-codes", . [theora] Xiph.org Foundation, "Theora". Boyaci & Schulzrinne Expires April 30, 2009 [Page 37] Internet-Draft RTP Payload Format for App Sharing October 2008 Authors' Addresses Omer Boyaci Columbia University Dept. of Computer Science 1214 Amsterdam Avenue New York, NY 10027 US Email: boyaci@cs.columbia.edu Henning Schulzrinne Columbia University Dept. of Computer Science 1214 Amsterdam Avenue New York, NY 10027 US Email: schulzrinne@cs.columbia.edu Boyaci & Schulzrinne Expires April 30, 2009 [Page 38] Internet-Draft RTP Payload Format for App Sharing October 2008 Full Copyright Statement Copyright (C) The IETF Trust (2008). This document is subject to the rights, licenses and restrictions contained in BCP 78, and except as set forth therein, the authors retain all their rights. This document and the information contained herein are provided on an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY, THE IETF TRUST AND THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Intellectual Property The IETF takes no position regarding the validity or scope of any Intellectual Property Rights or other rights that might be claimed to pertain to the implementation or use of the technology described in this document or the extent to which any license under such rights might or might not be available; nor does it represent that it has made any independent effort to identify any such rights. Information on the procedures with respect to rights in RFC documents can be found in BCP 78 and BCP 79. Copies of IPR disclosures made to the IETF Secretariat and any assurances of licenses to be made available, or the result of an attempt made to obtain a general license or permission for the use of such proprietary rights by implementers or users of this specification can be obtained from the IETF on-line IPR repository at http://www.ietf.org/ipr. The IETF invites any interested party to bring to its attention any copyrights, patents or patent applications, or other proprietary rights that may cover technology that may be required to implement this standard. Please address the information to the IETF at ietf-ipr@ietf.org. Boyaci & Schulzrinne Expires April 30, 2009 [Page 39]