Internet-Draft Stephen Bailey (Sandburst) Expires: May 2003 Tom Talpey (NetApp) The Architecture of Direct Data Placement (DDP) And Remote Direct Memory Access (RDMA) On Internet Protocols draft-bailey-roi-ddp-rdma-arch-01 Status of this Memo This document is an Internet-Draft and is in full conformance with all provisions of Section 10 of RFC2026. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet- Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. Copyright Notice Copyright (C) The Internet Society (2002). All Rights Reserved. Abstract This document defines an abstract architecture for Direct Data Placement (DDP) and Remote Direct Memory Access (RDMA) protocols to run on Internet Protocol-suite transports. This architecture does not necessarily reflect the proper way to implement such protocols, but is, rather, a descriptive tool for defining and understanding the protocols. Bailey & Talpey Expires May 2003 [Page 1] Internet-Draft DDP & RDMA Architecture November 2002 Table Of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . 2 2. Architecture . . . . . . . . . . . . . . . . . . . . . . 3 2.1. Direct Data Placement (DDP) Protocol Architecture . . . 3 2.1.1. Transport Operations . . . . . . . . . . . . . . . . . . 5 2.1.2. DDP Operations . . . . . . . . . . . . . . . . . . . . . 6 2.1.3. Transport Characteristics in DDP . . . . . . . . . . . . 9 2.2. Remote Direct Memory Access Protocol Architecture . . . 10 2.2.1. RDMA Operations . . . . . . . . . . . . . . . . . . . . 11 2.2.2. Transport Characteristics in RDMA . . . . . . . . . . . 14 3. Security Considerations . . . . . . . . . . . . . . . . 14 4. IANA Considerations . . . . . . . . . . . . . . . . . . 15 5. Acknowledgements . . . . . . . . . . . . . . . . . . . . 15 References . . . . . . . . . . . . . . . . . . . . . . . 15 Authors' Addresses . . . . . . . . . . . . . . . . . . . 16 Full Copyright Statement . . . . . . . . . . . . . . . . 17 1. Introduction This document defines an abstract architecture for Direct Data Placement (DDP) and Remote Direct Memory Access (RDMA) protocols to run on Internet Protocol-suite transports [RDDP, ROM]. This architecture does not necessarily reflect the proper way to implement such protocols, but is, rather, a descriptive tool for defining and understanding the protocols. The first part of the document describes the architecture of DDP protocols, including what assumptions are made about the transports on which DDP is built. The second part describes the architecture of RDMA protocols layered on top of DDP. Before introducing the protocols, three definitions will be useful to guide discussion: o Placement - writing to a data buffer. o Delivery - informing the Upper Layer Protocol (ULP) (e.g. RDMA) that a particular message is available for use. Delivery therefore may be viewed as the "control" signal associated with a unit of data. Note that the order of delivery is defined more strictly than it is for placement. o Completion - informing the ULP or application that a particular RDMA operation has finished. A completion, for instance, may require the delivery of several messages, or it may also reflect that some local processing has finished. Bailey & Talpey Expires May 2003 [Page 2] Internet-Draft DDP & RDMA Architecture November 2002 The goal of the DDP protocol is to allow the efficient placement of data into buffers designated by Upper Layer Protocols (e.g. RDMA). This is described in detail in [ROM]. Efficiency may be characterized by the minimization of the number of transfers of the data over the receiver's system buses. The goal of the RDMA protocol is to provide the semantics to enable Remote Direct Memory Access between peers in a way consistent with application requirements. The RDMA protocol provides facilities immediately useful to existing and future networking, storage, and other application protocols. [DAFS, FIBRE, IB, MYR, SDP, SRVNET, VI] The DDP and RDMA protocols work together to achieve their respective goals. RDMA provides facilities to a ULP for identifying buffers, controlling the transfer of data between ULP peers, and providing completion notifications to the ULP. RDMA uses the features of DDP to steer payloads to specific buffers at the Data Sink. ULPs that do not require the features of RDMA may be layered directly on top of DDP. The DDP and RDMA protocols are transport independent. The following figure shows the relationship between RDMA, DDP, Upper Layer Protocols and Transport. +---------------------------------------------------+ | ULP | +---------+------------+----------------------------+ | | | RDMA | | | +----------------------------+ | | DDP | | +-----------------------------------------+ | Transport | +---------------------------------------------------+ 2. Architecture The Architecture section is presented in two parts: Direct Data Placement Protocol architecture and Remote Direct Memory Access Protocol architecture. 2.1. Direct Data Placement (DDP) Protocol Architecture The central idea of general-purpose DDP is that a data sender will supplement the data it sends with placement information that allows the receiver's network interface to place the data directly at its final destination without any copying. DDP can be used to steer received data to its final destination, without requiring layer- Bailey & Talpey Expires May 2003 [Page 3] Internet-Draft DDP & RDMA Architecture November 2002 specific behavior for each different layer. Data sent with such DDP information is said to be `tagged'. The central component of the DDP architecture is the `buffer', which is an object with beginning and ending addresses, and a method (set()) to set the value of an octet at an address. In many cases, a buffer corresponds directly to a portion of host user memory. However, DDP does not depend on this---a buffer could be a disk file, or anything else that can be viewed as an addressable collection of octets. Abstractly, a buffer provides the interface: typedef struct { const address_t start; const address_t end; void set(address_t a, data_t v); } ddp_buffer_t; address_t a reference to local memory data_t an octet data value. The protocol layering and in-line data flow of DDP is: Client Protocol (e.g. ULP or RDMA) | ^ untagged messages | | untagged message delivery tagged messages | | tagged message delivery v | DDP+---> data placement ^ | transport messages v Transport (e.g. SCTP, DCP, framed TCP) ^ | IP datagrams v . . . In addition to in-line data flow, the client protocol registers buffers with DDP, and DDP performs buffer update (set()) operations Bailey & Talpey Expires May 2003 [Page 4] Internet-Draft DDP & RDMA Architecture November 2002 as a result of receiving tagged messages. DDP messages may be split into multiple, smaller DDP messages, each in a separate transport message. However, if the transport is unreliable or unordered, messages split across transport messages may or may not provide useful behavior, in the same way as splitting arbitrary upper layer messages across unreliable or unordered transport messages may or may not provide useful behavior. In other words, the same considerations apply to building client protocols on different types of transports with or without the use of DDP. A DDP message split across transport messages looks like: DDP message: Transport messages: stag=s, offset=o, message 1: notify=y, id=i |type=ddp | message= |stag=s | |aabbccddee|-------. |offset=o | ~ ... ~----. \ |notify=n | |vvwwxxyyzz|-. \ \ |id=? | | \ `--->|aabbccddee| | \ ~ ... ~ | +----->|iijjkkllmm| | | + | message 2: \ | |type=ddp | \ | |stag=s | \ + |offset=o+n| \ \ |notify=y | \ \ |id=i | \ `-->|nnooppqqrr| \ ~ ... ~ `---->|vvwwxxyyzz| Although this picture suggests that DDP information is carried in- line with the message payload, components of the DDP information may also be in transport-specific fields, or derived from transport-specific control information if the transport permits. 2.1.1. Transport Operations For the purposes of this architecture, the transport provides: Bailey & Talpey Expires May 2003 [Page 5] Internet-Draft DDP & RDMA Architecture November 2002 void xpt_send(socket_t s, message_t m); message_t xpt_recv(socket_t s); msize_t xpt_max_msize(socket_t s); socket_t a transport address, including IP addresses, ports and other transport-specific identifiers. message_t a string of octets. msize_t (scalar) a message size. xpt_send(socket_t s, message_t m) send a transport message. xpt_recv(socket_t s) receive a transport message. xpt_max_msize(socket_t s) get the current maximum transport message size. Corresponds, roughly, to the current path Maximum Transfer Unit (PMTU), adjusted by underlying protocol overheads. Real implementations of xpt_send() and xpt_recv() typically return error indications, but that is not relevant to this architecture. 2.1.2. DDP Operations The DDP layer provides: void ddp_send(socket_t s, message_t m); void ddp_send_ddp(socket_t s, message_t m, ddp_addr_t d, ddp_notify_t n); ddp_recv_t ddp_recv(socket_t s); bdesc_t ddp_register(socket_t s, ddp_buffer_t b); void ddp_deregister(bhand_t bh); msizes_t ddp_max_msizes(socket_t s); Bailey & Talpey Expires May 2003 [Page 6] Internet-Draft DDP & RDMA Architecture November 2002 ddp_addr_t the buffer address portion of a tagged message: typedef struct { stag_t stag; address_t offset; } ddp_addr_t; stag_t (scalar) a Steering Tag. A stag_t identifies the destination buffer for tagged messages. stag_ts are generated when the buffer is registered, communicated to the sender by some client protocol convention and inserted in DDP messages. stag_t values in this DDP architecture are assumed to be completely opaque to the client protocol, and implementation-dependent. However, particular implementations, such as DDP on a multicast transport (see below), may provide the buffer holder some control in selecting stag_ts. ddp_notify_t the notification portion of a DDP message, used to signal that the message represents the final fragment of a multi-segmented DDP message: typedef struct { boolean_t notify; ddp_msg_id_t i; } ddp_notify_t; ddp_msg_id_t (scalar) a DDP message identifier. msg_id_ts are chosen by the DDP message receiver (buffer holder), communicated to the sender by some client protocol convention and inserted in DDP messages. Whether a message reception indication is requested for a DDP message is a matter of client protocol convention. Unlike stag_ts, the structure of msg_id_ts is opaque to DDP, and therefore, completely in the hands of the client protocol. bdesc_t a description of a registered buffer: Bailey & Talpey Expires May 2003 [Page 7] Internet-Draft DDP & RDMA Architecture November 2002 typedef struct { bhand_t bh; ddp_addr_t a; } bdesc_t; `a.offset' is the starting offset of the registered buffer, which may have no relationship to the `start' or `end' addresses of that buffer. However, particular implementations, such as DDP on a multicast transport (see below), may allow some client protocol control over the starting offset. bhand_t an opaque buffer handle used to deregister a buffer. ddp_recv_t an untagged message, a tagged message reception indication, or a tagged message reception error: typedef union { message_t m; ddp_msg_id_t i; ddp_err_t e; } ddp_recv_t; ddp_err_t indicates an error while receiving a tagged message, typically `offset' out of bounds, or `stag' is not registered to the socket. msizes_t The maximum untagged and tagged messages that fit in a single transport message: typedef struct { msize_t max_untagged; msize_t max_tagged; } msizes_t; ddp_send(socket_t s, message_t m) Bailey & Talpey Expires May 2003 [Page 8] Internet-Draft DDP & RDMA Architecture November 2002 send an untagged message. ddp_send_ddp(socket_t s, message_t m, ddp_addr_t d, ddp_notify_t n) send a tagged message. ddp_recv(socket_t s) get the next received untagged message, tagged message reception indication, or tagged message error. ddp_register(socket_t s, ddp_buffer_t b) register a buffer for DDP on a socket. The same buffer may be registered multiple times on the same or different sockets. Different buffers may also refer to portions of the same underlying addressable object (buffer aliasing). ddp_deregister(bhand_t bh) remove a registration from a buffer. ddp_max_msizes(socket_t s) get the current maximum untagged and tagged message sizes that will fit in a single transport message. 2.1.3. Transport Characteristics In DDP Certain characteristics of the transport on which DDP is mapped determine the nature of the service provided to client protocols. Specifically, transports are: o reliable or unreliable, o ordered or unordered, o single source or multisource, o single destination or multidestination (multicast or anycast). Some transports support several combinations of these characteristics. For example, SCTP [SCTP] is reliable, single source, single destination (point-to-point) and supports both ordered and unordered modes. In general, these transport characteristics equally affect transport and DDP message delivery. However, there are several Bailey & Talpey Expires May 2003 [Page 9] Internet-Draft DDP & RDMA Architecture November 2002 issues specific to DDP messages. A key component of DDP is how the following operations on the receiving side are ordered among themselves, and how they relate to corresponding operations on the sending side: o set()s, o untagged message reception indications, and o tagged message reception indications. These relationships depend upon the characteristics of the underlying transport in a way which is defined by the DDP protocol. For example, if the transport is unreliable and unordered, the DDP protocol might specify that the client protocol is subject to the consequences of transport messages being lost or duplicated, rather requiring different characteristics be presented to the client protocol. Multidestination data delivery is the other transport characteristic which may require specific consideration in a DDP protocol. As mentioned above, the basic DDP model assumes that buffer address values returned by ddp_register() are opaque to the client protocol, and can be implementation dependent. The most natural way to map DDP to a multidestination transport is to require all receivers produce the same buffer address when registering a multidestination destination buffer. Restriction of the DDP model to accommodate multiple destinations involves engineering tradeoffs comparable to those of providing non-DDP multidestination transport capability. 2.2. Remote Direct Memory Access (RDMA) Protocol Architecture Remote Direct Memory Access (RDMA) extends the capabilities of DDP with the ability to read from buffers registered to a socket (RDMA Read). This allows a client protocol to perform arbitrary, bidirectional data movement without involving the remote client. When RDMA is implemented in hardware, arbitrary data movement can be performed without involving the remote host CPU at all. In addition, RDMA protocols usually specify a transport-independent untagged message service (Send) with characteristics which are both very efficient to implement in hardware, and convenient for client protocols. The RDMA architecture is patterned after the traditional model for device programming, where the client requests an operation using Bailey & Talpey Expires May 2003 [Page 10] Internet-Draft DDP & RDMA Architecture November 2002 Send-like actions (programmed I/O), the server performs the necessary data transfers for the operation (DMA reads and writes), and notifies the client of completion. The programmed I/O+DMA model efficiently supports a high degree of concurrency and flexibility for both the client and server, even when operations have a wide range of intrinsic latencies. RDMA is layered as a client protocol on top of DDP: Client Protocol | ^ Sends | | Send reception indications RDMA Read Requests | | RDMA Read Completion indications RDMA Writes | | RDMA Write Completion indications v | RDMA | ^ untagged messages | | untagged message delivery tagged messages | | tagged message delivery v | DDP+---> data placement ^ | transport messages v . . . In addition to in-line data flow, read (get()) and update (set()) operations are performed on buffers registered with RDMA as a result of RDMA Read Requests and RDMA Writes, respectively. An RDMA `buffer' extends a DDP buffer with a get() operation that retrieves the value of the octet at address `a': typedef struct { const address_t start; const address_t end; void set(address_t a, data_t v); data_t get(address_t a); } rdma_buffer_t; 2.2.1. RDMA Operations The RDMA layer provides: Bailey & Talpey Expires May 2003 [Page 11] Internet-Draft DDP & RDMA Architecture November 2002 void rdma_send(socket_t s, message_t m); void rdma_write(socket_t s, message_t m, ddp_addr_t d, rdma_notify_t n); void rdma_read(socket_t s, ddp_addr_t s, ddp_addr_t d); rdma_recv_t rdma_recv(socket_t s); bdesc_t rdma_register(socket_t s, rdma_buffer_t b, bmode_t mode); void rdma_deregister(bhand_t bh); msizes_t rdma_max_msizes(socket_t s); Although, for clarity, these data transfer interfaces are synchronous, rdma_read() and possibly rdma_send() (in the presence of Send flow control), can require an arbitrary amount of time to complete. To express the full concurrency and interleaving of RDMA data transfer, these interfaces are also defined to be multithreaded. For example, a client protocol may perform an rdma_send(), while an rdma_read() operation is in progress. rdma_notify_t RDMA Write notification information, used to signal that the message represents the final fragment of a multi-segmented RDMA message: typedef struct { boolean_t notify; rdma_write_id_t i; } rdma_notify_t; identical in function to ddp_notify_t, except that the type rdma_write_id_t may not be equivalent to ddp_msg_id_t. rdma_write_id_t (scalar) an RDMA Write identifier. rdma_recv_t a Send message, an RDMA Write completion identifier, or an RDMA error: typedef union { message_t m; rdma_write_id_t i; rdma_err_t e; } rdma_recv_t; Bailey & Talpey Expires May 2003 [Page 12] Internet-Draft DDP & RDMA Architecture November 2002 rdma_err_t an RDMA protocol error indication. RDMA errors include buffer addressing errors corresponding to ddp_err_ts, and buffer protection violations (e.g. RDMA Writing a buffer only registered for reading). bmode_t buffer registration mode (permissions). Any combination of permitting RDMA Read (BMODE_READ) and RDMA Write (BMODE_WRITE) operations. rdma_send(socket_t s, message_t m) send a message, delivering it to the next untagged RDMA buffer at the remote peer. rdma_write(socket_t s, message_t m, ddp_addr_t d, rdma_notify_t n) RDMA Write to remote buffer address d. rdma_read(socket_t s, ddp_addr_t s, ddp_addr_t d) RDMA Read from remote buffer address s to local buffer address d. rdma_recv(socket_t s); get the next received Send message, RDMA Write completion identifier, or RDMA error. rdma_register(socket_t s, rdma_buffer_t b, bmode_t mode) register a buffer for RDMA on a socket (for read access, write access or both). As with DDP, the same buffer may be registered multiple times on the same or different sockets, and different buffers may refer to portions of the same underlying addressable object. rdma_deregister(bhand_t bh) remove a registration from a buffer. rdma_max_msizes(socket_t s) get the current maximum Send (max_untagged) and RDMA Read or Write (max_tagged) operations that will fit in a single Bailey & Talpey Expires May 2003 [Page 13] Internet-Draft DDP & RDMA Architecture November 2002 transport message. The values returned by rdma_max_msizes() are closely related to the values returned by ddp_max_msizes(), but may not be equal. 2.2.2. Transport Characteristics In RDMA As with DDP, RDMA can be used on transports with a variety of different characteristics that manifest themselves directly in the service provided by RDMA. Like DDP, an RDMA protocol must specify how: o set()s, o get()s, o Send messages, and o RDMA Read completions are ordered among themselves and how they relate to corresponding operations on the remote peer(s). These relationships are likely to be a function of the underlying transport characteristics. There are some additional characteristics of RDMA which may translate poorly to unreliable or multipoint transports due to attendant complexities in managing endpoint state: o Send flow control o RDMA Read These difficulties can be overcome by placing restrictions on the service provided by RDMA. However, many RDMA clients, especially those that separate data transfer and application logic concerns, are likely to depend upon capabilities only provided by RDMA on a point-to-point, reliable transport. 3. Security Considerations System integrity must be maintained in any RDMA solution. Mechanisms must be specified to prevent RDMA or DDP operations from impairing system integrity. For example, the threat caused by potential buffer overflow needs full examination, and prevention mechanisms must be spelled out. Because a Steering Tag exports access to a memory region, one critical aspect of security is the scope of this access. It must Bailey & Talpey Expires May 2003 [Page 14] Internet-Draft DDP & RDMA Architecture November 2002 be possible to individually control specific attributes of the access provided by a Steering Tag, including remote read access, remote write access, and others that might be identified. A specification must provide both implementation requirements relevant to this issue, and guidelines to assist implementors in making the appropriate design decisions. A number of other potential attacks have been envisioned and must be addressed. Some such examples are outlined in [RDMACON]. Resource issues leading to denial-of-service attacks, overwrites and other concurrent operations, the ordering of completions as required by the RDMA protocol, and the granularity of transfer are all within the required scope of any security analysis of RDMA and DDP. 4. IANA Considerations IANA considerations are not addressed in by this document. Any IANA considerations resulting from the use of DDP or RDMA must be addressed in the relevant standards. 5. Acknowledgements The authors wish to acknowledge the valuable contributions of David Black, Jeff Mogul and Allyn Romanow. 6. References [DAFS] Direct Access File System http://www.dafscollaborative.org http://www.ietf.org/internet-drafts/draft-wittle-dafs-00.txt [FIBRE] Fibre Channel Standard http://www.fibrechannel.com/technology/index.master.html [IB] InfiniBand Architecture Specification, Volumes 1 and 2, Release 1.0.a. http://www.infinibandta.org [MYR] Myrinet, http://www.myricom.com [RDDP] Remote Direct Data Placement Working Group charter, http://www.ietf.org/html.charters/rddp-charter.html [RDMACON] D. Black, M. Speer, J. Wroclawski, "DDP and RDMA Concerns", Bailey & Talpey Expires May 2003 [Page 15] Internet-Draft DDP & RDMA Architecture November 2002 http://www.ietf.org/internet-drafts/draft-black-rdma- concerns-00.txt, Work in Progress, June 2002 [ROM] A. Romanow, J. Mogul, T. Talpey, S. Bailey, "RDMA over IP Problem Statement", http://www.ietf.org/internet-drafts/draft- romanow-rdma-over-ip-problem-statement-01.txt, Work in Progress, November 2002 [SCTP] R. Stewart et al., "Stream Transmission Control Protocol", Standards Track RFC, http://www.ietf.org/rfc/rfc2960 [SDP] Sockets Direct Protocol v1.0 [SRVNET] Compaq Servernet, http://nonstop.compaq.com/view.asp?PAGE=ServerNet [VI] Virtual Interface Architecture Specification Version 1.0. http://www.viarch.org/html/collateral/san_10.pdf Authors' Addresses Stephen Bailey Sandburst Corporation 600 Federal Street Andover, MA 01810 USA USA Phone: +1 978 689 1614 Email: steph@sandburst.com Tom Talpey Network Appliance 375 Totten Pond Road Waltham, MA 02451 USA Phone: +1 781 768 5329 Email: thomas.talpey@netapp.com Bailey & Talpey Expires May 2003 [Page 16] Internet-Draft DDP & RDMA Architecture November 2002 Full Copyright Statement Copyright (C) The Internet Society (2002). All Rights Reserved. This document and translations of it may be copied and furnished to others, and derivative works that comment on or otherwise explain it or assist in its implementation may be prepared, copied, published and distributed, in whole or in part, without restriction of any kind, provided that the above copyright notice and this paragraph are included on all such copies and derivative works. However, this document itself may not be modified in any way, such as by removing the copyright notice or references to the Internet Society or other Internet organizations, except as needed for the purpose of developing Internet standards in which case the procedures for copyrights defined in the Internet Standards process must be followed, or as required to translate it into languages other than English. The limited permissions granted above are perpetual and will not be revoked by the Internet Society or its successors or assigns. This document and the information contained herein is provided on an "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Bailey & Talpey Expires May 2003 [Page 17]