Network File System Version 4 D. Noveck
Internet-Draft HPE
Intended status: Standards Track September 24, 2016
Expires: March 28, 2017

RPC-over-RDMA Extension to Manage Transport Properties


This document specifies an extension RPC-over-RDMA Version Two. The extension enables endpoints of an RPC-over-RDMA connection to exchange information which can be used to optimize message transfer.

Status of This Memo

This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.

Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at

Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."

This Internet-Draft will expire on March 28, 2017.

Copyright Notice

Copyright (c) 2016 IETF Trust and the persons identified as the document authors. All rights reserved.

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents ( in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License.

Table of Contents

1. Preliminaries

1.1. Requirements Language

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC2119].

1.2. Introduction

This document specifies an extension to RPC-over-RDMA Version Two. It allows each participating endpoint on a single connection to communicate various properties of its implementation, to request changes in properties of the other endpoint, and to notify the other endpoint of changes to properties made during operation.

The extension described herein specifies OPTIONAL message header types to implement this mechanism. The means by which the implementation support status of these OPTIONAL types is ascertained is described in [rpcrmdav2].

Although this document specifies the new OPTIONAL message header types to implement these functions, the precise means by which the presence of support for these OPTIONAL functions will be ascertained is not described here, as would be done more appropriately by the RFC defining a version of RPC-over-RDMA which supports protocol extension.

This document is currently written to conform to the extension model for RPC-over-RDMA Version Two as described in [rpcrdmav2].

1.3. Role Terminology

A number of different terms are used regarding the roles of the two participants in an RPC-over-RMA connection. Some of these roles are fixed for the duration of a connection while others vary from request to request or from message to message.

The roles of the client and server are fixed for the lifetime of the connection, with the client defined as the endpoint which initiated the connection.

The roles of requester and responder often parallel those of client and server, although this is not always the case. Most requests are made in the forward direction, in which the client is the requester and the server is the responder. However, backward-direction requests are possible, in which case the server is the requester and the client is the responder. As a result, clients and servers may both act as requesters and responders.

The roles of sender and receiver change from message to message. With regard to the types of messages described in this document, both the client and the server can act as sender and receiver. With regard to messages used to transfer RPC requests and replies, the requester sends requests and receives replies while the responder receives requests and sends replies.

2. Transport Properties

2.1. Property Model

A basic set of receiver and sender properties is specified in this document. An extensible approach is used, allowing new properties to be defined in future standards track documents.

Such properties are specified using:

typedef propid          uint32;

struct propval {
        propid          pv_which;
        opaque          pv_data<>;

typedef propval         propvalset<>;

typedef uint32          propvalsubset<>;


The following XDR types are used by operations that deal with transport properties:

A propid specifies a particular transport property. In order to allow easier XDR extension of the set of properties by concatenating XDR files, specific properties are defined as const values rather than as elements in an enum.

A propval specifies a value of a particular transport property with the particular property identified by pv_which, while the associated value of that property is contained within pv_data.

A pv_data which is of zero length is interpreted as indicating the default value or the property indicated by pv_which.

While pv_data is defined as opaque within the XDR, the contents are interpreted (except when of length zero) using the XDR typedef associated with the property specified by pv_which. The receiver of a message containing a propval MUST report an XDR error if the length of pv_data is such that it extends beyond the bounds of the message transferred.

In cases in which the propid specified by pv_which is understood by the receiver, the receiver also MUST report an XDR error if either of the following occur:

Note that no error is to be reported if pv_which is unknown to the receiver. In that case, that propval is not processed and processing continues using the next propval, if any.

A propvalset specifies a set of transport properties. No particular ordering of the propvals within it is imposed.

A propvalsubset identifies a subset of the properties in a previously specified propvalset. Each bit in the mask denotes a particular element in a previously specified propvalset. If a particular propval is at position N in the array, then bit number N mod 32 in word N div 32 specifies whether that particular propval is included in the defined subset. Words beyond the last one specified are treated as containing zero.

Propvalsubsets are useful in a number of contexts:

2.2. Transport Property Groups

Transport properties are divided into a number of groups

2.3. Operations Related to Transport Properties

There are a number of operations defined in Section 4 which are used to communicate and manage transport properties.

Prime among these is ROPT_CONNPROP (defined in Section 4.1 which serves as a means by which an endpoint's transport properties may be presented to its peer, typically upon establishing a connection.

In addition, there are a set of related operations concerned with requesting, effecting and reporting changes in transport properties:

Unlike many other operation types, the above are not used to effect transfer of RPC requests but are internal one-way information transfers. However, a ROPT_REQPROP and the corresponding ROPT_RESPROP do constitute an RPC-like remote call. The other operations are not part of a remote call transaction.

3. Basic Transport Properties

Although the set of transport properties is subject to later extension, a basic set of transport properties is defined below in Table 1.

In that table, the columns contain the following information:

property code XDR type default section
Receive Buffer Size 1 uint32 4096 3.1
Requester Remote Invalidation 2 bool false 3.2
Backward Request Support 3 enum bkreqsup BKREQSUP_INLINE 3.3

Note that this table does not provide any indication regarding whether a particular property can change or whether a change in the value may be requested (see Section 4.2). Such matters are not addressed by the protocol definition. An implementation may provide information about its readiness to make changed in a particular property using the opticonnpr_nochg field in the ROPT_CONNPROP message.

A partner implementation can always request a change but peers MAY reject a request to change a property for any reason. Implementations are always free to reject such requests if they cannot or do not wish to effect the requested change.

Either of the following will result in effective rejection requests to change specific properties:

With regard to unrequested changes in transport properties, it is the responsibility of the implementation making the change to do so in a fashion that which does not interfere with the other partner's continued correct operation (see Section 3.1).

3.1. Receive Buffer Size

The Receive Buffer Size specifies the minimum size, in octets, of pre-posted receive buffers. It is the responsibility of the participant sending this value to ensure that its pre-posted receives are at least the size specified, allowing the participant receiving this value to send messages that are of this size.


const uint32    PROP_RBSIZ = 1;
typedef uint32  prop_rbsiz;


The sender may use his knowledge of the receiver's buffer size to determine when the message to be sent will fit in the preposted receive buffers that the receiver has set up. In particular,

Because there may be pre-posted receives with buffer sizes that reflect earlier values of the buffer size property, changing this property poses special difficulties:

3.2. Requester Remote Invalidation

The Requester Remote Invalidation property indicates that the current endpoint, when in the role of a requester, is prepared for the responder to use RDMA Send With Invalidate when replying to an RPC-over-RDMA request containing non-empty chuck lists.

As RPC-over-RDMA is currently used, memory registrations exposed to peers are not established by the server and explicit RDMA operations are not done to satisfy backward direction requests. This makes it unlikely that servers will present non-default values of the PROP_REQREMINV property or that clients will take note of that value when presented by servers.


const uint32    PROP_REQREMINV = 2;
typedef bool    prop_reqreminv;


When the Requester Remote Invalidate property is set to false, a responder MUST use Send to convey RPC reply messages to the requester. When the Requester Remote Invalidate property is set to true, a responder MAY use Send With Invalidate instead of Send to convey RPC replies to the requester.

The value of the Requester Remote Invalidate property is not likely to change from the value reported by ROPT_INITPROP (see Section 4.2).

3.3. Backward Request Support

The value of this property is used to indicate a client implementation's readiness to accept and process messages that are part of backward-direction RPC requests.


enum bkreqsup {
        BKREQSUP_NONE    = 0,
        BKREQSUP_INLINE  = 1,
        BKREQSUP_GENL    = 2

const uint32    PROP_BRS = 3;
typedef bkreqsup prop_brs;                


Multiple levels of support are distinguished:

When information about this property is not provided, the support level of servers can be inferred from the backward- direction requests that they issue, assuming that issuing a request implicitly indicates support for receiving the corresponding reply. On this basis, support for receiving inline replies can be assumed when requests without read chunks, write chunks, or Reply chunks are issued, while requests with any of these elements allow the client to assume that general support for backward-direction replies is present on the server.

4. New Operations

The proposed new operations are set forth in Table 2 below. In that table, the columns contain the following information:

operation code XDR type msg section
Specify Properties at Connection 1 optinfo_connprop No 4.1
Request Property Modification 2 optinfo_reqprop No 4.2
Respond to Modification Request 3 optinfo_resprop No 4.3
Report Updated Properties 4 optinfo_updprop No 4.4

Support for all of the operations above is OPTIONAL. RPC-over-RDMA Version Two implementations that receive an operation that is not supported MUST respond with RDMA_ERROR message with an error code of RDMA_ERR_INVAL_OPTION as specified in [rpcrdmav2]

The only operation support requirements are as follows:

  • Implementations which send ROPT_REQPROP messages must support ROPT_RESPROP messages.
  • Implementations which support ROPT_RESPROP or ROPT_UPDPROP messages must also support ROPT_CONNPROP messages.

4.1. ROPT_CONNPROP: Specify Properties at Connection

The ROPT_CONNPROP message type allows an RPC-over-RDMA participant, whether client or server, to indicate to its partner relevant transport properties that the partner might need to be aware of.


const uint32     ROPT_CONNPROP= 1;

struct optinfo_connprop {
        propvalset      opticonnpr_start;
        propvalsubset   opticonnpr_nochg;

The message definition for this operation is as follows:

All relevant transport properties that the sender is aware of should be included in opticonnpr_start. Since support of this request is OPTIONAL, and since each of the properties is OPTIONAL as well, the sender cannot assume that the receiver will necessarily take note of these properties and so the sender should be prepared for cases in which the partner continues to assume that the default value for a particular property is still in effect.

Values of the subset of transport properties specified by opticonnpr_nochg is not expected to change during the lifetime of the connection.

Generally, a participant will send a ROPT_CONNPR message as the first message after a connection is established. Given that fact, the sender should make sure that the message can be received by partners who use the default Receive Buffer Size. The connection's initial receive buffer size is typically 1KB, but it depends on the initial connection state of the RPC-over-RDMA version in use. See [rpcrdmav2] for details.

Properties not included in opticonnpr_start are to be treated by the peer endpoint as having the default value and are not allowed to change subsequently. The peer should not request changes in such properties.

Those receiving an ROPT_CONNPR may encounter properties that they do not support or are unaware of. In such cases, these properties are simply ignored without any error response being generated.

4.2. ROPT_REQPROP: Request Modification of Properties

The ROPT_REQPROP message type allows an RPC-over-RDMA participant, whether client or server, to request of its partner that relevant transport properties be changed.

The rdma_xid field allows the request to be tied to a corresponding response of type ROPT_RESPROP (See Section 4.3.) In assigning the value of this field, the sender does not need to avoid conflict with xid's associated with RPC messages or with ROPT_REQPROP messages sent by the peer endpoint.

The partner need not change the properties as requested by the sender but if it does support the message type, it will generate a ROPT_RESPROP message, indicating the disposition of the request.


const uint32     ROPT_REQPROP = 2;

struct optinfo_reqprop {
       propvalset      optreqpr_want;

The message definition for this operation is as follows:

The propvalset optreqpr_want is a set of transport properties together with the desired values requested by the sender.

4.3. ROPT_RESPROP: Respond to Request to Modify Transport Properties

The ROPT_RESPROP message type allows an RPC-over-RDMA participant to respond to a request to change properties by its partner, indicating how the request was dealt with.


const uint32     ROPT_RESPROP = 3;

struct optinfo_resprop {
        propvalsubset   optrespr_done;
        propvalsubset   optrespr_rej;
        propvalset      optrespr_other;                        

The message definition for this operation is as follows:

The rdma_xid field of this message must match that used in the ROPT_REQPROP message to which this message is responding.

The optrespr_done field indicates which of the requested transport property changes have been effected as requested. For each such property, the receiver is entitled to conclude that the requested change has been made and that future transmissions may be made based on the new value.

The optrespr_rej field indicates which of the requested transport property changes have been rejected by the sender. This may be because of any of the following reasons:

  • The particular property specified is not known or supported by the receiver of the ROPT_REQPROP message.
  • The implementation receiving the ROPT_REQPROP message does not support modification of this property.
  • The implementation receiving the ROPT_REQPROP message has chosen to reject the modification for another reason.

The optrespr_other field contains new values for properties where a change is requested. The new value of the property is included and may be a value different from the original value in effect when the change was requested and from the requested value. This is useful when the new value of some property is not as large as requested but still different from the original value, indicating a partial satisfaction of the peer's property change request.

The sender MUST NOT include propvals within optrespr_other that are for properties other than the ones for which the corresponding property request has requested a change. If the receiver finds such a situation, it MUST ignore the erroneous propvals.

The subsets of properties specified by optrespr_done, optrespr_rej, and included in optrespr_other MUST NOT overlap, and when ored together, should cover the entire set of properties specified by optreqpr_want in the corresponding request. If the receiver finds such an overlap or mismatch, it SHOULD treat properties missing or within the overlap as having been rejected.

4.4. ROPT_UPDPROP: Update Transport Properties

The ROPT_UPDPROP message type allows an RPC-over-RDMA participant to notify the other participant that a change to the transport properties has occurred. This is because the sender has decided, independently, to modify one or more transport properties and is notifying the receiver of these changes.

const uint32     ROPT_UPDPROP = 4;

struct optinfo_updprop {
        propvalset        optupdpr_now;



The message definition for this operation is as follows:

optupdpr_now defines the new property values to be used.

5. XDR

This section contains an XDR [RFC4506] description of the proposed extension.


grep '^ *///' | sed 's?^ /// ??' | sed 's?^ *///$??'



This description is provided in a way that makes it simple to extract into ready-to-use form. The reader can apply the following shell script to this document to produce a machine-readable XDR description of extension which can be combined with XDR for the base protocol to produce an XDR that combines the base protocol with the optional extensions.


sh < ext.txt > charext.x



That is, if the above script is stored in a file called "" and this document is in a file called "ext.txt" then the reader can do the following to extract an XDR description file for this extension:

5.1. Code Component License


/// /*
///  * Copyright (c) 2010, 2016 IETF Trust and the persons
///  * identified as authors of the code.  All rights reserved.
///  *
///  * The author of the code is: D. Noveck.
///  *
///  * Redistribution and use in source and binary forms, with
///  * or without modification, are permitted provided that the
///  * following conditions are met:
///  *
///  * - Redistributions of source code must retain the above
///  *   copyright notice, this list of conditions and the
///  *   following disclaimer.
///  *
///  * - Redistributions in binary form must reproduce the above
///  *   copyright notice, this list of conditions and the
///  *   following disclaimer in the documentation and/or other
///  *   materials provided with the distribution.
///  *
///  * - Neither the name of Internet Society, IETF or IETF
///  *   Trust, nor the names of specific contributors, may be
///  *   used to endorse or promote products derived from this
///  *   software without specific prior written permission.
///  *
///  */



Code components extracted from this document must include the following license text. When the extracted XDR code is combined with other complementary XDR code which itself has an identical license, only a single copy of the license text need be preserved.

5.2. XDR Proper for Extension


/// * Core transport property types
/// */ 
///typedef propid          uint32;
///struct propval {
///        propid          pv_which;
///        opaque          pv_data<>;
///typedef propval         propvalset<>;
///typedef uint32          propvalsubset<>;
/// * Transport property codes for basic properties
/// */
///const uint32    PROP_RBSIZ = 1;
///const uint32    PROP_REQREMINV = 2;
///const uint32    PROP_BRS = 3;
/// * Other transport property types
/// */
///enum bkreqsup {
///        BKREQSUP_NONE    = 0,
///        BKREQSUP_INLINE  = 1,
///        BKREQSUP_GENL    = 2
/// * Transport property typedefs
/// */
///typedef uint32   prop_bsiz;
///typedef bool     prop_reqreminv;
///typedef bkreqsup prop_brs;                
/// * Optional operation codes
/// */
///const uint32     ROPT_CONNPROP = 1;
///const uint32     ROPT_REQPROP = 2;
///const uint32     ROPT_RESPRO = 3;
///const uint32     ROPT_UPDPROP = 4;
/// * Optional operation message structures
/// */
///struct optinfo_connprop {
///        propvalset      opticonnpr_start;
///        propvalsubset   opticonnpr_nochg;
///struct optinfo_reqprop {
///       propvalset      optreqpr_want;
///struct optinfo_resprop {
///        propvalsubset   optrespr_done;
///        propvalsubset   optrespr_rej;
///        propvalset      optrespr_other;                        
///struct optinfo_updprop {
///        propvalset      optupdpr_now;


6. Extensibility

6.1. Additional Properties

The set of transport properties is designed to be extensible. As a result, once new properties are defined in standards track documents, the operations defined in this document may reference these new transport properties, as well as the ones described in this document.

A standards track document defining a new transport property should include the following information paralleling that provided in this document for the transport properties defined herein.

  • The propid value used to identify this property.
  • The XDR typedef specifying the form in which the property value is communicated.
  • A description of the transport property that is communicated by the sender of ROPT_INITXCH and ROPT_UPDXCH and requested by the sender of ROP_REQXCH.
  • An explanation of how this knowledge could be used by the participant receiving this information.
  • Information giving rules governing possible changes of values of this property.

The definition of transport property structures is such as to make it easy to assign unique values. There is no requirement that a continuous set of values be used and implementations should not rely on all such values being small integers. A unique value should be selected when the defining document is first published as an internet draft. When the document becomes a standards track document working group should insure that:

  • The propids specified in the document do not conflict with those currently assigned or in use by other pending working group documents defining transport properties.
  • The propids specified in the document do not conflict with the range reserved for experimental use, as defined in Section 6.2.

Documents defining new properties fall into a number of categories.

  • Those defining new properties and explaining (only) how they affect use of existing message types.
  • Those defining new OPTIONAL message types and new properties applicable to the operation of those new message types.
  • Those defining new OPTIONAL message types and new properties applicable both to new and existing message types.

When additional transport properties are proposed, the review of the associated standards track document should deal with possible security issues raised by those new transport properties.

6.2. Experimental Properties

Given the design of the transport properties data structure, it possible to use the operations to implement experimental, possibly unpublished, transport properties.

propids in the range from 4,294,967,040 to 4,294,967,295 are reserved for experimental use and these values should not be assigned to new properties in standards track documents.

When values in this range are used there is no guarantee if successful interoperation among independent implementations.

7. Security Considerations

Like other fields that appear in each RPC-over-RDMA header, property information is sent in the clear on the fabric with no integrity protection, making it vulnerable to man-in-the-middle attacks.

For example, if a man-in-the-middle were to change the value of the Receive buffer size or the Requester Remote Invalidation boolean, it could reduce connection performance or trigger loss of connection. Repeated connection loss can impact performance or even prevent a new connection from being established. Recourse is to deploy on a private network or use link-layer encryption.

8. IANA Considerations

This document does not require any actions by IANA.

9. References

9.1. Normative References

[bidir] Lever, C., "Size-Limited Bi-directional Remote Procedure Call On Remote Direct Memory Access Transports", April 2016.

Work in progress.

[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997.
[RFC4506] Eisler, M., "XDR: External Data Representation Standard", STD 67, RFC 4506, DOI 10.17487/RFC4506, May 2006.
[rfc5666bis] Lever, C., Simpson, W. and T. Talpey, "Remote Direct Memory Access Transport for Remote Procedure Call", May 2016.

Work in progress.

[rpcrdmav2] Lever, C. and D. Noveck, "RPC-over-RDMA Version Two", June 2016.

Work in progress.

9.2. Informative References

[RFC5662] Shepler, S., Eisler, M. and D. Noveck, "Network File System (NFS) Version 4 Minor Version 1 External Data Representation Standard (XDR) Description", RFC 5662, DOI 10.17487/RFC5662, January 2010.
[RFC5666] Talpey, T. and B. Callaghan, "Remote Direct Memory Access Transport for Remote Procedure Call", RFC 5666, DOI 10.17487/RFC5666, January 2010.

Appendix A. Acknowledgments

The author gratefully acknowledges the work of Brent Callaghan and Tom Talpey producing the original RPC-over-RDMA Version One specification [RFC5666] and also Tom's work in helping to clarify that specification.

The author also wishes to thank Chuck Lever for his work resurrecting NFS support for RDMA in [rfc5666bis] and for his helpful review of and suggestions for this document.

The shell script and formatting conventions were first described by the authors of the NFSv4.1 XDR specification [RFC5662].

Author's Address

David Noveck Hewlett Packard Enterprise 165 Dascomb Road Andover, MA 01810 USA Phone: +1 781-572-8038 EMail: