Network File System Version 4 D. Noveck
Internet-Draft HPE
Intended status: Informational September 3, 2016
Expires: March 7, 2017

Issues Related to RPC-over-RDMA Internode Round Trips


As currently designed and implemented, the RPC-over-RDMA protocol requires use of multiple internode round trips to process some common operations. For example, NFS WRITE operations require use of three internode round trips. This document looks at this issue and discusses what can and what should be done to address it, both within the context of an extensible version of RPC-over-RDMA and potentially outside that framework.

Status of This Memo

This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.

Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at

Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."

This Internet-Draft will expire on March 7, 2017.

Copyright Notice

Copyright (c) 2016 IETF Trust and the persons identified as the document authors. All rights reserved.

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents ( in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License.

Table of Contents

1. Preliminaries

1.1. Requirements Language

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC2119].

1.2. Introduction

When many common operations are performed using RPC-over-RDMA, additional inter-node round-trip latencies are required to take advantage of the performance benefits provided by RDMA Functionality.

While the latencies involved are generally small, they are a reason for concern for two reasons.

Given this background, round trips beyond the minimum necessary need to be justified by corresponding benefits. If they are not, work needs to be done to eliminate those excess round trips.

We are going to look at the existing situation with regard to round-trip latency and make some suggestions as to how the issue might be best addressed. We will consider things that could be done in the near future and also explore further possibilities that would require a longer-term approach to be adopted.

2. Review of the Current Situation

2.1. Troublesome Requests

We will be looking at four sorts of situations:

These are all simple examples of situations in which explicit RDMA operations are used, either to effect Direct Data Placement or to respond to message size limits that derive from a limited receive buffer size.

We will survey the resulting latency and overhead issues in an RPC-over-RDMA Version One environment in Sections 2.2 and 2.3 below.

2.2. WRITE Request Processing Details

We'll start with the case of a request involving direct placement of request data. In this case, an RDMA READ is used to transfer a DDR-eligible data item (e.g. the data to be written) from its location in requester memory, to a location selected by the responder.

Processing proceeds as described below. Although we are focused on internode latency, the time to perform a request also includes such things as interrupt latency, overhead involved in interacting with the RNIC, and the time for the server to execute the requested operation.

To summarize, if we exclude the actual server execution of the request, the latency consists of two internode round-trip latencies plus two-responder-side interrupt latencies plus one requester-side interrupt latency plus any necessary registration/de-registration overhead. This is in contrast to a request not using explicit RDMA operations in which there is a single inter-node round-trip latency and one interrupt latency on the requester and the responder.

The processing of the other sorts of requests mentioned in Section 2.1 show both similarities and differences:

2.3. READ Request Processing Details

We'll now discuss the case of a request involving direct placement of response data. In this case, an RDMA WRITE is used to transfer a DDR-eligible data item (e.g. the data being read) from its location in responder memory, to a location previously selected by the requester.

Processing proceeds as described below. Although we are focused on internode latency, the time to perform a request also includes such things as interrupt latency, overhead involved in interacting with the RNIC, and the time for the server to execute the requested operation.

To summarize, in this case the additional latency that we saw in Section 2.2 does not arise. Except for the additional overhead due to memory registration and invalidation, the situation is the same as for a request not using explicit RDMA operations in which there is a single inter-node round-trip latency and one interrupt latency on the requester and the responder.

3. Near-term Work

We are going to consider how the latency and overhead issues discussed in Section 2 might be addressed in the context of an extensible version of RPC-over-RDMA, such as that proposed in [rpcrdmav2].

In Section 3.1, we will establish a performance target for the troublesome requests, based on the performance of requests that do not involve long messages or direct data placement.

We will then consider how extensions might be defined to bring latency and overhead for the requests discussed in Section 2.1 into line with those for other requests. There will be two specific classes of requests to address:

The optional features to deal with each of the classes of messages discussed above could be implemented separately. However, in the handling of RPCs with very large amounts of bulk data, the two features are synergistic. This fact makes it desirable to define the features as part of the same extension. See Sections 3.4 and 3.5 for details.

3.1. Target Performance

As our target, we will look at the latency and overhead associated with other sorts of RPC requests, i.e. those that do not use DDP, and that have request and response messages which do fit within the receive buffer limit.

Processing proceeds as follows:

In this case, there is only a single internode round-trip latency necessary to effect the RPC. Total request latency includes this round-trip latency plus interrupt latency on the requester and responder, plus the time for the responder to actually perform the requested operation.

Thus the delta between the operations discussed in Section 2 and our baseline consists of two portions, one of which applies to all the requests we are concerned with and the second of which only applies to request which involve use of RDMA READ, as discussed in Section 2.2. The latter category consists of:

The additional overhead necessary to do memory registration and deregistration applies to all requests using explicit RDMA operations. The costs will vary with implementation characteristics. As a result, in some implementations, it may desirable to replace use of RDMA Write with send-based alternatives, while in others, use of RDMA Write may be preferable.

3.2. Message Continuation

Using multiple RPC-over-RDMA transmissions, in sequence, to send a single RPC message avoids the additional latency and overhead associated with the use of explicit RDMA operations to transfer position-zero read chunks. In the case of reply chunks, only overhead is reduced.

Although transfer of a single request or reply in N transmissions will involve N+1 internode latencies, overall request latency is not increased by requiring that operations involving multiple nodes be serialized. Generally, these transmissions are pipelined.

As an illustration, let's consider the case of a request involving a response consisting of two RPC-over-RDMA transmissions. Even though each of these transmissions is acknowledged, that acknowledgement does not contribute to request latency. The second transmission can be received by the requester and acted upon without waiting for either acknowledgment.

This situation would require multiple receive-side interrupts but it is unlikely to result in extended interrupt latency. With 1K sends (Version One), the second receive will complete about 200 nanoseconds after the first assuming a 40Gb/s transmission rate. Given likely interrupt latencies, the first interrupt routine would be able to note that the completion of the second receive had already occurred.

3.3. Send-based DDP

In order to effect proper placement of request or reply data within the context of individual RPC-over-RDMA transmissions, receive buffers need to be structured to accommodate this function

To illustrate the considerations that could lead clients and servers to choose particular buffer structures, we will use, as examples, the cases of NFS READs and WRITEs of 8K data blocks (or the corresponding NFSv4 COMPOUNDs).

In such cases, the client and server need to have the DDP-eligible bulk data placed in appropriately aligned 8K buffer segments. Rather than being transferred in separate transmissions using explicit RDMA operations, a message can be sent so that bulk data is received into an appropriate buffer segment. In this case, it would be excised from the XDR payload stream, just as it is in the case of existing DDP facilities.

Consider a server expecting write requests which are usually X bytes long or less, exclusive of an 8K bulk data area. In this case the payload stream will most likely be less than X bytes and will fit in a buffer segment devoted to that purpose. The bulk data needs to be placed in the subsequent buffer segment in order to align it properly, i.e. with the appropriate alignment, in the DDP target buffer. In order to place the data appropriately, the sender (in this case, the client) needs to add padding of length X-Y bytes where Y is the length of payload stream for the current request. The case of reads is exactly the same except that the sender adding the padding is the server.

To provide send-based DDP as an RPC-over-RDMA extension, the framework defined in [xcharext] could be used. A new "transport characteristic" could be defined which allowed a participant to expose the structure of his receive buffers and to identify the buffer segments capable of being used as DDP targets. In addition, a new optional message header would have to be defined. It would be defined to provide:

3.4. Feature Synergy

While message continuation and send-based DDP each address an important class of commonly used messages, their combination allows simpler handling of some important classes of messages:

To accommodate these situations, it would be best to have the definition of the headers to support message continuation interact with data structures to support send-based DDP as follows:

A potential extension supporting these features interacting as described above can be found in [rtrext].

3.5. Feature Selection and Negotiation

Given that an appropriate extension is likely to support multiple OPTIONAL features, special attention will have to be given to defining how implementations which might not support the same subset of OPTIONAL features can successfully interact. The goal is to allow interacting implementations to get the benefit of features that they both support, while allowing implementation pairs that do not share support for any of the OPTIONAL features to operate just as base Version Two implementations could do in the absence of the potential extension.

It is helpful if each implementation provides characteristics defining its level of feature support which the peer implementation can test before attempting to use a particular feature. In other similar contexts, the support level concerns the implementation in its role as responder, i.e. whether it is prepared to execute a given request. In the case of the potential extension discussed here, most characteristics concern an implementation in its role as receiver. One might define characteristics which indicate,

Use of such characteristics might allow asymmetric implementations. For example, a client might send requests containing DDP-eligible data items using send-based DDP without being able to accept messages containing data items using send-based DDP. That is a likely implementation pattern, given the greater performance benefits of avoiding use of RDMA Read.

Further useful characteristics would apply to the implementation in its role of responder. For instance,

Because of the potentially different needs of operations in the forward and backward directions, it may be desirable to separate the receiver-based characteristics according the direction of operation that they apply to.

A further issue relates to the role of explicit RDMA operations in connection with backwards operation. Although, no current protocols require support for DDP or transfer of large messages when operating in the backward direction, the protocol is designed to allow such support to be developed in the future. Since the protocol, with the extension discussed here is likely to have multiple methods of providing these functions, we have a number of possible choices regarding the role of chunk-based methods of providing these functions

4. Possible Future Development

Although the reduction of explicit RDMA operation reduces the number of inter-node round trips and eliminates sequences of operations in which multiple round-trip latencies are serialized with server interrupt latencies, the use of connected operations means that round-trip latencies will always be present, since each message is acknowledged.

One avenue that has been considered is use of unreliable-datagram (UD) transmission in environments where the "unreliable" transmission is sufficiently reliable that RPC replay can deal with a very low rate of message loss. For example, UD in Infiniband specifies a low enough rate of frame loss to make this a viable approach, particularly for use in supporting protocols such as NFSv4.1, that contain their own facilities to ensure exactly-once semantics.

With this sort of arrangement, request latency is still the same. However, since the acknowledgements are not serving any substantial function, it is tempting to consider removing them, as they do take up some transmission bandwidth, that might be used otherwise, if the protocol were to reach the goal of effectively using the underlying medium.

The size of such wasted transmission bandwidth depends on the average message size and many implementation considerations regarding how acknowledgments are done. In any case, given expected message sizes, the wasted transmission bandwidth will be very small.

When RPC messages are quite small, acknowledgments may be of concern. However, in that situation, a better response would be transfer multiple RPC messages within a single RPC-over-RDMA transmission.

When multiple RPC messages are combined into a single transmission, the overhead of interfacing with the RNIC, particularly the interrupt handling overhead, is amortized over multiple RPC messages.

Although this technique is quite outside the spirit of existing RPC-over-RDMA implementations, it appears possible to define new header types capable of supporting this sort of transmission, using the extension framework described in [rpcrdmav2].

5. Summary

We've examined the issue of round-trip latency and concluded:

As it seems that the features sketched out could put internode latencies and overhead for a large class of requests back to the baseline value for the RPC paradigm, more detailed definition of the required extension functionality is in order.

We've also looked at round trips at the physical level, in that acknowledgments are sent in circumstances where there is no obvious need for them. With regard to these, we have concluded:

As the features described involve the use of alternatives to explicit RDMA operations, in performing direct data placement and in transferring messages that are larger than the receive buffer limit, it is appropriate to understand the role that such operations are expected to have once the extensions discussed in this document are fully specified and implemented.

It is important to note that these extensions are OPTIONAL and are expected to remain so, while support for explicit RDMA operations will remain an integral part of RPC-over-RDMA.

Given this framework, the degree to which explicit RDMA operations will be used will reflect future implementation choices and needs. While we have been focusing on cases in which other options might be more efficient in some cases, it worth looking also at the cases in which explicit RDMA operations are likely to remain preferable:

6. Security Considerations

This document does not raise any security issues.

7. IANA Considerations

This document does not require any actions by IANA.

8. References

8.1. Normative References

[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997.
[rfc5666bis] Lever, C., Simpson, W. and T. Talpey, "Remote Direct Memory Access Transport for Remote Procedure Call", May 2016.

Work in progress.

8.2. Informative References

[RFC5666] Talpey, T. and B. Callaghan, "Remote Direct Memory Access Transport for Remote Procedure Call", RFC 5666, DOI 10.17487/RFC5666, January 2010.
[rfc5667bis] Lever, C., "Network File System (NFS) Upper Layer Binding To RPC-Over-RDMA", August 2016.

Work in progress.

[rpcrdmav2] Lever, C. and D. Noveck, "RPC-over-RDMA Version Two", June 2016.

Work in progress.

[rtrext] Noveck, D., "RPC-over-RDMA Extensions to Reduce Internode Round-trips", June 2016.

Work in progress.

[xcharext] Noveck, D., "RPC-over-RDMA Extension to Manage Transport Characterisitcs", August 2016.

Work in progress.

Appendix A. Acknowledgements

The author gratefully acknowledges the work of Brent Callaghan and Tom Talpey producing the original RPC-over-RDMA Version One specification [RFC5666] and also Tom's work in helping to clarify that specification.

The author also wishes to thank Chuck Lever for his work reviving RDMA support for NFS in [rfc5666bis], [rfc5667bis], and [rpcrdmav2], and for helpful discussion regarding RPC-over-RDMA latency issues.

Author's Address

David Noveck Hewlett Packard Enterprise 165 Dascomb Road Andover, MA 01810 USA Phone: +1 781-572-8038 EMail: