NFSv4 Working Group David L. Black Internet Draft Stephen Fridella Expires: April 2006 EMC Corporation October 21, 2005 pNFS Block/Volume Layout draft-black-pnfs-block-02.txt Status of this Memo By submitting this Internet-Draft, each author represents that any applicable patent or other IPR claims of which he or she is aware have been or will be disclosed, and any of which he or she becomes aware will be disclosed, in accordance with Section 6 of BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet- Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html This Internet-Draft will expire in April 2006. Abstract Parallel NFS (pNFS) extends NFSv4 to allow clients to directly access file data on the storage used by the NFSv4 server. This ability to bypass the server for data access can increase both performance and parallelism, but requires additional client functionality for data access, some of which is dependent on the class of storage used. The main pNFS operations draft specifies storage-class-independent extensions to NFS; this draft specifies the additional extensions (primarily data structures) for use of pNFS with block and volume based storage. Black Expires April 2006 [Page 1] Internet-Draft pNFS Block/Volume Layout October 2005 Conventions used in this document The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC-2119 [RFC2119]. Table of Contents 1. Introduction...................................................3 2. Background and Architecture....................................3 2.1. Data Structures: Extents and Extent Lists.................4 2.1.1. Layout Requests and Extent Lists.....................6 2.1.2. Client Copy-on-Write Processing......................7 2.1.3. Extents are Permissions..............................8 2.2. Volume Identification....................................10 3. Operations Issues.............................................11 3.1. Layout Operation Ordering Considerations.................12 3.1.1. Client Side Considerations..........................12 3.1.2. Server Side Considerations..........................13 3.2. Recall Callback Completion and Robustness Concerns.......14 3.3. Crash Recovery Issues....................................15 3.4. Additional Features - Not Needed or Recommended..........16 4. Security Considerations.......................................17 5. Conclusions...................................................17 6. Revision History..............................................18 7. Acknowledgments...............................................18 8. References....................................................18 8.1. Normative References.....................................18 8.2. Informative References...................................18 Author's Addresses...............................................19 Intellectual Property Statement..................................19 Disclaimer of Validity...........................................19 Copyright Statement..............................................20 Acknowledgment...................................................20 NOTE: This is an early stage draft. It's still rough in places, with significant work to be done. Black Expires April 2006 [Page 2] Internet-Draft pNFS Block/Volume Layout October 2005 1. Introduction Figure 1 shows the overall architecture of a pNFS system: +-----------+ |+-----------+ +-----------+ ||+-----------+ | | ||| | NFSv4 + pNFS | | +|| Clients |<------------------------------>| Server | +| | | | +-----------+ | | ||| +-----------+ ||| | ||| | ||| +-----------+ | ||| |+-----------+ | ||+----------------||+-----------+ | |+-----------------||| | | +------------------+|| Storage |------------+ +| Systems | +-----------+ Figure 1 pNFS Architecture The overall approach is that pNFS-enhanced clients obtain sufficient information from the server to enable them to access the underlying storage (on the Storage Systems) directly. See [PNFS] for more details. This draft is concerned with access from pNFS clients to Storage Systems over storage protocols based on blocks and volumes, such as the SCSI protocol family (e.g., parallel SCSI, FCP for Fibre Channel, iSCSI, SAS). This class of storage is referred to as block/volume storage. While the Server to Storage System protocol is not of concern for interoperability here, it will typically also be a block/volume protocol when clients use block/volume protocols. 2. Background and Architecture The fundamental storage abstraction supported by block/volume storage is a storage volume consisting of a sequential series of fixed size blocks. This can be thought of as a logical disk; it may be realized by the Storage System as a physical disk, a portion of a physical disk or something more complex (e.g., concatenation, striping, RAID, and combinations thereof) involving multiple physical disks or portions thereof. Black Expires April 2006 [Page 3] Internet-Draft pNFS Block/Volume Layout October 2005 A pNFS layout for this block/volume class of storage is responsible for mapping from an NFS file (or portion of a file) to the blocks of storage volumes that contain the file. The blocks are expressed as extents with 64 bit offsets and lengths using the existing NFSv4 offset4 and length4 types. Clients must be able to perform I/O to the block extents without affecting additional areas of storage (especially important for writes), therefore extents MUST be aligned to 512-byte boundaries, and SHOULD be aligned to the block size used by the NFSv4 server in managing the actual filesystem (4 kilobytes and 8 kilobytes are common block sizes). This block size is available as an NFSv4 attribute - see Section 7.4 of [PNFS]. This draft relies on the pNFS client indicating whether a requested layout is for read use or read-write use. A read-only layout may contain holes that are read as zero, whereas a read-write layout will contain allocated, but uninitialized storage in those holes (read as zero, can be written by client). This draft also supports client participation in copy on write by providing both read-only and uninitialized storage for the same range in a layout. Reads are initially performed on the read-only storage, with writes going to the uninitialized storage. After the first write that initializes the uninitialized storage, all reads are performed to that now- initialized writeable storage, and the corresponding read-only storage is no longer used. This draft draws extensively on the authors' familiarity with the the mapping functionality and protocol in EMC's HighRoad system. The protocol used by HighRoad is called FMP (File Mapping Protocol); it is an add-on protocol that runs in parallel with filesystem protocols such as NFSv3 to provide pNFS-like functionality for block/volume storage. While drawing on HighRoad FMP, the data structures and functional considerations in this draft differ in significant ways, based on lessons learned and the opportunity to take advantage of NFSv4 features such as COMPOUND operations. The support for client participation in copy-on-write is based on contributions from those with experience in that area, as HighRoad does not currently support client participation in copy-on-write. 2.1. Data Structures: Extents and Extent Lists A pNFS layout is a list of extents with associated properties. Each extent MUST be at least 512-byte aligned. Black Expires April 2006 [Page 4] Internet-Draft pNFS Block/Volume Layout October 2005 struct extent { offset4 file_offset;/* the logical location in the file */ length4 extent_length; /* the size of this extent in file and and on storage */ pnfs_deviceid4 volume_ID; /* the logical volume/physical device that this extent is on */ offset4 storage_offset;/* the logical location of this extent in the volume */ extentState4 es; /* the state of this extent */ }; enum extentState4 { READ_WRITE_DATA = 0, /* the data located by this extent is valid for reading and writing. */ READ_DATA = 1, /* the data located by this extent is valid for reading only; it may not be written. */ INVALID_DATA = 2, /* the location is valid; the data is invalid. It is a newly (pre-) allocated extent. There is physical space. */ NONE_DATA = 3, /* the location is invalid. It is a hole in the file. There is no physical space. */ }; The file_offset, extent_length, and es fields for an extent returned from the server are always valid. The interpretation of the storage_offset field depends on the value of es as follows: o READ_WRITE_DATA means that storage_offset is valid, and points to valid/initialized data that can be read and written. o READ_DATA means that storage_offset is valid and points to valid/ initialized data which can only be read. Write operations are prohibited; the client may need to request a read-write layout. Black Expires April 2006 [Page 5] Internet-Draft pNFS Block/Volume Layout October 2005 o INVALID_DATA means that storage_offset is valid, but points to invalid uninitialized data. This data must not be physically read from the disk until it has been initialized. A read request for an INVALID_DATA extent must fill the user buffer with zeros. Write requests must write whole blocks to the disk with bytes not initialized by the user must be set to zero. Any write to storage in an INVALID_DATA extent changes the written portion of the extent to READ_WRITE_DATA; the pNFS client is responsible for reporting this change via LAYOUTCOMMIT. o NONE_DATA means that storage_offset is not valid, and this extent may not be used to satisfy write requests. Read requests may be satisfied by zero-filling as for INVALID_DATA. NONE_DATA extents are returned by requests for readable extents; they are never returned if the request was for a writeable extent. The volume_ID field for an extent returned by the server is used to identify the logical volume on which this extent resides, see Section 2.2. The extent list lists all relevant extents in increasing order of the file_offset of each extent; any ties are broken by increasing order of the extent state (es). typedef extent extentList; /* MAX_EXTENTS = 256; */ TODO: Define the actual layout and layoutupdate data structures as extent lists. TODO: Striping support. Layout independent striping will not be added to [PNFS], but it can help compact layout representations when the filesystem is striped across block/volume storage. 2.1.1. Layout Requests and Extent Lists Each request for a layout specifies at least three parameters: offset, desired size, and minimum size (the desired size is missing from the operations draft - see Section 3). If the status of a request indicates success, the extent list returned must meet the following criteria: Black Expires April 2006 [Page 6] Internet-Draft pNFS Block/Volume Layout October 2005 o A request for a readable (but not writeable layout returns only READ_WRITE_DATA, READ_DATA or NONE_DATA extents (but not INVALID_DATA extents). A READ_WRITE_DATA extent MAY be returned by a pNFS server in a readable layout in order to avoid a subsequent client request for writing (ISSUE: Is that a good idea? It involves server second-guessing client, and the downside is the possible need for a recall callback). o A request for a writeable layout returns READ_WRITE_DATA or INVALID_DATA extents (but not NONE_DATA extents). It may also return READ_DATA extents only when the offset ranges in those extents are also covered by INVALID_DATA extents to permit writes. o The first extent in the list MUST contain the starting offset. o The total size of extents in the extent list MUST cover at least the minimum size and no more than the desired size. One exception is allowed: the total size MAY be smaller if only readable extents were requested and EOF is encountered. o Extents in the extent list MUST be logically contiguous for a read-only layout. For a read-write layout, the set of writable extents (i.e., excluding READ_DATA extents) MUST be logically contiguous. Every READ_DATA extent in a read-write layout MUST be covered by an INVALID_DATA extent. This overlap of READ_DATA and INVALID_DATA extents is the only permitted extent overlap. o Extents MUST be ordered in the list by starting offset, with READ_DATA extents preceding INVALID_DATA extents in the case of equal file_offsets. 2.1.2. Client Copy-on-Write Processing Distinguishing the READ_WRITE_DATA and READ_DATA extent types combined with the allowed overlap of READ_DATA extents with INVALID_DATA extents allows copy-on-write processing to be done by pNFS clients. In classic NFS, this operation would be done by the server. Since pNFS enables clients to do direct block access, it requires clients to participate in copy-on-write operations. When a client wishes to write data covered by a READ_DATA extent, it MUST have requested a writable layout from the server; that layout will contain INVALID_DATA extents to cover all the data ranges of that layout's READ_DATA extents. More precisely, for any file_offset range covered by one or more READ_DATA extents in a writable layout, the server MUST include one or more INVALID_DATA extents in the layout that cover the same file_offset range. The client MUST Black Expires April 2006 [Page 7] Internet-Draft pNFS Block/Volume Layout October 2005 logically copy the data from the READ_DATA extent for any partial blocks of file_offset and range, merge in the changes to be written, and write the result to the INVALID_DATA extent for the blocks for that file_offset and range. That is, if entire blocks of data are to be overwritten by an operation, the corresponding READ_DATA blocks need not be fetched, but any partial-block writes must be merged with data fetched via READ_DATA extents before storing the result via INVALID_DATA extents. Storing of data in an INVALID_DATA extent converts the written portion of the INVALID_DATA extent to a READ_WRITE_DATA extent; all subsequent reads MUST be performed from this extent; the corresponding portion of the READ_DATA extent MUST NOT be used after storing data in an INVALID_DATA extent. In the LAYOUTCOMMIT operation that normally sends updated layout information back to the server, for writable data, some INVALID_DATA extents may be committed as READ_WRITE_DATA extents, signifying that the storage at the corresponding storage_offset values has been stored into and is now to be considered as valid data to be read. READ_DATA extents need not be sent to the server. For extents that the client receives via LAYOUTGET as INVALID_DATA and returns via LAYOUTCOMMIT as READ_WRITE_DATA, the server will understand that the READ_DATA mapping for that extent is no longer valid or necessary for that file. ISSUE: This assumes that all block/volume pNFS clients will support copy-on-write. Negotiating this would require additional server code to cope with clients that don't support this, which doesn't seem like a good idea. 2.1.3. Extents are Permissions Layout extents returned to pNFS clients grant permission to read or write; READ_DATA and NONE_DATA are read-only (NONE_DATA reads as zeroes), READ_WRITE_DATA and INVALID_DATA are read/write, (INVALID_DATA reads as zeros, any write converts it to READ_WRITE_DATA). This is the only client means of obtaining permission to perform direct I/O to storage devices; a pNFS client MUST NOT perform direct I/O operations that are not permitted by an extent held by the client. Client adherence to this rule places the pNFS server in control of potentially conflicting storage device operations, enabling the server to determine what does conflict and how to avoid conflicts by granting and recalling extents to/from clients. Block/volume class storage devices are not required to perform read and write operations atomically. Overlapping concurrent read and write operations to the same data may cause the read to return a Black Expires April 2006 [Page 8] Internet-Draft pNFS Block/Volume Layout October 2005 mixture of before-write and after-write data. Overlapping write operations can be worse, as the result could be a mixture of data from the two write operations; this can be particularly nasty if the underlying storage is striped and the operations complete in different orders on different stripes. A pNFS server can avoid these conflicts by implementing a single writer XOR multiple readers concurrency control policy when there are multiple clients who wish to access the same data. This policy SHOULD be implemented when storage devices do not provide atomicity for concurrent read/write and write/write operations to the same data. A client that makes a layout request that conflicts with an existing layout delegation will be rejected with the error NFS4ERR_LAYOUTTRYLATER. This client is then expected to retry the request after a short interval. During this interval the server needs to recall the conflicting portion of the layout delegation from the client that currently holds it. This reject-and-retry approach does not prevent client starvation when there is contention for the layout of a particular file. For this reason a pNFS server SHOULD implement a mechanism to prevent starvation. One possibility is that the server can maintain a queue of rejected layout requests. Each new layout request can be checked to see if it conflicts with a previous rejected request, and if so, the newer request can be rejected. Once the original requesting client retries its request, its entry in the rejected request queue can be cleared, or the entry in the rejected request queue can be removed when it reaches a certain age. NFSv4 supports mandatory locks and share reservations. These are mechanisms that clients can use to restrict the set of I/O operations that are permissible to other clients. Since all I/O operations ultimately arrive at the NFSv4 server for processing, the server is in a position to enforce these restrictions. However, with pNFS layout delegations, I/Os will be issued from the clients that hold the delegations directly to the storage devices that host the data. These devices have no knowledge of files, mandatory locks, or share reservations, and are not in a position to enforce such restrictions. For this reason the NFSv4 server must not grant layout delegations that conflict with mandatory locks or share reservations. Further, if a conflicting mandatory lock request or a conflicting open request arrives at the server, the server must recall the part of the layout delegation in conflict with the request before processing the request. Black Expires April 2006 [Page 9] Internet-Draft pNFS Block/Volume Layout October 2005 2.2. Volume Identification Storage Systems such as storage arrays can have multiple physical network ports that need not be connected to a common network, resulting in a pNFS client having simultaneous multipath access to the same storage volumes via different ports on different networks. The networks may not even be the same technology - for example, access to the same volume via both iSCSI and Fibre Channel is possible, hence network address are difficult to use for volume identification. For this reason, this pNFS block layout identifies storage volumes by content, for example providing the means to match (unique portions of) labels used by volume managers. Any block pNFS system using this layout MUST support a means of content-based unique volume identification that can be employed via the data structure given here. A volume is content-identified by a disk signature made up of extents within blocks and contents that must match. block_device_addr_list - A list of the disk signatures for the physical volumes on which the file system resides. This is list of variable number of diskSigInfo structures. This is the device_addr_list<> as returned by GETDEVICELIST in [PNFS] typedef diskSigInfo block_device_addr_list; /* disksignature info */ where diskSigInfo is: struct diskSigInfo { /* used in DISK_SIGNATURE */ diskSig ds; /* disk signature */ pnfs_deviceid4 volume_ID; /* volume ID the server will use in extents. */ }; where diskSig is defined as: typedef sigComp diskSig; struct sigComp { /* disk signature component */ offset4 sig_offset; /* byte offset of component */ length4 sig_length; /* byte length of component */ Black Expires April 2006 [Page 10] Internet-Draft pNFS Block/Volume Layout October 2005 sigCompContents contents; /* contents of this component of the signature (this is opaque) */ }; sigCompContents MUST NOT be interpreted as a zero-terminated string, as it may contain embedded zero-valued octets. It contains sig_length octets. There are no restrictions on alignment (e.g., neither sig_offset nor sig_length are required to be multiples of 4). 3. Operations Issues NOTE: This section and its subsections are preserved for historical/review purposes only, as the [PNFS] draft has addressed all of these issues. The section and all subsections will be deleted in the next version of this draft. This section collects issues in the operations draft encountered in writing this block/volume layout draft. Most of these issues are expected to be resolved in draft-welch-pnfs-ops-03.txt . 1. RESOLVED: LAYOUTGET provides minimum and desired (max) lengths to server. 2. RESOLVED: Layouts are managed by offset and range; they are no longer treated as indivisible objects. 3. RESOLVED: There is a callback for the server to convey a new EOF to the client. 4. RESOLVED: HighRoad supports three types of layout recalls beyond range recalls: "everything in a file", "everything in a list of files", "everything in a filesystem". The first and third are supported in [PNFS] (set offset to zero and length to all 1's for everything in a file - [PNFS] implies this, but isn't explicit about it), and the second one can probably be done as a COMPOUND with reasonable effectiveness. LAYOUTRETURN supports return of everything in a file in a similar fashion (offset of zero, length of all 1's). 5. RESOLVED: Access and Modify time behavior. LAYOUTCOMMIT operation sets both Access and Modify times. LAYOUTRETURN cannot set either time - use a SETATTR in a COMPOUND to do this (Q: Can this inadvertently make time run backwards?). Black Expires April 2006 [Page 11] Internet-Draft pNFS Block/Volume Layout October 2005 6. RESOLVED: The disk signature approach to volume identification appears to be supportable via the opaque pnfs_devaddr4 union element. 7. RESOLVED: The LAYOUTCOMMIT operation has no LAYOUTRETURN side effects in -03. If it ever did, they were not intended. 3.1. Layout Operation Ordering Considerations This deserves its own subsection because there is some serious subtlety here. In contrast to NFSv4 callbacks that expect immediate responses, HighRoad layout callback responses are delayed to allow the client to perform any required commits, etc. prior to responding to the callback. This allows the reply to the callback to serve as an implicit return of the recalled range or ranges and tell the server that all callback related processing has been completed by the client. For consistency, pNFS should use the NFSv4 callback approach in which immediate responses are expected. As a result all returns of layout ranges MUST be explicit. 3.1.1. Client Side Considerations Consider a pNFS client that has issued a LAYOUTGET and then receives an overlapping recall callback for the same file. There are two possibilities, which the client cannot distinguish when the callback arrives: 1. The server processed the LAYOUTGET before issuing the recall, so the LAYOUTGET response is in flight, and must be waited for because it may be carrying layout info that will need to be returned to deal with the recall callback. 2. The server issued the callback before receiving the LAYOUTGET. The server will not respond to the LAYOUTGET until the recall callback is processed. This can cause deadlock, as the client must wait for the LAYOUTGET response before processing the recall in the first case, but that response will not arrive until after the recall is processed in the second case. The deadlock is avoided via a simple rule: RULE: A LAYOUTGET MUST be rejected with an error if there's an overlapping outstanding recall callback to the same client. The client MUST process the outstanding recall callback before retrying the LAYOUTGET. Black Expires April 2006 [Page 12] Internet-Draft pNFS Block/Volume Layout October 2005 Now the client can wait for the LAYOUTGET response because it will come in both cases. This RULE also applies to the callback to send an updated EOF to the client. The resulting situation is still less than desired, because issuance of a recall callback indicates a conflict and potential contention at the server, so recall callbacks should be processed as fast as possible by clients. In the second case, if the client knows that the LAYOUTGET will be rejected, it is beneficial for the client to process the recall immediately without waiting for the LAYOUTGET rejection. To do so without added client complexity, the server needs to reject the LAYOUTGET even if it arrives at the server after the client operations that process the recall callback; if the client still wants that layout, it can reissue the LAYOUTGET. HighRoad uses the equivalent of a per-file layout stateid to enable this optimization. The layout stateid increments on each layout operation completion and callback issuance, and the current value of the layout stateid is sent in every operation response and every callback. If the initial layout stateid value is N, then in the first case above, the recall callback carries stateid N+2 indicating that the LAYOUTGET response is carrying N+1 and hence has to be waited for. In the second case above, the recall callback carries layout stateid N+1 indicating that the LAYOUTGET will be rejected with a stale layout stateid (N where N+1 or greater is current) whenever it arrives, and hence the callback can be processed immediately. This per-file layout stateid approach entails prohibiting concurrent callbacks to the for the same file to the same client, as server issuance of a new callback could cause stale layout stateid errors for operations that the client is performing to deal with an earlier recall callback. ISSUE: Does restricting all pNFS client operations on the same file to a single session help? 3.1.2. Server Side Considerations Consider a related situation from the pNFS server's point of view. The server has issued a recall callback and receives an overlapping LAYOUTGET for the same file before the LAYOUTRETURN(s) that respond to the recall callback. Again, there are two cases: 1. The client issued the LAYOUTGET before processing the recall callback. The LAYOUTGET MUST be rejected according to the RULE in the previous subsection. Black Expires April 2006 [Page 13] Internet-Draft pNFS Block/Volume Layout October 2005 2. The client issued the LAYOUTGET after processing the recall callback, but it arrived before the LAYOUTRETURN that completed that processing. The simplest approach is to apply the RULE and always reject the overlapping LAYOUTGET. The client has two ways to avoid this result - it can issue the LAYOUTGET as a subsequent element of a COMPOUND containing the LAYOUTRETURN that completes the recall callback, or it can wait for the response to that LAYOUTRETURN. This leads to a more general problem; in the absence of a callback if a client issues concurrent overlapping LAYOUTGET and LAYOUTRETURN operations, it is possible for the server to process them in either order. HighRoad forbids a client from doing this, as the per-file layout stateid will cause one of the two operations to be rejected with a stale layout stateid. This approach is simpler and produces better results by comparison to allowing concurrent operations, at least for this sort of conflict case, because server execution of operations in an order not anticipated by the client may produce results that are not useful to the client (e.g., if a LAYOUTRETURN is followed by a concurrent overlapping LAYOUTGET, but executed in the other order, the client will not retain layout extents for the overlapping range). 3.2. Recall Callback Completion and Robustness Concerns The discussion of layout operation ordering implicitly assumed that any callback results in a LAYOUTRETURN or set of LAYOUTRETURNs that match the range in the callback. This envisions that the pNFS client state for a file match the pNFS server state for that file and client regarding layout ranges and permissions. That may not be the best design assumption because: 1. It may be useful for clients to be able to discard layout information without calling LAYOUTRETURN. If conflicts that require callbacks are rare, and a server can use a multi-file callback to recover per-client resources (e.g., via a multi-file recall operation based on some sort of LRU), the result may be significantly less client-server pNFS traffic. 2. It may be similarly useful for servers to enhance information about what layout ranges are held by a client beyond what a client actually holds. In the extreme, a server could manage conflicts on a per-file basis, only issuing whole-file callbacks even though clients may request and be granted sub-file ranges. Black Expires April 2006 [Page 14] Internet-Draft pNFS Block/Volume Layout October 2005 3. The synchronized state assumption is not robust to minor errors. A more robust design would allow for divergence between client and server and the ability to recover. It is vital that a client not assign itself layout permissions beyond what the server has granted and that the server not forget layout permissions that have been granted in order to avoid errors. OTOH, if a server believes that a client holds an extent that the client doesn't know about, it's useful for the client to be able to issue the LAYOUTRETURN that the server is expecting in response to a recall. At a minimum, in light of the above, it is useful for a server to be able to issue callbacks for layout ranges it has not granted to a client, and for a client to return ranges it does not hold. This leads to a couple of requirements: A pNFS client's final operation in processing a recall callback SHOULD be a LAYOUTRETURN whose range matches that in the callback. If the pNFS client holds no layout permissions in the range that has been recalled, it MUST respond with a LAYOUTRETURN whose range matches that in the callback. This avoids any need for callback cookies (server to client) that would have to be returned to indicate recall callback completion. For a callback to set EOF, the client MUST logically apply the new EOF before issuing the response to the callback, and MUST NOT issue any other pNFS operations before responding to the callback. ISSUE: HighRoad FMP also requires that LAYOUTCOMMIT operations be stalled at the server while an EOF callback is outstanding. 3.3. Crash Recovery Issues Client recovery for layout delegations works in much the same way as NFSv4 client recovery for other lock/delegation state. When an NFSv4 client reboots, it will lose all information about the layout delegations that it previously owned. There are two methods by which the server can reclaim these resources and begin providing them to other clients. The first is through the expiry of the client's lock/delegation lease. If the client recovery time is longer than the lease period, the client's lock/delegation lease will expire and the server will know to reclaim any state held by the client. On the other hand, the client may recover in less time than it takes for the lease period to expire. In such a case, the client will be required to contact the server through the standard SETCLIENTID protocol. The server will find that the client's id matches the id of the previous client invocation, but that the verifier is different. The server Black Expires April 2006 [Page 15] Internet-Draft pNFS Block/Volume Layout October 2005 uses this as a signal to reclaim all the state associated with the client's previous invocation. The server recovery case is slightly more complex. In general, the recovery process will again follow the standard NFSv4 recovery model: the client will discover that the server has rebooted when it receives an unexpected STALE_STATEID or STALE_CLIENTID reply from the server; it will then proceed to try to reclaim its previous delegations during the server's recovery grace period. However there is an important safety concern associated with layout delegations that does not come into play in the standard NFSv4 case. If a standard NFSv4 client makes use of a stale delegation, the consequence could be to deliver stale data to an application. However, the pNFS layout delegation enables the client to directly access the file system storage---if this access is not properly managed by the NFSv4 server the client can potentially corrupt the file system data or meta-data. Thus it is vitally important that the client discover that the server has rebooted as soon as possible, and that the client stops using stale layout delegations before the server gives the delegations away to other clients. To ensure this, the client must be implemented so that layout delegations are never used to access the storage after the client's lease timer has expired. This prohibition applies to all accesses, especially the flushing of dirty data to storage. If the client's lease timer expires because the client could not contact the server for any reason, the client MUST immediately stop using the layout delegation until the server can be contacted and the delegation can be officially recovered or reclaimed. 3.4. Additional Features - Not Needed or Recommended This subsection is a place to record things that existing SAN or clustered filesystems do that aren't needed or recommended for pNFS: o Callback for write-to-read downgrade. Writers tend to want to remain writers, so this feature may not be very useful. o HighRoad FMP implements several frequently used operation combinations as single RPCs for efficiency; these can be effectively handled by NFSv4 COMPOUNDs. One subtle difference is that a single RPC is treated as a single operation, whereas NFSv4 COMPOUNDs are not atomic in any sense. This can result in operation ordering subtleties, e.g., having to set the new EOF *before* returning the layout extent that contains the new EOF, even within a single COMPOUND. Black Expires April 2006 [Page 16] Internet-Draft pNFS Block/Volume Layout October 2005 o Queued request support. The HighRoad FMP protocol specification allows the server to return an "operation blocked" result code with a cookie that is later passed to the client in a "it's done now" callback. This has not proven to be of great use vs. having the client retry with some sort of back-off. Recommendations on how to back off should be added to the ops draft. o Additional client and server crash detection mechanisms. As a separate protocol, HighRoad FMP had to handle this on its own. As an NFSv4 extension, NFSv4's SETCLIENTID, STALE CLIENTID and STALE STATEID mechanisms combined with implicit lease renewal and (per- file) layout stateids should be sufficient for pNFS. 4. Security Considerations Certain security responsibilities are delegated to pNFS clients. Block/volume storage systems generally control access at a volume granularity, and hence pNFS clients have to be trusted to only perform accesses allowed by the layout extents it currently holds (e.g., and not access storage for files on which a layout extent is not held). This also has implications for some NFSv4 functionality outside pNFS. For instance, if a file is covered by a mandatory read-only lock, the server can ensure that only read-layout- delegations for the file are granted to pNFS clients. However, it is up to each pNFS client to ensure that the read layout delegation is used only to service read requests, and not to allow writes to the existing parts of the file. Since block/volume storage systems are generally not capable of enforcing such file-based security, in environments where pNFS clients cannot be trusted to enforce such policies, block/volume-based pNFS SHOULD NOT be used. 5. Conclusions 6. IANA Considerations There are no IANA considerations in this document. All pNFS IANA Considerations are covered in [PNFS]. Black Expires April 2006 [Page 17] Internet-Draft pNFS Block/Volume Layout October 2005 7. Revision History -00: Initial Version -01: Rework discussion of extents as locks to talk about extents granting access permissions. Rewrite operation ordering section to discuss deadlocks and races that can cause problems. Add new section on recall completion. Add client copy-on-write based on text from Craig Everhart. -02: Fix glitches in extent state descriptions. Describe most issues as RESOLVED. Most of Section 3 has been incorporated into the [PNFS] draft, add NOTE to that effect and say that it will be deleted in the next version of this draft (which should be a draft-ietf-nfsv4 draft). Cleaning up a number of things have been left to that draft revision, including the interlocks with the types in [PNFS], layout striping support, and finishing the Security Considerations section. 8. Acknowledgments This draft draws extensively on the authors' familiarity with the the mapping functionality and protocol in EMC's HighRoad system. The protocol used by HighRoad is called FMP (File Mapping Protocol); it is an add-on protocol that runs in parallel with filesystem protocols such as NFSv3 to provide pNFS-like functionality for block/volume storage. While drawing on HighRoad FMP, the data structures and functional considerations in this draft differ in significant ways, based on lessons learned and the opportunity to take advantage of NFSv4 features such as COMPOUND operations. The design to support pNFS client participation in copy-on-write is based on text and ideas contributed by Craig Everhart of IBM. 9. References 9.1. Normative References [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997. [PNFS] Goodson, G., et. al. "NFSv4 pNFS Extensions", draft-ietf- nfsv4-pnfs-00.txt, Work in Progress, October 2005. TODO: Need to reference RFC 3530. 9.2. Informative References OPEN ISSUE: HighRoad and/or SAN.FS references? Black Expires April 2006 [Page 18] Internet-Draft pNFS Block/Volume Layout October 2005 Author's Addresses David L. Black EMC Corporation 176 South Street Hopkinton, MA 01748 Phone: +1 (508) 293-7953 Email: black_david@emc.com Stephen Fridella EMC Corporation 32 Coslin Drive Southboro, MA 01772 Phone: +1 (508) 305-8512 Email: fridella_stephen@emc.com Intellectual Property Statement The IETF takes no position regarding the validity or scope of any Intellectual Property Rights or other rights that might be claimed to pertain to the implementation or use of the technology described in this document or the extent to which any license under such rights might or might not be available; nor does it represent that it has made any independent effort to identify any such rights. Information on the procedures with respect to rights in RFC documents can be found in BCP 78 and BCP 79. Copies of IPR disclosures made to the IETF Secretariat and any assurances of licenses to be made available, or the result of an attempt made to obtain a general license or permission for the use of such proprietary rights by implementers or users of this specification can be obtained from the IETF on-line IPR repository at http://www.ietf.org/ipr. The IETF invites any interested party to bring to its attention any copyrights, patents or patent applications, or other proprietary rights that may cover technology that may be required to implement this standard. Please address the information to the IETF at ietf- ipr@ietf.org. Black Expires April 2006 [Page 19] Internet-Draft pNFS Block/Volume Layout October 2005 Disclaimer of Validity This document and the information contained herein are provided on an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Copyright Statement Copyright (C) The Internet Society (2005). This document is subject to the rights, licenses and restrictions contained in BCP 78, and except as set forth therein, the authors retain all their rights. Acknowledgment Funding for the RFC Editor function is currently provided by the Internet Society. Black Expires April 2006 [Page 20]