INTERNET DRAFT draft-hufferd-iser-ib-01.txt John Hufferd Brocade Mike Ko IBM Corporation Yaron Haviv Voltaire Ltd September, 2005 Expires: March, 2006 Generalization of iSER for InfiniBand and other Network Protocols Status of this Memo By submitting this Internet-Draft, each author represents that any applicable patent or other IPR claims of which he or she is aware have been or will be disclosed, and any of which he or she becomes aware will be disclosed, in accordance with Section 6 of BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet- Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/1id-abstracts.html. The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. J.Hufferd et. al. Expires March 2006 1 Internet-Draft Generalization of the iSER Specification Sept. 2005 Abstract The iSCSI Extensions for RDMA document [iSER] currently specifies the RDMA data transfer capability for [iSCSI] over iWARP. This document generalizes the iSER document to permit it to be used with other RDMA capable protocols such as InfiniBand. J.Hufferd et. al. Expires March 2006 2 Internet-Draft Generalization of the iSER Specification Sept. 2005 Table of Contents 1 Motivation..................................................5 2 Overall generalizations needed within the iSER specification6 2.1 Generalization of Definitions...............................6 2.1.1 The iWARP term.............................................6 2.1.2 The RNIC term..............................................7 2.1.3 Steering Tag (STag)........................................7 2.1.4 Inbound RDMA Read Queue Depth (IRD) & Outbound RDMA Read Queue Depth (ORD).................................................8 2.1.5 RDMA Protocol (RDMAP)......................................8 2.1.6 RDMAP Layer................................................8 2.1.7 RDMAP Stream...............................................8 2.1.8 RDMAP Message..............................................8 2.2 The following is placed/updated in the Acronym Section......9 2.3 Connection Establishment, Login, and Transition to iSER.....9 2.4 Security considerations.....................................9 2.5 Adjustments to the iSER Appendix...........................10 2.6 Add Appendix B.............................................10 3 Additional detailed [iSER] document modification...........11 3.1 Adjustments to Section 1 on Definitions and Acronyms.......11 3.1.1 Adjustments to Section 1 on Definitions...................11 3.2 Adjustments to section 2...................................12 3.2.1 Adjustment to Section 2.1 Motivation......................12 3.2.2 Adjustment to Section 2.2 Architectural Goals.............12 3.2.3 Adjustment to Section 2.3 Protocol Overview...............12 3.2.4 Adjustment to Section 2.4 RDMA services and iSER..........13 3.2.5 Adjustment to Section 2.7 iSCSI/iSER Layering.............13 3.3 Adjustments to Section 3...................................14 3.3.1 Adjustment to Section 3.1.6...............................14 3.4 Adjustments to Section 4...................................14 3.4.1 Adjustment to Section 4.1.................................14 3.4.2 Adjustments to Section 4.2................................15 3.5 Adjustments to Section 5...................................15 3.5.1 Adjustments to section 5.1 iSCSI/iSER Connection Setup....15 3.5.2 Adjustment to Section 5.1.1 Initiator Behavior............16 3.5.3 Adjustment to Section 5.1.2 Target Behavior...............16 3.6 Adjustments to Section 7 on iSCSI PDU Considerations.......16 3.6.1 Adjustment to Section 7.3.9...............................16 3.7 Adjustments to Section 11 Security Considerations..........17 3.8 Adjustments to Section 12.2 Informational References.......17 4 IANA Considerations........................................18 5 References.................................................19 5.1 Informative References.....................................19 6 Appendix...................................................20 6.1 Architectural discussion of iSER over InfiniBand...........20 6.2 The Host side of the InfiniBand iSCSI & iSER connections...20 J.Hufferd et. al. Expires March 2006 3 Internet-Draft Generalization of the iSER Specification Sept. 2005 6.3 The Storage side of iSCSI & iSER mixed network environment.21 6.4 Discovery processes for an InfiniBand Host.................21 6.5 IBTA Connection specifications.............................22 7 Author's Address...........................................23 8 Acknowledgments............................................24 9 Full Copyright Statement...................................25 Table of Figures Figure 1 - Example of iSCSI/iSER Layering in Full Feature Phase..13 Figure 2 - iSCSI and iSER on IB..................................20 Figure 3 - Storage Controller with TCP, iWARP, and IB Connections21 J.Hufferd et. al. Expires March 2006 4 Internet-Draft Generalization of the iSER Specification Sept. 2005 1 Motivation Currently the work to define iSCSI extensions for RDMA [iSER] only considers using the iWARP protocol suite. While this objective meets the short term requirement since iSCSI is defined only for TCP, there is a huge benefit to generalize a standardized [iSER] so that it can be used with other types of RDMA capable Protocol layers now and in the future such as InfiniBand (with reliable connections, RC). The interest in using [iSER] for InfiniBand is based on exploiting the iSCSI protocol features and its discovery and management protocol instead of using the SCSI RDMA Protocol (SRP) which lacks the management and discovery support. Furthermore, with an iSCSI based protocol, the storage professional and/or administrator only needs to understand and support a single basic protocol, which has similar implementations across a suite of different network types (iWARP, InfiniBand, etc.). It was to enable this vision and desire for a single storage protocol that the proposed generalizations to [iSER] were created. J.Hufferd et. al. Expires March 2006 5 Internet-Draft Generalization of the iSER Specification Sept. 2005 2 Overall generalizations needed within the iSER specification This section will specify changes/adjustments that are to be made in the iSER document to make it more general. The goal of these changes is not to modify the basic operation of iSCSI/iSER when operating on iWARP, but to change/adjust the wording in such a way that iSCSI/iSER can be layered over a different RDMA-capable protocol layer such as InfiniBand. Except for the unique features of non iWARP protocols dealing with initial Login and Security, the rest of the iSER document is applicable to these other RDMA-Capable Protocols (such as InfiniBand.) The details of many of the suggested changes can be found in Section 3 of this document. 2.1 Generalization of Definitions It is required that some of the terminology be clarified as to the applicability of the terms to the actual transport layer used. 2.1.1 The iWARP term As currently defined, the iWARP term has a strong TCP centric bias. We are introducing a new, more generic term, known as RDMA-Capable Protocol (RCP) to denote the protocol layer that provides the RDMA functionality for iSER. The following term will be added to the Definition section: RDMA-Capable Protocol (RCP) - The protocol or protocol suite that provides a reliable RDMA transport functionality, e.g., iWARP, InfiniBand, etc. With these new definitions, the "iWARP" term is hereby generalized as follows: 1. Whenever the term "iWARP protocol suite" occurs in the iSER draft, it is hereby replaced by "RDMA-Capable Protocol". In addition, the phrase "such as the iWARP protocol suite" is hereby added only where necessary to denote cases that only apply for iWARP. 2. Whenever the term "iWARP layer" occurs in the iSER draft, it is hereby replaced by "RDMA-Capable Protocol layer". In addition, the phrase "such as the iWARP Layer" is hereby added only where necessary to denote cases that only apply for iWARP. J.Hufferd et. al. Expires March 2006 6 Internet-Draft Generalization of the iSER Specification Sept. 2005 3. Whenever the term "iWARP" is used as an adjective in other context, it is hereby replaced with RCP. E.g., "iWARP functionality" is replaced with "RCP functionality". 4. Whenever the term "iWARP" is used as shorthand for the iWARP protocol suite, it is hereby replaced by "RDMA-Capable Protocol". 2.1.2 The RNIC term The term "RNIC" has been generally accepted by the industry to mean an RDMA-enabled Network Interface Controller for the IP world. So to generalize iSER for any RDMA-Capable Protocol layer, we will introduce a new term known as RDMA-Capable Controller, defined as follows: RDMA-Capable Controller û A network I/O adapter or embedded controller with RDMA functionality. E.g., for iWARP, this could be an RNIC, and for InfiniBand, this could be a HCA (Host Channel Adapter) or TCA (Target Channel Adapter). Within the body of the iSER document the term RDMA-Capable Controller is hereby used whenever the intention is to refer to a general controller that provides RDMA functionality. In addition, the clause "such as an RNIC" is hereby added as necessary where the clear intent of the statement is to address an iWARP RDMA-Capable Controller. Within the body of the iSER document, the term RNIC is left unchanged if it specifically or implicitly refers to TCP/IP. 2.1.3 Steering Tag (STag) The Steering Tag (STag) term hereby has its definition extended so that it applies to both a Tag for a Remote Buffer, and the Tag for a Local Buffer. The following is a replacement for the existing Steering Tag definition in the definition section. Steering Tag (STag) - An identifier of a Tagged Buffer on a Node (Local or Remote) as defined in [RDMAP] and [DDP]. For other RDMA-Capable protocol layers, the Steering Tag may be known by different names but will be herein referred to as STags. For example, for InfiniBand, a Remote STag is known as an R-Key, and a Local STag is known as an L-Key and both will be considered STags. J.Hufferd et. al. Expires March 2006 7 Internet-Draft Generalization of the iSER Specification Sept. 2005 2.1.4 Inbound RDMA Read Queue Depth (IRD) & Outbound RDMA Read Queue Depth (ORD) To generalize on the terms Inbound RDMA Read Queue Depth (IRD) and the Outbound RDMA Read Queue Depth (ORD) for other RDMA-Capable protocol layers, the following is added to the definition for IRD: "For some RDMA-Capable Protocol layers, the term "IRD" may be known by a different name. For example, for InfiniBand, the equivalent for IRD is the Responder Resources". For ORD, the following is added: "For some RDMA-Capable Protocol Layer, the term "ORD" may be known by a different name. For example, for InfiniBand, the equivalent for ORD is the Initiator Depth." 2.1.5 RDMA Protocol (RDMAP) In the body of the document the term "RDMA-Capable Protocol", or "RCP" is hereby used whenever any RDMA wire protocol or RDMA protocol stack is applicable. Only when the document intends to explicitly address a specific iWARP wire protocol is the term [RDMAP] used. 2.1.6 RDMAP Layer In the body of the document the term "RDMAP Layer" is hereby replaced with the term "RCP Layer". 2.1.7 RDMAP Stream The following is hereby included in the definition section replacing the term "RDMAP Stream": RCP Stream - A single bidirectional association between the peer RDMA-Capable Protocol layers on two Nodes over a single transport-level stream. For iWARP, an RCP Stream is known as an RDMAP Stream, and the association is created when the connection transitions to iSER-assisted mode following a successful Login Phase during which iSER support is negotiated. In the body of the document, the term "RDMAP Stream" is hereby replaced by the term "RCP Stream". 2.1.8 RDMAP Message The following is included in the definition section to replace "RDMAP Message": J.Hufferd et. al. Expires March 2006 8 Internet-Draft Generalization of the iSER Specification Sept. 2005 RCP Message - One or more packets of the network layer comprising a single RDMA operation or a part of an RDMA Read Operation of the RDMA-Capable Protocol. For iWARP, an RCP Message is known as an RDMAP Message. In the body of the document, the term "RDMAP Message" is hereby replaced by the term "RCP Message". The exception to this is when the term "RDMAP Message" is used to describe the iSER Hello and HelloReply Messages. Here "RDMAP Message" is hereby replaced by "iSER Message" in order to accommodate transport layers that have message delivery capability such as [IB]. The iSCSI layer may use that messaging capability immediately after connection establishment before enabling iSER-assisted mode. In this case the iSER Hello and HelloReply Messages are not the first RCP Messages, but they are the first iSER Messages. 2.2 The following is placed/updated in the Acronym Section HCA Host Channel Adapter IB InfiniBand IPoIB IP over InfiniBand TCA Target Channel Adapter 2.3 Connection Establishment, Login, and Transition to iSER The discussion of connection establishment and the use of a messaging protocol for exchanging Login Request and Login Response PDUs for IB are inserted in this section, along with the extended specification of the transition of an IB connection to iSER mode. The suggested detail changes can be found in section 3.5.1 through section 3.5.3 of this document. 2.4 Security considerations The security consideration are updated to include requirements on security for transports other than TCP, the document now states that the security concerns must be addressed appropriately for different transport environments. However the iSCSI implementation requirements for IPsec are still required wherever an iSER Message enters an IP environment from a non IP one (such as IB). Further the iSCSI/iSER requirement for IPsec on IP based protocols such as TCP will continue to require IPsec as a must implement, but optional to use. There is now a SHOULD implement (optional to use) J.Hufferd et. al. Expires March 2006 9 Internet-Draft Generalization of the iSER Specification Sept. 2005 requirement on non IP networks for a packet by packet security facility that is at least as strong as that required by [iSCSI]. The exact wordage can be found in section 3.6 of this document. 2.5 Adjustments to the iSER Appendix. The current iSER appendix will hereby be renamed "Appendix A". 2.6 Add Appendix B A new informational appendix (Appendix B) is hereby added that explains how an InfiniBand RC connection can be used to carry the iSER protocol. The content of the new appendix B is that which is contained in the section 6 of this document. J.Hufferd et. al. Expires March 2006 10 Internet-Draft Generalization of the iSER Specification Sept. 2005 3 Additional detailed [iSER] document modification The new terms introduced in the subsections under section 2.1 will replace the existing ones in the [iSER] document where appropriate. In addition, the following changes and clarifications are needed. 3.1 Adjustments to Section 1 on Definitions and Acronyms The following paragraph in section 1 is to be removed: "Some of the following definitions are taken from [RDMAP]. In those definitions, the term ULP refers to the iSER layer." The term "ULP" in subsections under section 1 is replaced by the term "iSER layer". 3.1.1 Adjustments to Section 1 on Definitions The definition for Advertisement is replaced with the following: The act of informing a remote iSER Layer that a local node's buffer is available to it. A Node makes a buffer available for incoming RDMA Read Request Message or incoming RDMA Write Message access by informing the remote iSER Layer of the Tagged Buffer identifiers (STag, TO, and buffer length). Note that this Advertisement of Tagged Buffer information is the responsibility of the iSER Layer on either end and is not defined by the RDMA-Capable Protocol. A typical method would be for the iSER Layer to embed the Tagged Buffer's STag, TO, and buffer length in a Send Message destined for the remote iSER Layer. The definition for Invalidate Stag is replaced with the following: Invalidate STag - A mechanism used to prevent the Remote Peer from reusing a previous explicitly Advertised STag, until the iSER Layer at the local node makes it available through a subsequent explicit Advertisement. The definition for Tagged Buffer is replaced with the following: Tagged Buffer - A buffer that is explicitly Advertised to the iSER Layer at the remote node through the exchange of an STag, Tagged Offset, and length. The definition for Untagged Buffer is replaced with the following: J.Hufferd et. al. Expires March 2006 11 Internet-Draft Generalization of the iSER Specification Sept. 2005 Untagged Buffer - A buffer that is not explicitly Advertised to the iSER Layer at the remode node 3.2 Adjustments to section 2 3.2.1 Adjustment to Section 2.1 Motivation The fourth paragraph is hereby adjusted to: Supporting direct data placement is the main function of an RDMA-Capable Protocol (RCP). An RDMA-Capable Controller (such as an iWARP RNIC, or an InfiniBand HCA/TCA) can be used by any application that has been extended to support RDMA. 3.2.2 Adjustment to Section 2.2 Architectural Goals The following are changes for the numbered paragraphs: 1. Provide an RDMA data transfer model for iSCSI that enables direct in order or out of order data placement of SCSI data into pre-allocated SCSI buffers while maintaining in order data delivery. 5. Allow initiator and target implementations to utilize generic RDMA-Capable Controllers such as RNICs, or implement iSCSI and iSER in software (not require iSCSI or iSER specific assists in the RCP implementation or RDMA-Capable Controller). 6. Require full and only generic RCP functionality at both the initiator and the target. 3.2.3 Adjustment to Section 2.3 Protocol Overview The following change is hereby made to paragraph number 6: 6. RCP guarantees data integrity. (For example, iWARP includes a CRC-enhanced framing layer called MPA on top of TCP; and for InfiniBand, the CRCs are included in the Reliable Connection mode.) For this reason, iSCSI header and data digests are negotiated to "None" for iSCSI/iSER sessions. The following is added to paragraph number 7: (However, see section Error! Reference source not found. on the handling of SNACK Request PDUs.) J.Hufferd et. al. Expires March 2006 12 Internet-Draft Generalization of the iSER Specification Sept. 2005 3.2.4 Adjustment to Section 2.4 RDMA services and iSER The following change is hereby made to the first paragraph: iSER is designed to work with software and/or hardware protocol stacks providing the protocol services defined in RCP documents such as [RDMAP], [IB], etc. The following subsections describe the key protocol elements of RCP services that iSER relies on. 3.2.5 Adjustment to Section 2.7 iSCSI/iSER Layering The layering wordage needed additional generalization and the example needed to be made more general. Therefore, the following is the change in wordage and the replacement for Figure 1: "iSCSI Extensions for RDMA (iSER) is layered between the iSCSI layer and the RCP Layer. Note that the RCP layer may be composed of one or more distinct protocol layers depending on the specifics of the RCP. Figure 1 shows an example of the relationship between SCSI, iSCSI, iSER, and the different RCP layers. For TCP, the RCP is iWARP. For InfiniBand, the RCP is the Reliable Connected Transport Service. Note that the iSCSI layer as described here supports the RDMA Extensions as used in iSER." +-------------------------------------+ | SCSI | +-------------------------------------+ | iSCSI | DI ------> +-------------------------------------+ | iSER | +---------+--------------+------------+ | RDMAP | | | +---------+ Infiniband | | | DDP | Reliable | Other | +---------+ Connected | RDMA- | | MPA | Transport | Capable | +---------+ Service | Protocol | | TCP | | | +---------+--------------+------------+ | | Infiniband | Other | | IP | Network | Network | | | Layer | Layer | +---------+--------------+------------+ Figure 1 - Example of iSCSI/iSER Layering in Full Feature Phase J.Hufferd et. al. Expires March 2006 13 Internet-Draft Generalization of the iSER Specification Sept. 2005 3.3 Adjustments to Section 3 3.3.1 Adjustment to Section 3.1.6 The first sentence in the last paragraph is to be replaced with the following: The Final_Login_Response_PDU input qualifier is applicable only for a target, and contains the final Login Response PDU that concludes the iSCSI Login Phase. If the underlying transport is TCP, the final Login Response PDU must be sent as a byte stream as expected by the iSCSI Layer at the initiator. 3.4 Adjustments to Section 4 3.4.1 Adjustment to Section 4.1 The title of the section "Interaction with the iWARP Layer" is hereby changed to "Interaction with the RCP Layer". The first paragraph is hereby changed to: "The iSER protocol layer is layered on top of an RCP layer (see Figure 1) and the following are the key features that are assumed to be supported by any RCP Layer." The second * paragraph is hereby changed to: "The RCP layer provides reliable, in-order message delivery and direct data placement." The following paragraph is to be added after the second * paragraph: "When the iSER Layer issues an RDMA Read Operation following an RDMA Write Operation on one RCP Stream, the RDMA Read Response Message processing on the remote node will be started only after the preceding RDMA Write Message payload is placed in the memory of the remote node." The next to last * paragraph is hereby replaced with the following: * For a transport layer that operates in byte stream mode such as TCP, the RCP implementation supports the enabling of the RDMA mode after Connection establishment and the exchange of Login parameters in byte stream mode. For a transport layer that provides message delivery capability such as [IB], the RCP implementation supports the use of the messaging capability by J.Hufferd et. al. Expires March 2006 14 Internet-Draft Generalization of the iSER Specification Sept. 2005 the iSCSI Layer directly for the Login phase after connection establishment before enabling iSER-assisted mode. 3.4.2 Adjustments to Section 4.2 The following is a replacement for section 4.2 Interactions with the Transport Layer The iSER Layer does not directly setup the transport layer connection (e.g., TCP, or [IB]). During Connection setup, the iSCSI Layer is responsible for setting up the Connection. If the login is successful, the iSCSI Layer invokes the Enable_Datamover Operational Primitive to request the iSER Layer to transition to the iSER-assisted mode for that iSCSI connection. See section 5.1 on iSCSI/iSER Connection setup. After transitioning to iSER-assisted mode, the RCP Layer and the underlying transport layer are responsible for maintaining the Connection and reporting to the iSER Layer any Connection failures. 3.5 Adjustments to Section 5 3.5.1 Adjustments to section 5.1 iSCSI/iSER Connection Setup The following is to be added at the end of paragraph 1: "The same connection MUST be used for both the iSCSI Login phase and the subsequent iSER-supported full feature phase." The following text is hereby added after the second paragraph: "If the RDMAExtensions key is not negotiated to Yes, then for some RCP implementation (such as [IB]), the connection may need to be re-established in TCP capable mode. (For InfiniBand this will require an [IPoIB] type connection.) The following text is hereby added after the third paragraph: "Discovery sessions are always conducted using the transport layer as described in [RFC3720]." The following is a replacement for the first two sentences in the last paragraph: "When the RDMAExtensions key is negotiated to "Yes", the HeaderDigest and the DataDigest keys MUST be negotiated to "None" on all iSCSI/iSER connections participating in that J.Hufferd et. al. Expires March 2006 15 Internet-Draft Generalization of the iSER Specification Sept. 2005 iSCSI session. This is because, for an iSCSI/iSER connection, RCP provides error detection based on 32-bit CRC for all iSER Messages." 3.5.2 Adjustment to Section 5.1.1 Initiator Behavior The following are changes for the bullet 3: 3. If necessary, the iSER Layer should enable RCP and transition the connection to iSER-assisted mode. When the RCP is iWARP, then this step MUST be done. Not all RCPs may need it depending on the RCP Stream start-up state, e.g., [IB]. 3.5.3 Adjustment to Section 5.1.2 Target Behavior Bullets "3." & "4." are hereby replaced with the following: "3. The iSER Layer MUST send the final Login Response PDU in the native transport mode to conclude the iSCSI Login Phase. If the underlying transport is TCP, then the iSER Layer MUST send the final Login Response PDU in byte stream mode. 4. After sending the final Login Response PDU, the iSER Layer should enable RCP if necessary and transition the connection to iSER-assisted mode. When the RCP is iWARP, then this step MUST be done. Not all RCPs may need it depending on the RCP Stream start-up state. The last paragraph is hereby replaced with: "Note: In the above sequence, the operations as described in bullets 3 and 4 MUST be performed atomically for iWARP connections. Failure to do this may result in race conditions." 3.6 Adjustments to Section 7 on iSCSI PDU Considerations 3.6.1 Adjustment to Section 7.3.9 Replace the first sentence in the first paragraph with the following: If the underlying transport is TCP, the Login Request PDUs and the Login Response PDUs are exchanged when the connection between the initiator and the target is still in the byte stream mode. J.Hufferd et. al. Expires March 2006 16 Internet-Draft Generalization of the iSER Specification Sept. 2005 3.7 Adjustments to Section 11 Security Considerations The following paragraphs are replacement paragraphs for this section. "When iSER is layered on top of an RCP Layer and provides the RMDA extension to the iSCSI protocol, the security considerations of iSER are the same as that of the underlying RCP Layer. For iWARP, this is described in [RDMAP] and [RDDPSEC]. Since iSER-assisted iSCSI protocol is still functionally iSCSI from a security considerations perspective, all of the iSCSI security requirements as described in [RFC3720] and [RFC3723] apply. If the IPsec mechanism is used, then it MUST be established before the connection transitions to the iSER- assisted mode. If iSER is layered on top of a non-IP based RCP Layer, all the security protocol mechanisms applicable to that RCP Layer is also applicable to an iSCSI/iSER connection. If iSER is layered on top of a non-IP protocol, the IPsec mechanisms as specified in [RFC3720] MUST be implemented at any point where the iSER protocol enters the IP network (e.g., via gateways), and the non-IP protocol SHOULD implement (optional to use) a packet by packet security protocol equal in strength to the IPsec mechanism specified by [RFC3720]. To minimize the potential for a denial of service attack, the iSCSI Layer MUST NOT request the iSER Layer to allocate the connection resources necessary to support RCP until the iSCSI layer is sufficiently far along in the iSCSI Login Phase that it is reasonably certain that the peer side is not an attacker, as described in sections 5.1.1 and 5.1.2." 3.8 Adjustments to Section 12.2 Informational References Add the following references: [IB] InfiniBand Architecture Specification Volume 1 Release 1.2, October 2004 [IPoIB] H.K. Chu et al, "Transmission of IP over InfiniBand", IETF Internet-draft draft-ietf-ipoib-ip-over-infiniband-10.txt (work in progress), March, 2005 J.Hufferd et. al. Expires March 2006 17 Internet-Draft Generalization of the iSER Specification Sept. 2005 4 IANA Considerations The following items will require registration with IANA before the resulting draft can be approved to become an RFC: None are known at this time. J.Hufferd et. al. Expires March 2006 18 Internet-Draft Generalization of the iSER Specification Sept. 2005 5 References 5.1 Informative References [DA] M. Chadalapaka et al., "Datamover Architecture for iSCSI", IETF Internet-draft, draft-ietf-ips-iwarp-da-03.txt (work in progress), June 2005 [DDP] H. Shah et al., "Direct Data Placement over Reliable Transports", IETF Internet-draft draft-ietf-rddp-ddp-04.txt (work in progress), February 2005 [IPSEC] S. Kent et al., "Security Architecture for the Internet Protocol", RFC 2401, November 1998 [iSCSI] J. Satran et al., "iSCSI", RFC 3720, April 2004 [iSER] M. Ko et. al., "iSCSI Extensions for RDMA Specification", IETF Internet-draft draft-ietf-ips-iser-04.txt (work in progress), July 2005 [iSNS] Josh Tseng et. al., Internet Storage Name Service (iSNS), IETF Internet-draft, draft-ietf-ips-isns-22.txt (work in progress), February 2004 [MPA] P. Culley et al., "Marker PDU Aligned Framing for TCP Specification", IETF Internet-draft draft-ietf-rddp-mpa-02.txt (work in progress), February 2005 [RDMAP] R. Recio et al., "An RDMA Protocol Specification", IETF Internet-draft draft-ietf-rddp-rdmap-04.txt (work in progress), April 2005 [SAM2] T10/1157D, SCSI Architecture Model - 2 (SAM-2) [SLP] M. Bakke et. al., "Finding iSCSI Targets and Name Servers by Using SLPv2", RFC 4018, April 2005 [TCP] Postel, J., "Transmission Control Protocol", STD 7, RFC 793, September 1981 [VERBS] J. Hilland et al., "RDMA Protocol Verbs Specification", RDMAC Consortium Draft Specification draft-hilland-iwarp-verbs- v1.0a, May 2003 J.Hufferd et. al. Expires March 2006 19 Internet-Draft Generalization of the iSER Specification Sept. 2005 6 Appendix This entire appendix is hereby included as Appendix B in the iSCSI Extensions for RDMA document [iSER]. 6.1 Architectural discussion of iSER over InfiniBand This section explains how an InfiniBand network (with Gateways) would be structured. It is informational only and is intended to provide insight on how iSER is used in an InfiniBand environment. 6.2 The Host side of the InfiniBand iSCSI & iSER connections Figure 2 defines the topologies in which iSCSI and iSER will be able to operate on an InfiniBand Network. +---------+ +---------+ +---------+ +---------+ +--- -----+ | Host | | Host | | Host | | Host | | Host | | | | | | | | | | | +---+-+---+ +---+-+---+ +---+-+---+ +---+-+---+ +---+-+---+ |HCA| |HCA| |HCA| |HCA| |HCA| |HCA| |HCA| |HCA| |HCA| |HCA| +-v+ +-v-+ +-v-+ +-v-+ +-v-+ +-v-+ +-v-+ +-v-+ +-v-+ +-v-+ |----+------|-----+-----|-----+-----|-----+-----|-----+---> To IB IB| IB | IB | IB | IB | SubNet2 SWTCH +-v-----------v-----------v-----------v-----------v---------+ | InfiniBand Switch for Subnet1 | +---+-----+--------+-----+--------+-----+------------v------+ | TCA | | TCA | | TCA | | +-----+ +-----+ +-----+ | IB / IB \ / IB \ / \ +--+--v--+--+ | iSER | | iSER | | IPoIB | | | TCA | | | Gateway | | Gateway | | Gateway | | +-----+ | | to | | to | | to | | Storage | | iSCSI | | iSER | | IP | | Controller| | TCP | | iWARP | |Ethernet | +-----+-----+ +---v-----| +---v-----| +----v----+ | EN | EN | EN +--------------+---------------+----> to IP based storage Ethernet links that carry iSCSI or iWARP Figure 2 - iSCSI and iSER on IB In Figure 2, the Host systems are connected via the InfiniBand Host Channel Adapters (HCAs) to the InfiniBand links. With the use of IB switch(es), the InfiniBand links connect the HCA to InfiniBand Target Channel Adapters (TCAs) located in gateways or Storage J.Hufferd et. al. Expires March 2006 20 Internet-Draft Generalization of the iSER Specification Sept. 2005 Controllers. An iSER-capable IB-IP Gateway converts the iSER Messages encapsulated in IB protocols to either standard iSCSI, or iSER Messages for iWARP. An [IPoIB] Gateway converts the InfiniBand [IPoIB] protocol to IP protocol, and in the iSCSI case, permits iSCSI to be operated on an IB Network between the Hosts and the [IPoIB] Gateway. 6.3 The Storage side of iSCSI & iSER mixed network environment Figure 3 shows a storage controller that has three different portal groups: one supporting only iSCSI (TPG-4), one supporting iSER/iWARP or iSCSI (TPG-2), and one supporting iSER/IB (TPG-1). | | | | | | +--+--v--+----------+--v--+----------+--v--+--+ | | IB | |iWARP| | EN | | | | | | TCP | | NIC | | | |(TCA)| | RNIC| | | | | +-----| +-----+ +-----+ | | TPG-1 TPG-2 TPG-4 | | 9.1.3.3 9.1.2.4 9.1.2.6 | | | | Storage Controller | | | +---------------------------------------------+ Figure 3 - Storage Controller with TCP, iWARP, and IB Connections The normal iSCSI portal group advertising processes (via SLP, iSNS, or SendTargets) are available to a Storage Controller. 6.4 Discovery processes for an InfiniBand Host An InfiniBand Host system can gather portal group IP address from SLP, iSNS, or the SendTargets discovery processes by using TCP/IP via [IPoIB]. After obtaining one or more remote portal IP addresses, the Initiator uses the standard IP mechanisms to resolve the IP address to a local outgoing interface and the destination hardware address (Ethernet MAC or IB GID of the target or a gateway leading to the target). If the resolved interface is an [IPoIB] network interface, then the target portal can be reached through an InfiniBand fabric. In this case the Initiator can establish an iSCSI/TCP or iSCSI/iSER session with the Target over that InfiniBand interface, using the Hardware Address (InfiniBand GID) obtained through the standard Address Resolution (ARP) processes. J.Hufferd et. al. Expires March 2006 21 Internet-Draft Generalization of the iSER Specification Sept. 2005 If more than one IP address are obtained through the discovery process, the Initiator should select a Target IP address that is on the same IP subnet as the Initiator if one exists. This will avoid a potential overhead of going through a gateway when a direct path exists. In addition a user can configure manual static IP route entries if a particular path to the target is preferred. 6.5 IBTA Connection specifications It is outside the scope of this document, but it is expected that the InfiniBand Trade Association (IBTA) has or will define: . The iSER ServiceID . A Means for permitting a Host to establish a connection with a peer InfiniBand end-node, and that peer indicating when that end-node supports iSER, so the Host would be able to fall back to iSCSI/TCP over [IPoIB ]. . A Means for permitting the Host to establish connections with IB iSER connections on storage controllers or IB iSER connected Gateways in preference to [IPoIB] connected Gateways/Bridges or connections to Target Storage Controllers that also accept iSCSI via [IPoIB]. . A Means for combining the IB ServiceID for iSER and the IP port number such that the IB Host can use normal IB connection processes, yet ensure that the iSER target peer can actually connect to the required IP port number. J.Hufferd et. al. Expires March 2006 22 Internet-Draft Generalization of the iSER Specification Sept. 2005 7 Author's Address John Hufferd Brocade 1745 Technology Drive San Jose, CA 95110, USA Phone: +1-408-333-5244 Email: jhufferd@brocade.com Mike Ko IBM Corp. 650 Harry Rd. San Jose, CA 95120, USA Phone: +1-408-927-2085 Email: mako@us.ibm.com Yaron Haviv Voltaire Ltd. 9 Hamanofim St. Herzelia 46725, Israel Phone: +972.9.9717655 Email: yaronh@voltaire.com J.Hufferd et. al. Expires March 2006 23 Internet-Draft Generalization of the iSER Specification Sept. 2005 8 Acknowledgments David Black EMC Corporation 176 South St. Hopkinton, MA 01748, USA Phone: +1-508-293-7953 Email: black_david@emc.com Mallikarjun Chadalapaka Hewlett-Packard Company 8000 Foothills Blvd. Roseville, CA 95747-5668, USA Phone: +1-916-785-5621 Email: cbm@rose.hp.com Mike Krause Hewlett-Packard Company 43LN 19410 Homestead Road Cupertino, CA 95014, USA Phone: +1-408-447-3191 Email: krause@cup.hp.com Alex Nezhinsky Voltaire Ltd. 9 Hamanofim St. Herzelia 46725, Israel Phone: +972.9.9717637 Email: alexn@voltaire.com Renato J. Recio IBM Corp. 11501 Burnett Road Austin, TX 78758, USA Phone: +1-512-838-3685 Email: recio@us.ibm.com Tom Talpey Network Appliance 375 Totten Pond Road Waltham, MA 02451, USA Phone: +1-781-768-5329 EMail: thomas.talpey@netapp.com J.Hufferd et. al. Expires March 2006 24 Internet-Draft Generalization of the iSER Specification Sept. 2005 9 Full Copyright Statement Copyright (C) The Internet Society (2005). This document is subject to the rights, licenses and restrictions contained in BCP 78, and except as set forth therein, the authors retain all their rights. This document and the information contained herein are provided on an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. The IETF takes no position regarding the validity or scope of any Intellectual Property Rights or other rights that might be claimed to pertain to the implementation or use of the technology described in this document or the extent to which any license under such rights might or might not be available; nor does it represent that it has made any independent effort to identify any such rights. Information on the procedures with respect to rights in RFC documents can be found in BCP 78 and BCP 79. Copies of IPR disclosures made to the IETF Secretariat and any assurances of licenses to be made available, or the result of an attempt made to obtain a general license or permission for the use of such proprietary rights by implementers or users of this specification can be obtained from the IETF on-line IPR repository at http://www.ietf.org/ipr. The IETF invites any interested party to bring to its attention any copyrights, patents or patent applications, or other proprietary rights that may cover technology that may be required to implement this standard. Please address the information to the IETF at ietf- ipr@ietf.org. J.Hufferd et. al. Expires March 2006 25