A. Wilson Internet-Draft Adaptec, Inc. Document: May 2000 Category: Informational Expires October 27, 2000 The SCSI Encapsulation Protocol (SEP) Status of this Memo This document is an Internet-Draft and is in full conformance with all provisions of Section 10 of RFC2026 [1]. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet- Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet- Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. 1. Abstract This Internet-draft presents a protocol developed at Adaptec, Inc. as part of a proof-of-concept demonstration that Gigabit Ethernet can serve as a direct replacement for traditional storage interconnects. The draft discusses the scope of problems that the protocol addresses, the key design decisions that were made, and then documents the protocol details. 2. Introduction Historically, storage to computer interconnects have been perceived as requiring very different technology from that used in the networks which connect computers to each other. But, this perception is changing as network technology reaches performance levels comparable to that of existing direct attached storage interconnects. This document presents a SCSI Encapsulation Protocol (SEP) developed as part of a project started a couple of years ago at Adaptec to demonstrate how network technology can be used to create a high performance storage subsystem. Since this session layer protocol runs on a reliable transport protocol such as TCP/IP [2][3], it allows internet wide access to storage using the SCSI [4] command set. However, Adaptec believes it will be most useful in Computer Room and Campus size storage networks using gigabit Wilson Informational 1 The SCSI Encapsulation Protocol (SEP) MAY 2000 Ethernet as the link layer. The project has demonstrated a storage area network prototype achieving wire-speed performance, and its protocol is being offered to the Internet community as an example of a relatively simple yet effective SCSI encapsulation protocol for networks. The document will first discuss the scope of this project and the requirements imposed on its design. A general architectural overview and discussion of design alternatives is supplied in Section 5, while Section 6 provides a description of the protocol. 3. Scope This project is intended to create a storage architecture which spans the range from desktop systems through computer room SANs to global storage networks, while utilizing standard network hardware as the interconnection fabric. The resulting network must have performance comparable to alternative storage network architectures, with lower prices and better manageability. For the desktop and small server market, the basic requirement is to connect local storage class devices to computer systems, especially when the devices are housed outside of the main computer case. For the large server systems, the requirements of storage subsystem sharing and interprocess communication over campus area networks are added. The large server system space can include cross country SCSI as well. To motivate this broad scope, three possible applications domains for IPS will be described. 3.1. Desktop Connectivity Desktop systems can use IPS with Ethernet hubs and switches to allow connection to a nearly unlimited number of targets. In addition, multiple desktop hosts could use IPS to share peripherals, effectively producing a desktop SAN configuration. While it will be simplest to have an IPS SAN separate from other communications networks, there is a significant appeal to having a single, combined function Ethernet connector on the desktop computer. For example a desktop system could include a connection to an external network. However, there are significant security issues with such an approach. One answer could be to add a router / firewall to isolate the internal IPS traffic from the external network while still letting other communications traffic through, while another could be to use IPsec which requires setup and management, but allows Internet wide use of IPS. 3.2. Computer Room Storage By far the most important use of IPS will be in computer room and campus storage networks. A typical installation would use IPS as the Storage Area Network (SAN), connecting individual servers and server clusters to pools of shared storage or backup media. For example, Wilson Informational 2 The SCSI Encapsulation Protocol (SEP) MAY 2000 several clusters could be connected to a larger backbone network which also includes a shared RAID subsystem and a tape backup subsystem. Each cluster could be on its own subnet, with its own, mostly private, storage. The large RAID box shared by the clusters, and the automatic tape backup system would also be on their own separate subnets. All of the hosts in each cluster and the tape backup subsystem can access all of storage as need through the IP router. The backup host would determine which files need to be backed up, and could then set up 3rd party copies between the storage subsystems and the tape drives to cause the affected files to be written to tape. 3.3. Cross Country The computer room / campus SAN can be extended to communicate with storage devices housed in a geographically remote site for disaster recovery purposes. The remote site would contain disks which mirrored critical local site data, or archival storage devices. In both cases, most of the IPS traffic would be writes, which can be highly pipelined using SCSIÆs overlapped request feature to hide the long latencies induced by the long distances involved. 4. Requirements The following eight metrics are particularly important to the storage subsystem industry: 1. Low latency. Since applications are often blocked while accessing storage, storage subsystems strive to achieve latencies on the order of a couple hundred microseconds or less. With disk drive access times continuing to shrink, low latencies will become even more important. Even today, RAID boxes with large cache memories can often respond immediately to storage requests, placing a premium on low latency in the storage network. 2. High Bandwidth. Storage subsystem bandwidth needs are rapidly increasing. Storage architectures must meet the current needs, as well as have a growth path towards supporting future needs. 3. Relatively short distances in most cases. Because of the latency intolerance of most storage applications, distances between storage unit and host computers are typically less than one hundred meters. There are applications, such as remote backup that demand distances measured in kilometers, so ability to efficiently span a wide range of distances is highly desirable. 4. Data Integrity. For most storage applications, the data must be delivered completely and without corruption, though there are a few applications where some corruption is tolerable. 5. Low CPU Utilization. Typical storage systems can offload much of the processing, and put read data directly into user buffers, Wilson Informational 3 The SCSI Encapsulation Protocol (SEP) MAY 2000 avoiding any data copies. Storage system purchasers expect the resulting low level of CPU utilization in any candidate products. 6. Easy management by system administrators. System administration complexity is becoming a major concern with emerging storage networks. 7. Cost competitive with other storage network technologies. Many storage applications are quite cost sensitive. Any alternative storage technology will have to have an end user price similar to older technologies of similar capabilities. 8. Needs to support interprocess communication paradigms. A likely use for IPS will be in cluster configurations, where supporting both storage and IPC will reduce system cabling requirements. The need for low latency (#1) is easily met using todayÆs Ethernet switches and switching routers. Typical values for latency are 3-20 microseconds for commercially available Gigabit Ethernet switches. Even with several switches and a couple hundred meters of cable, the round trip delay will be less than 100 microseconds in the absence of congestion. These values are adequate for todayÆs storage applications. For longer distances, the speed of light will add unavoidable latency, but some applications (e.g. remote mirroring) can tolerate it. The bandwidth requirements imposed by future disk drives (#2) are difficult to predict, since they depend on advances in recording density and mechanical speed of the disk drives, as well as the access patterns of future applications. However, if historical improvement rates are assumed, applications which access data sequentially, or in very large random reads, are projected to nearly saturate a gigabit interconnect with a single drive by 2005. On the other hand, transaction processing applications will still be limited by disk access time to about 350 IOPS per drive, or one to five megabytes a second depending on request size. Even at five megabytes a second, over 20 drives could be supported by a single gigabit link. Thus, gigabit links should be adequate for small server installations, while large systems will need a mixture of 1 and 10 gigabit links. For large systems with external RAID boxes, the new 10 Gigabit Ethernet will be required. For clusters, and also for RAID management, some Interprocess communication may be required. Such communication doesnÆt require large bandwidths, but does require very low latencies (the lower the better) and efficient transport of short packets. This argues for a low latency, low overhead transport protocol, but also argues for QOS and priority features such as found in IP version 6. The typical distances encountered with todayÆs storage systems (#3) can be met with 1000 BaseT Ethernet. In addition, since SEP runs on standard networking components, there are no inherent distance Wilson Informational 4 The SCSI Encapsulation Protocol (SEP) MAY 2000 limitations, opening up the possibility of direct access to even globally distributed storage resources. In networks, error free and complete data transfer (#4) is provided by transport protocols, in particular by TCP on the internet. The TCP also provides an end-to-end checksum to guard against errors introduced in transit which have not otherwise been detected, for instance in unprotected data paths. For critical data storage, a CRC, which would provide better protection, could be added at the SEP level (Section 5.5.3). In order to equal the low CPU utilization of current host adapters (#5), the whole network stack will have to run on the IPS host adapter, ideally with special purpose hardware. To keep the hardware from becoming too complicated and expensive, a simple SCSI encapsulation protocol is required, such as described in this document. The manageability (#6) issue consists of several parts. One aspect of this is the ability to do hot insertion and removal, with automatic discovery of the insertion or removal. Ethernet provides the hot insertion and removal portion, but additional technology must be supplied for automatic discovery. Existing discovery protocols, such as the Service Locator Protocol(SLP)[5] proposed for use on the Internet or other SAN management protocols may suffice. Interfacing to existing storage management systems would also provide storage partitioning, external RAID management, and other services. Another aspect of manageability is the use of the Internet Protocol (IP). By including IP, an Ethernet based storage network can take advantage of a wide variety of network management software already developed for corporate Intranets. The IP layer also provides QoS and bandwidth management options, important to configuring an efficient storage network. Something like 80% of the storage market consists of small to medium size systems, where cost is a major issue. To be accepted in this market, a new storage network must have price points comparable to the existing storage solutions, parallel SCSI and Fibre Channel, as indicated by requirement 7. The use of off-the-shelf Ethernet components will help, but low cost, full bandwidth single chip implementations of host adapters and target controllers must be possible. Currently many generic cluster systems (those that donÆt rely on System Area Networks, which are mostly proprietary), use TCP/IP for their Interprocess Communication Traffic (IPC). Because a network based storage protocol can co-exist with other network traffic, the current TCP/IP based IPC could simply share the storage network with storage traffic, providing an easy way to meet requirement #8. Wilson Informational 5 The SCSI Encapsulation Protocol (SEP) MAY 2000 5. SEP Architecture The SEP architecture is designed to address the requirements of desktop, small server, campus wide and larger storage area networks. Since the most likely uses will be in the server and campus SAN environment, the architecture is focussed on providing a cost and performance competitive solution in those spaces. With typical server applications, CPU utilization by the host processors is a particularly big concern. In order to keep host CPU utilization at the same low level achieved by traditional storage adapters, the entire stack, from SEP on down must be processed on the host adapter. Partly this is because the most common network, Ethernet, has a maximum frame size of 1500 bytes, with header processing required for each frame. Since most storage transfers are for larger amounts of data, they must be broken into multiple 1500 byte frames, suffering processing overhead on each frame which would be absent from other storage interconnects. But the biggest contributor to CPU utilization is the network buffer to user buffer copy required on storage read operations. Since the ultimate location of the received data is not known until the packet headers have been processed, the normal approach of a simple NIC combined with main CPU header processing, cannot place the data directly into the user buffer, but must copy it there using the main CPU. This typically requires two to three times the CPU cycles of the header processing. While protocol processing can be done by a microprocessor on the host adapter, the best cost/performance will be achieved with ASIC based processing. To facilitate ASIC implementation, the SEP protocol has been kept as simple as possible, consistent with the twin goals of good performance and reliable delivery. Using Ethernet also keeps costs low, as it is the most widely deployed LAN technology and Ethernet components enjoy cost benefits that come with high volume. In the long run, IPS should cost less than current storage subsystems whose components are only manufactured for their specific storage technologies. There are also cost benefits from being able to share infrastructure with the traditional networking services, or at least using the same type of equipment and network management utilities in a separately configured storage infrastructure. This section will discuss a number of architectural aspects of SEP, including the general approach and several key issues. Some of the reasons for selecting particular mechanisms will be included. Section 6 will specify the actual protocol used in AdaptecÆs studies. 5.1. Encapsulation Approach Wilson Informational 6 The SCSI Encapsulation Protocol (SEP) MAY 2000 In order to make maximum use of existing network technology, SEP operates at the session layer of the network protocol stack, just above the TCP/IP layer, as shown in Figure 1. It relies on the transport layer for correct delivery, and concentrates on multiplexing SCSI command, data and status information. Because TCP/IP can be used on any Link and Physical layer, IPS implemented with SEP can provide SCSI service over any network which has adequate performance. SCSI identifies the type of information (phase) on the data lines with a set of encoded control lines. For a serial approach, it is much more efficient to send a type code at the beginning of the new phase, which applies till the next phase. This type code is part of a header that is appended to the beginning of data from each phase. This header also contains the SCSI tag, so that data and status segments can be matched to the correct command. Transport protocols such as TCP provide a stream of bytes arriving in the same order as they were sent. Interpretation of those bytes is left up to the higher layer, such as SEP. Thus, the headers need to be identifiable by SEP using information within the data stream. A count of bytes till the next header will be placed in the current header, and SEP will use this byte count to determine the location of the next header. The byte count method is used because it is fairly simple to implement, efficient, and completely self-contained. However, there may be cases where you want the transport layer to identify the location of an SEP header, in which case the urgent pointer in TCP will be used to indicate when an SEP header is the first item in a transport packet. An implementation must be able to force all SEP headers to be at the beginning of a transport packet if either initiator or target insist, and should indicate when a header happens to be at the beginning of a transport packet in any case. To make extracting a SEP header more efficient, it will always be placed on a four byte boundary, by the method described in Section 6.1.2. Host Target +------------+ +-----------+ | Device | | OS Driver | | Controller | +-----------+ +------------+ | SEP | | SEP | +-----------+ +------------+ | TCP | | TCP | +-----------+ +------------+ | IP | | IP | +-----------+---------------------+------------+ | Internet | +----------------------------------------------+ Figure 1: Relationship of SCSI Encapsulation Layer to Network Stack Wilson Informational 7 The SCSI Encapsulation Protocol (SEP) MAY 2000 5.2. Connection Endpoints As discussed in Section 4, data must be delivered completely and without corruption. This means that the underlying network must deliver data bytes in original transmission order to the SCSI layer, and must have facilities to detect and recover from lost or damaged bytes. There is one additional requirement imposed by the SCSI architecture, and that is that all commands heading to a logical unit arrive in transmission order. The InternetÆs TCP accomplishes this reliable delivery of data by establishing a logical connection between two endpoints (often called a session) which allows it to track the progress of data transfer through a set of state variables. Each transport level session can be thought of as a pipe, which passes bytes from one endpoint to another. The TCP will allow multiple instances to exist, providing independent pipes between a variety of endpoints. To send a SEP command or data segment from one endpoint to another simply requires specifying the appropriate pipe. The simple addressing and flow control provided by a transport protocol for each of its instances would argue for a separate instance for each SCSI command. However, aside from the overhead required to establish and terminate an instance of a transport protocol, there is also the requirement for ordering between commands to the same logical unit. But, because the pipes are independent, there is no ordering relationship enforced between pipes and hence between commands. So, when multiple commands can be sent to one LUN (i.e. tagged command queuing), all the commands must flow over the same transport protocol instance. SEP includes multiplexing features to accomplish this. When using LUN bridges, a single pipe could be used to send all data and commands to all targets and LUNs behind the bridge. However, LUN information would have to be added to all SCSI command and data segments, and flow control and buffer management would be more difficult. For this reason, an architecture that attaches one transport protocol instance to each LUN will be developed. For devices that present only one LUN, a single protocol instance will suffice. For continuously active devices, such as disk drives, the instance will be created shortly after the devices have been identified and will remain until LUN removal or system shutdown. Other devices, such as a shared printer, may require that an initiator connect to it with a session only when it has data to print. Such a session will then be terminated after successful printing, freeing up limited session resources for use in accepting sessions from other initiators. Figure 2 shows the set of transport instances which could exist in a two host, six disk system. Note that two of the disks are parallel Wilson Informational 8 The SCSI Encapsulation Protocol (SEP) MAY 2000 SCSI and connected to a RAID box, two more are parallel SCSI and connected through a LUN bridge and the last two are IPS disks. Both hosts share the RAID box, each with a single transport session to the RAID controllerÆs single logical disk drive. The LUN bridge gets two, one for each SCSI disk attached to it. +--------+ +------------+ ___ | | Sessions | RAID | |___| | Host 1 | | Controller | ___ | --|------------------------|--(vol. a) | |___| | --|-------\ /------|--(vol. b) | ___ | --|------\ \ / | | |___| +--------+ \ \ / +------------+ \ \ / +--------+ \ \ / +------------+ ___ | | \ X | LUN Bridge | /-|___| | Host 2 | X \----------|------------|-/ ___ | --|----------/ \-----------|------------|-->=|___| | --|------------------------|------------|-/ ___ | --|------------------------|------------|----|___| +--------+ +------------+ Figure 2: Use of SCSI sessions to connect disk drives to hosts 5.3. SEP Functions The Storage Encapsulation Protocol (SEP) is basically a method for including phase and tag information with the data stream. This is a session layer protocol which rides on the TCP transport layer. The Storage Encapsulation Protocol performs the following three functions: 1. Identifying the command, data, status and message information segments. 2. Associating commands with their data and status 3. Providing flow control between host data buffers and target data buffers. The first function implies appending a header with a type field to the data associated with each phase. The method used to identify those headers will be covered in Section 6.1.1. With SEP many of the messages used by parallel SCSI will not be necessary because their purposes are accomplished directly by SEP headers or are not applicable to a network based approach. However, a message type is included to handle the few which remain, as specified in Section 6.2.4. The second function, association of commands with data and status is necessary for devices that can queue up several commands, such as disk drives. The SCSI approach of adding a small tag to commands, Wilson Informational 9 The SCSI Encapsulation Protocol (SEP) MAY 2000 data and status is used in SEP, where the tags are in a field of the SEP header. SCSI disk drives are capable of queuing multiple commands, allowing them to process the commands in the most efficient order. Unfortunately, disk drives tend to have a limited amount of data buffering memory, which can be overwhelmed by the data associated with queued write commands. Parallel SCSI will disconnect when this happens and request a write commandÆs data only when it is processing that particular write command. An encapsulation protocol which just sends write data immediately after a write command and relies on the transport or link layer protocol to prevent data buffer overrun, will block future commands and write data from reaching the drive, even though it might be more efficient for the drive to process them first. This is why SEP provides session layer flow control as its third function. Section 5.4 explores approaches to such flow control. The method actually used by SEP to provide data buffer flow control, which allows the drives to ask for write data just before they need will be described in Section 5.4. 5.4. Target Buffer Management In traditional storage systems, hosts pre-allocate buffer space and hence initiators have no buffer management issues, but targets have limited space and need some form of high level flow control to prevent buffer overruns. The high level flow control implemented by SEP is called Get SCSI Data and allows the target to request discrete amounts of data from the host adapter (or stop a stream of data that is already being sent). The command is implemented symmetrically, so a host adapter can use it to meter a stream of data as well, should that become necessary. In the normal storage model, initiators are installed in hosts, with large amounts of memory. The hosts know how much buffer space will be needed by each command and can (and usually do) pre-allocate the space. Thus, Initiators can guarantee to source and sink all the data required by outstanding commands. However, targets often have very small buffers, and though the storage media will eventually be able to accept all of the write data being sent to it, there is not enough buffer space to store the data until the media is able to accept it. The transport protocol could be relied upon to prevent overflow of the buffers, however this stops all information from reaching the target, which can create problems, especially in a target that supports tag queuing. This problem is not unique to SEP, but exists with SCSI over Fibre Channel (FCP) and parallel SCSI. Both Fibre Channel and parallel SCSI have transport level flow control (RREADY in Fibre Channel and REQ/ACK in parallel SCSI). In addition, Fibre Channel uses FCP_XFER_READY commands to provide Host Buffer to Target Buffer flow control while Parallel SCSI uses disconnects for that purpose, in addition to their use in enabling SCSI bus multiplexing. That is, in parallel SCSI, if a Target is operating in Write Cache Enabled mode, Wilson Informational 10 The SCSI Encapsulation Protocol (SEP) MAY 2000 and its Cache fills up with write data, it will simply disconnect until it again has space in its write cache. In an IPS system the initiator may want to send many commands to the target, with the intent that the target will order them so as to maximize the transfer rate to the media. Thus, the commands will not be executed in the same order as they are sent. If many of them are writes, and the initiator sends the data along with the write command, the targetÆs buffer could fill up. The transportÆs packet buffers would then fill up and it would stop receiving any additional commands and data from the host, until some of the already received data for already received write commands was written to the media. What all of this means is that we need a mechanism to allow the target to separately manage its data buffer resources, instead of using the transport layer flow control for that purpose. Two possible approaches will be discussed and then compared. The safest approach is for the target to be in complete control of write data transfers. In parallel SCSI this is achieved by allowing the target to disconnect from the initiator whenever it is offered more write data than it can handle. In Fiber Channel, a solicited information unit request is used to request write data from the initiator. The P1394 architecture goes even further, putting the target in complete control of the data transfers by passing it a scatter/gather list and requiring it to fetch write data through physical read commands to the host system memory. For SEP, we propose to implement a version of the Fiber Channel model for target controlled write data transfer which will be called the SCSI Data Request. If an IPS target has a lot of buffer memory, it may want to decrease initial latency and improve cable bit efficiency by sending some or all of the write data along with the write command. If the target later discovered that it was running out of buffer space it could send the equivalent of an XOFF command to the Initiator. When buffer space again permitted, it would send an XON to re-start the flow of write data. The same SCSI Data Request command can be used for the XOFF and XON functions. To stop the flow, the target would send a SCSI Data Request command with the byte offset pointing to the last byte it will be able to accept. The initiator will attempt to stop sending at the specified byte offset, though it may be unable to because further data has already been committed to the transport layer for sending. If it is not able to stop at the specified byte offset, it will stop as soon as possible, and reset its internal byte pointer to the stop value, from where it will continue sending when asked to resume transmissions by a future SCSI Data Request (acting as an XON). This last feature will let it reset its scatter/gather list pointers, a potentially time consuming task, before receiving the XON. Wilson Informational 11 The SCSI Encapsulation Protocol (SEP) MAY 2000 To restart transmissions, a SCSI Data Request command with the byte offset pointing to the last byte the receiver is now able to accept, which presumably is further along in the stream than were the stream was previously stopped will be sent by the target. If the amount is more than is left to send, the transmission will stop when all data requested by the original write command has been sent. The target could set the stop point according to the size of its available data buffers, and effectively transition to the safe mode, or it could specify a larger stop point and use an XOFF later on if necessary. The conservative approach of requesting each chunk of write data from the host is probably best suited to targets with relatively small buffers, such as disk drives. If write caching is enabled, the disk could still ask for all the write data as soon as it determined that it had adequate space, overlapping the fetch of the write data with other operations, such as seeking. For devices with large buffers, such as RAID boxes, the more aggressive approach of sending the write data with the command would probably work best. If the RAID box becomes overwhelmed, it can still cut off the transfer with the SCSI Data Request command. 5.5. Other Issues 5.5.1. SCSI Messages For the most part SCSI messages are not needed, either because the information they contain is built into the SEP header or because they are not applicable to a serial interconnect. However a message system is still provided for the few messages that might need to be transported between the host and the target. The SCSI messages which are useful to send are the Abort, Abort Tag, Bus Device Reset, Initiate Recovery, Message Reject, Release Recovery, and Terminate I/O process. These messages are encapsulated in a message type SEP segment, and as such can be up to 64KB long. Because of this extended length option, other uses for the message system may be developed in the future. 5.5.2. Third Party Commands The SCSI-3 specification allows a number of third party commands, such as copy. These commands are sent from an initiator to a target, and cause the target to perform some operation with another target. With SEP, the initiator will first open a session with one of the targets, then instruct the target to open a session with the other target through a special SEP command. Finally the third party operations will be sent to the first target, which will use the session it has established with the second target to actually perform the commands. 5.5.3. Segment CRC Wilson Informational 12 The SCSI Encapsulation Protocol (SEP) MAY 2000 While many link level protocols include a CRC with each packet, intermediate routers and switches may not have full error checking on their data paths, leading to the possibility of undetected data corruption. The TCP includes a sixteen bit checksum in its header to detect such corruption, but checksums are not as strong a detection mechanism as CRCs. To protect critical data, an optional segment CRC is defined, which appends a four byte CRC onto every SEP segment. This CRC covers the SEP header as well as the associated data. 6. SEP Details 6.1. SEP fundamentals Command, status, message and data information is encapsulated as SEP segments with an SEP header at the beginning. The SEP header is a fixed length of eight bytes, as defined in this section. The information included with individual segment SEP types will be shown in Section 6.2. 6.1.1. Header Figure 3 shows the SEP header. 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Type | Flags | Tag | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Reserved | SEP Segment Length | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 3: SCSI encapsulation protocol header Type: These are the SCSI encapsulation segment types which indicate what kind of data is following the header. The following types have been defined: 0x1 Simple Tagged Command 0x2 Head of Queue Tagged Command 0x3 Ordered Tagged Command 0x4 SCSI Data 0x5 SCSI Status 0x6 SCSI Message 0x8 SCSI Data Request 0x9 SCSI Set Data Pointer 0x10 Connect & Negotiate 0x11 Negotiation Response 0x12 Third Party Open 0x13 Third Party Open Response 0x14 Third Party Close 0x15 Third Party Close Response Wilson Informational 13 The SCSI Encapsulation Protocol (SEP) MAY 2000 Flags: These are the SCSI encapsulation flags. Two flags are currently defined across multiple segment types: 0x80 Command Complete û no more data to be sent in this direction. 0x40 Good Status û status is good, so status segment wonÆt be sent. The range of 0x01 to 0x08 are reserved for segment type specific information. Flag usage will be further defined with each type for which they are valid. Tag: The same tag used in Parallel SCSI to associate data, messages and status with commands. Always contains valid information, even if only one outstanding command is allowed for this session. SEP Segment Length: The length in bytes of actual data in the segment. Additional pad bytes may be inserted between the end of the data and the next header, as discussed in Section 6.1.2, and a trailing CRC may also be inserted, as discussed in Section 5.5.3. Since all headers will have at least one byte of ôdataö, a zero in the length field will mean 65536. When transferring more than 65536 bytes, additional data headers will be required, however it is recommended that headers be spaced closer together, so that new commands can be sent to the target in a timely fashion. 6.1.2. Padding to four byte boundaries Hardware implementation of a protocol such as SEP can be greatly simplified if all fields are aligned on four byte or larger boundaries. Since simplified hardware results in cheaper and faster implementations, SEP preserves a four byte alignment. To make sure that all SEP headers and segments start on four byte boundaries, the payload in an SEP segment will be padded with zeros to the next byte boundary that is divisible by 4. The Next SCSI Header field will indicate the actual number of valid bytes, and the additional padding will be automatically applied and removed by SEP. An example of how a 6 byte CDB would be handled is shown in Figure 4. 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | + SEP Header, Type = 0x00, Length = 6 + | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | + Six Byte SCSI CDB +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | 0 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 4: Padding to four byte boundaries Wilson Informational 14 The SCSI Encapsulation Protocol (SEP) MAY 2000 6.1.3. SEP CRC If the optional SEP CRC is used, it is appended to the end of a segment, after any required pad bytes, as shown in . The length field in the header still refers to the amount of usable data in the segment, so the CRC constitutes an additional four bytes which must be added to the overall length till next header by the receiving SEP. 6.2. SEP Segment Formats 6.2.1. SCSI Command Segment 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | + SEP Header, Type = 0x00 - 0x02, Length = n + | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | \ n Byte SCSI CDB \ | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 5: SCSI Command Segment Format Three segment types (0x00, 0x01, 0x02) are used for SCSI Command Descriptor Blocks, which are followed by up to a 256 byte CDB payload as indicated in Figure 5. The tag field of a SEP header is always meaningful, so if an untagged command is desired, the simple tagged command will be sent. The following flag is valid: 0x80 Command Complete û If this is a command which does not send any data from the initiator to the target (e.g. a read command), this flag will be set. 6.2.2. SCSI Data segment 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | + SEP Header, Type = 0x04, Length = d + | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | \ d Byte data segment \ | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Wilson Informational 15 The SCSI Encapsulation Protocol (SEP) MAY 2000 Figure 6: SCSI Data Segment The SCSI Data segment (type: 0x04) will contain up to 65536 bytes of data, as shown in Figure 6. If the Command Complete flag is set, it indicates that this is the last Data segment in this direction. If Good Status is set on a read segment, it indicates that good status was received, and no status segment will follow. The following flags are valid: 0x80 Command Complete - Placed in the last SEP header of a data segment to indicate that this segment is the last one in this direction for this command. On a command which transfers data from an initiator to a target, the target can determine if an over- or under-run condition has occurred by comparing the actual amount transferred with the expected amount. If placed in a segment carrying data or status from a target to an initiator, it also indicates the SCSI command has completed. 0x40 Good Status û Placed in the last SEP header of read data (along with Command Complete) to indicate that good status was received. No status segment will follow. 6.2.3. SCSI Status 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | + SEP Header, Type = 0x05, Length = 4 + s + | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | SCSI Status | Reserved | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | \ s Byte Auto Sense data segment \ | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 7: SCSI Status and Sense Data Segment The SCSI Status segment (Figure 7, type: 0x05) has the status byte in the first transmitted byte of the 4 Byte minimum payload. Since a status segment, if sent, is always the last segment of a SCSI command, the Command Complete flag will always be set. If sense data is required, it will follow after the 4 bytes reserved for status and the length field will reflect the number of sense bytes plus the 4 byte status field. If no sense data is to be sent, the length field will be exactly 4. 0x80 Command Complete û always set. Wilson Informational 16 The SCSI Encapsulation Protocol (SEP) MAY 2000 6.2.4. SCSI Message 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | + SEP Header, Type = 0x06, Length = m + | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | \ m Byte message segment \ | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 8: SCSI Message Segment The SCSI Message segment (Figure 8, type: 0xB) simply has the message byte(s) as the payload. While we identified a couple of SCSI messages that we might want to send, we havenÆt yet implemented any. But, this segment type is available, and can handle messages of up to 65536 bytes in length. 6.2.5. SCSI Data Request 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | + SEP Header, Type = 0x08, Length = 8 + | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | + New Stop Pointer + | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 9: SCSI Data Request The SCSI Data Request segment (Figure 9, Type: 0xC) allows a device receiving data (usually a target) to control the time at which the data is delivered. The typical case occurs when write data is being sent to a target with inadequate buffer memory, where the device wants little or no data sent to it until it is ready to actually transfer the data to the media. The SCSI Data Request allows it to ask for appropriate size chunks of data at a more convenient time, rather than when the host adapter wants to send it. The SCSI Data Request sets a new value into a stop pointer, which is a byte offset from the beginning of the data stream for this command. Both the sending and receiving SEP layer will keep a current byte pointer indicating which byte they last sent or received. When the sending SEP layer receives a SCSI Data Request, it immediately updates its Stop Pointer, and checks to see whether Wilson Informational 17 The SCSI Encapsulation Protocol (SEP) MAY 2000 the current byte pointer is greater or less than the new Stop Pointer. If the current pointer is greater, it immediately stops sending data, and sets its current byte pointer back to the new value of the stop pointer. If the current pointer is less, it continues sending until the current pointer equals the stop pointer. If the SEP layer has stopped sending data because its data pointer was equal to the stop pointer, it will resume sending data from its current data pointer and continue up to the new stop pointer. Thus, the SCSI Data Request can be used to regulate host buffer to target buffer data flow in three ways: by not allowing the host to send write data until explicitly requested by the target, by allowing the host to send some initial write data then explicitly requesting the rest by the target, or by allowing the host to send write data immediately after the write command until stopped by the target. In the first and second approaches the SCSI Data Request is being used as an explicit data fetch mechanism, while in the third approach it is being used as an XOFF / XON signal. In the explicit fetch method the target will determine how much data it can initially accept and negotiate that amount, which may be zero, with the initiator. When the target is ready to receive the rest of the data, it will send a SCSI Data Request with a stop value equal to the number of bytes it can next accept. As additional data buffers become available, the target can increase the stop pointer by the amount of the additional buffers, by sending additional SCSI Data Requests with the new stop value. The SCSI Data Request acts immediately to increase the stop pointer, allowing transmission to continue uninterrupted. In the XOFF / XON approach, the target will normally be able to store all the delivered data, but if it should run out of buffer space, it will stop the delivery by sending a SCSI Data Request with a stop point set to the offset of the last byte it can accept. Any additional data received will be discarded. The target will later re-enable data flow by sending a SCSI Data Request with a higher stop point. 6.2.6. SCSI Set Data Pointer 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | + SEP Header, Type = 0x09, Length = 8 + | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | + New SCSI Data Pointer + | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 10: SCSI Set Data Pointer Wilson Informational 18 The SCSI Encapsulation Protocol (SEP) MAY 2000 The SCSI Set Data Pointer (Figure 10, Type: 0xDS) allows devices and LUN bridges to emulate the Modify Data Pointers, Save Data Pointers, and Restore Data Pointers messages from Parallel SCSI in the host adapter to target direction. Because network communications tend to be highly pipelined, the targetÆs byte count and the host adapterÆs will normally not be in synch, so a direct offset from the beginning of the commandÆs data stream will be used. A bridge can properly emulate the effect of Data pointer messages from SCSI targets by keeping local count of their progress through the data stream. This command will immediately update the senderÆs Data Pointer, so it should not be notified by the receiver until all desired data has been received from the last SCSI Data Request. The SCSI Set Data Pointer is also used to separate streams of SCSI data when there is a non-sequential point in the data flow. For example, if SCSI Data Request is used to stop data flow, and data beyond the stop point had already been sent, the sender will precede the next chunk of requested data with a SCSI Set Data Pointer since the new chunk will be for earlier bytes in the stream. The receiver will discard the extra data over the requested stop point until it receives the SCSI Set Data Pointer which matches the previous stop point. Because SCSI only transfers data in one direction at a time per command, there is no ambiguity in the two uses of SCSI Set Data Pointer. 6.2.7. Connect & Negotiate 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | + SEP Header, Type = 0x10, Length = 16 + | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | + SCSI LUN + | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | + Desired Parameter Values + | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 11: SCSI Connect and Negotiate The Connect & Negotiate SEP header is followed by a 16 byte SEP segment which specifies a LUN and Desired SEP level parameters for this session, as shown in Figure 11. This Information must be sent by an initiator at the beginning of a SEP session before any SCSI transfers are attempted. Attempts to send SCSI commands before the Connect and Negotiate command has been successfully acknowledged will fail. The Parameter values field (Figure 12 )is used to indicate SEP options which the Initiator would like to have enabled. Wilson Informational 19 The SCSI Encapsulation Protocol (SEP) MAY 2000 Generally this would be the set of features which the Initiator supports. The corresponding negotiation response will contain the set of features which the target is also able to support. 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | |B|A|N|N| | Reserved |A|S|R|W| | |S|N|L|L| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | MRPR | MWPS | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 12: SEP Session Negotiation Parameters Parameter Values include: BAS Every Ethernet Frame starts with a SEP header ASN Every SEP header starts a new Ethernet Frame NWL Allows Initiator to send as much data as it wants with a write by disabling the MWPS. NRL Allows the Target to send as much data as it wants on a read reply by disabling the MRPR. MWPS Maximum data bytes sent by the Initiator on a Write operation without receiving a Get SCSI Data command from the target (Can be zero). MRPR Maximum data bytes sent by the Target on a Read operation without receiving a Get SCSI Data command from the Initiator (Can be zero). 6.2.8. Negotiation Response 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | + SEP Header, Type = 0x11, Length = 8 + | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | + Required Parameter Values + | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 13: SEP Negotiation Response The Negotiation Response header is followed by an eight byte SEP segment (Figure 13) which specifies the SEP level parameters which will be used for this session. This Information is sent by a target in response to a Connect & Negotiate command. The Parameter values field is used to indicate SEP options which the session will use, as determined by the Target from its own supported feature set and that Wilson Informational 20 The SCSI Encapsulation Protocol (SEP) MAY 2000 desired parameter values supplied by the Initiator. In other words, the Parameter values represent the largest set of features supported by both the Target and the Initiator. 6.2.9. Third Party Open 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | + SEP Header, Type = 0x12, Length = 24 + | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | + + | | + ip address of device to open 3rd party session with + | | + + | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | + SCSI LUN + | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 14: SEP Third Party Open The third party open command instructs a target device to open a session between itself and the device specified by the IP address and LUN fields. A flag bit in the SEP header indicates whether an IP v. 4 address is included or an IP v. 6 address. 0x01 IP address is version 6 if flag asserted. 6.2.10. Third Party Open Response 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | + SEP Header, Type = 0x13, Length = 8 + | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | + 3rd party identifier or failure code + | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 15: SEP Third Party Open Response If the third party open is successful, a response segment is returned, which includes an eight byte session identifier. This Wilson Informational 21 The SCSI Encapsulation Protocol (SEP) MAY 2000 identifier can be used by the initiator in subsequent third party commands (such as copy) in the identifier field. If the third party open fails, a flag bit in the SEP header is asserted to indicate failure, and the eight byte field contains information about the failure. 0x01 Third party open failed. 6.2.11. Third Party Close 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | + SEP Header, Type = 0x14, Length = 8 + | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | + 3rd party session identifier + | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 16: SEP Third Party Close Once the third party session is no longer needed (i.e. the initiator has no more third party commands to execute), the initiator can close the session by sending this SEP segment with the appropriate third party session identifier. 6.2.12. Third Party Close Response 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | + SEP Header, Type = 0x15, Length = 8 + | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | + Close Status + | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 17: SEP Third Party Close Response Once the target has closed the third party session, it will respond with a close segment, with any status in the close status field. 7. Security Considerations Currently this protocol is assumed to be used on a private, storage area network, hence security issues have not been addressed in this document. Wilson Informational 22 The SCSI Encapsulation Protocol (SEP) MAY 2000 8. References 1 Bradner, S., "The Internet Standards Process -- Revision 3", BCP 9, RFC 2026, October 1996. 2 Postel, J., ôTransmission Control Protocol û DARPA Internet Program Protocol Specificationö, STD-007, RFC-793, September 1981. 3 Postel, J., ôInternet Protocol û DARPA Internet Program Protocol Specificationö, STD-005, RFC-791, September 1981. 4 ôAmerican National Standard for Information Technology - SCSI-3 Architecture Modelö, ANSI X3.270 û 1996. 5 Guttman, E. et al, ôService Location Protocol, Version 2ö, RFC- 2608, June 1999. 9. Acknowledgments The Author would especially like to thank Paul von Stamwitz of Adaptec for the many comments, ideas and suggestions exchanged with him during the development of this protocol. 10. Author's Addresses Andrew Wilson Adaptec, Inc. 691. S. Milpitas Blvd. Milpitas, CA 95035 Email: dwilson@corp.adaptec.com Wilson Informational 23 The SCSI Encapsulation Protocol (SEP) MAY 2000 Full Copyright Statement "Copyright (C) The Internet Society (date). All Rights Reserved. This document and translations of it may be copied and furnished to others, and derivative works that comment on or otherwise explain it or assist in its implementation may be prepared, copied, published and distributed, in whole or in part, without restriction of any kind, provided that the above copyright notice and this paragraph are included on all such copies and derivative works. However, this document itself may not be modified in any way, such as by removing the copyright notice or references to the Internet Society or other Internet organizations, except as needed for the purpose of developing Internet standards in which case the procedures for copyrights defined in the Internet Standards process must be followed, or as required to translate it into. Expires October 27, 2000 Wilson Informational 24