Internet-Draft Alexander Bogdanov draft-bogdanov-umsp-rfc3018-update-00.txt August 2003 Expires: February 2004 Unified Memory Space Protocol Status of this Memo This document is an Internet-Draft and is in full conformance with all provisions of Section 10 of RFC2026 [1]. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet- Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. Abstract UMSP gives the universal environment for integration of any Internet devices in one computing system. UMSP controls the distributed execution of applications at a level of network connections, gives the access to applications memory on remote nodes, and provides the network interaction on basis of the Remote Application Procedure Call. The submitted document is updating of the RFC3018 specification. The difference is the independence of the protocol from the format and the length of network address, the interaction between IPv4 and IPv6 hosts, and the interaction between hosts with public and non-public network addresses. Conventions used in this document The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC-2119 [2]. Table of Contents 1 Overview...........................................................3 2 The Differences from RFC3018 Specification.........................5 3 Terminology and Abbreviations......................................6 Bogdanov Expires: February 2003 [Page 1] Internet-Draft Unified Memory Space Protocol August 2003 4 The Model..........................................................6 4.1 The Computing Model............................................6 4.2 Executive System...............................................7 4.3 Virtual Network Addresses......................................8 4.4 Operations with Pointers.......................................8 4.5 Job Control....................................................9 4.6 Network Model.................................................11 5 Common Header Format..............................................12 6 Connections Control...............................................15 6.1 Connection Establishment......................................15 6.1.1 Establishment of Primary Connection........................16 6.1.2 TCP Use....................................................18 6.1.3 The New Job Registration on JCP............................18 6.1.4 Job Initiation on the Host.................................20 6.1.5 "Host-Host" and "Host-JCP-Host" Connection Establishment...21 6.2 Termination of Connection.....................................25 6.2.1 Termination of "Host-Host" Connection......................26 6.2.2 Termination of "Host-JCP-Host" Connection..................26 6.2.3 Termination of "JCP-Host" Connection.......................26 6.2.4 Job Termination............................................27 6.2.5 Termination of Primary Connection..........................27 6.2.6 The Rules of Connection Identifiers Reuse..................28 6.2.7 Notification about Termination of "JCP-Host" Connections...29 6.2.8 Termination Codes..........................................29 6.3 Use of Several JCP............................................30 6.4 Format of Connection Control Commands.........................30 6.4.1 INIT.......................................................30 6.4.2 INIT_ACK...................................................31 6.4.3 CONNECT....................................................31 6.4.4 CONNECT_ACK................................................32 6.4.5 COOKIE.....................................................33 6.4.6 CONTROL_REQ, JOB_REQ.......................................33 6.4.7 CONTROL_CONFIRM, JOB_CONFIRM...............................35 6.4.8 BIND.......................................................36 6.4.9 BIND_FWD...................................................38 6.4.10 BIND_ACK..................................................40 6.4.11 BIND_ACK_JCP..............................................41 6.4.12 DISCONNECT, ABORT.........................................43 6.4.13 DISCONNECT_ACK............................................44 6.4.14 TERMINATION_NOTIF.........................................44 6.4.15 ADDR_LIST.................................................45 7 Transfer of VM Instructions.......................................45 7.1 Format of Data Transmission Commands..........................48 7.1.1 DATA.......................................................48 7.1.2 ACK........................................................48 7.1.3 CONGESTION.................................................49 8 Activity Control..................................................50 8.1 ACTIVITY_REQ..................................................50 9 Job Lifetime......................................................51 9.1 LIFE_TIME_REQ.................................................51 9.2 LIFE_TIME_SET.................................................52 10 Identification of VM Type.......................................52 Bogdanov Expires: February 2003 [Page 2] Internet-Draft Unified Memory Space Protocol August 2003 10.1 VM_REQ.......................................................54 10.2 VM_NOTIF, VM_NOTIF_JCP.........................................54 11 Security Consideration..........................................56 References..........................................................56 Author's Address....................................................57 Full Copyright Statement............................................58 1 Overview UMSP is the network connection-oriented protocol of the application layer. It is intended for the organization of the distributed virtual memory and for the control of the distributed execution of applications. The protocol is based on a memory address, which includes the network host address and the local (usual) memory address on this host. UMSP offers two interdependent ways of interaction between hosts: (1) The immediate access to memory of applications on remote hosts. For memory, simple allocating/deallocating and reading/write operations or more complex specialized operations can be provided. (2) The Remote Application Procedure Call. The program loads its own code on the remote node. API of the executive system on this remote node and any application objects are available for this code. The application can call its own code on any node. The call parameters and the result are sent by the unstructured binary buffer. The submitted method allows solving the following tasks: O The application can create any object on a remote host and address to it by usual pointers and method calls. The host address is only the pointer attribute. O Interaction between hosts in application is simple access to variables and a call of application functions. The application does not call the functions of network interface at the level of the program source text. The compiler and/or the executive system provide the network interaction. The source text of objects does not differ for local and remote execution. The developer can consider the distributed system as one computer. O The logic of network interaction on the application layer doesn't require the protocol specification and can change dynamically. It cannot be achieved at a classical network interaction. O The network exchange is optimized for the decision of application tasks. Only necessary results are sent over the network. Bogdanov Expires: February 2003 [Page 3] Internet-Draft Unified Memory Space Protocol August 2003 O If the application must cooperate only with API of the executive system and immediate interaction with the other applications is not required, there is no need in data presentation layer. The functions call parameters and the result are sent by usual binary stack. O The uniting on the executive system level appreciably simplifies applications logic and resource management. If the application does not know about network presence, complicated network technologies are not necessary for the application. The distributed functions of the application level can be realized as usual API. The uniting of network computers in one computing system offers a maximum level of simplification at creation of distributed systems. In systems with UMSP support, it is possible to delimit the following layers: (1) The transport layer. UDP and TCP use is specified in this document. This layer is lower. (2) The UMSP layer. This layer provides a network interaction and an application control at a level of network connections. The virtual network addresses are distributed, and the initiation and termination of jobs are carried out on this layer. The UMSP layer realizes error-free non-duplicated acknowledged data transfer and traces shutdown or reboot of nodes and separate applications. (3) The executive system layer. First of all, these are virtual machines (VM) with byte-code. VM must provide pointers with the network address of node for working with UMSP. VM translates instructions with remote addresses into network commands for execution on remote host. Commands are sent by UMSP connections. VM controls memory of applications and executes code. (4) The application layer. User applications or autonomous programs are executed on this layer. These applications contain pointers with the network host address and the memory address. The operation repertoire with such pointers may be minimal (for example, only read and write), or it may be full- value pointers. VM type defines the operation repertoire. UMSP is universal technology for solving of home, corporate and scientific problems. The protocol may be realized in stationary and mobile computers of various types. UMSP can be used: O For the applications demanding simultaneous execution on several computers. Bogdanov Expires: February 2003 [Page 4] Internet-Draft Unified Memory Space Protocol August 2003 O In systems with complicated and dynamically changing logic of interaction between hosts. O For granting unused computing resources to the other computers. This way is absolutely protected from attacks, if only the processor time and the closed memory space are given for public access. O For interacting between the person and remote computer. Two variants or their combination are possible: (1) The remote computer loads its code at the client. This logic is similar to active Web pages. (2) The client loads its code at a remote server. O As a basis for the services on demand and the distributed object technologies. O For the remote control of simple and complicated devices. The remote device may independently carry out the base functions. Interaction with the control device is necessary for an execution of expanded functions only. O The protocol can give an infrastructure for self-organizing systems in the Internet. 2 The Differences from RFC3018 Specification The following tasks were set by development of new specification: O To provide interaction between nodes with public and non-public IP addresses. O To work with IPv6 addresses. O To simplify the specification without losing of basic protocol functions and network efficiency. The following changes were made to decide these problems: O 8-byte virtual network address is used instead of the real network address for identification of nodes. The Control Task IDentifier (CTID) [5] is chosen as a value of the virtual network address. This identifier has a unique value on Job Control Point (JCP) node. Any exchange between hosts begins from sending an initial package through JCP. JCP computes the real network addresses and sends a package to the addressee. If hosts are located in the same network addresses space, the subsequent exchange is carried out without JCP participation. Using of a third node at establishment phase of point-to-point connection is justified for systems with the centralized control at the application layer. One node carries out a Bogdanov Expires: February 2003 [Page 5] Internet-Draft Unified Memory Space Protocol August 2003 centralized control at the application layer and allocates virtual network addresses. O TCP use is inefficient as three nodes participate in establishment of network connection. Besides, UMSP does not require support of the ordered data flow. To decide these problems, the UDP use is specified in the document. O The VM layer is considered as an independent layer, non- included in UMSP. The format of VM network instructions is not specified in the document. Each VM type may use its own repertoire of network instructions. O The interaction between different VM types on UMSP layer is not provided. Nevertheless, applications on different VM may cooperate at the API level of the executive system. O The transaction functions, instructions fragmentation and work with objects have been deleted from the protocol. These functions must be realized on the VM layer if necessary. The submitted document is incompatible with the source specification RFC3018. 3 Terminology and Abbreviations Node - any computing device that implements UMSP. JCP - the node which carries out the centralized job control. Host - the node which is not JCP. One node can be simultaneously a host and JCP in different jobs. Virtual Machine (VM) - the common term for any system with UMSP support. It can be function library, a virtual machine, operational system or the hardware module. MTU - maximum packet size in bytes can be sent between adjacent nodes without fragmentation on lower layer. PMTU - minimum MTU of all path hops between a sender node and a receiver node. 4 The Model 4.1 The Computing Model It is based on the model of a cluster with the distributed virtual memory. Processors are connected over a standard network. Each processor has its own memory segment. All address space of memory is accessible to all processors. Memory addresses include the network address of the processor and the local memory address. The Bogdanov Expires: February 2003 [Page 6] Internet-Draft Unified Memory Space Protocol August 2003 instructions in the program may include the memory address of any processor. The instructions with memory addresses of this processor are executed on this processor immediately. The current processor sends request to the remote processor for an execution of the instruction with the memory address of the remote processor. In general case, it may be a reading/writing request or a control jump. The remote processor executes the request and sends a result. The UMSP layer does not provide a distributed swapping and a remote interlock of memory segments. For the application, the submitted cluster model can be considered as one computing system. The basic problem is the possibility of switching-off of any processor at serviceability preservation of the others processor in a cluster. In fact, any pointer in the application can become invalid at any moment of time. Traditional applications do not provide processing such situations. Change of exceptions processing and/or change of application architecture is required for porting of these applications to the distributed environment. 4.2 Executive System The basic protocol requirement is to control the operations with pointers. UMSP can be provided at the following levels: (1) At the libraries level of a standard programming language. For example, C++ can be used for realization. It is necessary to define the special pointer type with the overloaded set of all base operators. This pointer must additionally include the virtual network address and/or the reference to UMSP control object. The program source text must contain explicit syntax for creation of pointers to remote objects. The special code from library must be executed at operations with the distributed pointers. (2) At the compiler level. The compiler must create the pointers, which include the memory address and the node virtual network address. The compiler creates the code for a call of UMSP functions. Explicit syntax for operations with the distributed pointers in the program source text can be absent. (3) At the level of executive system. The executive system must consider pointers as special type of memory. This variant can be used in virtual machines with a byte-code. Pointers must include the network address of the node and the local address of memory. The executive system has the distributed virtual memory and executes all necessary operations for working with it. Explicit syntax for pointers to remote addresses in the program source text is absent. Bogdanov Expires: February 2003 [Page 7] Internet-Draft Unified Memory Space Protocol August 2003 The "executive system" term is equivalent to the "virtual machine" (VM) term in the text of this document. They mean any level, indicated above. 4.3 Virtual Network Addresses Virtual network addresses are allocated centrally on the Job Control Point (JCP) node. The virtual network address is 8 bytes in length. The host must set the connection with JCP, in order to be included in the distributed system. At establishment of connection, the nodes exchange with connection identifiers. The connection identifiers allocated on JCP are used as virtual network addresses. Use of virtual addresses solves the following problems: O Nodes with non-public network address and nodes with public network address can be included in system simultaneously. O If JCP node has IPv4 and IPv6 interfaces, the system can simultaneously include IPv4 and IPv6 hosts. All nodes of these networks are peer on the UMSP layer. In general case, JCP may unite networks of any type (WANs, LANs, with different protocols of network layer or with non-overlapping spaces of network addresses). O Pointers are fixed in length. Compilers can allocate static memory for variables without relation with definite network type. Two types of the virtual addresses are defined: (1) The public virtual network address. This address is replacement of the real network address. The host has one public address on each JCP. This address is not connected to certain job and must not be used in memory addresses. The public address may be indicated as the network address of the receiver in commands of an establishment of connections through this JCP. (2) The private virtual network address. The host has the separate private address in each job. This address may be used in memory addresses and in commands of establishment of connections with a binding to the job. The private address must not be used for an initiation of jobs. Each job has its own private addresses. Address spaces of different jobs are isolated. 4.4 Operations with Pointers The protocol does not work with pointers. Their format is defined by executive system completely. Presence of 8-byte virtual network address in the pointer is the only requirement. The following Bogdanov Expires: February 2003 [Page 8] Internet-Draft Unified Memory Space Protocol August 2003 special values of the virtual network address are defined ("?" indicates any hexadecimal digit): 0x????????00000000 - indicates the local address of this node. The assignment operator of this pointer carries out the following actions: (1) Simple copying must be carried out for the source of the pointer value located in local memory on this node. (2) For the pointer value received from other node, the virtual network address of the pointer must be set to value of the private virtual network address of the source. 0x????????FFFFFFF0 - 0x????????FFFFFFFE - reserved. 0x????????FFFFFFFF - indicates the invalid address (the address of a disconnected node). Any operation with the invalid address must result to exception. This value is NOT RECOMMENDED to use for unassigned pointers. Pointer values can be sent over a network in the binary buffer. The received pointer value must be assigned to a variable until sending to other node. Transit sending is not supposed. The executive system MUST control operations of pointer assignment. For any type variables, the pointer assignment operation MUST be executed only after an establishment of special connection between the object host and the pointer host. This connection must not be closed, if there is even one pointer related to it. The pointers counter is started from each side of connection to control pointers. Each connection side has its own pointer counter and does not synchronize its value with the opposite side. If the counter becomes zero, the request to end connection will be sent. If the counter is not zero on the other side of connection, connection will not be terminated. If connection has been closed emergency, the virtual network address of all pointers related with this connection must be set to 0x????????FFFFFFFF. The executive system must provide references between pointers and UMSP connection to assign this value. The protocol ensures detection of emergency reboot of the job on the remote node, but in this case, the new objects with the same addresses may be created on this node. 4.5 Job Control UMSP bases on model with the centralized control of the separate job. The control node is defined on the UMSP layer for each job. Job Control Point (JCP) carries out these functions. Job control is connected with allocation of virtual network addresses. The start node of the application can carry out JCP functions or it can be any other addressed node. JCP can have several interfaces with networks of different types and/or with different network address space. All nodes of these networks are peer on the UMSP layer. Bogdanov Expires: February 2003 [Page 9] Internet-Draft Unified Memory Space Protocol August 2003 The application must initiate UMSP job to start the distributed execution. One application has one UMSP job. The initiation of the UMSP job can be made simultaneously with application start or during its work (directly before the first operation with remote addresses). JCP allocates the private virtual network address to the host. Each JCP controls its closed space of the virtual network addresses. In each job one node has one private virtual network address. Address spaces of different jobs are nonintersecting on UMSP layer. After job initiation, the application can create objects on remote nodes and can execute operations with them. Memory for data and/or a code can be the objects. The first request to the remote host is sent through JCP. JCP carries out the following actions: O If this job has not been initiated on the destination host, JCP initiates the job. After that, JCP forwards the request. The response can be sent immediately to the source host without JCP participation. O If the job on the destination host has been already initiated by other host, JCP carries out simple request forwarding. The interaction between different jobs is not provided at UMSP layer. Nevertheless, jobs of one JCP can cooperate at a pointers level by API of the executive system. The exchange of pointers between jobs of different JCP is impossible. The host sends the job shutdown request to the JCP at the application shutdown. JCP sends the shutdown message to all other job hosts. If the host is not the job initiator, the job on this host may be terminated before common job termination. Normal termination means absence of any job objects on the host (for example, objects are deleted as a result of application work). The host is the initiator of such termination. If the job lifetime has expired or connection between JCP and host of the job initiator has been disconnected, JCP can be the initiator of common job termination. Bogdanov Expires: February 2003 [Page 10] Internet-Draft Unified Memory Space Protocol August 2003 4.6 Network Model The general scheme of interaction over a network is shown in the following figure: Node 1 Node 2 -------- -------- +--------------------+ +--------------------+ | user application 1 | | user application 1 | +-----------------------+ +-----------------------+ | user application N | | user application N | +--------------------+ +--------------------+ +----------------------------+ +----------------------------+ | VM | | VM | +----------------------------+ +----------------------------+ | | +--------------------------+ +--------------------------+ | | | | | +-----+ U M S P | | U M S P | | | JCP | | | | | +-----+ | +-------------+------------+ +-------------+------------+ | | +-----+-----+ +-----+-----+ | UDP/TCP | | UDP/TCP | +-----+-----+ +-----+-----+ | | | +-----------------/ | /------------------+ / | +-----+-----+ Node N | UDP/TCP | -------- +-----+-----+ | +------------+------------+ | +-----+ | | | JCP | U M S P | | +-----+ | +-------------------------+ The UMSP connections are used for data traffic. UDP is used on the lower level. TCP may be used for nodes with non-public network address. At establishment UMSP connection, each side allocates 8- byte identifier to the connection. All connection identifiers of one node for all jobs have the unique values in any moment of time. The allocation of identifiers on different nodes is independent. To initiate job on any host, the connection must be established between this host and JCP. The identifier of this connection on JCP Bogdanov Expires: February 2003 [Page 11] Internet-Draft Unified Memory Space Protocol August 2003 side is the private virtual network address of the host in this job. Different jobs use different connections. Termination of connection between JCP and host indicates job termination on the host. Termination of connection between JCP and the host of job initiator indicates common termination of job. In this case, JCP finishes all other connections of the job. "Host-host" connection is established for data exchange between VM and for support of operations with pointers. One "host-host" connection is established between two hosts in one job. Different jobs use different connections. The initial package to establish "host - host" connection is sent to JCP. This package can contain the following types of the address of destination host: O Public IP address. O Address in local network. In this case, JCP considers the destination situated in a local network of the source. O Public virtual network address. JCP carries out the following actions after receipt of initial packet: (1) Computes the real network address of the destination. (2) If connection between destination host and JCP is absent, such connection must be established. (3) The initial package forwards to the destination with the network address of the source host. If the source and destination are located in different spaces of network addresses, the real network address of the source is not indicated. If hosts are located in one space of network addresses, the subsequent exchange carries out without JCP participation. If hosts are located in different networks, the exchange is conducted through JCP. "Host-host" connection can be closed, if there are no pointers connected by this connection with objects of the remote host. Connection abort means loss of all pointers. Job end on the host terminates all connections "host-host" of this job. 5 Common Header Format UDP is used for sending of UMSP packages. Allocated IANA port is 2110. UDP packages must include the checksum. If the node has no public address and cannot use UDP, TCP connection by destination port 2110 can be used. The identical UMSP packet format is used for UDP and TCP. UMSP packet has the following common format: Bogdanov Expires: February 2003 [Page 12] Internet-Draft Unified Memory Space Protocol August 2003 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | + CONNECTION-ID + | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | ~ Commands ~ | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ CONNECTION-ID: 8 bytes Indicates the connection identifier allocated on the packet destination node. Commands: variable length One or several protocol commands. It may be control commands of the UMSP layer or VM instructions. One packet can include several commands. Each UMSP command has the following general format: 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |R|T| DATA-LENGTH | OPCODE | DATA-1 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | SEQUENCE-NUMBER | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | + SOURCE-VIRTUAL-ADDRESS + | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | + DEST-VIRTUAL-ADDRESS + | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | ~ DATA-2 ~ | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ R: 1 bit Reserved. This bit is analyzed only in the first command of packet. The value is set to 0. If this bit is set to 1, the packet must be silently discarded. T: 1 bit Bogdanov Expires: February 2003 [Page 13] Internet-Draft Unified Memory Space Protocol August 2003 Flag of transit sending. Used at an exchange between hosts through JCP. If T flag is set to 1, the command includes SOURCE-VIRTUAL-ADDRESS and DEST-VIRTUAL-ADDRESS fields. If it is set to 0, SOURCE-VIRTUAL-ADDRESS and DEST-VIRTUAL-ADDRESS fields are absent. The T flag can be set to 1 only in the first command of packet. All following commands of packet have T = 0. DATA-LENGTH: 14 bits Indicates full command length in 4-byte words. OPCODE: 1 byte Identifies the command opcode. The value of this field defines the purpose of command and format of the DATA-1 and DATA-2 fields. DATA-1: 1 byte This field contains the first data byte. SOURCE-VIRTUAL-ADDRESS : 8 bytes If T = 0, this field is absent. If T = 1, this field contains the source private virtual network address. DEST-VIRTUAL-ADDRESS : 8 bytes If T = 0, this field is absent. If T = 1, this field contains the destination private virtual network address. SEQUENCE-NUMBER: 4 bytes Indicates the command sequence number in connection. DATA-2: variable length This field contains the other data. The following ranges of OPCODE values are defined: 0 - 127 commands for obligatory processing 128 - 255 optional commands If the node knows the format definition for received OPCODE, command processing does not depend on range (obligatory or optional). If the node does not know the format definition for received OPCODE, packet processing on reception depends on range. If the command is obligatory, it must be discarded without further action. If the command is optional, the received sequence number must be included to cumulative confirmed number and the command must be discarded. The Bogdanov Expires: February 2003 [Page 14] Internet-Draft Unified Memory Space Protocol August 2003 destination MUST NOT form the selective acknowledgment for optional command with unknown OPCODE. All packet commands terminate on a 4-byte boundary. The general packet length must not exceed PMTU. SEQUENCE-NUMBER field is used for identification of responses and for detection of packet duplicates. This field contains the sequence packet number in connection. Arithmetic described in [13] must be used for calculation of the sequence number value. Zero value in SEQUENCE-NUMBER field indicates that necessity of response is absent. This value must be skipped at consecutive numeration. 6 Connections Control 6.1 Connection Establishment Connections can be primary and secondary. Primary connection is used only for sending the initiation commands of secondary connections. Secondary connections can be used for sending commands of other connections establishment and for sending VM instructions. The establishment of primary connection includes four steps with a possibility of data transfer in third step. The establishment of secondary connection includes two steps with a possibility of data transfer in first step. The purpose of any UMSP connection establishment is the exchange of connection identifiers. Each side allocates the 8-byte identifier to connection. All connections on one node use one connection identifiers space. Connection identifiers on different nodes are allocated independently. Only one primary connection MUST be between host and JCP. Secondary connections can be the following: "Host-JCP" - is established between the job initiator and JCP. This connection is equivalent to job existence. Closing of this connection finishes the job and everything connections relating to it. "JCP-host" - is established between JCP and a host which is not the job initiator. This connection is equivalent to job existence on the separate host. Closing of this connection finishes the job for this host. "Host-host" - is established between any two hosts to send VM instructions and to support operations of pointers assignment. All connected pointers are invalid after closing of this connection. This connection is the subordinate to "host-JCP" or "JCP-host" connection. Bogdanov Expires: February 2003 [Page 15] Internet-Draft Unified Memory Space Protocol August 2003 "Host-JCP-Host" - is similar to "host-host" connection at interaction through JCP. "Host-JCP-Host" is used, if immediate interaction between hosts is impossible, for example, hosts are located in different spaces of network addresses or in networks of different types. In the text of this document at the description of connection establishment, the initiator's node is named "initiator", and the node answering to request of connection establishment is named "respondent". 6.1.1 Establishment of Primary Connection Primary connection is the first connection between the host and JCP. Primary connection MUST NOT be established between two hosts. The connection initiator can be the host or JCP. The establishment of primary connection includes four steps. For establishment of primary connection the following commands are used: INIT - initiates the connection establishment INIT_ACK - acknowledges INIT command CONNECT - is sent after INIT_ACK reception CONNECT_ACK - is acknowledgement to the CONNECT command COOKIE - attaches to INIT_ACK and CONNECT commands for protection against the attacks related to falsification of the source address. The establishment of primary connection solves the following tasks: O The exchange of connection identifiers. O Definition of the used protocol version. The scheme of identifiers exchange is shown in the following diagram (T = 0 in all commands): Bogdanov Expires: February 2003 [Page 16] Internet-Draft Unified Memory Space Protocol August 2003 "Initiator" "Respondent" ----------- ------------ CONNECTION-ID = 0, INIT [SEQUENCE-NUMBER = Number11, DATA-2 = PrimaryID1] -------> <---------------- CONNECTION-ID = PrimaryID1, INIT_ACK [SEQUENCE-NUMBER = Number11, DATA-2 = TemporaryID] + COOKIE [COOKIE-VALUE = Value1] CONNECTION-ID = TemporaryID, CONNECT [SEQUENCE-NUMBER = Number11 + 1, DATA-2 = PrimaryID1] + COOKIE [COOKIE-VALUE = Value1] ---------> <------------ CONNECTION-ID = PrimaryID1, CONNECT_ACK [SEQUENCE-NUMBER = Number12, SOURCE-ID = PrimaryID2, SEQUENCE-NUMBER-CONF = Number11 + 1] After connection establishment: CONNECTION-ID = PrimaryID2, Command [SEQUENCE-NUMBER = Number11 + 2] ---> <------------ CONNECTION-ID = PrimaryID1, Command [SEQUENCE-NUMBER = Number12 + 1] COOKIE command is used for protection from attacks of source address substitution. COOKIE value is hashing function of time and unchangeable parameters of the "initiator" node. "Initiator" simply attaches received COOKIE to the packet with CONNECT command. "Respondent" sends CONNECT_ACK, only if COOKIE value is correct. COOKIE command is optional. It must not be used for IPsec packets. "Initiator" considers the connection established after CONNECT_ACK reception. Before CONNECT_ACK reception, "initiator" must not send other packets by this primary connection and secondary connections related to it. "Initiator" must discard other connection packets before CONNECT_ACK reception. "Respondent" must not save a connection state and allocate resources before CONNECT command receiving. "Respondent" considers connection established after CONNECT_ACK sending. The version flags field in INIT and INIT_ACK commands is used for the identification of UMSP version. One flag is intended for the indication of one version. "Initiator" in INIT command can set several flags of supported versions. "Respondent" in INIT_ACK Bogdanov Expires: February 2003 [Page 17] Internet-Draft Unified Memory Space Protocol August 2003 command MUST set one flag. This document defines one protocol version. Unused flags are reserved for the future versions. Primary connection can be public or private. Jobs of other hosts can be executed on this host over public connection. Jobs initiated on this host can only be executed for private connection. Jobs initiation commands of other hosts must be silently discarded on JCP. The logic of public and private connections establishment does not differ. 6.1.2 TCP Use TCP is intended for interaction between hosts without the public network address and JCP from a public network. TCP connection may be established between the host and JCP. The establishment of immediate TCP connection between two hosts is not specified in this document. Each TCP connection associates with a separate host. This host cannot have the real network address. All traffic between this host and a public network goes through JCP. TCP connection may be used by two manners: (1) As primary connection. Primary UMSP connection is not established. The host has no public virtual network address. This host is inaccessible to jobs initiation from other hosts. (2) As the transport protocol instead of UDP. The primary UMSP connection is established by TCP and the host has public virtual network address. This address can be used for a jobs initiation from other hosts. The commands format and the logic of exchange for TCP and UDP do not differ. 6.1.3 The New Job Registration on JCP The secondary "host-JCP" connection is established for registration of the new job. The identifier of this connection allocated on JCP is the private virtual network address of the host in the new job. The procedure of connection establishment includes two steps. The primary connection is used for initiation of this connection. The start node of the application is the initiator of connection establishment. For "host-JCP" connection establishment the following commands are used: CONTROL_REQ - is sent from the job initiator host to the JCP. CONTROL_CONFIRM - is sent from the JCP to the host as positive response to CONTROL_REQ. The "host-JCP" connection establishment solves the following tasks: Bogdanov Expires: February 2003 [Page 18] Internet-Draft Unified Memory Space Protocol August 2003 O The exchange of connection identifiers. O The setting of job lifetime. O Definition of VM type and VM version for the job execution. The scheme of identifiers exchange is shown in the following diagram (T = 0 in all commands): "Initiator" JCP ----------- ----- CONNECTION-ID = PrimaryID2, CONTROL_REQ [SEQUENCE-NUMBER = Number11 + 3, SOURCE-ID = SecondaryID1, INITIAL-NUMBER = Number21] ----> <------------ CONNECTION-ID = SecondaryID1, CONTROL_CONFIRM [SEQUENCE-NUMBER = Number22, SOURCE-ID = SecondaryID2, SEQUENCE-NUMBER-CONF = Number21] After connection establishment: CONNECTION-ID = SecondaryID2, Command [SEQUENCE-NUMBER = Number21 + 1] ---> <------------ CONNECTION-ID = SecondaryID1, Command [SEQUENCE-NUMBER = Number22 + 1] The packet with CONTROL_REQ and CONTROL_CONFIRM commands may include other commands of new connection. All commands located in the packet after CONTROL_REQ or CONTROL_CONFIRM, related to new secondary connection. One packet may contain several CONTROL_REQ commands. Each CONTROL_CONFIRM command is sent in a separate packet. If the packet contains several CONTROL_REQ commands, these connections are independent. At a simultaneous establishment of primary and secondary connection, CONTROL_REQ may be sent in a packet with CONNECT or CONNECT_ACK of primary connection. "Initiator" must not receive packets of new connection before reception of CONTROL_CONFIRM command. Such packets must be silently discarded. The connection is established for JCP after CONTROL_CONFIRM sending. If the application is initiated on JCP node, registration of the new job is beyond the scope of a network protocol. If the TCP connection is used instead of primary UMSP connection, PrimaryID2 is set to 0. Bogdanov Expires: February 2003 [Page 19] Internet-Draft Unified Memory Space Protocol August 2003 6.1.4 Job Initiation on the Host The secondary "JCP-host" connection is established for initiation of new job on a host. The identifier of this connection allocated on JCP is the private virtual network address of the host in this job. The connection establishment procedure includes two steps. For sending initiation commands of the new connection, the existing primary connection is used. If primary connection is absent, it must be established. JCP is the initiator of the "JCP-host" connection establishment. The following commands are used for "JCP-host" connection establishment: JOB_REQ - is sent from the JCP to the host for job initiation on the host. JOB_CONFIRM - is sent from host to JCP as positive response to JOB_REQ. The "JCP-host" connection establishment solves the following tasks: O Exchange of connection identifiers. O Setting of the job lifetime. O Setting VM type and VM version for execution of the job. The scheme of identifiers exchange is shown in the following diagram (T = 0 in all commands): JCP "Respondent" ----- ------------ CONNECTION-ID = PrimaryID3, JOB_REQ [SEQUENCE-NUMBER = Number3, SOURCE-ID = SecondaryID3, INITIAL-NUMBER = Number31, INIT-VIRTUAL-ADDR = SecondaryID2] ----> <------------ CONNECTION-ID = SecondaryID3, JOB_CONFIRM [SEQUENCE-NUMBER = Number32, SOURCE-ID = SecondaryID4, SEQUENCE-NUMBER-CONF = Number31] After connection establishment: CONNECTION-ID = SecondaryID4, Command [SEQUENCE-NUMBER = Number31 + 1] ---> <------------ CONNECTION-ID = SecondaryID3, Command [SEQUENCE-NUMBER = Number32 + 1] Bogdanov Expires: February 2003 [Page 20] Internet-Draft Unified Memory Space Protocol August 2003 The packet with JOB_REQ and JOB_CONFIRM may include other commands of new connection. All commands located in a packet after JOB_REQ or JOB_CONFIRM pertain to new secondary connection. One packet may contain several JOB_REQ commands. Each JOB_CONFIRM command is sent in a separate packet. If the packet contains several JOB_REQ commands, these connections are independent. At a simultaneous establishment of primary and secondary connection, JOB_REQ may be sent in a packet with command CONNECT or CONNECT_ACK of primary connection. JCP must not receive packets of new connection before JOB_CONFIRM reception. Such packets must be silently discarded. The connection is established for "initiator" after JOB_CONFIRM sending. If the TCP connection is used instead of primary UMSP connection, PrimaryID3 is set to 0. 6.1.5 "Host-Host" and "Host-JCP-Host" Connection Establishment Secondary connections "host-host" and "host-JCP-host" are intended for sending VM instructions. "Host-host" connection allows excluding JCP from exchange. "Host-host" and "host-JCP-host" connections are used to support VM operations with pointers. These connections allow calculating counters of references to remote objects and identifying the job abend on the remote host. Two hosts have only one "host-host" or "host-JCP-host" connection in one job. Different jobs use different connections. The host initiates "host-host" and "host-JCP-host" connection establishment. "Host-host" or "host-JCP-host" connection can be established if hosts are located in one network address space. If hosts are located in different network address spaces, "host-JCP-host" connection can be established only. Use of "host-host" and "host-JCP-host" connections for sending of VM instructions is analogous. The host with smaller value of the private virtual network address wins by encounter connection establishment of one type ("host-host" or "host-JCP-host") in one job. The virtual address is considered as unsigned integer. "Host-JCP-host" wins at an encounter "host-host" and "host-JCP-host" connections establishment. All packets of loser connection must be silently discarded. The following commands are used for the "host-host" and "host-JCP- host" connection establishment: BIND - is sent from the host to the JCP for connection initiation. BIND_FWD - is BIND forwarding from the JCP to the destination host. BIND_ACK - is sent immediately between hosts as the positive response to BIND_FWD command, for "host-host" connection. Bogdanov Expires: February 2003 [Page 21] Internet-Draft Unified Memory Space Protocol August 2003 BIND_ACK_JCP - is sent as the positive response to BIND_FWD command for "host-JCP-host" connection. For initiation of "host-host" and "host-JCP-host" connection establishment, "host-JCP"/"JCP-host" connection of this job is used. If JCP has received BIND command and there is no "JCP-host" connection (the job is not initiated) between JCP and the destination host, JCP analyzes the destination address type. If the real network address or the public virtual network address is specified in BIND command, JCP establishes "JCP-host" connection with BIND destination. If BIND contains the private virtual network address, the packet with BIND must be silently discarded. The sides solve the following tasks at "host-host" and "host-JCP- host" connection establishment: O The exchange of connection identifiers is carried out. Values of these identifiers are not used as virtual network addresses. O The PMTU value is established in "host-JCP-host" connection. Bogdanov Expires: February 2003 [Page 22] Internet-Draft Unified Memory Space Protocol August 2003 The scheme of identifiers exchange for "host-host" connection is shown in the following diagram (T = 0 in all commands): "Initiator" JCP "Respondent" ----------- ----- ------------ CONNECTION-ID = SecondaryID2, BIND [SEQUENCE-NUMBER = Number21 + 2, SOURCE-ID = SecondaryID5 INITIAL-NUMBER = Number51, DEST-NET-ADDR = RespondentAddress] -> CONNECTION-ID = SecondaryID4, BIND_FWD [SEQUENCE-NUMBER = Number31 + 2, SOURCE-ID = SecondaryID5, INITIAL-NUMBER = Number51, SOURCE-VIRTUAL-ADDR = SecondaryID2, SOURCE-NET-ADDR = InitiatorAddress] ---> <---------------------- CONNECTION-ID = SecondaryID5, BIND_ACK [SEQUENCE-NUMBER = Number52, SOURCE-ID = SecondaryID6, SEQUENCE-NUMBER-CONF = Number51, PRIVATE-SRC-VIRTUAL-ADDR = SecondaryID3, PUBLIC-SRC-VIRTUAL-ADDR = PrimaryID3] After connection establishment: CONNECTION-ID = SecondaryID6, Command [SEQUENCE-NUMBER = Number51 + 1] ---------> <----------- CONNECTION-ID = SecondaryID5, Command [SEQUENCE-NUMBER = Number52 + 1] JCP does not participate in an exchange after "host-host" connection establishment. The scheme of identifiers exchange for "host-JCP-host" connection is shown in the following diagram: Bogdanov Expires: February 2003 [Page 23] Internet-Draft Unified Memory Space Protocol August 2003 "Initiator" JCP "Respondent" ----------- ----- ------------ CONNECTION-ID = SecondaryID2, BIND [T = 0, SEQUENCE-NUMBER = Number21 + 3, SOURCE-ID = SecondaryID7, INITIAL-NUMBER = Number61, DEST-NET-ADDR = RespondentAddress] -> CONNECTION-ID = SecondaryID4, BIND_FWD [T = 0, SEQUENCE-NUMBER = Number31 + 3, SOURCE-ID = SecondaryID7, INITIAL-NUMBER = Number61, SOURCE-VIRTUAL-ADDR = SecondaryID2, SOURCE-NET-ADDR = InitiatorAddress] --> <--- CONNECTION-ID = SecondaryID7, BIND_ACK_JCP [T = 1, SEQUENCE-NUMBER = Number62, SOURCE-VIRTUAL-ADDRESS = SecondaryID3, DEST-VIRTUAL-ADDRESS = SecondaryID2, SOURCE-ID = SecondaryID8, SEQUENCE-NUMBER-CONF = Number61, PUBLIC-SRC-VIRTUAL-ADDR = PrimaryID3] <-------- CONNECTION-ID = SecondaryID7, BIND_ACK_JCP [without changes] After connection establishment: CONNECTION-ID = SecondaryID8, Command [T = 1, SOURCE-VIRTUAL-ADDRESS = SecondaryID2, DEST-VIRTUAL-ADDRESS = SecondaryID3, SEQUENCE-NUMBER = Number61 + 1] -> CONNECTION-ID = SecondaryID8, Command [without changes] ----------> <--- CONNECTION-ID = SecondaryID7, Command [T = 1, SOURCE-VIRTUAL-ADDRESS = SecondaryID3, DEST-VIRTUAL-ADDRESS = SecondaryID2, SEQUENCE-NUMBER = Number62 + 1] <------------ CONNECTION-ID = SecondaryID7, Commands [without changes] Bogdanov Expires: February 2003 [Page 24] Internet-Draft Unified Memory Space Protocol August 2003 The packet may include several BIND commands by sending from the host to JCP. The packet with "host-host" and "host-JCP-host" connection establishment commands may include other commands of new connection. These commands are located in the packet immediately after connection establishment command. "Initiator" must not receive the new connection packets before reception BIND_ACK or BIND_FWD_ACK. Such packets must be silently discarded. The connection is established for "respondent" after BIND_ACK or BIND_ACK_JCP sending. As "host-JCP-host" connection is established through an intermediate gateway (JCP), PMTU calculation is impossible at the network layer. The PMTU field from BIND_FWD and BIND_FWD_ACK commands is used for the PMTU calculation. This field is set to minimal PMTU value from "host-JCP" and "JCP-host" hops. If "initiator" and "respondent" are not located in one network, the packets size with connection initiation commands must not exceed minimum PMTU. If at a "host - host" connection establishment, sending of response from "respondent" to "initiator" through JCP is required (for example, at a keys exchange for the protected connection between hosts), BIND_ACK_JCP may be used instead of BIND_ACK. In this case, BIND_ACK_JCP contains the "respondent" real network address. The "initiator" chooses the connection type at first data packet sending. The establishment of the protected connections is beyond the scope of this document. BIND_ACK command is RECOMMENDED to send at a "host- host" connection establishment. 6.2 Termination of Connection The protocol provides graceful and abnormal connection termination. The following commands are used for connection termination: DISCONNECT - initiates graceful connection termination. DISCONNECT_ACK - acknowledges graceful connection termination or it is the negative response on DISCONNECT. ABORT - is unconditional abnormal connection termination. All connection types use one set of termination commands. The initiator of graceful termination sends DISCONNECT command, stops the traffic of outgoing VM instructions and waits the response. "Initiator" processes the entering traffic and sends packets with acknowledgement up to reception of closing acknowledgement. The acknowledgement to DISCONNECT is DISCONNECT_ACK, contrary DISCONNECT or ABORT. The connection is closed after acknowledgement reception. If response has not been received, two repeated sending with 10 seconds intervals are carried out. If incoming data traffic is absent, the interval in 10 seconds may be reduced up to a usual time Bogdanov Expires: February 2003 [Page 25] Internet-Draft Unified Memory Space Protocol August 2003 out. If response has not been received after repeated sending, ABORT must be transmitted and connection must be closed. DISCONNECT_ACK command may be the negative response on DISCONNECT in "host-host" and "host-JCP-host" connections. Termination codes are definite in section 6.2.8. If DISCONNECT_ACK is the negative response, connection remains opened and can be used for data transmission in both ways. ABORT command stops any connection traffic in both ways. The counter ABORT command is response to ABORT. If response is absent, ABORT sends repeatedly. If ABORT is sent in expectation of DISCONNECT_ACK, repeated sending is not carried out. ABORT may be sent as the negative response to CONNECT, CONTROL_REQ and JOB_REQ commands on the fourth step of primary connection establishment and on the second step of secondary connection established. DISCONNECT may be sent after connection establishment. 6.2.1 Termination of "Host-Host" Connection Graceful closing of "host-host" connection is carried out, if the references counter on the host set to zero. If the references counter is not zero on the "respondent", DISCONNECT_ACK command with ERRCODE = 2 is sent as the negative response and connection remains open. "Respondent" must send DISCONNECT_ACK response just after DISCONNECT reception with ERRCODE = 1 (zeroing of the references counter). Before response reception, the application on the "initiator" host cannot assign new references to objects of a remote host for this connection. Only one opened "host-host" connection may be between two hosts. Such references can be received by other connections and operations must be buffered before response reception. The abort of "host-host" connection is the consequence of the job abort on the host or the channel break. All references in the executed application related to connection must set to invalid value at abort. 6.2.2 Termination of "Host-JCP-Host" Connection "Host-JCP-host" connection termination is carried out similarly "host-host" connection termination. Difference is presence of the intermediate node: JCP. JCP carries out simple forwarding of termination commands. 6.2.3 Termination of "JCP-Host" Connection The termination of "JCP-host" connection finishes the job on the host simultaneously. This termination is carried out in the following cases: Bogdanov Expires: February 2003 [Page 26] Internet-Draft Unified Memory Space Protocol August 2003 O At the common job shutdown. In this case, JCP is the termination initiator. O If all resources of the job were released on the host. In this case, the host is the termination initiator. O At the host shutdown. The host must finish all "host-host" and "host-JCP-host" connections related to this job before sending of DISCONNECT or DISCONNECT_ACK. If even one such connection is not finished gracefully and the host has the exploitable job resources, ABORT command with ERRCODE = 11 must be used for "JCP-host" connection termination. If all job resources on the host are released, the host sends DISCONNECT or DISCONNECT_ACK. ABORT is sent only if resources connected with the job are on the host and there are open "host-host" and/or "host-JCP-host" connections. "JCP-host" connection is completed gracefully if the host has sent DISCONNECT or DISCONNECT_ACK. The command sent from the JCP to the host has no importance. 6.2.4 Job Termination The job termination is equivalent to connection termination between the job initiator host and JCP ("host-JCP" connection). The job terminates in the following cases: O At the termination of application. In this case, the host of job initiator is termination initiator. Special instructions for correct application termination may be provided on the VM layer. O At the expiration of job lifetime. In this case, JCP is the termination initiator. O At channel fault between JCP and the host of the job initiator. If the host is the termination initiator, it sends DISCONNECT or ABORT to the JCP and expects the response. JCP controls job termination. JCP carries out the following actions: (1) Terminates all "JCP-host" connections of this job. (2) Terminates "host-JCP" connection between JCP and the host of application start. If JCP is the termination initiator, connection between JCP and the job initiator host is terminated last. 6.2.5 Termination of Primary Connection Bogdanov Expires: February 2003 [Page 27] Internet-Draft Unified Memory Space Protocol August 2003 Termination of primary connection is equivalent to the node shutdown. The termination of primary connection terminates all "JCP- host"/"host-JCP" connections related to it. The termination commands are sent only on primary connection. Termination of TCP connection terminates all UMSP connections related to it. 6.2.6 The Rules of Connection Identifiers Reuse The node has one identifiers space for all connections. The identifier of the closed connection may be used at a new connection establishment. Initial sequence number of new connection MUST be more on 65536 from last sequence number of the closed connection. The following rules for reuse of connection identifiers are defined: (1) The connection identifier may be used for new connection right away after closing of the current connection. This rule is carried out in the following cases: O For any connection identifiers on hosts. O For public virtual network addresses. These identifiers are not used in memory addresses. If JCP distributes these addresses in VM_NOTIF_JCP commands (see section 10.2), it is recommended to establish static correspondence between the node and its public virtual network address. O For any job identifiers after job termination if all job connections are completed gracefully. (2) The identifier of closed connection may be used for new connection after the expiration of double maximal packet lifetime in a network. This rule is carried out in the following cases: O For private virtual network addresses if "JCP-host" connection is completed gracefully. These identifiers are used in application pointers. All related "host- host" and "host-JCP-host" connections are closed at graceful termination of "JCP-host" connection and all pointers are established to invalid value. O For any job identifiers after job termination. (3) If "JCP-host" connection is abend, the private virtual network address may be used for new connection. It may be used only after transmission to the hosts of the job the notification about connection termination. If the notification has not been transmitted, the private virtual network address of the closed connection can be saved in applications memory for a long time, and cannot be used repeatedly. Bogdanov Expires: February 2003 [Page 28] Internet-Draft Unified Memory Space Protocol August 2003 If the node is overloaded without state saving, the overload time must be no less than maximal packets lifetime in a network. 6.2.7 Notification about Termination of "JCP-Host" Connections If "JCP-host" connection is completed emergency, pointers related to the closed connection can be on other hosts. If such pointers present, the private virtual network address of the closed connection cannot be used repeatedly. If lifetime of the job is long, it can lead to exhaustion of JCP resources. JCP sends the notification about jobs termination on hosts for identifiers release. For this purpose the following commands are used: TERMINATION_NOTIF - is sent from the JCP to job hosts. This command contains the list of the released private virtual network addresses. ACK - is acknowledgement of TERMINATION_NOTIF. ADDR_LIST - is sent from the host to the JCP at "JCP-host" connection termination. ADDR_LIST includes the list of private virtual network addresses with which "host- host" and "host-JCP-host" connections are not terminated gracefully. ADDR_LIST is sent in one packet with ABORT command. If ADDR_LIST have not been transmitted, JCP sends TERMINATION_NOTIF command to all job hosts. If ADDR_LIST have been transmitted, TERMINATION_NOTIF is sent to hosts from the received list only. Each destination host must send ACK (see section 7.1.2) for acknowledgement of TERMINATION_NOTIF delivery. It is RECOMMENDED to send TERMINATION_NOTIF, only if loading of JCP resources has extreme value. 6.2.8 Termination Codes DISCONNECT and ABORT commands can have the following termination codes: 0 - normal termination. 1 - the references counter is zero. This code is sent in DISCONNECT command at deleting all pointers to destination host objects. 2 - the references counter is not zero. If the source host has pointers to the destination objects, this code will be sent in DISCONNECT_ACK as the negative response on DISCONNECT. Connection remains open. 3 - it is necessary to execute the Objects destructors. This code is sent in DISCONNECT_ACK as the negative response to DISCONNECT. Connection remains open. 4 - the node does not create new objects as it is in a job termination stage. 5 - unacceptable job lifetime. Bogdanov Expires: February 2003 [Page 29] Internet-Draft Unified Memory Space Protocol August 2003 6 - the establishment of primary connection without sending other commands is not supposed. 7 - unknown address type. 8 - the job is terminated, because the job lifetime has expired. 9 - the job is terminated because of channel break between JCP and the job initiator. 10 - JCP cannot prolong job lifetime. 11 - connection is terminated before closing of all related connections. 12 - connection is completed because of node inactivity. This code is sent in DISCONNECT command. Reciprocal DISCONNECT_ACK command can contain ERRCODE = 2/3 and connection remains open. 6.3 Use of Several JCP The general UMSP logic allows several JCP to control one job. These JCP must share common space of virtual network addresses. This document does not consider this question and does not specify JCP-JCP protocol. The logic of the host working does not depend on values of virtual network addresses. JCP chooses these values independently. 6.4 Format of Connection Control Commands 6.4.1 INIT INIT command is used to initiate primary connection between host and JCP. This command has the following format: CONNECTION-ID = 0 T = 0 DATA-LENGTH = 4 OPCODE = 1 DATA-1: indicates flags of the supported protocol versions and has the following format: +-+-+-+-+-+-+-+-+ |1| RESERVED |P| +-+-+-+-+-+-+-+-+ The first bit must be set to 1. It indicates the protocol version specified in this document. RESERVED : 6 bits Must be set to 0 on transmit. If one of these bits is set to 1 on receipt, command must be silently discarded. These bits are reserved for the following protocol versions. Each version uses one flag. Several flags can be set in INIT. The version is chosen by "respondent" in INIT-ACK. Bogdanov Expires: February 2003 [Page 30] Internet-Draft Unified Memory Space Protocol August 2003 P : 1 bit Bit of the private protocol version. It is intended for all protocol versions, non-intended for public use. This bit must be set to 0 for public versions. Authentication of the private protocol realizations is beyond the scope of this document. SEQUENCE-NUMBER : indicates initial sequence number for this connection. DATA-2 : 8 bytes Indicates the connection identifier allocated by the sender of this command. The packet with INIT command must not include other commands. 6.4.2 INIT_ACK INIT_ACK command is used to acknowledge the INIT command and has the following packet format: CONNECTION-ID : indicates the connection identifier from DATA-2 field of INIT command. T = 0 DATA-LENGTH = 4 OPCODE = 2 DATA-1 : indicates the flag of the chosen protocol version. This field MUST include only one nonzero bit. The format coincides with DATA-1 field from INIT command. SEQUENCE-NUMBER : value is taken from the SEQUENCE-NUMBER field of the INIT command. DATA-2 : 64 bits This field contains the temporary identifier allocated by the sender of this command. The protocol does not define constraints on this field values. The packet with INIT_ACK command may contain VM_REQ, VM_NOTIF or VM_NOTIF_JCP commands. 6.4.3 CONNECT CONNECT command is used on the third step of primary connection establishment. It has the following packet format: Bogdanov Expires: February 2003 [Page 31] Internet-Draft Unified Memory Space Protocol August 2003 CONNECTION-ID : indicates the temporary identifier, taken from the DATA-2 field of the INIT_ACK command. T = 0 DATA-LENGTH = 4 OPCODE = 3 - for public connection 4 - for private connection DATA-1 : indicates the flag of the chosen protocol version. Value of this field is copied from DATA-1 field of INIT_ACK command. SEQUENCE-NUMBER : indicates the sequence number. DATA-2 : 64 bits Indicates the connection identifier allocated by the sender of this command. Only the host can be the initiator of private connection (OPCODE = 4). JCP cannot be the initiator of private connection. The host or JCP can be the initiator of public connection (OPCODE = 3). The packet with CONNECT command may contain one or several CONTROL_REQ, JOB_REQ or BIND commands. The commands located in a packet after CONTROL_REQ, JOB_REQ or BIND relate to secondary connection. 6.4.4 CONNECT_ACK CONNECT_ACK command is used to acknowledge the CONNECT command. CONNECT_ACK has the following format: CONNECTION-ID : indicates the connection identifier, taken from DATA-2 field of CONNECT command. T = 0 DATA-LENGTH = 5 OPCODE = 5 DATA-1 : set to 0 on transmit and ignored on receipt. SEQUENCE-NUMBER : indicates the initial sequence number for this connection. DATA-2 has the following format: Bogdanov Expires: February 2003 [Page 32] Internet-Draft Unified Memory Space Protocol August 2003 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | + SOURCE-ID + | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | SEQUENCE-NUMBER-CONF | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ SOURCE-ID : 64 bits This field contains the connection identifier allocated by this command sender. SEQUENCE-NUMBER-CONF : 32 bits Value is taken from SEQUENCE-NUMBER field of CONNECT command. The packet with CONNECT_ACK command may contain one or several CONTROL_REQ, JOB_REQ, BIND or DATA commands. The commands located in a packet after CONTROL_REQ, JOB_REQ or BIND relate to secondary connection. 6.4.5 COOKIE COOKIE command is used for protection against the attacks related to falsification of the source address at establishment of primary connection. The command has the following format: T = 0 DATA-LENGTH = 2 - 255 OPCODE = 6 SEQUENCE-NUMBER : this field is absent. DATA-1 + DATA-2 : indicates the COOKIE value. The protocol does not define any restrictions on values of this field. It can be hashing function of time and unchangeable parameters of the "initiator" node. "Respondent" of primary connection forms COOKIE command and sends it in a packet with INIT_ACK. The initiator MUST copy this command to the packet with CONNECT command. 6.4.6 CONTROL_REQ, JOB_REQ CONTROL_REQ command is transmitted from the host to the JCP for initiation of "host-JCP" connection. This connection establishment Bogdanov Expires: February 2003 [Page 33] Internet-Draft Unified Memory Space Protocol August 2003 is the initiation of the new job on JCP. The job initiator host sends CONTROL_REQ. JOB_REQ command is transmitted from the JCP to the host for initiation of "JCP-host" connection. This connection establishment is the initiation of the job on the host. This host is not the primary initiator of the job. CONTROL_REQ and JOB_REQ commands are transmitted on primary connection. These commands have an identical format and differ only in OPCODE value and presence of INIT-VIRTUAL-ADDR field: CONNECTION-ID : indicates the identifier of primary connection allocated by the packet destination. T = 0 DATA-LENGTH = 7 - for CONTROL_REQ 9 - for JOB_REQ OPCODE = 7 - for CONTROL_REQ 9 - for JOB_REQ DATA-1 : set to 0 on transmit and ignored on receipt. SEQUENCE-NUMBER : indicates the sequence number of primary connection. DATA-2 has the following format: 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | + SOURCE-ID + | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | + INIT-VIRTUAL-ADDR + | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | INITIAL-NUMBER | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | RESERVED | JOB-LIFE-TIME | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | VM-TYPE | VM-VER | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ SOURCE-ID : 64 bits Indicates the identifier of new secondary connection, allocated by the sender of this command. For JOB_REQ, this Bogdanov Expires: February 2003 [Page 34] Internet-Draft Unified Memory Space Protocol August 2003 field is the private virtual network address of the host in this job. INIT-VIRTUAL-ADDR : 64 bits Indicates the host private virtual network address of the job initiator (the host which has sent CONTROL_REQ) in JOB_REQ. This field is absent in CONTROL_REQ. INITIAL-NUMBER : 32 bits Indicates the initial sequence number for new connection. RESERVED : 16 bits MUST be set to 0 on transmit. If this field is non-zero on receipt, command must be silently discarded. JOB-LIFE-TIME : 16 bits Indicates the lifetime of the job in seconds. For CONTROL_REQ, the value can be reduced in CONTROL_CONFIRM. VM-TYPE : 16 bits Indicates the VM type of the initiated job (see section 10). VM-VER : 16 bits Indicates the VM version of the initiated job (see section 10). CONTROL_REQ and JOB_REQ commands may be sent in a packet with CONNECT or CONNECT_ACK command. One packet may include several CONTROL_REQ and JOB_REQ. BIND command of new connection may be located after CONTROL_REQ command in one packet. 6.4.7 CONTROL_CONFIRM, JOB_CONFIRM CONTROL_CONFIRM command is transmitted from the JCP to the host as CONTROL_REQ acknowledgement. JOB_CONFIRM command is transmitted from the host to JCP as JOB_REQ acknowledgement. CONTROL_CONFIRM and JOB_CONFIRM have an identical format and differ only in OPCODE value and by presence of JOB-LIFE-TIME field: CONNECTION-ID : indicates the connection identifier from SOURCE-ID field of the CONTROL_REQ or JOB_REQ command. T = 0 DATA-LENGTH = 6 - for CONTROL_CONFIRM 5 - for JOB_CONFIRM Bogdanov Expires: February 2003 [Page 35] Internet-Draft Unified Memory Space Protocol August 2003 OPCODE = 8 - for CONTROL_CONFIRM 10 - for JOB_CONFIRM DATA-1 : set to 0 on transmit and ignored on receipt. SEQUENCE-NUMBER : indicates the initial sequence number of new connection. DATA-2 has the following format: 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | + SOURCE-ID + | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | SEQUENCE-NUMBER-CONF | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | RESERVED | JOB-LIFE-TIME | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ SOURCE-ID : 64 bits Indicates the identifier of new connection allocated by the sender of this command. This field in CONTROL_CONFIRM is the private virtual network address of the host in this job. SEQUENCE-NUMBER-CONF : 32 bits The value is copied from INITIAL-NUMBER field of the confirmed command. RESERVED : 16 bits Must be set to 0 on CONTROL_CONFIRM transmit. If this field is not zero on receipt, command must be silently discarded. This field is absent in JOB_CONFIRM. JOB-LIFE-TIME : 16 bits In CONTROL_CONFIRM, this field indicates established lifetime of the job in seconds. In JOB_CONFIRM, this field is absent. 6.4.8 BIND BIND command is transmitted from "initiator" to JCP, to establish "host-host" or "host-JCP-host" connection. "JCP-host"/"host-JCP" connection of this job is used for sending BIND. BIND has two OPCODE values. If OPCODE = 11, JCP or "respondent" chooses the type of Bogdanov Expires: February 2003 [Page 36] Internet-Draft Unified Memory Space Protocol August 2003 established connection ("host-host" or "host-JCP-host"). If OPCODE=12, "host-JCP-host" connection must be established. The command format for both OPCODE values coincides: CONNECTION-ID : contains the identifier of "JCP-host"/"host-JCP" connection allocated by the packet destination (JCP is the destination). T = 0 DATA-LENGTH - variable length OPCODE = 11 - for "host-host" or "host-JCP-host" connection 12 - only for "host-JCP-host" connection DATA-1 : this field indicates ADDR-TYPE value (the address type in NET-ADDRESS field). SEQUENCE-NUMBER : indicates the sequence number for "JCP- host"/"host-JCP" connection. DATA-2 has the following format: 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | + SOURCE-ID + | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | INITIAL-NUMBER | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ ~ DEST-NET-ADDRESS ~ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ SOURCE-ID : 64 bits Indicates the identifier of new connection allocated by the sender of this command. INITIAL-NUMBER : 32 bits Indicates the initial sequence number for new connection. DEST-NET-ADDRESS : variable length The network address of the destination host. ADDR-TYPE value from DATA-1 field defines the length and format of this field. Bogdanov Expires: February 2003 [Page 37] Internet-Draft Unified Memory Space Protocol August 2003 The address types are shown in the following table: -------------------------------------------------------------- | ADDR-TYPE | Len(NET-ADDRESS) | NET-ADDRESS | -------------------------------------------------------------- 1 4 Public IPv4 address 2 16 Public IPv6 address 4 variable DNS name 5 8 Virtual network address of the host on this JCP 33 4 Non-public IPv4 address 34 16 Non-public IPv6 address -------------------------------------------------------------- ADDR-TYPE values are divided into the following ranges: 1 - 31 - for public networks 33 - 63 - for local networks 32, 64 - 255 - reserved NET-ADDRESS field for DNS name (ADDR-TYPE = 4) has the following format: 0 1 2 3 4 5 6 7 +-+-+-+-+-+-+-+-+----------------/ /------------------+---/ /---+ | NAME-LENGTH | NAME | PADDING | +-+-+-+-+-+-+-+-+----------------/ /------------------+---/ /---+ NAME-LENGTH : 8 bits NAME field length in bytes. NAME : 1 - 235 bytes Indicates the DNS name. PADDING : 0 - 3 bytes Zero bits for alignment on a 4-byte boundary. If the destination address is private virtual network address, the packet with BIND command may include DATA commands of new connection. 6.4.9 BIND_FWD BIND_FWD command is transmitted from JCP to "respondent", to establish "host-host" or "host-JCP-host" connection. This command is BIND forwarding. BIND_FWD is sent by "host-JCP"/"JCP-host" connection. BIND_FWD has the following format: CONNECTION-ID : indicates the identifier of "JCP-host"/"host-JCP" connection allocated by the "respondent". Bogdanov Expires: February 2003 [Page 38] Internet-Draft Unified Memory Space Protocol August 2003 T = 0 DATA-LENGTH - depends on the SOURCE-NET-ADDRESS field length OPCODE = 13 DATA-1 : Indicates the type of source address in SOURCE-NET- ADDRESS field (see section 6.4.8). If DATA-1 is set to 0, SOURCE-NET-ADDRESS field is absent. SEQUENCE-NUMBER : indicates the sequence number in "JCP- host"/"host-JCP" connection. DATA-2 has the following format: 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | + SOURCE-ID + | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | INITIAL-NUMBER | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | RESERVED | PMTU | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | + SOURCE-VIRTUAL-ADDRESS + | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ ~ SOURCE-NET-ADDRESS ~ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ SOURCE-ID : 64 bits Copied from SOURCE-ID field of BIND command. INITIAL-NUMBER : 32 bits Copied from INITIAL-NUMBER field of BIND command. RESERVED : 16 bits Set to 0 on transmit and ignored on receipt. PMTU : 16 bits Indicates minimal PMTU value of "host-JCP" and "JCP-host" hops in bytes. SOURCE-VIRTUAL-ADDRESS : 64 bits Bogdanov Expires: February 2003 [Page 39] Internet-Draft Unified Memory Space Protocol August 2003 Indicates the private virtual network address of "initiator". SOURCE-NET-ADDRESS : variable length The real network address of the "initiator". The format of this field is defined by DATA-1 value. If "initiator" and "respondent" are located in disjoint spaces of network addresses, this field is absent. If SOURCE-NET-ADDRESS field is present in command, "respondent" may choose between "host-host" and "host-JCP-host" connection. If this field is absent, "host-JCP-host" connection must be established. The packet with BIND_FWD command may include other commands of new connection. These commands are copied from the packet with BIND command. 6.4.10 BIND_ACK BIND_ACK command is used to acknowledge the BIND_FWD command. BIND_ACK is sent from "respondent" to "initiator" to establish "host- host" connection. BIND_ACK has the following format: CONNECTION-ID : contains the connection identifier allocated by the packet destination. The value is copied from SOURCE-ID field of BIND_FWD command. T = 0 DATA-LENGTH = 7/9 - depends on presence of the public virtual network address. OPCODE = 14 DATA-1 : set to 0 on transmit and ignored on receipt. SEQUENCE-NUMBER : indicates the initial sequence number of new connection. Bogdanov Expires: February 2003 [Page 40] Internet-Draft Unified Memory Space Protocol August 2003 DATA-2 has the following format: 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | + SOURCE-ID + | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | + PRIVATE-SRC-VIRTUAL-ADDR + | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | + PUBLIC-SRC-VIRTUAL-ADDR + | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | SEQUENCE-NUMBER-CONF | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ SOURCE-ID : 64 bits Indicates the identifier of new secondary connection allocated by "respondent". PRIVATE-SRC-VIRTUAL-ADDR : 64 bits Indicates the private virtual network address of the "respondent" in this job. PUBLIC-SRC-VIRTUAL-ADDR : 64 bits If DATA-LENGTH = 6, this field indicates the public virtual network address of the "respondent". If DATA-LENGTH = 5, this field is absent. SEQUENCE-NUMBER-CONF : 32 bits This value is copied from INITIAL-NUMBER field of BIND_FWD command. The packet with BIND_ACK command may include other commands of new connection. 6.4.11 BIND_ACK_JCP BIND_ACK_JCP command is used to acknowledge the BIND_FWD command. BIND_ACK_JCP is sent from "respondent" to "initiator" through JCP to establish "host-JCP-host" connection. If at establishment of "host- host" connection the response of sending from "respondent" to "initiator" through JCP is required, BIND_ACK_JCP may be used instead of BIND_ACK. Bogdanov Expires: February 2003 [Page 41] Internet-Draft Unified Memory Space Protocol August 2003 BIND_ACK_JCP has the following format: CONNECTION-ID : contains the connection identifier allocated by "initiator". The value is copied from SOURCE-ID field of BIND_FWD command. T = 1 DATA-LENGTH - variable length OPCODE = 15 DATA-1 : Indicates the address type of the source in SOURCE-NET- ADDRESS field (see section 6.4.8). If DATA-1 is set to 0, SOURCE-NET-ADDRESS field is absent. DATA-1 MUST NOT be set to 5 (the virtual network address). If the establishment of immediate "host-host" connection between "initiator" and "respondent" is impossible, DATA-1 must be set to 0. SEQUENCE-NUMBER : indicates the initial sequence number of new connection. SOURCE-VIRTUAL-ADDRESS : indicates the private virtual network address of the source. DEST-VIRTUAL-ADDRESS : indicates the private virtual network address of the destination. DATA-2 has the following format: 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | + SOURCE-ID + | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | + PUBLIC-SRC-VIRTUAL-ADDR + | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | SEQUENCE-NUMBER-CONF | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ ~ SOURCE-NET-ADDRESS ~ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ SOURCE-ID : 64 bits This field contains the identifier of new secondary connection allocated by the "respondent". Bogdanov Expires: February 2003 [Page 42] Internet-Draft Unified Memory Space Protocol August 2003 PUBLIC-SRC-VIRTUAL-ADDR : 64 bits If (DATA-LENGTH - length(SOURCE-NET-ADDRESS)) = 11, this field indicates the public virtual network address of the "respondent". If (DATA-LENGTH - length(SOURCE-NET-ADDRESS)) = 9, this field is absent. SEQUENCE-NUMBER-CONF : 32 bits The value is copied from INITIAL-NUMBER field of the BIND_FWD command. SOURCE-NET-ADDRESS : variable length The real network address of the "respondent". DATA-1 value defines format of this field. If "initiator" and "respondent" are located in disjoint spaces of network addresses, this field is absent. If SOURCE-NET-ADDRESS field presents in the command, "initiator" may choose between "host-host" and "host-JCP-host" connection. If this field is absent, "host-JCP-host" connection must be established. The packet with BIND_ACK_JCP command may include other commands of new connection. 6.4.12 DISCONNECT, ABORT DISCONNECT is used for the graceful termination of connection. ABORT is used to initiate and acknowledge emergency termination of connection. DISCONNECT and ABORT have an identical format and differ only in OPCODE value: CONNECTION-ID : Contains the identifier of closed connection allocated by the packet destination. T = 0/1 DATA-LENGTH - variable length OPCODE = 16 - for DISCONNECT 17 - for ABORT DATA-1 : Indicates the termination code - ERRCODE (see section 6.2.8). SOURCE-VIRTUAL-ADDRESS : If T = 0, this field is absent. If T = 1, this field contains the private virtual network address of the source. Bogdanov Expires: February 2003 [Page 43] Internet-Draft Unified Memory Space Protocol August 2003 DEST-VIRTUAL-ADDRESS : If T = 0, this field is absent. If T = 1, this field contains the private virtual network address of the destination. SEQUENCE-NUMBER : indicates the sequence number. DATA-2 : Can contain non-formalized message on the error in ASCII symbols. This field can be absent. The packet with ABORT command must not contain other commands of closed connection. 6.4.13 DISCONNECT_ACK DISCONNECT_ACK command is used to acknowledge the DISCONNECT or ABORT commands. DISCONNECT_ACK has the following format: CONNECTION-ID : Contains the identifier of connection required to be terminate, allocated by the packet destination. T = 0/1 DATA-LENGTH = 3/7 OPCODE = 18 DATA-1 : indicates the termination code. SOURCE-VIRTUAL-ADDRESS : If T = 0, this field is absent. If T = 1, this field contains the private virtual network address of the source. DEST-VIRTUAL-ADDRESS : If T = 0, this field is absent. If T = 1, this field contains the private virtual network address of the destination. SEQUENCE-NUMBER : indicates the sequence number. DATA-2 : 4 bytes The value is copied from SEQUENCE-NUMBER field of the DISCONNECT or ABORT commands. 6.4.14 TERMINATION_NOTIF TERMINATION_NOTIF command is sent in the "JCP-host" connection from the JCP to the host. TERMINATION_NOTIF contains the private virtual network addresses list of the closed "JCP-host" connections. Destination of TERMINATION_NOTIF must close all related "host-host" and "host-JCP-host" connections without a sending of network Bogdanov Expires: February 2003 [Page 44] Internet-Draft Unified Memory Space Protocol August 2003 primitives and must set pointers for the closed connections to 0x????????FFFFFFFF. TERMINATION_NOTIF has the following format: CONNECTION-ID : Contains the connection identifier allocated by the packet destination. T = 0 DATA-LENGTH - variable length OPCODE = 19 DATA-1 : Set to 0 on transmit and ignored on receipt. SEQUENCE-NUMBER : Indicates the sequence number. DATA-2 : contains the list of private virtual network addresses of the closed "JCP-host" connections. Destination of TERMINATION_NOTIF must send ACK. It is NOT RECOMMENDED to detain ACK sending. 6.4.15 ADDR_LIST ADDR_LIST command is sent over "JCP-host" connection from the host to the JCP at emergency termination of connection. ADDR_LIST is sent in one packet with ABORT. ADDR_LIST includes the list of private virtual addresses for non-closed "host-host" and "host-JCP-host" connections. ADDR_LIST has the following format: CONNECTION-ID : Contains the connection identifier allocated by the packet destination. T = 0 DATA-LENGTH - variable length OPCODE = 128 DATA-1 : Set to 0 on transmit and ignored on receipt. SEQUENCE-NUMBER : Indicates the sequence number. DATA-2 : contains the list of the private virtual network addresses of the non-closed "host-host" and "host-JCP- host" connections. 7 Transfer of VM Instructions The following commands are used for transfer of VM instructions: DATA - contains the VM instructions. ACK - is used to acknowledge the delivery on the UMSP layer. Bogdanov Expires: February 2003 [Page 45] Internet-Draft Unified Memory Space Protocol August 2003 CONGESTION - the explicit notification of congestion sent from the JCP to the host. UMSP gives the unstructured binary buffer for sending of VM instructions. The protocol assumes that each VM type has its own set of network instructions. The VM data is sent in DATA command. "Host-host" or "host-JCP-host" connection is used for sending. Instructions with VM data may be sent simultaneously with instructions of connection establishment. The received data is sent to the VM layer without buffering on the UMSP layer. ACK command is used to acknowledge the DATA command on the UMSP layer. SEQUENCE-NUMBER field is used for responses identification and for detection of packet duplicates. The maximal window of the unconfirmed packets numbers on sending is 65535. All packets with displacement of sequence number from last cumulative acknowledged packet more than 65535 must be silently discarded on receipt. 65534- window is used for DATA commands on sending. One more number is used for sending of ABORT command at absence of responses. Bogdanov Expires: February 2003 [Page 46] Internet-Draft Unified Memory Space Protocol August 2003 VM data exchange on the "host-host" connection is shown in the following diagram (T = 0 in all commands): Host1 Host2 ------- ------- CONNECTION-ID = SecondaryID6, DATA [SEQUENCE-NUMBER = Number51 + 2, DATA-1 + DATA-2 = VM-Instructions] ---------> <----------- CONNECTION-ID = SecondaryID5, ACK [SEQUENCE-NUMBER = Number51 + 2] VM data exchange on the "host-JCP-host" connection is shown in the following diagram: Host1 JCP Host2 ------- ----- ------- CONNECTION-ID = SecondaryID8, DATA [T = 1, SOURCE-VIRTUAL-ADDRESS = SecondaryID2, DEST-VIRTUAL-ADDRESS = SecondaryID3, SEQUENCE-NUMBER = Number61 + 2, DATA-1 + DATA-2 = VM-Instructions] -> CONNECTION-ID = SecondaryID8, DATA [without changes] ------------------> <-- CONNECTION-ID = SecondaryID7, ACK [T = 1, SOURCE-VIRTUAL-ADDRESS = SecondaryID3, DEST-VIRTUAL-ADDRESS = SecondaryID2, SEQUENCE-NUMBER = Number62 + 2] <------------ CONNECTION-ID = SecondaryID7, ACK [without changes] For calculation of time-outs and number of repeated transfers, it is possible to use recommendations, given in [17]. The algorithms definite for TCP [6,7,8] are recommended for congestion control. As UMSP does not buffer received data and does not require ordered flows, the receiving window is set to infinite quantity. The congestions control is carried out separately for each UMSP connection. In addition, UMSP gives the special command for the notification of congestion on the JCP: CONGESTION. This command is sent from the JCP to the host. CONGESTION is processed the same as packet loss in procedure of congestion control on the host. Bogdanov Expires: February 2003 [Page 47] Internet-Draft Unified Memory Space Protocol August 2003 7.1 Format of Data Transmission Commands 7.1.1 DATA DATA command is used to send the VM instructions. The command is sent in the "host-host" and "host-JCP-host" connection. DATA has the following format: CONNECTION-ID : Contains the connection identifier allocated by the packet destination. T = 0/1 DATA-LENGTH - variable length OPCODE = 20 DATA-1 : Contains the VM data. SOURCE-VIRTUAL-ADDRESS : If T = 0, this field is absent. If T = 1, this field contains the private virtual network address of the source. DEST-VIRTUAL-ADDRESS : If T = 0, this field is absent. If T = 1, this field contains the private virtual network address of the destination. SEQUENCE-NUMBER : indicates the sequence number. DATA-2 : Contains the VM data. This field may be absent. 7.1.2 ACK ACK command is used to acknowledge the delivery of DATA commands on the UMSP layer. ACK includes cumulative confirmed number and may include selective acknowledgements. ACK has the following format: CONNECTION-ID : Contains the connection identifier allocated by the packet destination. T = 0/1 DATA-LENGTH - variable length OPCODE = 21 DATA-1 : Contains the number of one confirmed packet. Value is positive displacement from SEQUENCE-NUMBER of this command. Zero value is ignored. Bogdanov Expires: February 2003 [Page 48] Internet-Draft Unified Memory Space Protocol August 2003 SOURCE-VIRTUAL-ADDRESS : If T = 0, this field is absent. If T = 1, this field contains the private virtual network address of the source. DEST-VIRTUAL-ADDRESS : If T = 0, this field is absent. If T = 1, this field contains the private virtual network address of the destination. SEQUENCE-NUMBER : indicates the cumulative confirmed sequence number. All packets with this number and smaller numbers are delivered. DATA-2 : Includes the optional list of selective responses and has the following format: +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | GAP-BLOCK-1-START | GAP-BLOCK-1-END | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | ~ ~ | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | GAP-BLOCK-n-START | GAP-BLOCK-n-END | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ GAP-BLOCK-N-START + GAP-BLOCK-N-END : 16 + 16 bits These fields contain the beginning and the end (inclusive) of the selective acknowledgement block. Their value is positive displacement from CUM-SEQUENCE-NUM value of this command. The command may contain several blocks of selective acknowledgement. 7.1.3 CONGESTION CONGESTION command is used for the explicit notification of packets congestion on the JCP. The command is sent from the JCP to the host. CONGESTION is used in congestions control algorithm on the host and it is processed the same as packet loss. CONGESTION has the following format: CONNECTION-ID : contains the connection identifier allocated by the packet destination. T = 0 DATA-LENGTH = 1 OPCODE = 129 DATA-1 : Set to 0 on transmit and ignored on receipt. Bogdanov Expires: February 2003 [Page 49] Internet-Draft Unified Memory Space Protocol August 2003 SEQUENCE-NUMBER: is absent. Absence of this field is difference from common format of UMSP command (see section 5). DATA-2 : is absent. 8 Activity Control For activity check, the following commands are used: ACTIVITY_REQ - requests activity of the node. ACK - is acknowledgement of the node activity. ACTIVITY_REQ command is sent to check activity. Any received command acknowledges ACTIVITY_REQ. If response has not been received, repeated transfers are carried out. If response has not been received after the set quantity of repeated transfers, connection terminates. ACTIVITY_REQ command can be sent from the JCP to the hosts or between two hosts. ACTIVITY_REQ command is not used for connection control between JCP and host of the job initiator (on which the application have been started). The command of lifetime control is used for this purpose (see section 9). The activity control is RECOMMENDED to be carried out in connections with the long period of traffic absence and only if the resource use on the node has exceeded a limiting threshold. In all other cases, it is necessary to be guided by the job lifetime timer. The protocol defines maximal time of non-activity during 8 hours. If there is no any traffic during this time on connection, this connection can be terminated without sending of network primitives, irrespective of the job lifetime. If the node has a reserve of resources for maintenance of connections, it is not RECOMMENDED to send commands of the activity control. The activity controls of different jobs are not connected. 8.1 ACTIVITY_REQ ACTIVITY_REQ command is used to check the hosts' activity. It is sent from the JCP to the host or between hosts. ACTIVITY_REQ has the following format: CONNECTION-ID : Contains the connection identifier allocated by the packet destination. T = 0/1 DATA-LENGTH = 2/6 OPCODE = 22 Bogdanov Expires: February 2003 [Page 50] Internet-Draft Unified Memory Space Protocol August 2003 DATA-1 : Set to 0 on transmit and ignored on receipt. SOURCE-VIRTUAL-ADDRESS : If T = 0, this field is absent. If T = 1, this field contains the private virtual network address of the source. DEST-VIRTUAL-ADDRESS : If T = 0, this field is absent. If T = 1, this field contains the private virtual network address of the destination. SEQUENCE-NUMBER : indicates the sequence number. DATA-2 : is absent. 9 Job Lifetime The job lifetime is defined at registration of the job on JCP. The protocol does not define the job with unlimited lifetime, but it allows prolonging this time periodically. For lifetime control the following commands are used: LIFE_TIME_REQ - requests the information on the job lifetime. LIFE_TIME_SET - establishes a new lifetime of the job. JCP controls the lifetime of the job. If this time has expired, JCP sends LIFE_TIME_REQ command to the job initiator's host and waits LIFE_TIME_SET command or command of job termination. The job initiator's host can specify the new lifetime in LIFE_TIME_SET command. If lifetime has not been prolonged, JCP finishes the job. The host which is not a job initiator may request the new lifetime of the job if the current time has expired. The host sends LIFE_TIME_REQ to JCP, and waits LIFE_TIME_SET. If lifetime is not prolonged, the host will finish the job. It is recommended to check the job activity after some interval since lifetime has expired. The job initiator and the JCP should process LIFE_TIME_REQ command no more than five times during lifetime in one connection. If these restrictions are exceeded, LIFE_TIME_REQ commands are silently discarded. If sending queue is on the node, the LIFE_TIME_SET command must be sent with high priority. 9.1 LIFE_TIME_REQ LIFE_TIME_REQ command is used for request of job lifetime. LIFE_TIME_REQ has the following format: CONNECTION-ID : Contains the connection identifier allocated by the packet destination. T = 0 Bogdanov Expires: February 2003 [Page 51] Internet-Draft Unified Memory Space Protocol August 2003 DATA-LENGTH = 2 OPCODE = 23 DATA-1 : Set to 0 on transmit and ignored on receipt. SEQUENCE-NUMBER : indicates the sequence number. DATA-2 : is absent. 9.2 LIFE_TIME_SET LIFE_TIME_SET command is used for indication of the new job lifetime. LIFE_TIME_SET has the following format: CONNECTION-ID : Contains the connection identifier allocated by the packet destination. T = 0 DATA-LENGTH = 2/3 OPCODE = 24 DATA-1 : If DATA-LENGTH = 2, this field indicates the new life time of the job in minutes. If DATA-LENGTH = 3, this field is set to 0 on transmit and ignored on receipt. SEQUENCE-NUMBER : indicates the sequence number. DATA-2 : If DATA-LENGTH = 2, this field is absent. If DATA-LENGTH = 3, this field has the following format: +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | RESERVED | JOB-LIFE-TIME | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ RESERVED : 16 bits Set to 0 on transmit and ignored on receipt. JOB-LIFE-TIME : 16 bits Indicates the new job lifetime in seconds. 10 Identification of VM Type VM are the identified resources of the protocol. The VM standardization is not UMSP function. The protocol gives the transparent environment for code and data transfer of any type. Bogdanov Expires: February 2003 [Page 52] Internet-Draft Unified Memory Space Protocol August 2003 For VM identification, the following values are definite: o VM type. Values range is 1 û 65534. o VM version. Values range is 1 û 65534. The protocol requires obligatory bottom-up compatibility for VM of one type and different numbers of versions (VM with larger version number must execute the VM code with any smaller version number). The following numbers of VM types are defined: 1 - 1023 Assigned for standard VM. 1024 - 49151 Assigned for registered VM of users. 49152 - 65534 Free (defined for dynamic and/or private VM). Numbers of types and versions %x0000 and %xFFFF are reserved by the protocol. The host can request other hosts and inform about supported VM versions. The following commands are used for this purpose: VM_REQ - requests the list of supported VM. VM_NOTIF - informs the list of supported VM. This command may be sent independently or as the response to VM_REQ. VM_NOTIF_JCP - This command is similar to VM_NOTIF. The difference is the indication of the host virtual network address together with VM type. JCP sends this command. VM_REQ, VM_NOTIF and VM_NOTIF_JCP commands can be sent in the following way: O By the primary connection with JCP. If TCP is used as primary connection, CONNECTION-ID field must be set to 0. O Without a connection establishment. CONNECTION-ID field set to 0. Two methods of resources identification are possible: (1) Hosts may exchange VM_REQ and VM_NOTIF commands immediate. Broadcasting and multicast dispatch is possible. (2) The host may establish the primary connection with JCP. The identifier of this connection on JCP is the public virtual network address of the host. It is necessary to send VM_NOTIF on this connection. The subsequent interaction with this host is possible through this JCP. JCP notifies on resources of such hosts by means of the VM_NOTIF_JCP command. The public virtual network address may be used as the destination address at a connections establishment. This Bogdanov Expires: February 2003 [Page 53] Internet-Draft Unified Memory Space Protocol August 2003 method may be used for opaque networks and for interaction of IPv4 and IPv6 networks through JCP with two interfaces. 10.1 VM_REQ VM_REQ command is used for request of supported VM types. VM_REQ has the following format: CONNECTION-ID : Contains the connection identifier allocated by the packet destination. If the command is sent without UMSP primary connection, this field is set to 0. T = 0 DATA-LENGTH - variable length OPCODE = 25 DATA-1 : Set to 0 on transmit and ignored on receipt. SEQUENCE-NUMBER : Indicates the sequence number by sending by UMSP connection. If CONNECTION-ID = 0, this field is absent. DATA-2 : This field contains the list of required VM types. If DATA-LENGTH = 2, this field is absent. Each record in the list has the following format: +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | VM-TYPE | VM-VER | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ VM-TYPE : 16 bits Indicates required VM type. VM-VER : 16 bits Indicates the minimal necessary version for this VM type. If VM_REQ does not contain the list of required VM types, all types are requested. 10.2 VM_NOTIF, VM_NOTIF_JCP VM_NOTIF, VM_NOTIF_JCP commands are used for notification about supported VM types. VM_NOTIF, VM_NOTIF_JCP differ only in format of the list of supported VM: CONNECTION-ID : Contains the connection identifier allocated by the packet destination. Set to 0, if the command is sent without UMSP primary connection. T = 0 Bogdanov Expires: February 2003 [Page 54] Internet-Draft Unified Memory Space Protocol August 2003 DATA-LENGTH - variable length OPCODE = 26 - for VM_NOTIF 27 - for VM_NOTIF_JCP DATA-1: Set to 0 on transmit and ignored on receipt. SEQUENCE-NUMBER : If CONNECTION-ID # 0, the value is copied from SEQUENCE-NUMBER field of VM_REQ command. If the transmitted command is not the response on VM_REQ, this field is set to 0. If CONNECTION- ID = 0, this field is absent. DATA-2 : This field contains the list of supported VM types. For VM_NOTIF, each record in the list has the following format: +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | VM-TYPE | VM-VER | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ For VM_NOTIF_JCP, each record in the list has the following format: +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | + VIRTUAL-NETWORK-ADDR + | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | VM-TYPE | VM-VER | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ VIRTUAL-NETWORK-ADDR : 64 bits Indicates the public virtual network address of the host for this VM type. VM-TYPE : 16 bits Indicates the supported VM type. VM-VER : 16 bits Indicates the maximal version for this VM type. All records MUST be ordered on increase of VM-TYPE value. If VM_NOTIF or VM_NOTIF_JCP is the response on VM_REQ with the list of necessary VM, the response contains only the requested VM types. Bogdanov Expires: February 2003 [Page 55] Internet-Draft Unified Memory Space Protocol August 2003 11 Security Consideration IPsec use [14] is considered as the basic means of protection against the attacks. Presence of the protected connection between the host and the JCP is sufficient to protocol work. The protected connections between hosts allow reducing the network traffic. Use of protection on the application level is inefficient against the attacks directed to connections break. References [1] Bradner, S., "The Internet Standards Process -- Revision 3", BCP 9, RFC 2026, October 1996. [2] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", RFC 2119, March 1997. [3] Postel, J., "User Datagram Protocol", STD 6, RFC 768, August 1980. [4] Postel, J., "Transmission Control Protocol - DARPA Internet Program Protocol Specification", STD 7, RFC 793, September 1981. [5] Bogdanov, A., "Unified Memory Space Protocol Specification", RFC 3018, December 2000. [6] Allman, M., Paxson, V. and W. Stevens, "TCP Congestion Control", RFC 2581, April 1999. [7] Allman, M., Floyd, S., Partridge, C., "Increasing TCP's Initial Window", RFC 3390, October 2002. [8] Dawkins, S., Montenegro, G., Kojo, M., Magret, V. and Vaidya, N., "End-to-end Performance Implications of Links with Errors", RFC 3155, August 2001. [9] Stewart, R., Xie, Q., Morneault, K., Sharp, C., Schwarzbauer, H., Taylor, T., Rytina, I., Kalla, M., Zhang, L. and V. Paxson, "Stream Control Transmission Protocol", RFC 2960, October 2000. [10] Karn, P. and W. Simpson, "Photuris: Session-Key Management Protocol", RFC 2522, March 1999. [11] Mogul, J. and S. Deering, "Path MTU Discovery", RFC 1191, November 1990. [12] McCann, J., Deering, S. and J. Mogul, "Path MTU Discovery for IP version 6", RFC 1981, August 1996. [13] Elz, R. and R. Bush, "Serial Number Arithmetic", RFC 1982, August 1996. Bogdanov Expires: February 2003 [Page 56] Internet-Draft Unified Memory Space Protocol August 2003 [14] Kent, S. and R. Atkinson, "Security Architecture for the Internet Protocol", RFC 2401, November 1998. [15] Deering, S. and R. Hinden, "Internet Protocol, Version 6 (IPv6) Specification", RFC 2460, December 1998. [16] Dawkins, S., Montenegro, G., Kojo, M., Magret, V. and Vaidya, N., "End-to-end Performance Implications of Links with Errors", RFC 3155, August 2001. [17] Paxson, V. and M. Allman, "Computing TCP's Retransmission Timer", RFC 2988, November 2000. [18] Srinivasan, R., "RPC: Remote Procedure Call Protocol Specification Version 2", RFC 1831, August 1995. Author's Address Alexander Bogdanov 23-126, Generala Kuznetsova St. Moscow, Russia 109153 RU Phone: +7 095 706 1002 Email: a-bogdanov@umsp.net Web: www.umsp.net Bogdanov Expires: February 2003 [Page 57] Internet-Draft Unified Memory Space Protocol August 2003 Full Copyright Statement Copyright (C) The Internet Society (2003). All Rights Reserved. This document and translations of it may be copied and furnished to others, and derivative works that comment on or otherwise explain it or assist in its implementation may be prepared, copied, published and distributed, in whole or in part, without restriction of any kind, provided that the above copyright notice and this paragraph are included on all such copies and derivative works. However, this document itself may not be modified in any way, such as by removing the copyright notice or references to the Internet Society or other Internet organizations, except as needed for the purpose of developing Internet standards in which case the procedures for copyrights defined in the Internet Standards process must be followed, or as required to translate it into languages other than English. The limited permissions granted above are perpetual and will not be revoked by the Internet Society or its successors or assigns. This document and the information contained herein is provided on an "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Bogdanov Expires: February 2003 [Page 58]