Network Working Group Jim Boyle Internet Draft Craig White Expiration Date: January 2001 Joe Lawrence Level 3 July 2000 SONET Synchronous Transport Signal over IP draft-boyle-sts-ip-00.txt Status of this Memo This document is an Internet-Draft and is in full conformance with all provisions of Section 10 of RFC2026. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet- Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt. The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. Boyle, et al. [Page 1] Internet Draft draft-boyle-sts-ip-00.txt July 2000 Abstract A proposal is made to carry Synchronous Transport Signal (STS-N) services over packetized networks, in particular over IP networks. This has the potential to dramatically lower the price, ease the provisioning and grooming, and improve managability of delivering these types of circuits. Two proposals are put forth for discussion on how STS services could be transported over an IP network. The first proposal, referred to as SPE1/IP, is to break OC-N signals down into their STS-1 blocks, identify the SPE components and packetize these for transmission across a packet network. The second proposal, referred to as STSc/IP breaks down the STS signal into the underlying concatenated services and transports configurable length segments of the signal to the far end of the data network. With either of these approaches, different STS-N services on a given port can be connected to different ports across the network and the network will groom them onto the most optimal path available. Table of Contents 1 Specification of Requirements .......................... 3 2 Introduction ........................................... 3 3 SONET Background ....................................... 4 4 Encapsulation Alternatives ............................. 6 4.1 SPE-1 transport over IP (SPE1/IP) ...................... 6 4.2 STS-Nc transport over IP (STSc/IP) ..................... 8 4.3 Comparison of SPE1/IP and STSc/IP ...................... 8 5 Issues of Efficiency ................................... 9 5.1 Efficiency Losses ...................................... 9 5.2 Efficiency Gains ....................................... 9 6 Functionality Requirements ............................. 10 6.1 Playback buffers on IP/TDM conversion .................. 10 6.2 QoS in IP network ...................................... 10 6.3 Traffic engineered control plane ....................... 10 7 Security Considerations ................................ 11 8 Intellectual Property Disclaimer ....................... 11 9 Acknowledgements ....................................... 11 10 References ............................................. 11 11 Author Information ..................................... 11 Boyle, et al. [Page 2] Internet Draft draft-boyle-sts-ip-00.txt July 2000 1. Specification of Requirements The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119. 2. Introduction Anyone familiar with operating a private line network understands that, even though there maybe additional costs, technologies that ease grooming and provisioning are worth consideration. This is one of the areas where the next generation of transmission equipment is improving by leveraging IP based control protocols and head of service automated provisioning. However, there do exist some issues of multi-vendor compatibility with this approach. This makes an approach where private line services are converted into an open data protocol, such as RTP/IP and/or MPLS attractive. By converting the TDM service to IP, the provider is now also able to use network elements such as ethernet switches which are geared to much wider, mass markets and potentially ride a much more aggressive price/performance curve than standard telecommunication equipment. These benefits were the impetus of the authors investigation into converting high capacity private line services, STS1 and STSNc services in particular, into IP for transport over a data network. Investigation yielded the result that this could likely be done with rather plain technology, as is used in current transmission equipment, and might yield significant network savings due to signal compression potential. This memo is intended as a strawman statement of approach and essential functionality. The description, though written rather close to the physical and network layer, is actually intended to be more in line with a transport protocol, such as RTP. In fact, though this memo is written in the form of a protocol specification, it may well be that this functionality can, and should, be adapted to existing protocols to ease implementation. Boyle, et al. [Page 3] Internet Draft draft-boyle-sts-ip-00.txt July 2000 3. SONET Background SONET, as its name implies, is a hierarchical protocol for synchronously transporting TDM circuits across an optical network. Over the length of a circuit, there is a hierarchical breakdown of functionality which allows for multiplexing, service monitoring and alarming and signal degradation monitoring at high data rates. From end to end, at the highest layer, SONET provides monitoring service via it's Path overhead. This is transported to the customer and allows for continuity verification as well as error calculation over the service payload. The physical optical fiber is often in a 1:1 relationship with the section, and often directly corresponds to the line. The difference is that multiple lines usually terminate at a Digital Cross Connect (DACS) or Add Drop Multiplexer (ADM) where synchronization and alignment across lines is required. With the advent of WDM, the section no longer directly corresponds to a fiber, it refers to logically adjacent SONET equipment such as an ADM and a 3R. In fact, with some WDM products, it is possible for the WDM equipment to operate in a non section processing, or pass-through, mode in which case the WDM signal has 1:1 parity of Path, Line and Section. Here is a diagram to illustrate SONET's signal hierarchy: |---------------------- Path -----------------------| |--- Line ---|--- Line ---|--- Line ---|--- Line ---| |--- Sect ---| *** See Figure 2 *** |--- Line ---| Router------ADM----------ADM----------ADM------Router Figure 1 - SONET Signal Hierarchy Below is a blowup of one of the ADM-ADM Lines to illustrate how SONET section no longer corresponds directly to a fiber. ADM-----WDM-----ILA-----3R-----ILA-----WDM-----ADM |----------------------Line----------------------| |-----------Sect--------|------------Sect--------| Figure 2 - SONET Line and Section in WDM environment In SONET, the digital signal is referred to as an STS (Synchronous Transport Signal). This signal is broken down into a repetitive series of STS frames, each frame for an STS being generated every 125 us (8000 fps). Each frame can be thought of as having 90 columns and 9 rows. The first 3 columns contain the section and line overhead. The path overhead is contained within the Synchronous Payload Envelope (SPE). Boyle, et al. [Page 4] Internet Draft draft-boyle-sts-ip-00.txt July 2000 +-----------------------------------------------------+ | | | | ... | | SOH | ... | |_|_|_| ... | | | | | ... PATH OH and Payload | | | | | ... | | LOH | ... | | | | | ... | | | | | ... | | | | | ... | +-----------------------------------------------------+ Figure 3 - STS frame with transport overhead +-----------------------------------------------------+ | | | | | | SOH | | |_|_|_| _____________________________________| | | | |_________| | | | | | | |P| | | LOH | |O| STS-1 SPE | | | | | |H| | | | | | | | | |_|_|_|_ _ _ _ _|N|_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _| | | | | | | | | SOH | | | | |_|_|_| |_|___________________________________| | | | |_________| | | | | | ... | | LOH | ... | | | | | ... | | | | | ... | | | | | ... | +-----------------------------------------------------+ Figure 4 - STS frame including SPE Notice that the payload floats inside the STS frame. It is referenced in the LOH with a Path Payload Pointer, and can change. This is to allow slight variations in line rates on DACS or ADMs which a path traverses. The LTE can then either byte positive or negative stuff a signal. The basic building block of SONET is the STS-1. This is always carried over a physical facility, in particular a fiber operates at a bit rate which is a specified multiple of STS-1s. The STS-1s are byte multiplexed together as show in Figure 5. Boyle, et al. [Page 5] Internet Draft draft-boyle-sts-ip-00.txt July 2000 STS1-#01 + \ STS1-#02 +-+- #03#02#01 + / \ STS1-#03 + \ \ STS1-#04 + \ \ \ STS1-#05 +-+- #06#05#04 + \ / \___ \ STS1-#06 + \__\ __ + #12#09#06#03#11#08#05#02#10#07#04#01 STS1-#07 + ___/ / \ / / STS1-#08 +-+- #09#08#07 + / / / STS1-#09 + / / STS1-#10 + / \ / STS1-#11 +-+- #12#11#10 + / STS1-#12 + Figure 5 - STS-12 multiplexing of underlying STS-1s The byte pattern for the STS-12 presented across an OC-12 #12#09#06#03#11#08#05#02#10#07#04#01-> is a byte interleaving of all the STS-1s. In 125 us, a byte from each STS-1 is transmitted over the OC-12 signal. At each SONET switching element, it is customary for the signal to be demultiplexed into the STS-1s into a small buffer and then for the appropriate egress port to pull the signal out of the buffers during its multiplexing. 4. Encapsulation Alternatives 4.1. SPE-1 transport over IP (SPE1/IP) In this mode, the STS-1s from an STS-N signal are broken down and are written into a buffer. There is no need to forward any LOH or SOH, however it is required to identify and forward the POH so that the customer can monitor continuity and performance. A complete SPE is written out in several SPE segments to a buffer. An SPE segment refers to the portions of an SPE that reside in different STSs. Although the POH for different services within an STS-N structure may be located at different offsets from their respective LOH, the SPE for concatenated STSs must be treated as all at the same offset as the first STS in the concatenated service. The pointer to the start of the SPE is known for all STSs within an existing service, and a complete SPE has been written out to buffer. At this point, the SPE Boyle, et al. [Page 6] Internet Draft draft-boyle-sts-ip-00.txt July 2000 segments are encapsulated and forwarded across the network. The essential information for a header would include: SPE1/IP Header (6 bytes) o Version field (4 bits = 1) o Operation Code, OP, (2 bits) - 00 = standard user data format - 01 = Unused, reserved for extended data format - 10 = OAM - 11 = unused o Parity bit (2 bits) - P0 - bit 1, even parity across header - P1 - bit 2, even parity across entire PDU o STS reference (16 bits) o Sequence number (16 bits) +-------------+-------------+-------------+-------------+ |V |OP|PP | Unused | STS Reference | +-------------+-------------+-------------+-------------+ | Sequence Number | +-------------+-------------+ The sequence number is synchronized across all of the SPE segments of an SPE for a given concatenated service and increases by 1 every 125 us. At 8000 fps, it would wrap in just under 8 seconds. If a packet is dropped, then an all 1s segment is inserted in its place for multiplexing into the next line in the service. Bit errors within the frame are noticed by standard SONET monitoring facilities, in this case the B3 byte in the POH will alert the end-user with potentially 1 or 3 BIP-8 violations depending on if the bit error happened in the B3 byte of an SPE or not. For all 1s inserted segments that are multiplexed into an SPE, at least 3 BIP-8 violations would occur at the path layer. Bit errors in the encapsulation header are recognized in a non-recoverable fashion by the parity bit P0 which is set or cleared to create even parity across the rest of the header. The packet should be discarded if such a bit error is detected in the encapsulation header. The P1 bit can be used to check parity for bit error across the header and the SPE. This can be done for service management to determine if the data network has an unexpectedly high bit error rate. This is similar to section and line monitoring in a standard SONET network. The STS reference is similar to a UDP port and remains constant for a given STS traversing a node. It is set by the node that converts the STS from the TDM to the IP domain. Boyle, et al. [Page 7] Internet Draft draft-boyle-sts-ip-00.txt July 2000 4.2. STS-Nc transport over IP (STSc/IP) In this mode, the contents of the concatenated SPE are written out to a buffer upon demultiplexing of the TDM line into the TDM/IP conversion platform. The signal is then sampled at a configurable block size, 1000 bytes for example, and this information is encapsulated and sent across the IP network STSc/IP Header (8 bytes) o Version field (4 bits = 1) o Operation Code, OP, (2 bits) - 00 = standard user data format - 01 = Unused, reserved for extended data format - 10 = OAM - 11 = unused o Parity bit (2 bits) o STS reference (16 bits) o Sequence number (32 bits) +-------------+-------------+-------------+-------------+ |V |OP|PP | Unused | STS Reference | +-------------+-------------+-------------+-------------+ | Sequence Number | +-------------+-------------+-------------+-------------+ The number of frames per second are a function of the number of the sample size and the service bit-rate. Sequence number wrap can be derived from that, but with a range of 0 to 2^32-1, a worst case scenario of sampling an OC192 at 100 bytes yields wrap in over 5 minutes. Parity bits are as in section 3.2. The POH still must be identified in the transported payload. One approach would be to use integer math on the sequence number and have the concatenated STSc be segmented into chunks which together exactly construct the payload of one frames worth of SPE. Another might be to use an extended data format that points into the payload to identify the start of SPE for each frame. 4.3. Comparison of SPE1/IP and STSc/IP SPE1/IP is not very flexible, but in such an approach a certain simplicity is gained. In fact, most TDM gear today is built around breaking down services into STS-1 for switching. STSc/IP is very flexible, but that might add too much complexity to implementation and interoperability. Either approach requires that the service provider be intimate with the channel structure of the OC services they provide, either through static provisioning or through Boyle, et al. [Page 8] Internet Draft draft-boyle-sts-ip-00.txt July 2000 derivation from the LOH. 5. Issues of Efficiency 5.1. Efficiency Losses As SONET is non-hierarchically built around the STS-1 payload, with direct multiplexing, the overhead remains constant at around 4% (3 out of 90 columns, plus POH in first STS-1). It doesn't matter if the services are all STS-1s, or a concatenated service occupying the whole linespeed such as an STS-12c in an OC-12c, there exist LOH and POH for each STS. This creates a bit of an issue when the payload is encapsulated and forwarded out into an IP network. As an example, if one of the core links in a network was a OC-192c, it would not be able to carry 192 STS-1s of service, as the IP traffic must itself become SONET payload on this link. In this case, one provisioned service is going to have to take another route, which introduces the need for skew recovery buffers. This could be problematic for networks whose service bandwidth approaches their internal link speeds. Perhaps it is time to try to take use of all those D and E bytes in the SONET header? Even that wouldn't carry the additional 3-5% of overhead incurred by IP encapsulation. One should note that an OC-12 can be routed across a Gigabit Ethernet, and the assumption is that volume and target market of Gigabit Ethernet make it likely cheaper than the corresponding SONET interface. 5.2. Efficiency Gains The losses outlined in the previous section appear problematic for this application, however there exists a large potential for reducing the amount of network that has to be built out to support a set of private line services by using technology like this. It may be trivial to remove bandwidth from STS services being used for data transport by compressing predictable frame sequences. Suggested sequences include HDLC idle (0x7Es), STS services that are either in alarm (AIS) or are not in service (unequiped) (all ones), unequiped asynchronous DS3 and idle ATM cells. One approach for compression could be to send a packet that has a the base sequence written just once, and it is to be repeated out to fill out the remainder of the SPE. Another approach, especially for unequiped circuits would be to use signalling to establish state on either end of the circuit reflecting unequiped and transport nothing through the data network. For a carrier who has sold an OC12 structured as 12 STS1, of which only 8 are in service, a significant savings may be realizable. Boyle, et al. [Page 9] Internet Draft draft-boyle-sts-ip-00.txt July 2000 Further efficiency gains can be realized by relaxing the timing precision of the service if the STS payload is HDLC, PPP or FR [Martini]. However, this draft outlines an approach to exactly mimic a TDM based SONET service. 6. Functionality Requirements 6.1. Playback buffers on IP/TDM conversion The extent of playback buffering required (and user experienced latency) is somewhat traded off with QoS of IP network described in the following section. The playback logic must allow for packets received out of sequence to be ordered correctly and played out in sequence and insertion of all ones payload for packets that didn't arrive in time for enqueue. Also, if the contents of an STS-Nc service take multiple routes through the network, additional skew recovery buffers will be required. 6.2. QoS in IP network It is expected that the network is not chronically congested. For a multiservice network, the TDM/IP (and other delay sensitive traffic such as VoIP) can be forwarded without too much delay. However, in conjunction with the previous section, it might not be required that this be strict "next packet served" type queuing. 6.3. Traffic engineered control plane There must be a way to keep proper coordination of the traffic assigned to network bandwidth resources. The application requires a way to signal the bandwidth it requires from the network. One potential approach is use of the Resource Reservation Protocol [RSVP]. A more direct approach would be to have the host on which the application runs signal traffic engineered MPLS tunnels [RSVP- TE][CRLDP] between itself and other TDM/IP conversion devices with which it has common services. These tunnels could be sized to accommodate multiple STS services, or sized to carry just one, or a portion of one, STS service. MPLS tunneling would allow the TDP/IP device to find a route for their traffic across the network, potentially find backup routes, and groom their traffic onto new, more optimal paths in the network. The concern would be one of scalability as all TDM/IP devices would have to belong to a common TE-IGP domain. An alternate approach would be for the TDM/IP devices to communicate the bandwidth information of the service via loosely routed TE-LSPs, and to have switches along the way with more complete Boyle, et al. [Page 10] Internet Draft draft-boyle-sts-ip-00.txt July 2000 information insert the route the service should take. 7. Security Considerations 8. Intellectual Property Disclaimer This document is being submitted for use in IETF standards discussions. Level 3 Communications, Inc. has filed one or more patent applications relating to the technology outlined in this document. 9. Acknowledgements This draft has benefited from the discussions and assistance provided by Tom Issenhuth, Jack Waters and Danny McPherson, among others. 10. References [Martini] "Transport of Layer 2 Frames Over MPLS", draft-martini- l2circuit-trans-mpls-01.txt (work in progress) [RSVP] "Resource ReSerVation Protocol -- Version 1 Functional Specification", RFC2205 [RSVP-TE] "Extensions to RSVP for LSP Tunnels", draft-ietf-mpls-lsp- tunnel-06.txt (work in progress) [CR-LDP] "Constraint-Based LSP Setup Using LDP", draft-ietf-mpls-cr- ldp-04.txt (work in progress) 11. Author Information Jim Boyle Level 3 Communications, LLC. 1025 Eldorado Blvd. Broomfield, CO 80021 e-mail: jboyle@level3.net phone: 720.888.1192 Boyle, et al. [Page 11] Internet Draft draft-boyle-sts-ip-00.txt July 2000 Craig White Level 3 Communications, LLC. 1025 Eldorado Blvd. Broomfield, CO 80021 e-mail: craig.white@level3.com phone: 720.888.2375 Joe Lawrence Level 3 Communications, LLC. 1025 Eldorado Blvd. Broomfield, CO 80021 e-mail: joe.lawrence@level3.com phone: 720.888.1000 Boyle, et al. [Page 12]