Mbps to Tbps : Optical Transport Evolution

Toughest question to answer to my kid when he asks what "Dad what do you work on in office?". Simple answer which he can understand is "working on the laptop" or "programming" etc. But if we think deeper about our work, the correct answer should be that "whatever we do is used in transforming the life of a common man". Yes, you heard it right. We (telecom engineers) are transforming the world just by revolutionizing the communication and making difference to their daily life. Just imagine how your world would have been if there was no mobile, smart phone, internet? The unfortunate part is when a common man talks about is this revolution, he always talks about the end stuffs, like smart phone revolution, sleek design tablets, internet download speed and about the data plan which he is having. In terms of the devices we talk about our mobile models and wifi modem connectivity etc. Hardly a common man gives a thought on "who is enabling the Service Providers to move the large amount of data from our mobiles/laptops to it's destinaton?". It is "We" or "everyone working in the telecommunication space" who playing a big role in revolutionizing the networks. We work on the technologies/equipment's which transports the voice and data from every common man's laptop, mobile, tablet, smart TV, modem etc to its destination which could be as far as 10000 km away. For transporting your goods from one place to other, you need roads/railway lines and vehicles. On the same line to transport all your voice and data from one place to other you need "transport medium" and "transport technologies". In this article I would try to capture evolution of various transport technologies over few decades, thankfully most of which I have been part of while working over different products in my career.

Start with voice Transport : PDH

As the communication started with the simple telephone line, Telecommunication transport started majorly with the voice transport networks. During initial couple of decades data was not that pre-dominant but it was the voice transport for which transport networks were required.

What is voice over the wire?

First let's understand what voice translates to when it is converted as electrical/digital signal. In telephony usable voice frequency band ranges from 300 Hz to 3400 Hz. Thanks to Mr. Nyquist sampling theorem which dictated that "any analog signal can be recovered if it is sampled at twice of its highest rate". In order to recover the voice signal at the other end it should be sampled with the 8 KHz or 8 sample per seconds or 8*8(8 bits per sample)=64 kbps. That's how we got our voice bit rate running on the wire, it is 64 Kbps. T-1 carrier system started by Bell system in 1962 consists of 24 such telephone lines.

 Evolution of Telecommunication transport started primarily with the voice transport for which plesiochronous digital hierarchy was standardized to define the higher order digital voice signal multiplexing. PDH was developed in early 1960s. It drives its name from Greek term “Plesio” which means “near” and “Chronos” which means “time”. It was a time division multiplexing of lower order signals to higher order signals. Whole first few generation transport network was built using TDM multiplexing so it is very important to understand TDM First.

 How to ride the train? answer - have your timeslots.

 Figure 1: Time Division Multiplexing

As shown in the figure above, In the time division multiplexing scheme each stream gets opportunity to place its data in the time slot provided. What it means that at the receiver end it should be known that which stream to recover from which time slot. For this very purpose it is very important that transmitting device and receiving device are using the same frequency clock.

You use your clock I will use mine.

As shown in the figure below, in PDH, 64 kbps voice signals get TDM multiplexed at multiple levels and transported to the other end. However due to the fact that each end used their own clock sources, the frequency at both end used to be closer but exactly not the se. his leads to the timing slips and mismatches for which bit stuffing was used.

 Figure 2: PDH Multiplexing Hierarchy

 And PDH had major issues.. 

• Timing slips : Because if bit stuffing of timing slips due to independent clocks at sender and receiver end , to identify an individual channel from the higher bit-rate stream, it was required to demultiplex down to the level where your tributary channel lies. This was considered as the major in-efficiency.
• Lack of OAM : Because of no agreed standard for management or to monitor the performance.
• Lack of a standardized definition: There was no standard for PDH rates greater than 140 megabits per second. An alternative was needed.

 1980-90: Issues PDH Time to look for a new standard

Keeping in-view the limitation of the PDH, objectives were clear. A new transport technology was required where is synchronous, capable of supporting higher bit rates with great OAM capability and direct access to the low bit rate tributaries. It took nearly 10 years for the transport networking industry to migrate from PDH to SONET. 1984 breakup of the Bell Operating companies in USA created immense pressure for inter-exchange carriers to have the standard optical interface. Finally, Bell core drafted a standard called SONET in USA which went through revisions before emerging in late 80s and early 90s. A similar standard, Synchronous Digital Hierarchy (SDH), is used in Europe by the International Telecommunication Union Telecommunication Standardization Sector (ITU-T). SONET equipment is generally used in North America, and SDH equipment is generally accepted everywhere else in the world. SONET was introduced as a synchronous transmission system that could directly extract low-speed signals from multiplexed high-speed traffic.

SONET/SDH offered following advantage over the PDH

 - Capability to handle much Higher rates (~ 40 Gbps)

 - Overhead bytes for Strong OAM

 - Well defined standards allowing multi-vendor inter-operability

 - Efficient multiplexing and demultiplexing using well synchronized network wide timing from the highly accurate standard reference clock   

 A SONET/SDH transmission network is composed of several pieces of equipment, including:

• Terminal multiplexer (TM)
• Add-drop multiplexer (ADM)
• Repeater
• Digital cross-connect system (DCS)

 

Figure 3: SONET/SDH Equipment Definition

 1990s – Revolutionary deployment of SONET/SDH

Transport oriented features and capability to carry multiple protocol type very quickly made SONET/SDH de facto standard for voice, data and streaming media applications. By the end of 1990, a core network infrastructure typically has series of SONET/SDH protected rings (BLSR) carrying voice and data from various traffic sources. SONET/SDH which was primarily designed for the voice traffic stayed for quite a significant time. However, as the data networking started growing, demand for SONET/SDH to carry data started to be prevalent.

Next Gen SONET: Birth of MSPP

As the SONET grew a bit older, a new type of SONET cross-connect/ADMs evolved called as Multi Service Provisioning platforms. These MSPP offered only the traditional voice services ranging from DS1 to OC1-192 but also the data services like ATM and Ethernet. A typical first generation MSPP used to have the chassis with centralized cross-connect fabric with lower and higher order (VT and STS level) with the various optical (OC1,3), PDH, ATM and Ethernet line cards. Initially developed for the ILEC community, the MSPP has found applications within large enterprises-universities, governments, health, and financial institutions-and even utility customers, who use the MSPP in their internal communications networks to facilitate data collection and data backup. Due to photo copyright I have tried to draw a typical MSPP on my own here just to give a glimpse of how it looked initially.

 Figure 4 : A typical MSPP

  A typical MSPP has following characterstics

• Accepts a variety of tributary and line circuit packs to supports interfaces from DS1/DS3 to OC-192 (E1 to STM-64 for SDH markets).
• On the line side a shelf can be deployed as OC-3/12/48/192 (STM-1/4/16/64) in a ring protection(UPSR/BLSR).
• Central redundant switching fabrics that support STS-1 (VC-3) and VT1.5 (VC12) grooming and can be used as a digital cross-connect system (DCS) for grooming DS1s and DS3s.
• Supports data interfaces such as Ethernet (10/100/1000 Base-T, GigE, 10GE), ATM etc.

  Figure 5 : MSPP deployment in Ring

My encounter with SONET /SDH and MSPP (2002)

My face-off with the SONET/SDH first happened while I got an opportunity to work on the ATM line card going in into the one of the first few MSPPs in the world. Optical line cards used to have the fixed ports to connect the fiber whereas electrical line cards majorly used to get receive the electrical signals from the back through the connector on a patch panel. That was the reason a typical Lab with the MSPP look like a jungle of copper cables and fibers with the MSPP chassis. Optical OC-192 cards and XC cards providing STS and VT cross-connects normally used to take the double slot. I remember XC cards used to be too bulky and you sometime need the healthiest person of your team to pull it out of chassis. One thing worth mentioning here is that these first generation MSPPs used to have a line card of each optical rate as the concept of Flex port (SFP Pluggable) wasn’t there till that time.

 SONET and Data transport

SONET/SDH initially developed for the voice centric application did have a provision for the concatenated payloads to carry the data protocols like ATM, Ethernet. However due to the fixed bandwidth pipe, it proved in-efficient for carrying data on SONET. However, as the data demand grew Ethernet over SONET has been made possible by the development of generic framing procedure (GFP), link capacity adjustment scheme (LCAS), and virtual concatenation (VC). Using virtual concatenation, the SONET/SDH transport pipes may be "right-sized" for Ethernet transport.

SONET/SDH Limitations 

has several limitations.

• SONET/SDH cannot scale beyond 40 Gbps and has problems handling high-bandwidth clients such as video.
• SONET/SDH has limited Forward Error Correction (FEC) capabilities. FEC is an error correction method required to transmit 10 Gbps signals or greater over any reasonable distance.
• SONET/SDH standards could not be extended into WDM networks, including: Standards for mapping client traffic on to wavelengths, such as video, storage, and GDPS mapping standards. The absence of client mapping standards meant

that signals had to be de-multiplexed into their basic elements between the boundaries of carriers using incompatible equipment.

 “All Optical Network” a concept which never became reality but OTN was born

During the telecom bubble in late 90s and early 2000, after advent of the DWDM networking there was a very high hope and speculation that all optical networks would be reality very soon. This “All Optical Networking” model was based upon the assumption that client signals would be carried over the wavelengths and they will be switched across the network without converting it to the electrical domain. Problem with the switching in the pure optical domain was that if the client signal is sent over a wavelength in its native format it would be nearly impossible for the network operator to provide OAMP specific to the client signal.

Keeping in view this ITU-T SG15 developed series of OTN standards for wavelength division multiplexed network that covered physical layer, signal rate and format specification. OTN standard came up with the optical wrappers to carry different client signals and transport them transparently. Standard defined the specifications for both electrical as well as optical layers for client signal mapping, multiplexing and its transport over different wavelength.

 Figure 6: OTN Transparent Transport of various services

OTN: Advantage over SONET/SDH

There is a common misconception that OTN is just an enhanced version of SONET/SDH for higher speed. Although the multiplexing structure in OTN and SONET/SDH are quite similar but OTN was created as the carrier grade technology to work in the multi-vendor and multi domain environment with great transparency, reach, scalability and error correction. It has following clear advantages over SONET/SDH

• OTN supports FEC which corrects a very high number of errors due to noise in the transmission channel and hence extends the distance between repeaters.
• Provides transparency to all the client types through the digital wrapper.
• Enhances OAM
• Tendom connection monitoring

 OTN: Evolution

 Figure 7 : G.709 Evolution

Originally introduced in 2001, OTN has proven capable of continuing to evolve and adapt to cover new applications and technology in the transport network. In the figure above, By looking at the different G.709 versions launched we can see how OTN has evolved in last 15 years. As the time progressed additions not only done to support the higher bit rate (40 Gbps and 100 Gbps) wrapper specification but also to make OTN more granular(introduction of ODU0 and ODU Flex) and flexible to adopt to the other bit rate protocols. Introduction of ODU0 and ODU Flex also added the switching capability in the OTN devices.

OTN: Adoption to Ethernet

When OTN was invented fixed rate clients were limited to the SONET/SDH. There was an assumption made that any client will be first mapped to the SONET/SDH and then SONET/SDH will be mapped to OTN payload. However, standard did define direct mapping of GFP and ATM frames to the OPU payload. This situation was resolved in late 2008 when specification came to map 1GE ethernet client to map directly into ODU0 (a new payload type).

OTN Equipments

OTN Technology offers variety of equipments for electrical as well as optical domain transponders, multiplexers and cross-connects.

Transponders : Normally line cards which maps the clients signals to the OTN payload which is transmitted either from one of the OTN port on the same line card.

Mux : These line card can accept the veriety of client signals as well as OTN signals and multiplex them to the OTN port on the same of different slot having OTN port.

In optical domain, an optical transport network is comprised of a set of optical network elements (wavelength division multiplexers (WDM), optical amplifiers (OA), optical add-drop multiplexers (OADM), optical cross-connects (OXC), etc.) connected by optical fibre links and able to provide functionality of transport, multiplexing, routing, management, supervision and survivability of optical channels carrying client signals.

I got to work on variety of OTN products, all chassis based, Few transponders were simply having couple of 10G OTN ports with multiple client ports. Apart from the transponder functionality these devices also supported the protection at the ODU Layer. Client and line layer protections were supported through external optical relay switches. Typically in a chassis based solution variety of transponder cards can map their client signal to the higher b/w OTN port (40G, 100G) on the different slot. These higher bandwidth port card may drop the signal over a single wavelength to a ROADM for further optical layer processing.

ROADM Networking (DWDM)

So far, we have mostly talked about technologies used to map the client traffic and carry it over the a single wavelength. As we move from the from access->aggregation->metro->edge->core->long-haul, a bigger and bigger b/w pipe is required. We can imagine how voice signal is carried over the long haul network like following.

• At the access DS1 encapsulated in the DS3 payload and mapped to the STS1.
• At aggregation node multiple STS1s are multiplexed to the higher rate OC-48
• Several OC-48s at the Metro are mapped to the OTU3 OTN line at the core along with the other technologies like Ethernet etc. This all about the processing of the signal at the electrical layer. OTU-3 or OUTU-4 frames are now carried over a specific wavelength (OCH).

If the signal is transmitted over the long haul subsea cable, it is very much required to use the fiber bandwidth efficiently. That is why it is requires at the core to have the switched which can switch the optical signals at the wavelength layer. This is achieved by the ROADM(Reconfigurable Add drop multiplexer). ROADM adds, drops, switches and passes through the various wavelengths. It used DWDM multiplexers which can multiplex the different wavelength to the single wavelength, Wavelength selective switches (WSS) which can switch wavelengths on the optical domain. ITU-T has defined the 50 Ghz grid for ROADM to operate. Though ROADM evolved with a fixed architecture, attempts were made to make the optical layer switching flexible and dynamic. Today's ROADMs are Colorless, Direction less and Contention less

• Colorless ROADM allows allocation of any wavelength on any port.
• Directionless ROADM broadcasts all the wavelength aggregated signal to all the direction of add/drop Node.
• Contentionless ROADMs eliminate the potential problem of two identical wavelengths colliding in the ROADM

Now ITU-T is looking to define the Gridless architecture to enable "super channel" wavelengths ( great than 100G) to the ROADM.

OTN Challenges

Some of the major challenges for OTN going forward are

• Going beyond 100 Gbps, in the current 50 Ghz channel spacing grid imposes limits on transporting signals beyond reasonable distances.
• The new OTN format to carry 400GbE 
• Larger than 1.25 Gbps tributary size.
• OTUCn signal over multiple wavelengths
• FlexO
• OSMC for synchronization

 Packet Based Transport Networks

 Voice's "undisciplined" brother Data

Voice is a very disciplined 64 kbps constant bit rate stream. As a result, a transport frame can always have a time slot reservation inside the frame. Synchronous TDM based network could pretty well transport and manage the voice traffic. However, data is a bursty in nature means you can't predict when is coming when it not. Here are some other attribute differences between data and voice.

Therefore, In order to understand the need for the packet based transport networking it is required to first understand the basic difference between the voice and data and also difference between the TDM and statistical multiplexing. In contrast to the Time Division Multiplexing, statistical multiplexing allocates the bandwidth on demand. Data stream is basically random in nature and bursty, you can't predict when will data be received. Due to its bursty nature Data doesn't require the peak bandwidth all the time. Problem with the TDM based networks is that you allocate a fixed bandwidth for each service type which may go unused as data traffic may not be hitting the full rate. This will make the network highly in-efficient as a significant NE/port bandwidth is allocated but not used most of time. This triggers the need for the packet based transport network where bandwidth can be dynamically shared/prioritize and controlled between the different services.

TDM Based network as data carrier 

Even when SONET/SDH was pre-dominant, due to internal evolution packet networking was evolving there was a need to transport the packets though the existing SONET/SDH infrastructure.

POS (Packet over SONET), Frame Relay over SONET, Ethernet over SONET etc were the attempt in the same direction. SONET acted as the high bandwidth lines used to inter-connect high speed packet routers.

During first couple of decades, ethernet speed was typically 10 Mbps, 100 Mbps and 1Gbps, increasing factor of 10.However as discussed earlier SONET rates which were primarily optimized for the voice (OC-1 51 Mbps, OC-3 155 Mbps and OC12 622 Mbps) don’t match the rates to support Ethernet data stream. This rate mismatches made SONET very in-efficient in supporting ethernet transport. To further optimize the ethernet data transport over the SONET, VCAT, LCAS and GFP was introduced. VCAT was an inverse multiplexing where TDM lower order tributaries were clubbed together to match the right bandwidth requirement. LCAS was a signaling mechanism to support the dynamic adjustment of the bandwidth. Generic Framing procedure protocol was used for mapping the data packets into an SONET payload.

Evolution of Ethernet in WAN

Ethernet initially evolved as the LAN technology to connect the devices in the same geographical area, premises or building for data communication. Before 2000, Other protocols were developed to transport the data to the longer distances namely X.25, ISDN, Frame Relay and ATM. Continued proliferation of the data hungry applications like email, web and storage triggered the network transformation. Voice centric network using SONET, SDH proved in-efficient to carry data. In early 2000, Ethernet extended itself to the carrier space for the WAN connectivity. MEF forum closely worked towards making the Ethernet Technology as "Carrier" grade. Five major attributes were defined for carrier Ethernet

Figure 8 : Carrier Ethernet Defined Attributes

Ethernet was no more a technology used in the premises or building but it was serving as the "carrier" in the service provider network. Carrier Ethernet was defined for Service Providers as "A set of certified network elements that connect to transport Carrier Ethernet services for all users, locally & worldwide" and for the business users "

A ubiquitous, standardized, carrier-class Service and Network defined by five attributes that distinguish it from familiar LAN based Ethernet ".

Figure 8 above depicts the essential attrbutes defined for Ethernet use it a in the carrier space as a Carrier grade technology.

• Standardized Services : MEF defined the standard services which an Ethernet based carrier network provider will offer to the customer. For the customers, defined services EVC were E-LINE, E-LAN and E-Tree. For the operator to operator, for ethernet based handoff, OVC services were defined as E-Acces and E-Transit.
• Scalability : Scalability challenges were addressed through the Q-in-Q, Provider Backbone Bridging (MAC-in-MAC).
• Reliability: Carrier Ethernet had LAG (LACP) and Ring Based (G.8032) protection methods like to make it reliable in the carrier space.
• Quality of Service: Carrier Ethernet provides provisioning for defining SLA parameters, per port, per connection or per traffic flow.
• Service Management: To match SONET/SDH type OAM, Various OAM functionalities like 802.1ag CFM, Y.1731 Performance Monitoring, 802.1ah for monitoring and troubleshooting access links, RFC 2544/Y.1564 for testing services, RFC 5357 TWAMP for two-way performance measurement, LLDP to identify the capabilities of the connected devices etc.

MPLS-TP

Somewhere during 2000, when MPLS was created its main objective was to have a single control plane for IP and ATM routers. While some of the MPLS initial objectives are still relevant today they have become lesser important relative to the MPLS acting as a carrier for the ethernet services.

When bandwidth and packet demand emerged initially, they represented only a small percentage of the total traffic therefore it was natural to carry them over the existing circuit switched network. However, as IP/MPLS based packet services saw a tremendous growth, client traffic became increasingly and inherently packet based. Not to forget that SONET/SDH had set a very high standard for the "Transport Networks". Challenge was to develop a new "packet based transport" infrastructure giving high level of reliability and OAM like SONET . IP/MPLS was already evolving very fast as the resilient packet based infra structure. In 2006, ITU-T came up with the definition of T-MPLS as the first "Packet Based transport Infrastructure". In 2008, ITU-T stopped the work on T-MPLS and formed a joint working team with the IETF to examine the architectural consideration on "transport profile". This joint Team defined a new transport profile for MPLS called as the MPLS-TP.

How was MPLS-TP different from IP MPLS

Figure 9 : MPLS vs MPLS-TP Feature set comparision

As shown in the figure 9 above, MPLS-TP used the subset of features from IP MPLS and threw away the features like IP forwarding, PHP, ECMP and LSP Merge. However, MPLS-TP defined following extensions to the existing MPLS standard to match the transport network requirements.Comprehensive In-band OAM for fault detection, localization and troubleshooting.

• Transport grade sub 50ms recovery (1:1, 1+1 and 1:N protection)
• Separation of Control plane and data plane.
• Fully automated operation using NMS without the need of control plane.

Transport Other-way : TDM over Packet

There is a huge install base of the older TDM networks and they cannot be de-activated or removed over-night or in near future. While the core of the network moves to the packet centric infrastructure, question is what will happen to those huge TDM install base services. Keeping this in mind, circuit emulation technology was defined by the MEF and IETF to packetize the legacy voices services like PDH and SONET/SDH to the packet and then transport them over the carrier Ethernet or MPLS-TP network using Pseudo wire emulation. TDM time synchronization requirement in the packet network can be achieved through one of the packet synchronization method syncE or 1588.

What should be Transport Network: Packet or Optical ?

Current traffic flows in the networks may vary in different ways in priority, nature of burstiness, time of day usage, deterministic or non-deterministic. When it comes to the choice of transport technology there is no one size fits all. So far we saw three Technologies still predominant are

• Packet transport : For efficient b/w handling and control over traffic flows
• OTN : Multiservice transport with longer reach, transparent port based mapping and simpler OAM
• ROADM for wavelength level grooming or switching.

Question for network designers : what to choose Packet or optical

Final Answer : Packet and Optical both : Packet Optical

Emergence of ODU Flex allows the to switch/multiplex ODU container at 1.25 Gbps. But Problem arise when we want finer granularity traffic f lows to control. Therefore becomes essential to club the packet technology for traffic flow based switching using OTN as the interface based transport eventually going over a wavelength to the ROADM network if required. Traffic flows identified by the VLAN or LSP or Pseudowire get statistically multiplexed to the Ethernet payload go over the matching OTN rate. Platform which used the Packet and OTN/Optical capability both are referred as the converged packet optical. 

The packet transport architecture forced optical/OTN equipment vendor to put the packet functionality while for the Packet router leaders to hunt for the various optical technologies. That posed challenges not only to the equipment vendors but also to the services providers as all of them used to have their packet and transport organization different.

 Figure 10: Packet Optical Multiplexing Model

Packet Optical Equipments

A Packet optical equipment consists of following functionality

• Ethenrnet to OTN mapping, OTN switching/multiplexing at ODU0 and ODU Flex granularity.
• Packet Optical Fabric to be able to switch at packet as well as OTN layer.
• Ethernet/MPLS Layer switching

I worked on multiple types of packet optical equipments. Packet optical NE can be either a line card or a chassis based solution. In the line card based solution, along with the ethernet port, we find few OTN ports to map ethernet payload to the OTN format. Typically they acheive the functionality of the packet aggregration to 10GBE or OTU2/2e interfaces.

Chassis based packet optical product tend to have multilayer L0, L1 and L2 layer functionality in the same box. They have very high capacity packet optical fabric which can switch Packet/OTN traffic from the different line cards to the desired format.Sometimes these solution also support the ROADM based card in the same chassis to be able to do the aggregation at the wavelength level. In the current trend MPLS-TP is used as the packet transport and ethernet mostly used as the aggregator which does the hand-off of the ethernet to the MPLS for transport.Few packet optical boxes also support the line cards with the Legacy PDH/SONET/SDH interfaces which uses the Circuit emulation technology to packetisze the TDM time slots and switch/transport across the Ethernet/MPLS network.

Path Going forward : Intelligent Multi-Layer Transport (Transport SDN)

Today there are two types of network demands. One from the Legacy Telcos which require sophisticated multi-layer packet optical transport devices and on the other hand we have the Content Providers who need more packet centric server and switches (Spine and Leaf) based transport elements. Traditionally Larger Telcos used to have the well-equipped staff to provision the network whereas content based providers demand for automated provisioning. One important thing to remember that for the content providers network is not generating revenue but it is applications and contents which generate the revenue. Keeping in view the versatile network requirements from telco and non-telcos, there is a need to design the agile, dynamic, intelligent SDN based multi-layer transport network which can not only cater to the Telcom service Providers but also to the distributed cloud based architecture, data center inter connect and 5G networking. Many vendor have already started pitching-in solutions on the same line. vedict is clear ?Future Transport network is finally moving from ground to cloud!