ANALYZING THE LIMITS OF DATA TRANSMISSION IN THE PROCESS BUS

SUMMARY

This paper aims to discuss the limits of data transmission in the Process Bus, i.e. the network bandwidth occupied by the Sampled Values (SV) published through a Merging Unit (MU). Several tests were performed in order to demonstrate the network behavior and the bandwidth consumed, a test set was used to simulate the MUs. Issues like the use of VLANs, the SV frame configuration based on IEC 61850-9-2-LE or IEC 61869-9 formats as well as the maximum number of MUs allowed on the network were approached on the tests done. An equation was deduced in order to assist optimized analysis of the available network bandwidth for the Process Bus.

I?– INTRODUCTION

The IEC 61850 defines SAS (Substation Automation System) divided into three levels: process, bay and station. At the process level are all the devices which interface with the system like: sensor, actuators, CTs and VTs, and circuit breakers. At the bay are IEDs (Intelligent Electronic Devices) that perform various functions like protection, measurement and control. At the station is the supervisory system that performs a general monitoring and control of the substation and communicates with external control. Interfacing with the process and bay levels is the Process Bus that takes data from CTs, VTs, actuators and etc to the IEDs at the bay. To interconnect bay to the station level, there is the station bus connecting IEDs to the supervisory.

The Sampled Values are defined in the IEC 61850-9-2 as Ethernet frames containing the digitalized current and voltage values. Due to the implementation of IEC 61850, the paradigm was changed because the system no longer works with secondary analog signals, but with sampled values sent by the Ethernet network. In this way, hard copper wiring cable was replaced by Ethernet cables, resulting in money savings and simplicity.

The Sampled Values have three main features: SV messages are standardized in order to allow interoperability. In addition, they are Multicast frames with fixed range of Destination MAC addresses defined by the standard (from 01: 0C: CD: 04: 00: 00 to 01: 0C: CD: 04: 01: FF). Due to the importance of the SV messages function, they run only on the second layer of the Ethernet network (Link layer). It is high priority message (level 4) with critical time.

In 2004, UCA released an Implementation Guide with additional information to IEC 61850-9-2 which was known as “Light Edition”, becoming a reference for the Process Bus implementation. This guide introduced some definitions relative to the SV frame format, such as sample rates and number of samples per frame. In 2016, IEC 61869-9 was released as a complementary standard to IEC 61850-9-

2. This standard brings some changes related to increase the allowed sample rates and the number of ASDUs (Application Service Data Unit).

The aim of this work is to approach the limits of data transmission in the Process Bus in order to define the maximum number of MUs the network tolerates. Using Conprove test sets CE-6710 and CE-7024 to simulate Merging Units, several tests were performed to verify the Ethernet network behavior and bandwidth consumption.

In the proposed tests, issues like network consumption by the maximum number of MUs configured based on IEC 61850-9-2LE or IEC 61869-9, the influence of VLANs and Destination MAC Address filters were approached. A math equation was deduced to help the analysis of how to optimize the use of available bandwidth for Process Bus. Some concepts of data networks as Bandwidth, VLAN and Manageable Switches were also addressed in this paper.

A comparative study of the SV messages transmission times and the bandwidth occupied by MUs based on 9-2LE as well on IEC 61869-9 was done in this paper, verifying the maximum transmission capability in number of MUs for each standard. Besides, some issues were addressed like: what is the network behavior with and without VLAN? If a maximum number of MUs was sent to a Switch port, could the packets be read on another port? Also, if more Switch ports were used, what would be the network behavior?

II?– CONCEPTS OF DATA NETWORK

A network Bandwidth defines the data transmission capability in the time. Usually, the common Switch speed on a substation is 100 Mb/s or 1 Gb/s, what means that it is possible to transmit, at most, 100 Mbits of data or 1 Gbits of data in a second on this network, respectively.

In networks which demand high performance such as Process Bus, the Manageable Switches are required. The basic differences between a manageable switch and an ordinary one are in their features. The manageable switch has several applications that an ordinary one has not; some can be pointed as software management, security settings, port mirroring, multicast filters, redundancy protocols settings such as RSTP, PRP and HSR or even proprietary, automatic management of priority flags, bandwidth settings, VLANs and etc.

VLAN (Virtual Local Area Network) is pretty used in the substations for allowing logical separation of several local networks. Thus, several networks can be operated simultaneously in a way that one cannot interfere on another operation. Then, several devices of a same substation, grouped on different VLANs, can be connected in the same Switch and operate in different logical networks.

III – COMPARISON OF TRANSMISSION TIMES AND BANDWIDTHS FOR THE PROCESS BUS ACCORDING TO IEC 61850-9-2LE AND IEC 61869-9

It is a mistake to consider just the frame size when analyzing the bandwidth occupied by the SV frames, this is a simplistic approach. Sampled Values have a specific feature of high publishing rates, resulting in microsecond times between frames. So, the Switch ports latency must be take?into account.

In order to avoid network dimensioning mistakes, the Switch ports latency, also called Switch latency, must be considered. In applications with big frames, near to the 1512 bytes of Ethernet standard, the Switch latency can be ignored. However, when dealing with applications like the Sampled Values on the Process Bus, whose frames have a small size, around 10% of the 1512 Bytes but with a higher publishing rate, the Switch latency cannot be ignored anymore and must be considered.

Regarding 9-2LE, the sampling rate must be 80 points/cycle that is 4800 Hz, considering a nominal frequency of 60 Hz, with 1 ASDU which means one Sample Counter per frame. So, if the MU must publish 4800 frames per second a new frame must be transmitted each 208.33 μs. The Figure 1 below, taken from the 9-2LE, illustrates the sampling rate and the numbers of ASDUs for Protection.

In order to exemplify the transmission times analysis in the 9-2LE format, using the Conprove CE- 6710 test set to simulate a MU it is possible to verify that this SV frame has 125 Bytes (1000 bits) as indicated at Figure 2 below, taken from the Wireshark network protocol analyzer. Then, considering a 100 Mb/s network, each SV frame has 10 μs of transmission time. So, taking into account that the theorical Ruggedcom Switch latency is about 4 μs, this SV message has a total transmission time (composed by the sum of the SV frame transmission time plus the Switch latency) of 14 μs.

Considering above data, it is possible to estimate the maximum MUs number capacity on a 100 Mb/s network as well as the bandwidth occupied by these MUs. As already deduced, if a SV message must be published each 208.33 μs and the example transmission total time is 14 μs, the conclusion is that the 100 Mb/s network tolerates 14 MUs maximum (208.33 μs divided by 14 μs is equal to 14.88, then considering the integer value of 14 MUs). Related to the bandwidth occupied by these 14 MUs, as each frame is 1000 bits length and in a second 4800 frames are transmited, the conclusion is these MUs occupy 67.2 Mb/s of a 100 Mb/s network, that is just 67.2% of the network bandwidth.

About IEC 61869-9, the preferred sampling rate is 4800 Hz, considering a 60 Hz nominal frequency, with 2 ASDUs which means that a MU can transmit 2400 frames per second. So, each 416.67 μs a new frame SV is trasmitted. Figure 3 below, taken from IEC 61869-9, illustrates the preferred samplig rate for Protection as well as the number of ASDUs.

As an example of the transmission times analysis on the IEC 61869-9 format, using the Conprove CE- 6710 to simulate a MU, it is possible to verify that this SV frame has 223 Bytes (1784 bits) as indicated on Figure 4 below, taken from the Wireshark network protocol analyzer. So, considering a 100 Mb/s network, the SV frame is transmitted in 17.84 μs. Taking into account the theorical Switch latency is about 4 μs, the total transmission time of a SV message, composed by the sum of the SV frame transmission time and the Switch latency, is about 21.84 μs.

Considering above data, it is possible to estimate the maximum MUs number capacity in a 100 Mb/s network as well as the bandwidth occupied by these MUs. As already deduced, if a SV frame must be transmitted each 416.67 μs and the total transmission time is 21.84 μs, the conclusion is that the 100 Mb/s network tolerates 19 MUs maximum (416.67 μs divided by 21.84 μs is equal to 19.07, then considering the integer value of 19 MUs). Related to the bandwidth occupied by these 19 MUs, as each SV frame is 1784 bits length and in a second 2400 frames are transmitted, the conclusion is that these MUs occupy 81.35 Mb/s of a 100 Mb/s network, i.e. 81.35% of the network bandwidth.

Analyzing above data, it is possible to deduce an equation to calculate the maximum number of MUs tolerated by a network and derivate another equation to calculate the total bandwidth consumed by these MUs. These two results are function of the SV frames sampling rate, numbers of ASDUs, SV frame length, network bandwidth and the Switch latency.

The Equation 1 below calculates the maximum MUs number tolerated by a network.

nMaxMU = maximum MUs number on the network;

DtSV = transmission period between SV frames;

tTransm = total transmission time of a SV frame;

sRate = SV frames sampling rate;

nASDU = numbers of ASDUs;

Lframe = SV frames length in bits;

BWnet = network bandwidth;

tSW = Switch latency.

The Equation 2 below calculates the total bandwidth consumed by the maximum number of MUs on the network.

where,

BWmu = total bandwidth consumed by the maximum number of MUs on the network.

IV – TESTS

Several tests were done to investigate the network behavior and the bandwidth consumed in different situations related to the maximum MUs number in one or more Switch ports with or without VLANs. The tests were performed with Conprove CE-6710 and CE-7024 test sets, the Ruggedcom Switch RS- 940G and the software to subscribe the Sampled Values (Conprove MultimSV). The Conprove CE- GPS was used to perform the time synchronism.

All the tests were done on the IEC 61850-9-2LE context as also on the IEC 61869-9. The SV frames lengths were 1000 bits and 1784, respectively, as shown in the examples of item III above.

The CE-6710 and CE-7012 models have total support to IEC 61850, being capable to generate current and voltage analog signals or publish/subscribe Sampled Values on the 9-2LE or IEC 61869-9, simulating one or more Merging Units.

These test sets can be used on the process bus to verify the network, publishing and subscribing SV messages, simulating overload throw the setting of up to 10 MUs simultaneously, simulating frames fails (lost of sync, frame delayed, frame in duplicity, frame corrupted, lost of frames, changes to Quality Bits values and Simulation Bit value).

The MultimSV software acts literally as a Sampled Values multimeter, subscribing the SV frames and displaying the waveforms, measuring the amplitudes and phase angles, besides harmonic analysis and SV frame errors detection.

The Figure 5 illustrates the test system set up.

a) First Test

In the first test, a CE-6710 test set was used to simulate up to 10 MUs in 9-2LE format and up to 9 MUs in IEC 61869-9 format in one Switch port on 100 Mb/s network. In a second Switch port, the MultimSV software was used to verify if was possible to read all the MUs SV frames. Figure 6 below illustrates the first test scheme.

The MUs SV frames were published increasing one by one until the limit of MUs number according to the standard related. Each time a MU was added, all the previous were verified throw the MultimSV, filtering by the respectively Sampled Value ID (svID) and analyzing wheter or not SV frames were lost. The Figure 7 below illustrates.

After the analysis of all MUs with the MultimSV, the conclusion is that were not SV frames losses, because no SV frames errors were registered in the 9-2LE format as well as in the IEC 61869-9 format.

b) Second Test

In the second test, two CE-6710 and one CE-7024 were used to simulate up to 30 MUs in the 9-2LE format and up to 27 MUs in the IEC 61869-9 format using three Switch ports with 100 Mb/s network. In a fourth Switch port, MultimSV was used to monitor if was possible to read all the MUs SV frames. In order to guarantee the data analysis precision, the test sets clocks were synchronized by the CE- GPS. Figure 8 below illustrates the second test scheme.

The MUs SV frames were published one by one until the limit of MUs number according to the standard related. Each time a MU was added, all the previous were verified throw the MultimSV, filtering by the respectively Sampled Value ID (svID) and analyzing wheter or not SV frames were lost. The Figure 9 below illustrates.

After the analysis of all MUs with the MultimSV, the conclusion is that, in the 9-2LE format, SV frames starts to lose with 18 MUs on network, taking up 86.4% of the network bandwidth which means 86.4 Mb/s of a 100 Mb/s network. In the IEC 61869-9 format, the SV frames losses happened with 21 MUs on network, taking up 89.9% of the network bandwidth which means 89.9 Mb/s of a 100 Mb/s network.

Thus, considering this test set up using a 100 Mb/s network, the maximum MUs number possible in the 9-2LE format was 17 MUs, taking up 81.6% of the network bandwidth, and in the IEC 61869-9 format was 20 MUs, taking up 85.6% of the network bandwidth.

Based on this, it was estimated that the Ruggedcom Switch latency was less than the 4 μs specificated. Using the Equation 1 deduced on item III, it was possible to conclude that the real Switch latency was about 2.25 μs. It implies that it was possible to tolerate more than 14 MUs in the 9-2LE format and more than 19 MUs in the IEC 61689-9 format.

c) Third Test

In the third test, the same test scheme of the second test was used; however the three Switch ports which received the three test sets were separated on three VLANs. Therefore, each test set with their respectively MUs, was on a different VLAN ID. To do the MUs analyses on MultimSV, another three Switch ports were used each one on a configured VLAN. The Figure 10 below illustrates the third test scheme.

The same way, the MUs SV frames were published one by one until the limit of MUs number according to the standard related. Each time a MU was added, all the previous were verified throw the MultimSV, filtering by the respectively Sampled Value ID (svID) and analyzing wheter or not SV frames were lost. The Figure 11 below illustrates.

After the analysis of all MUs with the MultimSV on the three Switch ports separated by VLANs, it was possible to conclude that in the 9-2LE as much as in the IEC 61869-9 there was not any SV frames losses, because no SV frames error was registered.

So, even the MUs number being more than the network limits of 17 MUs for the 9-2LE format and 20 MUs for the IEC 61869-9 format, considering the real Switch latency of 2.25 μs, there was not any SV frames losses. Then, it is possible to say that separate MUs by VLANs allowed to enhance use of the network bandwidth for the Process Bus.

d) Considerations for a 1 GB/s Network

Just for comparation, based on the Equation 1 and Equation 2 in item III it is possible do deduce the maximum MUs number and the total network bandwidth consumed on a 1 Gb/s network, considering the Switch latency of 2.25 μs. Considering the 9-2LE format, the transmission time of a 1000 bits length SV frame is 1 μs, so the total transmission time is 3.25 μs. If a SV frame must be published each 208.33 μs, the maximum MUs number that a 1 Gb/s network tolerates is 64 MUs, taking up just 30.07% of the bandwidth, i.e. 307.2 Mb/s of the 1 Gb/s network. In the IEC 61869-9 format, the transmission time of a 1784 bits length SV frame is 1.78 μs, then the total transmission time is 4.03 μs. If a SV frame must be published each 416.67 μs, the maximum MUs number a 1 Gb/s network tolerates is 103 MUs, taking up just 44.1% of the bandwidth, i.e. 441 Mb/s of the 1 Gb/s network.

V??- CONCLUSIONS

Based on the results of this paper, it was possible to explore the limits of data transmission on the Process Bus, verifying the network behavior and analyzing the bandwidths in several situations on the IEC 61850-9-2LE and on the IEC 61869-9 universes.

Equations were deduced to calculate the maximum Merging Units number tolerated by the network and the bandwidth consumed, which allows a better analysis of the network consumption by the Process Bus and optimizes the available network use. Therefore, it was possible to demonstrate that considering just the bandwidth to calculate the maximum MUs number tolerated by the Process Bus can lead to mistaken results, because important variables as the transmission period between SV frames and the Switch latency also must be taking into consideration.