Virtualization in the next generation Digital Substation

Ensuring redundancy, standardization of hardware and reduction of maintenance costs using virtualization

? Welotec 2021

In modern substations, an increasing number of field devices and automation components and a higher degree of automation means that more and more computers are required. Communication components and operator terminals (HMIs) as well as SCADA systems require hardware that meets stringent requirements to ensure safe operation of the substation environment. In many substations, there are separate computers for each individual function, such as small embedded computers but also 19“ station computers.?

Through virtualization, many systems can be accommodated on one platform. Also, the construction of redundant systems becomes easier. Virtualization originally comes from the area of large IT data centers and abstracts IT resources by inserting an additional layer between application and hardware. This makes it possible to emulate and virtually provide services, operating systems on a common hardware. For the user, the virtual object behaves like a dedicated hardware or software object.

Development timeline of transformer station equipment

In substations, there are typically the following applications that have a specific need for computing power and different requirements for operating systems or hardware resources:

• Intelligent operator terminals (HMIs)
• Authentication server with Lightweight Directory Access Protocol (LDAP) and RADIUS protocol
• Workstations for maintenance technicians or operating personnel
• Historical data databases for analysis and monitoring
• Communication gateways and data concentrators
• SCADA systems for monitoring and control
• Other automation equipment, PQ management etc.

These applications are operated on individual computers in many substations. However, this entails high maintenance costs and is also not recommended for reasons of system redundancy. Each application uses hardware tailored to its needs, which however represents a single point of failure. In order to expand each application redundantly, a huge number of small computers must be installed, all of which would have to be kept in stock and maintained for possible replacement.

By using virtualization, applications on the same hardware can be operated completely separately. Different operating systems can be used for each application and the prioritization of the individual applications can be assigned.

This makes it possible to provide several applications with a uniform, then redundantly operated hardware infrastructure that is easy to maintain and adapt to specific conditions. Such an implementation, which is based on the picture above, would look as follows:

Benefits of using virtualized computing power in Digital Substations

This simpler and fail-safe hardware infrastructure with virtualized application level creates a lot of added value during planning, operation and the adaptation to new conditions:

• Reduction of investment costs through procurement of identical hardware
• Reduction of maintenance and repair costs
• Ensuring redundancy and reliability of individual components of the substation
• Unification of the IT and OT infrastructure
• Achievement of a security level that considers all compliance processes

Virtualization - A cost driver?

The investment and operating costs of redundant applications often appear to be high, since a large number of computers have to be procured and operated in existing systems. With high-performance hardware, however, different applications can be run on one and the same hardware platform (Fig. 2). The standardization of hardware, IT and OT technologies results in economies of scale that make the use of virtualization very interesting.

Investment protection in Digital Substations through virtual machines

Transformer substations generally use specialized software, which represents a significant portion of the total cost of the IT environment. The original manufacturer may have disappeared from the market and no longer provide new versions, or the existing license may not cover newer versions and runs only on legacy operating systems such as Windows NT, 98 or XP, for which no current hardware is available. This legacy hardware can be made available virtually, thus allowing the continued use of existing software.

Virtualizations in Transformer Stations - Design Considerations

Hardware requirements

The selection and dimensioning of the right hardware platform should play a major role in virtualization, as all redundancy and cost considerations are based on this:

• A high-performance platform with sufficient computing capacity, multiple processor cores and suitable memory equipment is required.
• Minimum 32 GB to 64 GB ECC-RAM (error-correcting code). ECC-RAM can correct data corruption in memory areas to increase reliability in substation environments.
• Flash-based, maintenance-minimized drive systems with redundant design. Single-level cell-based (SLC based) memory is characterized by a high number of available write cycles, high write speeds and long memory maintenance, but is also more expensive than other flash memory media such as iMLC (industrial multilevel cell). However, SLC-based drive technology is the preferred choice for some applications such as fast storage of large amounts of data for a long period of time. For disaster prevention purposes, the drives should at least be operated in a RAID 1 network. iSLC storage media represent a cost-effective middle course, offering many advantages of SLC media and placed between MLC and SLC regarding write cycles.
• The CPU, graphics unit, and I/O layer should natively support virtualization to enable multiple operating systems to access this shared infrastructure optimally. Typical examples of terms used to support virtualization technologies of a CPU include Intel VT-x. Substation computers that support this standard can be better used for virtualization purposes. Processors with VT-d also enable guest systems to provide dedicated access to hardware components such as LAN ports.
• The computing unit must be able to operate safely under all environmental conditions. Therefore, extreme temperature ranges from -30°C to +70°C, high electromagnetic compatibility as well as shock and vibration resistance are essential in this hardware. These requirements are described in the standards IEEE 1613, IEC 61850-3, IEC 61000-4 and IEC 60255.
• The computer platform requires NICs capable of performing VLAN tags and trunking. It must also be able to be equipped with several independent Ethernet ports. This supports the separation of different systems and services per port. Because Windows 10 does not support NIC Teaming anymore you should use NICs with integrated Teaming function.
• The network protocols HSR and PRP are required for new IEC 61850 applications. Station computers should support HSR and PRP to be seamlessly integrated and used on all station levels.
• The hardware must be designed to be maintenance friendly and modular. For example, the replacement of a defective power supply unit during operation via two redundant power supplies is required in order to maintain operation during maintenance.
• The long-term availability of components of the computing unit must be ensured for a long period of time. It is therefore important to select components that are as up-to-date as possible.

Hypervisor - But which one?

In many projects the virtualization solution Hyper-V played an important role. Hypervisors are basically divided into two groups depending on the needs of the respective applications: Type 1, where the hypervisor is placed directly on the hardware (so-called bare metal, see figure 3 on the left side), and Type 2, where a basic operating system lies between the hardware and the hypervisor (see figure 3 right side). The selection of the appropriate virtualization components should be carefully considered in the planning phase.

Storage and resource management

It is necessary to evaluate which applications require which memory and resources. It must also be checked whether the hardware used supports the hypervisor and hardware drivers. Network functions (including support for multiple VLANs and virtual switches as well as redundancy technologies such as HSR/PRP) should also be native to the hardware. Furthermore, the hardware used should meet the requirements of IEC 61850-3.

Redundancy architecture for the next generation Digital Substation

In the transmission and distribution grids that are very critical, it is common to use a high-availability redundancy architecture. This includes the appropriate design of intelligent electronic devices (IEDs), gateways, SCADA, systems, network technology, cabling and much more. This design is made possible using central, redundant systems for different applications, i.e. by virtualization, in a simple and cost-effective manner. However, extensive planning steps should be carried out before implementation in order to guarantee optimum results and safe and cost-effective operation. With a good redundancy architecture, network operators can protect themselves against unplanned hardware failures and reduce single points of failure.

VLAN architecture and cost reduction through virtual Switches

Network infrastructures in substations can become very complex. Some applications within such network stations are time-critical, other data flows have high bandwidth requirements. By using VLANS, data traffic can be prioritized and network performance increased. The virtual machines are not directly connected to a physical Ethernet interface, but via a so-called internal private virtual switch (vSwitch). This then connects the individual applications with the physical Ethernet port and serves as a bridge between the logical Ethernet ports of the VM and the physical interfaces. It is possible to configure multiple VLANs on a vSwitch and send GOOSE traffic with QoS requests over it.

Virtual switches reduce hardware costs by not requiring many physical Ethernet interfaces. Especially in HSR/PRP networks, which are built redundantly, many cost-intensive HSR/PRP ports are eliminated.

IT/OT Convergence

The IT departments of many network operators are responsible for the selection of hardware and software, patch management and the operation of communication networks. IT and OT technologies are moving closer using virtualization, resulting in synergy effects. Even though there are different requirements for IT and OT systems, numerous common elements can also be identified:

• Development of common standards and guidelines
• Use of the common infrastructure for maintenance and service
• Use of identical or similar technology platforms
• Unification of license agreements for software
• Development of common governance
• Managing IP address management for WAN integration

Security challenges and requirements

The North American Electric Reliability Corporation (NERC) has developed a series of standards that describe the protection of critical infrastructures against cyber threats. Parallel to this, there is the NIS Directive in Europe. The NERC standards are mandatory for companies in North America. For companies in Europe it is the NIS Directive. In addition, most utilities around the world have also committed themselves to these standards to varying degrees, as they represent an industry-recognized best practice approach to most cyber security issues. Cyber security design should include the following key components:

• Defense-in-depth concepts with firewall layer and demilitarized zone (DMZ)
• Role-based access management
• Encryption of remote access connections
• Secure authentication and authorization
• Protection with antivirus and anti-malware software
• Consistent patch management for all software and firmware components
• Attack detection through intrusion detection systems

Summary

The ever-increasing digitalization of the world will not stop at the power grids and Digital Substations either. E-mobility and renewable energies will increasingly drive the grid and thus also the demands on computing power for control. It is therefore important to make the right decisions when selecting computing elements in order to enable digitization and secure operation of substations. IT and OT projects in the energy industry are always complex and require extensive and careful planning. Therefore, close, reliable cooperation in product selection and adaptation is necessary.

Welotec Substation Automation Computer for Digital Substations

Information about the Welotec Rugged Substation Computer for Digital Substations