Fortigate VM as Network Security Layer for Substations in Virtualized Environments on Welotec Rugged Substation Automation Computer

? Welotec GmbH - Author: Jan Hille - Solution Architect at Welotec

More and more complex applications are realized with virtualization host platforms carrying multiple virtual machines in IEC 61850 digital substations. For securing the communication, hardware-based firewalls installed in addition to the virtualization host platforms are used. These hardware-based firewalls can monitor and control the traffic coming out of/going into the virtualization host’s ethernet interfaces.

Each virtual machine in such a virtual environment has its own very specific task. This leads to different requirements when it comes to communication protocols/services. With a firewall capable only to monitor/control traffic running through the host systems interfaces, it is impossible to define rules for specific traffic for each single virtual machine. The necessary protocols and services can only be defined for the external interfaces, which in most cases will be used by several different virtual machines. Furthermore, controlling the traffic between the virtual machines inside of a virtual environment is not possible on a protocol or service basis.

We extend the virtualization host system by a specialized firewall-based security layer to be able to control and monitor traffic more specific.

Realization of a Firewall VM for Substation Automation

A FortigateVM appliance installed in a virtualized environment on a Welotec RSAPC shall be used to monitor and control the whole network-based communication.

We want to prove that the virtualized firewall is capable to control the network-communication covering the following areas:

1. Virtual Machine <-> external networks attached to hostsystem’s ethernet ports (e.g. Internet Access)
2. Virtual Machine <-> Virtual Machine
3. Virtual Machine <-> hostsystem (applications)
4. Hostsystem <-> external networks attached to hostsystem’s ethernet ports (e.g. Internet Access)
5. External devices <-> External Devices (traditional hardware based firewall)

Setup Hostsystem with Hyper-V

The Welotec RSAPC has been developed with the goal of providing a datacenter level virtualization platform for the Substation environment. Based on an Intel Xeon CPU and Server Chipset, but still fanless, IEC61850-3 certified and with the operating temperature range demanded by those applications, while still bringing Server features like up to 64GB ECC Memory, redundant power supplies and Rapid Storage Premium RAID controller.

The Rugged Substation Server with Intel? Virtualization Technology (VT-x) allows one hardware platform to function as multiple “virtual” platforms. It offers improved manageability by limiting downtime and maintaining productivity by isolating computing activities into separate partitions.

Intel VT-d can help end users improve security and reliability of the systems and also improve performance of I/O devices in virtualized environments.

Hardware-platform: Welotec RSAPC

• CPU:                      Intel? XEON CPU E3-1505L v6
• RAM:                     32 GB
• SDD 1:          64 GB SSD carrying Host-Operating-System Hyper-V
• SSD 2+3: 256 GB SSD carrying Virtual Machine files (RAID1)
• Network-Adapters:              4x Intel I210 Gigabit Network Adapter
• OS:                                       Windows Server 2016 Standard Edition
• Virtual environment:           Hyper-V 2016 Configuration Version 9.0

The virtualization platform is running several virtual machines with different operating systems (e.g. Ubuntu 18.04, Windows 10). One of the virtual machines is running a Fortigate VM64-HV firewall appliance.

The virtual machines can be connected using virtual switches, which are defined within the Hyper-V manager

Those switches can be used to connect virtual machine to interfaces of the host system, to connect virtual machines to each other or in a combination of both.

Using virtual switches connected to the a Fortigate VM64-HV virtual machine we are now able to let all the traffic run through a transparent firewall instance to be able to monitor and control the traffic on a service/protocol basis.

Fortigate Firewall VM64-HV Virtual Machine

• CPU:         4 virtual processor cores (theoretical maximum of 64 virtual processor cores available according to Hyper-V CPU sizing best practice, highly depending on individual environment)
• RAM:        2 GB (minimum 1GB needed)
• Network:     8 Network Adapters attached to virtual switches

The Fortigate VM has been equipped with 4 virtual processor cores and 2 GB RAM. The used disk space is dynamically allocated meaning, that it will only consume the space it needs.

We reserve 512MB of RAM and 5% of CPU resources of the hostsystem to make sure that the VM is always able to operate even if any other VM is consuming a large amount of the resources.

Virtual machine -> external devices and networks

Using this setup allows us to control and monitor the traffic from and to any device/network attached to a physical interface of Welotec RSAPC. Rules can be defined limiting services/protocols as well as specific hosts can be allowed or prohibited.

Virtual Machine -> Virtual Machine

The same applies to the communication between virtual machines. As virtual machines might have to share specific information, an interconnection might be necessary. With this setup we can limit the communication to the protocols which are necessarily needed. Every other communication will be blocked.

Virtual machines -> applications on hostsystem

In some cases, it might be reasonable to display some information out of the virtual machines directly on the hostsystem without providing access to the virtual machine itself. This might be the case if some status information has to be displayed which is for example provided by a webservice running on a virtual machine.

Using the VM firewall appliance also allows us to monitor and control the traffic between virtual machines and the host operating system.

The Hypervisor can be configured to completely take over the control of one or more physical host interfaces and redirect the traffic through a virtual switch. This enables us to lead the whole traffic from the hostsystem’s applications through the virtualized firewall appliance. This means that we can also cover the security of the hostsystem using the Fortigate VM64-HV.

External devices -> external devices

The Fortigate VM64-HV appliance can also be used like a classic hardware firewall appliance to monitor and control the traffic between devices which are attached to the hostsystem’s ethernet ports. That means that the RSAPC can be used as a virtualization platform and a firewall appliance for the local network at the same time.

Defeating attacks with a Fortinet Firewall in a Digital Substation

A firewall is of course not only used to monitor and control the traffic in network environments. A very important task is also defending the network and the services within this network against attackers.

In order to check the abilities of the virtual firewall appliance with regards to defending attacks from external networks we want to perform a DoS-Attack in order to bring down a service running in a virtual machine. A DoS- (Denial of Service)-attack is an attack which floods a network device with network-packets. The goal behind that is consuming all the resources to make the service unavailable for others.

We are running a simple webserver which might be used to provide monitoring data from the assets for internal and external clients (right side of the picture).

We have a Virtual Machine (left side of the picture) as well as native client in the external network (bottom right side of the picture) which are consuming the service of the webserver, in our case by viewing the webpage provided.

The attacker is also located in the external network. The attack will try to bring the webserver down to make the service unavailable for the two client systems. We could verify that the attack causes the webserver to fail when it is not protected by the Fortigate VM64-HV.

The transparent firewall running in the virtual environment between the clients and the webserver is configured to only allow HTTP-traffic directed to the webserver.

In addition, a DoS-policy has been set up in order to monitor the traffic coming into the external LAN port of the RSAPC. This policy can recognize different patterns of DoS attacks as well as blocking those attacks to make sure that the desired service will be available at every time.

Attacking the service

The firewall appliance brings a built-in network monitoring tool. When we now start the attack, we see a huge load of packets coming into the LAN port.

The firewall appliance recognizes the attack and blocks it immediately. We can see a sudden rise of network-sessions when the attack is started.

Recognition and blocking of the attack cause a slight increase of CPU and memory usage starting with the attack and ending with the successful blocking of the attack. We can also see the attack within the anomaly logs even identifying the attackers IP-address.

The most important information: the webserver was available for both, the Client in the virtual environment and the client in the external network without any interruption. The overall performance of the firewall appliance has not been affected by the attack, all the other services in the network were working well all the time.

Conclusion

With a Fortigate VM64-HV virtual machine we will able to replace a hardware firewall. As we can redirect the whole communication from/to a physical interface on the hostsystem the firewall can monitor and control the whole traffic.

In addition to this we add a security layer to the virtual environment helping us to secure the communication between virtual machines and even between virtual machines and the hostsystem and external networks. As we can create a complex network environment inside the virtual structure, rules can be defined more precisely to meet the needs of controlling traffic for each virtual machine inside the environment.

The scenarios described above can also be combined. That means that the Welotec Rugged Substation Server RSAPC can be used as a virtualization platform, providing an additional security layer for the virtual machine communication and can at the same time be a firewall solution to secure the whole network traffic for external devices as well.

The Fortigate VM64-HV performs very good even with limited resources provided. Attacks are defeated having no or only minor impact on the appliances’ performance, depending on the provided resources. As we are running a virtual machine, we have all the benefits when it comes to scaling the solution to meet the needs of each individual installation.