IT/OT Integration Easier Done Than Said

There is much speculation that integrating operational infrastructure such as the control system and historian with IT infrastructure like the ERP and Business Intelligence (BI) to achieve digital transformation (DX) will be difficult, yet the IT and I&C departments in plants are collaborating successfully on digital transformation (digitalization) projects including Industrial IoT (IIoT). So IT/OT integration is not as hard as they say. How do these plants do this “IT/OT integration”? Here are my personal thoughts:

No reorganization required

One key aspect of digital transformation (digitization) and IIoT is that there is no need to merge the IT department with the I&C department looking after the operations infrastructure “OT”. This avoids challenges of reorganization and changing lines of responsibility. As far as departments and project execution is concerned it is more of IT/OT integration of the systems, and collaboration between departments and vendors. There is no merging of departments. The IT department remains specialized in their area of expertise and execute the IT infrastructure part of the DX and IIoT project while the I&C department remains specialized in their area of expertise and execute the I&C part of the project. At level 3.5 of the enterprise architecture is where the IT infrastructure meets the operational infrastructure of the control system, safety system, sensors, and condition monitoring analytics software etc. This also becomes the natural boundary of responsibility between the two departments and their vendors. The IT and I&C departments collaborate on the integration. The respective departments are then responsible for maintaining the expanded systems running. So for the people and process the term is IT/OT integration, not IT/OT “convergence”.

Most of digital transformation can be done by I&C alone but IIoT may require collaboration between I&C and IT departments

The Department of Homeland Security (DHS) National Cybersecurity and Communications Integration Center (NCCIC) and Industrial Control Systems Cyber Emergency Response Team (ICS-CERT) recommended practice Improving Industrial Control System Cybersecurity with Defense-in-Depth Strategies refers to level 0-3 of the enterprise architecture as the Manufacturing security zone (with OT) and level 4-5 as the Enterprise security zone (with IT). Level 3.5 is the boundary where they connect.

Data-Driven Work

Digital transformation is first and foremost about digitizing how the plant is run and maintained by operations. Changing from manual paper-based work practices to automated data-driven software-based digital procedures. Data-driven work practices means, for instance, decision to service a piece of equipment is based on the actual condition and performance of that piece of equipment as determined by analytics software. This analytics software gets its raw data from sensors. That is why hundreds or thousands of new sensors are required in the digital transformation of a plant. Thus digitalization is mostly about deploying operational infrastructure to enable the daily operational work procedures for maintenance, reliability, and integrity management as well as energy efficiency, safety, and production to be changed.

I&C responsibility

Deployment and support of the automation systems used by the operations teams in the plant such as production, maintenance, reliability, HS&E, and energy departments are the responsibility of the I&C department. This does not change with digital transformation and IIoT, but the scope goes larger since the operational infrastructure is extended with a second layer of automation enabling the digital transformation of remaining manual and paper-based tasks.

The I&C department already looks after the control system (DCS), safety system (SIS), package unit controllers (PLC), turbomachinery protection systems, motor controls, which includes all the process and equipment sensors and sensor networks such as fieldbuses as well as intelligent device management (IDM). For some processes there is also Model Predictive Control (MPC) software and Real-Time Optimization (RTO) software for the process as a whole, and may even include process simulation (e.g. HYSYS) etc. These systems may have been running in the plant for decades. The I&C department already has established relations with many approved I&C vendors that understands the production process and process equipment. Operations demand standard software with proven track record, custom configured, but no custom programming.

Pervasive Networking

In most plants the I&C department has already started to deploy wireless sensor networks for digital transformation of how the plant is run and maintained. Many new plants are using fieldbus for process control, and additional sensors can be connected to these networks. Sensors are critical to data-driven work processes.

Equipment Analytics

The maintenance and reliability engineers need real-time equipment analytics for condition monitoring and performance monitoring of equipment like pumps, compressors, fans/blowers, heat exchangers, cooling towers, and air-cooled heat exchangers etc. These are purpose-built predictive analytics apps for equipment monitoring for proactive intervention, usually model-based for predictable results providing descriptive equipment diagnostics of the problem and prescriptive directions. General purpose data analytics won’t do. The analytics aggregate data from multiple underlying sources such as the existing control system and the new sensors, either through the existing historian or directly from the source through OPC-UA. There is no need to deploy and maintain another costly layer of analytics platform.

Operations KPIs and Dashboards

Most plants already have a historian platform to which most new sensors installed for digital transformation are integrated, either directly or through the control system. The historian middleware is maintained by the I&C department and computes the operational Key Performance Indicators (KPIs) used by plant operations managers such as the production manager, reliability manager, maintenance manager, energy manager, quality manager, integrity manager, and plant manager etc. and are displayed on operations dashboards configured specifically for their roles. Some KPIs use information from the CMMS. More than a hundred different KPIs providing the insights that drives the work processes may be in use by operations personnel in various roles:

·      Production manager’s dashboard: Production rate, Quality. Waste & Rework

·      Reliability manager’s dashboard: Overall Equipment Effectiveness (OEE)

·      Maintenance manager’s dashboard: Maintenance Index, Availability, Predictive/Preventive/Reactive split, Planned/Unplanned Shutdown rate

·      Energy manager’s dashboard: Energy usage, Energy cost, Flaring, Energy Intensity Index

·      HS&E manager’s dashboard: Recordable incident rate, emissions

·      Quality manager’s dashboard: Process capability index, Quality, Waste & Rework

·      Plant manager’s dashboard: Production rate, OEE, energy cost, incident rate, and quality

IIoT Center

For companies that operate a fleet of many plants, mills, offshore installations, stations, factories, or other sites it may make sense for their I&C department to setup their own IIoT center with their own pool of experts in vibration, process equipment, control valves, analyzers, and steam traps etc. to monitor these equipment across all their sites. The subject matter experts at the IIoT center generates reports that drives the maintenance activities and other work in the plant.

Asset RFID

Intelligent devices like transmitters, valve positioners, and analyzers etc. have a microprocessor and digital communication such as fieldbus including a unique identifier (ID) so their data can be retrieved from their memory or referencing their ID with a document management system. However, lots of equipment in a plant do not have a microprocessor and digital communication. On these assets the I&C department mount RFID tags. For instance, RFID tags are mounted on control valves, containing the same information as on the traditional identification plate, and more. Basic valve and actuator data is readable directly from the RFID chip, but the identifier can also be used to retrieve documents from a document management system. RFID tags comply to the ISO 18000 standard.

For the industrial environment RFID tags have many advantages over other solutions such as QR code.

Plant Wi-Fi

Digital transformation of certain operations tasks in the plant requires use of laptop, tablet computer, video camera, smart glasses, or RTLS tags etc. In these plants the I&C department deploys industrial grade Wi-Fi infrastructure for the plant areas. Operations people in these plants use tablets instead of clipboards and paper for digital inspection on software data collection forms as well as digital checklists. Tablets are also used for document access, web browsing, and to report safety incidents, hazards, damage, and leaks etc.

In-plant Mobility

Within the plant, operations personnel use mobile devices like smartphones, tablets, and traditional handhelds getting the data across the plant-wide Wi-Fi infrastructure provided by the I&C department. This includes looking at the role-based KPI dashboards, checking equipment condition and performance through the analytics apps, getting instant messages (IM) w.r.t. equipment condition and performance, as well as monitoring process parameters.

Wearables

The Wi-Fi infrastructure in the plant is also used by maintenance personnel with wearable two-way video, with the camera and display mounted on their helmet for them to get live guidance from an expert in another location while their hands are free to do work. The expert sees exactly what the technician at sites sees, and the technician sees the expert in the other location or whatever the expert choses to show. This is another good example of how the Wi-Fi infrastructure provided by the I&C department supports plant operations.

Real-Time Locating Services (RTLS)

Some plants have deployed Real-Time Locating System (RTLS) in certain areas using the Wi-Fi infrastructure. Plant personnel, visitors, and contractors wear RTLS tags. This enables many new software functions such as digital mustering where the software tallies who has mustered and who is still trapped in the plant, and displays location of missing persons over a plant map or plot plan such that they can be rescued faster by a minimal search party. The RTLS system also provides geo-location and geo-fencing, meaning that if somebody strays into an area where they are not supposed to be, local and centralized alarms go off. Various reports can be extracted. The RTLS system is very much like a position sensor. The calibration and position accuracy of the locating system is the responsibility of the I&C department. Alarm can be triggered when a person has not moved for a preset period of time indicating a man-down situation, or when distress button is pressed. Traditional security access control cards can be combined into the RTLS tags.

Virtual Reality (VR) and 3D Scanning

Virtual Reality (VR) is digital transformation of learning; a new digital way to train and assess field operators to perform manual plant tasks like startup, shutdown, loading, and offloading etc. which require opening and closing of valves, and checking gauges etc. This is done indoor in a classroom environment. Trainees can make mistakes without making headlines. The virtual plant is a 3D software simulation of the physical plant but for the trainee wearing the VR goggles the experience is very immersive and real including vessels, piping, catwalks, monkey ladders, transmitters, and valves etc. The instrumentation and controls is sometimes integrated with the process simulation software part of the automation system such that when the valve is closed the flow and pressure reading on the instrumentation changes etc. I&C department which supports operations in the physical plant also supports operations in this virtual plant, including the integration with the process simulation through OPC-UA. The 3D virtual environment is for new plants developed from the 3D model used for plant design, or in the case of older plants for which no 3D model exists, one can be created through 3D laser scanning.

Augmented Reality

Augmented Reality (AR) is about field operators in the plant wearing smart glasses which superimposes live measurement data, equipment information, and instructions in their field of vision. You might say it is just another computer display you wear on your head, but the beauty is that your hands are free to do work, and the display automatically follows your gaze so when you turn your head and look at a tank it displays the fluid level inside it, when you walk up to a pump it displays its condition, and so on. That is, the I&C department integrate the AR smart glasses with the control system, historian, analytics apps, and other operational infrastructure data sources to provide the field operators the data they need. The live data is streamed across the plant Wi-Fi network. Incorporating new live data into the field of vision in the AR system as and when new measurements and analytics are added in the plant is the responsibility of the I&C department. If smart glasses are not available, AR is also possible with a smartphone or a tablet where data is superimposed on the screen over the image of whatever the built-in camera is aimed at. However, one hand is occupied holding the smartphone or tablet.

IT responsibility

Deployment and support of the business systems used by the admin teams such as finance, human resource (HR), and legal departments etc. are the responsibility of the IT department. This does not change with digital transformation and IIoT.

The IT department already looks after the Ethernet LAN, Wi-Fi, file servers, printers, and computers in the admin building including the Internet connection and associated security. There is also the phone system, electronic identity card access control, video conferencing, and CCTV. The IT department already has established relations with many approved IT vendors, IT consultants, and IT analysts that understand business and admin, but not process and equipment.

Enterprise Resource Planning (ERP) Business Platform

Most companies have an ERP system (e.g. SAP or Oracle) which is the business platform maintained by the IT department. The ERP system is typically a “platform” for software modules including employee Performance Management System, Learning Management System (LMS), permit to work, work order management, planning and scheduling, Supply Chain Management (SCM), Vendor Management System (VMS), and many more. The ERP platform modules typically require extensive custom programming by third-party developers which is very costly. This is very different from the I&C approach of using readymade software.

L4 Integration

The KPI information in the operations dashboards for the managers of the various operations departments as well as detail data and information for process equipment etc. used by engineers and technicians can also be made available on their computers at their desks in the admin building by going through a De-Militarized Zone (DMZ) of firewalls onto the office LAN at Purdue model level 4 (L4) of the enterprise architecture. Equipment diagnostic alarms and data related to work order management may go into the ERP through the same DMZ. The office LAN, computers, and ERP system are supported by the IT department while the data comes from operational infrastructure such as the historian and analytics apps at level 3 (L3) supported by the I&C department. The purpose of the DMZ is to protect the control system and other operational infrastructure at L3 and below. The DMZ sits right in between at L3.5. This is where I&C and IT systems connect together for data to pass through; “IT/OT integration”.

Note that the Office LAN and ERP integration can come at a later stage. Digital transformation typically starts with operational infrastructure simply integrating with the historian and control system, or using a dedicated server for equipment analytics. The historian may already be accessible from office computers in which case data and analytics connected to the historian automatically becomes accessible from the office desks.

Internet Connection

Some digital transformation solutions, but far from all, require an internet connection. Internet connection is required for any IIoT solution, cloud, and mobile/tablet access outside of the plant. A secure internet connection is provided by the IT department with a DMZ at L5.5.

Note that the internet connection can come at a later stage. Digital transformation typically starts within the plant; where the users of the data are all on-premise (“on-prem”) such as the maintenance, reliability, and energy efficiency engineers, as well as safety officers etc. In this case data does not leave the plant.

Also note that IIoT solutions in some cases need not be connected to the control system or other operational infrastructure in the plant. For instance, vibration, corrosion, and steam trap monitoring are often deployed as an independent system. In this case the IIoT solution is physically separate from the plant’s own networks. A physically separate IIoT solution cannot be a path into plant networks. This means the I&C department can easily deploy it themselves.

When the IIoT system is physically separate with its own internet connection not using plant networks then the I&C department can manage on their own and IT need not provide any connection. If the IIoT system uses plant networks then IT must support.

Many control systems running plants were designed 30 years ago, never meant to be connected to the Internet. Various companies have developed security solutions for connecting control systems to the Internet, such as using industrial data diode.

Cloud

Some digital transformation solutions, but far from all, require cloud computing or data storage. For some solutions data is sent to the company’s own cloud account, and for other services data is sent to a connected service provider’s cloud account. For instance, cloud is used for IIoT-based connected services such as equipment condition and performance monitoring by third-party service providers. Cloud may also be used for mobile/tablet instant messaging (IM) outside of the plant. For other solutions the company may have their own cloud account in which case the cloud servers are managed by the IT department. For instance, the L4 ERP system may be executing in cloud.

Business Intelligence (BI)

Business Intelligence (BI) is a form of “analytics” for operations-wide data discovery using visual software tools (e.g. Tableau, Tibco Spotfire, MS-Power BI, or QlikView) visualizing big datasets from the ERP platform as bar chart, trend chart, pie chart, map, scatter plot, Gantt chart, bubble chart, histogram, heatmap, and treemaps etc. to uncover new insights. The idea is to visualize data to spot correlation to possibly get new insights, perhaps identify new opportunities.

For instance, correlating transactional ERP records like site access records, shift/duty records, maintenance work order records etc. might perhaps be able to link equipment failure with who serviced it and who was operating when it failed - if any such relationship exists that is.

That is, L4 business intelligence is very different from L3 equipment diagnostics used by engineers and operations managers. But it is interesting to note how the equipment condition and performance information output from real-time L3 equipment analytics in the production-end feeds as input into L4 business intelligence analytics in the enterprise-end. Enterprise-wide analytics is easier to manage if done this way because data is analyzed in this decentralized manner, starting from the sensor where thousands of raw data samples are distilled into an indication of bearing impacting, and at the next level aggregating data from multiple sensors into pump condition, and at the BI level into overall state of the plant and business such that a trader can decide to place an order for opportunity crude or not.

This layered approach is more effective than just feeding raw data from all sensors into L4 analytics directly, hoping to “discover” some correlation which is probably already known (as many have discovered) and requiring huge data volumes to be transferred.

Run analytics as close to the data source as possible, in the sensor if supported. This reduces the data volume transferred and stored, and real-time data gives more accurate results and timely response.

Business KPIs and Dashboards

Most plants already have an ERP platform which holds accounting and transactions. The ERP platform is maintained by the IT department and computes the business KPIs used by finance manager, business development manager, procurement manager, commercial manager, marketing manager, sales manager, HR manager, IT manager, security manager, and traders etc. which are displayed on business dashboards specific to their roles. These KPIs and dashboards are very different from operations dashboards and KPIs. Business KPI may include EBIT, NPAT, gross margin, ROACE, and revenue etc.

Note that business KPIs are not based on measurements by sensors but are derived from the ERP system

Note the difference between real-time equipment analytics at the operations level feeding information to the transactional business intelligence at the enterprise level

Equipment analytics at the operations level is real-time while business intelligence at the enterprise level is transactional

Video Analytics

By migrating to digital CCTV cameras, video surveillance can go beyond today’s motion detection by using digital video analytics like facial recognition, license plate recognition, object recognition, and left-behind object detection etc.

Integration and Collaboration

The operational infrastructure and IT infrastructure meet and integrate at L3.5 of the enterprise architecture. The DMZ is the boundary of responsibility as illustrated in the Department of Homeland Security (DHS) National Cybersecurity and Communications Integration Center (NCCIC) and Industrial Control Systems Cyber Emergency Response Team (ICS-CERT) recommended practice Improving Industrial Control System Cybersecurity with Defense-in-Depth Strategies.

This is where the IT and I&C departments collaborate the most. There are in fact two DMZ which ICS-CERT refers to as the level 3.5 Control Center LAN DMZ and level 5.5 Internet DMZ.

The IT and I&C departments may already have collaborated to make the historian accessible from the office LAN in which case data and analytics connected to the historian automatically becomes accessible from the office desks without additional work. If the company has an enterprise historian as well, then information in every plant becomes accessible from any company office anywhere in the world across the company intranet.

For example, the equipment analytics software supported by the I&C department interfaces to the ERP system and Business Intelligence software supported by the IT department. Equipment alarm is integrated to the ERP system maintenance module through OPC-UA to trigger work order. Health and performance index for individual pieces of equipment are integrated into BI through OPC-UA, rolled up for plant unit, area, and plant operations as a whole. Access to full equipment diagnostics detail is provided as HTML5 pages through a web server.

IT/OT Integration Made Easy

So, IT/OT integration is not as hard as they say because it is about integration and collaboration between IT and OT, not about merging the IT and OT departments. Similarly, IT project vendors collaborate with OT project vendors. This collaboration between departments and between vendors is made easy by using agreed upon standard interfaces like OPC-UA which allow data to pass from operations to enterprise zones with full metadata and information model etc. Indeed if a single vendor did both the IT and OT projects, the interfaces between them would likely end up being proprietary APIs or proprietary data formats over MQTT transport. Proprietary technologies always becomes costly to support in the long run. Caveat emptor.

The future is digital, but don’t attempt to implement all digital technologies and solutions in a single project. Implement digital transformation in a few phases, with tracks for IT and for I&C (“OT”).

Start with an operational certainty discovery session to uncover the needs for digital transformation in the plant. Based on this a digital roadmap can be created for the plant. Well, that’s my personal opinion. If you are interested in digital transformation in the process industries click “Follow” by my photo to not miss future updates. Click “Like” if you found this useful to you and share it with others if you think it would be useful to them.