Enterprise OT - Structured Integration for Digital Transformation

Digital transformation (DX) includes both internal and customer facing changes managed by different teams and vendors. Manual and paper-based procedures are replaced with new digital ways of working. It may include centralized fleet management for multiple plants. These changes are under way right now. These transformations are made independently, at different time and pace, each requiring unique know-how, managed by different groups responsible for the systems involved. Yet these systems are connected. What is the proper way to address this? Here are my personal thoughts – so this is more of a “how to” piece than a “vision” piece.

Most enterprises have multiple manufacturing sites. Each site has an admin office and a production plant. The plant is responsible the Line of Business (LoB); production of the product, which also has several associated activities such as maintenance, reliability, personnel safety, energy management, integrity, and quality control etc. The office is responsible for admin and finance including: placing orders on suppliers and processing orders from customers. Therefore manufacturing sites use their IT vendors for office digitalization and I&C vendors for plant digitalization.

Image courtesy ISA95 (my colors)

Horizontal Customer Facing Digital Transformation

Office activities are customer facing, and supplier facing. Processing orders from customers, providing updates on shipment, and accounts receivable was traditionally very manual. Improving the customer experience such as with convenience, ease of use, and timely updates are now an important aspect of digital transformation. Similarly, placing orders on suppliers, tracking deliveries, and accounts payable was also very manual. Digitization means changing manual and paper-based tasks to digital commerce for productivity and speed.

Over the past decades most companies have already digitally transformed office work with an Enterprise Resource Planning (ERP) system business platform at level 4 of the ISA95/Purdue reference model. This was typically done in a phased approach over many years, perhaps starting with a module for finance & accounting, then Human Resource (HR), and Customer Relationship Management (CRM) and so on. One of the latest ERP modules may be Supply Chain Management (SCM) for end-to-end digital integration with other supplier and customer supply chain sites, and other vendors. Lately many companies in the retail-end have developed mobile apps and websites to digitalize the customer experience: digital commerce.

Since digitalization is about new ways of working, not the enabling technologies, digital transformation is best driven by the personnel whose work will be transformed. For the office this is the admin personnel such as accounting and finance, business development, procurement, commercial & marketing, sales, inventory & trading, HR, and so on. as they are accountable for the outcome and value of the investment. Moreover, the office infrastructure technology is office tools such as the office Wi-Fi, ERP business platform, business analytics, and business dashboards associated with business administration so the additional technology for the realization of horizontal integration and transformation of the customer facing digital commerce experience is best supported by the IT group too.

The ERP business platform storing the business data and which the business apps run on, MS Office, IP phones, video conferencing system, office LAN and Wi-Fi infrastructure, servers for Intranet and data storage, and associated cybersecurity are Information Technology (IT) managed by the IT department with all the associated annual licensing and service contracts.

In the structured integration model the SCM integration between sites and corporations is horizontal. It could be sites within the same company, or external suppliers and customers for digital commerce. An upstream wellsite or offshore platform can be digitally integrated with a liquefaction plant SCM system part of the level 4 ERP business platform, which is integrated with the pipeline automation, which is integrated with the automation in a storage terminal or a regas terminal, which is integrated with the systems in a power plant or a chemical plant, which is integrated with a factory, which is integrated with retail stores, which digitally interface with the ultimate consumer through customer facing apps. The SCM allows plants along the chain to manage the supply. There are companies that operate along the entire value chain from mining to retail, but most companies only play in part of the value chain.

ERP business platform are proprietary monoliths. The interfaces between the ERP modules are all proprietary API. There are no detailed ERP standards with data formats etc. Therefore modules from different ERP vendors do not work together. Business platform apps and mobile customer facing apps must be custom programmed by a developer team. Because the ERP is not an open system, the customer facing DX and horizontal integration between ERP is best done by the ERP vendor since they know the product best, managed by the site’s IT department. It may not be a good idea to get a third-party to do it. The ERP vendor also has the business process expertise.

Vertical Internal Plant Operations Digital Transformation

Plant activities are internal. Receiving feedstock, processing, and shipping product. The production process is highly automated in most industries. Particularly for continuous processing. Plants with batch processing tend to have some manual steps. The most interesting fact is that the other activities associated with production such as maintenance, reliability, personnel safety, energy management, integrity (corrosion and erosion), and quality control etc. until today tend to still be very manual and labor-intensive.

The Core Process Control (CPC) is performed by the DCS which is already digital. So internal digital transformation in the plant is now focusing on Monitoring and Optimization (M+O) for the other activities such as maintenance, reliability, personnel safety, energy management, integrity, and quality control etc. Plants deploy Digital Operational Infrastructure (DOI) to enable this digital transformation. This transformation is best done in a phased approach over many years, perhaps starting with readymade solutions for condition monitoring, location awareness, energy management, loss detection, corrosion monitoring, and reducing manual field operator rounds etc. with digital integration with the level 4 ERP business platform. This may include sending notifications to mobile apps on personnel’s smartphones. The advanced industrial sensors and software in the DOI may come from multiple vendors, often not the same as the DCS vendor.

Because digital transformation is about changing work practices, not the underlying technology, it must be led by the teams that actually do the work being transformed. In the plant this means the Line-of-Business operations teams like production, maintenance, and reliability etc. as they are responsible for results and ROI of the solutions. Also, the operational infrastructure technology is I&C like advanced industrial sensors, wireless sensor network, industrial Wi-Fi, historian platform, operational analytics apps, and operational dashboards closely related to process control so the additional technology for transformation of internal plant operations is best handled by the I&C department too. Operations and I&C must lead the digital transformation effort to ensure real problems are being solved so the solution gets used. A January 2019 McKinsey article "What separates leaders from laggards in the internet of things" finds "Understandably, many executive teams have perceived the IoT mainly as a technology challenge and put their chief information officers (CIOs) at the helm of their IoT efforts. Casting the IoT as a pure technology play, though, carries the risk that companies will overlook all the value they might capture by redesigning processes to take advantage of the IoT’s capabilities.

The sensors, actuators, DCS hardware (I/O and controller) and software (HMI etc.), historian, plus suite of other apps in the DOI, and associated ICS cybersecurity are Operation Technology (OT) managed by the I&C department.

In the structured integration model the integration between office systems and plant systems (IT and OT) is vertical. This integration is within the same site. So for instance, when analytics in the plant predict a problem with a pump it sends a work request to the CMMS/ERP in the office. Once approved by the maintenance scheduler a work order is sent to maintenance personnel in the plant.

DCS are integrated systems where hardware and software are purpose built and tested together by the same manufacturer. Yet DCS now support the open OPC-UA standard (IEC62541) interface which enable M+O software in the DOI to read data from the DCS, and the DCS to read data from the M+O software in the DOI. That is, OPC-UA enables software from different vendors to work together. The plant uses readymade software and readymade mobile apps so custom programming by a developer team is not required. Plant personnel value Proven Track Record (PTR) as opposed to custom made. Internal plant DX and vertical integration with the ERP system is best done by the I&C vendor that actually makes the readymade purpose-built analytics software, networking, and advanced sensors, since they know these products best, and best managed by the site’s I&C department. It may not be a good idea to get a third-party to do it. The I&C vendor also has production, process control, process equipment, maintenance, reliability, corrosion, and hazardous area etc. expertise.

Business Admin Digitalization

The ERP system is the business platform intended to cover all the company’s sites. The local site office IT and global enterprise IT teams work together, although site-IT may be outsourced.

Plant data from I&C systems is mostly just passing through the IT infrastructure at L4 across the Internet and optionally into a cloud data center. The important role of the IT team in digital transformation of the plant is to provide the connection to the Internet for the OT systems that need it, like Augmented Reality (AR), operational dashboards, and notifications etc. Therefore it is important for the I&C team to also involve the IT team in digital transformation, from the very beginning of the project, and for IT and OT teams to collaborate. Some plant data goes into the ERP business system. IT data (from ERP system and office suite etc.) and OT data (from I&C systems) flow across the same WAN. Today data is a utility just like water and electricity flowing through the plant - just as important.

Data is a utility just like water and electricity

The challenge of digitally integrating SCM between companies in a supply chain is that they may use different ERP business platform (SAP, Oracle, and Infor etc.) and SCM software, and because these are not open systems, the software APIs are proprietary. Thus this integration requires custom programming by a developer team. Even different versions of the same brand ERP may not be compatible. Because modules and apps get customized, a deployment by one system integrator (SI) at one site may not be compatible with a deployment by another SI in another site. The respective ERP vendors for these companies work together for the horizontal SCM integration between plants and companies. A joint development team will be required. A challenge is corporations from time to time divest and acquire sites as their product focus changes and trying to establish supply chain synergies etc. With a new site comes a new office, often with a different brand or version of ERP to integrate.

The ERP vendor is experienced working with office teams (accounting and finance, business development, procurement, commercial & marketing, sales, inventory & trading, HR etc.) as ERP vendors work with them every day. ERP vendor conducts workshops with the office teams to uncover customer and supplier facing challenges to advice on the right tools (apps) for staff in each department. The ERP vendor develops apps and the personalized business dashboards and notifications with the underlying analytics on ERP data. New customer facing website and digital commerce apps may also be developed. The ERP vendor provides training for the ERP system tools (apps) the personnel in the office will use.

Customer-centered design, customer experience, mass-customized product development, and minimum viable product (MVP) makes most sense at the manufacturing and retail-end of the supply/value chain which is facing consumers, not so much in the upstream mining and processing-end. Most process plants make a single or very few products. Some companies have their own retail arm like petrol kiosks (gas stations) which are a separate business unit in the other end of the long chain from production, transportation, and processing.

Fleet I&C: Enterprise OT

The DCS and plant historian make up the platform whose scope is the plant; a single site. The enterprise historian is the data aggregation platform intended to cover all the company’s sites; the entire fleet. The local plant I&C team and global I&C engineering team work together. Just like there is enterprise wide IT, there is also enterprise wide OT. Example of enterprise wide I&C (Enterprise OT) includes fleet management center covering multiple production sites, or for operational dashboards aggregating information from multiple plants, or for operational notifications coming from multiple plants etc. That is, I&C (OT) is no longer confined to a single plant, hence “Enterprise OT”. The term “enterprise” can be a bit confusing, because sometimes it is taken to refer to business admin activities, and other times it is used meaning multiple locations across the corporation. I will as much as possible use the term “fleet” when referring to multiple plants. So we could use “Fleet I&C” interchangeably with “Enterprise OT” or “Central Monitoring and Optimization” (Central M+O) as it is called in the NAMUR Open Architecture (NOA) for Industrie 4.0 (Industry 4.0).

Image courtesy NAMUR

When people say "IT" they often mean "digital". In the plant it is not IT products. It is digital technology, software and hardware, but it is OT. For example, the analytics used to analyze process data and plant equipment data to optimize and predict problems is not IT. It is not the same business analytics used on business data from the ERP system. Process and equipment analytics are purpose built for these tasks, with a huge amount of subject matter expertise built in. Sure, these systems use Ethernet, UDP/TCP/IP, and Windows etc. since 25 years ago, but the application protocols and APIs are standardized and different. Ethernet-APL will soon be used in field instruments.

When people say "IT" they often mean "Digital"

Fleet Management Center – Integrated Operations (iOps)

Many companies own and operate multiple sites; the mix depends on the industry and may include breweries, distilleries, factories, mills/plants/stations, offshore installations/platforms, mines/quarries, refineries, terminals, vessels, and well sites. One of the many objectives of digital transformation may be to centralize some of the operations and management of multiple plants to a fleet management center; a single central “control tower” with regional or global coverage to improve productivity and reduce travel cost etc. Some industries, like upstream oil & gas production, use the term “integrated operations” (iOps) instead of fleet management center. Operating sites from a central location requires a secure remote connection into the DCS at site. Most plants are not yet ready for this. Engineering this solution is not trivial, but DCS vendors are experienced in doing it.

Traditionally fleet management is only for process supervision, such as power generation. However, with digitalization of plant equipment, the fleet management center also does maintenance management such as predicting and diagnosing failure, and scheduling maintenance. Fleet management 4.0 if you will. Maintenance management is a level 3 activity (ISA95) regardless of where the maintenance office is located: in the plant itself, or a central maintenance control tower for a fleet of plants. So even when maintenance is managed from another location, and data goes across the Internet, it is still a level 3 activity.

Fleet Operational Dashboards

Digital integration of I&C systems (OT systems) across the corporation, at multiple sites within the same company, is typically done through an enterprise historian to get operations visibility across the fleet of multiple sites. Every site has a local plant historian, all the same brand, and they are all networked together across the corporate Wide Area Network (WAN) over the Internet. Plants may use different DCS, but the historian is the same brand at all sites. Plants today have a plant historian that stores time-series process variables and process alarms; a process historian if you will. Plants are now adding other data storage on the side; for non-process data like reliability data such as vibration spectrums; a reliability historian if you will. They complement each other. The data is not duplicated.

Also note that I&C systems are not connected between companies in the supply chain. Through the enterprise historian data from multiple plants is aggregated into a single dashboard; such that for instance an energy manager responsible for multiple sites gets data from all those sites in that single dashboard, and can then zoom down into individual sites, areas, units, and down to individual pieces of equipment. Dashboards are personalized, meaning they are designed with the real-time indexes and other information each person needs to do their job better, to meet their KPIs. For instance, the plant manager’s dashboard is different from that of the production manager or the integrity manager. These dashboards are webpages viewable on all kinds of devices from wall screens to smart phones, anywhere like in the plant, office, boardroom, or on your way to or from work.

Fleet Operational Notifications

Similarly, alarms & events from various I&C systems (OT systems) across the fleet from all the corporation’s plants organization-wide are captured and relayed through the cloud as notifications to personnel’s smartphone wherever they may be; such as on the plant floor, in the admin building, at the corporate headquarters, or traveling. Notifications are personalized, meaning they are relayed to the right person who can act. For instance, the reliability manager and engineers get notifications different from the safety manager or the quality manager. That is, personnel do not receive nuisance alarms not relevant to their responsibilities. Notifications from multiple sources are aggregated into a single notification app, so you don’t get the notifications scattered across multiple apps in different formats.

Notifications from multiple sites are aggregated into a single noticeboard; such that for example a maintenance manager responsible for multiple plants receive notifications from all those plants on that single noticeboard, together with relevant information required to act.

Connected Services

Plants may not have enough subject matter experts (SME) to interpret the equipment health and performance data such as associated with control valve, vibration, corrosion, and even steam trap monitoring etc. Some plants are therefore opting for Industrial Internet of Things (IIoT) based connected services; meaning a service provider with SMEs in another location monitor equipment in your plant, and others, and sends a report to the team in the plant to take preventive or corrective action when and where needed. This is enabled by installing permanent sensing in the plant, and sending the data across the Internet to analytics software in a data center in the cloud. In the case of control valve monitoring you install intelligent valve positioners, but then again an intelligent valve positioner is a package of sensors (position, actuator pressures, supply pressure, temperature, air mass flow, and pneumatic relay position) so the prediction is still based on sensor data. A service provider monitors plant equipment across all the company’s sites so this is also an organization-wide OT solution.

Note that the implementation of connected services is done such as to not pose a risk to the process control (DCS) and safety systems (SIS). For instance, vibration, corrosion, and steam trap monitoring plus many others are not related to process control so the associated sensors need not get connected to the existing DCS. Instead a separate Digital Operational Infrastructure (DOI) is deployed on the side of the existing DCS to separately send data to the cloud without providing a connection (path) to the DCS thus not creating additional risk. Moreover, sensors don’t impact the process. However, control valve monitoring is different. Plants have Intelligent Device Management (IDM) software on a local server connected to the DCS through which control valve positioner data is accessed by the analytics, and if a positioner were to be accessed maliciously it could impact the process. Therefore, connected services for control valves involves a data diode through which data is published (pushed) from the IDM system to the cloud without providing a return path to the DCS (i.e. unidirectional).

SplitOps

As I write this, companies in various countries, including plants, run in split operations mode (split ops) as part of their business continuity plan due to the COVID-19 coronavirus, meaning personnel are split into two teams taking turns where one team works in the plant for a week while the other works from home to reduce risk to personnel. For office job functions, working from home is relatively easy with VPN connection and web conferencing. However, for plant operations this is more of a challenge for sites because personnel cannot access I&C systems (OT systems) from home. Outside access into I&C systems is considered risky so VPN access is not provided.

Personnel will not be running the process from home (i.e. no DCS access) but with a Digital Operational Infrastructure (DOI) in place for fleet management, operational dashboards and notifications, connected services, and many other functions could also be performed from home whenever business continuity plans call for split ops:

That is, fleet I&C enables decentralized work. Even during normal operation, this same DOI enables plants to connect with experts in other locations outside the plant to help with tricky problems or simply reduce the burden on plant personnel at times when they may be struggling to keep up with the workload, such as in preparation for, and during, a turnaround.

Plant Solutions for Digital Transformation (DX)

I&C vendors are experienced working with the plant operations teams as they work with them every day. I&C vendor conducts workshops with the operations teams to uncover internal plant operational challenges which need new approaches. The I&C vendors help plants identify new more effective internal procedures for plant operations and then advice on the right data and digital solutions to be applied. Mostly proven readymade solutions, as available. New solutions are co-created together with plant personnel where none exist yet. Engineers have a knack for finding or creating solutions to problems. This may include fleet management center for centralized operations of multiple plants. The I&C vendor develops a detail roadmap for digitalization in each plant and helps in periodic review. Additional detailed workshops are conducted to identify where mobile devices and wearables are required in plant operations and determine which information is required in operational dashboards and notifications for each role. The I&C vendor provides learning & development for the new digital tools used internally by the plant operations teams such as the sensors, wireless infrastructure, and software apps. Optionally the I&C vendor provides connected services for plant operations functions such as condition, integrity, and performance monitoring etc.

Digital Transformation – A New Era in Automation

The Fourth Industrial Revolution (4IR) is in full swing. Digital transformation is now mostly about line-of-business: production, safety, maintenance, and energy in the plant itself. For the best outcome, consult the experts. Get your I&C team to work with I&C vendors for the DOI, and get your IT team to work with your ERP vendor for the ERP.

Most productivity improvement coming from digitalization is in the transformation of the manual work done by the operational teams in the plant. like maintenance and reliability. Additionally, the integration between the IT and OT systems does improve productivity a little further, but not nearly as much as digitalizing work within the plant itself. Therefore, start from the ground up by digitalizing manual work in the plant like data collection, data interpretation, and reporting etc. before connecting to the ERP/CMMS. Well, that’s my personal opinion. If you are interested in digital transformation in the process industries click “Follow” by my photo to not miss future updates. Click “Like” if you found this useful to you and “Share” it with others if you think it would be useful to them.