5 points to consider before starting your reinforcement learning project for process manufacturing optimization

Every Autonomous System (AS) project built using the?Microsoft Project Bonsai?AI toolchain (or any other tool) will comprise the same core set of components. Whether the goal is to control an extruder's specification drift, improve a paper mill start-up time, or guarantee the quality of a food manufacturing process, the following two high-level elements will always be present:

• A set of process inputs and outputs that provide both the correct view of its behavior and the right "levers" to control it.
• An effective process simulator to train the AI agent (aka Bonsai "brain") using deep reinforcement learning (DRL).

It is impossible to leverage this AI training strategy to generate meaningful results without those two things. They look pretty straightforward when taken at face value. However, both imply more granular aspects critical to improving a DRL project's probability of success.

Here are the top 5 to consider:

1. Process subject matter experts need to translate their expertise into measurable data points and decision-making criteria
2. Appropriate sensors and actuators must exist or be added before any further development can start
3. An accurate process simulation must exist or be developed
4. The project goals and ROI targets must be estimated and agreed upon before the project starts
5. Potential blocking issues need to be proactively identified early on

1. Translate subject matter experts' tacit knowledge into explicit, measurable data and decision-making criteria?

Process subject matter experts (SMEs) usually have developed heuristics that help them predict and spot manufacturing issues. Some of those heuristics, or shortcuts based on years of experience on the production floor, can be translated into hard data that existing sensors already capture.

Unfortunately, those heuristics are often the combination of hard data, such as melted plastic temperature and other elements. For instance, an operator could spot something about to go sideways due to a change in color output, machine noise, or perceived (eyeballed, not measured) viscosity.

To train an AI agent that can improve the process, the project team will need to extract those often intuitive (tacit) process expertise from the SMEs, codify them, and figure out how to measure them. It is a crucial element of the?Machine Teaching?aspect of the overall project.

2. Be ready to develop and add new IoT sensors to your production line?

Existing sensors and actuators will often not be enough to measure those "soft" process aspects, and new advanced intelligent IoT sensors will have to be added to the manufacturing line.

For instance, PepsiCo had to build a product visual characteristic measuring system to capture various Cheetos snack shape and color elements for its extrusion project. With this automated measuring system, both AI agent training and operation can now use those hard data points.

In our experience, this can be overlooked during the initial project phase. It's critical not to underestimate the complexity of translating human expertise into measurable process data. Sometimes, it is impossible to capture this "soft" data using today's technologies. In turn, this can make a DRL unfeasible.

3. Have or build an accurate process simulator?

Autonomous Systems use simulators to train their AI agents, or in Project Bonsai vernacular, their "brains." The animation below depicts a typical DRL training loop.

As the illustration shows, the agent can't be trained without a simulator that accurately models the process' behavior.

There are?multiple methods to build a process simulator?if no simulation exists. In our experience, an AI simulator (i.e., trained with the process' real-life data) is?often the best approach, but it's not the only one.

If the first two points in this list have been resolved, creating an initial usable process simulator can take anywhere from one to six months. The time and effort needed will correlate with the first two points, the simulator technology used, and the required performance.

For instance, in one of our projects, we quickly (less than two months) developed a simulator using a standard simulation tool (AnyLogic). However, the speed was inadequate to iterate enough on the brain training quickly enough. So, we scaled up the simulator cloud back-end (with the associated cost). At the same time, our data scientists developed an AI simulator using the teacher/student approach explained in more detail?here.

4. Estimate the ultimate project ROI, and communicate openly about the time needed to demonstrate it

Until all the previous steps are finalized, the AI agent can't be trained, and, therefore, the project team can't demonstrate visible returns. The simulator can allow the development of a process Digital Twin and enable "what if" analysis. That can have significant value on its own in some projects. However, it's often not enough to justify all the time, resources, and money spent.

Therefore, the project team must be transparent about expectations and timelines. The expected final project timeline, goals, and ROI need to be proactively (over)communicated to all stakeholders from the project onset. It will avoid spending months getting to the end of the simulator development only to have the project canceled because no clear ROI was demonstrated. Yet.

It can be hard to put precise ROI numbers behind the initial project. It is often the case with new technologies used on complex business challenges. DRL-trained AI agents are no different. The project team and their executive sponsors need to be conscious of this constraint from the beginning to avoid any unnecessary frustration later.

The benefits of this technology can be tremendous. However, it sometimes will require a bit of a leap of faith compared to traditional process manufacturing investments such as new PLCs, SCADA/MMI, sensors, etc.

5. Proactively identify potentially blocking issues

As mentioned above, quite a few roadblocks can show up and can derail an Autonomous Systems project. In addition to extracting and codifying SME's tacit knowledge, implementing the suitable IoT sensors and actuators, and building a simulator that accurately models the process, additional roadblocks can slow down or even stop a project.

These roadblocks can come in many shapes and sizes. The most common types are:

• Human: are SMEs available, capable, and willing to share their knowledge??
• Logistical: can project members (engineers, data scientists, etc.) access the manufacturing line during live production? For instance, during the pandemic, a project was delayed by almost a year because our Data Scientists could not physically travel to the plant where the Autonomous System was going to be deployed.?
• IT: Are the IT systems ready to collect, store, and share the raw data needed to create the models? Does a significant data migration and modernization step need to happen before the simulation development and bran training? The answer is often a resounding yes!?
• Process control: Are the exiting controllers able to share their inputs and receive commands from an external (edge) AI? Are all controls automated, or are some still manual??

If you are interested in learning more about mitigating those points,?this article?elaborates on our teams' specific steps to tackle them.

Autonomous Systems are a crucial component of Industry 4.0's promise to the process manufacturing industry. If manufacturers keep those five critical elements in mind when starting a new project, the chances of success are much higher. We learned them the hard way through our many successful and (fortunately) few unsuccessful Bonsai projects. Hopefully, our experience can help you avoid those pitfalls and increase your project success rate.

(This article was initially published on Neal Analytics blog)