How to transform AI from co-pilot to autopilot

I’m a mechanical automation engineer by training. Right out of graduate school, I worked in high-tech manufacturing automation for six years, building fully automated manufacturing lines that made components for hard disk drives. At first, I worked on cutting and grinding semiconductor wafers, then on building head-gimbal assemblies using vision-based robots. The parts we worked on were so small that you could only see them under microscopes, and the wires were thinner than human hair.?

In manufacturing, there is a term, “lights-out automation,” meaning full end-to-end automation that requires no human interaction. Since machines don’t need lights to see, if there are no human operators on the manufacturing floor, you might as well turn the lights off.?

Now that I’m in RevOps automation, I see a lot of parallel concepts from my old manufacturing automation days that can serve as a great model for how to think about RevOps automation, especially now that AI has been hyped to solve world hunger and soon, to shed light on the meaning of life. AI is certainly the most impactful technology advancement of this generation. What it will be able to accomplish is really only limited by your imagination. However, we still have a ways to go before we get there.?

In the meantime, AI is still a “lights-on automation” technology and will need a lot of help from other automation technologies to achieve lights-out automation. In this newsletter, I want to outline a framework to help you think about:

• The difference between lights-out and lights-on automation
• Why lights-out automation is so much harder than lights-on automation
• What it takes to achieve lights-out automation with today’s AI technology in RevOps

In short, how you can transform AI from a co-pilot into an autopilot.

Lights-out vs. lights-on automation

The difference between lights-out and lights-on automation is whether human intervention or assistance is required throughout the process. Humans can play different roles alongside automation, such as:

• Preparing the input - This can include picking the necessary parts, putting them into a jig/fixture, and putting the parts into the machine. This phase is often manual because this is where the very predictable environment of the manufacturing line interfaces with the very unpredictable environment of the manufacturing floor.
• Performing the task - This can involve humans either performing some tasks alongside fully automated tasks or giving machines a hand, such as fine-tuning what a machine recommends before the task is performed.
• Inspection - No automated machine can get things right 100% of the time. While you can automate quality inspection, tasks often need to be augmented by human inspectors.
• Remediation - When a machine does things wrong, or cannot perform a task because the input is beyond what it can accommodate, humans usually have to intervene to fix the problem, either by fixing the input so the machine can try again or by finishing the task manually.
• Handling the output - Just like it’s hard to automate putting things into the machine, the reverse is also true: it’s hard to automate taking things out of the machine and transition them into the human world.
• Monitoring - Automated manufacturing lines are monitored with sensors and cameras to ensure everything is running okay. This is almost always supplemented by human monitors, who either sit in front of a wall of computer screens or walk around the manufacturing floor observing the actions directly.

You don’t have lights-out automation unless the first 5.5 of these 6 areas are fully automated and the only human involvement is watching monitors in a control room. To put this framework into the RevOps context, consider a list loading use case:

• Preparing the input - Get the file from someone and somewhere. Make sure the data in the file meets minimum requirements. If it doesn’t, fix or fill in manually. Log in to the CRM and perform the loading action.
• Performing the task - Match the records to those in the CRM. Check for duplicates. Add, update, and merge the records accordingly.
• Inspection - Ensure the list has been properly loaded and there is no error.
• Remediation - Fix records that failed to load. Either resubmit the list after the data has been fixed, or manually add and edit the records in the CRM.
• Handling the output - Act on the new list: add it to a campaign, or download and send it to a marketer.
• Monitoring - Ensure the list is being processed, especially the list is large. Confirm there are no issues such as queue backup, excessive API calls, or syncing issues between the CRM and the marketing automation platform. Alert the submitter that her list has been loaded with details on which records were accepted and which records were rejected.

As you can see, a typical list loading process is, at best, a lights-on process due to the high degree of involvement from human operators. To achieve lights-out automation, the technology you use has to automate the following tasks:

• Fetch the file
• Inspect if the records all satisfy minimum requirements
• Reject failed records or files, and notify the submitter to remediate and resubmit
• Clean, standardize, and segment the data
• Enrich using one or more enrichment sources
• Unify data from multiple enrichment sources to create a golden record
• Match and deduplicate against the CRM
• Add, update, and merge against the CRM
• Handle errors from the CRM, including retry or perform alternative action
• Initiate follow-on actions such as add to campaign
• Alert operator and submitter of task completion
• Measure and log data for inspection and review

Why lights-out automation is so much harder

People and data are never perfect and real-life is messy. When you deal with real-world data and processes, there are a wide range of scenarios where things can go off the rails, such as:

• Input is missing or not as expected
• A technology along the chain is offline, slow to respond, or generating errors
• A human has intervened and the process is altered
• AI input or output may not perform consistently or even contains hallucination

Your automation technology needs to handle the vast majority of variations without asking for human intervention, if you’re to realize the scalability and throughput benefit of automation.

This is why it’s fairly easy to build and implement lights-on automation and much more difficult to achieve lights-out automation. Lights-on automation only has to worry about the majority of scenarios where the input is as expected, because it can always count on the human operator to fix things, deal with the unexpected, or just take over.?

Lights-on automation makes human operators more efficient and is often the only viable option if throughput is low. Light-out automation becomes mandatory as throughput increases. For example, if you’re loading 20 files a month, lights-on automation is just fine, but if you’re loading 200+ files a month, you will need lights-out automation.

What it takes to achieve lights-out automation with AI

This framework applies to any automation technology, and especially to AI technologies as we know them today. Today’s generative AI can do an amazing job in a lights-on environment, creating summaries, drafts, suggestions, and even ideas for humans to review, revise, and take action on. AI cannot yet perform lights-out automation without help from complementary automation technologies.?

This is why many AI products are called “co-pilot”: because today’s AI technology, especially the language models, have been optimized to provide a human-like response rather than to perform tasks precisely and predictably. It is not a deterministic technology. Ironically, its weakness in automation applications is precisely because it is too human-like. More specifically:

• It cannot produce predictable and repeatable results. You can run the same prompt 10 times and get 5 different results phrased 10 different ways.
• It is hard to get precision in its response. Generative AI is good at generating freeform response and even mimicking styles, but it’s difficult to make it follow precise instructions on how to construct the output in a reliable manner.
• It is sensitive to prompt construction. You need highly engineered prompts to achieve precise and repeatable results. This is why Prompt Engineer is a highly paid job now and it’s still more art than science..
• It hallucinates. It is very difficult to prevent AI hallucination even when the temperature is turned down to zero.
• It tries too hard. Beyond hallucination, AI really wants to perform the task it’s given, even if it means ignoring your explicit instructions about what not to do. Beyond the explicit policies built in by its trainer, AI often chooses to give a wrong response instead of not completing the request. Research even shows attempts to discipline AI for not following direction will even lead to lying.
• It doesn’t follow all instructions all the time. AI will randomly not follow certain parts of your instructions no matter how explicit you are and how many examples you provide in the prompt.
• It can be sloppy. Sometimes the response is just not clean, and you get random code or other text that doesn't belong.

Due to all these shortcomings, achieving lights-out automation with AI today means you will have to couple it with non-AI, deterministic automation technologies in every part of the automation framework:?

• Preparing the input - Get data from enterprise applications and databases. Ensure the data quality is good enough to create quality prompts for the AI.
• Performing the task - Construct the prompt for the tasks to be performed. Specify the format of the output and provide examples and rules on how to handle deviations from the norm.
• Inspection - Ensure the AI has performed the task correctly without deviating from the instructions, its output is in the format specified, and it did not hallucinate.
• Remediation - Fix any inconsistencies and bad response from hallucinations and failure to follow instructions.
• Handling the output - Prepare the AI response and make it usable for downstream technologies. This will include parsing/extraction, transformation, standardization, and correcting technical mistakes.
• Monitoring - AI needs operational monitoring like any other automation technology, but it also needs budgetary and security monitoring. Most AI technologies use token-based pricing, so costs can easily run away since, as the quality of the prompt gets better, it also gets longer and more iterative, especially with the latest reasoning models. An emerging area that also will require diligent monitoring is security breaches from prompt injection. This is a fairly new area of security vulnerability introduced by AI.

Turn AI from a co-pilot to an autopilot

AI is exciting and a valuable tool for RevOps teams. However, its current incarnation makes it better suited for lights-on automation, serving as a co-pilot. It is possible to use AI in lights-out automation, or as an autopilot that only requires light human supervision, but it will need to be used in conjunction with powerful automation technologies that can provide AI the data it needs, verify its work, correct its mistakes, and monitor its performance.